How do we feel about "Should trusted close/reopen voters and reviewers have their votes weighted more than others'?"

Question

On meta, a new information-gathering project about weighted voting for trusted users started that I think might affect us considerably later down the line: Should trusted close/reopen voters and reviewers have their votes weighted more than others'?

For your convenience, I quote the entire discussion prompt and invite you to discuss it here in particular regarding rpg.se.

[the following was posted by Catija, a Community Manager employed by Stack Exchange]

In Q3 2021 I finalized my research around three vote close/reopen and I was pretty happy with the outcomes - as were the sites that got to test it out. While the benefit to each site was different, the most important thing was that there were no negative impacts of that test and all of the sites wanted to keep the reduced numbers of votes to close. We will be continuing to make this change on other sites that have requested it in the near future, so if you're waiting for a 3-vote close/reopen test, it'll be coming soon.

One of the things that came up during that process was a question about expanding the gold badge close/reopen privileges to other users as an addition or alternative to three vote close/reopen. We've discussed it internally and want to do some research before deciding whether to ask the Public Platform team to build a new feature to allow this - unlike 3-vote close, there's nothing already built that will allow a change of this sort.

This question is the second part of that research - I asked a version of this question to the Moderators in November 2021 and am following up here to see what others think, too. Based on their initial feedback, they were concerned about a few things and it didn't seem there was a huge amount of interest in this path forward, at least as proposed below. I've adjusted the description below to answer questions the moderators had and concerns they expressed.

What is the goal?

First things first - this project is purely information gathering as mentioned in the Q4 roadmap. We're trying to see whether it's something that would be worth building at some point in the future and we have no specific timeline related to actually building it. We'll be gathering data and community sentiment before deciding whether to move forward. Additionally, this wouldn't be shipped network-wide if we did build it. Like 3-vote close/reopen, it'd be up to the site to opt-in. That out of the way, let's look at the goal of the feature...

As my research showed, there's a need across the network to make curation tasks easier to complete and require fewer people to participate. Three vote close/reopen is a way to do that and it's also a very even-handed one - essentially, everyone with the privilege to vote to close/reopen gets the same decision weight when it comes to participating in reviewing and voting to close and reopen.

In general, this is fine but it means that we're not necessarily recognizing the specialized knowledge and experience of people who are very involved in the curation process, and I would say that recognition is deserved!

So, in the end, the goal is: Make it easier for engaged curators to close and reopen questions more quickly, for any reason. Additionally, we hope that people who are working to curate the site will feel like their efforts are more valued and that there are fewer close/reopen review tasks needing their attention.

While we can't really hope to make reviewing fun, we can hope to make it feel less futile.

Why are we investigating weighting close votes as a solution?

Up to this point, weighting votes has been limited to the gold badge "duplicate hammer" - which, as its name implies, only has any power over duplicate closures.

There are other things about the gold badge hammer that may not be a good fit for this usage:

• It's only particularly useful on sites that are huge (like SO) or have some very deep interest and expertise in specific tags (like SFF) since getting a gold tag badge is a very high bar to set.
• Having a gold tag badge doesn't necessarily mean you know when to close/reopen posts - that said, having this only relate to duplicates minimizes that risk.
• It only relates to duplicates.
• It's unilateral. This is a pro and a con - for duplicates it's likely good but for other close reasons, it may not be as sensible.

That said, when we initially rolled out this feature, half of the answer announcing it says that we'd consider it for other uses in the future - and that was in 2014! Seems like it's about time.

Beyond this, we think this is a good potential solution for the goal we're trying to achieve. While it's not something we can start implementing tomorrow, since we'd have to get it built into the site, we do have some data and personal experience that we can look at from duplicate closures.

What are the initial feature ideas for this?

I've got some ideas floating around in my head of what I think might make a good feature set to ask for here and that's based on my personal experience and my thoughts about the duplicate hammer. I'm open to being convinced that I'm wrong about any of these points! I'll put them here in a numbered list with my thoughts so that y'all can call them out by number in your answers:

1. Increase the close/reopen vote weight to 2 for users who meet certain criteria - I think that making these unilateral has the risk of making it a bit too easy to close or reopen questions without a lot of oversight.
   • I could also go the route of making it a weight of 2 on sites with 3-vote close/reopen and 3 on sites with 5-vote close/reopen.
2. Unlocked by voting to close/reopen (including reviews) n times + - Since reopening is less common, this would get imbalanced if we relied on each action individually. We don't necessarily want to rely on reviewer badges only as they're not very common. Review stats:
   • Steward badges (1k reviews) - only 88 sites have at least one person with the badge and only 42 have ≥10 †.
   • Reviewer badges (250 reviews) - 132 sites with at least 1 and 81 with ≥10 †.
3. Requires maintaining n% of close/reopen votes & reviews deemed "valid" - Whether someone's reviews were actually "good" and whether someone should be able to robo-review and still get this privilege is a concern to me. I don't think they should.
   • "Good" defined by being validated by other community members (e.g. was the post closed/reopened?) We'd put some work into finding a reasonable way to define "good".
4. Alternative option: Only weight votes cast in review - We could make it so the privilege is only awarded for reviewing and would only weight votes cast in close/reopen reviews.
5. Applies to all close reasons - There's really no reason to restrict this to any specific close reason over the other. Except for duplicates, we don't necessarily want to incentivize people opting for one reason over the others.
   • This would not replace the gold badge hammer - so, no loss of privilege for those users when it comes to duplicate closures.

Those are the broad-stroke points. There are clearly minutiae, such as determining which close reason "wins" to poke at but those things seem like secondary bits that we can work out once we answer the big question - "Is this idea worth pursuing?"

^{† - These badge counts are queue agnostic, so it's likely an overcount if someone has a Steward or Reviewer badge in LQP instead of Close or Reopen.}

What do I need from y'all?

Poke at my ideas, question them. Point out things that I've missed or where I'm looking at things wrong.

• Is this concept (even ignoring the specific rules I've outlined above) good/bad/silly?
• Would this be something you think the sites you use would gain value from?
• Is there something different that you think will address the goals we've mentioned?

Don't feel like you have to answer any or all of these questions, they're here as ideas for what sort of feedback I'm looking for.

Answer 1

This is a feature for sites that struggle to get close (or reopen) votes

RPG SE is not that site. In fact this site seems to have one of the strongest moderation cultures across the entire network. Questions get closed very fast, especially if they are clearly off topic. There's not a problem with speed here that needs to be solved.

Like other recent per site features (3 vote close and unpinning the accepted answer), it doesn't make sense for all Stack Exchange sites to have it. That's fine.

(In fact I don't believe 3 vote closure ever had a meta discussion here. It didn't need to, because it's not necessary for this site either.)

You should evaluate weighted close votes in the context of other SE sites that you use, where it often takes longer to close questions unless a moderator votes. On some sites, votes even expire because too few people close/reopen vote.

But the bottom line is that RPG SE probably won't use this feature.

Answer 2

At this time, I don't feel like the stakes are high enough to warrant a robust analysis of the data, since SE isn't forcing this upon us; so I will leave a few thoughts based on my experience as one of the most active participants in site curation (though I'm sure my experience would be supported by the data if my SQL-Fu were anything special).

I very much agree with Laurel and Trish that this is definitely not a change we need at this time. When it is abundantly clear that a question ought to be closed, it is closed quickly, even without the involvement of the moderators. Based on JNat's annual analysis, around 10% (113/1135) of question closures in 2021 had the final vote cast by a moderator. To put that into perspective, I want to compare to another stack I read frequently: Fitness.SE. In 2021, 93% (199/214) of closures had a moderator as the last vote. Fitness would benefit from a system like the one proposed because they are already relying on users with weighted votes to handle closures. We are not. Our moderators could go completely AFK and the only people that would notice would be the ones who flag comments. Our question closure and reopen activities are moving along just fine without help from the users that already have weighted votes.

However, sometimes it does take a while for question to get closed. It's never when closure is the obvious choice. It is when there is room for us to disagree. Like most stacks, we have policies and guidance unique to our corner of the network, and different users interpret these things differently. So sometimes, we have a question that skirts the line of one of our stack-specific policies, or may be off-topic for one of SE's vanilla reasons if you read it a certain way. Just check out the specific-question tag here on meta: many of these question were asked here because a close-review cycle happened, another one has begun, and comments were insufficient to resolve the disagreement quickly. This is when it can take a while for closure to be resolved - either we go through multiple cycles and then discuss on meta, or enough people aren't sure either way and it does take a little while for enough people to form an opinion and flip the vote.

So, because we already have a thriving and active culture of curation and moderation, I think it is really important that everyone have the same vote weight (dupehammers notwithstanding). There is no evidence that any of our reviewers are acting in bad faith, and we don't have enough volume for "robo-reviewing" to be a thing. If you've participated enough to earn close vote privileges, my 90000 rep doesn't make my close vote any more valuable than yours – as long as we're all just trying our best to faithfully apply the guidance and policies we have (which there is no evidence that anyone is doing anything less than that), we should all have the same vote weight.

Answer 3

Currently, it's not needed here

On this site, there are several dozen people that vote to close at all times of the day. I have rarely seen a question staying in the Vote-To-Close state for more than a few hours before it was either closed, or kept open.

This is not 3D-Printing SE, where about 6 people cast closure votes at all, and 3 of those are mods. We don't need to make moderation faster in our current state.