We need to talk about late votes to “policy” metas

Question

Let me start by trying to give a definition. We’re here talking about meta discussions which strongly guide or dictate behaviour, and in particular where there are opposing, incompatible options and – by nature of the problem we’re talking about here – disagreement about which to go with. For lack of a better term, let’s call them “policies”.

We decide which option to go with based on vote consensus (the answer with the highest overall score once stabilised). However, it is still possible to vote on those posts, which would make it possible for the votes to change. Unless there exist old (and forgotten) examples, this has happened exactly twice – both in relation to “Don’t Guess the System”.

There are a number of issues or concerns which exists around this, and which need to be addressed and/or resolved:

• The “losing” side would be much more likely to be directed to that meta and vote. That may include any number of visiting users who do not take part in the curation and activity that the policy exists for. That makes the new scores unreliable for representing the community consensus, meaning it can't be taken as a new result without a full revisit.

• The resulting post stops working as an efficient signpost for the policy, as votes no longer reflect current praxis.

• We can’t have consensus continuously update according to which answer has the highest score at any given time. Neither having the policy flip-flop between solutions, nor expecting users to check every time they would take a relevant action, is a feasible solution. While there is no guarantee that would happen, I don’t think any of us would want to have such a lingering burden. Closure is important.

• We don’t want to (by principle) automatically start a revisit, as it would rapidly exhaust at least one side of the argument, which isn’t workable.

That last point in particular brings me back to the two cases we’ve had. First was the “5e”-specific clarification to DGtS. Once the change in votes became known (to me), I made the call to revisit that discussion. Enough time, discussion, and other changes had happened since the original that it seemed warranted. And the reception to that was quite good, and the consensus startlingly strong, given how divisive the topic has generally been.

When the same changeover in votes happened to the most recent DGtS revisit, it was not revisited, opting instead for stability (in terms of acting policy). This was largely due to the relatively short period of time that had passed since that re-visit and a general feel that the community was tired of discussing it for the time being. I think that’s part of why this discussion is so late to be started. For it being late, I can only apologize.

Now, with the major problems put forth, let’s start working towards solutions. To that end, let’s keep this discussion to the process. That is, please don’t expend energy here discussing the DGtS policy itself, though the discussion will inevitably include as an example, and apply to, the DGtS policy. We work best with specific examples, but let’s leave DGtS as only an example and not the actual meat of the discussion.

With all that out of the way:

How do we want to handle “policy” meta discussions when the votes change later on?

Answer 1

I believe changes to scores in policy metas should be observed as reflecting the ongoing, developing will of the community. Leaving these metas open and unlocked is useful for gauging that will, and I believe it's important for telling us when we need to take another look at something and have another discussion about it.

I'll get to a generalised proposal in the very last section.

First though, I need to address those concerns that were shared in the bullet points in this meta question. We were asked not to discuss the “Don't guess the system” policy here, but it's materially relevant to those concerns, and to respond to them I have to describe what's concretely happening in our community around that policy. So, I'm going to be talking about that stuff.

The assertion about new scores being unreliable disenfranchises voters

The “losing” side would be much more likely to be directed to that meta and vote. That may include any number of visiting users who do not take part in the curation and activity that the policy exists for. That makes the new scores unreliable for representing the community consensus, [meaning it can't be taken as a new result without a full revisit.]

I don't agree with this as given, with the exception of the last bit I put in square brackets. To the bit in square bracketes: yes, I believe a full revisit is appropriate when this occurs; I'll get to that in my last section.

But on the bit outside the square brackets—there is an enormous claim here that's unsubstantiated: that the “losing” side in specific is disproportionately affecting things, and that this makes the votes unreliable.

Supposing that there's a “losing side magnet” of some kind and that any votes attracted by that magnet don't count is a big assumption to make. It's a significant claim that disenfranchises community voters from having their say, and needs to be backed up. The votes didn't just ... come out of nowhere. They came out of real people having their say on the policy, and alleging their votes don't count just because they disagree with the position we initially took out of that meta is not appropriate.

Dear reader, if you share this concern, let me be clear I don't believe you'd necessarily make this claim in bad faith. I simply want to express the severity of this assertion and its implications that you may not have considered. I think it's fairly reliable that you'd be acting in good faith, dear reader, so I believe you'll take into serious consideration what I'm saying here—that dismissing voters is a big deal and cannot be done based on speculation.

The argument “oh, the data's not reliable, because {reason I decided here}” is readily available as a reason to disenfranchise any group, and we've seen it before, and it sucks. Back when Game Recommendations were banned, the collective community uproar was completely ignored by the diamond moderation team we had at the time. The diamond moderators ignored us specifically due to who we hitched our wagon to (here and here) despite the person to whom our wagon was hitched making a reasonable, heavily supported complaint. Further, individuals like me who spoke up about our upset were ignored because we were upset—the mods flipped the bozo bit on us (which means: deciding in perpetuity that someone's wrong because of who they are, not what they said). As that complaint lays out in point 3, the mods operated then on a basis of “anyone who's upset about this is one of those five, oops I mean six, oh I guess it's seven, always-upset people”. The diamond moderators disenfranchising people directly lead to such harm that it took the community several years to recover.

Although I don't believe anyone here is coming from the same place, I am worried that this concern may also come from flipping the bozo bit on people voting against the policy—we need to avoid making the same mistake of deciding “oh, the community input can be ignored here because of reasons that are not substantiated” again on a different basis.

The fact is, the community is in charge of determining the course of how we operate on the site. The diamond moderators are not, vocal individuals are not. The voting meta community is. The voting meta community is making it very clear what they want. It's not appropriate for any group to say “oh, well, that other group doesn't count actually because reasons.”

I believe instead it's significant that people are responding with discontent to the policy. This isn't an aberration—it's not like there's been any shortage of people expressing discontentment with our “don't guess the system” policy. By any measure at all (score, upvotes, or downvotes), popular support is now fully behind repealing the policy, and in fact the score is within 1-2 votes of being double the other policy's score:

• “Revert the policy, and treat these edits like any other” is at 44 points, +67 upvotes, -23 downvotes
• “I think it is a terrible idea to revert the policy” is at 23 points, +58 upvotes, -35 downvotes

It's my personal speculation that this change in scores over time reflects the community trying and testing the statements in both and figuring out that it's not actually going to be all that bad if the policy was repealed. But overall, it doesn't matter what my speculation is. It matters that people are voting and have expressed their stance.

Lastly, I don't believe we do have any evidence to substantiate that there's some kind of “losing side magnet” anyway. We haven't seen previous topics automatically attract votes in favour of the “losing” side just because it lost. We've seen votes side in favor of general community consensus, which has, in general, been the “winning” side.

The state of the meta has been reflected just fine

The resulting post stops working as an efficient signpost for the policy, as votes no longer reflect current praxis.

Re-revisiting the "don't guess the system" policy this paragraph at the end, and in fact this is why we added it:

After 3 months, the community's highest voted answer is sitting at +54/-22 and advocates for not changing the policy according to the moderators' proposal, which ranks below it in score at +47/-22. As the community failed to achieve a consensus to change the status quo, we will continue to enforce the policy as we have in the past. Separate refinements made notwithstanding.

I was still on the diamond team when this meta was developing, and I remember us discussing doing this eventually to record how things unfolded specifically because votes might change. (Rubiksmoose eventually added this paragraph a month after I left the team.) We discussed whether it would be more appropriate to add this paragraph or post a new meta, since both are reasonable options with precedent, and if my memory serves, we settled on adding this paragraph lest the new meta be seen as a place to re-contest the issue.

We're not witnessing a flip-flop, we're witnessing a community trend over a year in the making

We can’t have consensus continuously update according to which answer has the highest score at any given time. Neither having the policy flip-flop between solutions, nor expecting users to check every time they would take a relevant action, is a feasible solution. While there is no guarantee that would happen, I don’t think any of us would want to have such a lingering burden. Closure is important.

This is not what's happening though. The policy has not been flip-flopping. When votes on Re-revisiting the "don't guess the system" policy were neck and neck and it was not clear which one would prevail, we held off on determining our course forward until it was very clear one was the prevailing answer we should go with.

Now, there is a very clear and unambiguous sentiment that the community's will is changing. The trend has continued consistently. There is a noticeable gap. The scores are more unambiguous than when we decided what to go with in the first place. There is no flip-flopping like it's just going to happen every day—there's a clear answer emerging from the community, and it's different to last time, but it's still the answer emerging.

Arguments about exhaustion don't hold water

We don’t want to (by principle) automatically start a revisit, as it would rapidly exhaust at least one side of the argument, which isn’t workable.

I don't believe this to be a valid reason not to have a revisit. I don't think this is even true.

I believe it's an invalid claim to suggest that a side can be “exhausted”, as if the majority side can somehow get real tuckered out from having to write a meta justification every now and then, or as if all the words have run out and can't be re-used. If they're the majority, if they're right, if they have popular support, it'll be easy enough to explain why and get that support.

When we decided to go forward with asking Time to retire the [rules-as-written] tag? even though one user was suspended and unable to participate, the moderator team's position was that we needed to be able to discuss this anyway, and if not one other person could mount an argument in favor of keeping the tag, then what's that really indicate?

Coming from that—are we really suggesting that an entire half of the active community might be unable to successfully mount a position on the policy, and that's reason enough to not revisit the topic? Or that if we revisited it and nobody mounted a certain position, it'd mean the whole discussion's invalidated?

I've seen only one individual make repeated serious claims about exhaustion. I'll leave them unnamed for their benefit, but specifically, their expressed claim has been that one side is trying to win by exhausting their opposition.† That's an allegation that an entire side is acting in bad faith, which I don't think is fair in the least. Worse yet, it's an implication that any change of support in subsequent assessments is just “exhaustion” of one side rather than a change in community will.

(† I'd quote it, but again, I want the user to be able to remain anonymous. Diamond mods, if you think this is an unfair summary, please correct it or request that I do so. If I'm lying, may I be struck down for it—but I can link to this assertion being stated plainly.)

Neither side got exhausted in the last revisit. We just discussed things and saw where we were at. Neither side will be exhausted in the next revisit. It's been more than a year—I think this is completely reasonable to do one again.

I believe arguments of exhaustion carry the indication that once we've had the discussion once, the community will is set in stone indefinitely and even if the majority are now disagreeing, we can't change anything. That's counterproductive to a meaningful community democracy—if consensus changes, we should be able to have that reflected in how we operate. Proponents of the status quo, if they have support, should have nothing to fear on things being revisited.

Support for one side drying up is not an indication that the side is somehow exhausted or that the other has been on some campaign to exhaust them. Instead, it's just indication that support has dried up.

So what do we do?

I think it's actually useful for us to leave metas like this open and let people vote.

Recording the state of votes: We don't need to lock them to reflect where votes were at, since we already have options for this already:

• We can update the meta to specify how we decided on things. We did that in the last paragraph of Re-revisiting the "don't guess the system" policy: three months in, Rubiksmoose updated it to clearly say what the community will had settled on and what the votes were, making that information available even as votes changed.
• We can also post a follow-up meta confirming what's happened. This was done with game recommendations: we had a discussion, then ten days later mods made a post clarifying the new situation. The title for that one says “revisited”, but that's a reference to the four-year-prior policy post, not a reference to revisiting the discussion.

As we leave the meta open, votes will continue to come in, and that's fine. A great deal of our metas are about us deciding how to do things (and I don't mean policy posts, I mean everyday discussions of “what do we do here?” at various scales) and those don't get locked either.

If community sentiment remains in tune with that meta, well, I believe scores will continue to reflect that, as they have just about always. If community sentiment does change, that's something to pay attention to and something to cue that we should have another discussion on that matter to see where we're at.

We don't need to be afraid of revisits happening constantly, or constant flip-flops. Suppose we have a meta and decide something, and we're afraid of having a revisit after just one month—that would mean that the majority case lost support in just one month! We surely wouldn't even go with that case to begin with, or we've obviously discovered major opposition to it or that it doesn't work, and that's worth discussing.

In this concrete case, it took an entire year for the “don't guess the system” policy to reach this point. Heck, it took about eight months for the two positions to achieve parity again (Akixkisu notes in this answer that the votes were about even, with “repeal” ahead by just a vote or two, in November 2020 when this meta was posted.) I believe we should not be afraid of re-assessing something after this long, especially not after the response changes so significantly, and I don't believe we have reason to be afraid it will happen more often than this.

I think locking metas is the wrong call. Those votes coming in after we decided which direction to go are useful to have. Let me ask you, if we're concerned of having revisits too often: how do we know when to have them, if we don't have a metric like this? Those votes tell us when it's time for a revisit or if no revisit is needed. We only had the March 2020 re-revisit because circumstances seemed to be diffrent from what the September 2018 revisit predicted—that we'd have tons of edition confusion again when Pathfinder 2e came out. We're only talking about this now because the previous majority answer has lost support by such a significant margin.

A full revisit is probably preferable to just immediately changing policy. A change in will being expressed by voters indicates it's time to have a new look at the topic. This could mean the now-popular stance should be immediately adopted, but it's possible instead we're ready for a third option, one that reflects more clearly where we're at now—and we can only find that out through discussion with each other. The possibility of discovering that suggests we should probably prefer having a new revisit discussion. Maybe there will be situations in the future where we should just immediately change policy without a new discussion, but I don't think that's any current situation we're dealing with, and we'll know if and when that's needed.

So to summarise:

• Leave metas open after discussing them. Update the question or post a follow-up meta confirming the outcome at the time the community will seems clear.
• If a position on meta loses majority support, revisit them. The evidence we have at the moment suggests this takes a long time to happen, so we don't need to be concerned that this might happen constantly on any given issue.
• Put aside “exhaustion” arguments. They don't have merit and are undemocratic. Sides don't get exhausted. People can be exhausted, but anyone can be exhausted—they can take a break and the rest of the community can still act.
• Maintain that votes matter, and don't disenfranchise any group for any reason. Instead, take votes seriously and ensure the community has their say.

Answer 2

On second thought... is this important?

Don’t Guess the System is almost certainly due for a revisit regardless, and not only is it the only example of this that we have—it’s also just about the only conceivable controversy where this would even be a concern.

We simply don’t have policies as described in the question for much of anything—all of it is either

• network-wide rules (Be Nice, don’t engage in activity that might get the site in legal trouble, e.g. piracy, don’t answer in comments),

• what is on- or off-topic (which has rarely needed to be addressed repeatedly, and when it does everyone has seemed comfortable with “we’ll try this for a while and see how it goes”), or

• extremely non-controversial (i.e. don’t post iterations on homebrew-review questions more frequently than once every 72 hours, which is as far as I can tell our only site-specific policy outside topicality, and no one has ever pushed back on it as far as I know).

I am struggling to find anything else—the relevant Meta question doesn’t list anything else. The alignment discussion, maybe? I’d argue that’s a topicality question, and really just a recognition of where alignment falls on Good Subjective/Bad Subjective, but it’s the only other thing I can think of. Maybe we could or should revisit tool-recommendation—seeing as that never got a proper Meta hearing in the first place, and site consensus on that was never actually determined—but that’s been that way for years and no one seems eager to do so.

So all of these are very unlikely to change—are very unlikely to garner much interest in a change. Many of them we can’t change. Don’t Guess the System isn’t just the only concrete example we have, it’s also, as far as I can tell, the only theoretical example we have.

So what are we even discussing here? If we’re only discussing Don’t Guess the System, we should just discuss that. If we’re discussing unimagined hypothetical future scenarios, we probably shouldn’t. It certainly doesn’t seem pressing or important to do so.

Answer 3

I think this is very much an important subject.

I also think that trying to raise it at this point—when we have only one example of the problem, and it’s this example, and that example is very much currently unresolved—is simply not going to work. The arguments are not going to be appropriately separated from the specific example; the highest-rated existing answer certainly does not. And even if answers are, it’s highly unlikely that votes are. You’ll just see people voting in favor of whatever action they’d like to see taken with respect to the Don’t Guess the System policy—that is, act now, or not.

At this point, it might be better debated as a hypothetical, except hypotheticals are terrible for debates and is unlikely to be productive—just more productive, I think, than this is.

I have no idea what to do the next time it comes up. I don’t, actually, even have confidence that there is a good answer. But I don’t think the framing of the debate here is likely to yield particularly useful insights, and I don’t think we have a “good” alternative.

But at the very least, I think we should probably revisit Don’t Guess the System, settle that (and it does seem likely to get settled at this point), and then have this debate. As things currently stand, I suspect many votes here are going to be stand-in votes about Don’t Guess the System, since there isn’t currently any (active, meaningful) place to put those votes and that is something that (I assume) is nearer and dearer to most users’ hearts, considering the controversy around that policy.

Answer 4

Lock the policy question once the matter is mostly settled.

Locking a question prevents voting on the question and its answers, which would resolve this issue. Diamond mods can simply lock policy questions after voting has stabilized. If somebody really wants to make their voice heard on that issue after the question has been locked, they can put in the effort to ask a new question to re-open the matter for discussion.

We could use a variety of criteria to determine when to lock. A set amount of time after the question is created (1 week, 2 weeks)? A set amount of time after the last modification of the thread (question or answer)? A set amount of time after the last vote on any of the answers (seems ideal but hard to track)? After the question falls off the main page of meta? Some combination of the above?