# A Modest Proposal for answers that aren't backed up

This has happened many times, but I'm going to focus on the most recent question where answers have flooded in that aren't backed up but are highly upvoted.

If we do, in fact, want people to support their answers, then it feels like we need to do something more active when this happens.

As it stands, we are punishing the querent by closing their question because the answers aren't abiding by our policies. The problem is, those answers are still being positively reinforced by upvotes and rep generation.

The unsupported answer isn't being negatively reinforced, it's being rewarded. And if we really do want people to follow this policy, then it feels like we should do something.

Even if an answer is highly upvoted, should unsupported answers be removed? Or have their rep removed? My concern is that by leaving them up, we are not only positively reinforcing that type of answer for whomever submitted it, but it's also acting as an acceptable guidepost for those reading it and it can perpetuate that activity.

I'm not asking this lightly. I'm sure I've done this myself in the past (although I'm trying to only answer with cited experience now for good subjective questions), but it seems like that if we do want to curb this and the community won't do it through votes, something larger may need to be done.

• I did notice multiple answers on there claiming "just do this, it'll work" followed by several requests for sources or experience. I felt frustrated, from an aswerer's perspective, that so many people were answering without seeming to have (or at least cite) applicable experience – G. Moylan May 30 '19 at 14:51
• @G.Moylan How are you inferring that an answer poster does not have experience? The lack of a citation seems like weak evidence that an poster lacks experience. – GcL May 30 '19 at 14:54
• Did the question in question get HNQ-blocked? I don't remember seeing it come up on the HNQ feed in chat - I shudder a little at what the state of voting might've been if the current distribution is, in fact, what happens without the HNQ traffic it would have inevitably had. – Carcer May 30 '19 at 14:54
• @Carcer This happens regardless of HNQ or not. While HNQ may exacerbate the problem, it's still a problem. – NautArch May 30 '19 at 14:54
• @GcL probably by the way that none of the answerers so asked updated their answers to provide evidence of experience or explicitly said that they didn't actually have personal experience with the problem (in comments or in the answer itself) – Carcer May 30 '19 at 14:55
• @GcL Lacking cited experience is exactly the problem. We don't know that there's experience. Saying "i've got experience" isn't the same thing as discussing that experience. THe point is that answers need to be supported, and if you don't have the support, then they're unsupported. – NautArch May 30 '19 at 14:55
• – Sdjz May 30 '19 at 15:01
• @GcL that's why they're asked to talk about that experience and how they dealt with it/what happened for them, not just to say "I have experience". – Carcer May 30 '19 at 15:02
• @GcL Yes, that’s useless and not what’s being asked of answer writers. The point of the Back It Up rule is that we want people to talk about their experience, because that’s actually more valuable than whatever advice they give. A story about “I did X and Y outcome happened” is miles more valuable than “I have experience, which I will not bother sharing, and I think you should do X.” – SevenSidedDie May 30 '19 at 15:05
• @GcL The point is, we don't do "just good advice" here. Answers should be supported, and if not, they should either be not voted on or downvoted. Advice is basically idea generation. – NautArch May 30 '19 at 15:06
• @GcL I'm not sure it is. Unsupported advice seems awfully similar to "here's an idea I've got". – NautArch May 30 '19 at 15:10
• NautArch @GcL Might I suggest taking this to a chat room or into an answer? This is getting way too lengthy, even for Meta comments. – Rubiksmoose May 30 '19 at 15:11
• @GcL that's because you can trust an electrician is qualified (I assume, in whichever country you live in) because there are qualifying bodies that do the hard work of testing their knowledge and experience for you. Posting an answer on Stack Exchange does not have such a high barrier to entry. I also find it unreasonable that someone might have so much experience that they are unable to identify specific anecdotes of relevance to the situation, and part of what makes an answerer a good answerer is skill at condensing down to the relevant information. – Carcer May 30 '19 at 15:11
• Please continue debates about the content of the question in this chatroom instead of the comments. It has gotten far far too long to be coherent or productive and is clogging up the comments' intended purpose. – Rubiksmoose May 30 '19 at 15:27
• Just as a heads up, the moderators are almost certainly not going to do hard intervention deleting answers all over the site to resolve this. We put up with enough flak even on the small number of occasions where this is totally necessary. That and there's only four of us — we don't have the capacity to manage this. We're just here to be exception handlers and cannot moderate every post. The community itself needs to lead the charge on this and collectively set the standards it wants. – doppelgreener May 30 '19 at 19:17

I understand, I really do. I wish popularity contests didn’t break out occasionally and stir up trouble, and I wish voters would leave them to die down faster after a close.

By and large, the activity does drop off though. And it doesn’t really encourage asking that kind of question on purpose. Answering it might be encouraged a bit, but those who don’t practice the skill of backing up their answers are simply letting others surpass them as writers of quality answers.

But most importantly, the cure is worse for the community than the disease.

## Policing answers is awful, dirty work for mods, and often is worse for the site than some undeserved rep. On balance, it’s better to just let voters be a little irresponsible sometimes.

Having done this duty before, it sucks.

The situation: A popular question, with lots of activity, and many answers that people like.

To police that, mods have to do a close read of word counts in the tens of thousands, plus comments and possibly revision histories.

For each post, make a difficult judgement as fairly as possible on whether posts are meeting the GS/BS standard. Then decide what to do: nothing, cite-banner, comment only, banner + comment, delete, or deleted + comment. Then if a comment is involved, compose a quality comment that could lead to a positive outcome.

All while considering that passions and personalities are in play. And the more the page has exploded, the greater that factor is.

Then get ready to deal with the backlash that’s inevitable when popular posts get moderated.

Among the backlash too is always a small but numerically significant fraction of active users who experience this so badly that it’s reason to resent the mods involved, and that’s long-term pain for the site.

The entire process is exhausting and a minefield.

Doing all that is rough and draining at the best of times: when doing it to try avoiding closing a question as obviously drawing primarily unsupported opinions.

Doing all that after a question gets a Primarily Opinion-based close is just as dirty and has much smaller benefit.

## Even popular closed questions aren’t huge rep generators

I have a lot of answers and have answered a few in my time that in hindsight I should have not (because it was destined to be closed) or that I disagreed with the close. And they’re not really big rep generators after the initial activity.

Closed questions just don’t get the attention and activity of the same question that’s open. It bumps far less, inviting fewer voters.

## On balance, close then ignore is better for the site and community

Some undeserved rep is a small price to pay to avoid the drain of community goodwill and moderator energy.

• You know, your reasonableness is just unreasonable :) But I agree with a lot of what you're saying - and maybe a solution is for users to more directly comment on those answers and not jsut say "please back this up" but maybe to suggest to others "This answer has no support and does not follow site policy. It should not be upvoted." – NautArch May 30 '19 at 17:03
• @NautArch I just realized that the flipping the comment around might work as well. Saying "I really like the use of experience in this answer because it helped me see X" or something of the sort could act as a form of positive support and help make it more apparent why such answers are valuable. – Rubiksmoose May 31 '19 at 13:01
• @Rubiksmoose Absolutely! We need to strongly positively reinforce the good answers and strongly negatively reinforce the unsupported ones. Multiple comments on each :) – NautArch May 31 '19 at 15:23
• Just going to add that placing a comment like "This answer does not provide support and should not be upvoted." is positive punishment, not negative reinforcement. :) – Jason_c_o Jun 7 '19 at 6:20
• @NautArch I think it's fine. It's what the voting system/comments are for. I was just being a pedant. Adding an unpleasant stimuli (a critical comment) is positive punishment. Negative reinforcement is removing an unpleasant stimuli. – Jason_c_o Jun 8 '19 at 19:04
• I was considering adding an answer along the lines of "could mods clean up such answers" (this was before I'd read any answers and only seen the question), but your answer has reminded me of the dreaded [designer-reasons] questions and what a nightmare for mods that was (such as mxyzplk's efforts on my own infamous designer-reasons question), having to delete all of the answers that didn't have a tweet or book reference in it. I can see that getting mods to weigh in with their mod powers is not worth it... – NathanS Jun 12 '19 at 14:24

# Some Suggestions for Users

I don't have strong opinions on how, if at all, things should be changed on the Moderation side of things, so I'm not going to opine on that.

Instead, I'm going to make some suggestions for what we, as engaged members of the community, can do.

## More Aggressive Downvoting

I can't necessarily speak for other users, but speaking for myself, I'm often hesitant to issue downvotes to answers that violate the Good Subjective policy if it seems like they've been written in Good Faith. I don't like issuing downvotes unless the answer seems obviously incorrect or poorly thought out.

There's a couple reasons, though, why I think this instinct—being reluctant to issue downvotes to such answers—is a bad instinct:

• Pile-On Effects: Like it or not, a lot of highly voted answers (both positively and negatively) seem to accrue a lot of those votes from a sense of "supporting the consensus", where users will see that a lot of other users have voted a particular way on a specific answer and vote similarly to reinforce the trend. If an answer picks up a few upvotes before users begin to proactively vote to support the Good Subjective policy, the run-away effect of users promoting the consensus is probably already too powerful to overcome.
• Signal to Noise: A downvote or two issued when an answer is still at a low score tends to engender far more substantial response from the poster than when the answer is up at a high positive score. In addition to trying to mitigate the aforementioned Pile-On Effects, delaying the downvotes means changing the score delta from +18 to +98, which means the user is far less likely to take seriously the idea that there may be a critical issue with their answer.

For these reasons, I'm planning on being more aggressive with downvoting answers to such questions that don't clearly demonstrate that their response is coming from a position of experience with the issue the user is facing—and I would advise other users who approach these questions from a similar perspective to me to do the same.

## Proactive prompts for Users

Whenever I see a question that registers as clearly needing experience-driven answers, I try to show up in the comments and remind other users about the need to focus on such answers. As I did for the question linked by NautArch—after several answers had already come in.

My suggestion here is that we should all be making sure to "make sure the obvious doesn't go unstated", as can sometimes be the case: if an answer is in clear need of expertise-citation, let's be proactive in pointing that out so that people who are tempted to provide answers are already aware as such.

I said that I didn't have suggestions for changes to Moderation, but if I did have such a suggestion, I would probably be arguing that what would be really valuable would be a way to tag, formally or informally, a question to show a prompt whenever a user attempted to provide an answer. This prompt would affirmatively declare the expectation that answers to the question try to focus on experience, not just on Armchair "Here's what I would do!" responses. But like I said, I don't have such suggestions, so I won't mention that.

I would also suggest that we leave such a comment if we see only one additional other comment saying the same thing. A single comment can seem like the whim of an individual user, but if multiple users say the same thing, especially in a short span of time, it can call much more direct attention to the community consensus in a way that the Comment-Upvote number might not. This is mostly a play-it-by-ear suggestion: if the question is already an hour old and already has a comment saying as such, it's probably not necessary, but if, as is often the case, you're halfway through composing that comment and see someone else post it, go ahead and finish + post it anyways, to help reinforce the consensus.

• My concern is that we do often do this - both in question comments as a proactive and in answer comments as reactive. But that doesn't seem to be effective. – NautArch May 30 '19 at 16:20
• @NautArch We do—but my recollection from watching the question unfold over the course of yesterday was that it took us quite some time to do so. I was the first person to post on the question that answers should adhere to Good Subjective, and I wasn't the first engaged user to see the question—and many of the answers went almost half an hour before a user pointed out that they should be experience driven. My answer is not so much a "here's different things we should be doing" so much as "here's the things we are doing that we need to be doing faster." – Xirema May 30 '19 at 16:24
• Ive often done the proactive, but didn't get to it quickly enough because it didn't start off the 5e tag that I generally just watch. But on the ones I've commented for early...I haven't found it's made a difference. – NautArch May 30 '19 at 16:31
• I got aggressively down voted on my answer to the trigger question. Oddly enough, my answer has no mod flag on it. It amuses me to agree with you on advocating more aggressive down voting since you have saved me from writing an answer. (I ended up making an edit to "tone" thanks to NathanS's comment). – KorvinStarmast May 30 '19 at 17:01
• @KorvinStarmast Honestly, I think I had downvoted you at first because your answer when first posted didn't have support. But it does now and got my upvote. – NautArch May 30 '19 at 17:07
• @NautArch fair enough, I was a bit hasty and was getting toward a frame challenge at first, but thankfully Nathan gave me a tip on "tone" that got me to change my mind. But I still need to go and find that link to "is the DM always right" Q&A and maybe link to that also. Pondering .... – KorvinStarmast May 30 '19 at 17:18
• I do really like the idea of more than one user posting a comment. Reinforcement. – NautArch May 30 '19 at 19:13

## What you can do to save subjective questions

First of all - I think most people here understand our Good Subjective requirements, why we have them, and why they're good. See What are the citation expectations of answers on RPG Stack Exchange? for more on that, I'm not going to debate "whether that's right" (it is, for the record), this is about what to do when it's not followed.

## Vote, comment, vote to delete an answer, vote to close the question, edit

Vote them down immediately and comment why, linking the meta guidance as necessary. That should indicate to the poster and other voters what the problem is. Downvotes can be changed to upvotes after an edit, comments can be removed. If that doesn't work, then (rep allowing) vote to delete answers without backup as "not an answer." Deletions aren't permanent either, they can add backup and flag for undeletion. Learning, however, is forever.

Vote to put the question on hold so it can have some edits to make it draw better answers. Usually the OP or someone else editing the question can help when people are just spouting off opinions by adding a "I want specific experience on how you've handled this" or similar, it should go without saying but obviously it's not going without saying. To be honest sometimes this is "the question's fault" in that it's stated as looking for idea generation.

## But when the question becomes more trouble than it's worth...

It gets closed and multiple mods will vote to close it if necessary. Every question isn't sacred, and when it requires huge amounts of curation, it needs to go. The example question linked is at 11 flags, 5 reviews, 26 deleted comments, I have put 7 citation-needed post notices on it, 3 answers are deleted, and to be honest I'm done with it. We have many, many posts and users and if something goes that far off the tracks it's not worth the time of the mods or community trying to make it work any more, as SSD points out in his excellent answer. There's no objective "right" or "wrong" here, it's whether it's just causing too much disruption.

Every bad question we leave around means people will ask more bad questions. Every answer someone just spouts of an opinion that isn't challenged means they'll do it again. The only way to handle this is the community handling it - by the time a question like this gets to the mods, it's down to some blunt choices. If the community hasn't managed to get it in hand, we'll do some warnings and closes to be nice but eventually it may have to just get locked as "not a good on topic question for the site."

• This was my concern entirely - we leave bad answers up that are upvoted is essentially promoting that they are good answers. What do you think of this idea for how to approach comments? – NautArch May 30 '19 at 17:33
• Ah, yes, quite, edited. – mxyzplk May 30 '19 at 22:24
• We can definitely have a pro forma comment, and I think we were trying to get a more customized post notice in too but we need SE Central to implement it. (They can be identical.) – mxyzplk May 30 '19 at 22:24
• What would a pro forma comment look like? How would it work? – NautArch Jun 3 '19 at 14:10
• It would just be a standard blurb that's nice and informative about "We prefer answers that back themselves up with experience or citations following our site guidelins <link>" etc. There's no magic, it would be something documented here you can just cut and paste instead of having to roll your own each time. – mxyzplk Jun 3 '19 at 14:13

## We should treat these answers the same as those for a fact-based rules questions

If the rule is "include citation" then the answer should at least include some degree of "I have relevant experience because X" or "I've tried Y before and it did/didn't work."

The trick becomes how much "citation" in this case is adequate. As far as removal or manipulation, the rules should still apply the same here as for fact-based posts.

Perhaps we should flag the post as "citation needed" and let the poster fix their post. If it still hasn't been corrected after a certain period of time then it should be removed

• So you are suggesting to change nothing? – Akixkisu May 30 '19 at 15:05
• Doesn't the notice "Some of the information contained in this post requires additional references. Please edit to add citations to reliable sources that support the assertions made here. Unsourced material may be disputed or deleted." qualify as what you are proposing here? – Sdjz May 30 '19 at 15:07
• @Sdjz do you mean in a flag or in a comment on the answer? – G. Moylan May 30 '19 at 15:08
• The tricky part is that users have limited options to remove posts for any reason. High rep users can vote to delete, but only if the post has a negative score and it needs 3 votes to confirm deletion. The vast majority of these problem posts have a positive score. Any user can flag a post for a mod to look at, but currently it seems like the current moderation behavior would just result in "citation needed" banners, not deletion. That is the current situation. Are you proposing some kind of change to this or what specifically? – Rubiksmoose May 30 '19 at 15:09
• Are you saying to flag for mods, and if no action taken after a certain time, then mods remove? – NautArch May 30 '19 at 15:09
• If you scroll to the bottom of some of the answers in that question you will see the notice I mention: here is an example of such a notice in an answer. I believe this is something that moderators are able to add to posts – Sdjz May 30 '19 at 15:09
• I clarified. Or tried to. – G. Moylan May 30 '19 at 15:12
• The title was confusing, as it made it take work to figure out what you were advocating, so I switched it from a negative rhetorical question to a statement of what we ought to do. Feel free to revert. (But I think this is much clearer.) – SevenSidedDie May 30 '19 at 15:18

Leave the questions open.

The closure process is broken. It's just especially noticeable here. In theory, the process works by closing bad questions, so that they can be improved, so they can get good answers. In reality, bad questions get answered badly, then closed, then never updated because they already have whatever textual diarrhea squeaked through before closure. Worse, anyone who shot from the hip quick enough can then edit their answer, essentially giving the worst answerers a unique license to answer closed questions.

Not trying to pick on @korvinstarmast or say that his answer is bad, but you can see how he displayed this exact behavior.

So, despite the design intent, we have a system implementation which privileges and rewards bad answers. Furthermore, that they get answers (even bad ones) appears to dis-incentivize querents from fixing their questions.

If closed questions hid all answers (or even hid them from users below some rep threshold), I think this system would work as designed. As far as I understand, that would take an actual change in site behavior, which I see as out of scope.

Instead, I suggest that we stop making the problem worse. Leaving bad questions open will continue inviting bad answers, but at least it would allow good answers too. We will inevitably get bad answers, let's at least stop foreclosing good ones. Leaving questions open will let good answers drive out the bad ones.

Even if we don't want to just ignore bad questions, let's at least close them before there are any answers; once those bad-subjectives slither in, it's too late.

(Note that all of this applies no matter how you define "good" or "bad" answers.)

• Your chastisement is accepted. ;) The initial rash of down votes were perceived by me as based on advocacy of the DM being in charge. What got me to edit it was a comment from NathanS on tone. My answer did not attract the mod banners. I then thought of writing a totally different answer, that was based solely on avoiding conflict, but ended up having to fold it into the answer that initially attracted down votes. And yes, you are absolutely right: fastest gun in the west symptom is an SE feature. – KorvinStarmast May 30 '19 at 21:32
• @KorvinStarmast sorry! I didn't mean that your answer is bad. But when I saw your comment describing how you circumvented the system, it was too perfect not to mention. – fectin May 30 '19 at 21:50
• Heh, no worries, I kept trying to improve the answer as my brain got better engaged. If that's "circumventing the system" OK. I do see the point that you are making. 8^) – KorvinStarmast May 30 '19 at 22:28

# Let them stand

If upvotes can be considered proxies for "I like this answer", "I found this answer useful", or "I agree this is corrected", then removing the answers would be removing well liked, useful, or correct answers.

• my obvious concern with the status quo is that if we actually want people to follow our good subjective guidelines, then how does letting them stand do that? Can you talk about why an answer like that (regardless if others 'like it') is good for the community and our policies? – NautArch May 30 '19 at 14:53
• I don't see removing answers that the community have vote up as serving the community. – GcL May 30 '19 at 14:56
• Do you have a counterproposal for how to get people to follow? Because the status quo is reinforcing bad behavior. Unless you're saying we shouldn't have the policy or that we should allow this positive reinforcement for those who don't follow it. – NautArch May 30 '19 at 14:56
• @NautArch I don't view upvoting answers because they are useful, liked, or seem to be the correct answer as a bad behavior. – GcL May 30 '19 at 15:02
• Okay, just confirming then that you don't think the citation policy should apply - which it seems like you believe as long as the answer gets upvotes. – NautArch May 30 '19 at 15:04
• I don't mind the citation policy. I would let answers that are viewed as useful, liked, or agreed with based on the community interactions with them stand. A policy that gets rid of useful, well liked, or agreed upon answers seems questionable to me. – GcL May 30 '19 at 15:11
• If the policy can be ignored, why do we have it? – NautArch May 30 '19 at 15:11
• Note that the Back It Up rule exists to allow subjective topics like non-rule questions. It exists so that this site can exist at all. Popularity-based voting is different from correctness-based voting and is bad here. You’re advocating popularity voting. If you’re unfamiliar with the origin and fundamental need for the rule, or with why lots of upvotes on answers not following it is an existential problem for sites with topics like ours, you should read the foundational document allowing sites like RPG to be created. – SevenSidedDie May 30 '19 at 15:12
• @SevenSidedDie Who determines what is correct in "correctness-based voting" ? If it's a panel of users that sit in an off-white cylinder, perhaps change the requirement for submitting answers. A peer review or review queue might be more appropriate for ensuring correctness or blessedness. – GcL May 30 '19 at 15:17
• @NautArch A policy that can be ignored sounds like a guideline. I like guidelines. – GcL May 30 '19 at 15:18
• @GcL You’re talking about rewriting the entire network’s fundamental usage now. That’s not on the table. As for who decides correctness: voters. How that works is complex and you seem to be misunderstanding a large number of things. It sounds like you’re familiar with how to use the mechanics of the site, but unfamiliar with the reason the mechanics work the way they do. So I’m not sure what educational advice to give. Also, that was too fast to have actually read and considered the link, so consider doing that if you’re interested in having useful knowledge for participating in meta. – SevenSidedDie May 30 '19 at 15:24
• @SevenSidedDie I've read the blog post previously as well as the policy being addressed here. – GcL May 30 '19 at 15:26
• @GcL You don’t mention in your answer why the keystone of the site’s existence should be removed, but that’s what you’re advocating. How does the site stay open if we embrace popularity-based voting? – SevenSidedDie May 30 '19 at 15:50
• To clarify: I don’t disagree with the “leave them be” headline, but with the reason. The reason given is antithetical to the way the site works. – SevenSidedDie May 30 '19 at 16:35
• FWIW, I think that "useful" and "correct" answers don't seem to be the issue, but "well liked" but opinion based/not backed up answers are. It's hard to argue against a correct answer; usefulness varies with the reader. Example on that bit: I found none of the answers (beyond my own) useful to the trigger question until guildsbounty got to post his; I already have my own solution to players like this but guildsbounty opened me to a different angle on that). The "well liked" but "not backed up" is the core GS/BS issue (and it's not just an issue at RPG.SE. ) – KorvinStarmast May 31 '19 at 14:42