When do we close a question for drawing opinion-based answers and how can we try to get ahead of such closures?

Question

I'm not talking about opinion-based questions, but the flipside of when a question could generate good subjective answers, but doesn't.

This question does have several parts that are tied together:

1. What criteria is there for determining an answer is opinion-based bad subjective?
2. What do we do with those answers when the community upvotes them?
3. Should we have a harder line for those types of answers in order to stay within our good subjective requirements?
4. Will this help keep those types of questions open or will it drive away users?

It seems that we have a very broad and unclear process in that context. I can't figure out when we decide to close and when we don't. It's clear when an answer isn't supported by good subjective, but it doesn't stop the community from upvoting the good idea generation (or bad), and there are times when questions are closed and others when they are not.

Here are some examples:

Cases where questions remained open even with a large number of bad subjective opinion-based answers

What is the purpose of easy combat scenarios that don't need resource expenditure? - I asked this one and didn't put in the caveat about good subjective. Many answers did come in with experience, but many did not and this question stayed open.

How to design and run cinematic combats that move between zones? - ANother example where some are experienced by answers and others are not. But many many are not, and this was allowed to stay open and is effectively idea generation.

Are there any magical contraceptive birth control options available? - Another case where the vast majority of answers aren't experience-based. Question is remains open.

A Meta about when a question was closed due to a large number of bad subjective opinion-based answers

This meta covers the other side of this where we say we want to close questions when this occurs.

It very much seems like we don't have a process of determining when to close when not to close. I get that we don't want opinion-based answers without experience, but we do seem to allow them a lot which suggest that we do kinda want it (which is fine!) - but it leaves a weird taste in my mouth not knowing when to act and when not to.

Non-experience based idea generation

I think the problem is that many people like the ideas that get generated, even without experience to back them up (they can still be good ideas.) But if we, as a site, are saying that we don't want those answer then we need to decide if that's really true or not and draw a hard line. Currently, the line seems very weak and...opinion-based on when/if we will do something when those answers come up.

The case for the hard line

If we are serious about not wanting bad subjective answers, we need to have a harder line and start flagging those answers for removal.

This is the extreme case of saving the community from itself. If the community upvotes bad subjective answers, then the cycle will always repeat. The only way to end the cycle and promote good subjective is to show that bad subjective isn't something we will tolerate.

The case for a softer line

The problem with the hard line is it's not very nice in appearance and we may lose members because of it. That worries me, because the community does seem to like those answers and I'm not sure what the ultimate effect may be.

Answer 1

We should take a hard line on unsupported answers

I don't like questions drawing large numbers of unsupported answers having to be closed either. But last time we tried the alternate path of deleting unsupported answers, that generated a lot more blowback, so here we are.

We have solid guidance on citation needs for the site now, so I would suggest the community comment linking that guidance, downvote the unsupported answer, and if that does not avail, vote to delete unsupported answers. This will save these questions and ideally teach people how to answer in a Good Subjective, Bad Subjective manner.

But individual cases differ

Some of the questions you link have stayed open because they have at least some experience-based answers, because closing questions is a last resort we usually use when basically none, or none of the high voted, answers are GS/BS supported - it's "just an opinion-fest" in other words.

But some things don't require huge amounts of support (see your other recent question When do homebrew suggestions need to be playtested by an answerer? for a similar situation). If someone posits "Well, you could let divination serve as a birth control method to ask if conception is likely" - do they really need personal experience doing that? No, I don't think so.

• Having said that, the birth control question is certainly on the line given it's drawing increasingly ridiculous idea generation answers.
• The mobile combat question, the top 2 answers say "per my experience" but don't really go into that experience, leading me to not upvote either of them but not downvote either.
• The "easy combat scenarios" one is falling into the usual problem many designer-reasons questions did - there's one or two experience based answers but they're mostly unsubstantiated. But then again some answers may not need extensive citation? "Sandbox playstyles rely on a random assortment of foes above and below your level." Yep. This isn't Wikipedia, we don't need every sentence footnoted.

We as mods don't have time to trawl every question on the site, so we rely on you all to downvote, constructively comment, flag poor answers, edit to improve, and so on to keep things healthy. Not every opinion-fest comes to our attention, we need y'all for that (and ideally to handle it without mod intervention). If it gets out of control, vote to close the question.

In the end, if answers are successfully curated by the community to meet citation criteria, and there's good work on making questions more Good Subjective, then we usually don't need to close questions.

The proof is in the pudding however - a question may seem to be worded objectively but if the answers are an unremitting march of nonsense then it's not achieving our bar of good Q&A. I have questions I've asked on the site where I had to add an extra rider to the question demanding GS/BS answers to pull it over the line. Hot Network Questions can bring in more driveby answers. But be calm, use the tools we have to promote good content and improve poor content, and usually that can head off closure.

Answer 2

Questions getting closed because of runaway opinion-based answers

Things have changed very recently, such that hopefully this won't happen often anymore. To talk about that, I first need to talk about why this happens.

Basically: it's mostly HNQ's fault. Lots of our social and more challenging subjective questions hit HNQ. Sometimes when that happens we just start getting a deluge of crap answers, and see all the answers (including the bad ones) getting too many upvotes and no correction. The people posting these crap answers have been here before, so they have a bit of rep and don't get stopped by Protected status, but they're not familiar with our customs including our citation expectations. HNQ visitors frequently aren't experts on RPGs and don't know our customs, and can't downvote, so they just upvote everything that looks plausibly good.

Now, this doesn't only happen as a result of HNQ, but almost all the time this happens it's because of HNQ and I can't remember a time it didn't.

When that happens that is a bad situation. If the community doesn't or can't stop the cavalcade of crap, sometimes the community or diamond moderators get fed up and close the question. Closing was our only tool for addressing this situation so we used it, much to the disservice of the querent.

As of recent updates to the Hot Network Questions (Updating the Hot Network Questions List - now with a bit more network and a little less "hotness"!) this shouldn't actually really be a thing anymore because moderators can now remove questions from HNQ. Removal is permanent, the question will never return to HNQ after that. There's a question about when we should do so (When should a question be removed from the Hot Network Questions List?) and the only current answer says we should remove a question from HNQ “when HNQ harms the interests of the asker”. That's this situation.

So nowadays, when a question's getting runaway crap answers, instead of closing it, moderators should remove it from the HNQ to stop the tide.

A few days ago we booted our first question off the HNQ: How can I discourage/prevent PCs from using door choke-points?. It's a good example of exactly all of this happening and without this change it was heading for being closed.

Identifying & managing bad answers

1. What criteria is there for determining an answer is opinion-based bad subjective?

Good Subjective is defined in the blog post Good Subjective, Bad Subjective and re-explained in our Subjective citation expectations.

Bad Subjective is all the subjective stuff that isn't Good Subjective. It's when you provide absolutely no experience to back up how your solution works in actual practice. It's also when you break these guidelines:

To apply the Good Subjective, Bad Subjective's guidance on questions to answers as well, a good subjective answer should tend to:

1. Explain “why” and “how”
2. Be longer, not shorter
3. Have a constructive, fair, and impartial tone
4. Share experiences over opinions
5. Back up opinion with facts and references, including specific experiences

The most common kinds of bad subjective answer are:

• Soapboxing “here's what I think about this situation” in violation of that numbered list.
• Saying “here, try this” without citing any experience of how that works out in practice.
• “Here's how I'd rule this if I were DMing” which frequently both lacks explanation of why that's the right ruling and lacks citation of any experience.

2. What do we do with those answers when the community upvotes them?

Downvote them. They frequently get almost entirely ignored in terms of votes, maybe collect 1 or 2 upvotes. The community is okay at policing these sometimes, but frequently just doesn't police them at all.

Request they cite experience to meet our citation expectations. Link those citation guidelines to the author in a comment. And downvote them.

3. Should we have a harder line for those types of answers in order to stay within our good subjective requirements?

They're already outside our good subjective requirements. Y'all could stand to downvote them more.

When they're in HNQ the tide of upvotes is hard to keep at bay, but downvotes from the community can help a lot.

4. Will this help keep those types of questions open or will it drive away users?

As long as these bad answers get managed properly by the community requesting improvement and downvoting poor quality answers, it won't escalate to that — plus we can now boot questions off HNQ.