9
\$\begingroup\$

I'm not talking about opinion-based questions, but the flipside of when a question could generate good subjective answers, but doesn't.

This question does have several parts that are tied together:

  1. What criteria is there for determining an answer is opinion-based bad subjective?
  2. What do we do with those answers when the community upvotes them?
  3. Should we have a harder line for those types of answers in order to stay within our good subjective requirements?
  4. Will this help keep those types of questions open or will it drive away users?

It seems that we have a very broad and unclear process in that context. I can't figure out when we decide to close and when we don't. It's clear when an answer isn't supported by good subjective, but it doesn't stop the community from upvoting the good idea generation (or bad), and there are times when questions are closed and others when they are not.

Here are some examples:

Cases where questions remained open even with a large number of bad subjective opinion-based answers

What is the purpose of easy combat scenarios that don't need resource expenditure? - I asked this one and didn't put in the caveat about good subjective. Many answers did come in with experience, but many did not and this question stayed open.

How to design and run cinematic combats that move between zones? - ANother example where some are experienced by answers and others are not. But many many are not, and this was allowed to stay open and is effectively idea generation.

Are there any magical contraceptive birth control options available? - Another case where the vast majority of answers aren't experience-based. Question is remains open.

A Meta about when a question was closed due to a large number of bad subjective opinion-based answers

This meta covers the other side of this where we say we want to close questions when this occurs.

It very much seems like we don't have a process of determining when to close when not to close. I get that we don't want opinion-based answers without experience, but we do seem to allow them a lot which suggest that we do kinda want it (which is fine!) - but it leaves a weird taste in my mouth not knowing when to act and when not to.

Non-experience based idea generation

I think the problem is that many people like the ideas that get generated, even without experience to back them up (they can still be good ideas.) But if we, as a site, are saying that we don't want those answer then we need to decide if that's really true or not and draw a hard line. Currently, the line seems very weak and...opinion-based on when/if we will do something when those answers come up.

The case for the hard line

If we are serious about not wanting bad subjective answers, we need to have a harder line and start flagging those answers for removal.

This is the extreme case of saving the community from itself. If the community upvotes bad subjective answers, then the cycle will always repeat. The only way to end the cycle and promote good subjective is to show that bad subjective isn't something we will tolerate.

The case for a softer line

The problem with the hard line is it's not very nice in appearance and we may lose members because of it. That worries me, because the community does seem to like those answers and I'm not sure what the ultimate effect may be.

\$\endgroup\$
  • 1
    \$\begingroup\$ The one that's referred to in that meta didn't seem like a very convincing example to me. That one was pretty controversial. But... I'll defer to other participants in this Q/A whether there's enough context. I understand the problem you're talking about even without extra examples, so it's probably not necessary. \$\endgroup\$ – Bloodcinder Mar 18 at 15:39
  • 2
    \$\begingroup\$ FWIW I think the Birth Control question is, since the edits by user, an idea list question and needs to be closed. Birth control in the rules might be valid, but currently it just invites creative applications whose validity is up to opinion or imagination. \$\endgroup\$ – kviiri Mar 18 at 15:54
  • 2
    \$\begingroup\$ I'm unclear on what this meta Q is asking us. Your title is asking about how we judge opinion-based answers, and you say you're not asking about the questions. But you then talk about when we do and don't close the questions, so you're also asking about the questions, and about judging the questions, and you don't ask at all about the moderation criteria we apply as a community to opinion-based answers. So are we talking about when we do/don't close questions that are attracting opinion based answers, or something else? \$\endgroup\$ – doppelgreener Mar 18 at 15:55
  • \$\begingroup\$ @kviiri I kinda agree, but that's for another meta :) \$\endgroup\$ – NautArch Mar 18 at 15:56
  • 2
    \$\begingroup\$ @kviiri I've been vying to VtC it for a while, but ever since the bounty was added I can't. \$\endgroup\$ – Rubiksmoose Mar 18 at 15:56
  • 1
    \$\begingroup\$ @doppelgreener I'm trying to ask about when do we close a question for drawing opinion-based answers. But it's also about what do we do with opinion-based answers to prevent closure due to them. I'm honestly not sure how to separate them out as they seem tied together. \$\endgroup\$ – NautArch Mar 18 at 15:56
  • \$\begingroup\$ @NautArch Ok, cool. I'll await that edit since I have thoughts to share on that. Both of those things are OK to ask about together since they sorta go hand in hand. \$\endgroup\$ – doppelgreener Mar 18 at 15:57
  • \$\begingroup\$ @doppelgreener Tried to clean it up - worrried I made it worse. \$\endgroup\$ – NautArch Mar 18 at 16:01
  • 2
    \$\begingroup\$ @NautArch Thanks! That's an improvement, 'cause it's now clear that this is sort of a multi-part question. \$\endgroup\$ – doppelgreener Mar 18 at 16:03
6
\$\begingroup\$

We should take a harder line on unsupported answers

I don't like these questions having to go silent either. But last time we tried deleting unsupported answers, that generated a lot more blowback, so here we are.

We have better guidance on citation needs here now, so I would suggest the community comment linking that guidance, downvote, and if that does not avail, vote to delete unsupported answers. This will save these questions and ideally teach people how to answer in a Good Subjective, Bad Subjective manner.

But individual cases differ

Some of the questions you link have stayed open because they have at least some experience-based answers, because closing questions is a last resort we usually use when basically none, or none of the high voted, answers are GS/BS supported.

But some things don't require huge amounts of support (see your other recent question When do homebrew suggestions need to be playtested by an answerer? for a similar situation). If someone posits "Well, you could let divination serve as a birth control method to ask if conception is likely" - do they really need personal experience doing that? No, I don't think so.

  • Having said that, the birth control question is certainly on the line given it's drawing increasingly ridiculous idea generation.
  • The mobile combat question, the top 2 answers say "per my experience" but don't really go into that, leading me to not upvote either of them but not downvote either.
  • The "easy combat scenarios" one is falling into the usual problem many designer-reasons questions did - there's one or two experience based answers but they're mostly unsubstantiated. But then again some may not need that? "Sandbox playstyles rely on a random assortment of foes above and below your level." Yep. This isn't Wikipedia, we don't need every sentence footnoted.

In any event, we as mods don't have time to trawl every question on the site, so we rely on y'all to vote, comment, flag, edit, and so on. Not every opinion-fest comes to our attention, we need y'all for that (and ideally to handle it without mod intervention).

\$\endgroup\$
  • 1
    \$\begingroup\$ Thank you - this covers a lot of what I was thinking (including when not to do it because of good answers.) I'm going to try something new out in comments, but I'm also not going to get into arguments about citations with people. I'll post my notices and let the community decide if they think it's worthwhile to upvote/downvote/novote. But at a minimum, maybe they'll think more about how to construct answers and what makes an appropriately cited answer good. \$\endgroup\$ – NautArch Mar 19 at 13:26
4
\$\begingroup\$

Questions getting closed because of runaway opinion-based answers

Things have changed very recently, such that hopefully this won't happen often anymore. To talk about that, I first need to talk about why this happens.

Basically: it's mostly HNQ's fault. Lots of our social and more challenging subjective questions hit HNQ. Sometimes when that happens we just start getting a deluge of crap answers, and see all the answers (including the bad ones) getting too many upvotes and no correction. The people posting these crap answers have been here before, so they have a bit of rep and don't get stopped by Protected status, but they're not familiar with our customs including our citation expectations. HNQ visitors frequently aren't experts on RPGs and don't know our customs, and can't downvote, so they just upvote everything that looks plausibly good.

Now, this doesn't only happen as a result of HNQ, but almost all the time this happens it's because of HNQ and I can't remember a time it didn't.

When that happens that is a bad situation. If the community doesn't or can't stop the cavalcade of crap, sometimes the community or diamond moderators get fed up and close the question. Closing was our only tool for addressing this situation so we used it, much to the disservice of the querent.

As of recent updates to the Hot Network Questions (Updating the Hot Network Questions List - now with a bit more network and a little less "hotness"!) this shouldn't actually really be a thing anymore because moderators can now remove questions from HNQ. Removal is permanent, the question will never return to HNQ after that. There's a question about when we should do so (When should a question be removed from the Hot Network Questions List?) and the only current answer says we should remove a question from HNQ “when HNQ harms the interests of the asker”. That's this situation.

So nowadays, when a question's getting runaway crap answers, instead of closing it, moderators should remove it from the HNQ to stop the tide.

A few days ago we booted our first question off the HNQ: How can I discourage/prevent PCs from using door choke-points?. It's a good example of exactly all of this happening and without this change it was heading for being closed.

Identifying & managing bad answers

  1. What criteria is there for determining an answer is opinion-based bad subjective?

Good Subjective is defined in the blog post Good Subjective, Bad Subjective and re-explained in our Subjective citation expectations.

Bad Subjective is all the subjective stuff that isn't Good Subjective. It's when you provide absolutely no experience to back up how your solution works in actual practice. It's also when you break these guidelines:

To apply the Good Subjective, Bad Subjective's guidance on questions to answers as well, a good subjective answer should tend to:

  1. Explain “why” and “how”
  2. Be longer, not shorter
  3. Have a constructive, fair, and impartial tone
  4. Share experiences over opinions
  5. Back up opinion with facts and references, including specific experiences

The most common kinds of bad subjective answer are:

  • Soapboxing “here's what I think about this situation” in violation of that numbered list.
  • Saying “here, try this” without citing any experience of how that works out in practice.
  • “Here's how I'd rule this if I were DMing” which frequently both lacks explanation of why that's the right ruling and lacks citation of any experience.
  1. What do we do with those answers when the community upvotes them?

Downvote them. They frequently get almost entirely ignored in terms of votes, maybe collect 1 or 2 upvotes. The community is okay at policing these sometimes, but frequently just doesn't police them at all.

Request they cite experience to meet our citation expectations. Link those citation guidelines to the author in a comment. And downvote them.

  1. Should we have a harder line for those types of answers in order to stay within our good subjective requirements?

They're already outside our good subjective requirements. Y'all could stand to downvote them more.

When they're in HNQ the tide of upvotes is hard to keep at bay, but downvotes from the community can help a lot.

  1. Will this help keep those types of questions open or will it drive away users?

As long as these bad answers get managed properly by the community requesting improvement and downvoting poor quality answers, it won't escalate to that — plus we can now boot questions off HNQ.

\$\endgroup\$
  • 2
    \$\begingroup\$ Yet, many bad subjective as defined here have more than a couple upvotes. It seems the community just isn't self policing and voting on whether or not they like the ideas/answers and not being critical in the answer itself. \$\endgroup\$ – NautArch Mar 18 at 16:41
  • \$\begingroup\$ Do any of those bad subjective types put an answer solidly in VLQ flag territory? At what point (if any should we bring the question to a mod's attention for possible addition of a "citations needed" notice on the post? Sorry I know these are kinds of really squishy questions, but I feel like these tools might stand to be better understood in the context of this issue and, possibly, better used. \$\endgroup\$ – Rubiksmoose Mar 18 at 16:44
  • \$\begingroup\$ @Rubiksmoose Excellent question - and goes to the crux of my "What do?" bit. SHould we flag any non-experienced based idea generation answers? \$\endgroup\$ – NautArch Mar 18 at 16:46
  • 3
    \$\begingroup\$ @Rubiksmoose Short, crap, uncited answers are VLQ, other stuff is less so. Bear in mind VLQ flags mostly get handled by the community in review queues. Overall we already try to do quite a lot as a mod team on these questions, but we need more involvement from the community downvoting bad subjective answers and requesting improvement from them to meet our citation criteria. We're unlikely to unilaterally delete answers that have positive scores. We could do so if we have a community mandate to do so, but we're just four people, and we can't do all the post moderation ourselves. \$\endgroup\$ – doppelgreener Mar 18 at 16:48
  • 1
    \$\begingroup\$ @NautArch As a community y'all should be ensuring they are negatively scored and have citation requested in order to meet Good Subjective. It is your site to moderate as a community member. Diamond moderators should be primarily handling unusual exceptions: this is usual stuff that happens day to day. Like you say there's not enough self-policing going on. When it happens it's effective, but most posts it just doesn't happen to. \$\endgroup\$ – doppelgreener Mar 18 at 16:52
  • \$\begingroup\$ In those cases where the community isn't voting by the guidelines for what they probably should be doing, what should we do? Should we flag? Should we debate? If we want to make a change (do we?), then something different needs to happen because it doesn't seem like the standard approach is working. \$\endgroup\$ – NautArch Mar 18 at 16:56
  • 1
    \$\begingroup\$ @NautArch I'm not sure. Leave downvotes, add comments requesting citation, encourage others to do the same, is what I've been doing. \$\endgroup\$ – doppelgreener Mar 18 at 17:45
  • \$\begingroup\$ @doppelgreener Sure, but that seems to bring us back full circle. That's the system in place, and it doesn't seem to be doing the job. \$\endgroup\$ – NautArch Mar 18 at 17:46
  • 1
    \$\begingroup\$ @NautArch Sure, and I can only incrementally change that. Just to be clear, “diamond moderators do most of the work via flagging and deletion” isn't a viable alternative: we're meant to do as little as possible and we can't do that much for the site and the slack needs to be picked up by the community itself... by downvoting more and leaving comments requesting citation. I don't have a magical way to make that happen but it's what needs to happen. \$\endgroup\$ – doppelgreener Mar 18 at 17:47
  • \$\begingroup\$ That's kind of frustrating :( I recently flagged an answer I thought was problematic (that others don't), so I'm interested to see if I'm looking at it wrong or if this is a good example to discuss. \$\endgroup\$ – NautArch Mar 18 at 17:58
  • \$\begingroup\$ Let us continue this discussion in chat. \$\endgroup\$ – NautArch Mar 18 at 18:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .