Request for feedback on Good Subjective moderation

Question

The idea of Good Subjective questions and answers is core to how Stack Exchange works and is essentially what allows RPG.se to exist. We have this Q&A that sets the broad outline for how that idea manifests itself and generally how we moderate it:

Answering Subjective Questions

The blog post Good Subjective, Bad Subjective is the gold standard for understanding how to make good on topic answers to subjective questions. The heart of it is using the Back It Up! principle, which states that you should only answer based on:

• Something that happened to you personally
• Something you can back up with a reference

Here, "happened to you personally" means real play experience. We understand that some people believe that given their wide experience in games they think they can tell someone the right technique or system or solution to use for their particular need even though they haven't done it themselves - but when everyone does that, it leads to opinion-laden chaos. "Good Subjective" is constructive and is based on experience and expertise. "Bad Subjective" is speculative and is based on conjecture and opinion. Speculative answers will be downvoted, commented on ("Back It Up!"), and/or deleted.

Recently, however, we have noticed more and more friction about how moderation around subjective answers is handled. Users, both old and new are noticing changing norms on the site in how the GSBS principles are being moderated or enforced by the community or diamond moderators, and some disagreements surrounding this enforcement have arisen.

Given that we are in the process of getting a new post notice that we can specifically use for subjective answers, and given some of you raised objection over this issue there, we figured now would be a good time to hear from the community and see if any kind of course correction in how we are handling such posts is needed.

So, we want to hear from you: how are we doing here and how can we do better? If there are course corrections for the community and/or diamond moderators to make, what are they?

Answer 1

The purpose of GS/BS is to enable meaningful voting behaviors. That’s it: to make sure that the Stack Exchange system actually works, actually filters out noise and highlights strong signals. Can an expert judge the quality of this answer and how well it will likely help those who have the question? Or is impossible to judge because it’s just an expression of preference (which might suggest something wrong with the question, but is a bad answer in any event), or because it’s baseless speculation, or because it’s armchair hypotheticals? Then downvote, request for more back-up to the claims made, and sure, put a post notice.

But the friction isn’t from such answers. The friction comes from people getting pedantic about how people enable meaningful voting. It’s unnecessary, and more than sometimes, it’s rude. If an answer thoroughly details how to handle a situation, it is not truly necessary to end it with “and yes, I have done this myself.” Answers that come from experience and expertise are generally pretty easy to distinguish from answers that don’t. Ultimately, take a step back and just ask yourself, does an answer convince you that it’s coming from a place of knowledge and that this recommendation is a strong and useful solution? If so, upvote, if no, downvote, and if you have some confusion—you can downvote for being confused by an answer, answers should be clear—ask for clarification. Make suggestions, offer constructive criticism.

But don’t waste everyone’s time with “This all seems like great advice that really would help in this situation, but you have to explicitly say you’ve actually done this.” That isn’t improving the site. And yes, that sentiment is precisely the kind of thing I feel like I am seeing more and more often.

Answer 2

Good subjective bad subjective is sufficient for a good answer, but not necessary.

I find getting prodded on the issue unpleasant

As detailed in a previous answer about this issue. At least a few others find it unpleasant as well.

Some examples of good answers that don't explicitly certify personal experience

Good answer without attesting personal experience:

GM Considerations for a Human Barbarian Toddler PC

The answer is essentially, "here's something to try and stuff to watch out for", but does not include that the answerer has done something similar or experienced something similar. It's a great answer without personal experience support explicitly stated.

Good answer without attesting personal experience citing questions that have good answers that do not attest personal experience

Can the dungeon master have a player character?

The best part of this answer is that the answerer has not personally tried it, and wouldn't attempt it.

For perspective, I've been running games for over 25 years, and I do not feel confident that I could pull off a DMPC successfully, even with all that experience.

The answer is neither something that happened to them personally, and the references are to questions with answers that also do not refer to direct personal experience. They're great answers even though they don't satisfy the good subjective answer criteria:

• Something that happened to you personally
• Something you can back up with a reference

Some other good answers that do not meet the good subjective bad subjective criteria.

• This answer advice on a homebrew hook-shot equipment.
• This answer about avoiding playing a character that you don't like.
• This answer on giving non-accusatory feedback to a DM.

I find the post notice particularly unappealing

Answer with example post notice

The comments requesting certifying personal experience with the threat of, "okay but your answer might get deleted..." to be sufficient to put me off answering questions as it is. The post label, to me, appears to be a sticker implying "this answer is bull****". I do not expect it will contribute significantly to better answers, but I do expect it will not be well received.

Misapplied Post Notice

Using a post notice is likely to result in it being misapplied or remaining after the body of the question not longer has the issue. At the time of writing, This answer has a gsbs post notice, yet states clearly:

...it's a problem that's, by my experience on similar issues, much better solved by discussion than by invoking in-universe means for punishment.

The post notice was:

Some of the information contained in this post requires additional references. Please edit to add citations to reliable sources that support the assertions made here. Unsourced material may be disputed or deleted.

Moderate moderately

The good subjective bad subjective guide is a way to craft a good answer. It is not the only way. Do not moderate as if it was.

Answer 3

I have been thinking about this issue for a long time, because it has irritated me for a long time. But I have not been aggressive about it (by my definition of aggressive, anyway-- your mileages may vary) because:

• I do understand the desire, and agree in large part agree with the need, to prevent the site from degenerating into a freeform brainstorming environment, but also,
• I do not yet have a comprehensive or a workable suggestion, let alone a suggestion I think is both, and
• I don't like being the crazed iconoclast, burning things down without having something put to in their place.

But the subject has come up, because, at the time of this writing, the top "constructive" answer and the top "frame challenging" answer to this question are currently tied. And that, only because the frame challenge has a lot of downvotes on it, too. So with that said, let me at least try to articulate the nature of my irritation, in the hopes it sparks some good ideas.

The Anecdote Tax

I know that there is no formal requirement for experience to be in the form of anecdotes, and I know that there is no policy that experience MUST BE cited in the form of an anecdote. That said, that's the way it can come across in the aggregate. I don't think this can be in much dispute as a simple fact, since there is at least one case of self-deletion of a good (in my opinion) answer due to the perceived need to share enough details to constitute an anecdote.

Again, I understand that comments in that thread (now moved to chat) were not necessarily meant to say "Details and anecdote, or nothing!" but a neutral read of that thread tells me that nevertheless that is the overall impression that answerer got. That is why I made the comment I did (that GSBS answers do not require specific details or anecdotes) because I thought it would be detrimental for the site to see the answer deleted.

The Guidelines/Community Wiki Are Unclear

Two relevant quotes from the same source:

Not every claim you make will need citation. Many things we might say are common sense or common knowledge and nobody will request a citation for them.

And later:

This means when you provide a subjective solution you believe will resolve the situation, we expect that you cite analogous experience of how it has worked out in actual practice — your own experience or someone else's.

That both does (or nearly does) mandate detailed anecdotes, and also brushes aside the need to do so. I cut off the first quote for effect, so let me quote the whole thing here, with what I believe to be a serious problem highlighted:

Not every claim you make will need citation. Many things we might say are common sense or common knowledge and nobody will request a citation for them. For many things, however, nobody should have to take our word for it that what's being said is true; we need to show it is. If someone requests citation it's probably needed—revise your post to add that citation.

The bolded text is what is trying to reconcile these two completely contradictory approaches, and I think it is fundamentally flawed. I see no reason to automatically assume that every request for citation is a good and necessary request. All it takes is one user who makes it his mission to ask for detail and citation on every perceived GSBS question and/or answer to collapse this Solomonic attempt to have the best of both worlds into the worst of one world.

An Improper Focus On Noise Vs Signal, And On Mechanics-Like Policies

The purpose, I've gleaned from past reading here in meta, is to put a preponderance of effort (seemingly, all of it) on filtering noise-- noisy questions, noisy answers, noisy comments-- from signal. (System working as intended!) And the tendency that I have seen is a desire for perfect policies.

I'd like to suggest, without a chapter-length MathJAX-augmented dissertation, that the former is wrong-headed, when taken to extremes and that we are nearing the extremes now; and that the latter is futile. We'll end up with less valuable signal-- enough less to be detrimental to the site. I cite as evidence the situation above.

A Proposal, If I must

Maybe we can be a little slower on the trigger in insisting on detailed anecdotal references.

Answer 4

Don't 'enforce it' but do recommend it

I really like it when I see a question end with a link to the good subjective / bad subjective post (indeed I find links to 'how to stackexchange good' content are brilliant), and have seen some really good answers which have clearly benefited from that advice.

Equally I have seen answers which don't cite any evidence, but show a clear understanding of the problem and are answered more from life experience than anything else (And I think that is fine).

So I would suggest 'this answer could be improved by following this advice: link' rather than any veiled 'do what we say or your post may be deleted' threat, and let the voting system deal with the answers as they stand (as is the solution for other quality issues).

What I would NOT like to see is any action that leads to lower quality answers, and would willingly sacrifice all the good but not cited answers in exchange if it came to that. I wasn't around when the 'rule' was first agreed, so I don't know how bad things were or how effective it has been.

I added a suggested moderator notice on the other thread to cover this option.

Answer 5

Back to Stack Basics

1. GSBS guidelines: we have those already.

2. Encourage "back it up" with a link to those guidelines.

   This is never a bad idea. Some authors will upgrade their answers, some will not. The post notice that is in work should also help in raising awareness of the better ways to answer GS/BS questions.

3. Let the votes work. If the author won't back it up, don't pester the author about it. Whomever left the comment has reminded the author of a "best practice" and that suffices. Better answers will tend to get more votes.

4. Let the votes work. On a Stack Exchange, the votes are supposed to stack answers from best to least-best by number of votes. This latest crusade versus GS/BS looks to me like an attempt to indirectly micromanage votes, and voters, before the fact. Let the voters vote. Leaving the initial "please support this answer" comment visible will help voters, IMO.

5. If now and again a not great answer moves up the stack, that's not a problem. Other stacks run into this now and again. (There was a famous example on SO that I read a few years back where a wrong answer had like 200 votes. It happens, but it's rare). Let the votes do the stacking.

6. Use flags. They are a tool. For an answer that is just plain bad or utterly unsupported, flag it for mod attention so that it gets into the review queue as "low quality" or whatever. We have plenty of folks who get into the review queue (me included). Our community mod are generally effective. So, alert that community mod effort with a flag.

7. There isn't an easy button, but there is a suite of tools. Let's use them all.

   This latest effort of "give me another rule" looks to me like another vain quest for an Easy Button.

Don't Chase Exceptions

Chasing exceptions is what Deming called "tampering" and is a great way to make bad policy and bad "corrections" to a given process. I learned this the hard way in a number of real life organizational change efforts. I have been on the receiving end of this, and on the policy writing end of this. It stinks no matter what end of the stick you are holding.

This is a revision of a previous answer.

After a productive discussion with NautArch in chat this morning, it occurs to me that this answer applies here as well as its original location.

Answer 6

We should promote our requirements for Good Subjective support

I very much understand that this is a hot-button issue, but I am also a strong believer in why we should be backing up subjective responses.

While I do understand that many folks here have a lot of experience that they can bring to an answer, there remains a difference between generating ideas from experience and providing supported answers to help guide someone with an issue.

In the past, I have submitted answers that aren't supported and were well received. But as I spent more time on the site, I came to understand the purpose of why we should still support our ideas.

I believe that while we can come up with good ideas from our experience, we should hold ourselves to a higher standard and be able to provide better direction to folks asking questions. Even when I was in consulting and I worked at a place where there was a lot of institutional knowledge and experience, we were still required to back up our recommendations.

This is the same thing here. If we're going to suggest a course of action, we should be able to speak to it beyond the idea. Providing feedback on how the recommendation had went for us (or how we've seen it used) bridges the gap between an idea suggestion and an idea recommendation. I don't want to give unsupported recommendations because I think it's less valuable. It's much more valuable to provide both a recommendation and information on how it went. THis helps identify potential pitfalls as well as just the idea, which I believe improves the quality of our answers and our site.

But mostly, stackexchange does have direction on supporting good subjective. I guess we can opt not to enforce or encourage it, but there's excellent reasons for doing so and I don't see this as something we should just ignore and pretend doesn't exist.

So how do we back things up?

For those who do not agree with the recommendations on supporting good subjective as discussed in previous metas, what is the recommendation for how to support?

I am all for looking at something fresh, but the complaints I seem to read are that they don't agree with the current good subjective supports. I have yet to see a recommendation on other support options and would very much like to see that.

It's not all about personal experience

Personal experience as a form of good subjective support is one piece of it. But nowhere discussed has it been the only - but that is what a lot of folks see it as. This answer covers very well that various forms of good subjective support that have been discussed and personal experience is one of those, but not the only one.

Focusing solely on that and assuming it's the requirement or gold standard for good subjective is doing how we can look at good subjective support as a whole a disservice.

Answer 7

Answers should not always require backing.

In the GSBS article, it highlights the purpose behind the guideline was a board where every question was subjective and every answer was subjective (in regards to parenting). It nowhere mentions that each and every post must adhere to this rule, let alone every answer.

This sentence from the GSBS article sums up the most important idea for me:

they’re attempting to mold and shape their inherently subjective Q&A into something constructive, informative and helpful.

These three specific criteria are the measures by which the writer proposes an answer would be considered "good".

If a subjective answer is constructive to discussion, then it assists in building up a meaningful answer or route on how to get there. An answer that opens itself up to a discussion on how to employ a specific strategy, how to hook it into a story, or how to modify the answer to be applicable to the asker's specific situation is exactly what I envision from an RPG Q&A board.

If a subjective answer is informative to the general reader, then it contributes overall knowledge to someone interested enough to read both the question and the answer. General education is never a bad thing, especially for people learning a system but too shy to ask questions or otherwise contribute on the boards.

If a subjective answer is helpful to answer the question, then it stands by its own merits that it is useful to the question asker, which to me seems the most important. An objective answer that doesn't answer the question is akin to a useless answer, and an objective answer that doesn't address the subjective implications of a specific course of action misses the gaming aspect of this board.

This SE deals with creative interpersonal interactions.

This is not a hard science where the "best practice" is something codified by time and experience. Using our imagination and proposing things we haven't done before is sometimes the best answer. Proposals such as this inherently draw on our experience and mastery of the content, and we can apply completely different scenarios we've encountered in the past to the one at hand through inductive reasoning.

Backing personal experience is not needed.

We shouldn't have to exposit our gaming resumes when we answer a question subjectively that is backed by it. It wastes space and it isn't verifiable.

The first issue is that espousing personal experience is an unnecessary use of space. If you ask me a question, I can tell you I've ran into the same problem every week for the last 20 years and this is how I always resolve it or I can cut to the answer, but in the end, the answer to the question is the same.

This raises a second issue of verifiability. There is no professional organization that tracks a player's history and can support the claim of if they have personal experience. As with any credential someone places on their resume, if it can't be verified, you may as well not include it. No one who questions the audacity of my answer is going to be reaching out to my gaming group for references. They'll just downvote and move on.