How should question protection be used on RPG.SE?

Question

Goal for this discussion

Following a recent event, it became clear that RPG.SE lacks sufficient guidance and community consensus on how the protection system should be used here. The moderation team has created this meta to facilitate a discussion that we hope will result in establishing what best practices we want to have around protecting, unprotecting, and leaving questions protected. Our hope is that this will result in a better community understanding of this tool, which will result in more consistent, beneficial application of it.

Note again that we don’t necessarily need hard rules or hard numbers here. Also, we don’t need to work out every single edge case here and now; with a good set of guidelines, we can address corner cases as they come up. The goal here is to construct guidelines that will allow the community to apply their wisdom to use this tool consistently and to the greatest benefit to the site.

Background for protection discussions

Question protection is one of the tools provided by Stack Exchange to maintain the quality of questions on the network. Protecting questions is a privilege granted to users with 15k+ rep, and the links question on Meta Stack Exchange provides the following advice on how and when to use it:

When should I protect or unprotect a question?

• Do protect questions that are attracting a lot of non-answers or very poor answers (spam, etc.) from new users.
• Don't protect questions just because they're linked to on Hot Network Questions or a high-traffic news site.
• Do unprotect questions that aren't currently attracting a lot of attention and don’t have a long history of unproductive answers.

See also: Changes and guidelines for the Protected Question status

Here at RPG.SE, we have historically used protected questions as a defense against spam and for questions which attract low-quality answers (for any of a number of reasons) by 1-rep users. Rubiksmoose gathered some detailed data as a background to this conversation. A summary of this information can be viewed in this meta: How has RPG.se used protection? (crunching the numbers on spam and protection)

Automated Community protection

There is a community bot that will automatically protect questions that meet certain criteria. The criteria (as sourced from the same Meta.SE answer quoted above) is:

1. The number of deleted answers from users with <10 rep, plus the number of answers with helpful spam flags, is at least 3. (Note that spam answers from new users are counted twice.)

   • Generally, this means that three deleted answers from new users will cause auto-protection, but if at least one of those answers is spam, only two answers will trigger it.

2. Five answers from users with <10 rep were posted in the last 24 hours. [...]

The system will never unprotect a question automatically, even if the deleted answers are later undeleted or the spam flags are cleared.

The goal for us is not to be human versions of this bot. Abiding by hard and fast rules based on the number of deleted posts and traffic patterns is something the bot is already very good at. We want this discussion to guide the community on how to handle the questions where these criteria fail to meet our needs.

Example discussion points

Some points a good answer might consider (but it is not required to do so):

When should protection be used?

• Spam and low-quality answers are two already-in-place use cases.
• How do we best identify questions that have attracted spam answers that could benefit from protection?
• Does it make a difference what the content of the spam is (e.g. "spellcaster" spam, versus some actual human spamming a link to their RPG on DMsGuild)?
• Under what conditions should we protect a question due to low-quality deleted answers?
• Does HNQ (Hot Network Questions) status have any effect?
• Are there any other types of issues that protection should be used to prevent?

When should protection not be used?

• Are there any specific circumstances that we should define in which it isn't beneficial to protect a question?
• If a question has been protected in the past, should that have an impact on whether or not it should be protected again?

Under what conditions should we unprotect questions?

• When should we unprotect questions that have been automatically protected?
• When should manually protected questions be reassessed?
• Would some kind of regularly scheduled community check-in be beneficial? Or some sort of automated script?

(You may recognize a lot of these as suggestions left by Catija, one of the community managers working at Stack Exchange Inc., in one of the discussions on the community's response to the unprotected-questions issue.)

Once we establish some initial guidelines, we can work through some examples from our list of recently unprotected questions and start getting things reprotected. Please refrain from taking any major actions to re-protect questions until some community guidance has been established.

---

Related older meta from 2017: When should I NOT protect questions?

Answer 1

Use the site-wide guidelines

Protection was originally created to deal with high volume drive-by posting:

[Protection] is meant to be a reaction to persistent abuse from anonymous or unproven participants: when a page attracts a lot of noise or vandalism from outside the community, Protecting it reduces the amount of clean-up needed later on.

On our site, this is a bit of a solution looking for a problem. We don't have a problem with drive-by posting. We do not see volumes of 1 rep posting so high that we can't handle them.

When protection was introduced, an example screenshot was posted showing 10+ deleted answers by 1 rep users (most of which were 1 sentence long) on a post. An analysis of protected questions on the site showed that 99% of them had 5 or fewer deleted posts. The majority of them (66%) had 2 or fewer deleted posts. 11% of them had no deleted posts at all.

Keep in mind the warning given by former CM Josh Heyer (Shog9) who posted the site-wide guidelines:

Judicious use of this feature is critical to allowing these sites to handle large amounts of external attention, but over-using it breaks the system: Stack Exchange sites depend on a constant influx of new blood, both to answer new questions and provide updated information on old ones. When in doubt, err on the side of letting new users prove themselves before locking them out.

How do they apply to us?

The site-wide guidelines are simple, but missing a little nuance:

Do protect questions that are attracting a lot of non-answers or very poor answers (spam, etc.) from new users.

Don’t protect questions just because they’re linked to on a high-traffic news site like Reddit or Ars Technica. While there’s certainly some correlation between sudden spikes in popularity and associated non-answers, not all popular questions suffer from this.

Do unprotect questions that aren’t currently attracting a lot of attention and don’t have a long history of unproductive answers.

What constitutes "a long history"? What constitutes "currently"?

I believe the Protected queue system should inform out choices for these metrics. They currently list 2 important pieces of information: the number of answers in the past 30 days, and the number of deleted answers.

For convenience's sake, I think "currently" should mean "in the past 30 days", and "a long history" should mean "more than X deleted answers".

When determining the value of X we should consider the volume of drive-by posts our site members can handle. Currently, the queues sits at empty most of the time. In the past week, we have had 7 deleted low-quality posts. I think we have some room to set X a bit higher (10?) and see how it goes.

Keep in mind that we are not robots, so if you see a post get a lot of answers really quick, it's ok to act before it hits that threshold. Similarly, if a post has more than that number of deleted answers, but is really old, it's fine to unprotect it.

Answer 2

We have a high moderation capacity

One thing to remember about the protection feature is that it is a preventative measure that aims to reduce the amount of (reactive) moderation that is necessary. We have a high capacity for moderation – and most of that consists of community members using their flags and privileges. There’s very rarely more than one or two items in a given review queue, and very often nothing. And to scope out post deletion: we get north of 200 answers and 100 questions a week, and delete about 25 answers and 6 questions over the same time period on average. Therefore, we should limit the use of protection to questions that have attracted answers that had to be removed in the past and that and that are overwhelmingly likely to draw answers that will have to be removed in the future, because we have the capacity to let some things slip through.

Let’s talk about the answers from new users

Ok, I think that in order to take a look at this, we need to dissect the types of answers that come in to old questions from new users (or specifically, from accounts with <10 rep). These are gonna be broad-strokes categories, and yes, they include different categories of spam.

• Keyword spam. This is much of the spam previously discussed. Spammers look for keywords which relate to their spam and post it in the answer box below. Keywords being in the other answers probably also attract them. Our more prominent keywords are “spellcasting”, “vampire”, “resurrection”, “death spell”. These are primarily trying to sell (scam, obviously) services offering something to the relevant keyword – say, turning you into a vampire so you can get revenge on your ex. No, I’m not kidding.
• Network spam. This is basically keyword spam that finds keywords on other sites on the network (mainly SO), but posts it here through the vagaries of website crawling. Basically, this is “tech-y” spam that gets posted to us. This mostly happens to HNQ questions, as they’re linked from other network sites.
• Honest spam. This is spam that is posted on a related topic by a genuine human creator, but often falls under “link-only answer” and “fails to disclose affiliation”. This is fairly low-volume.
• Forum replies. A fair chunk of answers posted by new users are forum-like replies, which very often need to be removed. We can probably throw other honest, but (given the site format) misplaced, attempts at communicating – such as new questions posted as answers, and “thank you”s – into this bag for the discussion.
• Actual novel/useful answers. We do get new users answering old questions with genuine answers. These are great, but from subjective experience, not too common.
• Useful comments as answers. So that happens too. New users encounter a Q&A that is outdated (or has some other issue), and post an answer (because they can’t comment) pointing the thing out. I’ll argue that this happening is (usually) a good thing (because it gives us a chance to fix something), even if the process is awkward.
• Screams from the void. This category is mostly on here for completeness. We get some answers posted that just don’t make sense. Standouts from the top of my head are the word “Giraffe” and a long series of “V”s,^† but there’s also a bunch that are just foul language, which is where my name for them originates. They don’t, as far as I can understand, follow any pattern.

How do we decide what warrants protection?

Here’s the crux of the problem as I see it. Deciding to continuously protect something is a value judgement that we, the community, have to make. We can do stats for how big the problem is, but we have to decide how much value the chance for novel answers and useful comments has, against having to moderate the other categories. Deciding to protect a question (indefinitely) is essentially deciding that the likelihood and benefit of the two useful categories is not worth the likelihood and cost of having to deal with the others, on that specific question.

So what do I think warrants continuous protection?

In short: keyword spam and high proven likelihood of forum replies, or a high volume of honest spam or screams from the void over time, with a low likelihood of novel answers and useful comments.

Factors in favor of protection

In more detail, if a question has received keyword spam in the past, it still has those keywords on it and it will be spammed again. This simply follows from the simplest assumption of how keyword spammers work. If it’s obvious why the question is pulling keyword spam, I don’t think we need to wait for two instances of keyword spam to be posted before protecting, and don’t think we should unprotect them just to wait for another spam answer to come in. If it’s not obvious, there’s minimal harm in letting the auto-protection kick in on the second. The less we have of this stuff, even in deleted form, the better. As analysis shows, this doesn’t actually affect that great a volume of questions anyway (some hundred of our 40k questions); it’ll be fine.

The second category I think we should protect against is those questions shown historically to attract many forum-like replies and likely to continue to do so. And let me be clear: the “many” is kinda important here.

These questions would most likely be those that are getting continuous views and are often fairly subjective (and thus have many people who want to voice their opinion). Why? Because if we want to retain users, having their first interaction with the site be their answer getting deleted because they didn’t know what the site was like isn’t great. Actually, it sucks. If they’re forced to poke around the site, they have (IMO) a much better chance at picking up how the site works beforehand (say, by taking the tour or something).

If a question is for some (possibly unknowable) reason drawing a lot of honest spam answers or screams from the void, we should consider protecting it. It is probably rare and strange enough that having a dedicated meta to discuss each such question is fine, in part because there may be other things we should do to the question.

Factors against protection

We shouldn’t protect a question if it is likely to benefit from novel answers and useful comments in the future. This is generally much harder to evaluate and to talk about in general. We’ll have to use our game and Q&A expertise.

The easy categories here are questions like optimization, questions trying to compile all the game features that fit X criteria, and rules questions on ambiguous rules (as they may be updated by errata or receive relevant commentary). Also questions on smaller/rarer systems. If in doubt, lean towards not protecting these.

Factors that shouldn’t count either way

Maybe it goes without saying, but I don’t think the remaining categories should be counted either way when evaluating indefinite protection. But I’d still like to make some comments about why, and this is relevant because these will be counted by the auto-protection algorithm.

• Network spam. If there’s nothing on the question which seems to have beckoned the spam, protecting it is unlikely to prevent further spam. If a question is drawing a lot of spam of this type while on the Hot Network Questions list (HNQ), protect it while it is on HNQ, and unprotect it once it is off. Reminder: a question leaves the HNQ list if it is more than 3 days old. I don’t remember this happening with the new HNQ, but that’s how I think we should deal with it.

• Honest spam. These don’t warrant protection. They’re one-offs, topic relevant, and half the time could be turned into valid answers if the user put in the effort.

• One-off forum replies. If the question just gets one or two forum-like non-answers, especially while new, there’s no need for protection.

• Screams from the void. As these are both varied and weird, they don’t really say anything about the question needing protection. They don’t really say much about anything.

The exception is obviously if there are high volumes of any of these over time, as covered above.

---

PS: if anyone would like me to split my suggestion from the dissection above, so we can use that more neutrally, I’d be happy to.

†: Those are two different answers. I’m sure others have more examples, but I’ll ask you not to share them in the comments here.

Answer 3

Don't protect questions for esoteric systems

We've got several rpgs on here with almost no participation. I think I'm probably the only active user with extensive Polaris experience, for example. There was another guy who played Anima for a bit, but also one of those anti-optimization people at the same time so that's kinda dead now. I'm quite confident that nobody on this site plays Ki Khanga (that includes you BESW, but props for trying :P ).

Questions asked in these systems represent a huge opportunity for this site to slightly correct its awful and overwhelming 5e bias. They also represent an area where people coming to the site randomly from Google are most often a better source of answers than we are-- we don't play these games, and the internet does because the internet does everything. Protecting these questions does the most damage to the site that protection is capable of, because the new users prevented from answering are the new users whose one-time participation most helps the site, and furthermore whose long-term participation best contributes to the growth of the site.

What's an esoteric system is definitely a grey area, though. I'd argue, for example, that OD&D counts, barely: although we have a bit of a healthy community here for that, it's very small. Also The Dark Eye, Polaris, FATE but not Fate, maybe Dresden Files but that's a bit of a stretch, Phoenix Command, Amber Diceless, anything less common than these. What seems clear to me doesn't count is D&D 3.x,4,5, Dungeon&Apocalypse World at the very least of the pbta games, Fate Core, stuff we regularly see on the front page and gets answered, like, immediately with at least two different active high-rep users. That's just my thoughts on it, though, I think we need to decide for ourselves when our userbase's lack of expertise is a good justification for higher mantainance on a question.

Answer 4

Summarizing criteria that I find relevant

Things that are a +1 point in Protecting a question:

• The question already has an accepted answer, that is most likely up-to-date.
• The question already has a highly upvoted answer, that is most likely up-to-date.
• The question already has 2 or more deleted answers from low (<100) rep users (I'm using <100 instead of <10 because the user may have gotten some rep elsewhere and I find the behavior overall similar)
• The deleted answers are comments, not answers, or answers that don't add anything new to what has already been discussed in other answers.

Things that are a +1 point in Unprotecting or Not Protecting:

• The question doesn't have any well accepted (either by community or by asker) answer.
• The question is about a not-so-popular game system, thus bringing more experts in that game system to our site would be beneficial.
• The question had a spike of activity for some reason and that spike is now gone. For example, you can check the timestamp in the low quality answers. If all of them are pretty much clustered together, it may have been such a spike. If it has 3 bad answers, each separated by 1 year, then bad answers are likely to keep coming if we unprotect.

For what is a well accepted answer: usually, answers have considerably more votes than questions (if anyone can help with the data here I'll edit, but I'm 99% certain this is true, and I will guess by at least twice the votes), so, if the most upvoted answer doesn't have half the score of the question, I would say that question would benefit from new answers. But use your own judgement on it.

We don't get too much spam/poor quality answers from 1 rep users

Okay, let us start with stating my experience on it: we don't get much spam. Most of our questions don't have a deleted answer or worse, a spam answer. For that reason, protection should be using sparingly. That said, let me give my 50 cents on some of the discussion topics you have suggested.

How do we best identify questions that have attracted spam answers that could benefit from protection?

On the run, and in a reactive way. I've recently protected this question because it got a new, considerably low quality answer from a 1 rep user, and we had two deleted answers on it already.

My suggested approach is simply: you are going through the review queue, you notice a bad answer from a new user, you take a look in the question. Does it already have other similar bad answers from new users? If it does, go on and protect it. There are more criteria which I will talk about later. Stay with me please. But the point is that this should be reactive. Protecting before the question has had any kind of problem is, for me, a no-no. Doesn't matter how much it looks like it is going to be a problem, the frequency of this problem is so low here that we can afford being reactive instead of preventive.

Does HNQ (Hot Network Questions) status have any effect?

This is an important question, and my answer is: I don't know. Most likely people coming from HNQ will have the 101 starting rep anyway, so I'm not sure it matters at all. HNQ seems to inflate votes, some not-too-useful comments, but personally I haven't experienced a huge problem with HNQ leading to a bunch of bad new answers. My current inclination is that no, it doesn't have any effect. If HNQ brings up a problem, we can see the symptoms of the problem, and protect based on the symptoms, making the fact that it is in HNQ is meaningless. Again: reactive stance.

Are there any other types of issues that protection should be used to prevent?

This is included in "low quality answers", but I personally feel like the most common problem with 1 rep users are answers that are essentially comments. For me, questions that attract such comment-answers are the highest priority for protection.

If a question has been protected in the past, should that have an impact on whether or not it should be protected again?

No. But the fact that the question probably already has 2+ deleted bad answers should, so there is a correlation here. But simply the fact that it has been protected in the past shouldn't.

Are there any specific circumstances that we should define in which it isn't beneficial to protect a question?

The scenarios I have described in my summary. If a question has no good answers yet, then we would certainly want a user that can answer that question answering it, no matter if they are new to the website. Similarly, if the question is for something not too popular, I think it is more beneficial to take care in protecting it, since we may actually bring an expert from a obscure system, and they may be the only expert we are getting any time soon. PS: I say that as a D&D5e-user. Don't take it personally, fellows.

When should we unprotect questions that have been automatically protected?

Most likely never, unless we can identify that the automatic protection was linked to some extraordinary temporary event (idk, the question being posted in Kotaku or 9gag or another big communication vehicle) and such event has since then vanished.

About manually protected questions and/or scheduling check-ins, I have no formed opinion.