21
\$\begingroup\$

Goal for this discussion

Following a recent event, it became clear that RPG.SE lacks sufficient guidance and community consensus on how the protection system should be used here. The moderation team has created this meta to facilitate a discussion that we hope will result in establishing what best practices we want to have around protecting, unprotecting, and leaving questions protected. Our hope is that this will result in a better community understanding of this tool, which will result in more consistent, beneficial application of it.

Note again that we don’t necessarily need hard rules or hard numbers here. Also, we don’t need to work out every single edge case here and now; with a good set of guidelines, we can address corner cases as they come up. The goal here is to construct guidelines that will allow the community to apply their wisdom to use this tool consistently and to the greatest benefit to the site.

Background for protection discussions

Question protection is one of the tools provided by Stack Exchange to maintain the quality of questions on the network. Protecting questions is a privilege granted to users with 15k+ rep, and the links question on Meta Stack Exchange provides the following advice on how and when to use it:

When should I protect or unprotect a question?

  • Do protect questions that are attracting a lot of non-answers or very poor answers (spam, etc.) from new users.
  • Don't protect questions just because they're linked to on Hot Network Questions or a high-traffic news site.
  • Do unprotect questions that aren't currently attracting a lot of attention and don’t have a long history of unproductive answers.

See also: Changes and guidelines for the Protected Question status

Here at RPG.SE, we have historically used protected questions as a defense against spam and for questions which attract low-quality answers (for any of a number of reasons) by 1-rep users. Rubiksmoose gathered some detailed data as a background to this conversation. A summary of this information can be viewed in this meta: How has RPG.se used protection? (crunching the numbers on spam and protection)

Automated Community protection

There is a community bot that will automatically protect questions that meet certain criteria. The criteria (as sourced from the same Meta.SE answer quoted above) is:

  1. The number of deleted answers from users with <10 rep, plus the number of answers with helpful spam flags, is at least 3. (Note that spam answers from new users are counted twice.)

    • Generally, this means that three deleted answers from new users will cause auto-protection, but if at least one of those answers is spam, only two answers will trigger it.
  2. Five answers from users with <10 rep were posted in the last 24 hours. [...]

The system will never unprotect a question automatically, even if the deleted answers are later undeleted or the spam flags are cleared.

The goal for us is not to be human versions of this bot. Abiding by hard and fast rules based on the number of deleted posts and traffic patterns is something the bot is already very good at. We want this discussion to guide the community on how to handle the questions where these criteria fail to meet our needs.

Example discussion points

Some points a good answer might consider (but it is not required to do so):

When should protection be used?

  • Spam and low-quality answers are two already-in-place use cases.
  • How do we best identify questions that have attracted spam answers that could benefit from protection?
  • Does it make a difference what the content of the spam is (e.g. "spellcaster" spam, versus some actual human spamming a link to their RPG on DMsGuild)?
  • Under what conditions should we protect a question due to low-quality deleted answers?
  • Does HNQ (Hot Network Questions) status have any effect?
  • Are there any other types of issues that protection should be used to prevent?

When should protection not be used?

  • Are there any specific circumstances that we should define in which it isn't beneficial to protect a question?
  • If a question has been protected in the past, should that have an impact on whether or not it should be protected again?

Under what conditions should we unprotect questions?

  • When should we unprotect questions that have been automatically protected?
  • When should manually protected questions be reassessed?
  • Would some kind of regularly scheduled community check-in be beneficial? Or some sort of automated script?

(You may recognize a lot of these as suggestions left by Catija, one of the community managers working at Stack Exchange Inc., in one of the discussions on the community's response to the unprotected-questions issue.)

Once we establish some initial guidelines, we can work through some examples from our list of recently unprotected questions and start getting things reprotected. Please refrain from taking any major actions to re-protect questions until some community guidance has been established.


Related older meta from 2017: When should I NOT protect questions?

| |
\$\endgroup\$
  • 1
    \$\begingroup\$ If anybody needs any data crunched to support their proposed method beyond what I've already compiled here, I'm happy to try to get that data (or find someone who can get it). \$\endgroup\$ – Rubiksmoose Jul 21 at 14:27
  • \$\begingroup\$ Do we have any control over the bot's criteria? In other words, can we add to the bot's rules, or have an RPG.SE-specific bot, or anything like that, or is the bot just a general site-wide bot that we have no control over? You say that we don't want to become human versions of that bot, but on the other hand, if the bot can only do what it does, and we come up with more rules than what the bot already does, doesn't that mean we will become human bots after all? Or am I missing the point of what that section is saying? \$\endgroup\$ – NathanS Jul 21 at 14:50
  • \$\begingroup\$ @NathanS It is possible to modify those parameters to fit a specific site, IIRC from the MSE post on it \$\endgroup\$ – Someone_Evil Jul 21 at 14:56
  • \$\begingroup\$ @NathanS I'm unsure about the first bit (number of answers with flags + deleted low rep answers) but the second one (number of low-rep answers within 24 hours) is different on some sites. To quote "What is a “protected” or “highly active” question?" (emphasis mine): "[...] Five answers from users with <10 rep were posted in the last 24 hours. Three answers on English Language Learners and Workplace, and 20 answers on Code Golf. [...]" \$\endgroup\$ – Medix2 Jul 21 at 15:19
  • \$\begingroup\$ Thanks Someone_Evil and Medix2, that's good to know. I don't have anything answer-post-worthy to add, but at least if anyone is unaware of these things, hopefully they will now be more aware. \$\endgroup\$ – NathanS Jul 21 at 15:25
  • \$\begingroup\$ Should answers try to discuss only one thing or several? I'd just be unsure how to vote if I agreed with only some of the guidelines and ideas a given answer uses \$\endgroup\$ – Medix2 Jul 21 at 16:37
  • 2
    \$\begingroup\$ @NathanS to add to what has already been said, we do know that it has been done before, but we don't know how willing the company is to do it again or how hard it would be. But it seems to be within the "possible" realm at least. \$\endgroup\$ – Rubiksmoose Jul 21 at 16:43
  • 2
    \$\begingroup\$ @Medix2 I'd say that it would be up to the poster. If we end up with several highly voted fragments that would work well together, someone can always gather those into a single answer to be voted on as a group, or if it is clear that those are the only suggestions that have community backing possibly even just glued together after voting has settled. (1/2) \$\endgroup\$ – Rubiksmoose Jul 21 at 16:46
  • 2
    \$\begingroup\$ @Medix2 In the end, we want a comprehensive answer. If it is too much work to do everything in one shot, I don't see a huge issue with having some partial answers if necessary as long as the path to a comprehensive answer is clear. (2/2) \$\endgroup\$ – Rubiksmoose Jul 21 at 16:48
  • 1
    \$\begingroup\$ Please don't take this the wrong way, but after I began my review of the "OMG 700+" questions, I arrived at the feeling of "we don't have a problem here" and I wonder if we are overthinking this. My first impression. It is unprotection that is an issue, right? \$\endgroup\$ – KorvinStarmast Jul 22 at 14:18
  • 1
    \$\begingroup\$ @KorvinStarmast I definitely don't take it the wrong way. It's a valid question. From the numbers I'm seeing, I'd say that a discussion about how to use both protection and unprotection would be beneficial to the site, even if the discussion technically gets more involved then is really warranted for the impact protection has. After all, it's not only about solving current problem but also looking forward to future issues. \$\endgroup\$ – Rubiksmoose Jul 22 at 15:52
  • \$\begingroup\$ OK, just tossing that out there. Thanks. \$\endgroup\$ – KorvinStarmast Jul 22 at 15:52
7
\$\begingroup\$

Use the site-wide guidelines

Protection was originally created to deal with high volume drive-by posting:

[Protection] is meant to be a reaction to persistent abuse from anonymous or unproven participants: when a page attracts a lot of noise or vandalism from outside the community, Protecting it reduces the amount of clean-up needed later on.

On our site, this is a bit of a solution looking for a problem. We don't have a problem with drive-by posting. We do not see volumes of 1 rep posting so high that we can't handle them.

When protection was introduced, an example screenshot was posted showing 10+ deleted answers by 1 rep users (most of which were 1 sentence long) on a post. An analysis of protected questions on the site showed that 99% of them had 5 or fewer deleted posts. The majority of them (66%) had 2 or fewer deleted posts. 11% of them had no deleted posts at all.

Keep in mind the warning given by former CM Josh Heyer (Shog9) who posted the site-wide guidelines:

Judicious use of this feature is critical to allowing these sites to handle large amounts of external attention, but over-using it breaks the system: Stack Exchange sites depend on a constant influx of new blood, both to answer new questions and provide updated information on old ones. When in doubt, err on the side of letting new users prove themselves before locking them out.

How do they apply to us?

The site-wide guidelines are simple, but missing a little nuance:

Do protect questions that are attracting a lot of non-answers or very poor answers (spam, etc.) from new users.

Don’t protect questions just because they’re linked to on a high-traffic news site like Reddit or Ars Technica. While there’s certainly some correlation between sudden spikes in popularity and associated non-answers, not all popular questions suffer from this.

Do unprotect questions that aren’t currently attracting a lot of attention and don’t have a long history of unproductive answers.

What constitutes "a long history"? What constitutes "currently"?

I believe the Protected queue system should inform out choices for these metrics. They currently list 2 important pieces of information: the number of answers in the past 30 days, and the number of deleted answers.

For convenience's sake, I think "currently" should mean "in the past 30 days", and "a long history" should mean "more than X deleted answers".

When determining the value of X we should consider the volume of drive-by posts our site members can handle. Currently, the queues sits at empty most of the time. In the past week, we have had 7 deleted low-quality posts. I think we have some room to set X a bit higher (10?) and see how it goes.

Keep in mind that we are not robots, so if you see a post get a lot of answers really quick, it's ok to act before it hits that threshold. Similarly, if a post has more than that number of deleted answers, but is really old, it's fine to unprotect it.

| |
\$\endgroup\$
  • 5
    \$\begingroup\$ +1. Protection damages the site by inhibiting participation. Don’t use it based on some guess there might be a problem, use it when there really is a problem. I was a mod for many years on this site and I protected maybe a handful of questions. It is generally not necessary. \$\endgroup\$ – mxyzplk - SE stop being evil Jul 28 at 13:22
  • 1
    \$\begingroup\$ As an aside, is there a reason you referenced Shog by his full legal name instead of his Stack Exchange username (Shog9)? I think 99.9% of people will not recognize him by the former, especially since he's no longer a CM/employee of the company. \$\endgroup\$ – TylerH Jul 29 at 15:09
  • 2
    \$\begingroup\$ @TylerH Official blog posts use full name rather than usernames. I don't think it particularly matters who posted the site policy since I'm certain it was discussed internally. But I will add the username too for clarity. \$\endgroup\$ – gszavae Jul 30 at 0:10
  • 2
    \$\begingroup\$ I'm not sure how/if you want to use these but on the completed feature-request to auto-protect after N low-rep answers in 1 day Shog9 said "Expand the existing heuristics to protect questions that are demonstrating problematic tendencies, but haven't yet been proven problematic" and on the feature request to allow non-diamond-mods to protect questions less than 1 day old Shog9 said: "proactively protecting questions is a stupid idea, and you should never do it even if the system does allow you to do it." \$\endgroup\$ – Medix2 Jul 30 at 0:27
  • 1
    \$\begingroup\$ @Medix2 Great links, considering there are some strong opinions to the contrary here, I think having quotes from Shog9 calling pro-active protection "stupid" or "irresponsible" is probably not the greatest idea. Great further reading though, thanks for posting them. \$\endgroup\$ – gszavae Jul 30 at 0:35
  • 2
    \$\begingroup\$ @gszavae Yeah there are quite a few feature-requests about protected questions on Meta Stack Exchange ranging from completely removing non-diamond protection, to auto-removing protection, to auto-adding protection on questions with accepted answers, to letting people request a question be unprotected... and plenty more. That second post I linked was the first time I had seen somebody stating that protection is for well-answered questions with high traffic and high attention from new users \$\endgroup\$ – Medix2 Jul 30 at 0:37
5
\$\begingroup\$

We have a high moderation capacity

One thing to remember about the protection feature is that it is a preventative measure that aims to reduce the amount of (reactive) moderation that is necessary. We have a high capacity for moderation – and most of that consists of community members using their flags and privileges. There’s very rarely more than one or two items in a given review queue, and very often nothing. And to scope out post deletion: we get north of 200 answers and 100 questions a week, and delete about 25 answers and 6 questions over the same time period on average. Therefore, we should limit the use of protection to questions that have attracted answers that had to be removed in the past and that and that are overwhelmingly likely to draw answers that will have to be removed in the future, because we have the capacity to let some things slip through.

Let’s talk about the answers from new users

Ok, I think that in order to take a look at this, we need to dissect the types of answers that come in to old questions from new users (or specifically, from accounts with <10 rep). These are gonna be broad-strokes categories, and yes, they include different categories of spam.

  • Keyword spam. This is much of the spam previously discussed. Spammers look for keywords which relate to their spam and post it in the answer box below. Keywords being in the other answers probably also attract them. Our more prominent keywords are “spellcasting”, “vampire”, “resurrection”, “death spell”. These are primarily trying to sell (scam, obviously) services offering something to the relevant keyword – say, turning you into a vampire so you can get revenge on your ex. No, I’m not kidding.
  • Network spam. This is basically keyword spam that finds keywords on other sites on the network (mainly SO), but posts it here through the vagaries of website crawling. Basically, this is “tech-y” spam that gets posted to us. This mostly happens to HNQ questions, as they’re linked from other network sites.
  • Honest spam. This is spam that is posted on a related topic by a genuine human creator, but often falls under “link-only answer” and “fails to disclose affiliation”. This is fairly low-volume.
  • Forum replies. A fair chunk of answers posted by new users are forum-like replies, which very often need to be removed. We can probably throw other honest, but (given the site format) misplaced, attempts at communicating – such as new questions posted as answers, and “thank you”s – into this bag for the discussion.
  • Actual novel/useful answers. We do get new users answering old questions with genuine answers. These are great, but from subjective experience, not too common.
  • Useful comments as answers. So that happens too. New users encounter a Q&A that is outdated (or has some other issue), and post an answer (because they can’t comment) pointing the thing out. I’ll argue that this happening is (usually) a good thing (because it gives us a chance to fix something), even if the process is awkward.
  • Screams from the void. This category is mostly on here for completeness. We get some answers posted that just don’t make sense. Standouts from the top of my head are the word “Giraffe” and a long series of “V”s, but there’s also a bunch that are just foul language, which is where my name for them originates. They don’t, as far as I can understand, follow any pattern.

How do we decide what warrants protection?

Here’s the crux of the problem as I see it. Deciding to continuously protect something is a value judgement that we, the community, have to make. We can do stats for how big the problem is, but we have to decide how much value the chance for novel answers and useful comments has, against having to moderate the other categories. Deciding to protect a question (indefinitely) is essentially deciding that the likelihood and benefit of the two useful categories is not worth the likelihood and cost of having to deal with the others, on that specific question.

So what do I think warrants continuous protection?

In short: keyword spam and high proven likelihood of forum replies, or a high volume of honest spam or screams from the void over time, with a low likelihood of novel answers and useful comments.

Factors in favor of protection

In more detail, if a question has received keyword spam in the past, it still has those keywords on it and it will be spammed again. This simply follows from the simplest assumption of how keyword spammers work. If it’s obvious why the question is pulling keyword spam, I don’t think we need to wait for two instances of keyword spam to be posted before protecting, and don’t think we should unprotect them just to wait for another spam answer to come in. If it’s not obvious, there’s minimal harm in letting the auto-protection kick in on the second. The less we have of this stuff, even in deleted form, the better. As analysis shows, this doesn’t actually affect that great a volume of questions anyway (some hundred of our 40k questions); it’ll be fine.

The second category I think we should protect against is those questions shown historically to attract many forum-like replies and likely to continue to do so. And let me be clear: the “many” is kinda important here.

These questions would most likely be those that are getting continuous views and are often fairly subjective (and thus have many people who want to voice their opinion). Why? Because if we want to retain users, having their first interaction with the site be their answer getting deleted because they didn’t know what the site was like isn’t great. Actually, it sucks. If they’re forced to poke around the site, they have (IMO) a much better chance at picking up how the site works beforehand (say, by taking the tour or something).

If a question is for some (possibly unknowable) reason drawing a lot of honest spam answers or screams from the void, we should consider protecting it. It is probably rare and strange enough that having a dedicated meta to discuss each such question is fine, in part because there may be other things we should do to the question.

Factors against protection

We shouldn’t protect a question if it is likely to benefit from novel answers and useful comments in the future. This is generally much harder to evaluate and to talk about in general. We’ll have to use our game and Q&A expertise.

The easy categories here are questions like optimization, questions trying to compile all the game features that fit X criteria, and rules questions on ambiguous rules (as they may be updated by errata or receive relevant commentary). Also questions on smaller/rarer systems. If in doubt, lean towards not protecting these.

Factors that shouldn’t count either way

Maybe it goes without saying, but I don’t think the remaining categories should be counted either way when evaluating indefinite protection. But I’d still like to make some comments about why, and this is relevant because these will be counted by the auto-protection algorithm.

  • Network spam. If there’s nothing on the question which seems to have beckoned the spam, protecting it is unlikely to prevent further spam. If a question is drawing a lot of spam of this type while on the Hot Network Questions list (HNQ), protect it while it is on HNQ, and unprotect it once it is off. Reminder: a question leaves the HNQ list if it is more than 3 days old. I don’t remember this happening with the new HNQ, but that’s how I think we should deal with it.

  • Honest spam. These don’t warrant protection. They’re one-offs, topic relevant, and half the time could be turned into valid answers if the user put in the effort.

  • One-off forum replies. If the question just gets one or two forum-like non-answers, especially while new, there’s no need for protection.

  • Screams from the void. As these are both varied and weird, they don’t really say anything about the question needing protection. They don’t really say much about anything.

The exception is obviously if there are high volumes of any of these over time, as covered above.


PS: if anyone would like me to split my suggestion from the dissection above, so we can use that more neutrally, I’d be happy to.

†: Those are two different answers. I’m sure others have more examples, but I’ll ask you not to share them in the comments here.

| |
\$\endgroup\$
  • 1
    \$\begingroup\$ I think this is a great answer (+1), but I do have a small quibble. If I had my choice I'd change that spam section to be a bit more reactive. Data shows that 58% of all spammed questions on this site remains unprotected, which also means that all of those questions only ever receive 1 spammy answer ever. Unless you argue that all 58% of those are not keyword spam, then at least some keyword spammed posts never get spammed a second time. (1/2) \$\endgroup\$ – Rubiksmoose Jul 30 at 21:14
  • 1
    \$\begingroup\$ I see no downside to waiting in that case for it to get hit a second time and automatically protected. Given the extremely low spam volumes (only 4 a month including this month with almost 0 questions protected) it shouldn't ever be an issue. I don't think it's worth a downvote or writing my own answer but I truly believe it is an overreaction given the numbers I am seeing. (2/2) \$\endgroup\$ – Rubiksmoose Jul 30 at 21:15
  • 2
    \$\begingroup\$ @Medix2 It's about community as a whole, and it might only make sense if you've spent some time looking at smaller stacks where things can take much longer to be resolved (it's important to keep in mind on certain parts of the SE documentation). And on review queues, a lot of the things they are for are done outside of it. And for mod capacity is doesn't really matter who does it as long as it gets done (community aspect and bus factor are valid concerns to that though). \$\endgroup\$ – Someone_Evil Jul 30 at 21:23
  • \$\begingroup\$ @Someone_Evil: Gotcha, gotcha, thanks for clarifying that; comment removed then \$\endgroup\$ – Medix2 Jul 30 at 21:29
  • \$\begingroup\$ I agree we should rely on moderation more, we have a huge capacity that goes unused. I would be interested to see some numbers on keyword spam. I'm not sure about the idea that once a question receives X amount of spam answers, it will continue to receive them forever at a high rate, there doesn't seem to be a lot of risk in unprotected them after some time to reassess. \$\endgroup\$ – gszavae Aug 1 at 0:38
4
\$\begingroup\$

Don't protect questions for esoteric systems

We've got several rpgs on here with almost no participation. I think I'm probably the only active user with extensive Polaris experience, for example. There was another guy who played Anima for a bit, but also one of those anti-optimization people at the same time so that's kinda dead now. I'm quite confident that nobody on this site plays Ki Khanga (that includes you BESW, but props for trying :P ).

Questions asked in these systems represent a huge opportunity for this site to slightly correct its awful and overwhelming 5e bias. They also represent an area where people coming to the site randomly from Google are most often a better source of answers than we are-- we don't play these games, and the internet does because the internet does everything. Protecting these questions does the most damage to the site that protection is capable of, because the new users prevented from answering are the new users whose one-time participation most helps the site, and furthermore whose long-term participation best contributes to the growth of the site.

What's an esoteric system is definitely a grey area, though. I'd argue, for example, that OD&D counts, barely: although we have a bit of a healthy community here for that, it's very small. Also The Dark Eye, Polaris, FATE but not Fate, maybe Dresden Files but that's a bit of a stretch, Phoenix Command, Amber Diceless, anything less common than these. What seems clear to me doesn't count is D&D 3.x,4,5, Dungeon&Apocalypse World at the very least of the pbta games, Fate Core, stuff we regularly see on the front page and gets answered, like, immediately with at least two different active high-rep users. That's just my thoughts on it, though, I think we need to decide for ourselves when our userbase's lack of expertise is a good justification for higher mantainance on a question.

| |
\$\endgroup\$
  • 1
    \$\begingroup\$ As much as I am uninterested in AD&D (5e or otherwise), I think calling the site's demographic an 'awful' bias is a kind of zero-sum (or even negative-sum maybe) thinking that is needlessly harsh towards those who do come here with questions and/or answers for this system. I do agree with your objection to the system-assumption in the first part of the post though. \$\endgroup\$ – vicky_molokh- unsilence Monica Jul 30 at 17:50
  • 1
    \$\begingroup\$ I'm going to have to disagree with your first point. I'm not sure I see wrong-system answers as a special case that protection could help with distinct from other VLQ answers. From what I can remember many wrong-system answers are from users that have over 10 rep, so protection wouldn't help at all in those cases and that should be specified. Also I'm not sure I've seen many answers that have a rash of wrong-system answers, and I'd personally like to see at least a couple qualifying bad answers before something is protected. \$\endgroup\$ – Rubiksmoose Jul 30 at 20:15
  • 3
    \$\begingroup\$ Your second point is very good though. We should be very careful about protecting those less common systems, if we should even protect them at all and that was something that had never occurred to me. \$\endgroup\$ – Rubiksmoose Jul 30 at 20:16
  • \$\begingroup\$ @vicky_molokh-unsilenceMonica I don't think the people or demographic or whatever is awful, I just think we have an awful big bias in that direction. \$\endgroup\$ – Please stop being evil Jul 30 at 23:11
  • \$\begingroup\$ Even if this is a huge problem, I don't think we need a specific rule for this. Off-topic answers are by definition VLQ and should lead to deletion. If the overall policy is "protect when there are too many VLQ from new users to handle", then we don't need a policy for "protect when there are too many VLQ, of one specific type, from new users to handle." I also think we should be careful with hostility (or things that can be perceived as hostility) towards 5e - so perhaps avoid phrases like "awful bias". \$\endgroup\$ – gszavae Jul 31 at 1:31
  • \$\begingroup\$ @Rubiksmoose Here is an example: My Druid wants to transform into a blink dog. How do I rule this? \$\endgroup\$ – Please stop being evil Jul 31 at 1:48
  • \$\begingroup\$ @gszavae We aren't making a policy like that in any case. We are providing guidelines for what to look for in terms of when protection should be used. Specific, easily applied criteria (like a subset of VLQ answers) are more helpful than a general stance on VLQ answers. Which, I would argue, is bad advice anyways because the kind of VLQ answer does, in fact, matter in terms of if we should protect a question-- kinds that tend to symptomatic of a repeating problem should be protected more than kinds that don't tend to go that way. \$\endgroup\$ – Please stop being evil Jul 31 at 1:51
  • 2
    \$\begingroup\$ @Pleasestopbeingevil It may be more useful to discuss types of VLQ answers that don't matter then? It's hard for me to imagine a case where we see a lot of VLQ answers from low rep users but decide not to act. I would be interested to hear more about that situation. Ultimately you do hit the nail on the head with pointing out the value of new users. \$\endgroup\$ – gszavae Jul 31 at 1:55
  • \$\begingroup\$ @gszavae Well, if we get to 3 the automated system kicks in, so... \$\endgroup\$ – Please stop being evil Jul 31 at 2:02
  • 1
    \$\begingroup\$ @Pleasestopbeingevil Sorry, I'm not sure what your point is. Note that we can modify the system, and we can unprotect and ignore it. \$\endgroup\$ – gszavae Jul 31 at 2:13
  • 1
    \$\begingroup\$ @Pleasestopbeingevil I hope I was clear enough. I do understand that this happens and it definitely shouldn't and they should be removed. But what I was saying though was that I'm not sure it is a problem specific to new users and that this specific tool would be well suited to combating it apart from more general guidelines. (For example, you example Q shows two wrong system answers, only one of which protection would have helped with). Luckily I think the guidelines provided by at least two other answers would cover a new user wrong system answer as well as other types of VLQ posts. \$\endgroup\$ – Rubiksmoose Aug 2 at 4:54
  • 1
    \$\begingroup\$ @Rubiksmoose It's true to my experience that we get a lot of these from newish users with more than 10 rep (as well, though far fewer even per capita, as some higher rep users), I think that 1 rep users are still disproportionately likely to do this, but I'll remove it from the answer since it's not a big enough deal above and beyond normal VLQ protections to talk about right now. The main reason to distinguish it, I think, is that it's useful for telling when to ignore the advice in the second part of the answer, but that can be redressed in the future. \$\endgroup\$ – Please stop being evil Aug 2 at 6:12
3
\$\begingroup\$

Summarizing criteria that I find relevant

Things that are a +1 point in Protecting a question:

  • The question already has an accepted answer, that is most likely up-to-date.
  • The question already has a highly upvoted answer, that is most likely up-to-date.
  • The question already has 2 or more deleted answers from low (<100) rep users (I'm using <100 instead of <10 because the user may have gotten some rep elsewhere and I find the behavior overall similar)
  • The deleted answers are comments, not answers, or answers that don't add anything new to what has already been discussed in other answers.

Things that are a +1 point in Unprotecting or Not Protecting:

  • The question doesn't have any well accepted (either by community or by asker) answer.
  • The question is about a not-so-popular game system, thus bringing more experts in that game system to our site would be beneficial.
  • The question had a spike of activity for some reason and that spike is now gone. For example, you can check the timestamp in the low quality answers. If all of them are pretty much clustered together, it may have been such a spike. If it has 3 bad answers, each separated by 1 year, then bad answers are likely to keep coming if we unprotect.

For what is a well accepted answer: usually, answers have considerably more votes than questions (if anyone can help with the data here I'll edit, but I'm 99% certain this is true, and I will guess by at least twice the votes), so, if the most upvoted answer doesn't have half the score of the question, I would say that question would benefit from new answers. But use your own judgement on it.

We don't get too much spam/poor quality answers from 1 rep users

Okay, let us start with stating my experience on it: we don't get much spam. Most of our questions don't have a deleted answer or worse, a spam answer. For that reason, protection should be using sparingly. That said, let me give my 50 cents on some of the discussion topics you have suggested.

How do we best identify questions that have attracted spam answers that could benefit from protection?

On the run, and in a reactive way. I've recently protected this question because it got a new, considerably low quality answer from a 1 rep user, and we had two deleted answers on it already.

My suggested approach is simply: you are going through the review queue, you notice a bad answer from a new user, you take a look in the question. Does it already have other similar bad answers from new users? If it does, go on and protect it. There are more criteria which I will talk about later. Stay with me please. But the point is that this should be reactive. Protecting before the question has had any kind of problem is, for me, a no-no. Doesn't matter how much it looks like it is going to be a problem, the frequency of this problem is so low here that we can afford being reactive instead of preventive.

Does HNQ (Hot Network Questions) status have any effect?

This is an important question, and my answer is: I don't know. Most likely people coming from HNQ will have the 101 starting rep anyway, so I'm not sure it matters at all. HNQ seems to inflate votes, some not-too-useful comments, but personally I haven't experienced a huge problem with HNQ leading to a bunch of bad new answers. My current inclination is that no, it doesn't have any effect. If HNQ brings up a problem, we can see the symptoms of the problem, and protect based on the symptoms, making the fact that it is in HNQ is meaningless. Again: reactive stance.

Are there any other types of issues that protection should be used to prevent?

This is included in "low quality answers", but I personally feel like the most common problem with 1 rep users are answers that are essentially comments. For me, questions that attract such comment-answers are the highest priority for protection.

If a question has been protected in the past, should that have an impact on whether or not it should be protected again?

No. But the fact that the question probably already has 2+ deleted bad answers should, so there is a correlation here. But simply the fact that it has been protected in the past shouldn't.

Are there any specific circumstances that we should define in which it isn't beneficial to protect a question?

The scenarios I have described in my summary. If a question has no good answers yet, then we would certainly want a user that can answer that question answering it, no matter if they are new to the website. Similarly, if the question is for something not too popular, I think it is more beneficial to take care in protecting it, since we may actually bring an expert from a obscure system, and they may be the only expert we are getting any time soon. PS: I say that as a D&D5e-user. Don't take it personally, fellows.

When should we unprotect questions that have been automatically protected?

Most likely never, unless we can identify that the automatic protection was linked to some extraordinary temporary event (idk, the question being posted in Kotaku or 9gag or another big communication vehicle) and such event has since then vanished.

About manually protected questions and/or scheduling check-ins, I have no formed opinion.

| |
\$\endgroup\$
  • 1
    \$\begingroup\$ I'll agree that we currently don't get much spam, but can you support a bit why you dismiss VLQ answers by new users? I think this issue is magnitudes greater than spam though I haven't thought of a way to gather data yet. However, I can tell you that already we've had 5 VLQ answers from new users in 21 days just from the questions that recently got unprotected. That tells me that it might be more of an issue then this answer is letting on. What do you think? \$\endgroup\$ – Rubiksmoose Jul 21 at 22:46
  • 2
    \$\begingroup\$ @Rubiksmoose I don't dismiss them. To clarify: I think a preventive stance would only be justified if we got lots of spam. 5 VLQ in 21 days seems a low number, enough that we can afford to be reactive to it, instead of trying to prevent it. \$\endgroup\$ – HellSaint Jul 21 at 22:57
  • 1
    \$\begingroup\$ Firstly, sorry for misrepresenting your position it was not intentional! Secondly, note that that number was only counting VLQ questions from new users on those specific 700 posts in the last 21 days. I'm pretty sure we've gotten more than 5 VLQ new answers today from new users and that is pretty normal. Not that that actually invalidates your overall point. \$\endgroup\$ – Rubiksmoose Jul 21 at 23:13
  • \$\begingroup\$ @Rubiksmoose I have went through the list of recent answers and opened the negative voted answers. We had 3 in the last 2 days, although this doesn't count the deleted ones. I went on and protected this one as well because, well, it fits my criteria. Is there a way to filter for deleted answers in the Search? \$\endgroup\$ – HellSaint Jul 21 at 23:20
  • \$\begingroup\$ unfortunately the deleted search modifier only works for mods when searching for posts other than your own :-/ I was able to find 7 posts (pic) with -1 or lower score deleted in the last 24-48 hours. 2 of those answers were from users with > 10 site rep. The rest were VLQ from new users. \$\endgroup\$ – Rubiksmoose Jul 21 at 23:49
  • \$\begingroup\$ @Rubiksmoose I recognize 3 of these as higher rep users, at least. With the very little data we gathered manually, it feels like (at least if the voting system accurately describes the quality of an answer) we get as many answers from new users as from regular users. \$\endgroup\$ – HellSaint Jul 21 at 23:52
  • 1
    \$\begingroup\$ Nope, only 2 of those were greater then 10 rep. Just double checked. I guess my point is though that if we assume an average of 2 VLQ new user posts a day (I moved it a bit lower to account for low times during the weekend and stuff) that is still somewhere around 730 posts a year that need active moderation by the community which is not trivial effort-wise. I do agree with your main thrust. I think we should tend to be on the reactive side. I like new users, I wish they would contribute more quality stuff and protection stops that. But I still think we get more VLQ then we could. \$\endgroup\$ – Rubiksmoose Jul 21 at 23:58
  • 7
    \$\begingroup\$ "Most likely people coming from HNQ will have the 101 starting rep anyway" - Just FYI starting rep does not count towards answering protected questions. You must have earned at least 10 rep on this site to answer. \$\endgroup\$ – linksassin Jul 22 at 0:17
  • \$\begingroup\$ @linksassin Oh, that's something I didn't know. That may change my position on HNQ - although I would keep the idea of reactive, it may be a good idea to pull the trigger faster if the question is showing abnormal behavior and is on HNQ. Hm. \$\endgroup\$ – HellSaint Jul 22 at 0:30
  • 2
    \$\begingroup\$ One point of note, due to changes to the requirements for protection a while back, protection now has to be done reactively. A question cannot be protected if it's less than 24 hours old (Community may be able to bypass this restriction), and it cannot be protected unless it has at least one answer (deleted or not) from a low-rep (<10) user. \$\endgroup\$ – Oblivious Sage Jul 23 at 13:04
  • 2
    \$\begingroup\$ @ObliviousSage For more information on that there is this: "Why can only moderators protect questions less than 1 day old?" The currently highest scoring answer states: "proactively protecting questions is a stupid idea, and you should never do it even if the system does allow you to do it." which is also quoted in this feature-request to completely remove the ability for non-diamond moderators to protect questions at all \$\endgroup\$ – Medix2 Jul 23 at 15:14
  • \$\begingroup\$ @Rubiksmoose to clarify: "we've had 5 VLQ answers from new users in 21 days just from the questions that recently got unprotected" - out of those 5: only 4 were made by new users (the other user had >400 rep), 2 were questions in answers, the other 2 were valid answers. In 4 weeks we have seen 9 answers, although only 6 by new users, out of those 6 only 1 was a non-answer while the others were answers of varying quality. I'm not sure what the criteria for VLQ is, but hopefully that breakdown gives a little more nuance. \$\endgroup\$ – gszavae Jul 29 at 4:43
  • \$\begingroup\$ I disagree with the first two points. There is no reason to protect a question just because it has an accepted or highly upvoted answer. Protecting such questions encourages insular/right-think behavior in that it encourages "one true answer". I have seen that, even for questions asking for a rule, there can often be multiple valuable answers providing different perspectives/alternative solutions in addition to just spitting out what the rulebook might say. But the vast majority of questions on a site like RPG.SE are not ones where "one answer" is plenty. \$\endgroup\$ – TylerH Jul 29 at 15:05
  • \$\begingroup\$ @TylerH Sure, and you can't protect it just because it has an accepted or highly upvoted answer. Unless you are a mod, but then I trust you, as a mod, is able to make a fair judgment on yourself. What I am saying is that this contributes in favor of protecting it, as there is little reason to believe the question needs new answers. \$\endgroup\$ – HellSaint Jul 29 at 16:47
  • \$\begingroup\$ FWIW there actually is a feature-request to auto-protect old questions that have accepted answers. Not that it says much (and I can't fetch vote totals) but it does exist \$\endgroup\$ – Medix2 Jul 29 at 16:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .