We have a high moderation capacity

One thing to remember about the protection feature is that it is a preventative measure that aims to reduce the amount of (reactive) moderation that is necessary. We have a high capacity for moderation – and most of that consists of community members using their flags and privileges. There’s very rarely more than one or two items in a given review queue, and very often nothing. And to scope out post deletion: we get north of 200 answers and 100 questions a week, and delete about 25 answers and 6 questions over the same time period on average. Therefore, we should limit the use of protection to questions that have attracted answers that had to be removed in the past and that and that are overwhelmingly likely to draw answers that will have to be removed in the future, because we have the capacity to let some things slip through.

Let’s talk about the answers from new users

Ok, I think that in order to take a look at this, we need to dissect the types of answers that come in to old questions from new users (or specifically, from accounts with <10 rep). These are gonna be broad-strokes categories, and yes, they include different categories of spam.

• Keyword spam. This is much of the spam previously discussed. Spammers look for keywords which relate to their spam and post it in the answer box below. Keywords being in the other answers probably also attract them. Our more prominent keywords are “spellcasting”, “vampire”, “resurrection”, “death spell”. These are primarily trying to sell (scam, obviously) services offering something to the relevant keyword – say, turning you into a vampire so you can get revenge on your ex. No, I’m not kidding.
• Network spam. This is basically keyword spam that finds keywords on other sites on the network (mainly SO), but posts it here through the vagaries of website crawling. Basically, this is “tech-y” spam that gets posted to us. This mostly happens to HNQ questions, as they’re linked from other network sites.
• Honest spam. This is spam that is posted on a related topic by a genuine human creator, but often falls under “link-only answer” and “fails to disclose affiliation”. This is fairly low-volume.
• Forum replies. A fair chunk of answers posted by new users are forum-like replies, which very often need to be removed. We can probably throw other honest, but (given the site format) misplaced, attempts at communicating – such as new questions posted as answers, and “thank you”s – into this bag for the discussion.
• Actual novel/useful answers. We do get new users answering old questions with genuine answers. These are great, but from subjective experience, not too common.
• Useful comments as answers. So that happens too. New users encounter a Q&A that is outdated (or has some other issue), and post an answer (because they can’t comment) pointing the thing out. I’ll argue that this happening is (usually) a good thing (because it gives us a chance to fix something), even if the process is awkward.
• Screams from the void. This category is mostly on here for completeness. We get some answers posted that just don’t make sense. Standouts from the top of my head are the word “Giraffe” and a long series of “V”s,^† but there’s also a bunch that are just foul language, which is where my name for them originates. They don’t, as far as I can understand, follow any pattern.

How do we decide what warrants protection?

Here’s the crux of the problem as I see it. Deciding to continuously protect something is a value judgement that we, the community, have to make. We can do stats for how big the problem is, but we have to decide how much value the chance for novel answers and useful comments has, against having to moderate the other categories. Deciding to protect a question (indefinitely) is essentially deciding that the likelihood and benefit of the two useful categories is not worth the likelihood and cost of having to deal with the others, on that specific question.

So what do I think warrants continuous protection?

In short: keyword spam and high proven likelihood of forum replies, or a high volume of honest spam or screams from the void over time, with a low likelihood of novel answers and useful comments.

Factors in favor of protection

In more detail, if a question has received keyword spam in the past, it still has those keywords on it and it will be spammed again. This simply follows from the simplest assumption of how keyword spammers work. If it’s obvious why the question is pulling keyword spam, I don’t think we need to wait for two instances of keyword spam to be posted before protecting, and don’t think we should unprotect them just to wait for another spam answer to come in. If it’s not obvious, there’s minimal harm in letting the auto-protection kick in on the second. The less we have of this stuff, even in deleted form, the better. As analysis shows, this doesn’t actually affect that great a volume of questions anyway (some hundred of our 40k questions); it’ll be fine.

The second category I think we should protect against is those questions shown historically to attract many forum-like replies and likely to continue to do so. And let me be clear: the “many” is kinda important here.

These questions would most likely be those that are getting continuous views and are often fairly subjective (and thus have many people who want to voice their opinion). Why? Because if we want to retain users, having their first interaction with the site be their answer getting deleted because they didn’t know what the site was like isn’t great. Actually, it sucks. If they’re forced to poke around the site, they have (IMO) a much better chance at picking up how the site works beforehand (say, by taking the tour or something).

If a question is for some (possibly unknowable) reason drawing a lot of honest spam answers or screams from the void, we should consider protecting it. It is probably rare and strange enough that having a dedicated meta to discuss each such question is fine, in part because there may be other things we should do to the question.

Factors against protection

We shouldn’t protect a question if it is likely to benefit from novel answers and useful comments in the future. This is generally much harder to evaluate and to talk about in general. We’ll have to use our game and Q&A expertise.

The easy categories here are questions like optimization, questions trying to compile all the game features that fit X criteria, and rules questions on ambiguous rules (as they may be updated by errata or receive relevant commentary). Also questions on smaller/rarer systems. If in doubt, lean towards not protecting these.

Factors that shouldn’t count either way

Maybe it goes without saying, but I don’t think the remaining categories should be counted either way when evaluating indefinite protection. But I’d still like to make some comments about why, and this is relevant because these will be counted by the auto-protection algorithm.

• Network spam. If there’s nothing on the question which seems to have beckoned the spam, protecting it is unlikely to prevent further spam. If a question is drawing a lot of spam of this type while on the Hot Network Questions list (HNQ), protect it while it is on HNQ, and unprotect it once it is off. Reminder: a question leaves the HNQ list if it is more than 3 days old. I don’t remember this happening with the new HNQ, but that’s how I think we should deal with it.

• Honest spam. These don’t warrant protection. They’re one-offs, topic relevant, and half the time could be turned into valid answers if the user put in the effort.

• One-off forum replies. If the question just gets one or two forum-like non-answers, especially while new, there’s no need for protection.

• Screams from the void. As these are both varied and weird, they don’t really say anything about the question needing protection. They don’t really say much about anything.

The exception is obviously if there are high volumes of any of these over time, as covered above.

PS: if anyone would like me to split my suggestion from the dissection above, so we can use that more neutrally, I’d be happy to.

†: Those are two different answers. I’m sure others have more examples, but I’ll ask you not to share them in the comments here.