How has RPG.se used protection? (crunching the numbers on spam and protection)

Question

Very soon we intend to start a discussion about how protection is used on this site going forward. To inform that discussion, I've tried to dig deep and find some data that could help guide this discussion. It got so lengthy that I decided that posting it as its own post was probably the best way to go to get the most eyes on it.

Note that I am not a SEDE wizard. I have tried to verify the results for all the SEDE queries as best as I can, but it is entirely possible that somethings have slipped through. I'd appreciate any extra eyes on them at all and any suggestions for corrections or improvements. If you think additional queries would be helpful, drop a comment and I'll do my best to add them.

With that out of the way, here's more than you ever wanted to know about spam and protection on RPG.se. I hope it will be helpful.

Feel free to use the answers to point out anything interesting or to dig in deeper.

---

What is a protected question?

Luckily there is a Meta.SE post that goes into great detail.

To quote the essential part from it though:

all users must have earned 10 or more reputation on that specific site to answer a protected question.

The other important thing to note is when questions are automatically protected by the community user:

1. The number of deleted answers from users with <10 rep, plus the number of answers with helpful spam flags, is at least 3. (Note that spam answers from new users are counted twice.)
   • Generally, this means that three deleted answers from new users will cause auto-protection, but if at least one of those answers is spam, only two answers will trigger it.
2. Five answers from users with <10 rep were posted in the last 24 hours. [...]

The system will never unprotect a question automatically, even if the deleted answers are later undeleted or the spam flags are cleared.

---

Spam

Over the history of the site (~10 years) we have had 495 questions with at least one answer with a spam flag on it and 637 questions that have been marked as spam. That averages out to 113 posts that attract spam a year and 9 posts a month.¹

We don't really have any tools to prevent question spam, so if we just look at the answer spam (which we at least have some tools to prevent) the average is about 4 maybe-preventable spam posts a month.

Looking at 2019, we had 127 questions with spam answers and 43 spam questions for an average of 14 spam posts per month (11 spam answers per month). (2020 is so far tracking a bit lower with about 11 total spam posts/month)

Spam protection

Out of all these questions that have received spam² only 212 have ever been protected (so 58% of posts which receive spam are never protected).

Speaking of protection, over the history of the site we have had 943 questions ever be protected (out of ~40,300 currently undeleted questions, a bit over 2%). Given the previous data, this would mean that only 212/943 = 22% of protection activity involves spam.

Of all the protected questions, 149 questions, (15%) ever got unprotected (not counting the recent mass-unprotection event which will likely be reversed).³

Protection

~50% of all non-automatic protections on this site have been carried out by 1 user and the top 3 users combined make up 71% (the third user hasn't even been active on the site for years).

136 questions have been automatically protected in the past (14% of all protections). Of those, 25 posts have ever been unprotected.

Of all the 136 automatically protected questions, 79 (or 58%) had at least one spam answer on them.

Out of the 943 questions that have ever been protected, 834 have at least 1 deleted answer (88%). Out of all the questions that got unprotected, 130 have at least 1 deleted answer. 31 of the questions that got unprotected had at least one spam answer on them.

Since Feb. 28, 2019 (which is when the HNQ status started being tracked), 51 questions have hit HNQ and also been protected at some point. This represents ~2% of the 3052 HNQs we’ve had over that same period of time. 12 of these protections were done automatically by the community user.

---

^{1 - These numbers include 2018 where we went through an enormous spam wave, so this number is actually lower if we remove that outlier. If we exclude 2018 from our averaging we’d get 153 spam questions and 381 spam questions over 9 years, so 60 posts a year and 5 a month.}

^{2 - I’m not counting spam questions here because spam questions get deleted entirely, and are never protected.}

^{3 - Per Should we reprotect the 700 or so questions that were unprotected in the last few weeks?. For the full list of questions that were unprotected see this Google doc.}

Answer 1

Spam prevention doesn't seem to be the main use for protection (volume-wise)

Given the previous data, this would mean that only 212/943 = 22% of protection activity involves spam.

58% of posts which receive spam are never protected

This tells me that spam has been not the main use of protection for us in the past. In fact, it seems like a very minor or at least inconsistent use for it. Since we've only ever had 79 automatically protected questions with spam on it, this also tells me that this 58% of unprotected spam doesn't seem to be attracting a huge volume of repeat offenders. (If there was more than one deleted spam post, it would get automatically protected).

It is possible that we are just very good at picking out which spam posts will draw more spam and that could account for some of the low usage (and maybe even some of the 31 questions that got unprotected, even with spam on it).

This doesn't mean that we shouldn't protect spammy posts of course, but we should be aware that at our current very low usage of protection for spam, we still only have around 4 spammy answers a month. A very manageable amount.

Going forward in our discussions, I think this means that I would recommend us not putting major focus on increasing our coverage of spam questions since our current usage of this seems to be sufficient at keeping spam low.

Unprotecting questions is rare (but it shouldn't be)

To quote doppelgreener's meta answer here,

As BESW summed up in chat, questions tend to get protected for one of three reasons: (1) a question hits HNQ and attracts nonsense, (2) a question is controversial and needs a bit of brakes applied, and (3) a spam algorithm has identified that question as a target via keywords.

Assuming this is true (and both BESW and doppelgreener are certainly experienced enough users to have developed a feel for this), somewhere around 2/3 of our main uses of protection are explicitly temporary. HNQ only lasts 72 hours maximum and thus protection for this reason need not exceed that amount of time. Controversial questions are often the same, when the question becomes less active, the need for protection fades quickly away since the volume of posts slows to a trickle or stops completely.

However:

Of all the protected questions, 149 questions, (15%) ever got unprotected

Ideally this number should be higher following from the argument above. But that's not really entirely our fault. There is no timed or conditional modes for protection; it stays until it is sought out and removed by a user with enough rep to do so. Without any kind of notification or prompting to unprotect, it is unsurprising that it rarely gets done. Especially since the users that have the rep to do so are entirely unaffected by the protection.

We do need to remember that protection completely shuts out the potential for a new user to answer that question unless they receive an upvote somewhere else on the site first. We should be reducing the times this tool is used unnecessarily to reduce the barriers for new users.

Going forward, given the low rate of unprotection I think we should make a major point to try to figure out as a site how we can minimize questions getting and staying protected when they do not need to be.

Protections are carried out by only a handful of users

This whole discussion was kicked off by one user who unprotected 99% of our questions. However, it should be noted that ~ 50% of those protections were carried out by a single user as well.

~50% of all non-automatic protections on this site have been carried out by 1 user.

To be clear, there's nothing wrong with that user doing the protections. And there is a clear difference between the two cases because this user did these protections over the course of years as opposed to the unprotections which were done over the course of a couple hours.

But I bring it up because I think it shows how little transparency there is around this whole mechanic. The vast majority of users never even interact with it at all (except an unknowable number of 1-rep users and spammers who get blocked from answering).

I think this means that most of the community has not interacted with this tool and also likely doesn't have a full understanding how and when and when not to use. Setting up some community guidelines to further understand and align the community's use of this tool would therefore be beneficial.

Answer 2

We have to unprotect things intelligently

Out of the ~700 recently unprotected questions, 5 have gotten answers in just 21 days. All 5 of those answers have since been deleted and/or are at -1 or lower. This works out very roughly to around 2 Very Low Quality (VLQ) answers per week for this group of posts. All of these posts would have been prevented by protection.

Not including those 5, 54 questions have ever gotten an answer after being unprotected. This works out — again very roughly — to 0.1 answer per week for all of these. And I have no idea of the quality of these answers, (some of them could be quite good) or if protection would have even prevented them.

Even if we assume all of the 54 answers are junk though, the rate at which VLQ answers have come in from the recently unprotected question is magnitudes higher.

This tells me that selective unprotection, unsurprisingly, is the smart way to go about things here. Mass unprotection without consideration of post history will lead to more junk answers and more moderation needed by the community.

Answer 3

I just wanted to extract a few replies from the comments above:

From Medix2's comment on the number of deleted answers per protected question (note: this is all-time, not just the 700 recently unprotected questions)

• 11% had no deleted answers
• 26% had 1 deleted answer
• 29% had 2 deleted answers
• 18% had 3 deleted answers
• 9% had 4 deleted answers
• 4% had 5 deleted answers
• 1% had 6 or more deleted answers

From my comment on how old protected questions were (note: this is only for the 700 recently unprotected questions):

• 5% had been protected less than 6 months ago
• 9% had been protected 6 months to a year ago
• 28% had been protected 1-2 years ago
• 30% had been protected 2-3 years ago
• 12% had been protected 3-4 years ago
• 18% had been protected 4+ years ago

In the last 21 days, 5 recently unprotected questions (as calculated by Rubiksmoose) have received answers. The number of deleted answers from low-rep users that the question had before being unprotected is listed below, along with the total number of deleted answers, and the age of the question:

• 1 deleted answer from a low rep user, 2 total deleted answers in 5 years
• 1 deleted answer from a low rep user, 2 total deleted answers in 3 years
• 3 deleted answers from low rep users, 3 total deleted answers in 3 years*
• 0 deleted answers from low rep users, 0 total deleted answers in 3 years
• 2 deleted answers from low rep users, 2 total deleted answers in 6 years

*Note: The user who answered this question has over 100 rep, so protection would not have made a difference in this case.

Also note that so far none of the answers on the 5 unprotected questions that have received answers in the last 21 days, have been spam (2 were questions-in-answers, 3 were valid answers - it's worth pointing out that one is quite a lengthy and thought out answer).

Update: It has now been 4 weeks since the unprotection. 9 questions have been answered (including 3 by users with >10 rep):

• Q: recent answerer had >400 rep (non-answer), 1 deleted answer from a low rep user, 1 previously deleted answer (by a spambot) in 2 years
• Q: recent answerer had 1 rep (low-quality answer), 2 previously deleted answers (deleted account low quality answer and by a +20k rep user) in 5 years
• Q: recent answerer had 1 rep (legitimate answer), 3 previously deleted answers (deleted account comment-in-answer, 1 rep low-quality answer, +100k rep user) in 5 years
• Q: recent answerer had >3k rep (legitimate answer), 2 deleted answers (both low quality answers, 1 rep user and deleted user) in 5 years
• Q: recent answerer had 1 rep (comment/question in answer), 2 previously deleted answers (comment in answer by 1 rep user, +30k rep user) in 3 years
• Q: recent answerer had 1 rep (low quality answer), 2 deleted answers (both 1 rep questions-in-answers) in 3 years
• Q: recent answerer had >100 rep (legitimate answer), 3 deleted answers (1 deleted account, 2x 1 rep accounts, 2x low quality answers and 1 mistake that as deleted) in 3 years
• Q: recent answerer had 1 rep (legitimate answer), no deleted answers in 3 years
• Q: recent answerer had 1 rep (low-quality answer), 2 deleted answers (low-quality answer by deleted user, spam by 1 rep user) in 6 years

Total: 6/9 new answers were made by 1 rep users (that would have been prevented by protection). 0/6 of these were spam, 1/6 was a question in answers, 3/6 were low-quality answers, 2/3 were legitimate answers.

These numbers are so low that it's hard to gauge the effects. Approximately 0.9% of unprotected questions received an answer that they otherwise would not have. None of the questions received any spam (0% spam rate), all answers appear to have been made by legitimate, real users, trying to answer the question.