The current policies are inadequate
After previously working through the protected questions backlog, clearing out any questions that remain protected despite no longer needing to (as per site and stack policy), I discovered that the current policy is quite controversial.
- Do protect questions that are attracting a lot of non-answers or very poor answers (spam, etc.) from new users.
- Don't protect questions just because they're linked to on Hot Network Questions or a high-traffic news site.
- Do unprotect questions that aren't currently attracting a lot of attention and don’t have a long history of unproductive answers.
Our site deviates from these established conventions, instead, we have the following site policy:
Only protect questions when they have "gone wrong." Usually that's some lurid question that is bringing in loads of crap answers from new users seeing it in "hot network questions" or something and coming and dumping junk.
Other than that, don't protect, and we'll quickly unprotect questions that have protection for no clear compelling reason.
The current policies are ill-defined
The reasons for controversy seem to stem from difficulty in quantifying certain terms, these terms are:
- "a lot of"
- "a long history"
- "gone wrong"
- "clear compelling reason"
I interpreted these to mean essentially "more than we can handle with normal processes", be that a volume that is too large, or answers that are too damaging to allow to stay around before our deletion turnaround. To me, 0, 1 or even 2 deleted answers is not enough to constitute a lot of deleted answers, nor a long history, and certainly not a compelling reason. I fully accept that not everyone has this interpretation.
Oblivious Sage said the following of their methods:
My own criteria for protecting questions is similar to the Community criteria, merely tighter. Where Community looks for 3 deleted answers from low-rep users, with spam counting double, my own threshold is essentially 2 deleted answers from low-rep users, with spam counting double (that is, I immediately protect a question once it gets a spam answer[...])
doppelgreener describes their method as:
Myself, I just protect stuff that's accumulating one of our typical spam patterns (witch doctors, vampires, magic spells, etc) because the spammers historically revisit the same questions with special keywords repeatedly and ignore all the others. I also protect questions that are attracting a lot of bad attention, or highly popular questions which over several months or years have kept accumulating questions-in-answers from people who think this place works like a forum.
Even considering just 3 users, we have vastly different interpretations of the guidelines. Neither of them stated they would unprotected questions, in fact only 15% of all protected questions were ever unprotected. Clearly this ambiguity needs to be addressed.
The state of protected questions
In the protection number-crunching thread Medix2 calculated the following:
Using that data then (small ±1 errors may have happened): 242/834 (29.0%) have one deleted answer. 277/834 (33.2%) have two deleted answers. 167/834 (20.0%) have three deleted answers. 84/834 (10.1%) have four deleted answers. 35/834 (4.2%) have five deleted answers. 15/834 (1.8%) have six deleted answers. 13/834 (1.6%) have seven or more deleted answers.
That leaves some 109 questions which had no deleted answers.
However, Oblivious Sage notes:
It would be ideal if you could limit it to only deleted answers from low-rep users, but since that's apparently not an option this query will suffice. It's interesting to note that on most of the questions with a lot of deleted answers, the majority of the deleted answers come from non-low-rep (>100) users, and were deleted due to violating various policies (back up subjective answers with experience, don't advocate illegal activity or link to sites that engage in it, etc.), rather than anything that would get them auto-protected.
Looking at how long ago these answers had been protected:
1 had been protected in the last month, 33 had been protected less than 6 months ago, 94 had been protected less than a year ago.
195 had been protected 1-2 years ago, 216 2-3 years ago, 87 3-4 years ago, the remaining 128 were 4+ years
Keeping this in mind, in my opinion, it isn't clear that we have ever had more than a handful of questions that warranted protection. I see very little evidence of a volume of spam that is so high that we can't handle it through normal means like downvoting, flags, and deletion.
The way forward
What kind of policy should the site adopt in the future? How and when should we protect questions, and how and when should we unprotect them?
The following questions may help with discussion:
- What is the purpose of protection on this site?
- At what volume of spam should an answer be protected?
- Should we pre-emptively protect answers?
- Should we leave questions protected forever?
- Should questions be reassessed after some time?
- Should we fully repurpose protection for other goals (protecting questions with spam keywords for example)?
