Community self-evaluation: How are we doing with game recommendations?

Question

Our game recommendation guidelines are four years old this month. It's worth taking some time to reflect and review as a community how well they are working.

What works well? What doesn't? Have they achieved the goals we set for them? Is there any way our guidelines could be improved, in expression or in execution? Is the community satisfied with moderation around game-rec questions and answers? Is the community satisfied with the quality of game-rec questions and answers?

Any thoughts on the state and evolution of game recommendations are welcome.

Answer 1

Our guidelines serve two masters, and neither well:

1. Helping the asker find a game or games that will suit what they are aiming to do.
2. Ensuring that we can meaningfully judge and vote on the answers.

What I've observed is that how well it satisfies one goal is inversely proportional to how well it satisfies the other: the more perfectly answers follow the guidelines the less useful they are to the asker; the more useful to the asker, the less the answers follow the guidelines.

I believe this lies at the core of why the guidelines are no longer being followed — naturally, we are here to help people, and we will tend over time toward bending the rules in order to best help people.

Why are they inversely related? Because a helpful answer is timely and novel to the asker, while answers that correctly follow the guidelines (of a question that correctly follows the guidelines) are slow to arrive and derivative of requirements the asker can already imagine.

A brief history of game-rec guidelines

The guidelines were created because we found that the voting system didn't work — every answer was equally valid, which is the first reason our help centre mentions for why questions get closed:

Help Center > Asking >
What types of questions should I avoid asking?

[…] To prevent your question from being flagged and possibly removed, avoid asking subjective questions where …

• every answer is equally valid

Once we realised that recommendation questions were exactly off-topic, we had a choice: shrug and let them be closed as off-topic, or figure out how to make them not match a basic close reason. We chose the latter, and we forged some rules that would ensure that answers would not all be equally valid. To ensure that, the rules required that people refrain from posting answers that didn't follow the guidelines, or they'd be removed by delete votes and mods.

Why this may have been foolish

We made sure that game recommendation Q&As would fit how SE works. But SE doesn't work in a way that's useful for people looking for recommendations!

• SE is designed so that hard questions don't get answers until the rare expert who can answer it finds and answers it; SE is designed to be insensitive to time for that reason, which is why it is never too late to answer an “old” question.
• SE is also designed to solve a problem in the best way possible, and to make sure the solution isn't buried behind pages of posts; but a recommendation-seeker naturally wants to be educated about all the relevant games they don't already know about.
• There's no way to know which answer is the solution until they try it and it works. All answers are still equally valid, despite our efforts to craft guidelines to ensure otherwise.

Forcing game-recommendation questions and answers to fit the Stack Exchange format is forcing a round peg into a square hole.

So why not ditch the guidelines and just answer their question? That would just bring us full-circle: game recs are naturally off-topic.

Why are they off topic? Because in their natural state, their answers are all equally valid. But why is that bad? Because…

Stack Exchange is for what forums are bad at

Stack Exchange was invented to be good at the things discussion forums are bad at, and in exchange for that superpower sacrificed the ability to be good at what forums are already good at. This is a smart trade-off, because SE doesn't aim to replace forums, but to complement them. Having multiple sites that each do one kind of thing well, people can choose and use the right tool for the right job.

We're seeing that here, now. If we wrestle a game-rec Q&A to fit SE's system, they stop being very useful to the asker, because SE is simply not good at what a game-rec asker is normally looking for. Game recommendations work really well at discussion forums though, so that really shouldn't be surprising.

Stack Exchange is simply bad at purchase recommendations. This was observed half a decade ago by the co-founder, and hasn't changed. Forums excel at purchase recommendations, because they generate a lot of responses that are valuable because of the amount of material they generate, which the prospective buyer can then sift through to earn wisdom and ideas they can then move their exploration forward with.

When we see someone using a screwdriver like a hammer, why are we lecturing them on Best Practices For Using Screwdrivers to Hammer Nails? Why don't we just hand them a hammer?

In this case, our site is the screwdriver, which is good for solving specific problems, and the wealth of RPG discussion forums on the Internet is the hammer, which is good for gathering opinions, ideas, and further research material to help you make a new-game decision. Why don't we just direct them to a forum, where they can quickly get recommendations to their heart's content?

We should ditch game-rec guidelines and direct people to forums

Our problem is that we're trying to force well-meaning people to rewrite their questions for reasons that don't really make sense to them, and then we sit back, self-satisfied, and watch them not get useful answers very quickly, or shout at well-meaning users who write answers in good faith that actually help. When the asker wonders why we're not helping, we tell them that it's for their own good. When answerers wonder why we're not letting them just help already, dammit, we tell them that it's for the asker's own good, to ensure quality.

Really, none of that is for the asker's own good. It's all an effort to avoid saying that game-rec questions are off topic and closing their question, just so that we can give them weirdly-constrained answers slowly (or not at all) and pretend we're providing gold-plated help. Meanwhile, they put their faith in us and are poorly rewarded for it.

We really want to be good at game-recommendation questions, but we're not. Denial doesn't help the site, or the askers who innocently come here looking for help. Pretending these questions work here does nothing except lead askers on, when we could be honest and direct them to somewhere they can get help immediately.

We should not cling to these askers jealously. We should send them to a forum, with a smile and a “Good luck!”

Answer 2

The current guidelines promote objectivity and answers backed up with experience and rationale. Having been on open forums that discuss the same topics, I think this isn't just a good thing, it's necessary for the SE format and a reasoned discussion.

However, the guidelines do not currently suit the purpose of game recommendation questions. A rules question about diagonal movements with a fly speed is general question that is very likely to apply to other people and needs the best answer possible, regardless of how long it takes to get that answer. The current experience criteria for game recommendations are set up the same way: you should only answer if you have experience doing the exact same thing as the querent, regardless of how long it takes. This is counter-productive for two reasons:

1. Unlike rules questions, game recommendations often involve the creative ideas of the querent. The boundless limits of creativity mean that no one having done exactly what you are asking for is very likely. Under the current guidelines, the more creative the querent's idea, the less likely they are to get an answer, regardless of how well they listed the requirements needed from the system. If I know exactly where to get all the features they need but haven't run a game that way, I can't help them.
2. Unlike rules questions, game recommendations are time sensitive. If I ask for a game recommendation for a project I'm currently working on, it's only useful to me while I'm working on that project. Furthermore, if my questions is very specific, it may never be useful to someone else. If it is answered in 5 years by someone with the perfect experience, it has done me no good and may never do anyone else any good.

Rules questions are curated for the good of the community in posterity. Game recommendations are different. You are helping an individual querent with an individual need. If your response helps others later, that's great, but that isn't the expectation. If questions and answers designed only for an individual's needs are deemed inappropriate for the SE format, they should be banned outright. It is more fair to the querents to ban them outright than to allow them to post a question that will not likely be answered while it is still useful to them.

My concrete suggestion for improvement is to change the experience critera to the following: you must have experience with the system you are suggesting and must provide a reasoned, detailed argument for how your suggestions matches the listed needs of the querent. This still ensures objectivity and discourages popularity votes while allowing for timely responses to unique questions that are specific to the game recommendation type.

Answer 3

(Note: Please read the entire FAQ linked below on game-recs, including the Meta.SE blog post and linked Arqade metas linked from it, to be sufficiently informed for your opinion to be taken seriously on this topic.)

Our guidance on game-rec questions as explained in Are game recommendation questions on topic?, I believe, are still valid. We do have some debate about how tightly to interpret them, and this debate is largely driven by both the continued presence of low quality game-recs but also by a community minority that continues to kick against the rules and say "why not let people say whatever they want." This causes those who do enforce the rules to have to monitor more, and more tightly, which then causes those unhappy with the restrictions to become more unhappy, as a vicious cycle.

However, these aren't going well. On RPG.SE we've had 13,204 questions of which 1010 (7.6%) have been closed. We have had 427 game-rec questions, of which 106 (24.8%) have been closed. This means we're attracting low quality questions to that tag, and that it's requiring a disproportional amount of moderation.

We have a couple options, I think.

1. Refine/clarify the game-recs as needed and then have the community do a significantly better job of curating them. I have added a FAQ to the original game-rec rules to try to clarify frequently confused points about them.
2. Go the way of SE in general and disallow them entirely. If this is too complicated, then it's too complicated and we should cut scope.

I am leaning towards #2 unless someone can come up with a way for us to make these kinds of question higher quality and more sustainable in the long term.

Simply another note in the FAQ isn't going to help. In general people have already bypassed our site help, meta, the game-rec post notice, and some amount of community or mod guidance on their question or answer for a question to be permanently closed or an answer deleted - the problem is that the community isn't of an accord on how to handle these, and therefore is not leading adequately by example at a tactical level.

Answer 4

This answer does not entirely reflect my personal opinion, but I feel that it is important for the community to be able to vote on the idea of banning game-recs entirely.

We should ban game-recs because:

• They are obviously problematic. 25% of our game-recs are closed, and 36% have been closed at some point in their lifetime. And they have special rules, and everyone else has banned them, and there's this meta, and so on. I don't think anyone would deny that they're a problematic category.

• It's difficult for the community to judge (and therefore vote on) the answers. A well-written answer can convince me to upvote, but not having actually played that type of game in that system, I don't know for certain that the answer truly deserves that upvote. I'm judging the answer purely on its merits as an answer, not on the merits of what it's actually suggesting.

• Because game-recs have to be extremely specific, even a successful game-rec is only going to be useful to a very small group of people; sometimes, only the person who posted it.
• The moderators, and to some extent, the rest of us, are spending a disproportionately large amount of time policing these.

Anyone who can think of more reasons to add should feel free to do so.

Answer 5

I think our current guidelines are working well and the moderation is effective. I have had a game-req question answered usefully this year, and learned a lot from other people's recommendation questions (What is a good game for a conflict-avoidant, disaffectionate player was my question). Our restrictions/guidelines are wrong, I think, and both too restrictive in certain areas and not restrictive enough in others, but the errors are slight and the guidelines seem to be largely effective in making our users-- especially new users-- actually think about answering the question properly rather than posting a 'FAE is the best'/'4e sux' shallow and inexpert psuedoanswer. Moderation is very good, albeit not so much based off the actual game-rec guideline text so much as what that text is trying to model (which is better in some ways and worse in others).

Things to improve one:

1) We still have popularity contest voting issues on rare occasions. The new Fate stuff seems to get it worse (both up and down votes, but mostly up) in terms of popularity-based issues, and GURPS has similar problems. D&D {AD&D, 3.5, 4e, 5e, etc} oddly, doesn't seem to suffer from this much at all, possibly because everyone knows about it at least somewhat so it's rarely the right answer to a game-rec and very often where the querent is coming from. The populaity contest issues, though they exist, are really very infrequent and usually only effects answers that are at least somewhat reasonable, which is good.

2) It's really hard to know what criteria matters as a querent. You have the question for some reason, but the reason as initially explained is often 'too broad' because you don't realize the scope of the answer space within the initial bounds. The thing to do, then, is to add more possibly important details until it turns out that some detail you thought wouldn't have affected much turns out to limit the answer space sufficiently for the expert community. It's kind of like throwing darts in the dark and hoping to hit a small target, and there's really nothing we can ask querents to do before posting to fix this; knowing a variety of solutions fitting their stated requirements basically precludes asking the question.

3) This category of question is one of the kinds that's most often done flagrantly wrong by a new user who ignored all the signs and thinks the place is a normal forum (as in, when that kind of user posts, it's usually this, not as in most game-recs are that). That's an indication of something. Not sure what.

4) Our explanations of our guidelines and the reasoning behind them could be much clearer and more helpful.

Answer 6

I see two problems with our game-rec questions (and answers) as they stand.

It is perfectly possible to write what you think is an excellent question, with well thought out criteria that you think narrows down the field of possible answers, ticking all the boxes for our guidelines. However, by very definition of the fact you are asking in the first place, you can't possibly know this for certain, and questions do get closed by those who can look at those criteria and make the judgement, 'nope sorry, too many possible answers there'.

Due to the relatively niche nature of many of these types of questions it becomes extremely difficult for most people to vote with any accuracy on the quality of answers. A 'perfect' answer demonstrates how a particular system has been used in the exact way requested by the question. The only way I can know for certain whether that answer is good or not is to have done the same thing, and this is extremely unlikely in many of the cases we have. At the moment, I look at answers in very general terms and go through a mental check list - does it demonstrate good knowledge of the system? Does it show the actual answerer has used it in exactly the way the questioner asks? If not, does it show someone else has? If not does it show how it could be used it in that way?...........The thing is, without having done the same myself, any judgement I might make of the quality of an answer for voting is guesswork.

So we have a set of guidelines it's impossible to know whether you've followed, and answers that are being voted on by people who are very unlikely to be best placed to accurately judge their quality. I'd say we have an issue.

Answer 7

A Suggested Improvement on the Guidance for Game Rec Questions

With many thanks to @mxyzplk for the time and effort put forth to preserve game-rec questions, the following is an attempt to distill the guidance post into something like what's on our help page for question "how to" or "this is what we expect."

Feedback/review of this is desired. Wherever it fails to fit the original guidance post will hopefully be identified. Edits to clean it up shall be made. Comments very much desired.

---

Game Recommendation Questions (draft)

The following guidance shall be adhered to for Game Recommendation questions and answers on RPG.SE. This guidance covers questions, answers, and voting to ensure that SE quality standards are maintained.

Questions

• Questions must be specific enough that there can be a single or best answer. The purpose of RPG.SE is to help people with their actual gaming.
• Don't just ask about games. Add context about what your needs are that a game is expected to fulfill.
• Avoid Lists
  • If your question is looking for a list, rather than a “best choice,” it's a bad question. Polls or lists, or "community wiki" for game rec questions are not acceptable questions. Quizzing people for the sake of doing it or idle curiosity generate poor questions … which will be closed.

Examples of poor questions:

1. “What are all the medieval fantasy RPGs?"
2. "What's a fun game?

This kind of question will be immediately closed as "too broad" and/or "primarily opinion-based."

Outline of a good question:

• I want to try a game that has these specific attributes (or covers X genre)

  • List attributes (1)
  • Describe genre

• I want to use it in this way.

  • Describe style of play you seek, or kind of story you want this game to tell.

• Who has done this?

• What game would you recommend and why?"

The importance of listing requirements / attributes / criteria.

If you don’t establish what it is you want the game to help you do, the SE community cannot provide a high quality answer.

This Wild West game question is a good enough question to be answered.

It is specific about the kind of game it wants. It would be a better question if it explained more about the asker's needs and predilections. If the person asking this question wants "gritty combat" then explaining what is meant by “gritty combat” in some detail is necessary.

Game recommendation questions shall be tagged [game-recommendation].

Answers

Answers MUST adhere to the "Back It Up!" principle set forth in the SO blog post Good Subjective, Bad Subjective To whit:

1. Something that happened to you personally
2. Something you can back up with a reference

Under the SE premise of expert guidance from experts, you may only answer a game rec question if:

• You are familiar with the game you are recommending.
• You have actually done it or seen it done, or
• If you have a detailed reference from someone who has actually done it (with details like how and results).

Example of an unacceptable answer:

FATE would be great for your ultra gritty combat system needs. I've never done that or seen anyone that has, but it is SUCH A GREAT SYSTEM that I'm sure it will do it.

Answers of this sort can and should be vigorously downvoted, and even deleted.

If you have never even read the game, don’t answer.

Anyone can Google "Wild West RPG" and offer something they've vaguely heard of but never done. That isn't expert advice from experts.

This is not an appropriate answer.

I know someone put out a Western supplement for game X

This the best kind of answer.

We played a Wild West game, used system X, and here's how that turned out"

Stay on topic.

It doesn’t matter if you don't like a particular kind of gaming. If someone asks for gritty combat and you want to reply with "people don't like gritty combat" - move along.

The querent is looking for a particular style and needs a game to do it. If you don’t have experience and sound recommendations for how they can do it, and with what game, then you aren’t the one who can help them.

• Exception
  You can make a parenthetical warning about pitfalls as part of an otherwise legitimate answer if you want to, but first craft the acceptable answer, then add warnings and caveats.

Your answer will be down-voted and / or deleted if you do not follow these guidelines.

Voting on Game Rec Answers

Answers should be voted up based on the specific expertise they demonstrate.

1. Someone who played that kind of game for a while should be heavily up-voted
2. Someone who says that they have a game like that and has read it somewhat should be downvoted
3. Someone whose answer shows that they just know about the existence of a game should be downvoted.
4. Someone recommending their pet system with no supporting points relating to the requirements in the question should be nuked.

FAQs

Questions FAQ

"But what if I just want anything I could maybe hack to my requirements?"

Questions that try to be "flexible" (e.g. "anything I can conceivably adapt to my goals") are too broad.
When asking your question focus it down on the exact thing you want to accomplish - don't worry, people will suggest things that don't hit your criteria 100% as it is.

To prevent a game-rec questions from becoming a litany of "everyone's favorite game," you, the person asking the question, need to narrow down your needs - think about it a while, you probably are assuming some requirements you're not conveying.

Answers FAQ

But what if I just lie (or pull out of my backside) about having experience with their requirements?"

Review Good Subjective, Bad Subjective requirements.
An answer cannot just recommend a game for the poster's desired use; it must also explain how using that game for that purpose worked out.

1. Just saying "and I did it myself" isn't sufficient to help the poster. We at RPG.SE require more explanation than that,
2. We assume good faith here on RPG.SE. If you think lying about a game, or lying about your own experiences, will help someone else you are mistaken.

How close does my experience need to be to their target requirements?"

If the request is well formed, then you don't have to have used a game in the exact specific way they intend to use it, but you should have hit the major points and need to address in detail your deviations from any of their requested attributes, requirements or criteria.

• Example:

Someone wants a "Wild West game with mechanics that help reinforce the genre, medium crunchy, with Indian magic and stuff, undead, and grey-type aliens,"

It would be fair to say

"I've played a lot of Deadlands and it addresses most of your points (include detail); they don't include grey aliens out of the box but did have an adventure with mi-go, you could easily re-skin them as greys..."

The key virtue of this answer is experience.

This is not OK:

"I have played GURPS and know they sell a Wild West supplement and supplements for Undead and aliens both too, so I bet you could use it to do that"

This lacks relevancy of experience.

Your need to be hitting near the bullseye, not just getting somewhere on the target. If you're not addressing at least 2/3 of their requirements or criteria, you shouldn't be posting an answer.

Answer 8

The first suggestion for improvement I'd offer is in how much help is given to ask a good question. Because the bar has been set high, the necessity for many game rec questions to go through some liposuction, focus, and refining seems to be common.

With this in mind, it would be beneficial for encouragement to use comments for this dialogue while the question is on hold, so that significant focus and refinement can occur before the question is opened up for answers. I would ask that "take it to chat" guidance be deleted from the process while refinement dialogue is ongoing, and then resumed once the question is reopened for answers ... once it is constructed well enough.

The second suggestion, in the FAQ from the 4 year old post:

Recommend an edit to put this part up in the "Questions" part of the FAQ and not as an afterthought to the "Answers" section.

Also note the related question How to deal with questions that just don't understand the scope of the RPG landscape? - we get some game-rec (adventure-rec, etc.) questions that try to be "flexible" (e.g. "anything I can conceivably adapt to my goals") and as a result are too broad. When asking your question you should focus it down on the exact thing you want to accomplish - don't worry, people will suggest a bunch of things that don't hit your criteria 100% as it is.

My Third Suggestion:

The Answer to the question encompassed in this meta post is a discussion, and as such pretty well written. See my second answer for a recommended reorganization of that guidance in something more like the help format.

Last thought on "how is RPGSE doing" in game rec:

There is a meta post where a variety of helpful forums are listed as answers.

There may be value in a meta post (maybe there isn't) with links to style or category analysis of RPGs? With so many games out there, that sort of meta/faq may exceed the scope of the ability of SE to help.

Setting the bar high is a good idea. Getting a good question crafted, with community help, should make any answer useful to more than just the querent.

Answer 9

The game recommendation guidelines probably started out with good intentions. The rules are set to keep answer to more than just an opinionated "I like" or "it sucks". And that's great.

However, the implementation is bringing more problems than I am willing to tolerate.

• As the requirement is experience, I have at least twice been forced to add noise to my answers. Although my answer contained a lot of things that could only been learned through experience, people downvoted me because I did not put "I did that" in the post. So I did. Now it sits there and goes against the very spirit of SE: no noise, more information. The valuable thing should be the information, not the sentence "Oh, yeah, I did that". Especially as this is the internet. I could write a post on how I flew to the moon on pink elephants. It gets neither better nor worse when I put "yeah, I really did" as a last sentence.

• Our requirements are so strict, that by definition the voting can only be a popularity contest. Take the Westeros question as a good example: even if I had played DSA on Westeros and there would have been no other errors in the answer, who here would have been able to judge? Who else would have played DSA on Westeros to check if I'm correct or simply posting nonsense? Nobody. So voting on game rec is a popularity contest by definition of our own rules.

• The strict requirements go only for the poster. I need to demonstrate experience, yet anybody can come by and vote. You don't need to demonstrate anything at all for that.

• We are interpreting things into questions the OP probably never intended. Take the westeros question again. The OP has written a very clear and good question. His requirements are the bullet points he set. And yet we have moderators that say it does not matter what he asked for. You may not answer, if you cannot answer his question in the context of Westeros. However, that is purely their interpretation of the guideline. The OP only said he needs a system that follows his requirements. And the requirements he listed said nothing about Westeros at all. I'm pretty sure we have helped neither the poster nor the site, despite a good question.

So personally, I will stop participating in game rec at the current state. It's just too frustrating trying to fill in all the bureaucracy, dodge the trigger happy downvoters that wait for you to make a mistake and all to have a bunch of people holding their popularity contest. That is not what SE is supposed to be and it's not what other SE sites manage to handle under the same constraints.

To answer the comments as that would be too long for a comment:

I'm not really advocating to ditch game rec altogether. That would not help anybody. But my suggestion is not a game rec suggestion either but goes for the whole site and pretty much any post: be less of a trigger-happy dictatorship. Don't go after mistakes. Make suggestions on how to improve and let people improve.

Look at what is on hold and who put it there. The last time I counted, it were 6 questions and of those, only a single one had actually been closed because of community consensus of 5 people. 80% of them had been closed by a binding mod vote. Because it was "against the guidelines".

Look at the Westeros question. Yes, my answer is incomplete. Because I thought that as I recommended a system the asker already knew, some points on his list would be self-explanatory. So, it's incomplete. Not spam or rude or off-topic or factually incorrect. Simply incomplete. Yet nobody came in and commented "hey, your answer would be better, if you...". Instead, the answer was downvoted and the question closed. Not by the community. By the mods, in advance of the community, citing the guidelines.

Is this inside the limits of the SE software? Absolutely. Is this the SE spirit? No. Not at all.

We are spending time here trying to help people. For free. If a positive, constructive post written to help without factual errors is so bad that two people tell me to "get this piece of s... off this site" (which is the verbal translation of the downvote mechanic) because it violates the guidelines then well, those guidelines are too strict for me. Actually, not the guidelines. I like the guidelines. But it feels like there is a secret guideline police that is out to catch you on any tiny mistake you make. The guidelines are not there to improve your post or help the poster, they are there as a means to justify downvoting and closing stuff. Regardless of the actual content.

There are sites out there (and SO is one of them) that handle this waaay better. Where when I get a downvote I know I have made a big mistake. But even on factually wrong things, you may get a comment saying "hey, you might want to correct xxx, it's wrong, yyy would be correct.". And those commentators are 100K-700K rep users that could use the very same knowledge to make 1k $ an hour instead of helping you improve.

Professionally oriented sites seem to have a codex that says "if someone is trying to be helpful and spending his time, I will not downvote on minor mistakes. I will just upvote the correct answer and maybe leave a comment."

I am missing this attitude of professional courtesy here and I really don't advocate to do anything with the guidelines because in the absence of good, constructive behaviour, how we change the guidelines will not matter. They will continue to be used to put down people.