# Are we satisfied with the state of homebrew review questions?

For the past seven years, homebrew review questions (that is, questions of the form, "Is this homebrew balanced?") have been on-topic and thus allowed. As brought up in the latest "How is the community doing?" meta, however, they're becoming more common and arguably more problematic.

Are we happy with the homebrew review questions we're getting now? If not, what do we want to do about it?

Our existing guidance on asking these questions can be found here: "How can I ask a good homebrew review question?"

• As to your "more common and more problematic" point, if someone's got the time and energy to do a month-over-month or quarter-over-quarter look at # of homebrew questions and their closure rates, I think that'd be interesting to see. (Perhaps # of questions should be scaled to growing site traffic/activity.) – nitsua60 May 11 at 3:12
• @nitsua60 I took a potshot at doing this below. – Rubiksmoose May 12 at 16:53
• Maybe we should just make a generic "Is my homebrew thing balanced?" question with the answer of "Probably not, no", and then just close any new homebrew questions as being redundant with said question. – nick012000 May 21 at 6:08

We should probably improve the meta guidance we give to users for homebrew review requests. I done messed up in creating it the first time around.

See, I asked a generic guidance topic (How can I ask a good homebrew review question?) and then two weeks later asked a more specific guidance topic just for D&D 5e (How should I ask about my D&D 5e Homebrew being balanced?). However D&D 5e currently represents just about all of our homebrew review content, so while that seemed like a good idea at the time, the end result was I just wound up splitting our guidance for D&D 5e homebrew review questions across two different metas.

I think the right course of action here would be for me to post a Community Wiki answer to that second question which combines the guidance from the first meta with HellSaint's answer from the second. This would give us just one spot we point people to that has all the guidance.

That Community Wiki post might be revised further beyond just being a combination of those two posts. linksassin's answer on this meta brings up the idea of enforcing a minimum wait time between iterations in its last section: if someone posts version 1, then within a set number of days (e.g. 3 days) posts version 2, we close version 2 as a duplicate of version 1. If we do go with that being a thing, we'd further revise this post to give people a straightforward heads-up that we'll do that, and we ought to link that post to new homebrew review requests to bring that to their attention.

• I really like this for a couple of reasons. It 1)gives us a single place that's agreed upon for direction on how to approach and 2)it provides a system that must be adhered to. My biggest concern is enforcement and whether the community will hold themselves to following and upholding the guidelines. Unfortunately, if we don't, then we may have to reconsider allowing them. – NautArch May 14 at 15:38
• @NautArch Just a thought on that note: one thing I realised after game recs were banned was that applying stricter rules and heavier enforcement won't necessarily actually fix a category of question, it might just make more of a problem out of it as we notice more rules more regularly broken and needing more work to manage. So if we want this category to succeed, we'll have to find a way to manage it with a light touch. A good clear meta plus a bare minimum of easily community-enforced hard rules (e.g. a time limit on iterations), plus a component of going with the flow, seem important. – doppelgreener May 14 at 15:45
• That's a really good point - but is there anything we can do to try and not get "first drafts"? Or is that just par for the course? – NautArch May 14 at 15:50
• @NautArch I don't think we can really control that: we'll get what we get, our only options are how to react to it afterwards. At the very least controlling the iteration cycles may be helpful and require people to put more thought into it between iterations—and give us more opportunity for more than one person providing feedback before the question goes obsolete. (We could require people demonstrate that they did playtesting before we review their content? Not sure whether that would work or be one of those strict rules that exacerbates the problem. Playtested content can still be shoddy.) – doppelgreener May 14 at 15:58
• Paragraph 3 is a good idea. – KorvinStarmast May 14 at 22:20
• I think this is the best approach overall, centralise the good but distributed guidelines and add some enforcement rules. @NautArch To avoid 'first draft' I think we should stick with the existing tools available to us. Downvote as "lack of research/effort" and comment to that effect. User's will hopefully see those comments on other homebrew questions before posting theirs and know what we expect. – linksassin May 15 at 0:41
• I also feel like we should limit askers to their own homebrew, rather than helping them evaluate things they found on danddwiki. – Oblivious Sage May 16 at 15:52
• The answer in comments response shown here is typical of the homebrew question problem ... not sure if you want to fold that into the answer. – KorvinStarmast May 16 at 20:14

# I would be disappointed to see them go

I may be somewhat biased as I am one of the more active users on the homebrew tag since I joined the site. However, I do not agree that homebrew question in general are an issue on this site. Moreover, I think that banning them, as suggested by Oblivious Sage would only be a net-loss to the site.

Sure, homebrew questions aren't usually very popular. Experienced askers and answerers know this, and don't get a lot of rep for these question. I know that my homebrew answers often take a lot of work for not a lot of reward, but I am ok with that and am willing to put the work in anyway.

I posted some of it in chat but here are some of the reasons I think homebrew review questions are valuable:

• We answer them well because of our strict requirements on supporting your answer. Homebrew reviews on other sites are rarely supported and give a lot of "here's how I would do this instead" type advice which isn't as useful.
• They are a valuable topic for increasing participation. They are popular with new users and help grow our user base.
• Personally I also enjoy asking/answering them. I think it is a unique way to learn about game mechanics and the design consideration for homebrew.
• Any question that would provide the same information without a specific homebrew example would be too broad and get closed.

I think NautArch's answer describe why a lot of homebrew questions are bad is excellent. He said that most homebrew lacks sufficient research or tested before posting. I believe this makes the question a candidate for downvotes and we should use them. It does not make them off-topic. Similar to any other question we should work with askers to advise them why we consider it a poor question and how they can improve.

# Rapid iteration is a problem

On a site rather than question level, I can see the issue with rapid iteration. It spams the question feed with multiple questions from the same user. Downvoting all of them could get reversed as targeted voting. It prevents each individual question from getting enough attention (often homebrew questions only get 1 answer). Generally it also means that each iteration is smaller than it should have been, if they had waited longer for more feedback they could have avoided several generations.

I propose we vote to close iterations posted within 3 days of the previous as a duplicate of the first. While closed we encourage the user to reflect on the feedback of the previous version, test their new version and edit in their findings. Once three days has passed are there is sufficient distinction between the questions we can reopen the new one for answers.

• Interested to know if downvotes mean people disagree with my view, or my proposal? Perhaps I shouldn't have put both in one answer. – linksassin May 13 at 2:59
• Personally, I don't see your proposal helping - as mentioned in a few places on this page, trying to "fix" a site problem by adding rules to it has yet to work out well in the long run. – Miniman May 13 at 3:39
• @Miniman My problem with "ban them" is that there are users (including me) who enjoy them. Other's may not like these questions but I fail to see how they are causing harm to the site. I would say that they do work when asked well and we should use votes to encourage them to be asked better. Discouraging rapid iteration would be the only improvement I would make to the current system. – linksassin May 13 at 4:51
• There were users who enjoyed system-recommendation questions, but we banned those. There were users who enjoyed designer-reasons questions, but we banned those. Whether or not people like a type of question has no bearing on whether it fits in the SE framework. We're not a forum, we're not the absolute end-all be-all of RPG resources, and we're not trying to be either of those things. We should stick with what we're good at. – Oblivious Sage May 13 at 13:09
• @OblivousSage I fail to see how these are doing any harm to the site. I think this site can answer them well, and better than many others for the reason I state. Getting rid of them would hurt multiple users and for what benefit? I don't see one other than the people that dislike it don't have to see them anymore. They can ignore tags and have them not show up if they want to. – linksassin May 13 at 14:02
• To be fair, there are genuine problems that are often involved with homebrew questions - but unlike system-/tool-rec and designer-reasons questions, they're not inherent to the category of question, and unlike rules-as-written questions, we're not really imposing special rules on that content alone. We're just trying to apply the same rules to them that we apply to everything else, and I think it's just a matter of new users not understanding those rules. – V2Blast May 13 at 20:54
• @ObliviousSage I think you make some good points there, and I think "fit" needs to be thought of as a gradient, not just as a binary. I.e. some things fit very easily, some fit and often take a bit of work, and some never fit no matter how much work we put in. If homebrew, for example, were typically "fits but often needs some work" then the cost-benefit implied by linksassin mentioning the popularity/engagement of the tag would be valuable. (Personally I'm not sure where I think homebrew lies: I just wanted to push back a bit against fit/non-fit being a simple determination in all cases.) – nitsua60 May 14 at 0:52
• @nitsua60 That's a reasonable perspective, but as mxyzplk pointed out in the comments on my answer, "more work" has not historically solved our issues with problematic question types/tags, primarily because the users who ask those sorts of questions are almost entirely new users who don't even understand the general rules for the stack, much less whatever specific rules we come up with to try to make the "fits but needs work" tag actually fit. – Oblivious Sage May 14 at 1:03
• @ObliviousSage "the users who ask those sorts of questions are almost entirely new users" - and closing these as off-topic would drive these users away. We will always have some work to do to get new users up to speed. I don't see homebrew questions as any worse than any other category. – linksassin May 14 at 1:06
• @nitsua60 Furthermore, I think it's worth noting that there's no "homebrew review is doing fine" answer. Everyone agrees something is wrong with these questions, but we don't seem to have much consensus on what that is, much less how to fix it. That suggests to me that we're going to muddle to some sort of conclusion about holding these questions to higher standards, putting in more effort to police them, etc. and then end up right back at this same discussion later on because we haven't solved (and perhaps cannot solve) the underlying problem. – Oblivious Sage May 14 at 2:54
• @nitsua60 And that's exactly what we tried with game-recommendations, with designer-reasons, and especially with rules-as-written, and the result was we let people become attached to those questions so there was ultimately a lot more acrimony when we finally did ban them. That's why I think we should tear the bandaid off and ban homebrew review now. – Oblivious Sage May 14 at 2:56
• @ObliviousSage 3 of 4 points in my answer are those arguments. Do you disagree with them or not understand them? I'm happy to expand on them if required. – linksassin May 14 at 3:19
• +1 I agree with the last section, something like duplicate closing on a too-short iteration timeframe would be helpful. I did ask in our meta that people wait a few days before opening the next iteration but we don't currently enforce that in any way, and people usually iterate after just one answer instead of waiting for multiple. – doppelgreener May 14 at 14:50
• @ObliviousSage While I'm concerned about the chance of it not working out, I'd rather us not take “we might ban it in the future” as a reason to ban stuff now. It's a fairly pessimistic philosophy that would cost us those topics we could make work down the line. Let's not forget that at one point we might have banned ID questions but accepted them as-is, and might have banned all setting (or lore) questions but instead found a better way to handle them. I'd prefer we give each topic a generous chance. – doppelgreener May 14 at 16:27
• @doppelgreener Exactly. If "we might ban it later" becomes a precedent to ban things we may as well close the whole site as anything could be banned one day. – linksassin May 15 at 3:54

# Honestly, No.

My answer to another meta question covers the vast majority of my concerns in that many homebrew questions feel like they've been posted before they are ready to do so.

Just like you don't hand a paper around to be edited before you've done several rounds yourself, the same goes for homebrew questions.

We're not here to write the homebrew for you. We are definitely here to check for balance, but we really should only be doing that if people put in their due diligence before posting.

This goes for both initial posts on content as well as iterations.

## But this doesn't mean we shouldn't have them...yet

Ultimately, I feel we need to be better at downvoting content that seems not fully baked. I don't think we can put it on hold any more we can put a low-value question on hold. People can ask questions that don't have research effort and people can post homebrew that doesn't show enough effort.

But I do feel we need to be clearer via downvotes in what our expectations are. We can support that with the meta posts we have about how to ask for homebrew, but ultimately, homebrew is a part of many RPGs and people can and should be able to ask about it - regardless of how much effort they have put in initially.

We do need to keep an eye on this, though. If the community feels that we're getting more low quality homebrew balance questions than high quality, then we may need have this conversation again - but until then we should try our best to improve the homebrew questions by both showing folks positive reinforcement through what a good question is, what good iterations are, and also providing negative reinforcement for when they don't.

## In summary, put some effort in and show it

That means having questions that are formatted correctly, use common game terms, have been tested as thoroughly as possible before posting, and after taking in feedback.

• I'm conflicted on how to vote, because I agree with "honestly, no" but don't quite agree with this diagnosis of what's wrong or how to improve it. (I'm not sure I can put my finger on what's bothering me yet, though.) – nitsua60 May 11 at 1:19

# Homebrew statistics

I took a crack at hacking together a couple SEDE queries to pull some data that might be helpful (as suggested by @nitsua60). Below is a graph detailing the number of homebrew questions asked and closed on this site per month since the beginning of the site. The grey line represents the percentage of the questions closed that month. Note that data is not equalized for traffic.

The most obvious conclusion we are getting increasing quantities of homebrew questions in non-relative terms.

I would highly appreciate people checking my SEDE queries and methodologies, because I really am just hacking it together and could easily have made a mistake.

And the two SEDE queries I modified:

For open questions: here

For closed questions: here

• Now I kinda want to know if this pattern is unusual compared to all questions on the site. I admit to not paying too close attention, but I haven't seen a systematic issue with homebrew questions compared to RAW for instance – Saladani May 12 at 18:05
• It looks like the rate at which we get new homebrew questions started to really pick up about 18 months ago. As the total number of such questions has increased, the percentage of them that end up being closed has become less noisy, since each individual closed question represents a smaller fraction of the total. Just eyeballing it, the close rate looks like it will stabilize in the high teens or maybe low 20s. Do we have a comparable number for all questions, regardless of tags? – Oblivious Sage May 12 at 18:41
• I'm curious what people think this data indicates. Does a falling close rate mean the questions are better asking overall? Or that we are worse at moderating them due to the increase in questions? – linksassin May 13 at 0:08
• @linksassin I don't see the close rate as falling, necessarily, merely becoming less noisy as we get a larger sample size. – Oblivious Sage May 13 at 2:55
• @ObliviousSage That's a valid interpretation as well. I think it makes Saladani's point about how this compares to other tags more compelling though. Is it levelling out to similar levels to the rest of the site. – linksassin May 13 at 3:00
• @ObliviousSage fwiw I don't currently have this data for all questions/tags, but I could get it if people think it is useful. I'm really not the best at SEDE queries though so it takes me a while to get things to work. – Rubiksmoose May 13 at 3:02
• @Rubiksmoose Perhaps just repeating those queries for a popular tag like dnd-5e would be representative? It isn't a very scientific approach but it possibly the quickest and easiest. – linksassin May 13 at 3:04
• @linksassin Actually, dnd-5e may not be a good choice since the vast majority of our homebrew review is for 5e. Maybe pathfinder? – Oblivious Sage May 13 at 3:05
• @ObliviousSage Actually maybe a non-system tag like spells would be less biased? Really though any of the top 10 tags should do, ideally average the data from more than one of them. – linksassin May 13 at 3:07
• @linksassin My preference would be for a tag that has minimal overlap with homebrew review, so that we can make a proper comparison. – Oblivious Sage May 13 at 3:11
• I don't find this graph useful in addressing the question, though I do appreciate the effort you put into getting a quality-based metric into an easy to use form. The problem with homebrew questions is one of quality for the most part. SEDE can't measure that. – KorvinStarmast May 15 at 17:04
• @KorvinStarmast never said it was useful! :) Somebody requested data, I did my best to provide. It could have had some interesting results, but alone it is, at least, very hard to draw conclusions from. I would argue, however, that closures are at least an indirect measure of quality for that sliver of cases. – Rubiksmoose May 15 at 17:06
• @Rubiksmoose Or closure is an indicator of newness of the user ... hard to get a correlation, or causation, from the query. Again, thanks for putting that together, as at least it shows the volume ... – KorvinStarmast May 15 at 17:12

## Also, no

I'm going to posit an answer that's more or less a combination of what was posted above coupled with recommendations following my own experiences associated with attempting to answer these kinds of questions over the past year:

## Demonstrate your familiarity with the system

Our comments, suggestions, answers, etc. fall upon deaf ears if the querent cannot be bothered to learn the system they are proposing to homebrew well enough to ask a question that is coherent to other users of that system. For example, in 5e, if you aren't familiar with what Actions, Reactions, and Bonus Actions are then you should probably hold off on trying to homebrew an entirely new class until you get that figured out.

Your question, as written, should demonstrate knowledge of the system that you're working in. Treat the game's terminology the way you'd want it treated were it to be read by any other user of the system; certain systems the criteria is very well-defined, others, less so. Regardless, if 90% of people playing that system interpret your question in differing ways that's a clear sign that the querent has not posted a useful and answerable question and should be downvoted/closed.

## Let it breathe

So many times do I see someone post a homebrew that's going to require more than 10 minutes to knock out an answer for. I don't have time for it right now so I move on. Within a couple days, there's several answers that address multiple different issues; one is deemed accepted; and then there's a second draft issued as a subsequent question. Half the time, the question has also since been closed (probably because each answer seemed to focus on a different issue and thus the question was unclear).

Many times, the answers that did get in do not address some underlying issues and as a result, those issues carry forward into the next iteration. Furthermore, it complicates review of the second draft because you now need to refute the changes that were made since they didn't consider the underlying issue.

In general, I think we need to establish some sort of timeline for iterations as well as asking new iteration questions when an existing question is on hold. I think 1 week is reasonable for new iterations, but others are likely to disagree. I also don't think you should be permitted to post a new iteration if the original question was closed; edit that question so that it can be re-opened and new answers can be provided.

## Tell us why

So many homebrew questions are posted without any sort of stated goal. This goes back to my first point, if I don't know what problem with the system (perceived or otherwise) you're trying to fix with this homebrew it's very hard to provide useful comments.

I'm not saying that you cannot create new mechanics with homebrew, but if you elect to you need to fully explain the process by which you created those mechanics. Want to create power points in a 5e system, explain what logic you're using to balance that (perhaps the DMG's alternate rules on spell points).

If you're electing to create something out of whole cloth, I hope you can demonstrate some results of the playtesting you've done. Because if MacGuffin summonings are based on the phase of the moon in the Realm of Thanus, I can all but guarantee you that nobody else out there has done this already, so we are going to be more reviewing your playtest results and helping you identify ways to improve playtesting.

This may be the most unpopular suggestion since it targets respondents instead of querents, but we get a lot of answers to poorly phrased homebrew questions that are assuming a lot of unstated details. This disincentivizes subsequent and perhaps more thoughtful answers, but it also puts the querent in a weird spot with regards to editing their question to be more thorough. We don't want edits that invalidate existing answers, but if the question was poorly phrased, then it needs to be corrected before useful answers can be posted.

Comments, downvotes and delete votes are the tools we have available and should be using them on answers that may be premature. I believe diamond mods also have the means to denote deficiencies in an answer and should be using them in these scenarios; flags should be considered to help draw diamond mods to those premature answers.

• Not sure if the stack format allows, but I wonder if it would (or wouldn't) be helpful to create a closure for homebrew that gold badge users can use to close it rather than waiting for 5 votes. – NautArch May 15 at 15:57
• I think gold badges can only auto-close with a duplicate, but I'm not sure. I doubt you could have them (SE) alter the code to create a new classification or close mechanic simply for a niche issue unique to our site. Perhaps as a matter of policy we permits gold users to close with a standard duplicate question that stipulates the manner in which homebrew questions need to be posted; but I think that would need a separate meta with diamond mod blessing to permit. – Pyrotechnical May 15 at 16:35
• @NautArch We're also nowhere near having any users with a gold badge in the [homebrew] tag. We might have one or two silver badges floating around. – Oblivious Sage May 15 at 16:42
• @ObliviousSage True, but we do have gold badge users for the system tag they are homebrewing in. – NautArch May 15 at 16:43
• I would have to check, but I think the gold-tag auto-close requires both the closed question and the dupe question to have the tag you have gold in. If we went with a canonical dupe target for bad homebrew questions, as @Pyrotechnical is proposing, then we would need the most common system tags on the dupe target. – Oblivious Sage May 15 at 16:46
• While I was going to suggest 72 hours, before version 2, a week is fine also. – KorvinStarmast May 15 at 16:58
• @KorvinStarmast I suggested a week because my world, in general, tends to be weekly cycles with regards to when or if I'll have downtime to respond. 72 hours could be a hectic weekend for me and thus I'd never get a good chance. – Pyrotechnical May 15 at 16:59
• @Pyrotechnical I'd vote yes to a RPG.SE policy of "no second iteration until a week has passed" ... not sure how we get that into a policy statement. Insofar as focus goes, not sure if my Stampede homebrew is an OK example to use for a focus/scope one? – KorvinStarmast May 15 at 17:01
• @ObliviousSage Yeah i'm not sure on that - but as you've said the solution is rather simple just to include system tags - but I think the only requirement is for the question being marked as the duplicate to be a tag in which you hold gold. – NautArch May 15 at 17:29
• I think these are pretty fair guidelines. Though I don't love the idea of having a meta-type post on the main site as a duplicate target. I think we should stick with normal close mechanics and revisit if there is still a problem. Misuse of system terms -> VTC Unclear and downvote; Iteration too fast -> VTC duplicate of first; No goal or purpose -> VTC Unclear/Too broad; No process -> Use comments to request it; Answer on bad question -> Downvote, flag, VTD same as answers posted to questions without system tags. – linksassin May 16 at 1:21

## Homebrew review questions should probably be off-topic.

Like and before it, homebrew review is a type of question that in theory seems like something we would like to have on the site, but in practice tends to produce a lot of bad questions and/or bad answers. This problem results from a number of factors that make homebrew review a poor fit for RPG.SE.

First, homebrewing is (if done properly) an iterative process, coming up with a design and then testing it and getting feedback, making improvements, more testing/feedback, more improvements, and so on. The Stack Exchange system was not designed with this iterative process in mind. Askers either edit their question when they iterate (a big no-no, since it negates the existing answers), or they post multiple iterations of the same question. That's technically allowed, but it feeds into the second issue.

Homebrew review isn't useful in the long tail. The goal of the Stack Exchange system is not just to answer people's questions, but to create a searchable archive of questions and answers, so that later people with that same problem can discover the question, already asked, and use the answers, already provided. Homebrew review does not lend itself to this because it is almost never useful to anyone other than the asker.

Finally, a lot of homebrew is just too big for this site. Sometimes complex homebrew can be split into a multitude of similar questions; this at least results in answerable questions, even if it spams the front page (something that is technically allowed but that many of us find obnoxious, especially for closely-related questions). But we've also seen askers trying to post entire classes/archetypes for review, which can end up being simultaneously too broad to handle in one question, but too inter-connected to handle in multiple questions.

We could try to tighten up the rules around these questions and try to enforce them more. However, that approach has not historically worked for other problematic tags, largely because these questions tend to be asked by new users who don't know the rules. It won't matter what special rules we have for this tag, or any other specific tag, because there tends to be very little intersection between the set "users who ask questions with problematic tags" and the set "users who read the rules". We'll just end up having this same discussion again in a few months or years.

In conclusion, homebrew review is yet another tag that was worth a shot, and generated some good questions and answers in its day, but ultimately just doesn't work well on the Stack Exchange and would be better suited to a forum.

• I find myself strongly in agreement with your "long tail" paragraph, particularly since I rarely see homebrew querents actually describe their problem. "Is this balanced" is nowhere near as good as "I'm having this trouble making homebrew_item work at my table, here's how I know, here's what I've tried, how can I achieve table_result" as for the likelihood of generating an answer that's instructive for the ages. – nitsua60 May 11 at 1:16
• Interestingly, spamming the front page with questions all at once makes it impossible for them to be consistently downvoted, since it will get picked up as serial downvoting - so although it annoys many of us, it actually leads to less downvotes. – Miniman May 11 at 3:53
• I agree with this. The iteration spamming, the lengthy discussions, it all shows how we're trying to force something into the SE format that it's not suited for. We;re not the "anything related to RPG" site, we are the "RPG Q&A" site. Whenever we try to have complicated rules a kind of question has to follow, we fail, because most site users are casuals (and should be). – mxyzplk May 11 at 14:42
• @mxyzplk Well said. I understand many people like these questions, but trying to put more rules and better enforcement on them isn't going to fix the underlying problem. We'll just find ourselves revisiting drive-by users' inability to follow those rules in a few months or years and banning these questions then. We should just bite the bullet, save ourselves all that trouble, and ban them now. – Oblivious Sage May 11 at 15:30
• "Stack Exchange system was not designed with this iterative process in mind" - so? It wasn't designed with subjective roleplaying questions, problem players or a host of other things that we do well, in mind. Not a reason to ban it. "Homebrew review isn't useful in the long tail" - I regularly view old homebrew questions for inspiration, homebrew gotchas and innovate solutions to problems. "a lot of homebrew is just too big for this site" - agreed, and those questions should be closed as too broad. There is a sensible limit but we don't need to ban things under that limit. – linksassin May 14 at 3:17