What is the issue with questions looking for averages or statistics?

Question

Over my relatively short tenure here so far, it is my impression that there seem to be two schools of thought in the community of active contributors:

• one sees practical value in using averages based on generic assumptions or on collected statistics (this is my current perspective)
• the other sees such averages as meaningless or not of value, and recommends asking questions for either a concrete situation, or about immutable mathematical truths

The second perspective can also motivate people to vote to close questions as opinion-based, like this question about the average number of encounters per day in a D&D 5e game.

What I understood from discussion at the time is that this could vary widely from table to table, and, so the conclusion, therefore it would be of no value in providing an averaged view; which, I think, is a slightly different argument than it being opinion based, it is an argument that it is artificial and not useful – but there is no closure option for that.

What reminded me today of this subject was that Thomas Markov put a bounty on an answer that is the equivalent of "we cannot say" (in this case, for a question on what the average to-hit chance per level is). I am grateful that he is not going quite as far as voting to close the question as opinion based (the fate of a closely related one on if one can use an average AC per level chart in DPR calculations), but he is offering 500 of his hard-earned Internet-Points for making great answers to support that view.

If I understand it right, he is a professional statistician, or at least, works in a role that requires professional use of statistics, and this makes me wonder if there is something fundamental I am missing.

Staying with D&D 5e, as a thought experiment to support such a view, I could imagine a campaign world set after a magical cataclysm, where our heroes are among the only survivors, everything else they encounter is either a Zombie (AC 8), or a magically animated armor (AC 18). In this world the average encountered AC would be 13. Clearly a character would never get to fight an opponent with an actual AC 13, and an ability that allowed the character to always hit AC 8, 50 % hit AC 13, and never hit AC 18 would have a very different performance than what one would expect from that average in individual encounters with either monster type.

Such an extreme could of course exist, and here is exaggerated to make the point. In practice, I think extremes and outliers are rare, and because they are rare, there is still value in averaging, even if it will not work in a situation like this. But of course, to prove that with a low statistical chance of being wrong, one would need to sample a large number of modules or tables.

Most other questions that do not just ask for rulebook exegesis, for example how to handle problem players or how to deal with issues of player agency do not get closed as opinion-based, even though answers can often only be based on personal experience of one writer, and not on aggregated, objective knowledge across many tables, either.

But when the question is about game statistics encountered in play, it appears that personal experience from years of play is not enough any more. So what I'm wondering about are particularly about these related aspects:

1. Do questions about numbers have to be mathematically stringent, or are rough approximations that do not hold in some (maybe even many) cases acceptable, without counting as opinion-based?

2. Why would questions that are asking for statistics – either based on rulebooks, from surveys, or from collecting data – be opinion-based?

Answer 1

Many of these "what is the average of...?" question should be setting off your XY alarm.

You mentioned my occupation, so I'll give some insight into how it applies here. Part of my job involves collecting all sorts of data, finding meaningful patterns, and presenting those meaningful patterns to stakeholders so they can make good business decisions. Sometimes, those stakeholders have ideas about ways to look at the data, other descriptive statistics they want to see. Part of my job is to tell them if looking at the data that way can actually be helpful, or if it doesn't really apply to our process. As I have asked you many times, I have asked my vice president of operations: what problem are you trying to solve with this information? He wants to know if trying to use that information is going to be a waste of the company's time, and perhaps more importantly, if trying to use that information could lead to poor business decisions. So I ask why he wants to know, he tells me why he thinks that information could be useful, and I advise him on the limitations of the data, the limitations of the statistics he's looking for, and whether or not I think it can actually tell him what he thinks it's going to tell him. In this business, there is no such thing as information for information's sake. If it isn't going to make us more money, we aren't going to waste time trying to figure out.

Obviously, the stack is not a business, at least, not for us. We can handle questions that are looking for trivia. But, the guiding principles for asking good questions are very similar. Just as my boss wants to know if the answer to his questions are actually going to help him make sound business decisions, the best Q&As here are those that are going to lead to positive outcomes at the table of play. Of course, not every question has to even attempt to do that, What's the greatest number of hands I can have to annoy my mother-in-law with? comes to mind as one of our successful "just for fun" questions with zero consequences for the table of play (though I hope Bluemoon was able to patch things up with his mother in law).

But when an investigation does appear to have potential consequences or application at the table of play, it is the sort of question where knowing "what are you planning to do with this?" or "why do you want to know this?" provides answerers with the tools needed to write fantastic answers that improve your gaming experience. And this is where your XY detector should be going off. Consider your question What is the average number of encounters per day?. I explained in a chatroom on the question how it runs into the XY problem. You wrote in that question:

How much damage such a feature can be expected to contribute to combat depends heavily on how many encounters you'll have in a day, because once you run out of uses, it won't contribute any more damage. When planning your character and estimating average expected damage, you therefore often have to make an assumption about the number of encounters per day that will use up some of those limited resources.

I'll just quote my take from that chat:

From what I can tell, the problem you are trying to solve is "I want an idea of how many encounters per day I can expect when I am building a character, because different builds perform differently with different numbers of encounters per day".

That's your problem, X.

You have asked a question about what you think a good solution Y is, "what's the average encounters per day?", because you think knowing that will equip you to better build your characters.

Classic XY problem.

Instead, an on topic question might be something like "Some character builds are really powerful with only a few encounters per day, but taper off quickly as their resources are depleted. I want to take this into account during character creation, so how can I know how many encounters to expect on average at a given table?"

This is the sort of question where experience based answers are what we are after. Because it is experience solving your actual problem, not experience with a mostly unhelpful solution you thought of.

Your question asked "what is a good assumption for the number of encounters per day?", and the answer is "there is no good assumption because it will be wrong more often than it will be right". Instead, the best solution to your problem there, and the answer to the question I suggested in chat, is to just discuss it with the dungeon master and other players at a session zero or before the game starts so you can inform you character creation decisions accordingly. The average number of encounters per day varies from table to table, so just figure out what it is likely going to be at your particular table, rather than bringing to bear some arbitrary expectation that is probably going to be wrong for most tables.

The thing I really want to know is whether or not the answer to a question actually leads to worse outcomes at the table. Using a statistically accurate estimation of encounter per day as a tool for informing your character creation decisions is likely to lead to worse outcomes for you at the table of play. You will disappointed when the expectation you had was not met. This is why it is important to tell us why you want to know something. And this is why I bountied Theik's answer: they observe that bringing the expectation of 65% chance to hit to the table of play just isn't a realistic expectation, even if it is statistically accurate.

This is also my concern with your more recent question, What is the average damage per attack for monsters by CR?. You state in the question:

The DMG on page 274 has guidelines about damage per round for monsters of each Challenge Rating. However, most monsters deal that damage as a combination of multiple attacks (and sometimes other abilities like breath weapons).

The reason this information is given is because it motivates the CR calculation used for monsters, and CR is the primary metric used in encounter design. In other words, damage per round is an important consideration for assessing the challenge a monster brings to a fight. This is the metric you should be concerned about when designing encounters. However, your question asks about damage per attack, but doesn't give any motivation for why. You already know the expected DPR for a given CR, which is the metric I'd expect you to be using in encounter design, so I have no idea why you want to know damage per attack independent of damage per round. My XY alarm is going off. Damage per attack is obviously going to be an inferior metric to damage per round for the purpose of encounter design. If you are trying to solve a problem with encounter design using damage per attack, I want to know what it is so that I can tell you if damage per attack actually helps you. And if you just want to know for the sake of knowing, just say that. It's okay. I withheld my vote to close that question because it is a question about the averages of readily accessible numbers. It isn't opinion based. But I've downvoted it because I don't think it is a good question, and I think the correct answer to the question could lead to bad outcomes at the table of play if it is used in the wrong way. Or as the downvote helper text says:

This question does not show any research effort; it is unclear or not useful

Answer 2

Statistics questions that involve player & DM decisions are (usually) opinion-based.

From Wikipedia:

Two main statistical methods are used in data analysis: descriptive statistics, which summarize data from a sample using indexes such as the mean or standard deviation, and inferential statistics, which draw conclusions from data that are subject to random variation.

Notice a major commonality between those two methods: they work on data. No data means no statistics.

Consider that question about average chance-to-hit per level. To give a proper statistical answer to that question, we would need a dataset with a large number of sample attacks, including at least the attacker's level & attack bonus, the target's AC, and whether the attack has advantage or disadvantage. Further, that dataset would need to actually be relevant to the asker.

Now, it's admittedly possible that an answerer could get a dataset relevant to the asker by either A) conducting a highly expensive worldwide survey of gaming groups to get a statistically useful representative sample of attacks over the course of each group's campaign, or B) stalking & spying on the asker's group to build a representative sample of that group's attacks over the course of a campaign. In practice, however, that's not what happens; instead, answerers generally just come up with their best guess (i.e. their opinion) for a dataset.

Some statistics questions are answerable purely from publicly available data (e.g. "What's the average Constitution save bonus by level of monsters in the Monster Manual?"), and those are perfectly acceptable.