19
\$\begingroup\$

While our policy on homebrew-critique questions makes a lot of sense as the questions are certainly able to be answered effectively by experts and fit all the criteria for an on-topic question, these questions often take a sharp turn fast.

Soon after the answers to these questions come in we often get one of two paths.

  1. The answers say the homebrew is balanced, one answer may be accepted, and the question is largley left alone. (Example)
  2. The answers say the homebrew is unbalanced and the original poster sometimes makes edits to the question accounting for the points made in the answers. (Example)

The latter case is the problem I have with our current approach as it devalues current answers (which hold value not only in addressing the homebrew but for other homebrewers to get an idea for how to approach balancing their handiwork). It also leads to a seesaw of edits (Q > A > Q > A) that StackExchange does not handle well.

One thing I've done in the past is stop this chain early by just inviting the person to chat to refine the homebrew to a high level (which is also the most common choice for me when a question is off-topic leading me to believe something is up). (Example)

One option seems to be to close it once the OP has started down the seesaw chain (Example), but this doesn't seem right to me either.

Are there any ways to address this issue more formally or should we adjust our policy on homebrew questions as a whole? On the other hand, is our current method fine and are there few enough cases of this latter behavior that our policy is okay?

\$\endgroup\$
28
\$\begingroup\$

It’s often worth exploring the Stack Exchange network to see if other sites have found good solutions to a given problem.

For example, in this case, we have Code Review, which is basically an entire Stack devoted to “homebrew” critique. They therefore have had to address this issue, and appear to be quite successful. So taking a look at their policy could be useful.

So here is Code Review’s policy on editing in revisions:

You have several options for follow-ups:

  1. Accepting an answer

[...]

  1. Posting a new question

[...]

  1. Posting a self-answer

[...]

  1. Posting on an external site

[...]

In addition...

  • You must not edit the code in the question,

  • You also should not append your revised code to the question.


To end on a positive note, here is an example of what I consider to be a fruitful iterative review:

  1. Python Port Scanner
  2. Python Port Scanner 2.0
  3. Python Port Scanner 2.1

(see full policy answer for detailed descriptions, explanations, and examples)

Relevant to the situation here is #2, and the two don’ts. I think this is absolutely the approach we should generally take here. I don’t think there’s any particular risk of the site being inundated with these iterations; we don’t really get that many homebrew critique questions, and many of them don’t really require iteration. (For those that don’t really need iterating, the other suggestions are appropriate.)

\$\endgroup\$
  • 7
    \$\begingroup\$ It would be very helpful for us to have a guide similar to theirs on here. \$\endgroup\$ – doppelgreener May 28 '18 at 0:43
  • 2
    \$\begingroup\$ Wow, this is an excellent find and certainly something we should consider borrowing for this site \$\endgroup\$ – Wibbs May 30 '18 at 16:38
10
\$\begingroup\$

The stack should not be used for iterative design

The way the Q&A system works on this site is not intended for, nor should it be used (IMO), for iterative design processes.

Edits for clarity or for focusing a question or for major improvement (eg allowing a closed non-stackable question to be reopened) are all good reasons to edit and the answers should follow those edits. These edits are things that are terminal (in that they have a defined endpoint: when the question's quality issue is resolved) and have to do with improving the quality of the question.

However, a question should generally not be allowed to make major revisions that continuously change the core nature of the content of the question in response to feedback from answers. As you point out, the biggest offenders of this tend to be questions about homebrew in my experience.

In a previous meta on this topic (When a Question changes, should answers change, too?) this point was brought up:

If the question was changed because the OP has refined what they wanted to ask as a result of feedback on their question... That's a grey area, and we'll have to deal with it on a case-by-case basis. In some cases it'll be appropriate to revert the change and ask a new question; In others, to let the edit stand and update or downvote and delete old answers.

I believe this behavior is one of those cases that we can have a generalized response to. This behavior of trying to iterate homebrew designs using answers as a feedback loop fits into this category I'd say. And further, that this usage should not be allowed because our system is just not made to handle such things. If a question about homebrew content is allowed to continuously make changes based on the feedback in the answers, it is going to leave a trail of irrelevant and outdated answers and maybe not even end up getting an answer at all. This is not how the system is supposed to function.

How should we handle the behavior when it is observed?

I don't think we need a new system to handle this. In my opinion we already have all the tools we need.

Revert edits and comment

When we see OP making an edit on their homebrew question that incorporates major feedback from an answer such that it changes the core question in any significant way, that edit should be reverted. Then a comment should be left explaining to OP why the Stack should not be used in this manner and pointing them towards possibly writing a self-answer (see below) or asking a new question.

When writing answers, look beyond the obvious flaws (if possible)

It is somewhat easy to look at homebrew, find a glaring flaw, and tell OP that it is broken based on that. An answer that does only this can be useful for sure, but not always as useful as they can be.

For example, if the obvious flaw can easily be removed, suggest that and evaluate the rest of the homebrew based off of that so that OP gets feedback about the whole homebrew and not just one glaring flaw in it. Sometimes the flaw can be serious enough to sink the whole homebrew sure, but that is certainly not always the case. Giving a more complete answer might disincentivize OP from iterating their question because the answer has already suggested a fix and next steps and evaluated the rest based off of that.

Encourage self-answering

(Based on @Helwar's excellent answer)

As always, OP is allowed and encouraged to self-answer. This might be useful in the case that they have taken the feedback from one or multiple answers and come up with a drastically new homebrew that they are content with as a result. Instead of putting this in the question as an edit which one might be tempted to do, it could be appropriate to encourage OP to add it as an answer instead. However, it should be made clear that they cannot ask questions in their answer so what they are putting there should be a solution to their issue and not something that they still require assistance with.

\$\endgroup\$
  • 4
    \$\begingroup\$ I actually like this one, too. That way the question remains original and the answer gets the "update" and when accepted becomes the final version. Iterations should reside in the answer, but answerers need to include the full homebrew stats. \$\endgroup\$ – NautArch May 25 '18 at 14:24
9
\$\begingroup\$

I don't usually post in meta, so I'm not used to it and maybe my approach or format are wrong, but I felt like I could give my opinion here.

The problem about homebrew is that by design, you look for improvements on your work. So as suggestions on how to do it start to appear, an instinctive reaction would be to edit the question with the changes and ask "Is it good now?".

Obviously as you said this devalues the already given answers, so it's a bad practice. If the asker keeps editing until he gets the OK on the answers, all we get are pretty "OK" homebrews, with answers that say "It's fine", and nothing else. You lose the building process, as @David Coffron explained so well.


Self-answer homebrew questions

My suggestion would be to encourage self-answering homebrew questions if the answers from other users don't feel right to the asker. In that case, the asker can use all the knowledge the other answers have and build his improved homebrew, giving credit to whom it is deserved, in the answer.

This answer can be further edited, always adding the steps of the homebrew building, never deleting it alltogether and building anew, so not to lose the process.

In the end we would have the original question, an accepted answer with the final homebrew made with the help of the comunity, and all the other answers that are steps towards building that homebrew.

It's not perfect, but I think It's better than closing the question to edits, or closing it alltogether. The alternative is asking about your homebrew, getting answers, then asking again in a new question with your edited homebrew, getting answers, rinse and repeat.

\$\endgroup\$
  • 2
    \$\begingroup\$ Love the idea of leave the question alone, but have a self-answer to update with the changes suggested by other users. \$\endgroup\$ – NautArch May 25 '18 at 13:57
  • \$\begingroup\$ @NautArch it's a cool idea I'm just not sure if it fits with the idea of the stack as a Q&A. Might be a slippery slope (though I don't foresee exactly how at the moment) \$\endgroup\$ – David Coffron May 25 '18 at 14:01
  • 1
    \$\begingroup\$ This is a good suggestion that echoes the discussion here: rpg.meta.stackexchange.com/q/5276/23970. (I feel like there's another meta in that vein more fleshed-out somewhere around here, but the search is weak with this one.) \$\endgroup\$ – nitsua60 May 25 '18 at 14:07
  • \$\begingroup\$ @DavidCoffron Yeah, i was thinking about that, too. What answer do you accept? The OP's or the person who created it? Or now you have two identical answers. \$\endgroup\$ – NautArch May 25 '18 at 14:14
  • 4
    \$\begingroup\$ @NautArch, DavidCoffron It's a rare outcome, but it's a Stack-accepted practice to write a “this is what actually worked for me” answer that's a synthesis of answers. It has the benefit of being directly from experience, doesn't eclipse the other answers that might help other readers more, and if self-accepted still has to compete on score to rise to the top. We don't see that dynamic very often but it's a fit for SE. \$\endgroup\$ – SevenSidedDie May 25 '18 at 18:29
-4
\$\begingroup\$

Close until clear/refined is a valid SE course of action

We do that for a lot of other questions, so homebrew should not be exempt from that process.

One option seems to be to close it once the OP has started down the seesaw chain (Example), but this doesn't seem right to me either.

If the question isn't clear, or is/becomes a moving target, then the question does need to be closed for a while as the refinement process happens: we, the community, engage with the asker until clarity and appropriate scope is achieved. Since that feels right for most kinds of questions, I am not sure why it doesn't feel right for a homebrew question. (Perhaps that's just a perspective thing).

\$\endgroup\$
  • \$\begingroup\$ It's just that the scope of the new question may be clearly defined. Only taking into consideration both questions can we have an "Unclear" closing \$\endgroup\$ – David Coffron May 25 '18 at 14:11
  • 2
    \$\begingroup\$ Presumably the question already had proper scope otherwise it would have been closed before the process began right? I think this question is assuming a valid stackable question to begin with. \$\endgroup\$ – Rubiksmoose May 25 '18 at 14:11
  • \$\begingroup\$ @Rubiksmoose In an ideal case, yeah, but how often does that arise? \$\endgroup\$ – KorvinStarmast May 25 '18 at 14:15
  • 1
    \$\begingroup\$ @KorvinStarmast: I guess my main point is that we are already assuming that the question has an appropriate scope and that the edits made to it were based off of suggestions from the answers not quality improvement edits to make the question stackable. \$\endgroup\$ – Rubiksmoose May 25 '18 at 14:19
  • \$\begingroup\$ @Rubiksmoose OK, let's see how this sorts out. I am not sure of the extent of the problem, but in my limited experience here the "moving target question" is not unique to homebrew. \$\endgroup\$ – KorvinStarmast May 25 '18 at 14:31
  • 7
    \$\begingroup\$ This doesn't look like a correct response to me: it's not that the question needs refinement/correction, it's that a legitimate question morphs into a new question in response to legitimate answers. The morphing probably shouldn't happen. \$\endgroup\$ – doppelgreener May 25 '18 at 14:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .