Can we develop a system to avoid/discourage subsequent homebrew critique question edits?

Question

While our policy on homebrew-critique questions makes a lot of sense as the questions are certainly able to be answered effectively by experts and fit all the criteria for an on-topic question, these questions often take a sharp turn fast.

Soon after the answers to these questions come in we often get one of two paths.

1. The answers say the homebrew is balanced, one answer may be accepted, and the question is largley left alone. (Example)
2. The answers say the homebrew is unbalanced and the original poster sometimes makes edits to the question accounting for the points made in the answers. (Example)

The latter case is the problem I have with our current approach as it devalues current answers (which hold value not only in addressing the homebrew but for other homebrewers to get an idea for how to approach balancing their handiwork). It also leads to a seesaw of edits (Q > A > Q > A) that StackExchange does not handle well.

One thing I've done in the past is stop this chain early by just inviting the person to chat to refine the homebrew to a high level (which is also the most common choice for me when a question is off-topic leading me to believe something is up). (Example)

One option seems to be to close it once the OP has started down the seesaw chain (Example), but this doesn't seem right to me either.

Are there any ways to address this issue more formally or should we adjust our policy on homebrew questions as a whole? On the other hand, is our current method fine and are there few enough cases of this latter behavior that our policy is okay?

Answer 1

It’s often worth exploring the Stack Exchange network to see if other sites have found good solutions to a given problem.

For example, in this case, we have Code Review, which is basically an entire Stack devoted to “homebrew” critique. They therefore have had to address this issue, and appear to be quite successful. So taking a look at their policy could be useful.

So here is Code Review’s policy on editing in revisions:

You have several options for follow-ups:

1. Accepting an answer

[...]

2. Posting a new question

[...]

3. Posting a self-answer

[...]

4. Posting on an external site

[...]

In addition...

• You must not edit the code in the question,

• You also should not append your revised code to the question.

---

To end on a positive note, here is an example of what I consider to be a fruitful iterative review:

1. Python Port Scanner
2. Python Port Scanner 2.0
3. Python Port Scanner 2.1

^{(see full policy answer for detailed descriptions, explanations, and examples)}

Relevant to the situation here is #2, and the two don’ts. I think this is absolutely the approach we should generally take here. I don’t think there’s any particular risk of the site being inundated with these iterations; we don’t really get that many homebrew critique questions, and many of them don’t really require iteration. (For those that don’t really need iterating, the other suggestions are appropriate.)

Answer 2

The stack should not be used for iterative design

The way the Q&A system works on this site is not intended for, nor should it be used (IMO), for iterative design processes.

Edits for clarity or for focusing a question or for major improvement (eg allowing a closed non-stackable question to be reopened) are all good reasons to edit and the answers should follow those edits. These edits are things that are terminal (in that they have a defined endpoint: when the question's quality issue is resolved) and have to do with improving the quality of the question.

However, a question should generally not be allowed to make major revisions that continuously change the core nature of the content of the question in response to feedback from answers. As you point out, the biggest offenders of this tend to be questions about homebrew in my experience.

In a previous meta on this topic (When a Question changes, should answers change, too?) this point was brought up:

If the question was changed because the OP has refined what they wanted to ask as a result of feedback on their question... That's a grey area, and we'll have to deal with it on a case-by-case basis. In some cases it'll be appropriate to revert the change and ask a new question; In others, to let the edit stand and update or downvote and delete old answers.

I believe this behavior is one of those cases that we can have a generalized response to. This behavior of trying to iterate homebrew designs using answers as a feedback loop fits into this category I'd say. And further, that this usage should not be allowed because our system is just not made to handle such things. If a question about homebrew content is allowed to continuously make changes based on the feedback in the answers, it is going to leave a trail of irrelevant and outdated answers and maybe not even end up getting an answer at all. This is not how the system is supposed to function.

How should we handle the behavior when it is observed?

I don't think we need a new system to handle this. In my opinion we already have all the tools we need.

Revert edits and comment

When we see OP making an edit on their homebrew question that incorporates major feedback from an answer such that it changes the core question in any significant way, that edit should be reverted. Then a comment should be left explaining to OP why the Stack should not be used in this manner and pointing them towards possibly writing a self-answer (see below) or asking a new question.

When writing answers, look beyond the obvious flaws (if possible)

It is somewhat easy to look at homebrew, find a glaring flaw, and tell OP that it is broken based on that. An answer that does only this can be useful for sure, but not always as useful as they can be.

For example, if the obvious flaw can easily be removed, suggest that and evaluate the rest of the homebrew based off of that so that OP gets feedback about the whole homebrew and not just one glaring flaw in it. Sometimes the flaw can be serious enough to sink the whole homebrew sure, but that is certainly not always the case. Giving a more complete answer might disincentivize OP from iterating their question because the answer has already suggested a fix and next steps and evaluated the rest based off of that.

Encourage self-answering

(Based on @Helwar's excellent answer)

As always, OP is allowed and encouraged to self-answer. This might be useful in the case that they have taken the feedback from one or multiple answers and come up with a drastically new homebrew that they are content with as a result. Instead of putting this in the question as an edit which one might be tempted to do, it could be appropriate to encourage OP to add it as an answer instead. However, it should be made clear that they cannot ask questions in their answer so what they are putting there should be a solution to their issue and not something that they still require assistance with.

Answer 3

I don't usually post in meta, so I'm not used to it and maybe my approach or format are wrong, but I felt like I could give my opinion here.

The problem about homebrew is that by design, you look for improvements on your work. So as suggestions on how to do it start to appear, an instinctive reaction would be to edit the question with the changes and ask "Is it good now?".

Obviously as you said this devalues the already given answers, so it's a bad practice. If the asker keeps editing until he gets the OK on the answers, all we get are pretty "OK" homebrews, with answers that say "It's fine", and nothing else. You lose the building process, as @David Coffron explained so well.

Self-answer homebrew questions

My suggestion would be to encourage self-answering homebrew questions if the answers from other users don't feel right to the asker. In that case, the asker can use all the knowledge the other answers have and build his improved homebrew, giving credit to whom it is deserved, in the answer.

This answer can be further edited, always adding the steps of the homebrew building, never deleting it alltogether and building anew, so not to lose the process.

In the end we would have the original question, an accepted answer with the final homebrew made with the help of the comunity, and all the other answers that are steps towards building that homebrew.

It's not perfect, but I think It's better than closing the question to edits, or closing it alltogether. The alternative is asking about your homebrew, getting answers, then asking again in a new question with your edited homebrew, getting answers, rinse and repeat.