The stack should not be used for iterative design
The way the Q&A system works on this site is not intended for, nor should it be used (IMO), for iterative design processes.
Edits for clarity or for focusing a question or for major improvement (eg allowing a closed non-stackable question to be reopened) are all good reasons to edit and the answers should follow those edits. These edits are things that are terminal (in that they have a defined endpoint: when the question's quality issue is resolved) and have to do with improving the quality of the question.
However, a question should generally not be allowed to make major revisions that continuously change the core nature of the content of the question in response to feedback from answers. As you point out, the biggest offenders of this tend to be questions about homebrew in my experience.
In a previous meta on this topic (When a Question changes, should answers change, too?) this point was brought up:
If the question was changed because the OP has refined what they wanted to ask as a result of feedback on their question... That's a grey area, and we'll have to deal with it on a case-by-case basis. In some cases it'll be appropriate to revert the change and ask a new question; In others, to let the edit stand and update or downvote and delete old answers.
I believe this behavior is one of those cases that we can have a generalized response to. This behavior of trying to iterate homebrew designs using answers as a feedback loop fits into this category I'd say. And further, that this usage should not be allowed because our system is just not made to handle such things. If a question about homebrew content is allowed to continuously make changes based on the feedback in the answers, it is going to leave a trail of irrelevant and outdated answers and maybe not even end up getting an answer at all. This is not how the system is supposed to function.
How should we handle the behavior when it is observed?
I don't think we need a new system to handle this. In my opinion we already have all the tools we need.
Revert edits and comment
When we see OP making an edit on their homebrew question that incorporates major feedback from an answer such that it changes the core question in any significant way, that edit should be reverted. Then a comment should be left explaining to OP why the Stack should not be used in this manner and pointing them towards possibly writing a self-answer (see below) or asking a new question.
When writing answers, look beyond the obvious flaws (if possible)
It is somewhat easy to look at homebrew, find a glaring flaw, and tell OP that it is broken based on that. An answer that does only this can be useful for sure, but not always as useful as they can be.
For example, if the obvious flaw can easily be removed, suggest that and evaluate the rest of the homebrew based off of that so that OP gets feedback about the whole homebrew and not just one glaring flaw in it. Sometimes the flaw can be serious enough to sink the whole homebrew sure, but that is certainly not always the case. Giving a more complete answer might disincentivize OP from iterating their question because the answer has already suggested a fix and next steps and evaluated the rest based off of that.
(Based on @Helwar's excellent answer)
As always, OP is allowed and encouraged to self-answer. This might be useful in the case that they have taken the feedback from one or multiple answers and come up with a drastically new homebrew that they are content with as a result. Instead of putting this in the question as an edit which one might be tempted to do, it could be appropriate to encourage OP to add it as an answer instead. However, it should be made clear that they cannot ask questions in their answer so what they are putting there should be a solution to their issue and not something that they still require assistance with.