A frame challenge is where an author answers a question in a wholly different way the querent never asked for, or potentially expressly forbade, but in a way the author feels will actually solve the problem. (Or otherwise improve the querent's life quality or prevent them from making some terrible mistake.) This is as opposed to answers which answer the question at face value.
The “frame” being challenged here is specifically the way the question was framed — the way they put it forward, the parameters they offered, the kinds of answers they're driving for.
“Frame challenge” is Stack Exchange jargon. You might know this by some other more common phrases such as challenging the assumptions of a question, or questioning its premise.
Frame challenges are risky; sometimes they're a good idea and sometimes you just cop a whole lot of downvotes. We advise you also answer the question at face value, though you may prefer not to do so (and sometimes, it's a very good idea to not do so).
Q: My wizard needs to get from the Undead Asylum to Blighttown by sundown. What's the fastest steed for riding there?
- Regular answer: Ask for a bronze steed down the hill. It's very quick, but the earliest you can make it there is past midnight.
- Frame challenge answer: Don't use a steed to get there at all. There's a crow nearby who'll fly you there in minutes.
Q: This other player killed my character. I'm furious, and I want to harass him and give him hell until he cries and leaves the group never to return. How can I do that?
- Regular answer: Start by slipping poison into his character's drinks, and...
- Frame challenge answer: Don't do this at all, you'd be terrible for doing it and you'd bring the game down. Talk to the dude and with your GM to resolve this.
Q: How can I teach my warlock foo? I refuse to do (thing), it's bad because reasons, so don't answer with that.
- Regular answer: You can't. Sorry.
- Frame challenge answer (sort of): (thing) is the only way to do it. It's actually a good idea, so you should do this. Plus I think you're mistaken about those reasons, let me explain...