Designing for models that are wrong

Most AI features are designed as though the model will be right. The happy path gets a beautiful streaming animation. The failure path gets a thumbs-down icon.

But a model that is right ninety percent of the time will be wrong in front of your user within the first ten interactions. What happens then is not an edge case. It is the product.

The failure path is the product

This is a design problem before it is a modelling problem. Improving accuracy is expensive, slow and eventually asymptotic. Improving what happens when the output is wrong is comparatively cheap, entirely within your control, and it is what determines whether somebody trusts the feature enough to keep using it.

The useful reframing: you are not building a system that answers. You are building a system that proposes, plus a set of affordances for a person to accept, adjust or reject the proposal. Once the feature is described that way, the design work stops being about the output and starts being about the controls around it.

Correction should be cheap, specific and local

Cheap

One interaction, not a re-prompt. If the only way to fix a wrong output is to rewrite the request and run it again, the user pays the full cost of generation for a partial error — and loses everything that was right the first time along with the part that was not.

Specific

Let people point at the wrong part rather than rejecting the whole output. A thumbs-down tells you something failed and nothing about what. An interface that lets somebody select the offending sentence, field or row gives them agency, and gives you a far better signal than a binary ever will.

Local

The correction should apply where they are, without losing the surrounding work. Anything that resets state, closes a panel or discards nearby edits teaches the user that correcting is dangerous — and the lesson they take from that is to stop trusting the feature, not to correct more carefully next time.

Confidence is harder than it looks

Showing confidence sounds like the obvious answer to uncertainty, and it is usually the wrong one. Numeric confidence is close to meaningless to most users and frequently miscalibrated in the model itself, so you end up presenting a precise-looking number that is not precise.

Hedging language is better, and degrades quickly. If every response is wrapped in “I think” and “you may want to verify”, the hedge stops carrying information within a day and becomes texture that people read straight past.

The most useful pattern we have found is showing the basis rather than asserting a probability: the sources, the rows it used, the fields it read. That lets the user apply their own judgement, which they are far better placed to do than the model is, and it fails gracefully — a visible source that looks wrong is immediately actionable in a way that “72% confident” never is.

Refusal deserves design attention

A refusal that explains what it will not do, and offers the nearest thing it will, reads as competence. One that apologises at length reads as evasion. One that refuses silently reads as broken.

The same applies to partial capability. “I can do this part but not that part” is a genuinely useful response, and it is rarely designed for, because it does not fit an interface built around producing a single answer.

Undo is the most underrated control

Anywhere a model takes an action rather than producing text — moving data, sending something, changing a record — undo does more for trust than any amount of confidence display. It converts an irreversible risk into a reversible one, which is the difference between a feature people try and a feature people avoid.

Where undo is genuinely impossible, that is the moment for a confirmation step. Where it is possible, a confirmation step is usually just friction charged to every correct action in order to insure against the occasional wrong one.

Test with a model that is wrong on purpose

Most AI features are demoed on the happy path and reviewed on the happy path, which means the correction experience is frequently the only part of the product nobody has actually looked at. It ships untested because it was never on screen during a review.

The fix is procedural rather than clever: put deliberately wrong output in front of the team during design review. Stub a response with the wrong name in it, a fabricated row, a plausible-but-incorrect summary. Then watch how many interactions it takes to recover, and whether anything the user had already done gets lost on the way.

It is uncomfortable in a way that demos are designed not to be, and it surfaces the two failures that matter most: corrections that cost more than starting again, and errors the interface presents with exactly the same confidence as everything else.

Making the boundary legible

None of this is about making the model seem smarter. It is about making the boundary between the model's capability and the user's judgement legible enough that the user can work with it.

People are remarkably tolerant of a tool that is wrong in ways they can see and fix, and remarkably intolerant of one that is wrong in ways they cannot predict. Legibility, not accuracy, is what separates the two — which is why the correction path deserves the same care as the generation path, and usually gets a fraction of it.

This is the practical half of the argument in how AI is reshaping creative design: the model proposes, the person decides, and the interface's job is to make that division obvious. It is the shape of most of the AI product work we take on.