The Dog That Reached Production

A serious essay on labour-market outcomes went live on this publication with a photograph of a dog as the hero image. The model that produced the image hallucinated. Three separate checks should have caught it. None did, for reasons that turn out to be more interesting than the dog. The lesson is not that AI hallucinates. It is that reliable AI systems are built less by models that never fail and more by organisations that fail safely when they do.

The dog. The brief asked for a photograph illustrating a piece on labour-market displacement. The image model returned, with great confidence and excellent lighting, a dog. Not a metaphorical dog. A literal one. The piece shipped, the dog reached readers, and the dog has since acquired a name internally that I will not be repeating here.

Why image models do this. Hallucination is the term of art for when a generative model produces output that is fluent, plausible-looking, and untethered from the input. Image models hallucinate for the same structural reason language models do: they are trained to produce something that looks right rather than something that is right. There is no internal step where the model checks its output against the brief and notices a mismatch. The model does not know it has drawn a dog. It only knows the pixels score well on the patterns it learned.

This is not a bug that gets fixed in the next release. It is a property of how these systems work. Any production pipeline that assumes the model will not hallucinate is a pipeline that will, eventually, ship a dog.

The check that worked. Our QA agent — a second model whose job is to review outputs before a human sees them — caught it. It flagged, in plain text, that the hero image did not match the article's subject matter. This is the part of the story I want to be honest about: the safety layer did its job. The technology was not the failure.

The check that had been quietly weakened. A workflow change some weeks earlier had reclassified that specific QA warning from blocking (the piece cannot move forward until a human resolves the flag) to advisory (the flag appears in the review card, the human can proceed anyway). The reasoning at the time was reasonable: the QA agent had been over-flagging on stylistic image choices, and editors were getting alert fatigue. The fix was to soften the gate. The cost of that fix, paid weeks later, was that a real flag arrived wearing the same clothes as the false ones.

Blocking versus advisory is the single most consequential design choice in a human-in-the-loop pipeline. A blocking check assumes the human will sometimes be wrong and the system should refuse to proceed. An advisory check assumes the human will read carefully and decide well. The first is engineering; the second is hope.

The check that the interface defeated. The human reviewer opened the review card in Slack. The review card, due to a rendering issue, did not display the actual generated image. It displayed a placeholder. The reviewer, working at pace, saw the QA flag, saw no obvious problem in the card, and approved.

This is the layer that gets least attention in discussions of AI safety and matters most in practice. The reviewer was not negligent. The reviewer was working with an interface that, at the moment of the decision, was lying to them by omission. A human-in-the-loop is only as good as the loop. If the loop does not show the human what they are approving, the human is not really in it.

The Swiss-cheese model. James Reason's framework for accident analysis, borrowed from aviation and medicine, treats every safety layer as a slice of Swiss cheese. Each slice has holes — places where it fails. Most of the time the holes do not line up, and an error caught by one layer is stopped by the next. Accidents happen when the holes line up: the model hallucinates, and the gate has been softened, and the interface fails to render, and the reviewer is moving fast. Any one of those, alone, would have stopped the dog. All four, together, did not.

This is what people mean by defence-in-depth. The point is not that any single layer is reliable. The point is that the layers fail independently, so the joint probability of all of them failing on the same item is low. Defence-in-depth breaks down when the layers are not actually independent — when, for example, a workflow change weakens one layer and a UI bug weakens another and nobody has mapped how those interact.

What this says about reliable AI. The dominant public conversation about AI reliability is about the model: will it hallucinate, can we make it not hallucinate, how often does it hallucinate. This is the wrong unit of analysis for production systems. The model will hallucinate. The interesting question is what your organisation does on the occasions when it does.

Reliable AI deployments look less like reliable models and more like reliable hospitals: multiple overlapping checks, explicit gating decisions, interfaces designed for the case where the human is tired, and a culture that publishes its failures rather than burying them. None of this is exciting. All of it is the actual work.

What we changed. Three things, named so that the same holes cannot line up the same way. First, hero-image / subject-matter mismatch is now a blocking check, not advisory; an editor can override it, but the override is logged and reviewed weekly. Second, the review card now embeds the actual generated asset, and renders a visible error state — not a placeholder — when it cannot. Third, any reclassification of a QA check from blocking to advisory now requires sign-off from two editors and a documented rationale, on the principle that softening a gate is a load-bearing decision and should feel like one.

None of this guarantees we will not ship another dog. It guarantees that, if we do, it will be a different dog, arriving through a different combination of holes, which we will then close in turn. That is what the job actually is.

The dog, for what it is worth, was a very good dog. The piece it accompanied deserved better, and so did the readers who clicked on it expecting a photograph relevant to their working lives. We owe both an apology, and this essay is part of it.

Footnotes

¶