The Paradox of Instruction: Why Safe AI Needs Architectural Gates, Not Better Prompts
Start with two numbers that point in opposite directions. In a study published this year, an automated check bolted onto a language model got things wrong seventy per cent of the time in one specific situation. In that same situation, the plain instruction it was supposed to improve on did better. Yet on ordinary questions the model could answer perfectly well, that same instruction refused to answer one in ten of them, and the automated check answered every one correctly.
Two mechanisms, two blind spots, running in opposite directions, from the same underlying problem: a model cannot reliably tell, from the inside, when it is about to be confidently wrong.
This is the finding underneath a debate that has run for a couple of years now. When you want an AI system to stop making things up, you have two broad options. You can tell it not to: a system prompt that instructs the model to decline when it is unsure. Or you can build a separate check that looks at what the model produced and decides independently whether to let it through. Tell it to be careful, or gate it from the outside. The first is an instruction. The second is an architectural decision about where the check lives.
A March 2026 preprint by Angelina Hintsanen ran both, separately and combined, across three models and five deliberately engineered situations. It is not a peer-reviewed journal paper, and I will come back to how much weight that lets it carry. It is worth reading anyway, because it does something most safety claims do not: it isolates the instruction from the gate and measures each one, item by item.
The paradox, stated plainly
Here is the result that should stop anyone reaching for the simple headline. On the full fifty-item evaluation, for the two more capable models tested, the instruction on its own scored exactly the same as the full system with the extra check bolted on. Both reached ninety-eight per cent accuracy and zero fabricated answers. Adding the structural gate to an already well-instructed capable model produced no measurable improvement at all. Used entirely on its own, the gate was worse than simply telling the model to be careful.
If that were the whole story, prompting would already be enough, and this article would not exist. It is not the whole story, and the reason it is not is the actual argument. The aggregate score hides the failure. The two mechanisms fail in different places, and because each failure covers a small fraction of the fifty items, the net effect washes out when you average everything together. You have to look at where each one breaks.
Where each mechanism actually fails
The paper built one situation called Conflicting Evidence: the documents retrieved for the model genuinely disagree with each other. An old version of a fact sits in one place, a newer version in another, and nothing flags which is current.
In that situation, the structural gate on its own was wrong seventy per cent of the time for both capable models. The reason is precise and it matters. The gate computes what the paper calls a support-deficit score, built from three signals: whether the model answers the same way when asked repeatedly, whether the answer survives being paraphrased, and how much the answer's wording overlaps the retrieved text. Every one of those signals is measured from the model's own output. When the model picks one side of the conflict and states it confidently and consistently, all three signals look healthy. The score stays below the threshold. The gate waves it through. The model was, in the authors' own phrase, "confidently wrong," and a check computed entirely from the model's own behaviour has no way to see it. From the inside, a confident wrong answer looks exactly like a confident right one.
The instruction has the opposite blind spot. On plainly Answerable questions, the questions the model could get right, the instruction to decline when uncertain made the two capable models refuse ten per cent of items they would otherwise have answered correctly. Told to be careful, they became too careful. The gate, on those same questions, answered all of them. Combine the two mechanisms and the hallucination rate on the Conflicting Evidence situation drops to zero. Neither mechanism alone gets there. The authors summarise it cleanly: instruction can over-abstain on answerable items, while the gate can miss confident confabulation under conflicting evidence.
What happens under real pressure
There is one situation where the gap is least ambiguous. The paper ran a second, harder test: a hundred adversarial questions with the supporting context deliberately withheld, drawn from the TruthfulQA set, where the honest answer is usually to decline. Here the instruction alone, on the weakest model, managed correct refusal only sixty-two per cent of the time. It hallucinated on the rest. The configurations that included the structural gate held between ninety-eight and one hundred per cent correct refusal across all three models.
This is the half of the argument that carries least qualification. When the ground is deliberately pulled out from under the model, being told to be careful is not enough. The independent check, blind to the model's confidence and looking instead at whether there is anything to ground an answer, is what holds the line.
A second line of evidence, from a different direction
One small study on three now-superseded models is a thin foundation, and I will not pretend otherwise. So it is worth noting that a completely separate piece of research, using a different method on different failures, points the same way. One April 2026 study, from Digital Applied, ran five thousand prompts across five current frontier models and measured citation fabrication: how often a model invents a source. It found that grounding the model's claims in retrieved documents, an architectural decision about where the claims come from, cut citation fabrication by seventy-five to ninety per cent. Instructing the model to be careful, the "say you don't know" approach, cut it by five to fifteen per cent.
Two studies, two different failure modes, two different mechanisms, one converging conclusion: what checks a claim matters more than what tells the model to be careful. This is genuine methodological diversity rather than one result restated, and it is worth being honest that it is convergence, not replication. No second team has reproduced Hintsanen's specific experiment. The two lines of evidence support the same structural point by different routes; they do not verify each other.
What V.E.R.A. already treated as settled, and does differently
Regular readers will see where this lands. V.E.R.A., Andreas Hamberger's Verified Existence and Reason Architecture, is built on exactly this premise: that a check on a claim has to live somewhere the model's own confidence cannot reach it. Its Name-Type-Predicate framework, NTP for short, sorts every claim in an output by the kind of checking it needs. An e-type claim asserts that something is real, an entity exists, an event happened, a quantity was measured, and it requires an external lookup. An n-type claim follows logically from established premises and is checked by examining the logic. The Krampitz Load Analyser does that sorting before any verification runs.
The distinction with this week's gate is specific, not a claim of superiority. V.E.R.A.'s E! Verification Service takes an entity name and asks an external corpus, seeded from open knowledge bases, a categorical question: does this exist. It returns EXISTS, does NOT EXIST, or UNKNOWN, with no confidence score attached. It never asks how sure the generating model was, or how consistently it answered. That is the whole point. Because it never consults the model's own confidence, it cannot inherit the blind spot that made the paper's gate wave through a confident wrong answer under conflicting evidence. First-party benchmarks on the architecture record a 28.9 per cent hallucination reduction and a 21.1 per cent truthfulness improvement; those are not this paper's numbers and are not compared to them.
Say plainly what this is not. The paper does not validate V.E.R.A., and V.E.R.A. is not a more advanced version of the paper's gate. They decompose different things. The paper's method runs two checks on the same answer to cover each other's blind spot. V.E.R.A. routes different kinds of claim to different, purpose-built methods. The shared principle, do not verify a compound judgement as one undifferentiated block, is real, and it is the only bridge. V.E.R.A. does not verify its own reasoning, and this is a different system entirely from the several unrelated products that share the acronym.
What this does not prove
Honesty about the evidence is a credibility asset in this series, so state the limits without softening them. The primary study is fifty items across five situations, plus the hundred-item stress test. All three models come from a single vendor and are now behind the current frontier. It is a single-author preprint extending an earlier workshop paper; it is not, despite one description in circulation, published in a journal, and its author's institutional affiliation could not be confirmed from the paper itself. The method needs roughly twenty-two separate calls to the model for each query, and the author explicitly describes it as not production-ready, and does not claim it solves hallucination in general. This is a young, narrow, honestly scoped result. It is stronger for saying so.
The governance layer
The distinction shows up above the level of any single model, too. New Zealand's Government Chief Digital Office publishes Responsible AI Guidance for the Public Service, last updated in February 2025 and non-binding. Its recommended way to check an AI output is to "test your model and evaluate its outputs to assess performance across a range of scenarios and prompts," and to keep people in the loop. That is a scenario-and-prompt-based testing method. It is, described factually, the very approach this week's research shows has a blind spot under conflicting evidence, and the guidance does not mention an architecturally independent structural check as a complement. That is an observation about the current state of published guidance, not a call for anyone to mandate anything.
A contrasting example sits at the level of a whole national framework. China's Implementation Opinions on intelligent agents, issued jointly by three agencies and effective 15 July 2026, set a three-tier structure: some decisions stay with a human, some need human approval before the agent acts, and some run autonomously within a delegated scope with the user keeping override rights. Described factually, that is an external procedural check that exists independently of whatever instructions the agent itself runs under. Governance as architecture, not governance as instruction. It addresses agent authority rather than output truthfulness, and the two should not be conflated, but the structural logic is the same one this article has traced.
There is an open-source dimension worth naming before the wider implication. Every mechanism in this week's research can be trusted only if its internals are open to inspection. The structural gate's scoring, the instruction beside it, and the corpus a lookup checks against are all auditable only when someone can read them. V.E.R.A. is published under the GNU General Public Licence, version three, on GitHub, and its existence corpus is seeded from open knowledge bases, Wikipedia and Wikidata, so the question it answers about a named entity can be traced by anyone. The March preprint released its own method for independent reproduction. An architecturally separate check earns trust only when its logic is open to the people relying on it. A closed check asks for the same blind faith as the model it is meant to police.
The implication on the defence and sovereignty side is worth naming directly. The failure this research isolates, a system that is confidently wrong and cannot tell from the inside, is the exact failure that accountability frameworks for lethal autonomous weapons exist to prevent. At the United Nations, the Convention on Certain Conventional Weapons runs a Group of Governmental Experts on lethal autonomous weapons systems, and its work turns on one requirement: a decision to use force must rest on a determination that can be checked independently, not on a system's own confidence that its target is lawful. A target identification is an existence claim, this object is a lawful military target, and an instruction to be careful does not verify it. Governance asymmetries follow between states that treat that check as an architecturally separate step and states that treat a confident output as sufficient.
Arc 4 of this series has now asked four questions: what a verifier should output, how it reaches its grounding, how much verification effort is worth, and, this week, where the check has to live. The answer is not "always outside the model." It is narrower and more useful than that. The check has to live outside the model in the one place where being told to be careful and being confidently wrong look identical from the inside.
If you run an AI system that has behaved well for months, what is the one situation you have never tested it against: the case where careful and confidently wrong would look exactly the same? I would like to read how your team decided whether an independent check was worth building, and what finally settled the argument.
The views expressed in this article are entirely my own, informed by more than 30 years of professional experience in architecture, security, and technology leadership in New Zealand. They do not represent the views of my employer, any government agency, or the New Zealand government. My commentary on legislation and policy is analytical, drawing on publicly available sources and my professional expertise in architecture, security, and AI governance. I follow the Public Service Commissioner's Code of Conduct for the Public Sector and social media guidance.
Andreas Hamberger is a New Zealand leader in Architecture & Security and Associate Member of the Institute of Directors. V.E.R.A. (Verified Existence & Reason Architecture) is an open-source logic engine available on GitHub.
I use AI tools, including Sudowrite, Claude, Perplexity AI, DeepSeek AI, ChatGPT, Grok, Copilot, Openart and Gemini, as deliberate production tools, not ghostwriters. This is consistent with my position: AI amplifies human judgement; it does not replace it. The frameworks, arguments, and editorial decisions in this series are original work. AI accelerated the process. The thinking is mine.
References
[1] Hintsanen, A. "Hallucination as output-boundary misclassification: a composite abstention architecture for language models" (abstract). arXiv, 12 March 2026. https://arxiv.org/abs/2604.06195
[2] Hintsanen, A. "Hallucination as output-boundary misclassification: a composite abstention architecture for language models" (full text). arXiv, 12 March 2026. https://arxiv.org/html/2604.06195
[3] Digital Applied Team. "AI Hallucination Rate Benchmarks 2026." Digital Applied, 23 April 2026. https://www.digitalapplied.com/blog/ai-model-hallucination-rate-benchmarks-2026-study
[4] Rimon Law. "China AI Law Brief." Rimon Law, retrieved 31 July 2026. https://www.rimonlaw.com/china-ai-law-brief/
[5] Pebblous. "China AI Agent Decision Tiers." Pebblous blog, retrieved 31 July 2026. https://blog.pebblous.ai/blog/china-ai-agent-decision-tiers/en/
[6] Government Chief Digital Office. "Responsible AI Guidance for the Public Service: GenAI" (overview). digital.govt.nz, last updated 3 February 2025. https://www.digital.govt.nz/standards-and-guidance/technology-and-architecture/artificial-intelligence/responsible-ai-guidance-for-the-public-service-genai/overview
[7] Hamberger, A. "V.E.R.A. (Verified Existence and Reason Architecture) documentation." Te Pono Limited, January 2026. GitHub. (No captured document URL; cited in full without a URL.)
[8] United Nations. "Convention on Certain Conventional Weapons, Group of Governmental Experts on emerging technologies in the area of lethal autonomous weapons systems." Geneva. (No captured document URL; cited in full without a URL.)