Look It Up: Why the Model That Searched Beat the Model That Remembered
A model is handed a document and asked a plain question: does this document report a particular figure? It does not. The figure was never in the file. The model can do one of two things. It can say the figure is not there, or it can produce a number that sounds exactly like the kind of number a document like this would contain.
Eleven models were asked questions of this shape two hundred and four times. Ten of them fabricated at some rate. One answered correctly every single time. The difference between them was not which model had the best training run, the largest context window, or the highest score on any general leaderboard. The difference was what each model did with the document in front of it. Ten of them read it. One of them searched it.
That single distinction, reading against searching, is the whole finding, and it lands very close to a claim this series has made from its first episode: for the questions that assert something is real, looking it up beats trying to remember it.
The test: a metric that is not there
The benchmark is HALC-Bench, the "LLM Hallucination on Long-Context Retrieval Benchmark", published by the market-research firm AIMultiple and updated on 7 July 2026 [1]. It is worth being precise about what it is and is not. It is a commercial benchmark with a transparent method, not a peer-reviewed academic study, and at the time of writing it has not been independently replicated. I treat it here as a primary, publisher-sourced measurement, and I flag the single-source status rather than dress it up as settled science.
The design is deliberately uncomfortable. Researchers took fourteen real earnings-call transcripts, drawn from the conversational sections rather than the tidy summary bullets, so a model has to engage with actual source material. Into questions about those transcripts they planted two hundred and four fabricated metric claims: a carbon-emissions figure, a revenue sub-segment, a workforce statistic, each one plausible, contextually reasonable, and absent from the document. A model scores full marks only for correctly refusing to report the missing metric. Producing any value at all scores zero.
There is a second axis. Each trap was placed at three depths in the document, at one tenth, one half, and nine tenths of the way through the context, to test whether position changes the outcome. If the well-known "lost in the middle" effect were driving the failures, traps buried in the middle should draw more fabrications than traps near the ends.
This is the most dangerous failure mode to test, not the most dramatic. A model that invents an obviously absurd answer is easy to catch. A model that confidently reports a financial metric that was never measured does not look like a system that is malfunctioning. It looks like a competent analyst. The error is invisible precisely because the output is fluent.
What happened
The eleven configurations split into two groups that are not two settings of the same dial. They are two different answers to the question "does this metric exist in this document?" [5]
Ten of them were passive-context models. Each received the haystack inside its context window, with windows ranging from one hundred and fifty thousand to one million tokens, and answered directly: GPT-5.5, two Gemini variants, Claude Opus 4.8 and Sonnet 4.6, and five others. Every one of them fabricated on some fraction of the traps. The best performer in this group was GPT-5.5, reported at 98.1 per cent [3]. I flag that specific figure as reported rather than confirmed; the underlying page records GPT-5.5 as the strongest passive model but I have not pinned the exact percentage against it, and it belongs on the list of numbers to verify before anyone cites it as final. Even taken at face value, 98.1 per cent means the best passive model still fabricated on roughly four of the two hundred and four traps. Four out of two hundred and four is small as a percentage. It is not small if those four invented figures reach a decision.
The eleventh configuration was different in kind. It was a single system: the model paired with the Claude Code harness, given the eight-hundred-and-fifty-thousand-token document as an external file and a set of file-search tools. It did not read the document into its context and answer from memory. It searched the file for the metric it was asked about. It answered all two hundred and four traps correctly, at every position: 100.0 per cent [2].
One qualification matters enough to state plainly, because the article's title invites the wrong reading. The benchmark itself notes that the score measures the model together with its harness, not the model in isolation. This was not a controlled contest between one model and ten others on identical terms; the winning configuration used a different method, file search rather than passive reading, and the benchmark did not test that same model with its retrieval tools switched off. So the honest claim is not "one model is smarter than the rest". It is that the system that searched beat every system that read. When I say the model that searched, I mean the system that searched.
Then the position result, which is the quiet surprise. Across all ten passive models, there was no correlation between where a trap sat in the document and how often it was fabricated [1]. "Lost in the middle" did not explain who hallucinated. A model that invented a value did so at roughly the same rate whether the trap was near the start, the middle, or the end. The thing that separated the honest answers from the fabricated ones was not position in the context. It was whether the system looked the metric up or recalled it.
Why looking it up is not the same as reading it
Hold the two approaches next to each other, because the mechanism is the point.
A passive-context model reads the document and builds an internal representation of it. When you ask about a specific metric, it generates an answer from that representation blended with everything its training taught it about how companies of this kind usually report. If the metric is absent, the model has no explicit signal that says so. It has a strong prior about what a plausible value would be. Absence can therefore masquerade as presence: the model recalls the shape of the expected answer and produces it.
A system using retrieval tools does something structurally different. It queries the document for the specific metric. If the query returns nothing, the absence is not inferred; it is returned. An empty result set is a fact about the file, not a guess about the world. That is why the fabrication rate collapses to zero: the question "is this here?" is answered by searching, and a search that finds nothing reports nothing.
The companion benchmark from the same publisher makes the same point from the other side. RELC-Bench, the "Retrieval on Long Context Benchmark", tests whether a system can find a real number rather than refuse a fake one [2]. The same retrieval configuration scored 97.0 per cent overall, and, tellingly, 97.0 per cent at the beginning of the document, 97.1 per cent in the middle, and 97.0 per cent at the end. Position stops mattering, because a search does not attend to the start and end of a file more than its middle. It is not reading linearly at all.
None of this repeals the "lost in the middle" research for passive models on other tasks. On multi-fact integration, where a model must find and combine several needles at once, the degradation is real and documented: on needle-in-a-haystack tests at one-million-token scale as of April 2026, single-needle retrieval sits high, but asking for eight needles at once drops one frontier model by twenty-two points and another by thirty-three [3]. I date those figures deliberately, because benchmark baselines move; anyone deploying them as current should check for a more recent run. The reconciliation is simple. Position hurts passive models when they must integrate scattered facts. It does not explain the fabrication of absent ones. Two different findings, neither cancelling the other, and retrieval sidesteps both by not depending on positional attention in the first place.
What V.E.R.A. already separated
This distinction is not new to anyone who has read the architecture this series is built on. It is the founding move.
V.E.R.A., the Verified Existence and Reason Architecture, is an open-source logic engine that wraps a language model in three layers, grounded in Non-Traditional Predication Theory, the formal separation of existence from predication developed by Professor Horst Wessel at Humboldt University Berlin [4][5]. The first layer, the Krampitz Load Analyser, classifies every claim before any verification runs. It sorts each claim into one of two kinds. An e-type claim asserts that something is real: an entity exists, an event happened, a quantity was measured at a stated value. An n-type claim follows logically from premises already established. The two kinds need different treatment. An e-type claim cannot be checked by consulting the model that produced it, because that model is the thing under test; it has to be looked up against an independent record. An n-type claim needs only that the logic hold.
Every one of the two hundred and four HALC-Bench traps is an e-type question by this classification: does this metric exist in this document at this value? The correct response is not a probability drawn from what the model has seen before, and not a formal derivation. It is a lookup. The passive models performed a recall; they pattern-matched on how such metrics are usually reported and generated a plausible one. The retrieval system performed a search. The gap between recall and lookup is exactly the gap the framework names between what a model can describe from its training distribution and what it can assert as real from an external check.
V.E.R.A.'s own lookup mechanism is the E! Verification Service. Given an entity pulled from a claim, it queries a structured external corpus, seeded from sources such as Wikipedia and Wikidata, and returns a categorical verdict: the entity exists, does not exist, or is unknown. There is no "seventy-three per cent exists". On first-party benchmarks the architecture records a 28.9 per cent reduction in hallucination and a 21.1 per cent improvement in truthfulness [5]. Those are first-party measures of categorical existence checking, offered as evidence of that one job.
Here the boundary has to be marked in the text, not just noted in a file, because the resemblance is close enough to mislead. HALC-Bench does not validate V.E.R.A., and V.E.R.A. does not achieve a hundred per cent on it. The E! Verification Service searches a corpus of real-world entities to confirm that a named thing exists; the HALC-Bench system searches earnings transcripts for numeric metrics. Different stores, different query mechanisms, different domains. They share one principle and only one: a lookup against an external record beats a recall from training on questions that assert something is real. The architecture was designed from Wessel's formal framework, before this benchmark existed; the benchmark is consistent with the thesis, and consistency is not proof [F9]. The direction runs from the architecture to the result, not the other way. And to be clear about the firewall the whole design rests on, V.E.R.A. does not verify its own reasoning; a checker living inside the process it judges would inherit that process's blind spot, which is why the reference sits outside the loop. This V.E.R.A. is Andreas Hamberger's architecture, published by Te Pono Limited; it is not vera.ai, not DataGrail Vera, not VeraSafe. HALC-Bench, likewise, is AIMultiple's benchmark, distinct from the earlier academic paper that shares the letters "HALC".
The implication: a design choice, made or defaulted
If e-type questions are answered better by lookup than by recall, then anyone building a system that makes claims about whether things are real faces a design choice, whether or not they notice they are making it. They can route those claims through a recall mechanism, the model's context and training, or through a lookup mechanism, a retrieval tool over a source, a structured corpus, a verified reference. HALC-Bench puts a number on the cost of the first path and the benefit of the second.
The governance frameworks already assume someone will catch the difference. The European Union's AI Act, in its Article 14 human-oversight requirement, obliges high-risk systems to let a human "detect and address dysfunctions" [6]. A fabricated metric is such a dysfunction. But if the model underneath is reading passively and its best-case ceiling still lets roughly one trap in fifty through, the residual lands on the human overseer, and Article 14 does not say whether the system should have used retrieval grounding in the first place. Its Article 50 transparency obligations, incidentally, go operative on 2 August 2026, the day after this article's publication week.
Closer to home, there is a gap worth naming carefully. Searching the material I have to hand, I could not find a New Zealand instrument that specifies whether an AI system used for claims of fact in a regulated setting should retrieve from a source or may rely on passive context. I put that as an observation about what I could locate, not as a proven absence; it should be confirmed against forthcoming guidance before anyone treats it as settled. What does exist points at the shape of the problem. The Office of the Privacy Commissioner's 2026 survey found sixty-seven per cent of respondents concerned about automated decision-making [8], and a system that reports a figure no one measured is squarely inside that concern. The Reserve Bank's draft Operational Resilience Standard asks boards to set tolerance levels for AI failure [9]; a tolerance is a probabilistic frame, a decision about how much error is acceptable in aggregate. HALC-Bench sharpens the question a board should ask underneath it: is the failure rate we are tolerating a property of the method we chose, recall, that a different method, lookup, would remove? A Five Country advisory co-authored by New Zealand's National Cyber Security Centre put the underlying discipline plainly: it is not enough to have controls, you have to be sure they work under pressure [10]. A passive model fabricating a metric under the pressure of an unfamiliar query is the failure that advisory warns against.
The implication on the defence and sovereignty side runs along the same axis. A lethal autonomous weapon that classifies an object as a lawful military target is answering an existence claim: is this, in the external record, a combatant or a protected person? A system that answers from its learned representation is remembering what such a target looks like. A system that checks the claim against an independent reference is looking it up. The United Kingdom Ministry of Defence, in its Joint Service Publication nine three six on the governance of autonomous systems, asks for that verification to be demonstrable rather than assumed [11]. The retrieval-versus-recall result gives that requirement a harder edge: a regime that accepts a recalled classification accepts the exact failure mode this benchmark measured, in the one domain where a fabricated positive is not a reporting error but the use of lethal force.
The convergence is the finding. A benchmark built to catch fabricated numbers, and an architecture built from a 1992 theory of existence and predication, arrive at the same instruction from opposite directions: for a claim that something is real, search the record, do not trust the memory. Which raises the next question, and it is the one this series turns to next. If a claim of fact should be answered by a lookup rather than a recall, then in a system of many agents, each running its own retrieval configuration and each reporting that it has verified the claim, whose lookup governs, and who verifies the verifiers?
The views expressed in this article are entirely my own, informed by more than 30 years of professional experience in architecture, security, and technology leadership in New Zealand. They do not represent the views of my employer, any government agency, or the New Zealand government. My commentary on legislation and policy is analytical, drawing on publicly available sources and my professional expertise in architecture, security, and AI governance. I follow the Public Service Commissioner's Code of Conduct for the Public Sector and social media guidance.
Andreas Hamberger is a New Zealand leader in Architecture & Security and Associate Member of the Institute of Directors. V.E.R.A. (Verified Existence & Reason Architecture) is an open-source logic engine available on GitHub.
I use AI tools, including Sudowrite, Claude, Perplexity AI, DeepSeek AI, ChatGPT, Grok, Copilot, Openart and Gemini, as deliberate production tools, not ghostwriters. This is consistent with my position: AI amplifies human judgement; it does not replace it. The frameworks, arguments, and editorial decisions in this series are original work. AI accelerated the process. The thinking is mine.
References
[1] AIMultiple. "HALC-Bench: LLM Hallucination on Long-Context Retrieval Benchmark." Updated 7 July 2026. https://aimultiple.com/ai-hallucination
[2] AIMultiple. "RELC-Bench: Retrieval on Long Context Benchmark." Updated July 2026. https://aimultiple.com/ai-memory
[3] DigitalApplied. "Long-Context Retrieval 2026: Needle-in-a-Haystack Test." April 2026. https://www.digitalapplied.com/
[4] Wessel, H. "Existenz, Ununterscheidbarkeit, Identitat." Wissenschaftliche Zeitschrift der Humboldt-Universitat zu Berlin, Reihe Geistes- und Sozialwissenschaft 41, pp. 30 to 39. 1992.
[5] Hamberger, A. "V.E.R.A. (Verified Existence and Reason Architecture): benchmarks and documentation." Te Pono Limited. January 2026. https://github.com/ahamberger/vera
[6] European Union. "Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 14 (human oversight) and Article 50 (transparency)." 2024. https://artificialintelligenceact.eu/
[7] "A-RAG: Scaling Agentic Retrieval-Augmented Generation." arXiv 2602.03442. February 2026. https://arxiv.org/abs/2602.03442
[8] Office of the Privacy Commissioner. "2026 Privacy Concerns Survey: automated decision-making." 11 May 2026. https://www.privacy.org.nz/
[9] Reserve Bank of New Zealand. "Operational Resilience Standard 2027 (Deposit Takers Act, Tranche 3): exposure draft for consultation." June 2026. https://www.rbnz.govt.nz/
[10] National Cyber Security Centre New Zealand (Five Country Council joint advisory). "Ensuring controls work under pressure." 22 June 2026. https://www.ncsc.govt.nz/
[11] United Kingdom Ministry of Defence. "Joint Service Publication 936: Dependable Artificial Intelligence in Defence, governance of autonomous systems." 2024. https://www.gov.uk/government/publications
[12] Stanford University (RegLab). "Hallucination-Free? Assessing the Reliability of Legal AI Research Tools." 2024. https://reglab.stanford.edu/