Scoring the Reasoning: When the Verifier Becomes Probabilistic
A practitioner at a financial services firm is reading an AI-generated credit assessment. The report states that the borrower's employer has traded for more than ten years and that the declared income sits within the normal range for that company. The team lead is confident. "We put the model through three verification stages," she says. "A reward model scored every section above zero point eight five. A formal checker confirmed the arithmetic is internally consistent. And a human reviewer signed it off."
The practitioner asks one question. "Has anyone actually checked that the company exists?"
The room goes quiet, because the honest answer is no. The reward model scored the internal quality of the reasoning. The formal checker verified the numbers were consistent with each other. The human reviewer judged whether the argument hung together. Not one of the three produced a finding on whether the named employer is a real, currently trading entity. That is an existence claim, and it slipped through three verifiers because none of them was built to catch it.
This is not a hypothetical about one firm. In a single week in July 2026 the field published three ways to verify AI reasoning, each impressive, each external to the model being checked, and each producing a different kind of output. Reading them together is the fastest way to see why "has this been verified?" is not yet a well-formed question.
The probabilistic verifier: a score in the interval
On 6 July 2026 a paper titled "LLM-as-a-Verifier: A General-Purpose Verification Framework" was published to arXiv [1]. Its move is subtle. A standard model-as-judge is prompted to emit a discrete score, "this solution scores four out of five." LLM-as-a-Verifier instead computes the expectation over the distribution of the scoring tokens the model would produce, yielding a continuous score across candidate solutions at once. The verifier never says "correct." It says, in effect: across all the ways this model would express a quality judgement, the centre of mass sits here.
The results are strong. The framework reports 86.5 per cent on Terminal-Bench V2, 78.2 per cent on SWE-Bench Verified, and 87.4 per cent on RoboRewardBench, state of the art at submission on the first three, with 73.3 per cent on MedAgentBench [1]. It is training-free and drops into existing evaluation pipelines. Its Probabilistic Pivot Tournament dispatches many candidate trajectories in parallel and picks the best through a tournament ranked on those continuous scores, and it identifies three ways to scale the approach: finer score granularity, repeated evaluation to reduce variance, and decomposing an assessment into sub-criteria scored independently.
Now the structural limit. A score of zero point eight seven measures how confidently this model would express approval of this trajectory. It does not measure whether the trajectory's claimed result matches reality. If the model that produced the trajectory tends to approve of a certain kind of plausible-sounding answer, a plausible-sounding but false answer can score highly. The verifier and the generator draw from the same distribution. A high score is evidence about the model's confidence, not about the world.
The formal verifier: a proof or nothing
Three days earlier, on 3 July 2026, Mistral released Leanstral 1.5 under an Apache 2.0 licence, a mixture-of-experts model of 119 billion total parameters with 6 billion active, built to work in Lean 4, the interactive theorem-proving language [2]. Its output is not a score. It is a proof certificate: a formal derivation that a Lean 4 type-checker can verify mechanically. The type-checker accepts the derivation or it rejects it. There is no middle value.
The reported numbers show both the strength and the price. Leanstral 1.5 records 87 per cent on the FATE-H benchmark and produced formal proofs for 587 of 672 problems on PutnamBench, the William Lowell Putnam competition set [2]. Tested against 57 open-source repositories, it uncovered five previously unknown bugs whose existence it could formally prove. That is the promise of formal methods: not a higher score, but provable correctness inside a bounded domain.
The same release records 34 per cent on the harder FATE-X benchmark [2]. That gap is not a weakness of the model; it is the edge of the formal domain. What Leanstral can express in Lean 4's type theory, it proves completely. What it cannot formalise, it cannot verify at all. And most claims that matter in a credit report are of the second kind. "This company has traded for ten years" is a natural-language existence claim about the world. It is not, in general, expressible in dependent type theory without a separate and non-trivial formalisation step that presumes an external record already exists. A formal verifier authorises reliance within the domain it can express, and says nothing beyond it.
The human verifier, given better eyes
The third study is older but drew fresh attention in the same window. "Improving Human Verification of LLM Reasoning through Interactive Explanation Interfaces," on arXiv as 2510.22922, first appeared in October 2025 and reached its current version in January 2026 [3]. Across 125 participants it compared how well people spot an incorrect step in an AI reasoning chain under four interfaces. The iGraph condition, which lays the reasoning out as connected nodes and edges with consistent colour-coded variables and step controls, is the one to note, and it is distinct from the network-analysis library that shares the name.
Wrong-step identification rose from 66.1 per cent under plain chain-of-thought text to 85.2 per cent under iGraph, and readers were faster, 57.9 seconds a question against 64.7 [3]. The output here is a human judgement, and the finding is sobering rather than triumphant: even the gold standard of human oversight swings by nearly twenty percentage points on the same people, depending only on how the reasoning is shown to them. That is an argument for taking interface design seriously wherever a person is asked to check a machine, which is precisely what the European Union's AI Act, in its Article 14 human-oversight requirement, leaves to the implementer rather than specifying [6].
Three outputs that will not add up
Set the three side by side. A real number in an interval. A proof certificate that a type-checker accepts or rejects. A human correctness judgement mediated by an interface. All three are external to the model they check. All three beat their baselines. And all three are structurally incompatible: you cannot fold "0.87 from the reward model, plus a formal proof exists, plus 85.2 per cent human agreement" into a single verified-or-not signal without first specifying what each output is actually warranted to assert. This follows from what each system is, and needs no further source [F5]. A downstream system handed all three at once cannot tell whether their combination is a stronger warrant than any one alone.
It is tempting to read the LLM-as-a-Verifier idea of decomposing an assessment into sub-criteria as the same move that underlies claim-type separation. The resemblance is real but the mechanism is not [F6]. Criteria decomposition breaks one probabilistic judgement into smaller probabilistic judgements to improve the score. Claim-type separation asks a prior question: what kind of verification does this claim even need? One improves a single kind of check. The other decides which check applies. Reading the first as the second is exactly the error this series exists to name, so it is worth stating the boundary plainly: none of these three papers implements the architecture described next, and none validates it. What they share with it is one premise only, that verification must sit outside the thing being verified.
Where V.E.R.A. sits: a verdict, not a score
V.E.R.A. (Verified Existence and Reason Architecture) is an open-source logic engine that wraps a language model in three layers, grounded in Non-Traditional Predication Theory, the formal separation of existence from predication developed by Professor Horst Wessel at Humboldt University Berlin [4][5]. The first layer, the Krampitz Load Analyzer, classifies each claim as an existence claim that needs grounding or an inference that follows from established premises. The second, the E! Verification Service, checks the existence claims against a structured, sourced corpus that sits outside the model. The third, the D-Service, acts on the verdict.
The output of the E! Verification Service is none of the three above. It is not a score, not a proof certificate, and not a human judgement. It is a verdict: an entity either exists in the independent record, written E!, or it does not. There is no "73 per cent exists." When the service returns a positive verdict it is because it has located an independent record; when it cannot, the claim is flagged as unverified, and the D-Service returns unknown and stops rather than proceeding on an unconfirmed premise. On first-party benchmarks the architecture records a 28.9 per cent reduction in hallucination and a 21.1 per cent improvement in truthfulness [5]. These are measures of categorical existence checking, and they are offered as first-party evidence of that one job, not as a comparison against the three papers, which do a different job.
That boundary is the point, not a hedge. V.E.R.A. addresses existence claims. It is not a competitor to a trajectory-quality scorer, to a formal prover of propositions, or to an interface that helps a human read a reasoning chain. It also does not verify its own reasoning: a checker living inside the process it judges would inherit that process's blind spot, which is the whole reason the referent sits outside the loop. And this V.E.R.A. is Andreas Hamberger's architecture, published by Te Pono Limited; it is not vera.ai, not DataGrail Vera, not VeraSafe, and shares nothing with them but four letters.
The New Zealand angle: which output counts as assurance?
Here is a gap worth naming carefully. Searching the regulatory material I have to hand, I could not find a New Zealand instrument that specifies which verification output, a probability threshold, a formal proof, a human sign-off, or a categorical existence check, counts as adequate assurance for an AI-generated claim used in a regulated decision. I put that as an observation about what I could locate, not as a proven absence; someone should confirm it against the Privacy Commissioner's forthcoming guidance before treating it as settled.
What exists points at the shape of the gap. The Office of the Privacy Commissioner's 2026 survey found 67 per cent of respondents concerned about automated decision-making, up five points year on year [11]; that concern is about outcomes, not about output format. The Reserve Bank's draft Operational Resilience Standard asks boards to set tolerance levels for disruption to critical operations, and a tolerance is a probabilistic frame: how much error a system may produce before intervention [12]. That is a reasonable question, and it shows the probabilistic mode is not inadequate in itself. But a tolerance governs what a system is allowed to get wrong in aggregate. It does not tell you whether a specific claim, in a specific decision, is warranted. Those are different questions, and both need answering.
The implication on the defence and sovereignty side is worth drawing out, because it sharpens the whole argument. Consider lethal autonomous weapons systems, the class that Article 36 of the 1977 Additional Protocol to the Geneva Conventions requires each state to review before fielding, and that the International Committee of the Red Cross frames through the doctrine of meaningful human control. Ask what a verification output must be to discharge that obligation. A score of zero point eight seven that a target was correctly identified does not authorise a strike; it reports a model's confidence, not the reality of who stands in the frame. A formal proof covers only what can be formalised. The accountability standard here is closer to a verdict on one existence claim, is this a lawful target, and United States Directive 3000.09 asks for that assurance to be demonstrable, not merely probable.
The convergence of that same week is the finding. Three teams, working from probability, from formal logic, and from human factors, all reached for a verifier outside the model. None of them, and none of the benchmark numbers they report, tells you whether the entity at the centre of a claim is real. If a system runs several of these verifiers at once, each returning a different kind of output, who decides what the combination warrants, and who verifies the verifiers themselves?
The views expressed in this article are entirely my own, informed by more than 30 years of professional experience in architecture, security, and technology leadership in New Zealand. They do not represent the views of my employer, any government agency, or the New Zealand government. My commentary on legislation and policy is analytical, drawing on publicly available sources and my professional expertise in architecture, security, and AI governance. I follow the Public Service Commissioner's Code of Conduct for the Public Sector and social media guidance.
Andreas Hamberger is a New Zealand leader in Architecture & Security and Associate Member of the Institute of Directors. V.E.R.A. (Verified Existence & Reason Architecture) is an open-source logic engine available on GitHub.
I use AI tools, including Sudowrite, Claude, Perplexity AI, DeepSeek AI, ChatGPT, Grok, Copilot, Openart and Gemini, as deliberate production tools, not ghostwriters. This is consistent with my position: AI amplifies human judgement; it does not replace it. The frameworks, arguments, and editorial decisions in this series are original work. AI accelerated the process. The thinking is mine.
References
[1] "LLM-as-a-Verifier: A General-Purpose Verification Framework." arXiv 2607.05391. 6 July 2026. https://arxiv.org/abs/2607.05391
[2] Mistral AI. "Leanstral 1.5." 3 July 2026. https://mistral.ai/news/leanstral-1-5
[3] "Improving Human Verification of LLM Reasoning through Interactive Explanation Interfaces." arXiv 2510.22922. Version 4, January 2026. https://arxiv.org/abs/2510.22922
[4] Wessel, H. "Existenz, Ununterscheidbarkeit, Identitat." Wissenschaftliche Zeitschrift der Humboldt-Universitat zu Berlin, Reihe Geistes- und Sozialwissenschaft 41, pp. 30 to 39. 1992.
[5] Hamberger, A. "V.E.R.A. (Verified Existence and Reason Architecture): benchmarks and documentation." Te Pono Limited. January 2026. https://github.com/ahamberger/vera
[6] European Union. "Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 14: human oversight." 2024. https://artificialintelligenceact.eu/
[7] National Institute of Standards and Technology. "AI Risk Management Framework (AI RMF 1.0), Measure function." 2023. https://www.nist.gov/itl/ai-risk-management-framework
[8] International Committee of the Red Cross. "Article 36 of the 1977 Additional Protocol I to the Geneva Conventions: new weapons review." 1977. https://ihl-databases.icrc.org/
[9] International Committee of the Red Cross. "ICRC position on autonomous weapon systems and meaningful human control." 2021. https://www.icrc.org/
[10] United States Department of Defense. "Directive 3000.09: Autonomy in Weapon Systems." Updated 2023. https://www.esd.whs.mil/
[11] Office of the Privacy Commissioner. "2026 Privacy Concerns Survey: automated decision-making." 11 May 2026. https://www.privacy.org.nz/
[12] Reserve Bank of New Zealand. "Operational Resilience Standard 2027 (Deposit Takers Act, Tranche 3): exposure draft for consultation." June 2026. https://www.rbnz.govt.nz/