Smarter, and Wronger: The Deeper-Reasoning Paradox and the Case for Proof
In the same week, an AI model closed a mathematics question that had been open for twenty seven years, and three independent benchmarks confirmed that current-generation AI reasoning models, as a class, get plain facts wrong more often than the models they replaced. Read alone, either finding tells a familiar story: AI is getting smarter, or AI is getting less reliable. Read together, in the same seven days, from different evaluators testing different systems on different tasks, they say something more specific.
On 1 and 2 August 2026, OpenAI disclosed that an internal model it calls Astra had produced ten new results in mathematics and theoretical computer science, the headline result an explicit construction that closes a question Mikhail Gromov posed in 1999, twenty seven years ago. Every result carries a formal proof, checked step by step by the Lean 4 proof assistant, with a "sorry" count of zero: nothing left unproven. The same week, Vectara's updated hallucination leaderboard showed GPT-5, Claude Sonnet 4.5, Grok-4, Gemini-3-Pro and DeepSeek-R1 all exceeding 10% hallucination on straightforward, document-grounded questions, while a smaller, non-reasoning model scored 3.3%. Same technology, same lever, think longer, opposite results, depending entirely on what kind of question gets asked. That is this week's argument, and it turns out to be the argument V.E.R.A. was built on.
Two headlines that should not both be true
Astra has not been designated GPT-6 or any confirmed successor name, and this article does not use "GPT-6" as a synonym for it; OpenAI has said only that Astra is intended as its next major model. What OpenAI has confirmed is narrower and more interesting. Ten open problems, each unresolved for at least a decade, now carry machine-checked proofs. The token cost of generating all ten solutions was about US$2,000. Human researchers converted the model's raw output into publishable papers, and the results have not undergone independent peer review, a distinction worth holding onto: the Lean 4 verification itself is mechanical and checkable by anyone who downloads the certificates, but the significance framing, that this represents a step change in mathematical capability, rests on OpenAI's own account.
Set that beside Vectara's leaderboard. The dataset behind this week's figures relaunched on 19 November 2025, tested against more than 7,700 documents across technology, medicine, law and finance; newer 2026 models have been added to it since. Gemini-3-Pro scores 13.6%. Claude Sonnet 4.5, GPT-5, Grok-4 and DeepSeek-R1 all exceed 10%. The best performer on the page is Gemini-2.5-Flash-Lite, a smaller model with no extended reasoning mode, at 3.3%.
These are not two versions of the same finding. One evaluator tested one lab's mathematics results against a closed proof system. A different, independent evaluator tested many labs' general-purpose models against an open-domain factual benchmark. They measure different things, using different methods, in different domains. What they share is timing, and what the timing exposes is a boundary that reasoning alone does not respect: think longer inside a domain with fixed rules, and there is nothing false left to invent. Think longer inside a domain without them, and there is more room than ever to invent something plausible instead.
Why more reasoning cuts both ways
Vectara's own researchers are candid about the limits of a single comparison. Their January 2025 post naming DeepSeek-R1 at 14.3% hallucination against DeepSeek-V3, its own non-reasoning base model, at 3.9%, states plainly that "it is still too early to draw conclusions about any degradation of reasoning-enhanced LLM." Their evidence for reasoning causing the gap was indirect: R1's scores varied more across samples than V3's did, a standard deviation of 0.23 against 0.06, which shows more borderline outputs, not a demonstrated mechanism. That figure is eighteen months old and gets recirculated without its own hedge attached. It deserves to keep it.
The mechanism this series can state with more confidence is structural. An extended reasoning trace gives a model more opportunities to generate connective material that sounds right and is not, because nothing in an open-domain factual question stops an invented middle step from looking exactly like a correct one. Digital Applied's April 2026 study, testing five 2026 frontier models across 5,000 prompts, found that extended-thinking mode cuts factual-recall hallucination by roughly 30% to 60% depending on the model: GPT-5.5 Pro from 8.3% to 4.2%, Claude Opus 4.7 from 9.4% to 5.1%, DeepSeek V4 from 12.7% to 10.4%. That is a genuine improvement, and it is worth stating what it is not: none of those figures approaches zero, and none of them approaches the certainty a formal proof carries. A separate aggregator republishes the identical three model figures without its own methodology or attribution, which is not a second study agreeing with the first; it is one study, restated. Citing both as though they corroborate each other would overstate how much independent evidence actually exists.
Apply the same mechanism inside a closed formal system, and the story reverses. Every step of a Lean 4 proof is checked against a fixed, non-negotiable set of axioms. Reasoning for longer does not create more opportunities to invent a plausible middle step, because there is no room for a step that is not either provably true or rejected outright. That is the entire difference between the two headlines, expressed as a single structural fact rather than two unrelated stories about the same technology.
The domain that admits proof, and the one that doesn't
Mathematics is not a fair comparison for most of what enterprise AI systems are asked to do, and the week's third finding supplies the fairer one. Databricks built OfficeQA Pro V2 from roughly 1,400 US Treasury PDFs, about 120,000 pages spanning 1793 to 2024, with most of its ninety questions requiring evidence synthesised across six or seven separate documents. This is closer to what a claims-review tool, a compliance function or a research desk actually does than a competition mathematics problem is.
The results are worth reading in full rather than as a single headline number, because two different baselines are in play and conflating them would misstate the finding. Generic frontier-agent baseline harnesses average 26.0% accuracy. A stronger, purpose-built coding-agent harness, Claude Code or Codex used without OfficeQA-specific tuning, reaches 37.5%. An academic competition of eleven external teams, built specifically to optimise this one benchmark, averaged 41.1%, and the single winning team, its best-optimised configuration, reached 63.3%. That last figure is not an average and not what any system achieves without deliberate tuning for this exact task. It is the ceiling, under the most favourable conditions this study tested.
Sixty three per cent, achieved by the most heavily resourced configuration in a competition built to chase this specific number, is the strongest showing enterprise document synthesis produced this week. A Lean 4 certificate with a zero "sorry" count is not a strong showing; it is complete. That gap, not a claim that AI reasoning is getting worse in general, is the argument this article is making. In the one domain this week's evidence shows admits a genuine formal proof, closed, symbolic mathematics, the result eliminates ambiguity entirely. In every domain this week's evidence shows does not admit one, open-domain factual recall, enterprise document synthesis, more reasoning either fails to close the reliability gap anywhere near that standard, or measurably widens it against a non-reasoning baseline.
What V.E.R.A. already does with that boundary
This is the boundary V.E.R.A.'s architecture was built around, months before this week's news existed to illustrate it. Non-Traditional Predication Theory, developed by Professor Horst Wessel at Humboldt University Berlin, separates existence from predication: a sentence can be grammatically and semantically correct while asserting that something real exists when it does not. The Krampitz Analyser classifies every claim an AI system generates by which kind it is. An e-type claim asserts that something is real, an entity exists, an event occurred, a quantity was measured, and requires external grounding. An n-type claim follows logically from confirmed premises and is checked by examining the logic, not by looking anything up.
For e-type claims, the E! Verification Service does one narrow thing well. Given a named entity, it queries a corpus built from Wikipedia and Wikidata and returns one of three fixed verdicts: EXISTS, does not EXIST, or UNKNOWN. No confidence score. No probability. A categorical answer, or an honest admission that the corpus does not contain enough to answer at all.
Here is the distinction this article has to hold precisely, because it is the easiest thing in this whole piece to get wrong. Astra's Lean 4 certificates are proofs: derivations checked step by step against a closed system of mathematical logic, where a "sorry" count of zero means every step is mechanically verified against the system's own axioms. The E! Service is not that, and does not claim to be. An existence claim about a named entity in the open world is not a theorem. No formal system exists that could prove whether a given entity is real the way Lean 4 proves a group-theoretic construction. What the E! Service borrows from the proof-grade ideal is narrower: rather than a confidence score that degrades exactly the way this week's Vectara and OfficeQA Pro V2 findings document, it commits to a fixed, non-negotiable verdict against a fixed corpus. That is the strongest verdict the domain of open-world existence claims actually admits. It is not a proof, and nothing in this week's evidence licenses calling it one.
What this doesn't prove
A pairing this striking is worth checking piece by piece before it gets used, and several pieces here needed correction before they were safe to deploy. Astra has not been confirmed as GPT-6; OpenAI has assigned it no public release date, price, context window or model card, and this article has deliberately avoided the substitution. Vectara's headline leaderboard dataset relaunched in November 2025, nine months before this week; it is an ongoing, continuously updated measurement framework that newer models have since been tested against, not a fresh finding published this week. The frequently recirculated DeepSeek comparison is from January 2025, eighteen months old, and carries Vectara's own hedge about causation, which this article has kept rather than dropped in favour of a stronger claim. A secondary aggregator republishing Digital Applied's extended-thinking figures without independent methodology is not a second source agreeing with the first.
None of that weakens the domain-boundary argument. It strengthens the case for making it carefully. The correct, bounded claim is this: this week's evidence shows that where a domain admits a genuine formal proof, proof beats confidence decisively. V.E.R.A.'s architecture was built on the adjacent premise that most domains do not admit a formal proof, and that a categorical, non-probabilistic verdict is still the right target even where a full proof is not available. It is not evidence that V.E.R.A. performs theorem-proving, that the E! Service is a form of Lean 4 verification, or that Astra's results validate V.E.R.A.'s architecture in any direct sense. The two systems answer different kinds of question, in different domains, using different methods. Only the structural principle, refuse a probability where a categorical verdict is available, bridges them.
The governance gap
No New Zealand regulator or institution has yet addressed the specific distinction this article turns on, a formally verified proof against a probabilistically checked benchmark score. That gap is stated plainly here rather than filled with the nearest adjacent guidance. Targeted searches for Privacy Commissioner or National Cyber Security Centre material on reasoning-mode hallucination, formal verification, or this specific proof-versus-probability question returned only general AI guidance with no bearing on this week's topic. Two previously used anchors, the Government Chief Digital Officer's responsible AI guidance and the Reserve Bank's operational resilience standard, were considered and set aside again this week: neither has a natural connection to a proof-grade mathematics result, and forcing one in would repeat exactly the kind of thin link this series has flagged before. The gap is the finding, not an oversight to paper over.
The one part of this week's evidence that anyone outside OpenAI can actually check is open source. Lean 4, the proof assistant behind Astra's zero "sorry" count, is a community-maintained project with a public mathlib library; the certificate is not OpenAI's claim to take on faith, it is a file anyone can download and run themselves. That is the entire value of a checker nobody owns. V.E.R.A. rests on the same principle at a smaller scale: the NTP classification logic and the E! Verification Service are GPL-3.0, published on GitHub, checked against a corpus built from Wikipedia and Wikidata rather than a proprietary index. An external check earns trust only when its own logic is open to inspection, not merely when its answer sounds confident, which is exactly the property a closed benchmark score cannot offer.
That same distinction, open to inspection or not, checked categorically or not, is not staying inside mathematics for long. NSPM-11, the US National Security Presidential Memorandum on artificial intelligence issued 5 June 2026, requires assurance that an AI system's outputs meet a defined reliability standard before deployment in specified national security contexts. Earlier episodes in this series applied its assurance pillar to a reasoning-process property and to an action-path property. This week adds a third, harder question: assurance of a formally verifiable property is not the same guarantee as assurance of a probabilistically checked one, and NSPM-11 does not yet specify which of the two a given deployment must clear. A procurement standard that treats a benchmark score and a formal proof as interchangeable evidence of reliability is applying one word, assurance, to two different guarantees, and this week supplied the clearest illustration yet of exactly how different they are.
Arc 4 of this series has spent six episodes asking what a verifier should output, how it reaches its grounding, whether verification effort itself scales, where the check has to live, and whether independence from the model is enough once the asking user is not neutral either. This week adds a further question to that list: how certain can the check itself actually be, and does the domain even allow that certainty. The multi-agent identity question, who is checking whom when more than one system is involved, remains open, deferred now across six consecutive episodes, and is the leading candidate for wherever this series goes next.
If your organisation is about to accept a benchmark score as proof that an AI system is reliable enough to deploy, what would it take for someone on your team to notice, before the decision is made, that a benchmark and a proof are not the same kind of evidence? I would like to hear about the moment your team caught that difference, or the moment it caught you instead.
The views expressed in this article are entirely my own, informed by morethan 30 years of professional experience in architecture, security, andtechnology leadership in New Zealand. They do not represent the views ofmy employer, any government agency, or the New Zealand government. Mycommentary on legislation and policy is analytical, drawing on publiclyavailable sources and my professional expertise in architecture, security,and AI governance. I follow the Public Service Commissioner's Code ofConduct for the Public Sector and social media guidance.
Andreas Hamberger is a New Zealand leader in Architecture & Security andAssociate Member of the Institute of Directors. V.E.R.A. (VerifiedExistence & Reason Architecture) is an open-source logic engine availableon GitHub.
I use AI tools, including Sudowrite, Claude, Perplexity AI, DeepSeek AI,ChatGPT, Grok, Copilot, Openart and Gemini, as deliberate productiontools, not ghostwriters. This is consistent with my position: AI amplifieshuman judgement; it does not replace it. The frameworks, arguments, andeditorial decisions in this series are original work. AI accelerated theprocess. The thinking is mine.
[1] Vectara. "Introducing the Next Generation of Vectara's Hallucination Leaderboard." 19 November 2025. https://www.vectara.com/blog/introducing-the-next-generation-of-vectaras-hallucination-leaderboard
[2] Vectara. "DeepSeek-R1 hallucinates more than DeepSeek-V3." 30 January 2025. https://www.vectara.com/blog/deepseek-r1-hallucinates-more-than-deepseek-v3
[3] SiliconANGLE. "OpenAI's Astra solves 10 long-open math problems, publishes proofs." 2 August 2026. https://siliconangle.com/2026/08/02/openais-astra-solves-10-long-open-math-problems-publishes-proofs/
[4] DataCamp. "OpenAI's model Astra solved ten open math problems." 2 August 2026. https://www.datacamp.com/blog/open-ai-model-astra-solved-ten-open-math-problems
[5] Databricks. "Introducing OfficeQA Pro V2: A New Benchmark for Enterprise Grounded Reasoning." 6 August 2026. https://www.databricks.com/blog/introducing-officeqa-pro-v2-new-benchmark-enterprise-grounded-reasoning
[6] Digital Applied. "AI Model Hallucination Rate Benchmarks: 2026 Study." 23 April 2026. https://www.digitalapplied.com/blog/ai-model-hallucination-rate-benchmarks-2026-study
[7] AI Business Weekly. "AI Hallucination Statistics." Last updated 21 July 2026. https://aibusinessweekly.net/p/ai-hallucination-statistics (cited only to illustrate a same-origin restatement of source [6]'s figures, not as independent corroborating evidence)
[8] The White House. National Security Presidential Memorandum on Artificial Intelligence (NSPM-11). 5 June 2026. (No URL captured in this episode's source material; cited by title and date only, per URL integrity practice, rather than an invented or reconstructed link.)