Broken by Default: Why the Scanner Can't See What the Proof Can
- Series: V.E.R.A. Saturday (Arc 3, news-driven; Episode 23)
- Article type: Argument-from-analogy, empirically grounded (external-validation arc)
- NTP claim scan: All e-type claims sourced. Findings 1, 2, 3 are Confirmed (e-type) from arXiv 2604.05292 and RAND RRA4881-1. Finding 4 (Vectara) used in the confirmed formulation only ("every major reasoning model exceeded 10 per cent"); the unverified "12 per cent" composite figure is not used. Boundary discipline: V.E.R.A. does not verify code; the code-security study is a structural analogy, marked as such. NTP Existence-Predication Firewall: two mandatory statements present. No n-type claim presented as e-type.
- Defence angle paragraph (Section 27): present, 129 words, opposition test passed, reasonable observer test passed. Series defence angle cut: formal verification of autonomous-systems software as an artefact-level property a weapons review can demand, with the RAND scope boundary (artefact properties, not intent) as the analytical hinge. Per-series calibration applied: MODERATE BRIDGING REQUIRED (bridge explicitly signposted).
[NAVIGATION LINKS: Episode 22, Calibrated to Decline: Why "I Don't Know" Is the Optimal Answer | Episode 24, planned]
A development team ships a feature. The code was written with an artificial intelligence (AI) coding assistant, reviewed by a human, and run through the continuous integration and continuous delivery (CI/CD) pipeline. That pipeline runs six industry static application security testing (SAST) tools. Every one of them passes. The merge goes through. The code is in production by lunchtime.
The code was vulnerable from the moment it was generated. A specific execution path triggers a security flaw, and that path was provably present before a single scanner ran. The scanners did not miss it because they were misconfigured. They missed it because they were looking for something else.
This is not a hypothetical. In the largest study of its kind to date, it is the majority outcome.
The Scale of the Gap
In April 2026, Dominik Blain and Maxime Noiseux of Cobalt AI, a research organisation in Gatineau, Quebec, Canada, published "Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Code" on the arXiv preprint server (arXiv 2604.05292). The naming matters for anyone who searches it later: Cobalt AI, the research group, is unrelated to cobalt.io, a separate penetration-testing company that shares the word.
The study generated 3,500 code artefacts from 500 security-critical prompts, spanning five Common Weakness Enumeration (CWE) vulnerability categories, across seven different language models. Each artefact was then analysed by a pipeline built on the Z3 satisfiability modulo theories (SMT) solver, a formal reasoning tool from Microsoft Research. Z3 does not estimate. It does not pattern-match. It produces a satisfiability witness: a mathematical proof that a specific execution path exists which triggers the vulnerability condition.
The headline finding is that 55.8 per cent of the artefacts, 1,055 of the 3,500, contained formally-proven vulnerabilities. Not flagged as suspicious. Proven.
Then the same artefacts were run through six named industry SAST tools: Semgrep, Bandit, Cppcheck, the Clang Static Analyzer, FlawFinder, and CodeQL version 2.25.1. Combined, those six tools flagged 7.6 per cent of all artefacts. On the subset that Z3 had formally proven vulnerable, they missed 97.8 per cent, 88 of 90. The scanner and the proof were reading the same code. The proof found the flaw. The scanner saw almost nothing.
The per-model results close off the easy explanation. Vulnerability rates ranged from GPT-4o at 62.4 per cent, graded F, down to Gemini 2.5 Flash at 48.4 per cent, graded D. Every one of the seven models received a failing or near-failing grade. This is not a league table where one vendor disappoints and another saves the day. The failure is structural and it crosses every model tested.
Why the Scanner Cannot See It
SAST tools are not broken. They are doing exactly what they were built to do, against the wrong target.
A static analyser works by matching known patterns: buffer overflows written in familiar idioms, injection sinks in recognised frameworks, the textual shapes that a decade of human-written vulnerabilities have taught the tooling to recognise. The pattern library is the product. It encodes what insecure human code tends to look like.
AI-generated code does not look like human code. It is structured differently. It often avoids the surface patterns a scanner recognises while keeping the semantic flaw underneath. Pixee.ai research from January 2026, examining fifteen AI coding applications, put it plainly: traditional SAST flags generic patterns that AI already avoids. The vulnerability is still there. It is simply no longer wearing the costume the scanner was trained to spot.
Z3 operates at a different level. It does not ask whether the code resembles a known-bad example. It asks whether any execution path can satisfy the vulnerability condition, and it answers with a proof. The flaw lives at the semantic level, in what the code can actually do, and that is the level at which formal verification works. The scanner reads the surface. The proof reads the behaviour. The 97.8 per cent gap is the distance between those two levels, measured.
The Model Knows. It Just Does Not Tell You.
The most counter-intuitive result in the paper is not the size of the gap. It is what happens when you ask the model to look.
In a separate experiment, the researchers took 89 of the Z3-proven vulnerable artefacts and fed them back to the models that had generated them, this time asking each model to review its own output for security flaws. Seventy of the 89 were correctly flagged as vulnerable, a detection rate of 78.7 per cent. GPT-4o, the worst generator at 62.4 per cent vulnerable, detected 67 per cent of its own flaws in review mode. Mistral Large detected 100 per cent.
The paper names this the generation-review asymmetry. The model can identify the problem when explicitly asked to look. It does not look by default. The authors attribute this to instruction fine-tuning for code review not transferring to the generation task. In ordinary terms: the knowledge is in the system, but default generation mode does not reach for it.
Read that finding twice, because it is the structural heart of the matter. The model is not ignorant of the flaw. It produces the flaw anyway, then recognises it the moment a reviewer forces the question. The safety knowledge exists. It is simply not wired into the act of producing the output. Closing that gap is not a matter of more training or a larger model. It is a matter of architecture: of forcing the check to happen, externally, every time, rather than hoping the generator chooses to perform it.
V.E.R.A.'s Structural Response
This is the point where the analogy has to be drawn carefully, because it is an analogy and nothing more.
V.E.R.A. does not verify code. The Cobalt AI study is a code-security result using a formal solver on program semantics. V.E.R.A. is an ontological verification system for existence claims in natural language, built on Non-Traditional Predication Theory (NTP) and an external reference corpus. These are categorically distinct domains. The one principle that bridges them, and the only one, is this: external, formal verification catches what internal, heuristic checking misses.
With that boundary stated, the parallel is exact. A generating model, in default mode, can produce an existence claim it would flag as uncertain if explicitly prompted to check it. That is the generation-review asymmetry, transposed from program semantics to natural-language assertions. The knowledge is present under the right conditions; it does not manifest in default generation.
V.E.R.A.'s E! Verification Service is the architectural mechanism that forces the check. When the Krampitz Load Analyser classifies a claim in the generation stream as e-type, an existence, quantity, or occurrence claim, the E! service pauses generation, extracts the subject entity, and queries the external E! Corpus directly. It does not ask the generating model whether its claim is reliable. It asks the corpus, which returns EXISTS, NOT_EXISTS, or UNKNOWN. The model's internal state is never consulted, exactly as Z3 never asks the model whether its code is safe. The reference standard sits outside the system under test.
One point has to be stated plainly, because it is the discipline the whole architecture rests on: V.E.R.A. does not verify its own reasoning. It cannot, and it does not claim to. The E! Corpus is an external referent precisely because no system can stand as its own ground of truth. That is the same reason Z3 reads the code rather than asking the model: a check that runs inside the thing being checked is not a check.
The boundary holds in both directions. V.E.R.A. has no view on code security. The code-verification work has no view on existence claims. What they share is the shape of the solution: an external formal check beats internal state-based generation, each within its own domain.
Institutional Convergence
The empirical shock has an institutional counterpart, and it arrived in the same month.
In June 2026, the RAND Corporation published "Verified Machine Learning Infrastructure: Formal Methods for Trustworthy Artificial Intelligence Deployment" (report number RRA4881-1), authored by Gopal P. Sarma, Rachel Steratore, Sunny D. Bhatt, and Geoffrey Irving. Drawing on a survey of 23 experts across formal methods, AI infrastructure, cybersecurity, software engineering, hardware architecture, and policy, the report concludes that formal verification can address code-level correctness in machine learning inference and training stacks: memory safety, access control, protocol enforcement, and sandbox isolation. Weight integrity and supply-chain threats were rated most urgent.
One academic study and one expert-consensus institutional report, arriving in the same month from opposite directions, reach the same conclusion: formal, external verification is necessary, not optional. They did not coordinate. They converged.
The RAND report also draws the boundary that makes this argument precise. Formal methods, it notes, have limited reach into semantic challenges such as model sandbagging, jailbreaks, and intent. They work on definable properties checked against a specification. They do not adjudicate intent. That caveat is not a weakness for the argument here. It is the map. Formal external verification applies to artefact-level properties: code correctness in the RAND case, existence of named entities in V.E.R.A.'s. It does not apply to reasoning quality or motive, and neither system claims that it does. V.E.R.A.'s scope choice, existence claims rather than reasoning intent, sits exactly inside the boundary RAND draws.
There is a regulatory signal worth noting alongside it. The European Union Cyber Resilience Act activates vulnerability-reporting obligations for products with digital elements from September 2026. New Zealand firms that export software into the European market will fall under those obligations, and AI-generated code that ships with formally-provable vulnerabilities is precisely the artefact class the Act targets. The Five Country Council guidance on careful adoption of agentic AI services, co-authored by New Zealand's National Cyber Security Centre and published on 1 May 2026, already names supply-chain risk as one of its categories. A team that adopts an AI coding assistant is taking on a supply-chain exposure that standard scanning does not detect.
What This Means for Practitioners
For teams deploying AI coding assistants, the operational lesson is narrow and concrete: a clean SAST pass is necessary but no longer sufficient as a security gate. It tells you the code does not match known-bad human patterns. It does not tell you the code is safe. Those were always different statements; the Cobalt AI data shows how far apart they have grown.
The implication on the defence and sovereignty side is worth naming directly, because the same principle governs systems where the stakes are absolute. United States Department of Defense Directive 3000.09 requires verification and validation of autonomous weapon functions before fielding, and an Article 36 weapons review under the 1977 Additional Protocol I obliges a state to determine, before deployment, whether a new means of warfare can be used lawfully. Both ask for evidence about what software does, not assurances about what a vendor believes. That is an artefact-level property, exactly the class the RAND report says formal methods can address. The governance asymmetry between states becomes measurable here: a state that can formally verify autonomous-systems code can produce accountability evidence on demand; one relying on probabilistic performance claims cannot.
For anyone evaluating AI verification more broadly, the generation-review asymmetry is the durable finding. You cannot rely on the model to catch its own errors in the act of producing them. The knowledge being present is not the same as the knowledge being applied. External verification is an architectural requirement, not a quality layer you can add later. V.E.R.A. is one answer to that requirement in the existence-claim domain. The Z3 pipeline is another in the code domain. The shape is the same: put the check outside the system, make it formal, and run it every time.
A practitioner asked me recently: we run CodeQL on every pull request and have never had a false pass, so why would AI-generated code be different? The answer is that CodeQL, like every SAST tool, matches against known patterns. The Cobalt AI study gave those same tools the same task. They missed 97.8 per cent of the formally-proven vulnerabilities. That is not CodeQL failing. It is CodeQL doing exactly what it was designed to do. Z3 did something different, and found most of the code broken. A zero from your scanner is not a clean bill of health. It is a pattern-match on human-code idioms, run against code that no human wrote.
Join the Movement
Te Pono V.E.R.A. is open source. The architecture that forces an external existence check at every e-type claim, the Krampitz Load Analyser, the E! Verification Service, and the supporting modules, is in the repository. It is worth saying once more, because the claim is easy to overstate: V.E.R.A. does not verify its own reasoning and does not eliminate hallucination. It checks existence claims against an external corpus, and that is the whole of what it does. If you work in formal methods, program analysis, or AI verification, the structural question this episode raises is the one worth carrying: where in your own systems does a model's knowledge fail to reach the point of action, and what external check would close that gap?
The intellectual foundation is from 1992. The engineering is from 2026. Both are open.
Repository: https://github.com/andreas-linux/vera/
Ita est momentum veritatis.
Where have you watched a security gate pass code that turned out to be broken, or an AI system assert something it could not support? What did the gap cost, and what finally caught it?
The views expressed in this article are entirely my own, informed by more than 30 years of professional experience in architecture, security, and technology leadership in New Zealand. They do not represent the views of my employer, any government agency, or the New Zealand government. My commentary on legislation and policy is analytical, drawing on publicly available sources and my professional expertise in architecture, security, and AI governance. I follow the Public Service Commissioner's Code of Conduct for the Public Sector and social media guidance.
Andreas Hamberger is a New Zealand leader in Architecture & Security and Associate Member of the Institute of Directors. V.E.R.A. (Verified Existence & Reason Architecture) is an open-source logic engine available on GitHub.
I use AI tools, including Sudowrite, Claude, Perplexity AI, DeepSeek AI, ChatGPT, Grok, Copilot, Openart and Gemini, as deliberate production tools, not ghostwriters. This is consistent with my position: AI amplifies human judgement; it does not replace it. The frameworks, arguments, and editorial decisions in this series are original work. AI accelerated the process. The thinking is mine.
[1] Blain, D. and Noiseux, M. (Cobalt AI). "Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Code." arXiv preprint 2604.05292. 2026. https://arxiv.org/abs/2604.05292
[2] Sarma, G. P., Steratore, R., Bhatt, S. D. and Irving, G. "Verified Machine Learning Infrastructure: Formal Methods for Trustworthy Artificial Intelligence Deployment." RAND Corporation, RRA4881-1. June 2026. https://www.rand.org/pubs/research_reports/RRA4881-1.html
[3] Pixee.ai. "Why Traditional SAST Misses AI-Generated Vulnerabilities." January 2026. https://www.pixee.ai/
[4] Vectara. "Hallucination Evaluation Framework 2026." Vectara. 2026. https://www.vectara.com/
[5] arXiv. "Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation." arXiv preprint 2506.17088. 2026. https://arxiv.org/abs/2506.17088
[6] National Cyber Security Centre New Zealand and Five Country Council partners. "Careful Adoption of Agentic AI Services." 1 May 2026. https://www.ncsc.govt.nz/
[7] European Union. "Cyber Resilience Act: Obligations for Products with Digital Elements." 2024. https://digital-strategy.ec.europa.eu/en/policies/cyber-resilience-act
[8] Wessel, H. "Nichtstandard-Logik und Nichtstandard-Arithmetik: Neue Grundlagen und Anwendungen." Akademie-Verlag, 1992.
[9] United States Department of Defense. "Directive 3000.09: Autonomy in Weapon Systems." 2023. https://www.esd.whs.mil/
[10] Te Pono V.E.R.A. Logic Engine. GitHub repository. https://github.com/andreas-linux/vera/