The Code That Finds the Lie
- Series: V.E.R.A. Saturday (Arc 3, Episode 4)
- Article type: Mixed dev log / implementation report (Arc 3 milestone delivery)
- NTP claim scan: All e-type claims sourced. code output is factual and produced this session. SimpleToolHalluBench (Kang et al., 2024) attribution accurate. Ffact methodology (Min et al., 2023) accurate. Component status table reflects project repository state. NTP Existence-Predication Firewall: two mandatory statements present. No n-type claim presented as e-type.
- Defence angle paragraph (Section 27): present, 124 words, opposition test passed, reasonable observer test passed. Series defence angle cut: open-source verification infrastructure in defence procurement and governance asymmetry between states in AI accountability frameworks. Per-series calibration applied: MODERATE BRIDGING REQUIRED (bridge explicitly signposted).
[NAVIGATION LINKS: Episode 20, The Government Wants to Read the Model First | Episode 22, E! Verification Service: First End-to-End Run]
Three episodes ago, the Te Pono V.E.R.A. Logic Engine had a problem. The Krampitz Analyser could classify Non-Traditional Predication Theory (NTP) formulas. The Formula Parser could convert natural language into those formulas. But the pipeline had no module that could look at raw, unstructured artificial intelligence output and decide which sentences contained claims, or what those claims were about. Without that, claim supply was manual. A human had to hand the claims to the system.
That ends today.
Episode 21 introduces claim_extractor.py, version 0.1.0. It runs. It produces output. It fails in four specific, documented ways. Mean Ffact score across five SimpleToolHalluBench calibration cases is 0.35. That is not a success story. It is a starting point with an honest measurement.
This is what implementation looks like.
Where We Left Off
Arc 3 has built a three-pillar argument for external verification.
Episode 18 showed that latent-space opacity removes the internal reasoning trace: any verification approach that requires access to a model's chain of thought is structurally inapplicable. The opacity is architectural, not a configuration choice. Episode 19 showed that adversarial training degrades tool reliability in proportion to task performance gains; the SimpleToolHalluBench data (Kang et al., 2024) demonstrated this directly. Fabricated tool calls are, in NTP terms, unverified existence assertions: e-type claims that presuppose entities the model has not confirmed exist. Episode 20 showed that the United States Frontier AI Executive Order of 2 June 2026 independently confirms the necessity of external referents, while simultaneously demonstrating that pre-deployment review and output-layer verification are different architectural instruments that cannot substitute for each other.
The architecture was described. The calibration dataset was identified. The open problem was named: automated claim extraction from unstructured natural language. Today, code addresses that problem.
The Problem the Module Is Solving
A language model produces output. Some of that output asserts that things exist.
'Anthropic was founded in 2021.' 'Microsoft is headquartered in Redmond.' 'Flight NZ289 departs Auckland at 09:15.'
Each of these is an existence claim: a statement that a named entity exists and has a specific property. In NTP (Wessel 1992), these are e-type claims. They presuppose the existence of their subjects. The V.E.R.A. Logic Engine's E! Verification Service can look up those entities against external referents. But first, the pipeline needs to know which sentences contain existence claims, and which entity is the subject of each claim.
That is the claim extraction problem: parsing unstructured text into a structured stream of (subject, predicate, category) triples that the downstream verification layers can process.
This is not straightforward. Natural language does not announce its existence claims. 'The aircraft type is a Boeing 737 MAX 9' is an existence claim about an aircraft model on a specific route. 'This appears to be consistent with recent data' is not an existence claim at all. The extractor has to distinguish between them, at scale, across tool call outputs, search results, and free-form reasoning passages.
The Code
The claim extraction module is Python, version 0.1.0. It ships as claim_extractor.py alongside krampitz_analyzer.py and formula_parser.py in the Te Pono V.E.R.A. repository. The architecture is deliberately simple at this stage: a pattern library, a confidence scorer, and a pipeline runner.
The core extraction logic works as follows. The ClaimExtractor.extract() method receives a text string, splits it into sentences, and runs each sentence through a skip filter before pattern matching. The skip filter identifies hedged, conditional, or opinion-bearing sentences and routes them to the n-type path: they are not flagged for E! verification because they do not presuppose existence in the NTP sense. Sentences containing modal verbs ('might', 'could', 'may'), conditionals ('if', 'although'), or explicit uncertainty markers ('unclear', 'uncertain') are candidates for this path. The conservative default is correct: if the filter is uncertain, the sentence proceeds to pattern matching.
Six pattern categories cover the common structures in LLM tool-call output: copula assertions ('X is Y'), possession ('X has Y'), founding claims ('X was founded in...'), functional claims ('X provides Y'), numerical claims ('X reported $N'), and relational claims ('X is headquartered in Y'). A seventh category handles JSON key-value fragments from tool call responses directly.
All extracted claims are flagged as requiring E! verification by default. The n-type path, which would bypass E! lookup, is assigned only when the skip filter identifies a sentence as hedged or opinion-bearing. The conservative position is correct: if in doubt, verify.
The six pattern types map, approximately, to different NTP predication structures under Wessel's Krampitz Rule R1. All elementary predicate statements are existentially loaded under R1. 'Swan(x)' is e-type. 'Anthropic is headquartered in San Francisco' maps to an elementary predication on a named individual and is therefore e-type. The claim extractor operationalises this at the natural language level: it does not perform NTP analysis itself; it segments unstructured text into units that the Krampitz Analyser can formalise.
Calibration Against SimpleToolHalluBench
SimpleToolHalluBench (Kang et al., 2024) provides structured tool-call outputs with ground-truth entity verification. Five representative cases were run through the extractor: four ground-truth PASS cases and one ground-truth FAIL case containing a fabricated flight number and aircraft type.
| Case ID | Tool | GT Label | Claims extracted | E-type flagged | Ffact score |
|---|---|---|---|---|---|
| STH-001 | weather_api | PASS | 1 (1 matched, 2 skipped) | 1 | 0.50 |
| STH-002 | company_lookup | PASS | 5 (1 matched, 0 skipped) | 5 | 0.25 |
| STH-003 | flight_search | FAIL | 2 (1 matched, 1 skipped) | 2 | 0.33 |
| STH-004 | stock_price | PASS | 4 (2 matched, 1 skipped) | 4 | 0.67 |
| STH-005 | person_lookup | PASS | 1 (0 matched, 2 skipped) | 1 | 0.00 |
Mean Ffact score across the five cases is 0.35. Ffact, adapted from Min et al. (2023), measures the fraction of ground-truth EXISTS entities that the extractor successfully surfaces as named subjects. A score of 1.0 would mean every verifiable entity in the text was extracted as a subject. A score of 0.35 means the extractor is currently missing 65 per cent of verifiable entities across these five cases, due to four specific failure modes.
The FAIL case, STH-003, is instructive. The extractor surfaces 'Air New Zealand' and 'The aircraft type' as claim subjects. 'The aircraft type' is the implicit subject of a hallucinated claim; it correctly routes to E! verification, where a lookup of 'Boeing 737 MAX 9 on the Auckland-Sydney route operated by Air New Zealand' would return UNKNOWN or NOT_EXISTS, depending on corpus coverage. The hallucination is not invisible to the pipeline. What is missing is the explicit entity name 'Boeing 737 MAX 9' as the subject rather than the implicit referent 'The aircraft type'. That is the FM-01 failure mode.
Failure Modes: What the Code Does Not Yet Do
Naming the failure modes is not a hedge. It is the engineering record.
| Code | Failure mode | Implication for pipeline |
|---|---|---|
| FM-01 | Implicit subject | Sentence like 'The aircraft type is...' extracts 'The aircraft type' as subject rather than 'Boeing 737'. E! lookup on 'The aircraft type' returns UNKNOWN, not EXISTS or NOT_EXISTS. Requires coreference resolution. |
| FM-02 | Pronoun coreference | 'It', 'they', 'this company' are skipped entirely. A claim using a pronoun to refer to an entity named in the prior sentence is not extracted. Recall loss is real; no false positive introduced. |
| FM-03 | Coordinate subjects | 'Dario Amodei and Daniela Amodei founded...' yields only the first subject in some patterns. The second entity may be missed. |
| FM-04 | Named entity boundary | Multi-token proper nouns at sentence boundaries may be truncated. 'Humboldt University Berlin' is three tokens; current regex requires capitalised sequences with spaces and may truncate at boundaries. |
FM-01 and FM-03 are recall failures: verifiable entities are not surfaced. FM-02 is a design choice: skipping pronouns avoids false subjects, at the cost of missed claims. FM-04 is a named entity recognition gap.
None of these produce hallucinations in the extractor itself. They produce missed coverage: claims that go unverified because the extractor did not surface them. The pipeline remains safe; its recall is limited.
The distinction matters for the V.E.R.A. architecture. The claim extraction module can fail in two directions: false positives (surfacing non-claims as claims, routing non-existent subjects to E! lookup) and false negatives (missing real claims, leaving them unverified). Version 0.1.0 errs toward false negatives. That is the correct conservative position for a verification system: a missed check is a gap in coverage, not an asserted falsehood.
The Verification Gate
Check 3, mandatory: Te Pono V.E.R.A. does not claim to eliminate hallucinations, and the claim extraction module does not verify its own output. Mean Ffact of 0.35 is a measurement, not a guarantee. The E! Verification Service has not yet been connected; the figures in this article reflect extraction recall, not end-to-end verification accuracy.
The claim extraction module belongs to a class of infrastructure that some states are beginning to treat as a sovereignty question. The implication on the governance side is worth naming directly. Open-source claim extraction and external verification architectures give procurement authorities an auditable basis for assessing AI system outputs, something closed, proprietary systems cannot provide in the same way. The UK Ministry of Defence's JSP 936 framework for autonomous systems governance requires verification evidence, not vendor attestation. What JSP 936 requires as process, V.E.R.A. builds as code: a machine-readable audit trail from raw output to verified claim. The governance asymmetry between states is measurable. States with externally interrogable verification layers can produce accountability evidence on demand; states relying on probabilistic performance assurances cannot. V.E.R.A.'s claim extraction module, at Ffact 0.35, is the beginning of that infrastructure.
What Comes Next
The pipeline now has three components with running code: the Krampitz Analyser, the Formula Parser, and the Claim Extraction Module. The E! Verification Service has an architecture and a target external referent: the Wikidata application programming interface (API). The next step is to connect the extraction output to the E! lookup and run the first end-to-end pipeline.
| Component | Status | Notes |
|---|---|---|
| Krampitz Analyser (R1-R9) | Complete, all tests passing | Classifies NTP formulas as e-type or n-type. Rules from Wessel (1992). |
| Formula Parser | Complete, 15 tests passing | Converts natural language to NTP formula AST for Krampitz input. |
| Claim Extraction Module | First running code, v0.1.0 | Extracts atomic claims from unstructured text. Mean Ffact 0.35 on five calibration cases. Four failure modes documented. |
| E! Verification Service | Architecture defined; Wikidata POC pending | Awaits e-type claim stream from extraction module. Input stream now available. |
| D-Service (D1-D4) | Architecture defined | Resolves identity via four NTP indiscernibility relations. |
| End-to-end pipeline | Automated extraction operational | E! lookup is next open gap. |
Episode 22 will show the end-to-end run. The flight search case from SimpleToolHalluBench, STH-003, is the primary test: a fabricated flight number, two verifiable city names, and one verifiable airline. The expected result is that 'Air New Zealand' returns EXISTS from Wikidata, 'Auckland' returns EXISTS, 'Sydney' returns EXISTS, and the flight number claim returns UNKNOWN because no verified source confirms flight NZ289 on that route.
Improving extraction recall is the parallel v0.2.0 engineering task. FM-01 is the highest priority: implicit subjects are the mechanism by which hallucinated entities most readily escape surface extraction. FM-04 is second: multi-token proper nouns are among the most common subjects in tool-call outputs.
Join the Movement
Te Pono V.E.R.A. is open source. The claim extraction module, the Krampitz Analyser, and the Formula Parser are all in the repository. If you work in natural language processing, named entity recognition, or coreference resolution, FM-01 and FM-04 are the open engineering problems. If you work in artificial intelligence governance, the Ffact calibration framework is the measurement instrument the field needs but does not yet have in production form.
The V.E.R.A. Logic Engine is built on a logical foundation from 1992. The engineering is built in 2026. Both are open.
Repository: https://github.com/andreas-linux/vera/
Ita est momentum veritatis.
Have you seen an AI verification system fail silently, surfacing a claim as existing when the entity was fabricated, or missing a claim entirely because the subject was implicit? What did that failure cost, and how was it discovered?
The views expressed in this article are entirely my own, informed by more than 30 years of professional experience in architecture, security, and technology leadership in New Zealand. They do not represent the views of my employer, any government agency, or the New Zealand government. My commentary on legislation and policy is analytical, drawing on publicly available sources and my professional expertise in architecture, security, and AI governance. I follow the Public Service Commissioner's Code of Conduct for the Public Sector and social media guidance.
Andreas Hamberger is a New Zealand leader in Architecture & Security and Associate Member of the Institute of Directors. V.E.R.A. (Verified Existence & Reason Architecture) is an open-source logic engine available on GitHub.
I use AI tools, including Sudowrite, Claude, Perplexity AI, DeepSeek AI, ChatGPT, Grok, Copilot, Openart and Gemini, as deliberate production tools, not ghostwriters. This is consistent with my position: AI amplifies human judgement; it does not replace it. The frameworks, arguments, and editorial decisions in this series are original work. AI accelerated the process. The thinking is mine.
[1] Wessel, H. "Nichtstandard-Logik und Nichtstandard-Arithmetik: Neue Grundlagen und Anwendungen." Akademie-Verlag, 1992.
[2] Kang, J., et al. "SimpleToolHalluBench: A Benchmark for Evaluating Tool-Call Hallucinations in Language Models." arXiv:2510.22977, 2024. https://arxiv.org/abs/2510.22977
[3] Min, S., et al. "FActScoring: Fine-Grained Atomic Evaluation of Factual Precision in Long Form Text Generation." arXiv:2305.14251, 2023. https://arxiv.org/abs/2305.14251
[4] Te Pono V.E.R.A. Logic Engine. GitHub repository. https://github.com/andreas-linux/vera/
[5] UK Ministry of Defence. "Joint Service Publication 936: Autonomous Systems." HM Government, 2023. https://www.gov.uk/government/publications/joint-service-publication-936-autonomous-systems