Language Is Not Capability: Why AI Governance Documents Cannot Audit Themselves
Return to Episode 1: V.E.R.A. Series OverviewPrevious Article: Episode 13 - The Mechanism Behind the Lie
A research team at MIT fed more than 1,000 AI governance documents into a large language model and asked it to classify their coverage. The model gave high scores for governance failure coverage to documents that used phrases like "risk management," "oversight," and "accountability." Human expert reviewers applying the same task gave materially lower scores. They were looking for something the model was not: evidence that the document actually specified mechanisms to prevent governance failure, not just language that described governance as a concern.
The MIT AI Risk Repository Version 4, published April 2026, documented this pattern across the full dataset. The model was not wrong about the words. It was wrong about what the words proved.
Language is not capability. Governance vocabulary is not governance substance. This distinction matters enormously, and AI systems cannot reliably make it.
This is Episode 14 of V.E.R.A. Saturday. Episode 13 showed why AI systems cannot distinguish confidence from accuracy: the training incentive mechanism produces confident wrong answers as a structural outcome of the objective function, not as a performance failure. This episode extends that argument to a new domain. The same structural limitation that produces confident wrong answers also produces confident misclassifications of governance evidence. The architecture required to address both problems is identical.
What the MIT Team Actually Found
The MIT AI Risk Repository team used an LLM pipeline to classify documents from CSET's AGORA archive, a curated collection of AI governance documents from governments, standards bodies, and major institutions worldwide. The classification task was to assign coverage scores across a set of risk sub-domains, including Governance Failure.
The finding on Governance Failure coverage was specific. LLMs over-attributed coverage to this sub-domain relative to human expert reviewers. The source of the over-attribution was identifiable: the model was treating the presence of governance-related language as evidence that governance failure was substantively addressed. Human reviewers applied a different standard. They required either an explicit diagnosis of a governance breakdown or a specified mechanism designed to prevent one. Governance vocabulary was not sufficient. Governance mechanism was required.
The scale matters here. More than 1,000 documents. International coverage across governments, standards bodies, and corporate policy collections. The finding is not an artefact of a small sample or a narrow document selection. It holds at the scale that represents how governance documents are actually produced and assessed.
The MIT team also found limits in their own human calibration process. Their original five-point coverage scale was consolidated to a three-point scale after analysis showed that human raters could not draw consistent distinctions between "basic" and "minimal" coverage. This finding directly supports a design choice in V.E.R.A.'s architecture. V.E.R.A.'s binary e-type/n-type classification is deliberately simpler than the MIT team's scale, and that simplicity is a feature, not a limitation. The MIT finding provides external validation that finer-grained classifications introduce calibration instability which undermines consistency. A classification system that cannot be applied consistently by trained human reviewers will not be applied consistently by any system at all.
The gap between what the model credited and what the human reviewers required is the governance vocabulary gap at measured scale. Understanding why it exists requires going back to the first principles that sit at the core of V.E.R.A.'s architecture.
The NTP Frame: Why This Is an Existence Claim Problem
V.E.R.A. is built on Non-Traditional Predication Theory (NTP), developed by Professor Horst Wessel at Humboldt-Universität zu Berlin. I studied directly under Wessel from 1991 to 1996 and served as his research assistant during that period. NTP draws a formal distinction between two types of claim:
| Claim Type | Definition | Example |
|---|---|---|
| e-type | Existence claim. Asserts that something is real or present in the world. Requires source attribution and verification before carrying evidential weight. | "We have a risk management process." |
| n-type | Normative or analytical claim. Follows logically from stated premises. Does not assert external existence. Can be evaluated against its premises without external verification. | "If a risk management process exists and functions as specified, then governance failure in this domain is mitigated." |
When an AI governance document states "we have a risk management process," it is making an e-type claim. It asserts that a specific capability exists in the real world. Under V.E.R.A.'s framework, that claim requires a verifiable source before it can carry evidential weight: documentation of the process, an audit trail, or an accountable individual who can testify to its operation. The claim can be sincere. It can be well-intentioned. It still requires evidence.
The LLM in the MIT study was treating the e-type claim as sufficient evidence for the n-type inference. It was reading "we have a risk management process" as evidence that governance failure risk is addressed. That is the precise misclassification V.E.R.A.'s Krampitz Analyser is built to prevent.
V.E.R.A.'s Existence-Predication Firewall is the architectural term for this separation. The firewall sits between what a document says and what a document proves. A governance document that says "we apply the OECD Due Diligence Guidance for Responsible AI" has asserted an e-type claim. It has not provided evidence for that claim. Those are different logical objects. Standard LLM processing does not separate them. V.E.R.A. does.
The conflation happens because LLMs are trained on text. Text contains both e-type and n-type claims, frequently in close proximity. The training process optimises for language prediction, not for claim-type classification. The model learns that governance vocabulary co-occurs with governance coverage assessments. It does not learn to require evidence for the existence claims that make up the vocabulary. That is not a bug in the model. It is a structural consequence of what language models are built to do.
How the Krampitz Analyser Routes These Claims
The Krampitz Analyser, V.E.R.A.'s claim classification module, implements the NTP e-type/n-type distinction as a deterministic pre-processing step. Every claim presented to V.E.R.A. is classified before any evaluation begins. The classification is binary: e-type routes to the E! Verification Service; n-type is assessed against its stated logical premises.
The Krampitz Analyser never consults the model's confidence score. A claim that arrives at high confidence with no verifiable source is still an unverified e-type claim. Confidence and evidential adequacy are independent dimensions. Episode 13 documented this for factual claims and training incentive structures. Episode 14 documents the same principle for governance coverage claims. The architecture is identical because the logical problem is identical.
For governance coverage claims specifically, the Krampitz Analyser would classify "we have a risk management process" as e-type and route it to the E! Verification Service. The E! service would require a citation: a document, a system record, or an audit trail. Without that citation, the claim does not advance to evidential weight. The document may use governance vocabulary. The vocabulary does not carry the e-type claim across the evidential threshold.
This is precisely the function that the MIT team's human expert reviewers were performing intuitively. They were applying an evidential standard: mechanism required, not vocabulary. V.E.R.A. encodes that standard deterministically, at the point of claim intake, before any scoring or assessment begins.
The V.E.R.A. v0.1 codebase, released under GPL-3.0 in April 2026, implements the Krampitz Analyser for factual and analytical claims. The governance coverage claim extension is the direction this episode maps for the Coding Arc. It is a direction marker, not an implementation commitment. Episode 15 will show the first code.
The Jagged Frontier, Extended
The Jagged Frontier thesis, well established in AI capability research, holds that AI systems do not fail uniformly. Performance is strong in some domains and unexpectedly weak in adjacent ones. The capability frontier is uneven.
The MIT finding adds a second dimension to this picture. AI systems are also uneven in meta-cognitive capability. Specifically, they are unreliable judges of whether a document provides substantive evidence for a claim or merely uses vocabulary associated with that claim's domain.
This is distinct from performance failure in a specific task domain. The LLM classifying governance documents is not confused about what "risk management" means. It applies that vocabulary correctly as a pattern. The failure is in treating the pattern as a proof. An LLM that can draft governance documents, explain governance frameworks, and summarise governance research accurately may still be unreliable at distinguishing which governance documents actually provide evidence of governance capability rather than governance aspiration.
The two-dimensional Jagged Frontier, then, maps across two axes: task performance and meta-cognitive reliability. AI systems can score well on task performance while scoring poorly on meta-cognitive reliability. These are independent dimensions. The MIT finding demonstrates the independence empirically: in the governance failure sub-domain, task performance was adequate in the sense that the model correctly identified governance vocabulary. Meta-cognitive reliability failed in the sense that the model could not distinguish vocabulary from evidence.
V.E.R.A. addresses both dimensions by substituting deterministic NTP classification for AI meta-cognition. The question posed to the system is not "does this document sound as though it addresses governance failure?" The question is: "does this claim, in this document, meet the evidential standard required for an e-type assertion?" One question is answered by language pattern recognition. The other is answered by logic. These are different operations requiring different architecture. V.E.R.A. performs the second operation. Standard LLMs perform the first.
The Practical Problem: AI-Generated Compliance Summaries
The governance vocabulary problem is not confined to academic classification exercises. Organisations are generating compliance summaries, governance assessments, and maturity scores using LLM pipelines. Those outputs are being presented to boards, regulators, and procurement panels as evidence of governance substance.
Under the MIT finding, those summaries carry systematic over-attribution risk. An LLM reading an organisation's AI governance policy and asked to assess governance maturity will credit governance vocabulary as evidence of governance capability. The MIT team demonstrated this with 1,000 documents under controlled conditions, with expert comparison scores as the reference. In commercial practice, the assessments are not being compared to human expert reviews at all. Organisations are acting on the vocabulary score without knowing that it is a vocabulary score.
The reach of this problem extends to director liability contexts. Company directors in New Zealand are required to exercise reasonable care, diligence, and skill under the Companies Act 1993. The reasonable steps standard that courts and regulators apply requires that governance documentation reflect actual capability, not merely describe it. A board that commissioned an LLM-generated governance maturity assessment may believe it has taken reasonable steps to understand its AI governance position. If that assessment awarded high marks for governance failure coverage based on vocabulary rather than evidence, the board may have a misleading picture of its actual exposure. The assessment looks like due diligence. Under the MIT finding, it may not be.
The Stanford AI Index 2026 reported a sharp gap between expert and public assessments of AI impact: 56% of AI experts expressed positive views, compared to 10% of the general public. One interpretation: the people who build AI systems understand their limitations better than the people who commission governance reports about those systems. The gap in understanding is exactly where the vocabulary-substance confusion does the most operational damage. The people making governance decisions based on LLM compliance summaries are often not the same people who know what LLM pipelines can and cannot distinguish.
A New Zealand Dimension: IPP 3A Takes Effect
From 1 May 2026, Privacy Act 2020 changes under Information Privacy Principle 3A require agencies to notify individuals when personal information is collected indirectly. Any AI system that generates a compliance summary stating "we comply with IPP 3A" is making an e-type existence claim. That claim requires a verifiable source: documentation of the notification mechanism, evidence that it operates as specified, and attribution to an accountable individual who can confirm compliance.
The MIT finding indicates that AI-generated compliance summaries will not self-generate this evidential standard. They will produce language that describes compliance. Language that describes compliance is not compliance. V.E.R.A.'s framework treats these as categorically different claims and routes them accordingly. The vocabulary is not the evidence. The evidence is the evidence.
The New Zealand AI Strategy and associated guidelines use governance vocabulary consistently: transparency, accountability, fairness, human oversight. Under the MIT finding, documents using this vocabulary would receive elevated LLM coverage scores in a governance assessment pipeline. That is a structural observation, not a critique of the documents themselves. The observation applies to governance documents from every jurisdiction. No jurisdiction has yet specified the evidential standards that would allow governance vocabulary to be formally distinguished from governance substance at a regulatory level. New Zealand is not exceptional in this regard; the gap is global.
The practical implication for New Zealand organisations preparing for IPP 3A compliance: an AI governance assessment that relies entirely on LLM scoring of your policy documents is measuring vocabulary coverage, not operational maturity. Those are different things. The distinction matters for Privacy Act compliance obligations, for reasonable steps arguments under the Companies Act, and for the operational reality of whether your AI systems are doing what your governance documentation says they are doing.
Two Episodes, One Architectural Argument
Episodes 13 and 14 together make the complete case for V.E.R.A.'s NTP classification layer:
| Episode | Finding | Architectural Response |
|---|---|---|
| Ep 13 | AI systems cannot distinguish confidence from accuracy. Confident wrong answers are a structural outcome of training incentive mechanisms, not a performance edge case. | Route on claim type, not confidence. The Krampitz Analyser never consults the model's confidence score. |
| Ep 14 | AI systems cannot distinguish governance vocabulary from governance substance. Governance coverage claims receive evidential weight they have not earned. | Classify governance claims as e-type assertions requiring source attribution before evidential weight is assigned. |
| Combined | AI meta-cognition is unreliable across two independent dimensions: confidence calibration and evidence classification. Both dimensions require the same fix. | NTP-based deterministic claim classification substitutes for AI meta-cognition in both dimensions. The architecture is identical because the logical problem is identical. |
The connection to the Cyber Sunday board governance series is worth noting. The accountability gap that series has been mapping at the board level throughout 2026, between what an organisation documents and what its governance actually delivers, is structurally identical to the verification gap V.E.R.A. is addressing at the logical layer. Governance without verification is aspiration at both levels. The NTP architecture that makes V.E.R.A.'s verification auditable is the same architecture that would make board-level governance maturity assessments defensible.
A Worked Example: Classifying the Claims
Consider this statement from commercial AI analysis published in April 2026:
"AI does not create new vulnerabilities, it exposes existing ones."
What type of claim is this? It is n-type: a logical inference from the premise that vulnerabilities pre-exist AI deployment. AI changes the discovery rate and the exploitation rate. It does not change the existence rate of the underlying vulnerability. The claim follows analytically from that premise.
V.E.R.A. would classify it as n-type, assign it the confidence level warranted by the evidence for the underlying premise, and note that the inference holds only if the reader accepts that premise. The confident framing of the original statement is not treated as evidence that the premise is established. Confidence is not verification.
Now contrast: "Our AI system complies with the NIST AI Risk Management Framework." This is e-type. It asserts a real-world compliance relationship. Under V.E.R.A.'s framework, it routes to the E! Verification Service. Source attribution required before this claim carries evidential weight.
The NIST AI Risk Management Framework Profile for Trustworthy AI in Critical Infrastructure is currently at concept note stage as of April 2026. The criteria for adequate governance evidence under this framework are not yet specified at a regulatory level. V.E.R.A. would flag this accurately: the claim is e-type, source attribution is required, and the required evidential standard for this specific claim is undefined at the regulatory level. That is a precise and useful output. An LLM-generated compliance assessment would likely score the statement as governance coverage. These are different answers to different questions.
What Comes Next: The First Code
The practical engineering direction from Episode 14 is a governance coverage extension for V.E.R.A.'s claim-type classifier. The Krampitz Analyser was designed around factual and analytical claims. The MIT finding provides external validation that the same NTP classification architecture applies to governance coverage claims. The MIT team's dataset of more than 1,000 governance documents with human-expert coverage scores is, in principle, a calibration dataset against which V.E.R.A.'s governance claim classifier could be measured. That is a direction marker for a future Coding Arc episode.
Episode 15 is the first implementation episode: the claim extraction module, show the code, show the failure modes. The Ffact score is the success criterion. The claim extraction problem is the hardest unsolved problem in the current V.E.R.A. development arc, and Episode 15 will document the first attempt honestly, including the parts that do not work. That is how an open-source project earns the trust of contributors.
V.E.R.A. is open source under GPL-3.0. The repository is at https://github.com/andreas-linux/vera/. If you work in AI governance, AI verification, formal logic in production systems, or you just want to see what an NTP-grounded claim classifier looks like in working code, the project is open for contribution.
Next week in Episode 15: First Implementation Attempt: Show the Code, Show the Failure Modes. The claim extraction module goes into the repository. The Ffact score measures the result.
If you fed your organisation's AI governance policy into a language model and asked it to assess your governance maturity, what score would it give you? More importantly: how would you know if that score was right?
The MIT team knows the score was systematically wrong for more than 1,000 documents. They know because they had human expert reviewers applying an evidential standard. Most organisations running LLM governance assessments do not have that comparison. They have the vocabulary score. They are treating it as the substance score.
V.E.R.A.'s NTP classification layer is designed to close that gap, not by generating better vocabulary scores but by refusing to award evidential weight to claims that have not met an evidential standard. That is a different function from what LLMs are trained to do. It is also the function that governance actually requires.
Ita est momentum veritatis.
The views expressed in this article are entirely my own, informed by more than 30 years of professional experience in architecture, security, and technology leadership in New Zealand. They do not represent the views of my employer, any government agency, or the New Zealand government. My commentary on legislation and policy is analytical, drawing on publicly available sources and my professional expertise in architecture, security, and AI governance. I follow the Public Service Commissioner's Code of Conduct for the Public Sector and social media guidance.
Andreas Hamberger is a New Zealand-based enterprise architect, security practitioner, and technology strategist with more than 30 years of experience across government, banking, transport, and aviation. He holds a Master of Arts in Philosophy (Logic) from Humboldt-Universität zu Berlin, where he studied under Professor Horst Wessel, the founder of Non-Traditional Predication Theory. He is TOGAF-certified, holds IAPP credentials, and is an Associate Member of the Institute of Directors New Zealand. He is the creator and lead architect of V.E.R.A. V.E.R.A. (Verified Existence and Reasoning Architecture) is an open-source logic engine available on GitHub.
I use AI tools, including Sudowrite, Claude, Perplexity AI, DeepSeek AI, ChatGPT, Grok, Copilot, Openart and Gemini, as deliberate production tools, not ghostwriters. This is consistent with my position: AI amplifies human judgement; it does not replace it. The frameworks, arguments, and editorial decisions in this series are original work. AI accelerated the process. The thinking is mine.
[1] MIT AI Risk Repository. "AI Risk Repository Version 4." April 2026. https://airisk.mit.edu
[2] CSET. "AGORA: AI Governance Research Archive." Center for Security and Emerging Technology, Georgetown University. https://agora.cset.institute
[3] Wessel, H. "Ontology Without Borders." Akademie-Verlag, Berlin, 1992.
[4] Krampitz, H. "Formallogische Grundlagen der Pradikation." Humboldt-Universitat zu Berlin, 1990.
[5] Stanford University Human-Centered AI. "Artificial Intelligence Index Report 2026." April 2026. https://hai.stanford.edu/ai-index
[6] New Zealand Privacy Commissioner. "Information Privacy Principle 3A: Indirect Collection Notification." Office of the Privacy Commissioner, 2026. https://privacy.org.nz
[7] National Institute of Standards and Technology. "AI Risk Management Framework Profile for Trustworthy AI in Critical Infrastructure (Concept Note)." NIST, 7 April 2026. https://nist.gov/artificial-intelligence
[8] V.E.R.A. Project. "Verified Existence and Reasoning Architecture: v0.1 Repository." Te Pono Limited. https://github.com/andreas-linux/vera/