LLMs Explained for Security Professionals: What You Actually Need to Know
Return to Part 0: Table of ContentsPrevious Article: Chapter 1, "Your Security Team Cannot React Fast Enough: The 16-Minute Problem"
ZERO TRUST ARCHITECTURE FOR THE AGENTIC ENTERPRISE
Part One: Foundations, Chapter 2 of 39
LLMs Explained for Security Professionals: What You Actually Need to Know
The Security Architect's Field Guide to How Language Models Work, Fail, and Get Exploited
**
Forty-five per cent.
That is the proportion of AI-generated code that introduces security vulnerabilities, according to Veracode's 2025 GenAI Code Security Report, which tested over one hundred large language models across eighty coding tasks.[1] Not edge cases. Not obscure languages. Across Java, Python, C#, and JavaScript, the models chose the insecure implementation nearly half the time. Java was worst: a seventy per cent failure rate.[1] When the models generated code to defend against cross-site scripting, eighty-six per cent of samples failed. For log injection, eighty-eight per cent.[1]
These are not theoretical risks discovered in academic red-team exercises. These are vulnerabilities being shipped into production systems today, by developers who trust LLM output the way a previous generation trusted compiled code.
If you are a security architect reading this, you do not need a PhD in machine learning. You do not need to understand attention mechanisms, backpropagation, or the mathematics of transformer networks. But you do need to understand five things about how large language models work, because those five things determine where the attack surface lives, why traditional security controls fail, and what you must do differently.
Chapter 1 established the machine-speed conflict: the gap between how fast AI systems operate and how fast human security teams can respond. This chapter examines the engine at the centre of that conflict. Every subsequent chapter in this book builds on the mental model you develop here.
What Happens When You Send a Prompt
When a user (or an AI agent) sends a prompt to a large language model, the system does not "understand" the request in any human sense. It executes a pipeline, and every stage in that pipeline is a potential point of security failure.
The prompt first passes through a tokeniser, which breaks text into tokens (roughly, word fragments). The sentence "Security architects must understand LLMs" might become seven or eight tokens, each mapped to a numerical identifier. This is where the model's perception of the world begins. Tokenisation is not neutral. Different tokenisers handle languages, code, and special characters differently. Adversarial inputs that look benign to a human reviewer can exploit tokenisation quirks to change meaning at the numerical level.
**
The tokens then pass through an embedding layer, which converts each numerical identifier into a high-dimensional vector (a long list of numbers representing the token's "meaning" in relation to all other tokens). These embeddings are the model's internal representation of language. For security practitioners, the critical insight is that embeddings are not one-way transformations. Research has demonstrated that embedding inversion attacks can reconstruct the original text from its vector representation.[2] If your Retrieval-Augmented Generation (RAG) system stores embeddings of sensitive documents, those embeddings are not anonymised data. They are recoverable data.
The embedded tokens then pass through the model's attention layers, the mechanism that allows the model to weigh which parts of the input are most relevant to generating each word of the output. This is where the model "decides" what matters. It is also where prompt injection operates: by crafting inputs that manipulate attention weights, an attacker can redirect the model's focus from legitimate instructions to malicious ones. The model does not distinguish between your system prompt and an attacker's injected text. Both are tokens. Both receive attention weights. Both influence the output.
Finally, the model generates output one token at a time, each token selected based on probability distributions shaped by everything that came before it. The model is not retrieving facts from a database. It is predicting the next most likely token in a sequence. This distinction is the foundation of every security concern in this chapter.
Context Windows: The Attack Surface You Cannot See
Every LLM operates within a context window, a fixed-size buffer that holds everything the model can "see" at once: the system prompt, the user's input, any retrieved documents, conversation history, and the output generated so far. Modern models have context windows ranging from 128,000 to over one million tokens. That sounds generous. It is also the largest attack surface in your AI architecture.
The context window is a shared memory space with no access controls. The system prompt (which typically contains your security instructions, role definitions, and behavioural constraints) sits in the same buffer as the user's input, which sits alongside retrieved documents from your RAG pipeline, which sit next to tool outputs and conversation history. There is no privilege separation. No memory segmentation. No hardware-enforced boundary between "trusted instructions" and "untrusted input."
For security architects accustomed to network segmentation, this is the equivalent of running your firewall rules, your application logic, and your user input through the same processing pipeline with no isolation. Every input is an instruction. Every document retrieved into context is a potential command.
This is why indirect prompt injection is so effective. An attacker does not need to interact with the LLM directly. They can embed instructions in a document that gets retrieved into the context window, in an email that an AI assistant processes, or in a web page that an AI agent visits. The model treats the injected instructions with the same weight as legitimate system prompts because, at the token level, they are indistinguishable.
The OWASP Top 10 for LLM Applications (2025 edition) places Prompt Injection at position one for precisely this reason.[3] It is not simply the most common attack. It is architecturally fundamental: the way LLMs process input makes complete prevention unclear, as OWASP itself acknowledges.[3]
**
New Zealand practitioners face a specific dimension of this challenge. The 2025 cross-agency survey reported 272 AI use cases across 70 government agencies, with 55 now deployed and operational, up from 15 in 2024.[4] Many of these use AI-enhanced search to retrieve information from large document sets. Each retrieval operation feeds data into a shared context window, and in government environments, those documents frequently contain information classified under the New Zealand Information Security Manual (NZISM) and subject to the Privacy Act 2020. The context window does not respect your classification levels.
Training, Fine-Tuning, and Inference: Three Different Threat Models
Security architects must distinguish between three phases of an LLM's lifecycle, because each carries a fundamentally different threat model.
Training is the initial process where the model learns from massive datasets (often terabytes of text from books, websites, code repositories, and other sources). The security risk is foundational: if training data is poisoned, the model's behaviour is compromised at the deepest level, and the contamination is extremely difficult to detect or remediate after the fact. You cannot patch a poisoned model the way you patch a vulnerable application. The contamination is distributed across billions of parameters with no clear mapping between specific training data and specific outputs. The OWASP Top 10 for LLM Applications classifies this as LLM03: Supply Chain Vulnerabilities.[3]
Fine-tuning adapts a pre-trained model for a specific task or domain, using a smaller, curated dataset. This is where many organisations customise models for their environment. The threat model shifts: the attack surface is now the fine-tuning dataset and the fine-tuning process itself. A compromised fine-tuning dataset can introduce targeted behaviours (responding to specific trigger phrases, leaking specific types of data) that do not manifest during standard testing. For New Zealand organisations fine-tuning models with government data, the implications under the Privacy Act's Information Privacy Principles are direct: IPP 5 (Storage and Security) requires agencies to ensure personal information is protected against loss, unauthorised access, and misuse.[5] A fine-tuning dataset containing personal information creates a copy of that data, embedded in model weights, that cannot be deleted, searched, or audited in the way a database record can.
Inference is what happens every time someone sends a prompt and receives a response. This is the phase most security teams focus on, and the phase where the attack surface is broadest: prompt injection, jailbreaking, data leakage through outputs, and tool misuse all operate at inference time. IBM's 2025 Cost of a Data Breach Report found that 97 per cent of organisations experiencing AI-related security incidents lacked proper AI access controls.[6] Thirteen per cent of organisations reported breaches of AI models or applications, with a further eight per cent unsure whether they had been compromised.[6] Among AI-related breaches, 60 per cent resulted in data compromise and 31 per cent in operational disruption.[6]
The practical implication: your organisation needs a different security posture for each phase, and most organisations have controls for none of them.
**
Table: Security Implications of LLM Deployment Models
| Deployment Model | Data Residency | Customisation | Primary Threat Model | NZ Regulatory Considerations |
|---|---|---|---|---|
| Commercial API (e.g., OpenAI, Anthropic) | Provider's infrastructure, typically US | Prompt engineering only | Data leakage via prompts; provider supply chain; context window attacks | PSR (data offshore), Privacy Act IPP 12 (disclosure), NZISM cloud controls |
| Fine-tuned commercial model | Provider infrastructure | Dataset-specific behaviour | Fine-tuning data poisoning; model theft; training data extraction | All API concerns plus IPP 5 (embedded PII), IPP 9 (retention in weights) |
| Self-hosted open-weight (e.g., Llama, Mistral) | Your infrastructure | Full control | Infrastructure security; supply chain (model provenance); resource requirements | NZISM infrastructure controls, but data residency resolved |
| RAG-augmented (any base) | Split: model (provider) + data (yours) | Knowledge injection | Retrieval poisoning; embedding inversion; context overflow; indirect injection | IPP 5 and IPP 11 (disclosure of retrievable data), classification controls |
| Edge/on-device | Local device | Limited by hardware | Physical access; model extraction; limited update capability | Reduced network exposure but endpoint security critical |
Stochastic Parrots: What Probabilistic Generation Means for Trust
The phrase "stochastic parrot" was coined by Emily Bender and colleagues in 2021 to describe what LLMs actually do: generate text by probabilistically selecting the next token based on patterns in training data, without any understanding of meaning.[7] The term remains contentious (the capabilities question is genuinely complex), but for security architects, the stochastic nature of LLM output is not a philosophical debate. It is an operational fact with direct security consequences.
First, LLMs do not retrieve information. They generate it. When a model produces a factual statement, it is not looking up a record. It is predicting which tokens are most likely to follow the preceding tokens. This means the model can generate plausible, confident, and completely wrong output (hallucination) with no internal signal that it has done so. The model has no concept of "I do not know." The 2025 state of LLM security research estimates hallucination rates between 8 and 15 per cent in unverified deployments.[8] In a security context, a hallucinated configuration recommendation, a fabricated compliance requirement, or an invented vulnerability classification is not merely unhelpful. It is actively dangerous.
Second, stochastic generation means outputs are non-deterministic. The same prompt can produce different outputs on successive runs. For security practitioners accustomed to deterministic systems (where the same input always produces the same output), this breaks fundamental assumptions about testing, validation, and audit. You cannot test an LLM the way you test a firewall rule. A rule that passes today may behave differently tomorrow.
Third, and most relevant to the Zero Trust architecture this book builds: stochastic generation means LLM output cannot be trusted by default. Every output requires verification against ground truth, policy constraints, or external validation before it triggers action. This is not a design choice. It is a security requirement. Chapter 36 introduces V.E.R.A. (Verified Existence and Reason Architecture), which addresses this gap through triple-layer ontological verification.[9] The principle, however, applies from this chapter forward: treat LLM output as untrusted input.
Where the Attack Surface Lives
With the LLM pipeline understood, security architects can map the complete attack surface. It exists at every layer.
**
Input attacks target the prompt and context window. Direct prompt injection manipulates the model through crafted user input. Indirect prompt injection embeds instructions in data the model will process (documents, emails, web pages). Jailbreaking uses elaborate prompts to override safety controls. The GreyNoise threat intelligence team documented a campaign from late 2025 in which threat actors launched over 80,000 enumeration requests against LLM endpoints, systematically probing model infrastructure with the hallmarks of a professional reconnaissance operation.[10] They are building target lists.
Model attacks target the weights, training data, or fine-tuning process. Data poisoning introduces malicious patterns during training. Model theft extracts proprietary models through systematic querying. System prompt leakage (new in the OWASP 2025 Top 10) exposes the instructions that define model behaviour, potentially revealing security controls, access patterns, and business logic.[3]
Output attacks exploit the generated content. Insecure output handling (OWASP LLM05) occurs when LLM outputs are passed to other systems without validation, potentially triggering cross-site scripting, remote code execution, or SQL injection in downstream applications.[3] Excessive agency (OWASP LLM06) arises when LLMs are given tools and permissions beyond what their task requires, allowing compromised or hallucinating models to take real-world actions.[3]
Integration attacks target the connections between the LLM and enterprise systems. RAG poisoning manipulates the knowledge base the model draws from. Tool abuse exploits the model's ability to call APIs, query databases, or execute code. ServiceNow disclosed a vulnerability in late 2025 where a second-order prompt injection through a low-privilege agent caused a higher-privilege agent to export entire case files to an external URL, bypassing the usual access controls.[11] The attack exploited not a bug in the code but the trust relationship between agents.
The Monday-Morning Takeaway
You do not need to become an AI researcher to secure these systems. But you need to internalise five principles that will govern every technical decision in the remaining thirty-seven chapters of this book.
One: all context is untrusted. The context window has no privilege separation. Treat every input (system prompts, user messages, retrieved documents, tool outputs) as potentially adversarial. Chapter 25 introduces the Logic Firewall, which operationalises this principle.
Two: outputs require verification. Stochastic generation means LLM output is, by definition, unverified. Never allow LLM output to trigger privileged actions without external validation. Chapter 36 provides the V.E.R.A. framework for systematic verification.
Three: deployment architecture is security architecture. The choice between a commercial API, a self-hosted model, a RAG-augmented system, or an edge deployment is a security decision with direct implications for data residency, attack surface, and regulatory compliance. Chapter 11 maps these choices to New Zealand's Protective Security Requirements.
Four: traditional controls are necessary but insufficient. You still need network segmentation, authentication, encryption, and logging. But you also need controls that operate at the logic layer, examining not just who is accessing what, but what the model is reasoning about and why. Part Five of this book builds these controls.
Five: the threat model is active and evolving. Eighty thousand enumeration requests against LLM endpoints. Professional threat actors building target lists. State-sponsored operations weaponising AI assistants. This is not a future risk. Chapter 1 documented the machine-speed conflict. This chapter has shown you the machine that creates it.
**
The remaining thirty-seven chapters build the architecture that secures it. Chapter 3 brings the inverse perspective: Zero Trust fundamentals for AI practitioners who build these systems but lack security foundations.
**
Next week in Chapter 3: "Zero Trust Fundamentals for Data Scientists and ML Engineers." If Chapter 2 translated AI for security professionals, Chapter 3 returns the favour, giving AI practitioners the security foundations they need before building production systems.
What was the moment you realised that an LLM in your environment was doing something you had not anticipated? I would like to hear that story.
The views expressed in this article are entirely my own, informed by more than 30 years of professional experience in architecture, security, and technology leadership in New Zealand. They do not represent the views of my employer, any government agency, or the New Zealand government. My commentary on legislation and policy is analytical, drawing on publicly available sources and my professional expertise in architecture, security, and AI governance. I follow the Public Service Commissioner's Code of Conduct for the Public Sector and social media guidance.
Andreas Hamberger is a New Zealand leader in Architecture & Security and Associate Member of the Institute of Directors. Zero Trust Architecture for the Agentic Enterprise is the first book in The Hamberger Report series, providing practitioners with deployable patterns and configurations for securing AI-driven systems.
I acknowledge the role of AI tools, such as Sudowrite, Claude, Perplexity AI, DeepSeek AI, ChatGPT, Grok, Copilot, Openart and Gemini, which assisted in drafting, editing and reviewing. They accelerated the process, but the first draft, revisions, vision, voice and final decisions were mine alone.
Publication date: 19 February 2026
References
[1]: Veracode, "2025 GenAI Code Security Report," July 2025. https://www.veracode.com/resources/analyst-reports/2025-genai-code-security-report/
[2]: OWASP, "Vector and Embedding Weaknesses," OWASP Top 10 for LLM Applications 2025. https://genai.owasp.org/llm-top-10/
[3]: OWASP, "OWASP Top 10 for Large Language Model Applications 2025," 2025. https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/
[4]: Government Chief Digital Officer, "Full Results: 2025 Cross-Agency Survey for Artificial Intelligence (AI) Use Cases," Digital.govt.nz, August 2025. https://www.digital.govt.nz/dmsdocument/263~full-results-2025-cross-agency-survey-for-artificial-intelligence-ai-use-cases/html
[5]: Office of the Privacy Commissioner, "AI and the Information Privacy Principles," Privacy.org.nz. https://www.privacy.org.nz/publications/guidance-resources/information-privacy-principles-and-artificial-intelligence/
[6]: IBM Security / Ponemon Institute, "Cost of a Data Breach Report 2025," IBM, July 2025. https://www.ibm.com/reports/data-breach
[7]: Emily Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" Proceedings of FAccT '21, March 2021.
[8]: AI Verification Research, hallucination rate range cited in multiple 2025 industry analyses including Bright Security's "2026 State of LLM Security."
[9]: V.E.R.A. (Verified Existence and Reason Architecture), open-source project by Andreas Hamberger. See Chapter 36 for full technical reference.
[10]: GreyNoise Labs, "Threat Actors Actively Targeting LLMs," GreyNoise Intelligence, January 2026. https://www.greynoise.io/blog/threat-actors-actively-targeting-llms
[11]: Sombra Inc., "LLM Security Risks in 2026: Prompt Injection, RAG, and Shadow AI," January 2026. https://sombrainc.com/blog/llm-security-risks-2026
**