Your Experiment Tracker Is Production Infrastructure Now
Return to Part 0: Table of ContentsPrevious Chapter: Part 29, The Trust Boundary Is the CI Runner
On 17 August 2026, the United States Cybersecurity and Infrastructure Security Agency gave federal civilian agencies three days to patch a piece of software most of them did not know they were running. Three days is not the normal window. The standard federal remediation deadline for a Known Exploited Vulnerabilities entry is fourteen days, and CISA applied exactly that two days later, on 19 August, to a second flaw in a second AI development tool. Same catalogue, same week, same category of software, four and a half times the urgency on one of them. CISOBrief reports that CISA declined to explain the accelerated timeline publicly.
The two entries are CVE-2025-62593 in Ray, the distributed compute framework used to scale model training, and CVE-2026-64849 in MLflow, the experiment-tracking and model-registry platform that lives under the LF AI & Data Foundation. Both were added on evidence of active exploitation in the wild, not on disclosure alone. Both are fixed in current releases. Ray 2.52.0 closes the first, MLflow 3.15.0 the second, and both patches were available before the KEV entries appeared.
So the interesting question is not whether a patch existed. It is why an organisation would have applied it. Neither of these tools arrives through a procurement process. Neither generates a change record. Neither shows up in the architecture review that a new production system triggers, because neither is a production system in the sense the review's categories mean. They are the software a data science team installs on a Tuesday to get its job done, and they have been sitting in enterprises for years holding cloud credentials that nobody classified as cloud credentials.
That is the chapter. The vulnerabilities are the evidence.
Two flaws, one category error
Take MLflow first, because its mechanism is the cleaner illustration.
MLflow supports webhooks, and it validates a webhook's destination when you register it. The _validate_webhook_url() function confirms the URL resolves to a public address, which is the right check to make. The problem is when it makes it. The delivery logic that actually fires the webhook, reachable through the unauthenticated POST /api/2.0/mlflow/webhooks/{id}/test endpoint, follows HTTP redirects and re-resolves the hostname without pinning the address it validated a moment earlier. This is a time-of-check-to-time-of-use flaw, and it is one of the oldest shapes in the catalogue.
The exploitation path is short. An attacker registers a webhook pointing at a server they control, which passes validation cleanly because it genuinely is a public address. That server then answers with a redirect to 169.254.169.254, the cloud instance metadata endpoint, or to localhost. MLflow follows it and hands back whatever is there. On a cloud-hosted instance, what is there is the machine's own AWS, GCP or Azure credentials.
CVSS v3.1 base score 9.3, vector CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:L/A:N, per the IONIX threat centre. Unauthenticated, network-reachable, no user interaction, scope changed. Yordan Ganchev of watchTowr is quoted describing attackers "abusing this vulnerability to target cloud-hosted MLflow systems", in order, his account continues, to pull credentials and secrets from the account, with scanning activity beginning within hours of the CVE identifier being assigned on 17 August 2026.
Ray's flaw is stranger, and it is the one that got three days.
Ray's dashboard and job-submission endpoints, /api/jobs and /api/job_agent/jobs/, shipped without authentication. What stood in for authentication was a check on the HTTP User-Agent header: if the string started with "Mozilla", the request was treated as coming from a browser rather than a script, and browsers were assumed to be safe because of the same-origin policy. That assumption is the whole control.
DNS rebinding dissolves it. An attacker spoofs the User-Agent, then serves a page from a domain they control whose DNS record changes mid-session to point at the local or internal address where Ray is listening. A developer visits the page, or simply sees a malicious advertisement on a page they had every reason to trust, and their own browser submits a job that executes with the privileges of the Ray process. The developer's browser is the attack vector. Nothing was phished, nothing was downloaded, and no credential was stolen to make it work.
Ray is scored 9.4 under CVSS 4.0 and 8.8 under CVSS 3.1. Both figures are correct under their own scoring system and should not be presented as a discrepancy or averaged into a single number.
One detail needs stating precisely, because it is easy to read backwards. Resecurity reports that Bitsight observed the RondoDox botnet attempting to exploit this flaw before public disclosure, and that the technique RondoDox used at the time did not successfully bypass Ray's browser check. That failed early attempt is a separate event from the active exploitation CISA's KEV entry responds to. No source connects the two, and neither should we.
What the two flaws share is not a mechanism. MLflow's is a validation race, Ray's is a network-identity confusion, and they have almost nothing in common at the code level. What they share is the sentence that follows the incident report in every organisation that runs either one: nobody owned it, because it was not that kind of system.
The Sandbox Illusion
There is a name for this, and the name is more precise than "shadow IT" or "unmanaged asset".
The Sandbox Illusion is the belief that a test environment is a containment boundary rather than an obstacle, and the corresponding assumption that a system being evaluated warrants less governance than a system being deployed.
I named it in my Gen AI Tuesday series, at Part 31, The Boundary You Outsourced, where the failure was a test environment treated as a wall. This is its first use in an enterprise architecture setting, and the first time I have applied it to the question an asset register actually asks: what counts as a system.
Both halves are doing work here, and the second half is the one that produced these two KEV entries. An experiment tracker is unambiguously a system being evaluated in, not a system deployed to. It serves no predictions. It sits upstream of every pipeline that matters. Under any lifecycle-stage taxonomy, it is development tooling, and development tooling gets development-tooling governance: no inventory entry, no named patch owner, no subscription to the upstream advisory feed, no credential rotation schedule.
Meanwhile the credentials it holds are production credentials, because that was the only practical way to let the team log artefacts against real training data. The permission was granted once, for a good reason, by someone with the authority to grant it, and then nobody removed it, because removal has a cost and inaction has none. This is Architectural Debt in its most ordinary form: not a bad decision, a decision that was never revisited.
The illusion is not that anyone believed the experiment tracker was secure. It is that the question of whether it was secure was routed to a governance process calibrated for something that cannot hurt you.
An attacker does not consult your lifecycle taxonomy. An attacker enumerates what holds credentials and what can be reached, and finds that the experiment tracker scores well on both.
Where the trifecta fits, and where it does not
Readers of Chapter 5, "The Three Capabilities That Turn Your AI Into an Insider Threat," will recognise the shape of the MLflow case immediately, and it is worth being careful about the fit rather than forcing it.
The lethal trifecta, the term Simon Willison coined in June 2025 for the combination of private data access, untrusted content exposure, and external communication capability, describes the compound risk that appears when a single execution context holds all three at once. MLflow satisfies all three cleanly. It holds private data access through its cloud role. It takes untrusted content through an unauthenticated registration endpoint. It communicates externally by design, because delivering webhooks is the feature. Three properties, one context, no supervision.
Ray is a two-property case, and saying so is more useful than rounding it up. The untrusted input reaches a privileged execution environment, but the outbound leg is not a separate capability the attacker had to find. It is the same channel they arrived on, running in reverse. The trifecta's third property is supplied by the topology of the attack rather than by a feature of the software.
That variant deserves its own name eventually, and it is not this chapter's job to give it one. What matters for the architect is that a two-property environment is not two-thirds as dangerous as a three-property one. Where the inbound path and the outbound path are the same path, removing the third capability removes nothing, because there was never a third capability to remove. The control that works on Ray is authentication and network restriction, not egress filtering.
What the inventory was actually counting
Chapter 24, "Sixteen Hundred Agents and No Inventory: The Identity Layer Zero Trust Forgot," measured how far agent populations had outrun the registers meant to track them. Chapter 26, "The Agent Sprawl Nobody Provisioned," took the same measurement for tooling that arrives without a discrete deployment event. Chapter 28, "Ninety-One Per Cent of Your Identities Are Not People: Zero Trust for the Machine Majority," put a number on the machine side of the identity estate.
This chapter is those three arguments one layer further upstream, and the upstream position changes something.
An agent that appears without provisioning is at least recognisably an agent. Somebody, somewhere, understands that a thing which takes actions ought to have been registered. An experiment tracker triggers no such instinct, because it does not act. It records. It compares runs and stores artefacts and draws charts, and the entire mental model it invites is the one a data science lead reaches for by default: a lab notebook that happens to run on a server.
Lab notebooks do not hold IAM roles. This one does.
The gap that Chapter 12, "The Sixth Pillar: Agent Identity and the New Perimeter," identified as the missing Zero Trust pillar has a precondition nobody states, which is that you know the thing exists. Agent Identity as Perimeter is the right architecture and it is unreachable for any system your inventory has never named. You cannot scope a credential you have not enumerated, and you cannot rotate one whose owner is unassigned.
A blueprint: classify by what it holds
Here is the correction, and it is a schema rather than a policy, because policies about developer tooling have been written before and these two tools were exposed anyway.
The governing principle is one line. An asset's governance class is computed from what it holds and what can reach it. It is never inherited from its lifecycle stage.
# development-asset-classification.yml# Reference schema. Lifecycle stage is recorded and then ignored.asset:asset-id: mlflow-tracking-01owner-team: data-sciencelifecycle-stage: development # informational onlyfirst-deployed: 2024-11-03deployment-event: none # no change record, no architecture reviewholds-credentials:- credential-type: cloud-roleprovider: awsscope: s3-readresource-shared-with-production: truelast-rotated: never- credential-type: metadata-endpoint-reachableprovider: awsendpoint: 169.254.169.254egress-restricted: falsereachable-from:- source: unauthenticated-networkpath: http-apiauthentication: none- source: developer-browserpath: dns-rebindingauthentication: user-agent-string-checkpatch-management:inventory-entry: absentpatch-owner: unassignedupstream-advisory-feed: not-subscribedgovernance-class:computed-from:- holds-credentials- reachable-fromnever-computed-from:- lifecycle-stagerule: >An asset holding any credential whose scope reaches a production resource,and reachable by any source that is not authenticated, is classifiedproduction-privileged regardless of lifecycle-stage.result: production-privilegedenforcement:- inventory-entry: required- patch-owner: required- credential-scope: must-not-share-production-resources- upstream-advisory-feed: requiredRun the two KEV entries through it and the result is not subtle.
| Asset | lifecycle-stage | holds-credentials | reachable-from | governance-class |
|---|---|---|---|---|
| MLflow tracking server | development | cloud-role, production-shared | unauthenticated-network | production-privileged |
| Ray head node | development | cloud-role, cluster-execution | developer-browser, unauthenticated-network | production-privileged |
| Model-serving cluster | production | cloud-role, production-shared | authenticated-network | production-privileged |
Three systems, one governance class, and the column that every existing inventory sorted on is the column that carries no weight.
The schema is a starting shape, not a finished control, and it will need adapting to whatever asset register an organisation already runs. The point that transfers is the ordering of the fields. Ask what it holds before you ask what it is.
The evidence problem: how would you know?
Reclassification tells you what to govern. It does not tell you what actually happened inside an environment you have just discovered you were not watching.
Chapter 10 named that outcome Proof of Action: the ability to prove what an agent did, when, with which permissions, and on which inputs. It has been an architectural aspiration in this series for five months, mostly because the instrumentation to produce it did not exist in portable form.
On 25 August 2026, the Linux Foundation announced TRACE, contributed by OPAQUE and developed with AMD, Intel, Microsoft and the Technology Innovation Institute. It defines a format, an anchoring protocol and verification rules for a cryptographically provable, hardware-rooted artefact binding the runtime environment, software, policies, data classifications and tool usage a workload operated under, portable across cloud providers and confidential-computing environments. The Linux Foundation governs the specification; the technical workstream sits with the Coalition for Secure AI. Its reference library recorded close to 135,000 PyPI downloads in the ten weeks after its introduction at the June 2026 Confidential Computing Summit.
That is Proof of Action with hardware underneath it, and it is worth separating from Constraint Attestation, the construct Chapter 20 introduced for attestation at each delegation event in a multi-hop agent chain. Constraint Attestation asks whether this hop was permitted. TRACE asks what this environment was, in a form someone else can verify. Different questions, complementary answers, and conflating them would lose both.
Two qualifications, stated rather than buried.
TRACE was announced on 25 August 2026, eight and six days after the KEV entries this chapter opens with. There is no causal relationship, and nobody has demonstrated that TRACE would have surfaced an unpatched Ray or MLflow instance before an attacker's scanner did. The claim here is architectural fit assessed from its stated design goals, which is reasoning about structure, not evidence about outcome.
And foundation governance is not the control people assume it is. MLflow is an LF AI & Data Foundation project. It carried a 9.3 unauthenticated server-side request forgery flaw into active exploitation anyway. Vendor-neutral stewardship of a specification is a different thing from assurance about an implementation, and this week supplied the demonstration.
The New Zealand position
There is no New Zealand hook for these two vulnerabilities, and the honest thing is to say so rather than manufacture one.
The National Cyber Security Centre published "Cyber Readiness in the Frontier AI Era" on 4 June 2026. It directs organisations to review compliance against the New Zealand Information Security Manual and the NCSC Cyber Security Framework, to confirm executive accountability for AI-related risk, and to identify and remediate material gaps. It takes a governance posture rather than a technical-controls one. It does not mention Ray, MLflow, the KEV catalogue, or AI development-pipeline tooling by name, and it was not written in response to anything that happened in August.
That said, "confirm executive accountability for AI-related risk" is a question with a specific answer in this context, and it is not a comfortable one. Accountability for a system nobody has inventoried cannot be confirmed. It can only be assumed, which is what an executive attestation about an unenumerated estate actually is.
The applicable technical baseline remains NZISM v3.9. This chapter makes no claim about what any particular NZISM chapter requires of development-tooling patch cadence, because that text was not read for this piece.
Three consecutive chapters in this series have now reached a New Zealand section and found no local publication addressing the topic directly. That is worth noticing as a pattern rather than reporting three times as an absence.
What to do on Monday
Four things, in order, and the first one is the only one that is urgent.
Enumerate what your data science teams run. Not what was procured. What is running. MLflow, Ray, Kubeflow, Weights and Biases, a model registry somebody stood up in 2023, a Jupyter server on a VM with a name nobody recognises. Ask the teams directly, because the asset register will not have it.
For each one, answer two questions. What credentials does it hold, and what is the scope of each? What can reach it, and what authenticates that reach? Everything else in your classification can wait.
Reclassify on the answers, not on the lifecycle stage. Anything holding a credential that touches a production resource, reachable by anything unauthenticated, is production-privileged. Give it an inventory entry and a named patch owner today, and narrow the credential scope this quarter.
Then check the two specific things this month made urgent. Ray at 2.52.0 or later, with token authentication enabled and management interfaces network-restricted. MLflow at 3.15.0 or later. The federal deadlines were 20 August and 2 September 2026. Yours are whatever you set, and until last month most organisations had not set one, because the question had never been asked of these systems.
The data science lead in the audit did not fail her checklist. Her checklist asked which systems were production systems, and it got an honest answer to the wrong question.
Ray and MLflow are both open source, and this chapter has shown why that cuts two ways. Ray carries an Apache 2.0 licence under Anyscale's stewardship, the funding model most infrastructure-scale open projects depend on for maintenance. MLflow sits inside the Linux Foundation AI and Data Foundation, the vendor-neutral home built for projects too foundational for one company to own. Foundation governance answers a real question: whose roadmap this is, and who can be trusted not to capture it. It does not answer whether this month's code was reviewed closely enough to catch a validation-timing flaw before it reached a customer's cloud credentials. A foundation can steward a project's direction and still not staff the hours to catch its own defect first. Sustainability and review capacity are related. They are not the same question.
The dependency-privilege gap this chapter tracks in two AI tools is not new to defence and national-security thinking; it is a lineage with three prior chapters. The SolarWinds compromise, discovered December 2020, showed a routine network-management update could carry a nation-state implant into thousands of networks, because nobody had questioned what privilege a monitoring tool needed. The Log4j vulnerability, disclosed December 2021, showed the same gap one layer down: a logging library present in almost every Java application, tracked by almost nobody, held enough reach to compromise everything depending on it. The XZ Utils backdoor, found March 2024, showed it again, engineered into a compression library maintained by one volunteer, reaching the authentication path of much of the internet's Linux servers. Ray and MLflow extend the lesson to a fourth category, AI development tooling. The category a dependency sits in never predicted the privilege it is trusted with.
What is running in your environment right now that holds a production credential and has never appeared on a patch-management schedule, because nobody classified it as the kind of thing that needs one? I would like to hear what turned up when you went looking.
If your programme needs an independent architecture assurance review before its next gate, message me and I will send the scope and the fixed fee.
The views expressed in this article are entirely my own, informed by morethan 30 years of professional experience in architecture, security, andtechnology leadership in New Zealand. I write as director of Te PonoLimited; the views are personal and do not represent the position of anyclient, any government agency, or the New Zealand government. My commentaryon legislation and policy is analytical, drawing on publicly availablesources and my professional expertise in architecture, security, and AIgovernance, and it is politically neutral.
Andreas Hamberger is a New Zealand leader in Architecture & Security and Associate Member of the Institute of Directors. Zero Trust Architecture for the Agentic Enterprise is the first book in The Hamberger Report series, providing practitioners with deployable patterns and configurations for securing AI-driven systems. Through Te Pono he provides independent architecture assurance reviews and AI verification audits to boards and programmes; contact andreas@thehambergerreport.com for the scope and fixed fee.
This article was produced with AI assistance under my direction. Research,drafting and images pass through a pipeline I built and govern: automatedgates for source verification, forbidden language and political neutrality,and my own review before anything is published. The tools include Claude,Gemini and Openart. The frameworks, arguments and editorial judgements aremine and are the same discipline I apply to the AI systems I audit forclients. AI accelerated the work; the thinking, and the responsibility forit, are mine.
[1] The Hacker News, "Attackers Exploit MLflow SSRF Flaw to Steal Cloud Credentials and Secrets", August 2026. https://thehackernews.com/2026/08/attackers-exploit-mlflow-ssrf-flaw-to.html
[2] BleepingComputer, "CISA warns of hackers exploiting critical MLflow vulnerability", August 2026. https://www.bleepingcomputer.com/news/security/cisa-warns-of-hackers-exploiting-critical-mlflow-vulnerability/
[3] IONIX Threat Center, "CVE-2026-64849, Unauthenticated SSRF via Webhook Redirect Bypass". https://www.ionix.io/threat-center/cve-2026-64849/
[4] The Hacker News, "CISA Flags Actively Exploited Ray Flaw That Can Trigger Browser-Based RCE", August 2026. https://thehackernews.com/2026/08/cisa-flags-actively-exploited-ray-flaw.html
[5] Security Affairs, "U.S. CISA adds a Ray-Project Ray flaw to its Known Exploited Vulnerabilities catalog", August 2026. https://securityaffairs.com/197419/security/u-s-cisa-adds-a-ray-project-ray-flaw-to-its-known-exploited-vulnerabilities-catalog.html
[6] CISOBrief, "CISA imposes three-day fix deadline on actively exploited Ray framework RCE flaw", August 2026. https://cisobrief.com/cisa-imposes-three-day-fix-deadline-on-actively-exploited-ray-framework-rce-flaw/
[7] Resecurity, "Ray CVE-2025-62593: Critical Browser-Driven RCE via DNS Rebinding". https://www.resecurity.com/blog/article/ray-cve-2025-62593-critical-browser-driven-rce-via-dns-rebinding
[8] Simon Willison, "The lethal trifecta for AI agents: private data, untrusted content, and external communication", 16 June 2025. https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/
[9] Linux Foundation, TRACE (Trust, Runtime Attestation and Compliance Evidence), announced 25 August 2026.
[10] National Cyber Security Centre New Zealand, "Cyber Readiness in the Frontier AI Era", 4 June 2026.
[11] New Zealand Information Security Manual (NZISM) v3.9, 9 May 2025.