The Null Pointer
An announcement crossed my feed this week carrying the informational density of a null pointer. Anthropic, the AI research lab, had expanded third-party access to its frontier models for METR โ the Model Evaluation & Threat Research organization, formerly known as ARC Evals. The item shipped with exactly two facts attached: the access grant itself, and a prediction-market figure, "46.5% YES," appended without a named subject, platform, or resolution rule. No model versions. No access mechanism. No evaluation scope. No disclosure rights. No timestamp beyond the publication date.
I have spent the better part of a decade reading smart contract disclosures, and the habit that has served me best is treating the shape of what is omitted as data in its own right. A governance announcement that specifies everything except the mechanism is not a governance announcement. It is a positioning statement wearing the syntax of one. The signal here is not what Anthropic gave METR. It is what the disclosure chose to leave unspecified โ and in a domain where verifiability is the entire product, the omissions are the story.
That framing matters because this item sits at the intersection of two industries I work in daily: frontier AI safety, and the cryptographic machinery for proving that a computation happened the way someone claims it did. The first is where Anthropic operates. The second is where the blockchain field has spent fifteen years and several billion dollars learning expensive lessons about trust, attestation, and the persistent gap between a claim and a proof. Those lessons transfer directly here, and almost nobody in the AI commentary space is applying them.
What METR Actually Is
To understand the stakes, you need the institutional backdrop, which the announcement omitted entirely.
METR is not a casual reviewer. It is an independent nonprofit that specializes in evaluating frontier models for dangerous capabilities: autonomous replication, offensive cyber capability, biological and chemical uplift, deception, and sabotage. Its independence is the load-bearing wall of its credibility. An evaluation performed by a lab on itself is marketing. An evaluation performed by a third party with the freedom to publish negative findings is evidence. The entire distinction rests on that freedom, and on nothing else.
Anthropic has a framework that anticipates this. Its Responsible Scaling Policy, introduced in 2023, defines AI Safety Levels โ ASL-1 through ASL-4 and beyond โ and explicitly requires external evaluation at ASL-3 and above. Opening access to METR is not a spontaneous gesture; it is the execution of a standing policy commitment. METR has previously conducted pre-deployment evaluations of Anthropic's Claude models, including sabotage-risk assessments. So the accurate reading is not "first contact" but "deepened and expanded cooperation." The distinction matters because it changes the inference: this is not a lab discovering a new idea, but a lab operationalizing a commitment it already made.
There is also a regulatory clock, and it is loud. The EU AI Act's obligations for general-purpose AI models took effect in August 2025, and they include model evaluation, systemic-risk mitigation, and transparency reporting. A voluntary disclosure made in proximity to a mandatory requirement is a predictable move: it lets a lab frame compliance as leadership rather than obligation, and it pre-empts the criticism that regulators forced the behavior.
Layered on top sits the competitive field. OpenAI and Google carry their own safety narratives. In a market where raw capability gaps between the leading labs are narrowing, "safety and trust" becomes a differentiated brand โ particularly for enterprise and regulated buyers in finance, health, and government, who need auditable assurance before signing procurement contracts. METR access is a verifiable credential in that sales motion, or at least it is designed to look like one.
This cooperation also sits inside a broader institutional movement. METR, Apollo Research, the UK AI Safety Institute, and the US AI Safety Institute are collectively evolving from research organizations into quasi-certification bodies. They publish methodologies, they run red-team exercises, they advise governments. The direction of travel is clear: independent evaluation is becoming a recognized function, and the labs that engage with it early are buying both legitimacy and regulatory goodwill. Whether it functions as one depends entirely on details the announcement did not provide.
The Access Mechanism Is the Risk Surface
Here is the central technical question, and it is the same question I have spent years asking about on-chain systems: at what layer does verification actually occur, and what does the verifier get to see?
The announcement does not say whether METR received model weights, a constrained API, or a sandboxed environment. These are not interchangeable. They are different trust topologies with different failure modes, and conflating them is either ignorance or deliberate ambiguity.
If METR interacts with the model only through a rate-limited inference endpoint, then every evaluation is a black-box probe. You can test behavioral outputs. You cannot inspect the training process, the loss landscape, the data mixture, or any internal representation. You are testing a function's input-output behavior, not its construction. Most dangerous-capability elicitation โ the deliberate attempt to surface latent abilities โ can still be attempted this way, but the coverage is fundamentally partial. A model that behaves safely under API probing may contain internal structure that a weight-level auditor would flag immediately. Behavioral safety and structural safety are different properties, and only one of them is observable through an API.
Handing over weights is qualitatively different. It permits fine-tuning, gradient-based probing, activation analysis, and mechanistic interpretability work. It also multiplies risk: weights that leave a controlled environment can leak, and once leaked, cannot be recalled. This is the classic data-availability tradeoff, and it maps almost exactly onto a problem my field has litigated for years. A rollup that posts only state commitments to a base layer exposes far less surface than one that publishes full calldata โ but it also supports less independent reconstruction. I have argued for a long time that the data-availability narrative in rollups is oversold, because most rollups never generate enough data for the choice to matter. Here the same logic inverts into a real constraint: you cannot have maximal verifiability and minimal exposure simultaneously. The engineering question is where on that curve you choose to sit, and whether you are honest about the choice.
The sandbox is the middle path and, in my experience, the most common in practice. The lab provisions an isolated compute environment with some subset of capabilities โ inference, maybe limited fine-tuning, controlled logging โ and the evaluator works inside it. This is operationally sane. It is also the hardest configuration to audit from outside, because the boundaries of the sandbox define the boundaries of what can be found. If the environment cannot run a certain class of experiment, that class of risk is invisible by construction. This is the first unintended consequence: the access mechanism silently determines the risk surface, and a disclosure that omits the mechanism has omitted the only thing that would let an outside observer calibrate the evaluation's value.
The Verification Gap
Suppose the access is generous and the evaluation is thorough. The remaining question is the integrity of the result. How does anyone outside the two parties know that the published evaluation reflects the actual evaluation? In a purely institutional setting, you rely on reputation and contract law. Both are real, and both are weaker than they appear when the stakes are systemic.
Reputation is a lagging indicator. It tells you how a party behaved in the past, under conditions that may not recur. Contract law is a threat, not a proof; it deters some behavior and prices the rest. Neither mechanism lets a third party independently confirm that the artifact in front of them is the artifact that was produced. That is the verification gap, and it is the same gap that motivated the entire cryptographic apparatus of the blockchain field.
If an evaluation pipeline produces outputs, you can commit to those outputs: hash them, anchor the hash to a public ledger or a transparency log, and publish a proof that the committed artifact is the one generated under stated conditions. This is not exotic. It is the same primitive that underlies certificate transparency and, in a different form, the Merkle commitments that secure every rollup. You can go further with verifiable computation โ zero-knowledge proofs that a given inference or training step was executed faithfully, without revealing the weights or the inputs.
I built a proof-of-concept for exactly this in 2026, a minimal viable system for verifiable AI inference on-chain, and the experience taught me both the promise and the ceiling. The promise is that verification can be trust-minimized: you do not have to believe the lab, you can check the proof. The ceiling is cost and coverage. Proving a full frontier-model evaluation is currently impractical; proving a narrow, well-defined computation is tractable. The realistic near-term design is therefore a hybrid: cryptographic commitments to evaluation artifacts and their provenance, combined with institutional evaluation of the substance.
A concrete design would look like this. The lab and the evaluator agree on a manifest: the model identifier, the evaluation harness version, the test-suite hash, and the environment specification. Each artifact is hashed; the hashes are combined into a Merkle tree; the root is anchored to a public transparency log with a timestamp. Any subsequent claim about what was evaluated must reconcile against that root. If a party later disputes the record, the Merkle proof either reconstructs or it does not. This is not a complete solution โ it proves integrity, not honesty, and it says nothing about whether the chosen test suite was adequate โ but it converts an unverifiable claim into a falsifiable one. The difference between "we evaluated our model" and "here is a cryptographic commitment to what we evaluated, when, and with which harness" is the difference between a press release and an audit trail. The announcement contains none of it. There is no commitment scheme, no provenance record, no statement about who holds disclosure rights.
Who Holds the Disclosure Rights
That last point deserves its own section, because it is the crux of the entire disclosure.
Who decides whether METR's findings become public? Two possibilities exist. In the first, METR retains independent publication authority โ it can release negative conclusions on its own schedule and in its own words. In the second, the contract governs disclosure, and negative findings require the lab's consent before release. The first is transparency. The second is what I would call selective transparency, and it is not a distinction the word "transparency" survives.
A third possibility is the one that should worry anyone building in this space: a friendly-evaluator selection bias. If a lab chooses its evaluators, it can, consciously or not, favor institutions whose methodologies, risk models, or funding align with its own governance philosophy. This is not corruption. It is the ordinary gravitational pull of selection, and it produces systematic optimism. I have watched the same dynamic in smart contract audits: a project that selects its auditor optimizes for a clean report, not for a rigorous one. "Audit passed" and "system is safe" are different statements, and the distance between them is where the losses accumulate.
There is a sharper version of the concern. Does METR accept funding from the labs it evaluates? If so, the independence that justifies the entire exercise is compromised at the root. The announcement is silent, which is itself informative. An independence guarantee is cheap to state and valuable to state; its absence from a transparency-focused disclosure is a data point, not a null.
Security and Centralization Risks
My cybersecurity background forces a specific habit: every review gets a risks section, and this one earns it.
The first risk is nominal transparency. If the evaluation is confined to API access and the contract constrains disclosure, the entire apparatus collapses into a public-relations instrument with an institutional veneer. The probability is moderate; the impact is high, because it would teach the market to trust the wrong signal.
The second risk is compromised independence through funding. If METR's operating budget depends on the labs it evaluates, its negative findings become structurally unlikely regardless of anyone's intent. Independence is a property of governance and cash flow, not of mission statements.
The third risk is the window before mandatory regulation. Voluntary governance is a placeholder for enforcement, not a substitute. Between now and the full enforcement of the EU AI Act and any national analogues, labs can accumulate favorable framing that later complicates stricter rules. The window is not empty; it is where narratives harden.
The fourth risk is metadata centralization, a pattern I first dissected in NFT collections in 2021, where a Merkle root held on a single server quietly centralized an allegedly decentralized asset. The AI analogue is the evaluation record itself: if the only copy of what was evaluated lives in one lab's internal systems, the "transparency" is a pointer to a server you do not control. Verifiability without durable, independent custody of the evidence is a promise dressed as a proof. And there is a fifth, subtler risk: evaluation arbitrage. If methodologies are not standardized, labs will gravitate toward the evaluator whose criteria are loosest, and the certification will mean whatever the easiest path to it allows.
The Prediction Market Is Noise
I want to address the "46.5% YES" figure directly, because its inclusion is itself diagnostic.
The number sits near a coin flip, which tells you the market has no consensus on whichever question it references โ presumably which lab holds the best model by some future date. That is a reasonable state of belief given the current three-way scramble between Anthropic, OpenAI, and Google, with xAI and Meta chasing. But the figure arrives with no subject, no platform, no resolution criteria, and no liquidity disclosure. A prediction-market price without those four attributes is not information; it is decoration.
Its presence in a safety story is a tell about the publication. It signals that the audience is understood to care more about odds and speculation than about evaluation methodology. That is a content-factory pattern, and the item's total absence of bylines, interviews, or background reinforces the suspicion that it was assembled by an automated aggregator rather than reported. I have no way to confirm authorship, but the structural fingerprints โ two unrelated data points stitched together, no sourcing, no timestamps โ are consistent with machine aggregation. Which is a small irony: an article about AI transparency, possibly generated by an AI, disclosing almost nothing.
Contrarian: Promises Priced as Proofs
The conventional framing of this story is optimistic, and I want to push against it directly. The standard narrative runs like this: a frontier lab voluntarily opened itself to external scrutiny; this is good governance; other labs should follow.
I do not dispute the direction. I dispute the inference that voluntary disclosure is a substitute for structural verification. Voluntary governance is a promise; verifiable governance is a proof. The industry keeps celebrating promises as if they were proofs, and the gap between them is where the second unintended consequence lives.
The mechanism is straightforward. When an industry normalizes voluntary, self-selected, self-described evaluation as the gold standard, it creates a market for the appearance of safety rather than its substance. Labs compete on the legibility of their safety posture โ the press releases, the partnerships, the frameworks โ because those are cheap to produce and easy to market. The expensive part, the part that actually constrains behavior, is the mechanism: who can see what, who can publish what, and who can independently check it. Voluntary regimes, left to themselves, drift toward the cheap part, because the cheap part is what audiences reward.
I have watched this exact dynamic in DeFi. Liquidity mining programs subsidized the metric โ total value locked โ rather than the substance behind it, and the metric evaporated the moment the incentives stopped. Substitute "safety" for "TVL" and the behavior is identical. A safety narrative subsidized by public-relations pressure rather than constrained by independent verification is a number waiting to be gamed, and it will be gamed by the lab that is best at presentation rather than the lab that is safest in fact.
The uncomfortable corollary is that the crypto field, which spent a decade building the tooling to close this gap, is largely absent from the conversation. Verifiable computation, commitment schemes, transparency logs, independent attestation โ these are solved or partially solved problems in our domain. They are not yet applied to AI evaluation, and the labs have little incentive to apply them while voluntary gestures are rewarded at the same price as verified ones.
What to Watch
The question to carry forward is not whether Anthropic's cooperation with METR is sincere. It probably is. The question is what happens when the next lab makes a similar announcement, and the one after that, and the disclosures stay exactly this thin โ generous in framing, empty in mechanism. Watch for the artifacts that would falsify my skepticism: an independently published METR report, a stated access mechanism, a documented commitment scheme, a disclosure-rights clause, and a funding-independence statement. If those appear, the narrative was real, and the industry will have earned a genuine step forward. If they do not, we will have learned something colder and more useful โ that "transparency," like everything else in this industry, is only as strong as the layer at which you can verify it.