What it scored, honestly.

An evaluation score is a claim of the form: a model, applied to a test set under a scoring rule, obtained this number. Today that claim is checked by trust. The evaluator chooses the test set, may have trained on it, may have run the benchmark many times and reported the best run, may have changed the model between the run and the release, and may have changed the scoring rule. None of this is visible in the number.

Backtest overfitting and benchmark contamination are the same crime. A trading record mined across thousands of strategies and an evaluation score mined across thousands of checkpoints, prompts and seeds both report a maximum and call it a mean. VTR-1 closes this for trading records with four commitments: the procedure, the data, the cost rule and the number of trials, with the trial count bound in-circuit rather than reported. This profile applies the same four commitments to evaluation.

A conforming evaluation credential proves, without revealing the model: that a committed model was scored on a committed test set under a committed scoring rule, that the test set was committed before the model could have seen it, that the number of runs across checkpoints, prompts, seeds and configurations is bound, and that the reported score is the one the committed procedure produced. Every requirement carries a stable identifier, E-01 onward.

The key words MUST, MUST NOT, SHALL, SHOULD and MAY are used as in VTR-1.

This profile governs credentials whose statistic is a score of a model or agent on an evaluation set: accuracy, exact match, pass@k, win rate, calibration, Brier score, task success, cost-normalised success, or any scoring rule that is a deterministic function of model outputs and committed labels.

It governs the claim, not the model. It does not certify capability in general, safety, alignment, intent, or performance on data other than the committed set. A conforming credential proves that a score is honest and belongs to a specific, committed model. Nothing more.

Out of scope: scores that depend on a human rater's judgement unless the ratings are themselves committed before the model outputs exist; scores on sets the evaluator cannot commit publicly in hashed form; and any construction that proves the absence of training-set leakage, which no current construction does (see §07).

Model commitment. H(model_artifact ‖ model_salt): the complete inference procedure (weights or API identity and version, system prompt, tool configuration, decoding parameters, scaffolding) and 32 secret random bytes. Identical to the VTR-1 sealed-path commitment.

Test set commitment. A Merkle root over the test items, each leaf H(input ‖ label ‖ item_salt), published with a timestamp anchor before any scored run. Items MAY stay private; the root is public.

Scoring rule commitment. The hash of the deterministic scoring function and its parameters (normalisation, tolerance, k, judge model identity if a model judge is used).

Run. One execution of the committed model over the committed test set under the committed scoring rule with a committed seed and configuration.

Run ledger. A Merkle commitment to every run the evaluator executed on this test set with any model version, in order. Its leaf count is N_runs. Each leaf commits the model commitment, seed, configuration and the score obtained.

Reported score. The score the credential attests. It MUST be the score of a run named by index in the run ledger.

Selection-adjusted score. The reported score corrected for the number of runs, by the method in §05.

Holdout class. The relationship between the test set and the model's possible training data, declared and, where possible, proven (§04).

• E-01 (MUST). A credential binds the reported score to a model commitment, a test set commitment and a scoring rule commitment. A change to any byte of the model, set or rule is a new claim.
• E-02 (MUST). The scoring rule MUST be deterministic given model outputs and labels. A model-as-judge rule is permitted only when the judge is itself committed (identity, version, prompt, decoding) and its verdicts are included in the proven computation.
• E-03 (MUST). The proof MUST compute the score over every item of the committed set. Subsetting, filtering or weighting MUST be part of the committed rule, never applied afterward.
• E-04 (MUST). The credential MUST state the class. A surface that displays the score MUST display the class beside it.

• E-05 (MUST). The test set root MUST be anchored to a public timestamp authority and, for E3 and above, to a blockchain, before any scored run.
• E-06 (MUST). The credential MUST commit the model's declared training-data cutoff and the test set's creation date. E3 requires the cutoff to precede creation.
• E-07 (MUST). A verifier MUST present the holdout class as declared where it rests on the evaluator's statement and proven where it rests on anchors. Anchoring proves the set existed before the run; it does not prove the model never saw similar material. This profile does not claim to prove the absence of contamination (§07).
• E-08 (SHOULD). Test sets SHOULD include canary items: synthetic items created after the model's cutoff whose answers cannot be inferred from public data, so that an abnormal score on canaries is evidence of leakage. Canary results MUST be reported separately when present.
• E-09 (MUST). An evaluator that publishes the test items after scoring MUST publish them with the item salts so the root re-derives; a set published this way is retired for new E3 credentials from the publication date.

• E-10 (MUST). Every run on a committed test set MUST be appended to the run ledger before its score is known to the evaluator, by committing seed and configuration first and the score second.
• E-11 (MUST). The ledger is append-only. A verifier MUST reject a credential whose ledger root is not a descendant of the previously published root for the same evaluator and set.
• E-12 (MUST). N_runs is the leaf count and is bound in-circuit, not reported.
• E-13 (MUST). The credential MUST carry a selection-adjusted score: the expected maximum of N_runs draws under the null that runs are exchangeable, subtracted from the reported score, with the method and its assumptions committed. This is the evaluation analogue of the deflated Sharpe ratio. The adjustment MUST be shown beside the raw score; it MUST NOT replace it.
• E-14 (MUST). The reported score MUST be the score of a specific ledger leaf. A credential MUST NOT report an aggregate of several runs unless the aggregation is part of the committed rule.
• E-15 (SHOULD). Evaluators SHOULD pre-register the leaf index they will report before the run executes.

A party about to rely on a score (a buyer, a deployer, a leaderboard, an allocating agent) performs, before relying:

• E-16 (MUST). Retrieve the credential by digest and verify its hash against the issuer registry or an anchor.
• E-17 (MUST). Verify the proof against the named era and obtain the score, the class and N_runs exactly as issued.
• E-18 (MUST). Check the test set root against the set the relying party believes was used, and the scoring rule hash against the rule it expects.
• E-19 (MUST). Read the holdout class, the cutoff and creation dates, the canary result if present, and the selection-adjusted score.
• E-20 (MUST NOT). Treat any of the following as passing: a score without a ledger (below E2) when its policy requires E2 or above; a root mismatch; a rule mismatch; a selection-adjusted score below its threshold; a retired set.

The eval-check reads properties. Which class, adjustment or canary result is sufficient is the relying party's policy.

This profile does not certify, and a conforming credential MUST NOT be presented as certifying:

• general capability beyond the committed set;
• the absence of training-set contamination (E-07);
• safety, alignment, intent or legality;
• performance on inputs outside the committed distribution;
• that a model-as-judge is correct, only that it was committed and applied.

• E-21 (MUST). Every surface that displays a proven score MUST display, beside it, the class, N_runs and the holdout class.

• E-22 (MUST). An issuer MUST publish each evaluation credential as a verdict object with at least:

• E-23 (SHOULD). Verifiers SHOULD expose the eval-check as a single callable operation over at least one agent protocol (MCP tool, A2A skill, or HTTP), with the same shape as the VTR-1/A allocate-check.

Residual risks: paraphrased contamination that canaries do not catch; an evaluator running many test sets rather than many runs and reporting the friendliest set (the set-selection analogue; a future version may require a set ledger); collusion between evaluator and set author.

This profile adds E-01 to E-23. It reuses the sealed-path model commitment, the trial-ledger construction (as the run ledger), era law, the verdict transparency log, receipts and the check-before-reliance pattern unchanged. It removes nothing and weakens nothing in VTR-1. Governance follows VTR-1 §6.

1. Should E3 require a minimum gap between training cutoff and set creation?
2. Is a set ledger (every test set an evaluator has tried) feasible without revealing how many sets a lab holds?
3. Which selection-adjustment method should be normative at v1.0: expected maximum under exchangeability, or a bootstrap over the ledger's committed scores?
4. How should cost-normalised agent success commit to prices that change over time?