Does the overfitting statistic have teeth?

A fair skeptic, handed any overfitting test, should ask the only question that matters: does it actually work? A gate that passes everything is a rubber stamp. A gate that fails everything is noise. The Probability of Backtest Overfitting is only worth reporting if it can separate a genuine edge from a lucky one, and unlike most claims in this field, that is something you can check directly, because you can build worlds where you already know the answer.

01What PBO measures

The Probability of Backtest Overfitting (Bailey, Borwein, López de Prado & Zhu, 2014) asks a precise question: if you pick the configuration that looked best in-sample, how often does it turn out to be below-median out-of-sample? A configuration that only ever wins on the slice of history you happened to fit is not a strategy; it is a coincidence with good marketing. PBO puts a number on how likely that coincidence is.

The machinery is combinatorially-symmetric cross-validation (CSCV): chop the track into S blocks, and for every way of splitting those blocks into a training half and a testing half, find the in-sample winner and record its rank out-of-sample. With S = 16 that is C(16, 8) = 12,870 distinct splits — evaluated exactly, no sampling. PBO is the fraction of splits where the in-sample winner lands in the bottom half out-of-sample. Low is skill; high is overfitting.

02The test: two worlds where we know the truth

To prove the statistic has teeth, we run it on two synthetic searches constructed so the answer is not in question:

Both numbers are computed the same way, over all 12,870 splits, with a fixed seed so the result is identical on every run — download the script and run it yourself, offline, stdlib Python. The pure-noise search lands at a coin flip (≈0.5) — no genuine edge, exactly what luck scores. The genuine-skill search collapses to near zero, decisively cleared. The statistic neither blesses everything nor condemns everything; it does exactly the job it claims to.

A test you can only trust when you don't know the answer is not a test. This one you can check where you do.

03Where a real credential sits

Now the number that matters. Our flagship BTC trend book carries a real, composed-PBO credential in the live registry, a proof in which each per-trial STARK is verified in-circuit across S=16 blocks:

04Why we compute it in a proof

Everything above could be run by anyone with the papers and a weekend. The difference MIZAN makes is not the CSCV itself; it is that in an ordinary workflow, the person reporting PBO is the person it judges, and they choose the strategy pool it runs over. Inside a MIZAN proof, the pool and the committed positions are fixed before the statistic is computed, the computation happens in-circuit on Merkle-pinned prices, and the result is bound to a credential that reveals nothing about the strategy. The overfitting test stops being a courtesy the manager extends and becomes a property an allocator can check.

But within its remit, the statistic works, and it works demonstrably, not because we assert it, but because you can run it on a world whose answer you already know and watch it get that answer right.