BTC strategy research · 235 strategies · nine years

Selection Rules on Trial

Six ways to pick trading strategies, each given the same pool and the same five and a half years of evidence — then graded on three years of data none of them were allowed to see.

Result — the proposed rule failed

The idea under test was “prefer strategies whose second half beats their first”, on the theory that it would find things not living off the old bull market. On the holdout it scored −0.55 against plain in-sample Sharpe’s +1.03, and it is significantly anti-predictive (rank correlation −0.20, t = −3.1). It does not select strategies that improved. It selects strategies that were terrible and regressed toward the mean.

How the test was built

The trap in this question is circularity: you cannot screen on “the second half is stronger” and then grade on the second half. So the nine years are cut in three. Every rule sees the first two windows. Every rule is graded on the third, which no rule touches.

P1 · selection inputAug 2017 — Dec 2020 · 1,233 days
P2 · selection inputJan 2021 — Jun 2023 · 911 days
P3 · holdoutJul 2023 — Sep 2026 · 1,168 days
The pool is all 235 scored strategies, placebos and deliberate inverse controls included. Excluding the known failures first would itself have been a selection step.

Each rule’s portfolio, 2017 to 2026

Equity of an equal-risk blend of each rule’s ten picks, on a log scale. The shaded band is the holdout. Every curve looks good before it — that is what being chosen on that data buys you. Only the part inside the band is evidence.

positive on the holdout negative on the holdout the other five rules, for context holdout period

What survived the wall

The same six rules, in-sample Sharpe on the left and holdout Sharpe on the right. The slope is the decay. Everything falls — selection on any statistic flatters itself — but the ranking before the wall is not the ranking after it.

Note that “no selection at all” and a random ten sit at the bottom in both windows. On a raw pool containing junk, selecting is clearly better than not selecting — the failure is specific to which statistic you select on.

The numbers

Selection ruleP1P2 P3 holdoutFullCAGRMax DD

The seven disciplines, separately

The live book’s component sleeves over the same nine years, against buy-and-hold. These are not selection rules — they are the strategies themselves, shown so the blend can be read against its parts.

SleeveP1P2 P3 holdoutFullCAGRMax DD

Why both findings are true at once

An earlier version of this work, run on just the seven sleeves above, concluded that selection was anti-predictive in general — holding all seven beat picking the best three at four consecutive cutoffs. On 235 strategies with a clean holdout, that generalisation is wrong: in-sample Sharpe predicts the holdout at +0.43, t = +7.2.

Both results hold, and the reconciliation is the actual finding. Selection’s value lies in removing things that do not work, and it is exhausted once they are removed. Across a raw pool containing placebos and inverse controls, a Sharpe filter is genuinely strong. Inside a pre-screened elite, the surviving rank differences are mostly noise, and re-selecting on them chases it.

Which makes two earlier decisions right for reasons that were half wrong: screening 407 strategies down to seven was sound, and keeping all seven in the live book rather than dropping the weak-looking ones was sound. A third round of selection inside the seven is the thing that would have failed.