Why backtested strategies fail live: the 50,000-run test

Backtested trading strategies fail live most often because the backtest edge was noise: a lucky run that looks like a pattern and then fades. Finforge's Sophon-3 paper, Version 1.0 published July 2026, answers that with a mechanical reality check in Appendix B. It reruns every agent-versus-variant comparison 50,000 times through a moving-block bootstrap and reports the span a reader needs before calling a gap real.

An active return is portfolio return minus the return of the benchmark the agent trades. For each agent-variant pair, the paper lines up the daily difference of active returns on the pair's common dates, a sample of roughly 855 to 870 days. The question it asks is narrow: does the gap between an agent and a changed copy of itself sit above random noise, or on top of it?

Why do backtested strategies fail live?

A backtest has one job and one weakness. It returns a number, but a number from a single path through history can be stubborn luck. Markets do not move in clean, independent steps; today's return leans on yesterday's, so a naive test that treats each day as its own draw will call noise a signal. That misread is a big reason strategies that shine in the lab go quiet with real money.

Appendix B uses a moving-block bootstrap. It breaks the 855 to 870 daily differences into overlapping blocks of 21 trading days, about one month, which spans the ensemble's weighting cadence and the typical return autocorrelation. It draws blocks at random with replacement, stitches them into a series the same length as the original, and records the annualized mean, 252 times the average difference, scaled as a percentage.

That one resampling is a single guess. The procedure repeats it 50,000 times, building a distribution of what the annualized gap could plausibly be. The paper reports the 2.5th and 97.5th percentiles of that distribution, the 95% interval that brackets the guess. If zero sits inside it, the difference is not verified. If zero sits outside, the gap is real enough to print.

Why the pooled estimate is more honest than three separate runs

Finforge runs three agents, and it does not treat them as independent witnesses. For the pooled estimate it averages the three agents' daily differences date-wise first, then applies the identical bootstrap to the averaged series. Averaging before resampling preserves the cross-agent correlation inside each block, so the pooled interval reflects an effective sample of roughly 1.9 rather than a fiction of three separate trials. The paper also reports a t-statistic, the mean difference divided by the standard error, on the same series.

That discipline is why these backtests carry weight instead of dying live. A gap whose 95% interval swallows zero is not published as a win. Printing the interval next to the number is how Finforge lets a skeptic falsify the claim rather than take the chart on faith. Most strategies die leaving the lab, either from rules fitted to old data or from noise mistaken for edge; a bootstrap puts the noise where a reader can see it.

Questions people ask

What is a moving-block bootstrap? It is a way of generating many fake versions of a real history by cutting the history into blocks and resampling them at random, so the resampled series keeps the same correlation structure as the original. Repeating it 50,000 times shows how wide the measured gap could swing just from chance.

Why not just run the backtest twice? Two runs are still noise; luck can repeat. A bootstrap builds thousands of alternative histories and reports the range, so a single flattering stretch cannot masquerade as proof. It is the difference between one anecdote and a distribution.

Who produced these numbers?

Finforge Research wrote the Sophon-3 paper and runs its trading agents in public, wins and losses included. Four Sophon agents have traded since 21 April 2026 on founder capital in Alpaca paper accounts, so no customer money is traded before launch. You can check the live results, benchmarks and drawdowns for every Sophon agent, and the Sophon-3 research summary describes the bootstrap and the ablation suite in full. These statistics describe past simulated comparisons and are not a promise of future performance; past performance is not a guide to future returns.

Read the paper. Then check the agents.