There is an uncomfortable moment in every quantitative trading project: you discover your backtest does not measure what you think it measures.
Ours arrived like this.
The two numbers
Same configuration. Same historical data. Same time window. The only thing we changed was the simulator fill model in MetaTrader:
| Model | Result | Factor |
|---|---|---|
| Open prices M1 | +34,496 | 1.47 |
| Every tick | −93,797 | 0.38 |
That is not a difference in magnitude. It is a difference in sign. One model says the system wins 47 % more than it loses; the other says it loses almost three times what it wins.
Why it happens
A backtest does not simulate the market. It simulates your model of how orders get filled.
The open-prices model assumes only four things happen inside each candle: open, high, low, close, in an order the simulator invents. The every-tick model reconstructs the intra-candle path in far more detail.
If your strategy is indifferent to what happens inside the candle, both models agree. If your strategy depends on the intra-candle path, they diverge — and the more it depends, the more they diverge.
Ours depends. The server-side trailing target arms when price reaches a level and then follows the peak, cutting only on a retrace. That behaviour is pure intra-candle path. With open-price candles the simulator invents a benevolent path. With every tick, it has to face the real jerks.
It is telling that the piece carrying most of the modelled result is exactly the one that does not survive the stricter model: without the trailing target, net falls from 40,675 to 9,389. In other words, the bulk of the modelled gain is concentrated precisely where the model risk lives.
What to do about it
First, the answer is not "always use every tick". Every tick is also a model, with its own assumptions about spread, slippage and latency. Switching models does not give you truth, it gives you a different approximation.
What we do:
1. Always run both models. If they agree on sign, you have something. If they do not, the backtest is not evidence — it is a hypothesis.
2. Treat the divergence as the data. It is not noise to average away. It is telling you exactly which part of your strategy depends on assumptions you cannot verify.
3. Measure real execution. No simulator gives you your broker's slippage at your latency. We log requested price against filled price on every order. First live session, 62 gold fills: average slippage of +$0.0274 per order, leaving 92 % of the modelled expectancy. It is one session, on demo, where fills are optimistic — but it is the only number that does not come from a simulation.
4. Do not publish the pretty number alone. When the sign depends on the simulator, showing only the positive one is not optimism. It is omitting half the experiment.
The boring conclusion
A backtest whose sign depends on the simulator cannot support a deployment decision. However good the good number is.
A 65-configuration sweep plus out-of-sample validation gave us none meeting a factor above 1.2 with drawdown below 3 %. That is also a result, and we publish that too.
From building Cerberus at Tuurt Labs. High-risk instrument: trading leveraged CFDs can cost you your entire capital. This is not investment advice.