The backtest that lies, and the rules that stop it

Most inflated backtests I have seen came from one of three mistakes, and each one produces numbers that look technically respectable.

• Look-ahead inside the bar. The simulator enters and exits on information from the same bar, in whatever order suits it. That is time travel.
• Ghost quotes. The strategy trades the quote it saw at time t. By the time a real order reaches the book, that quote is often gone.
• Fantasy liquidity. A visible price is treated as an available fill, at any size.

I learned all three on Polymarket, first in 15-minute Up/Down markets and then in second-scale HFT research. None of the fixes is specific to prediction markets. They apply to any backtest where one bar of hindsight or a few hundred milliseconds can change the answer.

A backtest that fills at the quote it sees at time t is trading a ghost. A real order needs reaction time to compute, decide and send, plus network time of 50–200 ms at minimum, often more. By the time it reaches the book, that ask may be gone.

The clearest case came from the HFT pipeline. Buying at the ask at t and selling at the bid five seconds later, with no reaction latency, the hit rate looked like ~98%. Acting at t + 300 ms instead, it dropped to around 55%, which is much more realistic.

I left the rule as a comment in the backtest code: "Quant audit: use ≥ 300 ms so you don't inflate hit rate with ghost quotes (98% → ~55% real)." Reaction latency (reaction_latency_ms) belongs in version one of a backtest, not in the apology after live trading disappoints.

Add slippage, a minimum visible size (min_ask_size), queue effects or maker-fill uncertainty, and a strategy that looked clean can collapse very quickly. So every short-horizon backtest now gets 1–2% slippage and a liquidity filter from the start.

The label has to respect the same reality. In the HFT pipeline a sample only counts as a win if up_bid[t + h] > up_ask[t] + fee: buy at the ask, sell at the bid, pay the fee. Never at the mid.

Even that still overstates PnL. A bid on screen is not a fill: the market maker can cancel it, and confirmation can take 2–5 seconds.

A second class of lies comes from data alignment, and it matters as much as fills.

In an asynchronous system, two feeds almost never update in the same millisecond. The honest join is often last known value on local receipt time: each row sees only what my machine had actually received, and feature windows follow the same clock instead of a future-friendly exchange timestamp.

For the HFT work I recorded my own data rather than relying only on downloaded history. A 24-hour recorder stored local receipt time (local_receipt_ts_ms, L) and the exchange event time (E) when available, so network_latency_ms = L − E became a column. Rows above a 200–500 ms latency threshold were dropped so the model did not train on lag spikes, freshness columns made the remaining staleness explicit, and an explicit leakage check ran on top.

A model can be mathematically correct and still be temporally dishonest.

The practical rule: use the backtest to disqualify ideas aggressively. If the numbers survive realistic pain and realistic timestamps, the strategy has earned more attention. If not, the simulator already did its job.

That rule kept the 15-minute system grounded. Once fills were sequenced and latency, fees and slippage were charged together, expected value stayed near zero after costs, and I stopped the strategy on evidence rather than hope. It also made the HFT work much harder to fake: good classifier metrics still had to survive the step to executable PnL, and the HFT write-up shows where they did not.

Honest simulation is half of the playbook. The other half is operational, keeping a live bot in sync with the exchange: why a bot is infrastructure, not edge.