Latency and infrastructure2026Published8 min
Polymarket journey — part 3 of 3
HFT on Polymarket: Rust, timestamps and the 98% hit rate that wasn't
Second-scale trading research on Polymarket, built recorder-first: Rust on the hot path, Python for datasets and LightGBM. Good classifier metrics did not turn into profit, and a 98% backtest hit rate fell to about 55% with 300 ms of reaction latency.
- Role
- Research and systems engineer (independent)
- Stack
- Rust (PyO3), Python, Parquet, LightGBM
- Data
- Binance and Polymarket WebSocket feeds
- Scope
- Recorder, ETL, datasets, models, paper/live loop
- Rust
- PyO3
- Python
- LightGBM
- WebSockets
- Parquet
- Polars
- Adverse selection
- +1

- Question
- Could second-scale signals, such as Polymarket lagging Binance or order-book imbalance, hold an edge that survives execution?
- Approach
- Recorder-first: Rust for feeds and features, Python for ETL, prepared datasets and LightGBM, and every tick stamped with the time my machine received it.
- Outcome
- A 0.921-accuracy classifier with negative simulated PnL, and a 98% hit rate that fell to about 55% at 300 ms of latency. The branch never went live.
Context
The question narrowed to second-scale signals
Phase 1 hit a data bottleneck: for long-horizon prediction, the dataset had to be built before any model could be trained. Phase 2, a 15-minute trading system, found expected value near zero after fees. The next question was narrower: could second-scale signals still contain something useful?
I kept the 15-minute crypto Up/Down markets from phase 2, where UP and DOWN tokens trade between 0 and 1, and used Binance as the reference price. This is HFT in style rather than in the microsecond sense: most targets look 1 to 10 seconds ahead, and the network alone adds 50–200 ms or more.
Three hypotheses drove the work:
- Polymarket lags the reference market.
- Order-book imbalance and short lag features carry signal a model can use.
- The edge, if any, lives in data freshness and execution discipline rather than in prediction alone.
All three are about timing. The question stopped being "can I train a model?" and became "can I observe, align and execute honestly enough that the model means anything?" That made the project infrastructure-first.
Architecture
Rust on the hot path, Python around it
I split the stack by runtime constraint. The path from a market event to a feature vector went to Rust, exposed to Python as the poly_rust_core module through PyO3. Everything that benefits from fast iteration stayed in Python.
Rust: the hot path
- WebSocket clients for Binance and Polymarket
- Event-driven wait on the next Binance tick instead of polling
- A single source for the 40-feature HFT vector: spread, order-book imbalance, lag features
- Recorder and latency primitives
Python: everything around it
- Recorder orchestration and token refresh
- JSONL to Parquet conversion, including incremental ETL
- prepared_l1 and prepared_l2 datasets for train, val and test
- LightGBM training, experiment sweeps, visualization, paper and live wrappers
Data
The recorder mattered more than the model
The real improvement in this phase was the recorder and the ETL around it, not a model. Raw captures became prepared datasets before any model number was worth reading.
01
Record
Rust feeds, Python orchestration
- Binance and Polymarket WebSockets
- local_receipt_ts_ms on every tick
- Raw JSONL
02
Convert
Python
- JSONL to Parquet
- Incremental ETL
03
Prepare
prepared_l1 and prepared_l2
- Last-known-value alignment
- Freshness column per feed
- Latency filter, e.g. 200–500 ms
- Leakage checks
- Train, val and test splits
04
Train
LightGBM
- Several target families
- Experiment sweeps
Rust handles the feeds and the timing primitives; conversion, alignment, datasets and training are Python.
Timestamps
Two clocks: when it happened and when I knew
Each update has two times. Exchange event time (E) is when Binance or Polymarket says it happened. Local receipt time (L) is when my recorder got it. A strategy can only act on L, so the datasets were built on L, and the gap L − E was kept per row as network_latency_ms.
Two feeds almost never update in the same millisecond. Instead of pretending they share timestamps, alignment took the last known value of each feed, and freshness columns (binance_age_ms, poly_age_ms) turned that misalignment into explicit information. Rows could then be filtered by latency thresholds such as 200–500 ms, so training did not learn from lag spikes.
The rule is point-in-time: a feature may only use what had arrived when its row was built. Exchange timestamps are future-friendly, and a model built on them can be mathematically correct and still temporally dishonest.
Exchange clock (E)
My machine (L)
Binance tick
binance_ts_ms
Polymarket book update
up_ts_ms, dn_ts_ms
Binance tick received
local_receipt_ts_ms
Polymarket update received
local_receipt_ts_ms
Row built, decision at t
Order reaches the book
EBinance tick
binance_ts_ms
EPolymarket book update
up_ts_ms, dn_ts_ms
LBinance tick received
local_receipt_ts_ms
L − E = network_latency_ms
LPolymarket update received
local_receipt_ts_ms
L − E
LRow built, decision at t
binance_age_ms
poly_age_ms
EOrder reaches the book
reaction_latency_ms, ≥ 300 ms in backtests
- — Features use the last known value of each feed at t.
- — Rows above a latency threshold (e.g. 200–500 ms) are dropped from training.
Interactive
Inspect one recorded market
This is one real market from the recorder: slug sol_1771810200, a 15-minute SOL window exported from the L1 Parquet data behind prepared_l1, with 58,613 rows and 57 columns. Pick up to three series, compare raw values against Hampel, EWMA or rolling-median smoothing, and check how the two venues move together.
It is the surface I used to debug features before training. Try binance_ask against up_bid, or network_latency_ms against poly_age_ms. A full-width version lives on its own page.
Recorded L1 feature explorer: SOL, one 15-minute market
Open full screen
Plotly export of about 6 MB, loaded only on demand. The browser view embeds 12,000 of the 58,613 rows.
Models
Same features, different labels
On top of the prepared datasets, the HFT module supports several target families for LightGBM. The features stay the same; the label changes with the trading question.
- hft_1s, hft_5s, hft_10s
- Taker horizons. Will the UP bid after h seconds beat the ask paid now plus fees:
up_bid[t+h] > up_ask[t] + fee? Buy at the ask, sell at the bid, never at the mid. - maker_1s
- Maker protection. Will the bid collapse within a second? If so, cancel.
- adverse_bid_1s
- Adverse selection. Will the minimum bid over the next second drop?
- fair_up, expiry
- Fair-value and expiry-oriented variants.
Good classification, negative PnL
- Test accuracy
- 0.921
- Brier score
- 0.068
- Log loss
- 0.244
- Maker-style PnL
- < 0
- Simulated, same evaluation
phase1_adverse_bid_1s_lightgbm evaluation on the test split. Values rounded as reported; PnL comes from the evaluation, not from live trading.
This is the result I trust more than a flashy headline. The pipeline could produce a decent classifier and still refuse to confuse classification quality with executable edge. Knowing that the bid is about to drop is not the same as being filled at a price that pays; latency, queue position and actual fills decide that.
The same evaluation timed inference at tens of microseconds per batch. Next to 50–200 ms of network latency, the model was never the slow part of the loop.
Backtest
Latency turned a 98% hit rate into 55%
Short-horizon backtests are easy to flatter. Without reaction latency, the simulation bought the visible ask at time t, sold the bid at t + 5 s, and assumed it acted instantly. The hit rate came out at about 98%.
Real orders are not instant. The network adds 50–200 ms or more, before computing, deciding and sending. By the time an order reaches the book, the ask seen at t may be gone: the backtest was buying ghost quotes. With reaction_latency_ms set to at least 300, the simulation traded at the prices of t + 300 ms, and the hit rate fell to about 55%.
Hit rate
Same backtest; only the moment of execution changes. Both values are approximate (~98% and ~55%).| Category | Hit rate |
|---|---|
| No reaction latency | 98% |
| Reaction latency ≥ 300 ms | 55% |
Same backtest; only the moment of execution changes. Both values are approximate (~98% and ~55%).
Source: Quant audit comment in the backtest code: "98% → ~55% real".
Latency was only the first cut. Slippage and queue realism made the result more fragile, and the quant diagnostics exposed several ways a careless simulator fakes PnL.
The rule I settled on: before trusting a short-horizon model, run a stress backtest with 1 s of latency, 2% slippage and a minimum ask size of 500. If PnL collapses there, the model is not ready for live. The full checklist is in How a real backtest works.
Execution
Built for live trading, kept on paper
The HFT branch did not stop at offline models. real_trading_hft runs paper and live modes on the same architecture, and its hot loop never makes a direct API call: order work goes through a blocking action_queue to a secondary API worker.
01
Market feeds
WebSockets
- Binance ticks
- Polymarket order book
02
Hot loop
No direct API calls
- Build features
- Decide
- Same path in paper and live
03
action_queue
Blocking hand-off
- Place and cancel tasks
04
API worker
Secondary worker
- Talks to the exchange
API jitter stays in the worker instead of freezing the market loop.
Around the loop, four rails decided whether it could act at all:
- Pre-flight ping checks before live actions are enabled.
- A latency monitor that works as a circuit breaker.
- A price band: no trades outside roughly 0.10–0.90.
- A higher probability bar when the spread makes the trade expensive.
I never took the HFT branch live. By design, it had to survive the painful backtest first, then paper trading with the full order lifecycle, and only then a tightly controlled live test.
Takeaways
Timing decides whether a model means anything
This was the technically strongest phase of the Polymarket work, because it forced everything to become precise: timestamps, ETL, feature contracts, evaluation and execution. The full arc is in the case study.
01
Build features on receipt time, not event time
Align features on local receipt time and keep L − E as a column. Exchange time is future-friendly.
local_receipt_ts_ms on every tick; network_latency_ms per row
02
Classification quality is not edge
Accuracy, Brier score and log loss grade the label. Money depends on fills, queue position and latency.
adverse_bid_1s: 0.921 accuracy, negative maker-style PnL
03
Reaction latency belongs in version one of the backtest
Not in the apology after live trading disappoints. An edge that only exists at zero latency is not an edge.
~98% to ~55% hit rate at ≥ 300 ms
04
Keep the hot path small and never blocking
Rust for feeds and the feature path, Python for everything that changes often, API calls behind a queue.
poly_rust_core via PyO3; action_queue to an API worker
If the data path or the simulator is weak, the model is just decorating a timing artifact.
Related
HFT feature explorer: one recorded 15-minute SOL market
Fifteen minutes of Binance and Polymarket data from my HFT recorder, with feed freshness and derived features on one clock. Run the same kind of inspection that sat behind the HFT datasets.
Interactive appendix
3 min
The backtest that lies, and the rules that stop it
A condensed playbook for honest short-horizon backtests: execution order inside one bar, reaction latency, slippage, payout bugs, timestamp alignment and leakage audits. They are the rules that kept my Polymarket research honest.
Execution playbook · 1/2
7 min read
A trading bot is infrastructure, not an edge
Notes from a live 15-minute bot and an HFT stack on Polymarket. The signal is one component; most of the work is state, fills, reconciliation and keeping blocking calls off the hot loop.
Execution playbook · 2/2
7 min
Get in touch
Working on something similar? I am happy to compare notes.