I think I found an Edge and I can’t break it

[Gcampb]

Not been here for a long time! But had a stint automating a greyhound trading strategy, which eventually broke even and put it to bed for a while.

Recently I was able to develop a full 10 year dataset of UK & IE horse racing data, which I combined with Betfair historical data. My initial plan was to use it for a back to lay strategy, however I went deep into ML and started building and testing models…I was about 5 models in before my initial tests started to show some promise.

The model basically scores pre race data, adds relative features and makes selections, a second model acts as a gate and decides the picks. Not going to go into details.

I validated with a full walk forward fold from 2017-2024. I then introduced the BSP price after the walkthrough and found that there was a positive ROI, low drawdown and high enough frequency of picks.

I’ve done multiple break tests, checked for leakage.. and I can’t find anything wrong

Overall ROI across 9 years is 13.6%. Ranges from 7.8% to 19%(Covid year)…

I’m now at the point where im starting to test on new races, tiny stakes, logging everything. I expect to have 150 or so bets a month.. so perhaps in 6 months I’ll have enough data to verify the model truly works..

But there is a massive temptation to scale sooner. I’m scratching my head trying to find out what I’ve done wrong because I can’t quite believe I’ve found an edge.

Does anyone have advice? Would like to hear from anyone here who is building models

[MemphisFlash]

Believe me whatever "edge" you think you may have found, you haven't.
Why are you concentrating on past events, there gone, dead and buried,
along with most of the dogs in your database. Concentrate on what is
happening before your very eyes and you will be better off.

[Gcampb]

Ha fair enough, I know where you are coming from, but I will extrapolate.

My intention was to build a model for finding back to lay candidates. It’s an easy strategy, especially on US markets and I’ve been consistent enough there at certain tracks with minimal data.. saying this because I understand what you’re getting at, trading vs handicapping and just like trading any market, we can’t always predict the future based on history.. however,,,

I was building models to try and find horses more likely to have a price collapse in play. That’s not a big leap, if you can identify front runners, pre off at a higher price, you capture a bigger collapse..that was the aim.,

I’ve literally spent 5 hours now tearing the model apart, removing features, shuffle tests, time leakage tests, look ahead leakage tests…and the only conclusion left is that the edge is a structural ranking edge,..

Again, I’ve reran walk forwards over 9 years, train on 2 years test on following year etc and it’s held up again and again, consistently. It’s a predictive edge that’s situationally

Of course. That’s no use to anyone if the markets already account for this.. But I’ve ran it against the BSP and the market hasn’t accounted for it. Certain race structures have been persistently mis-ranked by the market, and that mis-ranking has not been arbitraged away - it’s persisted over the 9 years I’ve tested against.

[Euler]

Looks like you’ve put a lot of effort into this and fair play for doing proper walk-forward testing rather than just curve fitting a backtest.

One thing I’d say though, having gone down similar rabbit holes in the past, is that markets tend to punish long-term predictive models more than people expect.

The big difference for me has always been that backtests are measuring historical correctness, whereas trading profitability comes from how prices move in real time. A model can be “right” a lot and still struggle once you factor in entry timing, queue position, partial matches, late money and so on.

A 13.6% ROI sounds great on paper, but in live conditions you’ll almost certainly see some erosion from slippage and execution, especially if you’re betting close to the off. Your approach of running it live at small stakes first is exactly the right thing to do.

Personally I’ve had far more success focusing on what the market is doing now rather than what similar races did historically. Price behaviour, volume changes and market pressure tend to be more reliable than long-term race attributes once real money is involved.

Definitely keep testing it live and stay sceptical. If it holds up under real conditions, that’s when it gets interesting, but the exchange is very good at humbling even well-researched models.

[ShaunWhite]

People saying data is just the past or finding that backtest success then fails just aren't doing it right. The 'secret' is to always seperate training and validation data, and to remember that excel or other static analysis are a waste of time. Unless you can simulate execution accurately then you're not testing what's going to happen. As Peter said queue position, latency, and fill rates are what needs to be accurately simulated. You then of course add a degree of slippage (exec failures) iro 5% would be a start. And then you're good to go.

It's the basic proven approach used by the quant community. Tick data, an accurate simulator, rigorous seperation of data, the right modelling software selection and analysis of your live results for correlation to your model. Thankfully all free these days.

What modelling software are you using? How close is your sim to a live trial?

[jamesedwards]

Plus you're looking at data where you weren't part of the market. Becoming part of the market will push the needle against you - how much will depend on your stake and market fragility.

Commencing live trades at very low stakes will be the make or break. Interested to see how it goes, please keep the thread updated.

Good luck!