Here is a playlist of short videos from a guy called Martyn Tinsley on subjects such as sample size, optimisation, backtesting and the use of power analysis.
I hope it is useful
https://youtu.be/wqhnFX_HZk4?si=345v3WvJ-cAs6vcJ
Sample size, optimisation, backtesting and the use of power analysis
I am not the biggest fan of backtesting. The more parameters added to a system, the more likely of curve fitting the data to the results. The playlist of videos above shows this all too well.
However, say you do no backtesting but have an idea. Using very small stakes in order to record real results in the market, it would seem looking at the info from the video playlist above, that a decent amount of sample size is required to see if your idea truly has an edge or if it is just a random walk.
Does anyone have any further tips/advice on this area of testing?
Cheers
However, say you do no backtesting but have an idea. Using very small stakes in order to record real results in the market, it would seem looking at the info from the video playlist above, that a decent amount of sample size is required to see if your idea truly has an edge or if it is just a random walk.
Does anyone have any further tips/advice on this area of testing?
Cheers
-
ShaunWhite
- Posts: 10495
- Joined: Sat Sep 03, 2016 3:42 am
I backtest, but I prefer to divide that term into analysis (with static data probably in Excel), and simulation (with software and a saved price stream). Most people would have to settle for just the first but that's still better than nothing. The most important thing above anything else is to periodically randomly divide your data into an analysis set and a test set. That gives you a previously unseen set of data to test against and ensures you're not backfitting. People often thing the key to avoiding backfitting is just to use a vast sample but that's not the case at all.Frontier wrote: ↑Thu Sep 21, 2023 5:31 pmI am not the biggest fan of backtesting. The more parameters added to a system, the more likely of curve fitting the data to the results. The playlist of videos above shows this all too well.
However, say you do no backtesting but have an idea. Using very small stakes in order to record real results in the market, it would seem looking at the info from the video playlist above, that a decent amount of sample size is required to see if your idea truly has an edge or if it is just a random walk.
Does anyone have any further tips/advice on this area of testing?
Cheers
Testing something with live money doesn't really cut it for me because you can't re-run the same period. Markets change even if you don't change anything about your strategy, so you'll never tell exactly what effect your change had or what was changing anyway. It's maybe ok for something hard to analyse such as a scalping strategy but for the simple one or two bet stuff I'd say keep your money in your pocket and do the analysis. And only then do the implementation with cash and adjusting for the inevitable slippage as you transition from the clean world of your data to the messy real world.
I only skipped through the first video but it looks useful for anyone getting into it.
-
ShaunWhite
- Posts: 10495
- Joined: Sat Sep 03, 2016 3:42 am
"The more parameters added to a system"
Yes too many parameters are a killer, you won't have clue exactly which has an effect on what and the number of permutations quickly gets crazy.
I saw the name for the branch of mathematics that deals with combinations and permutations the other day, it's a good one ....
Combinatorics is a stream of mathematics that concerns the study of finite discrete structures. It deals with the study of permutations and combinations, enumerations of the sets of elements.
Yes too many parameters are a killer, you won't have clue exactly which has an effect on what and the number of permutations quickly gets crazy.
I saw the name for the branch of mathematics that deals with combinations and permutations the other day, it's a good one
....Combinatorics is a stream of mathematics that concerns the study of finite discrete structures. It deals with the study of permutations and combinations, enumerations of the sets of elements.
Yes, I see in the videos that walk forward analysis using untested data is important to see if there is any edge. I would recommend watching them.
I see in another post that you posted too, that you stream datadirectly from an api on betfair and store the data on a database, taking time to learn this over many months is beneficial. Are there any tutorials on how to do this? What software is required etc? I have used backtesting software before (MT4 strategy tester), but I have no experience with api's and databases.
I've just had a brush with combinatorics, problem of selection. Which type of buscuit to have with my cuppa
I see in another post that you posted too, that you stream datadirectly from an api on betfair and store the data on a database, taking time to learn this over many months is beneficial. Are there any tutorials on how to do this? What software is required etc? I have used backtesting software before (MT4 strategy tester), but I have no experience with api's and databases.
I've just had a brush with combinatorics, problem of selection. Which type of buscuit to have with my cuppa
-
ShaunWhite
- Posts: 10495
- Joined: Sat Sep 03, 2016 3:42 am
I wrote my own using C#, SQL as the database and the libraries Betfair provide. Took me a while because it was the first part of a whole trading system.
https://developer.betfair.com/en/exchange-api/
https://docs.developer.betfair.com/
But there's open source you can use if you know Python with recording, simulation and trading facilities.
https://github.com/betcode-org/flumine
ShaunWhite wrote: ↑Thu Sep 21, 2023 9:38 pmI wrote my own using C#, SQL as the database and the libraries Betfair provide. Took me a while because it was the first part of a whole trading system.
https://developer.betfair.com/en/exchange-api/
https://docs.developer.betfair.com/
But there's open source you can use if you know Python with recording, simulation and trading facilities.
https://github.com/betcode-org/flumine
Thank you for the info.