Modeling data (why & how?)

[Brovashift]

Hi all,

I have heard Peter say on many occasions about modeling the data, but Im curious to know why we'd want to do it, and how to go about it.

Is modelling data a way to program a trading idea into a type of simulator where we can e.g. find out if it would have a positive or negative expectancy over 100... 200... 500 trades for example?

If this is the general premise, then how would I start to model data? I found an interesting source on Betfair that talks you through the initial steps and suggests "R" or "Python" languages to get started. I have some experience with C#, and even less with Java, but I thought my days of stressing over 1 line of code for 18 hours, only for the answer to pop into my head at a randon moment while doing the washing up or some other mundane task, were well behind me lol.
Is modelling data where we separate the men from the boys with regards to trading success? I will if I have too lol.

TIA

[sionascaig]

Have a look at the BA Tennis or Soccer traders... They are good examples of why you may want to develop a model...

Both are driven by data & the model allows you to test out various scenarios so you can go into trades with a better understanding of what may happen.

For example, I put one together for test cricket.. Basically input the actual players and the model picks up their most recent averages etc and runs a stochastic simulation of the game several 1000 times to give you and idea of what the starting prices should be (expected runs etc) ... Its reasonably accurate compared to the market (before the toss anyway)...

It can also allow you to get a better understanding of how sensitive the game is to specific players performance etc..

It can be a good learning exercise in itself to develop a good model as you have to get a feel for what the key factors are that will impact it and how they change over time... All of which will feed into your actual trading...

[jimibt]

Hi all,

I have heard Peter say on many occasions about modeling the data, but Im curious to know why we'd want to do it, and how to go about it.

Is modelling data a way to program a trading idea into a type of simulator where we can e.g. find out if it would have a positive or negative expectancy over 100... 200... 500 trades for example?

If this is the general premise, then how would I start to model data? I found an interesting source on Betfair that talks you through the initial steps and suggests "R" or "Python" languages to get started. I have some experience with C#, and even less with Java, but I thought my days of stressing over 1 line of code for 18 hours, only for the answer to pop into my head at a randon moment while doing the washing up or some other mundane task, were well behind me lol.
Is modelling data where we separate the men from the boys with regards to trading success? I will if I have too lol.

TIA

the short answer is yes!! yes, as in, we do want to model our data to find patterns, deviations and outliers that we can attribute some sort of categorisation against.

I can't speak for PW, but when I was working on strats for BF, I stored everything related to horses and dogs into a database and then (using c#) created a backtesting engine against which I could deploy different coded strategies. Using the same database (in a different set of tables), I'd store the results of my simulations. In effect, these simulations were behavioural models from which I hoped to capitalise on in the live markets.

In many ways, such modelling leads to confirmation in a general sense, but obviously, the market is a fairly complex system and thus the transition from modelling to live bets often requires additional *data* that isn't captured in the raw price feeds. So yes, modelling does yield a lot of answers and saves a lot of wasted time (or rather, time that could be used elsewhere) trying out ideas. So modelling in a way gives you a reassurance when deploying a strategy as you'll have provided a multitude of data and parameters over a wide time horizon, and thus should be able to apply it to the market with greater confidence.

I'm sure Shaun will definitely have a lot more to add to this as he has majored on this for quite some time and has it down to a fine art.

[ShaunWhite]

I thought my days of stressing over 1 line of code for 18 hours, only for the answer to pop into my head at a randon moment while doing the washing up or some other mundane task, were well behind me lol.

After 25yrs I was sick of my IT job and swore I'd never look at code again. Tried trading the subjective way and wasn't very effective. So the last 4yrs I've been doing what's basically an IT job again and never actually press a button in anger to trade. The job I've ended up with is pretty much the job I had !

"Modelling" is broad and misused term. It's anything from a pricing model to try and outwit SP, to a "model" of the current supply/demand to predict the next tick direction. It's essentially a hard objective proof of something you'd otherwise to subjectively. For some it's a bit of Excel, for others it's hardcore maths and the latest analytical software. Just keep your focus on exactly what you want to get out of it, I've seen on Slack that a lot of them spend all day on the tech and fancy new acronyms and never seem to work on a strategy as such. It's easy to look busy and be doing nothing.

(Thx Jimi but I'm no expert, I'm very much a one trick pony. I'm as keen as anyone to read anything and everything people have got to say on the subject)

[Brovashift]

Looks like shit just got serious

Thanks for all the info guys... The thing thats peaked my interest was browsing through Tennisabstract after Dallas suggested it to help me dig deeper into my tennis trading. Looking at that was just like a sea of raw data and I was thinking how am I gona pick through all this and find something useful... thats when modelling jumped into my head

That's what im wanting to do, is dig deeper into a tennis trading strategy, but what I am looking to find, I dont know! Hoping for some kind of eureka moment.
I see C# mentioned, so can models be created in any language? I think Id pick that back up quicker than learning something completely new.

And are there basic models (tennis models) available on BA that I can use to get me started? As right now I still dont know where to begin...

"Hello World" ?? Lol

[getagrip68]

I think you need to pick the right tool for the job otherwise you waste many more hours than you need to. I analyse data in Excel, which can get complicated but isn't too difficult to learn if you're persistant (sites like Exeljet are a great resource). For me the 'why' is to try and come up with probabilities so I can price stuff up, but I'm more of a punter than a trader.

[Brovashift]

I definitely dont want to waste time, Ive done plenty of that over the years learning things once that I never used again.
I could scribble ideas down on paper, start with something basic like; how often does an underdog beat a top seeded player and at what tournaments. Or do something with/include Elo rating. But how Id code that to show me the results of 1000 matches I dont know..

I cant even remember what they are called now in visual studio (c#), is it functions? Are there built in functions that you can call when building models.
I just remember building a lot of Windows forms applications, and working on a few android apps.

[Brovashift]

Have a look at the BA Tennis or Soccer traders... They are good examples of why you may want to develop a model...

Both are driven by data & the model allows you to test out various scenarios so you can go into trades with a better understanding of what may happen.

For example, I put one together for test cricket.. Basically input the actual players and the model picks up their most recent averages etc and runs a stochastic simulation of the game several 1000 times to give you and idea of what the starting prices should be (expected runs etc) ... Its reasonably accurate compared to the market (before the toss anyway)...

It can also allow you to get a better understanding of how sensitive the game is to specific players performance etc..

It can be a good learning exercise in itself to develop a good model as you have to get a feel for what the key factors are that will impact it and how they change over time... All of which will feed into your actual trading...

Hi Sionascaig,

Do you mean examples in the Tennis / Soccer trading threads, or were you just referring to BA Tennis trader and Soccer Mystic as examples?

[sionascaig]

Tennis Trader & Soccer Mystic ==> they are models already set up and running for you to use

[sionascaig]

MegaRain posted this link a while ago: Making a Modeller... Always thought it was quite good...

https://www.youtube.com/playlist?list=P ... xhi70mfZtv

[ShaunWhite]

I see C# mentioned, so can models be created in any language? I think Id pick that back up quicker than learning something completely new.
" ?? Lol

I'm a c# guy too (recent convert after 2 decades+ of vb) , python is probably great but I don't have the bandwidth. I suggest you keep your data repository in SQL. It's fast, no size limit and industry standard so it can be pulled into excel or anything else. Excel is OKish for analysis but I wouldn't store my data in it.

That way you can create a db that contains the sport stats, and market prices (eventually your bet data too) and start to look at developing some sort of pricing model. Eg initially take each stat on its own and see if it has any correlation with price. Eg does the price change as it changes. Repeat for all stats to eliminate the unnecessary and focus on the stats that matter. Then convert each range of stats values to a value from 0-1 (easier than eventually handling many different value ranges) and begin to look at them in combinations (easier said than done). That's the rough idea. Famously, Mr Benter found 120+ parameters that affect horse prices!

The objective is to have a method of pricing the outcome, and that means coming up with a price yourself that as closely as possible matches the price the market said for prior events. You should then be able to price events that haven't happened yet, and find differences between your pricing model and the price on offer.

[Brovashift]

MegaRain posted this link a while ago: Making a Modeller... Always thought it was quite good...

https://www.youtube.com/playlist?list=P ... xhi70mfZtv

Thanks for this... this is the kind of thing I was looking for, somewhere basic to start from. I get it now, I can see what Im supposed to do. Not great at Excel but I know the basics and can scrape through with Google

[Brovashift]

I see C# mentioned, so can models be created in any language? I think Id pick that back up quicker than learning something completely new.
" ?? Lol

I'm a c# guy too (recent convert after 2 decades+ of vb) , python is probably great but I don't have the bandwidth. I suggest you keep your data repository in SQL. It's fast, no size limit and industry standard so it can be pulled into excel or anything else. Excel is OKish for analysis but I wouldn't store my data in it.

That way you can create a db that contains the sport stats, and market prices (eventually your bet data too) and start to look at developing some sort of pricing model. Eg initially take each stat on its own and see if it has any correlation with price. Eg does the price change as it changes. Repeat for all stats to eliminate the unnecessary and focus on the stats that matter. Then convert each range of stats values to a value from 0-1 (easier than eventually handling many different value ranges) and begin to look at them in combinations (easier said than done). That's the rough idea. Famously, Mr Benter found 120+ parameters that affect horse prices!

The objective is to have a method of pricing the outcome, and that means coming up with a price yourself that as closely as possible matches the price the market said for prior events. You should then be able to price events that haven't happened yet, and find differences between your pricing model and the price on offer.

Im surprised that nobody has developed a nice comprehensive db, with easy to use textboxes and drop downs, and charged a monthly subscription by now, to give all punters & traders easy access to model their trading ideas... There's a million dollar idea in there somewhere lol

Touching on horse racing, if trading in-play by reading the race and trying to identify the weak horse out the pack to L2B, as thats such a discretionary trading strategy I imagine that would be quite difficult to model? If even necessary.

[Update]
Thinking about it, I probably could model an in play strategy; how often does a short priced fav lose, and lose by how much, rider stats... even day of the week lol. I guess its possible to model anything you can think of...

[goat68]

Earlier in the year I created a greyhound model, and it modelled prices reasonably well compared to BSP on average. But using it requires some skill...doing a blind oh this selection is way too high compared to my model price, so Back it...and reverse for Lay etc... produced poor results. Typically when a selection is say 50% higher than your model price, it's typically because your model isn't accounting for something the market is...like all those dog insiders/track siders who know the dog is not well or whatever....does your model know that?

I can see things like SoccerMystic are great tools though, but not sure i'd quite call that a model, that's just plain "Maths" !

[ShaunWhite]

..like all those dog insiders/track siders who know the dog is not well or whatever....does your model know that?

Models are about averages rather than being some sort of all seeing eye. If your model produced poor results then the model was poor rather than the principal?

What parameters went onto your model? Owner, trainer, sire, weight, sectionals, age, days since last run, prior results weighed vs the models of the dogs they raced against.....