Number Crunching After Capturing and Sorting.

[CallumPerry]

Happy 100th post to me! Don’t know whether it’s still a thing (was reading an old thread earlier that was discussing the titles) but I’m nearing archangel status! Can’t wait to reach that status within the markets and my P/L too!

Anyways, for my 100th post I wanted to ask an advanced analysis question. Does anybody have any advice on forecast models? I had a look at several, including Holt-Winters Forecasting, but this only works for seasonal repetitive series.

Next, i’ve seen forum members ask about capturing data and I’ve read threads about breaking this up into different sports, race types, scenarios etc but what then? I think it makes for a good point of discussion and I'm very interested to read how other people number crunch.

For example, say I have 10,000 markets recorded with a very particular set up. I was thinking of having an ‘in hindsight (holy grail)’ line once all the data has been collected and then correlate my testing indicators (or generally my entrance and exit points) to this so I can clearly say: Ok strategy A returned 68% correlation to the holy grail line whereas strategy B only returned 64% so I should pursue and tweak strategy A. Is anyone willing to share how they analyse their entrance and exit points from past data to determine which strategy works better than others?

Basically what I’m asking is, are there any common practises adopted from the financial world once large data sets have been split up into it’s smallest nuances? Or does anyone have something unique they are willing to share.

Say I have 1 second information for a market, for the sake of simplicity, 6 grey hounds. I have each of their Last Traded Price information for 5 minutes out until start time for over 1,000 markets and then I put the favourites Last Traded Price line on a chart (Blue line).

Screen Shot 2018-08-30 at 23.36.56.png

A simple ‘holy grail’ line (Orange) would be B30 = A1, B31 = A2 etc so it is 30 seconds ahead of the actual price moves. I’ve added in a polynomial (order 6) trend line to smoothen out the bumps (Could just use an EMA).

What is the problem with pursuing this sort of simple method? Already I can see a few including: ‘why 30 seconds?’ and ‘my own stakes would themselves alter the past data if this was live’. Big shifts (selections being withdrawn from a race) could skew results but remember these many markets have already been sorted.

Can anybody add any more faults, solutions, advice, recommended reading or methods they use to spot patterns in big data sets once you’ve broken the data into specific areas? I have little idea what to do next.

Cheers for any replies,

CPerry.

[ruthlessimon]

Can anybody add any more faults, solutions, advice, recommended reading or methods they use to spot patterns in big data sets once you’ve broken the data into specific areas? I have little idea what to do next.

Cheers for any replies,

CPerry.

Personally; I don't think many people know either Cal (hence the lack of replies) - so you're defo not alone!

For a laugh, I thought I’d test the following idea:

“Horses which began with the letters B, E, T, A, N, G, E, L would’ve been profitable back trades during the liveshow – over the last 3mths.”

Before testing, I put a couple of controls on this trade idea.

1. The trades were only placed on the fav.
2. The entry time was always @ “xx:xx” time (remaining to post).
3. Exit always at 00:00.
4. Commission, direction, hedging would be factored into the formulas

Here were the results (return on stake):

A= -0.3%
B= 0.1%
C= 0.5%
D= 0.3%
E= 1.2%
F= -1.1%
G= 0.0%
H= 0.6%
I= 1.9%
J= 1.0%
K= -0.4%
L= 0.3%
M= 0.1%
N= 0.9%
O= 0.5%
P= -0.5%
Q= 0.3%
R= -0.4%
S= -0.3%
T= 0.6%
U= 1.6%
V= 1.7%
W= 0.3%
X= -9.0%
Y= -0.7%
Z= -1.1%

These results raise some serious questions.

Firstly, because commission has been factored, anything above 0.0% is technically an edge. & anything returning less than -0.6%(ish) probably could be reversed into a positive strategy. Notice how nearly every letter is an “edge” - & that’s dangerous.

Secondly, there’s a common argument that if data can be “grouped”/”logical” it provides more significance. Well, let’s look at the vowels (“AEIOU”). This returned 0.5% (435 trades); & “BETANGEL” returns 0.3% (899 trades). Also, “BETANGEL” (as expected ) was actually more profitable than “AEIOU”, because of the no. of trades. & if a 899 trade sample, isn’t enough to confirm an edge, I dunno what is!! Most people reckon they’ve cracked it in a day

Thirdly, there’s an argument that this data might be too generic. Including a second letter, will probably lead to an improved edge. Problem here, is that every refinement shrinks the sample size – leading to just as rubbish conclusions - as data that’s too generic. & it also wouldn’t be a guarantee of an improved p&l!

I could go on, but I hope that was sort o’ interesting/thought-provoking.

Ideally now, someone like Peter needs to come in with some pearls of wisdom/correct my errors – but this topic is a bit of rabbit hole!

[Jukebox]

I see that although M.U.M beats D.A.D. F.A.T.H.E.R. beats M.O.T.H.E.R.

[ruthlessimon]

Thought the following was quite interesting: https://elitedatascience.com/bias-variance-tradeoff

The way I understand it (I think) :

Low variance, High bias = i.e. backing every fav @ xx:xx (time)

High variance, Low bias = i.e. backing a fav which trades through price x.xx (making sure it didn't spike, or was too close to the price), between times xx:xx & xx:xx, between volumes £xxx,xxx & £xxx,xxx, & making sure the other runners in the market moved as expected (what a nightmare )

Again though, very few people have worked out that correct balance of over-fit & under-fit strategies (via data, usually (unfortunately) seems most just go by "hunches/intuition")

[CallumPerry]

@ ruthlessimon

That is all very thought-provoking! It is very counterintuitive too as I’ve always seen and understood that the way to find an edge is to record some data and look at niche areas until you find something that works over a good amount of time, if it’s something nobody else has done then you have your edge. A bit of common sense tells you placing bets on selections that start with the letter ‘I’ isn’t the holy grail but looking at those results, heck you could start a tipster service on social media and convince hundreds to blindly follow that as a legit strategy.

Now the bias-vairance-tradeoff read was equally as fascinating and another area that I hadn’t even considered, if it’ll help me hopefully it will also help others consider how they are viewing their data; I think working out how to balance and not over/under-fit my strategies is what I should look at next.

Does anybody else have anything worthwhile to add once any trader gets to this stage? Data has been collected, potential edges have been identified and … some sort of further testing now happens before going live. Once you have something that is working has anybody got any tips for modifying/improving overtime without then slipping into over/under-fitting their data?

P.S out of curiosity, when you created your alphabet idea, how did you work out your return on stake? Using my orange line (from the OP) as an example, how could I work out how much that line would have made/lost in theory compared to the blue line (LTP)? Would be really interesting to see how high I could get my return with silly ideas like your alphabet one over massive sample sizes to better understand bias/variance.

Cheers,

CPerry.

[ruthlessimon]

It is very counterintuitive too as I’ve always seen and understood that the way to find an edge is to record some data and look at niche areas until you find something that works over a good amount of time

That’s the mainstream view certainly. But I find simply defining a “good amount” of time pretty dang tricky. Too little, & everything’s just a noisy mess. Too much, & we’re powerless to a genuine shift in the market.

if it’s something nobody else has done then you have your edge.

I’m torn on this. Yes absolutely, too many people with a correlated strategy & someone won’t get filled – eventually leading to a market with 0 volatility; where prices just spike between levels. On the other hand, there’s the argument that a “genuine edge” has a bit of leeway - & missing the perfect fill won’t return it to a 0 expectancy – but the problem is most things aren’t a 0 expectancy! (i.e. random letters..)

A bit of common sense tells you placing bets on selections that start with the letter ‘I’ isn’t the holy grail but looking at those results, heck you could start a tipster service on social media and convince hundreds to blindly follow that as a legit strategy.

Yeah, just throw in the “past performance is no indication of future performance” & I’m made for life!

At the moment though, (apart from the “leeway argument”), I don’t have a way of justifying (to myself) why the letter "I" isn't an edge (i.e. it was a 3mth sample, with hundreds of trades, commission accounted for etc) - & therefore, sets some very dangerous precedents. An example is the following:

“It won’t work next month (turns out profitable), it definitely won’t work next month (turns out profitable), hmm maybe it is an edge, I’ll trade it” – at which point it returns to a 0 expectancy! Lol - Sounds awfully similar to punting a trend too late doesn’t it!

My personal opinion is that it was correlated to “something interesting” – but this is almost impossible to verify – considering the sheer number of variables.

P.S out of curiosity, when you created your alphabet idea, how did you work out your return on stake? Using my orange line (from the OP) as an example, how could I work out how much that line would have made/lost in theory compared to the blue line (LTP)? Would be really interesting to see how high I could get my return with silly ideas like your alphabet one over massive sample sizes to better understand bias/variance.

The formula I use on my sheets is a bit of a nightmare! I adjust the staking by direction & price (i.e. a 1 tick winner will always be £1 (after commission) etc); but the basic premise is:

Stake: £500 (backing)
Entry price: 1.86
Exit/Hedge price: 1.84

(Stake * (Entry price – Exit price)) / Hedge price = Profit before commission (If it’s positive *.95)

Therefore:

(500*(1.86-1.84))/1.84)/0.95 = £5.16 = roughly 1% RoS

HTH

[ruthlessimon]

An interesting website: https://machinelearningmastery.com/start-here/

"The simplest type of prediction is to use the mean value."

"Predictive modeling is the most common type of problem that we solve with machine learning (e.g. classification and regression problems)."

"Breakthroughs and innovation occur when assumptions and best practice are demonstrated to be wrong in the face of real data."

Problem is, I'd label a "backtested strategy" as a "predictive model" (the strategy is predicting future direction), but I'm pretty sure this differs to Peter's view (i.e. focusing on %'s rather than p&l - but this baffles me because strike-rate means nothing without risk/reward).

Come on Peter give us a steer!

[foxwood]

Come on Peter give us a steer!

He explains what he does in various blog posts he has made over the years - search the blogs for "value" and read the posts for clues

Any "predictive model" that isn't based on a "backtested strategy" is usually called a guess - not a bad thing though as long as you understand the rationale for the strategy. Some ideas are just too complex to backtest - quicker to throw a few bob at those and see what happens.

If you can't come up with at least a halfway sensible reason why a new strategy should work then you are probably backfitting to the data. Sadly though, a sensible reason doesn't mean it is a valid strategy longer term.

The ROI element is the reward and the risk is (depth + length of drawdowns) versus (bank + nerves of steel | jelly)

Strike rate is pretty meaningless since it is an averaging technique and hides the reality of consecutive multiple losses which are usually enough to make most people kill a strategy within a short time.

If in doubt use at least a year's data and graph out the cumulative p&l day by day - a picture's worth a thousand words.

[CallumPerry]

@ruthlessimon

I would argue that the letter ‘I’ was an edge (or the start of) because it had been recorded to have a positive R.O.I over a good sample size, is it a good, long-term, easily scalable clever edge? Absolutely not. I think you touch on a very real internal problem that number crunchers have… where do you draw a line? Once you have found an edge when are you happy with it working? Like you said, you could have joined the trend too late. Is it a reoccurring trend? I agree, I believe it correlates to something interesting and could very well be a by-product of some other true edge.

I am going to collect some more info over the next two weeks or so and start using a variation of your staking back testing with some of my ideas and see what crops up.

Some of the things I’m looking at involves mean reversion, that’s no secret. Best to keep it simple, always. I think an edge comes from an idea/feeling/theory first. You then collect data on the variables you think play a part in this (and many more that you are unsure about) before breaking it down, this was the point of my original post and where I am stuck. What now? How do you draw meaningful conclusions, on how much?

There has to be a point when you just say ok fuck it, I’ll give it a shot with small stakes (then comes the ordeal of working out how/when to scale and keeping everything in perspective as you tweak the original strategy).

When I get to the stage that my first big clever bot project goes live, I want to know what exactly I’m looking at. Like a pilot in a plane, if the thing starts to go down I would like to know which buttons I need to press before it hits the ground.

At the moment, on top of everything else discussed, I’m thinking of recording my current ‘if I were to green up now’ + bank balance figure each second to see how my total funds fluctuate (I’m assuming more correlated numbers means less variation in my strategy [not always bad if you have to hold a position sometimes] and the steeper the angle means more money). Not sure what else to look out for at this point so I welcome other people’s inputs. (Will also record green up + balance for each selection in order of favouritism)

Strike rate, t-stats, R.O.I, longest winning run, longest losing run. What are everyone’s thoughts on these sorts of figures? How important do you think each are, does anybody use multiple before going live? Have any additional ones?

I am not looking to overcomplicate my strategies as I believe the simple ideas do best but I do want to collect as much information as I can for each of my ideas in the future and be able to shape and read them as efficiently as possible and I haven’t really seen many threads discussing how to do this so please add conversation.

[CallumPerry]

@foxwood

Yeah I’d agree without back testing, a 'strategy' is just a guess. If something is too complex to backtest maybe it could be stripped down to a simpler idea or yeah if you throw a couple of quid at it and record it you could tweak it later on. I believe having an idea why the strategy could work beforehand is important. Investigating whether selections that begin with the same letter as your name is a waste of time because there doesn't seem to be any reason why this would hold up long term but maybe I'm being naive.

Do you record your ROI in this way exploring: depth, length of drawdowns, bank and 'how much I've bricked myself' indicator? I'm interested in how traders interpret these figures you see. Same with strike rate.

Best thing to do is keep a running total of your balance and hopefully it grows consistently. As I touched on with my previous message, if the range is thin it implies low variance in your strategy, less risk and the higher the incline the better the strike rate. I just need a green up formula now and I'm close to really doing some number crunching.

If anybody has got a formula handy it would save me some research time. Balance + current green up value across all selections.

[ShaunWhite]

I just need a green up formula now and I'm close to really doing some number crunching.

If anybody has got a formula handy it would save me some research time. Balance + current green up value across all selections.

GreenUp = Stake * ((Entry price - Exit price) / Exit price)

Do that for each open trade and sum them.

Commission on profit obv but multiple trades in the same market will reduce that a little as losses will reduce the gross gains on which commish is based.

[ruthlessimon]

I just need a green up formula now and I'm close to really doing some number crunching.

If anybody has got a formula handy it would save me some research time. Balance + current green up value across all selections.

The actual formulas I use are as follows (if you use it & find an error (or ways to simplify it) lemmi know btw )

LAY:
Stake: =LI20/0.95*INDEX(Calcs!A:A,MATCH(LI22,Calcs!A:A,0)-1)/(INDEX(Calcs!A:A,MATCH(LI22,Calcs!A:A,0)-1)-LI22)
P&L: =IF(LI21*(LI23-LI22)/LI23>0,LI21*(LI23-LI22)/LI23*0.95,LI21*(LI23-LI22)/LI23)

LI20: 1 tick profit target (i.e. £1)
LI21: Stake
LI22: Entry price
LI23: Hedge price
Calcs!A:A: Price list from 1000 - 1.01

BACK:
Stake: =LJ20/-0.95*INDEX(Calcs!A:A,MATCH(LJ22,Calcs!A:A,0)+1)/(INDEX(Calcs!A:A,MATCH(LJ22,Calcs!A:A,0)+1)-LJ22)
P&L: =IF(LJ21*(LJ22-LJ23)/LJ23>0,LJ21*(LJ22-LJ23)/LJ23*0.95,LJ21*(LJ22-LJ23)/LJ23)

LJ20: 1 tick profit target (i.e. £1)
LJ21: Stake
LJ22: Entry price
LJ23: Hedge price
Calcs!A:A: Price list from 1000 - 1.01

[ShaunWhite]

The questions around confidence in back tests and live results is a hardy perennial. Whether it can be reduced to a formula is doubtful imo. I think it just comes down to a range of confidence and experience.

Even a rock solid data edge may well deteriorate a little in live, how much will depend on what transactions are involved and experience would probably give you a guide as to how much. For that reason it's probably worth having a small deduction factored into your ideal world simulation.

As for confidence then it might be worth looking into some sort of simplified Kelly staking, back it off on the bad runs and ramp it up on the good ones. It might not be the best way to maximise returns but it will help you keep your sanity if a bot that appears to be struggling is only costing you a pound or two a day.

I've always liked the idea of having a R&D budget too, it's easier to handle lose a few quid trying an idea if you don't expect it back. Like paying to enter a competition, you don't expect a prize but it's great if you do, and how much you'll pay is related to how confident you are that you'll win something.

[ruthlessimon]

When I get to the stage that my first big clever bot project goes live, I want to know what exactly I’m looking at. Like a pilot in a plane, if the thing starts to go down I would like to know which buttons I need to press before it hits the ground.

Here's a "random" edge I'm looking currently looking at - in the context of the plane analogy - this is a full-blown engine malfunction at 20,000 feet (& why your question is so important)

The overall figures are the following:

Return on stake: -0.09%
Total trades: 2002

Now someone could say, yup no edge - move on.

But let's look at the equity curve



Suddenly it shows a nasty dilemma (had it been traded live)

The first 450 trades (big sample) = +3.92% RoS! (big edge)

Everything after 450 = -1.22% (perhaps enough of a -ve return to label it as an anti-edge)

Therefore, is it fair to label this as "random"?

[ruthlessimon]

Didn't Mark Douglas say a 20 trade sample was enough??