Automation Testing

[Andrew2966]

First of all, I have to say that I find Bet Angel and Guardian an absolutely fantastic piece of software so congratulations to everyone involved in developing this program. My question is based around the testing of automation files and may well have been asked before. Obviously, as I am developing a file, I run it in practice mode to make sure that the file is doing what I want it to do before I go anywhere near the real markets. My preference is to trade the horse racing markets, both pre off and in play. Since I am not one of those people who trade the markets for a living, I have to go out and work during the day. Therefore, a large chunk of the days racing is over by the time I return from work (and all of it during the winter months) and this ultimately leaves very little time to test any files I am working on. So my question is would it be possible for the software team to consider this and integrate some kind of test centre, that is not run in real time but works off 'recordings' of historical races or other sporting events. There's nothing more frustrating than having an automation idea in your head and then having to wait 18 hours to test it. Maybe this is something to consider for the next version of the software. PS, if this already exists then please point me in that direction

[Realrocknrolla]

Probably not gonna happen lol

This should help you!

viewtopic.php?f=54&t=8896

[Andrew2966]

Thanks for the link. Unfortunately, the youtube video is now set to private so cannot be viewed. It's a shame something like this isn't available at this point in time as I believe it would be a feature that all Bet Angel users would benefit from. I would have thought it should be easy for the tech guru's at Bet Angel to create something like this that runs off a set of historical data files rather than running piggy back off the actual Betfair site. It would also negate the worry of having your betfair account restricted for inactivity and constant use of the practice mode.

[Realrocknrolla]

Thanks for the link. Unfortunately, the youtube video is now set to private so cannot be viewed. It's a shame something like this isn't available at this point in time as I believe it would be a feature that all Bet Angel users would benefit from. I would have thought it should be easy for the tech guru's at Bet Angel to create something like this that runs off a set of historical data files rather than running piggy back off the actual Betfair site. It would also negate the worry of having your betfair account restricted for inactivity and constant use of the practice mode.

Try Power Query mate with excel and the data you want to churn!

[the4droogs]

I completely agree with this. Testing automation files is by far the worst part of building automation files in BetAngel. The interface for building the bot is fantastic. Testing it is absolutely awful. Any bot that has any sort of complexity - and most profitable bots do - needs to be tested thoroughly. You often need to allow for edge cases that occur rarely. The only way you can test for this is watch your bot on the market for dozens (or more) different markets and hope that the test case occurs, and that when it does you have the required info in the logs to figure out what happened. I put a request for features to help with this a while ago:

viewtopic.php?f=20&t=22432

I've also seen other users request similar things. Thus far I haven't seen any indication from BA whether that would be something they'd consider. Though from a business perspective I doubt it would be worth their while. I doubt that a testing suite would make a big difference to getting new customers through the door.

I currently test by setting up a screen recording which records both the logs and the ladder interface for the market the bot is working on. It's far from ideal, but at least I can walk away from the screen while it does it's thing. Then if the test case I'm looking for occurs I can replay the video and frame by frame to figure out what happened in the market and debug that way. Like I said, far from ideal but it's better than sitting their watching and hoping that the edge case occurs.

[Derek27]

I've had a pretty frustrating day testing my automation on markets that didn't go in-play or suspended at the start, or an early goal before the rules could come into play. But in all honesty I was expecting nothing more. Live markets are real world events and that's the best place to test. People with programming backgrounds are probably more familiar with the tedious job of testing code.

It might help to rely on logs. Add stored values (created for debugging purposes) to your log so you can sieve through them at the end of the day. But I think the sheer expense and complication of creating a test centre that accurately mimics the Betfair API and also updating it whenever the API changes for probably a small number of people who want to use it would make the idea unfeasible.

[Dallas]

I completely agree with this. Testing automation files is by far the worst part of building automation files in BetAngel. The interface for building the bot is fantastic. Testing it is absolutely awful. Any bot that has any sort of complexity - and most profitable bots do - needs to be tested thoroughly. You often need to allow for edge cases that occur rarely. The only way you can test for this is watch your bot on the market for dozens (or more) different markets and hope that the test case occurs, and that when it does you have the required info in the logs to figure out what happened. I put a request for features to help with this a while ago:

viewtopic.php?f=20&t=22432

I've also seen other users request similar things. Thus far I haven't seen any indication from BA whether that would be something they'd consider. Though from a business perspective I doubt it would be worth their while. I doubt that a testing suite would make a big difference to getting new customers through the door.

I currently test by setting up a screen recording which records both the logs and the ladder interface for the market the bot is working on. It's far from ideal, but at least I can walk away from the screen while it does it's thing. Then if the test case I'm looking for occurs I can replay the video and frame by frame to figure out what happened in the market and debug that way. Like I said, far from ideal but it's better than sitting their watching and hoping that the edge case occurs.

It was answered a few times in the reply of the link posted, the one at the bottom of page 3 about the overall cost to the user is the most limiting factor

[ShaunWhite]

the overall cost to the user is the most limiting factor

The dollar cost can be zero but if you value your time then that can really add up.

[ShaunWhite]

Any bot that has any sort of complexity - and most profitable bots do -

You often need to allow for edge cases that occur rarely.

Not true, the more simple they are the better they are. Some people make 7-figures with stuff that takes less words to explain than there were in your question.

How many edge cases can you think of? I can only think of a handful including flipping the house fusebox. Besides edge case are +ve and -ve and a decent strategy should have enough margin for blue moon events.

It sound's like you just need some decent data, testing doesn't tell you much that your analysis shouldn't have already told you or warned you about.

[goat68]

Any bot that has any sort of complexity - and most profitable bots do -

You often need to allow for edge cases that occur rarely.

Not true, the more simple they are the better they are.

I'd agree with this, I reckon part of my problem was getting way too complex on the price action, when I look back at what I was doing I think blimey.....

[the4droogs]

Not true, the more simple they are the better they are. Some people make 7-figures with stuff that takes less words to explain than there were in your question.

Fair point. I'm sure there are simple profitable bots. But of my bots the only profitable ones have had a fair bit of complexity, mostly around exiting the trade if the market goes against me.

As for data analysis, you're probably right. My MO is to come up with an idea, automate it and test it's validity over a large number of markets in practice mode or at very low stakes. This means I build a lot of automation files that get thrown away if they are not successful. So there is a lot of testing as I need to make sure it works right so I can trust the results. Maybe I need to change my method so I do a lot more analysis before building.

[Anbell]

My MO is to come up with an idea, automate it and test it's validity over a large number of markets in practice mode or at very low stakes. This means I build a lot of automation files that get thrown away if they are not successful. So there is a lot of testing as I need to make sure it works right so I can trust the results. Maybe I need to change my method so I do a lot more analysis before building.

You dont need to spend more time analyzing, IMHO.

Your original approach is the best: hypothesis > test > (understand) > (tweak) > scale (slowly).

Back-testing is problematic, and data-fitting is worse.

[Anbell]

My MO is to come up with an idea, automate it and test it's validity over a large number of markets in practice mode or at very low stakes. This means I build a lot of automation files that get thrown away if they are not successful. So there is a lot of testing as I need to make sure it works right so I can trust the results. Maybe I need to change my method so I do a lot more analysis before building.

You dont need to spend more time analyzing, IMHO.

Your original approach is the best: hypothesis > test > (understand) > (tweak) > scale (slowly).

Back-testing is problematic, and data-fitting is worse.

(My experience is pre-off horses. Other markets might be different)

[ShaunWhite]

But of my bots the only profitable ones have had a fair bit of complexity, mostly around exiting the trade if the market goes against me.

Maybe I need to change my method so I do a lot more analysis before building.

Trial by PM or live is too slow. You can't spend 2months on each idea waiting for results to accumulate. In that time you could test 10 ideas if you equip yourself with some decent pro level data.

Sounds like you see exits as being important. I suggest you look at your bets closely and see which are actually making the money. In my experience stops/exits are a waste of time. You stop as many that would have come back as those would have kept going against you. I need as much reason to flip my position (Eg exit/close/hedge or whatever you want to call it) as I did opening the position in the first place. If you want an exit to reduce your varience then just stick in an sp close. Sp is known 0EV so unless you find your exit bets are +EV then an sp close will make sure you're not giving money away.

Basically, stop thinking about the "trade', realise they're all just bets that run. Look at how the individual bets are performing rather than picking two bets and looking at the net of them. When you open and close for a net zero all month, chances are one bet type makes money and the other is losing it.

' stops' were originally devised so traders could get some sleep without worrying about a massive crash. They're not for making your normal buy/sell decisions. The idea you 'stop' losses is naive.

[the4droogs]

Sounds like you see exits as being important.

Yes I do. The reason for this is that I create a lot of scalping bots that are trying to gain one tick. So if I have a market that moves against me I could lose 20-30 successful trades with one unsuccessful trade. That said, you're right that I have no idea if these exit trades are positive or negative EV. Just that the overall net result is positive or negative. This inspires me to revive some of my older files that just about broke even and collect some data about the individual bets.

Good to have a conversation on here about some of the more abstract aspects of automation.