Programming/Coding on Betfair

Post Reply
PeterLe
Posts: 3715
Joined: Wed Apr 15, 2009 3:19 pm

In a recent post, i mentioned that it may be nice to have a separate topic for those interested in writing their own code to connect to the API for trading extracting data etc
So if you use C#; Python, Java or whatever, then maybe we can share ideas/help

The idea stemmed from this post by Liam/Wolf:-
[/quote]

viewtopic.php?f=55&t=13909

Basically you need to sign up to use the betfair API which is £300 and then build the software the access the API.
LinusP has a public github repo with some generic python code to access the API streaming data. I think his github repo has the code used to access the API only, so you would still need to develop a shed load of code for yourself once you have a set of working API connection functions in order to achive anything resembling what he has built for his own personal use.
[/quote]

Pretty much however flumine allows you to record the raw streaming data that betfair charges a fortune for. Recording parsed data as it comes through is good but having the raw data available is invaluable.

https://github.com/liampauling/betfair
https://github.com/liampauling/flumine
[/quote]

Ive been playing around with C#, but recently decided to take a look at Python too. It's early days but Im finding it easier to learn Python than C#
For anyone interested in learning Python, this course is interesting (on Udemy) :

The Modern Python 3 Bootcamp - by Colt Steele (Course cost £13)

Regards
Peter
User avatar
Derek27
Posts: 23476
Joined: Wed Aug 30, 2017 11:44 am
Location: UK

PeterLe wrote:
Wed Nov 13, 2019 12:30 pm
Basically you need to sign up to use the betfair API which is £300 and then build the software the access the API.
I haven't yet got round to looking at accessing the API directly but I thought it is free to use and test (with a 20-second delay) so you only need to cough up when you're ready to run or test live?
spreadbetting
Posts: 3140
Joined: Sun Jan 31, 2010 8:06 pm

Only thing that put me off python when I had a quick look was that it seemed very much command line type of stuff rather than a GUI. Can you code GUI with python and also what program/compiler would people recommend, my daughter at uni uses spyder(?).
User avatar
northbound
Posts: 737
Joined: Mon Mar 20, 2017 11:22 pm

I recommend learning Nodejs, which is Javascript that runs on a server, which was built from day one to be asynchronous.

For example, my horse Telegram alerts application is written in Nodejs, reading the Betfair Stream data. When it sees a fav getting beaten at low BSP, it hits the Telegram API to deliver the alert.

All the while it keeps reading the incoming Betfair Stream data, without waiting for the Telegram API request to complete its execution.

Asynchronous.

The added benefit of Javascript is that you can reuse some of the skills to write visual applications and charts that you can access with a web browser.

The learning process could be:

- Learn Javascript basics.

- Learn Nodejs basics.

- Learn basics of running a Nodejs application on Linux, as you can get a VPS for only $5/month.

- Write a Nodejs application that reads Betfair Stream data and stores it somewhere, e.g. snapshot of horse racing markets at post time.

- Make your Nodejs application expose an API that serves that data.

- Learn a Javascript front-end framework (Reactjs, Angular) to create a GUI that you can access in a browser to display that data served by your Nodejs API.
PeterLe
Posts: 3715
Joined: Wed Apr 15, 2009 3:19 pm

Derek,
I was lucky enough not to have paid anything for the API keys, I got mine converted before the charges were brought in
(I think they are £200/Key now??)

Northbound, Thanks, but I think if I try anything else just now, my head will explode! (just spent two hours just being able to get the latest version of python on my mac and getting Sublime working with it :D
The reason i wanted to try Python was because Liam has already uploaded files to github and I wanted to play around..

...There could be another Voler la Vedette error coming up soon from my account :D
PeterLe
Posts: 3715
Joined: Wed Apr 15, 2009 3:19 pm

spreadbetting wrote:
Wed Nov 13, 2019 4:19 pm
Only thing that put me off python when I had a quick look was that it seemed very much command line type of stuff rather than a GUI. Can you code GUI with python and also what program/compiler would people recommend, my daughter at uni uses spyder(?).
I had that question too Spreadbetting; id be keen to know the answer also
User avatar
ShaunWhite
Posts: 9731
Joined: Sat Sep 03, 2016 3:42 am

PeterLe wrote:
Wed Nov 13, 2019 5:43 pm
(I think they are £200/Key now??)
£299
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

PeterLe wrote:
Wed Nov 13, 2019 12:30 pm
you would still need to develop a shed load of code for yourself once you have a set of working API connection
In your opinion, if average joe was plucked off the street (& only has gcse maths).

What kinda timeframe/hrs are we looking at, for developing something similar to what Liam has at his disposal?

(i.e. absolutely nothing -> rough ability to simulate the ladders)
User avatar
ShaunWhite
Posts: 9731
Joined: Sat Sep 03, 2016 3:42 am

ruthlessimon wrote:
Wed Nov 13, 2019 7:02 pm
PeterLe wrote:
Wed Nov 13, 2019 12:30 pm
you would still need to develop a shed load of code for yourself once you have a set of working API connection
In your opinion, if average joe was plucked off the street (& only has gcse maths).
It's a tough question.
I wouldn't worry about maths, there's none in coding. It's more about people's ability to break tasks down into small steps and choosing the right actions to perform them, while considering every evenuality such as missing info etc.

I'd say the people most suited to coding are people who have an aptitude for languages rather than maths, afterall most programming languages are an assembly of verbs, performed on nouns which themselves have adjectives... Like Excel, cells (nouns) have properties (adjectives) and you do stuff to them (verbs).

Given decent sample lumps of code, learning in a structured way, to get average Joe with zero experience to the point where they can read/write data from SQL or an API you might be looking at.... a monthish, or two ???? ...but most of that is just getting used to the development envoironment/tools/language basics etc. All stuff which you then don't need to do again for the next stage, progress accelerates. A project can be as big as your imagination though so something major can take years.

The hardest part is usually the visualisation and user forms/buttons etc, personally my stuff doesn't even have much of a display, just text in a command window to show how it's doing. Creating something visual like a ladder you can click on that does stuff is a whole different world of pain.

So hard to estimate though, sat with someone who understands how you like to learn, with decent sample code, you could probably get something simple working like reading and writing a file, inside a couple of days. But like poker, mins to learn and a lifetime to master. The probelm when you start is that you need to have a decent plan/idea of how it's going to work, but without knowing what you can do or how you do it, that's a bit of a Catch22.

Best thing might be to try some beginners stuff on Udemy and it'll either click in a week or it won't. The choice of language will probably be dictated by what you want to do. As I said a dumb data collection app or even trading bot is a very different thing to something with a recognisable functional front-end display. VBA is never a bad place to start though to see if you have the knack, because Excel does a lot of the heavy lifting on the display/data handling side and there's a ton of samples and tutorials. But you will need to move on to do something more suitable pretty soon.
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

Really interesting take ty :)
User avatar
johnsheppard
Posts: 283
Joined: Mon Feb 04, 2019 6:00 am
Location: Cairns Australia

Hello there,

This post may seem a bit disjointed...in a rush...

My experience is with C#. I can tell you I spend more time learning to program than I do learning to trade :) I still have a loooong way to go to be a good trader....So it depends on what you want to do...I am sorta almost their in terms of being a half decent software engineer (but le me stress the half decent part) and I've been tinkering with programming for 15 years and have a computing science degree (didnt learn too much there tho)....

It's not just the syntax you have to learn...that stuffs easy...almost any idiot can do it.....its the architecture and design practices that you get caught on that cost you....and nothing will teach you that except long hard experience born from mistakes....

If you are deciding on a language based on UI, you should also question whether you really need a user interface (other than a textual console that is)... as soon as you need a user interface in C#/WPF you typically then have to learn things like MVVM (which is very confusing for the uninitiated)....just like trading, every software guy on the planet has a different idea of how to do things....and that means....you need to know every different way of doing things :)

Look into things like Domain Driven Design, Onion Architecture/Clean Architecture...etc...there are 600 page books on the subject and they don't learn cheap....

But it depends on how hardcore you want to get, and how side tracked your personality gets you....you WILL spend a lifetime becoming an experienced software engineer...

But its good fun...so I heartily recommend it :)
LinusP
Posts: 1871
Joined: Mon Jul 02, 2012 10:45 pm

In my option in python is the best language to learn as a beginner, there are so many brilliant open source libraries available, requests / pandas / scipy / Django. It allows you to do a lot but with minimal experience, this of course comes at a cost due to it being a high level language it is never going to be as quick as C.

I was lucky that I started learning when the new API was being rolled out, I had 2/3 strategies running on BA that I migrated across to python in a month (from zero programming experience) I then soon realized that the open source wrappers available were a load of crap and not maintained so wrote my own. I have then just kept on top of changes and was one of the first to develop streaming which opened up a lot of doors!

In regards to GUI’s this is not what python is for, tkinter will make you want to kill yourself. Instead you should look at building a web app on top of an API or database, this is my setup:

Web App (vueJS / d3 / highcharts / redux, this gives me a view on profit per market / strategy / errors / uptime etc.)

Trading Instances (multiple servers running multiple strategies all in docker containers, managed with portainer)

API (sits on top of db, never connect directly to a production db without an API!)

Transaction Database (all order / strategy / market / metadata)

CloudWatch (logs from instances / db / api / docker)

S3 (recorded raw streaming data, I have a lot!)

The above is missing a data lake which sits separately that receives all data from the transaction db and all logs which are joined together to allow detailed analysis, jupyter then sits on top for further analysis.

This is then all replicated in a dev env, terraform is a lifesaver, I then use serverless to handle restarts / daily scripts. Average AWS bill is around £250-500 a month depending on what I have been doing but the majority of that is from db costs. Hopefully this won’t put people off as you can start with the trading framework like I did and then slowly build out, this is 6/7 years of hard work :D

This is my web app which I use to monitor things whilst lying on the beach ;)
55740446-C6B3-4AF1-9A2E-D008C8A869BD.png
You do not have the required permissions to view the files attached to this post.
User avatar
northbound
Posts: 737
Joined: Mon Mar 20, 2017 11:22 pm

LinusP, your web app looks great, nice job. But I'm sure you scared people away with that £250+ monthly bill. 😁

So here's a little message to reassure beginners that you can still accomplish a lot with a tiny $5/month Linux server that runs your Python or Nodejs app.

You wouldn't probably be able to save all of the data that LinusP does, but you can definitely inspect the current state of the market, the traded range for a horse in the past X seconds and trigger things accordingly (bets, save to db, Telegram alerts, etc).
User avatar
johnsheppard
Posts: 283
Joined: Mon Feb 04, 2019 6:00 am
Location: Cairns Australia

Thanks LinusP. looks cool! I think your tools highlights a big difference between c# world (microsoft) and python world.

In a Microsoft world, you pretty much live in visual studio and most of the technology is Microsoft centric... it has its pro's and cons... in python looks like a bunch of different companies :)

I haven't been doing this as long as LinusP (and i'm not profitable yet either), but here are some of the things that I use;
- WPF,/Xaml, Devexpress UI controls and their MVVM framework, MVVM patterns
- Entity Framework (code first, no stored procs)
- MS-SQL Server....(thinking of ditching this for just plain XML files)...
- C#
- Selenium for webscraping
- MS Test Framework for unit testing

I like that I more or less can just do it all in visual studio. It keeps my anxiety at bay...but that's just horses for courses...

A lot of this stuff is opensource, and I think Microsoft do realise that it has them beat...myself I'm all for paying for whatever is best value open source or not.....I would suspect your commercial options are a whole lot better in Microsoft land...(that's just half assed speculation...dunno really)

For those that haven't been there and done this....each of these technologies there are usually 4 or 5 leading options on the market that need to be analysed and decided on...it takes ages to do all that stuff...(e.g. MVVM frameworks might be Caliburn Micro, Prism, MVVMLight, and a gazillion others, ORMS might be dapper, entity framework LLBLGEN, or a gazillion others, UI can be Winforms, WPF...or HTML if you wanna go webbased)

There are also plenty of opensource libraries around which might be useful: Tulip Indicators (for financial indicators)...deedle for time series analysis, etc... (I don't use these)

IMO, The best thing you can do is keep it simple...don't have a UI, don't have a database, don't have anything until you really need it...work on your domain models first and foremost...(one of my many mistakes)

And if you are deciding on a language or platform...I don't know for sure...but I reckon...it doesn't really matter....6-7 years of hard gaff is what is more important :)

Or maybe just use Betangel :)

Hope this information is useful to someone...
spreadbetting
Posts: 3140
Joined: Sun Jan 31, 2010 8:06 pm

LinusP wrote:
Thu Nov 14, 2019 7:17 am

In regards to GUI’s this is not what python is for, tkinter will make you want to kill yourself. Instead you should look at building a web app on top of an API or database, this is my setup:
That's a pity, when I started botting (pre api) I choose php simply because I needed some application that was free and able to scrape webpages. Php was the obvious choice as it was free and hundreds of coding smippets around as it was used to drive a lot of websites. The fact I could echo or print data to screen was a real bonus to me when the API came long to understand the structure of the returned data. Once things are running smoothly I happily lave them to it but some still echo to screen and run off meta refreshes so you can see how old they are.

Hardest part I've always found with botting is actually connecting with the API and the placebet/cancel/ bet routines, once you have those in place the rest is usually easily sorted with a few web searches. So the fact you've done the hard work and stuck the engine up for free is a real temptation to finally upgrade my coding. I'll have to have a look at tkinter or see if I can find some way of connecting python to a web page for debugging etc
Post Reply

Return to “Betfair Exchange API”