CONFUSED BY BETFAIR FOOTBALL DATA

Post Reply
neilovan
Posts: 13
Joined: Sun Nov 18, 2018 11:12 am

Hello All,

I am trying to acquire betfair data, specifically OVER 2.5 goal market data. I recently downloaded some TAR directories and files. I wrote a python script to extract the files and put their contents (about 550 directories) into one file. But looking at this file (that contains all the data) I am somewhat disappointed ... Kind of looks like this


4611115274,"[{u'marketDefinition': {u'discountAllowed': True, u'countryCode': u'GB', u'marketType': u'OVER_UNDER_25', u'marketBaseRate': 5.0, u'crossMatching': False, u'turnInPlayEnabled': True, u'timezone': u'Europe/London', u'bspMarket': False, u'eventId': u'28271222', u'marketTime': u'2017-08-12T14:00:00.000Z', u'persistenceEnabled': True, u'eventName': u'Arsenal v Leicester City', u'version': 1676850071, u'bettingType': u'ODDS', u'inPlay': False, u'status': u'OPEN', u'suspendTime': u'2017-08-12T14:00:00.000Z', u'complete': True, u'numberOfActiveRunners': 2, u'runnersVoidable': False, u'betDelay': 0, u'regulators': [u'MR_INT'], u'openDate': u'2017-08-12T14:00:00.000Z', u'runners': [{u'status': u'ACTIVE', u'sortPriority': 1, u'id': 47972, u'name': u'Under 2.5 Goals'}, {u'status': u'ACTIVE', u'sortPriority': 2, u'id': 47973, u'name': u'Over 2.5 Goals'}], u'name': u'Over/Under 2.5 Goals', u'bspReconciled': False, u'eventTypeId': u'1', u'numberOfWinners': 1}, u'id': u'1.132168328'}]",mcm,1497439675419
4622621371,"[{u'marketDefinition': {u'discountAllowed': True, u'countryCode': u'GB', u'marketType': u'OVER_UNDER_25', u'marketBaseRate': 5.0, u'crossMatching': True, u'turnInPlayEnabled': True, u'timezone': u'Europe/London', u'bspMarket': False, u'eventId': u'28271222', u'marketTime': u'2017-08-12T14:00:00.000Z', u'persistenceEnabled': True, u'eventName': u'Arsenal v Leicester City', u'version': 1676850071, u'bettingType': u'ODDS', u'inPlay': False, u'status': u'OPEN', u'suspendTime': u'2017-08-12T14:00:00.000Z', u'complete': True, u'numberOfActiveRunners': 2, u'runnersVoidable': False, u'betDelay': 0, u'regulators': [u'MR_INT'], u'openDate': u'2017-08-12T14:00:00.000Z', u'runners': [{u'status': u'ACTIVE', u'sortPriority': 1, u'id': 47972, u'name': u'Under 2.5 Goals'}, {u'status': u'ACTIVE', u'sortPriority': 2, u'id': 47973, u'name': u'Over 2.5 Goals'}], u'name': u'Over/Under 2.5 Goals', u'bspReconciled': False, u'eventTypeId': u'1', u'numberOfWinners': 1}, u'id': u'1.132168328'}]",mcm,1497645853032
4685208083,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.53, u'id': 47972}]}]",mcm,1498827468430
4685229667,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.3, u'id': 47972}]}]",mcm,1498827641788
4685234047,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.4, u'id': 47973}]}]",mcm,1498827705158
4714131637,"[{u'marketDefinition': {u'discountAllowed': True, u'countryCode': u'GB', u'marketType': u'OVER_UNDER_25', u'marketBaseRate': 5.0, u'crossMatching': True, u'turnInPlayEnabled': True, u'timezone': u'Europe/London', u'bspMarket': False, u'eventId': u'28271222', u'marketTime': u'2017-08-11T18:45:00.000Z', u'persistenceEnabled': True, u'eventName': u'Arsenal v Leicester City', u'version': 1693360159, u'bettingType': u'ODDS', u'inPlay': False, u'status': u'OPEN', u'suspendTime': u'2017-08-11T18:45:00.000Z', u'complete': True, u'numberOfActiveRunners': 2, u'runnersVoidable': False, u'betDelay': 0, u'regulators': [u'MR_INT'], u'openDate': u'2017-08-11T18:45:00.000Z', u'runners': [{u'status': u'ACTIVE', u'sortPriority': 1, u'id': 47972, u'name': u'Under 2.5 Goals'}, {u'status': u'ACTIVE', u'sortPriority': 2, u'id': 47973, u'name': u'Over 2.5 Goals'}], u'name': u'Over/Under 2.5 Goals', u'bspReconciled': False, u'eventTypeId': u'1', u'numberOfWinners': 1}, u'id': u'1.132168328'}]",mcm,1499340153754
4794481557,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.0, u'id': 47972}]}]",mcm,1500647920512
4815756165,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.0, u'id': 47973}, {u'ltp': 2.1, u'id': 47972}]}]",mcm,1500977934003


I can see that some rows look to be back price and other potentially back and lay price rows. IS this it or am I missing a step/s somewhere along the line. AM I on totally the wrong path here ?

Any help or direction adjustments appreciated.

Thanks,

Neil.
neilovan
Posts: 13
Joined: Sun Nov 18, 2018 11:12 am

ANYBODY :D :lol: :shock:
spreadbetting
Posts: 3140
Joined: Sun Jan 31, 2010 8:06 pm

The data files are in json format, iirc, so just get yourself a phython script to decode the json into some array type you're more familiar working with.
neilovan
Posts: 13
Joined: Sun Nov 18, 2018 11:12 am

spreadbetting wrote:
Fri Jan 18, 2019 1:08 pm
The data files are in json format, iirc, so just get yourself a phython script to decode the json into some array type you're more familiar working with.
Thanks for reply.

I can do that, essentially read the file into a PHP program line by line and split up the data as I like. But the data in the file is not telling me much (this was an over 2.5 goal download that had about 590 directories in the tar file). Is this all betfair has for free in the over 2.5 goal downloads ?
spreadbetting
Posts: 3140
Joined: Sun Jan 31, 2010 8:06 pm

They do different data plans, so the ones they charge for will have more data available. The second page of the specs below will show you what's available in each price plan

https://historicdata.betfair.com/Betfai ... cation.pdf
User avatar
northbound
Posts: 737
Joined: Mon Mar 20, 2017 11:22 pm

Betfair historical data is in the same format as the Betfair Streaming API’s.

This format is not like CSV where on every line you have one selection and its data.

Instead, each line is a JSON document that may contain the list of selections or simply changes to one or more selections’ metrics, e.g. LTP, Traded Volume.

Before running (parsing and using this data), you must learn to walk (understand how data stream messages work, what kind of info each message contains, etc).

Do not spend a penny more purchasing Betfair data until you understand the following:
https://docs.developer.betfair.com/plug ... ew/6094862
neilovan
Posts: 13
Joined: Sun Nov 18, 2018 11:12 am

northbound wrote:
Fri Jan 18, 2019 3:36 pm
Betfair historical data is in the same format as the Betfair Streaming API’s.

This format is not like CSV where on every line you have one selection and its data.

Instead, each line is a JSON document that may contain the list of selections or simply changes to one or more selections’ metrics, e.g. LTP, Traded Volume.

Before running (parsing and using this data), you must learn to walk (understand how data stream messages work, what kind of info each message contains, etc).

Do not spend a penny more purchasing Betfair data until you understand the following:
https://docs.developer.betfair.com/plug ... ew/6094862
Thanks for the reply.

As a layman, I don't think I have enough time in this lifetime to figure this out. Seriously, only one thing to say regarding betfair data .... WTF !

May as well send it in Chinese, which F____g moron thinks of this ?

A couple of questions...assuming by some miracle you could decipher this JSON stream

1) Is it a different format for each market. SO Win , different to CORRECT SCORE, different to OVER 2.5 ?
2) Do you need an API key if you are only using static files downloaded?

Thanks for your interest and responses,

Neil.
User avatar
northbound
Posts: 737
Joined: Mon Mar 20, 2017 11:22 pm

- You don't need an API key to read historical data, but the data format is the same. If you manage to understand Streaming API's, you'll be able to understand historical data.

- The data is in the same format for all markets. Here's a quick explanation about data format.

On Betfair:

Every event (e.g. Bayern v Dortmund) has markets (e.g. Match Odds, Over/Under 2.5 Goals, etc).
Every market (e.g. Match Odds) has a market id (e.g. 1.132199876).
Every market has runners (e.g. Bayern, Dortmund, The Draw).
Every runner has an id (e.g. 47972), which doesn't start with "1." like a market id.

Streaming API and historical data are basically a list of sequential messages.

Each message tells you that something has changed for a specific runner in a specific market.

For example it can tell you that ltp (last traded price) is now 1.50 for runner 47972 of market 1.132199876.

Every line contains a message.

Every message is in JSON format.

The first message usually contains general info such as the market name, id, event id, start time, etc.

Subsequent messages contain changes to odds, volume, etc.

And so on...
neilovan
Posts: 13
Joined: Sun Nov 18, 2018 11:12 am

Thanks for the reply,

If I am looking at a stream of data, similar to the following, is what I see, what I get ?
So ok, each row has a unique identifier
Then a market ID.
Then it either has a last traded price for either outcome (under or over 2.5) 47972 or 47973.

Can I just take a PHP program and parse this information or will JSON give me far more ? Sorry for my stupid question but as a non JSON programmer, I don't really get it.

4794481557,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.0, u'id': 47972}]}]",mcm,1500647920512
4815756165,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.0, u'id': 47973}, {u'ltp': 2.1, u'id': 47972}]}]",mcm,1500977934003
4815758856,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.46, u'id': 47972}]}]",mcm,1500977982821
4815775498,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.36, u'id': 47972}]}]",mcm,1500978159099
4836438162,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.58, u'id': 47972}]}]",mcm,1501316316128
4850966677,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.6, u'id': 47973}, {u'ltp': 2.66, u'id': 47972}]}]",mcm,1501487991266
4858348800,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.62, u'id': 47973}, {u'ltp': 2.6, u'id': 47972}]}]",mcm,1501619815113
4861093323,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.34, u'id': 47972}]}]",mcm,1501672961652
4866622956,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.63, u'id': 47973}, {u'ltp': 2.58, u'id': 47972}]}]",mcm,1501753023516
4869823684,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.52, u'id': 47972}]}]",mcm,1501795374325
4873553143,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.64, u'id': 47973}, {u'ltp': 2.56, u'id': 47972}]}]",mcm,1501859977446
4889376132,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.52, u'id': 47972}]}]",mcm,1502022707535
4890167935,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.52, u'id': 47973}]}]",mcm,1502027765818
4894298918,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.48, u'id': 47972}]}]",mcm,1502064014390
4894308738,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.54, u'id': 47973}]}]",mcm,1502064345882
4895520531,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.6, u'id': 47973}]}]",mcm,1502099448188
4895698422,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.63, u'id': 47973}, {u'ltp': 2.5, u'id': 47972}]}]",mcm,1502102913760
4896238339,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.67, u'id': 47973}]}]",mcm,1502112106383
4898181857,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.61, u'id': 47973}]}]",mcm,1502136410516
4898506447,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.6, u'id': 47973}]}]",mcm,1502143634063
4900011491,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.66, u'id': 47973}]}]",mcm,1502175551290
4900177223,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.63, u'id': 47973}, {u'ltp': 2.58, u'id': 47972}]}]",mcm,1502179375625
4900705076,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.66, u'id': 47973}, {u'ltp': 2.5, u'id': 47972}]}]",mcm,1502188608673
4900890758,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.64, u'id': 47973}, {u'ltp': 2.56, u'id': 47972}]}]",mcm,1502191357217
4900893719,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.65, u'id': 47973}]}]",mcm,1502191426886
4901205144,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.63, u'id': 47973}]}]",mcm,1502196368040
4901948050,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 1.66, u'id': 47973}]}]",mcm,1502204766261

Many thanks,

Neil.
User avatar
northbound
Posts: 737
Joined: Mon Mar 20, 2017 11:22 pm

You can parse JSON by using any programming language, so PHP is fine.

Before coding anything, you need to decide what you want to extract from a market.

High/Low traded odds?

Total traded volume for each runner?
neilovan
Posts: 13
Joined: Sun Nov 18, 2018 11:12 am

northbound wrote:
Fri Jan 18, 2019 11:16 pm
You can parse JSON by using any programming language, so PHP is fine.

Before coding anything, you need to decide what you want to extract from a market.

High/Low traded odds?

Total traded volume for each runner?
But this file does not give me that information.
SO my option is to purchase more in depth data, extract into one outfile using some python script.
Parse this extracted file.

Could anybody post a few lines of what a paid for JSON file looks like from betfair?

Many thanks,

Neil.
neilovan
Posts: 13
Joined: Sun Nov 18, 2018 11:12 am

Hi All,

Can anyone tell me the significance of the numbers 47973 and 47972 in the following lines?

4794481557,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.0, u'id': 47972}]}]",mcm,1500647920512

4815756165,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.0, u'id': 47973}, {u'ltp': 2.1, u'id': 47972}]}]",mcm,1500977934003

Thanks,

Neil.
LinusP
Posts: 1871
Joined: Mon Jul 02, 2012 10:45 pm

neilovan wrote:
Tue Jan 22, 2019 8:13 pm
Hi All,

Can anyone tell me the significance of the numbers 47973 and 47972 in the following lines?

4794481557,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.0, u'id': 47972}]}]",mcm,1500647920512

4815756165,"[{u'id': u'1.132168328', u'rc': [{u'ltp': 2.0, u'id': 47973}, {u'ltp': 2.1, u'id': 47972}]}]",mcm,1500977934003

Thanks,

Neil.
Selection Id, have you read the docs?
neilovan
Posts: 13
Joined: Sun Nov 18, 2018 11:12 am

I think I did, maybe I read wrong document.


it says 'id marketId Unique identifier for the market'

on https://historicdata.betfair.com/Betfai ... cation.pdf

It seems like this file does not tell you the competition.
User avatar
northbound
Posts: 737
Joined: Mon Mar 20, 2017 11:22 pm

Neil, Please look at my previous post on this thread:
https://betangel.com/forum/viewtopic.ph ... 74#p178074

It explains how a market id (e.g. Match Odds market's) differs from a runner id (e.g. Bayern Munich's).
Post Reply

Return to “Betfair Data”