Football Data (CSV, JSON) - UPDATED 16/08/17

Post Reply
User avatar
Kafkaesque
Posts: 886
Joined: Fri Oct 06, 2017 10:20 am

The best part of being a relative newbie on here is these brilliant topics from before my time which pop up every so often :D

Brilliant work, Welshboy!

An ever so slightly off topic question for ya. Whoscored.com has some stats, I would love to delve more into. Before I try to teach myself to scrape data, though, am I correct in assuming that they've specifically coded the site to make it highly problematic to scrape it for data?
elliotarcher1
Posts: 3
Joined: Thu Nov 09, 2017 4:34 pm

Whoscored happens to be the website I value the most but yes, unfortunately, it is very difficult to scrape!

Betfair do offer what I am trying to find (historical HT odds) but I am not really sure I am willing to pay £200/month for it at this stage in my trading career!

Ah well...
Angelbaby
Posts: 55
Joined: Wed Nov 08, 2017 7:49 pm

Thanks to the poster on this one good stuff for us newbies.

I have a question if I am looking to back the draw in a certain match and want to work out roughly when the first 1st goal would be scored.

Would you seasoned football traders look at the specific games involving the 2 teams (as this could be only 2 matches) or would you look at all games played by each team in this league and then take an average from that wider sample?
rolandimone
Posts: 4
Joined: Mon Nov 06, 2017 4:47 pm

welshboy06 wrote:
Sun Aug 13, 2017 9:45 am
Hi All,

UPDATED 16/08/17
ADDED CSV VERSIONS OF ALL LEAGUES

I've decided to start collecting football data. Mainly because I have an interest in the sport and also because of a certain jonnyg throwing stats around in a very hard to read format.

The data is quite simple and was just scraped from HKJC.

The data is in json format, but you should be able to convert it to csv and import it to Excel. I chose json as it reduces data duplication, is structured and can be read reasonably well by the human eye. It also works really well with Python (Which I'm using to scrape and also analyse the data)
The data contains the following info.

Code: Select all

League
Season
Game Date
Home Team
Away Team
And also Goals and Red cards (Player, Time and Team)
-----
Soon:
Total FT/HT Goals and Red Cards for HT and AT (Included in CSV's)
I've managed to scrape ALL leagues and Seasons from HKJC (Where specific goal time data is provided)
I may decide to scrape more data, but for most leagues there is 10+ years worth of games, with goal and red card time data.

The data is a little too big to post here, and I didn't fancy splitting everything into multiple zip files. So I've uploaded and shared to my Dropbox
Not sure if the mods/admins could increase the file size limit, so I can add directly to the thread.
The below is a direct link to the folder containing all the data I've scraped.
Each league is in a separate text file, and I've also included a .zip of ALL the files.

https://www.dropbox.com/sh/7sif9n90ehys ... bd6ba?dl=0

Contents of the above files

Code: Select all

Number of Countries: 20
Country: Argentine - Leagues: 1
League Name: Argentine Division 1 - Seasons: 11
(2009-10(Au), 2010-11 (Sp), 2010-11(Au), 2011-12(Au), 2012-13 (In), 2012-13(Fi), 2013-14 (In), 2013-14(Fi), 2014-15 (In), 2015, 2016)
Country: Australian - Leagues: 2
League Name: Australian Division 1 Playoffs - Seasons: 7
(2008-09, 2009-10, 2010-11, 2011-12, 2012-13, 2013-14, 2014-15)
League Name: Australian Division 1 - Seasons: 9
(2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017)
Country: Belgian - Leagues: 3
League Name: Belgian Division 1 Championship Playoffs - Seasons: 1
(2016)
League Name: Belgian Division 1 UE Cup Playoffs - Seasons: 1
(2016)
League Name: Belgian Division 1 - Seasons: 2
(2015-2016, 2016-2017)
Country: Brazilian - Leagues: 3
League Name: Brazilian Division 1 - Seasons: 9
(2005, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016)
League Name: Brazilian Paulista League Knockout stage - Seasons: 1
(2014)
League Name: Brazilian Paulista League - Seasons: 1
(2014)
Country: Chilean - Leagues: 1
League Name: Chilean Division 1 - Seasons: 6
(2014-15(AP), 2014-15(CL), 2015-16(AP), 2015-16(CL), 2016-17(AP), 2016-17(CL))
Country: Dutch - Leagues: 3
League Name: Dutch Cup - Seasons: 7
(2005-06, 2007-08, 2008-09, 2009-10, 2010-11, 2011-12, 2012-13)
League Name: Dutch Division 1 - Seasons: 10
(2005-2006, 2006-2007, 2008-2009, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017)
League Name: Dutch Division 2 - Seasons: 6
(2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017)
Country: Eng - Leagues: 4
League Name: Eng Championship - Seasons: 13
(2004-2005, 2005-2006, 2006-2007, 2007-2008, 2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017)
League Name: Eng League 1 - Seasons: 8
(2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017)
League Name: Eng League Cup - Seasons: 6
(2006-07, 2007-08, 2009-10, 2010-11, 2012-13, 2013-14)
League Name: Eng Premier - Seasons: 14
(2003-2004, 2004-2005, 2005-2006, 2006-2007, 2007-2008, 2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017)
Country: French - Leagues: 4
League Name: French Division 1 - Seasons: 12
(2005-2006, 2006-2007, 2007-2008, 2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017)
League Name: French Division 2 - Seasons: 6
(2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017)
League Name: French FA Cup - Seasons: 1
(2005-06)
League Name: French League Cup - Seasons: 1
(2008-09)
Country: German - Leagues: 3
League Name: German Cup - Seasons: 5
(2006-07, 2007-08, 2008-09, 2012-13, 2013-14)
League Name: German Division 1 - Seasons: 14
(2003-2004, 2004-2005, 2005-2006, 2006-2007, 2007-2008, 2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017)
League Name: German Division 2 - Seasons: 8
(2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017)
Country: Italian - Leagues: 2
League Name: Italian Cup - Seasons: 2
(2005-06, 2007-08)
League Name: Italian Division 1 - Seasons: 14
(2003-2004, 2004-2005, 2005-2006, 2006-2007, 2007-2008, 2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017)
Country: Japanese - Leagues: 3
League Name: Japanese Division 1 - Seasons: 14
(2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015-Stage 1, 2015-Stage 2, 2016-Stage 1, 2016-Stage 2)
League Name: Japanese Division 2 - Seasons: 9
(2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016)
League Name: Japanese League Cup - Seasons: 9
(2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014)
Country: Korean - Leagues: 1
League Name: Korean Division 1 - Seasons: 1
(2016)
Country: Mexican - Leagues: 1
League Name: Mexican Premier - Seasons: 6
(2014-15 (AP), 2014-15(CL), 2015-16(AP), 2015-16(CL), 2016-17(AP), 2016-17(CL))
Country: Norwegian - Leagues: 1
League Name: Norwegian Division 1 - Seasons: 12
(2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016)
Country: Portuguese - Leagues: 1
League Name: Portuguese Premier - Seasons: 9
(2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017)
Country: Russian - Leagues: 1
League Name: Russian Premier - Seasons: 3
(2014-2015, 2015-2016, 2016-2017)
Country: Scottish - Leagues: 2
League Name: Scottish League Cup - Seasons: 5
(2005-06, 2007-08, 2008-09, 2009-10, 2012-13)
League Name: Scottish Premier - Seasons: 12
(2005-2006, 2006-2007, 2007-2008, 2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017)
Country: Spanish - Leagues: 1
League Name: Spanish Division 1 - Seasons: 13
(2003-2004, 2004-2005, 2005-2006, 2006-2007, 2007-2008, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017)
Country: Swedish - Leagues: 1
League Name: Swedish Division 1 - Seasons: 12
(2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016)
Country: US - Leagues: 2
League Name: US Football League Playoffs - Seasons: 8
(2007, 2009, 2010, 2011, 2012, 2013, 2014, 2015)
League Name: US Football League - Seasons: 10
(2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016)
Okay so there are now two sets of CSV files.
One with reds and goals separated, and one where they are together as "Events" The columns for each are as follows:


Goals and Reds Separated

Code: Select all

SEASON, DATE, HOME, AWAY, HHT_GOALS, AHT_GOALS, HFT_GOALS, AFT_GOALS, TOTAL_GOALS, HOME_REDS, AWAY_REDS, TOTAL_REDS, RED_1_TIME, RED_1_TEAM, RED_2_TIME, RED_2_TEAM, RED_3_TIME, RED_3_TEAM, RED_4_TIME, RED_4_TEAM, RED_5_TIME, RED_5_TEAM, RED_6_TIME, RED_6_TEAM, RED_7_TIME, RED_7_TEAM, RED_8_TIME, RED_8_TEAM, RED_9_TIME, RED_9_TEAM, RED_10_TIME, RED_10_TEAM, RED_11_TIME, RED_11_TEAM, RED_12_TIME, RED_12_TEAM, RED_13_TIME, RED_13_TEAM, RED_14_TIME, RED_14_TEAM, RED_15_TIME, RED_15_TEAM, GOAL_1_TIME, GOAL_1_TEAM, GOAL_2_TIME, GOAL_2_TEAM, GOAL_3_TIME, GOAL_3_TEAM, GOAL_4_TIME, GOAL_4_TEAM, GOAL_5_TIME, GOAL_5_TEAM, GOAL_6_TIME, GOAL_6_TEAM, GOAL_7_TIME, GOAL_7_TEAM, GOAL_8_TIME, GOAL_8_TEAM, GOAL_9_TIME, GOAL_9_TEAM, GOAL_10_TIME, GOAL_10_TEAM, GOAL_11_TIME, GOAL_11_TEAM, GOAL_12_TIME, GOAL_12_TEAM, GOAL_13_TIME, GOAL_13_TEAM, GOAL_14_TIME, GOAL_14_TEAM, GOAL_15_TIME, GOAL_15_TEAM

Goals and Reds Together as Events

Code: Select all

SEASON, DATE, HOME, AWAY, HHT_GOALS, AHT_GOALS, HFT_GOALS, AFT_GOALS, TOTAL_GOALS, HOME_REDS, AWAY_REDS, TOTAL_REDS, EVENT_1_TIME, EVENT_1_TEAM, EVENT_1_TYPE, EVENT_2_TIME, EVENT_2_TEAM, EVENT_2_TYPE, EVENT_3_TIME, EVENT_3_TEAM, EVENT_3_TYPE, EVENT_4_TIME, EVENT_4_TEAM, EVENT_4_TYPE, EVENT_5_TIME, EVENT_5_TEAM, EVENT_5_TYPE, EVENT_6_TIME, EVENT_6_TEAM, EVENT_6_TYPE, EVENT_7_TIME, EVENT_7_TEAM, EVENT_7_TYPE, EVENT_8_TIME, EVENT_8_TEAM, EVENT_8_TYPE, EVENT_9_TIME, EVENT_9_TEAM, EVENT_9_TYPE, EVENT_10_TIME, EVENT_10_TEAM, EVENT_10_TYPE, EVENT_11_TIME, EVENT_11_TEAM, EVENT_11_TYPE, EVENT_12_TIME, EVENT_12_TEAM, EVENT_12_TYPE, EVENT_13_TIME, EVENT_13_TEAM, EVENT_13_TYPE, EVENT_14_TIME, EVENT_14_TEAM, EVENT_14_TYPE, EVENT_15_TIME, EVENT_15_TEAM, EVENT_15_TYPE, EVENT_16_TIME, EVENT_16_TEAM, EVENT_16_TYPE, EVENT_17_TIME, EVENT_17_TEAM, EVENT_17_TYPE, EVENT_18_TIME, EVENT_18_TEAM, EVENT_18_TYPE, EVENT_19_TIME, EVENT_19_TEAM, EVENT_19_TYPE, EVENT_20_TIME, EVENT_20_TEAM, EVENT_20_TYPE, EVENT_21_TIME, EVENT_21_TEAM, EVENT_21_TYPE, EVENT_22_TIME, EVENT_22_TEAM, EVENT_22_TYPE, EVENT_23_TIME, EVENT_23_TEAM, EVENT_23_TYPE, EVENT_24_TIME, EVENT_24_TEAM, EVENT_24_TYPE, EVENT_25_TIME, EVENT_25_TEAM, EVENT_25_TYPE, EVENT_26_TIME, EVENT_26_TEAM, EVENT_26_TYPE, EVENT_27_TIME, EVENT_27_TEAM, EVENT_27_TYPE, EVENT_28_TIME, EVENT_28_TEAM, EVENT_28_TYPE, EVENT_29_TIME, EVENT_29_TEAM, EVENT_29_TYPE, EVENT_30_TIME, EVENT_30_TEAM, EVENT_30_TYPE
Please let me know if I should make any changes to the below columns, or the CSV's at all
I've tried, best I can, to go through the data and make sure all games are there. I've not spotted any leagues with missing games as of yet. BUT PLEASE LET ME KNOW IF YOU SPOT ANY ISSUES.

Just to note: The data is all scraped from HKJC, so if there are any errors it would be down to the data they provided. (The dates I've been told are in HK time)

Cheers,
Adam
Link not working.
User avatar
Dallas
Posts: 22673
Joined: Sun Aug 09, 2015 10:57 pm
Location: Working From Home

rolandimone wrote:
Thu Nov 23, 2017 12:02 pm

Link not working.
I've just tried it and its still active and working
rolandimone
Posts: 4
Joined: Mon Nov 06, 2017 4:47 pm

Dallas wrote:
Thu Nov 23, 2017 12:06 pm
rolandimone wrote:
Thu Nov 23, 2017 12:02 pm

Link not working.
I've just tried it and its still active and working
Thanks problem was with my IP provider. It works when I turned VPN on.
luckycharms365
Posts: 1
Joined: Sat Dec 30, 2017 6:09 am

Hi guys, noob post here. How can I see goal times using the data that welshboy has scraped? HHT,AHT,HFT, and AFT I have no idea what they mean. Thanks in advance for any help as it would be greatly appreciated.
User avatar
Dallas
Posts: 22673
Joined: Sun Aug 09, 2015 10:57 pm
Location: Working From Home

luckycharms365 wrote:
Sun Dec 31, 2017 10:12 pm
Hi guys, noob post here. How can I see goal times using the data that welshboy has scraped? HHT,AHT,HFT, and AFT I have no idea what they mean. Thanks in advance for any help as it would be greatly appreciated.
It will be the number of goals
Home Half Time
Away Half Time
Home Full Time
Away Full Time
Dendy
Posts: 1
Joined: Thu Jan 18, 2018 3:24 pm

Hi guys

I'm very interested in this topic (one of the reasons why i entered on BetAngel Forum :D). I think would be interesting to have data from the initial odds from Over/Under Markets (0.5; 1.5; 2.5; etc.) and "Cards markets" (i don't know exactly the name for it, sorry) to try to find some correlation. At least to find a possible strategy based on this data. Do you think it's possible to scrap this data? I think oddsportal have some historical data but i don't know if it's possible or not to scrap data from there. At least it's possible to find some Over/Under odds.

Cheers guys ;)
User avatar
gstar1975
Posts: 615
Joined: Thu Nov 24, 2011 11:59 am

Has anyone been able to work out how to use the New Betfair Historical data website, Ive managed to download a .tar file and extracted it but its all gobbledeegoop. All I get is the .bz2 files in text file
You do not have the required permissions to view the files attached to this post.
LinusP
Posts: 1871
Joined: Mon Jul 02, 2012 10:45 pm

gstar1975 wrote:
Mon May 14, 2018 1:01 pm
Has anyone been able to work out how to use the New Betfair Historical data website, Ive managed to download a .tar file and extracted it but its all gobbledeegoop. All I get is the .bz2 files in text file
You need to extract the bz2 file.
User avatar
gstar1975
Posts: 615
Joined: Thu Nov 24, 2011 11:59 am

I have extracted the .Bz2 file but they extract as a .txt file

Can you take me through the exact steps please?
User avatar
gstar1975
Posts: 615
Joined: Thu Nov 24, 2011 11:59 am

gstar1975 wrote:
Tue Jun 05, 2018 10:09 am
I have extracted the .Bz2 file but they extract as a .txt file

Can you take me through the exact steps please?
Can someone please do a video on this please???
User avatar
Dallas
Posts: 22673
Joined: Sun Aug 09, 2015 10:57 pm
Location: Working From Home

gstar1975 wrote:
Tue Jun 05, 2018 10:09 am
I have extracted the .Bz2 file but they extract as a .txt file

Can you take me through the exact steps please?
I think megarain once wrote a detailed post on how to open Json files in SQL in a betfair data thread.

I use excel myself so never had any need to open Json files
spreadbetting
Posts: 3140
Joined: Sun Jan 31, 2010 8:06 pm

You can import json into excel but alot is dependant on your version of excel, just look on youtube they'll be lots of tutorials how to imort json files into your version of excel. There are even plenty of online or standalone convertors for json -> csv files out there too. No idea how big the files are and it may well be that you'd be better off using Access or some SQL database to query them
Post Reply

Return to “Betfair Data”