The best part of being a relative newbie on here is these brilliant topics from before my time which pop up every so often
Brilliant work, Welshboy!
An ever so slightly off topic question for ya. Whoscored.com has some stats, I would love to delve more into. Before I try to teach myself to scrape data, though, am I correct in assuming that they've specifically coded the site to make it highly problematic to scrape it for data?
Football Data (CSV, JSON) - UPDATED 16/08/17
-
- Posts: 3
- Joined: Thu Nov 09, 2017 4:34 pm
Whoscored happens to be the website I value the most but yes, unfortunately, it is very difficult to scrape!
Betfair do offer what I am trying to find (historical HT odds) but I am not really sure I am willing to pay £200/month for it at this stage in my trading career!
Ah well...
Betfair do offer what I am trying to find (historical HT odds) but I am not really sure I am willing to pay £200/month for it at this stage in my trading career!
Ah well...
Thanks to the poster on this one good stuff for us newbies.
I have a question if I am looking to back the draw in a certain match and want to work out roughly when the first 1st goal would be scored.
Would you seasoned football traders look at the specific games involving the 2 teams (as this could be only 2 matches) or would you look at all games played by each team in this league and then take an average from that wider sample?
I have a question if I am looking to back the draw in a certain match and want to work out roughly when the first 1st goal would be scored.
Would you seasoned football traders look at the specific games involving the 2 teams (as this could be only 2 matches) or would you look at all games played by each team in this league and then take an average from that wider sample?
-
- Posts: 4
- Joined: Mon Nov 06, 2017 4:47 pm
Link not working.welshboy06 wrote: ↑Sun Aug 13, 2017 9:45 amHi All,
UPDATED 16/08/17
ADDED CSV VERSIONS OF ALL LEAGUES
I've decided to start collecting football data. Mainly because I have an interest in the sport and also because of a certain jonnyg throwing stats around in a very hard to read format.
The data is quite simple and was just scraped from HKJC.
The data is in json format, but you should be able to convert it to csv and import it to Excel. I chose json as it reduces data duplication, is structured and can be read reasonably well by the human eye. It also works really well with Python (Which I'm using to scrape and also analyse the data)
The data contains the following info.I've managed to scrape ALL leagues and Seasons from HKJC (Where specific goal time data is provided)Code: Select all
League Season Game Date Home Team Away Team And also Goals and Red cards (Player, Time and Team) ----- Soon: Total FT/HT Goals and Red Cards for HT and AT (Included in CSV's)
I may decide to scrape more data, but for most leagues there is 10+ years worth of games, with goal and red card time data.
The data is a little too big to post here, and I didn't fancy splitting everything into multiple zip files. So I've uploaded and shared to my Dropbox
Not sure if the mods/admins could increase the file size limit, so I can add directly to the thread.
The below is a direct link to the folder containing all the data I've scraped.
Each league is in a separate text file, and I've also included a .zip of ALL the files.
https://www.dropbox.com/sh/7sif9n90ehys ... bd6ba?dl=0
Contents of the above filesOkay so there are now two sets of CSV files.Code: Select all
Number of Countries: 20 Country: Argentine - Leagues: 1 League Name: Argentine Division 1 - Seasons: 11 (2009-10(Au), 2010-11 (Sp), 2010-11(Au), 2011-12(Au), 2012-13 (In), 2012-13(Fi), 2013-14 (In), 2013-14(Fi), 2014-15 (In), 2015, 2016) Country: Australian - Leagues: 2 League Name: Australian Division 1 Playoffs - Seasons: 7 (2008-09, 2009-10, 2010-11, 2011-12, 2012-13, 2013-14, 2014-15) League Name: Australian Division 1 - Seasons: 9 (2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017) Country: Belgian - Leagues: 3 League Name: Belgian Division 1 Championship Playoffs - Seasons: 1 (2016) League Name: Belgian Division 1 UE Cup Playoffs - Seasons: 1 (2016) League Name: Belgian Division 1 - Seasons: 2 (2015-2016, 2016-2017) Country: Brazilian - Leagues: 3 League Name: Brazilian Division 1 - Seasons: 9 (2005, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016) League Name: Brazilian Paulista League Knockout stage - Seasons: 1 (2014) League Name: Brazilian Paulista League - Seasons: 1 (2014) Country: Chilean - Leagues: 1 League Name: Chilean Division 1 - Seasons: 6 (2014-15(AP), 2014-15(CL), 2015-16(AP), 2015-16(CL), 2016-17(AP), 2016-17(CL)) Country: Dutch - Leagues: 3 League Name: Dutch Cup - Seasons: 7 (2005-06, 2007-08, 2008-09, 2009-10, 2010-11, 2011-12, 2012-13) League Name: Dutch Division 1 - Seasons: 10 (2005-2006, 2006-2007, 2008-2009, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017) League Name: Dutch Division 2 - Seasons: 6 (2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017) Country: Eng - Leagues: 4 League Name: Eng Championship - Seasons: 13 (2004-2005, 2005-2006, 2006-2007, 2007-2008, 2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017) League Name: Eng League 1 - Seasons: 8 (2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017) League Name: Eng League Cup - Seasons: 6 (2006-07, 2007-08, 2009-10, 2010-11, 2012-13, 2013-14) League Name: Eng Premier - Seasons: 14 (2003-2004, 2004-2005, 2005-2006, 2006-2007, 2007-2008, 2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017) Country: French - Leagues: 4 League Name: French Division 1 - Seasons: 12 (2005-2006, 2006-2007, 2007-2008, 2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017) League Name: French Division 2 - Seasons: 6 (2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017) League Name: French FA Cup - Seasons: 1 (2005-06) League Name: French League Cup - Seasons: 1 (2008-09) Country: German - Leagues: 3 League Name: German Cup - Seasons: 5 (2006-07, 2007-08, 2008-09, 2012-13, 2013-14) League Name: German Division 1 - Seasons: 14 (2003-2004, 2004-2005, 2005-2006, 2006-2007, 2007-2008, 2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017) League Name: German Division 2 - Seasons: 8 (2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017) Country: Italian - Leagues: 2 League Name: Italian Cup - Seasons: 2 (2005-06, 2007-08) League Name: Italian Division 1 - Seasons: 14 (2003-2004, 2004-2005, 2005-2006, 2006-2007, 2007-2008, 2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017) Country: Japanese - Leagues: 3 League Name: Japanese Division 1 - Seasons: 14 (2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015-Stage 1, 2015-Stage 2, 2016-Stage 1, 2016-Stage 2) League Name: Japanese Division 2 - Seasons: 9 (2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016) League Name: Japanese League Cup - Seasons: 9 (2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014) Country: Korean - Leagues: 1 League Name: Korean Division 1 - Seasons: 1 (2016) Country: Mexican - Leagues: 1 League Name: Mexican Premier - Seasons: 6 (2014-15 (AP), 2014-15(CL), 2015-16(AP), 2015-16(CL), 2016-17(AP), 2016-17(CL)) Country: Norwegian - Leagues: 1 League Name: Norwegian Division 1 - Seasons: 12 (2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016) Country: Portuguese - Leagues: 1 League Name: Portuguese Premier - Seasons: 9 (2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017) Country: Russian - Leagues: 1 League Name: Russian Premier - Seasons: 3 (2014-2015, 2015-2016, 2016-2017) Country: Scottish - Leagues: 2 League Name: Scottish League Cup - Seasons: 5 (2005-06, 2007-08, 2008-09, 2009-10, 2012-13) League Name: Scottish Premier - Seasons: 12 (2005-2006, 2006-2007, 2007-2008, 2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017) Country: Spanish - Leagues: 1 League Name: Spanish Division 1 - Seasons: 13 (2003-2004, 2004-2005, 2005-2006, 2006-2007, 2007-2008, 2009-2010, 2010-2011, 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016, 2016-2017) Country: Swedish - Leagues: 1 League Name: Swedish Division 1 - Seasons: 12 (2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016) Country: US - Leagues: 2 League Name: US Football League Playoffs - Seasons: 8 (2007, 2009, 2010, 2011, 2012, 2013, 2014, 2015) League Name: US Football League - Seasons: 10 (2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016)
One with reds and goals separated, and one where they are together as "Events" The columns for each are as follows:
Goals and Reds SeparatedCode: Select all
SEASON, DATE, HOME, AWAY, HHT_GOALS, AHT_GOALS, HFT_GOALS, AFT_GOALS, TOTAL_GOALS, HOME_REDS, AWAY_REDS, TOTAL_REDS, RED_1_TIME, RED_1_TEAM, RED_2_TIME, RED_2_TEAM, RED_3_TIME, RED_3_TEAM, RED_4_TIME, RED_4_TEAM, RED_5_TIME, RED_5_TEAM, RED_6_TIME, RED_6_TEAM, RED_7_TIME, RED_7_TEAM, RED_8_TIME, RED_8_TEAM, RED_9_TIME, RED_9_TEAM, RED_10_TIME, RED_10_TEAM, RED_11_TIME, RED_11_TEAM, RED_12_TIME, RED_12_TEAM, RED_13_TIME, RED_13_TEAM, RED_14_TIME, RED_14_TEAM, RED_15_TIME, RED_15_TEAM, GOAL_1_TIME, GOAL_1_TEAM, GOAL_2_TIME, GOAL_2_TEAM, GOAL_3_TIME, GOAL_3_TEAM, GOAL_4_TIME, GOAL_4_TEAM, GOAL_5_TIME, GOAL_5_TEAM, GOAL_6_TIME, GOAL_6_TEAM, GOAL_7_TIME, GOAL_7_TEAM, GOAL_8_TIME, GOAL_8_TEAM, GOAL_9_TIME, GOAL_9_TEAM, GOAL_10_TIME, GOAL_10_TEAM, GOAL_11_TIME, GOAL_11_TEAM, GOAL_12_TIME, GOAL_12_TEAM, GOAL_13_TIME, GOAL_13_TEAM, GOAL_14_TIME, GOAL_14_TEAM, GOAL_15_TIME, GOAL_15_TEAM
Goals and Reds Together as EventsPlease let me know if I should make any changes to the below columns, or the CSV's at allCode: Select all
SEASON, DATE, HOME, AWAY, HHT_GOALS, AHT_GOALS, HFT_GOALS, AFT_GOALS, TOTAL_GOALS, HOME_REDS, AWAY_REDS, TOTAL_REDS, EVENT_1_TIME, EVENT_1_TEAM, EVENT_1_TYPE, EVENT_2_TIME, EVENT_2_TEAM, EVENT_2_TYPE, EVENT_3_TIME, EVENT_3_TEAM, EVENT_3_TYPE, EVENT_4_TIME, EVENT_4_TEAM, EVENT_4_TYPE, EVENT_5_TIME, EVENT_5_TEAM, EVENT_5_TYPE, EVENT_6_TIME, EVENT_6_TEAM, EVENT_6_TYPE, EVENT_7_TIME, EVENT_7_TEAM, EVENT_7_TYPE, EVENT_8_TIME, EVENT_8_TEAM, EVENT_8_TYPE, EVENT_9_TIME, EVENT_9_TEAM, EVENT_9_TYPE, EVENT_10_TIME, EVENT_10_TEAM, EVENT_10_TYPE, EVENT_11_TIME, EVENT_11_TEAM, EVENT_11_TYPE, EVENT_12_TIME, EVENT_12_TEAM, EVENT_12_TYPE, EVENT_13_TIME, EVENT_13_TEAM, EVENT_13_TYPE, EVENT_14_TIME, EVENT_14_TEAM, EVENT_14_TYPE, EVENT_15_TIME, EVENT_15_TEAM, EVENT_15_TYPE, EVENT_16_TIME, EVENT_16_TEAM, EVENT_16_TYPE, EVENT_17_TIME, EVENT_17_TEAM, EVENT_17_TYPE, EVENT_18_TIME, EVENT_18_TEAM, EVENT_18_TYPE, EVENT_19_TIME, EVENT_19_TEAM, EVENT_19_TYPE, EVENT_20_TIME, EVENT_20_TEAM, EVENT_20_TYPE, EVENT_21_TIME, EVENT_21_TEAM, EVENT_21_TYPE, EVENT_22_TIME, EVENT_22_TEAM, EVENT_22_TYPE, EVENT_23_TIME, EVENT_23_TEAM, EVENT_23_TYPE, EVENT_24_TIME, EVENT_24_TEAM, EVENT_24_TYPE, EVENT_25_TIME, EVENT_25_TEAM, EVENT_25_TYPE, EVENT_26_TIME, EVENT_26_TEAM, EVENT_26_TYPE, EVENT_27_TIME, EVENT_27_TEAM, EVENT_27_TYPE, EVENT_28_TIME, EVENT_28_TEAM, EVENT_28_TYPE, EVENT_29_TIME, EVENT_29_TEAM, EVENT_29_TYPE, EVENT_30_TIME, EVENT_30_TEAM, EVENT_30_TYPE
I've tried, best I can, to go through the data and make sure all games are there. I've not spotted any leagues with missing games as of yet. BUT PLEASE LET ME KNOW IF YOU SPOT ANY ISSUES.
Just to note: The data is all scraped from HKJC, so if there are any errors it would be down to the data they provided. (The dates I've been told are in HK time)
Cheers,
Adam
I've just tried it and its still active and working
-
- Posts: 4
- Joined: Mon Nov 06, 2017 4:47 pm
-
- Posts: 1
- Joined: Sat Dec 30, 2017 6:09 am
Hi guys, noob post here. How can I see goal times using the data that welshboy has scraped? HHT,AHT,HFT, and AFT I have no idea what they mean. Thanks in advance for any help as it would be greatly appreciated.
It will be the number of goalsluckycharms365 wrote: ↑Sun Dec 31, 2017 10:12 pmHi guys, noob post here. How can I see goal times using the data that welshboy has scraped? HHT,AHT,HFT, and AFT I have no idea what they mean. Thanks in advance for any help as it would be greatly appreciated.
Home Half Time
Away Half Time
Home Full Time
Away Full Time
Hi guys
I'm very interested in this topic (one of the reasons why i entered on BetAngel Forum ). I think would be interesting to have data from the initial odds from Over/Under Markets (0.5; 1.5; 2.5; etc.) and "Cards markets" (i don't know exactly the name for it, sorry) to try to find some correlation. At least to find a possible strategy based on this data. Do you think it's possible to scrap this data? I think oddsportal have some historical data but i don't know if it's possible or not to scrap data from there. At least it's possible to find some Over/Under odds.
Cheers guys
I'm very interested in this topic (one of the reasons why i entered on BetAngel Forum ). I think would be interesting to have data from the initial odds from Over/Under Markets (0.5; 1.5; 2.5; etc.) and "Cards markets" (i don't know exactly the name for it, sorry) to try to find some correlation. At least to find a possible strategy based on this data. Do you think it's possible to scrap this data? I think oddsportal have some historical data but i don't know if it's possible or not to scrap data from there. At least it's possible to find some Over/Under odds.
Cheers guys
Has anyone been able to work out how to use the New Betfair Historical data website, Ive managed to download a .tar file and extracted it but its all gobbledeegoop. All I get is the .bz2 files in text file
You do not have the required permissions to view the files attached to this post.
I think megarain once wrote a detailed post on how to open Json files in SQL in a betfair data thread.
I use excel myself so never had any need to open Json files
-
- Posts: 3140
- Joined: Sun Jan 31, 2010 8:06 pm
You can import json into excel but alot is dependant on your version of excel, just look on youtube they'll be lots of tutorials how to imort json files into your version of excel. There are even plenty of online or standalone convertors for json -> csv files out there too. No idea how big the files are and it may well be that you'd be better off using Access or some SQL database to query them