Data scraping Do's & Dont's?

[Brovashift]

Hi all,

Any advice or tips when data scraping? Im thinking along the lines of things to avoid to try and prevent things breaking if/when a site gets updated etc.
I see many things online using Python so was just going to start with that, but if anyone has any better (easier) ideas; in the words of Universal Soldier "Im all ears..."
I like to try and work smarter, not harder lol

TIA

[Frogmella]

A lot of people do this and I've been looking into it myself. The advice seems to be, where possible, instead of scraping use the web-site's own API. Apparently many have them.

[Brovashift]

A lot of people do this and I've been looking into it myself. The advice seems to be, where possible, instead of scraping use the web-site's own API. Apparently many have them.

Humm interesting, thanks for this...

I don't suppose you've got any recommendations have you? I want to pull some tennis data to display on my one-click screen. I was thinking of getting from Tennis Abstract as he collects some pretty unique data on there, but no live in-play player data. Doing a quick google search now Sportsdata.io looks quite promising, but I hate it when they don't give any sort of prices, just "Get In Touch".

[Realrocknrolla]

Sofascore mate!

[Brovashift]

Sofascore mate!

Cheers mate, nice one!!!

This isn't an api no? Just a site to have open, or scrape if necessary... I dont suppose you know anywhere that includes "Unforced Error's" do you? I cant remember where I've seen it now, possibly Wimbledon. But think its a good live stat to have to know if a player is winning/losing because of skillz or because they keep spooning it lol

Cheers

[Realrocknrolla]

https://stackoverflow.com/questions/590 ... ing-python

Have a read of this and this forum mate!

[greenmark]

Hi all,

Any advice or tips when data scraping? Im thinking along the lines of things to avoid to try and prevent things breaking if/when a site gets updated etc.
I see many things online using Python so was just going to start with that, but if anyone has any better (easier) ideas; in the words of Universal Soldier "Im all ears..."
I like to try and work smarter, not harder lol

TIA

My view is that scraping websites is reasonable. They don't change a lot (every change costs them a lot more than it does you).
I did it with Java and even though I had 20+ years IT experience, Java was new to me and a colossal pain in the bum.
One of my (highly experienced) former colleagues recommended Python, so that might be the right way to go.
But for sure, using API's is smart although you may have to pay.
All I'd say is if you go down the road of writing your own stuff, please, please document every step, it will benefit you long term (despite being a pain right now). Trust me!

[Brovashift]

https://stackoverflow.com/questions/590 ... ing-python

Have a read of this and this forum mate!

Been a while since Ive logged into StackOverFlow, a good few years lol. But I think I can understand how that code is working. I'll do a few basic Python tutorials to familiarise myself, see how I get on. Cheers RocknRoller, always appreciated

[Brovashift]

Hi all,

Any advice or tips when data scraping? Im thinking along the lines of things to avoid to try and prevent things breaking if/when a site gets updated etc.
I see many things online using Python so was just going to start with that, but if anyone has any better (easier) ideas; in the words of Universal Soldier "Im all ears..."
I like to try and work smarter, not harder lol

TIA

My view is that scraping websites is reasonable. They don't change a lot (every change costs them a lot more than it does you).
I did it with Java and even though I had 20+ years IT experience, Java was new to me and a colossal pain in the bum.
One of my (highly experienced) former colleagues recommended Python, so that might be the right way to go.
But for sure, using API's is smart although you may have to pay.
All I'd say is if you go down the road of writing your own stuff, please, please document every step, it will benefit you long term (despite being a pain right now). Trust me!

Cheers Greenmark

Ive dabbled with Java myself when developing an android app, but most of my experience is in C#. I was watching a video on YouTube yesterday about programming in finance and the guy was saying to get started Python is the way to go, simply because it just works, start it up and write your code. Problem with a lot of programming interfaces is you have to install all kinds of crap to get them to work... usually followed by an error because you've not got the right .net framework installed, or missing java update files or some nightmare lol.
I like the idea of hopefully copy & pasting some python code, make a few tweaks, loading data into a csv file, ready to be imported into guardian . Thats either naivety or over confidence talking

Documentation: check

[MemphisFlash]

if python is beyond you (as it is for me) then use Parse Hub or Octoparse.
That's how i get my data.

Capture.PNG

[sniffer66]

Also, if you want live tennis stats from the SofaScore API I've already done the work for you. Check the tennis automation sub forum.

I'm not a pro coder so it may not be the best/most efficient code but it does work. All the usual in play stats are available and I've provided a sample baf to show how you get the data into Guardian.

If you want to go your own route with Python, it will at least give you a starting point to port code and endpoints from.

[foxwood]

Stumbled across this and thought of you lol https://realpython.com/learning-paths/p ... -scraping/

I used to scrape with my own .net progs in vb/c# but quite a few sites are anti-scraping which is tedious to cater for.

This year I switched to python using selenium to handle the website connections. It looks to most sites like a proper user and I've not encountered any blocking since.

If you understand code and html document layout then is generally fairly simple, If not then maybe the path suggested by @Memphis would be easier for you.

[jimibt]

Stumbled across this and thought of you lol https://realpython.com/learning-paths/p ... -scraping/

I used to scrape with my own .net progs in vb/c# but quite a few sites are anti-scraping which is tedious to cater for.

This year I switched to python using selenium to handle the website connections. It looks to most sites like a proper user and I've not encountered any blocking since.

If you understand code and html document layout then is generally fairly simple, If not then maybe the path suggested by @Memphis would be easier for you.

i would also recommend using selenium with chromedriver under the covers. you can then (if you need to scrape async), save the session cookies and headers and continue with normal HttpClient libraries. Of course, if it's a site of only a few dozen *pages*, then I'd just use selenium for the whole shebang.

[Brovashift]

Cheers for these tips guys, much appreciated

I seem to have hit a bit of a flat spot in my progress, losing as much as I'm making and making as much as I'm losing, so taking a bit of time this week to work on my execution, see if I can get a bit confidence back. Will dig into these suggestions a bit later

Thanks

[Brovashift]

Does anyone here use or have used tennisprofits?

I don't supposed it possible to scrap data from behind a pay wall is it?