Data scraping Do's & Dont's?

Discussion regarding the spreadsheet functionality of Bet Angel.
sniffer66
Posts: 1666
Joined: Thu May 02, 2019 8:37 am

Brovashift wrote:
Thu Aug 04, 2022 3:00 pm
Does anyone here use or have used tennisprofits?

I don't supposed it possible to scrap data from behind a pay wall is it?

I use it for manual trading, and the quality of the data is excellent. Better than any other site I've seen

Possible to scrape it if you pay the sub, but given the warnings on the site you'll risk being blocked if you do. Though I couldn't say if they monitor that or not. Your risk.
User avatar
Brovashift
Posts: 475
Joined: Tue May 18, 2021 12:35 am

sniffer66 wrote:
Thu Aug 04, 2022 3:26 pm
Brovashift wrote:
Thu Aug 04, 2022 3:00 pm
Does anyone here use or have used tennisprofits?

I don't supposed it possible to scrap data from behind a pay wall is it?

I use it for manual trading, and the quality of the data is excellent. Better than any other site I've seen

Possible to scrape it if you pay the sub, but given the warnings on the site you'll risk being blocked if you do. Though I couldn't say if they monitor that or not. Your risk.
Cheers Sniffer,

Ye I've heard its got really good data, that why I was wondering about the scrapping lol. I've been watching a few of his videos and wouldn't mind giving TP's a try, once I've perfected my simple tennis trade entries, so better not burn any bridges just yet :)
You do not have the required permissions to view the files attached to this post.
User avatar
jimibt
Posts: 3641
Joined: Mon Nov 30, 2015 6:42 pm
Location: Narnia

sniffer66 wrote:
Thu Aug 04, 2022 3:26 pm
Brovashift wrote:
Thu Aug 04, 2022 3:00 pm
Does anyone here use or have used tennisprofits?

I don't supposed it possible to scrap data from behind a pay wall is it?

I use it for manual trading, and the quality of the data is excellent. Better than any other site I've seen

Possible to scrape it if you pay the sub, but given the warnings on the site you'll risk being blocked if you do. Though I couldn't say if they monitor that or not. Your risk.
stu -i presume(?!) that TP has an under the covers api at play also?? One thing I've discovered from my dayjob on this is that not all front end and backend systems are authenticated together, so you can find that you are able to obtain the api data endpoints when logged in, but then logoff and STILL use those same api endpoints that would not have been visible without the login. at least 3 of the major streaming platforms present this opportunity as we speak.
sniffer66
Posts: 1666
Joined: Thu May 02, 2019 8:37 am

jimibt wrote:
Thu Aug 04, 2022 4:39 pm
sniffer66 wrote:
Thu Aug 04, 2022 3:26 pm
Brovashift wrote:
Thu Aug 04, 2022 3:00 pm
Does anyone here use or have used tennisprofits?

I don't supposed it possible to scrap data from behind a pay wall is it?

I use it for manual trading, and the quality of the data is excellent. Better than any other site I've seen

Possible to scrape it if you pay the sub, but given the warnings on the site you'll risk being blocked if you do. Though I couldn't say if they monitor that or not. Your risk.
stu -i presume(?!) that TP has an under the covers api at play also?? One thing I've discovered from my dayjob on this is that not all front end and backend systems are authenticated together, so you can find that you are able to obtain the api data endpoints when logged in, but then logoff and STILL use those same api endpoints that would not have been visible without the login. at least 3 of the major streaming platforms present this opportunity as we speak.
That's interesting Jim, I always assumed that a paid for front end meant the same authenticated back end. Given it's a small, bespoke site that's geared up for manual trading, I'd guess no API - I've not really looked.

Brova - if you are interested it may be worth asking Paul Shires who is Tradeshark, and the main guy at TP, if he has an API. To give him credit, Paul is one of the few subscription traders I've come across who gives decent information and data that's worth paying for. Even for manual trading TP it's well worth the sub.
User avatar
wearthefoxhat
Posts: 3206
Joined: Sun Feb 18, 2018 9:55 am

I was always told to read the manual....

developer.png
You do not have the required permissions to view the files attached to this post.
User avatar
Brovashift
Posts: 475
Joined: Tue May 18, 2021 12:35 am

MemphisFlash wrote:
Sun Jul 31, 2022 9:47 am
if python is beyond you (as it is for me) then use Parse Hub or Octoparse.
That's how i get my data.


Capture.PNG
Hi Memphis, thanks for this, looks interesting...

I signed up to Parse Hub yesterday and did the initial tutorial, and seems very user friendly, probably the quickest and easiest way for me to pull some data right now so I don't go off too far down a rabbit hole lol. Just wondering, can I still use this to get live data during a tennis match? And if so, how does it do it?
I'm thinking once you pull data into a csv file and import to BA to display in custom columns, then thats static data. Which also leads me onto another question; Would this need to be done every day for every tennis match, or could it just be done once for e.g. top 100 players in the ATP rankings and then used throughout a tournament, maybe updated once a week or something?
Last edited by Brovashift on Fri Aug 05, 2022 12:30 pm, edited 1 time in total.
User avatar
Brovashift
Posts: 475
Joined: Tue May 18, 2021 12:35 am

wearthefoxhat wrote:
Thu Aug 04, 2022 11:46 pm
I was always told to read the manual....


developer.png
:lol: Now thats a manual :lol:
sniffer66
Posts: 1666
Joined: Thu May 02, 2019 8:37 am

Brovashift wrote:
Fri Aug 05, 2022 12:08 pm
MemphisFlash wrote:
Sun Jul 31, 2022 9:47 am
if python is beyond you (as it is for me) then use Parse Hub or Octoparse.
That's how i get my data.


Capture.PNG
Would this need to be done every day for every tennis match, or could it just be done once for e.g. top 100 players in the ATP rankings and then used throughout a tournament, maybe updated once a week or something?
If you used a BetAngel level SV , linked to the Player name then it would be available in every match, until you restarted BA. However you'd still need to apply the baf, to use the stats, to each match daily, so you might as well pull them in again once per match, at a selection level
User avatar
Brovashift
Posts: 475
Joined: Tue May 18, 2021 12:35 am

sniffer66 wrote:
Thu Aug 04, 2022 4:58 pm
jimibt wrote:
Thu Aug 04, 2022 4:39 pm
sniffer66 wrote:
Thu Aug 04, 2022 3:26 pm



I use it for manual trading, and the quality of the data is excellent. Better than any other site I've seen

Possible to scrape it if you pay the sub, but given the warnings on the site you'll risk being blocked if you do. Though I couldn't say if they monitor that or not. Your risk.
stu -i presume(?!) that TP has an under the covers api at play also?? One thing I've discovered from my dayjob on this is that not all front end and backend systems are authenticated together, so you can find that you are able to obtain the api data endpoints when logged in, but then logoff and STILL use those same api endpoints that would not have been visible without the login. at least 3 of the major streaming platforms present this opportunity as we speak.
That's interesting Jim, I always assumed that a paid for front end meant the same authenticated back end. Given it's a small, bespoke site that's geared up for manual trading, I'd guess no API - I've not really looked.

Brova - if you are interested it may be worth asking Paul Shires who is Tradeshark, and the main guy at TP, if he has an API. To give him credit, Paul is one of the few subscription traders I've come across who gives decent information and data that's worth paying for. Even for manual trading TP it's well worth the sub.
Ye thats not a bad idea Sniffer. I will have to subscribe to TP first to get a feel for his product, but if its as good as what I am hearing might well be worth it 👍. Trying to keep costs down at the min, hence the scrapping :) , and at a bit of a cross roads atm with pre-off, in-play, and tennis trading, so just chipping away at each until I make a final decision where to focus my time/energy.

In one of your threads, the football one I think, you mention about 'history lists', and I've seen ShaunWhite talk about them in the automation section, to build your own practice mode for back testing ideas. How do you add data to history lists, and can it be done in the background while trading manually day to day?
User avatar
Brovashift
Posts: 475
Joined: Tue May 18, 2021 12:35 am

sniffer66 wrote:
Fri Aug 05, 2022 12:13 pm
Brovashift wrote:
Fri Aug 05, 2022 12:08 pm
MemphisFlash wrote:
Sun Jul 31, 2022 9:47 am
if python is beyond you (as it is for me) then use Parse Hub or Octoparse.
That's how i get my data.


Capture.PNG
Would this need to be done every day for every tennis match, or could it just be done once for e.g. top 100 players in the ATP rankings and then used throughout a tournament, maybe updated once a week or something?
If you used a BetAngel level SV , linked to the Player name then it would be available in every match, until you restarted BA. However you'd still need to apply the baf, to use the stats, to each match daily, so you might as well pull them in again once per match, at a selection level
The first thing we need to sort out is 'what is SV'? :lol: I keep seeing it, is it for Servant?

And when you say 'once per match', could a baf file be created where I can load all my matches in to guardian in the morning and apply a baf file to all markets that will pull in the data for every match automatically?
sniffer66
Posts: 1666
Joined: Thu May 02, 2019 8:37 am

Brovashift wrote:
Fri Aug 05, 2022 12:22 pm
sniffer66 wrote:
Thu Aug 04, 2022 4:58 pm
jimibt wrote:
Thu Aug 04, 2022 4:39 pm


stu -i presume(?!) that TP has an under the covers api at play also?? One thing I've discovered from my dayjob on this is that not all front end and backend systems are authenticated together, so you can find that you are able to obtain the api data endpoints when logged in, but then logoff and STILL use those same api endpoints that would not have been visible without the login. at least 3 of the major streaming platforms present this opportunity as we speak.
That's interesting Jim, I always assumed that a paid for front end meant the same authenticated back end. Given it's a small, bespoke site that's geared up for manual trading, I'd guess no API - I've not really looked.

Brova - if you are interested it may be worth asking Paul Shires who is Tradeshark, and the main guy at TP, if he has an API. To give him credit, Paul is one of the few subscription traders I've come across who gives decent information and data that's worth paying for. Even for manual trading TP it's well worth the sub.


In one of your threads, the football one I think, you mention about 'history lists', and I've seen ShaunWhite talk about them in the automation section, to build your own practice mode for back testing ideas. How do you add data to history lists, and can it be done in the background while trading manually day to day?
In the same way that you'd would save an SV but rather than it being constantly overwritten it's a value in time. For instance, in tennis, I save the 2nd serve win% as a value to a history list every 1 minute. Then, in my automation, I can look back over x minutes to see if that value is increasing or decreasing, or plot the values in a chart. Visualise it like an excel spreadsheet, where the row is your value and Col A is now, Col B is 1 min ago, Col c is 2 mins ago, and so on

You can then save that entire History list as a csv for review
Last edited by sniffer66 on Fri Aug 05, 2022 1:41 pm, edited 1 time in total.
User avatar
Brovashift
Posts: 475
Joined: Tue May 18, 2021 12:35 am

sniffer66 wrote:
Fri Aug 05, 2022 12:33 pm
Brovashift wrote:
Fri Aug 05, 2022 12:22 pm
sniffer66 wrote:
Thu Aug 04, 2022 4:58 pm


That's interesting Jim, I always assumed that a paid for front end meant the same authenticated back end. Given it's a small, bespoke site that's geared up for manual trading, I'd guess no API - I've not really looked.

Brova - if you are interested it may be worth asking Paul Shires who is Tradeshark, and the main guy at TP, if he has an API. To give him credit, Paul is one of the few subscription traders I've come across who gives decent information and data that's worth paying for. Even for manual trading TP it's well worth the sub.


In one of your threads, the football one I think, you mention about 'history lists', and I've seen ShaunWhite talk about them in the automation section, to build your own practice mode for back testing ideas. How do you add data to history lists, and can it be done in the background while trading manually day to day?
In the same way that you'd would save an SV but rather than it being constantly overwritten it's a value in time. For instance, in tennis, I save the 2nd serve win% as a value to a history list every 1 minute. Then, in my automation, I can look back over x minutes to see if that value is increasing or decreasing, or plot the values in a chart. Visualise it like an excel spreadsheet, where the row is your value and Col A is now, Col B is 1mago, Col c is 2 mins ago, and so on

You can then save that entire History list as a csv for review
Ah right OK, I just found them in the User Guide and the blog 'Store value in history lists'... Thats pretty cool. Will make use of that, thanks.
Post Reply

Return to “Bet Angel - Spreadsheet / Excel chat”