I don't get it, Shaun. Shouldn't you be paying additional 12% up to 40% (your 28% + 12%) of PC2? Or maybe I'm totally don't understand this PC2 concept (it may explain some things).ShaunWhite wrote: ↑Mon Apr 12, 2021 2:34 pmNo DD can indicate that you're playing it too safe and with no noticable DD you'll be paying hardly any commission. Ideally your bot needs to do plenty of losing too by letting it trade all the breakeven stuff at the edge of your curve. Remember you get credited with ½ of the commission on losers and ½ of the winning commission. On average I generate about 28% commission, which keeps me safely out of the PC zone and keeps their mitts off my net weekly.
Trading What I see !?
- ShaunWhite
- Posts: 9731
- Joined: Sat Sep 03, 2016 3:42 am
I'm not at PC2, thats £¼m lifetime and I'm not there yet and like I said on the other thread I paid a lot of commission even before I broke even. Lifetime comm is still at over 40%
ok, after a bit of explanation please for some discrepancy I am seeing of my backtest offer fill algorithm compared with the live trading or a given market today, and I know this is an estimate but still, I can't explain this descrepancy:
So I was confused why according to the backtest of a market today my "fill&kill" £4 Lay order at best reverse price (5.5) was filled, but live trading it expired. From looking at the logging of the Queued and Traded money at 5.5:
Just after order placement the AlreadyTraded was £95 and Queued £431 at price 5.5. So if I make a worst case scenario since these values are log values from 1second after order placement, that my £4 is at the end of the £431 queue. It should thus mean the order will definitely be filled if Traded money at 5.5 exceeds £95+£431=£526, and that is what happened for my backtest about 30seconds later. However, live trading of this market the order expired at 1minute and was never matched. If I look at the traded amount at 1minute according to my log at it exceeds £750, so well in excess of £526.
What am I missing??
I've thought of several possible reasons:
1) My logging is wrong! I frequently get confused in Guardian for the Stored a value rule: So in this scenario i've placed a Lay order at best reverse price, so that money is "queued" behind the Current money available to Back at the current Best Back price? And also the money currently already Traded at that price is "The current Volume traded at the Best Back price"? Right???
2) My assumption that the worst case scenario for getting filled is "Already Traded" + "Queued ahead of me" at that price is wrong?
3) I do notice from the log several moments when there is a price "gap",ie.back=5.5, lay=5.7, this gap affects my numbers somehow..? The gap is 5.6 being taken by layers at that price. So can't see how that affect 5.5 orders?
Apologies if i've made that sound complex, probably cos i've miss-understood something.....?
thanks!
So I was confused why according to the backtest of a market today my "fill&kill" £4 Lay order at best reverse price (5.5) was filled, but live trading it expired. From looking at the logging of the Queued and Traded money at 5.5:
Just after order placement the AlreadyTraded was £95 and Queued £431 at price 5.5. So if I make a worst case scenario since these values are log values from 1second after order placement, that my £4 is at the end of the £431 queue. It should thus mean the order will definitely be filled if Traded money at 5.5 exceeds £95+£431=£526, and that is what happened for my backtest about 30seconds later. However, live trading of this market the order expired at 1minute and was never matched. If I look at the traded amount at 1minute according to my log at it exceeds £750, so well in excess of £526.
What am I missing??
I've thought of several possible reasons:
1) My logging is wrong! I frequently get confused in Guardian for the Stored a value rule: So in this scenario i've placed a Lay order at best reverse price, so that money is "queued" behind the Current money available to Back at the current Best Back price? And also the money currently already Traded at that price is "The current Volume traded at the Best Back price"? Right???
2) My assumption that the worst case scenario for getting filled is "Already Traded" + "Queued ahead of me" at that price is wrong?
3) I do notice from the log several moments when there is a price "gap",ie.back=5.5, lay=5.7, this gap affects my numbers somehow..? The gap is 5.6 being taken by layers at that price. So can't see how that affect 5.5 orders?
Apologies if i've made that sound complex, probably cos i've miss-understood something.....?
thanks!
Ok, I think I may have figured why from reading this article: https://betangel.kayako.com/article/8-c ... ing-engine
So I am actually wondering in my example even though at price 5.5 on the given runner by the time my fill&kill expiry hit it exceeded my predicted "traded amount" that should have filled it, it's possible Befair's matching engine matched bets from other runners at 5.5 on this runner, and thus not consuming the "queued" amount directly on this runner. I guess that's what's called "cross matching"? So it makes it very difficult to thus predict in reality when the bet might be matched, even in the worst case scenario due to cross-matching...
So I am actually wondering in my example even though at price 5.5 on the given runner by the time my fill&kill expiry hit it exceeded my predicted "traded amount" that should have filled it, it's possible Befair's matching engine matched bets from other runners at 5.5 on this runner, and thus not consuming the "queued" amount directly on this runner. I guess that's what's called "cross matching"? So it makes it very difficult to thus predict in reality when the bet might be matched, even in the worst case scenario due to cross-matching...
- ShaunWhite
- Posts: 9731
- Joined: Sat Sep 03, 2016 3:42 am
Your data sample rate could be an issue. 1s is a lifetime and there could be 20 orders/fills/cancellations in that time even at prices above and below touch price. At peak times I'm getting a market change message about once every 17ms, 60 a second. That's the frequency of mains buzz !
So that should not affect once my order is accepted at price 5.5, at a point in time after the order acceptance the traded at 5.5 is £95 and the Waiting queue is £431. So once £526 is traded at 5.5 "at least" my £4 should be matched...? I don't care someone came in after my order, neither some of the £431 were cancelled, as that would have just made my £4 match earlier than £526...ShaunWhite wrote: ↑Mon Apr 12, 2021 8:30 pmYour data sample rate could be an issue. 1s is a lifetime and there could be 20 orders/fills/cancellations in that time even at prices above and below touch price. At peak times I'm getting a market change message about once every 17ms, 60 a second. That's the frequency of mains buzz !
However, if cross-matching throws in Lay bets on this runner from Back bets by other people on other runners, then that does mess things up.
This discrepancy doesn't occur too often, so I am suspecting it is cross matching.
Emmm, maybe stupid question, but do you know that bf counts "traded" for both sides, so you should divide the traded amount (which bf provides) by 2?goat68 wrote: ↑Mon Apr 12, 2021 8:37 pmSo that should not affect once my order is accepted at price 5.5, at a point in time after the order acceptance the traded at 5.5 is £95 and the Waiting queue is £431. So once £526 is traded at 5.5 "at least" my £4 should be matched...? I don't care someone came in after my order, neither some of the £431 were cancelled, as that would have just made my £4 match earlier than £526...ShaunWhite wrote: ↑Mon Apr 12, 2021 8:30 pmYour data sample rate could be an issue. 1s is a lifetime and there could be 20 orders/fills/cancellations in that time even at prices above and below touch price. At peak times I'm getting a market change message about once every 17ms, 60 a second. That's the frequency of mains buzz !
However, if cross-matching throws in Lay bets on this runner from Back bets by other people on other runners, then that does mess things up.
This discrepancy doesn't occur too often, so I am suspecting it is cross matching.
I'm hoping this is a big DOH! moment on my half. So the "Volume traded at a given price" as returned in a stored value by Guardian, equates to the sum of back+lay at that price. So when it says £95 traded at 5.5, that means £47.5 of queued Lay bets were matched and £47.5 of market Back bets were matched against them?napshnap wrote: ↑Mon Apr 12, 2021 9:02 pmEmmm, maybe stupid question, but do you know that bf counts "traded" for both sides, so you should divide the traded amount (which bf provides) by 2?goat68 wrote: ↑Mon Apr 12, 2021 8:37 pmSo that should not affect once my order is accepted at price 5.5, at a point in time after the order acceptance the traded at 5.5 is £95 and the Waiting queue is £431. So once £526 is traded at 5.5 "at least" my £4 should be matched...? I don't care someone came in after my order, neither some of the £431 were cancelled, as that would have just made my £4 match earlier than £526...ShaunWhite wrote: ↑Mon Apr 12, 2021 8:30 pmYour data sample rate could be an issue. 1s is a lifetime and there could be 20 orders/fills/cancellations in that time even at prices above and below touch price. At peak times I'm getting a market change message about once every 17ms, 60 a second. That's the frequency of mains buzz !
However, if cross-matching throws in Lay bets on this runner from Back bets by other people on other runners, then that does mess things up.
This discrepancy doesn't occur too often, so I am suspecting it is cross matching.
G'damit I feel rather stupid if that's the case, doh!
So in my example I actually need to wait for a traded amount of £95+(2*£431)=£957, and you've just explained in one simple answer why it doesn't match after £750!!!
THANK YOU !
Just updated backtest prog, re-run, now matches "live trading" exactly, you star! Sometimes it's the simplest things you overlook!napshnap wrote: ↑Mon Apr 12, 2021 9:02 pmEmmm, maybe stupid question, but do you know that bf counts "traded" for both sides, so you should divide the traded amount (which bf provides) by 2?goat68 wrote: ↑Mon Apr 12, 2021 8:37 pmSo that should not affect once my order is accepted at price 5.5, at a point in time after the order acceptance the traded at 5.5 is £95 and the Waiting queue is £431. So once £526 is traded at 5.5 "at least" my £4 should be matched...? I don't care someone came in after my order, neither some of the £431 were cancelled, as that would have just made my £4 match earlier than £526...ShaunWhite wrote: ↑Mon Apr 12, 2021 8:30 pmYour data sample rate could be an issue. 1s is a lifetime and there could be 20 orders/fills/cancellations in that time even at prices above and below touch price. At peak times I'm getting a market change message about once every 17ms, 60 a second. That's the frequency of mains buzz !
However, if cross-matching throws in Lay bets on this runner from Back bets by other people on other runners, then that does mess things up.
This discrepancy doesn't occur too often, so I am suspecting it is cross matching.
Interestingly, after fixing that embarrassing misstake in my backtest offer logic where it was basically matching on half the required fill, and running a full dataset run, showed very little change in result, only 6 less trades matched out of 308, phew!
- ShaunWhite
- Posts: 9731
- Joined: Sat Sep 03, 2016 3:42 am
Might be worth running those backtests with different sized stakes, a tenner is pretty much always going to be matched or not matched but when you're doing a few hundered there'll be more partial fills.
I can't say for Guardian, but raw bf data counts this way.goat68 wrote: ↑Mon Apr 12, 2021 9:17 pmI'm hoping this is a big DOH! moment on my half. So the "Volume traded at a given price" as returned in a stored value by Guardian, equates to the sum of back+lay at that price. So when it says £95 traded at 5.5, that means £47.5 of queued Lay bets were matched and £47.5 of market Back bets were matched against them?napshnap wrote: ↑Mon Apr 12, 2021 9:02 pmEmmm, maybe stupid question, but do you know that bf counts "traded" for both sides, so you should divide the traded amount (which bf provides) by 2?goat68 wrote: ↑Mon Apr 12, 2021 8:37 pm
So that should not affect once my order is accepted at price 5.5, at a point in time after the order acceptance the traded at 5.5 is £95 and the Waiting queue is £431. So once £526 is traded at 5.5 "at least" my £4 should be matched...? I don't care someone came in after my order, neither some of the £431 were cancelled, as that would have just made my £4 match earlier than £526...
However, if cross-matching throws in Lay bets on this runner from Back bets by other people on other runners, then that does mess things up.
This discrepancy doesn't occur too often, so I am suspecting it is cross matching.
G'damit I feel rather stupid if that's the case, doh!
So in my example I actually need to wait for a traded amount of £95+(2*£431)=£957, and you've just explained in one simple answer why it doesn't match after £750!!!
THANK YOU !
Scaling bet size requires more thought, and is not easy to backtest as when bet size becomes significant compared to queued bets you're potentially influencing the market significantly which you can't predict. My assumption in scaling will be to only scale to a size I estimate is small compared to the liquidity, this should then still roughly hold true with the backtest figures.ShaunWhite wrote: ↑Tue Apr 13, 2021 12:58 amMight be worth running those backtests with different sized stakes, a tenner is pretty much always going to be matched or not matched but when you're doing a few hundered there'll be more partial fills.
Quite pleased with BackTest to Live correlation so far this week with the latest bot and backtest program.
Here's a chart rather than numbers Shaun!
Just the 3 days so far this week, but I like the correlation of backtest run on the recorded data vs the actual live trading of the bot, gives me more confidence:
Here's a chart rather than numbers Shaun!
Just the 3 days so far this week, but I like the correlation of backtest run on the recorded data vs the actual live trading of the bot, gives me more confidence:
- ShaunWhite
- Posts: 9731
- Joined: Sat Sep 03, 2016 3:42 am
I like the confidence of your Y-axis label. Most say PL.
If you're looking at the correlation between those two lines then maybe chart the difference between them expressed as a percentage of the backtest figure?
If you're looking at the correlation between those two lines then maybe chart the difference between them expressed as a percentage of the backtest figure?