Store Value (range calc) issue

[AngelsWithDirtyFaces]

I've noticed another issue with range calcs which may or may not be related to the problem already outlined above but I'm adding it to this thread just in case.

The steps I'm performing (every refresh) are:
Store a value into a market history list (HL1) In this case it happens to be the back book % of the market.

Store a value (Avg) for the market being the average of a set of values from a range of values in a history list (HL1). Sampling every second from 10 seconds ago to now.

I'm displaying the two values (HL1 NOW AND stored value "Avg") side by side in two custom columns on the one click screen. At all times the two values appear to be in absolute lockstep and there is never any difference between the two. It appears that stored value "Avg" is not calculating an average but just storing the most recent value in the history list. I've rechecked the the custom columns are correctly displaying the two discrete variables and not the same variable by accident.

Even in a frantic pre off racing market in the final few seconds these two variables never get out of sync.
I've tried changing both the sampling frequency and the sampling period and nothing solves the problem.

Having a single market loaded into guardian to avoid any refresh/recycling issues also doesn't make a difference.

As ever, grateful for your observations.
Cheers

I've just tried it using the settings you gave above and got this log, it was about 15mins out so there wasn't a great deal of movement but I've highlighted the area it changed once there was movement


Capture.JPG



This was the file I used
HL Avg.baf

Thanks Dalllas

I've used the logic in your file HL Avg in one of my own with the following changed parameters:
Store Market Volume in MARKET HL 1 and Store Value test-avg as a MARKET level value being the average of MARKET HL 1 for the last 10 seconds sampling every second. I've chosen Market Volume to test as this I know will constantly change.

I've set the rule to trigger100 times and rearm on every refresh
The end of my log output looks like this:

25.09.2021 14:08:54: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 136676.54
25.09.2021 14:08:54: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 136676.54
25.09.2021 14:08:55: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 137182
25.09.2021 14:08:55: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 137182
25.09.2021 14:08:55: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 137215.71
25.09.2021 14:08:55: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 137215.71
25.09.2021 14:08:55: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 137585.44
25.09.2021 14:08:55: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 137585.44
25.09.2021 14:08:56: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 137796.75
25.09.2021 14:08:56: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 137796.75
25.09.2021 14:08:56: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 137975.13
25.09.2021 14:08:56: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 137975.13
25.09.2021 14:08:56: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 137979.13
25.09.2021 14:08:56: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 137979.13
25.09.2021 14:08:56: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 139333.46
25.09.2021 14:08:56: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 139333.46
25.09.2021 14:08:57: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 139468.74
25.09.2021 14:08:57: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 139468.74
25.09.2021 14:08:57: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 139469.72
25.09.2021 14:08:57: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 139469.72
25.09.2021 14:08:57: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 139554.63
25.09.2021 14:08:57: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 139554.63
25.09.2021 14:08:58: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 139642.91
25.09.2021 14:08:58: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 139642.91
25.09.2021 14:08:58: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 139709.9
25.09.2021 14:08:58: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 139709.9
25.09.2021 14:08:58: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 139947.09
25.09.2021 14:08:58: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 139947.09
25.09.2021 14:08:59: [G_Auto 2] : Store Value (Shared) in History List 1 for market = 140204.35
25.09.2021 14:08:59: [G_Auto 2] : Store (range calc) Value (Shared) for market: test_avg = 140204.35

This rule was applied to the 14:10 at Market Rasen so was only a minute from the off time. I had this market loaded on the one-click screen to ensure constant refreshing. As you can see, the test-avg variable returns the same value as the latest value in HL1?

So I'm puzzled. Is the problem only apparent in MARKET level values and lists? Or is it because I've got the rearm set to every refresh combined with the one second sampling of the data which is out of sync?

Here is my file

HL Avg v02.baf

[Dallas]

The second rule in the file you uploaded was armed to trigger 50 times and rearm every refresh - so when I ran it it was out of triggers in under 2 secs

So I've edited mine to Store the Value on the market and this was the log I got

Capture.JPG

[Dallas]

Sorry forgot to add the file used to produce that log

[AngelsWithDirtyFaces]

This is getting interesting.

I noted your point about the refresh time and only 50 triggers but my one-click screen was refreshing at 200m and the particular instance of guardian I ran it on had all the days uk racing markets loaded with a 20ms refresh. Anyway I edited the rule to trigger 300 times with a one second rearm and applied it to the 15:40 at Newmarket in the few minutes before the off.

Again, I am logging market volume which I know will almost certainly increase every second. So I would expect the average of the last 10 seconds to always be lower than the latest value in Market HL1.

The log which I have attached here shows 600 lines of output (2 lines per each iteration of the rule) in just over five minutes. So that seems to be fine. But when you look at the log, the overwhelming number of the outputs of test_avg are identical to the latest value in Market HL 1. But here is where it gets a bit weird. Just occasionally the test_avg output will show a different value compared to Market HL1.

By way of an example from my log. Row 300 of the log with a time stamp of 15:36:41 shows test_avg < Market HL 1. Within a few seconds, however, the two values are back in lockstep.

When you only look at the end of a log and see identical outputs, you don't realise that some further up might be different.

I feel I must be missing something obvious but for the life of me I can't see why the average keeps reporting the latest HL value.

My log:

HL1 Log.xlsx

[Dallas]

I've just shut my BA down for the night so can't check the file till morning

Just to answer this part though from the end of your post
When you only look at the end of a log and see identical outputs, you don't realise that some further up might be different.


The SV is showing the average just from now till 10 secs (or at least that's what the earlier files where doing), so the average could be different anywhere within the log and return back to same values several times

ie, if you get a lull in activity then that's going to cause the average value get closer to the latest stored value and after 10 secs (at the most) it will be back to exactly the same as the latest stored value

[AngelsWithDirtyFaces]

I've just shut my BA down for the night so can't check the file till morning

Just to answer this part though from the end of your post
When you only look at the end of a log and see identical outputs, you don't realise that some further up might be different.


The SV is showing the average just from now till 10 secs (or at least that's what the earlier files where doing), so the average could be different anywhere within the log and return back to same values several times

ie, if you get a lull in activity then that's going to cause the average value get closer to the latest stored value and after 10 secs (at the most) it will be back to exactly the same as the latest stored value

Thanks for your observations again Dallas. Always appreciate how much time and effort you put into the forum.

To address your point about lulls in activity. My log was tracking reported volume in the last minutes before the off of a race at Newmarket.

In the final 30 seconds of my log total volume increased by approximately £58,000.

In every second of that final 30, total volume as stored in HL 1 was higher than the previous second. And in every one of those final 30 seconds the figure reported by test_avg was identical to HL 1 to the penny.

So I'm at a loss with these findings when I would expect the average for the last 10 seconds to always be lower than HL1 if HL 1 is increasing during this period. I'm assuming here that the average will add up the last 10 HL1 readings and divide by 10 in my sample rule?

Thanks again

[Dabbla]

It seems like it should work as it is but doesn’t. Even storing the book% as a separate SV doesn’t seem to work.

To be honest I didn’t even know you could use averages I this way, so maybe im missing something.

I have got it to work by changing to selection rather than market. I have not done the maths but it looks right.


25/09/2021 21:00:05: [G_Auto 4] : Store Value (Shared) in History List 2 for 1. Coney Kroll = 102.086
25/09/2021 21:00:05: [G_Auto 4] : Store (range calc) Value (Shared) for 1. Coney Kroll: test_avg 2 = 101.91
25/09/2021 21:00:06: [G_Auto 4] : Store Value (Shared) in History List 2 for 1. Coney Kroll = 102.086
25/09/2021 21:00:06: [G_Auto 4] : Store (range calc) Value (Shared) for 1. Coney Kroll: test_avg 2 = 101.972
25/09/2021 21:00:08: [G_Auto 4] : Store Value (Shared) in History List 2 for 1. Coney Kroll = 101.459
25/09/2021 21:00:08: [G_Auto 4] : Store (range calc) Value (Shared) for 1. Coney Kroll: test_avg 2 = 101.794
25/09/2021 21:00:09: [G_Auto 4] : Store Value (Shared) in History List 2 for 1. Coney Kroll = 101.795
25/09/2021 21:00:09: [G_Auto 4] : Store (range calc) Value (Shared) for 1. Coney Kroll: test_avg 2 = 101.775

HL Avg v06.baf

[AngelsWithDirtyFaces]

It seems like it should work as it is but doesn’t. Even storing the book% as a separate SV doesn’t seem to work.

To be honest I didn’t even know you could use averages I this way, so maybe im missing something.

I have got it to work by changing to selection rather than market. I have not done the maths but it looks right.


25/09/2021 21:00:05: [G_Auto 4] : Store Value (Shared) in History List 2 for 1. Coney Kroll = 102.086
25/09/2021 21:00:05: [G_Auto 4] : Store (range calc) Value (Shared) for 1. Coney Kroll: test_avg 2 = 101.91
25/09/2021 21:00:06: [G_Auto 4] : Store Value (Shared) in History List 2 for 1. Coney Kroll = 102.086
25/09/2021 21:00:06: [G_Auto 4] : Store (range calc) Value (Shared) for 1. Coney Kroll: test_avg 2 = 101.972
25/09/2021 21:00:08: [G_Auto 4] : Store Value (Shared) in History List 2 for 1. Coney Kroll = 101.459
25/09/2021 21:00:08: [G_Auto 4] : Store (range calc) Value (Shared) for 1. Coney Kroll: test_avg 2 = 101.794
25/09/2021 21:00:09: [G_Auto 4] : Store Value (Shared) in History List 2 for 1. Coney Kroll = 101.795
25/09/2021 21:00:09: [G_Auto 4] : Store (range calc) Value (Shared) for 1. Coney Kroll: test_avg 2 = 101.775

HL Avg v06.baf

Yes, I believe it's definitley MARKET HL related as well.
As another test I've used the "Seconds until event start" value and stored it in both a SELECTION and MARKET HL. Trigger 50 times rearm every second. Create two separate SVs. One to average the SELECTION HL and one to average the MARKET HL. They should of course give the same result. The average of the SELECTION HL appears to work but the average of the MARKET HL carries on reporting the latest value of MARKET HL.

Glad someone else has spotted the issue. Thought it was just me.

[Dallas]

I've just shut my BA down for the night so can't check the file till morning

Just to answer this part though from the end of your post
When you only look at the end of a log and see identical outputs, you don't realise that some further up might be different.


The SV is showing the average just from now till 10 secs (or at least that's what the earlier files where doing), so the average could be different anywhere within the log and return back to same values several times

ie, if you get a lull in activity then that's going to cause the average value get closer to the latest stored value and after 10 secs (at the most) it will be back to exactly the same as the latest stored value

Thanks for your observations again Dallas. Always appreciate how much time and effort you put into the forum.

To address your point about lulls in activity. My log was tracking reported volume in the last minutes before the off of a race at Newmarket.

In the final 30 seconds of my log total volume increased by approximately £58,000.

In every second of that final 30, total volume as stored in HL 1 was higher than the previous second. And in every one of those final 30 seconds the figure reported by test_avg was identical to HL 1 to the penny.

So I'm at a loss with these findings when I would expect the average for the last 10 seconds to always be lower than HL1 if HL 1 is increasing during this period. I'm assuming here that the average will add up the last 10 HL1 readings and divide by 10 in my sample rule?

Thanks again

I had to logon early this morning to check something so done some more testing while I was there.

As in my earlier tests when storing and averaging either the Selection or the Market both were working but I did notice something when;

Creating 'two pairs' of Stored values (both on same rule) with logging
one to store value 'A' on selection in HL1
and average the selection in HL1

other to store value 'A' on market in HL2
and average the market in HL2

I noticed the two averages deviated over time, so as value 'A' continually changes the average value widens especially as you use a longer history period ie, 120-180 secs
Using a 10sec period and if there's little activity as I noticed yesterday as the value and average converged so do both the averages before starting again as Value 'A' changes

But if you set it to average over say 180 secs then the deviation only continues to get wider as time goes by

So to me it would look like one of the two averages is wrong - but that could be the selection average or the market average

It may be something as simple as a different number of decimal places (out of sight of the user) being used for market and selection and with rounding over time its pushing them out of alignment.
I'll flag it to dev tomorrow morning to run through the debugger as I don't think any further testing will shed much more light on it or provide anything more concreate

[AngelsWithDirtyFaces]

I had to logon early this morning to check something so done some more testing while I was there.

As in my earlier tests when storing and averaging either the Selection or the Market both were working but I did notice something when;

Creating 'two pairs' of Stored values (both on same rule) with logging
one to store value 'A' on selection in HL1
and average the selection in HL1

other to store value 'A' on market in HL2
and average the market in HL2

I noticed the two averages deviated over time, so as value 'A' continually changes the average value widens especially as you use a longer history period ie, 120-180 secs
Using a 10sec period and if there's little activity as I noticed yesterday as the value and average converged so do both the averages before starting again as Value 'A' changes

But if you set it to average over say 180 secs then the deviation only continues to get wider as time goes by

So to me it would look like one of the two averages is wrong - but that could be the selection average or the market average

It may be something as simple as a different number of decimal places (out of sight of the user) being used for market and selection and with rounding over time its pushing them out of alignment.
I'll flag it to dev tomorrow morning to run through the debugger as I don't think any further testing will shed much more light on it or provide anything more concreate

Thanks Dallas,

It's probably worth pointing out to the dev team that Summing the stored values also doesn't work for me in the Market HL. Indeed with hindsight I should have checked the sum function earlier because of course summing values over ten seconds isn't affected by activity lulls like the average would be as you pointed out yesterday. To summarise, summing ten seconds of values in a SELECTION HL works but summing 10 seconds of MARKET HL reports only the latest HL value.

Cheers

[Dallas]

Just to update you on this dev managed to find the cause of this, there was a bug and in short when checking the range of values on the market (selections were fine) it was starting one step past the end of the data queue and looping round to the start, once the reason was found it was a simple fix and just needed '-1' to correct it starting one step past the end when looping through the sample.
It will have been there since added in V1.56 so it was a good spot and will be fixed before V1.57 is formally released.

[AngelsWithDirtyFaces]

Just to update you on this dev managed to find the cause of this, there was a bug and in short when checking the range of values on the market (selections were fine) it was starting one step past the end of the data queue and looping round to the start, once the reason was found it was a simple fix and just needed '-1' to correct it starting one step past the end when looping through the sample.
It will have been there since added in V1.56 so it was a good spot and will be fixed before V1.57 is formally released.

Apologies for the late response Dallas.

Many thanks to you and the Dev team in turning this issue around in quick time. I've downloaded 1.57.0.b2 and all looks well.
Cheers.