Automation with a twist - FileSystemWatcher

[jimibt]

I searched and couldn't find any mention of this anywhere.

As we all know, BA is a fantastic *control surface* for managing almost any aspect of a traders requirements and to this end, it makes no sense to reinvent the wheel, even when using your own handrolled trading tools. However, one frustration that I face daily is the fact that I have a few bespoke tools that I reference in order to place trades within BA. The process is basically, one of eyeballing the bespoke UI and then flipping over to BA, navigating to the market and placing the trade. This works fine but is still an inconvenience.

However, I'm 100% certain that a simple mechanism could be added to BA in order for it (BA), to pick up a json file that contained instructions on the trade. I would see this as utilising the FileSystemWatcher class and picking up and processing any json files that were dropped in a predefined folder. The filespec for the json would obviously have to be defined, but could be as simple as capturing:

MarketID, SelectionId, BetType, OffsetTicks, FillOrKillSeconds, Amount, UseGlobalSettingsFlag

On such a file being dropped, BA would use this as if it were being entered manually and would output a response (to a response file in the folder, using a paired naming convention) that the bespoke app could validate and act upon.

I know there are Excel solutions, but these are quite heavyweight by comparison. A solution like this would be lightweight and provide a psuedo api which would be useful for many strategies.

Anyway, hope this filters back to the powers that be - this would solve 90% of my current duplication of effort and am certain that others would benefit in equal fashion.

[LinusP]

If you open up a .bat file in notepad you can see that automation pretty much does exactly this.

[jimibt]

If you open up a .bat file in notepad you can see that automation pretty much does exactly this.

yes, tho that is command driven via the file contents rather than being a cargo object as such (i'm assuming you meant a .baf file, rather than a .bat file?!). my proposal really is for BA to monitor file drops (using FileSystemWatcher) and then on finding a new json file (that matches a known spec), it pulls it into the program and processes the *order* based on the values specified. it then pings out a response file which the 3rd party application can then pull into its process in order to reconcile the transaction - rinse and repeat.

[LinusP]

Yeh, probably meant .baf

Lots of complications here, why not just learn to program? You won't regret it...

[jimibt]

Yeh, probably meant .baf

Lots of complications here, why not just learn to program? You won't regret it...

i actually am a .net developer by trade (but definitely could learn more ). this interaction is to allow me to utilise the BA internals due to the functionality it offers (on greening etc). i use my own bespoke code to identify targets against various technical signals etc... and then map those manually to BA at present.

... i guess i was thinking about a less involved way to allow BA to pick up external content that can be used to populate the bet placement engine. since BA is hosted on a windows platform, FileSystemWatcher seemed like the most pragmatic way to monitor and process dropped files (be those manually, or as would be the case, programatically dropped). i also make a wide ranging assumption here that the BA bet placement engine is loosely coupled to its UI, therefore allowing it to be hooked to other processes as required (this same assumption leads me to imagine that this it isn't a huge challenge to add the FileSystemWatcher process as described and have the engine process as required)

at the end of the day, i probably should bite the bullet and code up a full bet placement engine and altho i use the api-ng to do small placements, i don't have a full management system that i can rely on - thus why i thought that using BA as the management centre would be a far more sensible idea (that way, i also get to use any new features that BA add for free).

anyway, will pop further thoughts down later. thanks for the input tho, appreciated.

[dkedwards]

This sounds like it could be interesting.

Would anyone care to (briefly) translate for the 'non IT techy' members of us out here?

Cheers

[jimibt]

This sounds like it could be interesting.

Would anyone care to (briefly) translate for the 'non IT techy' members of us out here?

Cheers

Hi Dave - hope all going well. basically, what i'm proposing is that BA is extended to allow it to pick up an external cargo file that contains parameters pertaining to individual (or a List of) orders. the premise behind this is to allow 3rd party and bespoke applications to use the bet placement functionality of BA, thus saving a ton on the development of a bet placement engine oneself.

i'm suggesting that this could be implemented as a fairly trivial (and to most users - invisible) update as i firmly believe that the BA engine will be decoupled from the application logic and thus should be able to be used for many other processes that do not require UI interaction.

so, in short, the non techy translation is that i'd like to propose an entry point to the BA engine that can be used by other applications thus allowing those other applications to concentrate on their bespoke functionality and BA acting as a middle tier for placing the orders generated by said 3rd party apps - phew, all sounds a bit techy again

[edit] as an aside. this little update would serve as a perfect test-bed for future BA web-based apps as it would by it's nature require the engine to work against json documents and thus wrinkles would be sorted out by the 3rd party developers that utilised this functionality. win, win!!

[foxwood]

Since you are a .NET developer, if the Excel interface provides all you need ie placing / cancelling / greening etc then I'd use that rather than looking for BA changes.

Since two processes trying to update the same spreadsheet doesn't work well use MS SQL as an intermediary.

Pop your json "message" into a table as a timestamped row.

Using vba in Excel you can then poll the table and process the rows on a fifo basis executing the instructions in the table by placing the appropriate commands on the BA sheet(s).

The same mechanism could be used to get info from BA / Excel into SQL and visible to your other software. Lots of other mechanisms you could use but SQL would be an easy starting point for proof of concept.

Sounds long winded at first sight but if you can automate your "message" generation then should certainly be faster than manual execution - and it's all under your control so you can change it to suit.

[jimibt]

Since you are a .NET developer, if the Excel interface provides all you need ie placing / cancelling / greening etc then I'd use that rather than looking for BA changes.

Since two processes trying to update the same spreadsheet doesn't work well use MS SQL as an intermediary.

Pop your json "message" into a table as a timestamped row.

Using vba in Excel you can then poll the table and process the rows on a fifo basis executing the instructions in the table by placing the appropriate commands on the BA sheet(s).

The same mechanism could be used to get info from BA / Excel into SQL and visible to your other software. Lots of other mechanisms you could use but SQL would be an easy starting point for proof of concept.

Sounds long winded at first sight but if you can automate your "message" generation then should certainly be faster than manual execution - and it's all under your control so you can change it to suit.

tbh - that's quite a good interim solution. i'll try rolling something up from this tmro. i still would prefer a mechanism similar to what i describe, but this would certainly work as a poc - thanks for being my extended brain

all the best...

[edit] - in fact, i could probably use the winApi via vba to achieve the same end result without having to use sql (i.e. using FindFirstChangeNotification from the kernel32 lib), then, should BA implement something similar, i'd just port over the code to my client app...

[ShaunWhite]

That's basically what i've done Jimibt, but only being a humble vb6 programmer I did it the easy/lazy way and just have a second .xlsm (db & code) which takes data from the BA_multi.xls, chews it up and writes instructions back to it. (I could have used vb/sql but it's years since I did all that and I can't bring myself to learn it all again, and haven't really needed to). I'm just using BA_multi as my i/o to BA and a seperate xlsm back end. Isn't that all you need, or am I missing the point?

I'm using this via Guardian atm so polling time is a bit of an issue......to save me some time, do you know if the main BA interface can connect to a different xls other than that used in Guardian? or does it have to be the same xls with the current market going to the 'Bet Angel' worksheet and the Guardian polled markets going to the other tabs/worksheets.

You mention that you've had issues with two sources updating the one xls, I guess you mean BA and your own app....I've not had any problems with that (yet). Could you tell me what issues you've had?

and do you capture your manually executed trades externally somehow Jimibt, to combine with your algo trading? eg to ignore a selection or a market if manual trades exist, eg to manually take control of the autopilot in an interesting scenario?

sorry so many questions

[jimibt]

That's basically what i've done Jimibt, but only being a humble vb6 programmer I did it the easy/lazy way and just have a second .xlsm (db & code) which takes data from the BA_multi.xls, chews it up and writes instructions back to it. (I could have used vb/sql but it's years since I did all that and I can't bring myself to learn it all again, and haven't really needed to). I'm just using BA_multi as my i/o to BA and a seperate xlsm back end. Isn't that all you need, or am I missing the point?

I'm using this via Guardian atm so polling time is a bit of an issue......to save me some time, do you know if the main BA interface can connect to a different xls other than that used in Guardian? or does it have to be the same xls with the current market going to the 'Bet Angel' worksheet and the Guardian polled markets going to the other tabs/worksheets.

You mention that you've had issues with two sources updating the one xls, I guess you mean BA and your own app....I've not had any problems with that (yet). Could you tell me what issues you've had?

and do you capture your manually executed trades externally somehow Jimibt, to combine with your algo trading? eg to ignore a selection or a market if manual trades exist, eg to manually take control of the autopilot in an interesting scenario?

sorry so many questions

Shaun,

thanks for the reply. i do not want to incur cpu cycles or delays via any polling mechanisms (there in lives the road to h3ll ). my bespoke app uses the betfair api-ng to directly place trades, but it is all very basic and my main objective is to extend the logic of the code to cherry pick candidates, rather than reinvent a 1st class bet management engine (i.e. BA).

basically what i'll end up doing is that i'll create a little com-interop dll in c# and the reference it as an object in VBA. this will contain the relevant calls to FileSystemWatcher. the vba process will then piggyback this and process the cargo that is flagged up by the vba object. an example of this approach can be found here: https://social.msdn.microsoft.com/Forum ... forum=vsto

i'll be perfectly honest, i've never really used the excel integration with BA, so this will be a little learning curve. i'm still hopeful that my original suggestion makes enough sense to warrant the devs at BA using a wet wednesday afternoon to try a poc on

watch this space - boooom

[foxwood]

basically what i'll end up doing is that i'll create a little com-interop dll in c# and the reference it as an object in VBA. this will contain the relevant calls to FileSystemWatcher. the vba process will then piggyback this and process the cargo that is flagged up by the vba object. an example of this approach can be found here: https://social.msdn.microsoft.com/Forum ... forum=vsto

Ummmm - as you probably know, the underlying problem is that vba is single threaded.

The code in that link will get you a file watcher running and pop up a message box when it changes but that doesn't get the info into the main calling vba process / onto the sheet(s).

Sadly I got intrigued and looked deeper ...

http://stackoverflow.com/questions/5721 ... ing-in-vba good discussion on various techniques

http://analystcave.com/excel-multithrea ... -vs-c-net/ exhaustive analysis of techniques and timings

http://analystcave.com/excel-vba-multithreading-tool/ have a look at Example 3 shows how to write back to the master sheet(s) - the key thing needed to get BA to act.

Going back to my own nightmares now - enjoy

[jimibt]

basically what i'll end up doing is that i'll create a little com-interop dll in c# and the reference it as an object in VBA. this will contain the relevant calls to FileSystemWatcher. the vba process will then piggyback this and process the cargo that is flagged up by the vba object. an example of this approach can be found here: https://social.msdn.microsoft.com/Forum ... forum=vsto

Ummmm - as you probably know, the underlying problem is that vba is single threaded.

The code in that link will get you a file watcher running and pop up a message box when it changes but that doesn't get the info into the main calling vba process / onto the sheet(s).

Sadly I got intrigued and looked deeper ...

http://stackoverflow.com/questions/5721 ... ing-in-vba good discussion on various techniques

http://analystcave.com/excel-multithrea ... -vs-c-net/ exhaustive analysis of techniques and timings

http://analystcave.com/excel-vba-multithreading-tool/ have a look at Example 3 shows how to write back to the master sheet(s) - the key thing needed to get BA to act.

Going back to my own nightmares now - enjoy

thanks for the info - as you say, the single threaded nature causes roadblocks. the item above (example 3) looks kinda promising and might be worth pursuing. ok, will let your own personal nightmare resume