Betfair planned outage 7am to 11am tomorrow (12th Nov)

[jamesedwards]

There is a planned outage in place on Tuesday 12th November between 07:00am - 11:00am GMT in order to carry out some essential platform work.

https://forum.developer.betfair.com/for ... -11-00-gmt

[conduirez]

There is a planned outage in place on Tuesday 12th November between 07:00am - 11:00am GMT in order to carry out some essential platform work.

https://forum.developer.betfair.com/for ... -11-00-gmt

I have a question for James maybe Jimibt as well, or anyone who works on servers really.

When I use to work in infrastructure maintenance a lot of banks used to have what I would call backup sites, so their main servers would be in London and they would have another set of servers mirroring their main servers in another location, in case of terrorism etc.
Maintenance was not such a problem as they could carry out maintenance without to much impact on their clients because of their back up servers. Nowadays companies such as Betfair even banks such as NatWest have outages for maintenance, do they no longer have back up servers?

[PBRSreeetGang]

Normally planned maintenance is to make a change the to system, to either fix something, or upgrade a service. Redundancy (backup, mirror) is normally used if a site were to go down, or because of a hardware failure.

[Crazyskier]

Now saying 11.30 : - (

CS

[jamesedwards]

Obviously something gone wrong...

Thought it a bit odd to take out a whole morning, must be something important. Racing is less than one hour away and plenty of Greyhound action missed.

[Crazyskier]

Sportsbook back up, so hopefully not too much longer for exchange...

CS

[Crazyskier]

UP!

CS

[Crazyskier]

Haha - until you try to log in!

NOT up.

CS

[Crazyskier]

Finally IN!

CS

[ODPaul82]

I have a question for James maybe Jimibt as well, or anyone who works on servers really.

When I use to work in infrastructure maintenance a lot of banks used to have what I would call backup sites, so their main servers would be in London and they would have another set of servers mirroring their main servers in another location, in case of terrorism etc.
Maintenance was not such a problem as they could carry out maintenance without to much impact on their clients because of their back up servers. Nowadays companies such as Betfair even banks such as NatWest have outages for maintenance, do they no longer have back up servers?

FTSE250 insurance company I work for have a mixture of on & off-site redundancy for business critical (Tier 1) systems but it depends on the system/service.

There is a big difference between going to backup and maintenance.

Backup is what it says it is, it's backup.
Our finance systems which I manage we're cloud driven on the Oracle side of things. We have immediate access to the backup servers and can tell users to use them immediately and supply URL but it will be prior evenings data for anything not transmitted so far, plus I would have to reconfigure OICS to point to all of the DR instances.
Our closest "on-prem" for where our cloud data is stored is Slough which is one of the main choices for MS/Oracle/etc due to their location in Reading. Our backup is located in Amsterdam. We have spares in Frankfurt if we do require them.
Betfair use the same software for their financial reporting as (a) the formatting is the standard formatting for Oracle FCCS and (b) I meet their finance guys on a regular basis at EPM meetings so they will have a similar setup.

Reverting to backup in a DR scenario (Disaster Recovery) is a pain, it's enough of a pain doing it as a yearly exercise due to FCA requirements without having to do it properly.

Our AS/400 mainframe which has been in place since 80s has tape based backup again geographically different places in UK. Only now are we migrating onto something more cloud based.

The newer cloud based system called Guidewire we have running on Azure and can spin up and down as we so wish but again, getting to close to a live state is not immediate, you're talking petabytes of data.

A lot of places are now using systems such as docker where you can build up how you want items to be deployed based on live infrastructure for mainteance but you will still need to take systems down, things are a lot more complex then back in the day. Additionally there's so many layers of governance in regards to how you perform items. Cowboy days of just throwing in a fix are well over.

[conduirez]

I have a question for James maybe Jimibt as well, or anyone who works on servers really.

When I use to work in infrastructure maintenance a lot of banks used to have what I would call backup sites, so their main servers would be in London and they would have another set of servers mirroring their main servers in another location, in case of terrorism etc.
Maintenance was not such a problem as they could carry out maintenance without to much impact on their clients because of their back up servers. Nowadays companies such as Betfair even banks such as NatWest have outages for maintenance, do they no longer have back up servers?

FTSE250 insurance company I work for have a mixture of on & off-site redundancy for business critical (Tier 1) systems but it depends on the system/service.

There is a big difference between going to backup and maintenance.

Backup is what it says it is, it's backup.
Our finance systems which I manage we're cloud driven on the Oracle side of things. We have immediate access to the backup servers and can tell users to use them immediately and supply URL but it will be prior evenings data for anything not transmitted so far, plus I would have to reconfigure OICS to point to all of the DR instances.
Our closest "on-prem" for where our cloud data is stored is Slough which is one of the main choices for MS/Oracle/etc due to their location in Reading. Our backup is located in Amsterdam. We have spares in Frankfurt if we do require them.
Betfair use the same software for their financial reporting as (a) the formatting is the standard formatting for Oracle FCCS and (b) I meet their finance guys on a regular basis at EPM meetings so they will have a similar setup.

Reverting to backup in a DR scenario (Disaster Recovery) is a pain, it's enough of a pain doing it as a yearly exercise due to FCA requirements without having to do it properly.

Our AS/400 mainframe which has been in place since 80s has tape based backup again geographically different places in UK. Only now are we migrating onto something more cloud based.

The newer cloud based system called Guidewire we have running on Azure and can spin up and down as we so wish but again, getting to close to a live state is not immediate, you're talking petabytes of data.

A lot of places are now using systems such as docker where you can build up how you want items to be deployed based on live infrastructure for mainteance but you will still need to take systems down, things are a lot more complex then back in the day. Additionally there's so many layers of governance in regards to how you perform items. Cowboy days of just throwing in a fix are well over.

Thank you for the detailed reply, most enlightening.

I do remember the IT guys doing tape back ups and security vans coming to collect them.

"getting to close to a live state is not immediate, you're talking petabytes of data."

The main thing that I must say I did not think about was amount of data involved which you have made clear.

[jimibt]

... Cowboy days of just throwing in a fix are well over.

this bit did make me reflect on my itv days from the late 90's. things have changed.