Promote My Place?

[booboo]

PromoteMyPlace wrote: "Unfortunately we experienced a server failure today which has resulted in us having to rebuild everything from our backups. Rest assured we have been working on this as a matter of urgency and we hope to have everything back to normal very soon. We will update you and all our customers with full details once it has been resolved. Please accept our sincere apologies for the inconvenience."

[Nigel Goodwin]

Thanks - but shouldn't they have a backup server? My main worry when going with them was the fact that they would be hosting my site. I don't think Google likes it when a site goes down, either.

I don't know about web hosting, apart from the fact that it should always work....maybe this will prompt them to up their game.

Still not working at 20:20, and it went down before midday.

I wonder what their backup strategy is, and what their redundancy implementation is. From this experience, they need to call in some experts or outsource the server hosting.

Web server hosting is very specialized and technical, and a small company would quickly become stretched if it is not their main focus and their business grows.

outsource, outsource, outsource.

To be fair, was not impressed by holidaylettings in the last few days either, but that at least involved software/hardware upgrades.

From the FAQ (cached):

How reliable are the PromoteMyPlace servers?

Our servers are very reliable. They are fast, managed and monitored 24 hours per day, permanently connected to the internet and are fully protected by the latest firewall technology.

[Alvaro]

A new problem. I thought this enterprise was more reliable. I couldn´t find myself in google and then I find out the whole thing is down.

[Nigel Goodwin]

Let me be clear - I love their services and tools.

But i would willingly pay £100 p.a. if that is what it takes to get their servers as reliable as they should be - off site backups, redundant discs, redundant servers, redundant off site servers, redundant offsite hosting, redundant networks, redundant switches, etc etc etc.

Let me rephrase - I would be very happy to pay £100 pa for them to outsource to an excellent web hosting company, and let them concentrate on the things they are good at, holiday lettings.

Cheapness will always bring you problems.

[FelicityA]

I am basically a big fan of PMP but I am disappointed in them in that we are all expected to rush to Facebook to get answers.

Surely they should have a record elsewhere (on old fashioned paper?) of all the email addresses in order to contact people to let them know.

I have had to do the Facebook thing with Holiday Lettings and I am not someone who has more than a passing acquaintance with FB (and certainly not Twitter) so it is a last resort for me to go there.

[Hells Bells]

I hardly opened my computer yesterday so I was unaware of any problems at all until I read LMH this morning. I have had my website with PMP for over 2 years, and this is the first time to my knowledge that there has been a problem like this.

[Nigel Goodwin]

Stuff happens, hardware fails, software has bugs. That's not the issue.

The issue is that it shouldn't take so long to get it up and running. Professional web hosting is designed for online business, and puts as much effort into avoiding the once a year problem, and getting up and running again within minutes, as anything else.

The fact that it happens once in two years is not the issue. The issue is that they didn;t seem to have any redundancy.

I know RBS or something had computer issues recently - but that was software bugs (but still very embarrassing, shows they didn't test it properly). If somebody like RBS went down because of server failures - well, it just wouldn't happen, it is well known how to avoid that. Redundancy, redundancy, redundancy. It costs, but is mandatory.

Any decent web hosting site has excellent tolerance against hardware failures which will always be happening.

Also - presumably the number of clients has been increasing, what works when you start off is not what works when you have built a decent client base. It's called scaleability. Statistics on failures in the last two years is not a good indication of how the future will be.

Good news - it is up and running again. I would be interested to know what changes will be made and investments to avoid this again.

[Nigel Goodwin]

I am basically a big fan of PMP but I am disappointed in them in that we are all expected to rush to Facebook to get answers.

Surely they should have a record elsewhere (on old fashioned paper?) of all the email addresses in order to contact people to let them know.

I have had to do the Facebook thing with Holiday Lettings and I am not someone who has more than a passing acquaintance with FB (and certainly not Twitter) so it is a last resort for me to go there.

Maybe all their systems were on the same server, so they couldn;t access email addresses

[booboo]

Does this mean we all have to send ourselves a test email now?

[PromoteMyPlace]

We noticed the discussion here on LayMyHat regarding the recent downtime to the PromoteMyPlace service and we wanted to join in the conversation to let everybody know what happened and what the situation is. I will share the relevant information we sent to our customers below which covers many of the points raised by Nigel.

----------------

Unfortunately at 11:53am Sunday 7th April we experienced a catastrophic server failure which resulted in us having to rebuild the PromoteMyPlace system from our backups. We did our very best to get everything back online as quickly as possible, working with our hosting company throughout the day and night to get a new server in place. We can now confirm that full service resumed at 1.50am Monday 8th April.

Please accept our sincere apologies for the downtime.

We have been with our hosting company for 4.5 years now and this is the first time we have had a major problem. We currently have a 99.9% uptime record.

We have had some questions about our failover and disaster recovery strategy, so we will address these for people that are interested in the details.

We have our own dedicated server maintained by one of the top hosting companies in the UK (UKFast). We have built in redundancy using RAID arrays to protect us from loss of data due to disk failures and we have a disaster recovery plan that allows us to rebuild the server in the event of a catastrophic failure. What we do not have is a system in place to offer 100% uptime.

To offer 100% uptime we would need multiple failover servers, at least one of which would need to be off site and this would prove very costly. We do not believe our customers would want to pay the additional costs required to implement such a solution, especially since such failures happen rarely and are only likely to result in 0.1% loss of uptime. In the event that this does happen we fall back to our disaster recovery plan, which allows us to rebuild the server from our daily backups.

Having said that we are disappointed with the length of time it took to get the websites back online and the fact that we were unavailable via our website and usual email address. We had expected everything to be back up and running within 4-6 hours. We will therefore be reviewing our disaster recovery strategy with our hosting company to see if any improvements can be made, if this were to happen again in the future.

----------------

We accept that it shouldn't have taken so long to get it back up and running and we are discussing the reasons for this with our hosting company. Unfortunately we had no access to the server to be able to do anything ourselves so it was down to them to resolve the issue for us. We should receive a full incident report from them soon which we hope will give us a clearer picture of what happened and how it can be avoided. Please be assured we feel your frustration and as mentioned in the email will be reviewing our disaster recovery plan for the future.

As a final point we have already implemented a separate ticket based support system to ensure we are contactable at all times.

[Nigel Goodwin]

That's all very good news, and I'm happy you are outsourcing the hosting. Thanks also for communicating in an honest and open way - it all adds to confidence.