The low end hosting industry took a couple of black eyes this past week when stories circulated around the web over the accidental loss of customer information. It didn’t happen just once, it happened twice. It all started with a post on Server Fault , where an unidentified user called Bleemboy asked:
“I run a small hosting provider with more or less 1535 customers… All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script)… How I can recover from a rm -rf / now in a timely manner?”
Responses from the community ranged from helpful to… well, read for yourself:
“To be honest. I’m not sure you’re real. If you are, you’re probably in the wrong job…” – leftcase
“I feel sorry to say that your company is now essentially dead.” – Sven
“You’re going out of business. You don’t need technical advice, you need to call your lawyer.” – Michael
For those of us that don’t follow the specifics of that particular a rm -rf / command, it is a powerful Unix-based command that could indeed wipe out all the information on a system. It is a command that can be unforgiving, removing files without prejudice.
Thankfully, the story turns out to be a cautionary tale, a fire drill of sorts for the community, and a half-serious chuckle. Well, not for long.
Not a Fire Drill: 123-Reg
No sooner than that story went semi-viral did an even worse, true story hit the web. The UK-based hosting company 123-Reg accidentally wiped out the Servers belonging to an untold large number of customers, on their VPS platform. A VPS is basically a virtualized server and the company’s policies on this matter suggest that customers back up their own systems.
The company accidentally deleted data through a cleanup script, which was supposed to only remove the systems belonging to customers that weren’t in their master database. One error in that script wiped out hundreds of hours of work, client company information, intellectual assets, web pages, and much more. Several days after the fact, the company is still struggling to restore all of the systems.
A Backup Story
Every customer should indeed keep backups wherever critical information exists. Customers should also pay attention to what hosting providers are doing in terms of backups, including details of utilization and how far back files can be recovered.
Customers should also be conducting systems operations well. Things happen, which is why we plan, we test, and we backup everything. People cut corners though and not all hosting providers and services are the same. You need to remember that when picking a hosting provider. Don’t get lured by price, or hype. Make sure they have good, and proven customer references. These examples above are the type of hosting providers you have to stay away from, no matter what. These are very real examples with real affected customers.
Thankfully, there is a thing call Darwinism, and in technology, this principle states that the unfit will die off at some point. There can be no better ending in this scenario. A lot of the blame has to do with certain low end hosting providers focused low end pricing, and not focused on service and the rigor to operate a massive platform with tons of customers.
A solid and reputable hosting company doesn’t cut corners with its procedures or the quality of its personnel, and they will backup systems diligently with robust tools. Codero suggests the following: do your research, make sure your hosting provider is following the prime rules of administration, and don’t be lured by cheap prices.