The Day That Media Temple Stood Still
About two years ago I decided I would test out Media Temple’s grid service package, mainly because it was highly recommended by WordPress users, and seemed to perform quite well. In fact, for the first few weeks I was a Grid Service user, I thought things ran actually pretty smoothly. Unfortunately though, over the next few months, things would degrade rather rapidly. My site was plagued by ever increasing latency, and would often go down for 4- 6 hours at a time due to MySQL outages or random, seemingly unannounced maintenance periods.
It got so bad at one point that I actually came to the conclusion I could do a better job myself by hosting my website at home, and transitioned this website to a little server in the corner of my living room. With the exception of a blown power supply that took my site down for a few hours, it ran for the most part without any hiccups for a year and a half or so.
Obviously running a full blown hosting company is a lot harder than running a single machine in the comfort of your own home. But I would like to think that we use hosting companies to host our websites because these companies represent the experts, and that they have the technical know-how to keep our sites running smoothly. As recent events have proven, that is not always the case.
Shortly after noon on Saturday, Media Temple posted this on their blog:
We are currently working on restoring availability to all of Cluster 2 at this time. We will have an update shortly with more details and an estimated time of resolution.
Since then, Media Temple has posted various updates regarding the issue, most of which firmly place the blame on their storage vendor:
According to Bluearc engineers our recent firmware update introduced a new bug that we have never seen. Their developers are currently determining our best course of action. We are considering a rollback of the recent change we made or working with Bluearc to patch our current running version.
At this very moment we are trying to restore all services on the Cluster by isolating out Storage Segment 2*, which appears to be the problematic segment. This will ensure that a great majority of those affected will have their service restored. Those on Storage Segment 2 might have to wait a while longer pending the actions we take with Bluearc.
Thinking back over the last few years, I am fairly certain that Media Temple has routinely blamed their storage vendor for all their problems. And while it very well may be true that 100% of their problems are related to the storage, ultimately Media Temple has to take responsibility for these problems and to make corrective actions to fix them. Given that these issues have plagued Media Temple ever since I first signed on (which has to be at least two years ago at this point), you have to wonder why they haven’t been able to fix things. In fact, if this current situation is any indication, things at Media Temple are getting worse, not better.
Most shared hosting companies in the same league as Media Temple (and I use that term loosely, since in my opinion, most of those companies outperform Media Temple) are priced around $7/month. Media Temple charges $20. In the old days, that price reflected the promise that Media Temple’s Grid Service would be the first incarnation of shared hosting that didn’t completely suck. But if time has proven anything, it’s that the Grid Service performance really hasn’t matched the hype. Also tragic in this is that the cluster that is affected the most by this appears to have some of the oldest (and subsequently most loyal) Media Temple customers on it.
We pretty much take for granted these days that websites should be always on. When I punch a URL into a browser, rarely do I ever think that I won’t get a response from the server. Had BraveNewCode been affected by this (and it is actually located on Media Temple), I would be livid at this point, primarily because I know that having a dead website ultimately translates into losing real money when you are a business. Thankfully none of our Media Temple websites went offline, but it’s not lost on me that possibly thousands of businesses were affected by this, and ours might have been too.
At current count, we are approaching 48 hours of downtime on Media Temple’s servers, something that I don’t think I’ve ever seen happen to this extent. Given that most hosting companies guarantee 99.9% uptime (at least in the brochures), that basically means that Media Temple has exhausted their downtime for the next 5.5 years.
I’m really interested to see what Media Temple has to say tomorrow, and what they are willing to offer their customers for this outage. I’ve seen what Media Temple has Twittered about over the past 36 hours, and have also been reading up on their blog. But so far, while they have been apologetic, I haven’t seen any mention of compensation, which is something I think they are going to need to address soon.
I really don’t know what the fallout will be from this. I suspect a lot of people are extremely angry and disappointed with Media Temple now. I for one don’t plan to be caught with my Media Temple pants down in the future, and started the process of moving a few of my sites off of their service tonight. I’ve given Media Temple countless chances over the years to show me that they can improve, but at each turn they continue to disappoint me, and it’s not something I plan to let happen again. And I suspect that tomorrow, when Media Temple finally gets those websites back online, that you’re going to see an unprecedented amount of anger and disappointment in the blogosphere directed at Media Temple.
If you’re on Media Temple and looking for some other web companies, here’s a small list: