How I lost to huge traffic and what one can learn from it

This blog went down and crashed because of increased traffic and a lot of visitors yesterday.  Today as I post to the blog running on the dedicated server I still have high CPU usage. And so seems to be the trend going forward. When this happend I started getting mails from our hosting provider saying that one or more directories were blocked because of increased usage. The hosting being on a shared server was affecting other clients. I had the option to move to a dedicated server amongst others and I opted for it. Here’s my experience from the incident and what I learned which you can use to prepare your blog for a traffic increase.

What went wrong?

1. One directory is blocked – I had a backup of my data and was quick to create a sub-domain, restore the data to it and route the traffic there.
2. The sub-domain was blocked – I fell down to my ancient Google Pages hosted website. I simultaneously ordered a dedicated server to be set up. I escalated the ticket with my hosting provider. Meanwhile I get another mail saying that my entire account was blocked. I didn’t have a backup for the entire account. Actually I did but I could remember where I put it. Either ways I couldn’t host it elsewhere.
3. Getting the dedicated server up – The site had been down for 6 hours now and it was past mid-night. My eyes were straining through the glasses waiting for that mail. I’ve already had 6 chat sessions and 8 tickets with the hosting provider. A mail finally entered the inbox with the details of the dedicated server.
4. Moving to the dedicated – Now it’s one this to order a dedicated but it’s a completely different story migrating to it. You have to restore your files, your databases, your domains and sub-domains. You also get new private nameservers which you have to register with your domain provider. I spent a while figuring that out with a tech over a chat session who asked me to email Yahoo (my domain provider) to register the private nameservers. I sent them an email but the response time is like 24 hours so I decided to call them.
5. I requested my hosting provider to move my data to the dedicated. Took another 3 hours. The site now had been down for 9 hours. The wanted me to verify the data and change my nameservers.
6. The Yahoo session – I dialed yahoo and was in the queue for 58 minutes. I disconnected and tried their international line. Some one picked after 15 minutes and put me on hold for another 15. We got the private servers registered in a total of 88 minutes equivalent to 90 minutes. I then changed the nameservers. The nameserver change is a very slow thing. Your nameservers propagate through the internet and synchronise between godzillion servers. The entire process takes upto 72 hours to stabilize and fluctuates till then, but you’d be able to get it to work sooner. So while you may be able to browse your site, some visitors in the other side of the world will be getting strange errors trying to open up your site.
7. The wait – All had been done – the server installed, data migrated, nameservers set. I was my turn to wait for the propagation of the nameservers settings. I decided to spend this time sleeping – that was the best thing I could do for my puffy red eyes. The site had been down for 12 hours by now.

When I woke up I had my daily cuppa coffee, checked-open my site and went away to listen to some music. I realized when it can cost you to not be prepared for the growing traffic.

So here are a few things I learnt

1. Always check the terms and conditions of your hosting provider. See how cooperative they can be in times of a traffic increase. My provider blocked my directories and didn’t provide any access. Even on the dedicated server I had to file a ticket to get those directories unlocked.
2. Always backup your data – If you have it you couldn’t care less about your hosting provider.
3. Keep your site optimized. If you are running a blog or a generic site use some caching plugins. My site has a few php tools and some other developer stuff that I can’t enable complete caching.
4. If you can, have an alternative hosting if you see your site running good traffic. It is better than going for a dedicated. In my case I had been contemplating moving to a dedicated due to the grown popularity of the blog. But that’s not necessarily the right choice for everyone. Have a mirror for your site. If your site goes down all you need to do is change the nameservers and hopefully visitors will start getting routed to it within 30 minutes (practically; the actual time is 72 hrs).

The bright site

I have a pretty good hosting provider. They were able to get a dedicated server all set up and assigned within 4 hours against the standard 48 hours. They migrated my data like experts and the migration was seamless. They have an excellent tech support. When such an incident occurs your site could be down from anywhere between 48 to 72 hours. I was up and running in 12. So stick with a good hosting provider with good level of support.

Photo by Steve Webel