Learning to fail
The first thing we noticed was a drop in traffic. Then other symptoms that seemed unrelated at the time were also being reported like search not working. it wasn't until we got a notice from Google Webmaster that we realized we'd been hacked. Again.
The reality
After taking the site down for the second time in two weeks, it dawned on me, if we are to continue we’d better learn to fail.
It’s simple, a website with our set -up running a popular content management system on a public facing website might be compromised. It could happen for a variety reasons both in and out of our control. The important thing then is getting the site back to a point where it is safe as soon as possible while being 99% sure that whatever compromised the site in the first place has been removed.
Enter SVN and versioning
Versioning is not the first think I think of as being essential to a content site, benefits aside, there are somethings that can only add more work that do not equal the benefits. Simply Good Media is by any definition is a small shop with just a couple of developers, there are very few conflicts and if there are, a quick IM, clears everything up and one of us backs off for a while.
Also, in tech circles, versioning is for software teams, not a blog. The Budget Fashionista is not just a blog or a website and more than a few people make a living off the site and many more thousands enjoy it on a daily basis. Additionally years and years of goodwill that could be wiped out in the blink of an eye if our site is distribute malware installed by a hacker. It is our responsibility to keep things up and running.
The new workflow
So with that in mind, I set out for TBF to fail. The first thing I looked for was a hosted versioning system. After trying a couple out on personal projects, I settled on beanstalk. No only did I like the design(sorry, the graphic designer in me is alive and kicking) It was defacto endorsed by 37signals and integration with basecamp, which we use.
I also set up a staging server, and refrained from making any changes to the files on the website. The new workflow consisted of the repository where I would make local changes, upload the file to staging to make sure it’s right and them commit to Beanstalk.
With Beanstalk I update the site with a push of a button, or can roll it back to a safe copy if necessary. I also did some testing to make sure when we fail we can get back up. So something that use to take hours or days could now can be done from anywhere from any computer in minutes. We’ve cut down on mistakes making it to the live site and can push several changes at once without the fear of a conflict.
Conclusion
All in all it’s been a good exercise and turning something that has brought the site to a grinding halt and destroying our goodwill as a site to an opportunity to normalize the site and to make it that much more stable by using versioning and installing a few other safe guards to help mitigate the effects of the inevitable.
Posted on Wednesday, June 29, 2011 in Business • Featured • Productivity • Projects • Strategy • Technology