Ten Years Dark

It's been ten years since the great blackout on the east coast. It's odd that I can still remember it like yesterday.It happened when I was at work with everyone. It was still early days at ComparisonMarket (which later became Insurance.com) and it started a day like any other.Then the lights went out."Save..." was said in a dejected voice to remind you to save your work -- what you just lost from your desktop.Power outages in Solon, Ohio weren't a big surprise. The city even sued the electric company to get us on a better grid at some point, but that's not related to this blackout.Our site was up and things were running fine in our new server room that we had just built.No worries, the new beefy UPSs will keep things alive for an hour and a half if memory serves me right.Then we heard the news: this was the big one. And our generator was scheduled to get hooked up the next week.Assessing the situation we had two small Honda generators we had used as emergency backup for our old server room. We lacked a good supply of gas though. Even then, we didn't have much time on batteries once we realized what we were up against.A mad dash to turn off non-essential or redundant servers began.Soon the UPS was reporting around four hours of run time.Tony and I got into his truck to find not only gas, but gas containers. Driving around for an hour we found a Home Depot in Macedonia that had power somehow. We got all of the gas jugs. Another hour of driving we found a gas station with both power and gas.Meanwhile, back at camp, the troops were getting restless. An expedition was launched to the nearby gas station -- that had gas but no power to pump. We acquired all the beer from the slowly warming cooler. Cash on the barrel-head.Tony and I got back to base around 10 or 11 at night with the gas. The servers, by this point, were growing thin. We had the drive array with the session database on it, a web server, a database server and enough routing gear to keep us on the net. Even the mail server had been shut down after a short deliberation. This was the bare minimum to stay up.The generators were running in the warehouse and an electrician was called to hook of the idle 3/4 MW generator on an emergency basis.We took turns watching the gear progressively getting more drunk as time passed. There wasn't much else to do at this point but tend to the gear as best we could.Looking out the dock door we could see a glow in the distance of a patch that had power.As the crew was about to turn on the generator the power came back up. It was just getting light out -- maybe 6 or 7 in the morning.All told we were down and off the net for a total of an hour or so. We were pretty proud of that achievement given what we were up against!