Well, last night at about oh, 10-10:30pm, one of my businesses servers (phoenix) went down. The datacenter techs took a look at it, and it had a failing raid array. Now, normally, this is something that you really don’t want to have happen. Especially not this late at night, since its a 2.5hr drive to the cities.
So my buddy Chris and I sped down to Minneapolis around 11pm, and arrived at US Internet by about 1:30am. Sure enough, the raid array was borked. We tried rebuilding it, and that seemed to fix it, temporarily, but there were some massive inconsistencies. We checked both disks, both had current data on it, but they just varied slightly. So we re-imaged one disk over the other. This took about an hour and a half just for the re-imaging process. Good thing we were able to keep ourselves busy by tweaking our firewall (dagron). Unfortunately, we were still unable to figure out the one glitchy rule that was causing problems with some sites. That will have to wait for another adventure on another day.
Well, the re-imaging completed, but the array was still failing. I guess one of the drives is actually bad. So we re-hashed the setup to run off a single drive, got the system back up, and everything was all hunky-dory by 4am. Well, except for the bad hard drive, that will need to be replaced once I get a chance to order a new pair of 120gb hard drives.
So we clean up the rack, and head home. But first we needed to get gas. We stopped at a holiday station before we arrived at the Datacenter, but it was closed, and their pumps were turned off. I’ve never seen a holiday station that didn’t have at least 24/7 pay at the pump. Luckily, we were able to find another normal gas station (or petrol station as they’re called across the pond) and get fuel for my thirsty car. Our next point of business was to get fuel for our tummys. We found a perkins, but no one appeared to be there. We heard voices in the back of the kitchen, but no one came up to help us, so we decided to find another. The only other perkins we knew how to get to was in Roseville, all the way on the other side of the cities. So we went there, and ate breakfast, and had a pot of coffee.
The ride back was a blur, travelling in a caffeine induced sleepless state. We finally got back to duluth at about 7:30. I went straight to the bathroom because my can of Sierra Mist, plus a pot of coffee, plus a red bull and an amp energy drink were all crying to get out. All of those had been since 11pm when we left my house the night before.
So here I am now, at work (which started at 8am), no sleep, wired on caffeine. I’m probably going to read this post tomorrow and be like “What was I smoking!?!”
