Well, the past 3 weeks have definately been interesting. This is going to be a pretty long post, since it covers a lot, so I’ll try to break it up.
So on Monday August 22nd, I received my new Dell Poweredge 750 server I ordered. But when it arrived, there were 2 boxes. Both had the same packing slip (with my server listed on it), but they were 2 different boxes. I opened the first one and found my Poweredge 750 server as it should be. But in the 2nd box, I was suprised to find a Dell Precision 650 workstation with Dual Xeon processors. I definately did not order this (a $2000+ system). So I called my friend Chad who is a Dell reseller and asked him about it. He said he’s heard of people being shipped extra things, and said Dell often says to keep it.
I wanted to do the right thing, and only keep it if Dell said yes. So I called them up, explain the situation to them. I read them the service tag on the workstation (they didn’t even have record of that service tag in existence!). They scheduled DHL to come pick it up the next day. Boo hoo, oh well, I didn’t get to keep it, at least I was doing the right thing and returning it. It’s not like I had ordered it and never received it.
Well, we got the new server installed. We being Tim and myself, Tim is my new system administrator. We drove down at 9pm Tuesday night to US Internet to install it in the rack. We got done a little after midnight and started the 2.5hr drive back to Duluth. When I turned off of Hwy 694 onto 35W to go towards Duluth, there were a lot of cop cars parked on the shoulders. So we slowly drove past them and got on our way. About 3 minutes later, a car comes rushing up behind me, I check my speed, and I’m doing 78 in a 65. “Oh Great!” I said as the cherries lit up on the cop car behind me.
When the cop walked up to my car, he asked us where we were coming from. So I said “We’re coming from Duluth, headed back to Duluth” The cop did a little double take “Say what?”. So I explained that we’re from Duluth, and came down to install a new server in Minnetonka for our internet business. The cop then asked for both Tim’s and my drivers licenses, and said that there was a crime committed in the area, and my car matched the description. So he took our licenses, did some background checks, asked to look in my trunk and then let us go. I was kind of bummed I didn’t get to sit in the back of the squad car, since it seems like something fun to do at 1am, especially when you know you’re innocent.
So Tim and I were finally on our way back to Duluth, we were making good time, doing about 80mph, when all of a sudden 2 deer appear on the freeway. Tim yelled “Lookout!” and I swerved, but ended up hitting one of them. I didn’t hit my brakes until after hitting the deer. It was so fast it was one of those “What was that?” moments. I slowed down, but didn’t stop driving. I looked over the hood of my car and couldn’t see any damage, but I stopped at the next exit to get gas and check the damage.
Luckily, all that happened was a small dent in my hood (barely noticeable) and my front bumper is cracked. The headlight is fine, albeit a little loose now, but still properly aligned. It looked as though I ended up driving over the deer. There were some guts in my intake/radiator, and blood streaking along the side of my car. (I drive a white car). So after getting gas, I got a carwash to wash off the blood/guts. Thankfully the rest of the trip back was uneventful.
Aside from all the excitement driving late at night, another thing that has been happening the past few weeks is server problems. And lots of them! It all started when some malicious user uploaded an IRC bot to my main webserver ‘Phoenix’. This crashed the server, so we went in and cleaned that up. In the process of installing a new firewall, the kernel died, so I had to drive down to the cities at midnight to fix it. Yay! I got back to Duluth at 8am, and headed up to the end of the gunflint trail (4hrs north of Duluth, spitting distance from canada) to work on a contract for a client, and got back to Duluth at 9pm. Then we went through our other servers updating them to make sure the same kind of malicious files couldn’t be uploaded. The same kernel problem happened on another server, so at 10pm I got to drive back to the cities to fix the issue. I got back to Duluth at 5am and was able to sleep for the first time in 40hrs.
Fast forward to the past week, that initial attack on Phoenix left the server a wreck. Luckily, we just installed a new web server ‘Unicorn’ so we began migrating customer account off of Phoenix and onto Unicorn. Along with the migration came all the problems with DNS propagation, IP changes etc. Couple this with a buggy Apache update from Psoft (the makers of H-Sphere), apache kept dying on Unicorn. This did not bode well with customers that were promised the move to Unicorn would solve the server issues.
Well, finally almost all of the accounts have been successfully migrated to Unicorn, another Apache update has come out so Apache is running stable now. But last night I was adding the final few IP’s to Unicorn, and for some reason, the server went down. The Tech at USInternet said it was booting just fine, but wasn’t showing up on the network. So at 10pm last night, I got to drive to the cities yet again. I brought along the new Extreme Networks Summit24e2 managed switch I bought to replace the unmanaged Dell powerconnect we were using.
I get to the datacenter about midnight thirty (12:30am). Log into Unicorn, and find out I made a mistake in my IP configuration file with the netmask, which unbound all of the IP’s from the server. Great! After 1 change, the server was back online (without even needing to reboot!). I then installed the new Extreme Networks Switch, and took the Powerconnect with me. I was out of the datacenter by 1am.
About 1:30, I’m driving through Blaine, just got onto 35w (about where I was pulled over last time) and I received a phone call. It was the datacenter telling me that none of my servers came back online (they all went down while I was there). I instantly knew it was the switch. I failed to test/check and make sure everything was working properly. So I turn around, and get back to the datacenter again at 2:15am.
One look at the switch and I smack myself in the head. My Dell powerconnect switch was auto-sensing with auto-crossover. I had a patch cable going from USI’s switch to mine, and on my new switch, that port was dark. I moved that cable over to port #1, and it the crossover button and the port lights up. Instantly all my servers come back online. Ugh, lesson learned here, even though it’s late, and you’re tired, and you have a 2.5hr drive home. Take the time to double check everything is working.
So I took that time to use a workstation in the NOC to make sure all my servers were online, and I responded to a few trouble tickets/emails. I didn’t get out of there until 3am, due to chatting with the USI Tech.
Passing through Blaine again at 3:30am, I decided I couldn’t make it all the way back, so I stopped to sleep in a Home Depot parking lot. Woke up at 7:30 and made it back to Duluth by 10am.
For some reason, I’m not looking forward to school starting next Tuesday. My life has been hectic enough without having to worry about homework, and class, and grades. I’m thinking of taking the semester off and just getting things stablized with my business. I can’t wait for Tim to come back from his vacation so he can worry about the server stuff, and I can focus on everything else that I have on my plate.