Archives for September 2005

For some odd reason, today has just been the bugger of all mondays. I get in to the office this morning, only to find I have no internet access. Jim and Lyle reconfigured the firewall for the building’s T1, so now my office has no internet. I’m supposed to change my IP settings in my router to fix it. It should be fixed, but now the CSU/DSU for the T1 is giving errors, and no one has internet in the building.

What makes it even more fun is that I finally have my business phone line routed to my office, rather than my house. So the line is plugged into the PBX box for the building. Unfortunately, Jim is to busy to set up the PBX system, so when you call my phone number, it gets picked up and automatically routed to New Tech Concepts. Great! I’ve already received calls on my cell from customers asking me about this.

Last week, I purchased a copy of ACT! by sage. It is supposed to be this great CRM software for managing client relations. Well, it would be great if the software worked properly. For some reason, it is having problems rendering things on the screen. The images get screwed up when viewing the data fields, and the calendar does not show up at all.

What a way to start the week

Ahh yes, the week after labor day! School starts once again. I just need to keep reminding myself “I’m $30,000 in debt with student loans, I need to finish my degree”. I guess I now know how Bill Gates kinda felt when he was in college. With the pressures of my business demanding more and more of my time, I honestly have no idea how I’m going to find time for class this year.

I have 18 credits left to get my Bachelors degree (that is unless I need to retake some of the classes I got D’s in). These are going to be the longest 18 credits of my entire college career. This fall semester, I’m taking ‘Archeology of Cities’, ‘History of the Holocaust’, and ‘Strategic Management’. Why in the world I’m taking the first two classes, I don’t know. But in order to graduate from UMD’s School of Business and Economics, you need to have so many upper division electives from other colleges. And those 2 classes were some of the only ones that didn’t require lower division pre-requisites.

The temptation to just quit school and continue my business full time is a big one. But then I’d be $30,000 in debt for nothing. At least when I get my degree, I’ll have a $30,000 paper that I’ll probably never use, but I’ll at least have something.

I just got off the phone with EV1.net tech support. Do I have a server with EV1.net? No. But I did about 2 years ago.

The reason I called EV1 is because I had been receiving complaints of spam and hacking attempts coming from my servers. But I couldn’t find any evidence of such.

Finally, I found out why. The person who was experiencing the spam/hacking attempts was getting them from a certain IP: 216.127.XXX.137. And when they did a reverse DNS on the IP, it came up with one of my domain names. So they contacted me!

I didn’t recognize the IP address in question, it definately was not one of mine. So I did a reverse DNS/IP whois on the domain, and lo and behold, it belongs to EV1! Apparently, when I cancelled my server with them 2 years ago, they didn’t remove their reverse DNS entries. So the new owner of the server was causing trouble, but my domain still showed in the reverse DNS.

EV1 is now removing those reverse DNS records, and I commend their support, as it was a rather painless phonecall to make to get everything cleared up.

So, the lesson to be learned here, if you have ever had servers somewhere, make sure that they remove the reverse DNS records after you leave their services!

Well, the past 3 weeks have definately been interesting. This is going to be a pretty long post, since it covers a lot, so I’ll try to break it up.

So on Monday August 22nd, I received my new Dell Poweredge 750 server I ordered. But when it arrived, there were 2 boxes. Both had the same packing slip (with my server listed on it), but they were 2 different boxes. I opened the first one and found my Poweredge 750 server as it should be. But in the 2nd box, I was suprised to find a Dell Precision 650 workstation with Dual Xeon processors. I definately did not order this (a $2000+ system). So I called my friend Chad who is a Dell reseller and asked him about it. He said he’s heard of people being shipped extra things, and said Dell often says to keep it.

I wanted to do the right thing, and only keep it if Dell said yes. So I called them up, explain the situation to them. I read them the service tag on the workstation (they didn’t even have record of that service tag in existence!). They scheduled DHL to come pick it up the next day. Boo hoo, oh well, I didn’t get to keep it, at least I was doing the right thing and returning it. It’s not like I had ordered it and never received it.

Well, we got the new server installed. We being Tim and myself, Tim is my new system administrator. We drove down at 9pm Tuesday night to US Internet to install it in the rack. We got done a little after midnight and started the 2.5hr drive back to Duluth. When I turned off of Hwy 694 onto 35W to go towards Duluth, there were a lot of cop cars parked on the shoulders. So we slowly drove past them and got on our way. About 3 minutes later, a car comes rushing up behind me, I check my speed, and I’m doing 78 in a 65. “Oh Great!” I said as the cherries lit up on the cop car behind me.

When the cop walked up to my car, he asked us where we were coming from. So I said “We’re coming from Duluth, headed back to Duluth” The cop did a little double take “Say what?”. So I explained that we’re from Duluth, and came down to install a new server in Minnetonka for our internet business. The cop then asked for both Tim’s and my drivers licenses, and said that there was a crime committed in the area, and my car matched the description. So he took our licenses, did some background checks, asked to look in my trunk and then let us go. I was kind of bummed I didn’t get to sit in the back of the squad car, since it seems like something fun to do at 1am, especially when you know you’re innocent.

So Tim and I were finally on our way back to Duluth, we were making good time, doing about 80mph, when all of a sudden 2 deer appear on the freeway. Tim yelled “Lookout!” and I swerved, but ended up hitting one of them. I didn’t hit my brakes until after hitting the deer. It was so fast it was one of those “What was that?” moments. I slowed down, but didn’t stop driving. I looked over the hood of my car and couldn’t see any damage, but I stopped at the next exit to get gas and check the damage.

Luckily, all that happened was a small dent in my hood (barely noticeable) and my front bumper is cracked. The headlight is fine, albeit a little loose now, but still properly aligned. It looked as though I ended up driving over the deer. There were some guts in my intake/radiator, and blood streaking along the side of my car. (I drive a white car). So after getting gas, I got a carwash to wash off the blood/guts. Thankfully the rest of the trip back was uneventful.

——-

Aside from all the excitement driving late at night, another thing that has been happening the past few weeks is server problems. And lots of them! It all started when some malicious user uploaded an IRC bot to my main webserver ‘Phoenix’. This crashed the server, so we went in and cleaned that up. In the process of installing a new firewall, the kernel died, so I had to drive down to the cities at midnight to fix it. Yay! I got back to Duluth at 8am, and headed up to the end of the gunflint trail (4hrs north of Duluth, spitting distance from canada) to work on a contract for a client, and got back to Duluth at 9pm. Then we went through our other servers updating them to make sure the same kind of malicious files couldn’t be uploaded. The same kernel problem happened on another server, so at 10pm I got to drive back to the cities to fix the issue. I got back to Duluth at 5am and was able to sleep for the first time in 40hrs.

Fast forward to the past week, that initial attack on Phoenix left the server a wreck. Luckily, we just installed a new web server ‘Unicorn’ so we began migrating customer account off of Phoenix and onto Unicorn. Along with the migration came all the problems with DNS propagation, IP changes etc. Couple this with a buggy Apache update from Psoft (the makers of H-Sphere), apache kept dying on Unicorn. This did not bode well with customers that were promised the move to Unicorn would solve the server issues.

Well, finally almost all of the accounts have been successfully migrated to Unicorn, another Apache update has come out so Apache is running stable now. But last night I was adding the final few IP’s to Unicorn, and for some reason, the server went down. The Tech at USInternet said it was booting just fine, but wasn’t showing up on the network. So at 10pm last night, I got to drive to the cities yet again. I brought along the new Extreme Networks Summit24e2 managed switch I bought to replace the unmanaged Dell powerconnect we were using.

I get to the datacenter about midnight thirty (12:30am). Log into Unicorn, and find out I made a mistake in my IP configuration file with the netmask, which unbound all of the IP’s from the server. Great! After 1 change, the server was back online (without even needing to reboot!). I then installed the new Extreme Networks Switch, and took the Powerconnect with me. I was out of the datacenter by 1am.

About 1:30, I’m driving through Blaine, just got onto 35w (about where I was pulled over last time) and I received a phone call. It was the datacenter telling me that none of my servers came back online (they all went down while I was there). I instantly knew it was the switch. I failed to test/check and make sure everything was working properly. So I turn around, and get back to the datacenter again at 2:15am.

One look at the switch and I smack myself in the head. My Dell powerconnect switch was auto-sensing with auto-crossover. I had a patch cable going from USI’s switch to mine, and on my new switch, that port was dark. I moved that cable over to port #1, and it the crossover button and the port lights up. Instantly all my servers come back online. Ugh, lesson learned here, even though it’s late, and you’re tired, and you have a 2.5hr drive home. Take the time to double check everything is working.

So I took that time to use a workstation in the NOC to make sure all my servers were online, and I responded to a few trouble tickets/emails. I didn’t get out of there until 3am, due to chatting with the USI Tech.

Passing through Blaine again at 3:30am, I decided I couldn’t make it all the way back, so I stopped to sleep in a Home Depot parking lot. Woke up at 7:30 and made it back to Duluth by 10am.

————-

For some reason, I’m not looking forward to school starting next Tuesday. My life has been hectic enough without having to worry about homework, and class, and grades. I’m thinking of taking the semester off and just getting things stablized with my business. I can’t wait for Tim to come back from his vacation so he can worry about the server stuff, and I can focus on everything else that I have on my plate.