Category: Life News

Surfacing!

You can take that in a couple of ways. First, at least chronologically, we have surfaced from our dives and our vacation in fine diving form. In fact, we both took and passed our Advanced Open Water diving certification!

Yay, us!

It was a very good refresher course for us, so much so that Laura even got so excited about diving that she didn’t want to stop!

We loved it in the Keys – it was very relaxing and laid back. We are definitely going back… just not to Miami. Miami wasn’t so great – way too hectic and crowded for a vacation.

Secondly, and most important, I’m typing this blog entry on my brand new Surface! Full disclosure dictates that I tell you that Microsoft is my employer, but even so – this thing is way cool. I’m going on a couple of hours with hardly a dent in the battery. It’s a little small, but that’s to be expected of a tablet device.

Any way you slice it, It’s a cool machine.

It’s Difficult To Be A System Administrator

[WARNING: Geeky Content Ahead]

Sometimes, things go well – most of the time, usually. But then, you have things like massive power outages when you’re in the middle of doing something to prevent catastrophic data logs when the power goes out. And what you were doing in the middle of the power outage actually CAUSES catastrophic data loss. 😦

Two weeks ago, one of my hard drives on my personal file server failed. It’s a 1.5TB hardware RAID1 (mirror) array. For those who don’t know what that is, I have a special device in my server that allows me to build redundant arrays so that in case of the failure of a single disk, no data is lost. That’s what I have: two 1.5TB drives mirrored to provide full redundancy. One of them failed and the server started to beep… telling me “fix my drive!” So I did. I got a new drive and it spent the next 20 hours or so rebuilding the 1TB of data that is on the drive. Problem solved!

Then, I did something dangerous. I started to think.

I thought “Hmmm… that hard drive failed way too early in it’s lifecycle. What if my other ones fail to?” You see: the drive I just replaced was only my personal data [which is very important] but it wasn’t part of my infrastructure. I could use that data on any machine, but I’d have to rebuild a ton of stuff to get to it if I had many more failures.

Just so that you understand an overview of my “home server farm”, I have [for purposes of this discussion] 2 host servers that serve up virtual machines (VMs). I have somewhere around 30 VMs spread across these hosts. Most are work-related, helping me to design and build solutions for my customers and do research, and some are for my personal use, such as a desktop, web server, email, and the file server mentioned above. There’s also a few machines to maintain the farm – Domain Controllers, DNS, Certificate Authorities, etc.

Each one of these computers is actually a file on one of the two host boxes – and those files are rather large. If something were to happen, say a power outage, there’s no guarantee that my battery power would last long enough for me to shut down the machines cleanly.

With all that in mind, and the idea that in this instance a “server” is actually a big file on the order of 40GB to 200GB in size, I thought that making the disk upon which these files sit a mirrored array would be a smart thing to do. Which it is.

So: on Friday afternoon, I began the process of creating mirrors on two of my servers. One server, with most of my work VMs on it, has no RAID card. On that one, I used the Windows OS to mirror two 1TB drives after shutting down the machines and moving the files off of one. Once I had that started, I moved to the other server, my personal one, and did the same thing – shut down the VMs and moved them to another disk. Then, I repurposed the vacant disk and joined it to the one which now held the VMs and began building a mirrored volume.

Now, with the RAID card, I can do this on the fly while the disk is available. Before I turned the machines on, it said it would take about 2.5 hours. I turned the machines on. Now, it said it would take about 20 hours.

I should have left them off.

Well after 2.5 hours later, at about 10:30 that night, the power went off. 14 or so hours later, it came back. I went down to power things on. It all “looked” okay for a while – I was getting email again, but the web server was wonky and slow and some other things were just kind of weird.

Looking further, it appeared that the rebuild had to be restarted since it had lost power. I restarted it. [Note: the work server using the Windows OS RAID simply came back on automatically and began rebuilding the mirror and it completed with zero errors.] About 2 hours later, there was a loud obnoxious beeping from the server closet. The rebuild had failed and the drive simply dropped offline. Gah!

All my VMs disappeared for a moment. Rescanning the array with the utility made it come back, but now I was very worried. Since the VMs were all off, I copied all the files to a second drive and build the array from scratch [after several attempts to find and fix whatever bad secords or corrupt tables were on the drives]. I moved the files back after it completed. I turned on the VMs only to have half of the machines not come back – the half that mattered of course. One domain controller, my web server, desktop and the email server were the biggest losses. I had the old disks, but I had to actually reinstall the OS on all of them and begin the slow, painful process of restoration.

Which is where I am today. I have the web server working [obviously] and we now have email with empty mailboxes. I have a recovery database ready to go, but there are issues with the old database so I need to finish patching the Exchange server so I can get it to the same version that it was before so the recovery tools will work properly. That’s what I’m doing now.

All of this work has taken 5 days or so to get things back up. I now have most of the critical VMs housed on RAID drives. I just need one more to complete the process.

At least I learned more about doing Exchange server mailbox recovery.

Power Outage

Many of you may have noticed the site down for several days. That is directly due to the power outage. Not that we’ve been without power, mind you. Power came back within 12 hours of loss. (Yay! Air Conditioning!) What happened was that I was in the middle of moving critical files from one disk to another when the power failed. The failure damaged several of my servers including the web server and the email server. I have no email for now, but hope to have it up soon – even if I have to do without the old stuff.

But, as you can now see: the web site is up and running.

Thank you for your patience!

Travel Rememberences

Today, Laura and I sit in our room overlooking the San Diego Marina and try to relax a little. This is a very different travel experience than I’ve had recently.

Up until the a couple of weeks ago, I’d been assigned to several projects at once and the resulting travel was chaotic, frequent, tiring, and unsatisfying. It brought up memories of working for IBM – back when I was on the road every week.

This travel “binge”, however, seemed worse to me. I think it was worse because I was never in the same place twice and was therefore always feeling uprooted and unable to form a workable routine. My workout routine practically died, and I gained some of my weight back. I was always tired and unable to focus on anything but work.

This trip, though, isn’t about the customer: it’s about me. That make a big difference. I’m here for training, which while work related, means that I’m here to learn and take in instead of “give out” and I need some of that now.

I’ve finished one rather unsatisfying project, and worked myself out of another, so I’m down to a single, primary project and that takes a lot of stress off of my shoulders.

After a week at home to rest up a bit, going on this trip with Laura feels almost like a vacation – even though I’ll be spending my days learning things.

Oh: and I get to add a new airport to my list as well… 🙂

Lots of Travel

Normally, I’d travel to a new place and blog about it and such and have a good bit to say. Maybe you’ve noticed, but I’ve been a bit quiet here lately. That’s because I’ve been doing TOO much travel. Yes, that’s right. Too much.

I’ve been going from place to place so much, that I’ve been so tired and emotionally drained that I haven’t had the will to sit down and type.

In fact, I don’t really have it now, but I felt that I needed to put something down.

In the last month, I’ve been to Fayetteville, NC, spent the weekend in DC for Photoshop World, spent a day in Laramie, WY, a day in Breckenridge, CO, and a week in Denver, CO. Then, I just returned from Burlington, VT last night. Next week, I’ll be in Fayetteville again, followed by a weekend jaunt to go plant trees in Pennsylvania. Then I should have a break… for a week or so. Then, Laura and I will be headed to San Diego, CA. I’ll be there for a conference for most of the time, but we’ll be extending a couple of days to celebrate our 9th anniversary. This will, of course, add another airport to my airport list. Again. Just like Burlington did.

I’ll try to post more later when I have some time about the places I’ve been over the last few weeks.

Maybe I’ll even feel up to it. 🙂

Progress

I’ve been trying to exercise more lately and I think I may have made some progress. How do I know? Other than loosing a few pounds, which I have, one of the other ways I know I’ve been exercising a lot is that my running shoes are now toast. Fried. Completely worn out.

The last run I did, which was 4.35 miles, really put my poor feet close to some large blisters. The shoes I was using worked very well for me up until lately once I started running again. Now that I’m running more, the age and mileage on the shoes began to chafe a bit.

So, I bought a new pair today. I’ve got to break them in, so I think I’ll be doing a few short runs during the week next week before I hit the 4 mile mark next Saturday.

Go, shoes, go!

Old Black Water, Keep On Rollin’

I don’t think it’s what the Doobie Brothers were thinking about, but I’m sitting at my desk with a glass of black water. Yes, BLACK water. One day a couple of months ago, Laura was shopping and came across a black bottle of water with a label that said it had a high pH level: 9.0 – which is 2 points above the normal 7 of regular water. This is supposed to be healthy due to the fact that we eat too much acidic food (like sugar, etc.) So she bought it to try out.

She got it home and I see this bottle sitting on the counter. “What’s that?” I say. “Oh. It’s a new high-pH water I saw. I want to try it out.” Great. I pick it up to look at it. The bottle is so dark, you can’t see anything through it… until I look closer. Seems it’s not the bottle that’s dark, but the water. It’s actually BLACK WATER. For real. See?

202-1290

It’s been sitting on the counter for two months after I noticed that the actual water was black. Tonight, it bothered me enough that it was just sitting there, so I popped it open and poured it over some ice. It’s not bad. It has a very minerally taste. Like there’s sand in the water. But, it’s healthy. Right?

If you’re curious, go to their site and read more.