Meandering Chatter

I meant to post this last week, but we know how that goes…

The original title was going to be, iSCSI Boot w/ an HP DL380 G5.

We first purchased three of these units back in …  < SEE PO DB > and got them racked and powered.  We allocated one of them to replace NISC40, an image server for a Linux Lab.  The other unit we christened CP3 (a replacement for CP2) which will run CPanel for our Academic Faculty, Staff, & Depts..  We never really got around to configuring the 3rd one.  I believe I spent some time trying to get iSCSI boot working but I’m not really certain.

Several years later, it’s time for me to revisit the project.  I began by downloading the latest binary from HP and reading the updated documentation.  The process is fairly straight forward.  You configure the Option ROM on the NIC by providing the necessary iSCSI parameters.  You tell the BIOS you want to boot iSCSI from the NIC, you reboot and hope that you establish a connection and that somehow your system recognizes the iSCSI LUN as a boot disk and off you go.

I was able to update all of the firmware and what not on the box using a USB key and an HP FirmWare CD image.  I was able to find out what IP Address we had configured the iLO2 port for and uploaded the iSCSI Boot configuration file.  I then modified the BIOS to enable iSCSI Boot and to load the Option ROM, but I wasn’t able to establish a connection.  I tried changing a few parameters in the iSCSI Boot configuration file but was unsuccessful.  To top it off, there wasn’t any error messages displayed on the console, so I didn’t know what exactly was going on.

So, in order to try and find out what was going on I configured an old Mac laptop w/ Wireshark (an tedious task that ended up consuming a day and a half, but thank you MacPorts) and decided to sniff the network traffic to see what exactly was going on.  I believe I discovered two major issues.

1.  It appears that the Option ROM doesn’t support iSCSI Load Balancing.  That is to say it cannot communicate w/ our VIP and then change its connection to the specific module, as told by the VIP.  No problem, we’ll just tell it to communicate w/ a specific module.  That workaround, however, raises the question as to what happens after initialization, after the OS boots if redirection has to occur, will the OS comply.  There are a few issues with this, but for the moment I have a workaround.

2.  I suspected that network initialization between the NIC and the port on the switch doesn’t occur fast enough for the Option ROM to establish a connection and login to the iSCSI target.  This doesn’t occur if I have a small hub as an intermediary since the link is already active.  Aaron suspect that we might be able to hard code the port to Gig speeds @ full duplex to get around this issue.  Before I do that, I’d like to make sure that I get iSCSI Boot to work.  However, I’m now thinking that this might be more important to verify now, rather than later.  It would be a great pain to have come all this way to get iSCSI Boot working, only to find out that there is no way to delay the initialization of the Option ROM.  Being stuck at 10 Base-T @ half duplex would not be ideal.

Well, with those two out of the way, my attempts to configure iSCSI boot have so far failed.  My first attempt was to simply copy the existing OS installation onto the iSCSI LUN, make the appropriate changes, and hopefully boot that way.  Wishful thinking…  There is still far too much that is unknown for that to successfully work.

My next step was to configure iSCSI Boot and try a clean OS install, as documented in the documentation.  There is a KickStart file that helps configure the installation to support an iSCSI LUN from the Option ROM.  I managed to get that initial portion to work, however during the end of the install the post scripts failed and brought the entire thing to a screeching halt.  It looks as if the installation didn’t properly create a kernel image on the /boot lun (no initrd file to be found).  I spent some time mucking about the install trying to see if I can create said kernel, but w/o the working install directly I didn’t get too far.

I decided that I would try again w/ a more recent OS release (RHEL 5.4).  So, at the end of the day on Friday (10/2/09) I downloaded a newer install image and burnt it to DVD, on Monday.  Hopefully the post install scripts in the KS.cfg file will work w/ this release.  If they don’t, I’ll be forced to post to HP and give their support line a call to see if they can help.  Thankfully, the install completed and the post install scripts of the KS.cfg file ran.  Unfortunately, iSCSI boot still didn’t work.  The iSCSI connection was initialized at boot by the Option ROM, but GRUB failed to load.  All I was able to get was a black screen w/ the letter GRUB in the upper left hand corner.  No prompt (GRUB>) just GRUB.  I troubleshooted this for a bit on Monday.  I though that perhaps the MBR wasn’t properly installed on the iSCSI LUN.  Somewhere in the process of trying to fix that I hosed the /boot partition on the physical disks installed in the server and had to spend a few hours getting that back to normal.

If I can get iSCSI boot to work, that would be great.  It would mean that any future RHEL builds (although none are planned and it is highly unlikely that we will continue to run RHEL unless specific software required it) can be booted from the SAN and have backups (via SAN snapshots) taken care of.  The reality is that we will probably employ Ubuntu Server on most new Linux boxes.  The unfortunate aspect of this is that even if I do manage to get iSCSI boot working, I’ll have to perform a new clean install of CPanel in order to begin migrating existing Academic hosts from our older install.  Regardless, it is a worth while project to spend my time on.

The good news is that since the RHEL install worked and the post scripts ran I had a somewhat capable initrd image to work with. I tried copying that to the /boot partition of the internal disks and tried using a local /boot to mount an iSCSI root (/).  That failed, but at least it would get to a certain point and then fail.  I decided that I could probably take a look at the initrd image and hack it in order to initialize an iSCSI connection and possibly mount /root from iSCSI.  I would have gotten to it sooner except for the fact that a backup volume on my backup server filled and I was forced to repair the volume / server for the entire day on Tuesday.

Which brings me up to today.  I’m happy to report that I have successfully managed to get this system booting /root from iSCSI.  Now, if I can modify one of the original initrd files to include the additional drivers and scripts of the latest RHEL install I might be able to copy the local root to another iSCSI LUN and try to boot root off of iSCSI w/ the OS and our CPanel build installed and configured.

Joy,

–Raf

I was going to post some massive update to cover June, July, Aug., & Sept. but I need coffee at the moment.  Oh well, maybe next time (in January).

–Raf

I’m still feeling ridiculously busy, even though work is pretty quiet because it’s Spring Break.  There are many projects that we’re finalizing at work, and more along the way.

We finally got around to finishing up dovecot testing w/ various e-mail clients, two weeks ago.  Vasantha and I put the new dovecot in place on Sunday while Aaron & Sarah updated out core network infrastructure, things look good.  Aaron is on schedule to replace our routing core tonight.

Regarding Dovecot, there was a moment where we though we might have had to update / patch our mail server, however we were able to get around that.  We’ll probably have no alternative but to patch our mail server once it comes time to update sendmail.  We did eventually discover an issue w/ the subscriptions list, but we were able to work around it pretty quickly.  Like I said, things look good.

On the home front, Mali and I have finished phase 2 of our kitchen redo.  We’ve both been working pretty hard the last couple of weeks putting it all together.  We still have to put down our floating floor, but that will wait until later this week.

Part of the reason that I feel so stressed and busy is that I also have a lot of ideas regarding some house related projects that I would like to undertake, in addition to my regular work during the week.

I’d like to begin planning on the initial stages to plan out the back yard and build a few planters for Mali’s garden.  I’ve been thinking about making some kind of book shelving monstrosity for the library/study/lounge.  Then there is the fact that we’re rearranging the layout and purpose of the rooms we initially setup.

The “Entertainment Room” is going to be converted into a formal dining room and the “Entry Room / Grand Room” is going to be converted into a formal grand room / living room.   Both of these require moving the TV.  My TV is broke and so I have to see about getting it fixed or simply getting rid of it.  It would be ashamed to not be able to get some money for it.  Additionally, there doesn’t appear to be any way to move it into the living room w/o complicating the layout of the furniture, so it might just have to go away.

Our library / study / lounge will continue to be just that.  Eventually, we’ll need to rearrange the things that are in there and figure out what to do with the rest of the room.  I’m pretty certain that all of these will take more time to accomplish that we anticipate.  We’ll just have to see how much we can get done this summer / year.

We had a big party this Sunday to celebrate our new kitchen an a few birthday parties.  We made another huge batch of tamales w/ the help of some friends.  We’re all talking about possibly making more tamales later this month.

Always a blast,

–Raf

Emotionally, I feel busy, very busy…  Not quite over-whelmed, however I’ll have some long days and nights ahead of me if I’m to finish all that I’d like to finish.

Friday -

Friday evening, Mali and I when shopping for most of the hardware we needed to finish up our kitchen.  We purchased our faucet and a small base cabinet unit, to use next to our fridge, from Ikea.  We’ll probably return once more to purchase a bar table once we get the kitchen in order.  We then went to Lowe’s to pick up the rest of the hardware that we needed for the plumbing and electrical work that we planned on doing this weekend.

Saturday -

We tore out our kitchen sink on Saturday.  I managed to get most, of the plumbing done, and I prepped the dishwasher for installation.  We cut the piece of wood that we’re going to use as our end panel and Mali began painting it, as well as the drawers from the buffet table that we’ll be using in what will become our formal dining room.

Sunday -

Finished the plumbing work and began prepping the sink for installation.  We also cut notches out of the counter top to accommodate the sink and prepared the sink cabinet by removing the front and rear boards that provided stiffening support (I’m not too worried about it, but we’ll see how well it holds up).  The Ikea sink provides to long rods to use a stiffeners; I installed one of them, but will have to wait to see where to install the 2nd.   Unfortunately, since we were still waiting for paint to dry, we didn’t get much else done.  Had lunch w/ Gabe & Rahul @ Cuba Libre.  Upon returning from lunch,  I installed the new faucet on our sink, as well as the included strainer and the strainer/garbage disposal mounting unit.

Monday -

• Tapes
• Started using our new secure-wireless network, now w/ 802.11 A and N; not that I have the hardware to utilize these new features.
• RAM Upgrade for a PowerBook G4 = DONE

At home, I installed the power cord for the garbage disposal.  I’ll have to spend some time working on routing the new power receptacle and switch later this week.  I then read up for class.

Tuesday -

• Updated Ubuntu desktop, took some time to get VMware Workstation back in order (it always does).

Not much else planned for today.  I have to help Vasantha test a new version of Dovecot and write a short paper for my class tonight.  I won’t be doing much @ home tonight since I’ll be getting home so late in the evening.

–Raf

Tuesday -

• Tape Inventory
• More EmuMail Migration work…  Updated the list of valid LDAP users (2462) w/ EMU Mail folders (776) that have been accessed in the past year (225).  My initial count was 295 but that was from an old list of valid LDAP users that included 3268 users in all; before the deletion of last year’s class, back from September of 2007.

Wednesday -

• Backups
• NISC8 reboots – NISC8 wasn’t cleanly coming up after its reboots.  I converted its daily reboot schedule to a weekly one to minimize the amount of attention we spend on it every morning.  A weekly reboot should be suffice to clear its memory leak and return it to normal service.
• AHT & ASD Images on bootable USB keys.  Apple has a new procedure to use external USB drives for its Apple Hardware Test (both OS & EFI versions).  We purchased a few 8 Gig USB thumb drives and I was able to get the new AHT on them, but I still need a fairly recent computer to test them on.
• RAM Upgrade for an old PB G4 (12″, 1.33 GHz)
• EMUMail conversion – We ran another EMUMail conversion (our 2nd since announcing our phase out plan).  Things went well except for the fact that EMUMail will sometimes fail to update the number of messages in each folder after message deletions.  It makes accounting a bit of a pain.

Thursday -

• Replacing a SuperDrive in a MacBook (13″)

This week has flown by.

One of the disks in my Mac @ home has died a horrible death, making all kinds of grinding noises and the like.  It is not allowing the other drive to show up so, either the bus is dead, or its a single drive that is preventing the other from showing up.  I should pick up a 500 Gig ATA drive before they become extinct.

On the homefront end of things, Mali and I will be working on the kitchen this weekend.  I’d like to get the the countertops installed, the dishwasher prepped for installation, and the end panel of our cabinets (to hide the dishwasher) figured out.  If we can manage all of that we can probably begin to do the plumbing and install our new sink on Sunday.  We’ll see how much progress we make.

Laters,

–Raf