Sunday, June 30, 2013

Maintenance on Dell PowerEdge running ESXi

VMWare ESXi is really nice and easy to manage. Easy to create and maintain the VMs.
But as a bare-metal virtualization software, it makes several tasks a bit more complicated onto the server maintenance itself.
Should you have a “standard” OS such as Windows or Linux, there’s always packages to update drivers and firmware.
There’s no straightforward solution when running ESXi. We have to find a work around.
Also, the Open Manage Server Administrator to maintain the Dell server is not that simple anymore. We should have to add one more layer.

1. ESXi backup
Firs off, the backup.
I installed ESXi on an internal USB stick (8Gb). The server i configured to automatically boot on it.
That saves a little bit of space on hard drive, but it offers a much more nicer option. The backup. It is easier than it could look at a first glance.
I found a small and very efficient software to copy an entire USB stick as an image. It’s of course possible to restore it on an other USB stick (same size or greater).
Should your USB key burn out for some reason, just copy back your image to an other one, restart the server, and done !
Nothing more easy, find out more: http://www.alexpage.de/usb-image-tool/

2. Open Manage Server Administrator (OMSA)
Managing the OMSA on a server running can be a little bit tricky.
It must be done in two steps.
First , you should have an available server to host a webserver. Then install a small agent onto the remote ESXi host itself.

I have a very small CentOS guest OS (1Gb RAM defined, 10Gb hard drive) on my ESXi which host this webserver (this is also configured as a vmclient for some remote commands).

So, here I downloaded from the Dell support website (depends of the brand of server) the two pieces of software :
a) OpenManage Server Administrator Managed Node (RHEL6 - 64bit),v7.3.
That’s the webserver part. To be installed onto my CentOS guest OS.
b) OpenManageServer Administrator vSphere Installation Bundle (VIB) for ESXi 5.1,v7.3  
That’s the agent part to be installed on the remote ESXi host.

2.1 On the CentOS webserver host, I first uninstall any previously installed version of OMSA:
rpm -e $(rpm -qa | grep srvadmin)
Then, once extracted the tar.gz file, install the new one:
./setup.sh, choose default answer for Web Installation.

2.2 On ESXi, stop all the the VMs, put the host in maintenance mode, connect onto the host with ssh.
Under /vmfs/volumes/vm, create a new directory dedicated to the agent installer of OMSA, and copy inside, the new agent. Don’t unzip the given file.
Run the following command (here for the OMSA7.3):
esxcli software vib install -d /vmfs/volumes/506c2d23-01dbe48d-7940-001ec9deb63b/OM-SrvAdmin-Dell-Web-7.3.0-333_A00.VIB-ESX51i/OM-SrvAdmin-Dell-Web-7.3.0-333_A00.VIB-ESX51i.zip

Reboot the host, and check the new OMSA version:
esxcli software vib list|grep -i open
Exit the maintenance mode, restart the VMs you want to restart, here I restarted my CentOS guest hosting the webserver.
Finally, check the new version online (you may have to accept a new certificate from within the browser).

From the OMSA login page, the “About” gives the version of OMSA from the webserver side:
OMSA_002
OMSA_003

Connected to the managed host (the ESXi host):
OMSA_004

Once connected, click on the top right “About”, it will give the OMSA version from the managed agent (the ESXi host).
OMSA_005
OMSA_006     

3. Firmware update
This is probably the trickiest part and task to achieve. 
There’s no way to install any rpm on ESXi host, no way to run a direct single command to check the new firmware version from the Dell Support website.
Until I discover a pretty  nice tool, Dell Repository Manager. It is supported only on Windows (2003 and above), quite easy to install. It allows us to create our own packages according to our hardware specificities from a bundle of firmware/drivers update.

Here below, I’ll work from an example which came out.
From the Open Manage Server Administrator, a new firmware version is reported for the Perc6i Integrated piece of hardware embedded on the PowerEdge 2900III server.

 DRM_001

This is obviously available in the Dell support website, you can download it, but there won’t be any way to run it from the ESXi host. From any of the guests OS neither, they have no idea about the hardware.

For the good of the example, let’s go to a Windows 2008R2 guest OS.
Open the Dell Repository Manager (DRM): 
DRM_002

It checks for new catalog update immediately, download it.
DRM_003
 DRM_004

It creates a new repository with the newly imported catalog, open it.
DRM_005

There’re now all the newest bundles available for the Dell servers.
Let’s select only and only the one(s) we could be interested of. My PowerEdge 2900III is a tower server:
DRM_006 

Choose your server brand, mine is a 2900 tower series, take the Linux OS and the most recent date. Only one bundle is now given (here the one from the 6th of June 2013):
DRM_008

You may want to export the bundle and create an ISO file right away, that’s quite possible and probably easier.
But I want to do it only the component update that I’m looking for (as shown above, Perc6i Integrated). Go to the Components tab, be sure the previously given bundle is selected:
DRM_009

Again, select your brand of server, here tower server 2900, select the device you want the firmware for, here the Perc6i Integrated is a storage controller. It gives one and only one row, with the expected version as shown earlier by the OMSA alert (6.3.3-002, from the 29th of January 2013). See the size, it’s only 10Mb whereas the entire bundle was about 450Mb.
Select the component, and click on “Copy to”: 
DRM_011

Now, we will create a new bundle from the selected component.
Here I have only one component but you may do the same for more components:
DRM_012

So, create the NEW bundle is what we want:
DRM_013

Choose a bundle name which say something to you for further reference. Choose Linux OS, here it’s a bundle specification, it will be use later to create the ISO file:
DRM_014

Do not select any of the specific OS:
DRM_015

Once more select the server you want to update (surprisingly, it’s at least the third time we have to do it):
DRM_016

Check if everything is correct, and click “Finish”:
DRM_017

Finally, the bundle is created and ready to be exported:
DRM_018

Go back to the “Bundles” tab, select the newly created bundle and click on “Export
DRM_019

Now, we will create an ISO file that will be use to boot the server with and update the firmware. That would have been the case also if you exported the standard delivered bundle:
DRM_020

To do so, check the “Bootable ISO” option (remember, you created the bundle with Linux OS type):
DRM_021

If any new plugins are detected, they are downloaded:
DRM_022

We do not have any customized script:
DRM_023

Happy ? Click “Finish”:
DRM_024

In the end of the process, a job is submitted:
DRM_025 

At the bottom of the page, check and wait for the job completion:
DRM_026

Once it is finished, we can check the file and size in the directory. Note the size, 200Mb, much more than the size of the component update (10Mb)…:
DRM_027
Burn the ISO as an image on a CD disk. 

Put the ESXi in maintenance mode:
 DRM_028
DRM_029

Put the ISO disk in the server track and reboot the server:
DRM_030
DRM_031

After few seconds only (hey, only one component is updated here), the server is rebooted again from within the script (auto-executed from the ISO disk).
Remove the disk out of the track, exit the server from maintenance mode and restart the VMs you want to have up and running. Here I’m restarting the CentOS guest OS which host the OMSA webserver.
DRM_032

In the end, come back to OMSA and check the new version of the component which has been previously reported as obsolete. It is now the newest one:
DRM_033

Whether it looks tedious, it is not. There’re of course a lot of tasks, but it’s all for the good !

Enjoy,

Nicolas.

No comments: