Hypervisor to the Rescue!

I recently asked my fifteen year old daughter what she thought hypervisor was. Her response was pretty funny.

“Is that one of your silly comic book characters, Dad?”

To that I laughed and said no, not exactly. However, the whole idea of hypervisor being a “superhero” got me to thinking (always a dangerous path for me). In a lot of ways in the eyes of many organizations and systems administrators, hypervisor has become something of a superhero. It’s minus the cape, bright spandex, and catch phrase, but a super hero never the less.

In the virtual superhero world there are actually two hypervisors. The first is the powerhouse superhero (much like Superman) of the two and is referred to as type 1. Type 1 boasts the power to be the “native” or “bare-metal” software that runs directly on a given hardware platform and along with many other powers has the ability to act as an operating system control program. It can do this all while residing underneath the operating system (OS) on the bare metal of the server hardware itself, which in itself is a powerful ability. VMware Virtual Infrastructure and XEN are just two of the “secret” identities that hypervisor type 1 goes by in normal society.

Type 2 hypervisor is the second of these superheroes. If type 1 is Superman, then type 2 is Batman. It doesn’t possess all the cool powers, but it does have some nifty gadgets in its “utility belt”. Type 2 lacks the nifty super ability to perform its duties underneath the OS, but still is able to get the job done of putting “rogue” servers in their place by running on top of a “host” OS such as Windows or Linux. In normal society type 2 hypervisor has several “secret” identities including VMware Server, VMware Workstation, Microsoft Virtual Server, and Microsoft Virtual Workstation.

Now you might ask how type 1 and type 2 hypervisors are super heroes. What do they possess that goes far beyond the realm of mortal programs and operating systems? The answer to this is simple. They both possess the ability to take one physical workstation or server and turn it into multiple workstations or servers. I run VMware Workstation 6.0 on my desktop at work. The desktop before VMware Workstation 6.0 is a 3.00 GHz, dual proc, dual core server with 5 gigabytes of RAM running Windows XP 64 Bit (yes, I haven’t upgraded to Vista… yet). With VMware Workstation 6.0 I also run four Windows 2003 R2 Standard Edition servers, two Windows XP workstations, and a Vista workstation. So, I have a total of eight separate systems running on my one workstation at the same time! Also, each of these virtual machines running my workstation is just a collection of files that can be copied and moved to other systems. Since they run “bare-metal”, or in normal terms, without the need to interact with the physical hardware of the workstation, (they instead relying on the hypervisor layer to supply them with RAM, Video Cards, Processors, Network Interface Cards (NIC), and all other hardware needed to run a system) these virtual machines can easily be ported to other workstations or servers of a completely different model, chip set, and vendor should the need arise.

How does all of this allow hypervisor to “serve(r) and protect” your friendly neighborhood systems administrator? For starters, once setup correctly, deploying a virtual server takes minutes instead of hours, saving the systems administrators time and energy and allowing he/she to focus more on other projects. Next, when properly thought out hypervisor will allow a disaster recovery plan to have all of the critical Unix, Linux, and Windows servers up and running within a matter of minutes at a disaster recovery site, potentially saving an organization millions of dollars and making the systems administrator appear to be a “superhero” in their own right. Another “serve and protect” feature is that the administration that goes with maintaining physical hardware in a datacenter is taken away by using hypervisor, be it as VMware, Xen, Virtual Iron, or Microsoft Virtual Server. If there are 10, 20, 30, or 60 servers residing on just one physical server then that leaves only one physical server for the systems administrator to maintain. Granted, that does leave just one point of failure for all those servers. However, most enterprise level hypervisor products allow and recommend an infrastructure of multiple servers to be setup in order to reduce the possibility of downtime. Some even automate the process of failover to existing servers in the event of a server failure. With proper design a virtual infrastructure has many of the same feature sets as a clustered environment and can allow for greater total uptime for the virtual servers deployed on them.

As Hypervisor (said in my best loud and commanding superhero voice) scans the horizon looking for organizations to rescue, a sense of peace and security pours through the hearts of systems administrators everywhere. Be it Type 1 or Type 2, both will assist the systems administrator in his daily routine and create a more flexible and stable environment for the organization he serves. With this peace and security organizations and systems administrators can sleep soundly at night knowing that hypervisor is on duty protecting them from the evils that lurk in the darkest corners of information technology.

Virtualization in IT: Where is it Going?

Five years ago virtualized architecture in an IT shop was “virtually” unheard of. Today, virtualization is the new buzzword. IT shops the world over are turning to virtualization to answer many of their needs for server consolidation, server density, power savings, and rapid deployment of server solutions.

In the past I’ve seen 20 virtual servers run on a blade with 4 gigabytes of RAM and 2 dual core processors. Whiles these virtual servers were only providing printing, DNS, WINS, and domain functionalities, the thought that they could all be run on one server that cost no more than $5K to this day amazes me. With virtualization today allowing for a more segmented approach to deploying applications, the days of bundling multiple server based applications on one server are quickly going the way of the Dodo bird.

So, where is this leading? What does the future hold for virtualization? Those are extremely good questions.

Since virtualization has already proven itself in the server arena, the next proving ground will naturally be the virtualization of the user’s desktops. This can already be evidenced by VMware’s VDI (Virtual Desktop Infrastructure) environment that leverages VMware’s ESX Virtual Enterprise environment to provide users with a virtual desktop that all their own. While centralizing the desktop environment of users has already been accomplished with products like Citrix, VDI serves to provide the user with a centralized virtual desktop that’s all their own and not “shared” with any other users.

This in turn will eventually go one step further with scaled down laptops that connect through the internet to a company’s server architecture and connect from anywhere to their virtual desktop. A move such as this would cut the costs of today’s laptops drastically by allowing for a scaled down version running a minimal configuration for hardware. A move such as this will make the days of important files being lost to hard drive crashes while hardening security for systems. When an employee termination is performed, a simple click of a mouse not only disables their account but also cuts access to their computer and any company information they had on the system.

Turning back to the server world, virtualization will only improve. With companies like VMware and XEN fighting it out over the next several years for market dominance innovations are sure to come quickly. Current problems such as how to quickly and efficiently backup virtual servers will quickly be solved (probably by borrowing from snapshot technologies like those employed by Network Appliance) and new features and functionality will begin to pour out with every code release.

All of this makes it easier on the guys in the trenches during all of this; the systems administrators. I have been told that one system administrator can realistically support 200 virtual servers by himself without significantly impacting his other responsibilities. While this may or may not be true, virtualization certainly does make a system administrator’s job easier by providing him with less hardware to support and an easy means of supporting and maintaining the virtual servers that are running.

In the last five years virtualization has come so far and integrated itself into information technology to a degree that IT shops would never consider going back to standard server deployments. One can only imagine where virtualization will five years from now.

Interesting Quirks I have found when setting up High Availability (HA) in VMware

While HA is a wonderful tool for giving servers in your virtual enterprise a “close to clustered” environment, the initial setup has proven to be a bit flaky. I have had to scratch my head in wonder at times when attempting to setup HA. Everything is done correctly, everything has been checked and rechecked, and when I finally begin the setup I get an error message telling me the setup has failed. At times like this I want to pull my hair out. However, since I really can't afford to do that (I sit in the express chair at the barber shop... the 12 hairs or less chair) I have been forced to rethink my strategies. Should you run into issues with HA not setting up on your servers correctly even though you know that you have done everything correctly, follow the below instructions.

1. Remove HA from the enterprise and then set it up again. Yes, this sounds hokey and silly, but it has worked for me in the past. Sometimes I think HA thinks of the first attempt as a trial run.
2. Should this not work, in Virtual Center right click on each Host server in your environment you wish to add to HA and select “Reconfigure for HA”. I have found in the past that this will resolve the issues after the initial setup of HA.
3. Should this not work, perform a reboot of the servers and attempt the two steps above again.

The last step is one I HATE to do in a production environment. Often, if you are running a large scale environment this means playing Vmotion leapfrog with all of the servers in your environment to accomplish the reboot.

So, You Deleted a VM and Now Want to Get IT Back.....

If you've done this (as I have in the past) you will need to reregister the existing Virtual Machine in VirtualCenter again. To do this, follow the steps below.

1. Select the ESX server in the VirtualCenter where the VM files are located.
2. On the summary tab right-click on the datastore where the VM files are located and click “Browse Datastore”.
3. Select the folder where the VM files are located.
4. Right-click on the VMX file for the VM and select “Add to Inventory".
5. Select a name and location for the VM and click Next.
6. Select the Host or Cluster for the VM and click Next.
7. Select a Resource Pool and click Next.
8. Click Finish and the VM will now reappear in VirtualCenter.

Virtual Sprawl

The life of a VMware administrator can become one request after another for serving up Virtual Machines. At times I myself have felt like a fast food drive through. Various people will pull up to my office and place their order. “I’d like a Windows 2003 Server, extra RAM, extra disk, hold a processor.” I’ve often been inclined to ask if they wanted fries with that order.

With all of these orders coming in for virtual environments, it becomes easy to lose track of exactly what each server in your VMware Infrastructure does. After looking through all of the servers you’ve built out over several months and realizing that you have one or two that you just don’t remember building, you can start to feel as if you’ve lost control of the environment.

To counteract this process I began adopting a few simple approaches in order to maintain control and reason within my virtual infrastructure. The first method of control is to force the “customer” to submit a written request either by e-mail or through a formal change request process to me detailing exactly what they need and when they need it by. While a request like this usually will generate a phone call by me to the “customer” asking for further clarification or to explain to them that no, indeed, they did not NEED 16 gigabytes of RAM for their IIS server. Basically, the written request allows me to respond back to them with the pertinent information they will need to work with their server while at the same time giving me a way to track my servers should I need to.

The second method I use to contain Virtual Sprawl is to create an inventory database for virtual machines that tells me all relevant information I need for the systems. Yes, I know this can all be found in the Virtual Console. However, should something happen that I cannot access the virtual console I have a backup plan. Also, the virtual console doesn’t reveal to me the backup methodology employed, the persons responsible for the server, or what the scheduled reboot cycle of the server is. Trust me when I say this has saved my bacon on more than one occasion.

The final method I use is to simply put the server role in the Notes section of the Virtual Console. When I click on the server everything I need to know about it is displayed there. What its function is, who to call in the event of a problem, and any special things that my poor, tired brain may not remember in five months time all reside right there in the Notes.

All of three of these tips make my job a little easier whle at the same time keeping me from reaching for the tweezers and pulling what little bit of hair I have left out. Also, they allow for a sanity check once a quarter to ensure that all vm’s in the environment are being used and aren’t just dead weight. Most importantly they reduce “virtual sprawl” in the environment and keep everything nice and tidy.

3 Ways To Kill a Stuck Virtual Machine in ESX 3.0

Process 1

1. Login to the service console using Putty (found here http://www.putty.nl/latest/x86/putty.exe).
2. Check the Virtual Machine's state by typing “vmware-cmd /path to the VM/servername.vmx getstate”.
3. Type “ps -ef grep ”.
4. The second column is the PID of the vmkload_app of the Virtual Machine (To see all running processes type “ps –eaf” ).
5. Type “kill -9 ”.
6. Check the Virtual Machine's state again by typing “vmware-cmd /path to the VM/servername.vmx getstate”.
7. The Virtual Machine's state should now be off.
8. Type “vmware-cmd /path to the VM/servername.vmx start” to power on Virtual Machine from the console.

Process 2

1. Login to the service console.
2. Type “vmware-cmd –l” to retrieve a list of all Virtual Machine's and there paths.
3. Check the Virtual Machine's state by typing “vmware-cmd /path to the VM/servername.vmx getstate”.
4. Forcibly stop the virtual machine by typing vmware-cmd /path to the VM/servername.vmx stop hard”.
5. Check the Virtual Machine's state again by typing “vmware-cmd /path to the VM/servername.vmx getstate”.
6. The Virtual Machine's state should now be off.
7. Type “vmware-cmd /path to the VM/servername.vmx start” to power on VM.
   

Proecess 3

1. Login to the service console using Putty.
2. Get the vmid of the Virtual Machine to be killed by typing “vm-support –x”
3. Kill the Virtual Machine and generate core dumps and logs by typing “vm-support –X ”
4. A prompt will appear asking if you want to include a screenshot of the Virtual Machine, send an NMI to the Virtual Machine, and send an ABORT to the Virtual Machine. Answer Yes to the ABORT question to kill the Virtual Machine.
5. The entire process will take about 5-10 minutes to run and will create a a tar archive in the directory it is run in.

How to Register an existing Virtual Machine in VirtualCenter

1. Select the ESX server in the VirtualCenter where the VM files are located.
2. At the summary tab right-click on the datastore where VM files are located and click “Browse Datastore”
3. Select the folder where VM files are located.
4. Right-click on the VMX file for the VM and select “Add to Inventory”.
5. Select a name and location for the VM and click Next.
6. Select the Host or Cluster for the VM and click Next.
7. Select a Resource Pool and click Next.
8. Click Finish. The VM will be back in VirtualCenter and available for management.
   

VMware and SAP Form Global Technology Partnership

"VMware, Inc., the virtualization software leader, today announced that SAP AG will provide immediate full support to its solutions in 64-bit Windows- and Linux-based production environments running on VMware ESX Server. Servers from Dell, Fujitsu-Siemens, HP, IBM and Sun have achieved hardware platform certification for SAP® solutions running on Windows and Linux with VMware ESX Server, a component of the VMware Infrastructure software suite. VMware Infrastructure supports SAP solutions with both Windows and Linux on industry-standard hardware. SAP and VMware will assist joint customers in cooperative support services and problem resolution, backed by a global technology partnership agreement and dedicated support staffing......."
read the rest of the article:
http://www.vmware.com/company/news/releases/sap_fullsupport.html

Upgrading from ESX Server 3.x Using esxupdate

You can upgrade from ESX Server 3.x to ESX Server 3.5 using the esxupdate utility provided with ESX Server 3.x. The esxupdate utility is for ESX Server 3. For ESX Server 3i, use the vihostupdate utility instead.

Useful esxupdate options
If you need to upgrade multiple ESX Server 3.x hosts in-place, you can extract the zip archive onto an NFS share or an HTTP server. Then point the esxupdate command to the shared location. This saves you from having to download and extract the archive onto each ESX Server host. For example:

esxupdate -r http://bitserver.mycompany.com/pub/ESX3.5/upgrade update

Other useful options:

• -n/--noreboot Disable the automatic reboot after update. If you include this option, you must reboot the host manually for the changes to take effect.
• -x Exclude a package.

To upgrade using esxupdate

1. Download the esxupdate zip file. This zip file is not the same as the tarball used to upgrade from ESX Server 2.x to ESX Server 3.5. Ensure that you have downloaded the correct file before proceeding.
2. Extract the compressed zip file and change to the newly created directory.The easiest way to do this is to run the unzip command: unzip VMware-esx-upgrade-from-esx3-<build>.zip
3. Type esxupdate update. When the update command completes, the software reboots the ESX Server host.

VI 3.5 Install, Upgrade, Administration, and Patch Guides for ESX Server 3.5 and VirtualCenter 2.5

Release Notes - http://www.vmware.com/support/vi3/doc/vi3_esx35_vc25_rel_notes.html

ESX Server 3 Installation Guide - http://vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_installation_guide.pdf

Upgrade Guide - http://vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_upgrade_guide.pdf

ESX Server 3 Patch Management Guide - http://vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_esxupdate.pdf

Configuration Maximums for VMware Infrastructure 3 - http://vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_config_max.pdf

Basic System Administration - http://vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_admin_guide.pdf

Virtual Infrastructure Web Access Administrator's Guide - http://vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_web_access.pdf

ESX Server 3 Configuration Guide - http://vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_3_server_config.pdf

Virtual Machine Backup Guide - http://vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_vm_backup.pdf

VMware Infrastructure 3 Primer - http://vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_prim.pdf

VMware Converter Enterprise for VirtualCenter 2.5 Admin Guide - http://vmware.com/pdf/vi3_vec_10_admin_guide.pdf

VMware Update Manager Admin Guide - http://www.vmware.com/pdf/vi3_vum_10_admin_guide.pdf