Archive for the ‘VMware’ Category

Adventures in Virtualization

A long long time ago, I virtualized all my home infrastructure onto an ESXi 4.0 server. It has run perfectly fine, minus one hard drive failure, for quite a few years. Lately, though, I had been wanting to upgrade it because it’s not terribly fast and I have run out of resources to be able to add new VMs. It was running on a dual cpu machine (single core) with 160Gb HDD and 4Gb of ram, and I was just using it all up. No more ram for new stuff.

I decided that I would upgrade the matching spare server I had and try out KVM because I had used it a bit for RedHat training and it worked so well. Of course, Fessenden’s law, as opposed to Murphy’s law, stated simply that “Something will go wrong.” And it did. Over and over again.

First off, let me say that on an enterprise class server system, if it says it needs registered ECC ram, it is NOT kidding. I must have swapped ram around in that server 50 times before I noticed 2 sticks of non-registered ram in there. Once I got over that, I had 8Gb of ram and a new 250Gb HDD and I was ready to rock! Or so I thought.

I decided to use CentOS 6 as my virtualization host OS and that went right on but I soon discovered that my CPU doesn’t support virtualization. Ugh. So I decided that I would switch gears and go with virtualbox instead so that I could continue using my current hardware. I have often used virtualbox on other machines and it is a fantastic platform. I set about getting things running.

When I installed the base OS, I did a minimal install. No GUI, etc.. There is no sense in putting stuff on there you don’t need on a server right? Well, the very first thing I found was that I could not use the virtualbox gui controls because I did not have any X installed. To rectify that:

yum -y install xorg-x11-xauth dejavu-lgc-sans-fonts

You need the auth to be able to forward your X session, and need the fonts to be able to actually see words on your app.

Next I copied all my vmdk files to the new server. This takes a LONG time for old servers to move around 100Gb. Once there, however, I discovered that virtualbox cannot read native vmdk files. Ugh again.

yum -y install qemu-kvm

And then I could convert the vmdks to raw images, and then again to native vdi files for virtualbox.

qemu-img convert machine-flat.vmdk machine.bin
vboxmanage convertfromraw --format VDI machine.bin machine.vdi

I put all my machines together and noticed that virtualbox was complaining about uuid on some of the disk images. To fix that:

vboxmanage internalcommands sethduuid machine.vdi

The first machine I started up was a CentOS 6 machine and that fired right up, however, udev immediately reassigned my ethernet device to eth1. In order to get thatr back where it was supposed to be I had to go into /etc/udev/rules.d/70-persistent-net.rules and delete the ethernet rules in there and reboot.

Along about this time my server powered off. No idea why. It powered itself back on again about 30 seconds later. I checked everything on the server and it looked fine. Curious, but I kept on going.

Next I tried to start up my remaining Centos 5 VMs. These were problematic. The very first thing I noticed here was that they were barking because I never uninstalled the vmware drivers. I fired them back up on the original server and ran the program. I turned them back off and spent hours re copying the over, and then reformatting the vmdk files into vdi.

Starting them back up, I found that, again, they would not run. This time I received the error that it could not locate any LVM partitions. This, it turns out, is because the initrc files did not have the appropriate drivers in them. Fixing this was fun. First off, you need to add a cdrom drive to the vm and put a CentOS rescue cd/dvd there. Boot it up in rescue mode, chroot to the /mnt/sysimage and then fix the /etc/modprobe.conf file:

alias scsi_hostadapter mptbase
#alias scsi_hostadapter1 mptspi
#alias scsi_hostadapter2 ata_piix
alias scsi_hostadapter1 mptscsih
alias scsi_hostadapter2 mptscsih

The entries with the #s are the ones I had to change. Then I needed to rebuild all of the initrd images.

cd /boot
for file in $(ls init* | cut -d'-' -f2,3 | cut -d'.' -f1-6); do mkinitrd -v -f /boot/initrd-$file.img $file; done

After that, the machines came right up! Of course, the host powered right off. Several times over the next day. Grrr.

I figured that there was a hardware issue with the host somewhere and resolved to buy myself a new server. I picked an open box refurb from microcenter that had 8Gb ram, a 750Gb HDD and a nice quad core cpu that supported virtualization. Wohoo! I can now switch to KVM!

I set up the new machine and installed KVM and started copying vmdk files over again and, bingo, kernel panic. I rebooted and the machine would not even get past bios. This went on for a couple days until I took the machine back to microcenter. I picked up a different machine, better quad core with 12Gb of ram and 1Tb HDD and set about getting it running.

This time, success! I set up CentOS 6 and KVM, added the bridged networking and copied over the vmdk files. KVM will read vmdk files but I decided to convert to a more native format, qcow2, the preferred format for qemu, anyhow. that is fairly simple to do.

qemu-img convert -O qcow2 machine-flat.vmdk machine.qcow2

I put all the machines back together again and started them back up. I still had to do the initrd fixes on the CentOS 5 VMs to get them going, but after that all has been running fantastically!

Somewhere along the line here I figured out that my issue with my secondary server powering off was a bad port on my UPS.

KVM is really easy to run and manage for a Linux geek as opposed to VMware 4. The native gui tools do the job just fine, although they are not quite as intuitive to me as VMWare’s VIC. I am quite happy, though, with the switch. I now have more than twice the resources of my initial virtualization environment. Now I am good to go for several more test VMs and the new machine is nice and quiet and doesn’t have to hide under my couch 🙂

Sunday, August 5th, 2012

Playing catch-up

I decided that on my vacation I would do some catch-up work. I have many times mentioned that I am a consummate procrastinator, and if you combine that with me being just generally whooped tired after 12 hours away from home on any average day, you understand why my computers seem to go uncared for. I think it’s the same as the whole “the mechanics car is never fixed” thing.

I mentioned a couple days ago that I installed ESXi on one of my home servers (redundant servers) to fix a strange problem I had been having with VMware Server 2.x. That was the first job I needed to so, or at least the most important, and so far it has been doing beautifully.

Next on the list was Mint 8 on the old laptop. It has been running Mint 7 since the distro was released and it was time for an upgrade. Everything was working just fine on 7, I just wanted to catch up the latest/greatest. As expected, the upgrade was a no-brainer and it’s running gorgeously, as Mint does.

Today, so far, I decided to upgrade my desktop machine to Mint 8. This machine, a P4 3Ghz with 3Gb of ram runs like absolute crap. I don’t exactly know why, but it always has. Now I have replaced the cpu fan a couple times and also the power supply at least twice. The computer is noisy, whiny, but not physically broken that I can tell. It just seems to run slower than hell and always has. The installation of Mint 8 on it did make it prettier, but sure didn’t make it seem to run any faster. I think it just dogs over the dual display and craptasticly old Nvidia card. Perhaps if I bought it a new quiet power supply, a better working and quieter cpu fan, a new better video card and a new dvdrom drive (yeah that’s pretty broken too), I could resuscitate this thing so that I could stand using it again. But then again, I could probably buy a whole new desktop computer for what I would spend on repairs to this one. Dang.

So, what’s next? Well, I should install ESXi on my redundant server now that I am satisfied with how the other one is running. I should also upgrade to Mint 8 on my Acer Aspire All In One netbook (notice a pattern here). Other than that, I am not sure.. Maybe work on some code projects I have been stringing along for months and months.

So what kinds of great computery projects are you all up to? Or what SHOULD you be up to 🙂

Monday, February 8th, 2010

VMware ESXi – a sigh of relief!


A couple days ago I relayed the story about how my VMware Server 2 infrastructure was suffering some issues. Basically it would randomly just shut down my VMs. I don’t know why. I absolutely poured over the logs for days on end while simultaneously searching google for *any* inkling or hint of an idea on how to remedy the situation or even why it was happening. Nothing….

Frustrated, I was searching around for a different solution and after passing on Virtualbox, Parallels, KVM and others for various reasons, not the least of which was the learning curve on some, I settled on ESXi. I run a lot of ESX and some ESXi at work, so the familiarity is there and it’s been my experience that it’s a rock solid and stable platform, not to mention that it’s bare metal and wickedly fast.

There were some drawbacks. ESX(i) requires a Windows management interface (or Virtual Infrastructure Controller – VIC) and I wasn’t even sure my hardware would accomidate. You see, ESXi has only a certain set of hardware that it will work with.

Well, after a bit of research, I was mostly convinced that my hardware would work, albeit with a little tweak to get the IDE drive recognised. I registered for, and downloaded the free ESXi 4.x release from, burned it to a cd and I was off to the races.

The installation was completely a no-brainer. Just put the cd in, boot it up and go. It really is an almost no-touch install. I was also pleasantly surprised that it recognised my IDE drive automatically with no tweaking whatsoever. When the install was done, there were only a couple settings to adjust like configuring the IP address and root password, and they are all accessed and changed in a very plain and simple text interface. All in all, in less than a half an hour and with 1 reboot I had an ESXi server just begging me for some VMs.

Once it was up and running I decided I would try everything possible NOT to have to resort to running Windows at home for a management interface. Luckily, other people have decided the same and there is good information available on the web on using the built in command line tools to do what you need to. And they aren’t difficult at all.

First, I needed to be able to access the command line tools on ESXi, and that required turning on SSH access. I followed the instructions here:

After that, I needed to get my VMware Server 2.x VMs on the ESXi box. I turned to VMware Converter for that. Downloaded it (again free) from VMware and installed it on my VMware Server 2.x host machine so that the converter would have access to the local VM files.

I shut down the VMs and used vmware converter to convert them to the ESXi box. Each conversion of a 12GB VM took approximately 40 minutes (give or take). Since the converter is a GUI app, I did a “ssh -Y vmwareserver2host” to run the converter console on my local machine because my vmwareserver2 machine is a headless server.

When the VMs were converted to the ESXi box, I took a cue from this page:
to add vncserver to each VM, which allowed me to connect to the VMs and make 1 integral change to each virtual machine when they were running.

To get the machines running I used ESXi’s “vim-cmd vmsvc/getallvms” command on the ESXi box, which listed all the VMs I copied there with their assigned vm number. “Then, I ran vim-cmd vmsvc/power.on #” where number is the vm number listed from the getallvms command.
Once they were started, I used vncviewer to connect to the VMs, log in and fix their networking. You see when you move a vm to a different host machine, the mac address gets reassigned and hoses up your VMs network config. Once that was quickly fixed, I rebooted the VMs and they were good to go!

There are a couple other things that I need to get tweaked, like adding my registration number to ESXi, which I found directions for at I also noticed that vmware adds some filesystem into the VMs /etc/hosts file which errors out on boot. Just comment that out and it’s fine. Lastly, since I migrated the VMs from Server 2.x, they already had the vmware tools from that loaded in and I noticed a little barking about those tools while the VMs were booting, so I disabled them by doing a “service vmware-tools stop ; chkconfig vmware-tools off” on my VMs which are CentOS, so your method of disabling those tools may vary.

My impressions so far: Although this all sounded hard, long and technical, nothing could be farther from the truth. It was extremely easy – much more than I had initially hoped. And, if my VMs *stay running* now, it will be well worth it. I also believe that these VMs ABSOLUTELY SCREAM compared to how they ran before. They are much more responsive now in every way. The change was well worth it!

Friday, February 5th, 2010

Arrgh VMware!



Some times you just can’t win.

A while back I got some decent server hardware and decided I was going to migrate my main server to a VM, so I could make it more portable and more reliable, etc.. Well that worked out really well, for the most part, until lately.

I set up the 64 bit version of VMware Server 2.X, created a brand new VM server, got all my services running on it and all was well initially. For some reason, though, recently my VMs have just been randomly powered off. I cannot find anything in the logs, and the host machine is still running. This has happened a couple times to some other people as well. It’s very puzzling and quite frustrating to me. The only thing I have noticed about the problem is that, on my server, it seems that the VMs stay powered on longer if the management console is not running. When it *is* running, the VMs get shut down in a few hours (always when I am not around to watch them or restart them it seems) and the management interface gets stopped at the same time.

Searching the internet has been absolutely no help on this one, so if anyone has any hints, tips or information, please shoot me an email before I rip out what is left of my hair.

Friday, January 29th, 2010

Throw some Rocks at it!

One of the parts of my day job is dealing with and managing our HPC cluster. This is an 8 node Rocks cluster that was installed maybe a week after I started. Now I was a bit green still at that point and failed to get a better grasp on some things at the time, like how to maintain and upgrade the thing, and I have recently been paying for that 🙂

Apparently, the install we have doesn’t have a clear-cut way to do errata and bug fixes. It was an early version of the cluster software. Well, after some heated discussions with our Dell rep about this, I decided what I really needed to do was a bit of research to see what the deal really was and if I could get us upgraded to something a bit better and more current.

Along came my June 2009 issue of The Linux Journal which just happened to have a GREAT article in it about installing your very own Rocks Cluster (YAY!). Well, I hung on to that issue with the full intention of setting up a development/testing cluster when I had the chance. And that chance came just the other day.

Some of you probably don’t have a copy of the article, and I needed to do some things a bit different anyhow, so I am going to try and summarize here what I did to get my new dev cluster going.

Now what I needed is probably a little different that what most people will, so you will have to adjust things accordingly and I’ll try and mention the differences as I go along where I can. First off, I needed to run the cluster on RedHat proper and not CentOS, which is much easier to get going. I also am running my entire dev cluster virtually on an ESX box and most of you would be doing this with physical hardware.

To start things off I headed over to The Rocks CLuster website where I went to the download section and then to the page for Rocks 5.2 (Chimichanga) for Linux. At this point, those of you who do not need specifically RedHat should pick the appropriate version of the Jumbo DVD (either 32 or 64 bit). What I did was to grab the iso’s for the Kernel and Core Rolls. Those 2 cd images plus my dvd image for RHEL 5.4 are the equivalent to your one Jumbo DVD iso on the website that uses CentOS as the default Linux install.

Now at this point, you can follow the installation docs there (which are maybe *slightly* outdated(?), or just follow here as the install is pretty simple really. You will need a head node and one or more cluster nodes for your cluster. Your head node should have 2 interfaces and each cluster node 1 network interface. The idea here is that your head node will be the only node of your cluster that is directly accessible on your local area network and that head node will communicate on a separate private network with the cluster nodes. With 2 interfaces, plug your eth0 interface on all nodes, head and cluster into a separate switch and plug eth1 of your head node into your LAN. Turn on your head node and boot it up from the Jumbo DVD, or in the case of the RHEL people, from the Kernel cd.

The Rocks installer is really quite simple. Enter “build” at the welcome screen. Soon you will be at the configuration screen. There you will choose the “CD/DVD Based Rolls” selection where you can pick from your rolls and such. I chose everything except the Sun specific stuff (descriptions on which Rolls do what are in the download section). Since I was using RHEL instead of CentOS on the jumbo dvd, I had to push that “CD/DVD” button once per cd/dvd and select what I needed from each one.

Once the selections were made it asks you for information about the cluster. Only the FQDN and Cluster name are really necessary. After that you are given the chance to configure your public (lan) and private network settings, your root password, time zone and disk partitioning. My best advice here would be to go with default where possible although I did change my private network address settings and they worked perfectly. Letting the partitioner handle your disk partitioning is probably best too.

A quick note about disk space: If you are going to have a lot of disk space anywhere, it’s best on the head node as that space will be put in a partition that will be shared between compute nodes. Also, each node should have at least 30gb of hdd space to get the install done correctly. I tried with 16gb on one compute node and the install failed!

After all that (which really is not much at all), you just sit back and wait for your install to complete. After completion the install docs tell you to wait a few minutes for all the post install configs (behind the scenes I guess) to finish up before logging in.

Once you are at that point and logged into your head node, it is absolutely trivial to get a compute node running. First, from the command line on your head node, run “insert-ethers” and select “Compute”. Then, power on your compute node (do one at a time) and make sure it’s set to network boot (PXE). You will see the mac address and compute node name pop up on your insert-ethers screen and shortly thereafter your node will install itself from the head node, reboot and you’ll be rockin’ and rollin’!

Once your nodes are going, you can get to that shared drive space on /state/partition1. You can run commands on the hosts by doing “rocks run host uptime”, which would give you an uptime on all the hosts in the cluster. “rocks help” will help you out with more commands. You can ssh into any one of the nodes by simply doing “ssh compute-0-1” or whichever node you want.

Now the only problem I have encountered so far is I had an issue with a compute node that didn’t want to install correctly (probably because I was impatient). I tried reinstalling it and it and somehow got a new nodename from insert-ethers. In order to delete my bad info in the node database that insert-ethers maintains I needed to do a “rocks remove host compute-0-1” and then “rocks sync config” before I was able to make a new compute-0-1 node.

So now you and I have a functional cluster. What do you do with it? Well, you can do anything on there that requires the horsepower of multiple computers. Some things come to mind like graphics rendering and there are programs and instructions on the web on how to do those. I ran folding at home on mine. With a simple shell script I was able to setup and start folding at home on all my nodes. You could probably do most anything the same way. If any of you find something fantastic you like to run on your cluster, be sure to pass it along and let us know!

Friday, November 13th, 2009

Pukwudgie roars into life

Last night I finally finished cutting all my server services to their new residence on Pukwudgie (my spectacular CentOS 5.3 based server VM). I turned off my old Thinkpad server, which has been doing the job reliably for over 2 years, rebooted pukwudgie just to make sure everything starts up correctly unattended and that was that. My first impressions are that everything seems to run faster. I really expected that, though, because there are a lot more resources available to Pukwudgie than there were to the old server. I am loving it so far and it sure is nice to have an up-to-date server. The old server was running Ubuntu 6.10, which was so old I couldn’t even get security patches for it anymore and this new CentOS server is completely current.

Hopefully this is a move for the better, and I can probably offer the old lappy/server on too!

Friday, August 28th, 2009


Late last ‘week I noticed that my new nagios server was not responding anymore. Well, I checked it and it was down. Not only that, it was a vm on my test server and the entire server was down as well. Arrrgh.

Usually I use this as a foray to tell you all to remember to do your backups. Well, in this case I didn’t do them either. Hey, it’s a test vm server right? Yeah, well I am kicking myself about that anyhow. I just got nagios working really well the way I wanted. Oh well, I guess I get to practice some more right 🙂

Well, as it turns out, my server had a catastrophic drive failure. I did EVERYTHING to try and resuscitate this thing. To start with, it had no partition table at all. Luckily I bought 2 of these servers and they were identically configured, so I checked out the partition table of the one and used fdisk to apply it to the broken one. After that I was able to fsck one partition, but as it would happen, that partition was only boot. Feh. The other partition had lost all it’s superblock info. I couldn’t even use a backup superblock. Nada. I noticed that mkfs had a command line switch of -S, which writes the superblock info on a artition without formatting or touching the inodes. I tried that and it appeared to be successful. At leat I could run fsck on the partition now and it was fixing the inodes. YAY! except that after a few hours of fixing, I still got nothing but a few system files in a pile under the lost-n-found directory. Shortly thereafter the drive lost it’s partition info again anyway. That’s life I guess.

So, it was off to Microcenter to get a new hdd. I brought that home and did a fresh CentOS 5.3 32 bit install and played with it a bit and thought to myself, hey, maybe I should run some kind of burn-in test on this server before I go investing a lot of time into it again.

That is where Sys_Basher comes in. Sys_Basher is a multithreaded memory and disk exerciser. That’s what the website says. It makes a pretty good burn in program by continually testing your memory and disk (which pushes on your cpu as well) for any length of time you specify. I kinda like it actually, and that is a good thing because there are woefully few burn-in or stress test type programs available to the Linux community. In fact, if you are a programmer and looking for a great project, you could generate a lot of traffic and interest by making one. Not that I don’t like Sys_Basher, mind you, but variety is the spice of life and certainly the way of open source!

Anyway, I ran Sys_Basher overnight on my new machine which passed with flying colors. Then, this morning, I decided that maybe I should run 64bit Linux on this box. Some days I am so fickle, but I decided it would be in my best interest to change up the OS before building a bunch of new test vms on there 🙂

Maybe this time I’ll even back the darn thing up too! Wish me luck and, btw, do your backups!

Sunday, July 12th, 2009

More servers available

A little while back I picked up a couple Appro servers from for an outstanding price and also managed to convince Dann to do the same (after making a public plea on TLLTS to raise the moolah). Shortly thereafter, the deal was a bust and there were no more of those servers available. Well, today, Tyler, a listener of TLLTS, sent me an email letting me know that there were more of them available at and they are only $109 a piece! Well, what better blog post to break the no-blogging ice with than a great deal like that! Let me say that these servers, are referbished and work GREAT. I have been using mine as a VM server for months now and it works flawlessly. Thanks again for the heads up Tyler!

Need some great and inexpensive servers? You can pick yours up if you hurry at:

Sunday, July 5th, 2009

VMWare for the win



I always push VMWare because I use it a *LOT* and believe it to be a superior product. My friend Dann was no exception. When it came time for him to get a new server I suggested he could do several virtual machines on one good physical box and save his resources.

He did the same type of install on his server that I did, and that was to install CentOS 5.3 as a base and put VMWare Server 2 on it to handle his virtual infrastructure. Now for me, this has been bulletproof. Dann has had a couple minor instances where his VMs lost their registration after a power failure and he had some issue connecting to the web interface to fix it.

Anyway, the reason for this post is that there have been a couple instances where managing VMWare Server 2 from the command line have been a big help. It’s not readily apparent that you *can* manage it via the command line and the information was hard to come by, so I thought putting it here for my future reference was probably a good idea.

The command you mostly need to be concerned with is “vmrun”. The tricky part is the switches afterward. Some examples:

List Running VMs:
vmrun -T server -h https://localhost:8333/sdk -u root -p rootpassword list
List Registered VMs:
vmrun -T server -h https://localhost:8333/sdk -u root -p rootpassword listRegisteredVM
Start a VM:
vmrun -T server -h https://localhost:8333/sdk -u root -p rootpassword start "[standard] ogopogo/ogopogo.vmx"
Stop a VM:
vmrun -T server -h https://localhost:8333/sdk -u root -p rootpassword stop "[standard] ogopogo/ogopogo.vmx"

The biggest issue is just getting the syntax correct in as far as the name and location of the VM. My suggestion would be to list them, and then copy that output and enclose in quotes for your start and stop commands. In particular, vmware seems to use [standard] as it’s default volume name reference, so, as is listed above, my machine Ogopogo is listed and referenced at “[standard] ogopogo/ogopogo.vmx” or vmrun will just error out.

Hope this helps!

Friday, June 5th, 2009

CentOS 5.x + VMware Server



CentOS 5.x makes a great VM server. Here’s a quick recipe to make it happen:

First, install CentOS, which is an enterprise class Linux distribution that is basically a rebadged/rebranded RedHat, compiled from sources with a few tweaks. I did the generic server only install. After the install, make sure to do all your updates! (yum update)

Once that is done, head on over to VMware’s site and pick up the VMware Server 2. As far as I am concerned, VMware stil has the corner on the market for machine virtualization. There are plenty of other products out there, but I have done a *lot* of virtualization with VMware and it’s rock solid, and it’s free version has a really great price point for what it provides too! Sign yourself up and pick up the free server rpm and make sure to get your registration code as well.

Once that is done, the rest is easy:
Log in as root.
yum groupinstall “Development Libraries”
yum groupinstall “Development Tools”
yum install kernel-devel xinetd
rpm -Uvh VMware-server-2.xxxxxxxxxxxxxxxxxx

Run the perl config script that the RPM install will tell you about and pick the default settings (worked for me anyway). That’s it!

Point your firefox browser to https://yourCentOSserver:8333 and login using your root account and password.

One thing to note as you start creating VM’s is you may need to have access to iso images to do installs from. Place them into your /var/lib/vmware/Virtual\ Machines/ directory and you’ll be able to find them in your vmware datastore.

Wednesday, April 1st, 2009