LincolnBlogs
From the twisted mind of me!

A long long time ago, I virtualized all my home infrastructure onto an ESXi 4.0 server. It has run perfectly fine, minus one hard drive failure, for quite a few years. Lately, though, I had been wanting to upgrade it because it’s not terribly fast and I have run out of resources to be able to add new VMs. It was running on a dual cpu machine (single core) with 160Gb HDD and 4Gb of ram, and I was just using it all up. No more ram for new stuff.

I decided that I would upgrade the matching spare server I had and try out KVM because I had used it a bit for RedHat training and it worked so well. Of course, Fessenden’s law, as opposed to Murphy’s law, stated simply that “Something will go wrong.” And it did. Over and over again.

First off, let me say that on an enterprise class server system, if it says it needs registered ECC ram, it is NOT kidding. I must have swapped ram around in that server 50 times before I noticed 2 sticks of non-registered ram in there. Once I got over that, I had 8Gb of ram and a new 250Gb HDD and I was ready to rock! Or so I thought.

I decided to use CentOS 6 as my virtualization host OS and that went right on but I soon discovered that my CPU doesn’t support virtualization. Ugh. So I decided that I would switch gears and go with virtualbox instead so that I could continue using my current hardware. I have often used virtualbox on other machines and it is a fantastic platform. I set about getting things running.

When I installed the base OS, I did a minimal install. No GUI, etc.. There is no sense in putting stuff on there you don’t need on a server right? Well, the very first thing I found was that I could not use the virtualbox gui controls because I did not have any X installed. To rectify that:

yum -y install xorg-x11-xauth dejavu-lgc-sans-fonts

You need the auth to be able to forward your X session, and need the fonts to be able to actually see words on your app.

Next I copied all my vmdk files to the new server. This takes a LONG time for old servers to move around 100Gb. Once there, however, I discovered that virtualbox cannot read native vmdk files. Ugh again.

yum -y install qemu-kvm

And then I could convert the vmdks to raw images, and then again to native vdi files for virtualbox.

qemu-img convert machine-flat.vmdk machine.bin
vboxmanage convertfromraw --format VDI machine.bin machine.vdi

I put all my machines together and noticed that virtualbox was complaining about uuid on some of the disk images. To fix that:

vboxmanage internalcommands sethduuid machine.vdi

The first machine I started up was a CentOS 6 machine and that fired right up, however, udev immediately reassigned my ethernet device to eth1. In order to get thatr back where it was supposed to be I had to go into /etc/udev/rules.d/70-persistent-net.rules and delete the ethernet rules in there and reboot.

Along about this time my server powered off. No idea why. It powered itself back on again about 30 seconds later. I checked everything on the server and it looked fine. Curious, but I kept on going.

Next I tried to start up my remaining Centos 5 VMs. These were problematic. The very first thing I noticed here was that they were barking because I never uninstalled the vmware drivers. I fired them back up on the original server and ran the vmware-uninstall.pl program. I turned them back off and spent hours re copying the over, and then reformatting the vmdk files into vdi.

Starting them back up, I found that, again, they would not run. This time I received the error that it could not locate any LVM partitions. This, it turns out, is because the initrc files did not have the appropriate drivers in them. Fixing this was fun. First off, you need to add a cdrom drive to the vm and put a CentOS rescue cd/dvd there. Boot it up in rescue mode, chroot to the /mnt/sysimage and then fix the /etc/modprobe.conf file:

alias scsi_hostadapter mptbase
#alias scsi_hostadapter1 mptspi
#alias scsi_hostadapter2 ata_piix
alias scsi_hostadapter1 mptscsih
alias scsi_hostadapter2 mptscsih

The entries with the #s are the ones I had to change. Then I needed to rebuild all of the initrd images.

cd /boot
for file in $(ls init* | cut -d'-' -f2,3 | cut -d'.' -f1-6); do mkinitrd -v -f /boot/initrd-$file.img $file; done

After that, the machines came right up! Of course, the host powered right off. Several times over the next day. Grrr.

I figured that there was a hardware issue with the host somewhere and resolved to buy myself a new server. I picked an open box refurb from microcenter that had 8Gb ram, a 750Gb HDD and a nice quad core cpu that supported virtualization. Wohoo! I can now switch to KVM!

I set up the new machine and installed KVM and started copying vmdk files over again and, bingo, kernel panic. I rebooted and the machine would not even get past bios. This went on for a couple days until I took the machine back to microcenter. I picked up a different machine, better quad core with 12Gb of ram and 1Tb HDD and set about getting it running.

This time, success! I set up CentOS 6 and KVM, added the bridged networking and copied over the vmdk files. KVM will read vmdk files but I decided to convert to a more native format, qcow2, the preferred format for qemu, anyhow. that is fairly simple to do.

qemu-img convert -O qcow2 machine-flat.vmdk machine.qcow2


I put all the machines back together again and started them back up. I still had to do the initrd fixes on the CentOS 5 VMs to get them going, but after that all has been running fantastically!

Somewhere along the line here I figured out that my issue with my secondary server powering off was a bad port on my UPS.

KVM is really easy to run and manage for a Linux geek as opposed to VMware 4. The native gui tools do the job just fine, although they are not quite as intuitive to me as VMWare’s VIC. I am quite happy, though, with the switch. I now have more than twice the resources of my initial virtualization environment. Now I am good to go for several more test VMs and the new machine is nice and quiet and doesn’t have to hide under my couch

Lately I have been working so hard that I haven’t even had any desire to do any fun computering at home. Today that changed a bit.

I decided this morning that it was high time I upgraded my all time favorite rss feed reader, tiny tiny rss. Well, wouldn’t you know it, after I did the install I found it required a version of php higher than I had available on my server. Time to upgrade.

I run Centos 5 on my main server and, by default, that carries a php 5.1.x. I needed 5.2 or greater. As it happens, php 5.3 is available in the repos, so I did the upgrade. For the uninitiated, that entails doing a “yum list installed | grep php”, which gives you a list of what you *have* installed. Next you remove php by doing “yum remove <and name all the packages in the prior list here>”. This, followed by “yum install <list of files for php 5.3>”. For example, I had php-common.i386 and php.i386 installed, so I did a “yum remove php-common php” and then “yum install php53-common php53″ to get all my php 5.3 packages on there. This was followed by a quick “service httpd restart” to make sure my webserver was using the new version.

Murphy’s law states that “something will go wrong if it can”. Well, *MY* law states that “something will go wrong”, and it did. As it turns out, I had built a whole bunch of php applications maybe 7 years ago that my wife uses almost daily. In the olden days of php, you could declare a php script at the top by doing a “<?”. NOW, you need to declare it by doing “<?php”. Consequently, nothing I had written worked. It only took me a minute or two to identify why the problem was occurring, but fixing it was another story.

So, how do you find all the files you have to fix? Well, I used the “grep” command. More specifically, egrep. I went to my html root directory and searched by doing “egrep -r “<\?” * | egrep -vi “<\?php” | egrep -vi “<\?xml” | grep -v inary”. What does all that do? The first stanza looks recursively through the directory structure at every file and outputs the ones that have any “<?”‘s in them. The second takes that output but does NOT pass through any that are “<?php”. Why, because they would already be ok! The third takes the results and doesn’t pass through any that contain “<?xml”. The last one doesn’t pass through results from binary files. The end result is I had a list of directory / file / line information of all the files I had to change / update. A few minutes later, after using vim, the best text editor around, I was back up and running!

This week I am helping out my Mother, who has just had knee replacement surgery, so, oddly enough, I actually have a small bit of time to actually post something here again!

Right before I left, I decided I wanted to press an old Dell laptop into service for giggles. It is a Dell Latitude D810, which has a 2.mumble Ghz processor, 60 or 80Gb hdd and 1.5 Gb of ram (I confess I did upgrade that a bit from 1Gb). The really great thing about this particular laptop is it has a 1900×1200 resolution on it’s 15.4 inch screen. This makes for a great work laptop because it has so much screen realestate.

Well, what to run on this machine? It originally came with Windows XP, but, as you can imagine, I just cannot see fit to run Windows on my laptops, so which Linux? I decided on trying CentOS. Specifically CentOS 6.2 i386, which is current as of this writing. As it turns out, that was a really great choice!

The CentOS install was exactly what I had come to expect from it, short and sweet. The only issue of note is an error you get about half way through that says “insufficient memory to start Kdump” or some such thing. This is really nothing to worry about and can be safely ignored. Other than that, after a reboot from the installer, the system was ready to rock-n-roll. As in completely! Everything was working. I had sound, wireless, X, you name it! No muss, no fuss. To me this was wonderful as the last time I put CentOS on a laptop, I had to mess around a little bit to get wireless going. This time I ended up in luck because apparently my Intel Corporation PRO/Wireless 2915ABG is completely supported out of the box. Woot!

Anyhow, I ended up following my own recommendations I blogged about earlier here to get all my required softwares going like I prefer. The only noteable there is that, apparently, the EPEL repo (that’s Extra Packages for Enterprise Linux for those in the know) has changed the URL to:

http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-6.noarch.rpm

or

http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-6.noarch.rpm

respectively.

Also of note, I seemed to have a bit of trouble getting dropbox on here. The solution was to NOT use the dropbox package available in the repos, but to go directly to dropbox and download the package available for Fedora, which worked just fine.

Once all that was done, this old laptop worked WONDERFULLY, and, in fact, I am typing and working on it now.

Often I hear people say things like “get more use out of your old computer with Linux”, and, while this is certainly true, I tend to discourage that because running a modern operating system like Linux on older hardware can many times give the wrong impression. Things run slow and laggy and often that is interpreted as being Linux’s fault, when, in fact, you are just trying to shoehorn it on to your old 486. You REALLY get a superior Linux experience when running it on modern hardware and it becomes VERY easy to contrast it then to other modern operating systems. At any rate, I wanted to say that this is NOT the case here. Linux on this older hardware runs so phenomenally well that I am seriously inclined to recommend purchasing this particular laptop used for persons looking for an inexpensive but capable mobile computing environment. I see them listed on ebay and amazon for very close to $200, which is, in my opinion, quite a bargain compared to the performance I experience from my much newer and more expensive netbook.

In conclusion, I would have to say, don’t discount older hardware, and certainly don’t discount CentOS as a viable, capable and speedy desktop (yes I did say desktop) Linux distribution!

Probably you have either listened to me or read my thoughts or both for several years now, but it occurred to me today that someone out there might be interested in seeing what actually drives the LincGeek.

I currently live in Pennsylvania, but I was born and raised in Upstate NY, with a brief stint in Washington state. New Yorkers and hillbillies are my people and I understand them. Washington is some of the most beautiful country I ever spent time in and I hope to at least visit out there again someday.

Well, first and foremost, computers and Linux are my personal crack. I started on a life long obsession with computers back in 1983 with my first Vic=20 (Thank you William Shatner). I learned to program in BASIC and from there it was all over until I met Linux in the 90s, then that added into the mix.

I like the fastest computers I can get my hands on. I like Apple computers (more for their quality and aesthetics than OS – they do tend to run Linux very well). I love my Kindle, my Android phone and my Asus TF300T Linux Mint is probably the nicest version of Linux I have ever run and I use that almost exclusively as my desktop OS of choice. I am RedHat certified and use RHEL and CentOS for the vast majority of my enterprise and personal server needs, because, IMHO, it’s better than the rest.

I am a music lover. I dig 50s, 60s, 70s, 80s, Big Band, Jazz, Funk, Disco, Bluegrass and Classical music. I was a hardcore low brass musician and vocalist in my school years, even making it into “Who’s Who In Music” in my senior year in high school, and those are some of my most cherished and fondest memories. Rap is *NOT* music, by the way.

I have been married once, to my college sweetheart, for almost 20 years now and have an adult (she thinks so at least) daughter, currently in college. I am a Conservative Libertarian, politically, and a proud Christian.

Although I am now diabetic and stick mostly to various forms of Chicken and veggies, I LOVE good food. my favorites are good Irish cooking like my Grandma used to make. Corned Beef and Cabbage. And she made a monster macaroni and cheese too. I would literally hurt someone for some of that again. I strongly believe that vegetables are what food eats.

I like my coffee with (nonfat) milk and sweet-n-low. Buy it from Wawa because Starbucks coffee is overpriced and bitter yuppie coffie IMHO. I like an occasional good cigar (Acid Blondie) and enjoy them most when I can smoke them and hang out with my friends. (Edit, I am a confirmed Vaper now – RY4 absolutely ROCKS!)

I am not a drinker. If and when I do imbibe, I do so with Scotch or Whiskey as I believe beer must be what urine tastes like.

As you can probably surmise, I am highly opinionated, and as I have a monster sized guilty conscience and I am not at all politically correct, so if you ask my opinion, you are liable to actually get it.

I still think the occasional fart joke is funny. I hate unproductive meetings and long phone conversations. I try very hard to be honest, forthright, fair and maintain integrity.

I am a pet guy and love small furry mammals of all kinds. I have and have had cats, dogs, rabbits, mice, rats, ferrets and even a smattering of budgies and small lizards.

And now you know all about me!

I had a friend a few weeks ago who asked me for help getting the Transmission bittorrent client working on CentOS 6. I took these notes then and am sharing them now.

cd /etc/yum.repos.d/
wget http://geekery.altervista.org/geekery-el6.repo
NOTE: RHEL/CentOS 6 x86_64 users have to replace $arch with $basearch in the repo file
yum install transmission transmission-gtk

I love love RHEL and CentOS on servers but surprisingly, CentOS 6 makes a nice desktop as well! Here are some notes I took getting things going the way I like on my CentOS laptop.

Where I work, if you push your machine name to DHCP, it will register with DNS as well. This is a nice feature if you are in the habbit of ssh-ing to your boxes like I am. In order to accomplish this task, you change a setting in the dhclient.conf file, which for a long time has been found in /etc/dhcp or /etc/dhcp3. Well, things have been moved around a bit and it’s now located in /etc and the filename is dhclient-eth0.conf where eth0 is the interface you are using. My suspicion is that this was done to frustrate the 5 people in the world like myself who actually use this

I really have no need for SELinux on this machine and therefor turn it off to keep it out of my way. This is easily accomplished by editing the /etc/selinux/config file and setting SELINUX=disabled. (then performing a reboot)

I also have no use for the default firewall ruleset. I normally do a chkconfig iptables off && service iptables stop and just address my firewall concerns later on.

What I *DO* need, often, if not always, are the development tools. Things like compilers and make, etc.. Get them by doing yum -y groupinstall “Development tools”.

Install the EPEL repo:
rpm -Uvh http://download.fedora.redhat.com/pub/epel/6/i386/epel-release-6-5.noarch.rpm
or
rpm -Uvh http://download.fedora.redhat.com/pub/epel/6/x86_64/epel-release-6-5.noarch.rpm
depending on your archetecture.

Gotta have clusterssh. If you don’t use it you should!
yum -y install clusterssh

I also must have my chrome web browser:
Install chrome from website https://www.google.com/chrome?&brand=CHMB&utm_campaign=en&utm_source=en-ha-na-us-sk&utm_medium=ha

And Thunderbird too!
install thunderbird from website http://www.mozilla.org/en-US/thunderbird/

Add the RPMForge repository:
rpm –import http://apt.sw.be/RPM-GPG-KEY.dag.txt
rpm -Uvh http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.i686.rpm
or
rpm -Uvh http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
depending on your archectecture.

And, lastly, if you are somewhere where this is legal, you can install all the codecs that make using your computer nice!
yum -y install compat-libstdc++-33 libdvdcss libdvdread libdvdplay libdvdnav lsdvd libquicktime flash-plugin mplayerplug-in mplayer mplayer-gui gstreamer-ffmpeg gstreamer-plugins-bad gstreamer-plugins-ugly
wget www1.mplayerhq.hu/MPlayer/releases/codecs/mplayer-codecs-20061022-1.i386.rpm (or x86_64)
rpm -ivh mplayer-codecs-20061022-1.i386.rpm (or x86_64)
wget www1.mplayerhq.hu/MPlayer/releases/codecs/mplayer-codecs-extra-20061022-1.i386.rpm (or x86_64)
rpm -ivh mplayer-codecs-extra-20061022-1.i386.rpm (or x86_64)

A couple days ago I was reading a post from Knightwise and he mentioned using twidge on his server to do some fun stuff with his twitter account. Well! That sounded to me like just the thing for me to get some use from my neglected twitter account. Unfortunately, twidge is really best used on a debian type system and *my* server runs CentOS 5. This is a simple recipe to shoehorn twidge onto a CentOS 5 server.

I downloaded the twidge binary from https://github.com/downloads/jgoerzen/twidge/twidge-1.0.6-linux-i386-bin

The binary requires libcurl-gnutls which CentOS just doesn’t have. I snuck around that by doing

ln -s /usr/lib/libcurl.so.3.0.0 /usr/lib/libcurl-gnutls.so.4

Then the binary told me it needed libffi. This I could get from the epel repository. Do that by doing

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm

and then

yum install libffi

That gets twidge working ….. mostly. Because of the sneaky I pulled with that libcurl-gnutls thing, twidge generates an error message on each run. It still works fine, but gives me a message on each run:

twidge: /usr/lib/libcurl-gnutls.so.4: no version information available (required by bin/twidge)

Undaunted, the easy fix for that is to dump the unneeded error to /dev/null like so

twidge lsrecent 2> /dev/null

And there you have it! For those of you looking to employ twidge on CentOS or similar Linux, this will get you going pretty quickly. Enjoy and I’ll tweet ya later!

Bitten by the rsync bug? I was. Apparently in the new RHEL 5.7, and I am sure the RH clones like CentOS, Scientific Linux and ClearOS(?) as well, there is a bug in rsync when you use it with ssh transport like so:

rsync -avz -e ssh remotehost:/data /data

The fix is to make sure to append a username to your host and then it magically starts working properly again.

rsync -avz -e ssh username@remotehost:/data /data

Enjoy!

A friend asked me for a quick primer on how to set up a windows accessible share under RHEL 5, so I thought I would include it here for the benefit of anyone interested.

• sudo yum -y install samba
• sudo vim /etc/samba/smb.conf
• replace the file with something like so:

[global]
workgroup = SOMEWORKGROUPNAME
server string = SERVERHOSTNAME Samba Server Version %v
security = user
netbios name = CALLMESOMETHING
[data]
comment = my data share
path = /data
read only = no
writable = yes
guest ok = no
available = yes
valid users = USERNAME

• add a local user to the box: sudo useradd USERNAME
• add the local user to samba and give password: sudo smbpaswd -a USERNAME
• restart samba service: sudo service smb restart
• make sure samba starts at boot: sudo chkconfig smb on
• adjust your firewall settings if necessary

At this point you should be able to access the share at //servername/data.
Have fun!

Last night on the TechShow I was asked about providing some info on a decent default server build. Here are some quick notes to get people going. Adjust as necessary.

Just for ease, here, lets assume you are installing CentOS 5, a nice robust enterprise class Linux for your server needs.

CentOS 5 / RHEL 5 / Scientific Linux, etc., does a really great job picking the defaults, so sticking with those is just fine and has worked well for me on literally hundreds of servers.

• I let the partitioner remove all existing partitions and chose the default layout without modification.
• Configure your networking appropriately, make sure to set your system clock for the appropriate timezone (no I do not generally leave my hardware clock set to UTC).
• When picking general server packages I go for web server and software devel. I do not, generally, pick virtualization unless there is a specific reason to. I find that the web and devel meta server choices provide a robust background with all the tools I need to set up almost any kind of server I want without having to dredge for hundreds of packages later on.
• The install itself at this point should take you about 15 minutes depending on the speed of your hardware.
• Once installed, reboot the server and you should come to a setup agent prompt. Select the firewall configuration. Disable the firewall and SELinux completely (trust me here). Once that is done, exit the setup agent (no need to change anything else here), login to the machine as root and reboot it. This is necessary to completely disable SELinux.

From this point on it’s all post install config…:

• Add any software repositories you need to.
  I not only have my own repo for custom applications, but also have a local RedHat repo for faster updates and lower network strain/congestion.
• Install your firewall.
  I use an ingress and egress firewall built on iptables. While mine is a custom written app, there are several iptables firewall generator apps out there you can try.
• Install your backup software.
  Doesn’t matter if this is a big company backup software like TSM or CommVault, or you are just using tar in a script. Make sure your system is not only being backed up regularly, but that you can actually restore data from those backups if you need to.
• Add your local admin account(s).
  Don’t be an idiot and log into your server all the time as root. Make a local account and give yourself sudo access (and use it).
• Fix your mail forwarding.
  Create a .forward file in your root directory and put your email address in there. You will get your servers root emails delivered to you so you can watch the logwatch reports and any cron results and errors. This is important sysadmin stuff to look at when it hits your inbox.
• Stop unnecessary services.
  Yes, if you are running a server you can probably safely stop the bluetooth and cups services. Check through what you are running with a “service –status-all” or a “chkconfig –list” (according to your runlevel) and turn off / stop those services you are not and will not be using. This will go a long way toward securing your server as well.
• Install OSSEC and configure it to email you alerts.
• No root ssh.
  Change your /etc/ssh/sshd_config and set “PermitRootLogin no”. Remember, you just added an admin account for yourself, you don’t need to ssh into this thing as root anymore. Restart your sshd service after making the change in order to apply it.
• Set runlevel 3 as default.
  You do not need to have a GUI desktop running on your server. Run the gui on your workstation and save your server resources for serving stuff. Make the change in /etc/inittab “id:3:initdefault:”.
• Fix your syslog.
  You really should consider having a separate syslog server. They are easy to set up (hey, Splunk is FREE up to so much usage) and it makes keeping track of whats happening on multiple servers much easier (try that Splunk stuff – you’ll like it).
• Set up NTPD.
  Your server needs to know what time it is. ‘Nuff said.
• Install ClamAV.
  Hey, it’s free and it works. If you do ANYTHING at all with handling emails or fileshares for windows folks on this machine, you owe it to yourself and your users to run Clam on there to help keep them safer.
• Do all your updates now.
  Before you go letting the world in on your new server, make sure to run all the available updates. No sense starting a new server instance with out of date and potentially dangerous software.
• Lastly, update your logbook.
  You should have SOME mechanism for keeping track of server changes, whether it be on paper or in a wiki or whathaveyou. Use it RELIGIOUSLY. You will be glad someday you did.