Archive for the ‘System Administration’ Category

My aching butt? No more!

Knoll Generation

Knoll Generation

I have long held the opinion that you should not skimp on the things that you use professionally and frequently. For instance I often tell my peers to make sure they buy decent computers and not just bargain basement models. For some reason it never occurred to me to apply this reasoning to my posterior. That is, until recently.

I have been using a most uncomfortable office chair for the longest time, in fact, for the last 5 years. It’s one of those “conference room” models, which, although built plenty rugged, are not necessarily built for comfort. The real problem with this is I sit in the thing almost 8 hours a day. I even had to buy a pillow for it because I have what I like to refer to as “sysadmin’s butt”, which is essentially the lack of a posterior portion of your anatomy, due mostly to parking it in a chair 8 hours a day for many years.

A few days ago, a friend of mine who just happens to work for a great company called Knoll, got me hooked up with a review unit of the Knoll Generation chair. Let me just say I don’t know why I waited so long for a good chair!

This chair is the epitome of office comfort and has so many options to aid you in that regard it’s difficult to mention them all. Of course it offers the standard amenities such as height adjustment and a reclining back, but wait, there is more! It has these cool adjustable arms that not only raise and lower but they also have arm pads that adjust in depth, width and pivot. The back of the seat has this comfort type netting which is flexible, cool and very comfortable. The top of the back flexes almost over backwards and it a great place to rest your arm while sitting sideways in high comfort. I think the feature that threw me the most is the flex seat. Unlike most chairs with a solid unyielding seat, this seat has some degree of movement or side to side pivot. I almost thought that my chair was broken until I realized that this was intentional! This pivot to the seat allows comfort and support for how people sit. What I mean by that is people sit different ways and on different angles with their feet up, legs crossed and what have you, and this seat allows some motion in that regard to keep even support and pressure where it should be, fully on your posterior instead of on your legs when you are not seated “flat”. In addition to that, the seat depth adjusts as well.

Although this chair may be on the expensive side, I believe it’s well worth it, especially for people who are confined to such devices for much of their day. I wish they had a travel version available, I would buy it in a minute 😉 I think the best testimony is that every person in my office has now sampled the chair and ALL of them want one! If you are in need, you can’t go wrong with one of these. It’d most likely be the last office chair you would ever need to buy.

My only hope and wish is that the Knoll folks see this review and decide to send me one of their other products to test as well! Other than that, I’ll be happy to hang out in my own Genertion!

Sunday, October 2nd, 2011

Lost your Mint password?

First time this happened! A coworker asked me today how to get into his Linux Mint box after he forgot his password. Of course I rattled off the old GRUB way to get things done, but, what?? This is GRUB 2! No so fast there! Turns out it’s quite different.

You hold down the shift key while booting to get to the grub menu.
You hit ‘e’ to edit your boot options.
You change the kernel line options on the very end of the kernel line to read “rw init=/bin/bash”.
You press F10 to boot.

Once booted you are dropped immediately into a shell prompt where you can change your password with the “passwd username” command. Reboot and you’re home free!

Monday, August 22nd, 2011

Rsync bug



Bitten by the rsync bug? I was. Apparently in the new RHEL 5.7, and I am sure the RH clones like CentOS, Scientific Linux and ClearOS(?) as well, there is a bug in rsync when you use it with ssh transport like so:

rsync -avz -e ssh remotehost:/data /data

The fix is to make sure to append a username to your host and then it magically starts working properly again.

rsync -avz -e ssh username@remotehost:/data /data


Wednesday, July 27th, 2011

Mint 11 Boot Splash

I have seen a lot of commenting on the decision to use a black screen as the boot splash on Linux Mint 11. One person summed it up well when they said that that black screen is indicative in many other operating systems as something going wrong and it tends to scare people. Well, that being the case, if you are scared or if you just like to see what’s going on behind the scenes like I do sometimes this is how you can fix that fairly easily.

Open up a terminal and do a:
vi /etc/defaults/grub

scroll down to the line that says:
and change it to:
and save the file.

After that do a:

When that is finished, reboot your machine and enjoy watching the text based boot process as it occurs. Pay close attention, though, ’cause it sure doesn’t last long! 😉

Friday, June 10th, 2011

Linux Shell Scripting Cookbook

Linux Shell Scripting Cookbook

Linux Shell Scripting Cookbook

   As a full time Senior Linux System Administrator in real life I was quite interested to get my fingers on this book for a review. After all, the job of a smart sysadmin pretty much dictates scripting away as much of your work as possible. We are a lazy bunch and we call that being efficient 🙂

   This is the first book I have reviewed by Packt Publishing or the author, Sarath Lackshman, I wasn’t really sure what I was in for. In fact I was slightly put off by the price, which I initially thought overly hefty at $45 US. For that kind of scratch I am used to seeing a much more substantial sized book from the sort of publishers I normally review for. I started making my way through the book anyway, and I am glad I did.

   What makes this book really cool is the premise behind it. Inside, as a “cookbook” should, you have these “recipes” for scripts. These are not what I have normally seen in many scripting books before, which are generally theoretical and sometimes lengthy examples, but these recipes are pretty straight forward, real world examples of things you might want to do, and how to handle those efficiently. The recipes are also small enough that you could easily piece meal things out to compose another script and I am certain that would be a great help to novice scripters.

   As nice as I think this book would be for novice scripters, there is a lot of smart stuff in there, stuff that had never occurred to me through my years of command line use. I actually got really excited to try some of the examples in there and to put them into practice. I particularly liked the little tricks here and there, like the “subshell trick” and I was absolutely thrilled that this book used modern syntax and variable manipulation, dropping the deprecated stuff like putting commands into back ticks. Good form!

   This book is certainly a keeper and I would recommend it highly to anyone who wants to become proficient on the command line. Some days you actually *do* get what you pay for, and I believe people will find this book to be a good example of that. This book was truly fun for me to work my way through and I sure hope they have more like it in store for the future. Go buy yourself a copy. I know I will be hanging on to this one for a while 🙂

Friday, March 25th, 2011

RHEL 5 quick and dirty samba primer



A friend asked me for a quick primer on how to set up a windows accessible share under RHEL 5, so I thought I would include it here for the benefit of anyone interested.

  • sudo yum -y install samba
  • sudo vim /etc/samba/smb.conf
  • replace the file with something like so:

server string = SERVERHOSTNAME Samba Server Version %v
security = user
netbios name = CALLMESOMETHING
comment = my data share
path = /data
read only = no
writable = yes
guest ok = no
available = yes
valid users = USERNAME

  • add a local user to the box: sudo useradd USERNAME
  • add the local user to samba and give password: sudo smbpaswd -a USERNAME
  • restart samba service: sudo service smb restart
  • make sure samba starts at boot: sudo chkconfig smb on
  • adjust your firewall settings if necessary

At this point you should be able to access the share at //servername/data.
Have fun!

Tuesday, March 22nd, 2011

System Administration: Information

Probably 50% of a SysAdmin’s job revolves around information. Knowing what is going on with your systems can make all the difference. Just don’t make the mistake of thinking that the more information the better. What you really need is the *correct* information at the appropriate time and it shouldn’t be obfuscated by extra information.

Good info sources:
Use OSSEC and Nagios. These products will notify you about security issues and outages.

There is a child’s fable about the boy who cried wolf. To make a long story short, the boy made false alarms several times drawing attention until when he really saw a wolf, nobody would come. There is an important lesson in there about information too. After a while, if you are flooded with info you don’t need, you tend to stop paying attention and may miss something important.

The right stuff:
Make sure that you set up your source filters or rules well. Use your mail filters wisely and set them up as you go along to remove non essential notifications. And most importantly, read and pay attention to those alerts and notifications you get!

Wednesday, March 9th, 2011

Server Build

Last night on the TechShow I was asked about providing some info on a decent default server build. Here are some quick notes to get people going. Adjust as necessary.

Just for ease, here, lets assume you are installing CentOS 5, a nice robust enterprise class Linux for your server needs.

CentOS 5 / RHEL 5 / Scientific Linux, etc., does a really great job picking the defaults, so sticking with those is just fine and has worked well for me on literally hundreds of servers.

  • I let the partitioner remove all existing partitions and chose the default layout without modification.
  • Configure your networking appropriately, make sure to set your system clock for the appropriate timezone (no I do not generally leave my hardware clock set to UTC).
  • When picking general server packages I go for web server and software devel. I do not, generally, pick virtualization unless there is a specific reason to. I find that the web and devel meta server choices provide a robust background with all the tools I need to set up almost any kind of server I want without having to dredge for hundreds of packages later on.
  • The install itself at this point should take you about 15 minutes depending on the speed of your hardware.
  • Once installed, reboot the server and you should come to a setup agent prompt. Select the firewall configuration. Disable the firewall and SELinux completely (trust me here). Once that is done, exit the setup agent (no need to change anything else here), login to the machine as root and reboot it. This is necessary to completely disable SELinux.

From this point on it’s all post install config…:

  • Add any software repositories you need to.
    I not only have my own repo for custom applications, but also have a local RedHat repo for faster updates and lower network strain/congestion.
  • Install your firewall.
    I use an ingress and egress firewall built on iptables. While mine is a custom written app, there are several iptables firewall generator apps out there you can try.
  • Install your backup software.
    Doesn’t matter if this is a big company backup software like TSM or CommVault, or you are just using tar in a script. Make sure your system is not only being backed up regularly, but that you can actually restore data from those backups if you need to.
  • Add your local admin account(s).
    Don’t be an idiot and log into your server all the time as root. Make a local account and give yourself sudo access (and use it).
  • Fix your mail forwarding.
    Create a .forward file in your root directory and put your email address in there. You will get your servers root emails delivered to you so you can watch the logwatch reports and any cron results and errors. This is important sysadmin stuff to look at when it hits your inbox.
  • Stop unnecessary services.
    Yes, if you are running a server you can probably safely stop the bluetooth and cups services. Check through what you are running with a “service –status-all” or a “chkconfig –list” (according to your runlevel) and turn off / stop those services you are not and will not be using. This will go a long way toward securing your server as well.
  • Install OSSEC and configure it to email you alerts.
  • No root ssh.
    Change your /etc/ssh/sshd_config and set “PermitRootLogin no”. Remember, you just added an admin account for yourself, you don’t need to ssh into this thing as root anymore. Restart your sshd service after making the change in order to apply it.
  • Set runlevel 3 as default.
    You do not need to have a GUI desktop running on your server. Run the gui on your workstation and save your server resources for serving stuff. Make the change in /etc/inittab “id:3:initdefault:”.
  • Fix your syslog.
    You really should consider having a separate syslog server. They are easy to set up (hey, Splunk is FREE up to so much usage) and it makes keeping track of whats happening on multiple servers much easier (try that Splunk stuff – you’ll like it).
  • Set up NTPD.
    Your server needs to know what time it is. ‘Nuff said.
  • Install ClamAV.
    Hey, it’s free and it works. If you do ANYTHING at all with handling emails or fileshares for windows folks on this machine, you owe it to yourself and your users to run Clam on there to help keep them safer.
  • Do all your updates now.
    Before you go letting the world in on your new server, make sure to run all the available updates. No sense starting a new server instance with out of date and potentially dangerous software.
  • Lastly, update your logbook.
    You should have SOME mechanism for keeping track of server changes, whether it be on paper or in a wiki or whathaveyou. Use it RELIGIOUSLY. You will be glad someday you did.

Thursday, February 24th, 2011

ESXi and Subsonic

In continuation, somewhat, of my last post and a brief review on the last TechShow, I wanted to jot down some notes about my newest encounter with ESXi and Subsonic.



I wanted to try out Subsonic, so I really needed to put together a new machine to play with it a bit. As a RL System administrator, some things carry over into my home computing environment, and paranoia is one of them. I just *have* to test things outside of my “production” servers at home too. Since I run my servers in a virtualized environment, this shouldn’t be too much of a problem.

I run ESXi at home for my virtualization platform, and the norm there is to use virtualcenter (or the vic) to create and manipulate VMs. The problem there is I am just not a Windows fan (no kidding). I had gotten around this problem initially by creating a VM on VMware Server (running on Linux) and then using VMware Converter to move that VM to my ESXi machine. This time, I did a little more digging on the subject of using the command line to create those VMs natively and I actually found some great information that let me do just that. What I found was these two links that contain all the information I needed:
ESXi – creating new virtual machines (servers) from the command line

Without rehashing a lot of the detail provided in those two sites, the basics are using vmkfstools to create a disk image for you to use and then building a small minimal vmx file with enough info in it to get things going. To do the install, make sure have your vmx start an iso image from the cdrom drive and turn on vnc for the box. From there it’s quite easy to get an install working.

The server I decided upon installing is CentOS 5.5. I chose the standard server install and the only things that were required to get Subsonic working on it were:
yum install java-1.6.0-openjdk
and then to download and install the rpm from Subsonic’s website. A little later on I found that Subsonic would not stream my ogg files and that was easily fixed by:
rpm –import
rpm -Uvh rpmforge-release-0.5.2-2.el5.rf.i386.rpm
yum install lame ffmpeg

After all that, pointing your web browser to http://:4040 and you are rocking and rolling with the big boys. The thing that really impressed me with the setup is when you tell Subsonic where your music is. On every other music server install this is the part where it takes a while to scan and index your music. With Subsonic this was surprisingly almost instantaneous! You tell it where the music is and *whamo* your music shows up, ready to be played. Fantastic! The other great piece is the ability to add album art. You can just tell subsonic to change your album art and it finds some suggestions on the web and will let you pick the correct one and save it to your collection. It’s very nice and a complete time grabber 🙂

Friday, February 11th, 2011

Diagnosis: Paranoia

You know, there are just some things you do not need first thing on a Monday morning. This was one of them…

I came and and started reviewing my reports and was looking at an access report, which is basically a “last | grep $TheDateIWant” from over the weekend. I keep a pretty tight ship and want to know who is accessing what servers and when (and sometimes why). What I saw was monstrously suspicious! I saw MYSELF logged in to 3 different servers 3 times each around 5am on Sunday morning – while I was sleeping.

This is the kind of thing to throw you into an immediate panic first thing on a Monday morning, but I decided to give myself 10 minutes to investigate before completely freaking out.

The first thing I noticed was that the access/login times looked suspiciously like the same times I ran my daily reports on the machines, however, the previous week I had changed the user that runs those reports and this was still saying it was me. I double, triple and quadruple checked and searched all the report programs to make absolutely sure there was no indication that they were still using my personal account (which was probably bad practice to begin with btw). Then I scoured all the cron logs to see what was actually running at those times, and oddly enough, it was just those reports.

I looked through the command line history on those machines and checked again the “last | head” to see who was logging on those machines. Nothing out of place BUT with the “last| head” I was NOT listed as being on the machine on that date! So I ran the entire report command again “last | grep $TheDateIWant” and there I was again, listed right under the logins of the report user.

Anyone catching this yet?

What I had stumbled upon were a few machines that are used so infrequently that the wtmp file, which is what the “last” command uses for data, had over 1 year of entries. My search of “last | grep ‘Oct 31′” was returning not only this year, but my own logins from last year as well.


Moral of the story? Mondays stink – Just stay home!

Monday, November 1st, 2010