Archive for the ‘work’ Category

So you want to be a Linux admin…

This is somewhat of a reprisal of some thoughts I shared on a recent episode of the LinuxLink TechShow.

I have been asked many times about being a Linux admin. After a few years of walking the walk and being in on a lot of interviews, I have compiled a few mental notes and thought I would share…

  • Get a cert.
    I have been doing Linux for a LONG time but I never had so many job offers until I got my RedHat cert and put it on LinkedIn. Once that happened I get, some weeks, upwards of 5 job offers per week. Seriously. These offers are also local – not like people are calling me to move out of state or even out of the area. The jobs are out there folks. Linux people are currently on the hot list. Just do it right and you should be a shoe in.
  • Know your stuff.
    Here’s the deal. You MIGHT run into a company where you can snow them into thinking you are a serious Linux guy even though you don’t know how to tell what directory you are in on the command line, but it sure won’t be MY company. I ask potential candidates lots of questions – ones that I am convinced that anyone that *actually uses Linux* should know. Make sure that you do. You should know all kinds of common Linux things down absolutely cold and this includes things like common userland commands, problem diagnosis and resolutions. If you don’t know the fix to a problem, you should be able and prepared to demonstrate that you can quickly find the correct answer / resolution.
  • Don’t rely on the gui.
    I used to think this was a given but after a dozen interviews it bears mentioning. You *cannot* correctly administer a hundred servers if you need to rely on gui tools. They may be handy in a pinch, but they are wildly inefficient. On the same tolkin, you should be familiar and comfortable with at least basic scripting. One of the questions I generally ask is if another administrator left your company, how would you change the root password on 100 servers in a hurry?
  • Do be familiar with Desktop Linux.
    Although I think it’s extremely important to be command line savvy on the server end of things, I am also convinced that a Linux guy should be comfortable with using it on the desktop as well. It always strikes me as strange when I ask a Linux guy what kinds of computers he has at home and what he uses them for and he (or she – it’s just a figure of speech) says they have a windows laptop that they only use for browsing the web and email.
  • At least feign interest.
    In my opinion, a Linux system administrator should be interested in Linux and system administration. Things like playing at home with different linux distributions, running your own home server, setting up , learning about and trying different Linux services are all big plusses.
  • Don’t BS on your resume (or resume inflation).
    A friend of mine I work with and I have this theory that a person’s actual skill level with Linux is conversely proportionate to the size of their resume. Actually, this goes back to that “Know your stuff” rule as well. Put the relevant things you know on your resume and *actually know them*. Trust me, I will ask you technical questions about the things you list on your resume and I *will* find out if you are lying. Inflated resumes may impress H.R. people but not the people who actually have to weed through them.
  • Shake hands like you mean it.
    When you come for an interview, if you offer to shake hands or take an offer to shake a hand, actually do it. Nothing weirds me out more than someone giving me one of those limp wristed, pantywaist, palm tickle handshakes. Grip my hand like you mean it and give it a good shake like you are happy to be there.
  • Be genuine.
    Don’t try and be someone you are not during the interview. Be yourself, relax a little, be honest. Don’t be overly cocky, snarky, apologetic or overtly eager. Also, try and dress decently and speak well. ๐Ÿ™‚

That’s all I can think of right now – Knock ’em dead!

Thursday, February 24th, 2011

What’s with the Lemur?

System 76 Lemur

System 76 Lemur

Nope, I am not talking about the curious little Madagascan primate, I am talking about the one from System 76!

It has been a while since I have done a review, but that doesn’t mean I haven’t been working one up ๐Ÿ™‚ At this past years Ohio Linux Fest I got to rub elbows with Carl Richell from System 76 who promised me the opportunity to review one of their masterpieces. After some killer anticipation, the unit arrived on my doorstep and it turned out to be their Lemur Ultra Thin laptop.

This lappy has a GORGEOUS 13″ display, a core-i3 proc, and the all the assorted (and working) ports, wifi, ethernet, sound, SUSPEND, etc., that you would come to expect. I did say working didn’t I? That’s important because, as retailer of Linux computers, it’s important to make the distinction that there is NO guesswork as to whether or not Linux will run on perfectly on it. It does ๐Ÿ™‚

System 76 was nice enough to let me demo this thing for a long time, so it’s safe to say that I tested this thing out really well. I used it extensively at home to do my normal web surfing, video watching and music playing. I also used it for work where it was my portal for a bunch of system administration work, rdesktop and ssh sessions galore, plenty of terminals open with configuration scripts and php programming, connected through every kind of free and paid wifi you can think of, not to mention my work vpn. And a lot of that was all at the same time! This system performed more than admirably. I even used it at a work conference where I did splunk installs and testing without issue.

I am not sure what kinds of proprietary things that System 76 provides in its own packages, however everything in Ubuntu, the Linux distribution that System 76 ships with by default, runs perfectly. Then again, so did Mint 10, the other Linux distribution I installed and tested with. This left me with, literally, nothing whatsoever that didn’t meet my personal satisfaction ๐Ÿ™‚

This laptop is very light, perfectly functional, very good looking and stunningly well designed and put together. It feels to me like a MacBook Air with a warp drive, and at literally half the price. In short, this is the laptop that I want to carry around (did I mention it’s light too)? I am hoping that Mrs. LincolnClaus is reading this. It would look great under the tree this year! I would gladly get rid of most of the rest of my laptop entourage to be able to carry one of these.

Hey, anyone want to buy a pristine condition used netbook? Or two? ๐Ÿ™‚

Monday, December 20th, 2010

Resize iscsi volume on RHEL 5

I have this ISCSI volume mounted on a RHEL 5 system that is running out of space. How do you grow your mounted iscsi volume? Good question!

* Unmount the volume. In this case it was /dev/sdb1 for me.
umount /dev/sdb1

* Grow the volume size on your san/nas (however your san/nas does this).
In my case - "Hey SanAdmin, can you add another 100gb of space to $volume?"

* In order to resize, your server needs to see that there is more volume space available, so you need to “service iscsi restart”.
[root@nile ~]# service iscsi restart
Logging out of session [sid: 1, target:, portal: nnn.nnn.nnn.nnn,3260]
Logout of [sid: 1, target:, portal: nnn.nnn.nnn.nnn,3260]: successful
Stopping iSCSI daemon:
iscsid dead but pid file exists [ OK ]
Starting iSCSI daemon: [ OK ]
[ OK ]
Setting up iSCSI targets: Logging in to [iface: default, target:, portal: nnn.nnn.nnn.nnn,3260]
Login to [iface: default, target:, portal: nnn.nnn.nnn.nnn,3260]: successful
[ OK ]

* fdisk /dev/sdb and delete the old partition (yes, delete it).
fdisk /dev/sdb
Command (m for help): d
Selected partition 1

* Create a new bigger partition over top / in place of the original.
Command (m for help): n
Command action
e extended
p primary partition (1-4)
Partition number (1-4): 1
First cylinder (1-26109, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-26109, default 26109):
Using default value 26109

* Run e2fsck on the partition.
e2fsck -f /dev/sdb1

* Resize it.
resize2fs /dev/sdb1

* Finally, mount it back up!
mount -a (yes mine was listed in fstab)

Thursday, December 9th, 2010

RHEL 6 is here!

As many of you know, RedHat released RHEL 6 recently. I just finally got a chance to install the production version and thought I would share some of my immediate notes:

RHEL 6 Installation Notes: (text/net install)

No boot.iso available. Must use the ENTIRE installation DVD to boot, even for a network install.

Press tab at the boot splash to enter extra parameters – โ€œlinux text askmethodโ€ worked appropriately.

Askmethod prompts for URL rather than http or ftp and has you put the entire URL in one line instead of splitting into server / location like RHEL 5 did.

Installer does not ask for registration number – must be done through rhn_register *after* installation has completed.

Install does not ask you for โ€œtypesโ€ like RHEL 5 did (webserver, virtualization, development).

Post install does not have configuration menu where you can change authentication, firewall/selinux, system services, etc..

That’s about where I am with this right now. The install is reminiscent of RHEL 4 in a lot of ways. I am sure things will change and improve like they always do. The one clearly needed addition right now, though, as far as I am concerned is a boot/netinstall.iso image.

Tuesday, November 16th, 2010

Diagnosis: Paranoia

You know, there are just some things you do not need first thing on a Monday morning. This was one of them…

I came and and started reviewing my reports and was looking at an access report, which is basically a “last | grep $TheDateIWant” from over the weekend. I keep a pretty tight ship and want to know who is accessing what servers and when (and sometimes why). What I saw was monstrously suspicious! I saw MYSELF logged in to 3 different servers 3 times each around 5am on Sunday morning – while I was sleeping.

This is the kind of thing to throw you into an immediate panic first thing on a Monday morning, but I decided to give myself 10 minutes to investigate before completely freaking out.

The first thing I noticed was that the access/login times looked suspiciously like the same times I ran my daily reports on the machines, however, the previous week I had changed the user that runs those reports and this was still saying it was me. I double, triple and quadruple checked and searched all the report programs to make absolutely sure there was no indication that they were still using my personal account (which was probably bad practice to begin with btw). Then I scoured all the cron logs to see what was actually running at those times, and oddly enough, it was just those reports.

I looked through the command line history on those machines and checked again the “last | head” to see who was logging on those machines. Nothing out of place BUT with the “last| head” I was NOT listed as being on the machine on that date! So I ran the entire report command again “last | grep $TheDateIWant” and there I was again, listed right under the logins of the report user.

Anyone catching this yet?

What I had stumbled upon were a few machines that are used so infrequently that the wtmp file, which is what the “last” command uses for data, had over 1 year of entries. My search of “last | grep ‘Oct 31′” was returning not only this year, but my own logins from last year as well.


Moral of the story? Mondays stink – Just stay home!

Monday, November 1st, 2010




I had the opportunity to check out some encrypted filesystem stuffs recently. The one that really stood out as easy to install. manage and use, for me, was EncFs. Now this post is mostly for posterity, but I wanted to share that, unless you are trying to get it running on RHEL, it’s pretty easy to get set up. I mostly referred to this site and had it up and going lickety-split.. I really am thrilled with how easy this actually was…


I tried getting it running on RHEL 5. I will spare you all the gory details about how it took hours of peeling through the dependency issues with nonstandard RHEL packages, but you get the idea. What I will leave you with here is what actually made it work:

yum -y install fuse fuse-devel fuse-libs
rpm -Uvh rlog-1.3.7-1.el5.rf.i386.rpm
rpm -Uvh fuse-encfs-1.4.1-1.el5.rf.i386.rpm
modprobe fuse
useradd -G fuse your_user_name

And that was it! Bask in the glory!!!

Tuesday, October 5th, 2010

Server Names

It has certainly been a while since I last posted, so I thought I would find something either interesting or funny to get things going here again. This happens to be funny (I think). I came across this tidbit of information I wrote at work some time ago and thought I would share.

Retired Server Names
Much like sports jersey numbers, some server names are never to be used again… Mostly because they appear to be secretly cursed.

I am not sure if there is some sort of bad mojo associated with naming a server with some kind of Godly connotation, but this VMWare GSX server would crash violently almost once an hour at its peak. As far as I know, nobody ever found out what the problem was, and after the name was changed it started working admirably.

Kashmir (pronounced “Cash-mere”)
This was an old RHEL 3.9 or AS 2.1 server that would crash almost as fast as you could start it back up again. We called this server “Crashmere”. The reason I am not sure of the OS level is it was hard to keep it running long enough to check. In it’s defense, I believe it had bad HDD’s before it was finally decommissioned, however, we were too paranoid to try reusing the hardware whether or not we attempted to fix it first.

Odessa was, for the most part, our entire early implementation of an Identity Management System. It was based on an out of date and buggy Opensource LDAP and some poorly written custom code from some interns. Consequently it quickly became widely used and relied upon, and never updated as a result. Literally, this was the *beast* the infrastructure team worked to keep fed and happy. Eventually, we moved to a different IDM environment and Odessa was retired, the name never to be used again because we never want to see another single machine gain that much power over anyone again. Odessa is surely the precursor to Skynet.

This is obvious – Just try and say the name. This is the reason one member of our team is never EVER allowed to pick server names again. We spent over a month trying to pronounce this in conversation until we finally gave up and changed the name outright.

Those are all the good ones I have right now but I am eager to hear any that any of you may have to contribute. Perhaps we could start some master list somewhere and save ourselves and others the tragedy of stumbling upon the reuse of one of these cursed names. ๐Ÿ™‚

Tuesday, May 11th, 2010




i just love cssh. This is the way updates should be run.

Wednesday, March 17th, 2010


Todays desktop

Todays desktop

I don’t know why, but I have always been fascinated by what people have on their desktops and how they have them configured. Well, today I ran into a link to a site I hadn’t visited in a long time, and that is Unixporn. No, it’s not porn, it’s a bunch of people sharing how cool their desktop looks. Eye candy. Anyhow I thought I would post my desktop so far today. I’d put it on the Unixporn site, but hey, then what would I post here? ๐Ÿ™‚ Show us all what your desktop looks like!

Addendum: I probably should mention what is actually running there. Gnome, Nautilus, xterms using vim for programming, transparent gnome-term running mocp for music, thunderbird email, pidgin instant messaging, and google chrome for browsing.

Thursday, February 4th, 2010

Ugly Sweater Day

For quite sometime, as a little morale booster, we have been having Hawaiian Shirt Fridays at work. It’s fun and I, personally like Hawaiian shirts. The gaudier the better.

Well, recently, it has been decided that in lieu of the warmer weather, perhaps we should temporarily switch to Ugly Sweater Fridays instead. What you see included in this post is my first attempt. I asked my wife the day before to run to the thrift store and get me the ugliest sweater she could find.

She really deserves the credit here, although I did wear it proudly. I was *easily* given the title for the most hideous sweater. We’ll have to see what happens next week, but I believe this will be hard to beat unless someone finds one of those light-up Rudolph holiday sweaters.

Let’s see the pics of YOUR ugly sweaters!

Tuesday, November 24th, 2009