Archive for the ‘Programming’ Category

BashPodder: A new update

BashPodder

BashPodder


Yes it certainly has been a while since I have done *anything* with BashPodder. I have, however, received a few requests for things and some for access to the code on GitHub so it could be worked into some actual distribution packages. Queue this post. BashPodder – the original, now on GitHub. https://github.com/lincgeek/bashpodder

Have at it folks!

Saturday, June 29th, 2013

Multi-threaded scripts using screen


Recently I had to do some more email migrations, and I wanted to script them. But how do you take hundreds of migrations and run, say, 15 at a time, scripted, so you don’t beat up your mail server? Good question!

What I settled on was writing a bash script to call a certain number of background screen processes. I figured that way I could be sure that disconnecting from my session didn’t kill my processes, AND I could always jump into any screened process easily to see what was going on. It wasn’t as hard to get going as you might think and it worked brilliantly!

#!/bin/bash

while read command
do
if [ $(cat totalthreads.txt) -gt $(screen -ls | grep Socket | tail -n1 | cut -d’ ‘ -f1 | sed -e ‘s/No/0/g’) ]
then
echo $command
screen -d -m $command &
fi
while [ $(cat totalthreads.txt) -eq $(screen -ls | grep Socket | tail -n1 | cut -d’ ‘ -f1 | sed -e ‘s/No/0/g’) ]
do
sleep 60
done
while [ $(cat totalthreads.txt) -lt $(screen -ls | grep Socket | tail -n1 | cut -d’ ‘ -f1 | sed -e ‘s/No/0/g’) ]
do
sleep 60
done
done < commandlist.txt

So, what you see here is the entire shebang. What’s happening is all the commands I want to run, which in my case are imap copies but you could do anything really, are listed in a file called “commandlist.txt” individually, each on their own line. It looks like this:

copy this file to that file
copy file 2 to anotherfile 2
copy file 3 to anotherfile 3
copy file 4 to anotherfile 4

The script grabs the commands you want to run and checks a file called “totalthreads.txt”, which simply contains a number. This is the number of threads you want running at any given time. So, say I have 100 commands (copys in the example) I want to run, but only 20 at a time so I don’t eat up all my I.O. I would put all the commands in the commandlist.txt file and a “20” in the totalthreads.txt file, then execute the script itself, which I called “threads.sh” (get it HERE). This script starts up 20 of the commands in the commandlist.txt file, each in their own screen session. When one of your processes ends, another will be started, maintaining that limit listed in the totalthreads.txt file, as long as their are enough commands in the commandlist.txt file to do so.

The nice part about this is it is the thread count is somewhat interactive. This means that if you notice you want more threads running, simply increase the number in the totalthreads.txt file and after a minute or two, you will see them increase. Conversely, if you want less, decrease the number and as processes finish, new threads will not be started until the number running is lower than the number in the totalthreads.txt file.

Wednesday, November 7th, 2012

Screenshots

I have long been fascinated by different peoples computing environments. Somehow I believe it shows a little glimpse into someone’s mind. With that in mind, I thought it might be interesting to other people as well so I polled a group of my friends who are some of the most influential computing buddies I have. Here is what they sent:

Name: A.W.
What do you do?:
I’m a NetApp Wrangler and Windows Sysadmin by trade. Looking to add storage admin as well (EMC/Cisco).
Tell me about your DE?:
My main workstation is my MacBook. I identify with this machine the most and my desktop environments tend to show my personality and style choices. I like IBM style green on black terminals which I have been addicted to ever since I installed my first AIX machine (a POWERStation 320 that I got for free from my ex-girlfriend’s office). The desktop is a stylized Sylvanas the Banshee Queen of the Undead from World of Warcraft. I don’t currently play the game but I’m into zombies and undead stuff as art and game play (and hot pale powerful gothy women). My Windows 7 machine is a gaming machine and also used to do my work as it’s the best machine to log into our VPN with. It’s an Alienware with the Phobos Red theme and the LEDs are currently all set to red with a pulsating skull on the front. It’s kind of Darth Vader. Alienware does nice themes and some of the nicest pre-installs I’ve ever seen (yes, the first time I didn’t wipe the OS that came with the system)… It has no shovelware. I’ve owned the Powermac G5 Quad for years and bought it to be the last and best PowerPC machine. Eventually I was no longer using it as I supplanted its use with my MBP which I can carry all over the house and use wireless N with. Wanting to breathe new life into it, it became a PPC Linux test box and I’ve found the best environment with Fedora Core 17 Beefy Miracle. I’ve replaced the desktop graphic with something nicer than the default fireworks that is still Fedora themed. The Firefox window is a shot of my home file server control panel. It’s a red aluminum cased custom AMD A4 build with 8 GB of RAM, 6 x 2TB Seagates (SATA3)ZFS RAID6 and a memory stick to hold FREENAS 8.0.4 x64 MULTIMEDIA. Since it’s red I named it after my favorite Motts discontinued beverage: Beefamato.
aw2
aw3
aw1

Name: D.C.
What do you do?:
Programmer and professional Bearded Curmudgeon.
Tell me about your DE?:
vim is my IDE, and I have a window open full screen, split into up to eight or so buffers on my main screen. On a second screen I have terminals for running my code’s tests, viewing logs, and for talking to colleagues who work all over the world – my team is split between Utah, the UK, Moscow, and anywhere else that we can find good people. My windows are all slightly transparent when inactive, as it makes it easier to find stuff if I can see it when it’s behind something else. I do, of course, use focus-follows-pointer and click to bring to front, but almost all my navigation is via the keyboard. When I do need to move the pointer,I use a trackball. Desktop? Yeah, there’s one under there somewhere, but I hardly ever see it. It’s a plain neutral colour with no icons on it so it doesn’t interfere with window transparency.
dc

Name: J.B.
What do you do?:
Senior Software Engineer working on cloud managed digital media systems for the retail environment.
Tell me about your DE?:
Windows 7. I run Linux on my desktop, but I never felt like having the distribution to work to change what’s on my laptop, and I use the laptop the vast majority of the time.
jb

Name: J.F.
What do you do?:
Solutions Architect, Enterprise Services, HP.
Tell me about your DE?:
I alternate between a black desktop and this photo of my favorite car. A friend collects vintage gas station equipment and provided the setting when I took this picture. I try to keep my desktop clean and maintain a folder called “desktop-stuff” for all the junk that would normally accumulate.
jf

Name: J.S.
What do you do?:
Retired network engineer now part time Asterisk/VOIP and wireless consultant.
Tell me about your DE?:
Windows 7 for the most part, but I have a Ubuntu 12 VM running X11RDP so I use Remote Desktop rather than VNC. That’s where I do the majority of my compiling & code editing in Xemacs.
js

Name: K.H.
What do you do?:
I’m a senior engineer on the Enterprise Infrastructure Team for a state government. I wrangle Tivoli Storage Manager, VMWARE, DNS, Linux/Apache/MySQL/PHP, legacy and modern UNIX/Linux, SANs, some LAN/WAN, provide support to the CISO in all areas of infosec as needed, and function as troubleshooter of last resort for any given problem.
Tell me about your DE?:
Windows 7 would not be my first choice, but since I have to use Windows-only apps in the execution of my duties, it is the best for the job. I run two monitors, which have different resolutions, but this is the best that can be managed on a restricted budget. Ideally there would be two 23″ monitors, but if we’re dealing in ideals, I would have an Alienware laptop instead of a Dell. The theme is a transparent space-based theme courtesy of NASA, but the background is an image from Stickman featuring some of my favorite tools. Rather than hide the start bar, I leave it up all the time for quick access.
kh

Name: L.F.
What do you do?:
I.T.Manager and Senior Linux Admin, LAMP developer, scripter and all miscellaneous duties as assigned.
Tell me about your DE?:
Mint #newest_version running my usual slew of apps and xterms on 2 dual monitor machines. Dark wallpaper is currently a “black leather”. I like dark unobtrusive wallpapers best to avoid distraction. Windows running in a vm, where it belongs. Just can’t have enough desktop real estate you know! And, yes, that’s mutt for email – best client out there.
lf

Name: M.H.
What do you do?:
I’m an I/T support specialist and dispatcher.
Tell me about your DE?:
I have quite a number of different desktops really. In fact I always have had. When they get cluttered I throw things into folders and eventually archive them if I don’t want to delete them. (My folder structures in my home directories is horrible.) Each system I use has a different purpose. The desktop here is my home daily driver. Multiple screens often dictate what wallpaper I use though frustratingly it’s hard to span wallpaper across multiple monitors. At home I usually use single displays but at the office I use four screens total. Working on adding another one. 😉 As for colors I prefer a darker theme with light lettering. For terminals I prefer a black background with amber text or as close as I can get using a color picker. Green if I don’t have amber as a choice. Translucent terminals look nice initially but are a pain for me to focus on.
mh

Monday, September 24th, 2012

Rsync bug

rsync

rsync


Bitten by the rsync bug? I was. Apparently in the new RHEL 5.7, and I am sure the RH clones like CentOS, Scientific Linux and ClearOS(?) as well, there is a bug in rsync when you use it with ssh transport like so:

rsync -avz -e ssh remotehost:/data /data

The fix is to make sure to append a username to your host and then it magically starts working properly again.

rsync -avz -e ssh username@remotehost:/data /data

Enjoy!

Wednesday, July 27th, 2011

Linux Shell Scripting Cookbook

Linux Shell Scripting Cookbook

Linux Shell Scripting Cookbook


   As a full time Senior Linux System Administrator in real life I was quite interested to get my fingers on this book for a review. After all, the job of a smart sysadmin pretty much dictates scripting away as much of your work as possible. We are a lazy bunch and we call that being efficient 🙂

   This is the first book I have reviewed by Packt Publishing or the author, Sarath Lackshman, I wasn’t really sure what I was in for. In fact I was slightly put off by the price, which I initially thought overly hefty at $45 US. For that kind of scratch I am used to seeing a much more substantial sized book from the sort of publishers I normally review for. I started making my way through the book anyway, and I am glad I did.

   What makes this book really cool is the premise behind it. Inside, as a “cookbook” should, you have these “recipes” for scripts. These are not what I have normally seen in many scripting books before, which are generally theoretical and sometimes lengthy examples, but these recipes are pretty straight forward, real world examples of things you might want to do, and how to handle those efficiently. The recipes are also small enough that you could easily piece meal things out to compose another script and I am certain that would be a great help to novice scripters.

   As nice as I think this book would be for novice scripters, there is a lot of smart stuff in there, stuff that had never occurred to me through my years of command line use. I actually got really excited to try some of the examples in there and to put them into practice. I particularly liked the little tricks here and there, like the “subshell trick” and I was absolutely thrilled that this book used modern syntax and variable manipulation, dropping the deprecated stuff like putting commands into back ticks. Good form!

   This book is certainly a keeper and I would recommend it highly to anyone who wants to become proficient on the command line. Some days you actually *do* get what you pay for, and I believe people will find this book to be a good example of that. This book was truly fun for me to work my way through and I sure hope they have more like it in store for the future. Go buy yourself a copy. I know I will be hanging on to this one for a while 🙂

Friday, March 25th, 2011

Stay Tuned!

Linux Shell Scripting Cookbook

Linux Shell Scripting Cookbook

I have been asked to review the “Linux Shell Scripting Cookbook” by Packt Publishing. It’s supposed to be coming in a couple days, so here’s your teaser to stay tuned! Packt Pub vs. Curmudgeonly SyaAdmin, a dead tree death match, only here at lincgeek.org/blog.

Wednesday, March 9th, 2011

My Web Workers Toolkit

Ahh, it’s been far too long since I have had anything to say here. I have to say that I have been *legitimately* busy this time. As many of you know, we have come to a close on the first season of The LinuxLink TechShow. That’s 365 (about) 2 hour long episodes over the course of the last 6 years. We are due to start our next 365 at the Ohio Linux Fest in a month. This leaves an enormous amount of prep work and a fair bit of reorganization to keep things exciting and help us start out with a bang.

One of the *surprises* in store brings me to my current topic at hand, my web workers toolkit.

People all have differing opinions about what you really need to do decent web work. As an old commandline jockey, I thought I would share my own.

1) Vim.
Quite possibly the best text editor in the world, I use vim for darn near everything. As a system administrator, it’s indispensable (for scripting) and I find it similarly necessary for web work. Vim has a fantastic (imho) syntax highlighting system which does quite well for html and php highlighting. The only caveat is to make sure to set “set background=dark” in your .vimrc file, unless, of course, you are one of those wierdos who uses a light background in your terminal.

2) tidy or the w3c validator.
I DEFY you to write good code without one of these. There is NOTHING as nice as standards compliant code and without a good validator, you will have nothing like standards compliant code. The reason I listed both of these is that tidy is a program you can use locally to check your code and the w3c validator will check any pages that are accessible via the web.

3) Many different browsers.
Unfortunately, all browsers are not made equal. You can be sure that all mozilla based browsers like Firefox, etc., will display things very similarly, and maybe even throw Google Chrome into that mix, but you may really want to check your code with Safari and IE to be sure things still look the way you had intended, and let’s not forget about a text browser like lynx or w3m to make sure your pages are readable and navigable that way too.

4) Lastly, for me, some good reference material.
One can hardly be expected to remember everything and having some reference material handy for those odd css commands and perhaps php/perl/python/someotherprogramminglanguage could really save you some time and frustration. Never underestimate keeping your old code around for example and never ever underestimate the power of the power of the Google Search!

In a nutshell, that’s generally what keeps me cranking out websites and webpages. What kinds of things do you use? What am I missing out on? Send a long a comment and let us all know what works for you! ( Unless, of course, you use emacs 😀 )

Tuesday, August 17th, 2010

Playing catch-up

I decided that on my vacation I would do some catch-up work. I have many times mentioned that I am a consummate procrastinator, and if you combine that with me being just generally whooped tired after 12 hours away from home on any average day, you understand why my computers seem to go uncared for. I think it’s the same as the whole “the mechanics car is never fixed” thing.

I mentioned a couple days ago that I installed ESXi on one of my home servers (redundant servers) to fix a strange problem I had been having with VMware Server 2.x. That was the first job I needed to so, or at least the most important, and so far it has been doing beautifully.

Next on the list was Mint 8 on the old laptop. It has been running Mint 7 since the distro was released and it was time for an upgrade. Everything was working just fine on 7, I just wanted to catch up the latest/greatest. As expected, the upgrade was a no-brainer and it’s running gorgeously, as Mint does.

Today, so far, I decided to upgrade my desktop machine to Mint 8. This machine, a P4 3Ghz with 3Gb of ram runs like absolute crap. I don’t exactly know why, but it always has. Now I have replaced the cpu fan a couple times and also the power supply at least twice. The computer is noisy, whiny, but not physically broken that I can tell. It just seems to run slower than hell and always has. The installation of Mint 8 on it did make it prettier, but sure didn’t make it seem to run any faster. I think it just dogs over the dual display and craptasticly old Nvidia card. Perhaps if I bought it a new quiet power supply, a better working and quieter cpu fan, a new better video card and a new dvdrom drive (yeah that’s pretty broken too), I could resuscitate this thing so that I could stand using it again. But then again, I could probably buy a whole new desktop computer for what I would spend on repairs to this one. Dang.

So, what’s next? Well, I should install ESXi on my redundant server now that I am satisfied with how the other one is running. I should also upgrade to Mint 8 on my Acer Aspire All In One netbook (notice a pattern here). Other than that, I am not sure.. Maybe work on some code projects I have been stringing along for months and months.

So what kinds of great computery projects are you all up to? Or what SHOULD you be up to 🙂

Monday, February 8th, 2010

Ubuntu 9.10 and Grub 2

ubuntu
Yes, another post about Ubuntu 9.10. I know I tried it out before, but I put it on this new (old) laptop and am giving it a little better run this time. I still believe 9.10 (Karmic) to be a fine running distribution and this time I got to test out my method of installing all the codecs I want on there, along with messing with Grub 2 a little bit.

When you are travelling abroad where it’s legal to do so, as i was just the other day, you might want to have access to all those codecs that make life worth living on a linux box. Things like listening to your mp3s and watching your dvds and miscellaneous media files are very dificult without them.

I realise that Ubuntu has, for some time now, been able to detect that you need so and so codec to play so and so media and ask you if you really want it installed, but I find that particularly irritating. I like to already have that functionality there when I want to use it. To do that, I have a little script that I use that generally takes care of that for me, along with installing most of the programs I need to make my day to day use hassle free.

#!/bin/bash
sudo wget http://www.medibuntu.org/sources.list.d/karmic.list -O /etc/apt/sources.list.d/medibuntu.list
sudo apt-get update && sudo apt-get install medibuntu-keyring && sudo apt-get update
sudo apt-get install mozilla-thunderbird php5-common php5-cli php-pear subversion openssh-server clusterssh imagemagick vim synergy smbfs curl vlc libdvdcss2 ubuntu-restricted-extras w32codecs mplayer mencoder build-essential sqlite dia expect mysql-client

Feel free to modify and use this, but basically I derived this from paying attention to the programs I need and use and making a list. It really does save a lot of time to do this.

The other thing I wanted to mention is Grub 2. For some reason, someone decided it was time to move from the original Grub to Grub 2. Time alone will tell whether that was a smart move or not. I know I certainly had a tough time of it for a day or two. Everything has moved and the methodology has changed as well. The short of it is you have some config files in /etc/grub.d that you can now manipulate, along with issuing a “update-grub”, that will build your /boot/grub/grub.cfg, which is pretty much the equivalent of the old /boot/grub/menu.lst file. The fun part is figuring out how all this works because, as it happens with open source many times, the documentation sucks.

What I needed to do was to add another linux distribution to grub so I could dual (or multi) boot it. This is accomplished in that /etc/grub.d directory. Now it’s worth mentioning here that if you do multiple OS installs on your machine and just issue a “update-grub” on your base Grub 2 enabled OS, it will (or at least mine did) auto detect this installation by default and add a boot option for it into the grub boot menu. The problem is, like mine, it probaly won’t boot your other OS.

The way to fix this is to go into /etc/grub.d and “chmod -x 30_os-prober”. After that you won’t be auto-genning entries. Next you can make a copy of the 40_custom file (I named mine 41_centos) and edit that file to have the correct boot parameters to boot your other OS. This is especially fun without having a good grasp of the correct syntax. For instance it took me hours to figure out that the “kernel” line that the old Grub used has been replaced with a “linux” line now. Other than that, though, just make sure that if you are booting another linux to use the correct root label and kernel and initrd image names and locations. My correct and working CentOS entry looks like this for reference:

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the ‘exec tail’ line above.
menuentry “CentOS 5.4” {
set root=(hd0,3)
linux /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=/ rhgb quiet
initrd /boot/initrd-2.6.18-164.el5.img
}

Have fun!

Tuesday, December 1st, 2009

Coding FLB style.

In between bouts of making Linc’s World Famous Potato Soup, I had a few minutes to do some catch-up coding today on FreeLinuxBox. It desperately needed an rss feed, so that is what I coded up. How else are you supposed to know there is new stuff there right? Well, all finished and added the feed to LinuxPlanet.org, which you should be subscribing to if you aren’t already 🙂 If you are just looking for the FLB feed, you can find it at http://freelinuxbox.org/rss/rss.php.

Speaking of Free Linux Boxes, Russ, The Techie Geek, was the latest person to put a box up on FLB and he has a GREAT idea. He wants local pickup (because of weight no doubt), but he said he’d be wiling of delivering to the Ohio Linux Fest. Outstanding idea. If you, like me, have some boxes you are putting off giving out because of the hassle in shipping, perhaps following Russ’ example could be the answer!

Saturday, August 15th, 2009