LincolnBlogs

Recently I had to do some more email migrations, and I wanted to script them. But how do you take hundreds of migrations and run, say, 15 at a time, scripted, so you don’t beat up your mail server? Good question!

What I settled on was writing a bash script to call a certain number of background screen processes. I figured that way I could be sure that disconnecting from my session didn’t kill my processes, AND I could always jump into any screened process easily to see what was going on. It wasn’t as hard to get going as you might think and it worked brilliantly!

#!/bin/bash

while read command
do
if [ $(cat totalthreads.txt) -gt $(screen -ls | grep Socket | tail -n1 | cut -d’ ‘ -f1 | sed -e ‘s/No/0/g’) ]
then
echo $command
screen -d -m $command &
fi
while [ $(cat totalthreads.txt) -eq $(screen -ls | grep Socket | tail -n1 | cut -d’ ‘ -f1 | sed -e ‘s/No/0/g’) ]
do
sleep 60
done
while [ $(cat totalthreads.txt) -lt $(screen -ls | grep Socket | tail -n1 | cut -d’ ‘ -f1 | sed -e ‘s/No/0/g’) ]
do
sleep 60
done
done < commandlist.txt

So, what you see here is the entire shebang. What’s happening is all the commands I want to run, which in my case are imap copies but you could do anything really, are listed in a file called “commandlist.txt” individually, each on their own line. It looks like this:

copy this file to that file
copy file 2 to anotherfile 2
copy file 3 to anotherfile 3
copy file 4 to anotherfile 4

The script grabs the commands you want to run and checks a file called “totalthreads.txt”, which simply contains a number. This is the number of threads you want running at any given time. So, say I have 100 commands (copys in the example) I want to run, but only 20 at a time so I don’t eat up all my I.O. I would put all the commands in the commandlist.txt file and a “20” in the totalthreads.txt file, then execute the script itself, which I called “threads.sh” (get it HERE). This script starts up 20 of the commands in the commandlist.txt file, each in their own screen session. When one of your processes ends, another will be started, maintaining that limit listed in the totalthreads.txt file, as long as their are enough commands in the commandlist.txt file to do so.

The nice part about this is it is the thread count is somewhat interactive. This means that if you notice you want more threads running, simply increase the number in the totalthreads.txt file and after a minute or two, you will see them increase. Conversely, if you want less, decrease the number and as processes finish, new threads will not be started until the number running is lower than the number in the totalthreads.txt file.