I came and and started reviewing my reports and was looking at an access report, which is basically a “last | grep $TheDateIWant” from over the weekend. I keep a pretty tight ship and want to know who is accessing what servers and when (and sometimes why). What I saw was monstrously suspicious! I saw MYSELF logged in to 3 different servers 3 times each around 5am on Sunday morning – while I was sleeping.
This is the kind of thing to throw you into an immediate panic first thing on a Monday morning, but I decided to give myself 10 minutes to investigate before completely freaking out.
The first thing I noticed was that the access/login times looked suspiciously like the same times I ran my daily reports on the machines, however, the previous week I had changed the user that runs those reports and this was still saying it was me. I double, triple and quadruple checked and searched all the report programs to make absolutely sure there was no indication that they were still using my personal account (which was probably bad practice to begin with btw). Then I scoured all the cron logs to see what was actually running at those times, and oddly enough, it was just those reports.
I looked through the command line history on those machines and checked again the “last | head” to see who was logging on those machines. Nothing out of place BUT with the “last| head” I was NOT listed as being on the machine on that date! So I ran the entire report command again “last | grep $TheDateIWant” and there I was again, listed right under the logins of the report user.
Anyone catching this yet?
What I had stumbled upon were a few machines that are used so infrequently that the wtmp file, which is what the “last” command uses for data, had over 1 year of entries. My search of “last | grep ‘Oct 31′” was returning not only this year, but my own logins from last year as well.
Moral of the story? Mondays stink – Just stay home!