Anton Security Tip of the Week #16: Virtually There - Journey Into VMWare ESX Log Analysis
by Anton Chuvakin
Following the new "tradition" of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it "pay it forward" to the community.
So, Anton Security Tip of the Day #16: Virtually Screwed - Journey Into VMWare ESX Log Analysis
CISecurty guide for VMWare (here) and DISA STIG for virtual machines (here) both mandate collection and analysis of VM platform logs; none goes into enough details on what to look for in logs. Let's try to shed some light on security-focused log analysis of VMWare ESX v. 3.x logs.
First, at least until ESXi becomes the default choice, one needs to keep in mind that ESX as "Linux-inside" and thus diving into /var/log will not reveal any "alien technology" (well, not much :-)). However, one of the most useful logs is /var/log/hostd.N which is not a descendant of Linux standard logs. Extensive VM event records are written into this file.
Let's focus on various types of logins to the ESX platform and identify logs that indicate a successful and failed attempts to log in. Here are a few useful examples to analyze:
- May 30 09:20:42 esx2 su(pam_unix): session opened for user root by jhonny(uid=1626)
This is a classic Linux root login message; you can watch for these by searching VMWare ESX logs for "session AND opened AND user AND root." Notice the user name of the user who switched to root.
- May 30 09:20:34 esx2 sshd(pam_unix): session opened for user jhonny by (uid=0)
This is also a classic Linux message for a normal (non-root) user login.
- [2008-05-25 06:57:48.774 'ha-eventmgr' 111639472 info] Event 40645 : User firstname.lastname@example.org logged in
This is a VMWare -specific application login to ESX. You can track such events by username, by event ID or by keywords "event AND logged AND user" (if you are using search)
- May 30 09:20:31 esx2 sshd: Failed password for jhonny from 220.127.116.11 port 54773 ssh2
Another classic Linux message from the ESX system; a failure to login due to incorrect password.
- May 27 12:06:59 esx2 sshd: Failed password for illegal user jonny from 18.104.22.168 port 30594 ssh2
A message indicating a failure to login due to incorrect username (note a typo).
- May 25 07:03:48 esx1 sudo: jhonny : 3 incorrect password attempts ; TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash
This ESX Linux platform message should also be familiar to Linux/Unix admins: it indicates multiple sudo password failures; look for such messages in the logs.
BTW, do you need to be reminded to track NOT only failed, but also successful login events?!
Overall, you must prepare for the future by learning to analyze VMWare logs, just like you handled "legacy OS", such as Linux/Unix and Windows.