Top 11 Reasons to Analyze Your Logs
by Anton Chuvakin
- Seen an obscure log message lately? Me too - in fact, everybody have. How do you know what it means (and logs usually do mean something) without analysis? At the very least, you might need to bring additional context to know what some logs mean (example: IP address -> hostname -> server owner)
- Logs often measure in gigabytes and soon will in terabytes; log volume grows all the time - it definitely passed the limit of what a human can read a long time ago, it then made simple filtering 'what logs to read' impossible as well: automated log analysis is the only choice.
- Do you peruse your logs in real time? This is simply absurd! However, automated real-time analysis is entirely possible (and some logs do crave for your attention ASAP - e.g. major system failures, confirmed intrusions, etc)
- Can you read multiple logs at the same time? Yes, kind of, if you print them out on multiple pages to correlate (yes, I've seen this done :-)). Is this efficient? God, no! Correlation across logs of different types is one of the most useful approaches to log analysis.
- A lot of insight hides in "sparse" logs, logs where a single record barely matters, but a large aggregate does (e.g. from one "connection allowed" firewall log to a scan pattern). Thus, the only way to extract that insight from a pool of data is through algorithms that "condense" that collection of logs into usable knowledge (some say, visualization is the way to go)
- Ever did a manual log baselining? This is where you read the logs for a while and learn which ones are normal for your environment. Wonna do it again? Thought so :-) Log baseline learning is a useful and simple log analysis technique, but humans can only do it for so much before burning out.
- OK, let's pick the important logs to review. Which ones are those? The right answer is "we don't know, until we see them." Thus, to even figure out which logs to read, you need automated analysis.
- Log analysis for compliance? Why, yes! Compliance is NOT only about log storage (e.g. see PCI DSS). How to highlight compliance-relevant messages? How to see which messages will lead to a violation? How do you satisfy those "daily log review" requirements (again, see PCI DSS)? Through automated analysis, of course!
- Logs allow you to profile your users, your data and your resources/assets. Really? Yes, really: such profiling can then tell you if those users behave in an unusual manner (in fact, the oldest log analysis systems worked like that). Such techniques may help reach the holy grail of log analysis: have the system automatically tell you what matters for you!
- Ever tried to hire a log analysis expert? Those are few and far between. What if your junior analysts can suddenly analyze logs just as well? One log analysis system creator told me that his log data mining system enabled exactly that. Thus, saving a lot of money to his organization.
- Finally, can you predict future with your logs? I hope so! Research on predictive analytics is ongoing, but you can only do it with automated analysis tools, not with just your head alone (no matter how big :-)) ...
Past top 11 reasons:
- Top 11 Reasons to Collect and Preserve Computer Logs
- Top 11 Reasons to Look at Your Logs
- Top 11 Reasons to Secure and Protect Your Logs
You forgot the 12th reason!