Ideal Log Management Tool?

by Anton Chuvakin

The idea came from Jeremiah Grossman (here) when he described "The Best Web Application Vulnerability Scanner in the World" thus: "Within a few moments of pressing the scan button it’ll find every vulnerability, with zero false positives, generate a pretty looking report, and voila you’re compliant with GLBA, HIPAA, and PCI-DSS. Of course, we all know such a web application scanner is simply not possible to create for a variety of reasons."

So, let's imagine the idea log management application.

  1. Logging configuration: the ideal log app will go and find all possible log sources (systems, devices, applications, etc) and then enable the right kind of logging on them according to a high level policy given to it (required: God-like powers)
  2. Log collection: it will collect all the above logs securely (and without using any risky super-user access ) and with little to no impact to networks and systems (required: God-like powers)
  3. Log storage: it can security store the above logs in the original format for as long as needed and in a manner allowing quick access to them - in both raw and summarized/enriched form (required: plenty of hardware)
  4. Log analysis: this ideal application will be able to look at all kinds of logs, known to it and previously unseen, from standard and custom log sources, and tell the user what they need to know about their environment and based on their needs: what is broken? what is hacked? where? what is in violation of regulations/policies? what will break soon? who is doing this stuff? The analysis will power all of the following: automated actions, real-time notifications, long-term historical analysis as well as compliance relevance analysis (required: AI)
  5. Information presentation: this tool will distill the above data, information and conclusions generated by the analytic components and present then in a manner consistent with the user's role: from operator to analyst to engineer to executive. Interactive visual and drillable text-based data presentation across all log sources. The users can also customize the data presentation based on their wishes and job needs, as well as information perception styles (required: nothing more than a bunch of daring UI designers)
  6. Automation: the ideal log management tool will be able to take limited automated actions to resolve discovered and confirmed issues as well as generate guidance to users so that they know what actions to take, when full-auto mode is not appropriate. The responses will range from full-auto actions to assisted actions ('click here to fix it') to issuing detailed remediation guidance. The output will include a TODO-list of discovered items complete with actions suggested, ordered by priority (required: AI + some luck + some user stupidity :-))
  7. Compliance: this tool can also be used directly by auditors to validate or prove compliance with relevant regulations by using regulation-specific content and all the collected data. The tool will also point at gaps in data collection as it applies to specific regulations that the user is interested in complying (required: God-like powers)

In other words, this magic black box will have crap shoveled from one side and will have answers to questions about the meaning of Life :-) coming out the other side...

What? :-) Am I nuts? Well, can I dream for a second? :-)

Technorati tags: , , , ,




3 Comments

Matt
2007-11-06 08:27:47
syslog-ng + (my)sql + an as-yet-not-created php application gets you pretty close. Omnipotence is left as an exercise to the reader.
rwo
2007-11-15 08:30:39
How about XpoLog. If you believe their sales literature, they can do most of what you list above except maybe autodiscover your logs for you. I've never worked with it, but I am curious about it...
Anton Chuvakin
2007-11-15 09:31:18
>How about


Well, I did look at it and I can say this: ... IF and ONLY IF you can believe the sales literature, the world will indeed be a different place :-) Better, in fact :-)