Markdown Content - Command Line Log Analysis Meets the Age of Agents

---
# Command Line Log Analysis Meets the Age of Agents

**URL:** https://crunchtools.com/command-line-log-analysis-age-of-agents/
Date: 2026-09-27
Author: fatherlinux
Post Type: post
Summary: Petit, my command line log analysis tool from 2009, is back. It shrank logs for sysadmins, and now it shrinks them for AI agents too.Continue Reading "Command Line Log Analysis Meets the Age of Agents" →
Categories: Articles, Software
Tags: AI/ML, Large Language Models, Logging, Open Source Software, Python, Security, Systems Administration
Featured Image: https://crunchtools.com/wp-content/uploads/2026/09/gemini_gen_20260927_121849_a29a4c7b.png
---

Back in 2010, I was a Systems Administrator at a local company called [eyemg.com](https://www.eyemg.com/). We had about 150 servers at two different data centers, and we ran a central Syslog server that collected log text from across our servers, firewalls, routers, and switches. The problem wasn't altogether different from what I faced at my previous job, back in 2005, as a Systems Administrator at [americangreetings.com](https://www.americangreetings.com/) with about a thousand Linux servers all needing to meet the recently enforced, and dreaded, PCI compliance! Every morning before I planned my day, or dug into whatever problem was threatening to ruin my day, I would read a quick, simple report from a nightly cron job. The job would chew through the raw text from all of the centrally managed logs, and spit out a clean morning report. It used an elegant little command line tool for log analysis called petit. That report took me about 3 to 5 minutes to read. An interesting byproduct of the methodology used by petit is that the length of the report held steady as the fleet grew. Petit worked because the number of unique patterns in a cluster does not grow linearly with the number of servers, routers, firewalls, and switches. Generally, a bigger fleet breaks down in the same ways as a smaller one. And, that's the crux of the value of petit. Nobody can read a mountain of raw logs, not even an AI agent now that [tokenmaxxing is dead](https://www.ibm.com/think/insights/tokenmaxxing-dead-long-live-valuemaxxing), but even a tired sysadmin, or a token-thrifty agent, can read a dozen summarized patterns and quickly analyze it for problems.

The tool actually started life as a scrappy Perl script I called lt, which was heavily inspired by Marcus Ranum's classic writing on [artificial ignorance](https://www.ranum.com/security/computer_security/papers/ai/). Ranum's thesis was pretty simple: you throw away everything you already know is boring, and then you spend your limited human energy reading whatever weird debris is left over. When I rewrote the tool in Python, I summarized that core loop in the 2009 README by saying that petit "quantitatively removes certainty, thereby leaving uncertainty, which by necessity requires qualitative analysis from a systems administrator [in 2026, I'd also say: or an agent]."  In plain English, the script hashes lines by replacing variable numbers, timestamps, network addresses, and anything else I can easily identify, with a simple hash symbol, which collapses thousands of identical log records into a single line with a count next to it.

## Putting Petit to Work Today

If you have never seen the tool run on a raw terminal, the core workflow takes about five seconds to understand.

petit hashing a 1500 line sshd log down to 8 lines, then graphing it, then reporting by daemon and by word.

That video demonstrates collapsing 1500 lines of noisy authentication traffic down to 8 distinct patterns. If you are auditing that bastion host, you do not need to read 537 successful public key lines, because they represent the normal operating state of the box. What immediately jumps out from the output is that line showing 6 password logins, which tells you right away that somebody logged in using a password on a machine that is supposed to be locked down to keys, and that is where you spend your time digging.

If you want to run petit on your own machines, we have built fresh, native packages available from our [signed package repositories](https://crunchtools.github.io/packages/) covering Fedora, RHEL, SUSE, Debian, and Ubuntu. If you are on Fedora or RHEL, you can set up the repo and install it directly:

```
$ sudo curl -fsSLo /etc/yum.repos.d/crunchtools.repo \
    https://crunchtools.github.io/packages/rpm/crunchtools.repo
$ sudo dnf install petit
```

If you prefer to avoid system package managers or you are working on an ephemeral test box, you can install it as a tool using uv on any system running Python 3.11 or newer:

```
`$ uv tool install petit-log-crunchtools`
```

For those of you who are building your own AI agent workflows or custom MCP servers, the core pattern hashing engine is also exported as a standard Python library, so you can pull it straight into your code:

```
`from petit import hash_lines`
```

Beyond basic pattern hashing, the utility can also draw ASCII graphs right in your terminal, count entries by daemon and by host, and run word-frequency counts to help you isolate weird error strings. Crafty sysadmins, and agents, can plumb these options together in creative new ways. 

## Where Petit Came From

While it originally started life as a perl script called "lt" (log tool, IIRC), I renamed the project to petit in August 2009, put the [first intro video on YouTube](https://www.youtube.com/watch?v=5hI5sUPuzGc) on 2009-08-12, and then gave a talk titled "Science in Systems Administration" at the Akron Linux Users Group that September. By the summer of 2010, I was doing a tutorial called [Log Analysis with Python](https://archive.org/details/pyvideo_514___pyohio-2010-log-analysis-with-python) at PyOhio and documenting our [morning report](https://crunchtools.com/centralizing-log-files/) on Crunchtools. People seemed to find it useful, and the community stepped up to help package it for the wider world. Sandro Mathys submitted the package review for Fedora, which landed petit in both Fedora and EPEL in 2010, while Carl Chenet packaged it for Debian, meaning it shipped in Debian and landed in every Ubuntu release from 11.04 through 19.10.

For a while there, petit was just part of the standard toolkit people grabbed when a disk was filling up with junk. Folks like [Jason Antman in 2012](https://blog.jasonantman.com/2012/02/petit-for-log-analysis/) and Aaron Kili on [Tecmint in 2017](https://www.tecmint.com/petiti-log-analysis-tool-for-linux-sysadmins/) wrote great tutorials showing how to use it for incident triage, and it felt like a stable piece of plumbing. But software maintenance is a relentless grind, and when the major distributions went through the great Python 2 purge between 2018 and 2020, unmaintained Python 2 packages got swept out with the trash. Fedora retired petit in December 2018, Debian and Ubuntu dropped it soon after, and for a couple of years the code sat dormant until I finally sat down in October 2022 to port the whole codebase to Python 3.

## I Thought Petit Was Dead as a Doorknob

Over the years, I let the project flounder as I moved into a different career at Red Hat, eventually becoming a Product Manager for Red Hat Enterprise Linux, focused on things like Podman, Red Hat Universal Base Image, RHEL 10, and now [Red Hat Hardened Images](https://images.redhat.com/). I'd had many dreams of reviving the project, but just didn't have time. I made a few half-hearted attempts, like porting it to Python 3 in 2022. 

Right after I finished that Python 3 port, generative AI exploded into the mainstream. When ChatGPT took off in early 2023, I honestly thought petit was dead as a doorknob. My operating assumption at the time was that sysadmins, platform engineers, and developers were simply going to dump raw log files into a chat box or Claude Code, ask the model what went sideways, and go back to eating lunch. During tokenmaxxing, that was true. 

But, then things changed...

## Teaching Agents How to Read Logs

It dawned on me about 6 or 7 months ago, in March 2026, when I started building [Trentina](https://github.com/crunchtools/mcp-trentina). Trentina started its life as a security tool, a gateway that sits between my AI agents and the tools they call through MCP servers. My original motivation for writing it was purely defensive, because I was expanding my use of Claude Code, OpenClaw, and Hermes to automate huge swaths of my life. I wasn't comfortable giving these agents access to important pieces of my life, without some defense against prompt injection. But as the tokenmaxxing era started to die, meaning people finally stopped throwing unlimited tokens at every minor problem and actually started caring about cost, they also started caring about what gets crammed into a context window. I realized that an AI agent reading a log file (or PDF file, or HTML file, or JSON file, etc) has similar constraints as a human sysadmin on third shift.

When an agent executes a command to inspect a container log or dump a system journal, it pays for every single line of text, both in cold hard cash and in context window it can't use for thinking. Most of that text is pure, predictable certainty that the agent has zero reason to read word by word. So in September 2026, I pulled petit into Trentina as a pre-processor that runs on tool output before the agent ever sees it.

There is an interesting property here that I like quite a bit. Petit is designed to only normalize tokens that carry deterministic meaning, things like bare numbers, hex strings, IP addresses, UUIDs, and timestamps. It never normalizes words. If an attacker tries to hide a prompt injection attack inside 10,000 lines of boilerplate web server logs, the malicious payload retains its unique English words, retains its distinct fingerprint, and gets handed to the security scanners instead of being washed away in a flood of duplicate noise. Petit and Trentina go together like a hand in a glove.

To give you an idea of what this looks like in practice, I pulled the numbers from roughly 16 hours of Trentina's audit logs. During that window, my agents ran 15 container-log reads through the podman backend, which pulled in 2.26 MB of raw text off the hosts. By the time Trentina's preprocessing was done, with petit doing the heavy lifting on the log lines, the data delivered to the agent was just 274 KB, which is roughly an 88% reduction in payload size. Across all 670 tool calls recorded during that same window, Trentina ingested 8.86 MB of raw data and emitted 2.46 MB to the agents. Now, please be patient with me here, because that wider number includes other pipeline steps like converting raw HTML into clean Markdown, so I cannot credit all of that savings to petit alone, but the container log numbers speak for themselves.

## The Work That Lasts

It is funny how infrastructure tooling works out over the long haul. Petit is a 17-year-old tool, built back in 2009 when we were all trying to figure out how to stop our central syslog servers from melting down. You would think something that old would be completely irrelevant in a world full of autonomous agents, but keeping things legible is an evergreen engineering problem. The newest software entities in the room have the exact same problem the human operators had back in 2009, which is that raw logs are mostly certainty, and certainty is a waste of everybody's attention.

I plan on writing more about how Trentina uses petit alongside our other MCP pre-processors, but that is for a future article. If you want to poke at the source code, file an issue, or read through how the tool has changed since 2009, you can check out the [petit GitHub repository](https://github.com/crunchtools/petit) and read through the full [project history page](https://github.com/crunchtools/petit/blob/master/docs/history.md).

---

## Categories

- Articles
- Software

---

## Navigation

- [Home](https://crunchtools.com/)
- [Articles](https://crunchtools.com/category/articles/)
- [Events](https://crunchtools.com/category/events/)
- [News](https://crunchtools.com/category/news/)
- [Presentations](https://crunchtools.com/category/presentations/)
- [Software](https://crunchtools.com/software/)
- [Beaver Backup](https://crunchtools.com/software/beaver-backup/)
- [Check BGP Neighbors](https://crunchtools.com/software/check-bgp-neighbors-nagios/)
- [Chev](https://crunchtools.com/software/chev-check-vulnerabilities-script/)
- [Graph BGP Neighbors](https://crunchtools.com/software/grpah-bgp-neighbors/)
- [Graph MySQL Stats](https://crunchtools.com/software/graph-mysql-stats/)
- [Graph Sockets Pipes Files](https://crunchtools.com/software/graph-sockets-pipes-files/)
- [MCP Servers](https://crunchtools.com/software/mcp-servers/)
- [Petit](https://crunchtools.com/software/petit/)
- [Racecar](https://crunchtools.com/software/racecar/)
- [Shiva](https://crunchtools.com/software/shiva/)
- [About](https://crunchtools.com/about/)

## Tags

- AI/ML
- Large Language Models
- Logging
- Open Source Software
- Python
- Security
- Systems Administration