Files

2.5 KiB
Raw Permalink Blame History

Structured Log Parser

This project demonstrates how to parse raw log lines into typed events using Pydantic v2 and LangChain's structured output.
It supports two event types:

Event Fields
HttpOkEvent kind="ok", status=200, path, duration_ms
HttpErrorEvent kind="error", status (4xx/5xx), path, error_message

The parser uses a LangChain prompt to convert each log line into a JSON object that matches one of the schemas. The output is then validated with Pydantic.

Features

  • Discriminated Union: ApiEvent is a union of HttpOkEvent and HttpErrorEvent with a kind discriminator.
  • Structured Output: Uses LangChain's PydanticOutputParser to enforce schema.
  • Batch Parsing: Handles multiple log lines in a single run.
  • CLI: Accepts example logs, a file, or custom text and prints a table of parsed events.

Installation

# Clone the repo
git clone https://github.com/yourusername/structured-log-parser.git
cd structured-log-parser

# Create a virtual environment (optional but recommended)
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Usage

1. Using the builtin example

python -m src.main --example

2. Parsing a log file

python -m src.main --file path/to/log.txt

3. Parsing custom text

python -m src.main --text "2026-08-31 12:00:01 INFO /api/users 200 123ms"

The output will be a Markdownstyle table:

| kind | path          | status | duration_ms / error_message |
|------|---------------|--------|-----------------------------|
| ok   | /api/users    | 200    | 123 ms                      |
| error| /api/orders   | 404    | Not Found                   |
| error| /api/payments | 500    | Internal Server Error       |
| ok   | /api/products | 200    | 45 ms                       |

Environment Variables

The project uses OpenAI's API. Set the following variable before running:

export OPENAI_API_KEY="sk-..."

Alternatively, create a .env file in the project root:

OPENAI_API_KEY=sk-...

The python-dotenv package will load it automatically.

Project Structure

src/
├── __init__.py
├── main.py          # CLI entry point
├── cli.py           # Argument parsing and table rendering
├── parser.py        # Log parsing logic
└── models.py        # Pydantic event models

License

MIT License