2.5 KiB
2.5 KiB
Structured Log Parser
This project demonstrates how to parse raw log lines into typed events using Pydantic v2 and LangChain's structured output.
It supports two event types:
| Event | Fields |
|---|---|
HttpOkEvent |
kind="ok", status=200, path, duration_ms |
HttpErrorEvent |
kind="error", status (4xx/5xx), path, error_message |
The parser uses a LangChain prompt to convert each log line into a JSON object that matches one of the schemas. The output is then validated with Pydantic.
Features
- Discriminated Union:
ApiEventis a union ofHttpOkEventandHttpErrorEventwith akinddiscriminator. - Structured Output: Uses LangChain's
PydanticOutputParserto enforce schema. - Batch Parsing: Handles multiple log lines in a single run.
- CLI: Accepts example logs, a file, or custom text and prints a table of parsed events.
Installation
# Clone the repo
git clone https://github.com/yourusername/structured-log-parser.git
cd structured-log-parser
# Create a virtual environment (optional but recommended)
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
Usage
1. Using the built‑in example
python -m src.main --example
2. Parsing a log file
python -m src.main --file path/to/log.txt
3. Parsing custom text
python -m src.main --text "2026-08-31 12:00:01 INFO /api/users 200 123ms"
The output will be a Markdown‑style table:
| kind | path | status | duration_ms / error_message |
|------|---------------|--------|-----------------------------|
| ok | /api/users | 200 | 123 ms |
| error| /api/orders | 404 | Not Found |
| error| /api/payments | 500 | Internal Server Error |
| ok | /api/products | 200 | 45 ms |
Environment Variables
The project uses OpenAI's API. Set the following variable before running:
export OPENAI_API_KEY="sk-..."
Alternatively, create a .env file in the project root:
OPENAI_API_KEY=sk-...
The python-dotenv package will load it automatically.
Project Structure
src/
├── __init__.py
├── main.py # CLI entry point
├── cli.py # Argument parsing and table rendering
├── parser.py # Log parsing logic
└── models.py # Pydantic event models
License
MIT License