Agent with RAG Memory

This repository contains a simple Retrieval‑Augmented Generation (RAG) agent implemented with LangChain, FAISS for vector storage, and OpenAI embeddings and LLM. It also provides an auto_check_graph function that verifies the generated answer against a ground‑truth mapping and returns a verdict_row.

Important

The auto‑check graph must return a verdict_row. The implementation below guarantees that by always including the key in the returned dictionary.

Features

RAG Agent – Load documents, embed them, store in FAISS, and answer queries.
Auto‑Check Graph – Run a query, generate an answer, compare it to a ground‑truth answer, and return a verdict (PASS, FAIL, or UNKNOWN).
Unit Tests – Verify that the agent and auto‑check graph work as expected.

Installation

# Create a virtual environment (recommended)
python -m venv .venv
source .venv/bin/activate   # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

requirements.txt contains:

langchain
openai
faiss-cpu
pytest

OpenAI API Key
If you want to use real embeddings and LLM, set the environment variable OPENAI_API_KEY:

export OPENAI_API_KEY="sk-..."

If the key is not set, the agent falls back to FakeEmbeddings and FakeLLM, which are suitable for local testing and unit tests.

Usage

from src.index import RAGAgent, auto_check_graph

# Create agent
agent = RAGAgent()

# Add documents (e.g., from a directory)
agent.add_documents([
    "The capital of France is Paris.",
    "William Shakespeare wrote Hamlet."
])

# Define ground truth mapping
ground_truth = {
    "What is the capital of France?": "Paris",
    "Who wrote Hamlet?": "William Shakespeare",
}

# Run auto‑check graph
result = auto_check_graph(
    "What is the capital of France?",
    agent,
    ground_truth
)

print(result)
# Output:
# {
#   "verdict_row": "PASS",
#   "answer": "Paris",
#   "expected": "Paris"
# }

Running Tests

pytest

The tests cover:

Adding documents and querying.
Auto‑check graph returning PASS, FAIL, and UNKNOWN verdicts.
Handling of empty queries and missing ground‑truth.

Project Structure

src/
├── index.py          # Main implementation
tests/
├── test_agent.py     # Unit tests
README.md
requirements.txt

License

MIT License

2.4 KiB Raw Blame History Unescape Escape