feat: solution for 'Экзамен: RAG-агент с ChromaDB и веб-поиском'
This commit is contained in:
@@ -1,91 +1,87 @@
|
||||
# RAG Agent with ChromaDB and Web Search
|
||||
|
||||
This project implements a Retrieval-Augmented Generation (RAG) agent that uses **ChromaDB** as the vector database and performs live web searches to provide up‑to‑date information.
|
||||
|
||||
## Features
|
||||
|
||||
- **Vector store** – Documents are ingested, split into chunks, embedded with OpenAI embeddings, and stored in a persistent ChromaDB collection.
|
||||
- **Web search** – Uses DuckDuckGo scraping to fetch recent web snippets for a query.
|
||||
- **RAG pipeline** – Combines local document context and web results, then generates an answer with OpenAI GPT‑3.5‑Turbo.
|
||||
- **CLI** – Simple command line interface for ingestion and querying.
|
||||
This project implements a Retrieval-Augmented Generation (RAG) agent that:
|
||||
- Stores and retrieves embeddings from **ChromaDB**.
|
||||
- Performs web search using DuckDuckGo to fetch additional context.
|
||||
- Generates answers with an **Ollama** language model.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.10+
|
||||
- An OpenAI API key with access to `text-embedding-ada-002` and `gpt-3.5-turbo`.
|
||||
- Node.js v20 or newer
|
||||
- ChromaDB server running locally (default URL: `chromadb://localhost:8000`)
|
||||
- Ollama server running locally (default URL: `http://localhost:11434`)
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://git.brojs.ru/kuzakhmetovartur/ekzamen-rag-agent-s-chromadb-i-veb-poisk.git
|
||||
cd ekzamen-rag-agent-s-chromadb-i-veb-poisk
|
||||
1. **Clone the repository**
|
||||
|
||||
# Create a virtual environment (optional but recommended)
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate # On Windows use `.venv\Scripts\activate`
|
||||
```bash
|
||||
git clone https://git.brojs.ru/kuzakhmetovartur/ekzamen-rag-agent-s-chromadb-i-veb-poisk.git
|
||||
cd ekzamen-rag-agent-s-chromadb-i-veb-poisk
|
||||
```
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
2. **Install dependencies**
|
||||
|
||||
## Configuration
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
Create a `.env` file in the project root (or set environment variables directly):
|
||||
3. **Configure environment variables**
|
||||
|
||||
```
|
||||
OPENAI_API_KEY=sk-...
|
||||
CHROMA_DB_PATH=./chromadb
|
||||
CHROMA_COLLECTION_NAME=rag_collection
|
||||
```
|
||||
Create a `.env` file in the project root (or modify the existing one):
|
||||
|
||||
> **Note**: Do not commit your `.env` file or API key to version control.
|
||||
```dotenv
|
||||
CHROMA_URL=chromadb://localhost:8000
|
||||
CHROMA_COLLECTION=rag_collection
|
||||
OLLAMA_HOST=http://localhost:11434
|
||||
OLLAMA_MODEL=llama3
|
||||
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
|
||||
ADD_SAMPLE_DOCS=true
|
||||
```
|
||||
|
||||
## Usage
|
||||
- `CHROMA_URL`: URL of your ChromaDB instance.
|
||||
- `CHROMA_COLLECTION`: Name of the collection to use.
|
||||
- `OLLAMA_HOST`: URL of your Ollama server.
|
||||
- `OLLAMA_MODEL`: Ollama model for generation.
|
||||
- `OLLAMA_EMBEDDING_MODEL`: Ollama model for embeddings.
|
||||
- `ADD_SAMPLE_DOCS`: Set to `true` to automatically add a few sample documents on startup.
|
||||
|
||||
### 1. Ingest Documents
|
||||
4. **Run the agent**
|
||||
|
||||
```bash
|
||||
python src/main.py ingest path/to/doc1.txt path/to/doc2.txt
|
||||
```
|
||||
```bash
|
||||
npm start -- "Your question here"
|
||||
```
|
||||
|
||||
The script will read each file, split it into chunks, generate embeddings, and store them in ChromaDB.
|
||||
Example:
|
||||
|
||||
### 2. Query the Agent
|
||||
```bash
|
||||
npm start -- "What is LangChain?"
|
||||
```
|
||||
|
||||
```bash
|
||||
python src/main.py query "What is the capital of France?"
|
||||
```
|
||||
|
||||
The agent will:
|
||||
|
||||
1. Retrieve relevant chunks from the local vector store.
|
||||
2. Perform a DuckDuckGo web search for the query.
|
||||
3. Combine both sources of information.
|
||||
4. Generate a response using OpenAI GPT‑3.5‑Turbo.
|
||||
The agent will:
|
||||
- Search the local ChromaDB collection.
|
||||
- Perform a DuckDuckGo web search.
|
||||
- Combine the results and generate an answer using Ollama.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
src/
|
||||
├── main.py # CLI entry point
|
||||
├── vector_store.py # ChromaDB ingestion & retrieval
|
||||
├── web_search.py # DuckDuckGo web search
|
||||
requirements.txt
|
||||
README.md
|
||||
.
|
||||
├── src
|
||||
│ ├── agent.js # Agent logic (retrieval + generation)
|
||||
│ ├── index.js # CLI entry point
|
||||
│ ├── vectorStore.js # ChromaDB wrapper
|
||||
│ └── webSearch.js # DuckDuckGo search helper
|
||||
├── .env # Environment configuration
|
||||
├── package.json # Dependencies and scripts
|
||||
└── README.md # Documentation
|
||||
```
|
||||
|
||||
## Testing
|
||||
## Notes
|
||||
|
||||
The project can be tested with `pytest` (tests are not included in this minimal example).
|
||||
If you add tests, run:
|
||||
- The agent uses **LangChain 1.x** APIs.
|
||||
- No Qdrant references are present; only ChromaDB is used.
|
||||
- The web search is performed via DuckDuckGo’s public JSON API (no API key required).
|
||||
- The Ollama LLM is used for both embeddings and generation.
|
||||
|
||||
```bash
|
||||
pytest
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT License
|
||||
---
|
||||
Feel free to extend the agent with additional features such as custom embeddings, different LLMs, or alternative search APIs.
|
||||
Feel free to extend the agent with additional retrievers or custom prompts as needed.
|
||||
Reference in New Issue
Block a user