feat: solution for 'Экзамен: RAG-агент с ChromaDB и веб-поиском'
This commit is contained in:
@@ -1,110 +1,73 @@
|
||||
# RAG Agent with ChromaDB and Web Search
|
||||
|
||||
This project implements a Retrieval-Augmented Generation (RAG) agent that uses a local ChromaDB vector store for document retrieval and falls back to DuckDuckGo web search when the local store does not provide sufficient context.
|
||||
This project demonstrates a Retrieval-Augmented Generation (RAG) agent built with **LangChain 1.x**, **ChromaDB** as the vector store, and **SerpAPI** for web search integration.
|
||||
|
||||
## Features
|
||||
|
||||
- **Local Retrieval** – Store and query embeddings in a persistent ChromaDB collection.
|
||||
- **Web Search Fallback** – If local retrieval fails to find relevant context, the agent performs a DuckDuckGo search and uses the snippets.
|
||||
- **OpenAI Integration** – Uses OpenAI embeddings (`text-embedding-ada-002`) and the `gpt-3.5-turbo` model for generation.
|
||||
- **CLI** – Simple command line interface for ingesting documents and asking questions.
|
||||
- Stores documents in ChromaDB and generates embeddings using OpenAI.
|
||||
- Retrieves relevant documents via a vector store tool.
|
||||
- Performs live web searches with SerpAPI.
|
||||
- Combines both sources to answer user queries.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.9+
|
||||
- An OpenAI API key
|
||||
- (Optional) Internet access for web search
|
||||
- Node.js v18+ (ES modules support)
|
||||
- A running ChromaDB instance (default: `localhost:8000`)
|
||||
- OpenAI API key
|
||||
- SerpAPI key
|
||||
|
||||
## Installation
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://git.brojs.ru/kuzakhmetovartur/ekzamen-rag-agent-s-chromadb-i-veb-poisk.git
|
||||
cd ekzamen-rag-agent-s-chromadb-i-veb-poisk
|
||||
1. **Clone the repository**
|
||||
|
||||
# Create a virtual environment (recommended)
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate # On Windows use `.venv\Scripts\activate`
|
||||
```bash
|
||||
git clone https://github.com/your-username/rag-agent-chromadb-websearch.git
|
||||
cd rag-agent-chromadb-websearch
|
||||
```
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
2. **Install dependencies**
|
||||
|
||||
`requirements.txt` contains:
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
3. **Configure environment variables**
|
||||
|
||||
Create a `.env` file in the project root:
|
||||
|
||||
```dotenv
|
||||
OPENAI_API_KEY=your_openai_api_key
|
||||
CHROMA_HOST=localhost
|
||||
CHROMA_PORT=8000
|
||||
SERPAPI_KEY=your_serpapi_key
|
||||
```
|
||||
|
||||
4. **Run the agent**
|
||||
|
||||
```bash
|
||||
npm start
|
||||
```
|
||||
|
||||
The agent will add sample documents to ChromaDB, then answer a sample query using both the vector store and web search.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
openai
|
||||
chromadb
|
||||
duckduckgo-search
|
||||
beautifulsoup4
|
||||
requests
|
||||
src/
|
||||
├── index.js # Entry point
|
||||
├── agent.js # Agent construction
|
||||
├── vectorStore.js # ChromaDB interactions
|
||||
└── webSearch.js # SerpAPI web search
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
## Customization
|
||||
|
||||
| Variable | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `OPENAI_API_KEY` | Your OpenAI API key | `sk-...` |
|
||||
| `CHROMA_DB_PATH` | Directory where ChromaDB stores data | `./chromadb` |
|
||||
| `CHROMA_COLLECTION_NAME` | Name of the collection | `rag_collection` |
|
||||
| `TOP_K` | Number of top documents to retrieve | `5` |
|
||||
| `SIMILARITY_THRESHOLD` | Minimum similarity to consider a document relevant | `0.5` |
|
||||
| `WEB_SEARCH_MAX_RESULTS` | Max number of web snippets to fetch | `3` |
|
||||
|
||||
Set them in your shell or create a `.env` file and load with `dotenv` (optional).
|
||||
|
||||
## Usage
|
||||
|
||||
### Ingest Documents
|
||||
|
||||
Place your plain text files (`.txt`) in a folder, then run:
|
||||
|
||||
```bash
|
||||
python src/index.py ingest /path/to/text/files
|
||||
```
|
||||
|
||||
The script will read all `.txt` files, split them into chunks, embed them, and store them in ChromaDB.
|
||||
|
||||
### Ask a Question
|
||||
|
||||
```bash
|
||||
python src/index.py ask "What is the capital of France?"
|
||||
```
|
||||
|
||||
The agent will:
|
||||
|
||||
1. Query the local vector store for relevant passages.
|
||||
2. If none are found above the similarity threshold, perform a DuckDuckGo search.
|
||||
3. Combine the retrieved context into a prompt.
|
||||
4. Call OpenAI’s `gpt-3.5-turbo` to generate an answer.
|
||||
|
||||
## Example
|
||||
|
||||
```bash
|
||||
$ python src/index.py ingest ./data
|
||||
INFO:root:Added 12 documents to collection 'rag_collection'.
|
||||
|
||||
$ python src/index.py ask "Explain the theory of relativity."
|
||||
Answer:
|
||||
The theory of relativity, developed by Albert Einstein, consists of two parts: special relativity and general relativity. ...
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Unit tests are provided in the `tests/` directory. To run them:
|
||||
|
||||
```bash
|
||||
pytest tests/
|
||||
```
|
||||
|
||||
(If you don't have `pytest` installed, run `pip install pytest`.)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **No documents ingested** – Ensure the folder path is correct and contains `.txt` files.
|
||||
- **OpenAI errors** – Verify that `OPENAI_API_KEY` is set and that you have sufficient quota.
|
||||
- **Web search fails** – Check your internet connection and that DuckDuckGo is reachable.
|
||||
- **Adding Documents**: Use `addDocuments` from `vectorStore.js` to add your own documents.
|
||||
- **Changing LLM**: Replace `OpenAI` with another LLM provider supported by LangChain.
|
||||
- **Adjusting Retrieval**: Modify the number of results returned by the vector store or web search.
|
||||
|
||||
## License
|
||||
|
||||
MIT License
|
||||
MIT License
|
||||
---
|
||||
Happy coding!
|
||||
Reference in New Issue
Block a user