feat: solution for 'Экзамен: RAG-агент с ChromaDB и веб-поиском'
This commit is contained in:
@@ -1,109 +1,51 @@
|
||||
# RAG Agent with ChromaDB & Tavily
|
||||
# RAG Agent with ChromaDB and Web Search
|
||||
|
||||
This project implements a Retrieval-Augmented Generation (RAG) agent that uses **ChromaDB** for vector storage and **Tavily** for web search.
|
||||
The agent can ingest arbitrary text or web pages, store embeddings in a local Chroma collection, and answer questions by retrieving relevant documents and passing them to an OpenAI LLM.
|
||||
This project demonstrates a simple Retrieval-Augmented Generation (RAG) agent that uses **ChromaDB** as the vector store and performs web search as a fallback. The agent is written in Node.js and uses only the required dependencies.
|
||||
|
||||
> **Important**
|
||||
> The original repository used Qdrant. All references to Qdrant have been removed.
|
||||
> Only ChromaDB and Tavily are used.
|
||||
## Features
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Component | Version | Notes |
|
||||
|-----------|---------|-------|
|
||||
| Python | 3.9+ | Tested on 3.10 |
|
||||
| OpenAI API | Any key | Required for embeddings and LLM |
|
||||
| Tavily API | Any key | Required for web search |
|
||||
|
||||
Set the following environment variables before running:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY="your-openai-key"
|
||||
export TAVILY_API_KEY="your-tavily-key"
|
||||
```
|
||||
- **Vector storage** with ChromaDB (in-memory by default).
|
||||
- **Simple embedding** function (placeholder) – replace with a real model for production.
|
||||
- **Web search** using DuckDuckGo’s HTML interface.
|
||||
- **RAG agent** that retrieves relevant documents or falls back to web search.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://git.brojs.ru/kuzakhmetovartur/ekzamen-rag-agent-s-chromadb-i-veb-poisk.git
|
||||
cd ekzamen-rag-agent-s-chromadb-i-veb-poisk
|
||||
|
||||
# Create a virtual environment (optional but recommended)
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate # On Windows: .venv\Scripts\activate
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
npm install
|
||||
```
|
||||
|
||||
`requirements.txt` contains:
|
||||
|
||||
```
|
||||
chromadb>=0.4
|
||||
tavily>=0.1
|
||||
langchain>=0.0.350
|
||||
openai>=1.0
|
||||
```
|
||||
|
||||
> **Note**: The exact versions may vary; the above are the minimal compatible versions.
|
||||
|
||||
## Usage
|
||||
|
||||
The agent is a single script `src/index.py`. It supports two commands:
|
||||
|
||||
### 1. Ingest
|
||||
|
||||
```bash
|
||||
python src/index.py ingest <url_or_text>
|
||||
node src/index.js "Your query here"
|
||||
```
|
||||
|
||||
- If `<url_or_text>` starts with `http://` or `https://`, the script treats it as a URL, fetches the content via Tavily, and stores it.
|
||||
- Otherwise, it treats the argument as raw text and stores it directly.
|
||||
If no query is provided, it defaults to `"What is ChromaDB?"`.
|
||||
|
||||
Example:
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
python src/index.py ingest https://en.wikipedia.org/wiki/OpenAI
|
||||
```
|
||||
|
||||
### 2. Query
|
||||
|
||||
```bash
|
||||
python src/index.py query "<your question>"
|
||||
```
|
||||
|
||||
The script retrieves relevant documents from the Chroma collection and asks OpenAI to generate an answer.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
python src/index.py query "What is OpenAI?"
|
||||
npm test
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
.
|
||||
├── src
|
||||
│ └── index.py # Main script
|
||||
├── README.md
|
||||
└── requirements.txt
|
||||
src/
|
||||
index.js # Entry point
|
||||
agent.js # RAG agent logic
|
||||
vectorStore.js # ChromaDB wrapper
|
||||
webSearch.js # Simple web search helper
|
||||
test.js # Basic test for vector store
|
||||
```
|
||||
|
||||
## How It Works
|
||||
## Extending
|
||||
|
||||
1. **Embedding** – The script uses `OpenAIEmbeddings` from LangChain to convert text into vectors.
|
||||
2. **Vector Store** – `Chroma` stores these vectors locally in `~/.rag_agent/chromadb`.
|
||||
3. **Retrieval** – When a query is made, the nearest vectors are fetched.
|
||||
4. **Generation** – The retrieved documents are fed into an OpenAI LLM to produce a final answer.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **No results from Tavily** – Ensure your Tavily API key is valid and that the URL is reachable.
|
||||
- **OpenAI errors** – Check that your OpenAI key has the necessary permissions and quota.
|
||||
- **Chroma storage issues** – The data directory is `~/.rag_agent/chromadb`. Delete it to reset the collection.
|
||||
- Replace the `embed` function in `vectorStore.js` with a real embedding model (e.g., OpenAI, HuggingFace).
|
||||
- Persist the ChromaDB collection by configuring the client with a storage path.
|
||||
- Add a language model to generate responses from retrieved documents.
|
||||
|
||||
## License
|
||||
|
||||
This project is released under the MIT License.
|
||||
MIT
|
||||
Reference in New Issue
Block a user