135 lines
3.5 KiB
Markdown
135 lines
3.5 KiB
Markdown
# RAG‑Agent with ChromaDB and Web Search
|
||
|
||
A lightweight Retrieval‑Augmented Generation (RAG) agent that scrapes web pages, stores their embeddings in a local **Chroma** vector store, and answers user questions using OpenAI’s GPT‑3.5 Turbo.
|
||
The project demonstrates how to combine:
|
||
|
||
* **BeautifulSoup** – for HTML parsing
|
||
* **OpenAI embeddings** – to convert text into vectors
|
||
* **ChromaDB** – fast local vector search
|
||
* **LangChain** – orchestration of LLM, embeddings and QA chain
|
||
|
||
---
|
||
|
||
## 📦 Installation
|
||
|
||
```bash
|
||
# 1. Clone the repo (or copy solution.py)
|
||
git clone https://github.com/your‑repo/rag-agent.git
|
||
cd rag-agent
|
||
|
||
# 2. Create a virtual environment (recommended)
|
||
python -m venv .venv
|
||
source .venv/bin/activate # Windows: .venv\Scripts\activate
|
||
|
||
# 3. Install dependencies
|
||
pip install -r requirements.txt
|
||
```
|
||
|
||
`requirements.txt`:
|
||
|
||
```text
|
||
beautifulsoup4==4.12.3
|
||
langchain==0.2.0
|
||
openai==1.30.0
|
||
chromadb==0.5.6
|
||
rich==13.7.1
|
||
requests==2.32.3
|
||
```
|
||
|
||
> **Note**
|
||
> You must have an OpenAI API key set in the environment:
|
||
|
||
```bash
|
||
export OPENAI_API_KEY="sk-..."
|
||
# Windows: set OPENAI_API_KEY=sk-...
|
||
```
|
||
|
||
---
|
||
|
||
## 🚀 Running the Agent
|
||
|
||
The project contains a single script `solution.py`. It performs three actions depending on the command‑line argument:
|
||
|
||
| Argument | Action |
|
||
|----------|--------|
|
||
| `scrape` | Scrapes URLs from a file, stores embeddings in ChromaDB. |
|
||
| `ask` | Loads the vector store and asks a question via the console. |
|
||
| `clean` | Deletes the local Chroma database (optional). |
|
||
|
||
### 1. Prepare a list of URLs
|
||
|
||
Create a plain text file (`urls.txt`) with one URL per line, e.g.:
|
||
|
||
```text
|
||
https://en.wikipedia.org/wiki/Artificial_intelligence
|
||
https://www.python.org/
|
||
```
|
||
|
||
### 2. Scrape and index the pages
|
||
|
||
```bash
|
||
python solution.py scrape urls.txt
|
||
```
|
||
|
||
The script will:
|
||
|
||
1. Download each page (respecting `robots.txt` is **not** enforced).
|
||
2. Extract visible text with BeautifulSoup.
|
||
3. Generate embeddings via OpenAI.
|
||
4. Store them in `./chroma_db`.
|
||
|
||
### 3. Ask a question
|
||
|
||
```bash
|
||
python solution.py ask
|
||
```
|
||
|
||
You will be prompted to type your question. The agent retrieves the most relevant chunks from ChromaDB and feeds them into GPT‑3.5 Turbo, printing the answer.
|
||
|
||
> **Tip** – If you want to see the raw retrieved snippets, set `console.print(chunks)` in the script before the QA chain.
|
||
|
||
### 4. Clean up
|
||
|
||
```bash
|
||
python solution.py clean
|
||
```
|
||
|
||
Deletes the local Chroma database (`./chroma_db`).
|
||
|
||
---
|
||
|
||
## 📄 Example Session
|
||
|
||
```text
|
||
$ python solution.py scrape urls.txt
|
||
[green]Scraping and indexing 2 URLs...[/green]
|
||
[green]Done! 3 chunks stored in chroma_db.[/green]
|
||
|
||
$ python solution.py ask
|
||
Enter your question: What is the main goal of artificial intelligence?
|
||
[cyan]Retrieving relevant documents...[/cyan]
|
||
Answer:
|
||
Artificial Intelligence (AI) aims to create systems that can perform tasks requiring human-like intelligence, such as learning, reasoning, problem‑solving, perception, and language understanding. The primary goal is to develop machines that can adapt, learn from experience, and make decisions in complex environments.
|
||
|
||
```
|
||
|
||
---
|
||
|
||
## 🛠️ Customization
|
||
|
||
| Setting | Where to change |
|
||
|---------|-----------------|
|
||
| Embedding model | `EMBEDDINGS_MODEL` in `solution.py` |
|
||
| LLM model | `LLM_MODEL` in `solution.py` |
|
||
| Chroma directory | `CHROMA_DIR` in `solution.py` |
|
||
|
||
Feel free to replace the OpenAI models with local alternatives (e.g., HuggingFace) by adjusting the LangChain wrappers.
|
||
|
||
---
|
||
|
||
## 🤝 Contributing
|
||
|
||
Pull requests are welcome!
|
||
Please open an issue first to discuss major changes or new features.
|
||
|
||
--- |