RAG‑Agent with ChromaDB and Web Search

A lightweight Retrieval‑Augmented Generation (RAG) agent that scrapes web pages, stores their embeddings in a local Chroma vector store, and answers user questions using OpenAI’s GPT‑3.5 Turbo.
The project demonstrates how to combine:

• BeautifulSoup – for HTML parsing
• OpenAI embeddings – to convert text into vectors
• ChromaDB – fast local vector search
• LangChain – orchestration of LLM, embeddings and QA chain

---

📦 Installation

# 1. Clone the repo (or copy solution.py)
git clone https://github.com/your‑repo/rag-agent.git
cd rag-agent

# 2. Create a virtual environment (recommended)
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate

# 3. Install dependencies
pip install -r requirements.txt

requirements.txt:

beautifulsoup4==4.12.3
langchain==0.2.0
openai==1.30.0
chromadb==0.5.6
rich==13.7.1
requests==2.32.3

Note

You must have an OpenAI API key set in the environment:

export OPENAI_API_KEY="sk-..."
# Windows: set OPENAI_API_KEY=sk-...

---

🚀 Running the Agent

The project contains a single script solution.py. It performs three actions depending on the command‑line argument:

Argument	Action
scrape	Scrapes URLs from a file, stores embeddings in ChromaDB.
ask	Loads the vector store and asks a question via the console.
clean	Deletes the local Chroma database (optional).

1. Prepare a list of URLs

Create a plain text file (urls.txt) with one URL per line, e.g.:

https://en.wikipedia.org/wiki/Artificial_intelligence
https://www.python.org/

2. Scrape and index the pages

python solution.py scrape urls.txt

The script will:

1. Download each page (respecting robots.txt is not enforced).
2. Extract visible text with BeautifulSoup.
3. Generate embeddings via OpenAI.
4. Store them in ./chroma_db.

3. Ask a question

python solution.py ask

You will be prompted to type your question. The agent retrieves the most relevant chunks from ChromaDB and feeds them into GPT‑3.5 Turbo, printing the answer.

Tip

– If you want to see the raw retrieved snippets, set console.print(chunks) in the script before the QA chain.

4. Clean up

python solution.py clean

Deletes the local Chroma database (./chroma_db).

---

📄 Example Session

$ python solution.py scrape urls.txt
[green]Scraping and indexing 2 URLs...[/green]
[green]Done! 3 chunks stored in chroma_db.[/green]

$ python solution.py ask
Enter your question: What is the main goal of artificial intelligence?
[cyan]Retrieving relevant documents...[/cyan]
Answer:
Artificial Intelligence (AI) aims to create systems that can perform tasks requiring human-like intelligence, such as learning, reasoning, problem‑solving, perception, and language understanding. The primary goal is to develop machines that can adapt, learn from experience, and make decisions in complex environments.

---

🛠️ Customization

Setting	Where to change
Embedding model	EMBEDDINGS_MODEL in solution.py
LLM model	LLM_MODEL in solution.py
Chroma directory	CHROMA_DIR in solution.py

Feel free to replace the OpenAI models with local alternatives (e.g., HuggingFace) by adjusting the LangChain wrappers.

---

🤝 Contributing

Pull requests are welcome!
Please open an issue first to discuss major changes or new features.

---