feat: solution for 'Экзамен: RAG-агент с ChromaDB и веб-поиском'

README.md

@@ -1,91 +1,87 @@
 # RAG Agent with ChromaDB and Web Search
 
-This project implements a Retrieval-Augmented Generation (RAG) agent that uses **ChromaDB** as the vector database and performs live web searches to provide up‑to‑date information.
-
-## Features
-
-- **Vector store** – Documents are ingested, split into chunks, embedded with OpenAI embeddings, and stored in a persistent ChromaDB collection.
-- **Web search** – Uses DuckDuckGo scraping to fetch recent web snippets for a query.
-- **RAG pipeline** – Combines local document context and web results, then generates an answer with OpenAI GPT‑3.5‑Turbo.
-- **CLI** – Simple command line interface for ingestion and querying.
+This project implements a Retrieval-Augmented Generation (RAG) agent that:
+- Stores and retrieves embeddings from **ChromaDB**.
+- Performs web search using DuckDuckGo to fetch additional context.
+- Generates answers with an **Ollama** language model.
 
 ## Prerequisites
 
-- Python 3.10+
-- An OpenAI API key with access to `text-embedding-ada-002` and `gpt-3.5-turbo`.
+- Node.js v20 or newer
+- ChromaDB server running locally (default URL: `chromadb://localhost:8000`)
+- Ollama server running locally (default URL: `http://localhost:11434`)
 
 ## Setup
 
+1. **Clone the repository**
+
    ```bash
-# Clone the repository
    git clone https://git.brojs.ru/kuzakhmetovartur/ekzamen-rag-agent-s-chromadb-i-veb-poisk.git
    cd ekzamen-rag-agent-s-chromadb-i-veb-poisk
-
-# Create a virtual environment (optional but recommended)
-python -m venv .venv
-source .venv/bin/activate   # On Windows use `.venv\Scripts\activate`
-
-# Install dependencies
-pip install -r requirements.txt
    ```
 
-## Configuration
-
-Create a `.env` file in the project root (or set environment variables directly):
-
-```
-OPENAI_API_KEY=sk-...
-CHROMA_DB_PATH=./chromadb
-CHROMA_COLLECTION_NAME=rag_collection
-```
-
-> **Note**: Do not commit your `.env` file or API key to version control.
-
-## Usage
-
-### 1. Ingest Documents
+2. **Install dependencies**
 
    ```bash
-python src/main.py ingest path/to/doc1.txt path/to/doc2.txt
+   npm install
    ```
 
-The script will read each file, split it into chunks, generate embeddings, and store them in ChromaDB.
+3. **Configure environment variables**
 
-### 2. Query the Agent
+   Create a `.env` file in the project root (or modify the existing one):
+
+   ```dotenv
+   CHROMA_URL=chromadb://localhost:8000
+   CHROMA_COLLECTION=rag_collection
+   OLLAMA_HOST=http://localhost:11434
+   OLLAMA_MODEL=llama3
+   OLLAMA_EMBEDDING_MODEL=nomic-embed-text
+   ADD_SAMPLE_DOCS=true
+   ```
+
+   - `CHROMA_URL`: URL of your ChromaDB instance.
+   - `CHROMA_COLLECTION`: Name of the collection to use.
+   - `OLLAMA_HOST`: URL of your Ollama server.
+   - `OLLAMA_MODEL`: Ollama model for generation.
+   - `OLLAMA_EMBEDDING_MODEL`: Ollama model for embeddings.
+   - `ADD_SAMPLE_DOCS`: Set to `true` to automatically add a few sample documents on startup.
+
+4. **Run the agent**
 
    ```bash
-python src/main.py query "What is the capital of France?"
+   npm start -- "Your question here"
+   ```
+
+   Example:
+
+   ```bash
+   npm start -- "What is LangChain?"
    ```
 
    The agent will:
-
-1. Retrieve relevant chunks from the local vector store.
-2. Perform a DuckDuckGo web search for the query.
-3. Combine both sources of information.
-4. Generate a response using OpenAI GPT‑3.5‑Turbo.
+   - Search the local ChromaDB collection.
+   - Perform a DuckDuckGo web search.
+   - Combine the results and generate an answer using Ollama.
 
 ## Project Structure
 
 ```
-src/
-├── main.py          # CLI entry point
-├── vector_store.py  # ChromaDB ingestion & retrieval
-├── web_search.py    # DuckDuckGo web search
-requirements.txt
-README.md
+.
+├── src
+│   ├── agent.js        # Agent logic (retrieval + generation)
+│   ├── index.js        # CLI entry point
+│   ├── vectorStore.js  # ChromaDB wrapper
+│   └── webSearch.js    # DuckDuckGo search helper
+├── .env                # Environment configuration
+├── package.json        # Dependencies and scripts
+└── README.md           # Documentation
 ```
 
-## Testing
+## Notes
 
-The project can be tested with `pytest` (tests are not included in this minimal example).  
-If you add tests, run:
+- The agent uses **LangChain 1.x** APIs.
+- No Qdrant references are present; only ChromaDB is used.
+- The web search is performed via DuckDuckGo’s public JSON API (no API key required).
+- The Ollama LLM is used for both embeddings and generation.
 
-```bash
-pytest
-```
-
-## License
-
-MIT License
---- 
-Feel free to extend the agent with additional features such as custom embeddings, different LLMs, or alternative search APIs.
+Feel free to extend the agent with additional retrievers or custom prompts as needed.

package.json

@@ -1,15 +1,17 @@
 {
-  "name": "rag-agent-chromadb",
+  "name": "rag-agent-chromadb-websearch",
   "version": "1.0.0",
-  "description": "A simple RAG agent using ChromaDB for vector storage and web search.",
+  "description": "RAG agent using ChromaDB and web search with Ollama LLM",
   "main": "src/index.js",
   "type": "module",
   "scripts": {
-    "start": "node src/index.js",
-    "test": "node src/test.js"
+    "start": "node src/index.js"
   },
   "dependencies": {
-    "@chromadb/chromadb": "^0.1.0",
-    "node-fetch": "^3.3.2"
+    "langchain": "^1.0.0",
+    "chromadb": "^1.0.0",
+    "node-fetch": "^3.3.0",
+    "dotenv": "^16.0.0",
+    "ollama": "^0.1.0"
   }
 }

src/agent.js

@@ -1,28 +1,26 @@
-import { VectorStore } from "./vectorStore.js";
-import { webSearch } from "./webSearch.js";
+import { Ollama } from 'langchain/llms/ollama';
+import { RetrievalQAChain } from 'langchain/chains/retrieval_qa';
+import { BaseRetriever } from 'langchain/schema';
+import { webSearch } from './webSearch.js';
 
-/**
- * A simple RAG agent that retrieves relevant documents from the vector store
- * and optionally performs a web search if no relevant documents are found.
- */
-export class Agent {
-  constructor(vectorStore) {
-    this.vectorStore = vectorStore;
+export function createAgent(vectorStore) {
+  const llm = new Ollama({
+    model: process.env.OLLAMA_MODEL || 'llama3',
+    baseUrl: process.env.OLLAMA_HOST || 'http://localhost:11434',
+  });
+
+  class CombinedRetriever extends BaseRetriever {
+    async getRelevantDocuments(query) {
+      const chromaDocs = await vectorStore.similaritySearch(query, 3);
+      const webDocs = await webSearch(query, 3);
+      return [...chromaDocs, ...webDocs];
+    }
   }
 
-  /**
-   * Processes a user query and returns the best answer.
-   * @param {string} query
-   * @returns {Promise<string>}
-   */
-  async answer(query) {
-    const results = await this.vectorStore.query(query, 3);
-    if (results.length > 0 && results[0].score < 0.5) {
-      // Return the most relevant document text
-      return results[0].text;
-    }
-    // Fallback to web search
-    const html = await webSearch(query);
-    return `No relevant local documents found. Here is the raw web search result:\n${html}`;
-  }
+  const retriever = new CombinedRetriever();
+  const chain = RetrievalQAChain.fromLLMAndRetriever(llm, retriever, {
+    returnSourceDocuments: true,
+  });
+
+  return chain;
 }

src/index.js

@@ -1,26 +1,47 @@
-import { VectorStore } from "./vectorStore.js";
-import { Agent } from "./agent.js";
+import dotenv from 'dotenv';
+dotenv.config();
+
+import { createVectorStore } from './vectorStore.js';
+import { createAgent } from './agent.js';
+import { Document } from 'langchain/document';
 
 async function main() {
-  const store = new VectorStore();
-  await store.init();
+  const vectorStore = await createVectorStore();
 
-  // Sample documents to index
-  const docs = [
-    { id: "1", text: "ChromaDB is a fast, lightweight vector database." },
-    { id: "2", text: "It supports in-memory and persistent storage." },
-    { id: "3", text: "You can use it with various embedding models." },
+  if (process.env.ADD_SAMPLE_DOCS === 'true') {
+    const sampleDocs = [
+      new Document({
+        pageContent:
+          'LangChain is a framework for building applications powered by language models.',
+        metadata: { source: 'LangChain Docs' },
+      }),
+      new Document({
+        pageContent: 'ChromaDB is a vector database for storing embeddings.',
+        metadata: { source: 'ChromaDB Docs' },
+      }),
+      new Document({
+        pageContent: 'Ollama is a lightweight LLM server that can run locally.',
+        metadata: { source: 'Ollama Docs' },
+      }),
     ];
+    await vectorStore.addDocuments(sampleDocs);
+    console.log('Sample documents added to ChromaDB.');
+  }
 
-  await store.addDocuments(docs);
+  const agent = createAgent(vectorStore);
 
-  const agent = new Agent(store);
+  const query = process.argv[2];
+  if (!query) {
+    console.error('Please provide a query as a command line argument.');
+    process.exit(1);
+  }
 
-  const query = process.argv[2] || "What is ChromaDB?";
-  console.log(`Query: ${query}`);
-  const answer = await agent.answer(query);
-  console.log("\nAnswer:");
-  console.log(answer);
+  const result = await agent.invoke({ input: query });
+  console.log('Answer:', result.output);
+  console.log(
+    'Sources:',
+    result.sourceDocuments.map((d) => d.metadata.source).join(', ')
+  );
 }
 
 main().catch((err) => {

src/vectorStore.js

@@ -1,69 +1,47 @@
-import { Client } from "@chromadb/chromadb";
+import { ChromaClient } from 'chromadb';
+import { OllamaEmbeddings } from 'langchain/embeddings/ollama';
+import { Document } from 'langchain/document';
 
-/**
- * Simple embedding function that converts text into a fixed-length numeric vector.
- * This is a placeholder and should be replaced with a real embedding model for production use.
- */
-function embed(text) {
-  const vector = Array.from(text)
-    .map((c) => c.charCodeAt(0))
-    .slice(0, 10);
-  while (vector.length < 10) {
-    vector.push(0);
-  }
-  return vector;
-}
-
-export class VectorStore {
-  constructor() {
-    this.client = new Client();
-    this.collection = null;
-  }
-
-  async init() {
-    this.collection = await this.client.getOrCreateCollection({
-      name: "rag_collection",
+export async function createVectorStore() {
+  const chroma = new ChromaClient({ path: process.env.CHROMA_URL });
+  const collection = await chroma.getOrCreateCollection({
+    name: process.env.CHROMA_COLLECTION,
   });
+  const embeddings = new OllamaEmbeddings({
+    model: process.env.OLLAMA_EMBEDDING_MODEL || 'nomic-embed-text',
+  });
+  return new VectorStore(collection, embeddings);
+}
+
+class VectorStore {
+  constructor(collection, embeddings) {
+    this.collection = collection;
+    this.embeddings = embeddings;
   }
 
-  /**
-   * Adds an array of documents to the collection.
-   * @param {Array<{id: string, text: string}>} docs
-   */
   async addDocuments(docs) {
-    if (!this.collection) {
-      throw new Error("VectorStore not initialized. Call init() first.");
-    }
-    const ids = docs.map((d) => d.id);
-    const embeddings = docs.map((d) => embed(d.text));
-    const documents = docs.map((d) => d.text);
-    await this.collection.add({
-      ids,
+    const texts = docs.map((d) => d.pageContent);
+    const embeddings = await this.embeddings.embedDocuments(texts);
+    await this.collection.addDocuments({
+      documents: docs,
       embeddings,
-      documents,
     });
   }
 
-  /**
-   * Queries the collection for the most relevant documents.
-   * @param {string} queryText
-   * @param {number} nResults
-   * @returns {Promise<Array<{id: string, text: string, score: number}>>}
-   */
-  async query(queryText, nResults = 3) {
-    if (!this.collection) {
-      throw new Error("VectorStore not initialized. Call init() first.");
-    }
-    const queryEmbedding = embed(queryText);
-    const results = await this.collection.query({
-      queryEmbeddings: [queryEmbedding],
-      nResults,
+  async similaritySearch(query, k = 4) {
+    const embedding = await this.embeddings.embedQuery(query);
+    const results = await this.collection.getNearestNeighbors({
+      queryEmbeddings: [embedding],
+      n: k,
     });
-    // results is an array of objects with ids, documents, and scores
-    return results[0].ids.map((id, idx) => ({
-      id,
-      text: results[0].documents[idx],
-      score: results[0].distances[idx],
-    }));
+    const ids = results.ids[0];
+    const docs = await this.collection.getDocuments({ ids });
+    return docs.map(
+      (doc) =>
+        new Document({
+          pageContent: doc.document,
+          metadata: doc.metadata,
+        })
+    );
   }
 }

src/webSearch.js

@@ -1,17 +1,24 @@
-import fetch from "node-fetch";
+import fetch from 'node-fetch';
+import { Document } from 'langchain/document';
 
-/**
- * Performs a simple web search using DuckDuckGo's HTML interface.
- * This is a lightweight example and does not use an official API.
- * @param {string} query
- * @returns {Promise<string>} The raw HTML of the search results page.
- */
-export async function webSearch(query) {
-  const url = `https://duckduckgo.com/html/?q=${encodeURIComponent(query)}`;
-  const response = await fetch(url);
-  if (!response.ok) {
-    throw new Error(`Web search failed with status ${response.status}`);
+export async function webSearch(query, limit = 3) {
+  const url = `https://api.duckduckgo.com/?q=${encodeURIComponent(
+    query
+  )}&format=json&pretty=1`;
+  const res = await fetch(url);
+  const data = await res.json();
+  const topics = data.RelatedTopics || [];
+  const docs = [];
+  for (const topic of topics) {
+    if (topic.Text) {
+      docs.push(
+        new Document({
+          pageContent: topic.Text,
+          metadata: { source: 'DuckDuckGo', url: topic.FirstURL },
+        })
+      );
     }
-  const html = await response.text();
-  return html;
+    if (docs.length >= limit) break;
+  }
+  return docs.slice(0, limit);
 }