feat: solution for '8. Самописный поисковый агент на основе deep agents from scratch'
This commit is contained in:
@@ -1,68 +1,98 @@
|
||||
# Deep Search Agent – LangChain Implementation
|
||||
# Самописный поисковый агент на основе deep agents from scratch
|
||||
|
||||
This repository contains a minimal implementation of a **search agent** built with LangChain, following the “Deep Agents from Scratch” template.
|
||||
The agent can answer arbitrary questions by performing a web search and reasoning over the results.
|
||||
## Описание
|
||||
|
||||
## Features
|
||||
Это простая реализация поискового агента, использующего трансформерный энкодер для векторизации документов и запросов. Поиск осуществляется по косинусному сходству между векторами.
|
||||
|
||||
- Uses **OpenAI GPT‑4o‑mini** as the language model.
|
||||
- Performs web searches via **SerpAPI** (Google/SerpAPI).
|
||||
- Maintains conversation context with a memory buffer.
|
||||
- Implements the **Zero‑Shot React** agent pattern.
|
||||
- Simple command‑line interface for interactive use.
|
||||
## Структура проекта
|
||||
|
||||
## Prerequisites
|
||||
```
|
||||
src/
|
||||
├── index.py # Основной код агента и API
|
||||
data/
|
||||
└── documents.txt # Текстовый файл с документами (один документ на строку)
|
||||
README.md
|
||||
```
|
||||
|
||||
- Python 3.10+
|
||||
- An OpenAI API key.
|
||||
- A SerpAPI key (free tier available).
|
||||
> **Важно**: файл `data/documents.txt` должен существовать и содержать хотя бы несколько строк текста. Если его нет, агент не запустится.
|
||||
|
||||
## Setup
|
||||
## Установка
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://git.brojs.ru/kuzakhmetovartur/8.-samopisnyy-poiskovyy-agent-na-osnove-<repo>.git
|
||||
cd <repo>
|
||||
# Создайте виртуальное окружение (рекомендуется)
|
||||
python -m venv venv
|
||||
source venv/bin/activate # Windows: venv\Scripts\activate
|
||||
|
||||
# Create a virtual environment (optional but recommended)
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate # On Windows: .venv\\Scripts\\activate
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
# Установите зависимости
|
||||
pip install -U pip
|
||||
pip install torch transformers fastapi uvicorn pydantic numpy
|
||||
```
|
||||
|
||||
Create a `.env` file in the project root with your credentials:
|
||||
> Если у вас есть GPU, убедитесь, что установлена версия `torch` с поддержкой CUDA.
|
||||
|
||||
```
|
||||
OPENAI_API_KEY=sk-...
|
||||
SERPAPI_KEY=your-serpapi-key
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
Run the agent interactively:
|
||||
## Запуск
|
||||
|
||||
```bash
|
||||
python -m src.agent
|
||||
python src/index.py
|
||||
```
|
||||
|
||||
You will be prompted to enter a question. The agent will search the web and return a concise answer.
|
||||
Сервер будет доступен по адресу `http://0.0.0.0:8000`.
|
||||
|
||||
## Example
|
||||
## API
|
||||
|
||||
```
|
||||
Enter your question: What is the capital of France?
|
||||
Processing...
|
||||
### POST `/search`
|
||||
|
||||
=== Answer ===
|
||||
The capital of France is Paris.
|
||||
**Запрос**
|
||||
|
||||
```json
|
||||
{
|
||||
"query": "пример запроса",
|
||||
"top_k": 5
|
||||
}
|
||||
```
|
||||
|
||||
## Testing
|
||||
- `query` – строка запроса.
|
||||
- `top_k` – количество возвращаемых документов (по умолчанию 5).
|
||||
|
||||
The agent can be tested programmatically by importing `create_search_agent` from `src.agent` and calling `agent.run("your question")`.
|
||||
**Ответ**
|
||||
|
||||
## License
|
||||
```json
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"document": "текст найденного документа",
|
||||
"score": 0.8723
|
||||
},
|
||||
...
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Пример использования
|
||||
|
||||
```bash
|
||||
curl -X POST "http://0.0.0.0:8000/search" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query":"машинное обучение", "top_k":3}'
|
||||
```
|
||||
|
||||
## Как добавить документы
|
||||
|
||||
1. Откройте файл `data/documents.txt`.
|
||||
2. Добавьте новые строки – каждая строка будет рассматриваться как отдельный документ.
|
||||
3. Перезапустите сервер, чтобы обновления вступили в силу.
|
||||
|
||||
## Тесты
|
||||
|
||||
Тестов в проекте нет, но вы можете быстро проверить работу:
|
||||
|
||||
```python
|
||||
from src.index import SearchAgent
|
||||
|
||||
agent = SearchAgent()
|
||||
print(agent.search("пример", top_k=3))
|
||||
```
|
||||
|
||||
## Лицензия
|
||||
|
||||
MIT License
|
||||
+95
-77
@@ -1,92 +1,110 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Simple search agent implementation.
|
||||
|
||||
This module provides a minimal command‑line interface that accepts a search
|
||||
query and returns a list of dummy results. It is intentionally lightweight
|
||||
to satisfy the assignment requirements while demonstrating a clear
|
||||
structure that can be expanded in the future.
|
||||
|
||||
Author: Artur Kuzakhmetov
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import numpy as np
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModel
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from pydantic import BaseModel
|
||||
from typing import List
|
||||
|
||||
|
||||
def search(query: str, limit: int = 5) -> List[str]:
|
||||
"""
|
||||
Perform a mock search for the given query.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
class SearchRequest(BaseModel):
|
||||
query: str
|
||||
The search string.
|
||||
limit : int, optional
|
||||
Maximum number of results to return. Defaults to 5.
|
||||
top_k: int = 5
|
||||
|
||||
Returns
|
||||
-------
|
||||
List[str]
|
||||
A list of fake search results.
|
||||
class SearchResult(BaseModel):
|
||||
document: str
|
||||
score: float
|
||||
|
||||
Notes
|
||||
-----
|
||||
This function does not perform real network requests. It simply
|
||||
generates deterministic placeholder results so that the module can be
|
||||
tested without external dependencies.
|
||||
class SearchResponse(BaseModel):
|
||||
results: List[SearchResult]
|
||||
|
||||
class SearchAgent:
|
||||
"""
|
||||
if not query:
|
||||
raise ValueError("Query must not be empty")
|
||||
A simple deep‑agent search engine that uses a transformer encoder
|
||||
to embed documents and queries, then ranks documents by cosine
|
||||
similarity.
|
||||
"""
|
||||
def __init__(self,
|
||||
data_path: str = "data/documents.txt",
|
||||
model_name: str = "sentence-transformers/all-MiniLM-L6-v2"):
|
||||
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
||||
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||
self.model = AutoModel.from_pretrained(model_name).to(self.device)
|
||||
self.documents = self._load_documents(data_path)
|
||||
self.embeddings = self._embed_documents(self.documents)
|
||||
|
||||
# Generate deterministic dummy results
|
||||
results = [f"{query} result {i+1}" for i in range(limit)]
|
||||
def _load_documents(self, path: str) -> List[str]:
|
||||
if not os.path.exists(path):
|
||||
raise FileNotFoundError(f"Data file not found: {path}")
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
docs = [line.strip() for line in f if line.strip()]
|
||||
return docs
|
||||
|
||||
def _embed_documents(self, docs: List[str]) -> np.ndarray:
|
||||
batch_size = 32
|
||||
embeddings = []
|
||||
for i in range(0, len(docs), batch_size):
|
||||
batch = docs[i:i+batch_size]
|
||||
inputs = self.tokenizer(batch,
|
||||
padding=True,
|
||||
truncation=True,
|
||||
return_tensors="pt").to(self.device)
|
||||
with torch.no_grad():
|
||||
outputs = self.model(**inputs)
|
||||
token_embeddings = outputs.last_hidden_state
|
||||
attention_mask = inputs["attention_mask"].unsqueeze(-1)
|
||||
sum_embeddings = torch.sum(token_embeddings * attention_mask, dim=1)
|
||||
sum_mask = torch.clamp(attention_mask.sum(dim=1), min=1e-9)
|
||||
batch_embeddings = sum_embeddings / sum_mask
|
||||
embeddings.append(batch_embeddings.cpu().numpy())
|
||||
return np.vstack(embeddings)
|
||||
|
||||
def _embed_query(self, query: str) -> np.ndarray:
|
||||
inputs = self.tokenizer([query],
|
||||
padding=True,
|
||||
truncation=True,
|
||||
return_tensors="pt").to(self.device)
|
||||
with torch.no_grad():
|
||||
outputs = self.model(**inputs)
|
||||
token_embeddings = outputs.last_hidden_state
|
||||
attention_mask = inputs["attention_mask"].unsqueeze(-1)
|
||||
sum_embeddings = torch.sum(token_embeddings * attention_mask, dim=1)
|
||||
sum_mask = torch.clamp(attention_mask.sum(dim=1), min=1e-9)
|
||||
query_embedding = sum_embeddings / sum_mask
|
||||
return query_embedding.cpu().numpy()
|
||||
|
||||
def search(self, query: str, top_k: int = 5) -> List[SearchResult]:
|
||||
query_emb = self._embed_query(query)
|
||||
dot = np.dot(self.embeddings, query_emb.T).squeeze()
|
||||
norms = np.linalg.norm(self.embeddings, axis=1) * np.linalg.norm(query_emb)
|
||||
similarities = dot / norms
|
||||
top_indices = np.argsort(similarities)[::-1][:top_k]
|
||||
results = [SearchResult(document=self.documents[idx],
|
||||
score=float(similarities[idx]))
|
||||
for idx in top_indices]
|
||||
return results
|
||||
|
||||
app = FastAPI(title="Deep Agent Search API")
|
||||
|
||||
def main(argv: List[str] | None = None) -> int:
|
||||
# Instantiate the agent once at startup
|
||||
agent = SearchAgent()
|
||||
|
||||
@app.post("/search", response_model=SearchResponse)
|
||||
async def search_endpoint(request: SearchRequest):
|
||||
"""
|
||||
Entry point for the command‑line interface.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
argv : List[str] | None
|
||||
List of command‑line arguments. If None, sys.argv[1:] is used.
|
||||
|
||||
Returns
|
||||
-------
|
||||
int
|
||||
Exit code (0 for success, 1 for error).
|
||||
Search endpoint that accepts a JSON payload:
|
||||
{
|
||||
"query": "your search query",
|
||||
"top_k": 5
|
||||
}
|
||||
Returns the top_k most relevant documents.
|
||||
"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Simple search agent – returns mock results for a query."
|
||||
)
|
||||
parser.add_argument(
|
||||
"query",
|
||||
type=str,
|
||||
help="Search query string",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-n",
|
||||
"--limit",
|
||||
type=int,
|
||||
default=5,
|
||||
help="Number of results to return (default: 5)",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
try:
|
||||
results = search(args.query, args.limit)
|
||||
except ValueError as exc:
|
||||
print(f"Error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
for idx, result in enumerate(results, start=1):
|
||||
print(f"{idx}. {result}")
|
||||
|
||||
return 0
|
||||
|
||||
results = agent.search(request.query, request.top_k)
|
||||
return SearchResponse(results=results)
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8000)
|
||||
Reference in New Issue
Block a user