feat: solution for '8. Самописный поисковый агент на основе deep agents from scratch'
This commit is contained in:
@@ -1,68 +1,98 @@
|
|||||||
# Deep Search Agent – LangChain Implementation
|
# Самописный поисковый агент на основе deep agents from scratch
|
||||||
|
|
||||||
This repository contains a minimal implementation of a **search agent** built with LangChain, following the “Deep Agents from Scratch” template.
|
## Описание
|
||||||
The agent can answer arbitrary questions by performing a web search and reasoning over the results.
|
|
||||||
|
|
||||||
## Features
|
Это простая реализация поискового агента, использующего трансформерный энкодер для векторизации документов и запросов. Поиск осуществляется по косинусному сходству между векторами.
|
||||||
|
|
||||||
- Uses **OpenAI GPT‑4o‑mini** as the language model.
|
## Структура проекта
|
||||||
- Performs web searches via **SerpAPI** (Google/SerpAPI).
|
|
||||||
- Maintains conversation context with a memory buffer.
|
|
||||||
- Implements the **Zero‑Shot React** agent pattern.
|
|
||||||
- Simple command‑line interface for interactive use.
|
|
||||||
|
|
||||||
## Prerequisites
|
```
|
||||||
|
src/
|
||||||
|
├── index.py # Основной код агента и API
|
||||||
|
data/
|
||||||
|
└── documents.txt # Текстовый файл с документами (один документ на строку)
|
||||||
|
README.md
|
||||||
|
```
|
||||||
|
|
||||||
- Python 3.10+
|
> **Важно**: файл `data/documents.txt` должен существовать и содержать хотя бы несколько строк текста. Если его нет, агент не запустится.
|
||||||
- An OpenAI API key.
|
|
||||||
- A SerpAPI key (free tier available).
|
|
||||||
|
|
||||||
## Setup
|
## Установка
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Clone the repository
|
# Создайте виртуальное окружение (рекомендуется)
|
||||||
git clone https://git.brojs.ru/kuzakhmetovartur/8.-samopisnyy-poiskovyy-agent-na-osnove-<repo>.git
|
python -m venv venv
|
||||||
cd <repo>
|
source venv/bin/activate # Windows: venv\Scripts\activate
|
||||||
|
|
||||||
# Create a virtual environment (optional but recommended)
|
# Установите зависимости
|
||||||
python -m venv .venv
|
pip install -U pip
|
||||||
source .venv/bin/activate # On Windows: .venv\\Scripts\\activate
|
pip install torch transformers fastapi uvicorn pydantic numpy
|
||||||
|
|
||||||
# Install dependencies
|
|
||||||
pip install -r requirements.txt
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Create a `.env` file in the project root with your credentials:
|
> Если у вас есть GPU, убедитесь, что установлена версия `torch` с поддержкой CUDA.
|
||||||
|
|
||||||
```
|
## Запуск
|
||||||
OPENAI_API_KEY=sk-...
|
|
||||||
SERPAPI_KEY=your-serpapi-key
|
|
||||||
```
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
Run the agent interactively:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python -m src.agent
|
python src/index.py
|
||||||
```
|
```
|
||||||
|
|
||||||
You will be prompted to enter a question. The agent will search the web and return a concise answer.
|
Сервер будет доступен по адресу `http://0.0.0.0:8000`.
|
||||||
|
|
||||||
## Example
|
## API
|
||||||
|
|
||||||
```
|
### POST `/search`
|
||||||
Enter your question: What is the capital of France?
|
|
||||||
Processing...
|
|
||||||
|
|
||||||
=== Answer ===
|
**Запрос**
|
||||||
The capital of France is Paris.
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"query": "пример запроса",
|
||||||
|
"top_k": 5
|
||||||
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
## Testing
|
- `query` – строка запроса.
|
||||||
|
- `top_k` – количество возвращаемых документов (по умолчанию 5).
|
||||||
|
|
||||||
The agent can be tested programmatically by importing `create_search_agent` from `src.agent` and calling `agent.run("your question")`.
|
**Ответ**
|
||||||
|
|
||||||
## License
|
```json
|
||||||
|
{
|
||||||
|
"results": [
|
||||||
|
{
|
||||||
|
"document": "текст найденного документа",
|
||||||
|
"score": 0.8723
|
||||||
|
},
|
||||||
|
...
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Пример использования
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST "http://0.0.0.0:8000/search" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"query":"машинное обучение", "top_k":3}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Как добавить документы
|
||||||
|
|
||||||
|
1. Откройте файл `data/documents.txt`.
|
||||||
|
2. Добавьте новые строки – каждая строка будет рассматриваться как отдельный документ.
|
||||||
|
3. Перезапустите сервер, чтобы обновления вступили в силу.
|
||||||
|
|
||||||
|
## Тесты
|
||||||
|
|
||||||
|
Тестов в проекте нет, но вы можете быстро проверить работу:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from src.index import SearchAgent
|
||||||
|
|
||||||
|
agent = SearchAgent()
|
||||||
|
print(agent.search("пример", top_k=3))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Лицензия
|
||||||
|
|
||||||
MIT License
|
MIT License
|
||||||
+96
-78
@@ -1,92 +1,110 @@
|
|||||||
#!/usr/bin/env python3
|
import os
|
||||||
"""
|
import json
|
||||||
Simple search agent implementation.
|
import numpy as np
|
||||||
|
import torch
|
||||||
This module provides a minimal command‑line interface that accepts a search
|
from transformers import AutoTokenizer, AutoModel
|
||||||
query and returns a list of dummy results. It is intentionally lightweight
|
from fastapi import FastAPI, HTTPException
|
||||||
to satisfy the assignment requirements while demonstrating a clear
|
from pydantic import BaseModel
|
||||||
structure that can be expanded in the future.
|
|
||||||
|
|
||||||
Author: Artur Kuzakhmetov
|
|
||||||
"""
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import sys
|
|
||||||
from typing import List
|
from typing import List
|
||||||
|
|
||||||
|
class SearchRequest(BaseModel):
|
||||||
|
query: str
|
||||||
|
top_k: int = 5
|
||||||
|
|
||||||
def search(query: str, limit: int = 5) -> List[str]:
|
class SearchResult(BaseModel):
|
||||||
|
document: str
|
||||||
|
score: float
|
||||||
|
|
||||||
|
class SearchResponse(BaseModel):
|
||||||
|
results: List[SearchResult]
|
||||||
|
|
||||||
|
class SearchAgent:
|
||||||
"""
|
"""
|
||||||
Perform a mock search for the given query.
|
A simple deep‑agent search engine that uses a transformer encoder
|
||||||
|
to embed documents and queries, then ranks documents by cosine
|
||||||
Parameters
|
similarity.
|
||||||
----------
|
|
||||||
query : str
|
|
||||||
The search string.
|
|
||||||
limit : int, optional
|
|
||||||
Maximum number of results to return. Defaults to 5.
|
|
||||||
|
|
||||||
Returns
|
|
||||||
-------
|
|
||||||
List[str]
|
|
||||||
A list of fake search results.
|
|
||||||
|
|
||||||
Notes
|
|
||||||
-----
|
|
||||||
This function does not perform real network requests. It simply
|
|
||||||
generates deterministic placeholder results so that the module can be
|
|
||||||
tested without external dependencies.
|
|
||||||
"""
|
"""
|
||||||
if not query:
|
def __init__(self,
|
||||||
raise ValueError("Query must not be empty")
|
data_path: str = "data/documents.txt",
|
||||||
|
model_name: str = "sentence-transformers/all-MiniLM-L6-v2"):
|
||||||
|
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
||||||
|
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||||
|
self.model = AutoModel.from_pretrained(model_name).to(self.device)
|
||||||
|
self.documents = self._load_documents(data_path)
|
||||||
|
self.embeddings = self._embed_documents(self.documents)
|
||||||
|
|
||||||
# Generate deterministic dummy results
|
def _load_documents(self, path: str) -> List[str]:
|
||||||
results = [f"{query} result {i+1}" for i in range(limit)]
|
if not os.path.exists(path):
|
||||||
|
raise FileNotFoundError(f"Data file not found: {path}")
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
docs = [line.strip() for line in f if line.strip()]
|
||||||
|
return docs
|
||||||
|
|
||||||
|
def _embed_documents(self, docs: List[str]) -> np.ndarray:
|
||||||
|
batch_size = 32
|
||||||
|
embeddings = []
|
||||||
|
for i in range(0, len(docs), batch_size):
|
||||||
|
batch = docs[i:i+batch_size]
|
||||||
|
inputs = self.tokenizer(batch,
|
||||||
|
padding=True,
|
||||||
|
truncation=True,
|
||||||
|
return_tensors="pt").to(self.device)
|
||||||
|
with torch.no_grad():
|
||||||
|
outputs = self.model(**inputs)
|
||||||
|
token_embeddings = outputs.last_hidden_state
|
||||||
|
attention_mask = inputs["attention_mask"].unsqueeze(-1)
|
||||||
|
sum_embeddings = torch.sum(token_embeddings * attention_mask, dim=1)
|
||||||
|
sum_mask = torch.clamp(attention_mask.sum(dim=1), min=1e-9)
|
||||||
|
batch_embeddings = sum_embeddings / sum_mask
|
||||||
|
embeddings.append(batch_embeddings.cpu().numpy())
|
||||||
|
return np.vstack(embeddings)
|
||||||
|
|
||||||
|
def _embed_query(self, query: str) -> np.ndarray:
|
||||||
|
inputs = self.tokenizer([query],
|
||||||
|
padding=True,
|
||||||
|
truncation=True,
|
||||||
|
return_tensors="pt").to(self.device)
|
||||||
|
with torch.no_grad():
|
||||||
|
outputs = self.model(**inputs)
|
||||||
|
token_embeddings = outputs.last_hidden_state
|
||||||
|
attention_mask = inputs["attention_mask"].unsqueeze(-1)
|
||||||
|
sum_embeddings = torch.sum(token_embeddings * attention_mask, dim=1)
|
||||||
|
sum_mask = torch.clamp(attention_mask.sum(dim=1), min=1e-9)
|
||||||
|
query_embedding = sum_embeddings / sum_mask
|
||||||
|
return query_embedding.cpu().numpy()
|
||||||
|
|
||||||
|
def search(self, query: str, top_k: int = 5) -> List[SearchResult]:
|
||||||
|
query_emb = self._embed_query(query)
|
||||||
|
dot = np.dot(self.embeddings, query_emb.T).squeeze()
|
||||||
|
norms = np.linalg.norm(self.embeddings, axis=1) * np.linalg.norm(query_emb)
|
||||||
|
similarities = dot / norms
|
||||||
|
top_indices = np.argsort(similarities)[::-1][:top_k]
|
||||||
|
results = [SearchResult(document=self.documents[idx],
|
||||||
|
score=float(similarities[idx]))
|
||||||
|
for idx in top_indices]
|
||||||
return results
|
return results
|
||||||
|
|
||||||
|
app = FastAPI(title="Deep Agent Search API")
|
||||||
|
|
||||||
def main(argv: List[str] | None = None) -> int:
|
# Instantiate the agent once at startup
|
||||||
|
agent = SearchAgent()
|
||||||
|
|
||||||
|
@app.post("/search", response_model=SearchResponse)
|
||||||
|
async def search_endpoint(request: SearchRequest):
|
||||||
"""
|
"""
|
||||||
Entry point for the command‑line interface.
|
Search endpoint that accepts a JSON payload:
|
||||||
|
{
|
||||||
Parameters
|
"query": "your search query",
|
||||||
----------
|
"top_k": 5
|
||||||
argv : List[str] | None
|
}
|
||||||
List of command‑line arguments. If None, sys.argv[1:] is used.
|
Returns the top_k most relevant documents.
|
||||||
|
|
||||||
Returns
|
|
||||||
-------
|
|
||||||
int
|
|
||||||
Exit code (0 for success, 1 for error).
|
|
||||||
"""
|
"""
|
||||||
parser = argparse.ArgumentParser(
|
|
||||||
description="Simple search agent – returns mock results for a query."
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"query",
|
|
||||||
type=str,
|
|
||||||
help="Search query string",
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"-n",
|
|
||||||
"--limit",
|
|
||||||
type=int,
|
|
||||||
default=5,
|
|
||||||
help="Number of results to return (default: 5)",
|
|
||||||
)
|
|
||||||
args = parser.parse_args(argv)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
results = search(args.query, args.limit)
|
results = agent.search(request.query, request.top_k)
|
||||||
except ValueError as exc:
|
return SearchResponse(results=results)
|
||||||
print(f"Error: {exc}", file=sys.stderr)
|
except Exception as e:
|
||||||
return 1
|
raise HTTPException(status_code=500, detail=str(e))
|
||||||
|
|
||||||
for idx, result in enumerate(results, start=1):
|
|
||||||
print(f"{idx}. {result}")
|
|
||||||
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
sys.exit(main())
|
import uvicorn
|
||||||
|
uvicorn.run(app, host="0.0.0.0", port=8000)
|
||||||
Reference in New Issue
Block a user