169 lines
4.2 KiB
Markdown
169 lines
4.2 KiB
Markdown
# Human‑in‑the‑Loop Middleware
|
||
|
||
Human‑in‑the‑Loop (HITL) middleware is a lightweight Python framework that lets you build conversational agents on top of **Qdrant** vector store and **Ollama** LLMs.
|
||
The project demonstrates how to:
|
||
|
||
* Store and retrieve documents with Qdrant.
|
||
* Embed text using the `nomic-embed-text` model from Ollama.
|
||
* Generate responses with the `llama3` chat model via LangChain.
|
||
* Wrap everything in a simple HTTP client that can be used by external services or UI front‑ends.
|
||
|
||
The core logic lives in two files:
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| **agent.py** | Implements the LangChain pipeline: embedding → vector search → LLM generation. |
|
||
| **client.py** | Exposes a minimal FastAPI server that accepts user queries and returns responses from `agent.py`. |
|
||
|
||
---
|
||
|
||
## 📦 Prerequisites
|
||
|
||
| Component | Minimum Version | Notes |
|
||
|-----------|-----------------|-------|
|
||
| Python | 3.10+ | Tested on 3.12 |
|
||
| pip | – | Use the system package manager or `pipx` |
|
||
| **Ollama** | Latest | Install from https://ollama.ai/ |
|
||
| **Qdrant** | 1.7+ | Run locally (`docker run -p 6333:6333 qdrant/qdrant`) or use a managed instance |
|
||
|
||
> **Important:**
|
||
> * The Ollama image must expose the `llama3` and `nomic-embed-text` models.
|
||
> ```bash
|
||
> ollama pull llama3
|
||
> ollama pull nomic-embed-text
|
||
> ```
|
||
> * Qdrant should be reachable at `http://localhost:6333` (or set via the `QDRANT_URL` env var).
|
||
|
||
---
|
||
|
||
## ⚙️ Installation
|
||
|
||
```bash
|
||
# Clone the repo
|
||
git clone https://github.com/your-org/hitl-middleware.git
|
||
cd hitl-middleware
|
||
|
||
# Create a virtual environment (optional but recommended)
|
||
python -m venv .venv
|
||
source .venv/bin/activate # Windows: .venv\Scripts\activate
|
||
|
||
# Install dependencies
|
||
pip install -r requirements.txt
|
||
```
|
||
|
||
`requirements.txt` contains:
|
||
|
||
```text
|
||
langchain==0.2.*
|
||
langchain-ollama==0.1.*
|
||
langchain-qdrant==0.1.*
|
||
fastapi==0.* # for client.py
|
||
uvicorn==0.* # ASGI server
|
||
python-dotenv==1.*
|
||
```
|
||
|
||
---
|
||
|
||
## 🚀 Running the Project
|
||
|
||
### 1️⃣ Start Qdrant (if not already running)
|
||
|
||
```bash
|
||
docker run -d --name qdrant \
|
||
-p 6333:6333 \
|
||
qdrant/qdrant
|
||
```
|
||
|
||
> Make sure the container is healthy before proceeding.
|
||
|
||
### 2️⃣ Run the Agent
|
||
|
||
The agent can be executed as a script or imported into other code.
|
||
It will automatically load embeddings, connect to Qdrant, and expose a `process_query` function.
|
||
|
||
```bash
|
||
python agent.py
|
||
```
|
||
|
||
> The script prints a simple “Agent ready” message and waits for input if run directly.
|
||
|
||
### 3️⃣ Run the Client (FastAPI)
|
||
|
||
The client exposes an HTTP endpoint `/query`.
|
||
It forwards incoming requests to the agent and returns the LLM response.
|
||
|
||
```bash
|
||
uvicorn client:app --host 0.0.0.0 --port 8000
|
||
```
|
||
|
||
You should see:
|
||
|
||
```
|
||
INFO: Started server process [12345]
|
||
...
|
||
INFO: Application startup complete.
|
||
```
|
||
|
||
Now you can send requests to `http://localhost:8000/query`.
|
||
|
||
---
|
||
|
||
## 📄 Example Usage
|
||
|
||
### Using the HTTP API
|
||
|
||
```bash
|
||
curl -X POST http://localhost:8000/query \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"question": "What is the capital of France?"}'
|
||
```
|
||
|
||
**Response**
|
||
|
||
```json
|
||
{
|
||
"answer": "The capital of France is Paris."
|
||
}
|
||
```
|
||
|
||
### Using the Agent Directly (Python)
|
||
|
||
```python
|
||
from agent import process_query
|
||
|
||
response = process_query("Explain quantum computing in simple terms.")
|
||
print(response)
|
||
# Output: "Quantum computing uses qubits..."
|
||
```
|
||
|
||
---
|
||
|
||
## 🔧 Configuration
|
||
|
||
All configuration values can be overridden via environment variables or a `.env` file placed at the project root.
|
||
|
||
| Variable | Default | Description |
|
||
|----------|---------|-------------|
|
||
| `QDRANT_URL` | `http://localhost:6333` | Qdrant endpoint |
|
||
| `OLLAMA_HOST` | `http://localhost:11434` | Ollama API host |
|
||
| `LLM_MODEL` | `llama3` | Chat model name |
|
||
| `EMBEDDING_MODEL` | `nomic-embed-text` | Embedding model name |
|
||
|
||
Example `.env`:
|
||
|
||
```dotenv
|
||
QDRANT_URL=http://qdrant:6333
|
||
OLLAMA_HOST=http://ollama:11434
|
||
LLM_MODEL=llama3
|
||
EMBEDDING_MODEL=nomic-embed-text
|
||
```
|
||
|
||
---
|
||
|
||
## 📚 Further Reading
|
||
|
||
* [LangChain Docs](https://langchain.com/)
|
||
* [Ollama Quickstart](https://github.com/ollama/ollama)
|
||
* [Qdrant Documentation](https://qdrant.tech/documentation/)
|
||
|
||
Feel free to extend the middleware with custom prompts, retrieval strategies, or additional LLMs. Happy hacking! |