Human‑in‑the‑Loop Middleware Demo

This repository contains a minimal FastAPI application that demonstrates how to implement a Human‑in‑the‑Loop (HITL) mechanism using custom middleware.
The middleware allows you to pause the processing of an incoming request and resume it later, which is useful for scenarios where a human operator needs to review or approve data before the LLM generates a final response.

TL;DR – Send a request with X-HITL-Interrupt: true to pause.
Then send another request with X-HITL-Resume: <request-id> to resume processing.

📦 Installation

# 1️⃣ Clone the repo
git clone https://github.com/yourname/hitl-middleware-demo.git
cd hitl-middleware-demo

# 2️⃣ Create a virtual environment (recommended)
python -m venv .venv
source .venv/bin/activate   # On Windows: .\.venv\Scripts\activate

# 3️⃣ Install dependencies
pip install -r requirements.txt

Dependencies

fastapi – Web framework.

uvicorn[standard] – ASGI server.

langchain – LLM wrapper (OpenAI).

qdrant-client – Vector store client.

rich – Pretty console output.

If you don't have a Qdrant instance running locally, install it via Docker:

docker run -p 6333:6333 qdrant/qdrant

🚀 Running the Application

uvicorn solution:app --reload

The API will be available at http://127.0.0.1:8000.

Endpoints

Method	Path	Description
`POST /process`	Accepts a JSON body with a `text` field. The request is processed by the HITL middleware and forwarded to an OpenAI LLM via LangChain.

📄 Example Usage

Below are curl examples that illustrate how to interrupt and resume a request.

1️⃣ Send a request that will be interrupted

curl -X POST http://127.0.0.1:8000/process \
     -H "Content-Type: application/json" \
     -H "X-HITL-Interrupt: true" \
     -d '{"text":"Explain quantum entanglement."}'

Response (queued)

{
  "status": "queued",
  "request_id": "abcd1234"
}

The middleware stores the request in memory and returns a request_id that can be used to resume later.

2️⃣ Resume the queued request

curl -X POST http://127.0.0.1:8000/process \
     -H "Content-Type: application/json" \
     -H "X-HITL-Resume: abcd1234" \
     -d '{"text":"Explain quantum entanglement."}'

Response (processed)

{
  "status": "completed",
  "response": "Quantum entanglement is a physical phenomenon..."
}

The LLM processes the text and returns the answer.

📚 How It Works

Middleware (HumanInLoopMiddleware)
- Checks for X-HITL-Interrupt or X-HITL-Resume headers.
- If interrupted, stores the request body in a dictionary keyed by a generated UUID.
- If resumed, retrieves the stored body and forwards it to the downstream route.
Route (/process)
- Receives the text payload.
- Calls llm.invoke() from LangChain to generate a response.
- Returns the LLM output in JSON.
Qdrant
- The example includes an initialized Qdrant client, but it is not used in this minimal demo.
- In a real-world scenario you could store embeddings or metadata there.

📦 Project Structure

.
├── solution.py          # Main FastAPI app with HITL middleware
├── requirements.txt     # Python dependencies
└── README.md            # This file

🔧 Customization

LLM – Replace OpenAI(api_key="YOUR_OPENAI_API_KEY") with another provider supported by LangChain.
Storage – Swap the in‑memory dict for Redis, PostgreSQL, or any persistence layer to survive restarts.
Security – Add authentication/authorization headers before allowing resume operations.

🎯 Use Cases

Content moderation – Pause a request until a human moderator approves it.
Legal review – Let lawyers vet LLM outputs before they are sent to clients.
Data privacy – Inspect sensitive data for compliance before processing.

Happy hacking! 🚀

4.1 KiB Raw Blame History Unescape Escape