A sovereign medical RAG assistant that cites its source or says "I don't know"
A retrieval-augmented assistant for clinical guidelines that runs entirely on the organization's own hardware: local language model, local embeddings, local vector store, no cloud API. Each answer cites its document and page, and a question outside the corpus gets a plain "I don't know" instead of an invented answer.
The problem
Hospitals hold their protocols and guidelines in long PDFs that clinicians cannot search quickly, and generic AI assistants are not an option: the documents cannot leave the building, an answer without its source cannot be checked, and an assistant that invents a dosage is worse than no assistant. The goal was to find out what it really takes to build one that a hospital security team could approve, not just one that demos well.
The system at a glance
Ingest
Official guidelines chunked and embedded on the machine
Retrieve
Qdrant vector search, about 75 to 110 ms
Gate
Nothing relevant enough: "I don't know", the model is never called
Generate
Mistral 7B through Ollama, on local hardware
Cite
Every answer names its document and page
Approach
Hadirou built the whole chain, from model and embeddings to vector store, API, interface and deployment, then spent most of the effort trying to break it: reading the code critically, measuring, and asking questions the system should not be able to answer. Retrieval decides before generation, so the system refuses before the language model is even called.
- Answers grounded in a corpus of seven official French publications (HAS and SFMU)
- Document and page cited for every answer
- Relevance threshold calibrated on the corpus to switch from answering to refusing
- Answer language set by configuration, not hard-coded
Architecture
A Next.js interface talks only to its own origin; a server-side relay adds the API key and forwards to a FastAPI back end, which queries Qdrant for vectors and Ollama for generation. The multilingual embedding model ships inside the Docker image, because a hospital network may have no route to the internet and a system that needs one to start is not sovereign.
- Next.js front end with a server-side relay, so no key reaches the browser
- FastAPI back end, answers streamed token by token
- Qdrant vector store and Ollama for local generation
- Separate liveness and readiness probes, so a database outage does not restart healthy pods
- Generation concurrency capped, since CPU generations slow each other down
Deployment and operations
The stack runs end to end with Docker Compose. Production mode refuses to start with a weak API key or an open CORS policy. Logs carry a request id and never contain document content, and Prometheus metrics track latency, model errors and the share of unanswered questions. Reference Kubernetes manifests are included; they render with kustomize but have not been applied to a live cluster yet, and that is stated in the repository.
- Docker Compose, non-root containers, security headers and a CSP with no external origin
- Prometheus metrics, including the share of questions the corpus cannot answer
- CI: lint, tests, both images built, and a check that no key appears in the browser bundle
- Reference Kubernetes manifests, not yet run on a cluster
Business value
It shows a regulated organization what a sovereign document assistant involves before it commits: the measured performance, the security choices, and an honest list of where it fails. Nothing in the design is specific to medicine. Documents that cannot leave the building, answers that must cite their source and a system that can say "I don't know" describe most regulated document work.
Lessons learned
A similarity threshold is a first filter, not a guard rail: one out-of-scope question scored 0.5873, above a legitimate one at 0.5591, so a cross-encoder reranker is the next step.
Measuring finds what a demo hides: moving from 61 synthetic chunks to the 992 of the real corpus broke two demo questions, and installing the CPU build of PyTorch cut the image from 9.67 GB to 2.94 GB.
Re-indexing a revised protocol must purge the old version, or the assistant can quote a superseded dosage.