A language model is only as good as what it knows, and its training data froze months ago. Retrieval-augmented generation — RAG — fixes that by fetching relevant passages from your own documents at query time and handing them to the model alongside the question. This guide builds a complete, working RAG pipeline from open-source parts: sentence-transformer embeddings, Qdrant as the vector database, and a local LLM for the final answer. Nothing here phones home.
You’ll finish with two scripts: ingest.py, which chunks documents and stores them, and query.py, which answers questions with cited sources. It’s aimed at developers comfortable with Python and the command line.
Prerequisites
- OS: Linux or macOS. (Windows works via WSL2.)
- Python: 3.10 or newer, plus
pip. - RAM: 8 GB minimum; the embedding model is small (~90 MB) and runs on CPU fine.
- Docker: for running Qdrant. Alternatively, Qdrant offers a single binary download.
- A local LLM: Ollama running with any chat model pulled — the query script talks to it over HTTP.
Step 1: Set up the project and start Qdrant
mkdir mini-rag && cd mini-rag
python -m venv venv && source venv/bin/activate
pip install qdrant-client sentence-transformers pypdf requests langchain-text-splittersStart Qdrant in Docker:
docker run -d --name qdrant -p 6333:6333 qdrant/qdrantQdrant is a proper vector database written in Rust: it stores vectors, filters on metadata, and serves nearest-neighbour search. The dashboard at http://localhost:6333/dashboard is worth a look once data is in.
Step 2: Ingest documents — chunk, embed, store
RAG lives or dies on chunking. Whole documents are too coarse to retrieve precisely, so we split text into overlapping chunks, embed each chunk into a vector, and store the vectors. Create ingest.py:
from pathlib import Path
from langchain_text_splitters import RecursiveCharacterTextSplitter
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
from pypdf import PdfReader
# 1. Load documents (PDFs in ./docs, plus plain text)
texts = []
for pdf in Path("docs").glob("*.pdf"):
reader = PdfReader(str(pdf))
full = "\n".join(page.extract_text() or "" for page in reader.pages)
texts.append({"text": full, "source": pdf.name})
# 2. Chunk: ~500 chars with 50 chars of overlap so context survives splits
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = []
for doc in texts:
for piece in splitter.split_text(doc["text"]):
chunks.append({"text": piece, "source": doc["source"]})
# 3. Embed locally with a small, well-tested model (384-dim vectors)
embedder = SentenceTransformer("BAAI/bge-small-en-v1.5")
vectors = embedder.encode([c["text"] for c in chunks], show_progress_bar=True)
# 4. Store in Qdrant
client = QdrantClient(url="http://localhost:6333")
client.recreate_collection(
collection_name="knowledge",
vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)
client.upsert(
collection_name="knowledge",
points=[
PointStruct(id=i, vector=vectors[i].tolist(),
payload={"text": chunks[i]["text"], "source": chunks[i]["source"]})
for i in range(len(chunks))
],
)
print(f"Ingested {len(chunks)} chunks from {len(texts)} documents.")Drop a couple of PDFs into docs/ and run it:
mkdir docs # put your PDFs here
python ingest.py
# Ingested 214 chunks from 3 documents.Two details matter here. First, the embedding model’s dimension (384 for bge-small) must match the collection’s size — change one and you must change both. Second, recreate_collection wipes the collection, which is what you want while iterating but not in production.
Step 3: Retrieve and generate answers
Create query.py. It embeds the question, pulls the top chunks from Qdrant, stuffs them into a prompt, and asks the local LLM:
import sys, requests
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient
question = " ".join(sys.argv[1:]) or "What does the document say about refunds?"
client = QdrantClient(url="http://localhost:6333")
embedder = SentenceTransformer("BAAI/bge-small-en-v1.5")
qvec = embedder.encode(question).tolist()
hits = client.query_points(
collection_name="knowledge", query=qvec, limit=4
).points
context = "\n\n---\n\n".join(
f"[source: {h.payload['source']}]\n{h.payload['text']}" for h in hits
)
prompt = (
"Answer the question using ONLY the context below. "
"If the answer isn't in the context, say so.\n\n"
f"Context:\n{context}\n\nQuestion: {question}\nAnswer:"
)
r = requests.post(
"http://localhost:11434/api/generate",
json={"model": "llama3.1:8b", "prompt": prompt, "stream": False},
timeout=300,
)
print(r.json()["response"])
print("\nSources:", sorted({h.payload["source"] for h in hits}))python query.py "What is the refund policy?"
# ...an answer grounded in your PDFs, with the source filenames listed.The “answer only from context” instruction is the cheapest hallucination control you have. It’s not perfect, but it changes the failure mode from confident invention to an honest “I don’t know”.
Step 4: Sanity-check retrieval quality
Before trusting answers, check that retrieval itself works. Ask questions you know the answers to and inspect which chunks come back — print the raw hits in query.py temporarily. Common fixes when retrieval is poor: smaller chunks (300 chars) for factoid questions, larger chunks (800+) for summaries, and a better embedding model (bge-base or bge-large) at the cost of speed. Hybrid search (vectors plus keyword BM25) is the next upgrade when pure vector search misses exact terms like product codes.
One tuning detail worth knowing: the chunk_overlap of 50 characters exists so that a sentence straddling a chunk boundary still appears whole in at least one chunk. If your answers keep missing the second half of key sentences, raise the overlap to 100–150 before touching anything else. And keep a tiny eval set — ten questions with known answers over your own documents — that you re-run after every change to chunking, embedding model, or top-k. Retrieval tuning without a fixed test set is just vibes, and vibes don’t survive contact with new documents.
Going further: hybrid search and metadata filters
Pure vector search has a known blind spot: exact terms. Ask about “invoice INV-2024-118” and embeddings may rank a semantically similar but wrong invoice above the exact match, because vectors capture meaning, not strings. The standard fix is hybrid search — combine vector similarity with keyword matching (BM25) and merge the rankings. Qdrant supports this natively with sparse vectors alongside dense ones, though wiring it up is a project in itself.
The cheaper upgrade is metadata filtering. Because we stored source in each point’s payload, you can restrict search to specific documents: “answer only from the 2024 policy handbook” becomes a filter on source plus vector search within the survivors. In Qdrant you’d add a Filter with a FieldCondition on the payload key. Filters also enable the classic enterprise pattern — per-user or per-tenant document access — where each chunk carries an owner field and queries are scoped to it.
A third lever is re-ranking: retrieve 20 candidates with the fast embedding model, then score them with a heavier cross-encoder (like a ms-marco reranker) and keep the top 4. It roughly doubles retrieval latency but noticeably improves which chunks the generator sees. Add it when you’ve measured a retrieval problem, not before.
Where things go wrong
Qdrant dimension mismatch. VectorParams(size=...) must equal the embedding model’s output dimension. bge-small is 384, bge-base 768, bge-large 1024. The error message says this plainly — read it.
Empty or garbage PDF text. pypdf can’t read scanned PDFs (they’re images, not text). For those you need OCR first — or a layout-aware parser like Docling, which handles tables and scanned pages far better.
Answers ignore the context. Small local models sometimes do. Lower the temperature ("options": {"temperature": 0.1} in the Ollama request), shorten the context to the top 3 chunks, and make the system instruction blunter.
Slow first query. The embedding model and the LLM both load lazily; the first request pays the startup cost. Subsequent ones are fast.
Wrap-up
You now have the full RAG loop running locally: documents in, grounded answers out, with sources attached. From here, the usual production hardening applies — metadata filters in Qdrant, re-ranking the top hits, and evaluating answer quality on a fixed set of test questions. If your documents are messy PDFs rather than clean text, our document processing guide shows how to parse them properly before they ever reach this pipeline.
Featured image: rows of bookshelves in the Mary R. Schiff Library and Archives (Jason Zhang, via Wikimedia Commons — CC BY-SA 4.0).
