Privacy-First RAG: chatting with your local document repositories safely

Retrieval-Augmented Generation (RAG) is the gold standard for talking to your documentation. It queries files, extracts the most relevant segments and passes them to the LLM as reference context. But when you use cloud solutions, you upload financial reports, medical files or legal records to a third party.

In this tutorial, we will set up a 100% private RAG system. Your documents remain encrypted on your hard drive, analyzed by a local embedding model and answered by a local LLM orchestrated directly by Ollama.

The architecture of private RAG

A RAG pipeline consists of three core components, all of which can run locally:

1. Document Parser & Vector Database: Splices PDFs or text files into digestible chunks and indexes them using mathematical embeddings (using ChromaDB or LanceDB).
2. Embedding Model: A specialized, small model (like nomic-embed-text) served by Ollama, that converts text passages into vectors.
3. Inference LLM: A conversational model (like mistral) served by Ollama, that reads the retrieved context and answers your query.

Step-by-step setup with Ollama

Ollama exposes the /api/embeddings endpoint for vectorization and /api/chat for inference out of the box. Combined with a thin Python script and ChromaDB, you get a complete RAG pipeline that is scriptable and has no graphical dependency. Open WebUI can still be layered on top if you prefer a web interface, but it is no longer required.

1. Pull the models via Ollama

Open your command line and pull the embedding model and the inference LLM. Ollama will run them locally and serve them via its REST API:

ollama pull nomic-embed-text
ollama pull mistral

nomic-embed-text runs extremely fast on CPU or GPU. mistral answers questions based on the retrieved context.

2. Start the Ollama daemon

If Ollama is not already running, start it in headless mode. It will listen on http://localhost:11434:

ollama serve

3. Install ChromaDB for vector storage

In a Python environment, install ChromaDB. No network connection is required beyond the install:

pip install chromadb requests

4. Ingest and index your documents

The Python script below reads a folder of files, chunks them, generates embeddings via Ollama and stores them in ChromaDB. Everything stays on your machine:

import os
import requests
import chromadb

OLLAMA = "http://localhost:11434"
chroma = chromadb.PersistentClient(path="./chroma_db")
collection = chroma.get_or_create_collection("documents")

def embed(text):
    return requests.post(f"{OLLAMA}/api/embeddings", json={
        "model": "nomic-embed-text",
        "prompt": text,
    }).json()["embedding"]

for filename in os.listdir("./docs"):
    with open(f"./docs/{filename}") as f:
        content = f.read()
        chunks = [content[i:i+500] for i in range(0, len(content), 500)]
        for idx, chunk in enumerate(chunks):
            collection.add(ids=[f"{filename}-{idx}"], embeddings=[embed(chunk)], documents=[chunk])

5. Ask a question against your documents

Once the vector store is built, every question goes through three steps: embedding the question, retrieving the most similar chunks from ChromaDB, then sending them to the Ollama LLM as context:

import requests

def ask(question):
    q_emb = embed(question)
    results = collection.query(query_embeddings=[q_emb], n_results=3)
    context = "\n".join(results["documents"][0])
    response = requests.post(f"{OLLAMA}/api/chat", json={
        "model": "mistral",
        "messages": [{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}],
        "stream": False,
    })
    return response.json()["message"]["content"]

print(ask("What was the Q3 revenue?"))