Obsidian + local LLM: your notes as the model's external memory

A context window is the smallest, most expensive memory a computer has had in decades. A 32k-token window holds roughly 25,000 words — about fifty pages. Everything the model knew before that and everything it will need next, has to live somewhere else.

That somewhere is usually a database you can't read. It doesn't have to be. It can be a folder of Markdown files you own, edit by hand and keep in git.

Andrej Karpathy put the analogy in the clearest possible terms when he described using LLMs to build personal knowledge bases:

"Obsidian is the IDE, the LLM is the programmer, the wiki is the codebase."

It's a good frame and it maps onto something concrete: the model's context window is RAM, your vault is the disk and deciding what to write down is paging. This guide follows that idea to its practical limit on hardware you own — what works locally today, what works well and the one layer where a frontier model still wins.

Why Markdown is the format a model digests best

This part is almost too convenient. The format text-editor purists settled on twenty years ago happens to be the ideal substrate for a language model:

• It's plain text. No extraction step, no parser, no lossy conversion. The bytes on disk are the tokens the model reads.
• Headings are a free outline. ## boundaries are semantic boundaries, which means you can chunk a document without guessing where the ideas end.
• Wikilinks are a graph, written by a human. [[some-note]] is an explicit statement that two ideas are related — a signal no embedding model can reconstruct reliably.
• Frontmatter is structured metadata sitting right next to the prose, so filtering by date, tag or project needs no separate index.
• It diffs. When a model rewrites a note, git diff shows you exactly what it changed. Try auditing that in a vector store.

A vault is therefore not "documents you feed to an AI." It is a knowledge base that happens to be readable by both of you.

The three layers, from cheapest to hardest

Karpathy's framing describes an agent that maintains a wiki. That's the top of a ladder, not the first rung. Local models handle the lower rungs comfortably today; be honest with yourself about the top one.

Layer 1 — Semantic search over your own notes

The goal: ask a question in your own words and get the right note back, even when you don't remember its title or the words you used.

This runs entirely on your machine and it's fast. Measured on a real vault of 383 notes / 2.4 MB of Markdown, with nomic-embed-text on Ollama:

Step	Result
Chunks produced	2,218
Full index build	18.7 s (119 chunks/s)
Query, model warm	~150 ms
Query, first call (cold model)	~4 s
Index size on disk	a few MB of float vectors

Eighteen seconds to make three years of notes searchable by meaning. Re-indexing after edits is incremental, so in practice you pay it once.

The honest caveat: retrieval quality is decent, not magic. In the same test, asking "which local model should I pick for my RAM?" returned a note about RAM budgets in embedded firmware — topically adjacent, completely useless as an answer. Embeddings match aboutness, not truth. When your vault doesn't contain the answer, a similarity search will still hand you its three closest guesses with no signal that it failed. Treat the results as candidates to read, not as answers.

Layer 2 — Reading, summarising and drafting

Once retrieval works, you can put the retrieved passages in front of a local model and ask real questions: summarise this cluster of notes, draft a synthesis page, find contradictions between two documents.

This is where model choice starts to matter and where the honest advice is to match the model to the job rather than to the benchmark leaderboard:

• 8–16 GB RAM — a 7–8B model (Qwen 3 8B, Gemma 12B quantised). Good at summarising a passage you hand it. Don't expect it to hold your whole vault's structure in mind.
• 24–32 GB — a 24–32B model. This is the sweet spot for drafting and reorganising notes.
• 48 GB+ — a 70B-class model, or the larger Qwen/Gemma variants. Now the summaries stop feeling like summaries.

For note work specifically, prioritise context length over raw parameter count. A 12B model with a 128k window is more useful for "read these nine notes and reconcile them" than a 32B model capped at 8k. Ollama exposes this per model; check it before blaming the model's intelligence.

Layer 3 — An agent that maintains the wiki

This is Karpathy's actual proposition and it's a different kind of task: ingest a source, decide which existing pages it affects, rewrite each of them, create the missing concept pages, fix the cross-links — one input touching a dozen files in a single pass.

This is the layer where local models still fall short and pretending otherwise wastes your afternoon. It isn't about prose quality. It's that the job is agentic: many sequential tool calls, each depending on the last, with no human to catch a wrong turn at step four. A model that's 95% reliable per step is 54% reliable over twelve steps. Frontier models are not magically better writers here — they're better at not derailing.

If you want that layer today, it means a frontier model with filesystem access. That's a real trade-off, not a footnote: an agent that rewrites your vault has read everything in it. Which is exactly the reasoning in local first, cloud as a fallback — the answer isn't "never use the cloud", it's knowing which of your notes you'd rather it never saw.

A workable middle path: let a local model handle ingestion and first-pass summaries (the high-volume, low-stakes work) and reserve the frontier model for restructuring passes you actually review.

Setting it up, from zero

1. Ollama and the two models you need

curl -fsSL https://ollama.com/install.sh | sh

# The embedding model — small, fast, does the retrieval
ollama pull nomic-embed-text

# The generation model — pick per the RAM table above
ollama pull qwen3:8b

Two distinct models, two distinct jobs. The embedding model turns text into vectors and is never asked to write anything; it's 270 MB and runs on a CPU. Don't try to embed with a chat model — most refuse outright and the ones that comply are worse at it.

Verify the embedding endpoint before going further:

curl -s http://localhost:11434/api/embed \
  -d '{"model":"nomic-embed-text","input":"hello"}' | head -c 80

You want a JSON array of floats back. If you get This server does not support embeddings, your Ollama server was started with embeddings disabled — restart it plainly with ollama serve.

2. Wire it into Obsidian

Two community plugins do the job and they answer different needs:

• Copilot for Obsidian — a chat panel beside your notes, with the vault as context. Point it at Ollama: set the provider to a custom OpenAI-compatible endpoint at http://localhost:11434/v1, then name your chat model and nomic-embed-text as the embedding model. This is the one to install first.
• Smart Connections — no chat, just a "related notes" panel that surfaces links you never made. Lower effort and honestly the one that changes daily habits more.

Both index locally and both talk to Ollama without an API key. Install from Settings → Community plugins and expect the first full index to take a minute or two on a large vault.

3. Give the vault a shape the model can navigate

This matters more than the plugin choice. A model reading a flat pile of 400 notes performs about as well as you would.

The structure that holds up in practice has three layers, mirroring the one Karpathy describes:

sources/     raw captured material — never edited by hand
notes/       your own writing, one idea per file
index.md     the entry point: what lives where and why

Then two rules that do the real work:

Write the index by hand. A model can read 400 files, but it costs it (and you) far more than reading one file that says where to look. This is the single highest-return thing in the whole setup — an entry-point note listing what each area contains, maintained by you.

Write down the why, never the what. If a fact is recoverable from the source material, don't copy it into a note; the source is the truth and it will drift. Constraints, rationale, what you rejected and why, the thing that bit you last March — none of that is derivable from anywhere and it's what future-you will actually search for.

The counterintuitive part: less context, better answers

Here's the finding that surprised me most, from running this setup daily.

The instinct with a knowledge base is to load it: inject a summary of recent activity at every session start, keep a running recap, front-load everything the model might need. It feels like helping.

It isn't. On the vault measured above, an automatic chronological recap was being injected into every single session — a few thousand tokens of "here's what happened lately," every time. Removing it entirely made the setup better and not by a small margin. Two effects compounded:

1. The obvious one. Thousands of tokens per session, freed for actual work.
2. The one I didn't predict. With the automatic recap gone, the reflex changed — to reading the index and the relevant note, on demand. The passive recap had been a crutch that answered "where were we?" adequately enough that nobody reached for the source built to answer it properly.

Deleting a mediocre solution beat adding a good one. You don't have to train the better habit if you stop bypassing it.

The general rule this points at: retrieved-on-demand beats always-injected. Context spent before you know what the question is, is context spent on a guess. A vault with a good index is a system for spending it after.

What this actually buys you

Strip away the tooling and the change is this: your notes stop being a place you file things and become a place you ask things.

Not because the model got smart about your life — it didn't and the retrieval will hand you a firmware note when you ask about RAM. But because finding stopped depending on remembering. A note you wrote in 2024 and forgot entirely surfaces because it's about what you're asking, not because you recalled its title.

The parts that run locally — search, summarising, drafting — run on hardware you own, over files you own, with no subscription and nothing leaving the machine. The part that doesn't yet run locally is the fully autonomous librarian and that gap will close. Meanwhile the vault you build for a frontier model today is the same vault a local model will read tomorrow, because it's just Markdown in a folder. That's the quiet advantage of a boring format: it outlives the tool you chose this year.