MCP for local AI: give your model hands, not just a memory

If you've read our guide on privacy-first RAG, you already know how to make a local model aware of your documents — you find the relevant text and feed it into the prompt. That solves one problem: knowledge.

But that kind of knowledge is passive. A RAG-powered model answers from the passages you retrieved for it. It can't decide on its own to open a different file right now, run a fresh query against a live database, and act on the answer.

That's the gap the Model Context Protocol (MCP) fills. Where RAG gives your model a memory, MCP gives it hands — a standard way to reach out and use tools, read live data, and take actions. And despite being born in the cloud-AI world, MCP runs beautifully with a fully local model. No cloud, no API keys, nothing leaving your machine.

What MCP actually is

The official metaphor is the best one: MCP is a USB-C port for AI applications.

Before USB-C, every device needed its own special cable. Before MCP, every AI app that wanted to talk to your files, your database, or some API needed its own bespoke, hand-written integration — and none of them were reusable. MCP is the standard plug. Build a connector once, and any MCP-compatible AI can use it.

It's an open protocol (not owned by any single company) supported across a wide range of tools — Claude, ChatGPT, VS Code, Cursor, and, importantly for us, local-LLM setups. That's the whole point: build a tool once, and it works everywhere.

MCP vs RAG: knowing versus doing

This is the distinction that makes everything click, especially if you came here from the RAG guide. They are not competitors — they answer different questions.

	RAG	MCP
Answers the question	"What does my model know?"	"What can my model do?"
How it works	You inject relevant text into the prompt	The model calls tools and reads data on demand
The data	Retrieved passages from documents you've indexed	Live — a database query, a file read, an API call, right now
Can it act?	No. Read-only context.	Yes. It can write a file, run a query, trigger a workflow.

A simple way to hold both in your head: RAG is a library card; MCP is a set of car keys. One lets the model look things up. The other lets it go somewhere and do something. The most capable local setups use both — RAG for deep knowledge of your documents, MCP for live actions.

How MCP is built: host, client, server

MCP uses a clean client–server design with three roles. Once you see them, every MCP setup makes sense.

• MCP Host — the AI application you actually use. It runs (or talks to) the model and coordinates everything. Think of it as the cockpit.
• MCP Client — a connector inside the host. The host spins up one client per server it connects to, and each client holds a dedicated line to its server.
• MCP Server — a small program that exposes a capability: your filesystem, a database, a web search, a calendar. It can run locally on your machine or remotely.

So a single host might run three clients at once — one talking to a filesystem server, one to a database server, one to a search server — each over its own connection.

What a server exposes: the three primitives

An MCP server can offer three kinds of things, and it's worth knowing the names because you'll see them everywhere:

• Tools — executable functions the model can invoke to take action: read a file, run a query, call an API. This is the one that gives your model hands.
• Resources — data sources that provide context: a file's contents, a database record, an API response. Read-only information.
• Prompts — reusable templates that structure an interaction, like a pre-built system prompt or a set of few-shot examples.

Under the hood it all runs on JSON-RPC 2.0 — a simple, well-established message format. You almost never touch this directly (the tools handle it for you), but it's why MCP is so portable: the same messages work whether the server is a local script or a remote service.

The catch nobody tells you: Ollama is not an MCP client

Here's the single most important thing to understand before you try this — and the detail most tutorials get wrong.

Ollama does not speak MCP. Ollama is an inference server: it runs the model and exposes a chat API with tool calling support. That's it. It has no idea what an MCP server is.

So you can't "plug MCP into Ollama" directly. You need a piece in the middle — an MCP host/client that:

1. speaks MCP to your servers (filesystem, database, etc.), and
2. speaks Ollama's tool-calling API to your local model.

  Your tools          The bridge              Your local model
┌─────────────┐     ┌──────────────┐        ┌────────────────┐
│ MCP servers │◄───►│  MCP host /  │◄──────►│     Ollama     │
│ (files, DB) │ MCP │   client     │ tool-  │ (runs the LLM) │
└─────────────┘     └──────────────┘ calling└────────────────┘

The bridge translates between the two worlds. Once you internalize this, the whole landscape of "Ollama + MCP" projects suddenly makes sense — they're all bridges.

A concrete local example

Let's give a local model the ability to read your files — live, on demand, fully offline.

You need three things:

1. Ollama, running a tool-capable model:

   ollama pull qwen2.5
   

2. A bridge — an MCP client built for Ollama. The most purpose-built option is mcp-client-for-ollama (its command is ollmcp), a terminal app that connects local Ollama models to one or more MCP servers and supports all three primitives — tools, resources, and prompts — plus an agent mode.

3. An MCP server — for this example, the official filesystem server, which exposes your chosen folder as tools the model can use (list files, read contents, and so on).

You point the bridge at both: tell it which Ollama model to use, and which MCP server(s) to connect to. From that moment, when you ask "summarize the notes in my project folder," the model doesn't hallucinate — it calls the filesystem tool, reads the actual files, and answers from what it found. Every byte stays on your machine.

Don't like the terminal?

The bridge doesn't have to be a command-line tool. Two friendlier options:

• Open WebUI — a self-hosted web interface for local models, with native MCP support. A clean choice if you want a ChatGPT-style window instead of a terminal.
• LM Studio — the popular desktop GUI for local models also speaks MCP, so you can wire up servers from a graphical app.

Whichever you pick, the architecture from the section above is identical — only the cockpit changes.

Why this matters for sovereign AI

It's worth stepping back. MCP was designed for cloud assistants, but running it locally is where it gets interesting for privacy:

• A cloud assistant with MCP access to your files means your files are reachable from someone else's servers.
• A local model with MCP access to your files means a capable agent that reads your documents, queries your databases, and automates your tasks — and none of it ever leaves your machine.

That's the dream of local AI taken one step further. RAG made your model informed. MCP makes it useful. Together, on hardware you own, they turn a chatbot into a private assistant that can actually get things done.

Where to go next

• Not set up with Ollama yet? Start with the complete Ollama guide for tool-calling models.
• Want the knowledge side too? Revisit privacy-first RAG — pairing it with MCP is the most powerful local setup there is.
• Build a real agent — once your model can call tools, you're one step from a true local AI agent that reads, writes, and automates on your behalf.

Knowledge plus action, running entirely on your own machine. No subscription, no cloud, no one looking over your shoulder. That's the whole idea.