If you already have Ollama installed and you've chatted with your first model. That's the easy part — and honestly, it's where most tutorials stop.

But running ollama run mistral once is not the same as using Ollama. The tool quietly does a lot more: it manages a local library of models, lets you reshape any model's personality and settings without retraining anything and — the part most people miss — it runs a full HTTP server on your machine that your own scripts and apps can talk to.

This guide picks up where the install leaves off. No cloud, no API keys, no per-token billing. Everything below runs on your hardware and stays there.

Ollama is a server, not just a chat box

The single most useful thing to understand about Ollama: when it's running, there's a background server listening on your machine — by default at http://localhost:11434.

The ollama run command you typed is just a client talking to that server. Your own code can talk to the exact same server. That's the whole reason Ollama is more than a toy: it turns your computer into a private, offline AI backend.

You can start the server explicitly with:

ollama serve

On most installs (the desktop app on macOS/Windows, or the systemd service on GNU/Linux) the server is already running in the background, so you rarely call serve by hand. But it's good to know it's there — every command and every API call below depends on it.

Managing your model library

Models are downloaded once and cached locally. Over time you'll accumulate several and you'll want to keep the collection under control. Here are the commands that matter.

Download a model

ollama pull [model]

This fetches the model from the registry and stores it locally. After this, the model works fully offline — no internet required to run it.

See what you have

ollama ls

(ollama list does the same thing.) This shows every model on disk, its size and when you last modified it. Useful when you're wondering where your disk space went — these files are large.

See what's running right now

ollama ps

Models get loaded into RAM/VRAM when you use them and stay resident for a few minutes in case you ask again. ps shows what's currently loaded and how much memory it's eating. If your machine feels sluggish, this tells you whether a model is still parked in memory.

Stop a running model

ollama stop [model]

Unloads the model from memory immediately instead of waiting for it to time out. Handy when you want your RAM/VRAM back now.

Delete a model

ollama rm [model]

Removes it from disk entirely. You'll re-download it if you want it again.

Inspect a model

ollama show [model]

Prints the model's details — its parameters, the template it uses, its license. This is the quickest way to peek under the hood of any model you've pulled.

💾 Where do the files actually are stored?

Ollama stores models outside your normal Downloads folder, which surprises people. By default:

macOS: ~/.ollama/models
GNU/Linux: ~/.ollama/models (or /usr/share/ollama/.ollama/models when Ollama runs as a system service)
Windows: %USERPROFILE%\.ollama\models

If your system drive is small and you'd rather keep models on another disk, set the OLLAMA_MODELS environment variable to a new path (more on env vars below).

Customizing a model with a Modelfile

Here's where Ollama gets genuinely powerful — and where most people never venture.

A Modelfile is a small text recipe that takes an existing model and tweaks it: giving it a permanent personality, locking in settings like creativity level or pinning a larger context window. You're not retraining anything. You're wrapping a base model in a configuration that travels with it.

Create a plain text file named Modelfile (no extension):

# Start from an existing model
FROM mistral

# Make it less random and more focused (0 = deterministic, 1 = creative)
PARAMETER temperature 0.3

# Give it a bigger memory of the conversation
PARAMETER num_ctx 8192

# Define its permanent personality and rules
SYSTEM """
You are a concise technical writing assistant.
Answer in plain English. Prefer short sentences.
Never apologize. If you don't know, say so directly.
"""

Then build your custom model from it:

ollama create writer -f Modelfile

Now you have a new model called writer that behaves exactly that way every time:

ollama run writer

The instructions you'll actually use

Instruction	What it does
`FROM`	Required. The base model to build on (e.g. `mistral`, or a path to a `.gguf` file).
`SYSTEM`	The permanent system message — the model's personality and rules.
`PARAMETER`	Sets a runtime setting (see the table below). You can have many.
`TEMPLATE`	Advanced: overrides how prompts are formatted before reaching the model. Most people never touch this.
`MESSAGE`	Seeds a fake conversation history to steer the model's tone.
`ADAPTER`	Attaches a LoRA fine-tuning adapter. Advanced.
`LICENSE`	Records the license text for the model.

Parameters worth knowing

Parameter	Effect	Typical value
`temperature`	Randomness. Low = focused and repeatable, high = creative and surprising.	`0.3`–`1.0`
`num_ctx`	Context window — how much text the model can "remember" at once. Bigger uses more memory.	`4096`–`8192`
`num_predict`	Maximum number of tokens to generate in a reply.	`256`+
`top_k` / `top_p`	Fine control over word-choice diversity.	`40` / `0.9`
`repeat_penalty`	Discourages the model from looping the same phrases.	`1.1`
`seed`	Fix this and (with low temperature) you get reproducible answers — great for testing.	any integer
`stop`	A string that, when generated, halts the response.	e.g. `"User:"`

One detail that trips people up: the Modelfile is not case sensitive and instructions can appear in any order. The convention is uppercase keywords (FROM, SYSTEM) purely for readability.

Getting the most out of your model

Installing a model gets you a generic setup. Tuning it gets you one that fits your machine and your task. There are three knobs that matter — and the good news is you've already met the controls: they're the same PARAMETER lines from the Modelfile above, plus one decision you make before you even download.

Think of it in this order: pick the right model → make it run smoothly → make it answer well → make it behave the way you want.

Pick the right model first

No amount of tuning fixes a model that's too big for your hardware. Before anything else, match the model to your machine.

The variable that decides everything is how much memory the model needs versus how much you have (VRAM if you have a GPU, otherwise system RAM). A model that fits in your GPU's VRAM runs fast; one that spills over into system RAM runs much slower; one that doesn't fit at all won't load.

Two levers control a model's footprint:

Parameter count — the 7b, 13b, 70b in a model's name (billions of parameters). Bigger is generally smarter but heavier.
Quantization — how tightly the model's numbers are compressed, shown as tags like q4_0, q4_K_M, q8_0. Lower numbers = smaller and faster, at a small cost in quality.

A practical rule of thumb:

Your memory (VRAM or RAM)	Comfortable model size
8 GB	7B at q4 (e.g. `mistral`, `llama3.2`)
16 GB	13B at q4, or 7B at q8 for higher quality
24 GB+	30B+ models, or smaller models with lots of context

For most people on a typical laptop or gaming PC, a 7B model at q4 (the default Ollama pulls for many models) is the sweet spot: fast, capable and it fits. Our Hardware guide goes deeper on matching models to your exact setup.

🧠 Quick mental model of quantization

q4 is like a well-compressed photo — slightly softer if you pixel-peep, but indistinguishable for everyday use and a quarter of the size. Start at q4; only move up to q8 if you can spare the memory and notice the quality gap.

Make it run smoothly (speed & memory)

Once a model fits, these settings squeeze out speed and keep memory in check:

num_ctx — the biggest memory lever. The context window grows memory use fast. If a model is borderline on your hardware, lowering num_ctx (say from 8192 to 4096) is often what makes it fit and run smoothly. Only raise it when you genuinely need the model to remember long documents.
num_predict — cap the reply length. Limiting how many tokens the model can generate stops it from rambling and keeps responses snappy.
OLLAMA_KEEP_ALIVE — avoid reload lag. By default a model unloads from memory after a few minutes idle, so your next prompt pays a reload delay. If you're using a model on and off all day, set this longer (e.g. 30m) to keep it warm. Working with several models on tight memory? Set it short (or 0) so each unloads promptly and frees room for the next.
Let the GPU do the work. Ollama offloads to your GPU automatically when it can. Run ollama ps while a model is loaded — it tells you whether the model sits in GPU memory (fast) or has spilled to CPU (slow). If it spilled, that's your signal to drop to a smaller model or a lower quantization.

A measurable starting point: run a prompt, watch the tokens-per-second Ollama can report, then change one setting and compare. Tuning blind is guesswork; tuning one knob at a time is engineering.

Make it answer well (quality & consistency)

These settings shape what the model says, not how fast. They're the difference between a model that hallucinates wildly and one you can rely on. Set them as PARAMETER lines in a Modelfile so they stick:

Parameter	Turn it down for…	Turn it up for…
`temperature`	Facts, code, data extraction — anything where you want the same sensible answer every time (try `0.1`–`0.3`)	Brainstorming, fiction, creative writing where you want variety (try `0.8`–`1.0`)
`top_p`	Tighter, safer word choices (`0.5`)	More adventurous phrasing (`0.95`)
`repeat_penalty`	—	Bump to `1.2`–`1.3` if the model keeps looping the same phrase

Two combinations cover most real needs:

# A precise, repeatable assistant — coding, analysis, factual Q&A
PARAMETER temperature 0.2
PARAMETER top_p 0.5

# A creative writing partner — varied, surprising output
PARAMETER temperature 0.9
PARAMETER top_p 0.95

And when you're testing — comparing two prompts or two settings — pin the randomness so the only thing changing is what you meant to change:

PARAMETER seed 42
PARAMETER temperature 0

With a fixed seed and temperature 0, the same prompt gives the same answer every time. That's how you tell whether your prompt improved, not just whether you got lucky.

Make it behave the way you want (persona & rules)

Speed and sampling settings shape the form of answers. To shape their content and tone — to make a model consistently act as a code reviewer, a translator, a terse assistant — use the SYSTEM instruction. This is the single highest-leverage tuning you can do and it costs nothing in performance.

A vague system prompt produces a vague assistant. Be specific about role, format and boundaries:

FROM mistral

SYSTEM """
You are a senior Python code reviewer.
For each snippet, reply with exactly three sections:
  1. Bugs — concrete problems, or "none".
  2. Risks — things that could break later.
  3. One suggestion — the single highest-impact improvement.
Be blunt. No preamble, no praise. If the code is fine, say so in one line.
"""

You can go a step further and show the model the behavior you want using MESSAGE, which seeds an example exchange into its memory:

MESSAGE user "def add(a,b): return a-b"
MESSAGE assistant "1. Bugs — subtraction where addition is intended. 2. Risks — silent wrong results, no type checks. 3. Suggestion — return a + b and add type hints."

Now the model has a concrete template to imitate. Combined, SYSTEM + a couple of MESSAGE examples turn a generic model into a specialist — no fine-tuning, no GPU hours, just a text file.

🎯 Putting it together

The real workflow is to bundle all four layers into one Modelfile — your chosen base model, the speed settings that fit your hardware, the sampling settings for your task and the persona. Build it once with ollama create and you have a purpose-built model you can run anywhere with a single command.

Calling Ollama from your own code (the REST API)

This is the feature that turns Ollama from a chat tool into a building block. While it's running, the server exposes an HTTP API at http://localhost:11434. Any language that can make an HTTP request can drive a local LLM through it.

No API key. No rate limit. No data leaving your machine.

Generate a single response — `/api/generate`

The simplest endpoint. Send a prompt, get a completion.

curl http://localhost:11434/api/generate -d '{
  "model": "mistral",
  "prompt": "Explain what a context window is in one sentence.",
  "stream": false
}'

Setting "stream": false makes Ollama return one complete JSON object. Leave it out (or set true) and the server streams the answer back token by token — which is how chat UIs show text appearing live.

Have a conversation — `/api/chat`

For multi-turn chat, /api/chat takes a list of messages with roles, just like the major cloud APIs:

curl http://localhost:11434/api/chat -d '{
  "model": "mistral",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "What is local AI?" }
  ],
  "stream": false
}'

Because the message format mirrors the OpenAI-style schema, a lot of existing tools and libraries can be pointed at Ollama with little more than a URL change.

Other endpoints you'll meet

Endpoint	Method	What it's for
`/api/generate`	POST	One-shot text completion.
`/api/chat`	POST	Multi-turn conversation with message history.
`/api/tags`	GET	List the models installed locally (the API version of `ollama ls`).
`/api/show`	POST	Get a model's details and parameters.
`/api/pull`	POST	Download a model programmatically.
`/api/ps`	GET	List models currently loaded in memory.
`/api/embed`	POST	Turn text into embedding vectors — the foundation of local search and personal knowledge bases.

That last one, /api/embed, is quietly important: it's what powers private document search (RAG) without ever sending your files to a cloud service.

Useful environment variables

Ollama is configured through environment variables. A few are worth knowing:

Variable	Purpose
`OLLAMA_HOST`	Change the address/port the server binds to. Set it to `0.0.0.0:11434` to let other machines on your network reach your Ollama server.
`OLLAMA_MODELS`	Move your model storage to a different folder or drive.
`OLLAMA_KEEP_ALIVE`	How long a model stays loaded in memory after use (default is a few minutes). Set it longer to avoid reload delays, or `0` to unload immediately.

🔐 A note on OLLAMA_HOST

Binding to 0.0.0.0 exposes the server to your whole local network. That's great for using your beefy desktop's GPU from a laptop — but only do it on a network you trust. The whole point of local AI is that you control who can reach it.

How you set these depends on your OS and whether Ollama runs as an app or a service. Run ollama serve --help to see the variables Ollama currently recognizes on your version.

You now have the full picture: Ollama is a local model manager, a customization tool and a private AI server rolled into one.

Everything you've set up here runs entirely on your hardware. No subscription, no cloud, no one watching. That's the point.

The Local AI experts Wiki

No articles found