Tools matter more than models: the lesson from Mozilla's open AI report

Mozilla published its "State of Open Source AI" report, aggregating benchmarks, external studies and a SlashData survey of 1,410 developers (PDF). The headline finding is remarkable: open-weight models have nearly caught up with proprietary ones. But what makes this report truly fascinating lies elsewhere. If the performance gap between the two families of models has never been this small, it is not model power that decides whether an AI project succeeds. It is everything around it: the harness, the tools, the integration and how you actually use it.

This article draws on Mickael Bazoge's analysis on Next, which details the report's data and the geopolitical context.

Open models stick with the leading pack

The numbers are unambiguous. On the Artificial Analysis Intelligence index, the four top-performing models are proprietary and American: Claude Opus 5 and Fable 5, GPT-5.6 Sol and Grok 4.6, scoring between 61 and 63. Right behind them sit four open models, all Chinese: Kimi K3, GLM-5.3, Qwen3.8 Max and GLM-5.3 Flash, between 57 and 60. The gap is measured in benchmark points, not in real-world capability.

According to METR and Epoch studies cited by Mozilla, the best closed models complete long tasks of roughly 8 to 12 hours today; the best open models catch up to that level with a lag of about four months. In practice: for anything that needs to be done in under 8 hours, both families already deliver. The window where a closed model provides a real advantage keeps shrinking.

The model becomes a commodity

The price angle is even more striking. With Kimi K3, a million input tokens costs 30% of Fable 5's price. On OpenRouter, eight of the ten most-used models of August were open-weight and DeepSeek V4 Flash processed nearly twice as many tokens as the top closed model in the ranking. On Hugging Face, Qwen was downloaded 942 million times in a single month.

When open models cost a fraction of a cloud frontier model's price while reaching 95% of its performance, the model stops being a competitive advantage. It becomes a swappable component, like a database engine or a compiler. Nobody picks their tech stack solely on raw compiler speed.

What blocks adoption is not model power

This is where the report gets genuinely interesting. Mozilla asked developers why they abandon open models. The answer is neither cost, nor security, nor raw power. It is day-to-day maintenance work: updates, deployment complexity, difficulty integrating the model into existing systems, evaluation under real conditions.

Production numbers show it: in small companies, open and closed models are nearly tied (53% versus 54% go to production). The gap widens with size: 55% versus 66% in mid-sized companies, 57% versus 73% in large enterprises. That is not a capability ceiling — it is a polish deficit, a lack of turnkey product, compliance and contractual responsibility.

In other words: open models rarely fail on their benchmarks. They fail because the whole experience around them demands work that closed models take care of. The problem is not the engine. It is the rest of the car.

The tool makes the model

This reading matches exactly what we see on the ground with local AI. The formula popularized by modern agent frameworks says it well: Model + Harness = Agent. The model does the thinking; the harness handles everything else — file access, terminal commands, tool calling, session memory, MCP integrations, context management.

The strongest signal comes from the players themselves: when DeepSeek open-sourced a new project in August 2026, it was not a new model but a harness, DeepSeek Harness, able to turn any LLM into a working agent. The big labs, meanwhile, lock their finished products to their own models. The open players are building the tooling that makes any model useful.

Our article on the OpenAI-Hugging Face incident offers a striking illustration. When Hugging Face's security team had to analyze 17,000 attack events, the APIs of top commercial models blocked the requests — their guardrails cannot tell an incident responder submitting real exploit payloads from an attacker asking for them. The team turned to an open-weight model running on its own infrastructure and an analysis that would have taken days was done in about an hour. The model was not smarter than the ones that refused to answer. It was in an environment that let it do the job.

A mid-range model with well-crafted tooling regularly outperforms a frontier model with poor tooling. It is true for coding, where the quality of a well-equipped editor (code search, test execution, repair loop) weighs more than a few index points. It is true for agents, where reliability comes from the tool loop and verification, not just reasoning. It is true for RAG, where indexing and chunking quality decide the outcome long before the model choice does.

Where closed models still lead

Let's stay honest: the window still exists. On high-value professional tasks, Fable 5 keeps a 92-point Elo lead over Kimi K3 on the GDPval-AA v2 benchmark. On finding information scattered across a very long context (one million tokens), Gemini 3.1 Pro reaches 89% while the best open model measures 41%. And closed models come with a counterpart who bears part of the responsibility.

But that lead lies precisely in cases where context and polish matter more than raw power — exactly the domain where good tooling closes the gap. And it shrinks by about four months with every generation.

What it means for local AI

For local AI users, the report reads as a validation. If the model is no longer the limiting factor, there is no point chasing the biggest model possible. The winning strategy fits in four points:

1. Pick a sufficient model, not the best one. A 7-14B model is enough for targeted code edits, summarization, classification or simple tool calls. The extra performance of a frontier model often disappears once the model is embedded in a tool. Our coding models guide helps you size things to your hardware.
2. Invest the saved time in the harness. A good agent framework, well-chosen MCP servers, clean session memory and regular evaluations turn a mid-range model into a genuinely useful assistant. That is exactly the niche of multi-provider agent frameworks like DeepSeek Harness.
3. Polish tool calling. The reliability of a local agent depends first on tool-call quality: formats, schemas, error handling. Configuration details make more difference than the choice between two neighboring models.
4. Route smartly. For the 5% of tasks that genuinely need a frontier model, a hybrid local and cloud flow keeps the best of both worlds: heavy reasoning at a provider, sensitive data and the day-to-day on your machine.

There is also the sovereignty argument. Open weights run on your hardware, with no telemetry and no kill switch and the tooling around them — Ollama, agent frameworks, MCP — is open source itself. You are no longer a customer of a provider: you are the operator of your own stack, much like administering your own GNU/Linux servers rather than renting a closed service.

Run these models with Ollama

The four leading open models are all available on Ollama. Kimi K3, GLM-5.3 and GLM-5.3 Flash only ship as cloud versions there (hosted on Ollama's servers, an account is required) while Qwen3.8 runs entirely locally:

# Via Ollama Cloud (account required)
ollama run kimi-k3:cloud
ollama run glm-5.3:cloud
ollama run glm-5.3-flash:cloud

# Qwen3.8 (27B) — fully local, about 18 GB on disk
# The newest model of the pack, released in September 2026
ollama run qwen3.8

For a 100% local stack, Qwen3.8 is therefore the natural pick of the four — and it covers most everyday needs. Our coding models guide helps you size it to your hardware.

The bottom line

The Mozilla report confirms a shift: model performance is no longer the differentiating factor. Open models score 57-60 while closed ones score 61-63, at a fraction of the price and they already carry more than a third of production tokens. What separates a project that works from one that fails is the work around the model: the harness, the tools, the integration and the evaluation.

The practical conclusion is simple. Stop comparing leaderboard tables and look at your tooling. A well-tooled local model, with solid tool calls, well-managed context and honest evaluations, will do more for your productivity than the latest frontier model accessed through an API. The model is a commodity. The tool is your advantage.