An AI agent is a model that decides, on its own, when to call a tool instead of just answering in text — and building one locally means three separable choices: a model that’s actually trained for tool use, a way to guarantee its output parses cleanly, and an interface for wiring both to real tools. Get the first two wrong and the third doesn’t matter: an agent framework wrapped around a model that wasn’t trained to call tools will hallucinate function names and arguments no matter how well the framework is built.
Start with a model that supports tool use
Not every local model handles function calling equally well. Our model catalogue tags this explicitly rather than leaving it to guesswork: Llama 3.1, Llama 3.3, Qwen 3, Mistral Small, Mistral Large, Granite 3.3, Command R, and Ministral 3 are all tagged tools for general tool-use support. A smaller group goes further and is tagged function-calling specifically — Hermes 3 and Phi-4 Mini — meaning tool use was a dedicated part of their fine-tuning rather than a side effect of broad instruction tuning. For multi-step, self-directed workflows rather than a single tool call per turn, look at models tagged agentic: GLM-5 and its point releases, the Kimi K2 family, GPT-OSS, Devstral, Cogito, and minimax M2.7/M3 are all built with sustained tool-use loops in mind, not just one-shot function calls.
The practical difference shows up under load: a tools-tagged general model will usually call the right function on a simple, well-specified task, but an agentic-tagged model degrades more gracefully across a long chain of tool calls where earlier results have to inform later decisions — the kind of loop a coding agent or a research assistant actually runs.
Guarantee the output, don’t hope for it
A tool call is only useful if its arguments parse. Two local libraries solve this by constraining generation itself rather than trusting the model to format things correctly:
| Outlines | Guidance | |
|---|---|---|
| Publisher | dottxt-ai | Microsoft |
| Mechanism | Finite-state machine compiled from your schema | Template DSL interleaving fixed text with constrained generation blocks |
| Guarantee | 100% schema-valid output by construction | Structure guaranteed by the template; content within it is model-generated |
| Best fit | JSON schemas, regex, context-free grammars | Programs where you want fine control over exactly what varies |
| License | Apache-2.0 | MIT |
Both mask invalid tokens at the sampling step rather than validating after the fact, which is the actual guarantee: the model is mathematically unable to emit a token that breaks the schema, so there’s no retry loop or JSON-repair step needed downstream. Pick Outlines when your target is a clean JSON schema or grammar; pick Guidance when you want the fixed and generated parts of an output interleaved in a template you author directly.
Wiring it together: function calling versus MCP
Two integration paths solve the “how does the model actually reach a tool” problem, and they are not mutually exclusive:
Direct function calling is the simpler path. llama-cpp-python’s server implements OpenAI-compatible function calling directly, which is what makes it work with frameworks like LangChain and LlamaIndex that already expect that interface — you define a tool schema, the model returns a structured call, your code executes it and feeds the result back. This is the fastest way to get a single agent working end to end, and it’s the same pattern Llamafu uses to bring tool calling to Flutter apps on Android and iOS, so a mobile agent that reads local files or queries device APIs uses conceptually the same loop as a server-side one.
MCP (Model Context Protocol) standardizes that loop at a higher level: instead of wiring each tool into each client separately, an MCP server exposes a set of tools and data sources once, and any MCP-compatible frontend can use it without custom integration code. LibreChat implements this directly — its MCP support is what lets it hand a model file-system access, database queries, or web browsing through a standard interface rather than a bespoke plugin per capability. The trade-off is setup cost: writing a one-off function-calling integration is faster for a single tool, and MCP pays for itself once you’re maintaining several.
A worked example: a local research assistant
A minimal version of this looks like:
1. User asks a question that needs current data.
2. Model (Qwen 3 or Hermes 3, both tools-tagged) decides
a web-search tool call is needed and emits it.
3. Outlines constrains that emission to your tool-call
JSON schema, so it always parses.
4. Your code executes the actual search and returns
results as a tool message.
5. Model reads the results and either calls another tool
or answers.Served through Ollama, this loop runs entirely on local hardware except for the search call itself, which is the only step that has to leave the machine. Swapping the web-search tool for a local document index turns the same loop into a fully offline RAG-with-agency setup, using the same model and the same schema.
Limitations
Local agentic models still trail frontier hosted models on long, multi-step plans that require holding a lot of intermediate state — the gap is real and widest on tasks with more than five or six sequential tool calls where an early mistake compounds. Structured-output libraries guarantee the shape of a tool call, not that the arguments inside it are correct; a schema-valid call to the wrong function, or with a plausible but wrong argument, still happens and still needs error handling in your own code. And MCP is a young enough standard that not every frontend supports it yet — check a specific tool’s own documentation before assuming MCP support rather than inferring it from general popularity.
Bottom line
Pick a model tagged for tool use over a general chat model, use Outlines or Guidance rather than trusting raw JSON output, and start with direct function calling — reach for MCP once you’re wiring more than a couple of tools into more than one frontend. All of it runs on the same local hardware you’d use for chat; agentic capability is a property of the model and the schema, not of the hosting.