Best Local Models for Hermes Agent in 2026
Hermes Agent rejects any model under 64,000 tokens of context. Which open-weight models clear that bar on 8GB, 16GB, 24GB and 32GB of VRAM.
Most guides to running Hermes Agent on your own hardware start with the model. That is the wrong end. Hermes publishes a hard requirement that disqualifies candidates before capability enters the argument, and it is the one thing no other guide to this question mentions.
Nous states it plainly: Hermes Agent requires at least 64,000 tokens of context for agent use with tools, and smaller windows are rejected at startup. Not degraded. Rejected.
The rule that disqualifies most local models
The wording in the Nous documentation is worth quoting, because the strictness is the point: “Hermes Agent requires at least 64,000 tokens of context for agent use with tools. Smaller windows are rejected at startup because the system prompt, tool schemas, and working conversation state need enough room for reliable multi-step workflows.”
An agent turn is not a chat turn. Before your question arrives, the window already holds a system prompt, the JSON schemas for forty-odd tools, the running conversation, and the output of whatever the last tool returned. A 8K window is consumed before the model does any work.
Three ways to set it, all documented:
| Setup | Command |
|---|---|
| Ollama, one-off | OLLAMA_CONTEXT_LENGTH=64000 ollama serve |
| Ollama, systemd | Environment="OLLAMA_CONTEXT_LENGTH=64000" |
| Ollama, Modelfile | PARAMETER num_ctx 64000 |
And the trap underneath all three: you cannot set context length through the OpenAI-compatible /v1/chat/completions API. It has to be configured server-side, before the server starts. Nous calls this the single biggest source of confusion when connecting Ollama to tools like Hermes.
Two official sources that contradict each other
This is the part worth knowing before you spend an evening debugging.
Ollama ships a first-party Hermes integration. Its documentation page gives you ollama launch hermes as a one-command setup, and for manual setup it says to leave context length blank and let it auto-detect.
Ollama’s auto-detected default, per the Nous provider documentation, is 4,096 tokens on any GPU with less than 24GB of VRAM. It only reaches 32,768 in the 24GB to 48GB band, and only clears the Hermes floor at 48GB and above.
So on a 12GB or 16GB card, following Ollama’s own advice produces a 4,096-token window, sixteen times below what Hermes will accept, and Hermes refuses to start. Both pages are official. Both were checked on September 4, 2026. Set the variable by hand and ignore the auto-detect advice unless your card has 48GB.
What actually survives: the VRAM funnel
Here is the filter applied to eleven current builds from the Ollama library, every one of which carries the tool-calling capability tag and a native context window at or above 64K.
This first pass counts weights only, against raw card capacity. That is the generous reading, and it is deliberate: the next section adds the context cache and the picture gets considerably worse.
| Stage | Tasks remaining | Percent of candidate pool kept |
|---|---|---|
| Tool-capable builds (Carry the Ollama tools tag) | 11 | 100% |
| Clear 64K context (Native window at or above the floor) | 11 | 100% |
| Weights fit 32GB (RTX 5090 class) | 11 | 100% |
| Weights fit 24GB (RTX 3090 or 4090) | 10 | 91% |
| Weights fit 16GB (Mainstream laptop or GPU) | 5 | 45% |
| Weights fit 8GB (Entry gaming GPU) | 4 | 36% |
The shape is the finding. The context requirement removes nothing from this set, because modern open-weight models ship large windows as standard. What removes them is having to hold that window in memory at the same time as the weights.
The context window is not free, and this is where it falls apart
This is the step almost every local-model guide skips, and once you do the arithmetic most of the recommendations you will read stop working.
Every token in the conversation sits in the KV cache, and its size follows directly from the model architecture: 2 x layers x KV heads x head dimension x tokens x bytes per element. Those three architecture numbers come out of the model config, so this is checkable rather than a rule of thumb. The RAM sizing guide works the formula through in full.
Run it at the 64,000 tokens Hermes demands:
| Model | Per token | Cache at 64K, FP16 | Cache at 64K, 8-bit | Q4 weights |
|---|---|---|---|---|
| Llama 3.1 8B | 0.125 MB | 8.0 GB | 4.0 GB | 5.0 GB |
| Qwen3.8-27B | 0.25 MB | 16.0 GB | 8.0 GB | 17.1 GB |
| Llama 3.3 70B | 0.3125 MB | 20.0 GB | 10.0 GB | 42.0 GB |
Read the Qwen3.8-27B row against a 24GB card. The weights are 17.1GB and fit comfortably. Add the cache Hermes requires and you need 33.1GB at full precision, or 25.1GB with an 8-bit cache. A 27B model does not run Hermes on a 24GB card, even though every guide that stops at the download size says it does.
Two levers if a model almost fits. Quantise the cache, with --cache-type-k q4_0 --cache-type-v q4_0 on llama.cpp, which roughly quarters it for a quality drop most people do not notice at agent workloads. Or drop a size class, which is usually the better trade: a 12B model running at a full 64K beats a 27B model that swaps to system RAM every turn.
Note also that a bigger model does not automatically mean a bigger cache. Llama 3.3 70B is two and a half times Qwen3.8-27B on parameters but only 25 percent worse per token, because grouped-query attention uses four times the KV heads at half the head dimension. Check the architecture, not the parameter count.
Picks by card
Sized for weights plus a 64,000-token cache, not weights alone. The cache column assumes 8-bit quantisation, which is the setting that makes most of these viable.
| Your hardware | Run this | Weights | Realistic at 64K |
|---|---|---|---|
| 8GB card | granite4.1:3b |
2.1GB | Yes, but a 3B model is weak at multi-step tool sequences |
| 12GB card | granite4.1:8b |
5.3GB | Yes, with room for the cache. Apache 2.0 |
| 16GB card | gemma4:12b-it-q4_K_M |
7.6GB | The value pick. 256K native window, vision and tools |
| 24GB card | gemma4:26b-a4b-it-qat |
16GB | Marginal. A 26B MoE with 4B active, needs a quantised cache |
| 32GB card | qwen3.6:27b-q4_K_M |
17GB | Comfortable. The build Ollama names for Hermes |
| 32GB, best | gemma4:31b-it-q4_K_M |
20GB | The pick Nous names, and the class that genuinely needs 32GB |
| Always-on | nemotron-3.5-lightning:30b-a3b |
25GB | 3B active and a 1M native window. Built for agent duty cycles |
The honest headline for anyone shopping: 12GB runs a real Hermes agent, 16GB runs a good one, and 24GB is the point where a 27B model stops being the obvious answer rather than the point where it starts.
The official guide is out of date
Worth flagging, because it is the first table most people find. The Nous local Ollama guide names gemma4:31b as “currently the best local option with tool-call support”, which is sound. The comparison table below it is not: it still lists gemma2:27b, gemma2:9b and llama3.2:3b, and the guide correctly marks all three as having no tool calling at all.
Three of the four rows in the official table are models that cannot run Hermes. They are there as context for a hardware ladder, but read quickly it looks like a shortlist. The current Ollama library has tool-capable builds at every one of those sizes, which is what the table above uses.
A tool tag is not tool calling
The last failure mode, and the most frustrating, because the model looks fine and simply never calls anything.
A model advertised as supporting tools still produces no tool calls if the server is started without the right flags. The Nous provider documentation is specific about each backend:
| Backend | Required flags |
|---|---|
| llama.cpp | --jinja, without which llama-server ignores the tools parameter entirely |
| vLLM | --enable-auto-tool-choice and --tool-call-parser <name> |
| SGLang | --context-length 65536 --tool-call-parser qwen |
| LM Studio | lms load <model> --context-length 64000 |
vLLM ships parsers named hermes, llama3_json, mistral, deepseek_v3, deepseek_v31, xlam and pythonic. Pick the one matching your model family. Without both flags, the documentation notes, the model emits its tool calls as plain text and the agent does nothing with them. On llama.cpp you can confirm the server took the flag by checking http://localhost:8080/props.
The honest ceiling
Local models are further behind on agentic work than on chat, and the gap is not mainly about raw intelligence. It is about holding a tool chain together over many steps without drifting, which is a documented weak point for local models in agentic coding.
Set expectations accordingly. A 12B model on a 12GB card will handle summarising, tagging, file wrangling and routine edits inside Hermes. It will not hold a long refactor together the way a frontier API model does. If the privacy requirement is real, that is a fair trade. If it is a preference, the hosted picks start at roughly five dollars a month in tokens, which is less than the electricity for a card running all day.
For hardware sizing beyond Hermes, the RAM sizing guide has the general arithmetic, and Qwen3.8-27B covers what free open weights do to the economics.
Frequently asked questions
Frequently asked questions
- What is the best local model for Hermes Agent?
- On 16GB, gemma4:12b is the value pick. On 32GB, gemma4:31b, which Nous names as the best local option with tool-call support. The usual advice to run a 27B model on a 24GB card does not survive the arithmetic: a 27B model needs about 16GB for a 64,000-token cache at full precision on top of 17GB of weights, so 24GB is not enough at the context Hermes requires.
- Why does Hermes Agent refuse to start with my local model?
- Almost always the context window. Hermes rejects any model offering under 64,000 tokens at startup, and Ollama defaults to 4,096 tokens on a GPU with less than 24GB of VRAM. Set OLLAMA_CONTEXT_LENGTH=64000 before starting the server. It cannot be set through the OpenAI-compatible API.
- Can I run Hermes Agent on 8GB of VRAM?
- Barely. Four of eleven current tool-capable builds fit 8GB on weights alone, but once a 64,000-token cache is added only the 3B class really works, and a 3B model is weak at the multi-step tool sequences Hermes is built around. 12GB is a far more realistic floor and 16GB is comfortable.
- Why does my local model never call any tools?
- The server flags, not the model. llama.cpp ignores the tools parameter entirely without --jinja. vLLM needs --enable-auto-tool-choice together with a --tool-call-parser matching the model family. Without them the model writes its tool calls out as ordinary text and the agent never executes them.
- How much VRAM does a 64,000-token context actually need?
- It follows from the architecture: 2 x layers x KV heads x head dimension x tokens x bytes per element. At 64,000 tokens that is about 8GB for Llama 3.1 8B, 16GB for Qwen3.8-27B and 20GB for Llama 3.3 70B at full precision, halving with an 8-bit cache. Quantising with --cache-type-k q4_0 and --cache-type-v q4_0 is the usual fix when a model almost fits.
- Should I use ollama launch hermes?
- It is a genuine first-party shortcut and it works, but do not follow its advice to leave context length blank. Ollama auto-detects 4,096 tokens below 24GB of VRAM, which Hermes rejects. Set the context variable explicitly first, then launch.
Sources
- Nous Research (2026). Hermes Agent documentation: LLM and model providers (the 64,000-token minimum, Ollama context defaults, tool-calling flags per backend, parallel-slot division). Verified 2026-09-04. https://nousresearch.github.io/hermes-agent/docs/integrations/providers
- Nous Research (2026). Hermes Agent documentation: run Hermes locally with Ollama (the gemma4:31b recommendation and the hardware table). Verified 2026-09-04. https://nousresearch.github.io/hermes-agent/docs/guides/local-ollama-setup
- Nous Research (2026). Hermes Agent documentation: run local LLMs on Mac (KV cache sizes at f16, q8_0 and q4_0). Verified 2026-09-04. https://nousresearch.github.io/hermes-agent/docs/guides/local-llm-on-mac
- Ollama (2026). Hermes Agent integration (the
ollama launch hermescommand and the auto-detect guidance). Verified 2026-09-04. https://docs.ollama.com/integrations/hermes - Ollama (2026). Model library (build sizes, quantisations, native context windows and tool capability tags for gemma4, qwen3.6, qwen3.8, nemotron-3.5-lightning and granite4.1). Read 2026-09-04. https://ollama.com/library
- Capital & Compute. Local LLM hardware dataset (card memory and the 1GB driver reserve used in the funnel arithmetic), asOf 2026-09-03. https://capitalandcompute.net/ai-models/
Build sizes, context windows and documentation quotes were verified against the sources above on September 4, 2026. The funnel counts weights against card capacity only; cache cost is computed separately from the published KV formula and the per-model architecture figures it takes, and both bases are stated where they appear. See the editorial standards.