Why does my model slow down or run out of memory with a long conversation?

The KV cache, not the model's own weights, is almost always the cause. Weight memory is fixed once a model loads. KV cache memory grows with every token of context you're using, so a long conversation or a big pasted document can push a model past your available VRAM well after it loaded successfully with a short prompt.

Does a bigger context window mean the model remembers more accurately?

Not proportionally. Every model we catalogue has a maximum context window it was trained to accept, but recall quality within that window is not uniform — most current models are noticeably better at using information near the start and end of a long context than information buried in the middle, an effect often called 'lost in the middle.' A 128K context window means the model can accept 128K tokens, not that it uses all of them equally well.

What's the difference between context window and KV cache?

The context window is a property of the model: the maximum number of tokens it was trained to handle, listed in our model catalogue for each entry (for example, 128K for Llama 3.3, 512 for MxBai Embed Large). KV cache is what your runtime allocates in memory to actually hold that context while generating — it's not a fixed number, it grows with how much of the context window you're actually using in a given conversation.

How do I reduce KV cache memory usage without losing context?

Quantizing the KV cache itself (separately from quantizing the model's weights) is the main lever — llama.cpp and several other engines support 8-bit or 4-bit KV cache, which cuts KV cache memory roughly in half or more at a small quality cost. The other lever is simply using less context: summarizing or trimming older conversation turns instead of keeping the full history.

Context Windows and KV Cache Explained

A model’s advertised context window — 8K, 128K, even the 10M figure attached to Llama 4 in our catalogue — is not free at inference time: every token of context a model is actually holding requires memory for its key-value (KV) cache, and that memory grows independently of the model’s own weight size. This is the single most common reason a model that loaded fine with a short prompt runs out of memory or slows down dramatically as a conversation or a document gets long. It’s the other half of the VRAM story our quantization guide doesn’t cover — quantization shrinks the weights you load once, while KV cache is what grows while you use them.

What the context window number actually means

Every model in our catalogue lists a context window — the maximum number of tokens (roughly three-quarters of a word each, the same working ratio our model pages use to translate token counts into a practical word count) it was trained and architected to accept in a single pass. The range across our catalogue is wide: MxBai Embed Large caps out at 512 tokens (a few hundred words), general chat models cluster around 8K to 32K, most current frontier and mid-size models — Llama 3.3, Qwen 3, GLM-5, Mistral Large among them — advertise 128K, Qwen 3-Coder goes to 256K specifically for large-codebase and agentic work, and Llama 4 is catalogued at 10M, an outlier built for ingesting extremely long documents in one pass.

A larger context window is a capability ceiling, not a performance guarantee. It says the architecture can accept that many tokens; it says nothing about how well the model uses information spread across all of them, or what it costs you in memory to actually fill it. Check a specific model’s own context window on its model page before assuming your use case needs the largest figure in the catalogue.

Why KV cache, not model weight, is what runs out

When a model generates text, it doesn’t recompute attention over the entire prompt from scratch for every new token — that would make long generations impossibly slow. Instead, the model caches the key and value tensors it computed for every token already processed, and reuses them. That cache is the KV cache, and its memory footprint scales with:

Context length — every additional token held in context adds its own key/value tensors to the cache. Double the tokens in context, and (at a fixed model and precision) you roughly double the KV cache memory.
Model depth and width — a model with more transformer layers, or a wider attention mechanism, needs a proportionally larger cache per token than a smaller model does.
Precision — an FP16 KV cache uses twice the memory per entry that an 8-bit quantized KV cache does, independent of the model’s own weight quantization.

This is why the throughput and VRAM figures our hardware benchmark pages report specifically call out headroom for “the runtime and KV cache” on top of raw model weight size — a GPU that holds a model’s weights with nothing to spare will fail or fall back to CPU offload the moment a real conversation exceeds a short opening prompt.

A concrete illustration

Two models with identical parameter counts and identical Q4_K_M weight footprints can have very different practical VRAM needs once real conversations run, purely because of how much of their advertised context window gets used and at what KV cache precision. A short back-and-forth chat barely touches the cache regardless of the model’s advertised ceiling; a long document-analysis session or a multi-turn agent loop that keeps full history in context can grow the cache to a meaningful fraction of the weight size itself, especially on a model with a large context window and an unquantized cache. That’s the mechanism behind a pattern many local-AI users notice empirically: the same model and the same GPU handle a quick question fine and then stall or error out on a long-running session.

What to do about it

Three practical levers, in order of impact:

Quantize the KV cache, not just the model weights. llama.cpp and other engines expose this as a separate setting from weight quantization — an 8-bit or 4-bit KV cache roughly halves or better its own memory footprint at a modest, usually acceptable quality cost, and it’s the single highest-leverage change if you’re hitting a memory wall on long context specifically.
Don’t confuse “supports 128K context” with “needs 128K context.” If your actual use case is short exchanges, running at a much smaller configured context limit than the model’s maximum reduces KV cache memory proportionally and costs nothing in quality for that use case.
Trim or summarize history in long-running sessions rather than keeping every prior turn verbatim in context — an agent loop or chat application that periodically compresses older turns keeps effective context useful without linearly growing the cache for the life of the session.

Limitations

This explains the mechanism, not exact numbers for every model in our catalogue: KV cache size depends on architecture details (layer count, attention head configuration) that vary per model and that we don’t republish here, so treat the guidance above as a way to reason about why memory grows with context, not a formula to compute an exact gigabyte figure without your engine reporting it directly. Runtimes also differ in how aggressively they manage cache eviction and reuse, so two engines running the same model at the same context length will not necessarily show identical memory use.

Bottom line

A model’s context window is a ceiling on what it can accept, not a fixed memory cost — the actual cost is the KV cache, and it grows with how much of that ceiling a real session actually uses. Before assuming you need a bigger GPU for a long-context use case, check whether your engine quantizes the KV cache and whether you’re actually configuring the context length you need rather than the model’s maximum by default.