llama-cpp-python Fixes a Subtle Cached-Prompt State Bug

What changed

abetlen/llama-cpp-python merged a fix for recurrent and hybrid models where the full prompt was already cached. The change adds explicit tracking for when restored or truncated state still needs evaluation before sampling, adjusts prefix matching to include the full token sequence, and adds a substantial test block around the behaviour.

The commit touched llama_cpp/llama.py, the changelog, and tests: 323 additions and 16 deletions. This is not a shiny feature release. It is a correctness patch.

Why it matters

Prompt caching is supposed to reduce repeated work. In agent systems, that can mean faster local tool use, cheaper repeated context handling, and less friction when running local models through Python.

The catch: if cached state is wrong, the model may sample from stale or incomplete internal state. That is the kind of bug that looks like “the model is being weird” rather than “the cache path is broken”. Those are painful to diagnose.

For Foundry, Hermes, OpenClaw, and similar local-inference paths, this matters whenever llama.cpp is driven from Python and stateful generation is in play.

My read

This is worth updating for if you are using llama-cpp-python with recurrent or hybrid architectures, or if your stack relies heavily on prompt reuse. If you only run straightforward stateless completions on mainstream transformer models, it is more of a watch item than an emergency.

The strong signal here is the test coverage: nearly 300 lines of tests were added, which suggests the maintainers are pinning down a real behavioural edge case rather than just tidying code.

Bottom line

Small patch, practical consequence. If local Python inference is part of your agent stack, track this and update once it lands in your pinned release path. Cache bugs are boring right up until they cost you a day.

Commit: https://github.com/abetlen/llama-cpp-python/commit/9be3cd135bb87ef5c97662c8e60f5ec9689e94e5