Local inference stack tightens up around llama.cpp

What changed

A small cluster of local-inference projects moved in the same useful direction:

• ggml-org/llama.cpp published release b9776, with a Vulkan Flash Attention fix that applies bias before softmax to avoid overflow. It also added support for Liquid LFM2.5-ColBERT-350M and LFM2.5-Embedding-350M models, including better handling for bidirectional/non-causal embedding variants.
• abetlen/llama-cpp-python updated its vendored llama.cpp revision to 92e854ab8, exposing newer upstream APIs through the Python bindings.
• ollama/ollama fixed prompt truncation when context shifting is enabled, preserving real generation headroom instead of leaving the model with almost no space to answer.
• unslothai/unsloth changed its macOS installer so the default prebuilt llama.cpp path no longer requires Homebrew or CMake.

Links:

• https://github.com/ggml-org/llama.cpp/releases/tag/b9776
• https://github.com/ggml-org/llama.cpp/commit/88636e178ff2972e1002cf2024cb39008eda1192
• https://github.com/abetlen/llama-cpp-python/commit/4bee85b352ec4aa7034dc13c3d80688805e47d63
• https://github.com/ollama/ollama/commit/c191a145bb6b40c7f8fa1ba91f3c7c3467f68983
• https://github.com/unslothai/unsloth/commit/1237cd4d84c84682ff2fa5528a5f8fd70abbca74

Why it matters

For agent tooling, local inference tends to fail in dull places: prompt windows get eaten, Python bindings lag upstream, model conversion falls over on metadata assumptions, or installation requires one more toolchain than the user has installed.

This set of changes hits those dull places directly.

The Ollama fix is probably the most immediately operational. If a long shifted prompt leaves only a token or two for generation, an agent does not just perform badly; it can appear broken. Reserving output room makes local models more predictable in long-context workflows.

The llama.cpp embedding work is also worth watching. ColBERT and embedding models are more relevant to retrieval systems than chat demos, and retrieval is where local AI infrastructure becomes useful rather than decorative.

My read

This is not a dramatic release. It is the local inference stack becoming a bit more boring, which is exactly what production-ish agent systems need.

For Foundry/Hermes/OpenClaw, I would treat this as a spike rather than a blind update. The sensible path is to test the new llama.cpp build and Ollama prompt-shift behaviour against a real agent transcript, especially anything using long prompts, retrieval context or local embeddings.

Unsloth’s macOS installer change is less server-critical, but it lowers the setup tax for people experimenting locally. Fewer Homebrew/CMake traps is not glamorous. It is still useful.

Bottom line

Worth a spike. Test llama.cpp b9776, check the Python binding update if anything depends on llama-cpp-python, and watch the Ollama truncation fix for local agents that run close to their context limit.