llama.cpp adds EAGLE3-v3 speculative decoding support for GPT-OSS

What changed

ggml-org/llama.cpp has merged support for EAGLE3-v3 speculative decoding with GPT-OSS. The implementation explicitly references NVIDIA’s gpt-oss-120b-Eagle3-v3 draft model.

This is a default-branch commit, not part of the newly published b10156 release. That build was cut several hours earlier and carries a separate HIP maths change, so operators should not assume downloading b10156 includes the EAGLE3-v3 work.

The commit adds the pieces needed to convert and run this newer draft architecture:

• conversion reads the draft model’s explicit auxiliary hidden-state layer IDs instead of always deriving three fixed positions from the target model;
• the GGUF metadata can record whether the draft requires RMS normalisation before its feature-fusion projection;
• the runtime can extract the target model’s final hidden state when the draft configuration asks for the layer immediately beyond the numbered transformer blocks;
• the EAGLE3 graph applies the optional pre-projection normalisation and loads its new tensor;
• the GPT-OSS mixture-of-experts graph exposes the required final hidden state without prematurely narrowing it to selected output rows.

In plain English, llama.cpp can now represent and execute assumptions made by NVIDIA’s newer GPT-OSS draft model rather than treating it like an older EAGLE3 variant.

Why it matters

Speculative decoding uses a smaller draft model to propose several tokens while the larger target model verifies them. When the target accepts enough proposals, generation becomes faster without changing the target model’s final token distribution. It is an implementation optimisation, not a cheaper model pretending to be the larger one.

That is relevant to agent systems because much of their visible latency sits in repeated generation: deciding on a tool, producing arguments, reading the result and continuing. Reducing decode time on GPT-OSS could improve interactive response times or increase throughput on a fixed local GPU estate.

The catch is acceptance rate. A draft that predicts ordinary prose well may perform differently on JSON arguments, code, shell commands and short tool-selection turns. Extra model memory and verification work can erase the benefit if the target rejects too many proposed tokens. Support landing in the runtime is therefore the start of the evaluation, not proof of a speed-up.

My read

Worth a spike once this reaches a tagged build. The change is technically substantive and directly connected to local GPT-OSS throughput, but there are no end-to-end performance figures in the commit. Updating a production llama.cpp service to an arbitrary default-branch revision would turn a useful experiment into needless operational risk.

A sensible Foundry test should compare the same GPT-OSS quantisation with and without the EAGLE3-v3 draft across three workload shapes:

1. normal conversational responses;
2. structured tool calls and JSON-heavy output;
3. longer coding or diagnostic tasks typical of Hermes and OpenClaw.

Record time to first token, generated tokens per second, draft acceptance rate, peak memory and total wall-clock time. Also replay known tool-calling cases to confirm that the optimisation has not exposed a conversion or runtime edge case.

If the gain only appears on long prose, it may still be useful for report generation but less valuable for an operator loop. If acceptance remains high across tool calls, it becomes a credible way to make a large local model feel less cumbersome without changing the model serving API.

Bottom line

llama.cpp now has the core support required to pair GPT-OSS with NVIDIA’s EAGLE3-v3 draft model. That creates a plausible acceleration route for local agent workloads, but the commit is newer than the latest tagged build and ships without workload benchmarks. Watch for the next release, then spike it against real tool-calling traces before changing any production runtime.