llama.cpp keeps quantised embedding lookups on the GPU

What changed

llama.cpp merged CUDA support for quantised GET_ROWS, the operation used for row and embedding lookups inside a model graph.

The CUDA backend already handled the basic Q4, Q5 and Q8 types. This change extends the direct device path to the K-quants, I-quants and MXFP4 formats used by common GGUF recipes. It covers Q2_K through Q6_K, nine I-quant variants and MXFP4, bringing CUDA’s type coverage into line with CPU, Metal, Vulkan and SYCL.

That sounds like a narrow kernel change. The operational effect can be much larger. A Q4_K_M GGUF may store its token embedding table as Q6_K. Before this patch, CUDA rejected that quantised lookup, so the scheduler moved the operation to the host and could copy the full embedding matrix back on every generated token.

The pull request reports Qwen3 1.7B decode latency falling from 6.18 ms/token to 1.72 ms/token on the contributor’s hardware. That is a project-reported result for one model and setup, not an independent benchmark, but it shows why a small-looking fallback can dominate a local inference path.

The implementation reuses the existing CUDA dequantisation routines and enables the previously skipped backend tests against the CPU reference. There is one versioning wrinkle: the watched b10082 release was published about 15 minutes before this commit landed, so b10082 does not contain the new CUDA path. Use a later build or compile from a commit that includes c5a4a0b.

Why it matters

Local inference performance is not only about the headline matrix-multiplication kernels. A single unsupported graph operation can force a host fallback, add transfers and quietly erase the advantage of running the rest of the model on the GPU.

That matters for agent runtimes such as Hermes and OpenClaw because interactive tool use is latency-sensitive. Agents often generate many short turns around tool calls rather than one long uninterrupted answer. Per-token stalls compound across planning, tool selection, retries and summaries.

The patch also narrows a backend consistency gap. A quantisation that behaves well on Metal or Vulkan should not unexpectedly make CUDA slower because its embedding format falls outside one operation’s supported-type list. Full type coverage makes hardware comparisons less misleading and deployment recipes less brittle.

My read

Worth a spike for any NVIDIA local-inference host running K-quant, I-quant or MXFP4 GGUF models. It is not a reason to rebuild a stable Apple Silicon or CPU deployment.

Test the exact model and quantisation already used in production. Compare a build before c5a4a0b with one after it, keeping prompt, context length, batch settings and GPU layers fixed. Record prompt processing, decode milliseconds per token, tokens per second, VRAM, CPU utilisation and output parity. The expected signal is lower CPU activity and better decode latency where GET_ROWS previously fell back.

Do not treat b10082 as the upgrade vehicle: it predates the merge. Wait for the next tagged build if reproducibility matters more than speed, or pin a known post-merge commit for the spike. Update now only if profiling already shows the host fallback and the test passes on the deployed GPU and GGUF. Otherwise, watch only until the patch appears in a later release.

Bottom line

This is the useful kind of local-AI optimisation: not a vague speed claim, but the removal of a specific device-to-host fallback with a measurable reason for the gain. On affected quantised models, CUDA can now keep embedding lookups where they belong — on the GPU. Benchmark the real agent model, pin the build, and promote it only after output parity and latency both hold.