bitsandbytes 0.50 makes 4-bit inference faster across more hardware

What changed

bitsandbytes 0.50.0 turns several months of backend work into a major release for quantised inference.

The headline change is a fused 4-bit dequantisation and GEMM path for CUDA. It replaces the older GEMV and separate dequantize + F.linear routes for small and medium batches. The maintainers report up to 4x faster inference at batch sizes from 2 to 64 across NVIDIA architectures from Turing to Blackwell, with gains at batch size one in many cases. Runtime dispatch chooses the kernel based on tensor shape, GPU architecture and available SMs.

The same small-batch path now has a fused SIMT implementation for ROCm, and AMD support is no longer labelled preview. ROCm wheels expand to Windows and more consumer and data-centre GPUs.

Apple Silicon is also much less of an afterthought. All 4-bit and LLM.int8() configurations now work through MPS. On macOS 26 or later, the optional kernels package enables optimised Metal kernels; otherwise bitsandbytes falls back to a slower implementation. This path requires PyTorch 2.9 or newer.

CPU quantisation and dequantisation have been reworked for x86-64 and ARM64, with published improvements ranging from 1.1x to more than 20x depending on the operation, data type and processor. Host-side Python dispatch overhead is lower too.

There are real compatibility costs:

• the minimum supported PyTorch version is now 2.4;
• the old research module is gone;
• non-blockwise optimisers and legacy dynamic quantisation APIs have been removed;
• legacy sparse operations have been removed;
• several integer GEMM helpers are now deprecated.

Why it matters

Quantisation only helps operationally if the compressed model still runs quickly and predictably. A smaller checkpoint paired with a slow dequantisation path simply trades one bottleneck for another. Fusing dequantisation into matrix multiplication attacks that overhead directly, especially at the small batch sizes common in interactive agent work.

The broader hardware support matters just as much. Local evaluation should not require a narrow NVIDIA configuration before a model can be tested on real Hermes or OpenClaw workflows. Better ROCm, ARM CPU and Metal paths make it easier to run representative tests on the hardware already available, including Jason’s Apple Silicon environment.

This does not automatically improve an agent. It improves the economics of finding out whether a quantised model is good enough: more models can be tested, more prompts can be replayed, and throughput or latency can be measured without immediately renting larger infrastructure.

My read

Worth a spike, not an estate-wide update. The release is material, but the right result is a benchmark table rather than an upgraded lockfile.

For an existing 4-bit workload, compare 0.49.2 and 0.50.0 using the same model, quantisation, prompts and concurrency. Record cold-load time, first-token latency, tokens per second, peak memory and output quality. On Apple Silicon, test both the optimised Metal-kernel path and the fallback so deployment assumptions are explicit.

Before changing production dependencies, search for removed APIs and verify the PyTorch floor throughout the environment. A performance win is not useful if it quietly invalidates training scripts, optimiser configuration or an older runtime image.

For Foundry, the strongest near-term experiment is a local agent-model replay on Apple Silicon and one CUDA machine. If the reported small-batch gains survive a real tool-calling workload, 0.50 becomes a sensible base for the next quantised-model evaluation stack.

Bottom line

bitsandbytes 0.50.0 is the first release where the recent 4-bit, ROCm, CPU and Apple Silicon work arrives as a coherent operational upgrade. It can make constrained local inference meaningfully faster and broaden the machines that can run it. Spike it with real agent traces, check the breaking changes, and only then update pinned environments.