Quantization Is a Kernel Contract, Not a File Format

Here's a deployment story that cost us twelve launch attempts and a lot of beliefs about how quantization works.

The setup: a new ARM-based, unified-memory GPU appliance. The target: a 27B-parameter model in NVFP4 format — a 4-bit quantization scheme co-designed by NVIDIA for their newest silicon. On disk: 21GB. The box has 128GB of unified memory. On paper, it fits with 6x headroom.

The model loaded beautifully. Every time. Then it died at the first 4-bit matrix multiply.

What "Supported" Actually Means

The inference runtime listed NVFP4 as a supported format. The checkpoint passed validation. Weight loading completed without a warning. Then, at the first compute-heavy operation, one of three things happened:

• Autotuning mode: kernel autotuner segfaulted hunting for an fp4 GEMM implementation that didn't exist.
• Graph capture mode: CUDA graph capture segfaulted for the same reason.
• Eager mode: the worst one — memory climbed toward 100GB and the OOM killer shot the scheduler. On a unified-memory box, that collateral damage takes out everything else on the machine too.

The root cause: this runtime relies on just-in-time compiled kernels. For this particular silicon generation, no native fp4 GEMM kernel existed in the JIT repository. So the runtime did the worst thing a "compatible" stack can do: it silently dequantized every tensor back to bf16 before compute.

A 21GB 4-bit model that suddenly needs bf16-sized working memory isn't a model that "runs with reduced efficiency." It's a model that cannot run — it just fails slowly enough to look like a memory leak.

The Contract You Didn't Know You Signed

Everyone treats quantization as a property of the model file. FP8, INT4, NVFP4 — check the box in the model card, ship it. But quantization is really a contract between three parties:

1. The checkpoint — the weights, in their compressed representation
2. The runtime — the inference server that claims to load the format
3. The silicon — which must have actual kernels (compiled or compilable) to compute in that format

Model cards and runtime release notes tell you about parties 1 and 2. Almost nothing in the standard deployment checklist tells you about party 3. And the failure mode of a broken party-3 contract isn't a clean error — it's a silent fallback to dequantization that trades your memory budget for the runtime's compatibility promise.

We've written before about silent fallbacks in AI infrastructure. This is the sharpest version of it we've seen: a fallback that's technically correct (the model does produce output, briefly) and operationally fatal (nothing else on the box survives).

Dequant Math, Quickly

Why does a 21GB model need ~100GB? Three multiplications stack:

• 4-bit → 16-bit: dequantizing weights for compute inflates them ~4x. 21GB becomes ~84GB of bf16 working set.
• Activations: computed in bf16 regardless — KV cache and intermediate tensors don't shrink because the weights were 4-bit on disk.
• Overhead: dequant scratch buffers, allocator fragmentation, and (if you're unlucky) a runtime that holds both representations during the transition.

On a discrete-GPU server with 80GB HBM, you'd get a clean out-of-memory at worst. On unified memory, the "GPU" allocation competes with the OS page cache, your desktop session, and every other process — so the failure cascades into whatever the kernel's OOM heuristic decides to kill first. During our twelve attempts, the casualties included the desktop audio stack. Twice.

The Three Checks That Would Have Saved Us Twelve Attempts

1. Verify the native kernel for YOUR silicon, not the format checkbox

"NVFP4 supported" in a runtime's README means somewhere, on some hardware. The question is whether a native fp4 kernel exists for your exact compute capability — and whether your runtime ships it compiled (AOT) or attempts to fetch/compile it (JIT). JIT-only stacks are fragile precisely because "no kernel found" doesn't fail loudly; it falls back.

2. Watch memory at the FIRST matmul, not at load

Model loading validates the file, not the compute path. A successful load followed by a memory explosion at the first forward pass is the fingerprint of dequant fallback. If your monitoring only samples after warmup completes, you'll see "OOM during inference" and go hunting for batch-size bugs in the wrong place.

3. Test whether "unsupported" fails loudly or silently

Take a small layer, force an unsupported path, and see what happens. A runtime that errors clearly is a good citizen. A runtime that quietly dequantizes is telling you it values perceived compatibility over your operational envelope. That preference will bite you at 3AM on the biggest model you own.

What We Ended Up Doing

The pragmatic answer on this hardware: FP8. Native kernels, verified. Weighted against the theoretical 2x memory advantage of 4-bit, an 8-bit format that actually computes in-format beats a 4-bit format that dequantizes on every layer. Add speculative decoding with an FP8 draft model and the serving throughput exceeded what the 4-bit config promised on paper.

There's a path to real 4-bit on this box — the vendor's prebuilt containers ship ahead-of-time compiled fp4 kernels — but that's a vendor-lock-in decision, not a deployment default. Knowing which one you're making is the point.

The Generalizable Lesson

Silicon vendors ship new formats every 18 months. Runtimes race to list compatibility. Model zoos race to publish checkpoints in the newest format. Nobody in that pipeline is incentivized to tell you when the claim is load-bearing.

So treat every quantized deployment as unverified until you've watched it compute in-format under production-shaped load. Your memory graph at the first matmul is the only support matrix that matters.

Quantization is a kernel availability contract. Read the fine print — or watch it silently rewrite your memory budget.