Here's a benchmark result that sounds like a mistake but isn't: a 295-billion-parameter model, expertly distributed across two unified-memory AI nodes, generating tokens 3x slower than a 26-billion-parameter model running on just one of those same nodes.
Same hardware family. Same quantization philosophy. Both systems running at 60%+ of their theoretical ceiling. Nobody misconfigured anything. The 295B model is simply doing 5x more work per token — and no amount of extra GPUs fixes arithmetic.
If you're buying or building inference infrastructure in 2026, this is the math that decides whether your deployment feels instant or unusable — before you spend a single dollar more on hardware.
When a large language model generates text, it produces one token at a time. For each token, the hardware must stream-read every parameter that's active for that token from memory. The ceiling on generation speed is therefore brutally simple:
Decode speed ≈ memory bandwidth ÷ active weight bytes per token
Not total parameters. Active parameters. Most modern frontier-ish models are Mixture-of-Experts (MoE): they have enormous total parameter counts but only activate a fraction per token. The fraction is what your bandwidth bill charges you.
Our two benchmark models illustrate the trap. Both route to 8 experts per token — identical expert counts — but their experts are wildly different sizes:
| Model A (295B MoE) | Model B (26B MoE) | |
|---|---|---|
| Active params/token | ~21B | ~4B |
| Weight bytes read/token | ~15.5 GB | ~2.3 GB |
| Nodes required | 2 (tensor-parallel) | 1 |
| Measured decode | 21.5 tok/s | 61 tok/s |
| Time to first token | 2.3s | 0.57s |
Same expert count. 5x the active bytes. One third the speed. The lesson: "8 active experts" tells you nothing — expert size is the multiplier. Always compute active bytes per token, not expert counts, when you size hardware.
Model A didn't fit on one node — its weights are ~178GB against ~121GB of usable memory. So it runs tensor-parallel across two nodes, which means every transformer layer, every token, pays a network round-trip for the all-reduce.
That's the second law of right-sizing: a model that fits on one device always beats the same model split across two, even when the interconnect is excellent. We verified the interconnect at 71.6 Gb/s of raw RDMA throughput mid-benchmark — the fabric was never the problem. The problem is that splitting a model adds latency proportional to layer count to the per-token critical path, and per-token work is where interactivity lives or dies.
This is why "just add another GPU" so often makes latency worse: you're adding capacity you didn't need, at the cost of a per-token coordination tax you'll pay forever.
There's a third memory cost that surfaces only at real conversation lengths. Every generated token must also re-read the entire KV cache — the compressed memory of everything the model has attended to so far.
In our benchmark, short prompts showed Model A at a healthy ~22 tok/s. At ~35k tokens of conversation context, the same model fell to ~14 tok/s. Extrapolate to 100k+ context documents and the KV read alone can exceed the weight read — halving throughput again with zero configuration changes.
Practical implications: size your KV cache headroom for your real context lengths, not your model's maximum; consider KV quantization (halves the penalty); and expect every long-context feature to cost per-token speed even when the marketing says otherwise.
The same benchmark session produced the most valuable failure of the month. We enabled the model's built-in speculative decoding (MTP) — a feature specifically designed to accelerate token generation, with published 1.5–2x speedup claims.
Result: 4x slower. Single-stream decode collapsed from 22.3 to 5.1 tok/s. The draft-and-verify cycle was running unfused kernels while the main path ran optimized ones — and as a bonus, the draft model quietly consumed enough memory to shrink the KV cache to near-useless concurrency.
The feature was off within minutes of measuring. That's the real lesson of right-sizing: every optimization claim is a hypothesis until it's benchmarked on your hardware, your model, your context lengths. The flag that helps a datacenter H100 cluster can cripple a unified-memory edge box — and vendors publish numbers from the hardware they have, not the hardware you have.
One final distinction that trips up teams constantly: concurrency and latency are different axes. Raising concurrent slots from 4 to 8 raised our aggregate throughput 41% — and changed single-turn response time by exactly zero. A waiting human is a request of concurrency one.
If your product feels slow to one user, more slots won't help. Faster tokens per second on a right-sized model will.
The reflex in AI infrastructure is to over-provision: bigger model, more GPUs, more nodes. The physics says the opposite. The fastest interactive system is the smallest active-parameter model that meets your quality bar, on the fewest devices it fits — with every optimization flag verified by measurement instead of marketing.
We ran every number in this post on real hardware, in one night, with a benchmark script and a willingness to be wrong. Right-sizing isn't a procurement exercise. It's a measurement discipline.
VibeComputing benchmarks and manages local AI infrastructure with real measurements — active-parameter analysis, capacity planning, and configuration tuning that starts from physics, not vendor claims.
Get in touch