Teams keep budgeting for hardware when the real lever is the serving stack. Before you buy more GPUs, three configuration-level upgrades routinely deliver more tokens per second than a new card — and they cost nothing but understanding.
Autoregressive generation is fundamentally serial: one token at a time, each waiting on the last. Speculative decoding breaks this bottleneck with an asymmetric trick:
The result: the big model's quality, with throughput approaching draft-model speed on accepting sequences. Modern implementations now beat multi-token-prediction (MTP) across the full concurrency range on many workloads — and day-one support on new hardware is serving 27B-class models at 38+ tokens/s on desktop-class silicon.
Where it wins: structured output, code, RAG answers — anywhere the target model is confidently predictable. Where it doesn't: high-entropy creative generation, where draft acceptance rates crater and you pay the draft overhead for nothing.
Under concurrent load, your inference server's scheduler — not the model — determines throughput. Static batching waits for a whole batch to finish before starting the next; sequences finish at different times, and GPU cycles idle while stragglers run. Continuous batching admits new requests the instant a slot frees.
The trap: default concurrency limits. We've seen a 16-stream benchmark spread diagnosed as a "scheduling imbalance" that was actually a server capped at 4 concurrent sequences — 12 streams queueing politely behind a config default nobody read. Three requests were fast; everything after position four was waiting in line.
Check your max_num_seqs (or engine equivalent) before concluding anything about model performance. A mis-set scheduler can silently eat an entire GPU's worth of throughput, and your dashboards will happily blame the model.
Production prompts share structure: system instructions, tool definitions, few-shot examples, RAG templates. Without prefix caching, every request recomputes the key-value cache for that shared prefix — identical work, repeated thousands of times per hour.
Prefix caching stores computed KV blocks keyed by prompt prefix. Requests that share a prefix reuse them: first request pays full price, everything after skips straight to the novel suffix. For agentic workloads with 2-4K token system prompts, this alone can halve median latency and dramatically raise effective capacity.
One caveat: measure your hit rate. Adversarially varied prefixes (timestamped headers, randomised ordering) can defeat the cache. Instrument before and after.
Benchmark before trusting any optimisation claim. Paper speedups routinely evaporate on real hardware — we've measured a "1.5-2x win" feature deliver a 4x regression on-box with a specific tensor-parallel configuration. Every one of these techniques interacts with model size, quantisation, sequence length distribution, and concurrency profile differently.
The 30-minute protocol:
VibeComputing designs, benchmarks, and operates inference infrastructure — from single-node deployments to multi-GPU clusters with tuned serving stacks.
Talk to VibeComputing →