Distributed ServingNetworked NVMe Fabrics for Shared KV Cache in Multi-Node Serving Clusters
Shared NVMe fabric turns isolated GPU cache into a cluster-wide resource.
Tobias Wrenner·
Distributed ServingTensor Parallelism Degree Selection for Llama and Mixtral Multi-GPU Deployments
Tobias Wrenner·
Long-Context ServingMulti-Round Conversation State Management at Scale for Long Contexts
Managing KV cache across memory tiers prevents latency from becoming the bottleneck.
Tobias Wrenner·
Model-to-Hardware FitFP8 vs BF16 Inference Throughput and Memory Tradeoffs for Llama and Qwen Models
FP8 cuts memory but batch size and hardware determine whether throughput gains are real.
Tobias Wrenner·
Long-Context ServingDisaggregated Prefill and Decode for Long-Context Request Handling
Separating prompt and token generation onto different hardware eliminates bottleneck conflicts.
Tobias Wrenner·
BenchmarkingDesigning a KV Cache Throughput Harness for Production Serving Nodes
Naive peak-memory tests miss the churn that actually hurts latency.
Tobias Wrenner·
BenchmarkingBenchmarking NVMe Sequential and Random Read Performance for KV Cache Patterns
How to test NVMe drives against real KV cache offload I/O.
Tobias Wrenner·
Inference RuntimeSpeculative Decoding Overhead in Production vLLM Deployments
Speculative decoding speeds up single requests but can cut throughput under heavy load.
Tobias Wrenner·
Inference RuntimeLlama.cpp KV Cache Quantization and Memory Tradeoffs
Quantizing the KV cache cuts memory use sharply but slows token generation at long context lengths.
Tobias Wrenner·
Inference RuntimeTensorRT-LLM In-Flight Batching vs vLLM Continuous Batching
Continuous batching is the real win—the rest is architecture and memory management choices.
Tobias Wrenner·