Long-Context ServingEviction Policy Design When KV Cache Growth Saturates GPU Memory in Serving Clusters
How to choose among retaining, offloading, and discarding KV blocks.
Mei-Ling Tsao·
Long-Context ServingPer-Request KV Cache Budget Enforcement as the Primary Cost Lever in Long-Context Serving
Capping KV memory per request is the biggest cost lever in long-context serving.
Mei-Ling Tsao·
Distributed ServingLoad Balancing Strategies for Heterogeneous Multi-Node LLM Serving Fleets
Choosing the model and the instance together beats splitting the decision.
Mei-Ling Tsao·
Distributed ServingWhy Storage Latency Becomes the Binding Constraint in Production LLM Inference Pipelines
Long-context KV caches outgrow HBM, so I/O speed decides latency.
Mei-Ling Tsao·
Long-Context ServingBeyond GPU Memory Walls the Hidden Bottlenecks Throttling Long-Context Transformer Deployments
GPU memory bandwidth, not capacity, becomes the real constraint for long-context model serving.
Mei-Ling Tsao·
Long-Context ServingStorage-First Architecture Decisions for LLM Serving Systems Targeting Million-Token Contexts
Organizing KV cache across memory tiers unlocks efficient million-token LLM serving.
Mei-Ling Tsao·
Long-Context ServingKV Cache Storage Tiering Across DRAM HBM and SSD in Million-Token Serving Pipelines
Tiering KV cache across storage requires careful I/O engineering to beat recomputation costs.
Mei-Ling Tsao·
Long-Context ServingKV Cache Memory Pooling and Allocation Strategies for High-Throughput LLM Serving
Pooling and quantization strategies reclaim wasted GPU memory for higher concurrent LLM throughput.
Mei-Ling Tsao·
Distributed ServingRDMA Over InfiniBand for KV Cache Transfer Between Prefill and Decode Nodes
Disaggregated inference requires efficient KV cache transfer to keep decode nodes from stalling.
Mei-Ling Tsao·
Long-Context ServingAttention Kernel Efficiency at Long Sequence Lengths in Flash Attention 3
FlashAttention-3 reaches GPU compute limits on long sequences through asynchronous execution.
Mei-Ling Tsao·
Long-Context ServingChunked Prefill Strategies for 128K Token Inputs in vLLM
Balancing GPU memory, cache capacity, and latency when processing prompts longer than 128K tokens.
Mei-Ling Tsao·
Model-to-Hardware FitSelecting NVMe Drive Count and Topology for KV Cache Offloading Workloads
Bandwidth, not storage capacity, determines how many NVMe drives you need for KV cache offloading.
Mei-Ling Tsao·
Distributed ServingPipeline Parallelism Latency Bubbles and Mitigation in Multi-Node Inference
Recent systems show that pipeline bubbles stem from two separate problems requiring different fixes.
Mei-Ling Tsao·
BenchmarkingTokens per Second vs Latency Tradeoffs in Throughput-Optimized Serving Configs
Optimizing for throughput often means sacrificing the latency users actually feel.
Mei-Ling Tsao·
BenchmarkingReproducing MLPerf Inference Benchmarks on Open-Source Serving Stacks
Official MLPerf scores reflect vendor co-design, and reproducers face a real gap.
Mei-Ling Tsao·