Long-Context ServingCross-Request KV Cache Persistence Patterns for Session-Aware LLM Serving
Persisting KV state pays off only when fetching beats recomputing.
Darius Okonkwo·
Long-Context ServingDisaggregated Storage Backends Versus Monolithic GPU Memory the Infrastructure Stack Vendors Are Actually Shipping
How vendors tier KV cache across HBM, DRAM, and NVMe.
Darius Okonkwo·
Distributed ServingWeight Streaming Architectures That Eliminate GPU Idle Time During Model Loading
Streaming weights concurrently from object storage can reclaim most GPU minutes lost to cold starts.
Darius Okonkwo·
Long-Context ServingDistributed KV Cache Coordination Across GPU Node Boundaries in Multi-Host Inference Clusters
Cross-node KV reuse hinges on routing, metadata, and fast transfer.
Darius Okonkwo·
Long-Context ServingThe Storage Layer Rethink That Unlocks Context Scaling Beyond What GPU Memory Can Offer
Long-context serving needs tiered KV cache storage, and PCIe is the real bottleneck.
Darius Okonkwo·
Distributed ServingTiered Storage Layout Decisions That Determine Sustainable Throughput Ceilings in LLM Inference
KV cache placement and eviction policy determine LLM inference throughput at scale.
Darius Okonkwo·
Long-Context ServingKV Cache Offloading Tools Ranked by Production Inference Overhead
Choose your KV offloading tool by workload: prefix reuse, agent sessions.
Darius Okonkwo·
Distributed ServingDisaggregated KV Cache Architectures Across GPU Memory and NVMe Tiers
Disaggregated KV cache tiers trade memory bandwidth for storage capacity strategically.
Darius Okonkwo·
Long-Context ServingWhy Long-Context LLM Serving Is a Storage Architecture Problem Before a Model Problem
KV cache storage, not model size, determines long-context serving capacity.
Darius Okonkwo·
Long-Context ServingPrefix Caching Effectiveness for Repeated Long System Prompts
Prefix caching saves latency only when matched KV blocks stay resident on GPU.
Darius Okonkwo·
Model-to-Hardware FitGPU Direct Storage Integration With Open-Source Inference Runtimes
Eliminating the CPU bottleneck in GPU storage access unlocks faster inference on longer contexts.
Darius Okonkwo·
Model-to-Hardware FitLlama 3.1 405B Multi-Node Serving Hardware Requirements
Four sequential hardware layers determine whether serving works or merely looks feasible on paper.
Darius Okonkwo·
Model-to-Hardware FitDeepSeek-R1 MoE Routing and NVMe Bandwidth Requirements
Sparse expert routing requires keeping all weights accessible, not just active ones.
Darius Okonkwo·
Long-Context ServingNVMe Offloading for KV Cache in Long-Context vLLM Deployments
NVMe offloading overcomes GPU memory limits for serving multiple long-context requests concurrently.
Darius Okonkwo·
Long-Context ServingKV Cache Memory Scaling From 32K to 1M Token Contexts
Tiered memory strategies keep long-context LLM inference tractable as token windows scale.
Darius Okonkwo·
Long-Context ServingKV Cache Compression Techniques for Long-Context LLM Serving
Quantization and pruning trade memory for speed in different ways depending on your workload.
Darius Okonkwo·
BenchmarkingIdentifying Storage I/O as the Bottleneck in GPU Utilization Profiles
Storage I/O, not GPU compute, limits inference throughput.
Darius Okonkwo·
BenchmarkingMeasuring Time-to-First-Token and Inter-Token Latency Under Sustained Load
Understanding how prefill and decode phases behave separately under server load.
Darius Okonkwo·
Inference RuntimevLLM Chunked Prefill Scheduling and TTFT Optimization
Chunked prefill moves prefill and decode from competing priorities to cooperative scheduling phases.
Darius Okonkwo·