Inside DeepSeek's 155 Trillion Token Engine: What They Didn't Open Source

Описание: 🚀 What if the most important lesson from an open-source inference release isn’t the code they shipped—but the two components they admitted were missing?

DeepSeek serves over 155 trillion tokens a day behind their production API. When they open-sourced DeepInfer under Apache 2.0, the release notes included an unusually candid confession: the global KV-cache system and the disaggregated load balancers were withheld because they are "tightly coupled to internal infrastructure".

In this Skildcore deep dive, we audit the full DeepInfer architecture—what shipped, what stayed behind the firewall, and how DeepSeek coordinates multi-vendor hardware, quadratic prefill schedulers, and a 100 PiB storage cluster across four specialized fleets.

🔥 WHAT YOU'LL LEARN
The 155T tokens/day milestone: How DeepInfer powers production serving for V4.1-Flash
What shipped: Dual-vendor kernels for NVIDIA SM100 and Huawei Ascend 950
The honest gap: Why the distributed KV-cache system and PD orchestration stayed closed-source[cite: 17]
The 4-fleet blueprint: Stateless Encoders, 8-GPU Prefill nodes, Wide-EP Decode supernodes, and 3FS storage[cite: 17]
Scheduling against a clock: Why prefill chunks are sized by nanosecond budgets using quadratic cost models rather than token counts[cite: 17]
DSpark speculative decoding: Using confidence-head filtering to verify only high-payoff draft tokens (-29% verification overhead)[cite: 17]
Bounded replay: Slashing SWA cache-miss penalties to a flat 128-token recompute regardless of context length[cite: 17]
The 100 PiB cache: 10+2 erasure-coded 3FS cluster delivering 14.4 TB logical cache per GPU over GPUDirect RDMA[cite: 17]
Why modern frontier inference is no longer an attention math problem, but a distributed data-movement problem[cite: 17]

⚡ THE CENTRAL INSIGHT
At frontier scale, inference stops being a model execution problem and becomes a distributed data-movement problem[cite: 17]. DeepInfer proves that you don't need exotic all-to-all tensor parallelism when you disaggregate prefill from decode, schedule requests against quadratic wall-clock budgets, and stream 72-hour persistent KV caches over 100 PiB RDMA storage fabrics[cite: 17]. DeepInfer gives you the engine; the fleet orchestration is where the battle is actually won[cite: 17].

🎯 WHO SHOULD WATCH?
AI infrastructure engineers, LLM serving architects, CUDA/Ascend systems developers, distributed systems engineers, and platform teams operating vLLM, SGLang, or TensorRT-LLM in production.

⏱️ CHAPTERS
00:00 - 155 Trillion Tokens / Day: The DeepInfer Confession[cite: 17]
00:20 - Section 01: The Release (Apache 2.0 & V4.1-Flash)[cite: 17]
00:47 - Section 02: What Shipped (SM100, Ascend 950 & Rust Gateway)[cite: 17]
01:35 - Section 03: The Honest Gap (What "Not Included" Means)[cite: 17]
02:08 - Section 04: Fleet Blueprint (Encoder, Prefill, Decode & 3FS)[cite: 17]
03:03 - Section 05: Prefill Scheduling (Quadratic Time Budgets & Async Decode)[cite: 17]
04:04 - Section 06: DSpark (Confidence-Priced Speculative Decoding)[cite: 17]
04:51 - Section 07: Bounded Replay (Sliding Window KV Recovery at Constant Cost)[cite: 17]
05:39 - Section 08: The 100 PB Cache (3FS Cluster, 10+2 Erasure Coding & GPUDirect)[cite: 17]
06:39 - Section 09: The Boundary (Disclosure, Not Product)[cite: 17]
07:14 - Section 10: Data Movement (The Architectural Lesson Under the Engine)[cite: 17]

🔔 Subscribe to Skildcore for deep technical breakdowns of GPU systems, LLM serving, and runtime architecture.

#DeepSeek #DeepInfer #LLMInference #AIInfrastructure #GPUArchitecture #MachineLearning #SystemsEngineering #vLLM