DeepSeek open-sources DeepInfer inference engine, claims 155T tokens/day at 80% margin
DeepSeek released DeepInfer, the inference engine behind V4.1 Flash, as open source on the first day of PyTorch Conference. The engine supports distributed inference, speculative decoding, and structured output on NVIDIA and Huawei Ascend platforms. According to DeepSeek's translated blog post, their deployment achieves 155 million tokens per day with a claimed inference margin exceeding 80%, generating approximately $22 million in daily revenue at off-peak prices. The architecture combines vLLM with SGLang and TRRTLLM, though DeepSeek notes their most ultra-optimized KV cache and load balancing components remain proprietary.
DeepSeek's release of DeepInfer provides the first open-source benchmark for large-scale inference economics, revealing 80% margins and 155T daily throughput that challenges closed-source competitors. The disclosure of architecture details and hardware-specific optimizations offers critical insight into the feasibility of open-source AI infrastructure and the cost structure of next-generation AI services.
Source: Latent Space · latent.space