DeepSeek open-sources DeepInfer inference engine

DeepSeek released DeepInfer on October 20, 2026, the software it uses to run its own AI models in production, as open source. The code is under the Apache 2.0 license and supports both Nvidia and Huawei chips. It shows outsiders how DeepSeek serves its own models, though a few key parts are held back.

What DeepInfer is

An inference engine is the software that runs a trained AI model and turns requests into answers. It decides how the model is split across chips, how requests are batched together, and how fast tokens come out. That work largely sets the cost of an AI service.

"DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services," says the project's GitHub page. The repository says it was originally built mainly on vLLM, a popular open-source engine, with ideas from SGLang and Nvidia's TensorRT-LLM.

For now it supports one model: DeepSeek-V4.1-Flash, using the V4.1-Flash-1017 update as its default. DeepSeek plans to land code for future models in the same framework, according to its announcement as relayed by TechFlow.

Nvidia and Huawei hardware

DeepInfer runs on two kinds of chips: Nvidia's SM100-generation data center GPUs and Huawei's Ascend 950 NPUs. An NPU is a processor built for AI work. Support for Huawei's chips matters because DeepSeek has planned large orders of them.

Requirement	Detail
Operating system	Linux
Compiler	GCC 13 or newer
Python	3.12
Nvidia software	CUDA 13
Huawei software	CANN 9.2
License	Apache 2.0

Setup is short, per the README. You set DEEPINFER_ACCELERATOR to nvidia or ascend, run ./build.sh, then run python examples/offline_inference.py. The default model downloads from Hugging Face on first run.

Features in the release

The README lists the main pieces:

• Custom kernels. Hand-tuned chip code, including FlashMLA, DeepGEMM, DeepEP and kernels written in TileLang.
• Parallelism. Work is split across chips by data and by experts. DeepSeek's models are mixture-of-experts designs, which route each token to a few specialist sub-networks.
• Speculative decoding. A feature called DSpark guesses several tokens ahead and checks them, which speeds up output.
• Structured outputs. Answers can be forced to match a regular expression, a JSON schema or a grammar.
• Repeatable results. On Nvidia chips, a batch-invariant mode with fixed seeds gives identical outputs for identical inputs.

DeepSeek also released deepseek-recipe, a tool that converts text to tokens and back, built to match its official API, Crypto Briefing reports.

What is missing

The release is not a full service. "This open-source release provides offline inference with the core features needed for serving at scale," the README says. It names three parts needed for a complete service that are "not included in this release":

• A KV cache system, which stores earlier work so repeated prompts are cheaper.
• Prefill and decode disaggregation, which runs the reading and writing stages on separate machines.
• Load balancing across copies of the model.

Performance claims are also unclear. The README shows only charts, measured on 4 SM100 GPUs for prompt processing and 64 for output. Crypto Briefing cites figures such as 155 trillion tokens a day per cluster and a 96% input cache hit rate. Those numbers do not appear in the README, and a Hacker News thread traced the token figure to a post on Zhihu, a Chinese Q&A site.

What this means for developers

DeepInfer is less a drop-in server than a detailed look at how DeepSeek runs its models cheaply. Its value depends on your hardware and how much you are willing to build.

• Check your hardware first. It targets SM100-class Nvidia GPUs and Ascend 950 NPUs only. Older GPUs are not listed, so teams without that hardware cannot run it as is.
• Plan to build the missing pieces. Without a KV cache, disaggregation or load balancing, you will need your own serving layer for production traffic.
• Read the kernels, even if you never deploy it. Teams running mixture-of-experts models on vLLM or SGLang can study the parallelism and speculative decoding code for ideas.
• Use the repeatable mode for testing. Identical outputs for identical inputs make regression tests on model changes much easier on Nvidia hardware.
• Treat throughput claims as unverified. Run the repository's sweep script on your own chips before comparing it with your current engine.