Overview of DeepInfer

DeepInfer can be divided into several layers.

1. Entrypoint.
   • entrypoints/llm.py is the entrance for offline batch inference.
2. Engine layer.
   • engine/llm_engine.py is the scheduler core, consisting of scheduling, async execution, and state management
   • engine/arg_utils.py is for argument parsing
3. Worker layer.
   • worker/model_runner.py is responsible for model forward, sampling, and graph capturing
   • worker/cache_engine.py is for KV cache management, like SWA, compression, and indexing
   • worker/cuda_graph_runner.py, dflash_cuda_graph_runner.py captures CUDA Graph
4. Model and Executor layer, which consists of models and kernels.
   • models/ and v4_modules/ modules for models.
   • layers/ + tilelangs/ Kernels in TileLang, CUDA or Ascend
   • eplb/ Expert Parallelism load balance
   • input_metadata/
5. Support layer.
   • ve/ vision encoder.
   • structure_output/ xgrammar for structured output and mask
   • distributed/ is shared memory broadcast
6. C/C++ & Hardware layer, which abstracts hardware.
   • csrc/ contains C++ scheduler, sequences, block management, and pybind.
   • deep_cluster/ abstracts hardware devices, including Nvidia, Huawei Ascend, CPU, etc.