deepseek-ai / DeepGEMM

# DeepGEMM
DeepGEMM is a unified, high-performance tensor core kernel library that brings together the key computation primitives of modern large language models — GEMMs (FP8, FP4, BF16), fused MoE with overlapped communication (Mega MoE), MQA scoring for the lightning indexer, HyperConnection (HC), and more — into a single, cohesive CUDA codebase. All kernels are compiled at runtime through DeepJIT, requiring no CUDA compilation during installation.
DeepGEMM leverages some concepts from CUTLASS and CuTe, but avoids heavy reliance on their templates or algebras. The library is designed for simplicity, with only a limited number of core kernel functions, making it a clean and accessible resource for learning NVIDIA GPU kernel optimization techniques.
Despite its lightweight design, DeepGEMM's performance matches or exceeds expert-tuned libraries across various matrix shapes.
## News
- 2026.09.30
 - DeepGEMM Ascend is available! Check DeepGEMM-Ascend for more details
 - Add more optimizations, including locality domain features, check #462 for more details
- 2026.09.10: Sparse Indexer, Mega Gate, Mega mHC, DeepJIT, MoE and Indexer optimizations and more.
 - Please see #432 for more details.
- 2026.04.16: Mega MoE, FP8xFP4 GEMM, FP4 Indexer, PDL, faster JIT compilation and more.
 - Please see #304 for more details.
 - For Mega MoE benchmarks, refer to #316.
- 2025.09.28: DeepGEMM now supports scoring kernels (weighted ReLU MQA logits) for the lightning indexer for DeepSeek v3.2.
 - Please see #200 for more details.
- 2025.07.20: DeepGEMM now supports both SM90/SM100, and has a full refactor with a low-CPU-overhead JIT CPP module.
 - As NVCC 12.9 will automatically do the FFMA interleaving, all post optimizations will be no longer supported.
 - Please see #112 for more details.
- 2025.05.14: DeepGEMM now offers weight gradient kernels for dense and MoE backward! See #95 for details.
- 2025.04.18: DeepGEMM now achieves up to **1550 TFLOPS** on H800! See #74, #78, #81, #86 and 340d988 for details.
## Quick start
### Requirements
- NVIDIA SM90 or SM100 architecture GPU
- Python 3.8 or higher
- Compilers and standard libraries with C++20 ` ` support
- CUDA Toolkit 12.9 or higher
- PyTorch 2.3 or higher
- CUTLASS 4.0 or higher (could be cloned by Git submodule)
### Development
 [code] 
### Installation
 [code] 
Then, import `deep_gemm` in your Python project, and enjoy!
## Interfaces
#### Notices
This library provides optimized GEMM kernels for NVIDIA GPUs with a naming convention: `D = C + A @ B`. The input shape layout is NT (non-transposed A, transposed B). While the SM90 implementation supports only the NT memory layout (row-major, col-major), the SM100 implementation supports all memory layouts (NT, TN, NN, TT). For example, `fp8_gemm_nt` will do a `D = C + A @ B.T`
For both architectures, the LHS scaling factor is required to have a TMA-aligned and transposed layout. And the data format for the scaling factor of SM90 and SM100 is different:
- SM90 r