The repository `ryanzhou/deepseek-v4-flash-mi300x` provides a production-ready configuration and set of patches to run the `DeepSeek-V4-Flash-0731` model on a single AMD MI300X GPU. ### Why It Matters The MI300X is a compelling alternative to NVIDIA's H100 for this specific model due to its hardware specifications: * **Memory Capacity:** It features 192 GB of HBM3 memory, which is 2.4× the capacity of an H100 SXM5, allowing the 304B-parameter model to fit entirely in HBM without needing PCIe weight streaming or layer offload. * **Cost-Efficiency:** It is estimated to cost roughly half as much as an H100 at list price. * **Performance:** The repository demonstrates that a single MI300X can handle 2–8 typical concurrent streams and bursts of up to 64 streams, with a single-stream decode speed of 168.6 tok/s. ### Key Changes and Fixes The official vLLM recipe is primarily optimized for NVIDIA hardware and newer AMD GPUs (like the MI325X and MI355X). Running the model on the MI300X required several critical fixes, which this repository collects and pins: * **FP8 Correctness:** The MI300X (CDNA3) uses a different FP8 variant (`fnuz`) compared to the OCP-standard FP8 used in newer hardware. The repository includes fixes to ensure correct scaling. * **AITER Tuning:** It provides missing AITER GEMM tuning tables for `gfx942` shapes and an OGS geometry override for the MXFP4 experts. * **Hybrid KV Strategy:** It implements a hybrid KV cache strategy (20 GB GPU cache + 96 GB CPU offload) and includes a load-path fencing fix that was documented but never merged in upstream vLLM. * **Stability:** It addresses HIP-graph hazards in sparse MLA decode, MoE routing bugs at high concurrency, and CPU-KV synchronization issues. ### Strongest Sources * [GitHub: ryanzhou/deepseek-v4-flash-mi300x](https://github.com/ryanzhou/deepseek-v4-flash-mi300x) (The primary repository containing the configuration, patches, and performance data). * [Fergus Finn's MI300X worklog](https://fergusfinn.com/blog/deepseek-v4-flash-mi300x/) (Identified the initial FP8 incompatibilities and other hardware-specific challenges). * [Doubleword repository](https://github.com/doublewordai/vllm-amd-blog-doubleword) (Provided foundational work on MI300X performance tuning). The repository is designed for production use, providing a Docker Compose stack with digest-pinned artifacts to ensure a stable and reproducible environment.