[Klaud Cold] [AMD] Bump DSV4 MI355X vLLM 8k/1k to the 2026-09-01 nightly / [Klaud Cold] [AMD] 将 DSV4 MI355X vLLM 8k/1k 更新至 2026-09-01 nightly - #2792
Conversation
Update the fixed-sequence STP and MTP image pins to the 2026-09-01 ROCm nightly. Keep the existing AITER flags because the newly merged DSV4 optimizations auto-enable through that path. 中文：将 MI355X 固定序列 8k/1k 的 DSV4 STP 与 MTP 镜像更新到 2026-09-01 ROCm nightly。保留现有 AITER 标志，新合入的 DSV4 优化会通过该路径自动启用。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
jiacao-amd
added
AMD
full-sweep-fail-fast
full-sweep-enabled
and removed
full-sweep-fail-fast
labels
|
Claude finished @jiacao-amd's task in 1m 56s —— View job PR Review
LGTM - no blocking issues found. Verification details:
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Looks good, straightforward image bump. Reviewed the amd-master.yaml diff (only the two image: tags and their explanatory comments change; concurrency ranges, TP, and scenario fields are untouched) and confirmed the perf-changelog.yaml entry is appended at the tail per the append-only convention, matching the two config keys touched.
Extended reasoning...
Overview
The diff touches only configs/amd-master.yaml and perf-changelog.yaml. In amd-master.yaml, the two dsv4-fp4-mi355x-vllm and dsv4-fp4-mi355x-vllm-mtp entries have their image: field bumped from one ROCm nightly tag to another (both digest-suffixed, per the repo's stated digest-pinning requirement), plus rewritten explanatory comments describing why no new serving flags are needed. No other YAML fields (model, precision, framework, search-space/concurrency ranges, spec-decoding) changed. perf-changelog.yaml gets one new entry appended at the very tail, listing the two config keys, scenario-type, old/new image tags, digest, and a note that existing AITER env vars/flags are unchanged — consistent with AGENTS.md's append-only, byte-sensitive changelog rule.
Security risks
None. This is a version/tag bump in a YAML config plus a changelog entry; no code execution paths, auth, or secrets are touched.
Level of scrutiny
Low. This is exactly the class of change AGENTS.md and the guidelines call out as approvable: a mechanical image-tag bump with matching documentation update, no serving-flag or scenario changes in the YAML body itself. The model.container must equal image rule from AGENTS.md applies only to multi-node srt-slurm recipes, which is not the case here (these entries have no model.container field), so that invariant is inapplicable rather than violated.
Other factors
The bug hunter reported zero findings. I independently verified via git diff that the only body (non-comment) changes are the two image: lines, and that the perf-changelog entry was appended after the prior tail entry (PR #2634) without altering any preceding bytes. No CODEOWNERS check was needed since this is a generic recipe config, and the PR description's claims (digest, upstream vLLM commits) are not independently verifiable from the diff alone, but they don't affect the correctness of the config-file mechanics being reviewed here.
Add shared-expert fusion and INT4 quick-reduce environment variables to the MI355X 8k/1k STP and MTP paths, and document the explicit recipe settings.\n\n中文：为 MI355X 8k/1k 的 STP 与 MTP 路径加入 shared-expert fusion 和 INT4 quick-reduce 环境变量，并同步记录显式启用的配方设置。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33478188035 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33480422577 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33537691400 |
Reuse the completed full sweep after synchronizing the PR with main. 中文：PR 与 main 同步后复用已完成的完整扫描，跳过重复扫描。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33681446902 |
|
/stage-results 33538769698 |
|
@chunfangamd staged run 33538769698: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-09-01~r33538769698 This run remains available across future |
|
/reuse-sweep-run 33538769698 |
chunfangamd
left a comment
chunfangamd
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33538769698
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33538769698
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Doesn't apply (For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.)
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
Signed: @chunfangamd
❌❌❌ REJECTED ❌❌❌@chunfangamd — blocking: the sign-off contains no link to a merged/published upstream recipe for the single-node dsv4 MI355X vLLM recipes. The Additional detail section is still the template placeholder. Please post an updated sign-off linking the merged recipe PR in vllm-project/recipes or the published page on recipes.vllm.ai that covers this launch config. ✅ Check 0 (CODEOWNER): PASS — @chunfangamd is a named owner of |
|
/reuse-sweep-run |
Resolve conflicts from #2792 and #2841. Both bumped DSv4 MI355X vLLM images to nightlies that predate the FHMoE commit vllm-project/vllm@de69e821 (7c5dc571 by 266 commits, e962733e by 28), which would self-disable the fusion, so all three entries stay on nightly-d9105ea8. Keep the VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 knob that #2792 added to the 8K/1K STP and MTP scripts. perf-changelog.yaml stays append-only with the #2851 entries at the tail. 将 main 合并进 DSv4 MI355X FHMoE 分支，解决 #2792 与 #2841 带来的冲突。 这两个 PR 都把 DSv4 MI355X vLLM 镜像升级到了早于 FHMoE 提交 vllm-project/vllm@de69e821 的 nightly（7c5dc571 落后 266 个提交，e962733e 落后 28 个），会导致该融合自动关闭，因此三个条目继续固定在 nightly-d9105ea8。 保留 #2792 为 8K/1K STP 与 MTP 脚本新增的 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 开关。perf-changelog.yaml 保持仅追加，#2851 的条目位于文件末尾。 Co-authored-by: Cursor Agent <cursoragent@cursor.com>
4 participants
Summary
Update both fixed-sequence DeepSeek-V4-Pro FP4 MI355X vLLM configurations for the 8k/1k workload:
dsv4-fp4-mi355x-vllm(STP)dsv4-fp4-mi355x-vllm-mtp(MTP)Image update:
vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faavllm/vllm-openai-rocm:nightly-7c5dc571cbd1064ecc8a9b1045637ff647aa22cbsha256:f0bdaf5217a09949842b45c1ea1f12260d3205ec81f143b320dfc2eb3ec95e552026-09-01T05:31:02Z7c5dc571The 8k/1k TP8 concurrency range remains unchanged at 4-512 for both STP and MTP.
vLLM DSV4 optimization review
The new nightly includes the following relevant changes merged after the 2026-08-17 nightly used by #2590:
The newly merged DSV4 kernels are selected automatically through the existing
VLLM_ROCM_USE_AITER=1,VLLM_ROCM_USE_AITER_MOE=1, and--moe-backend aitersettings. For parity with the latest ROCm recipe, both STP and MTP now also set:VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1. The currentdeepseek-ai/DeepSeek-V4-Procheckpoint exposes FP4 routed experts with an FP8 block-quantized shared-expert path, so the latest vLLM compatibility check self-disables this fusion when the path is not eligible.VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4to enable INT4 quantized quick all-reduce; the full sweep evals cover the accuracy impact.Validation
/home/jiacao/.local/bin/uv run --with 'pydantic>=2' --with pyyaml \ python3 utils/matrix_logic/generate_sweep_configs.py full-sweep \ --config-files configs/amd-master.yaml \ --model-prefix dsv4 \ --framework vllm \ --precision fp4 \ --runner-type mi355x \ --min-conc 4 --max-conc 512 \ --seq-lens 8k1k --no-evalsgit diff --checkpasses.中文说明
更新 MI355X 上 DeepSeek-V4-Pro FP4 的两条固定序列 8k/1k vLLM 配置：
dsv4-fp4-mi355x-vllm（STP）dsv4-fp4-mi355x-vllm-mtp（MTP）镜像变更：
vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faavllm/vllm-openai-rocm:nightly-7c5dc571cbd1064ecc8a9b1045637ff647aa22cbsha256:f0bdaf5217a09949842b45c1ea1f12260d3205ec81f143b320dfc2eb3ec95e552026-09-01T05:31:02Z7c5dc571STP 和 MTP 的 8k/1k TP8 并发范围均保持 4-512 不变。
vLLM DSV4 优化核对
与 #2590 使用的 2026-08-17 nightly 相比，新镜像包含以下相关优化：
新合入的 DSV4 kernel 会通过现有的
VLLM_ROCM_USE_AITER=1、VLLM_ROCM_USE_AITER_MOE=1和--moe-backend aiter自动选用。为了与最新 ROCm 配方保持一致，STP 和 MTP 现在还会显式设置：VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1。当前deepseek-ai/DeepSeek-V4-Pro检查点采用 FP4 routed experts 与 FP8 block-quantized shared-expert 路径；当该路径不满足条件时，最新 vLLM 的兼容性检查会自动关闭此融合。VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4，启用 INT4 量化 quick all-reduce；完整扫描中的准确率评估会覆盖其精度影响。中文验证
git diff --check通过。Note
Low Risk
Benchmark and config pinning only; changes affect perf harness env vars and container image, not application auth or data paths.
Overview
Repins the fixed-sequence DeepSeek-V4-Pro FP4 MI355X vLLM benchmarks
dsv4-fp4-mi355x-vllm(STP) anddsv4-fp4-mi355x-vllm-mtp(MTP) to ROCm nightly7c5dc571, replacing the older09663abdedigest so sweeps pick up newer automatic DSV4 ROCm kernels (mHC/RMSNorm fusion, C4A top-k, compressor GEMMs, SWA quant fusion, cooperative top-k tuning).STP and MTP launch scripts now export
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1andVLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4alongside the existing AITER flags, matching the updated recipe notes inamd-master.yaml; shared-expert fusion is expected to no-op for this checkpoint when ineligible.perf-changelog.yamlrecords the image bump, optimization pickup, and new env knobs for thefixed-seq-lenscenario on both config keys.Reviewed by Cursor Bugbot for commit 8ae9c25. Bugbot is set up for automated code reviews on this repo. Configure here.