fix(amd): enable DSpark speculative decoding for DeepSeek-V4-Pro-0813 MI355X disaggregated AgentX - #3191
fix(amd): enable DSpark speculative decoding for DeepSeek-V4-Pro-0813 MI355X disaggregated AgentX#3191ichbinblau wants to merge 6 commits into
Conversation
(cherry picked from commit ebace4f)
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
Correct the pr-link picked up from the cherry-picked #3170 entry now that this DeepSeek-V4-Pro-0813 DSpark change has its own PR. Signed-off-by: Theresa Shan <theresa.shan@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
ichbinblau
added
AMD
agentx
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| if [[ "${SPEC_DECODING:-}" == "draft_model" ]]; then | ||
| if [[ -z "${MODEL_DSPARK_FLAGS// }" ]]; then | ||
| echo "FATAL: SPEC_DECODING=draft_model but model '${model_name}' has no dspark_flags in models.yaml." >&2 | ||
| exit 1 |
There was a problem hiding this comment.
🟡 (optional) The new fail-fast guard for missing dspark_flags does not actually stop the run: exit 1 at line 389 runs inside build_server_config, which is only ever invoked via command substitution (PREFILL_SERVER_CONFIG=$(build_server_config ...) at line 454). exit there only kills that subshell; with no set -e in this script and no $? check after the assignment, the script prints the FATAL line to stderr but keeps running with PREFILL_SERVER_CONFIG/DECODE_SERVER_CONFIG left empty (the function returns before reaching its final echo "$full_config"), so the server launches with a broken/missing config instead of aborting. Fix: after each build_server_config call, check the exit status (e.g. `PREFILL_SERVER_CONFIG=$(build_server_config ...) …
Extended reasoning...
…|| exit 1`) or have build_server_config return a status the caller checks, so a missing dspark_flags case (or any future error path in this function) actually halts the job for both the prefill and decode invocations at lines 454-455.
build_server_config is called via command substitution at lines 454-455, which bash always runs in a subshell. When SPEC_DECODING=draft_model and MODEL_DSPARK_FLAGS is empty, the function hits exit 1 at line 389 before it ever reaches the trailing echo "$full_config". That exit only terminates the subshell created for the command substitution; the parent server_sglang.sh script has no set -e and never inspects $? right after the assignment, so execution continues. PREFILL_SERVER_CONFIG (and likewise DECODE_SERVER_CONFIG from the second call) end up empty or truncated instead of containing --tp-size/base flags/etc. The script then proceeds to launch the sglang server with this incomplete config, producing a confusing downstream failure (or a badly configured server) instead of the clear early FATAL abort the PR intends for a model without…
Verification: nit. The candidate's mechanism is factually correct. The new fail-fast guard at server_sglang.sh:387-389 does exit 1 inside build_server_config(), but that function is invoked only via command substitution — PREFILL_SERVER_CONFIG=$(build_server_config ...) at line 454 and DECODE_SERVER_CONFIG=$(build_server_config ...) at line 455 — which bash runs in a subshell. The exit 1 therefore… | nit.…
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35111925590 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35111925590 |
2 participants
Description
SPEC_DECODING=draft_model) speculative decoding forDeepSeek-V4-Pro-0813on the MI355X disaggregated AgentX recipe (dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp): newdspark_flagsinmodels.yaml,--speculative-dspark-block-size/--speculative-num-steps 1wiring inserver_sglang.sh, and a fail-fast guard when a model has nodspark_flagsconfigured forSPEC_DECODING=draft_model.lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260913and rerun the disaggregated MTP sweep with DeepSeek-V4-Pro-0813 + DSpark, aligningconfigs/amd-master.yamlwith the internal sweep.AIPERF_EXPERIMENTAL_FAST/REQUIRE_POWERacross container boundaries viaINFERENCEX_RUNTIME_ENV_VARS(cherry-picked from fix(amd): restore native MTP checkpoint and collect all node logs #3170) so workflow-owned AgentX fast-mode and power requirements reach the client container.Follow-up to #2823.
中文
dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp）上的DeepSeek-V4-Pro-0813接入 DSpark（SPEC_DECODING=draft_model）推测解码：models.yaml新增dspark_flags，server_sglang.sh接入--speculative-dspark-block-size/--speculative-num-steps 1，并在模型缺少dspark_flags时快速失败。lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260913，使用 DeepSeek-V4-Pro-0813 + DSpark 重跑分离式 MTP 扫描，并使configs/amd-master.yaml与内部扫描保持一致。INFERENCEX_RUNTIME_ENV_VARS跨容器转发AIPERF_EXPERIMENTAL_FAST/REQUIRE_POWER（从 fix(amd): restore native MTP checkpoint and collect all node logs #3170 cherry-pick），使工作流所有的 AgentX 快速模式与功耗要求能到达客户端容器。