[AgentX GB200] enable Kimi-K3 power on the remaining recipes and re-sweep the full curve / 为剩余配方启用实测功耗并重跑完整曲线 - #3287
edwingao28 wants to merge 5 commits into
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Looks good — straightforward, mechanical config change that exactly mirrors the already-merged pattern from #3045. Reviewed: the four new telemetry: blocks and ENABLE_AGENTX_POWER/REQUIRE_POWER env entries byte-match the schema used in the seven sibling recipes; confirmed benchmark_lib.sh already validates these two env vars via check_env_vars (no launcher/script change needed); confirmed configs/nvidia-master.yaml needs no update since the single kimik3-fp4-gb200-dynamo-vllm-agentic job key already covers all four recipe files; perf-changelog.yaml entry is a clean append-only addition with a valid config-key.
Extended reasoning...
Overview
This PR touches four GB200 Kimi-K3 vLLM AgentX srt-slurm recipe YAMLs plus perf-changelog.yaml. Each recipe gets an identical, additive 14-line telemetry: block (dcgm_exporter power collection config) and two new benchmark.env entries (ENABLE_AGENTX_POWER, REQUIRE_POWER). No launcher, script, or master-config code is touched. This directly extends a pattern from the immediately preceding commit (8d70414, PR #3045) which added the identical block to seven sibling recipes in the same family.
Security risks
None. This is pure declarative YAML configuration for benchmark telemetry collection (interval, timeout, exporter image/port) — no code execution paths, auth, or secrets are introduced or modified.
Level of scrutiny
Low-to-moderate scrutiny is appropriate: the change is config-only, small, and mechanically identical across the four files, with a strong existing precedent to compare against. I verified the new telemetry: blocks are byte-identical to the schema already used in the seven sibling recipes (e.g. agg-dcp16-nospec-mooncake.yaml), confirmed via configs/nvidia-master.yaml that a single job key kimik3-fp4-gb200-dynamo-vllm-agentic already covers all four affected recipe paths (so no master-config coupling is needed, consistent with the prior 7-recipe PR which also didn't touch nvidia-master.yaml), and confirmed benchmarks/benchmark_lib.sh already validates ENABLE_AGENTX_POWER/REQUIRE_POWER via check_env_vars with no fallback defaults, so the new env vars are properly gated at the consuming end. The perf-changelog.yaml entry is appended at the tail without disturbing prior bytes, uses a config-key that matches the actual job key in nvidia-master.yaml, and follows the bilingual description convention used by all other entries.
Other factors
The pr-link: .../pull/XXX placeholder is expected at this stage since the CONTRIBUTING.md changelog template itself shows this exact placeholder pattern for use before the PR number is known — not a defect, just something the author fills in once the PR exists. No CODEOWNERS-restricted paths are touched (nvidia-master.yaml, the only owned file relevant to this area, is unmodified). The PR description's own stated testing status (matrix/changelog checks only, full-sweep hardware validation still pending) is a deployment/rollout concern rather than a code-correctness one, and doesn't change the assessment that the diff itself is simple and correct.
This review covers commit 1366206, which is no longer the latest commit on this pull request; later commits are not covered by it.
edwingao28
added
the
priority
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35492708340 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35492708340 |
edwingao28
added
skip_queue
ci-patchwork-waived
full-sweep-enabled
and removed
full-sweep-fail-fast
full-sweep-enabled
labels
edwingao28
changed the title
… cadence Run 35492708340 showed every completed point of the 500 ms / 1 s lanes failing publication with sample_gap_exceeded while the seven 1000 ms / 2 s Kimi-K3 recipes on the same fleet passed. The misses are all on the head node, whose dcgm-exporter shares the node with the frontend, NATS/etcd and the AIPerf client and answers in 1.0-1.6 s under load; a 1 s request timeout censors that tail into repeated timeouts (242 'broken pipe' late responses in the exporter log of the TP16 c1 job). The collector is fixed-cadence, so the earlier 'sleep after scrape' premise for 500/1 was also wrong. Restore 1000/2, matching the passing siblings, and rewrite the budget test as invariants that are true on their own: same cadence as the passing sibling, request timeout above the observed exporter tail, interval inside the coverage limit, join budget covering two cycles.
5338791 to
e318545
Compare
One sampling gap past MAX_SAMPLE_GAP_SECONDS rejects the window on every GPU of the job. This lane lost every point that way: one node's exporter answered in 3.26 s and 3.10 s during a 3640 s window, 0.175% of it, with the benchmark reporting no error. The lane that passed in the same run had a larger 3.22 s excursion and survived only because it landed in warmup, outside the window, so which points publish is chance rather than data quality. Baseline cadence is healthy on both: p99 gap 1.03-1.05 s. An over-long gap is interpolated coverage, not a corrupt measurement, and its per-device energy error is bounded by the dynamic range times the gap: under 0.04% for 3.3 s of a 3640 s window. A collector that actually stopped is a different failure and still has to be caught, so reject the window when one gap passes MAX_SAMPLE_GAP_HARD_SECONDS (10 s), or when the time inside over-long gaps passes MAX_OVERLONG_GAP_FRACTION (0.5%) of the window. The budget is a fraction, so a short window still rejects the gap a long one absorbs. The rule only ever relaxes, so no already-published point can regress. The stored audit still reports the largest gap per device, unchanged. The gap rule is a producer contract, so a consumer-only relaxation would mismatch the stored verdict and fail the point as package_recompute_invalid. The submodule pin advances exactly one commit from main's 984180e5 -- that commit is its parent, not the fork's main, so no other srt-slurm change enters the runtime. Tracked in SemiAnalysisAI/srt-slurm#23; move the pin to the merged sha before this merges. Verified by replaying the four retained H200 power packages of run 35532102407: the two rejected points now pass with no reason codes, the two that already passed are unchanged. 中文：将超长功耗采样间隙改为预算制，不再因单次间隙作废整个基准点。此前只要有一次 采样间隙超过 MAX_SAMPLE_GAP_SECONDS，整个作业所有 GPU 的窗口都被判废。本条 lane 的每个点都是这样丢的：某节点的 exporter 在 3640 秒窗口内有两次 3.26 秒和 3.10 秒 的响应，仅占 0.175%，而 benchmark 本身没有报错。同一次运行中通过的那条 lane 反而 有更大的 3.22 秒抖动，只因落在 warmup、窗口之外而幸免——哪些点能发布取决于运气而 非数据质量。两条 lane 的基线节奏都健康，p99 间隙 1.03-1.05 秒。 超长间隙属于插值覆盖而非损坏的测量，其单设备能量误差上界为动态范围乘以间隙长度， 3640 秒窗口中的 3.3 秒间隙低于 0.04%。采集器真正停止是另一类故障，仍需拦截，因此 改为：单次间隙超过 MAX_SAMPLE_GAP_HARD_SECONDS（10 秒），或超长间隙占用时间超过 窗口的 MAX_OVERLONG_GAP_FRACTION（0.5%）时拒绝该窗口。预算按比例计算，因此短窗口 仍会拒绝长窗口能够吸收的间隙。 该规则只会放宽，已发布的点不会回退。存储的审计记录仍按设备报告最大间隙，保持不变。 间隙判定属于生产端契约，只放宽消费端会与存储结论不一致，该点会以 package_recompute_invalid 失败。submodule pin 相对 main 的 984180e5 只前进一个 commit——该 commit 正是其父提交，而非 fork 的 main，因此不会引入任何其他 srt-slurm 改动。跟踪于 SemiAnalysisAI/srt-slurm#23；本 PR 合并前需将 pin 改为合并后的 sha。 验证方式：用 run 35532102407 留存的四个 H200 功耗包回放——两个原本被拒的点现在无 任何 reason code 通过，两个原本通过的结果不变。
Both GB200 DEP16 recipes set max-num-batched-tokens: 16384; every GB200 Kimi-K3 recipe that survived run 35492708340 uses 8192. FlashInfer's MXFP4 MoE workspace is allocated per forward, outside the memory profiler's budget, and scales with the batch's token count. At 16384 it requested 3.64 GiB against 3.50 GiB free (177.50 of 184.00 GiB in use) and killed rank DP1/TP3/EP7 24 minutes into the run. The surviving ranks hung in a collective until the NCCL heartbeat monitor aborted the job, which is why this reads as a mid-run engine death rather than an allocation failure. TP4/DP4 keeps four times the non-expert weights per GPU that a TP16 recipe does, so the headroom the larger chunk needs is not there; agg-tep16-balanced and agg-tp16-latency run the same kernel at 8192 and complete. Halving the chunk rather than lowering gpu-memory-utilization leaves KV cache capacity untouched. 中文：两个 GB200 DEP16 配方的 max-num-batched-tokens 为 16384，而 run 35492708340 中存活的 GB200 Kimi-K3 配方全部使用 8192。FlashInfer 的 MXFP4 MoE workspace 按每次 forward 分配，不在内存 profiler 的预算内，且大小随 batch 的 token 数增长：在 16384 下它申请 3.64 GiB，而当时只剩 3.50 GiB free（184.00 GiB 中已占用 177.50 GiB），在开跑 24 分钟后杀掉了 DP1/TP3/EP7 这个 rank。其余 rank 随后卡在集合通信上，直到 NCCL 心跳监控 终止作业——这也是它表现为运行中途引擎死亡、而非一次分配失败的原因。 TP4/DP4 每张卡驻留的非专家权重是 TP16 配方的四倍，没有更大 chunk 所需的 余量；agg-tep16-balanced 与 agg-tp16-latency 在 8192 下运行同一个 kernel 且能跑完。选择砍 chunk 而不是降 gpu-memory-utilization，是为了不影响 KV cache 容量。
1 participant
Description
The four GB200 Kimi-K3 AgentX recipes without power collection (
agg-tp16-latency,agg-tep16-balanced,agg-dep16,agg-dep16-vllm-simple-offload) now carry thetelemetry:block from #3045 at the fleet-standard 1000 ms / 2 s, the same cadence as the seven GB200 Kimi-K3 recipes that already pass the power gate. Also keepsinfx.results.powerimportable under the runners' python3.10.Those four recipes are the single
kimik3-fp4-gb200-dynamo-vllm-agentickey. AgentX resolves one curve per hardware, so the changelog selects all four GB200 keys and the sweep re-measures all 31 points.What run 35492708340 (head
5338791, which already included the 500 ms / 1 s commit885d227) showed. 19 of 31 jobs failed, in three groups:sample_gap_exceeded: TP16 c1/c2/c4/c8, TEP16 c16/c24/c32, vllm-simple c192, tp8pp2-mooncake c72/c96.max-num-batched-tokens: 16384.ProfileAbortedwithmeasurement_window_incomplete, no OOM and no worker crash in any of its 16 logs. It is an aiperf-side abort, not an engine failure.endpoint_http_errorappears in all 29 power manifests, passing jobs included; it is the collector's first polls racing the exporter start and does not gate.The gated points. All misses are on the head node, whose dcgm-exporter shares the node with the frontend, NATS/etcd, the AIPerf client and the collector; it answers in 1.0–1.6 s under load (
max_scrape_duration_secondson the passing 1000/2 lanes, 242write: broken pipelate responses in the TP16 c1 exporter log). A 1 s request timeout turns that tail into repeated misses, and the collector is fixed-cadence, so the "sleeps after each scrape" premise behind 500/1 was wrong as well. This PR restores 1000/2 (e318545).Restoring the cadence is not sufficient on its own:
_check_coveragevoids a window as soon as one device's largest gap passesMAX_SAMPLE_GAP_SECONDS, with no budget, so a single late scrape still discards every GPU of the job.b63c1768adds a ceiling plus a fractional budget and pins the matching producer change, SemiAnalysisAI/srt-slurm#23, until it merges. The pin advances exactly one commit from main's984180e5. That change also covers the tp8pp2-mooncake c72/c96 stalls, which no longer need separate srt-slurm work.The DEP16 OOM. FlashInfer's MXFP4 MoE workspace is allocated per forward, outside the memory profiler's budget, and scales with the batch's token count. At 16384 it requested 3.64 GiB against 3.50 GiB free (177.50 of 184.00 GiB in use) and killed rank DP1/TP3/EP7 24 minutes into DEP16 c32; the surviving ranks then hung in a collective until the NCCL heartbeat monitor aborted the job, which is why these read as mid-run engine deaths rather than allocation failures. TP4/DP4 keeps four times the non-expert weights per GPU that a TP16 recipe does, so the headroom the larger chunk needs is not there —
agg-tep16-balancedandagg-tp16-latencyrun the same kernel at 8192 and complete.1731889drops both DEP16 recipes to 8192, which every surviving GB200 recipe already uses and which leaves KV cache capacity untouched. This supersedes the memory follow-up in #3322.Testing: changelog, matrix plan (31 benchmark + 31 eval jobs),
utils/test_gb200_k3_telemetry_budget.pyand the python-floor tests pass locally. For the gap budget,utils/test_aggregate_power_multinode.pypasses with five added cases, the four retained H200 power packages of run 35532102407 replay as expected through the patched_check_coverage, and the producer side is 131 passed with the constants byte-identical on both sides. The OOM classification above is read from the retained artifacts of DEP16 c32 and vllm-simple c128; the other six share both the recipe and the signature. Neither fix has been exercised on GB200 yet — that is what this full sweep is for.中文
GB200 上还没有功耗采集的四个 Kimi-K3 AgentX 配方（
agg-tp16-latency、agg-tep16-balanced、agg-dep16、agg-dep16-vllm-simple-offload）现在使用 #3045 的telemetry:配置，采用集群标准的 1000 ms / 2 s，与已通过功耗闸门的七个 GB200 Kimi-K3 配方一致。同时修复infx.results.power在 runner 的 python3.10 下无法导入的问题。这四个配方就是
kimik3-fp4-gb200-dynamo-vllm-agentic这一个 key。AgentX 每个硬件只解析出一条曲线，因此 changelog 选中全部四个 GB200 key，sweep 重测 31 个点。Run 35492708340（head
5338791，已包含 500 ms / 1 s 的885d227）的实际结果。 31 个 job 中 19 个失败，分三类：sample_gap_exceeded失败：TP16 c1/c2/c4/c8、TEP16 c16/c24/c32、vllm-simple c192、tp8pp2-mooncake c72/c96。max-num-batched-tokens: 16384的两个配方的全部点。ProfileAborted结束，伴随measurement_window_incomplete，其 16 份日志中既无 OOM 也无 worker crash。这是 aiperf 侧的中止，不是引擎失败。endpoint_http_error出现在全部 29 个功耗 manifest 中（含通过的 job），是采集器首轮轮询早于 exporter 就绪的记录，不参与闸门判定。被拦下的点。 所有 miss 都在头节点上，其 dcgm-exporter 与 frontend、NATS/etcd、AIPerf 客户端和采集器共用该节点，负载下响应 1.0–1.6 s（通过的 1000/2 lane 的
max_scrape_duration_seconds；TP16 c1 的 exporter 日志有 242 条write: broken pipe迟到响应）。1 s 超时把这条长尾变成反复 miss；而采集器是固定节奏的，500/1 所依据的"每次抓取后再 sleep"前提也不成立。本 PR 恢复 1000/2（e318545）。只恢复节奏并不够：
_check_coverage只要某个设备的最大间隙超过MAX_SAMPLE_GAP_SECONDS就判废整个窗口，且没有任何预算，因此一次迟到的 scrape 仍会作废该作业所有 GPU 的数据。b63c1768增加了一个上限加一个按比例的预算，并在生产端改动 SemiAnalysisAI/srt-slurm#23 合入前先 pin 该提交；pin 相对 main 的984180e5只前进一个 commit。该改动同时覆盖 tp8pp2-mooncake c72/c96 的停顿，这两个点不再需要单独的 srt-slurm 工作。DEP16 的 OOM。 FlashInfer 的 MXFP4 MoE workspace 按每次 forward 分配，不在内存 profiler 的预算内，且大小随 batch 的 token 数增长。在 16384 下它申请 3.64 GiB，而当时只剩 3.50 GiB free（184.00 GiB 中已占用 177.50 GiB），在 DEP16 c32 开跑 24 分钟后杀掉了 DP1/TP3/EP7 这个 rank；其余 rank 随后卡在集合通信上，直到 NCCL 心跳监控终止作业——这也是它们表现为运行中途引擎死亡、而非一次分配失败的原因。TP4/DP4 每张卡驻留的非专家权重是 TP16 配方的四倍，没有更大 chunk 所需的余量——
agg-tep16-balanced与agg-tp16-latency在 8192 下运行同一个 kernel 且能跑完。1731889把两个 DEP16 配方降到 8192，这正是所有存活的 GB200 配方已在使用的值，且不影响 KV cache 容量。此项取代 #3322 中的显存后续工作。测试： changelog、矩阵规划（31 个 benchmark + 31 个 eval 作业）、
utils/test_gb200_k3_telemetry_budget.py以及 python 版本下限测试在本地通过。gap 预算方面，utils/test_aggregate_power_multinode.py新增五例后全部通过，run 35532102407 留存的四个 H200 功耗包在打过补丁的_check_coverage下回放结果符合预期，生产端 131 passed 且两侧常量逐字一致。上述 OOM 分类读自 DEP16 c32 与 vllm-simple c128 的留存产物，其余六个点配方与特征均相同。两项修复都尚未在 GB200 上验证过——这正是本次完整 sweep 的目的。Related Issue
Follows #3045; supersedes the memory follow-up in #3322 / 承接 #3045；取代 #3322 中的显存后续工作。
Type of Change