Conversation
Pin lmcache 0.5.5rc5+rocm7.2 from the versioned v0.5.5rc5-rocm release. rc5 is the merge commit of LMCache #4936, which fixes the DCP cache identity: the scheduler and the workers derived different model names, so every lookup missed and the DRAM tier was write-only at concurrency 44-52. The previous pin also no longer resolves. nightly-rocm keeps only its newest wheel, so 0.5.5.dev89+rocm7.2 disappeared when that tag rebuilt. Also derive the MP server ports from $PORT rather than the fixed 6555/8090, and raise the L1 read-lock TTL to 3600s.
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
sammshen
added
AMD
agentx
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Looks good — straightforward dependency re-pin plus port/TTL fixes for the Kimi-K3 MI355X LMCache benchmark. Reviewed: version-string parity between the script and configs/amd-master.yaml, the $PORT-derived LMCache port arithmetic against how $PORT is already made unique per concurrent runner cell (runners/launch_mi355x-amds.sh), and the perf-changelog.yaml entry — it's a pure tail append using the exact pull/XXX placeholder convention documented in CONTRIBUTING.md/utils/validate_perf_changelog.py.
Extended reasoning...
Overview
The diff touches three files for a single logical change: re-pinning the LMCache KV-offload dependency used by the Kimi-K3 FP4 MI355X agentic-MTP benchmark from a nightly dev build to an immutable versioned pre-release (0.5.5rc5+rocm7.2), moving the LMCache MP-server ports to be derived from $PORT, raising --l1-read-ttl-seconds to 3600, and appending a matching perf-changelog.yaml entry. No source/runtime logic outside this one benchmark path is touched.
Security risks
None identified. This is a version pin and shell arithmetic change in a benchmark script plus a YAML config/changelog update — no auth, crypto, or permission-sensitive code paths are involved. The LMCache index URL is switched from a nightly to a versioned immutable release tag, which if anything reduces supply-chain risk (immutable artifact vs. a rolling "latest wheel" tag).
Level of scrutiny
Low-to-medium is appropriate: config/version-pin changes with matching recipe updates are the well-established, mechanical pattern this repo uses per AGENTS.md/CONTRIBUTING.md, and the change is self-contained to one benchmark cell. I did verify the two substantive risk points myself: (1) the version string is identical across the benchmark script and configs/amd-master.yaml (both updated together as required), and (2) the $PORT-derived port arithmetic (PORT+13000, PORT+14000) is safe because $PORT is already guaranteed unique per concurrently-running cell on a shared runner (runners/launch_mi355x-amds.sh derives it from RUNNER_NAME), which is the same invariant the main vLLM server's --port "$PORT" already relies on — so the new LMCache ports inherit that uniqueness without introducing a new collision class.
Other factors
The perf-changelog.yaml diff is a clean tail-append (git diff shows only additions after the prior last entry, no altered bytes), and the pull/XXX placeholder exactly matches PR_LINK_PLACEHOLDERS in utils/validate_perf_changelog.py and the example in CONTRIBUTING.md, so it is not something requiring hand-editing. I also confirmed the --use-chat-template convention mentioned in the task's "conventions noted" does not apply to this file — this script calls run_eval, not run_benchmark_serving (the same pattern as every other *_mtp.sh file under benchmarks/single_node/agentic/), so that note was a false lead for this particular script. Given the mechanical nature of the change, consistency with existing patterns, and no unresolved objections in the PR timeline, this is a reasonable approve candidate; I'm choosing not to add a "what was reviewed" style defer/approve verdict beyond a light one since nothing here appears to need deeper human scrutiny.
This review covers commit 565db2a, which is no longer the latest commit on this pull request; later commits are not covered by it.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33931292450 |
The worker pings the MP server with a timeout equal to the heartbeat interval, and one timeout is enough to enter degraded mode. At the 10s default, the 42s and 55s server stalls seen on c4 and c48 dropped every worker, and the recovery that follows re-registers KV caches and leaves in-flight loads invalid. vLLM's recovery path for those invalid blocks does not support hybrid models, so it kills EngineCore on Kimi-K3. Avoid entering it at all: 60s heartbeat interval, and a 300s worker reap timeout to stay above 3x the interval.
…-dcp-retrieve-fix # Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34300098540 |
Move concurrency 44/48/52 into kimik3-fp4-mi355x-vllm-agentic-dcp-mtp at dram-utilization 0.40 (1199 GB). The shared 0.60 block reserved 1799 GB against a measured 698 GB peak, and LMCache pins that unswappably in ~9 GB steps: the ramp stalls the server past the heartbeat deadline and the residual pressure stalls the host. Pin lmcache 0.5.5rc5+rocm7.2 from the versioned v0.5.5rc5-rocm release for the DCP cache-identity fix (LMCache #4936); the previous nightly-rocm dev89 pin no longer resolves. Derive the MP server ports from $PORT, raise the L1 read-lock TTL to 3600s, and raise the heartbeat interval to 60s with a 300s worker reap timeout. 将并发 44/48/52 拆分到 kimik3-fp4-mi355x-vllm-agentic-dcp-mtp，并将 dram-utilization 降至 0.40（1199 GB）。共享的 0.60 配置预留了 1799 GB， 而实测峰值仅为 698 GB；LMCache 会以约 9 GB 为步长将其锁定为不可交换内存： 锁定过程会使服务器停顿并超过心跳超时，残留的内存压力还会导致整台主机停顿。 将 lmcache 固定为版本化 v0.5.5rc5-rocm 发布中的 0.5.5rc5+rocm7.2，以获得 DCP 缓存标识修复（LMCache #4936）；此前的 nightly-rocm dev89 固定版本已无法解析。 将 MP 服务器端口改为从 $PORT 派生，将 L1 读锁 TTL 提高到 3600 秒， 并将心跳间隔提高到 60 秒，同时将 worker 回收超时设为 300 秒。
sammshen
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34302035958 |
|
Hi @sammshen, thanks for the contribution to InferenceX. As today is the last day for the DSv4 8K1K scenario, we need to prioritize the related runs. We canceled your run and will resume it later. According to the current queue, it will be delayed by around 8 hours |
…-dcp-retrieve-fix # Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34399043668 |
Pairs with the existing lmcache point at the same concurrency, so the DRAM tier's throughput contribution reads directly off the two. Keeping it in the DCP config key holds the sweep at 4 points; adding it to the no-offload arm of kimik3-fp4-mi355x-vllm-agentic-mtp would have pulled that key's whole curve in, since append-only cannot mix with a non-append-only entry. 与同一并发下已有的 lmcache 数据点配对，从而可直接由两者读出 DRAM 层对吞吐的 贡献。将其保留在 DCP config key 中可使本次 sweep 维持在 4 个数据点；若添加到 kimik3-fp4-mi355x-vllm-agentic-mtp 的 no-offload 配置中，则会连带触发该 key 的整条曲线，因为 append-only 无法与非 append-only 条目混用。
66dff98 to
2216f82
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34411479662 |
| - dram-utilization: 0.40 | ||
| search-space: | ||
| - { tp: 8, ep: 1, dcp-size: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.5rc5+rocm7.2" }, conc-list: [44, 48, 52], spec-decoding: mtp } | ||
| - { tp: 8, ep: 1, dcp-size: 8, kv-offloading: none, conc-list: [52], spec-decoding: mtp } |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34411759033 |
…-dcp-retrieve-fix # Conflicts: # benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh # configs/amd-master.yaml # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34551846607 |
| @@ -638,7 +638,27 @@ kimik3-fp4-mi355x-vllm-agentic-mtp: | |||
| search-space: | |||
| - { tp: 8, ep: 1, dcp-size: 1, kv-offloading: none, conc-list: [1] , spec-decoding: mtp} | |||
| - { tp: 8, ep: 1, dcp-size: 1, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.5.dev114+rocm7.2" }, conc-list: [4, 8, 10, 12, 14], spec-decoding: mtp } | |||
There was a problem hiding this comment.
🟡 (optional) This dram-offload arm still declares kv-offload-backend version "0.5.5.dev114+rocm7.2", but the shared launcher script now hardcodes LMCACHE_VERSION=0.5.5rc5+rocm7.2 for every kv-offloading:dram invocation regardless of the config's version field, so this arm will actually install/run rc5 while its provenance/dashboard label still says dev114. Fix: keep the installed lmcache version and the declared kv-offload-backend.version in sync for every arm sharing kimik3_fp4_mi355x_mtp.sh -- either have the script read/validate against the config's version or bump this field to 0.5.5rc5+rocm7.2 too. [also at: benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh:152 - LMCACHE_VERSION is one hardcoded constant shared by every lmcache-offload arm, so bumping it to 0.5.5rc5+rocm7.2 also silently changes what installs for the untouched non-DCP arm in kimik3-fp4-mi355x-vllm-agentic-mtp (conc 4/8/10/12/14), whose config still declares kv-offload-backend.version…]
Extended reasoning...
Before this diff, both the base arm (conc 4/8/10/12/14) and the DCP8 arm declared "0.5.5.dev114+rocm7.2", matching the script's then-hardcoded LMCACHE_VERSION, so provenance was accurate. This diff bumps only the script's hardcoded LMCACHE_VERSION (line 152) and LMCACHE_ROCM_INDEX to rc5, and updates the new dcp-mtp config key's version field to rc5, but leaves the untouched base arm's search-space entry at line 640 declaring dev114. require_agentic_kv_offload_backend only checks the backend NAME ('lmcache'), never the version, so nothing catches the mismatch. Any run of the base arm now installs and runs lmcache 0.5.5rc5+rocm7.2 while every downstream record (perf-changelog cross-reference, matrix labeling, CONFIGS.md's 'version must be the component's exact release') still shows dev114, per CONFIGS.md's requirement that this field be exact.
Verification: normal. The launcher hardcodes the installed LMCache version and the config's version field is provenance-only, so this diff makes the base arm's declared version diverge from what actually runs. - benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh:152-153 now hardcodes LMCACHE_VERSION=0.5.5rc5+rocm7.2 (and the rc5 index) unconditionally for every lmcache / kv-offloading: dram…
2 participants
Runs only the DCP8 points (concurrency 44/48/52), now in their own config key.
What changed
dram-utilization0.60 → 0.40 (1799 GB → 1199 GB). Measured peak was 698 GB. LMCache pins the whole budget unswappably in ~9 GB steps; that ramp stalls the server past the heartbeat deadline, and 60% of the node pinned stalls the host.0.5.5.dev89+rocm7.2→0.5.5rc5+rocm7.2, from the versionedv0.5.5rc5-rocmrelease. It carries LMCache#4936, without which the scheduler and workers derive different cache keys under DCP and the DRAM tier is write-only. The old pin no longer resolves —nightly-rocmkeeps only its newest wheel.$PORT(were hardcoded 6555/8090, which collide between cells on one runner).Sweep scope
Splitting the arm into its own config key schedules 3 points instead of 9.
_matrix_visual_series_keydoes not include the config key, so the dashboard curve is unaffected.Known issue, not fixed here
When LMCache degrades it flags in-flight loads for recompute, which is correct. vLLM's recovery path
_update_requests_with_invalid_blocksassumes a single KV cache group and raisesValueError: too many values to unpack (expected 1)on Kimi-K3's four groups, killing EngineCore. This PR reduces how often LMCache degrades; it does not close that path.仅运行 DCP8 数据点（并发 44/48/52），现已拆分到独立的 config key 中。
变更内容
dram-utilization0.60 → 0.40（1799 GB → 1199 GB）。实测峰值为 698 GB。LMCache 会以约 9 GB 为步长将整个预算锁定为不可交换内存；该锁定过程会使服务器停顿并超过心跳超时，而锁定节点 60% 的内存还会导致整台主机停顿。0.5.5.dev89+rocm7.2→0.5.5rc5+rocm7.2，来自版本化的v0.5.5rc5-rocm发布。该版本包含 LMCache#4936；若缺少此修复，DCP 下 scheduler 与 worker 会推导出不同的缓存键，DRAM 层将只写不读。此前的固定版本已无法解析——nightly-rocm仅保留最新的 wheel。$PORT派生（原为硬编码的 6555/8090，同一 runner 上的多个单元会冲突）。Sweep 范围
将该配置拆分到独立的 config key 后，只会调度 3 个数据点而非 9 个。
_matrix_visual_series_key不包含 config key，因此仪表盘曲线不受影响。已知问题（本 PR 未修复）
LMCache 降级时会将进行中的加载标记为需要重算，这是正确行为。但 vLLM 的恢复路径
_update_requests_with_invalid_blocks假定只有一个 KV cache group，在 Kimi-K3 的四个 group 上会抛出ValueError: too many values to unpack (expected 1)，导致 EngineCore 崩溃。本 PR 降低了 LMCache 降级的频率，但并未消除该路径。Note
Low Risk
Benchmark and config-only changes for MI355X agentic sweeps; no production serving code paths, though LMCache version and timeout tweaks affect benchmark stability under high concurrency.
Overview
Splits the Kimi-K3 MI355X DCP8 + LMCache agentic sweep into a new config key
kimik3-fp4-mi355x-vllm-agentic-dcp-mtp, withdram-utilization0.40 (down from 0.60 on the shared matrix) so LMCache’s pinned DRAM ramp does not stall the host past MP heartbeats. The basekimik3-fp4-mi355x-vllm-agentic-mtpentry drops the DCP8 points; the new key runs DCP8 LMCache at conc 44/48/52 and adds a GPU-only conc 14 baseline for A/B against the existing LMCache conc 14 arm.Updates
kimik3_fp4_mi355x_mtp.shfor LMCache: pins0.5.5rc5+rocm7.2from the versioned ROCm release (DCP cache-identity fix; replaces the unresolvable nightly pin), derives MP/HTTP ports from$PORT, sets L1 read TTL 3600s, worker reap 300s, andlmcache.mp.heartbeat_interval60s in the vLLM connector config. Documents the new key inperf-changelog.yaml.Reviewed by Cursor Bugbot for commit cc03338. Bugbot is set up for automated code reviews on this repo. Configure here.