[AgentX] DeepSeek-V4 B200 SGLang update - #2703
Conversation
…terval Retune the existing B200 SGLang AgentX recipe on the 0820 nightly. Use MegaMoE with the FP4 indexer and bounded prefill delay for DP attention, preserve an 8192-token per-rank chunk budget, and size decode CUDA graphs through the configured maximum concurrency 196. Keep the published TP-only rows, remove the DP no-offload arm, and expand the DEP8 HiCache sweep through concurrency 196.
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Claude finished @nvpohanh's task in 2m 22s —— View job PR Review
LGTM - no blocking issues found Checks performed: master-config edit is paired with a |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM, straightforward AgentX benchmark recipe retune.
What was reviewed: the DP-attention MoE backend switch from deepep to megamoe (env vars + CLI flags moved consistently), the chunked-prefill-size formula change (8192 * TP generalizes the prior fixed 65536 for TP=8, matching the stated per-rank budget), the new cuda-graph-max-bs-decode override path for DP attention, and the config/perf-changelog sweep-point updates. Also confirmed the removed MODEL_ARGS array and its former conditional flag are still applied (now inlined in both TP-only and DP-attention branches).
Extended reasoning...
Overview
This PR retunes the existing B200 SGLang AgentX benchmark recipe for DeepSeek-V4-Pro: it switches the DP-attention MoE backend from DeepEP to MegaMoE with FP4 activations, adds a bounded prefill-delayer for DP attention, generalizes the chunked-prefill-size formula, adds a decode-specific CUDA graph max batch size for DP attention, bumps the SGLang image to a dev nightly, and updates the concurrency sweep list in configs/nvidia-master.yaml plus the corresponding perf-changelog entry.
Security risks
None. This only touches a benchmark launcher shell script, a benchmark sweep config YAML, and a changelog — no production inference code, auth, or data handling paths are affected.
Level of scrutiny
Low-to-moderate is appropriate: this is benchmark/tuning infrastructure, not a critical serving path, and the PRs content is already merged into main as commit cf000c1 (this PR appears to be the source PR being backfilled with its link in a follow-up commit). The bug-hunting pass found nothing, and my own read of the diff found the env-var/flag migrations and formula changes to be internally consistent (e.g., 8192*8=65536 preserves the prior TP8 behavior; the new cuda-graph-max-bs-decode 196 matches the new max concurrency in the sweep list).
Other factors
No CODEOWNER-restricted files are touched beyond the standard NVIDIA AgentX recipe files, and no outstanding reviewer comments are pending in the timeline.
…tx-megamoe-delayer # Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32443504991 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32454316647 |
移除 B200 SGLang AgentX DEP8 HiCache 扫描中的 c128、c160 和 c196 点。
|
/reuse-sweep-run 32454316647 |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:Upstream recipe (single-node requirement). The published SGLang cookbook page https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4 covers this recipe and is already the cited upstream source for the sibling
Two flags are not on the page and I am recording them as InferenceX-specific per-lane tuning rather than major deployment args: One deliberate deviation from the upstream recipe, driven by InferenceX policy. The cookbook wires MegaMoE into its high-throughput preset, whose documented spec-decode setting is "MTP disabled — at saturation the verify step costs more than it saves". This recipe composes MegaMoE with EAGLE MTP because Validation run. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/32454316647 on commit Evals. gsm8k Golden AL. Image provenance. Changelog. New entry appended at the physical end of Non-blocking, but @nvpohanh please read — I would rather see this hardened than merely green. The c128/c160/c196 failures are a genuine HBM ceiling, not flakes: all three died with the same signature on three different runners, a CUDA OOM inside the DSv4 FP4 indexer during chunked prefill, followed by a DP-attention-wide wedge that burned the full four-hour AIPerf warmup grace ( Two smaller notes. Signed: |
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
Ankur-singh
left a comment
Ankur-singh
left a comment
There was a problem hiding this comment.
Approving — CODEOWNER sign-off posted above and verified PASS by codeowner-signoff-verify (12 PASS / 1 N/A). Reuse is pinned to run 32454316647, so no re-run is needed.
Nice work on the golden-AL handling in particular: the EVAL_ONLY guard around SGLANG_SIMULATE_ACC_* is verifiably correct in the artifacts, not just in the code — throughput server logs carry all three variables and the eval logs carry none.
The one thing I would still like to see followed up, as noted in the sign-off and not blocking this merge: CUDA_GRAPH_MAX_BS = 2 * CONC is now unbounded, which is what pushed c128/c160/c196 past the HBM ceiling, and the surviving c96 point logged 213 allocator OOM retries. A cap (or a lower mem-fraction-static on the DP-attention arm) would be more durable than a shorter concurrency list.
|
/stage-results 32454316647 |
|
@cquil11 staged run 32454316647: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-21~r32454316647 This run remains available across future |
4 participants
DeepSeek-V4 B200 SGLang update on Agentx