Add MiniMax-M3 NVFP4 GB300 vLLM Disaggregated AgentX with EAGLE3-GQA MTP / 新增 MiniMax-M3 NVFP4 GB300 vLLM 分离式 AgentX EAGLE3-GQA MTP 配置 - #2663
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| command: "bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh" | ||
| env: | ||
| INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace" | ||
| RESULT_DIR: "/logs/agentic" | ||
| PORT: "8000" | ||
| IS_MULTINODE: "true" | ||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache" | ||
| AIPERF_SERVER_METRICS_COLLECTION_INTERVAL: "1.0" | ||
| HF_HUB_CACHE: "/hf_hub_cache" | ||
| WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126" |
There was a problem hiding this comment.
🔴 All 10 new MiniMax-M3 GB300 agentic recipes under gb300-fp4/ omit AIPERF_EXTRA_INPUTS: "thinking:true" from benchmark.env, unlike every existing MiniMax-M3 agentic recipe (e.g. the GB200 sibling disagg-1p1d-dep8-dep4-agentic.yaml line 113, and all agg-*-agentic.yaml files). Since MiniMax-M3 is a reasoning model and this flag is what tells aiperf to exercise thinking mode, these runs will silently benchmark with thinking disabled or defaulted, producing output-length/acceptance-length/latency numbers that are not comparable to the model's real agentic-coding behavior or its own GB200 curve; add the flag to all 10 recipes' benchmark.env blocks before merging.
Extended reasoning...
The bug. Every one of the 10 new recipes under benchmarks/multi_node/srt-slurm-recipes/vllm/minimax-m3/agentic/gb300-fp4/ (5 throughput + 5 eval variants) defines a benchmark.env block that ends at WEKA_LOADER_OVERRIDE and never sets AIPERF_EXTRA_INPUTS. Every pre-existing MiniMax-M3 agentic recipe does set it — confirmed by grep, AIPERF_EXTRA_INPUTS: "thinking:true" appears in all 6 files under minimax-m3/gb200-fp4/agentic/ (the 4 agg-*-agentic.yaml files and disagg-1p1d-dep8-dep4-agentic.yaml line 113), and nowhere else in the repo except benchmarks/benchmark_lib.sh (the consumer).
How the flag is used. benchmark_lib.sh:2052-2053 shows the mechanism directly:
if [ -n "${AIPERF_EXTRA_INPUTS:-}" ]; then
REPLAY_CMD+=" --extra-inputs $AIPERF_EXTRA_INPUTS"
fiThis is a plain conditional with no fallback/default — if the recipe's benchmark.env doesn't set AIPERF_EXTRA_INPUTS, aiperf never receives --extra-inputs thinking:true and there is no other code path that injects it. This is exactly why every existing MiniMax-M3 recipe sets it explicitly: thinking mode is not on by default for this client-side aiperf flag.
Why the shared dataset doesn't save it. One might assume the WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126" dataset implicitly drives thinking behavior, but the direct GB200 sibling recipe (disagg-1p1d-dep8-dep4-agentic.yaml) uses the exact same dataset value AND still sets thinking:true explicitly — so the dataset alone doesn't enable it. reasoning-parser: "minimax_m3" (present in both GB200 and GB300 recipes) is a separate, server-side vLLM response-parsing setting; it does not make aiperf send requests requesting thinking mode.
Why this looks like an authoring accident rather than an intentional change. The new GB300 benchmark.env blocks are structurally different from the GB200 ones — they drop AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING, AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS, and AIPERF_REQUIRED_SERVER_METRIC_PREFIX, while adding AIPERF_SERVER_METRICS_COLLECTION_INTERVAL. This is consistent with the recipes having been templated from the DeepSeek-V4 agentic vLLM recipes (a non-reasoning model family that never sets this flag) rather than from the MiniMax-M3 GB200 agentic pattern. Nothing in the PR description documents disabling thinking mode as an intended change.
Impact. These recipes exist specifically to produce a GB300 curve comparable to the GB200 MiniMax-M3 agentic curve, and to power a "real-verification" eval twin plus a tuned synthetic_acceptance_length: 2.78 for the thinking workload. Benchmarking with thinking silently off means: (1) the throughput numbers (output length, acceptance length, latency) reflect a different workload than intended and aren't comparable to the GB200 sibling; (2) the eval twin verifies accuracy under the wrong mode; (3) the synthetic acceptance length tuning (2.78) was presumably chosen for the thinking workload and may not apply.
Proof walkthrough.
- Recipe
gb300-fp4/1p1d-tep4-tp4-c1-agentic.yamlsetsbenchmark.env(lines 147-157) withWEKA_LOADER_OVERRIDEas the last key — noAIPERF_EXTRA_INPUTS. runners/launch_gb300-nv.shexports everybenchmark.envkey as an environment variable for the agentic benchmark job (bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh).- Inside that flow,
benchmark_lib.sh:2052checks[ -n "${AIPERF_EXTRA_INPUTS:-}" ]— since the var was never exported, this is false, and--extra-inputs thinking:trueis never appended toREPLAY_CMD. - aiperf therefore issues requests without the thinking-mode extra input, while the GB200 sibling recipe (same dataset, same model) does append it and does exercise thinking mode.
- Result: the GB300 run's output-length/acceptance-length/latency distribution reflects non-thinking (or default) behavior, not comparable to the GB200 curve it's meant to sit alongside.
Fix. Add AIPERF_EXTRA_INPUTS: "thinking:true" to the benchmark.env block of all 10 new gb300-fp4 recipe files, matching the GB200 pattern.
| served-model-name: "nvidia/MiniMax-M3-NVFP4" | ||
| kv-cache-dtype: "fp8" | ||
| block-size: 128 | ||
| trust-remote-code: true | ||
| enable-prefix-caching: true | ||
| language-model-only: true | ||
| no-enable-flashinfer-autotune: true | ||
| reasoning-parser: "minimax_m3" | ||
| max-cudagraph-capture-size: 512 |
There was a problem hiding this comment.
🔴 All 10 new gb300-fp4 MiniMax-M3 agentic/eval-agentic recipes set only reasoning-parser: minimax_m3 in prefill and decode, but omit dyn-tool-call-parser: minimax_m3 and dyn-reasoning-parser: minimax_m3 — args every existing MiniMax-M3 dynamo+vllm agentic recipe sets (e.g. gb200-fp4/agentic/disagg-1p1d-dep8-dep4-agentic.yaml lines 71-73, 97-99). Without these, the Dynamo frontend falls back to default tool-call/reasoning parsing for an agentic-coding workload that depends on correct tool-call extraction, which will misparse MiniMax-M3 output (especially in the real-verification -eval twins). Please add both dyn-tool-call-parser: "minimax_m3" and dyn-reasoning-parser: "minimax_m3" to the prefill and decode vllm_config blocks in all 10 new files.
Extended reasoning...
What the bug is: The 10 new benchmarks/multi_node/srt-slurm-recipes/vllm/minimax-m3/agentic/gb300-fp4/*.yaml recipes configure vLLM's engine-side reasoning-parser: "minimax_m3" in both the prefill and decode vllm_config blocks, but never set dyn-tool-call-parser or dyn-reasoning-parser. These two dyn-* keys are distinct, Dynamo-frontend-side directives — they tell the Dynamo frontend (not the vLLM engine) how to extract tool-call and reasoning content out of the raw model output stream before it's returned to the client/benchmark harness. reasoning-parser alone only configures vLLM's own internal parsing; it does not populate the Dynamo frontend's extraction path.\n\nWhere this diverges from precedent: Every existing MiniMax-M3 dynamo+vllm agentic recipe in this repo sets all three keys together, in both prefill and decode blocks. For example, gb200-fp4/agentic/disagg-1p1d-dep8-dep4-agentic.yaml sets reasoning-parser, dyn-tool-call-parser, and dyn-reasoning-parser, all to "minimax_m3", at lines 71-73 (prefill) and 97-99 (decode). Grepping all six gb200-fp4/agentic/*.yaml files (the agg-* and disagg-* siblings) confirms the same triad in every one. Grepping the 10 new gb300-fp4 files for dyn-tool-call-parser/dyn-reasoning-parser returns zero matches — only reasoning-parser (at line 83 prefill / line 103 decode in the representative 1p1d-tep4-tp4-c1-agentic.yaml) is present.\n\nWhy nothing else catches this: There's no schema validation on these recipe YAMLs enforcing that a given reasoning-parser value must be paired with matching dyn-* frontend args — the pairing is purely a convention followed by hand in every prior recipe. Nothing in launch_gb300-nv.sh or the srt-slurm framework injects a default derived from reasoning-parser; omitting the dyn-* keys just means the frontend uses whatever its built-in default parser is (likely none, or a generic one), not MiniMax-M3's actual tool-call format.\n\nImpact: These recipes exist specifically to benchmark an agentic-coding workload, where the benchmark harness needs to correctly extract tool calls from the model's output to measure/verify tool use. Without dyn-tool-call-parser: minimax_m3, the Dynamo frontend will fail to correctly delimit and extract MiniMax-M3's tool-call blocks, corrupting the agentic benchmark's traces. This is doubly important for the five *-eval-agentic.yaml twins, which run real MTP verification and rely on correctly-parsed tool calls to score correctness — a frontend parsing mismatch there would silently produce wrong eval results rather than an obvious crash.\n\nStep-by-step proof:\n1. Open gb200-fp4/agentic/disagg-1p1d-dep8-dep4-agentic.yaml (an existing, presumably-correct MiniMax-M3 dynamo+vllm agentic recipe) and look at the prefill vllm_config block: it sets reasoning-parser: "minimax_m3", dyn-tool-call-parser: "minimax_m3", and dyn-reasoning-parser: "minimax_m3" together (lines 71-73); the decode block repeats the same triad (lines 97-99).\n2. Now open the new 1p1d-tep4-tp4-c1-agentic.yaml added by this PR: the prefill block (around line 83) has only reasoning-parser: "minimax_m3"; the decode block (around line 103) has only the same single key. dyn-tool-call-parser and dyn-reasoning-parser do not appear anywhere in the file.\n3. Repeating this diff across the other 9 new gb300-fp4 files (both agentic and eval-agentic twins for each of the 5 topologies) shows the identical pattern: zero occurrences of either dyn-* key.\n4. Since reasoning-parser is vLLM-engine-scoped and the dyn-* keys are Dynamo-frontend-scoped (as evidenced by every other MiniMax-M3 recipe setting them independently and together), the frontend in these 10 new recipes has no MiniMax-M3-specific tool-call/reasoning extraction configured and falls back to its default behavior — which will misparse this model's actual output format.\n\nFix: Add dyn-tool-call-parser: "minimax_m3" and dyn-reasoning-parser: "minimax_m3" alongside reasoning-parser: "minimax_m3" in both the prefill and decode vllm_config blocks of all 10 new files, matching the established pattern in the gb200-fp4 agentic recipes.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32194995479 |
There was a problem hiding this comment.
Beyond the previously flagged findings on this PR, this run also checked and ruled out three additional candidates: the new gb300-fp4 recipes omitting max-model-len (consistent with the existing GB200 MiniMax-M3 agentic recipes, which also omit it), the decode-side kv-transfer-config using kv_role: kv_both at the top level (the sub-connector that actually consumes, MooncakeStoreConnector, is explicitly set to kv_consumer; the top-level role only governs the MultiConnector/NixlConnector pairing), and the omission of a cudagraph_mode compilation-config key (not set in any other MiniMax-M3 agentic recipe either).
Extended reasoning...
Checked three candidate issues raised by finder agents against the existing MiniMax-M3 GB200 agentic recipes and the recipe'''s own kv-transfer-config structure: none diverge from established precedent in this repo, so none were escalated as bugs. This is separate from the two previously-posted findings on this PR (missing AIPERF_EXTRA_INPUTS thinking flag, missing dyn-tool-call-parser/dyn-reasoning-parser), which remain as inline comments and are unaffected by this note.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32194995479 |
合并 main，并解决 PR #2663 中的配置、性能变更日志和 GB300 启动脚本冲突。
…AL / 修复 MiniMax-M3 GB300 AgentX 思考模式以匹配黄金 AL
The 10 GB300 disagg recipes pin synthetic_acceptance_length 2.78, which is
golden_al_distribution/minimaxm3_eagle3_gqa.yaml minimax-m3.thinking_on[3],
but nothing enabled thinking, so the sweep served thinking-off traffic
(thinking_off[3] = 2.97). Eval logs from run 32194995479 confirm it: ~99.4%
of gsm8k responses opened with a bare </mm:think> empty reasoning block.
Adopt the GB200 MiniMax-M3 AgentX pattern, which is the established shape for
this dynamo-vllm agentic stack:
- AIPERF_EXTRA_INPUTS: thinking:true in benchmark.env
- dyn-tool-call-parser / dyn-reasoning-parser alongside reasoning-parser
in both the prefill and decode vLLM configs
All 6 merged gb200-fp4/agentic recipes carry all three; these 10 carried none.
Requires a fresh full sweep: acceptance target and token profile both change.
…yn-default-thinking-mode 覆盖评测配置 AIPERF_EXTRA_INPUTS only reaches the aiperf replay. The eval twin runs lm_eval, whose payload srt-slurm builds from a fixed env allow-list (cli/do_sweep.py), so benchmark.env can never reach it. Left alone, the accuracy gate would keep validating a different thinking mode than the throughput arm it is meant to certify. --dyn-default-thinking-mode is absorbed by dynamo.vllm's parse_known_args before the strict AsyncEngineArgs parse, published to the runtime config for non-prefill workers only (vllm/main.py:699-707), and applied by the frontend to any chat request that carries no explicit thinking control. Both arms hit the same frontend, so decode-side is enough to cover them. Set on the decode block only; prefill workers never publish it. The flag landed in dynamo 32c8091a8, before the 1.4.0.dev20260730 nightly cut this config pins, and is absent from the 1.3.1 the GB200 siblings use, which is why they had no server-side option available.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32505173269 |
|
/reuse-sweep-run 32505173269 |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:Disclosure first. Two commits on this branch are mine — Recipe link is N/A. This is a multi-node, disaggregated submission: all ten benchmark files are under Image. Golden AL, and the thinking-mode correction. The recipes pin As originally submitted this config pinned that Commits I verified the fix took effect from the run artifacts rather than assuming it. The obvious
Visible What this fix does not do is move the published numbers, and I want that on the record. The AgentX replay forces output length from the trace — Evals. gsm8k Sweep. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/32505173269 on head Changelog. Appended at the physical end; no historical entry edited. Outstanding, and not resolved by this sign-off: the PR is currently One residual caveat. Signed: |
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
…ynamo-vllm-agentic-mtp-disagg # Conflicts: # perf-changelog.yaml
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — Note: the sign-off was posted against |
5 participants
Description
Add
minimaxm3-fp4-gb300-dynamo-vllm-agentic-mtp-disagg: MiniMax-M3 NVFP4 GB300 disaggregated AgentX config.新增
minimaxm3-fp4-gb300-dynamo-vllm-agentic-mtp-disagg：MiniMax-M3 NVFP4 GB300 分离式 AgentX 配置。minimaxm3-fp4-gb300-dynamo-vllm-agentic-mtp-disaggvllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7Related Issue
N/A
Type of Change
Checklist
perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/reuse-sweep-runon this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.Note
Medium Risk
Mostly additive benchmark recipes, but
launch_gb300-nv.shnow remapsCONFIG_FILEwhenEVAL_ONLYis set and copies MiniMax-M3 recipes into srt-slurm, which can change GB300 agentic job selection.Overview
Adds
minimaxm3-fp4-gb300-dynamo-vllm-agentic-mtp-disagg: disaggregated AgentX on GB300 with Dynamo-vLLM, NIXL P2P, and Mooncake DRAM KV offload.Five Pareto points (1p1d / 1p3d / 2p5d, TP/EP mixes, conc 1–120) each have a throughput recipe with synthetic EAGLE3-GQA AL 2.78 and a real-verification eval twin. Thinking is pinned on (
dyn-default-thinking-modeplusAIPERF_EXTRA_INPUTS=thinking:true).launch_gb300-nv.shmaps MiniMax-M3 FP4 to/scratch/models/MiniMax-M3-NVFP4, copies the new srt-slurm recipes, and selectsEVAL_CONFIG_FILEwhenEVAL_ONLY=true. Changelog entry appended.Reviewed by Cursor Bugbot for commit 9727eb0. Bugbot is set up for automated code reviews on this repo. Configure here.