-
Notifications
You must be signed in to change notification settings - Fork 27
Pull requests: mlcommons/endpoints
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
Distributed SWE bench by sharding SWE tasks across cpus for faster runtime
size/very-large
PR Review Policy: >1500 lines or >50 files
#458
opened Aug 20, 2026 by
leopck
Collaborator
Loading…
1 of 4 tasks
fix(lcb-service): pin base image to Python 3.11 to match LiveCodeBench
size/normal
PR Review Policy: <=500 non-test lines & <=20 files
#457
opened Aug 19, 2026 by
liayan
Member
Loading…
10 tasks
fix(swebench-service): give Pyxis container creation its own deadline
size/normal
PR Review Policy: <=500 non-test lines & <=20 files
#456
opened Aug 19, 2026 by
leopck
Collaborator
Loading…
10 tasks
fix(benchmark): an accuracy run that produces no number must fail
size/normal
PR Review Policy: <=500 non-test lines & <=20 files
#454
opened Aug 19, 2026 by
leopck
Collaborator
Loading…
1 of 4 tasks
fix(benchmark): don't reject an accuracy-only run for listing many endpoints
size/normal
PR Review Policy: <=500 non-test lines & <=20 files
#453
opened Aug 19, 2026 by
leopck
Collaborator
Loading…
1 of 4 tasks
fix(swebench-service): let srun find its config and honour proxy policy
size/normal
PR Review Policy: <=500 non-test lines & <=20 files
#452
opened Aug 19, 2026 by
leopck
Collaborator
Loading…
1 of 4 tasks
docs: steady-state metrics reporting design proposal
#447
opened Aug 13, 2026 by
nv-alicheng
Collaborator
Loading…
2 of 10 tasks
fix(lcb-service): distinguish infra crashes from submission failures in grading children
size/normal
PR Review Policy: <=500 non-test lines & <=20 files
#433
opened Jul 29, 2026 by
liayan
Member
Loading…
4 of 7 tasks
Add License Scanning and Reporting for Endpoints Changes on Main.
#412
opened Jul 14, 2026 by
arav-agarwal2
Collaborator
Loading…
4 of 10 tasks
feat(config): consolidate all time knobs into settings.timeouts; make --timeout a real run watchdog
priority: P0
Critical — blocks release or users
size/very-large
PR Review Policy: >1500 lines or >50 files
#409
opened Jul 13, 2026 by
viraatc
Collaborator
Loading…
9 tasks done
fix: Salt failure is a hard error
size/normal
PR Review Policy: <=500 non-test lines & <=20 files
#406
opened Jul 10, 2026 by
arekay-nv
Collaborator
Loading…
4 of 10 tasks
feat :Testing workflow to push images
size/normal
PR Review Policy: <=500 non-test lines & <=20 files
#405
opened Jul 9, 2026 by
arekay-nv
Collaborator
Loading…
10 tasks
fix(endpoint-client): retry pre-response connection resets (opt-in)
#404
opened Jul 8, 2026 by
Palanivelg
Contributor
Loading…
3 tasks done
DRAFT: feat(power): vendor-agnostic power/energy monitoring
#383
opened Jun 30, 2026 by
viraatc
Collaborator
Loading…
Feat: Support creation of run_metadata.json at the end of benchmark run
#374
opened Jun 23, 2026 by
anandhu-eng
Contributor
Loading…
5 of 10 tasks
feat: add configurable live metrics publishing for CPU servers
#363
opened Jun 22, 2026 by
attafosu
Collaborator
Loading…
10 tasks
feat: Enable CI for release branches
size/normal
PR Review Policy: <=500 non-test lines & <=20 files
#352
opened Jun 12, 2026 by
arekay-nv
Collaborator
Loading…
10 tasks
DeepSeek-V4-Pro benchmarking example (vLLM + SGLang)
#347
opened Jun 8, 2026 by
karverma-amd
Loading…
3 tasks
feat: add -vvv trace mode dashboard for client overhead
#334
opened Jun 4, 2026 by
viraatc
Collaborator
Loading…
4 of 5 tasks
Feat(Urgent): Add support for auto populating system description and run metadata
priority: P0
Critical — blocks release or users
type: feature
New feature or capability
#327
opened May 28, 2026 by
anandhu-eng
Contributor
Loading…
7 of 10 tasks
ProTip!
Exclude everything labeled
bug with -label:bug.