Skip to content

Isolate node and clients on separate CPUs for the locust perf test - #8161

Draft
Amaury Chamayou (achamayou) wants to merge 18 commits into
mainfrom
achamayou-cpu-isolation-for-perf
Draft

Isolate node and clients on separate CPUs for the locust perf test#8161
Amaury Chamayou (achamayou) wants to merge 18 commits into
mainfrom
achamayou-cpu-isolation-for-perf

Conversation

@achamayou

@achamayou Amaury Chamayou (achamayou) commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Important

Corrected local measurements show no measurable benefit at the latency-bound 100ms point: isolated and unisolated runs both settle around 1260 tx/s with sub-1% CV. This change is intended to remove node/client CPU contention from the CPU-bound CI measurements; it does not claim a 100ms throughput improvement.

Summary

Pins the CCF node and Locust load generators to disjoint CPU sets for pi_basic_blocking_locust, so the benchmark is not affected by how the node and clients happen to share cores.

  • Adds opt-in, best-effort CPU isolation using taskset.
  • Keeps isolation off by default, but enables auto for pi_basic_blocking_locust only.
  • Preserves main's current 2ms, 20ms, and 100ms signature-interval sweep.
  • Allocates whole SMT sibling groups so the node and clients do not share a physical core.
  • Prefixes both node and Locust commands; Locust workers inherit the master's affinity.
  • Warns and runs unisolated if the host cannot be split or taskset is unavailable.
  • Records the selected CPU plan in bencher.json under a test-specific metadata key, preserving metadata written by earlier benchmarks.
  • Installs util-linux in the test image to provide taskset.

No privileges or writable cgroups are required: taskset applies affinity before exec, and the mask is inherited by threads and child processes.

Update against main

The branch now includes main at f5a815bab. The overlaps from the squash-merged benchmark and radar work (#8158 and #8171) were resolved by retaining main's benchmark intervals and radar implementation, then applying only the CPU-isolation changes on top.

The net PR diff is limited to the isolation implementation and its integration. The previously unregistered standalone test is now wired into CTest as cpu_isolation_test in bucket_c.

Measurement status

On a corrected 12-thread local setup, the 100ms configuration showed no meaningful difference:

mode mean tx/s CV
unisolated 1261.2 0.2%
isolated 1257.4 0.5%

The earlier 5ms experiment saturated that host and was too noisy to support a conclusion. Fresh CI measurements are still needed for the current CPU-bound 2ms and 20ms points, so the PR remains a draft.

Testing

  • tests/infra/test_cpu_isolation.py: 12 cases pass after the merge, including real child/grandchild affinity, topology handling, graceful fallback, environment overrides, and metadata persistence.
  • cpu_isolation_test is registered in bucket_c for both virtual CI configurations.
  • Black, Ruff, Gersemi, Python syntax, and whitespace checks pass for the updated diff.

No CHANGELOG.md entry: this only changes test infrastructure.

@achamayou Amaury Chamayou (achamayou) mentioned this pull request Aug 14, 2026
Merged
eddyashton
Eddy Ashton (eddyashton) reviewed Aug 14, 2026
View reviewed changes
Comment thread CMakeLists.txt
eddyashton
Eddy Ashton (eddyashton) reviewed Aug 14, 2026
View reviewed changes
Comment thread CMakeLists.txt Outdated
@achamayou
Amaury Chamayou (achamayou) force-pushed the achamayou-cpu-isolation-for-perf branch 3 times, most recently from 6a803b7 to 82c3896 Compare August 14, 2026 16:44
@achamayou
Amaury Chamayou (achamayou) force-pushed the achamayou-cpu-isolation-for-perf branch from 82c3896 to 35256d4 Compare August 14, 2026 16:45
@achamayou
Add Basic Blocking Locust perf test
8cee892 
Adds pi_basic_blocking_locust, which measures the same blocking-write
workload as pi_basic_blocking but drives it with locust rather than
piccolo, so the number of concurrent clients can be varied.

The load is defined in tests/infra/basicperf_locustfile.py and uses
FastHttpUser, since HttpUser cannot drive enough requests per second to
saturate the service. tests/basicperf_locust.py owns the network, runs
locust against it, and converts locust statistics into bencher metrics
(throughput, latency, memory).

The key space helper shared with basicperf.py moves to
tests/infra/key_space.py, since basicperf.py can only be imported from
tests/infra.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Normalise new test files to LF line endings
0504b1a 
The three files added by this branch were committed with CRLF, unlike
every other Python file under tests/. No functional change.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Measure for a fixed window after users have spawned
1123d10 
Locust --run-time starts counting when locust starts, so it includes the
ramp, and --reset-stats discards the statistics gathered during the ramp
without extending the deadline. The measurement window was therefore
shorter than requested, and shrank as the spawn rate was lowered, until
it disappeared entirely.

This mattered because varying the client count is the point of this
test: at --users 128 --spawn-rate 4 the run ended mid-ramp and reported
127 tx/s instead of the ~1250 tx/s that 128 users actually sustain, and
did so without failing.

Start the shutdown timer from locust spawning_complete instead, so the
window is the same length whatever the spawn rate is, and rename
--run-time-s to --measure-time-s to describe what it now does.
--run-time is kept as a backstop against a run which never finishes
spawning.

Also fail, rather than report, when a run ends without having spawned
all users, or when the window measured is shorter than the one asked
for. Both produce plausible looking figures that do not describe steady
state.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Bind the locust master to a free port
cd4d789 
Locust workers reach the master on port 5557 by default, so a second
locust run anywhere on the same machine fails to bind. Pick a free port
per run instead, via the existing infra.net helper.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Fail with a diagnosis when locust reports N/A
13b2506 
Locust writes N/A rather than a number in the statistics CSV when it has
too few samples to compute a percentile. float() then raised a bare
ValueError, after the network had already been stopped, losing the run
with no indication of what had gone wrong.

Read the numeric columns through a helper which reports the column and
value, and says that the run did not gather enough data. Found by a run
which produced almost no samples.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Snapshot as rarely as the piccolo perf tests do
8ecbead 
add_piccolo_test passes --snapshot-tx-interval 10000 for every piccolo
perf test. add_e2e_test does not pass it at all, and e2e_args defaults
it to 10, which is sensible for functional tests but not for a
benchmark.

This test was therefore writing and fsyncing a ~213KB snapshot every 10
transactions for the whole run, which measures the disk rather than the
service, and makes the figure incomparable with Basic Blocking.

Found by the vegeta comparison work, which hit the same defect.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Measure at 5ms, 100ms and 1s signature intervals
35e9aae 
Blocking writes return once their transaction commits, and commit cannot
outpace the signature interval, so a single interval only measures one
regime. Sweeping three separates them: at 1s and 100ms the workload is
latency-bound and throughput is simply the client count divided by the
interval, while at 5ms the node becomes the limit and the benchmark
measures capacity instead.

Each interval gets its own network, since the interval is fixed in the
node configuration at startup. consensus_update_timeout_ms moves with
it, as in commit_latency.py, because commit cannot be observed faster
than the primary sends updates.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Plot benchmarks which main has no history for
afb4ad2 
A benchmark added by a branch has no main runs to build an EWMA baseline
from, so render_chart skipped it entirely. That made a new benchmark
invisible on the very pull request which adds it, which is when it is
most worth seeing.

Plot such a benchmark against the branch's own earliest run instead, so
its movement across the branch's runs is visible, and mark it as new. It
carries no standard deviation band and is never coloured as an
improvement or a regression, because there is nothing on main to compare
it against.

Borrowing a related benchmark's baseline was considered and rejected: an
axis normalised against something which measures a different thing shows
a difference which is not a change in CCF, and because the chart scale
follows the largest axis, one such axis compresses every other benchmark
into illegibility.

Also truncate long axis labels in the middle rather than at the end.
Benchmarks measured at several settings differ only in their suffix, so
truncating the end left the 100ms and 1000ms axes indistinguishable.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Drive 320 clients from 10 locust processes
ab07e8d 
The blocking workload is latency-bound at the longer signature intervals,
so throughput there is set by the client count: 320 clients raises the
100ms point from ~1260 to ~3100 tx/s and the 1s point from ~130 to ~314,
both within a few percent of clients divided by interval.

Ten sending processes rather than four keeps locust from becoming the
limit while driving that many clients, since each process drives all of
its users from a single thread.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Measure the fastest signatures at 2ms rather than 5ms
6df04a0 
The shortest interval is the point at which the node, rather than the
signature timer, becomes the limit, so it is the one which measures
capacity. Shortening it to 2ms pushes further past the latency-bound
regime. The node ticks every 1ms in these tests, so a 2ms interval is
representable.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Normalize new benchmarks against the branch's first run
46167f1 
The previous version reported a benchmark absent from main by its value
alone, marked (new), with no percentage. That read differently from every
other axis, and the marker made the label long enough to be truncated,
which is exactly what it should not have been for benchmarks whose names
differ only in a suffix.

Treat such a benchmark like any other axis instead, using this branch's
earliest run as its reference in place of the main EWMA baseline, so it
is normalized, labelled and coloured identically. The chart description
records that the reference differs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Elide the middles of words to fit long axis labels
6dd8606 
Truncating the middle of the whole label cut out several words at once.
Elide word by word from the left instead, keeping each word's first and
last letter and replacing the middle with a single ellipsis character, so
a label degrades gradually and the last word, which is what distinguishes
one setting of a benchmark from another, stays readable longest.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Give new benchmarks a band like every other axis
829f0b8 
A benchmark with no main history had its standard deviation hardcoded to
zero, so all four band curves collapsed to a single point at the baseline
while every other axis carried a spread. That left a visible pinch in the
band and, because nothing fell inside a zero-width noise threshold, any
movement between branch runs was coloured as an improvement or a
regression.

Measure its spread the same way as for a benchmark with main history,
across the runs available, which for such a benchmark are the branch's
own. The radial zoom already covered these axes, since their values were
always part of the data the scale is fitted to.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
Add opt-in CPU isolation for the locust perf test
500605e 
Pin the CCF node and the locust clients to disjoint sets of CPUs, so that the
Basic Blocking Locust benchmark measures the node rather than how the node and
the load generator happened to divide the machine between them.

The mechanism is a taskset prefix, which applies the mask before exec so it is
inherited by every thread and grandchild. That needs no privileges and no
cgroup access, so it works inside the unprivileged CI container.

Isolation is off by default and enabled only for pi_basic_blocking_locust, so
other tests are unaffected. Where it cannot be applied the test runs
unisolated with a warning, and the mode is recorded in bencher.json.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Amaury Chamayou (achamayou) force-pushed the achamayou-cpu-isolation-for-perf branch from 35256d4 to 500605e Compare August 18, 2026 16:07
Base automatically changed from locust-blocking-perf to main August 19, 2026 10:10
@achamayou
Merge main into CPU isolation branch
35ff105 
Keep main's 2ms, 20ms, and 100ms benchmark intervals while applying CPU isolation, and restore the standalone isolation test to bucket_c.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Merge remote-tracking branch 'origin/main' into achamayou-update-pr-8161
bf4e54b
@achamayou
Update CI test bucket inventory
372aaab 
Register cpu_isolation_test in the checked-in bucket_c snapshot so the CI inventory matches CMake.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b591ca67-881e-413e-b387-e4f091201815
@achamayou
Preserve CI bucket snapshot ending
6a28fdc 
Keep the generated snapshot's final blank line so the byte-for-byte CI comparison remains stable.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b591ca67-881e-413e-b387-e4f091201815
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eddyashton Eddy Ashton (eddyashton) eddyashton left review comments
Copilot code review Copilot
Copilot will automatically review once the pull request is marked ready for review

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@achamayou @eddyashton