Skip to content

CLDSRV-981: Log Scuba health checks through the server logger - #6261

Closed
anurag4DSB wants to merge 2 commits into
development/9.3from
bugfix/CLDSRV-981-scuba-pinned-startup-logger
Closed

CLDSRV-981: Log Scuba health checks through the server logger#6261
anurag4DSB wants to merge 2 commits into
development/9.3from
bugfix/CLDSRV-981-scuba-pinned-startup-logger

Conversation

@anurag4DSB

Copy link
Copy Markdown
Contributor

Intent: why does this change exist?

The quota client held the werelogs RequestLogger that server.js builds at boot, and its health-check timer logged through it forever. Same bug class as CLDSRV-979, found in the same RD-2240 investigation, much smaller blast radius.

System impact: what's affected, including downstream?

lib/utilization/scuba/wrapper.js and its one call site in lib/server.js. ScubaClientImpl.setup() no longer takes a logger argument — lib/server.js:403 is the only production caller. Health-check log lines now carry the server logger's fields instead of the startup request's, so they no longer have a req_id; anything grepping these two messages by req_id would be affected, though that req_id was misleading anyway since it belonged to an unrelated boot-time request.

Preserved behavior: what explicitly stays the same?

The health check itself — cadence, staleness threshold, and when quotas get enabled or disabled. Both messages keep their text and their levels.

Intended change: what's different after this PR?

Background health-check logging goes through the module-level server logger, which writes through and drops sub-level entries rather than buffering them. Nothing that outlives a request holds a RequestLogger.

Verification: how do we know this worked, or how would we know if it didn't?

Reproduced first: 200 simulated health-check flaps against the pinned startup logger left 199 buffered entries, every one carrying the same req_id — which is the signature of this bug generally, and a cheap thing to grep for elsewhere.

Two regression tests cover it. While in the file I also revived the four _healthCheck cases that had been silently failing on development/9.3: sinon reports healthCheck as non-existent because it sits deep in the scubaclient prototype chain, so they assign the method directly instead. Those four cover the function this PR changes, which is why they are in scope rather than left for later. tests/unit goes from 5114 passing / 15 failing to 5120 / 11; the remaining 11 are the pre-existing bucketPut and bucketUpdateQuota quota-metric-seeding failures, untouched by this. Lint clean, 0 errors.

@bert-e

bert-e commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Hello anurag4dsb,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval
/bypass_build_status Bypass the build and test status
/bypass_commit_size Bypass the check on the size of the changeset TBA
/bypass_incompatible_branch Bypass the check on the source branch prefix
/bypass_jira_check Bypass the Jira issue check
/bypass_peer_approval Bypass the pull request peers' approval
/bypass_leader_approval Bypass the pull request leaders' approval
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue CLDSRV-981 contains:

  • None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 9.3.17

  • 9.4.2

Please check the Fix Version/s of CLDSRV-981, or the target
branch of this pull request.

@codecov

codecov Bot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.19%. Comparing base (24fb6b3) to head (4b4ebb8).
✅ All tests successful. No failed tests found.

Additional details and impacted files

Impacted file tree graph

Files with missing lines Coverage Δ
lib/server.js 85.43% <100.00%> (ø)
lib/utilization/scuba/wrapper.js 100.00% <100.00%> (ø)

... and 2 files with indirect coverage changes

@@                 Coverage Diff                 @@
##           development/9.3    #6261      +/-   ##
===================================================
+ Coverage            85.17%   85.19%   +0.02%     
===================================================
  Files                  206      206              
  Lines                13424    13423       -1     
===================================================
+ Hits                 11434    11436       +2     
+ Misses                1990     1987       -3     
Flag Coverage Δ
file-ft-tests 62.38% <25.00%> (-6.02%) ⬇️
kmip-ft-tests 28.36% <25.00%> (+<0.01%) ⬆️
mongo-v0-ft-tests 69.58% <25.00%> (+0.02%) ⬆️
mongo-v1-ft-tests 69.53% <25.00%> (-0.07%) ⬇️
multiple-backend 36.83% <25.00%> (+<0.01%) ⬆️
s3c-ft-tests-v0 64.08% <25.00%> (+<0.01%) ⬆️
s3c-ft-tests-v0-null-compat 64.14% <25.00%> (+<0.01%) ⬆️
s3c-ft-tests-v1 64.06% <25.00%> (+<0.01%) ⬆️
sur-tests 36.00% <100.00%> (+0.02%) ⬆️
sur-tests-inflights 37.80% <100.00%> (-0.04%) ⬇️
unit 71.09% <100.00%> (+<0.01%) ⬆️
utapi-v2-tests 34.62% <25.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bert-e

bert-e commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

ScubaClientImpl.setup(log) stored the werelogs RequestLogger that
server.js creates at boot for initiateStartup, and the periodic health
check timer then logged through it on every enable/disable transition.
RequestLogger buffers every entry it is handed and only drains on an
error-level write, so the buffer grew for the process lifetime and was
never reclaimed. Measured: 199 entries retained after 200 health check
flaps.

The health check runs on a timer, not inside a request, so it takes the
module-level server logger, which writes through instead of buffering.
setup() no longer needs a logger argument.
Adds regression tests: the client retains no logger it is handed, health
check transitions do not write through one, and the enabling-quotas line
fires on the disabled-to-enabled transition.

The four existing _healthCheck cases had been failing on development/9.3
because sinon reports healthCheck as non-existent - it sits deep in the
scubaclient prototype chain - so they never ran. They assign the method
directly instead. They cover the function this change touches, so they
are worth having alive: tests/unit goes from 5114 passing / 15 failing to
5120 / 11.
@anurag4DSB
anurag4DSB force-pushed the bugfix/CLDSRV-981-scuba-pinned-startup-logger branch from 28f4fba to 4b4ebb8 Compare August 20, 2026 14:21
@anurag4DSB anurag4DSB closed this Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants