Attach jemalloc allocator stats to excessive-memory Sentry events - #15165
Attach jemalloc allocator stats to excessive-memory Sentry events#15165warp-agent-staging[bot] wants to merge 2 commits into
Conversation
The heap profile attached to the 'Excessive memory usage detected' Sentry event only accounts for sampled *live* jemalloc allocations at dump time, so it can look tiny (e.g. 0.39 GB) even when the OS memory footprint that tripped the alert is >=10 GB. There was no way to tell apart memory that was freed between the threshold trip and the dump, memory retained (but not actively used) by the allocator, and genuinely non-heap memory. Read jemalloc's stats.allocated/resident/retained/mapped via tikv-jemalloc-ctl (advancing the epoch first so the reads aren't stale), plus a freshly-read OS memory footprint, and attach them as a 'memory_diagnostics' Sentry context alongside the existing memory_breakdown context. A failure to read any individual stat is logged and does not prevent the event from being sent, and the event is still sent unconditionally (no suppression based on heap content). Fixes CORE-3791. Co-Authored-By: Warp <agent@warp.dev>
|
@/tmp/pr_comment_xyNsYJ |
…filing tikv-jemallocator/stats (and jemalloc's --enable-stats) was previously listed under jemalloc_with_profiling, which also gates jemalloc_auto_heap_profiling -- an unrelated, pre-existing profiling configuration that doesn't use the new diagnostic reads. --enable-stats carries measurable allocation-path overhead, so this imposed an unnecessary production cost on that build. Move it to jemalloc_pprof, which is the only feature that actually reads jemalloc allocator stats (via heap_usage_tracking). Verified with 'cargo tree -e features -i tikv-jemalloc-sys' that jemalloc_auto_heap_profiling no longer pulls in the stats feature, while heap_usage_tracking still does. Co-Authored-By: Warp <agent@warp.dev>
Cost of enabling jemalloc
|
Description
Attaches jemalloc allocator stats and OS memory footprint samples to the "Excessive memory usage detected" Sentry event, so it's possible to tell apart memory that was freed before the heap dump was collected, memory retained (but not actively used) by the allocator, and genuinely non-heap memory usage.
Today the pprof heap profile attached to that event only accounts for sampled live jemalloc allocations at dump time, so it can look tiny (e.g. 0.39 GB) even when the OS footprint that tripped the 10 GB threshold is much larger, with no way to explain the gap.
This change calls
epoch::advance()and then readsstats.allocated,stats.resident,stats.retained, andstats.mappedviatikv-jemalloc-ctl, plus a freshly re-read OS memory footprint (alongside the footprint that originally tripped the threshold), and attaches all of it as a newmemory_diagnosticsSentry context next to the existingmemory_breakdowncontext. Per the linked issue's explicit non-goal, the event is still always sent -- nothing about the heap content is used to suppress or drop it. A failure reading any individual jemalloc stat is logged and does not prevent the event from being sent.Linked Issue
CORE-3791: https://linear.app/warpdotdev/issue/CORE-3791/excessive-memory-sentry-events-cant-distinguish-freed-before-dump-from
Testing
app/src/profiling_tests.rs, a unit test asserting the newmemory_diagnostics_for_sentryhelper attaches the triggering and freshly-read footprints and successfully reads jemalloc'sstats.allocated/resident/retained/mapped.cargo check -p warp --no-default-features --features heap_usage_tracking --lib-- passedcargo check -p warp --lib(default features, jemalloc/heap tracking disabled) -- passedcargo check -p warp --no-default-features --features jemalloc_auto_heap_profiling --lib(jemalloc enabled without heap_usage_tracking) -- passedcargo test -p warp --no-default-features --features heap_usage_tracking --lib memory_diagnostics-- passed (1 test)cargo clippy -p warp --no-default-features --features heap_usage_tracking --lib --tests -- -D warnings-- passed, no warnings./script/format --check-- passedheap_usage_tracking->jemalloc_pprof->jemalloc_with_profiling->jemalloc) and the newtikv-jemalloc-ctldependency were added to the same non-platform-gated feature flags already used cross-platform by the existingjemalloc_pprof/crash_reportingfeatures, andtikv-jemalloc-ctlis already a transitive dependency of thejemalloc_pprofcrate that's compiled on all platforms today.Screenshots / Videos
N/A -- backend-only change.
Agent Mode