Skip to content

Add RAM sentinel daemon for OOM prevention - #160

Open
idelcano wants to merge 2 commits into
masterfrom
script/add_oom_ram_control_script
Open

Add RAM sentinel daemon for OOM prevention#160
idelcano wants to merge 2 commits into
masterfrom
script/add_oom_ram_control_script

Conversation

@idelcano

Copy link
Copy Markdown
Contributor

📌 References
Issue: https://app.clickup.com/t/4528615/869e36g3p

📝 Implementation
Adds ram_sentinel.py, a daemon that monitors available RAM and stops
Monit-managed services when memory drops critically low, then restores
them once RAM recovers. Configured per-environment via systemd ExecStart
flags — no code changes needed between servers.

Key design decisions:

  • Two stop strategies: kill (SIGKILL via Monit PID, for Java/Tomcat)
    and stop (clean monit stop, for Docker/Postgres)
  • Services stopped in tier order, restored in reverse
  • State persisted to /run/stopram/state.json so an interrupted recovery
    is resumed on daemon restart
  • --dry-run mode reads RAM and PIDs for real but skips all destructive
    commands

How to install

  1. Copy ram_sentinel.py to /usr/local/bin/ram_sentinel.py
  2. Run a dry-run with the trigger set above current available RAM to
    force an emergency cycle and verify tier names and PID resolution:
    python3 /usr/local/bin/ram_sentinel.py
    --emergency-trigger <above_current_available_MB to test>
    --safe-recovery <target_MB>
    --tier kill:<monit_name>
    --dry-run
  3. Confirm output shows correct PID found and [DRY-RUN] would... for
    each action
  4. Create the appropriate .service file for the server to
    /etc/systemd/system/ram-sentinel.service
  5. Enable and start the service:
    systemctl daemon-reload && systemctl enable --now ram-sentinel
    journalctl -u ram-sentinel -f

@idelcano
idelcano requested a review from cgbautista August 18, 2026 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant