Context: ObolNetwork/obol-infrastructure#3008 — a number of partner nodes are pushing metrics to vm.monitoring.gcp.obol.tech/write with credentials vmauth rejects. Per @OisinKyne, for most of them it's not that we invalidated a token — they never set one at all.
That's a launcher UX problem, and it's fixable here.
Why it happens
prometheus/prometheus.yml.example declares remote_write unconditionally, and prometheus/run.sh hard-exits when PROM_REMOTE_WRITE_TOKEN is empty. So today:
- No token → Prometheus crash-loops, and the operator loses their local Grafana too, for a feature they may never have wanted.
- Junk token (sample placeholder,
Bearer <tok> pasted whole, trailing newline) → renders fine, ships 401s at vmauth indefinitely.
- Either way the only feedback is a line in
docker compose logs prometheus, which nobody reads.
And nothing in the repo says where a token comes from. .env.sample.mainnet:201 reads "Prometheus remote write token used for accessing external prometheus" — it doesn't say it's Obol's, doesn't say it's optional, doesn't link anywhere. Zero hits for PROM_REMOTE_WRITE_TOKEN in any .md.
Proposal
A. Make token presence the switch
Move the remote_write block out of prometheus.yml.example into prometheus/remote-write.yml.example, and have run.sh append it only when the token is set. Drop the exit 1.
if [ -n "$PROM_REMOTE_WRITE_TOKEN" ]; then
sed -e "s|\$PROM_REMOTE_WRITE_TOKEN|${PROM_REMOTE_WRITE_TOKEN}|g" \
/etc/prometheus/remote-write.yml.example >> /etc/prometheus/prometheus.yml
else
echo "WARN: PROM_REMOTE_WRITE_TOKEN unset - local monitoring only." >&2
echo "WARN: Obol-hosted dashboards and alerts are disabled. Request a token: <link>" >&2
firun.sh is our {{ if }} — the conditional layer already exists, we just aren't using it. The local stack keeps working either way (the write_relabel_configs only forward charon/mev-boost metrics, so nothing local is lost), and operators who never onboarded stop reaching vmauth at all.
E. Fix the .env.sample comment
Say what the token is, that it's optional, what you give up without it, and where to get one. Both .env.sample.mainnet and .env.sample.hoodi.
A and E together should remove most of the rejected population, since that population mostly consists of people who never had a credential in the first place.
Not proposing now, but on the table
- B — format validation. A
case in run.sh rejecting the sample placeholder, a Bearer prefix, whitespace, wrong length. Catches the paste-the-whole-header class.
- C — verify at startup. One
curl -o /dev/null -w '%{http_code}' -X POST against the write endpoint before exec'ing Prometheus; on 401, print the actual diagnosis and the re-request link. Anything else (a 400 on an empty protobuf body means auth succeeded) proceeds. Warn-and-continue, never fatal — a monitoring outage must not stop a validator. This is what turns a silent server-side mystery into a message on the operator's own terminal.
- D — surface it post-start.
prometheus_remote_storage_samples_failed_total is already scraped; one alert rule and one local-Grafana panel ("Obol remote-write: OK / 401"). The only option here that catches rotation breakage months after setup, which C cannot.
- F — server-side, tracked in obol-infrastructure#3008. Per-tenant vmauth logging and tokens carrying operator identity, so a 401 is attributable without correlating source IPs against onboarding records. Needed for diagnosis; doesn't improve operator UX on its own.
- G — remove the shared secret. Charon already signs monitoring payloads with the ENR key (
charon alpha test --publish does exactly this). Auth by ENR / cluster-lock signature means nothing to distribute and nothing to rotate, and unregistered nodes become impossible by construction. The real fix; worth scoping once F gives us a baseline to measure against.
Mirroring
lido-charon-distributed-validator-node carries the same prometheus/run.sh, and helm-charts/charts/dv-pod has its own path to the same endpoint. Whatever lands here needs to go to both, or we fix a third of the operators.
Context: ObolNetwork/obol-infrastructure#3008 — a number of partner nodes are pushing metrics to
vm.monitoring.gcp.obol.tech/writewith credentials vmauth rejects. Per @OisinKyne, for most of them it's not that we invalidated a token — they never set one at all.That's a launcher UX problem, and it's fixable here.
Why it happens
prometheus/prometheus.yml.exampledeclaresremote_writeunconditionally, andprometheus/run.shhard-exits whenPROM_REMOTE_WRITE_TOKENis empty. So today:Bearer <tok>pasted whole, trailing newline) → renders fine, ships 401s at vmauth indefinitely.docker compose logs prometheus, which nobody reads.And nothing in the repo says where a token comes from.
.env.sample.mainnet:201reads "Prometheus remote write token used for accessing external prometheus" — it doesn't say it's Obol's, doesn't say it's optional, doesn't link anywhere. Zero hits forPROM_REMOTE_WRITE_TOKENin any.md.Proposal
A. Make token presence the switch
Move the
remote_writeblock out ofprometheus.yml.exampleintoprometheus/remote-write.yml.example, and haverun.shappend it only when the token is set. Drop theexit 1.run.shis our{{ if }}— the conditional layer already exists, we just aren't using it. The local stack keeps working either way (thewrite_relabel_configsonly forward charon/mev-boost metrics, so nothing local is lost), and operators who never onboarded stop reaching vmauth at all.E. Fix the
.env.samplecommentSay what the token is, that it's optional, what you give up without it, and where to get one. Both
.env.sample.mainnetand.env.sample.hoodi.A and E together should remove most of the rejected population, since that population mostly consists of people who never had a credential in the first place.
Not proposing now, but on the table
caseinrun.shrejecting the sample placeholder, aBearerprefix, whitespace, wrong length. Catches the paste-the-whole-header class.curl -o /dev/null -w '%{http_code}' -X POSTagainst the write endpoint before exec'ing Prometheus; on401, print the actual diagnosis and the re-request link. Anything else (a400on an empty protobuf body means auth succeeded) proceeds. Warn-and-continue, never fatal — a monitoring outage must not stop a validator. This is what turns a silent server-side mystery into a message on the operator's own terminal.prometheus_remote_storage_samples_failed_totalis already scraped; one alert rule and one local-Grafana panel ("Obol remote-write: OK / 401"). The only option here that catches rotation breakage months after setup, which C cannot.charon alpha test --publishdoes exactly this). Auth by ENR / cluster-lock signature means nothing to distribute and nothing to rotate, and unregistered nodes become impossible by construction. The real fix; worth scoping once F gives us a baseline to measure against.Mirroring
lido-charon-distributed-validator-nodecarries the sameprometheus/run.sh, andhelm-charts/charts/dv-podhas its own path to the same endpoint. Whatever lands here needs to go to both, or we fix a third of the operators.