Skip to content

VAPI-3808 VAPI-3804 fix(signaling): stop auto-reconnecting when the gateway returns 409 - #17

Merged
smoghe-bw merged 2 commits into
mainfrom
fix/stop-reconnect-storm-on-409
Aug 20, 2026
Merged

VAPI-3808 VAPI-3804 fix(signaling): stop auto-reconnecting when the gateway returns 409#17
smoghe-bw merged 2 commits into
mainfrom
fix/stop-reconnect-storm-on-409

Conversation

@smoghe-bw

@smoghe-bw smoghe-bw commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Problem

Signaling.connect builds its client with unlimited auto-reconnect:

const ws = new JsonRpcClient(websocketUrl, {
  autoconnect: true,
  reconnect: true,
  max_reconnects: 0, // Unlimited
});

but the error handler only ever treated one status as terminal:

if (error.message === "Unexpected server response: 403") {

So a 409 — the gateway rejecting the connection because another device already holds that endpoint — was treated as retryable. The SDK reconnected several times a second against a gateway that was correctly telling it the connection could not be established.

Gateway logs for a single endpoint show 21 rejections in 90 seconds across three hosts. And it was self-sustaining: each retry that won a socket before being rejected left an inert connection behind — no open handler, so it never called setMediaPreferences, never created peers, never answered the heartbeat — which held the endpoint and caused the next round of rejections. That's the same failure mode already described in the _disconnect comment added by #13; this is the other path into it.

Change

  • Collect terminal handshake failures into a single FATAL_HANDSHAKE_ERRORS table keyed by the ws error message, covering 403 and now 409.
  • On a match: disable auto-reconnect, close, reject with a clear error, and disconnect without calling leave.
  • Add the missing return, so a fatal error no longer falls through into the generic Websocket error: ... log below it (403 was double-logging).

Retry policy is deliberately left to the application. A 409 clears when the other device disconnects, and the app knows better than the SDK whether and when re-attempting is appropriate.

Note this only works under Node — browsers don't expose the HTTP status of a failed websocket upgrade to the error handler. Same pre-existing limitation as the 403 branch; called out in a comment.

Test plan

  • npm test — 72 passed
  • npx tsc --noEmit clean, prettier --check clean
  • New test: a 409 error closes with 409 and calls setAutoReconnect(false), mirroring the existing 403 test
  • Existing generic-error test kept (renamed to "non-fatal"), still asserting setAutoReconnect is untouched for recoverable errors

🤖 Generated with Claude Code

The client is constructed with `reconnect: true, max_reconnects: 0`, and the
error handler only ever treated 403 as terminal. A 409 — the gateway
rejecting the connection because another device already holds the endpoint —
was therefore treated as retryable, so the SDK reconnected several times a
second against a gateway that was correctly telling it the connection could
not be established.

Gateway logs for one endpoint show 21 such rejections in 90 seconds across
three hosts. Worse, each retry that did get a socket before being rejected
left an inert connection behind, which held the endpoint and caused the next
round of rejections — the same self-sustaining loop described in the
_disconnect comment added by #13.

Collect the terminal handshake failures into one table, disable reconnect
and surface the error for both, and return so a fatal error no longer falls
through into the generic error log. Retry policy is left to the app.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@smoghe-bw
smoghe-bw requested review from a team as code owners August 20, 2026 15:34
@bwappsec

bwappsec commented Aug 20, 2026

Copy link
Copy Markdown

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues
Code Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

connect() overwrote this.ws without closing the previous JsonRpcClient,
leaving its unlimited auto-reconnect loop running in the background.
That orphaned client's "open" handler calls setMediaPreferences() via
this.ws, which by then points at the new client, so the orphan's own
socket never sends anything and just idles until the gateway reaps it
— producing repeated "new websocket connection" / "never called
setMediaPreferences" storms against the same endpoint.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@smoghe-bw

Copy link
Copy Markdown
Contributor Author

Investigated a production incident (gateway logs 08:06–08:08 today) showing repeated new websocket connectionwebsocket never called setMediaPreferences; disconnecting idle connection (30s later) mixed with endpoint already has an active connection; rejecting connection bursts, all against the same endpoint.

Root cause: Signaling.connect() overwrote this.ws with a new JsonRpcClient without ever closing a pre-existing one. Since each client is constructed with reconnect: true, max_reconnects: 0 (unlimited), if connect() is ever called a second time on the same instance (e.g. app-level retry after an error/close — there's no built-in reconnect handling, see the // TODO: handle reconnections note), the old client is orphaned but keeps auto-reconnecting forever against the same endpointToken.

The orphan's open handler calls setMediaPreferences(), which internally reads this.ws — but by then this.ws points at the new, active client, so the RPC actually goes out over the wrong socket. The orphan's own connection never sends anything, sits idle, and gets reaped by the gateway's 30s idle watchdog — exactly matching the log pattern. Its reconnect loop then immediately retries, which is also the source of the repeated endpoint already has an active connection rejections.

Fix: connect() now calls this._disconnect(false) on any existing this.ws before constructing the new client, so there's never more than one live (and reconnecting) client per Signaling instance.

@smoghe-bw smoghe-bw changed the title fix(signaling): stop auto-reconnecting when the gateway returns 409 VAPI-3804 fix(signaling): stop auto-reconnecting when the gateway returns 409 Aug 20, 2026
@smoghe-bw
smoghe-bw merged commit e152c08 into main Aug 20, 2026
5 checks passed
@smoghe-bw
smoghe-bw deleted the fix/stop-reconnect-storm-on-409 branch August 20, 2026 17:54
@smoghe-bw smoghe-bw changed the title VAPI-3804 fix(signaling): stop auto-reconnecting when the gateway returns 409 VAPI-3808 VAPI-3804 fix(signaling): stop auto-reconnecting when the gateway returns 409 Aug 20, 2026
smoghe-bw added a commit to Bandwidth/swift-brtc-sdk that referenced this pull request Aug 20, 2026
… from 403 (#19)

The JS SDK (Bandwidth/javascript-brtc-sdk#17) fixed a reconnect storm caused
by unlimited auto-reconnect treating a 409 (another device holds the
endpoint) as retryable. The Swift SignalingClient has no auto-reconnect at
all, so that storm can't happen here, but it also gave the app no way to
tell a 403 (bad token) or 409 (endpoint occupied) apart from any other
WebSocket drop — both non-retryable conditions worth surfacing distinctly.

Capture the rejected handshake's HTTP status via WebSocketProtocol.response,
add BandwidthRTCError.endpointOccupied, and thread the classification through
the "close" event to a new BandwidthRTCClient.onDisconnected callback.

Co-authored-by: smoghe-bw <smoghe-bw>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants