[Narasimha-sc]

Problem

With private routing, a connection can stay at "connecting…" until the app is restarted, and joining a link can fail on every "Retry". The agent keeps one proxy per destination relay. When that proxy can't forward to the relay, every retry goes through it again. Seen with public servers on 2026-10-14 (smp4 → smp10).

The existing delayed resets don't help here. A failed connection is retried after 3 s (persistErrorInterval) to the same remembered proxy. A connection is dropped only after 3 timeouts and 15 minutes with no response, and a proxy that answers with errors keeps its connection alive. When the proxy's own connection to the relay is dropped, the client gets NO_SESSION and by design (#1178) makes a new session through the same proxy.

Fix

Forget the chosen proxy when it keeps failing, so the next attempt chooses again:

1. The connection to the proxy fails (proxy offline or unreachable): forget it at once. Without a live connection, this proxy normally hasn't received the destination, so choosing again adds no proxy that learns it. A request that only waits for another request's connection doesn't count its own wait timing out.
2. A forwarded command fails with PROXY (BROKER e) other than a host error (e.g. the proxy times out reaching the relay): the first failure starts a failure streak, stored next to the chosen proxy. A forward that reaches the relay clears it: a success, an error from the destination relay inside the response, or PROXY (PROTOCOL _). A failure at least proxyFailureInterval (new AgentConfig field, 30 s) after the start of the streak forgets the proxy. So a single timeout, or a burst of errors from forwards in flight when the relay restarts, doesn't move the destination to another proxy (a failed PRXY still forgets it at once, as before).

Updates are guarded: an entry is changed only while it still names the proxy that failed or succeeded (also in smpClientDisconnected and newProxiedRelay), so a late result from the old proxy can't change a new choice. After PROXY NO_SESSION, the dead relay session is removed from the proxy client that handled the request. The retry through the same proxy (#1178) doesn't record that proxy as the choice when there is none, so a proxy forgotten concurrently isn't put back.

Details: plans/2026-10-15-proxy-repick-on-failure.md.

Tests

• New SMPProxyTests:
  • "chooses another proxy when the chosen proxy is offline";
  • "chooses another proxy when the chosen proxy cannot forward to destination relay";
  • "keeps the chosen proxy until it fails for proxyFailureInterval", a direct test of the streak rule (needs proxiedRelayFailed and proxiedRelaySucceeded exported).
• Mutation checks (on the previous head 5c613348, which has the same fix before the merge of 8bc4d360): with the 30 s forget disabled, "cannot forward" fails (2/2). With the connect-failure forget disabled, "offline" fails (2/2). With the fix, all 3 pass (3/3 runs).
• CI on f9aa3fe5: passed.
• Local run on 5c613348: SMP proxy tests 52 examples, 0 failures. Full suite 896 examples, 8 failures, none caused by this change. 2 pass when re-run alone. 3 are environment failures on this machine that also fail on the base code (batching 200 subscriptions, XRCP multicast, SMP server CLI static files). The other 3 also fail on the base-code binary when run alone: delivery receipts concurrently (2/2), XFTP resume deleting after restart (2/2) and join when reply queue creation fails (1/2).

Desktop app (simplex-chat with this branch at 5c613348, before the merge of 8bc4d360; local servers)

Destination relay R; proxy P1 whose forwarding to R is black-holed (a SOCKS relay passes the handshakes and then drops all bytes); good proxy P2. The app's message servers are only P1 and P2. Trials where the first pick was P1:

Scenario	master	this branch
The contact joins the app's 1-time link (the app sends its reply in the background): "connecting…"	not connected after 10 min (2/2)	connected after 64, 64, 129 and 192 s, no user action (4/4)
The app joins a link, user taps "Retry" after each error	not connected after 6 Retries (2/2)	connected after 3, 4 and 6 Retries (86–177 s), and once after 61 s without a Retry (4/4)

The longer trials picked P1 again after forgetting it: the failed proxy isn't excluded, so with two proxies that happens half of the time.

Recordings (the app's proxies: 127.0.0.1 = P1, broken, and 127.0.0.2 = P2; the red box marks what to look at):

"connecting…" after the contact joins the app's link. master stays "connecting…":

This branch forgets P1 after 30 s of failures and chooses again; in this recording, going by the timing, the first re-pick chose P1 again, so the contact connects after 130 s, with no user action:

Joining a link with Retry. master fails on every Retry:

This branch: the first "Private routing timeout" starts the streak, the next one 30 s later forgets P1, and the next Retry connects:

Android

The APK built from simplex-chat/simplex-chat#7739 at 26f470276 (before its later master merge, same simplexmq pin), with the core library built from it (this branch at f9aa3fe5), on Waydroid with local servers; P1 (here 192.168.241.1) can't reach the contact's server, P2 can. Before: the master APK, which bundles an older core library (core 7.1.0.6, September 2026) without this change; there is no master core library built for Android here.

Scenario	master app (older core)	simplex-chat#7739
P1 failed 4 times (2 "Connection timeout", then 2 "Private routing timeout" alerts 32 s apart), then only P2 configured, no restart; join another link	fails via P1 (3 of 3)	connects at once (2 of 2)
A contact stuck on P1, then only P2 configured	still "connecting…" 240 s later (2 of 2)	connects 64 s after the contact joined (5 s after the switch), when the app forgets P1 and the re-pick can only choose P2 (2 of 2)
Join a link, P1 and P2 configured, Retry after each error (up to 6)	still failing after 6 Retries (4 of 4)	connects after 4, 4, 6 and 6 Retries; 2 trials still failing at the 6-Retry cap; in a run allowed up to 14 Retries, one trial connected after 10
A contact joins the app's 1-time link, no user action (up to 300 s)	still "connecting…" (4 of 4)	connects after 63, 63, 63, 67 and 128 s

The trials that needed more Retries are those where the re-pick chose P1 again (the re-pick is random, so with two proxies it can choose P1 again; here, in the Retry trials, it did in 9 of 14 re-picks). Regression: messages each way before and after an app restart, no crashes.

Open questions

• Interactive recovery takes a few Retries. Client timeouts don't count, so a join needs the proxy's report twice, 30 s apart. In the trials: 3–6 Retries on desktop and 4–10 on Android, up to about 5 minutes; 2 Android trials were still failing at the 6-Retry cap.

• The failed proxy can be chosen again. Excluding it from the next pick would make recovery take one streak instead of about two on average with two proxies.

• The streak doesn't expire. With rare sends, one timeout, then another hours later with nothing reaching the relay in between, forgets the proxy.

• Not covered by tests:

  • not counting a waiter's own timeout;
  • clearing on relay errors;
  • the deleteRelaySession change;
  • the disconnect and PRXY guards;
  • not recording an explicit proxy.

  The re-pick tests also don't assert that the bad proxy was tried first; they rely on the seeded pick.

• Unchanged: a host error that the proxy reports about the relay keeps the entry (e.g. an onion-only relay through a proxy without Tor).