auth_permanent errors (e.g. API_KEY_INVALID) can be caused by transient
provider outages rather than genuinely revoked credentials. Previously
these used the same 5h-24h billing backoff, which left providers disabled
long after the upstream issue resolved.
Introduce separate authPermanentBackoffMinutes (default: 10) and
authPermanentMaxMinutes (default: 60) config options so auth_permanent
failures recover in minutes rather than hours.
Fixes#56838
Instead of exponential backoff guesses on Codex 429, probe the WHAM
usage API to determine real availability and write accurate cooldowns.
- Burst/concurrency contention: 15s circuit-break
- Genuine rate limit: proportional to real reset time (capped 2-4h)
- Expired token: 12h cooldown
- Dead account: 24h cooldown
- Probe failure: 30s fail-open
This prevents cascade lockouts when multiple agents share a pool of
Codex profiles, and avoids wasted retries on genuinely exhausted
profiles.
Closes#26329, relates to #1815, #1522, #23996, #54060
Co-authored-by: ryanngit <ryanngit@users.noreply.github.com>
When the backoff saturates at 60 min and retries fire every 30 min
(e.g. cron jobs), each failed request was resetting cooldownUntil to
now+60m. Because now+60m < existing deadline, the window kept getting
renewed and the profile never recovered without manually clearing
usageStats in auth-profiles.json.
Fix: only write a new cooldownUntil (or disabledUntil for billing) when
the new deadline is strictly later than the existing one. This lets the
original window expire naturally while still allowing genuine backoff
extension when error counts climb further.
Fixes#23516
[AI-assisted]
When an auth profile hits a rate limit, `errorCount` is incremented and
`cooldownUntil` is set with exponential backoff. After the cooldown
expires, the time-based check correctly returns false — but `errorCount`
persists. The next transient failure immediately escalates to a much
longer cooldown because the backoff formula uses the stale count:
60s × 5^(errorCount-1), max 1h
This creates a positive feedback loop where profiles appear permanently
stuck after rate limits, requiring manual JSON editing to recover.
Add `clearExpiredCooldowns()` which sweeps all profiles on every call to
`resolveAuthProfileOrder()` and clears expired `cooldownUntil` /
`disabledUntil` values along with resetting `errorCount` and
`failureCounts` — giving the profile a fair retry window (circuit-breaker
half-open → closed transition).
Key design decisions:
- `cooldownUntil` and `disabledUntil` handled independently (a profile
can have both; only the expired one is cleared)
- `errorCount` reset only when ALL unusable windows have expired
- `lastFailureAt` preserved for the existing failureWindowMs decay logic
- In-memory mutation; disk persistence happens lazily on the next store
write, matching the existing save pattern
Fixes#3604
Related: #13623, #15851, #11972, #8434
