* fix(failover): classify HTTP 402 as rate_limit when payload indicates usage limit (#30484)
Some providers (notably Anthropic Claude Max plan) surface temporary
usage/rate-limit failures as HTTP 402 instead of 429. Before this change,
all 402s were unconditionally mapped to 'billing', which produced a
misleading 'run out of credits' warning for Max plan users who simply
hit their usage window.
This follows the same pattern introduced for HTTP 400 in #36783: check
the error message for an explicit rate-limit signal before falling back
to the default status-code classification.
- classifyFailoverReasonFromHttpStatus now returns 'rate_limit' for 402
when isRateLimitErrorMessage matches the payload text
- Added regression tests covering both the rate-limit and billing paths
on 402
* fix: narrow 402 rate-limit matcher to prevent billing misclassification
The original implementation used isRateLimitErrorMessage(), which matches
phrases like 'quota exceeded' that legitimately appear in billing errors.
This commit replaces it with a narrow, 402-specific matcher that requires
BOTH retry language (try again/retry/temporary/cooldown) AND limit
terminology (usage limit/rate limit/organization usage).
Prevents misclassification of errors like:
'HTTP 402: exceeded quota, please add credits' -> billing (not rate_limit)
Added regression test for the ambiguous case.
---------
Co-authored-by: Val Alexander <bunsthedev@gmail.com>
* fix(agents): recognize connection errors as retryable timeout failures
## Problem
When a model endpoint becomes unreachable (e.g., local proxy down,
relay server offline), the failover system fails to switch to the
next candidate model. Errors like "Connection error." are not
classified as retryable, causing the session to hang on a broken
endpoint instead of falling back to healthy alternatives.
## Root Cause
Connection/network errors are not recognized by the current failover
classifier:
- Text patterns like "Connection error.", "fetch failed", "network error"
- Error codes like ECONNREFUSED, ENOTFOUND, EAI_AGAIN (in message text)
While `failover-error.ts` handles these as error codes (err.code),
it misses them when they appear as plain text in error messages.
## Solution
Extend timeout error patterns to include connection/network failures:
**In `errors.ts` (ERROR_PATTERNS.timeout):**
- Text: "connection error", "network error", "fetch failed", etc.
- Regex: /\beconn(?:refused|reset|aborted)\b/i, /\benotfound\b/i, /\beai_again\b/i
**In `failover-error.ts` (TIMEOUT_HINT_RE):**
- Same patterns for non-assistant error paths
## Testing
Added test cases covering:
- "Connection error."
- "fetch failed"
- "network error: ECONNREFUSED"
- "ENOTFOUND" / "EAI_AGAIN" in message text
## Impact
- **Compatibility:** High - only expands retryable error detection
- **Behavior:** Connection failures now trigger automatic fallback
- **Risk:** Low - changes are additive and well-tested
* style: fix code formatting for test file
Classify Anthropic's 529 status code as "rate_limit" so model fallback
triggers reliably without depending on fragile message-based detection.
Closes#28502
When an OAuth auth profile returns HTTP 403 with permission_error
(e.g. expired plan), the error was not matched by the authPermanent
patterns. This caused the profile to receive only a short cooldown
instead of being disabled, so the gateway kept retrying the same
broken profile indefinitely.
Add "permission_error" and "not allowed for this organization" to
the authPermanent error patterns so these errors trigger the longer
billing/auth_permanent disable window and proper profile rotation.
Closes#31306
Made-with: Cursor
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
* fix: treat HTTP 502/503/504 as failover-eligible (timeout reason)
When a model API returns 502 Bad Gateway, 503 Service Unavailable, or
504 Gateway Timeout, the error object carries the status code directly.
resolveFailoverReasonFromError() only checked 402/429/401/403/408/400,
so 5xx server errors fell through to message-based classification which
requires the status code to appear at the start of the error message.
Many API SDKs (Google, Anthropic) set err.status = 503 without prefixing
the message with '503', so the message classifier never matched and
failover never triggered — the run retried the same broken model.
Add 502/503/504 to the status-code branch, returning 'timeout' (matching
the existing behavior of isTransientHttpError in the message classifier).
Fixes#20999
* Changelog: add failover 502/503/504 note with credits
* Failover: classify HTTP 504 as transient in message parser
* Changelog: credit taw0002 and vincentkoc for failover fix
---------
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
