* fix(agents,failover): propagate sessionId/lane/provider attribution through FailoverError
Adds optional `sessionId` and `lane` fields to `FailoverError` and threads
them — together with the existing `provider`, `model`, `profileId` — through
`describeFailoverError` and `coerceToFailoverError` context, so structured
error log ingestion can attribute exhausted-fallback wrapper errors back
to the originating request instead of dropping the per-profile metadata
when the final wrapper is built.
Fixes#42713.
* fix: preserve failover error attribution
---------
Co-authored-by: Altay <altay@uinaf.dev>
* fix: add CJK error patterns to failover classification
Chinese LLM providers (ZhipuAI/GLM, Bailian, Kimi/Moonshot, DeepSeek,
etc.) return error messages in Chinese. The existing failover
classification only matches English patterns, causing these errors to
fall through as unclassified — surfacing raw provider errors to users
instead of triggering model fallback.
Real production example: ZhipuAI error code 1234 returns
'网络错误，错误id：xxx，请联系客服。' (network error). This was not
matched by the existing 'network error' English pattern, so no failover
was triggered despite having a configured fallback model.
Changes:
- Add Chinese patterns to all error categories in failover-matches.ts:
timeout, serverError, rateLimit, billing, auth, overloaded
- Add Chinese network error detection in formatTransportErrorCopy()
for user-friendly error messages
- Add comprehensive test coverage for all CJK error categories
Follows the existing precedent set by Chinese context overflow patterns
in isContextOverflowError().
* fix: narrow billing pattern and fix placeholder issue URL
- Change '账户余额' to '账户余额不足' to avoid false positives on
messages that merely mention account balance (per greptile review)
- Replace XXXXX placeholder with actual issue #56242
* fix: wire CJK auth failover patterns
* fix: classify CJK provider failover errors
* fix: place failover changelog entry in unreleased
---------
Co-authored-by: Altay <altay@uinaf.dev>
Rewrites the stale branch on top of current `main` and preserves the original issue as regression coverage for the exact OpenRouter JSON 404 payload from #51571.
No production behavior changes are introduced here; current `main` already classifies this payload as `model_not_found`, and this merge locks that in across the shared matcher, failover classifier, and fallback loop.
Co-authored-by: 屈定 <mrdear@users.noreply.github.com>
Co-authored-by: Altay <altay@uinaf.dev>
* fix(agents): classify generic provider errors for failover
Anthropic returns bare 'An unknown error occurred' during API instability
and OpenRouter wraps upstream failures as 'Provider returned error'. Neither
message was recognized by the failover classifier, so the error surfaced
directly to users instead of triggering the configured fallback chain.
Add both patterns to the serverError classifier so they are classified as
transient server errors (timeout) and trigger model failover.
Closes#49706Closes#45834
* fix(agents): scope unknown-error failover by provider
* docs(changelog): note provider-scoped unknown-error failover
---------
Co-authored-by: Aaron Zhu <aaron@Aarons-MacBook-Air.local>
Co-authored-by: Altay <altay@uinaf.dev>
HTTP 500 (Internal Server Error) was not triggering model fallback,
causing agents to fail outright instead of trying the next candidate.
This is inconsistent with TRANSIENT_HTTP_ERROR_CODES which already
includes 500. Aligns the direct status check with that constant.
Co-authored-by: Craig McWilliams <craigamcw@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(failover): classify HTTP 402 as rate_limit when payload indicates usage limit (#30484)
Some providers (notably Anthropic Claude Max plan) surface temporary
usage/rate-limit failures as HTTP 402 instead of 429. Before this change,
all 402s were unconditionally mapped to 'billing', which produced a
misleading 'run out of credits' warning for Max plan users who simply
hit their usage window.
This follows the same pattern introduced for HTTP 400 in #36783: check
the error message for an explicit rate-limit signal before falling back
to the default status-code classification.
- classifyFailoverReasonFromHttpStatus now returns 'rate_limit' for 402
when isRateLimitErrorMessage matches the payload text
- Added regression tests covering both the rate-limit and billing paths
on 402
* fix: narrow 402 rate-limit matcher to prevent billing misclassification
The original implementation used isRateLimitErrorMessage(), which matches
phrases like 'quota exceeded' that legitimately appear in billing errors.
This commit replaces it with a narrow, 402-specific matcher that requires
BOTH retry language (try again/retry/temporary/cooldown) AND limit
terminology (usage limit/rate limit/organization usage).
Prevents misclassification of errors like:
'HTTP 402: exceeded quota, please add credits' -> billing (not rate_limit)
Added regression test for the ambiguous case.
---------
Co-authored-by: Val Alexander <bunsthedev@gmail.com>
* fix(agents): recognize connection errors as retryable timeout failures
## Problem
When a model endpoint becomes unreachable (e.g., local proxy down,
relay server offline), the failover system fails to switch to the
next candidate model. Errors like "Connection error." are not
classified as retryable, causing the session to hang on a broken
endpoint instead of falling back to healthy alternatives.
## Root Cause
Connection/network errors are not recognized by the current failover
classifier:
- Text patterns like "Connection error.", "fetch failed", "network error"
- Error codes like ECONNREFUSED, ENOTFOUND, EAI_AGAIN (in message text)
While `failover-error.ts` handles these as error codes (err.code),
it misses them when they appear as plain text in error messages.
## Solution
Extend timeout error patterns to include connection/network failures:
**In `errors.ts` (ERROR_PATTERNS.timeout):**
- Text: "connection error", "network error", "fetch failed", etc.
- Regex: /\beconn(?:refused|reset|aborted)\b/i, /\benotfound\b/i, /\beai_again\b/i
**In `failover-error.ts` (TIMEOUT_HINT_RE):**
- Same patterns for non-assistant error paths
## Testing
Added test cases covering:
- "Connection error."
- "fetch failed"
- "network error: ECONNREFUSED"
- "ENOTFOUND" / "EAI_AGAIN" in message text
## Impact
- **Compatibility:** High - only expands retryable error detection
- **Behavior:** Connection failures now trigger automatic fallback
- **Risk:** Low - changes are additive and well-tested
* style: fix code formatting for test file
Classify Anthropic's 529 status code as "rate_limit" so model fallback
triggers reliably without depending on fragile message-based detection.
Closes#28502
When an OAuth auth profile returns HTTP 403 with permission_error
(e.g. expired plan), the error was not matched by the authPermanent
patterns. This caused the profile to receive only a short cooldown
instead of being disabled, so the gateway kept retrying the same
broken profile indefinitely.
Add "permission_error" and "not allowed for this organization" to
the authPermanent error patterns so these errors trigger the longer
billing/auth_permanent disable window and proper profile rotation.
Closes#31306
Made-with: Cursor
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
* fix: treat HTTP 502/503/504 as failover-eligible (timeout reason)
When a model API returns 502 Bad Gateway, 503 Service Unavailable, or
504 Gateway Timeout, the error object carries the status code directly.
resolveFailoverReasonFromError() only checked 402/429/401/403/408/400,
so 5xx server errors fell through to message-based classification which
requires the status code to appear at the start of the error message.
Many API SDKs (Google, Anthropic) set err.status = 503 without prefixing
the message with '503', so the message classifier never matched and
failover never triggered — the run retried the same broken model.
Add 502/503/504 to the status-code branch, returning 'timeout' (matching
the existing behavior of isTransientHttpError in the message classifier).
Fixes#20999
* Changelog: add failover 502/503/504 note with credits
* Failover: classify HTTP 504 as transient in message parser
* Changelog: credit taw0002 and vincentkoc for failover fix
---------
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
