* fix: escalate to model fallback after rate-limit profile rotation cap
Per-model rate limits (e.g. Anthropic Sonnet-only quotas) are not
relieved by rotating auth profiles — if all profiles share the same
model quota, cycling between them loops forever without falling back
to the next model in the configured fallbacks chain.
Apply the same rotation-cap pattern introduced for overloaded_error
(#58348) to rate_limit errors:
- Add `rateLimitedProfileRotations` to auth.cooldowns config (default: 1)
- After N profile rotations on a rate_limit error, throw FailoverError
to trigger cross-provider model fallback
- Add `resolveRateLimitProfileRotationLimit` helper following the same
pattern as `resolveOverloadProfileRotationLimit`
Fixes#58572
* fix: cap prompt-side rate-limit failover (#58707) (thanks @Forgely3D)
* fix: restore latest-main gates for #58707
---------
Co-authored-by: Ember (Forgely3D) <ember@forgely.co>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
feat(cron): support persistent session targets for cron jobs (#9765)
Add support for `sessionTarget: "current"` and `session:<id>` so cron jobs can
bind to the creating session or a persistent named session instead of only
`main` or ephemeral `isolated` sessions.
Also:
- preserve custom session targets across reloads and restarts
- update gateway validation and normalization for the new target forms
- add cron coverage for current/custom session targets and fallback behavior
- fix merged CI regressions in Discord and diffs tests
- add a changelog entry for the new cron session behavior
Co-authored-by: kkhomej33-netizen <kkhomej33-netizen@users.noreply.github.com>
Co-authored-by: ImLukeF <92253590+ImLukeF@users.noreply.github.com>
When the gateway restarts with many overdue cron jobs, they are now
executed with staggered delays to prevent overwhelming the gateway.
- Add missedJobStaggerMs config (default 5s between jobs)
- Add maxMissedJobsPerRestart limit (default 5 jobs immediately)
- Prioritize most overdue jobs by sorting by nextRunAtMs
- Reschedule deferred jobs to fire gradually via normal timer
Fixes#18892
When an isolated cron agent returns HEARTBEAT_OK (nothing to announce),
the direct delivery is correctly skipped, but the fallback path in
timer.ts still enqueues the summary as a system event to the main
session. Filter out heartbeat-only summaries using isCronSystemEvent
before enqueuing, so internal ack tokens never reach user conversations.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(cron): move session reaper to finally block so it runs reliably
The cron session reaper was placed inside the try block of onTimer(),
after job execution and state updates. If the locked persist section
threw, the reaper was skipped — causing isolated cron run sessions to
accumulate indefinitely in sessions.json.
Move the reaper into the finally block so it always executes after a
timer tick, regardless of whether job execution succeeded. The reaper
is already self-throttled (MIN_SWEEP_INTERVAL_MS = 5 min) so calling
it more reliably has no performance impact.
Closes#31946
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: strengthen cron reaper failure-path coverage and changelog (#31996) (thanks @scoootscooob)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* feat(cron): add failure destination support with webhook mode and bestEffort handling
Extends PR #24789 failure alerts with features from PR #29145:
- Add webhook delivery mode for failure alerts (mode: 'webhook')
- Add accountId support for multi-account channel configurations
- Add bestEffort handling to skip alerts when job has bestEffort=true
- Add separate failureDestination config (global + per-job in delivery)
- Add duplicate prevention (prevents sending to same as primary delivery)
- Add CLI flags: --failure-alert-mode, --failure-alert-account-id
- Add UI fields for new options in web cron editor
* fix(cron): merge failureAlert mode/accountId and preserve failureDestination on updates
- Fix mergeCronFailureAlert to merge mode and accountId fields
- Fix mergeCronDelivery to preserve failureDestination on updates
- Fix isSameDeliveryTarget to use 'announce' as default instead of 'none'
to properly detect duplicates when delivery.mode is undefined
* fix(cron): validate webhook mode requires URL in resolveFailureDestination
When mode is 'webhook' but no 'to' URL is provided, return null
instead of creating an invalid plan that silently fails later.
* fix(cron): fail closed on webhook mode without URL and make failureDestination fields clearable
- sendCronFailureAlert: fail closed when mode is webhook but URL is missing
- mergeCronDelivery: use per-key presence checks so callers can clear
nested failureDestination fields via cron.update
Note: protocol:check shows missing internalEvents in Swift models - this is
a pre-existing issue unrelated to these changes (upstream sync needed).
* fix(cron): use separate schema for failureDestination and fix type cast
- Create CronFailureDestinationSchema excluding after/cooldownMs fields
- Fix type cast in sendFailureNotificationAnnounce to use CronMessageChannel
* fix(cron): merge global failureDestination with partial job overrides
When job has partial failureDestination config, fall back to global
config for unset fields instead of treating it as a full override.
* fix(cron): avoid forcing announce mode and clear inherited to on mode change
- UI: only include mode in patch if explicitly set to non-default
- delivery.ts: clear inherited 'to' when job overrides mode, since URL
semantics differ between announce and webhook modes
* fix(cron): preserve explicit to on mode override and always include mode in UI patches
- delivery.ts: preserve job-level explicit 'to' when overriding mode
- UI: always include mode in failureAlert patch so users can switch between announce/webhook
* fix(cron): allow clearing accountId and treat undefined global mode as announce
- UI: always include accountId in patch so users can clear it
- delivery.ts: treat undefined global mode as announce when comparing for clearing inherited 'to'
* Cron: harden failure destination routing and add regression coverage
* Cron: resolve failure destination review feedback
* Cron: drop unrelated timeout assertions from conflict resolution
* Cron: format cron CLI regression test
* Cron: align gateway cron test mock types
---------
Co-authored-by: Tak Hoffman <781889+Takhoffman@users.noreply.github.com>
Add recomputeNextRunsForMaintenance() call in run() so that stale
runningAtMs markers (from a crashed Phase-1 persist) are cleared by the
existing normalizeJobTickState logic before the already-running guard.
Without this, a manual cron.run() could be blocked for up to
STUCK_RUN_MS (2 hours) even though no job was actually running.
Fixes#17554
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(cron): wrap computeJobNextRunAtMs in try-catch inside applyJobResult
Without this guard, if the croner library throws during schedule
computation (timezone/expression edge cases), the exception propagates
out of applyJobResult and the entire state update is lost — runningAtMs
never clears, lastRunAtMs never advances, nextRunAtMs never recomputes.
After STUCK_RUN_MS (2h), stuck detection clears runningAtMs and the job
re-fires, creating a ~2h repeat cycle instead of the intended schedule.
The sibling function recomputeJobNextRunAtMs in jobs.ts already wraps
computeJobNextRunAtMs in try-catch; this was an oversight in the
applyJobResult call sites.
Changes:
- Error-backoff path: catch and fall back to backoff-only schedule
- Success path: catch and fall through to the MIN_REFIRE_GAP_MS safety net
- applyOutcomeToStoredJob: log a warning when job not found after forceReload
* fix(cron): use recordScheduleComputeError in applyJobResult catch blocks
Address review feedback: the original catch blocks only logged a warning,
which meant a persistent computeJobNextRunAtMs throw would cause a
MIN_REFIRE_GAP_MS (2s) hot loop on cron-kind jobs.
Now both catch blocks call recordScheduleComputeError (exported from
jobs.ts), which tracks consecutive schedule errors and auto-disables the
job after 3 failures — matching the existing behavior in
recomputeJobNextRunAtMs.
* test(cron): cover applyJobResult schedule-throw fallback paths
---------
Co-authored-by: Tak Hoffman <781889+Takhoffman@users.noreply.github.com>
