Commit Graph

10 Commits

Author SHA1 Message Date
Altay
6e962d8b9e fix(agents): handle overloaded failover separately (#38301)
* fix(agents): skip auth-profile failure on overload

* fix(agents): note overload auth-profile fallback fix

* fix(agents): classify overloaded failures separately

* fix(agents): back off before overload failover

* fix(agents): tighten overload probe and backoff state

* fix(agents): persist overloaded cooldown across runs

* fix(agents): tighten overloaded status handling

* test(agents): add overload regression coverage

* fix(agents): restore runner imports after rebase

* test(agents): add overload fallback integration coverage

* fix(agents): harden overloaded failover abort handling

* test(agents): tighten overload classifier coverage

* test(agents): cover all-overloaded fallback exhaustion

* fix(cron): retry overloaded fallback summaries

* fix(cron): treat HTTP 529 as overloaded retry
2026-03-07 01:42:11 +03:00
Josh Avant
806803b7ef feat(secrets): expand SecretRef coverage across user-supplied credentials (#29580)
* feat(secrets): expand secret target coverage and gateway tooling

* docs(secrets): align gateway and CLI secret docs

* chore(protocol): regenerate swift gateway models for secrets methods

* fix(config): restore talk apiKey fallback and stabilize runner test

* ci(windows): reduce test worker count for shard stability

* ci(windows): raise node heap for test shard stability

* test(feishu): make proxy env precedence assertion windows-safe

* fix(gateway): resolve auth password SecretInput refs for clients

* fix(gateway): resolve remote SecretInput credentials for clients

* fix(secrets): skip inactive refs in command snapshot assignments

* fix(secrets): scope gateway.remote refs to effective auth surfaces

* fix(secrets): ignore memory defaults when enabled agents disable search

* fix(secrets): honor Google Chat serviceAccountRef inheritance

* fix(secrets): address tsgo errors in command and gateway collectors

* fix(secrets): avoid auth-store load in providers-only configure

* fix(gateway): defer local password ref resolution by precedence

* fix(secrets): gate telegram webhook secret refs by webhook mode

* fix(secrets): gate slack signing secret refs to http mode

* fix(secrets): skip telegram botToken refs when tokenFile is set

* fix(secrets): gate discord pluralkit refs by enabled flag

* fix(secrets): gate discord voice tts refs by voice enabled

* test(secrets): make runtime fixture modes explicit

* fix(cli): resolve local qr password secret refs

* fix(cli): fail when gateway leaves command refs unresolved

* fix(gateway): fail when local password SecretRef is unresolved

* fix(gateway): fail when required remote SecretRefs are unresolved

* fix(gateway): resolve local password refs only when password can win

* fix(cli): skip local password SecretRef resolution on qr token override

* test(gateway): cast SecretRef fixtures to OpenClawConfig

* test(secrets): activate mode-gated targets in runtime coverage fixture

* fix(cron): support SecretInput webhook tokens safely

* fix(bluebubbles): support SecretInput passwords across config paths

* fix(msteams): make appPassword SecretInput-safe in onboarding/token paths

* fix(bluebubbles): align SecretInput schema helper typing

* fix(cli): clarify secrets.resolve version-skew errors

* refactor(secrets): return structured inactive paths from secrets.resolve

* refactor(gateway): type onboarding secret writes as SecretInput

* chore(protocol): regenerate swift models for secrets.resolve

* feat(secrets): expand extension credential secretref support

* fix(secrets): gate web-search refs by active provider

* fix(onboarding): detect SecretRef credentials in extension status

* fix(onboarding): allow keeping existing ref in secret prompt

* fix(onboarding): resolve gateway password SecretRefs for probe and tui

* fix(onboarding): honor secret-input-mode for local gateway auth

* fix(acp): resolve gateway SecretInput credentials

* fix(secrets): gate gateway.remote refs to remote surfaces

* test(secrets): cover pattern matching and inactive array refs

* docs(secrets): clarify secrets.resolve and remote active surfaces

* fix(bluebubbles): keep existing SecretRef during onboarding

* fix(tests): resolve CI type errors in new SecretRef coverage

* fix(extensions): replace raw fetch with SSRF-guarded fetch

* test(secrets): mark gateway remote targets active in runtime coverage

* test(infra): normalize home-prefix expectation across platforms

* fix(cli): only resolve local qr password refs in password mode

* test(cli): cover local qr token mode with unresolved password ref

* docs(cli): clarify local qr password ref resolution behavior

* refactor(extensions): reuse sdk SecretInput helpers

* fix(wizard): resolve onboarding env-template secrets before plaintext

* fix(cli): surface secrets.resolve diagnostics in memory and qr

* test(secrets): repair post-rebase runtime and fixtures

* fix(gateway): skip remote password ref resolution when token wins

* fix(secrets): treat tailscale remote gateway refs as active

* fix(gateway): allow remote password fallback when token ref is unresolved

* fix(gateway): ignore stale local password refs for none and trusted-proxy

* fix(gateway): skip remote secret ref resolution on local call paths

* test(cli): cover qr remote tailscale secret ref resolution

* fix(secrets): align gateway password active-surface with auth inference

* fix(cli): resolve inferred local gateway password refs in qr

* fix(gateway): prefer resolvable remote password over token ref pre-resolution

* test(gateway): cover none and trusted-proxy stale password refs

* docs(secrets): sync qr and gateway active-surface behavior

* fix: restore stability blockers from pre-release audit

* Secrets: fix collector/runtime precedence contradictions

* docs: align secrets and web credential docs

* fix(rebase): resolve integration regressions after main rebase

* fix(node-host): resolve gateway secret refs for auth

* fix(secrets): harden secretinput runtime readers

* gateway: skip inactive auth secretref resolution

* cli: avoid gateway preflight for inactive secret refs

* extensions: allow unresolved refs in onboarding status

* tests: fix qr-cli module mock hoist ordering

* Security: align audit checks with SecretInput resolution

* Gateway: resolve local-mode remote fallback secret refs

* Node host: avoid resolving inactive password secret refs

* Secrets runtime: mark Slack appToken inactive for HTTP mode

* secrets: keep inactive gateway remote refs non-blocking

* cli: include agent memory secret targets in runtime resolution

* docs(secrets): sync docs with active-surface and web search behavior

* fix(secrets): keep telegram top-level token refs active for blank account tokens

* fix(daemon): resolve gateway password secret refs for probe auth

* fix(secrets): skip IRC NickServ ref resolution when NickServ is disabled

* fix(secrets): align token inheritance and exec timeout defaults

* docs(secrets): clarify active-surface notes in cli docs

* cli: require secrets.resolve gateway capability

* gateway: log auth secret surface diagnostics

* secrets: remove dead provider resolver module

* fix(secrets): restore gateway auth precedence and fallback resolution

* fix(tests): align plugin runtime mock typings

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-03-03 02:58:20 +00:00
Evgeny Zislis
4b4ea5df8b feat(cron): add failure destination support to failed cron jobs (#31059)
* feat(cron): add failure destination support with webhook mode and bestEffort handling

Extends PR #24789 failure alerts with features from PR #29145:
- Add webhook delivery mode for failure alerts (mode: 'webhook')
- Add accountId support for multi-account channel configurations
- Add bestEffort handling to skip alerts when job has bestEffort=true
- Add separate failureDestination config (global + per-job in delivery)
- Add duplicate prevention (prevents sending to same as primary delivery)
- Add CLI flags: --failure-alert-mode, --failure-alert-account-id
- Add UI fields for new options in web cron editor

* fix(cron): merge failureAlert mode/accountId and preserve failureDestination on updates

- Fix mergeCronFailureAlert to merge mode and accountId fields
- Fix mergeCronDelivery to preserve failureDestination on updates
- Fix isSameDeliveryTarget to use 'announce' as default instead of 'none'
  to properly detect duplicates when delivery.mode is undefined

* fix(cron): validate webhook mode requires URL in resolveFailureDestination

When mode is 'webhook' but no 'to' URL is provided, return null
instead of creating an invalid plan that silently fails later.

* fix(cron): fail closed on webhook mode without URL and make failureDestination fields clearable

- sendCronFailureAlert: fail closed when mode is webhook but URL is missing
- mergeCronDelivery: use per-key presence checks so callers can clear
  nested failureDestination fields via cron.update

Note: protocol:check shows missing internalEvents in Swift models - this is
a pre-existing issue unrelated to these changes (upstream sync needed).

* fix(cron): use separate schema for failureDestination and fix type cast

- Create CronFailureDestinationSchema excluding after/cooldownMs fields
- Fix type cast in sendFailureNotificationAnnounce to use CronMessageChannel

* fix(cron): merge global failureDestination with partial job overrides

When job has partial failureDestination config, fall back to global
config for unset fields instead of treating it as a full override.

* fix(cron): avoid forcing announce mode and clear inherited to on mode change

- UI: only include mode in patch if explicitly set to non-default
- delivery.ts: clear inherited 'to' when job overrides mode, since URL
  semantics differ between announce and webhook modes

* fix(cron): preserve explicit to on mode override and always include mode in UI patches

- delivery.ts: preserve job-level explicit 'to' when overriding mode
- UI: always include mode in failureAlert patch so users can switch between announce/webhook

* fix(cron): allow clearing accountId and treat undefined global mode as announce

- UI: always include accountId in patch so users can clear it
- delivery.ts: treat undefined global mode as announce when comparing for clearing inherited 'to'

* Cron: harden failure destination routing and add regression coverage

* Cron: resolve failure destination review feedback

* Cron: drop unrelated timeout assertions from conflict resolution

* Cron: format cron CLI regression test

* Cron: align gateway cron test mock types

---------

Co-authored-by: Tak Hoffman <781889+Takhoffman@users.noreply.github.com>
2026-03-02 09:27:41 -06:00
0xbrak
4637b90c07 feat(cron): configurable failure alerts for repeated job errors (openclaw#24789) thanks @0xbrak
Verified:
- pnpm install --frozen-lockfile
- pnpm check
- pnpm test -- --run src/cron/service.failure-alert.test.ts src/cli/cron-cli.test.ts src/gateway/protocol/cron-validators.test.ts

Co-authored-by: 0xbrak <181251288+0xbrak@users.noreply.github.com>
Co-authored-by: Tak Hoffman <781889+Takhoffman@users.noreply.github.com>
2026-03-01 08:18:15 -06:00
NIO
ea3955cd78 fix(cron): add retry policy for one-shot jobs on transient errors (#24355) (openclaw#24435) thanks @hugenshen
Verified:
- pnpm install --frozen-lockfile
- pnpm check
- pnpm test -- --run src/cron/service.issue-regressions.test.ts src/config/config-misc.test.ts

Co-authored-by: hugenshen <16300669+hugenshen@users.noreply.github.com>
Co-authored-by: Tak Hoffman <781889+Takhoffman@users.noreply.github.com>
2026-03-01 06:58:03 -06:00
Gustavo Madeira Santana
eff3c5c707 Session/Cron maintenance hardening and cleanup UX (#24753)
Merged via /review-pr -> /prepare-pr -> /merge-pr.

Prepared head SHA: 7533b85156
Co-authored-by: gumadeiras <5599352+gumadeiras@users.noreply.github.com>
Co-authored-by: shakkernerd <165377636+shakkernerd@users.noreply.github.com>
Reviewed-by: @shakkernerd
2026-02-23 22:39:48 +00:00
Advait Paliwal
bc67af6ad8 cron: separate webhook POST delivery from announce (#17901)
* cron: split webhook delivery from announce mode

* cron: validate webhook delivery target

* cron: remove legacy webhook fallback config

* fix: finalize cron webhook delivery prep (#17901) (thanks @advaitpaliwal)

---------

Co-authored-by: Tyler Yust <TYTYYUST@YAHOO.COM>
2026-02-16 02:36:00 -08:00
Advait Paliwal
115cfb4430 gateway: add cron finished-run webhook (#14535)
* gateway: add cron finished webhook delivery

* config: allow cron webhook in runtime schema

* cron: require notify flag for webhook posts

* ui/docs: add cron notify toggle and webhook docs

* fix: harden cron webhook auth and fill notify coverage (#14535) (thanks @advaitpaliwal)

---------

Co-authored-by: Tyler Yust <TYTYYUST@YAHOO.COM>
2026-02-15 16:14:17 -08:00
Gustavo Madeira Santana
e19a23520c fix: unify session maintenance and cron run pruning (#13083)
* fix: prune stale session entries, cap entry count, and rotate sessions.json

The sessions.json file grows unbounded over time. Every heartbeat tick (default: 30m)
triggers multiple full rewrites, and session keys from groups, threads, and DMs
accumulate indefinitely with large embedded objects (skillsSnapshot,
systemPromptReport). At >50MB the synchronous JSON parse blocks the event loop,
causing Telegram webhook timeouts and effectively taking the bot down.

Three mitigations, all running inside saveSessionStoreUnlocked() on every write:

1. Prune stale entries: remove entries with updatedAt older than 30 days
   (configurable via session.maintenance.pruneDays in openclaw.json)

2. Cap entry count: keep only the 500 most recently updated entries
   (configurable via session.maintenance.maxEntries). Entries without updatedAt
   are evicted first.

3. File rotation: if the existing sessions.json exceeds 10MB before a write,
   rename it to sessions.json.bak.{timestamp} and keep only the 3 most recent
   backups (configurable via session.maintenance.rotateBytes).

All three thresholds are configurable under session.maintenance in openclaw.json
with Zod validation. No env vars.

Existing tests updated to use Date.now() instead of epoch-relative timestamps
(1, 2, 3) that would be incorrectly pruned as stale.

27 new tests covering pruning, capping, rotation, and integration scenarios.

* feat: auto-prune expired cron run sessions (#12289)

Add TTL-based reaper for isolated cron run sessions that accumulate
indefinitely in sessions.json.

New config option:
  cron.sessionRetention: string | false  (default: '24h')

The reaper runs piggy-backed on the cron timer tick, self-throttled
to sweep at most every 5 minutes. It removes session entries matching
the pattern cron:<jobId>:run:<uuid> whose updatedAt + retention < now.

Design follows the Kubernetes ttlSecondsAfterFinished pattern:
- Sessions are persisted normally (observability/debugging)
- A periodic reaper prunes expired entries
- Configurable retention with sensible default
- Set to false to disable pruning entirely

Files changed:
- src/config/types.cron.ts: Add sessionRetention to CronConfig
- src/config/zod-schema.ts: Add Zod validation for sessionRetention
- src/cron/session-reaper.ts: New reaper module (sweepCronRunSessions)
- src/cron/session-reaper.test.ts: 12 tests covering all paths
- src/cron/service/state.ts: Add cronConfig/sessionStorePath to deps
- src/cron/service/timer.ts: Wire reaper into onTimer tick
- src/gateway/server-cron.ts: Pass config and session store path to deps

Closes #12289

* fix: sweep cron session stores per agent

* docs: add changelog for session maintenance (#13083) (thanks @skyfallsin, @Glucksberg)

* fix: add warn-only session maintenance mode

* fix: warn-only maintenance defaults to active session

* fix: deliver maintenance warnings to active session

* docs: add session maintenance examples

* fix: accept duration and size maintenance thresholds

* refactor: share cron run session key check

* fix: format issues and replace defaultRuntime.warn with console.warn

---------

Co-authored-by: Pradeep Elankumaran <pradeepe@gmail.com>
Co-authored-by: Glucksberg <markuscontasul@gmail.com>
Co-authored-by: max <40643627+quotentiroler@users.noreply.github.com>
Co-authored-by: quotentiroler <max.nussbaumer@maxhealth.tech>
2026-02-09 20:42:35 -08:00
Peter Steinberger
bcbfb357be refactor(src): split oversized modules 2026-01-14 01:17:56 +00:00