Commit Graph

358 Commits

Author SHA1 Message Date
Spencer Fuller
3374f78ad8 fix(queue): prevent cron saturation from starving hook dispatch (#116666)
* feat(hooks): dispatch hook agent runs into a dedicated command lane

Hook agent runs passed lane:"cron", which resolveCronAgentLane remaps to
cron-nested — the same lane cron's own inner agent work uses, capped at the
hardcoded cron budget of 8. Eight busy cron turns therefore starved every hook.

Adds CommandLane.HookDispatch and dispatches hook runs into it. Neither lane
resolver needs changing: resolveCronAgentLane (agents/lanes.ts:15-22) and
resolveGlobalLane (embedded-agent-runner/lanes.ts:11-18) special-case only
"cron" and pass every other lane through.

This is lane identity only. It does NOT yet bound aggregate capacity — that is
the capacity group in the following commits. On its own this widens total
command-lane concurrency by the hook lane's width (1).

Consumers that inferred cron-ness from the lane, both preserved rather than
silently changed:
- heartbeat-runner-execution: HookDispatch added to the busy-lane check so hook
  work still suppresses heartbeats; only the lane it occupies changed.
- session-suspension: explicit resume concurrency and gateway-managed-lane
  membership, instead of falling through to the custom-lane default.

server-lanes publishes the lane at width 1: the guarantee is that a hook can
always start under cron saturation, not that hooks run concurrently.

Test: server.hooks-lane.test.ts asserts the dispatched lane and that it survives
cron lane resolution unremapped, with a positive control on the inputs that DO
remap. Mutation-verified — reverting the call site to "cron" fails it with
'expected cron to be hook-dispatch'. Nothing else in the suite reads the
dispatched lane, so without this assertion the change regresses silently.

Refs: openclaw#98813, openclaw#43235

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(queue): capacity groups with hard per-member reservations

Adds optional capacity groups to the command queue: lanes in a group share one
hard aggregate budget, and a member may hold a non-borrowable reservation
within it. This is what makes a separate hook lane safe — it bounds hook and
cron-nested work together at the existing cron cap instead of adding a slot
outside it (openclaw#98813 maintainer audit measured cron-nested=8 + hook-a=1 +
hook-b=1 = 10).

Group capacity is always DERIVED from members' activeTaskIds, never a separate
counter. Timeout, abort, clear, reset and stale-generation completion therefore
release capacity for free, because they all remove the task id; the only
remaining obligation is that those paths re-drain the group.

- setCommandLaneGroup / clearCommandLaneGroup / drainCommandLaneGroup
- admission: lane max, then group budget, then sibling reservations. A member
  may burst above its own reservation only into unreserved capacity.
- both completion paths (success AND error) wake group siblings; freed capacity
  belongs to the group, so a lane-local pump would strand a queued sibling
  behind capacity that is already free. resetCommandLane likewise.
- membership lives in the queue singleton keyed by lane name, NOT in LaneState,
  so setCommandLaneConcurrency cannot detach a member from its group.
- deadlock guard: rejects cron/main/subagent/nested and session:*/nested:*/
  context-engine-turn-maintenance:* — lanes that can be synchronously awaited,
  where a group wait would become a deadlock.
- rejects sum(reservations) > budget rather than starving silently.
- snapshot exposes group/groupActive/groupBudget/reservedForLane/blockedBy.

Tests: 9 new, all mutation-verified — dropping group admission fails 5,
removing the sibling wake fails 2, making reservations borrowable fails 2
("expected 4 to be 3": an idle sibling's reserve being borrowed). 57 pass
across all command-queue suites.

Not yet wired: no group is configured by default. That, plus group-wait
visibility and atomic publish, are the following commits.

Refs: openclaw#98813, openclaw#43235

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(queue): blockedBy answers hypothetical immediate admission, not queue state

Round-4 review (costaff-lapclaw-001) named this as the precision requirement
that decides whether the wait-visibility fix is vacuous:

  noteLaneWaitIfBusy runs BEFORE enqueue, so it snapshots the lane with
  queuedCount === 0. If blockedBy were populated only for an already-queued
  head entry, that snapshot would read "not blocked", no
  onLaneWait(waiting:true) would fire, and agent-watchdog's setup-timeout
  suppression would never engage — a run merely waiting on group capacity
  would take a false setup timeout.

resolveLaneBlockReason already answers "could this lane start work right now?"
independent of queue contents; these tests pin that contract:

- 7 cron active, hook holding the group's reserved slot: cron reports
  sibling-reservation with queuedCount 0 and activeCount < maxConcurrent, while
  the hook reports null (so the assertion discriminates rather than being
  always-truthy).
- a member lane that was never enqueued or was retired while idle still reports
  its group block state, instead of the not-found path returning a bare default
  that reads as free.

Mutation-verified: gating blockedBy on queue.length > 0 fails 3 tests with
'expected null to be sibling-reservation' — the exact symptom predicted.

Refs: openclaw#98813

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(queue): atomic lane publication, group-aware wait visibility, opt-in group

Closes the remaining two round-2 blockers and wires the default group.

publishLaneConfiguration({lanes, groups, clearGroups}) applies lane maxima and
group definitions as ONE transaction: install with dispatch suppressed, then a
single commit-time drain. The per-lane setter drains the instant a lane goes
positive and gateway publication was sequential, so a member could be widened
and dispatch BEFORE its group existed — admitting work above the budget the
group was meant to enforce. Validation throws before any drain, so a rejected
configuration cannot strand lanes widened and ungoverned.

lane-controller's noteLaneWaitIfBusy now also emits a wait when
snapshot.blockedBy != null. A group-blocked member has
activeCount < maxConcurrent and can have queuedCount === 0, so both lane-local
terms were false while the task genuinely could not start. This is not just
observability: agent-watchdog.ts suppresses the cron setup timeout only while
waitingForLane is true, so an invisible group wait produced a FALSE setup
timeout for cron-shaped runs.

The group is opt-in on hooks.enabled. The reservation is a real cost — it
withholds a slot from cron inner work even while the hook lane is idle — so it
is only paid where it buys something. This surfaced as a genuine regression:
server-lanes.test.ts asserts cron-nested alone reaches all 8, which an
unconditional group breaks. With hooks off, no group is installed, cron keeps
the entire budget, and such a deployment sees no behaviour change at all.
Turning hooks off on reload tears the group down (clearGroups).

With hooks ON, cron inner work trades one slot for the guarantee that hooks
cannot be starved. Aggregate stays exactly the pre-existing cron cap — no slot
added outside it, which is what openclaw#98813 was held for.

Tests: 6 new (3 publication, 3 opt-in), 135 passing across all affected suites.
Mutation-verified:
- sequential per-lane publication: 'expected 12 to be less than or equal to 8',
  the exact 8+4 additive leak, caught at PEAK not post-state as review required
- unconditional group: 'expected cron-hooks to be undefined'

Refs: openclaw#98813, openclaw#43235

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(queue): pin group-blocked lane waits to the setup-timeout suppression chain

Round-4 review (fiducian-spencer-001) asked for the 8-cron / hook-holding-
reserve / 8th-cron-waiting regression asserting no false setup timeout.

The chain spans three files:
  lane-controller.noteLaneWaitIfBusy -> onLaneWait({waiting:true})
  -> timer-job-runner.noteLaneState  -> watchdog.noteLaneWait()
  -> agent-watchdog:159-164          -> waitingForLane = true, clear timeout
  -> agent-watchdog:98               -> setup timeout suppressed

The watchdog end is already covered by agent-watchdog.test.ts. The link this
change introduced is the FIRST one, and it is the one that fails silently: a
group-blocked lane looks idle to a lane-local view, so no wait is reported and
a healthy run queued behind group capacity takes a false setup timeout.

The predicate was an inline closure, so it was untestable without the full
runner harness — and asserting a copy of it in a test would prove nothing.
Extracted as shouldNoteLaneWait(snapshot) and driven with real snapshots from a
real group:

- 7 cron active, hook holding the reserve: the test asserts explicitly that
  BOTH lane-local terms are false (activeCount 7 < maxConcurrent 8,
  queuedCount 0) and that the predicate still reports a wait.
- a hook blocked by a full group budget reports a wait.
- negative control: lanes that can start immediately report no wait, so a
  predicate hardcoded to true would fail.
- ordinary lane-local saturation still reports a wait (pre-existing behaviour).

Mutation-verified: reverting to the lane-local predicate fails 3 tests with
'expected false to be true'. 182 pass across all affected suites.

Refs: openclaw#98813

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(queue): make rejected publication a no-op; always tear down the group on hooks-off

Round-5 implementation review (fiducian-spencer-001, CHANGES REQUESTED) found
one real atomicity bug and two test gaps. Both bugs are fixed and both are now
mutation-guarded.

BLOCKER — rejected publishLaneConfiguration left lane maxima mutated.
Phase 1 widened lanes, then setCommandLaneGroup could throw (e.g.
sum(reservations) > budget) with no rollback. No drain ran, so the old test's
activeCount === 0 assertion passed — but the lane sat at the new width governed
by NO group, and the next unrelated drain trigger would dispatch its preserved
queue ungoverned. The function comment promised exactly what the code did not
do. Validation is now a distinct phase 0 over every group spec before anything
is mutated; validateCommandLaneGroupSpec/installCommandLaneGroup split out so
setCommandLaneGroup and the transaction share one validation path.

BUG — hooks-off skipped group teardown when the grouped lane was suspended.
applyGatewayLaneConcurrency published only when the lane map was non-empty. With
hooks off, cron-nested is the only lane that can enter it, so a suspended
cron-nested left the map empty and clearGroups was never published. A previously
installed cron-hooks group survived, and the member resumed still paying a
reservation for a hook lane receiving no work. Now publishes whenever hooks are
disabled, regardless of the lane map.

Also (review item 4): a lane may now belong to at most one group.
installCommandLaneGroup removes it from any prior owner's members, which
otherwise kept counting its active tasks toward a budget it had left. Not
reachable with the single default group, but this is a public API.

Tests: 3 new. Mutation-verified —
- validating during install instead of before: 'expected 8 to be +0'
- restoring the non-empty-lane-map guard: 'expected cron-hooks to be undefined'
  (this one initially SURVIVED; the first version of the teardown test never
  simulated suspension, so it could not see the bug. Now seeds a cleared lane
  resume and publishes via the gatewayStart path.)
- hooks-off now proves it DRAINS the work its teardown releases, not just that
  membership was deleted.

Test teardown fixed: resetAllLanes preserves queued entries by design, so work
on a lane that never opened never settles. clearCommandLane rejects it instead.

Typecheck clean (tsgo core + core test, exit 0).

Refs: openclaw#98813

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(queue): make setCommandLaneGroup self-waking; guard the clearGroups+invalid case

Round-5 implementation review from costaff-lapclaw-001 independently found the
same two bugs fiducian did (rejected-publish partial mutation, and hooks-off
teardown skipped when the grouped lane is suspended) — both already fixed in
0eb96a7. It raised two things fiducian did not:

1. The exported setCommandLaneGroup primitive was not self-waking. Replacing a
   group can FREE capacity — wider budget, dropped reservation, removed member —
   and queued members must not sit behind capacity that is already available.
   publishLaneConfiguration drains at commit, but the bare primitive is exported
   and its "replace" semantics silently stranded members until an unrelated
   poke. Now drains the union of previous and next members.

2. clearGroups combined with an invalid replacement was the worst case: the old
   group could be removed before the new one threw, leaving BOTH lane width and
   group membership partially committed. Phase 0 validation already ran before
   the clear after 0eb96a7, but nothing pinned it.

Also documents a limitation costaff noted: reservations are not validated
against the member's own maxConcurrent, because lane widths and group
definitions are published together and the width may not be applied yet at
validation time. A too-large reservation is accepted but partly unusable.

Tests: 2 new. Mutation-verified —
- removing the self-wake: 'expected 2 to be 5'
- validating during install instead of before the clear:
  'expected undefined to be cron-hooks' (the existing group torn down by a
  rejected replacement — costaff's exact worst case)

Reviewer agreement on the rest: admission arithmetic sound for the concrete
config, clearCommandLane correctly not wired (frees no active capacity), the
peak-occupancy publication test is the right shape, the static deadlock deny set
matches the known synchronous-wait lanes, and shouldNoteLaneWait's export is the
right trade over asserting a copied closure.

Typecheck clean (tsgo core + core test, exit 0).

Refs: openclaw#98813

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* style(queue): satisfy oxlint in the capacity-group tests

`check-lint` was failing on 21 errors across the four new suites:

- 17 `curly`: single-statement `for`/`for-of` bodies without braces.
- 4 `no-promise-executor-return`: `new Promise((resolve) => setTimeout(resolve, 0))`
  implicitly returns the Timeout handle from the executor. Rewritten to the
  braced form already used ~85 times elsewhere in the repo, e.g.
  `src/plugins/install-paths.test.ts:41`.

No behaviour change; the suites pass unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(queue): split capacity groups and shared state out of command-queue.ts

`check-lint` was failing `max-lines` on src/process/command-queue.ts: 918
counted lines against the repo cap of 700. The file was already only 51 lines
under the cap before this branch, so the new capacity-group code could not fit
in it.

Two pure moves, no logic change:

- `command-queue.state.ts` — the globalThis-backed queue singleton
  (`getQueueState`, same `Symbol.for` key), `normalizeLane`, and the
  `QueueEntry` / `LaneState` / `ActiveTaskWaiter` / `CommandLaneTaskMarker`
  types. Lets the group policy read lane state without importing the queue.
- `command-queue.capacity-groups.ts` — the group registry, eligibility policy,
  spec validation, install, and the block-reason computation.

The four near-identical "drain these member lanes" loops collapse into one
`drainMembers` helper. It keeps the load-bearing part of each original: the
lane is looked up rather than created, because `drainLane` goes through
`getLaneState` and would resurrect a scoped lane that
`retireIdleScopedCommandLane` had just removed. The one loop that additionally
tested `maxConcurrent > 0` loses that check, which was an optimisation only —
a zero-width lane's pump admits nothing.

The dependency on `drainLane` is passed in as a parameter rather than imported,
so the new modules stay acyclic; `setCommandLaneGroup`, `clearCommandLaneGroup`
and `drainCommandLaneGroup` remain exported from command-queue.ts as thin
wrappers, and every previously exported symbol is still exported from there.

command-queue.ts is now 676 counted lines; all three modules are under the cap.

Verified: `tsgo:core`, `tsgo:core:test`, `oxfmt --check`, `oxlint` all clean;
1007 tests across the 43 suites that touch command-queue pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(queue): allow concurrent hook dispatch within cron budget

* test(gateway): prove hook burst concurrency stays bounded

* refactor(queue): keep capacity groups internal

* test(gateway): isolate steady-state hook admission

* fix(gateway): close hook lane on disable

* fix(gateway): retarget suspended hook resumes

* fix(gateway): restore hook group before lane resume

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
2026-08-01 11:49:16 +08:00
Peter Steinberger
c402688894 feat(media): probe duration and dimensions for playback metadata (#115728)
* feat(media): probe playback metadata

* fix(media): satisfy CI gates

* fix(media): satisfy lint rules
2026-07-29 04:52:17 -04:00
Peter Steinberger
34c90a8cb3 fix(process): prevent orphaned Windows child process trees (#115535)
* test(process): reproduce Windows taskkill process-tree leak

* fix(process): stop leaked Windows child process trees

Escalate only when Windows taskkill reports that graceful process-tree termination failed. Preserve awaited taskkill completion, grace-period fallback, one-shot signaling, and PID-reuse protection.

Closes #110789

Supersedes #112202

Co-authored-by: Mohammed Alkindi <alkndymhmd692@gmail.com>

---------

Co-authored-by: Mohammed Alkindi <alkndymhmd692@gmail.com>
2026-07-29 00:26:32 -04:00
Peter Steinberger
ee81498487 refactor(process): remove obsolete process cancellation paths (#115416)
* refactor(process): unify cancellation and terminal state

* test(cli): align cancellation supervisor with active runs
2026-07-28 18:12:48 -04:00
Peter Steinberger
8bfb4a550e fix(process): preserve the first process cancellation reason (#115317)
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-28 13:35:08 -04:00
Peter Steinberger
4cea6daf5a fix(process): preserve long supervisor timeout deadlines (#115230)
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-28 10:38:43 -04:00
Peter Steinberger
8dac8967f9 fix(process): never start cancelled queued replacements (#114951) 2026-07-28 01:58:10 -04:00
Peter Steinberger
7bc36b76be fix(process): honor cancellation during process startup (#114917) 2026-07-28 00:26:25 -04:00
Peter Steinberger
1292262996 fix(process): prevent concurrent scoped replacement leaks (#114845) 2026-07-27 22:02:33 -04:00
Peter Steinberger
d2dc61cddf chore(process): drop Bun spawn-encoding workaround (fixed upstream in oven-sh/bun#36050) (#114606) 2026-07-27 10:36:58 -04:00
Peter Steinberger
a05cfd4756 feat(runtime): run OpenClaw under Bun runtimes that provide node:sqlite (#114256)
* feat(runtime): allow Bun runtimes that provide node:sqlite

* fix(process): drop execa buffer encoding under Bun spawn (Bun rejects non-spawn options)

* chore(process): cite oven-sh/bun#36049 in bun spawn workaround

* docs(install): bun with node:sqlite can run openclaw; bun install workspace caveat

* fix(process): clear execa buffer encoding under Bun without mutating read-only options
2026-07-26 23:44:09 -04:00
Peter Steinberger
c4f148368a fix(agents): retire completed session work lanes (#114051) 2026-07-26 02:27:42 -04:00
Peter Steinberger
3b7b2a2a1f chore: update dependencies and migrate major contracts (#112963)
* build(deps): complete latest dependency migrations

* fix(deps): satisfy updated dependency types

* fix(deps): hold incompatible build tooling

* fix(deps): preserve portable tooling contracts

* build(deps): allow reviewed fresh transitive releases

* fix(deps): repair major upgrade validation

* build(deps): regenerate current dependency graph

* fix(logging): keep tslog adapter type private

* fix(agents): narrow grep subprocess handle

* fix(codex): prefer pinned managed binary

* fix(codex): fence managed native provenance

* build(deps): align codex ACP with managed harness

* fix(slack): use socket-mode Undici runtime

* fix(slack): detect cross-runtime responses

* fix(slack): bridge package-owned fetch types

* fix(deps): retain tslog v4 JSON contract

* build(plugin-sdk): refresh logging API manifest
2026-07-23 21:21:01 -07:00
Peter Steinberger
ad505a7b55 fix(swarm): keep collector results reliable through races and restarts (#112989)
* fix(swarm): harden collector lifecycle and dashboards

* fix(swarm): initialize collector completion state

* test(swarm): satisfy cross-environment type checks

* test(codex): allow direct request handler calls

* style(ui): avoid Swarm widget shadowing

* test(swarm): keep internal helpers private

* refactor(ui): own Swarm roster helpers in runtime
2026-07-23 06:26:31 -07:00
Peter Steinberger
3ed2a144ac fix(process): keep secret pipe errors handled (#112550) 2026-07-21 23:59:19 -07:00
Jason (Json)
1a42e005fb fix(anthropic): forward selected profiles to Claude CLI (#112458)
* fix(anthropic): forward Claude CLI auth profiles

* fix(system-agent): inject CLI auth route stores

* fix(claude-cli): pass profile credentials by descriptor

* fix(anthropic): repair selected profile CI coverage

* fix(anthropic): preserve profile owner validation

* test(system-agent): preserve selected profile fixtures

* test(system-agent): narrow selected profile fixture

* test(system-agent): resolve profile store merge

* fix(anthropic): forward profiles to node Claude runs

* fix(system-agent): reconcile profile route projection

* test(system-agent): thread profile store through projection

* fix(anthropic): make selected profile authoritative

* fix(system-agent): type auth setup failures

* fix(system-agent): type setup auth failures

* style: format Claude profile maintenance

* fix(anthropic): keep gateway credentials off nodes

* fix(anthropic): clear ambient auth for selected profiles

* fix(anthropic): secure paired-node Claude auth

* fix(node-host): type Claude fd spawn streams

* style(node-host): satisfy Claude spawn lint

* fix(process): capture exit before secret delivery

* fix(anthropic): preserve node-native Claude auth
2026-07-21 23:27:37 -06:00
Peter Steinberger
edecdbd05e refactor(config): config-surface reduction tranche 3 — product consolidations (review request) (#111527)
* refactor(config): consolidate media model lists

* refactor(config): unify memory configuration

* refactor(config): consolidate TTS ownership

* refactor(config): move typing policy to agents

* refactor(config): retire product-level config surfaces

* refactor(config): share scoped tool policy type

* chore(config): refresh generated baselines

* fix(config): honor agent typing overrides

* fix(config): migrate sibling config consumers

* refactor(infra): keep base64url decoder private

* fix(config): strip invalid legacy TTS values

* chore(config): refresh rebased baseline hash

* fix(doctor): route legacy messages.tts.realtime voice to talk during tts move

* refactor(config): polish final layout names

* refactor(config): freeze retired tuning defaults

* feat(config): add fast mode default symmetry

* refactor(config): key agent entries by id

* docs(config): update final layout reference

* test(config): cover final layout migrations

* chore(config): refresh final layout baselines

* fix(config): align final layout runtime readers

* fix(config): align remaining readers

* fix(config): stabilize final layout migrations

* fix(config): finalize config projection proof

* fix(config): address final layout review

* docs(release): preserve historical config names

* fix(config): complete keyed agent migration

* fix(config): close final migration gaps

* fix(config): finish full-branch review

* fix(config): complete runtime secret detection

* fix(config): close final review findings

* fix(config): finish canonical docs and heartbeat migration

* fix(config): integrate latest main after rebase

* refactor(env): isolate test-only controls

* refactor(env): isolate build and development controls

* refactor(env): collapse process identity indirection

* refactor(env): remove duplicate config and temp aliases

* docs(env): define the operator-facing allowlist

* ci(env): ratchet production variable count

* fix(env): remove stale provider helper import

* fix(env): make ratchet sorting explicit

* test(env): keep test seam in dead-code audit

* test(env): cover ratchet growth and boundary; document surface budgets

* docs(config): document tier-eval consolidations

* docs(config): clarify speech preference ownership

* test(memory): align retired tuning fixtures

* refactor(memory): freeze engine heuristics

* refactor(config): apply tier-eval tranche

* refactor(tts): move persona shaping to providers

* refactor(compaction): move prompt policy to providers

* test(config): align hookified prompt fixtures

* chore(deadcode): classify test-only exports

* chore(github): remove unused spawn helper

* chore(deadcode): classify queue diagnostics

* chore(deadcode): remove unused lane snapshot export

* chore(plugin-sdk): ratchet consolidated surface

* fix(config): integrate latest main after rebase
2026-07-21 20:28:43 -07:00
Peter Steinberger
98742bc2c7 feat(cron): stream schedule sources with durable source identity (#112387)
* feat(cron): stream schedule sources (supervised command stdout)

Add gated argv stream schedules with bounded line batching and trigger.streamBatch composition.

Reuse the gateway ProcessSupervisor for source ownership, deterministic teardown, capped restart backoff, and schedule-key guarded batch execution. Expose additive protocol, CLI, tool, UI, docs, and generated snapshot surfaces without storage DDL.

Contract: stream schedules are event-driven, require cron.triggers.enabled, reject command payloads, and retain at most one bounded pending batch.

* fix(cron): reject retired stream source epochs at run admission

Thread an invalidatable per-owner source-generation token (ownerNonce.generation)
through cron.run admission alongside the schedule key. A batch handed to cron.run
under one source epoch can wait behind another run while its owner is stopped; a
disable→re-enable or A→B→A edit leaves the schedule key unchanged, so the key
check alone would admit the retired epoch's batch. The token is persisted in
job.state on every lifecycle write and compared at every admission site plus the
executeJobCore guard, so a stale epoch's batch is skipped.

Also fix direct stream-job mutations recording the wrong lifecycle status when
global cron is off but triggers are on: extract resolveStreamStopReason so the
direct path reports the remediable cron-disabled state like reconcile does.

* fix(cron): close stream admission windows from round-10 review

- Persist the retired source generation before draining stop teardown, so a
  batch queued behind another cron run cannot gain admission during the up-to-10s
  in-flight-batch wait (server-cron routed stop path).
- Add streamSourceGeneration to the closed gateway response schema (excluded from
  the writable patch schema) so a running stream job passes strict result
  validation without letting callers spoof source identity.
- Close the mutation-epoch ABA: track an eviction epoch so a snapshotted absent-0
  is trusted as unchanged only when no LRU eviction happened during the await.

* refactor(cron): stream sources own a durable logical identity

Split the conflated restart-generation/admission token into two concepts:
a persisted streamSourceIdentity owned by cron store mutations (rotates on
enable/disable, source replacement, once-trigger auto-disable, and explicit
retirement; stable across supervised child restarts) and a watcher-local
process generation used only to fence stale child callbacks. Admission now
requires schedule key + identity together at every window, closing the
A-to-B-to-A and restart-flush races from review rounds 8-11.

Also: match-mode regexes now see raw source text (the [truncated] marker is
applied after matching), stop-timeout failures set the live restartExhausted
mirror so shutdown preserves the terminal diagnostic, and the watcher is
split into owner/output/registry modules under the max-lines budget.

* fix(cron): harden stream teardown and intake from round-4 review

An exit queued ahead of a requested stop no longer counts toward restart
exhaustion (the synchronous stop fence owns it), overlapping cron.stop and
stopAndDrain share one memoized shutdown drain instead of double-stopping
every owner, raw output intake is bounded at 4x the batch cap so normal
64 KiB pipe reads stop losing complete lines to OS chunk boundaries, and
persisted-shape quarantine coverage for unsafe match expressions is pinned.

* fix(cron): keep watcher-internal owner disposal from retiring live identity

Disposing an obsolete owner while a start replaces it is not a durable
removal; a retiring stop there rotated the live job's identity and stranded
the replacement behind the CAS ownership guard. Also align the docs with the
implemented match semantics: complete lines match on full text past the
batch cap, only intake-cut prefixes are unmatchable.

* fix(cron): make oversized-line matching independent of pipe chunking

Partial lines are retained up to the raw-intake bound rather than the
delivery cap, so a complete over-cap line matches identically whether it
arrives in one callback or several; only a line the intake bound itself cut
remains an unprovable prefix. Reconcile also contains schedule-replacement
stop failures per job, matching the other stop branches.

* fix(cron): bound assembled lines, drop stale payload override, barrier stopAll

- enforce the 4x raw-intake per-line cap while assembling split callbacks,
  so an oversized line stays an unprovable prefix regardless of chunking
- stop passing the watcher-cached payload as a cron.run override; the run
  snapshots the persisted payload under its admission lock
- stopAll waits for every owner stop to settle before surfacing failures
- split cron-stream-output interleaving tests into their own file (max-lines)

* fix(cron): address first full-CI round (lint, knip, schema test, unused param)

* chore(cron): refresh codex prompt snapshots for stream schedule schema

* fix(cron): keep the first clean line after an intake drop ending at a newline

* fix(cron): fence stream reconcile list snapshots against direct mutation routes

A cron.list snapshot captured across the reconcile await could be applied
after a direct add/update route already started the owner, stopping it as
removed and retiring its live identity. A mutation revision bumped at every
direct route start invalidates the stale snapshot; reconcile re-lists
(bounded) instead of applying it.

* style(cron): format stream owner imports

* fix(cron): discard severed stream prefixes at EOF

* fix(cron): retry failed stream shutdown drains

* fix(cron): honor stream stop fence after output drain

* chore(cron): refresh landing checks
2026-07-21 15:01:30 -07:00
mikasa
a8a6e2a828 fix(process): bound Windows exec timeout cleanup (#104234)
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-21 09:26:04 -07:00
Peter Steinberger
6f43c50f37 fix(cli): preserve failure exit semantics (#112210) 2026-07-21 01:48:25 -07:00
LZY3538
8c9ecd9242 fix(process): use a valid PTY type when TERM is blank (#111261)
* fix(process): ignore blank PTY terminal names

* fix(process): normalize PTY terminal identity

Co-authored-by: LZY3538 <liu.zhenye@xydigit.com>

* fix(process): keep PTY default private

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-19 22:58:24 -07:00
Peter Steinberger
9a94beace7 fix(process): preserve descendant output under event-loop stalls (#111040)
* fix(process): drain buffered descendant output after exit

* test(process): cover deferred output release phase
2026-07-18 17:00:08 -07:00
mushuiyu886
4e11da2872 fix(process): UTF-8 command output corrupts at Windows byte limits (#105274)
* fix(process): preserve byte-capped UTF-8 output on Windows

* fix(process): skip codepage probe for UTF-8 output

Co-authored-by: 杨浩宇0668001029 <yang.haoyu@xydigit.com>

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-18 22:44:12 +01:00
Harjoth Khara
00eb33fe8e fix(cron): stop a cron job's own marker from blocking its awaited wake (#109440)
* fix(cron): deliver owning heartbeat synchronously

* fix(test): drop await on synchronous cron.stop()

CronService.stop() is synchronous (src/cron/service.ts:74); awaiting it
trips oxlint typescript(await-thenable) on check-lint. Matches sibling
cron tests, which all call cron.stop() bare.

* fix(cron): harden awaited wake ownership

Co-authored-by: harjoth <harjoth.khara@gmail.com>

* test(cron): track heartbeat sandbox cleanup

* fix(cron): scope awaited wake queue ownership

* chore: leave cron notes to release generation

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-18 07:22:59 +01:00
Peter Steinberger
72335d3d4a test(process): avoid exec timeout race (#110161) 2026-07-17 22:56:44 +01:00
Wynne668
ccb251c570 fix(process): report actual elapsed time for early lane timeouts (#109287)
* fix(process): clarify command lane timeout cause

* fix(process): complete timeout cause formatting

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-16 17:03:43 -07:00
thomas.szbay
b06fe2a673 fix(kill-tree): verify process group leader before using group kill to prevent gateway SIGTERM (#76259) (#94697)
* fix(kill-tree): verify process group leader before group kill to prevent gateway SIGTERM (#76259)

- Add isProcessGroupLeader() to killProcessTree/signalProcessTree: ps -p <pid> -o pgid= primary check with /proc/<pid>/stat fallback on Linux. Group kill only when the PID is its own process group leader; non-leaders fall back to single-pid kill, preventing accidental gateway SIGTERM when a non-detached child shares the gateway's process group.
- Propagate detached: true to all detached-spawn cleanup callers (exec-termination, agent-bundle LSP, mcp-stdio, bash, supervisor pty, agent-core nodejs) so detached group cleanup survives leader exit.
- Gateway/daemon cleanup paths (schtasks, restart-health) keep the leader-checked default (detached omitted).

Closes #76259

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(process): tighten process-group ownership checks

* refactor(daemon): split restart diagnostics

* refactor(daemon): isolate restart health types

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-16 12:30:51 -07:00
Peter Steinberger
5d9c114fa6 fix: keep isolated gateways quiet and Codex terminals interactive (#108871)
* fix(gateway): harden session recovery paths

* fix(terminal): normalize padded TERM values

* chore: defer gateway release notes

* test(terminal): type TERM test environments
2026-07-16 03:35:17 -07:00
Peter Steinberger
bae9752c5a refactor(deadcode): enforce repository hard zero (#108641) 2026-07-15 22:40:00 -07:00
Peter Steinberger
957cc81175 test: speed up slow unit and browser coverage (#108563) 2026-07-15 19:59:16 -07:00
Dallin Romney
ad34552473 refactor(qa): migrate Matrix scenarios into QA Lab (#103589)
* refactor(qa): migrate Matrix scenarios into QA Lab

* fix(qa): build Matrix boundary declarations

* fix(qa): preserve Matrix preview boundaries

* fix(qa): preserve Matrix hot reload semantics

* fix(qa): harden Matrix destructive scenario failures

* fix(qa): harden Matrix scenario isolation

* fix(qa): close Matrix negative scenario blind spots

* fix(qa): isolate Matrix substrate state

* fix(qa): harden Matrix transport substrate

* fix(qa): preserve Matrix profile and event parity

* fix(qa): preserve explicit scenario models

* fix(qa): align Matrix scenario coverage taxonomy

* fix(qa): format Matrix allowlist cleanup

* fix(qa): satisfy migrated Matrix CI contracts

* fix(qa): reconcile Matrix migration with current main

* fix(qa): break scenario flow import cycle

* fix(qa): reconcile Matrix max-lines ownership

* fix(qa): address Matrix review boundaries

* fix(qa): remove stale Matrix lint suppression

* fix(qa): adopt split Matrix E2EE flows

* fix(qa): export Matrix scenario record guard

* fix(qa): align Matrix migration with privatized helpers

* refactor(qa): finish Matrix QA Lab ownership

* fix(qa): preserve Matrix suite defaults

* fix(qa): reconcile Matrix cleanup with current main

* test(qa): follow canonical Matrix profile size

* fix(qa): guard stale Matrix QA package output

* docs(qa): redirect retired Matrix QA pages

* refactor(qa): finish Matrix runner rename

* test(qa): assert Matrix defaults through profile resolver

* docs: refresh QA cleanup map

* fix(qa): privatize Matrix storage discovery
2026-07-15 01:22:20 -07:00
Peter Steinberger
ccefa7c028 test: speed exec no-output timer (#108037) 2026-07-14 22:48:11 -07:00
Peter Steinberger
a6a0716486 feat(setup): rename Crestodian to OpenClaw system agent
User-facing name is now OpenClaw (the system speaks); internal code name is
system-agent. Gateway methods crestodian.* -> openclaw.chat/openclaw.setup.*,
agent tool -> openclaw, reserved agent ids openclaw + retired crestodian.
openclaw setup routes: onboarding flags -> onboard, -m/--yes -> system agent,
bare configured interactive -> OpenClaw chat, unconfigured -> onboarding.
Hidden crestodian CLI and /crestodian TUI aliases kept; docs moved to
docs/cli/openclaw.md with redirect stub. macOS/Android strings in lockstep.

Refs #107237
2026-07-14 11:03:02 -07:00
Peter Steinberger
98de5832a7 refactor(process): remove internal export seams (#107456) 2026-07-14 05:23:29 -07:00
Ayaan Zaidi
319a796079 fix(gateway): never leave the restart admission fence closed without a restart
A failed, refused, superseded, or thrown restart emission could leave the
reversible restart-signal admission fence closed forever: concurrent emitters
could overwrite the live rollback lease with a dead stand-in, the fenced body
had no try/finally, and the outer catch swallowed errors precisely because the
stuck fence made isGatewayRestartDraining() true. The gateway then rejected
every new task with GatewayDrainingError - silently - until an operator
restarted the process.

beginGatewayRestartSignalAdmission now returns null instead of stand-in
leases (single fence owner), emitPreparedGatewayRestart reopens the fence on
every non-delivery path via try/finally while preserving it whenever a queued
SIGUSR1 is unconsumed, refused-signal cleanup force-clears orphaned fences,
and admission close/reopen transitions are logged with their reason. The
self-contained SQLite restart-intent persistence moves to restart-intent.ts
to keep restart.ts within the LOC ratchet.

Fixes #107322
2026-07-14 16:14:48 +05:30
Peter Steinberger
489690fa16 fix(process): apply undefined env removals (#107258)
Co-authored-by: Pavan Kumar Gondhi <pavangondhi@gmail.com>
2026-07-14 00:43:30 -07:00
Michael Appel
99ca3599d6 fix: block cloud sdk workspace env controls (#103918)
Co-authored-by: Pavan Kumar Gondhi <pavangondhi@gmail.com>
2026-07-14 12:46:26 +05:30
Peter Steinberger
d8d2f83cc1 feat(terminal): open Codex/Claude catalog sessions in a terminal on their owning host
Catalog session rows (sidebar context menu + click), the built-in viewer
header, and a new "Open Codex/Claude sessions in" preference can launch the
native CLI (codex resume / claude --resume) in the operator terminal on the
machine that owns the session.

- Gateway-local sessions spawn through the existing terminal launch policy
  (sandbox/enabled gates preserved) with the resume command in the session cwd.
- Paired-node sessions run through a new seq-ordered node PTY relay: a
  duplex node-host command streams PTY output via node.invoke.progress and
  receives keystrokes/resize via a new node.invoke.input event, behind the
  unchanged terminal.* client protocol (TerminalSessionManager gains a backend
  abstraction; node relay reuses the streaming-invoke controller).
- Owner boundary: each plugin owns its resume command and builds argv from a
  validated thread id; the gateway routes node opens through the node command
  allowlist and plugin invoke policy (no advertisement-only trust), and nodes
  re-verify session eligibility before spawning.
- UI setting catalogOpenTarget + canOpenTerminal capability gate every entry
  point; capability requires the owning host to actually have the CLI.

Node PATH is normalized before command-availability probes, Windows .cmd/.bat
shims spawn via ComSpec, and catalog terminal opens reattach persisted tabs
before opening the new tab.
2026-07-13 22:20:50 -07:00
Peter Steinberger
704b17d80f test(process): harden descendant timeout proof (#107011) 2026-07-13 20:01:28 -07:00
Peter Steinberger
d1684f48a3 refactor: delete dead infra and config exports (#106019)
* refactor: delete dead infra and config exports

* refactor: preserve live infra and config contracts

* refactor(config): remove obsolete file-store lifecycle APIs

* refactor(infra): finish current-main dead export cleanup
2026-07-13 12:00:47 -07:00
Peter Steinberger
db02a96c4c refactor(process): route bounded commands through Execa (#106495)
* refactor(process): centralize bounded command execution

* refactor(process): migrate core one-shot commands

* refactor(plugins): migrate one-shot commands

* fix(process): await Windows tree termination

* chore(plugin-sdk): refresh process runtime surface

* refactor(process): migrate remaining bounded commands

* refactor(process): normalize command result handling

* refactor(process): split execution responsibilities

* chore(plugin-sdk): refresh API baseline

* chore(process): remove release-owned changelog entry

* fix(process): narrow binary command input checks

* fix(process): cap sandbox command output

* fix(qa-lab): preserve exact node probe env

* chore(ci): refresh dead export baseline

* fix(process): preserve force-kill command deadlines

* fix(process): avoid post-exit timeout reclassification

* test(process): update scp staging wrapper mock

* test(process): update remaining wrapper mocks

* refactor(qa-lab): preserve Execa tar execution
2026-07-13 11:07:35 -07:00
Peter Steinberger
16ef091cbf refactor(pty): use published node-pty contracts (#106592) 2026-07-13 09:59:06 -07:00
Peter Steinberger
4f287dd740 refactor(process): adopt Execa execution layer (#105939)
* refactor(process): adopt execa execution layer

* fix(process): preserve launch error semantics

* chore(process): ratchet exec size baseline

* fix(process): preserve Windows and error contracts

* chore(plugin-sdk): ratchet wildcard surface budget

* test(process): allow resolved Windows executable paths

* fix(process): preserve Windows shim completion

* test(process): isolate Execa mocks

* style(process): format Windows exec test

* style(process): apply Windows test formatting

* test(process): normalize Windows system path casing

* fix(process): harden execa edge handling

* test(tui): preserve ESM fixture semantics

* fix(process): preserve PATHEXT lookup contract

* fix(process): keep invocation type internal
2026-07-13 02:21:08 -07:00
Peter Steinberger
e2ec8283c4 refactor(deadcode): trim mid-size src exports (#105888)
* refactor(deadcode): trim auto-reply and CLI exports

* refactor(deadcode): trim cron and task exports

* refactor(deadcode): trim fleet and process exports

* test(deadcode): exercise live task and process seams

* test(fleet): cover stream redaction through owner module

* refactor(security): trim dead internal exports

* refactor(secrets): trim dead internal exports

* refactor(deadcode): trim remaining src exports

* refactor(deadcode): remove test-only runtime exports

* refactor(deadcode): trim pairing test exports

* refactor(deadcode): reconcile refreshed baseline

* test(auto-reply): deduplicate queue state imports
2026-07-13 00:42:56 -07:00
Peter Steinberger
32c84b0f41 feat(skills): capture reusable techniques from successful work (#105674)
* feat(skills): capture reusable experience safely

* feat(skills): review completed work for reusable learning

* docs(skills): explain self-learning

* docs: clarify self-learning runtime scope

* fix(skills): harden autonomous workshop reviews

* test(skills): align review prompt fixture
2026-07-13 00:22:06 -07:00
Peter Steinberger
3616fba951 fix(gateway): make hot reload transactional (#105289)
* fix(gateway): make hot reload transactional

Replace partial reload side effects with a deferred transaction that publishes config, secrets, auth, and subsystem state together, and drains in-flight reload work before shutdown.

Co-authored-by: LZY3538 <293718838+LZY3538@users.noreply.github.com>

* fix(auth): preserve state-only credential ownership

Keep derived runtime snapshots in place for main-store state mutations so order refreshes do not look like credential replacement.

* fix(gateway): close reload transaction gaps

* fix(gateway): close merged reload gaps

* chore: move reload note to PR context

* fix(gateway): exclude restart emission root

---------

Co-authored-by: LZY3538 <293718838+LZY3538@users.noreply.github.com>
2026-07-12 18:16:15 -07:00
Vincent Koc
26c4187297 refactor(process): trim internal type surface (#105599) 2026-07-12 20:28:13 +02:00
Peter Steinberger
6b95f98fe7 fix(core): make indexed access explicit across remaining src (NUIA phase 3b) (#104773)
* fix(core): make indexed access explicit in auto-reply, infra, and config

Part 1/3 of the src NUIA phase-3b burn-down (#104600): iteration and
destructuring over index reads, boundary guards on parsed input, and
named invariants. Config path walkers bind the path head once; SQLite
migration key handling is hoisted without query-shape changes.

* fix(core): make indexed access explicit in cli, gateway, commands, security, shared

Part 2/3: argv/token selection restructured, gateway event/attachment
invariants named, security parsers stay fail-closed (invariant
violations throw), edit-distance matrices access checked entries.

* fix(core): make indexed access explicit across remaining src surfaces

Part 3/3: channels, plugins, process, cron, plugin-sdk, media, logging,
tui, hooks, daemon, and small directories. Latent bug fixed: a tailnet
resolver could leak undefined through a string|null contract and now
fails with a descriptive local error.

* fix(core): keep optional boundaries optional after per-commit review

Review findings: expectDefined misused where absence is a legitimate
state. CLI --profile/route-args missing next tokens take their existing
miss paths; help normalization compares --help against the last
positional again; first-time plugin install spreads absent cfg.plugins;
denylist scan iterates manifest dependency entries instead of throwing
on omitted sections; tailnet resolver returns a guaranteed string at
the source instead of a caller-side undefined throw.

* refactor(core): closed-key provider labels and honest optional passthroughs

PROVIDER_LABELS becomes a satisfies-typed closed record (static reads
provably defined; dynamic lookups go through providerUsageLabel with
honest string|undefined). Status-scan overview passes its optional
params through unchanged instead of asserting them.

* fix(channels): make getChatChannelMeta honestly optional

The original signature claimed ChatChannelMeta while leaking undefined
on bundled channel id metadata drift; three of four callers already
handled absence. The return type now says so, and the one assuming
caller falls back to the raw channel label.

* fix(core): index-safety for post-rebase main drift

Covers the sqlite-sessions flip and auth-source-plan code that landed
mid-phase, plus the channel-validation test consuming the now honestly
optional getChatChannelMeta.

* refactor(channels): split chat-meta accessors along the SDK contract

getChatChannelMeta keeps its shipped plugin-SDK signature (defined for
bundled ids, fail-loud on impossible misses); new findChatChannelMeta
carries the drift-tolerant optional contract for core auto-enable and
formatting paths.

* fix(qa-channel): own channel metadata instead of a guaranteed-undefined catalog lookup

qa-channel spread getChatChannelMeta over an id that is never in the
bundled catalog, shipping an empty setup meta by accident; the fail-loud
SDK accessor exposed it. The channel now declares its metadata once.

* fix(gateway): heartbeat projection lookahead is optional at the transcript tail

expectDefined wrapped messages[i + 1] whose absence on the final message
is the normal case; the adjacent ternary already handled it. Restores
the plain optional read with an explicit guard in the pair condition.

* fix(plugin-sdk): channel plugin factory tolerates non-bundled channel ids again

createChannelPluginBase spreads bundled catalog meta for ANY channel id,
where absence is the normal case for external plugins; the resolver is
honestly optional again while the exported bundled-id accessor keeps the
fail-loud contract.

* fix(core): spreads of optional config sections stay optional

Fresh-setup and first-install paths (crestodian setup inference, hook
installs, agent config base, target agent models) legitimately lack the
section being rebuilt; spreading undefined is the shipped {} semantics.
Removes the remaining gratuitous assertion wraps found by tree audit.
2026-07-11 20:47:34 -07:00
xingzhou
2077ebe8b4 fix(exec): long retained process output corrupts boundary emoji (#104531)
* fix(process): preserve Unicode in captured output tails

* test(process): consolidate captured output coverage

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-11 15:35:59 -07:00
Peter Steinberger
96f0983a85 fix(onboarding): skip setup for configured gateways and require inference first (#102883)
* fix(crestodian): keep onboarding RPCs restart-safe

* fix(profiles): isolate approval state migrations

* fix(crestodian): bypass configured gateway setup

* test(crestodian): type onboarding mocks

* fix(onboarding): require inference before Crestodian

* fix(onboarding): enforce verified inference handoff

* fix(macos): reset setup on gateway endpoint edits

* chore(i18n): refresh native source inventory

* fix(gateway): keep socket on request cancellation

* test(packaging): require workspace templates

* fix(onboarding): bind setup to verified inference

* fix(onboarding): align inference gate contracts

* fix(crestodian): classify concurrent policy rejection

* test(crestodian): expect registry restoration

* fix(onboarding): bind setup to configured gateways

* fix(codex): preserve startup phase deadlines

* test(crestodian): match fail-closed policy ordering

* test(onboarding): assert bound gateway handoff

* fix(codex): bind runtime resolution to spawn cwd

* test(crestodian): assert policy rejection order

* fix(cli): preserve gateway routing across restarts

* fix(macos): fail closed during gateway edits

* test(macos): cover gateway route generation races

* chore: keep release notes out of onboarding PR

* fix(ci): refresh onboarding generated checks

* style(swift): align gateway channel formatting

* fix(ci): refresh plugin SDK surface budgets

* fix(ci): resync native string inventory

* refactor(swift): split gateway channel support

* test(doctor): isolate plugin compatibility registry

* test(macos): isolate gateway onboarding fixtures

* test(macos): assert gateway lease health ordering

* fix(codex): reconcile computer-use startup changes
2026-07-11 10:25:14 -07:00