* fix(mcp): reject sandbox CSP metadata that normalizes to no policy
A present ?csp= value that decodes to valid JSON but is not a usable CSP
(e.g. null or a wrong-shaped object) normalized to undefined and was
indistinguishable from an absent parameter, so the gateway served proxy
HTML under the default policy. encodeCsp omits the query param entirely
for such values, so a present-but-empty policy is never legitimate;
throw so the sandbox endpoint fails closed with 400. Found by review on
the revert of 73685b4e7c946; the gap predates that commit.
* fix(mcp): treat empty csp query value as malformed, not absent
?csp= with an empty value passed the falsy absent-guard and served proxy
HTML under the default policy. Only a truly absent parameter (null) may
skip validation; an empty string now falls through to JSON.parse and
fails closed with 400.
* refactor(gateway): drop redundant sandbox CSP handler guard
decodeMcpAppSandboxCsp now throws for every present-but-unusable value
(1ca26c508d added the same fail-closed behavior at the handler seam),
so the handler-level 'present but falsy' check is unreachable. Keep the
invariant in the decoder, which owns policy decode semantics.
* fix(test): reset diagnostic listener-presence mirror between non-isolated files
resetOpenClawGlobalDiagnosticState clears the listener sets and deletes
the diagnostic-events state key, but the listener-presence counts live
under a separate globalThis record and survived, so
hasInternalDiagnosticEventListeners() stayed true for every later file
in the worker once any file leaked a registration (e.g. the import-time
listener in src/logging/diagnostic-run-activity.ts whose stop handle is
lost to the module-registry reset). Zero the counts to match the cleared
sets. Root cause of the model-call-diagnostics flake in CI run
29624203224; #110288 added the victim-side reset, this fixes the class.
* perf(ci): cut the pre-fan-out critical path on canonical runs
Two changes to the run head and matrix shape:
1. runner-admission was a hosted 90s sleep every run queued behind
(observed 1.7min hosted-queue latency before the sleep started). The
debounce now lives at the tail of preflight: heavy jobs all need
preflight, so a superseding main push still cancels the run before
fan-out while only one 4 vCPU runner has been spent, and preflight's
own work usually exceeds the window so the residual sleep is zero.
security-fast (hosted, dependency-free) starts immediately.
2. Canonical main pushes now use the compact bin plan like PRs: the
82-job named matrix drained the runner pool for ~4.5min (job starts
trickled from minute 5.2 to 9.7 in run 29592647843) with no
branch-protection consumer for per-shard names on main. Coverage is
identical; dispatch/release-gate runs keep the full named matrix.
* perf(test): boot TUI PTY suite fixtures concurrently
tsx+TUI startup dominated the harness file's wall time and the three
suite PTYs booted serially. Boot them concurrently in beforeAll
(allSettled so a failed boot still assigns survivors for afterAll
cleanup); the env-specific fixtures never receive input, so their tests
only await their own readiness output. The slow-startup test now proves
frame ordering on the append-only output, which the old sequential
waits did not. File wall 10.2s -> ~4.9s, 3/3 repeat runs green.
* docs(ci): align gate and debounce descriptions with the removed admission job
* test(tooling): wait for readiness file content, not existence
writeFileSync creates the file before its bytes land, so the existence
poll raced the child's write on loaded runners and read an empty ready
file (observed in compact-small-4, run 29615028678). Poll for non-empty
content at both readiness sites.
A landed change removed the no-useless-assignment suppression in
extensions/reef/src/transport.ts without pruning the allowlist tail,
failing checks-node-compact-small-7 on every PR (merge-skew).
* feat(onboarding): recommend plugins and skills from installed apps
Scan installed macOS apps during classic onboarding (TCC-free), gather
candidates from official catalogs + ClawHub search, let the configured
model pick genuine matches, and offer an opt-in multiselect install step.
Adds a device.apps node-host command (default-off sharing, Android-parity
envelope) so remote gateways can request a paired Mac's inventory, and a
wizard.appRecommendations kill switch. Custom setup-inference completions
no longer inherit the 32-token verification-probe output cap.
* feat(onboarding): recommend apps in guided flow
* fix(onboarding): harden app recommendations against ClawHub self-promotion
Third-party ClawHub skills are never pre-selected regardless of model tier
(publisher-controlled listing text reaches the matcher prompt and could
promote itself); their labels now say they install third-party code.
Installed-app scans follow symlinked .app bundles. Matcher output stays
bounded by the resolved model's own maxTokens budget (documented invariant).
* fix(onboarding): key official catalog candidates by resolved plugin id
Real catalog entries are package manifests without a top-level id; keying the
candidate map and channel/provider classification by entry.id collapsed the
whole official catalog into one undefined-keyed entry, so no official plugin
or channel was ever recommended. Regression test runs against the bundled
catalogs.
* fix(onboarding): satisfy lint, types, deadcode, and migration gates
Split the guided-onboarding test into a self-contained custodian suite to stay
under max-lines. Narrow app-recommendation exports (drop dead node-payload
normalizer, unexport internal types/helpers, route candidate tests through the
public API), replace map-spread with a helper, unexport device.apps result
types, add installedAppsSharing to node-host migration expectations, cast the
wizard multiselect mock, and regenerate the docs map.
* test(onboarding): register new live test in the shard classifier
* test: serialize concurrent vitest mock resolution in the non-isolated runner
Root-causes the subagent-orphan-recovery flake (main run 29566318167, shard
agentic-agents-core-subagents): "includes last human message in resume when
available" and "adds config change hint..." received the bare resume template
because prod code called a different readSessionMessagesAsync mock instance
than the one the test configured.
Mechanism: vitest's BareModuleMocker.resolveMocks has no in-flight guard.
pendingIds is cleared only after all parallel resolveId RPCs settle, and every
registration re-invalidates the mock module node. In a shared isolate:false
worker, leaked async work from an earlier file (a real-timer dynamic import)
triggers a fetchModule while the next file's vi.mock registrations are still
pending, starting a second concurrent resolution pass over the same
pendingIds. The slower pass re-registers each manual mock and wipes the
already-evaluated mock module mid-import-chain: importers evaluated before the
wipe (the test file's own binding, subagent-orphan-recovery.test.ts:8) keep
factory instance A while later importers (subagent-orphan-recovery.ts via
test line 16) instantiate a fresh instance B. Only inline-factory mocks split;
vi.hoisted-stable mocks are immune, matching the CI failure signature exactly.
Fix: OpenClawNonIsolatedRunner installs a coalescing wrapper around
moduleRunner.mocker.resolveMocks (once per worker) so concurrent callers share
one pass and registration happens exactly once before any awaiting import
proceeds.
Proof:
- Deterministic repro (poison file leaving a 1ms recurring dynamic-import
timer + resolveId jitter on session-transcript-readers) reproduced the exact
CI assertion failures at lines 755/772 on the unfixed runner and passes 5/5
with the fix.
- New unit tests cover coalescing, idempotent install, and fresh passes.
- Stress: 10x subagent-orphan-recovery.test.ts and 5x the full
agentic-agents-core-subagents file set (63 files, 1220 tests) at
OPENCLAW_VITEST_MAX_WORKERS=6, all green.
* test: requeue mock ids queued during an in-flight resolveMocks pass
Review follow-up to the resolveMocks serialization pin: upstream snapshots the
pendingIds contents at pass start and reassigns the static to [] at the end,
so ids queued during the pass's RPC window land in the abandoned array. Pure
coalescing would silently drop those registrations (upstream's racy second
pass previously processed them). The wrapper now captures the queue reference
before each pass, requeues ids pushed past the processed count, and drains
until the queue is empty, so every coalesced caller's vi.mock registration is
applied before its fetch proceeds.
Proof: new unit test "registers ids queued while a pass is in flight before
callers proceed"; deterministic zombie+jitter repro still passes 5/5; full
agentic-agents-core-subagents file set green at OPENCLAW_VITEST_MAX_WORKERS=6.
* test: chain resolveMocks passes per caller instead of sharing one pass
The coalescing pin regressed shard agentic-agents-core-auth
(models-config.providers.auth-provenance.test.ts, runs 29570918219 and
29571513381): sharing one in-flight pass broke the mocker's freshness
invariant. Upstream gives every resolveMocks caller a pass that starts at or
after its call, so ids the caller queued (vi.doUnmock before dynamic imports
in that file's beforeAll/beforeEach) are always registered before its fetch
proceeds. A coalesced caller could instead ride a pass snapshotted before its
ids were queued and import with mock state unresolved. Reproduced locally on a
cold transform cache: the file then loaded the real provider-auth warm worker
plus a live oauth-manager refresh guarded by a 120s withRefreshCallTimeout
(the 105-142s stalls and the "Test timed out in 120000ms" CI failure), and the
real resolveProviderSyntheticAuthWithPlugin returned undefined (the
mode:"none" assertion failure at line 348).
The wrapper now chains each caller onto its own sequential pass. This keeps
both invariants: serialization (a pass queued behind an in-flight one sees the
cleared queue and no-ops, so a snapshot is never registered or its mock
modules invalidated twice - the original subagent-orphan-recovery disease) and
freshness (each caller's pass starts at or after its call). Abandoned-id
requeue is preserved so ids pushed during a pass's RPC window are registered
by the next chained pass instead of dropped.
Proof:
- Cold-cache auth shard (37 files, 528 tests): failed 105-142s with the
coalescing pin (also with a coalescing-only variant), passed 20s with the
unfixed runner, passes 3/3 in ~20-30s with the chained pin.
- Long-timer instrumentation pinpointed the stall: provider-auth warm worker
spawn plus oauth-manager withRefreshCallTimeout delay=120000.
- Original zombie+jitter orphan-recovery repro still passes 5/5.
- Warm agentic-agents-core-subagents (63 files, 1220 tests) and auth shard
green at OPENCLAW_VITEST_MAX_WORKERS=6; wrapper unit tests updated to
assert serialization, freshness, requeue, idempotent install.
The installation hit Blacksmith's backing-disk cap because sticky keys
embedded PR numbers and content hashes, minting a new disk per PR and
per input change until every mount 429-failed fleet-wide. node-deps
(#109752) and ext-boundary (#109804) were already re-keyed; this
converts the last two minters and audits the third suspect.
Keys changed:
- Vitest fs transform cache: `vitest-fs-v2-<pr-N|protected>-...` ->
the one existing `vitest-fs-v2-protected-<os>-<arch>-node-<ver>`
disk. Entries are content-hash keyed (vitest sha1 over
id+content+NODE_ENV+version+env config), so cross-PR sharing is safe
by construction: PRs now mount the protected snapshot directly
(read-only, commit false) and the separate PR-only seed mount is
deleted. Writers (non-PR shard writer + scheduled warm run) keep
explicit commit true. Disk count: 1 + O(open PRs) -> 1.
- Gradle: `gradle-v1-<task>-<pr-N|protected>-<hashFiles(deps)>` ->
`gradle-v2-<task>`. Task scope stays (light ktlint must not seed
heavy build lanes); PR number and dependency hash leave the key. The
dependency hash moved to an in-job fingerprint marker that makes the
non-PR writer rebuild its snapshot cold when inputs change, bounding
disk growth; PR mounts are read-only and safely reuse stale
snapshots because Gradle caches are content-addressed. Disk count:
O(tasks x PRs x dependency bumps) -> O(tasks) (7 today).
- Docker builder (useblacksmith/setup-docker-builder): audited, out of
scope. The action owns its key internally and always uses the repo
name only (src/setup_builder.ts getStickyDisk), so it is already one
disk per installation repo and cannot mint per-PR disks.
PR warm-path behavior changes:
- Vitest: PRs lose only PR-local warm entries for files the PR itself
changed (previously persisted on the pr-N disk between pushes);
changed files re-transform once per push, unchanged files still hit
the protected snapshot. PRs touching transform inputs
(lockfile/tsconfig/package.json) run cold per push since the
generation wipe is now local to the discarded clone.
- Gradle: dependency-bump PRs get warmer (stale-but-valid protected
snapshot instead of a cold fresh key); other PRs are unchanged.
Deleted `.github/workflows/pr-cache-cleanup.yml`: it existed for the
per-PR cache layer (#109425) and only deleted GitHub actions/cache
archives, which GitHub's own LRU/TTL eviction already handles for the
remaining fork/Windows per-PR archive paths.
Guards: pinned the O(1) vitest key + non-PR commit gate, added a
Gradle per-task key/writer/fingerprint guard, added a repo-wide scan
asserting no useblacksmith/stickydisk key ever contains
github.event.pull_request.number or hashFiles(), and replaced the
cleanup-workflow assertions with a stays-deleted check.
Group walls measured from compact run 29564411446 (post-#109769):
core-unit-fast-isolated runs 78.6s on 4 vCPU because fork-per-file
isolation parallelizes poorly there; pin it to the 8 vCPU class where
the same segment runs ~50s. Unit-fast stripes measure ~25-37s (hinted
100/60) and tooling stripes 71-87s; refresh both so the packer stops
overfilling small bins (the 299s compact tail this run produced).
Cheap stripes may now co-locate in one bin, so the compact test only
requires their presence.
A. check-extension-package-tsc-boundary "force-kills aborted async node
step process groups" (run 29568640449): two mechanisms behind the
expected-false-to-be-true at the child-alive assertion.
- Pid-file existence polls can observe writeFileSync's open-truncate
0-byte window, parse NaN, and process.kill(NaN, 0) throws, so
isProcessAlive reads false (the #109140 class, same as tsdown-build's
b1e6c6883b). Added waitForPidFile (poll until content parses to a
positive pid) and applied it to all three pid-read sites in the file.
- The fail-fast step exited on a 150ms fuse after the sibling's pid file
appeared, racing the test's aliveness assertion: a worker descheduled
>150ms observes the abort chain's SIGKILL first. The step now exits
only after the test writes abort.ack, which happens strictly after the
assertion (event-driven; the step's 5s timeout bounds a wedged run).
Forced proof: injecting a 300ms stall between pid-file wait and the
assertion reproduced the exact CI failure, and passed post-fix with
the same stall in place.
B. telegram-user-crabbox-proof "stops Crabbox recording when the desktop
probe fails" (waitFor timeout on recorder.term): the fake recorder
published its pid file before installing its SIGTERM handler. The probe
throws as soon as the pid file exists and recordProbeVideo SIGTERMs the
recorder in its finally, so a recorder preempted between publish and
process.on dies to the default disposition and never writes
recorder.term. Handler now arms before the pid publish. Forced proof: a
100ms spin between publish and process.on reproduced the exact CI
failure deterministically; moving the spin after the reordered handler
passes. Applied the same handler-before-readiness ordering to the three
sibling fakes in the file, and made the gateway-grandchild pid read
parse-validated inside its poll (a NaN pid skipped both the dead-check
and the finally SIGKILL cleanup, leaking the grandchild).
C. openclaw-live-updater "treats an owner file creation race as a normal
overlap" (run 29569279237): prod bug in acquireMaintenanceLock. Only
ENOENT got the bounded creation-window retry; a reader hitting the
racing writer's open-truncate window read an empty owner.json, JSON.parse
threw SyntaxError, and the script escalated to invalid_lock instead of
overlap. Any owner read/parse failure now re-reads within the same
bounded budget before declaring the lock invalid. New regression test
freezes the window (pre-created empty owner.json, delayed writer):
fails with the exact CI error pre-fix, passes post-fix.
Proof: focused runs green; 10x per-file stress at
OPENCLAW_VITEST_MAX_WORKERS=6 green for all three files (30/30).
check-additional-extension-package-boundary is the light-run critical-path
pole. Measured on runs 29564442266/29564411446/29564259862 (8 vCPU, sticky
mounts 429-failing): Setup Node ~40s cold install, shard step 142-161s =
~101s cold plugin-sdk dts prep (nine parallel tsgo builds) + ~58s compiling
122 plugins at concurrency 6 + ~1s canary; warm actions/cache path on a main
push (29551077288) runs the shard in 67s.
Levers chosen from that data:
- Re-key the ext-boundary sticky disk from per-PR/per-config keys (v1) to one
O(1) protected key (v2), following the node_modules snapshot fix in
#109752. Blacksmith's installation-wide 1000-disk cap is saturated and
every mount 429s; per-PR keys are a direct contributor. The config/scripts/
lockfile hash moves from the key into the in-job .source-trees marker next
to the existing tree-OID + node/pnpm gate. Single semantic writer: the
boundary lane commits explicitly (true, not if-missing) on protected pushes
only; PR mounts and the check-lint consumer stay read-only, and the seed
step is now writer-only (saves ~11s per PR cold run).
- Bump the lane 8 -> 32 vCPU and raise OPENCLAW_EXTENSION_BOUNDARY_CONCURRENCY
6 -> 16: both the parallel dts prep and the 122 plugin compiles scale with
cores at similar billed core-minutes (same precedent as check-lint/deps).
Expected pole time: warm PR runs (protected snapshot tracks every main push)
drop to roughly checkout + install-skip + restore + incremental shard, well
under 2 minutes; cold runs that touch boundary trees shrink via the core
bump. Guarded in ci-workflow-guards so the O(1) key, single-writer commit,
and fingerprint composition cannot silently regress.
