The installation hit Blacksmith's backing-disk cap because sticky keys
embedded PR numbers and content hashes, minting a new disk per PR and
per input change until every mount 429-failed fleet-wide. node-deps
(#109752) and ext-boundary (#109804) were already re-keyed; this
converts the last two minters and audits the third suspect.
Keys changed:
- Vitest fs transform cache: `vitest-fs-v2-<pr-N|protected>-...` ->
the one existing `vitest-fs-v2-protected-<os>-<arch>-node-<ver>`
disk. Entries are content-hash keyed (vitest sha1 over
id+content+NODE_ENV+version+env config), so cross-PR sharing is safe
by construction: PRs now mount the protected snapshot directly
(read-only, commit false) and the separate PR-only seed mount is
deleted. Writers (non-PR shard writer + scheduled warm run) keep
explicit commit true. Disk count: 1 + O(open PRs) -> 1.
- Gradle: `gradle-v1-<task>-<pr-N|protected>-<hashFiles(deps)>` ->
`gradle-v2-<task>`. Task scope stays (light ktlint must not seed
heavy build lanes); PR number and dependency hash leave the key. The
dependency hash moved to an in-job fingerprint marker that makes the
non-PR writer rebuild its snapshot cold when inputs change, bounding
disk growth; PR mounts are read-only and safely reuse stale
snapshots because Gradle caches are content-addressed. Disk count:
O(tasks x PRs x dependency bumps) -> O(tasks) (7 today).
- Docker builder (useblacksmith/setup-docker-builder): audited, out of
scope. The action owns its key internally and always uses the repo
name only (src/setup_builder.ts getStickyDisk), so it is already one
disk per installation repo and cannot mint per-PR disks.
PR warm-path behavior changes:
- Vitest: PRs lose only PR-local warm entries for files the PR itself
changed (previously persisted on the pr-N disk between pushes);
changed files re-transform once per push, unchanged files still hit
the protected snapshot. PRs touching transform inputs
(lockfile/tsconfig/package.json) run cold per push since the
generation wipe is now local to the discarded clone.
- Gradle: dependency-bump PRs get warmer (stale-but-valid protected
snapshot instead of a cold fresh key); other PRs are unchanged.
Deleted `.github/workflows/pr-cache-cleanup.yml`: it existed for the
per-PR cache layer (#109425) and only deleted GitHub actions/cache
archives, which GitHub's own LRU/TTL eviction already handles for the
remaining fork/Windows per-PR archive paths.
Guards: pinned the O(1) vitest key + non-PR commit gate, added a
Gradle per-task key/writer/fingerprint guard, added a repo-wide scan
asserting no useblacksmith/stickydisk key ever contains
github.event.pull_request.number or hashFiles(), and replaced the
cleanup-workflow assertions with a stays-deleted check.
Group walls measured from compact run 29564411446 (post-#109769):
core-unit-fast-isolated runs 78.6s on 4 vCPU because fork-per-file
isolation parallelizes poorly there; pin it to the 8 vCPU class where
the same segment runs ~50s. Unit-fast stripes measure ~25-37s (hinted
100/60) and tooling stripes 71-87s; refresh both so the packer stops
overfilling small bins (the 299s compact tail this run produced).
Cheap stripes may now co-locate in one bin, so the compact test only
requires their presence.
A. check-extension-package-tsc-boundary "force-kills aborted async node
step process groups" (run 29568640449): two mechanisms behind the
expected-false-to-be-true at the child-alive assertion.
- Pid-file existence polls can observe writeFileSync's open-truncate
0-byte window, parse NaN, and process.kill(NaN, 0) throws, so
isProcessAlive reads false (the #109140 class, same as tsdown-build's
b1e6c6883b). Added waitForPidFile (poll until content parses to a
positive pid) and applied it to all three pid-read sites in the file.
- The fail-fast step exited on a 150ms fuse after the sibling's pid file
appeared, racing the test's aliveness assertion: a worker descheduled
>150ms observes the abort chain's SIGKILL first. The step now exits
only after the test writes abort.ack, which happens strictly after the
assertion (event-driven; the step's 5s timeout bounds a wedged run).
Forced proof: injecting a 300ms stall between pid-file wait and the
assertion reproduced the exact CI failure, and passed post-fix with
the same stall in place.
B. telegram-user-crabbox-proof "stops Crabbox recording when the desktop
probe fails" (waitFor timeout on recorder.term): the fake recorder
published its pid file before installing its SIGTERM handler. The probe
throws as soon as the pid file exists and recordProbeVideo SIGTERMs the
recorder in its finally, so a recorder preempted between publish and
process.on dies to the default disposition and never writes
recorder.term. Handler now arms before the pid publish. Forced proof: a
100ms spin between publish and process.on reproduced the exact CI
failure deterministically; moving the spin after the reordered handler
passes. Applied the same handler-before-readiness ordering to the three
sibling fakes in the file, and made the gateway-grandchild pid read
parse-validated inside its poll (a NaN pid skipped both the dead-check
and the finally SIGKILL cleanup, leaking the grandchild).
C. openclaw-live-updater "treats an owner file creation race as a normal
overlap" (run 29569279237): prod bug in acquireMaintenanceLock. Only
ENOENT got the bounded creation-window retry; a reader hitting the
racing writer's open-truncate window read an empty owner.json, JSON.parse
threw SyntaxError, and the script escalated to invalid_lock instead of
overlap. Any owner read/parse failure now re-reads within the same
bounded budget before declaring the lock invalid. New regression test
freezes the window (pre-created empty owner.json, delayed writer):
fails with the exact CI error pre-fix, passes post-fix.
Proof: focused runs green; 10x per-file stress at
OPENCLAW_VITEST_MAX_WORKERS=6 green for all three files (30/30).
check-additional-extension-package-boundary is the light-run critical-path
pole. Measured on runs 29564442266/29564411446/29564259862 (8 vCPU, sticky
mounts 429-failing): Setup Node ~40s cold install, shard step 142-161s =
~101s cold plugin-sdk dts prep (nine parallel tsgo builds) + ~58s compiling
122 plugins at concurrency 6 + ~1s canary; warm actions/cache path on a main
push (29551077288) runs the shard in 67s.
Levers chosen from that data:
- Re-key the ext-boundary sticky disk from per-PR/per-config keys (v1) to one
O(1) protected key (v2), following the node_modules snapshot fix in
#109752. Blacksmith's installation-wide 1000-disk cap is saturated and
every mount 429s; per-PR keys are a direct contributor. The config/scripts/
lockfile hash moves from the key into the in-job .source-trees marker next
to the existing tree-OID + node/pnpm gate. Single semantic writer: the
boundary lane commits explicitly (true, not if-missing) on protected pushes
only; PR mounts and the check-lint consumer stay read-only, and the seed
step is now writer-only (saves ~11s per PR cold run).
- Bump the lane 8 -> 32 vCPU and raise OPENCLAW_EXTENSION_BOUNDARY_CONCURRENCY
6 -> 16: both the parallel dts prep and the 122 plugin compiles scale with
cores at similar billed core-minutes (same precedent as check-lint/deps).
Expected pole time: warm PR runs (protected snapshot tracks every main push)
drop to roughly checkout + install-skip + restore + incremental shard, well
under 2 minutes; cold runs that touch boundary trees shrink via the core
bump. Guarded in ci-workflow-guards so the O(1) key, single-writer commit,
and fingerprint composition cannot silently regress.
* fix(docker): bound smoke installer curl with connect timeout
Add --connect-timeout 30 and --max-time 300 to the direct installer
one-liner curl in the smoke test runner. The run_installer_for_package_spec
function is already bounded by timeout(1), but this direct invocation had
no curl-level or wrapper-level timeout.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(docker): wrap complete installer pipeline in timeout, not just curl
Replace the stand-alone curl timeout flags on the direct installer one-liner
with a timeout(1) wrapper around the entire curl|bash pipeline, matching the
run_installer_for_package_spec pattern. This bounds the complete pipeline
(download + install) rather than just the network transfer.
The curl-level --connect-timeout 30 and --max-time 300 are retained as
defence-in-depth inside the timeout wrapper.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor: centralize installer smoke pipeline bounds
Co-authored-by: LZY3538 <liu.zhenye@xydigit.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* perf(sdk): load Claude CLI identity from a narrow plugin artifact
src/plugin-sdk/anthropic-cli.ts snapshots CLAUDE_CLI_BACKEND_ID at module
scope through the sync facade loader, which jiti-evaluates the full
anthropic api.js barrel on source checkouts: 130.86s of self time per
cold worker on CI (Testbox import profile), silently stalling every job
whose graph reaches cli-runner/prepare.ts. A narrow cli-api.js artifact
carries the two static facts; Testbox proof: the reliability+helpers
agents-core pair drops from 157.6s to 14.3s.
* perf(ci): stripe unit-fast and tooling node tests on the full plan
core-fast ran the import-bound unit-fast graph as one job (247s vitest,
181s module evaluation) and core-tooling as one serial job (241s); both
now stripe on the full plan like the compact plan, so the compact-only
expansion is deleted and the docker helper config rides with the
isolated shard on both plans. Group hints refreshed from main run
29551077288; stale per-file commands hints dropped (the consolidation
landed); cache-writer selection and the warm workflow track the stripe
names.
* fix(state): tolerate vanished sqlite sidecars in agent-db permission sweep
existsSync+chmodSync raced SQLite's own WAL/SHM cleanup: a checkpoint or
close between the two calls throws ENOENT (observed from the transcript
reconcile worker in server-startup-web-fetch-bind on CI). chmod directly
and swallow only ENOENT, which removes the TOCTOU window.
* perf(ci): serve node_modules snapshots from O(1) protected sticky disks
Every Blacksmith mount of the dependency sticky disk currently 429s:
the v2 key minted one backing disk per PR and per manifest hash, which
saturated Blacksmith's installation-wide 1000-sticky-disk budget (run
29559333389, job checks-node-compact-small-15: "Sticky disk limit
exceeded"). Same-repo shards then fall back to a cold, storeless pnpm
install (~40s) on every run - worse than the ~22s actions/cache path
the sticky rollout replaced.
Re-key the snapshot to one stable disk per node-version and move the
install inputs into a runtime fingerprint marker:
- Key is now `<repo>-node-deps-bind-v3-<node-version>`; dependency
changes refresh the disk in place instead of allocating a new one.
- The fingerprint (manifest hashFiles set + node version + lockfile
mode) is evaluated on the bind step, before the mount lands, so the
'**/package.json' glob cannot sweep snapshot-internal manifests.
- Consumers mount read-only (commit: false); on fingerprint match they
restore importer archives and skip pnpm install entirely, on mismatch
they install against the clone's on-disk store (warm store, no
actions/cache download) without capturing.
- Writers are trusted non-PR jobs only: build-artifacts on canonical
pushes plus the scheduled vitest-cache-warm run as a deadline writer
(main pushes cancel each other under merge traffic and would starve
the snapshot). commit: on-change keeps warm no-op runs from
committing.
- Roll the consumer flag out to every Linux Blacksmith lane that
installs dependencies (build-artifacts, check-shard,
check-additional-shard, check-docs, checks-ui, control-ui-i18n,
native-i18n, qa-smoke-ci-profile, checks-fast-core, both contract
shards) with the same fork/dispatch gates as the nondist shard.
Guards now enumerate the consumer set, pin the O(1) key shape, enforce
single-writer commit expressions, and cover the fingerprint marker in
the importer capture/restore helper.
Verified locally: importer archive capture 0.96s/228K and restore
0.07s against the real 2.0GB hoisted tree; a snapshot untarred into a
fresh workspace resolves modules, runs bin shims, `pnpm exec`, and a
Vitest suite (one-time pnpm reconcile only when the absolute path
changes, which Blacksmith's fixed /home/runner/_work layout avoids).
* fix(ci): commit dependency snapshots explicitly, not via on-change heuristic
stickydisk's on-change mode compares allocated disk bytes with a 4KB
threshold, so a fingerprint refresh whose reinstall keeps usage stable
(metadata-only manifest edits, same-sized dependency swaps) could be
silently discarded, stranding every consumer on a stale marker and a
permanent reinstall path. Writers now commit explicitly, mirroring the
Vitest transform disk's rationale; warm no-op writer runs re-commit an
identical snapshot, which is cheap and safe. Also document that the
non-PR commit gate binds cooperating code only - the enforced trust
boundary stays the fork/dispatch runner gate, matching the protected
node-compile disk's posture.
* test(ci): guard sticky consumers against writerless node-version key splits
Reviewer follow-up: the snapshot key is partitioned by node-version and
both writers (build-artifacts, vitest-cache-warm) rely on the action
default, so a consumer pinning any other version would split onto a
key nobody seeds and silently regress to permanently cold installs.
Pin the action default to 24.x and assert every sticky consumer
resolves to that same key segment.
* test: narrow sticky-consumer step.with before indexing
The filter's runtime narrowing did not carry into the map callback, so
step.with indexing failed check-test-types (TS18048). Collect a narrowed
stepWith instead.
Measured breakdown of check-prompt-snapshots in run 29557851276 (5m03s):
- 37s checkout + Node env setup
- 4m25s inside one `pnpm prompt:snapshots:check` process:
- ~35s tsx import of the generator graph
- ~3min first createOpenClawCodingTools call: the message-tool schema
build resolves the scenario channel plugin, the empty loaded registry
falls back to the bundled-channel jiti loader
(extensions/telegram/index.ts), and jiti re-transpiles the core source
graph (136.6s jiti self time in --cpu-prof; second generation call in
the same process takes 0.3s)
- ~1min more for the cold provider-plugin lookup in
normalizeAgentRuntimeTools; no provider plugin owns tool-schema hooks
for the `codex` harness provider, so the jiti load only rediscovers a
no-op
- <2s actual rendering + oxfmt + drift compare
Split axis decision: the planned 2-3 way scenario shard matrix cannot
lower the pole because the cost is a one-time per-process transpile;
every shard would pay the full ~4min again (jiti fsCache does not help:
a warm-cache rerun still took 200s). Fix the generator instead:
- Preload each scenario's channel plugin surface (channel-plugin-api.js,
resolved via the bundled-plugin public-surface resolver) through the
ambient tsx/vitest ESM graph, and pin exactly that channel in a scoped
loaded registry while the scenario's tools are built. Snapshot bytes
are unchanged: the loaded plugin object is the same export the bundled
jiti fallback materializes, and cross-channel action discovery still
sees no other loaded channels.
- Pass allowProviderRuntimePluginLoad=false to normalizeAgentRuntimeTools;
registry-only resolution returns the same undefined plugin instantly.
Coverage guard: test/scripts/prompt-snapshots.test.ts now generates the
snapshot files and asserts zero jiti plugin-loader calls (with the
offending transform targets in the failure message), so a future scenario
or channel that misses the preload fails the suite instead of silently
restoring the 4-minute lane.
Proof: `pnpm prompt:snapshots:check` passes byte-identical against the
committed fixtures in 54s wall / 35s CPU locally (was 354s wall / 306s
CPU); focused vitest suites (32 tests incl. new guard), tsgo core-test +
extensions-test lanes, scoped oxlint/oxfmt, and `git diff --check` are
green. Expected CI pole when the lane fires: ~5.1min -> ~1.5-2min.
* perf(ci): move the three light-run pole jobs to 32 vCPU runners
check-lint (~186s oxlint shards), check-dependencies (~157s concurrent
knip), and build-artifacts (~126s tsdown+checks) are the critical path of
every light PR run; all three parallelize with cores at similar billed
core-minutes. Guard test updated.
* test: align prerelease-plan guard with the 32 vCPU check-dependencies row
Add --connect-timeout 30 and --max-time 120 to the curl command that
downloads the Opengrep install script and pipes it to bash. Both the
PR-diff (opengrep-precise) and full-scan (opengrep-precise-full)
workflows are covered. A stuck TLS handshake or stalled download
could previously occupy a CI runner until the job-level timeout.
Co-authored-by: Claude <noreply@anthropic.com>
* test: stabilize four flaky tests at root cause
- test/non-isolated-runner.ts: move shared-worker cleanup from onAfterRunSuite
to onAfterRunFiles. Vitest early-returns runSuite for files that fail during
collection, skipping onAfterRunSuite, so a crashed sibling left its evaluated
real modules cached and the next file's vi.mock factories silently never
applied — the "res.setHeader is not a function" failures in
http-utils.authorize-request.test.ts. onAfterRunFiles fires per file
regardless of collect/run outcome. Regression meta-test spawns a child
vitest run (collect-crash sibling + mock-dependent file) and fails on the
old runner with flavor:real vs flavor:mocked.
- src/gateway/http-utils.authorize-request.test.ts: export every http-common
binding http-auth-utils imports so the factory stays isolate:false-safe.
- src/gateway/session-message-events.test.ts: drop the beforeAll 60s override;
the suite harness cold-imports the full gateway server graph, which can
legitimately exceed 60s under contention. Sibling suites use the shared
project hookTimeout (120s/180s) for the same boot.
- src/gateway/server-methods/agent.sessions-and-models.test-utils.ts: the
finalize-throw test polled a 2s waitForAssertion for an off-turn background
run; signal finalize via a promise resolved from the spy (event-driven,
bounded by the test timeout) and keep the bounded poll for the follow-up
respond/warn observations.
- ui/src/pages/chat/chat-responsive.browser.test.ts: budget cold-app first
renders at 30s (real-app cases boot a cold Vite dev server whose first
transform can starve past 10s under 6-worker contention) and replace
one-shot isVisible reads with bounded locator waits for the state-driven
hover reveal.
Proof: focused runs green; 5x repeats of the three gateway files and 5x
repeats of chat-responsive with OPENCLAW_VITEST_MAX_WORKERS=6 all green;
runner regression test fails pre-fix, passes post-fix.
* test: isolate runner meta-test child env from GitHub Actions
The child vitest inherited GITHUB_ACTIONS/CI from the runner job, which
turned on ANSI colors (breaking the plain-substring assertions) and let
the child's github-actions reporter emit ::error annotations the parent
job rendered as its own failures. Drop GITHUB_ACTIONS/FORCE_COLOR and set
NO_COLOR, which overrides every tinyrainbow enable path.
* test: wait for parseable pid in tsdown-build pid-file polls
waitForFile existence polling can catch writeFileSync's open-truncate
0-byte window, yielding NaN pids and false isProcessAlive failures (the
same class as #109140). Poll until the file parses to a positive pid;
covers all three sibling pid-read sites in the suite.
* perf(build): cut write-cli-startup-metadata wall time via launcher-boot browser help and parallel renders
* perf(ci): fetch preflight parent metadata blob-less instead of a full depth-2 snapshot
* test(ci): expect the blob-less preflight parent fetch in checkout guards
* fix(build): default the bundled root-help render env
RootHelpRenderOptions marks env optional but spawnText requires one;
check-test-types caught the undefined assignment.
* feat(android): add Wear OS companion
Co-authored-by: Sebastian Schubotz <git@sibbl.net>
Co-authored-by: IWhatsskill <284122573+IWhatsskill@users.noreply.github.com>
* style(android): format Wear release gate test
* test(android): cover Wear release CI contracts
* test(android): import Wear JSON fixture type
* fix(android): harden Wear proxy event actor
* test(android): stabilize Wear proxy actor tests
* test(android): isolate Wear proxy actor lifecycles
* test(android): own Wear actor dispatchers
* test(android): use real time for Wear actor tests
* test(android): own Wear actors in test scope
* test(android): stop Wear actors explicitly
* test(android): drain Wear actor cancellation
* test(android): isolate Wear actors from test scheduler
* test(android): inject Wear test clock
---------
Co-authored-by: IWhatsskill <284122573+IWhatsskill@users.noreply.github.com>
Two direct-landed changes left every PR red: release-candidate-checklist.d.mts
declared isDirectReleaseCandidateExecution twice with other declarations
between them (adjacent-overload-signatures), and the parallels smoke model
test still asserted the workspace-state.json seeding that b6535fb8de
deliberately retired. Drop the duplicate and flip the assertion to guard
the retirement.
* fix(pr): close the review follow-ups on early gate proof
Three fail-safe gaps from #109331's review that silently disabled the
speedup: a stale terminal failure blocked a gate-proven pending rerun
(now superseded only by a SCHEDULED run whose own gate passed — manual
runs still cannot mask unresolved failures, guard test intact); the
attempt jobs lookup read one page while full-scope runs sit near the
100-job cap (now pages until the gate job is visible); and same-attempt
completion between snapshot and re-read discarded valid gate evidence
(now only an advanced attempt or non-success completion rejects).
* fix(pr): only newer gate-proven reruns supersede a terminal failure
Autoreview P1: without an ordering bound, a stalled older run's passed
gate could mask a newer completed failure. The supersede now requires
the gate-proven scheduled run to be created after the failed run;
regression test covers the stalled-older-run case.
