Success-only gating starved the deps/vitest snapshots: under merge
traffic every main push cancels the prior CI run, so the deadline
writer skipped for hours (15:34->18:45 observed) while PRs paid full
reinstalls against stale fingerprints. The warm job checks out main
tip regardless of the triggering run, so cancelled completions are
safe refresh signals; failed runs still skip.
The installation hit Blacksmith's backing-disk cap because sticky keys
embedded PR numbers and content hashes, minting a new disk per PR and
per input change until every mount 429-failed fleet-wide. node-deps
(#109752) and ext-boundary (#109804) were already re-keyed; this
converts the last two minters and audits the third suspect.
Keys changed:
- Vitest fs transform cache: `vitest-fs-v2-<pr-N|protected>-...` ->
the one existing `vitest-fs-v2-protected-<os>-<arch>-node-<ver>`
disk. Entries are content-hash keyed (vitest sha1 over
id+content+NODE_ENV+version+env config), so cross-PR sharing is safe
by construction: PRs now mount the protected snapshot directly
(read-only, commit false) and the separate PR-only seed mount is
deleted. Writers (non-PR shard writer + scheduled warm run) keep
explicit commit true. Disk count: 1 + O(open PRs) -> 1.
- Gradle: `gradle-v1-<task>-<pr-N|protected>-<hashFiles(deps)>` ->
`gradle-v2-<task>`. Task scope stays (light ktlint must not seed
heavy build lanes); PR number and dependency hash leave the key. The
dependency hash moved to an in-job fingerprint marker that makes the
non-PR writer rebuild its snapshot cold when inputs change, bounding
disk growth; PR mounts are read-only and safely reuse stale
snapshots because Gradle caches are content-addressed. Disk count:
O(tasks x PRs x dependency bumps) -> O(tasks) (7 today).
- Docker builder (useblacksmith/setup-docker-builder): audited, out of
scope. The action owns its key internally and always uses the repo
name only (src/setup_builder.ts getStickyDisk), so it is already one
disk per installation repo and cannot mint per-PR disks.
PR warm-path behavior changes:
- Vitest: PRs lose only PR-local warm entries for files the PR itself
changed (previously persisted on the pr-N disk between pushes);
changed files re-transform once per push, unchanged files still hit
the protected snapshot. PRs touching transform inputs
(lockfile/tsconfig/package.json) run cold per push since the
generation wipe is now local to the discarded clone.
- Gradle: dependency-bump PRs get warmer (stale-but-valid protected
snapshot instead of a cold fresh key); other PRs are unchanged.
Deleted `.github/workflows/pr-cache-cleanup.yml`: it existed for the
per-PR cache layer (#109425) and only deleted GitHub actions/cache
archives, which GitHub's own LRU/TTL eviction already handles for the
remaining fork/Windows per-PR archive paths.
Guards: pinned the O(1) vitest key + non-PR commit gate, added a
Gradle per-task key/writer/fingerprint guard, added a repo-wide scan
asserting no useblacksmith/stickydisk key ever contains
github.event.pull_request.number or hashFiles(), and replaced the
cleanup-workflow assertions with a stays-deleted check.
check-additional-extension-package-boundary is the light-run critical-path
pole. Measured on runs 29564442266/29564411446/29564259862 (8 vCPU, sticky
mounts 429-failing): Setup Node ~40s cold install, shard step 142-161s =
~101s cold plugin-sdk dts prep (nine parallel tsgo builds) + ~58s compiling
122 plugins at concurrency 6 + ~1s canary; warm actions/cache path on a main
push (29551077288) runs the shard in 67s.
Levers chosen from that data:
- Re-key the ext-boundary sticky disk from per-PR/per-config keys (v1) to one
O(1) protected key (v2), following the node_modules snapshot fix in
#109752. Blacksmith's installation-wide 1000-disk cap is saturated and
every mount 429s; per-PR keys are a direct contributor. The config/scripts/
lockfile hash moves from the key into the in-job .source-trees marker next
to the existing tree-OID + node/pnpm gate. Single semantic writer: the
boundary lane commits explicitly (true, not if-missing) on protected pushes
only; PR mounts and the check-lint consumer stay read-only, and the seed
step is now writer-only (saves ~11s per PR cold run).
- Bump the lane 8 -> 32 vCPU and raise OPENCLAW_EXTENSION_BOUNDARY_CONCURRENCY
6 -> 16: both the parallel dts prep and the 122 plugin compiles scale with
cores at similar billed core-minutes (same precedent as check-lint/deps).
Expected pole time: warm PR runs (protected snapshot tracks every main push)
drop to roughly checkout + install-skip + restore + incremental shard, well
under 2 minutes; cold runs that touch boundary trees shrink via the core
bump. Guarded in ci-workflow-guards so the O(1) key, single-writer commit,
and fingerprint composition cannot silently regress.
* perf(sdk): load Claude CLI identity from a narrow plugin artifact
src/plugin-sdk/anthropic-cli.ts snapshots CLAUDE_CLI_BACKEND_ID at module
scope through the sync facade loader, which jiti-evaluates the full
anthropic api.js barrel on source checkouts: 130.86s of self time per
cold worker on CI (Testbox import profile), silently stalling every job
whose graph reaches cli-runner/prepare.ts. A narrow cli-api.js artifact
carries the two static facts; Testbox proof: the reliability+helpers
agents-core pair drops from 157.6s to 14.3s.
* perf(ci): stripe unit-fast and tooling node tests on the full plan
core-fast ran the import-bound unit-fast graph as one job (247s vitest,
181s module evaluation) and core-tooling as one serial job (241s); both
now stripe on the full plan like the compact plan, so the compact-only
expansion is deleted and the docker helper config rides with the
isolated shard on both plans. Group hints refreshed from main run
29551077288; stale per-file commands hints dropped (the consolidation
landed); cache-writer selection and the warm workflow track the stripe
names.
* fix(state): tolerate vanished sqlite sidecars in agent-db permission sweep
existsSync+chmodSync raced SQLite's own WAL/SHM cleanup: a checkpoint or
close between the two calls throws ENOENT (observed from the transcript
reconcile worker in server-startup-web-fetch-bind on CI). chmod directly
and swallow only ENOENT, which removes the TOCTOU window.
* perf(ci): serve node_modules snapshots from O(1) protected sticky disks
Every Blacksmith mount of the dependency sticky disk currently 429s:
the v2 key minted one backing disk per PR and per manifest hash, which
saturated Blacksmith's installation-wide 1000-sticky-disk budget (run
29559333389, job checks-node-compact-small-15: "Sticky disk limit
exceeded"). Same-repo shards then fall back to a cold, storeless pnpm
install (~40s) on every run - worse than the ~22s actions/cache path
the sticky rollout replaced.
Re-key the snapshot to one stable disk per node-version and move the
install inputs into a runtime fingerprint marker:
- Key is now `<repo>-node-deps-bind-v3-<node-version>`; dependency
changes refresh the disk in place instead of allocating a new one.
- The fingerprint (manifest hashFiles set + node version + lockfile
mode) is evaluated on the bind step, before the mount lands, so the
'**/package.json' glob cannot sweep snapshot-internal manifests.
- Consumers mount read-only (commit: false); on fingerprint match they
restore importer archives and skip pnpm install entirely, on mismatch
they install against the clone's on-disk store (warm store, no
actions/cache download) without capturing.
- Writers are trusted non-PR jobs only: build-artifacts on canonical
pushes plus the scheduled vitest-cache-warm run as a deadline writer
(main pushes cancel each other under merge traffic and would starve
the snapshot). commit: on-change keeps warm no-op runs from
committing.
- Roll the consumer flag out to every Linux Blacksmith lane that
installs dependencies (build-artifacts, check-shard,
check-additional-shard, check-docs, checks-ui, control-ui-i18n,
native-i18n, qa-smoke-ci-profile, checks-fast-core, both contract
shards) with the same fork/dispatch gates as the nondist shard.
Guards now enumerate the consumer set, pin the O(1) key shape, enforce
single-writer commit expressions, and cover the fingerprint marker in
the importer capture/restore helper.
Verified locally: importer archive capture 0.96s/228K and restore
0.07s against the real 2.0GB hoisted tree; a snapshot untarred into a
fresh workspace resolves modules, runs bin shims, `pnpm exec`, and a
Vitest suite (one-time pnpm reconcile only when the absolute path
changes, which Blacksmith's fixed /home/runner/_work layout avoids).
* fix(ci): commit dependency snapshots explicitly, not via on-change heuristic
stickydisk's on-change mode compares allocated disk bytes with a 4KB
threshold, so a fingerprint refresh whose reinstall keeps usage stable
(metadata-only manifest edits, same-sized dependency swaps) could be
silently discarded, stranding every consumer on a stale marker and a
permanent reinstall path. Writers now commit explicitly, mirroring the
Vitest transform disk's rationale; warm no-op writer runs re-commit an
identical snapshot, which is cheap and safe. Also document that the
non-PR commit gate binds cooperating code only - the enforced trust
boundary stays the fork/dispatch runner gate, matching the protected
node-compile disk's posture.
* test(ci): guard sticky consumers against writerless node-version key splits
Reviewer follow-up: the snapshot key is partitioned by node-version and
both writers (build-artifacts, vitest-cache-warm) rely on the action
default, so a consumer pinning any other version would split onto a
key nobody seeds and silently regress to permanently cold installs.
Pin the action default to 24.x and assert every sticky consumer
resolves to that same key segment.
* test: narrow sticky-consumer step.with before indexing
The filter's runtime narrowing did not carry into the map callback, so
step.with indexing failed check-test-types (TS18048). Collect a narrowed
stepWith instead.
Measured breakdown of check-prompt-snapshots in run 29557851276 (5m03s):
- 37s checkout + Node env setup
- 4m25s inside one `pnpm prompt:snapshots:check` process:
- ~35s tsx import of the generator graph
- ~3min first createOpenClawCodingTools call: the message-tool schema
build resolves the scenario channel plugin, the empty loaded registry
falls back to the bundled-channel jiti loader
(extensions/telegram/index.ts), and jiti re-transpiles the core source
graph (136.6s jiti self time in --cpu-prof; second generation call in
the same process takes 0.3s)
- ~1min more for the cold provider-plugin lookup in
normalizeAgentRuntimeTools; no provider plugin owns tool-schema hooks
for the `codex` harness provider, so the jiti load only rediscovers a
no-op
- <2s actual rendering + oxfmt + drift compare
Split axis decision: the planned 2-3 way scenario shard matrix cannot
lower the pole because the cost is a one-time per-process transpile;
every shard would pay the full ~4min again (jiti fsCache does not help:
a warm-cache rerun still took 200s). Fix the generator instead:
- Preload each scenario's channel plugin surface (channel-plugin-api.js,
resolved via the bundled-plugin public-surface resolver) through the
ambient tsx/vitest ESM graph, and pin exactly that channel in a scoped
loaded registry while the scenario's tools are built. Snapshot bytes
are unchanged: the loaded plugin object is the same export the bundled
jiti fallback materializes, and cross-channel action discovery still
sees no other loaded channels.
- Pass allowProviderRuntimePluginLoad=false to normalizeAgentRuntimeTools;
registry-only resolution returns the same undefined plugin instantly.
Coverage guard: test/scripts/prompt-snapshots.test.ts now generates the
snapshot files and asserts zero jiti plugin-loader calls (with the
offending transform targets in the failure message), so a future scenario
or channel that misses the preload fails the suite instead of silently
restoring the 4-minute lane.
Proof: `pnpm prompt:snapshots:check` passes byte-identical against the
committed fixtures in 54s wall / 35s CPU locally (was 354s wall / 306s
CPU); focused vitest suites (32 tests incl. new guard), tsgo core-test +
extensions-test lanes, scoped oxlint/oxfmt, and `git diff --check` are
green. Expected CI pole when the lane fires: ~5.1min -> ~1.5-2min.
* perf(ci): move the three light-run pole jobs to 32 vCPU runners
check-lint (~186s oxlint shards), check-dependencies (~157s concurrent
knip), and build-artifacts (~126s tsdown+checks) are the critical path of
every light PR run; all three parallelize with cores at similar billed
core-minutes. Guard test updated.
* test: align prerelease-plan guard with the 32 vCPU check-dependencies row
Add --connect-timeout 30 and --max-time 120 to the curl command that
downloads the Opengrep install script and pipes it to bash. Both the
PR-diff (opengrep-precise) and full-scan (opengrep-precise-full)
workflows are covered. A stuck TLS handshake or stalled download
could previously occupy a CI runner until the job-level timeout.
Co-authored-by: Claude <noreply@anthropic.com>
* perf(build): cut write-cli-startup-metadata wall time via launcher-boot browser help and parallel renders
* perf(ci): fetch preflight parent metadata blob-less instead of a full depth-2 snapshot
* test(ci): expect the blob-less preflight parent fetch in checkout guards
* fix(build): default the bundled root-help render env
RootHelpRenderOptions marks env optional but spawnText requires one;
check-test-types caught the undefined assignment.
* feat(android): add Wear OS companion
Co-authored-by: Sebastian Schubotz <git@sibbl.net>
Co-authored-by: IWhatsskill <284122573+IWhatsskill@users.noreply.github.com>
* style(android): format Wear release gate test
* test(android): cover Wear release CI contracts
* test(android): import Wear JSON fixture type
* fix(android): harden Wear proxy event actor
* test(android): stabilize Wear proxy actor tests
* test(android): isolate Wear proxy actor lifecycles
* test(android): own Wear actor dispatchers
* test(android): use real time for Wear actor tests
* test(android): own Wear actors in test scope
* test(android): stop Wear actors explicitly
* test(android): drain Wear actor cancellation
* test(android): isolate Wear actors from test scheduler
* test(android): inject Wear test clock
---------
Co-authored-by: IWhatsskill <284122573+IWhatsskill@users.noreply.github.com>
* perf(ci): carry Android Gradle caches on a Blacksmith sticky disk
The android lane's setup-java gradle cache almost never restores under the
repo's churned GitHub cache quota, so every run re-resolves the full
dependency graph (~501s observed for android-build-play). Mount a
task-scoped Blacksmith sticky disk at /var/tmp/openclaw-gradle and point
GRADLE_USER_HOME at it for same-repo non-fork non-dispatch runs, mirroring
the extension-boundary sticky pattern (per-PR-or-protected key,
commit: if-missing, identical trust gate). Fork PRs and manual dispatches
keep the existing GitHub cache path and non-Blacksmith runners.
* fix(ci): let the Gradle sticky snapshot keep learning
Autoreview P2: if-missing froze the snapshot at first hydration, so
later build-cache entries never persisted. on-change commits when the
filesystem changed; content-addressed entries and task-scoped keys keep
it race-safe.
* fix(ci): address reviewer findings across the perf-lever batch
Seven follow-ups from Codex review feedback on the landed CI perf PRs:
docker packaging switches to the full build profile (the ciArtifacts
step env clobbered its canonical-dts requirement); the boundary sticky
restore gates on generative source tree OIDs in both lanes, making
restored artifacts valid by construction (kills the clock-skew mtime
false-skip and ghost declarations from deleted sources); the node-test
worker budget derives from nproc instead of the runner label (labels
over-promise under load) and keeps the 2-worker cap when plans can
overlap; the deadcode task falls back to serial Knip below 8 cores;
the lint-lane i18n verify also runs for any ui-touching diff (keys are
extracted from all ui sources, not just ui/src/i18n); the oxlint
concurrency diagnostic moves to stderr; and workspace packages join
the prompt-snapshot always-run surface.
* fix(ci): fingerprint every generative source the boundary artifacts consume
Autoreview flagged inputs outside the tree-OID gate; toolchain, config,
and script inputs were already covered by the sticky key hash, but the
old actions/cache key also fingerprinted src/plugins/types.ts and
src/video-generation sources the OID list missed. Add those plus
src/channels (the plugin-sdk declaration graph reaches its public
types) to the fingerprint in all three sites.
* fix(ci): fold runner toolchain identity into the boundary sticky fingerprint
Autoreview: the sticky key hashes config/scripts/package.json/lockfile,
but a node/pnpm toolchain bump (pinned in the workflow, not the key)
could leave the key unchanged and bless artifacts built with the prior
toolchain. Record node/pnpm versions alongside the source tree OIDs in
all three fingerprint sites so a toolchain-only change forces a cold
build.
