The installation hit Blacksmith's backing-disk cap because sticky keys
embedded PR numbers and content hashes, minting a new disk per PR and
per input change until every mount 429-failed fleet-wide. node-deps
(#109752) and ext-boundary (#109804) were already re-keyed; this
converts the last two minters and audits the third suspect.
Keys changed:
- Vitest fs transform cache: `vitest-fs-v2-<pr-N|protected>-...` ->
the one existing `vitest-fs-v2-protected-<os>-<arch>-node-<ver>`
disk. Entries are content-hash keyed (vitest sha1 over
id+content+NODE_ENV+version+env config), so cross-PR sharing is safe
by construction: PRs now mount the protected snapshot directly
(read-only, commit false) and the separate PR-only seed mount is
deleted. Writers (non-PR shard writer + scheduled warm run) keep
explicit commit true. Disk count: 1 + O(open PRs) -> 1.
- Gradle: `gradle-v1-<task>-<pr-N|protected>-<hashFiles(deps)>` ->
`gradle-v2-<task>`. Task scope stays (light ktlint must not seed
heavy build lanes); PR number and dependency hash leave the key. The
dependency hash moved to an in-job fingerprint marker that makes the
non-PR writer rebuild its snapshot cold when inputs change, bounding
disk growth; PR mounts are read-only and safely reuse stale
snapshots because Gradle caches are content-addressed. Disk count:
O(tasks x PRs x dependency bumps) -> O(tasks) (7 today).
- Docker builder (useblacksmith/setup-docker-builder): audited, out of
scope. The action owns its key internally and always uses the repo
name only (src/setup_builder.ts getStickyDisk), so it is already one
disk per installation repo and cannot mint per-PR disks.
PR warm-path behavior changes:
- Vitest: PRs lose only PR-local warm entries for files the PR itself
changed (previously persisted on the pr-N disk between pushes);
changed files re-transform once per push, unchanged files still hit
the protected snapshot. PRs touching transform inputs
(lockfile/tsconfig/package.json) run cold per push since the
generation wipe is now local to the discarded clone.
- Gradle: dependency-bump PRs get warmer (stale-but-valid protected
snapshot instead of a cold fresh key); other PRs are unchanged.
Deleted `.github/workflows/pr-cache-cleanup.yml`: it existed for the
per-PR cache layer (#109425) and only deleted GitHub actions/cache
archives, which GitHub's own LRU/TTL eviction already handles for the
remaining fork/Windows per-PR archive paths.
Guards: pinned the O(1) vitest key + non-PR commit gate, added a
Gradle per-task key/writer/fingerprint guard, added a repo-wide scan
asserting no useblacksmith/stickydisk key ever contains
github.event.pull_request.number or hashFiles(), and replaced the
cleanup-workflow assertions with a stays-deleted check.
check-additional-extension-package-boundary is the light-run critical-path
pole. Measured on runs 29564442266/29564411446/29564259862 (8 vCPU, sticky
mounts 429-failing): Setup Node ~40s cold install, shard step 142-161s =
~101s cold plugin-sdk dts prep (nine parallel tsgo builds) + ~58s compiling
122 plugins at concurrency 6 + ~1s canary; warm actions/cache path on a main
push (29551077288) runs the shard in 67s.
Levers chosen from that data:
- Re-key the ext-boundary sticky disk from per-PR/per-config keys (v1) to one
O(1) protected key (v2), following the node_modules snapshot fix in
#109752. Blacksmith's installation-wide 1000-disk cap is saturated and
every mount 429s; per-PR keys are a direct contributor. The config/scripts/
lockfile hash moves from the key into the in-job .source-trees marker next
to the existing tree-OID + node/pnpm gate. Single semantic writer: the
boundary lane commits explicitly (true, not if-missing) on protected pushes
only; PR mounts and the check-lint consumer stay read-only, and the seed
step is now writer-only (saves ~11s per PR cold run).
- Bump the lane 8 -> 32 vCPU and raise OPENCLAW_EXTENSION_BOUNDARY_CONCURRENCY
6 -> 16: both the parallel dts prep and the 122 plugin compiles scale with
cores at similar billed core-minutes (same precedent as check-lint/deps).
Expected pole time: warm PR runs (protected snapshot tracks every main push)
drop to roughly checkout + install-skip + restore + incremental shard, well
under 2 minutes; cold runs that touch boundary trees shrink via the core
bump. Guarded in ci-workflow-guards so the O(1) key, single-writer commit,
and fingerprint composition cannot silently regress.
* perf(sdk): load Claude CLI identity from a narrow plugin artifact
src/plugin-sdk/anthropic-cli.ts snapshots CLAUDE_CLI_BACKEND_ID at module
scope through the sync facade loader, which jiti-evaluates the full
anthropic api.js barrel on source checkouts: 130.86s of self time per
cold worker on CI (Testbox import profile), silently stalling every job
whose graph reaches cli-runner/prepare.ts. A narrow cli-api.js artifact
carries the two static facts; Testbox proof: the reliability+helpers
agents-core pair drops from 157.6s to 14.3s.
* perf(ci): stripe unit-fast and tooling node tests on the full plan
core-fast ran the import-bound unit-fast graph as one job (247s vitest,
181s module evaluation) and core-tooling as one serial job (241s); both
now stripe on the full plan like the compact plan, so the compact-only
expansion is deleted and the docker helper config rides with the
isolated shard on both plans. Group hints refreshed from main run
29551077288; stale per-file commands hints dropped (the consolidation
landed); cache-writer selection and the warm workflow track the stripe
names.
* fix(state): tolerate vanished sqlite sidecars in agent-db permission sweep
existsSync+chmodSync raced SQLite's own WAL/SHM cleanup: a checkpoint or
close between the two calls throws ENOENT (observed from the transcript
reconcile worker in server-startup-web-fetch-bind on CI). chmod directly
and swallow only ENOENT, which removes the TOCTOU window.
* perf(ci): serve node_modules snapshots from O(1) protected sticky disks
Every Blacksmith mount of the dependency sticky disk currently 429s:
the v2 key minted one backing disk per PR and per manifest hash, which
saturated Blacksmith's installation-wide 1000-sticky-disk budget (run
29559333389, job checks-node-compact-small-15: "Sticky disk limit
exceeded"). Same-repo shards then fall back to a cold, storeless pnpm
install (~40s) on every run - worse than the ~22s actions/cache path
the sticky rollout replaced.
Re-key the snapshot to one stable disk per node-version and move the
install inputs into a runtime fingerprint marker:
- Key is now `<repo>-node-deps-bind-v3-<node-version>`; dependency
changes refresh the disk in place instead of allocating a new one.
- The fingerprint (manifest hashFiles set + node version + lockfile
mode) is evaluated on the bind step, before the mount lands, so the
'**/package.json' glob cannot sweep snapshot-internal manifests.
- Consumers mount read-only (commit: false); on fingerprint match they
restore importer archives and skip pnpm install entirely, on mismatch
they install against the clone's on-disk store (warm store, no
actions/cache download) without capturing.
- Writers are trusted non-PR jobs only: build-artifacts on canonical
pushes plus the scheduled vitest-cache-warm run as a deadline writer
(main pushes cancel each other under merge traffic and would starve
the snapshot). commit: on-change keeps warm no-op runs from
committing.
- Roll the consumer flag out to every Linux Blacksmith lane that
installs dependencies (build-artifacts, check-shard,
check-additional-shard, check-docs, checks-ui, control-ui-i18n,
native-i18n, qa-smoke-ci-profile, checks-fast-core, both contract
shards) with the same fork/dispatch gates as the nondist shard.
Guards now enumerate the consumer set, pin the O(1) key shape, enforce
single-writer commit expressions, and cover the fingerprint marker in
the importer capture/restore helper.
Verified locally: importer archive capture 0.96s/228K and restore
0.07s against the real 2.0GB hoisted tree; a snapshot untarred into a
fresh workspace resolves modules, runs bin shims, `pnpm exec`, and a
Vitest suite (one-time pnpm reconcile only when the absolute path
changes, which Blacksmith's fixed /home/runner/_work layout avoids).
* fix(ci): commit dependency snapshots explicitly, not via on-change heuristic
stickydisk's on-change mode compares allocated disk bytes with a 4KB
threshold, so a fingerprint refresh whose reinstall keeps usage stable
(metadata-only manifest edits, same-sized dependency swaps) could be
silently discarded, stranding every consumer on a stale marker and a
permanent reinstall path. Writers now commit explicitly, mirroring the
Vitest transform disk's rationale; warm no-op writer runs re-commit an
identical snapshot, which is cheap and safe. Also document that the
non-PR commit gate binds cooperating code only - the enforced trust
boundary stays the fork/dispatch runner gate, matching the protected
node-compile disk's posture.
* test(ci): guard sticky consumers against writerless node-version key splits
Reviewer follow-up: the snapshot key is partitioned by node-version and
both writers (build-artifacts, vitest-cache-warm) rely on the action
default, so a consumer pinning any other version would split onto a
key nobody seeds and silently regress to permanently cold installs.
Pin the action default to 24.x and assert every sticky consumer
resolves to that same key segment.
* test: narrow sticky-consumer step.with before indexing
The filter's runtime narrowing did not carry into the map callback, so
step.with indexing failed check-test-types (TS18048). Collect a narrowed
stepWith instead.
Measured breakdown of check-prompt-snapshots in run 29557851276 (5m03s):
- 37s checkout + Node env setup
- 4m25s inside one `pnpm prompt:snapshots:check` process:
- ~35s tsx import of the generator graph
- ~3min first createOpenClawCodingTools call: the message-tool schema
build resolves the scenario channel plugin, the empty loaded registry
falls back to the bundled-channel jiti loader
(extensions/telegram/index.ts), and jiti re-transpiles the core source
graph (136.6s jiti self time in --cpu-prof; second generation call in
the same process takes 0.3s)
- ~1min more for the cold provider-plugin lookup in
normalizeAgentRuntimeTools; no provider plugin owns tool-schema hooks
for the `codex` harness provider, so the jiti load only rediscovers a
no-op
- <2s actual rendering + oxfmt + drift compare
Split axis decision: the planned 2-3 way scenario shard matrix cannot
lower the pole because the cost is a one-time per-process transpile;
every shard would pay the full ~4min again (jiti fsCache does not help:
a warm-cache rerun still took 200s). Fix the generator instead:
- Preload each scenario's channel plugin surface (channel-plugin-api.js,
resolved via the bundled-plugin public-surface resolver) through the
ambient tsx/vitest ESM graph, and pin exactly that channel in a scoped
loaded registry while the scenario's tools are built. Snapshot bytes
are unchanged: the loaded plugin object is the same export the bundled
jiti fallback materializes, and cross-channel action discovery still
sees no other loaded channels.
- Pass allowProviderRuntimePluginLoad=false to normalizeAgentRuntimeTools;
registry-only resolution returns the same undefined plugin instantly.
Coverage guard: test/scripts/prompt-snapshots.test.ts now generates the
snapshot files and asserts zero jiti plugin-loader calls (with the
offending transform targets in the failure message), so a future scenario
or channel that misses the preload fails the suite instead of silently
restoring the 4-minute lane.
Proof: `pnpm prompt:snapshots:check` passes byte-identical against the
committed fixtures in 54s wall / 35s CPU locally (was 354s wall / 306s
CPU); focused vitest suites (32 tests incl. new guard), tsgo core-test +
extensions-test lanes, scoped oxlint/oxfmt, and `git diff --check` are
green. Expected CI pole when the lane fires: ~5.1min -> ~1.5-2min.
* perf(ci): move the three light-run pole jobs to 32 vCPU runners
check-lint (~186s oxlint shards), check-dependencies (~157s concurrent
knip), and build-artifacts (~126s tsdown+checks) are the critical path of
every light PR run; all three parallelize with cores at similar billed
core-minutes. Guard test updated.
* test: align prerelease-plan guard with the 32 vCPU check-dependencies row
Add --connect-timeout 30 and --max-time 120 to the curl command that
downloads the Opengrep install script and pipes it to bash. Both the
PR-diff (opengrep-precise) and full-scan (opengrep-precise-full)
workflows are covered. A stuck TLS handshake or stalled download
could previously occupy a CI runner until the job-level timeout.
Co-authored-by: Claude <noreply@anthropic.com>
* perf(build): cut write-cli-startup-metadata wall time via launcher-boot browser help and parallel renders
* perf(ci): fetch preflight parent metadata blob-less instead of a full depth-2 snapshot
* test(ci): expect the blob-less preflight parent fetch in checkout guards
* fix(build): default the bundled root-help render env
RootHelpRenderOptions marks env optional but spawnText requires one;
check-test-types caught the undefined assignment.
* feat(android): add Wear OS companion
Co-authored-by: Sebastian Schubotz <git@sibbl.net>
Co-authored-by: IWhatsskill <284122573+IWhatsskill@users.noreply.github.com>
* style(android): format Wear release gate test
* test(android): cover Wear release CI contracts
* test(android): import Wear JSON fixture type
* fix(android): harden Wear proxy event actor
* test(android): stabilize Wear proxy actor tests
* test(android): isolate Wear proxy actor lifecycles
* test(android): own Wear actor dispatchers
* test(android): use real time for Wear actor tests
* test(android): own Wear actors in test scope
* test(android): stop Wear actors explicitly
* test(android): drain Wear actor cancellation
* test(android): isolate Wear actors from test scheduler
* test(android): inject Wear test clock
---------
Co-authored-by: IWhatsskill <284122573+IWhatsskill@users.noreply.github.com>
* perf(ci): carry Android Gradle caches on a Blacksmith sticky disk
The android lane's setup-java gradle cache almost never restores under the
repo's churned GitHub cache quota, so every run re-resolves the full
dependency graph (~501s observed for android-build-play). Mount a
task-scoped Blacksmith sticky disk at /var/tmp/openclaw-gradle and point
GRADLE_USER_HOME at it for same-repo non-fork non-dispatch runs, mirroring
the extension-boundary sticky pattern (per-PR-or-protected key,
commit: if-missing, identical trust gate). Fork PRs and manual dispatches
keep the existing GitHub cache path and non-Blacksmith runners.
* fix(ci): let the Gradle sticky snapshot keep learning
Autoreview P2: if-missing froze the snapshot at first hydration, so
later build-cache entries never persisted. on-change commits when the
filesystem changed; content-addressed entries and task-scoped keys keep
it race-safe.
* fix(ci): address reviewer findings across the perf-lever batch
Seven follow-ups from Codex review feedback on the landed CI perf PRs:
docker packaging switches to the full build profile (the ciArtifacts
step env clobbered its canonical-dts requirement); the boundary sticky
restore gates on generative source tree OIDs in both lanes, making
restored artifacts valid by construction (kills the clock-skew mtime
false-skip and ghost declarations from deleted sources); the node-test
worker budget derives from nproc instead of the runner label (labels
over-promise under load) and keeps the 2-worker cap when plans can
overlap; the deadcode task falls back to serial Knip below 8 cores;
the lint-lane i18n verify also runs for any ui-touching diff (keys are
extracted from all ui sources, not just ui/src/i18n); the oxlint
concurrency diagnostic moves to stderr; and workspace packages join
the prompt-snapshot always-run surface.
* fix(ci): fingerprint every generative source the boundary artifacts consume
Autoreview flagged inputs outside the tree-OID gate; toolchain, config,
and script inputs were already covered by the sticky key hash, but the
old actions/cache key also fingerprinted src/plugins/types.ts and
src/video-generation sources the OID list missed. Add those plus
src/channels (the plugin-sdk declaration graph reaches its public
types) to the fingerprint in all three sites.
* fix(ci): fold runner toolchain identity into the boundary sticky fingerprint
Autoreview: the sticky key hashes config/scripts/package.json/lockfile,
but a node/pnpm toolchain bump (pinned in the workflow, not the key)
could leave the key unchanged and bless artifacts built with the prior
toolchain. Record node/pnpm versions alongside the source tree OIDs in
all three fingerprint sites so a toolchain-only change forces a cold
build.
Platform-aware install kinds: Linux AppImage and macOS self-install in place;
Windows defers — the update downloads and verifies in the background, then the
NSIS installer runs only from the user-confirmed restart (Tauri's installer
launch exits the process, so it must never fire behind a silent auto-check).
Linux system packages stay notify-only.
The release workflow gains dispatch-gated unsigned macOS/Windows test bundles
signed with the same minisign key, and a separate desktop-test update channel:
latest.json stays Linux-only on every release, while an opt-in fixed
'desktop-test' prerelease hosts latest-desktop-test.json (monotonic version
guard) so Linux-only releases can never strand distributed test builds. Build
jobs check out the SHA the validation job resolved, closing the tag-move race.
* perf(ci): run the deadcode Knip scans concurrently
check-dependencies spent ~194s running seven Knip child processes
strictly serially (three package scripts, each iterating its scans).
The scans are independent processes over separate configs: both
deadcode scripts now Promise.all their scans, the CI task launches the
three package scripts concurrently with buffered logs, and the lane
moves to the 16 vCPU class for core/memory headroom. The
missing-exports-script contract violation now fails fast before
launching scans. Local: unused-files 218% CPU, 37s wall.
* test(ci): align prerelease plan with the 16 vCPU dependencies lane
* feat(android): add Wear proxy protocol
Introduce a bounded, versioned phone/watch RPC contract and wire its test, lint, and build gates before either runtime endpoint lands.
Co-authored-by: Sebastian Schubotz <git@sibbl.de>
* test(android): lock Wear wire names
Cover every supported RPC method and event discriminator so phone and watch cannot silently drift.
Co-authored-by: Sebastian Schubotz <git@sibbl.net>
* fix(android): bound Wear JSON depth
Reject excessively nested Data Layer envelopes before kotlinx.serialization can recurse through them.
* fix(android): enforce Wear depth on encode
Keep outbound messages inside the same JSON nesting contract enforced by the decoder.
* fix(android): preflight Wear payload depth
Traverse arbitrary JSON payloads iteratively before serialization so deeply nested local trees fail safely.
---------
Co-authored-by: Sebastian Schubotz <git@sibbl.de>
Co-authored-by: Sebastian Schubotz <git@sibbl.net>
* feat(migrate): add Hermes memory-only import and a shared memory-import core
* feat(onboarding): offer detected memory imports during CLI setup and guided onboarding
* feat(ui): show a first-run memory-import offer in Control UI onboarding mode
* feat(linux-app): open the first-run dashboard in onboarding mode
* feat(macos): add the onboarding memory-import page
* docs: document the onboarding memory-import page across surfaces
* chore(i18n): translate onboarding memory-import strings for control-ui and native locales
* refactor: keep memory-import internals unexported for deadcode gates
* fix(ci): resolve lint findings in onboarding memory import
* chore(i18n): refresh native inventory after lint refactor
* fix(ci): skip updater artifacts in the unsigned Linux companion PR lane
* chore(i18n): reconcile locale artifacts after rebase
* perf(ci): carry extension boundary artifacts on a Blacksmith sticky disk
The boundary lane spends ~113s of its 214s rebuilding plugin-sdk
declaration artifacts that an actions/cache was meant to carry — but
that cache is evicted so fast under the repo quota that even its prefix
restore never hits. Same-repo runs now keep the artifacts on a sticky
disk with a coarse toolchain key (source changes intentionally hit a
stale snapshot: restored mtimes predate the fresh checkout, so the
mtime-based freshness predicate always routes into the incremental
rebuild — a stale snapshot can never false-skip the compile). Fork PRs
keep the GitHub cache path and can never write repository-global
snapshots. Seed/restore round-trip verified locally with rsync -aR
layout mirroring.
* perf(ci): restore boundary artifacts for the lint lane too
check-lint's shard runner spawns the same plugin-sdk boundary artifact
prep the boundary lane snapshots (~72s of its wall clock); it now
mounts the shared sticky read-only-by-convention (only the boundary
lane seeds) and restores before the run step.
* fix(ci): make the lint boundary-sticky mount strictly read-only
Autoreview caught a first-hydration race: with commit if-missing, a
cold-key lint job racing the boundary lane could commit an empty,
marker-less snapshot that if-missing then never repairs. commit false
makes lint a pure reader; only the boundary lane can seed.
* perf(ci): unblock parallel oxlint shards and gate the lint-lane i18n verify
check-lint is the slowest static lane (median 252s): the control-UI
i18n verify runs 86s unconditionally and the three oxlint shards run
serially (89s) because the parallel gate demands 48GB of memory — a
local-Mac threshold that Blacksmith's 32GB 16 vCPU class can never
meet. CI now parallelizes shards at 8+ cores / 24GB+, and the lint case
reuses the existing run_control_ui_i18n manifest gate so diffs that
cannot affect generated catalogs skip the verify while oxlint always
runs.
* chore(ci): print oxlint shard concurrency decision
The lint-lane validation run still executed shards serially on a 16
vCPU runner; one log line exposes the resolved concurrency and the
host resources the decision saw.
