* ci: add hourly PR CI sweeper for dropped pull_request runs
Fresh PRs race GitHub's merge-ref computation: the open-event CI run can
drop entirely or be created as an un-rerunnable startup_failure (~10-16
runs daily). The sweeper lists recently updated open PRs hourly, finds
heads whose only pull_request-event CI runs are startup failures (or
missing), and re-fires the event by close/reopen with the Barnacle app
token (GITHUB_TOKEN events would not trigger workflows).
Safety: 10-minute quiet window, 24h lookback, skips drafts, merge
conflicts, pending mergeability, and auto-merge PRs (close cancels
auto-merge); revalidates state, head, and CI attachment immediately
before mutating; per-PR budget of two sweeper closes and a per-sweep cap
of ten; reopen-on-unknown ownership so a stranded close (silent) always
loses to a spurious reopen (visible); manual dispatch supports dry_run.
Accepted tradeoffs are documented inline: shared-SHA PR topologies can
mask a dropped run (skip-only miss; run.pull_requests matching would
break fork PRs), and app-auth failover at worst doubles the close budget.
* test(ci): exercise pr-ci-sweeper runner with a faked client
* fix(test): lint-clean pr-ci-sweeper runner fakes
* fix(ci): bound all openclaw-npm-release git fetch operations with timeout
Wrap all six git fetch commands with a 120-second deadline
and 10-second termination grace period, matching the pattern
established in #108940.
Fixes#109176
* test(ci): cover npm release fetch deadlines
Co-authored-by: wanyongstar <wan.yong@xydigit.com>
* ci: raise Control UI startup budget
Unblock exact-head builds after the compressed startup bundle reached the previous 370 KiB ratchet.
* ci: drop superseded Control UI budget bump
Current main now carries the smaller 340 KiB startup ratchet from #110528, so remove the temporary 372 KiB adjustment while keeping the PR head conflict-free.
---------
Co-authored-by: Peter Steinberger <steipete@gmail.com>
## Summary
- Problem: The `git fetch` in `macos-release.yml` had no timeout bound, risking indefinite CI hangs on network stalls. Same pattern already fixed for Mantis Crabbox workflows in #108940.
- Solution: Add `timeout --signal=TERM --kill-after=10s 120s` before the `git fetch` command.
- What changed: Wrapped the `git fetch --no-tags origin` command with timeout.
- What did NOT change: All release logic remains unchanged.
## Real behavior proof
- **Behavior addressed:** CI macos-release could hang during git fetch.
- **Evidence:** Same timeout pattern as merged Crabbox fix (#108940 / 16c2bbb).
- **Change:** `timeout --signal=TERM --kill-after=10s 120s git fetch --no-tags origin` replaces `git fetch --no-tags origin`.
## Root Cause
- The git fetch was added before the timeout-bounding pattern was standardized across CI workflows.
## Summary
- Problem: The `git clone` of `openclaw/docs.git` in `docs-sync-publish.yml` had no timeout bound, risking indefinite CI hangs on network stalls. Same pattern already fixed for Mantis Crabbox workflows in #108940.
- Solution: Add `timeout --signal=TERM --kill-after=10s 120s` before the `git clone` command.
- What changed: Wrapped the `git clone` command with timeout, matching the bounded pattern from #108940.
- What did NOT change: All retry logic, publish sync steps, and non-network operations remain unchanged.
## Real behavior proof
- **Behavior addressed:** CI docs-sync-publish workflow could hang indefinitely during git clone of external docs repo.
- **Real environment tested:** Pattern analysis and cross-reference with already-merged Crabbox fix (#108940 / 16c2bbb).
- **Exact change:** `if timeout --signal=TERM --kill-after=10s 120s git clone \` replaces `if git clone \`.
- **Evidence after fix:** Same `timeout --signal=TERM --kill-after=10s 120s` wrapper applied to Crabbox source retrieval.
- **What was not tested:** Live CI run.
## Root Cause
- The git clone was added before the timeout-bounding pattern was standardized across CI workflows. Missing automated check for unbounded external git operations.
* perf(ci): cut the pre-fan-out critical path on canonical runs
Two changes to the run head and matrix shape:
1. runner-admission was a hosted 90s sleep every run queued behind
(observed 1.7min hosted-queue latency before the sleep started). The
debounce now lives at the tail of preflight: heavy jobs all need
preflight, so a superseding main push still cancels the run before
fan-out while only one 4 vCPU runner has been spent, and preflight's
own work usually exceeds the window so the residual sleep is zero.
security-fast (hosted, dependency-free) starts immediately.
2. Canonical main pushes now use the compact bin plan like PRs: the
82-job named matrix drained the runner pool for ~4.5min (job starts
trickled from minute 5.2 to 9.7 in run 29592647843) with no
branch-protection consumer for per-shard names on main. Coverage is
identical; dispatch/release-gate runs keep the full named matrix.
* perf(test): boot TUI PTY suite fixtures concurrently
tsx+TUI startup dominated the harness file's wall time and the three
suite PTYs booted serially. Boot them concurrently in beforeAll
(allSettled so a failed boot still assigns survivors for afterAll
cleanup); the env-specific fixtures never receive input, so their tests
only await their own readiness output. The slow-startup test now proves
frame ordering on the append-only output, which the old sequential
waits did not. File wall 10.2s -> ~4.9s, 3/3 repeat runs green.
* docs(ci): align gate and debounce descriptions with the removed admission job
* test(tooling): wait for readiness file content, not existence
writeFileSync creates the file before its bytes land, so the existence
poll raced the child's write on loaded runners and read an empty ready
file (observed in compact-small-4, run 29615028678). Poll for non-empty
content at both readiness sites.
Success-only gating starved the deps/vitest snapshots: under merge
traffic every main push cancels the prior CI run, so the deadline
writer skipped for hours (15:34->18:45 observed) while PRs paid full
reinstalls against stale fingerprints. The warm job checks out main
tip regardless of the triggering run, so cancelled completions are
safe refresh signals; failed runs still skip.
The installation hit Blacksmith's backing-disk cap because sticky keys
embedded PR numbers and content hashes, minting a new disk per PR and
per input change until every mount 429-failed fleet-wide. node-deps
(#109752) and ext-boundary (#109804) were already re-keyed; this
converts the last two minters and audits the third suspect.
Keys changed:
- Vitest fs transform cache: `vitest-fs-v2-<pr-N|protected>-...` ->
the one existing `vitest-fs-v2-protected-<os>-<arch>-node-<ver>`
disk. Entries are content-hash keyed (vitest sha1 over
id+content+NODE_ENV+version+env config), so cross-PR sharing is safe
by construction: PRs now mount the protected snapshot directly
(read-only, commit false) and the separate PR-only seed mount is
deleted. Writers (non-PR shard writer + scheduled warm run) keep
explicit commit true. Disk count: 1 + O(open PRs) -> 1.
- Gradle: `gradle-v1-<task>-<pr-N|protected>-<hashFiles(deps)>` ->
`gradle-v2-<task>`. Task scope stays (light ktlint must not seed
heavy build lanes); PR number and dependency hash leave the key. The
dependency hash moved to an in-job fingerprint marker that makes the
non-PR writer rebuild its snapshot cold when inputs change, bounding
disk growth; PR mounts are read-only and safely reuse stale
snapshots because Gradle caches are content-addressed. Disk count:
O(tasks x PRs x dependency bumps) -> O(tasks) (7 today).
- Docker builder (useblacksmith/setup-docker-builder): audited, out of
scope. The action owns its key internally and always uses the repo
name only (src/setup_builder.ts getStickyDisk), so it is already one
disk per installation repo and cannot mint per-PR disks.
PR warm-path behavior changes:
- Vitest: PRs lose only PR-local warm entries for files the PR itself
changed (previously persisted on the pr-N disk between pushes);
changed files re-transform once per push, unchanged files still hit
the protected snapshot. PRs touching transform inputs
(lockfile/tsconfig/package.json) run cold per push since the
generation wipe is now local to the discarded clone.
- Gradle: dependency-bump PRs get warmer (stale-but-valid protected
snapshot instead of a cold fresh key); other PRs are unchanged.
Deleted `.github/workflows/pr-cache-cleanup.yml`: it existed for the
per-PR cache layer (#109425) and only deleted GitHub actions/cache
archives, which GitHub's own LRU/TTL eviction already handles for the
remaining fork/Windows per-PR archive paths.
Guards: pinned the O(1) vitest key + non-PR commit gate, added a
Gradle per-task key/writer/fingerprint guard, added a repo-wide scan
asserting no useblacksmith/stickydisk key ever contains
github.event.pull_request.number or hashFiles(), and replaced the
cleanup-workflow assertions with a stays-deleted check.
check-additional-extension-package-boundary is the light-run critical-path
pole. Measured on runs 29564442266/29564411446/29564259862 (8 vCPU, sticky
mounts 429-failing): Setup Node ~40s cold install, shard step 142-161s =
~101s cold plugin-sdk dts prep (nine parallel tsgo builds) + ~58s compiling
122 plugins at concurrency 6 + ~1s canary; warm actions/cache path on a main
push (29551077288) runs the shard in 67s.
Levers chosen from that data:
- Re-key the ext-boundary sticky disk from per-PR/per-config keys (v1) to one
O(1) protected key (v2), following the node_modules snapshot fix in
#109752. Blacksmith's installation-wide 1000-disk cap is saturated and
every mount 429s; per-PR keys are a direct contributor. The config/scripts/
lockfile hash moves from the key into the in-job .source-trees marker next
to the existing tree-OID + node/pnpm gate. Single semantic writer: the
boundary lane commits explicitly (true, not if-missing) on protected pushes
only; PR mounts and the check-lint consumer stay read-only, and the seed
step is now writer-only (saves ~11s per PR cold run).
- Bump the lane 8 -> 32 vCPU and raise OPENCLAW_EXTENSION_BOUNDARY_CONCURRENCY
6 -> 16: both the parallel dts prep and the 122 plugin compiles scale with
cores at similar billed core-minutes (same precedent as check-lint/deps).
Expected pole time: warm PR runs (protected snapshot tracks every main push)
drop to roughly checkout + install-skip + restore + incremental shard, well
under 2 minutes; cold runs that touch boundary trees shrink via the core
bump. Guarded in ci-workflow-guards so the O(1) key, single-writer commit,
and fingerprint composition cannot silently regress.
* perf(sdk): load Claude CLI identity from a narrow plugin artifact
src/plugin-sdk/anthropic-cli.ts snapshots CLAUDE_CLI_BACKEND_ID at module
scope through the sync facade loader, which jiti-evaluates the full
anthropic api.js barrel on source checkouts: 130.86s of self time per
cold worker on CI (Testbox import profile), silently stalling every job
whose graph reaches cli-runner/prepare.ts. A narrow cli-api.js artifact
carries the two static facts; Testbox proof: the reliability+helpers
agents-core pair drops from 157.6s to 14.3s.
* perf(ci): stripe unit-fast and tooling node tests on the full plan
core-fast ran the import-bound unit-fast graph as one job (247s vitest,
181s module evaluation) and core-tooling as one serial job (241s); both
now stripe on the full plan like the compact plan, so the compact-only
expansion is deleted and the docker helper config rides with the
isolated shard on both plans. Group hints refreshed from main run
29551077288; stale per-file commands hints dropped (the consolidation
landed); cache-writer selection and the warm workflow track the stripe
names.
* fix(state): tolerate vanished sqlite sidecars in agent-db permission sweep
existsSync+chmodSync raced SQLite's own WAL/SHM cleanup: a checkpoint or
close between the two calls throws ENOENT (observed from the transcript
reconcile worker in server-startup-web-fetch-bind on CI). chmod directly
and swallow only ENOENT, which removes the TOCTOU window.
* perf(ci): serve node_modules snapshots from O(1) protected sticky disks
Every Blacksmith mount of the dependency sticky disk currently 429s:
the v2 key minted one backing disk per PR and per manifest hash, which
saturated Blacksmith's installation-wide 1000-sticky-disk budget (run
29559333389, job checks-node-compact-small-15: "Sticky disk limit
exceeded"). Same-repo shards then fall back to a cold, storeless pnpm
install (~40s) on every run - worse than the ~22s actions/cache path
the sticky rollout replaced.
Re-key the snapshot to one stable disk per node-version and move the
install inputs into a runtime fingerprint marker:
- Key is now `<repo>-node-deps-bind-v3-<node-version>`; dependency
changes refresh the disk in place instead of allocating a new one.
- The fingerprint (manifest hashFiles set + node version + lockfile
mode) is evaluated on the bind step, before the mount lands, so the
'**/package.json' glob cannot sweep snapshot-internal manifests.
- Consumers mount read-only (commit: false); on fingerprint match they
restore importer archives and skip pnpm install entirely, on mismatch
they install against the clone's on-disk store (warm store, no
actions/cache download) without capturing.
- Writers are trusted non-PR jobs only: build-artifacts on canonical
pushes plus the scheduled vitest-cache-warm run as a deadline writer
(main pushes cancel each other under merge traffic and would starve
the snapshot). commit: on-change keeps warm no-op runs from
committing.
- Roll the consumer flag out to every Linux Blacksmith lane that
installs dependencies (build-artifacts, check-shard,
check-additional-shard, check-docs, checks-ui, control-ui-i18n,
native-i18n, qa-smoke-ci-profile, checks-fast-core, both contract
shards) with the same fork/dispatch gates as the nondist shard.
Guards now enumerate the consumer set, pin the O(1) key shape, enforce
single-writer commit expressions, and cover the fingerprint marker in
the importer capture/restore helper.
Verified locally: importer archive capture 0.96s/228K and restore
0.07s against the real 2.0GB hoisted tree; a snapshot untarred into a
fresh workspace resolves modules, runs bin shims, `pnpm exec`, and a
Vitest suite (one-time pnpm reconcile only when the absolute path
changes, which Blacksmith's fixed /home/runner/_work layout avoids).
* fix(ci): commit dependency snapshots explicitly, not via on-change heuristic
stickydisk's on-change mode compares allocated disk bytes with a 4KB
threshold, so a fingerprint refresh whose reinstall keeps usage stable
(metadata-only manifest edits, same-sized dependency swaps) could be
silently discarded, stranding every consumer on a stale marker and a
permanent reinstall path. Writers now commit explicitly, mirroring the
Vitest transform disk's rationale; warm no-op writer runs re-commit an
identical snapshot, which is cheap and safe. Also document that the
non-PR commit gate binds cooperating code only - the enforced trust
boundary stays the fork/dispatch runner gate, matching the protected
node-compile disk's posture.
* test(ci): guard sticky consumers against writerless node-version key splits
Reviewer follow-up: the snapshot key is partitioned by node-version and
both writers (build-artifacts, vitest-cache-warm) rely on the action
default, so a consumer pinning any other version would split onto a
key nobody seeds and silently regress to permanently cold installs.
Pin the action default to 24.x and assert every sticky consumer
resolves to that same key segment.
* test: narrow sticky-consumer step.with before indexing
The filter's runtime narrowing did not carry into the map callback, so
step.with indexing failed check-test-types (TS18048). Collect a narrowed
stepWith instead.
Measured breakdown of check-prompt-snapshots in run 29557851276 (5m03s):
- 37s checkout + Node env setup
- 4m25s inside one `pnpm prompt:snapshots:check` process:
- ~35s tsx import of the generator graph
- ~3min first createOpenClawCodingTools call: the message-tool schema
build resolves the scenario channel plugin, the empty loaded registry
falls back to the bundled-channel jiti loader
(extensions/telegram/index.ts), and jiti re-transpiles the core source
graph (136.6s jiti self time in --cpu-prof; second generation call in
the same process takes 0.3s)
- ~1min more for the cold provider-plugin lookup in
normalizeAgentRuntimeTools; no provider plugin owns tool-schema hooks
for the `codex` harness provider, so the jiti load only rediscovers a
no-op
- <2s actual rendering + oxfmt + drift compare
Split axis decision: the planned 2-3 way scenario shard matrix cannot
lower the pole because the cost is a one-time per-process transpile;
every shard would pay the full ~4min again (jiti fsCache does not help:
a warm-cache rerun still took 200s). Fix the generator instead:
- Preload each scenario's channel plugin surface (channel-plugin-api.js,
resolved via the bundled-plugin public-surface resolver) through the
ambient tsx/vitest ESM graph, and pin exactly that channel in a scoped
loaded registry while the scenario's tools are built. Snapshot bytes
are unchanged: the loaded plugin object is the same export the bundled
jiti fallback materializes, and cross-channel action discovery still
sees no other loaded channels.
- Pass allowProviderRuntimePluginLoad=false to normalizeAgentRuntimeTools;
registry-only resolution returns the same undefined plugin instantly.
Coverage guard: test/scripts/prompt-snapshots.test.ts now generates the
snapshot files and asserts zero jiti plugin-loader calls (with the
offending transform targets in the failure message), so a future scenario
or channel that misses the preload fails the suite instead of silently
restoring the 4-minute lane.
Proof: `pnpm prompt:snapshots:check` passes byte-identical against the
committed fixtures in 54s wall / 35s CPU locally (was 354s wall / 306s
CPU); focused vitest suites (32 tests incl. new guard), tsgo core-test +
extensions-test lanes, scoped oxlint/oxfmt, and `git diff --check` are
green. Expected CI pole when the lane fires: ~5.1min -> ~1.5-2min.
