Second calibration pass after the cli-runner reliability whale fix
(#109772) and the stripe-wall correction (#109899): the compact group
hints drifted below loaded-fleet reality, so FFD packed the heaviest
groups into ~300s bins (checks-node-compact-large-2 at 242-314s shard
wall, up to 389s job wall) while tail bins idled.
Method: parsed [shard:*] begin/end timestamps from all 147 compact job
logs across seven green CI runs whose head SHAs carry the whale-fix
hints (runs 29605136624, 29605203485, 29605983019, 29606701461,
29611308972, 29611457693, 29611500865; 7 samples per group). Each hint
is the per-group mean after dropping cache-warm/contention outliers
outside [median/1.5, median*1.5].
Packing cap: with honest loaded walls a 220s cap no longer fits either
pool (adds one job to each); 235s keeps the same job count (6 large +
15 small + dist) and flattens the ceiling. Median job setup overhead
measured 60s (p90 87s), so a 235s bin stays near the 5-minute PR
budget.
Predicted bin shard walls (sum of measured group means):
before: large 220/301/200/219/257/112, small heavies 251/235/218/217/215, max 301
after: large 235/232/229/228/228/156, small heavies 236/234/234/234/233, max 236
large-pool max/mean 1.38 -> 1.08
Biggest hint deltas (old -> new): startup-core 98->156, core-tooling-4
71->125, core-unit-fast-isolated 50->90, core-tooling-3 82->108,
runtime-server 14->29, storage-state 55->70, media-ui 113->124,
tui-pty 103->116, runner-cli-1 18->8, unit-src-security 108->95.
Exclusive bins keep their 150s cap and 5-bin count: tui-pty (116) no
longer shares a bin with tooling-isolated (166s combined measured);
FFD now pairs tooling-2 with tooling-isolated at 144s.
STRIPE_FILE_SECONDS_HINTS left unchanged: measured stripe walls sit in
exclusive bins far below the packing ceiling (tooling stripes
94-125s), and refitting per-file hints would reshuffle stripe
membership and invalidate the measured group means above.
Group walls measured from compact run 29564411446 (post-#109769):
core-unit-fast-isolated runs 78.6s on 4 vCPU because fork-per-file
isolation parallelizes poorly there; pin it to the 8 vCPU class where
the same segment runs ~50s. Unit-fast stripes measure ~25-37s (hinted
100/60) and tooling stripes 71-87s; refresh both so the packer stops
overfilling small bins (the 299s compact tail this run produced).
Cheap stripes may now co-locate in one bin, so the compact test only
requires their presence.
8858944a81 imports formatErrorMessage from scripts/openclaw-npm-postpublish-verify.ts,
but the .mjs had no sibling .d.mts like every other scripts/lib module imported
from .ts, so check-test-types fails repo-wide with TS7016.
* perf(sdk): load Claude CLI identity from a narrow plugin artifact
src/plugin-sdk/anthropic-cli.ts snapshots CLAUDE_CLI_BACKEND_ID at module
scope through the sync facade loader, which jiti-evaluates the full
anthropic api.js barrel on source checkouts: 130.86s of self time per
cold worker on CI (Testbox import profile), silently stalling every job
whose graph reaches cli-runner/prepare.ts. A narrow cli-api.js artifact
carries the two static facts; Testbox proof: the reliability+helpers
agents-core pair drops from 157.6s to 14.3s.
* perf(ci): stripe unit-fast and tooling node tests on the full plan
core-fast ran the import-bound unit-fast graph as one job (247s vitest,
181s module evaluation) and core-tooling as one serial job (241s); both
now stripe on the full plan like the compact plan, so the compact-only
expansion is deleted and the docker helper config rides with the
isolated shard on both plans. Group hints refreshed from main run
29551077288; stale per-file commands hints dropped (the consolidation
landed); cache-writer selection and the warm workflow track the stripe
names.
* fix(state): tolerate vanished sqlite sidecars in agent-db permission sweep
existsSync+chmodSync raced SQLite's own WAL/SHM cleanup: a checkpoint or
close between the two calls throws ENOENT (observed from the transcript
reconcile worker in server-startup-web-fetch-bind on CI). chmod directly
and swallow only ENOENT, which removes the TOCTOU window.
Recalibrate COMPACT_GROUP_SECONDS_HINTS and the cli-runner stripe file
hints against mean [shard:*] begin->end wall clock from five green
Blacksmith compact PR runs (29557851276, 29558164241, 29558528472,
29558634980, 29558677406).
Measured before (job wall, 5-run mean): 26 jobs; tail compact-large-6 =
317s (cli-runner stripe with the whale file) vs siblings 139-290s;
compact-small-10 = 277s. Key drifts: cli-runner.reliability.test.ts
measures 245s solo (import-dominated; stripe hint said 7s, so LPT packed
it with extra groups), auto-reply commands stripes 26/9/12s vs 92s
hints, core-unit-fast 142s vs 190s, tui-pty 103s vs 200s.
Computed after (planner output, measured-mean walls): 23 jobs (-4 fixed
~60s setups); reliability stripe isolated as its own 305s bin (floor set
by that single file's 245s module import), every sibling bin <= 280s;
large-runner class max/mean = 1.145. Pinned-worker and exclusive-bin
invariants unchanged.
* fix(ci): address reviewer findings across the perf-lever batch
Seven follow-ups from Codex review feedback on the landed CI perf PRs:
docker packaging switches to the full build profile (the ciArtifacts
step env clobbered its canonical-dts requirement); the boundary sticky
restore gates on generative source tree OIDs in both lanes, making
restored artifacts valid by construction (kills the clock-skew mtime
false-skip and ghost declarations from deleted sources); the node-test
worker budget derives from nproc instead of the runner label (labels
over-promise under load) and keeps the 2-worker cap when plans can
overlap; the deadcode task falls back to serial Knip below 8 cores;
the lint-lane i18n verify also runs for any ui-touching diff (keys are
extracted from all ui sources, not just ui/src/i18n); the oxlint
concurrency diagnostic moves to stderr; and workspace packages join
the prompt-snapshot always-run surface.
* fix(ci): fingerprint every generative source the boundary artifacts consume
Autoreview flagged inputs outside the tree-OID gate; toolchain, config,
and script inputs were already covered by the sticky key hash, but the
old actions/cache key also fingerprinted src/plugins/types.ts and
src/video-generation sources the OID list missed. Add those plus
src/channels (the plugin-sdk declaration graph reaches its public
types) to the fingerprint in all three sites.
* fix(ci): fold runner toolchain identity into the boundary sticky fingerprint
Autoreview: the sticky key hashes config/scripts/package.json/lockfile,
but a node/pnpm toolchain bump (pinned in the workflow, not the key)
could leave the key unchanged and bless artifacts built with the prior
toolchain. Record node/pnpm versions alongside the source tree OIDs in
all three fingerprint sites so a toolchain-only change forces a cold
build.
* test: fix environment-dependent failures on built and partial-clone checkouts
Canvas A2UI fixtures now pin the resolved root through a new canvas
test-api hook: dev checkouts with a locally built a2ui.bundle.js won
candidate resolution and shadowed the fixture root. Speech-core prefs
tests resolve the canonical tmpdir (macOS /tmp symlink breaks fs-safe
store roots), and the bundled plugin install test asks discovery for
the canonical bundled path instead of hardcoding the source tree.
* fix(release): keep evidence verifier blob reads local-deterministic
git show against a forged or missing SHA lazily fetched from the
promisor remote on partial clones, hanging a security check on network
state (266s observed). GIT_NO_LAZY_FETCH pins verification to local
objects; missing blobs fail closed as before.
* perf(test): pay the command-handler barrel once and merge subagent command tests
commands-name and commands-login imported the full handler barrel
(~1,700 modules) for registration assertions; those now live in a
dedicated registration test, dropping the two files from 12.6s/4.6s to
1.4s/1.0s locally. Five subagent command test files merged into one so
the auto-reply module graph evaluates once instead of five times.
* perf(ci): stripe node test bins by measured file cost
Round-robin striping packed the whale files together, leaving
auto-reply-commands-1 at ~220s while sibling stripes idled at ~30s.
Greedy LPT over advisory per-file seconds hints balances the commands,
cli-runner, and tooling stripes deterministically; group hints for the
restriped bins are equalized at the old totals.
check-prompt-snapshots costs 200-310s per PR regenerating real prompt
stacks, but snapshots only change when the generator's import graph or
its fixtures do. The manifest now computes that reach (same pattern as
QA smoke gating: always-run surface regex incl. the codex extension
whose test API loads via a dynamic module id, plus import-graph walk
from the snapshot helper; deletions and null diffs fail safe to
running). Unaffected PR diffs skip the lane with an explicit log line;
pushes and dispatches always run.
The agents-core graph can stall emitting nothing after warming Vitest's
module cache; a cli-runner stripe sat silent for the full 300s watchdog
window before its retry passed in seconds (run 29499091157, +5min on a
green PR). Give the family the same 60s no-output timeout the gateway
startup-health group already uses for this stall class.
Fleet-contended runners inflate measured group runtimes ~20-25% over
the serial-run hints (bins hit 300-358s on a loaded 12:1xZ run against
a 260s packed estimate). A 220s cap keeps the slowest bin near the
5-minute PR wall-clock budget under load at the cost of two extra
jobs (24 -> 26).
* perf(ci): skip global declaration emission in the CI artifacts build
Declaration emission is ~95% of the tsdown wall clock (measured 132.5s
with dts vs 6.0s without on the main graph; 224.7s in CI). PR CI's dist
consumers only execute runtime JS, so the ciArtifacts profile now skips
global dts and the plugin-sdk export gate validates self-built scoped
declarations (write-plugin-sdk-entry-dts non-canonical mode, 45s local).
The step cache is dropped in self-build mode because its outputs depend
on the SDK source graph the canonical cache inputs do not cover.
Release and docker packaging builds keep canonical declarations.
Local end-to-end: build:ci-artifacts 92s with all gates green.
* fix(ci): pin gateway suites to the proven 2-worker Vitest budget
gateway-core and gateway-methods flaked at 6 workers (hook timeouts,
mock-state res.setHeader TypeError, waitForAssertion misses) on the
first run after the worker-budget raise while passing on two prior
validation runs — the same intermittent contention class media-ui
showed. Pin both like the other timing-sensitive groups.
Validation run 29487321249 measured stripe 2 at 179s (hint said 110)
while stripes 1/3 run light; the index striping concentrates import cost
unevenly. Weight the hints accordingly so no compact bin overpacks past
the 260s target.
* perf(ci): scale Vitest workers with runner class and share serial module cache
Compact node-test jobs pinned OPENCLAW_VITEST_MAX_WORKERS=2 regardless of
runner size, so import-bound suites (30-45s of module-graph import per
file) crawled: runner-cli spent 284s on 5s of tests, commands-1 296s on
10s. With serial plans the budget now scales with the class (6 on 8vcpu,
3 on 4vcpu); timing-sensitive groups (tooling, tui-pty, infra-process)
stay pinned to 2 via plan-level env. Serial bins share one Vitest
fs-module-cache path so later plans reuse the first plan's transforms.
The import-bound commands stripes and security suite move to the 8vcpu
class, and packing hints refresh from serial-run measurements (run
29481835688).
* perf(ci): stripe cli-runner suite and pin media-ui worker budget
The agents-core config runs files serially (fileParallelism false guards
shared module state), so raising the worker budget cannot help its
import-heavy cli-runner suite (~35s module import per file, 213s serial).
Stripe it three ways so bins parallelize the imports instead. media-ui
hosts browser locator tests that timed out at 6 workers on the first
validation run; pin it to the proven 2-worker budget. commands-1's hint
drops to its measured 6-worker runtime.
* refactor(plugin-sdk): narrow wildcard barrels to explicit used exports
* refactor(tools): delete dead tool-planning module exposed by barrel narrowing
* fix(plugin-sdk): restore deprecation tag on OpenClawSchemaType alias
* test(agents): drop test for deleted runtime proxy module
* refactor(tools): trim descriptor types to cache consumers
* refactor(deadcode): harvest exports orphaned by barrel narrowing
* refactor(deadcode): harvest exports orphaned by barrel narrowing (rest)
* fix(agents): restore sdk imports and test markers via public predicate
* fix(plugin-sdk): named type re-exports in plugin-entry; trim types barrel precisely
* chore(plugin-sdk): account unmasked deprecated provider types in budgets
* fix(plugins): name star-only type rows for dts bundling
* fix(plugins): restore host-hook surface; unexport internal api compositions
* fix(plugins): named type imports for api composition; restore needed source exports
* fix(plugins): knip-visible type imports for registry surfaces
* test: adapt tests to privatized media and command internals
* fix(qa-lab): re-export snapshot conversation type
* style: format sessions sdk imports
* fix(plugins): restore smoke entry export; pin budgets to exact actuals
* fix(plugins): canonical smoke-entry import; drop orphaned root shims
* fix(plugins): allowlist manifest probe, repoint qa web import, drop dead browser barrels
* fix(plugin-sdk): pin codex auth marker and scaffold provider type
* fix(qa-lab): keep web-facing model-selection shim within boundary rules
* fix(plugin-sdk): preserve merged contracts through narrowed barrels
* chore(plugin-sdk): pin post-rebase surface budgets
The repository hard-zero sweep (bae9752c5a, #108641) left five CI lanes
red on every PR: a stale max-lines baseline entry for the shrunken
live-cache-regression-runner, session-accessor debt baseline above the
real count, two sqlite-reliability contract types un-exported while
scripts/lib still imports them, a deleted-file entry on the production
lint-suppression allowlist, and a stale wildcard-barrel assertion in the
parallels smoke model test.
