Commit Graph

31 Commits

Author SHA1 Message Date
Peter Steinberger
ea682182d0 fix: isolate npm plugin installs per package (#87647)
* fix: isolate npm plugin installs per package

* test: assert isolated npm plugin projects in upgrade survivor

* test: assert plugin lifecycle npm project roots

* test: resolve npm project deps in live assertions

* fix: resolve codex bins from isolated npm projects

* docs: document isolated npm plugin projects

* ci: configure testbox workflow for crabbox

* fix: stabilize npm project fingerprint

* fix: keep fetch runtime import side-effect free

* test: keep dynamic live model unit hermetic

* ci: handle empty node toolcache roots

* test: make nounset toolcache probe deterministic
2026-05-28 21:16:07 +01:00
Peter Steinberger
1d21224de3 perf: reduce runtime metadata hotpath churn
Reduce runtime metadata hotpath churn by freezing loaded plugin metadata snapshots once and returning the memoized object without clone-on-hit. Reuse persisted package file signatures while preserving realpath containment, cache normalized Jiti alias maps by identity, and defer Discord realtime turn retention/logging until audio starts.

Verification:
- node scripts/run-vitest.mjs src/talk/turn-context-tracker.test.ts src/plugins/plugin-metadata-snapshot.memo.test.ts src/plugins/manifest-registry-installed.test.ts src/plugins/sdk-alias.test.ts src/plugins/installed-plugin-index-records.test.ts
- node scripts/run-vitest.mjs src/plugins/plugin-metadata-snapshot.memo.test.ts
- pnpm test extensions/discord/src/voice/manager.e2e.test.ts --testNamePattern "keeps realtime playback alive|interrupts realtime playback|does not interrupt realtime provider state"
- pnpm lint --threads=8
- pnpm exec oxfmt --check src/plugins/plugin-metadata-snapshot.ts src/plugins/plugin-metadata-snapshot.memo.test.ts src/plugins/manifest-registry-installed.ts src/plugins/installed-plugin-index-record-builder.ts src/plugins/sdk-alias.ts extensions/discord/src/voice/realtime.ts
- pnpm tsgo:core
- pnpm tsgo:extensions
- pnpm build
- autoreview --mode commit --commit HEAD
- PR CI green on head 7dd3e44a78
2026-05-25 23:59:45 +01:00
Peter Steinberger
1531fe2525 perf: reduce runtime cache churn 2026-05-25 23:35:06 +01:00
Peter Steinberger
77d9ac30bb refactor: reuse shared coercion helpers (#86419)
* refactor: share talk event metric extraction

* refactor: reuse shared coercion helpers

* refactor: reuse shared primitive guards

* refactor: reuse shared record guard

* refactor: reuse shared primitive helpers

* refactor: reuse shared string guards

* refactor: reuse shared non-empty string guard

* refactor: share plugin primitive coercion helpers

* refactor: reuse plugin coercion helpers

* refactor: reuse plugin coercion helpers in more plugins

* refactor: reuse channel coercion helpers

* refactor: reuse monitor coercion helpers

* refactor: reuse provider coercion helpers

* refactor: reuse core coercion helpers

* refactor: reuse runtime coercion helpers

* refactor: reuse helper coercion in codex paths

* refactor: reuse helper coercion in runtime paths

* refactor: reuse codex app-server coercion helpers

* refactor: reuse codex record helpers

* refactor: reuse migration and qa record helpers

* refactor: reuse feishu and core helper guards

* refactor: reuse browser and policy coercion helpers

* refactor: reuse memory wiki record helper

* refactor: share boolean coercion helpers

* refactor: reuse finite number coercion

* refactor: reuse trimmed string list helpers

* refactor: reuse string list normalization

* refactor: reuse remaining string list helpers

* refactor: reuse string entry normalizer

* refactor: share sorted string helpers

* refactor: share string list normalization

* test: preserve command registry browser imports

* refactor: reuse trimmed list helpers

* refactor: reuse string dedupe helpers

* refactor: reuse local dedupe helpers

* refactor: reuse more string dedupe helpers

* refactor: reuse command string dedupe helpers

* refactor: dedupe memory path lists with helper

* refactor: expose string dedupe helpers to plugins

* refactor: reuse core string dedupe helpers

* refactor: reuse shared unique value helpers

* refactor: reuse unique helpers in agent utilities

* refactor: reuse unique helpers in config plumbing

* refactor: reuse unique helpers in extensions

* refactor: reuse unique helpers in core utilities

* refactor: reuse unique helpers in qa plugins

* refactor: reuse unique helpers in memory plugins

* refactor: reuse unique helpers in channel plugins

* refactor: reuse unique helpers in core tails

* refactor: reuse unique helper in comfy workflow

* refactor: reuse unique helpers in test utilities

* refactor: expose unique value helper to plugins

* refactor: reuse unique helpers for numeric lists

* refactor: replace index dedupe filters

* refactor: reuse string entry normalization

* refactor: reuse string normalization in plugin helpers

* refactor: reuse string normalization in extension helpers

* refactor: reuse string normalization in channel parsers

* refactor: reuse string normalization in memory search

* refactor: reuse string normalization in provider parsers

* refactor: reuse string normalization in qa helpers

* refactor: reuse string normalization in infra parsers

* refactor: reuse string normalization in messaging parsers

* refactor: reuse string normalization in core parsers

* refactor: reuse string normalization in extension parsers

* refactor: reuse string normalization in remaining parsers

* refactor: reuse string normalization in final parser spots

* refactor: reuse string normalization in qa media helpers

* refactor: reuse normalization in provider and media lists

* refactor: reuse normalization for remaining set filters

* refactor: reuse normalization in policy allowlists

* refactor: reuse normalization in session and owner lists

* refactor: centralize primitive string lists

* refactor: reuse lowercase entry helpers

* refactor: reuse sorted string helpers

* refactor: reuse unique trimmed helpers

* refactor: reuse string normalization helpers

* refactor: reuse catalog string helpers

* refactor: reuse remaining string helpers

* refactor: simplify remaining list normalization

* refactor: reuse codex auth order normalization

* chore: refresh plugin sdk api baseline

* fix: make shared string sorting deterministic

* chore: refresh plugin sdk api baseline

* fix: align host env security ordering
2026-05-25 21:20:41 +01:00
Peter Steinberger
313762282c fix(plugins): only memoize complete metadata snapshots 2026-05-25 17:29:46 +01:00
Peter Steinberger
159e4406ab perf(plugins): reuse derived metadata snapshots 2026-05-25 17:20:39 +01:00
Peter Steinberger
e7c696a5b0 fix(plugins): clear metadata memo at lifecycle boundaries 2026-05-25 10:39:42 +01:00
Dallin Romney
8ccb11cbfc perf(plugins,gateway): thread metadata snapshot + discovery through hot paths + plugin owner fixes (#84649)
* perf(plugins): thread metadata snapshot and discovery through hot paths

With the snapshot memo now actually hitting, route the snapshot's
manifestRegistry and discovery through the helper chains that already
had fast paths for them. Eliminates redundant per-call rebuilds at
two big amplifiers.

- Provider resolve paths (resolvePluginProviders /
  isPluginProvidersLoadInFlight / resolveOwningPluginIdsForProvider /
  resolveExternalAuthProfilesWithPlugins) self-service a snapshot once
  at the public entry, then thread it as a separate required arg
  through resolvePluginProviderLoadBase,
  resolveExplicitProviderOwnerPluginIds, and the setup/runtime load
  state helpers. Inner reads change from
  'params.pluginMetadataSnapshot?.x' to 'snapshot.x', no more
  enrichedParams clone. loadPluginManifestRegistryForInstalledIndex
  fires drop ~685 -> ~10 per cold start.

- Bundled-channel / auto-enable chain accepts an optional
  PluginDiscoveryResult. discoverOpenClawPlugins is fired once during
  snapshot building (resolveInstalledPluginIndexRegistry already
  produced it internally; now bubbled up through
  loadInstalledPluginIndexWithDiscovery, PluginRegistrySnapshotResult,
  and onto PluginMetadataSnapshot.discovery). load-context reads
  metadataSnapshot.discovery and passes it through
  applyPluginAutoEnable, so the bundled-channel cascade
  (collectConfiguredChannelIds, listBundledChannelIdsWith*,
  listPotentialConfiguredChannelPresenceSignals) short-circuits
  instead of each leaf re-firing discovery. Persisted-cache path is
  unchanged: no discovery on the snapshot, downstream chain handles
  its own fallback (pre-PR behavior on that path).

* test(plugins): isolate snapshot memo across tests that mock manifest registry

The snapshot memo is now process-scoped and effective (~98% hit rate).
Three test files were depending on cache misses (because the broken
cache returned them) — each test would set up its own
loadPluginManifestRegistry mock and expect a fresh derive. With the
cache fixed, an earlier test's mocked registry now leaks into later
tests in the same file.

- io.write-config.test.ts: afterEach now clears the snapshot memo so
  the 'demo' plugin mocked in the first test does not survive into
  'keeps shipped plugin install config records when index migration
  fails', which expects an empty registry to surface the 'plugin not
  found: demo' warning.

- gateway/model-pricing-cache.ts: resetGatewayModelPricingCacheForTest
  also clears the memo. Tests in model-pricing-cache.test.ts assert
  loadPluginManifestRegistryForInstalledIndex was called; the memo
  hit otherwise skips the call.

- providers.test.ts: vi.doMock loadPluginMetadataSnapshot to wrap the
  existing loadPluginManifestRegistryMock fixture. The plumbing
  commit added an auto-fetch fall-through in
  resolveOwningPluginIdsForProvider; without the mock, providers
  tests hit real disk reads and return empty registries (which is
  what surfaced as 9 unrelated-looking failures in the prior CI
  run).

* fix(plugins): preserve setup.cliBackends owner matching in provider scan

resolveOwningPluginIdsForProvider now also checks plugin.setup?.cliBackends.
The pre-PR no-registry fallback used resolvePluginContributionOwners which
includes both top-level cliBackends and setup.cliBackends; the PR's manifest
scan replacement was missing the setup case.

* fix(plugins): inherit active registry workspaceDir before loading metadata snapshot

isPluginProvidersLoadInFlight and resolvePluginProviders now resolve
env and workspaceDir once at the entry point (falling back to
getActivePluginRegistryWorkspaceDir) and pass them into both
loadPluginMetadataSnapshot and resolvePluginProviderLoadBase. Pre-fix
the snapshot used params.workspaceDir raw while the load base inherited
the active workspace, so workspace-scoped provider plugins could be
absent from the snapshot manifest registry even though owner resolution
expected them.

Regression test asserts the snapshot mock receives the active
workspaceDir when the caller omits it.

* perf(gateway): thread discovery into applyPluginAutoEnable call sites

Every gateway applyPluginAutoEnable call now passes the snapshot's
PluginDiscoveryResult so the bundled-channel cascade (collectConfiguredChannelIds
→ listBundledChannelIdsWith* → listPotentialConfiguredChannelPresenceSignals)
short-circuits instead of each leaf re-firing discovery.

Startup-time sites pull discovery from the snapshot/lookup-table they already
hold:
- server-plugin-bootstrap.ts (pluginLookUpTable)
- server-startup-plugins.ts (pluginMetadataSnapshot)
- server-startup-config.ts (pluginMetadataSnapshot)
- server-plugins.ts (pluginLookUpTable, both call sites)

Per-RPC sites (server.impl getRuntimeConfig callback, server-methods/channels
status + start handlers, server-methods/send) source discovery via
getCurrentPluginMetadataSnapshot using the runtime config to validate
compatibility. Falls through to the original slow path when the snapshot is
absent or incompatible.
2026-05-24 13:44:03 -07:00
Peter Steinberger
4314674054 perf: reuse plugin metadata snapshots (#85843)
* perf: reuse plugin metadata snapshots

* test: update plugin metadata snapshot mocks
2026-05-23 23:34:19 +01:00
Vincent Koc
bf64de9191 fix(plugins): keep derived metadata snapshots fresh
Keep derived plugin metadata snapshots out of the process memo/current snapshot cache so newly added plugins under derived discovery paths are found without restart.
2026-05-22 17:00:09 +08:00
Kaspre
5734193fdf fix(plugins): keep metadata snapshot memo fresh
* fix(plugins): keep metadata memo freshness

* fix(plugins): keep metadata memo freshness

* fix(plugins): resolve metadata memo review gaps

* fix(plugins): scope metadata memo watches to env

* fix(plugins): tighten metadata memo fingerprint return type

`resolvePersistedRegistryFastMemoFingerprint` was annotated `: unknown`
but always returns object literals (`{ disabled: true }` or
`{ index, npmPackageJson }`). Spreading the unknown-typed result on
line 478 (`...fastFingerprint`) was rejected by tsgo with TS2698, which
cascaded across every check that runs the project compile (build,
tsgo:prod, check:test-types, lint, all node test shards).

Tighten the return type to `Record<string, unknown>` to match the
function's actual return shapes and unblock the spread.

* test(gateway): tolerate ENOENT in sessions.list spy predicate

The `sessions.list configuredAgentsOnly hides disk-discovered
unregistered agent stores` test spies on `fsSync.readFileSync` and
predicates with `fsSync.realpathSync.native(file) === realDiskOnlyStorePath`
for every captured read. The native realpath call throws on missing
files, so any new readFileSync of a path that may not exist (e.g. the
persisted plugin install records probe added in this PR) crashes the
predicate before the assertion runs.

Wrap the predicate in ENOENT tolerance so the test stays robust against
any future readFileSync of files that may not exist on disk.

* fix(plugins): refresh memo from cached registry

* fix(plugins): use high resolution memo fingerprints

* test(plugins): stabilize memo freshness regression

* test(cli): satisfy config mutation mock hash contract

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-05-15 07:12:31 +01:00
Kaspre
f71df80522 perf(plugins): memoize metadata snapshots narrowly 2026-05-14 16:15:50 +08:00
Shakker
c795a1a8ef fix: propagate diagnostics timeline phase 2026-05-06 17:33:46 +01:00
Peter Steinberger
2294f5c95a fix: typecheck sparse plugin metadata handling 2026-05-02 08:21:25 +01:00
Peter Steinberger
c58319ff50 fix: tolerate sparse plugin metadata snapshots 2026-05-02 08:19:40 +01:00
Peter Steinberger
ebb45a8a28 refactor: unify plugin metadata snapshot callers 2026-05-02 07:51:17 +01:00
Peter Steinberger
eaf1f53d60 fix: stabilize plugin metadata release checks 2026-05-02 07:27:27 +01:00
Peter Steinberger
71da5af164 refactor: reuse plugin metadata snapshots 2026-05-02 07:13:17 +01:00
Peter Steinberger
dc848c94b8 refactor: hide plugin helper internals 2026-05-02 07:04:54 +01:00
Peter Steinberger
fecac7e40a refactor: unify plugin startup metadata planning 2026-05-02 06:36:03 +01:00
Peter Steinberger
9989512a37 refactor: simplify plugin cache boundaries 2026-05-02 06:00:53 +01:00
Peter Steinberger
f11046e0bf refactor: unify plugin control-plane cache context 2026-05-02 04:10:49 +01:00
Peter Steinberger
97a34e0f50 fix: tighten plugin metadata cache invalidation 2026-05-02 03:42:39 +01:00
Shakker
e69da9d578 fix: honor config timeline diagnostics 2026-04-29 19:53:55 +01:00
Shakker
097eed8cd8 feat: emit diagnostics timeline 2026-04-29 19:53:55 +01:00
Peter Steinberger
147752ecc3 refactor: split plugin metadata normalizer 2026-04-27 18:27:29 +01:00
Shakker
6e893eaee4 refactor: expose model catalog aliases in plugin lookup 2026-04-27 17:13:11 +01:00
Shakker
94591c3cb3 fix: fingerprint plugin metadata index reuse 2026-04-27 17:02:57 +01:00
Shakker
58b4407cda fix: reject stale plugin metadata inventory 2026-04-27 17:02:57 +01:00
Shakker
ab28cfa9d4 fix: guard plugin metadata snapshot reuse 2026-04-27 17:02:57 +01:00
Shakker
440fc73448 refactor: extract plugin metadata snapshot 2026-04-27 17:02:57 +01:00