docs(skills): add plugin pre-release test plan

2026-05-06 07:50:43 +00:00 · 2026-04-28 02:39:42 -07:00
parent cfbf4d1fa4
commit 17811480da
2 changed files with 238 additions and 0 deletions
--- a/.agents/skills/openclaw-pre-release-plugin-testing/SKILL.md
+++ b/.agents/skills/openclaw-pre-release-plugin-testing/SKILL.md
@@ -0,0 +1,234 @@
+---
+name: openclaw-pre-release-plugin-testing
+description: Plan and run pre-release OpenClaw plugin validation across bundled plugins, package artifacts, lifecycle commands, doctor/fix, config round-trip, gateway startup, SDK compatibility, Docker E2E, Package Acceptance, and Testbox proof.
+---
+
+# OpenClaw Pre-Release Plugin Testing
+
+Use this skill when Vincent asks for plugin release confidence, plugin lifecycle
+sweeps, package-artifact plugin proof, or "what else should we test before
+release?" It complements `openclaw-testing`; use that skill too when choosing
+the cheapest safe runner or debugging a failing lane.
+
+## Goal
+
+Prove the plugin system as a product surface, not just as source tests:
+
+- bundled plugin lifecycle: install, inspect, enable, disable, uninstall
+- package artifact behavior from a clean `HOME`
+- doctor/fix/config validation and idempotence
+- config discovery and config round-trip
+- status/log visibility and diagnostics
+- gateway startup/bootstrap with plugin metadata snapshots
+- public SDK compatibility for real external plugins
+- live-ish provider/channel probes only when safe credentials exist
+
+## First Checks
+
+From the OpenClaw repo root:
+
+```bash
+pnpm docs:list
+git status --short --branch
+readlink node_modules
+pnpm changed:lanes --json
+```
+
+In Codex worktrees under `.codex/worktrees`, `node_modules` must be a symlink to
+the main OpenClaw checkout. Do not run `pnpm install` there. For broad or
+package-heavy proof, use Blacksmith Testbox or GitHub Actions.
+
+## Runner Choice
+
+Prefer this order:
+
+1. **GitHub Package Acceptance** for installable-package product proof.
+2. **`ci-build-artifacts-testbox.yml` Testbox** when Docker/package lanes need
+   seeded `dist`, `dist-runtime`, and package caches.
+3. **`ci-check-testbox.yml` Testbox** for source checks, targeted Vitest,
+   package-boundary checks, or focused Docker lanes.
+4. **Local targeted commands only** for small format/static/unit probes.
+
+Avoid long package Docker runs from a stale sparse worktree. If Testbox sync
+reports hundreds of changed files or starts deleting package inputs, stop and
+warm a fresh box from current `main`, or switch to Package Acceptance.
+
+## Existing Baseline
+
+Run or verify these before inventing new coverage:
+
+```bash
+OPENCLAW_TESTBOX=1 pnpm check:changed
+pnpm run test:extensions:package-boundary:canary
+pnpm run test:extensions:package-boundary:compile
+pnpm test:docker:plugins
+OPENCLAW_PLUGINS_E2E_CLAWHUB=0 pnpm test:docker:plugins
+pnpm test:docker:plugin-update
+pnpm test:docker:bundled-channel-deps:fast
+```
+
+For full bundled install/uninstall proof, shard the packaged sweep:
+
+```bash
+OPENCLAW_BUNDLED_PLUGIN_SWEEP_TOTAL=8 \
+OPENCLAW_BUNDLED_PLUGIN_SWEEP_INDEX=<0-7> \
+pnpm test:docker:bundled-plugin-install-uninstall
+```
+
+Expected current packaged scope: 116 public bundled plugins over shards `0-7`.
+Private QA plugins are source-mode only unless a package explicitly includes
+them.
+
+## Confidence Matrix
+
+Use this matrix for pre-release signoff. Record pass/fail, run URL/Testbox ID,
+package SHA/version, and skipped-live reason.
+
+| Surface | Proof | Preferred runner |
+| --- | --- | --- |
+| Package artifact | Package Acceptance `suite_profile=package` or custom lanes | GitHub Actions |
+| Bundled lifecycle | 8-shard `test:docker:bundled-plugin-install-uninstall` | Testbox or release Docker |
+| External plugins | `test:docker:plugins` and `plugins-offline` | Testbox/package acceptance |
+| Update no-op | `test:docker:plugin-update` | Testbox/package acceptance |
+| Channel runtime deps | `test:docker:bundled-channel-deps:fast` plus key channels | Testbox/package acceptance |
+| Doctor/fix | seeded bad configs + `doctor --fix --non-interactive` | new Docker/Testbox harness |
+| Config round-trip | `config set/get`, inspect, doctor, reload, diff hash | new Docker/Testbox harness |
+| Gateway bootstrap | clean `HOME`, plugin groups enabled/disabled, status JSON | new Docker/Testbox harness |
+| SDK compatibility | directory, tgz, and `file:` external plugins using SDK subpaths | `test:docker:plugins` plus new smoke |
+| Live-ish | redacted provider/channel probes only for present env | Testbox live lanes |
+
+## Package Acceptance Plan
+
+Use this when validating a release branch, beta, or candidate package:
+
+```bash
+gh workflow run package-acceptance.yml \
+  --repo openclaw/openclaw \
+  --ref main \
+  -f workflow_ref=main \
+  -f source=ref \
+  -f package_ref=<branch-or-sha> \
+  -f suite_profile=custom \
+  -f docker_lanes='plugins-offline plugin-update bundled-channel-deps-compat doctor-switch update-channel-switch config-reload mcp-channels npm-onboard-channel-agent' \
+  -f telegram_mode=mock-openai
+```
+
+Use `source=npm -f package_spec=openclaw@beta` for published beta proof. Keep
+`workflow_ref` as trusted current harness code unless the release process says
+otherwise.
+
+## New Testbox Harness Plan
+
+If more certainty is needed, add or run a `plugin-lifecycle-matrix` Docker lane
+that uses one package tarball and sharded plugin lists. Per plugin:
+
+1. Start with a clean `HOME`.
+2. Capture `plugins list --json`.
+3. `plugins install <id>`.
+4. `plugins inspect <id> --json`.
+5. `plugins disable <id>`, then assert disabled visibility.
+6. `plugins enable <id>`, except config-required plugins without config.
+7. `plugins registry --refresh`.
+8. `doctor --non-interactive`.
+9. `plugins uninstall <id> --force`.
+10. Assert no config entry, allow/deny residue, install record, managed dir, or
+    bundled `dist/extensions/...` load path remains.
+11. Assert diagnostics contain no `level: "error"` and output redacts
+    secret-looking values.
+
+Keep `memory-lancedb` special: it is config-required. First assert install does
+not enable it without embedding config, then run a second configured case.
+
+## Doctor/Fix Matrix
+
+Seed bad states and require `doctor --fix --non-interactive` to repair them,
+then run doctor again and require idempotence:
+
+- stale `plugins.allow`
+- stale `plugins.entries`
+- stale channel config for missing channel plugin
+- invalid `plugins.entries.<id>.config`
+- packaged bundled path in `plugins.load.paths`
+- legacy `plugins.installs`
+- disabled channel/plugin config that must not stage runtime deps
+- root-owned global package tree that must remain unmodified
+
+## Gateway Bootstrap Matrix
+
+Start packaged OpenClaw in Docker with clean state:
+
+- provider plugins enabled, no credentials: ready with warnings, no crash
+- channel plugins configured disabled: no runtime deps staged
+- startup-activation plugins enabled: ready and reflected in status
+- invalid single plugin config: bad plugin skipped/quarantined, others remain
+
+Assert:
+
+- gateway reaches ready
+- `openclaw status --json` includes plugin diagnostics
+- `openclaw plugins inspect --all --json` is parseable
+- package tree is not mutated
+- logs contain no raw tokens
+
+## Config Round-Trip Representatives
+
+Use representative plugin families instead of every plugin for deep config
+round-trip:
+
+- providers: `openai`, `anthropic`, `mistral`, `openrouter`
+- channels: `telegram`, `discord`, `slack`, `whatsapp`
+- memory: `memory-lancedb`
+- feature/runtime: `browser`, `acpx`, `tokenjuice`
+
+For each representative:
+
+1. Write config through CLI when possible.
+2. Read it back through `config get` or JSON.
+3. Run `plugins inspect`.
+4. Run `doctor --non-interactive`.
+5. Trigger gateway config reload if applicable.
+6. Compare config hash before/after no-op commands.
+
+## External SDK Smoke
+
+In a package Docker lane, create tiny external plugins and install them from:
+
+- local directory
+- `.tgz`
+- `file:` npm spec
+
+Cover CJS and ESM shapes, plus at least one plugin importing focused
+`openclaw/plugin-sdk/*` subpaths. Assert `plugins inspect` sees its tool,
+gateway method, CLI command, or service.
+
+## Live-Ish Probe Rules
+
+Before live-ish work, source allowed env in Testbox and generate a redacted
+availability matrix: present/missing only, never values.
+
+Only run probes for credentials that exist. Prefer auth/catalog/status probes
+over sending user-visible messages. If a probe might contact an external user,
+channel, or workspace, stop and ask Vincent.
+
+## Reporting
+
+Report in this shape:
+
+```text
+package/ref:
+tbx ids / run urls:
+matrix:
+  bundled lifecycle:
+  package acceptance:
+  doctor/fix:
+  gateway bootstrap:
+  config round-trip:
+  sdk external:
+  live-ish:
+failures:
+skips:
+next highest-value gap:
+```
+
+Say clearly when a failure is Testbox sync/env damage rather than product
+behavior, and prove that with a clean rerun or current-main comparison.
--- a/.agents/skills/openclaw-pre-release-plugin-testing/agents/openai.yaml
+++ b/.agents/skills/openclaw-pre-release-plugin-testing/agents/openai.yaml
@@ -0,0 +1,4 @@
+interface:
+  display_name: "OpenClaw Plugin Pre-Release Testing"
+  short_description: "Plan plugin release validation"
+  default_prompt: "Use $openclaw-pre-release-plugin-testing to plan or run pre-release OpenClaw plugin validation across package, lifecycle, doctor, gateway, SDK, and live-ish proof."