From 720ab99307ff69a253cf070f0b2d499452bed8c0 Mon Sep 17 00:00:00 2001 From: Peter Steinberger Date: Mon, 27 Apr 2026 05:07:16 +0100 Subject: [PATCH] docs: explain release validation entrypoints --- docs/ci.md | 108 ++++++++++++++++++++++++++++++++++++ docs/reference/RELEASING.md | 73 ++++++++++++++++++------ 2 files changed, 164 insertions(+), 17 deletions(-) diff --git a/docs/ci.md b/docs/ci.md index 72dfa26d556..67bc394173f 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -26,6 +26,114 @@ workflow checkout. Profiles cover smoke, package, product, full, and custom Docker lane selections. The optional Telegram lane is published-npm only and reuses the `NPM Telegram Beta E2E` workflow. +## Package Acceptance + +Use `Package Acceptance` when the question is "does this installable OpenClaw +package work as a product?" It is different from normal CI: normal CI validates +the source tree, while package acceptance validates a single tarball through the +same Docker E2E harness users exercise after install or update. + +The workflow has four jobs: + +1. `resolve_package` checks out `workflow_ref`, resolves one package candidate, + writes `.artifacts/docker-e2e-package/openclaw-current.tgz`, writes + `.artifacts/docker-e2e-package/package-candidate.json`, uploads both as the + `package-under-test` artifact, and prints the source, workflow ref, package + ref, version, SHA-256, and profile in the GitHub step summary. +2. `docker_acceptance` calls + `openclaw-live-and-e2e-checks-reusable.yml` with `ref=workflow_ref` and + `package_artifact_name=package-under-test`. The reusable workflow downloads + that artifact, validates the tarball inventory, prepares package-digest + Docker images when needed, and runs the selected Docker lanes against that + package instead of packing the workflow checkout. +3. `npm_telegram` optionally calls `NPM Telegram Beta E2E`. It runs only when + `telegram_mode` is not `none`, and only for `source=npm`, because that lane + installs a published package spec. +4. `summary` fails the workflow if package resolution, Docker acceptance, or + the optional Telegram lane failed. + +Candidate sources: + +- `source=npm`: accepts only `openclaw@beta`, `openclaw@latest`, or an exact + OpenClaw release version such as `openclaw@2026.4.27-beta.2`. Use this for + published beta/stable acceptance. +- `source=ref`: packs a trusted `package_ref` branch, tag, or full commit SHA. + The resolver fetches OpenClaw branches/tags, verifies the selected commit is + reachable from repository branch history or a release tag, installs deps in a + detached worktree, and packs it with `scripts/package-openclaw-for-docker.mjs`. +- `source=url`: downloads an HTTPS `.tgz`; `package_sha256` is required. +- `source=artifact`: downloads one `.tgz` from `artifact_run_id` and + `artifact_name`; `package_sha256` is optional but should be supplied for + externally shared artifacts. + +Keep `workflow_ref` and `package_ref` separate. `workflow_ref` is the trusted +workflow/harness code that runs the test. `package_ref` is the source commit +that gets packed when `source=ref`. This lets the current test harness validate +older trusted source commits without running old workflow logic. + +Profiles map to Docker coverage: + +- `smoke`: `npm-onboard-channel-agent`, `gateway-network`, `config-reload` +- `package`: `install-e2e`, `npm-onboard-channel-agent`, `doctor-switch`, + `update-channel-switch`, `bundled-channel-deps`, `plugins`, `plugin-update` +- `product`: `package` plus `mcp-channels`, `cron-mcp-cleanup`, + `openai-web-search-minimal`, `openwebui` +- `full`: full Docker release-path chunks with OpenWebUI +- `custom`: exact `docker_lanes`; required when `suite_profile=custom` + +Release checks call Package Acceptance with `source=ref`, +`package_ref=`, `workflow_ref=`, and +`suite_profile=package`. That profile is the GitHub-native replacement for most +Parallels package/update validation. Cross-OS release checks still cover +OS-specific onboarding, installer, and platform behavior; package/update +product validation should start with Package Acceptance. + +Examples: + +```bash +# Validate the current beta package with product-level coverage. +gh workflow run package-acceptance.yml \ + --ref main \ + -f workflow_ref=main \ + -f source=npm \ + -f package_spec=openclaw@beta \ + -f suite_profile=product + +# Pack and validate a release branch with the current harness. +gh workflow run package-acceptance.yml \ + --ref main \ + -f workflow_ref=main \ + -f source=ref \ + -f package_ref=release/YYYY.M.D \ + -f suite_profile=package + +# Validate a tarball URL. SHA-256 is mandatory for source=url. +gh workflow run package-acceptance.yml \ + --ref main \ + -f workflow_ref=main \ + -f source=url \ + -f package_url=https://example.com/openclaw-current.tgz \ + -f package_sha256=<64-char-sha256> \ + -f suite_profile=smoke + +# Reuse a tarball uploaded by another Actions run. +gh workflow run package-acceptance.yml \ + --ref main \ + -f workflow_ref=main \ + -f source=artifact \ + -f artifact_run_id= \ + -f artifact_name=package-under-test \ + -f suite_profile=custom \ + -f docker_lanes='install-e2e plugin-update' +``` + +When debugging a failed package acceptance run, start at the `resolve_package` +summary to confirm the package source, version, and SHA-256. Then inspect the +`docker_acceptance` child run and its Docker artifacts: +`.artifacts/docker-tests/**/summary.json`, `failures.json`, lane logs, phase +timings, and rerun commands. Prefer rerunning the failed package profile or +exact Docker lanes instead of rerunning full release validation. + QA Lab has dedicated CI lanes outside the main smart-scoped workflow. The `Parity gate` workflow runs on matching PR changes and manual dispatch; it builds the private QA runtime and compares the mock GPT-5.5 and Opus 4.6 diff --git a/docs/reference/RELEASING.md b/docs/reference/RELEASING.md index dd8bf68a4c6..8eb149dea6b 100644 --- a/docs/reference/RELEASING.md +++ b/docs/reference/RELEASING.md @@ -66,9 +66,9 @@ the maintainer-only release runbook. 6. Run `OpenClaw NPM Release` with `preflight_only=true`. Before a tag exists, a full 40-character release-branch SHA is allowed for validation-only preflight. Save the successful `preflight_run_id`. -7. Run `Full Release Validation` for the release branch, tag, or full commit - SHA. This is the umbrella run for the four big release test boxes: Vitest, - Docker, QA Lab, and Package. +7. Kick off all pre-release tests with `Full Release Validation` for the + release branch, tag, or full commit SHA. This is the one manual entrypoint + for the four big release test boxes: Vitest, Docker, QA Lab, and Package. 8. If validation fails, fix on the release branch and rerun the smallest failed file, lane, workflow job, package profile, provider, or model allowlist that proves the fix. Rerun the full umbrella only when the changed surface makes @@ -96,15 +96,14 @@ the maintainer-only release runbook. - Run `pnpm build && pnpm ui:build` before `pnpm release:check` so the expected `dist/*` release artifacts and Control UI bundle exist for the pack validation step -- Run the manual `Full Release Validation` workflow before release approval - when you need the whole release validation suite from one entrypoint. It - accepts a branch, tag, or full commit SHA, dispatches manual `CI`, and - dispatches `OpenClaw Release Checks` for install smoke, package acceptance, - Docker release-path suites, live/E2E, OpenWebUI, QA Lab parity, Matrix, and - Telegram lanes. - Provide `npm_telegram_package_spec` only after a package has been published - and the post-publish Telegram E2E should run too. - Example: `gh workflow run full-release-validation.yml --ref main -f ref=release/YYYY.M.D` +- Run the manual `Full Release Validation` workflow before release approval to + kick off all pre-release test boxes from one entrypoint. It accepts a branch, + tag, or full commit SHA, dispatches manual `CI`, and dispatches + `OpenClaw Release Checks` for install smoke, package acceptance, Docker + release-path suites, live/E2E, OpenWebUI, QA Lab parity, Matrix, and Telegram + lanes. Provide `npm_telegram_package_spec` only after a package has been + published and the post-publish Telegram E2E should run too. Example: + `gh workflow run full-release-validation.yml --ref main -f ref=release/YYYY.M.D` - Run the manual `Package Acceptance` workflow when you want side-channel proof for a package candidate while release work continues. Use `source=npm` for `openclaw@beta`, `openclaw@latest`, or an exact release version; `source=ref` @@ -221,8 +220,9 @@ Validation` or from the `main`/release workflow ref so workflow logic and ## Release test boxes -`Full Release Validation` is the manual umbrella that operators use when they -want all release validation from one entrypoint: +`Full Release Validation` is how operators kick off all pre-release tests from +one entrypoint. Run it from the trusted `main` workflow ref and pass the release +branch, tag, or full commit SHA as `ref`: ```bash gh workflow run full-release-validation.yml \ @@ -236,9 +236,48 @@ gh workflow run full-release-validation.yml \ The workflow resolves the target ref, dispatches manual `CI` with `target_ref=`, dispatches `OpenClaw Release Checks`, and optionally dispatches post-publish Telegram E2E when -`npm_telegram_package_spec` is set. A full run is only acceptable when both -child workflows succeed or an intentionally skipped optional child is recorded -in the summary. +`npm_telegram_package_spec` is set. `OpenClaw Release Checks` then fans out +install smoke, cross-OS release checks, live/E2E Docker release-path coverage, +Package Acceptance, QA Lab parity, live Matrix, and live Telegram. A full run is +only acceptable when the `Full Release Validation` summary shows `normal_ci` and +`release_checks` as successful, and any optional `npm_telegram` child is either +successful or intentionally skipped. + +Use these variants depending on release stage: + +```bash +# Validate an unpublished release candidate branch. +gh workflow run full-release-validation.yml \ + --ref main \ + -f ref=release/YYYY.M.D \ + -f workflow_ref=main \ + -f provider=openai \ + -f mode=both + +# Validate an exact pushed commit. +gh workflow run full-release-validation.yml \ + --ref main \ + -f ref=<40-char-sha> \ + -f workflow_ref=main \ + -f provider=openai \ + -f mode=both + +# After publishing a beta, add published-package Telegram E2E. +gh workflow run full-release-validation.yml \ + --ref main \ + -f ref=release/YYYY.M.D \ + -f workflow_ref=main \ + -f provider=openai \ + -f mode=both \ + -f npm_telegram_package_spec=openclaw@YYYY.M.D-beta.N \ + -f npm_telegram_provider_mode=mock-openai +``` + +Do not use the full umbrella as the first rerun after a focused fix. If one box +fails, use the failed child workflow, job, Docker lane, package profile, model +provider, or QA lane for the next proof. Run the full umbrella again only when +the fix changed shared release orchestration or made earlier all-box evidence +stale. ### Vitest