improve(auto-qa): coordinate live root-cause testing campaigns (#113727)

* refactor(skills): require root-cause fixes in auto qa campaigns

* improve(auto-qa): coordinate current-main root-cause campaigns

---------

Co-authored-by: Peter Steinberger <steipete@golden-gate.local>

.agents/skills/auto-qa/SKILL.md

@@ -5,7 +5,7 @@ description: "Continuously audit, live-test, and stress-test the current OpenCla
 
 # OpenClaw Auto QA
 
-Run a continuous, current-`main` OpenClaw product campaign. Treat a reviewer finding as a hypothesis, a passing test as evidence only for its actual head, and a merge as complete only when the canonical repository confirms it.
+Run a continuous, current-`main` OpenClaw product campaign. Treat a reviewer finding as a hypothesis, a passing test as evidence only for its actual head, and a merge as complete only when the canonical repository confirms it. Repair the actual root cause in its canonical owner; a smaller patch is not better if it leaves sibling paths, lifecycle invariants, or the defective abstraction intact.
 
 ## Start with the moving source
 
@@ -17,9 +17,9 @@ Run a continuous, current-`main` OpenClaw product campaign. Treat a reviewer fin
    - `$crabbox` for remote, Docker, packaged, cross-platform, and live proof.
    - `$autoreview` for fresh independent review before publishing or landing.
    - `$openclaw-pr-maintainer` for authorized maintainer-side GitHub actions.
-2. Check `git status -sb`. When network access is authorized, refresh with `git fetch origin main`; record the full `origin/main` SHA and inspect the current source, package scripts, scenario inventory, workflows, and scoped guides at that SHA. For an explicitly offline or read-only assignment, record the existing remote-tracking SHA and disclose that remote freshness is unverified; never fetch or contact a provider without authorization. Never pull, rebase, or switch a checkout that another agent is using.
-3. Make one requested, fresh `codex/` worktree per implementation task. Keep reviewer workers read-only. When network access is authorized, refresh `origin/main` periodically and recreate or revalidate candidates against the new head. Otherwise revalidate against the locally recorded remote-tracking SHA and explicitly report that remote freshness is unverified.
-4. Initialize or resume the user-requested Markdown report and numbered ledger. Read [references/campaign-evidence.md](references/campaign-evidence.md) and [references/evidence-ledger.md](references/evidence-ledger.md) before accepting a finding.
+2. Check `git status -sb`. When network access is authorized, let the orchestrator refresh with `git fetch origin main`; record the full `origin/main` SHA and inspect the current source, package scripts, scenario inventory, workflows, and scoped guides at that SHA. For an explicitly offline or read-only assignment, record the existing remote-tracking SHA and disclose that remote freshness is unverified; never fetch or contact a provider without authorization. Never pull, rebase, or switch a checkout that another agent is using.
+3. Make one requested, fresh `codex/` worktree per implementation task. Keep reviewer workers read-only. Serialize all shared `origin/main` refreshes through the orchestrator; pause worker fetches before repository-native PR review, preparation, or landing. After a successful landing, fetch `origin/main` again, verify the recorded merge commit is an ancestor of that fetched ref, and only then broadcast its full immutable SHA and resume workers. Revalidate candidates against each new head without changing a sibling's active worktree. When offline, disclose that remote freshness is unverified.
+4. Initialize or resume the user-requested Markdown report and numbered ledger. Read [references/campaign-evidence.md](references/campaign-evidence.md) and [references/evidence-ledger.md](references/evidence-ledger.md) before accepting a finding. If the operator increases the target, update the active denominator and existing numbered ledger without erasing historical time-stamped progress.
 
 ## Keep at least ten lanes active
 
@@ -45,18 +45,20 @@ Read [references/live-proof-routing.md](references/live-proof-routing.md) before
 
 ## Turn findings into verified fixes
 
-1. Deduplicate against the current ledger, `origin/main`, current open and merged GitHub work, and sibling root causes. Do not count the same defect once per model, platform, route, symptom, test, or PR.
-2. Have an independent worker validate the reported behavior from the current source and user path. Add a focused regression that fails before the repair and passes after it when practical.
-3. Fix the canonical owner in an isolated worktree. Preserve public configuration, plugin ownership, gateway protocol, migration, provider, and external dependency contracts. Avoid new options, speculative compatibility, parallel state formats, dependency changes, or hard-coded symptoms.
-4. Run appropriately scoped proof on the exact candidate head. Route Docker, live providers, packaging, full checks, typechecking, broad suites, and browser work through the appropriate existing remote workflow. Inspect real exit status, scenario counts, and artifacts.
-5. Run a fresh `$autoreview` on the complete final diff. Resolve actionable findings; rerun it after any production, test, or head change. Verify formatting, `git diff --check`, exact-head CI, and the latest ClawSweeper rank-up moves.
-6. Create a focused PR with the repository's actual template, repro, user impact, frozen head, proof, and risk. Use only the current repo-native `scripts/pr` review, artifact, prepare, and merge workflow for authorized main landing.
-7. Autonomously merge only when the user authorized it **and** the defect is individually reproduced, small, low-risk, independently reviewed, current-main-compatible, and has green required exact-head proof. Verify the resulting canonical merge SHA before incrementing the ledger.
+1. Deduplicate against the current ledger, `origin/main`, current open and merged GitHub work, and sibling root causes. Count one broken invariant once, even when it produces multiple model, platform, route, lifecycle, or UI symptoms.
+2. Independently reproduce the actual current-main user path. Map the entry point, canonical owner, callers, callees, sibling implementations, state lifecycle, existing regressions, shipped contracts, and relevant direct upstream source. Identify why the current design fails before proposing a repair.
+3. Refactor the canonical owner in an isolated worktree. Repair all affected sibling paths in the same coherent change, simplify or remove the defective abstraction, and carry authoritative facts through the existing lifecycle. Prefer the appropriately sized root-cause solution over a minimal guard, special case, extra cache, fallback, compatibility shim, or narrowly passing test.
+4. Preserve public configuration, plugin ownership, gateway protocol, migrations, provider contracts, persistent state, and external dependencies. When a correct root-cause repair would change a sensitive contract or requires a product decision, prepare it for operator review; do not disguise that risk as a small autonomous fix.
+5. Add authentic regression coverage for the original reproduction, affected siblings, lifecycle cleanup, and unchanged legitimate behavior. Run appropriately scoped proof on the exact candidate head. Route Docker, real providers, packaging, full checks, typechecking, broad suites, and browser work through the existing remote workflow; inspect actual exit status, nonzero scenario counts, and artifacts.
+6. Run a fresh `$autoreview` on the complete final refactor. Resolve actionable findings; rerun review after any production, test, or head change. Personally read the latest ClawSweeper review, satisfy each applicable rank-up move with real evidence, and update the existing PR body before landing.
+7. Check existing open PRs, current author counts, and the actual repository automation before publishing. Read both the current labeler and response policy; verify the authenticated author association, repository permission, account type, automation branch prefix, and actual override label. Apply only exemptions proved by that current policy, including eligible owners, maintainers, collaborators, bots or apps, approved automation branches, and explicit overrides. Never infer capacity from a truncated list or assume that one privileged role represents every exemption. Reuse and repair an existing candidate PR for the same cause. When a real cap applies, hold reviewed worktrees and finish or land existing verified work first.
+8. Create a focused PR with the repository's actual template, canonical cause, user impact, frozen head, completed proof, and risk. Use only the current repo-native `scripts/pr` review, artifact, prepare, and merge workflow for authorized main landing.
+9. Autonomously merge only when the user authorized it **and** the canonical root-cause refactor is individually reproduced, low-risk, independently reviewed, current-main-compatible, and has green required exact-head proof. Evaluate risk by ownership and behavioral impact, not by whether the diff is the smallest possible. Verify the resulting canonical merge SHA before incrementing the ledger.
 
-Prepare but **do not autonomously merge** security or authentication changes; SQLite integrity, migration, schema, or persistent-state changes; public configuration or plugin SDK compatibility changes; protocol changes; architectural refactors; product decisions; broad fixes; uncertain diagnoses; or a candidate with pending, skipped, stale, conflicting, or failing proof. Mark these **user review required** and keep them outside the accepted-bug count.
+Prepare but **do not autonomously merge** security or authentication changes; SQLite integrity, migration, schema, or persistent-state changes; public configuration or plugin SDK compatibility changes; protocol changes; architectural or product decisions; broad high-impact refactors; uncertain diagnoses; or a candidate with pending, skipped, stale, conflicting, or failing proof. A bounded low-risk root-cause refactor may be autonomously landable when explicitly authorized; sensitive or uncertain impact is not. Mark higher-risk changes **user review required** and keep them outside the accepted-bug count.
 
 ## Maintain accurate campaign state
 
 Update the requested report throughout the campaign, not only at the end. Follow [references/campaign-evidence.md](references/campaign-evidence.md) for campaign and active-lane evidence and [references/evidence-ledger.md](references/evidence-ledger.md) for verified bug states. Separate discovered hypotheses, reproduced bugs, review-required PRs, exact-head validated fixes, and actually merged fixes. Report the frozen main SHA, worker lanes, resource use, live model and gateway proof, soak timestamps, actual failures, independent review, CI run IDs, PR links, and canonical merge SHAs.
 
-Count only distinct, verified, authorized, **actually merged** low-risk bugs toward an explicit target. Never count the Auto QA skill PR, observations, pending PRs, skipped checks, existing unrelated merges, hypotheses, or review-required fixes. Keep iterating on this skill from real campaign evidence in a separate skill-only worktree and PR.
+Count only distinct, verified, authorized, **actually merged** low-risk root-cause fixes toward the operator's current explicit target. Never count the Auto QA skill PR, observations, pending PRs, skipped checks, existing unrelated merges, individual symptoms of the same defect, hypotheses, or review-required fixes. Keep iterating on this skill from real OpenClaw campaign evidence in a separate skill-only worktree and PR.

.agents/skills/auto-qa/references/campaign-evidence.md

@@ -4,7 +4,7 @@ Update the operator-requested report throughout the run. Never place credentials
 
 ## Campaign header
 
-Record the user-approved scope, requested fix target, actual start time, requested soak duration, current immutable `origin/main` SHA, authorized landing policy, report location, and machine-load budget. Update the current baseline after every safe refresh.
+Record the user-approved scope, current requested fix target, actual start time, requested soak duration, current immutable `origin/main` SHA, authorized landing policy, report location, and machine-load budget. Record the orchestrator responsible for serialized remote-ref updates. After each landing, record the post-merge fetched baseline only after proving that it contains the recorded merge commit. If the user changes the target, update the active goal and every current ledger denominator while preserving historical time-stamped progress; never treat the old target as campaign completion.
 
 ## Audit lane
 
@@ -23,6 +23,8 @@ lane:
 
 Maintain at least ten active, differently scoped lanes whenever independent worker execution is authorized. Treat permission to fetch, contact a provider, or run an externally hosted model as a separate network constraint. Track blocked worker, network, remote, and device capacity explicitly. A finished worker, future worker, unstarted process, sequential inspection, or duplicate subsystem is not an active lane. For a single-agent task, inspect ten distinct surfaces but report the actual concurrency and independent-verification limitation.
 
+Freeze one baseline per worker wave. A worker may inspect that immutable SHA without independently refreshing shared remote refs. During native PR preparation or merge, pause worker fetches and let the orchestrator own `origin/main`. After a verified landing, fetch again, prove the merge commit is contained in the fetched ref, broadcast that full new SHA, and then resume workers.
+
 ## Bug ledger
 
 Count a product bug only after every required field is proven:
@@ -37,6 +39,8 @@ observed:
 expected:
 independent_verification:
 root_cause:
+affected_callers_and_siblings:
+canonical_owner_refactor:
 regression_or_live_proof_after:
 exact_reviewed_head:
 exact_head_hosted_checks:
@@ -46,7 +50,7 @@ risk: low
 status: merged
 ```
 
-Keep an independent `review-required` section for persistence, migrations, auth, security, SDK, protocol, broad changes, uncertain ownership, and other user decisions. Give the exact reproduction, proposed PR, real completed validation, risk, and outstanding gates. Do not include them in the merged-fix count.
+Require evidence that the repair eliminates the canonical cause, not only the observed symptom. Record which sibling paths were checked, which were fixed together, and which are unaffected; count a shared invariant once. Keep an independent `review-required` section for persistence, migrations, auth, security, SDK, protocol, high-impact architectural changes, uncertain ownership, and other user decisions. Give the exact reproduction, proposed PR, real completed validation, risk, and outstanding gates. Do not include them in the merged-fix count.
 
 ## Long-running evidence
 

.agents/skills/auto-qa/references/evidence-ledger.md

@@ -4,18 +4,20 @@ Maintain one Markdown ledger in the user-requested report. Resume its existing n
 
 Record campaign-wide facts:
 
-- Requested target and minimum soak duration.
+- The operator's current requested target and minimum soak duration; update the active target without erasing earlier historical progress.
 - Exact current `origin/main` SHA and refresh time.
+- The single refresh coordinator; native-operation pauses and proof that each post-merge fetched baseline contains the recorded merge SHA.
 - At least ten named, meaningfully distinct active subsystem lanes.
 - Owned gateway ports, isolated state, health, CPU/memory/load, and remote lease state.
 - Actual live provider and selected model without credential material.
 - Soak start, elapsed time, pass/fail/skip counts, and whether completion was actually reached.
+- Both current author-limit workflows, complete author count, and actual author, repository-role, bot/app, branch-prefix, or override exemption, if one is proved.
 
 For each candidate use one explicit state:
 
 - **Hypothesis:** worker reported a potential issue; no independent reproduction.
 - **Reproduced:** failing current-main user path or focused regression established.
-- **Fix validated:** exact branch head passes relevant user-path, regression, independent review, and required checks.
+- **Fix validated:** the canonical-owner root-cause refactor passes relevant user-path, sibling regressions, independent review, and required exact-head checks.
 - **Review required:** large, sensitive, uncertain, compatibility-affecting, or explicitly owner-reviewed work; link the separate PR without counting it.
 - **Merged:** exact hosted checks passed, native maintainer landing succeeded, and canonical main contains the merge SHA.
 - **Rejected or duplicate:** record the actual reason and canonical root cause; do not increment progress.
@@ -27,6 +29,7 @@ For every accepted merge include:
 subsystem: <canonical owner and affected user surface>
 main baseline: <full SHA>
 root cause: <repo-root source paths and current behavior>
+canonical refactor: <owner, shared invariant, affected callers and siblings>
 before: <actual failing user repro or regression>
 after: <exact passing product path and focused regression>
 live proof: <model, nonzero scenario/test counts, or exact packaging proof>
@@ -37,4 +40,4 @@ merged main SHA: <verified canonical full SHA>
 risk: low; autonomous landing explicitly authorized
 ```
 
-Do not accept mere workflow dispatch, queued CI, old-head success, a green summary with zero executed scenarios, mocked model responses, stale screenshots, a test skipped for missing credentials, or a GitHub merge request without verified canonical main state. Stop the count at the actual number of verified merges; never round up toward the user-requested target.
+Do not accept a symptom-only guard, one-sided workaround, compatibility shim, duplicate count, mere workflow dispatch, queued CI, old-head success, a green summary with zero executed scenarios, mocked model responses, stale screenshots, a test skipped for missing credentials, or a GitHub merge request without verified canonical main state. Stop the count at the actual number of verified root-cause merges; never round up toward the user-requested target.

.agents/skills/auto-qa/references/subsystem-lanes.md

@@ -1,6 +1,6 @@
 # Current-source subsystem lanes
 
-Freeze `origin/main` before starting a wave. Inspect root and scoped `AGENTS.md`, owner boundaries, tests, callers, sibling implementations, package scripts, and current GitHub history. File locations are discovery starting points, not a claim that a particular file, scenario ID, route, or model still exists.
+Freeze `origin/main` before starting a wave. Let only the orchestrator update the shared remote ref; pause refreshes during native PR preparation and merge. Fetch again after landing, verify that the fetched baseline contains the canonical merge commit, and give each resumed worker that final frozen SHA. Inspect root and scoped `AGENTS.md`, owner boundaries, tests, callers, sibling implementations, package scripts, and current GitHub history. File locations are discovery starting points, not a claim that a particular file, scenario ID, route, or model still exists.
 
 Maintain at least ten distinct active investigations. Split any large area into smaller leaf tasks:
 
@@ -21,6 +21,6 @@ Additional lanes may cover context assembly, memory, SDK consumers, browser auto
 
 Give read-only Codex CLI reviewers a narrow prompt equivalent to:
 
-> At frozen OpenClaw main `<full-sha>`, independently audit `<single ownership surface>` for real correctness regressions. Read the complete root and relevant scoped guides, complete changed modules, a caller, a callee, sibling paths, tests, and direct dependency source where relevant. Do not modify files, execute heavy tests, access or print secrets, alter an operator gateway, invoke a remote lease, or assume another reviewer's conclusion. Return only concrete current-main defects with repo-root paths, reproduction, impact, root cause, focused regression, duplicates, and low-risk versus user-review classification. Explicitly return no verified bug if the evidence is insufficient.
+> At frozen OpenClaw main `<full-sha>`, independently audit `<single ownership surface>` for real correctness regressions. Read the complete root and relevant scoped guides, complete changed modules, entry point, callers, callees, sibling paths, state lifecycle, tests, and direct dependency source where relevant. Do not modify files, execute heavy tests, access or print secrets, alter an operator gateway, invoke a remote lease, or assume another reviewer's conclusion. Return only concrete current-main defects with repo-root paths, user-path reproduction, canonical root cause, all affected siblings, a coherent owner-boundary refactor, authentic regression, duplicates, and low-risk versus user-review classification. Reject symptom-only patches and count a shared invariant once. Explicitly return no verified bug if the evidence is insufficient.
 
 Use the current installed CLI and verify its supported flags with `codex exec --help`. Keep reviews ephemeral and bounded. Set a read-only sandbox for reviewers; reserve writable isolated worktrees for authorized fixers. When an offline, no-network, or no-independent-worker assignment forbids starting Codex CLI reviewers, inspect the subsystem slices directly and explicitly report that simultaneous independent lane capacity is unavailable.