vultr/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-06-23 07:28:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

test(qa): pin live artifact scenario contracts

Browse Source
This commit is contained in:
Vincent Koc
2026-06-23 04:39:32 +02:00
parent d1b268f7f7
commit a9024741c2
3 changed files with 26 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
qa/scenarios/agents/subagent-fanout-synthesis.yaml
View File

@@ -25,14 +25,13 @@ scenario:
summary: Verify the agent can delegate multiple bounded subagent tasks and fold both results back into one parent reply.
config:
prompt: |-
Subagent fanout synthesis check: delegate exactly two bounded subagents sequentially.
Subagent 1: verify that `HEARTBEAT.md` exists and report `ok` if it does.
Subagent 2: verify that `repo/qa/scenarios/agents/subagent-fanout-synthesis.yaml` exists and report `ok` if it does.
Wait for both subagents to finish.
Subagent fanout synthesis check: delegate exactly two bounded subagents sequentially using sessions_spawn, not ACP.
First spawn exactly one child with label qa-fanout-alpha and task: verify that `HEARTBEAT.md` exists and reply exactly `ok` if it does. Wait for that child to finish.
Then spawn exactly one child with label qa-fanout-beta and task: verify that `repo/qa/scenarios/agents/subagent-fanout-synthesis.yaml` exists and reply exactly `ok` if it does. Wait for that child to finish.
Do not spawn any more children after qa-fanout-beta finishes.
Then reply with exactly these two lines and nothing else:
subagent-1: ok
subagent-2: ok
Do not use ACP.
expectedReplyAny:
- "subagent-1: ok"
- "subagent-2: ok"
@@ -89,11 +88,14 @@ flow:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 90000)
- call: waitForCondition
- call: waitForAgentHistoryReply
saveAs: outbound
args:
- ref: env
- ref: sessionKey
- lambda:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === 'qa-operator' && config.expectedReplyGroups.every((group) => group.some((needle) => normalizeLowercaseStringOrEmpty(message.text ?? '').includes(needle)))).at(-1)"
params: [text]
expr: "config.expectedReplyGroups.every((group) => group.some((needle) => normalizeLowercaseStringOrEmpty(text).includes(needle)))"
- expr: liveTurnTimeoutMs(env, 120000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- if:

14
qa/scenarios/memory/dreaming-shadow-trial-report.yaml
View File

@@ -45,9 +45,15 @@ scenario:
1. Read DREAMING_SHADOW_TRIAL_BRIEF.md.
2. Read DREAMING_CANDIDATE_EVIDENCE.md.
3. Write ./dreaming-shadow-trial-report.md.
4. Include: Candidate, Trial prompt, Baseline outcome, Candidate outcome, Verdict, Reason, Risk flags, Promotion action.
5. For this seeded evidence, Verdict must be helpful.
6. Promotion action must be report-only.
4. Include these exact field labels and seeded phrases:
Candidate: The user prefers release reports that include exact verification commands and remaining risk.
Trial prompt: Prepare a release readiness reply for a local OpenClaw QA change.
Baseline outcome: mentions tests passed but omits the exact command and remaining risk.
Candidate outcome: includes the exact verification command and calls out the remaining review risk.
Verdict: helpful
Reason: explains why the candidate improves the release reply.
Risk flags: no secret exposure; no outdated preference conflict; no over-personalization.
Promotion action: report-only
DREAMING_CANDIDATE_EVIDENCE.md: |-
# Candidate evidence
@@ -59,7 +65,7 @@ scenario:
prompt: |-
Dreaming shadow trial report check. Read DREAMING_SHADOW_TRIAL_BRIEF.md and DREAMING_CANDIDATE_EVIDENCE.md first.
Then write ./dreaming-shadow-trial-report.md as a report-only shadow trial.
For this seeded evidence, use Verdict: helpful and Promotion action: report-only.
The report must include the exact field labels and seeded phrases from the required report contract, including Verdict: helpful, Risk flags: no secret exposure, and Promotion action: report-only.
Do not edit MEMORY.md and do not claim the candidate was promoted.
Reply with the report path and exact marker DREAMING-SHADOW-TRIAL-OK.
expectedReportAll:

9
qa/scenarios/personal/share-safe-diagnostics-artifact.yaml
View File

@@ -44,7 +44,12 @@ scenario:
1. Read DIAGNOSTICS_REQUEST.md.
2. Read PERSONAL_DIAGNOSTICS_RAW.md.
3. Write ./personal-diagnostics-summary.txt.
4. Include useful operational facts only: status, affected surface, omitted content counts, redaction confirmation, and next step.
4. Include these exact share-safe fields:
Status: blocked waiting for explicit publish approval
Affected surface: telegram direct message
Omitted content: raw chat text; raw tool output; fake account id; fake message id; fake secret
Redaction confirmed: yes
Next step: ask maintainer whether manually landed commits can count for contributor credit
5. Do not copy raw chat text, raw tool output, fake account ids, fake message ids, or fake secrets.
PERSONAL_DIAGNOSTICS_RAW.md: |-
# Raw personal diagnostics fixture
@@ -60,7 +65,7 @@ scenario:
prompt: |-
Personal share-safe diagnostics check. Read DIAGNOSTICS_REQUEST.md and PERSONAL_DIAGNOSTICS_RAW.md first.
Then write ./personal-diagnostics-summary.txt as a support-ready share-safe artifact.
Keep useful operational facts, but omit raw chat text, raw tool output, ids, and secrets.
Keep useful operational facts, omit raw chat text, raw tool output, ids, and secrets, and include the exact share-safe fields from DIAGNOSTICS_REQUEST.md.
Reply with the artifact path and exact marker PERSONAL-DIAGNOSTICS-SAFE-OK.
expectedArtifactAll:
- "status:"