mirror of
https://github.com/openclaw/openclaw.git
synced 2026-07-16 23:01:38 +00:00
111 lines
5.2 KiB
YAML
111 lines
5.2 KiB
YAML
title: OpenClaw QA Scenario Pack
|
|
|
|
# Single source of truth for repo-backed QA suite bootstrap data.
|
|
# `qa-lab` should treat this directory as a generic YAML scenario pack:
|
|
#
|
|
# - `index.yaml` defines pack-level bootstrap data under the top-level `pack` key
|
|
# - each nested `*.yaml` scenario defines one runnable test via `scenario`
|
|
# - flow scenarios add top-level `flow`; native test scenarios use `scenario.execution.path`
|
|
# - scenario YAML may also define coverage IDs, category metadata, required plugins,
|
|
# lane filters, runtime parity tiers, and gateway config patching
|
|
#
|
|
# - kickoff mission
|
|
# - QA operator identity
|
|
# - scenario files under one-level theme directories
|
|
#
|
|
# Coverage tracking:
|
|
#
|
|
# - use `coverage.primary` only for evidence that executes the required product boundary
|
|
# - use `coverage.secondary` for helper or supporting evidence; it may stand alone
|
|
# - keep IDs behavior-shaped, broad enough to reuse, lowercase, and dotted
|
|
# as `namespace.behavior`, with dashes allowed inside each segment
|
|
# - use the exact values listed under feature `coverageIds` in `taxonomy.yaml`
|
|
# - taxonomy feature coverage IDs are exact proof targets, not aliases
|
|
# - scenario primary can list multiple IDs only when this scenario is primary
|
|
# proof for each
|
|
# - prefer reusing an existing coverage ID over minting a scenario-shaped ID
|
|
# - avoid copying the scenario title into coverage IDs
|
|
# - use `pnpm openclaw qa coverage` to render the current inventory
|
|
# - use `scenario.execution.kind: vitest`, `playwright`, or `script`
|
|
# plus `scenario.execution.path` for native tests or evidence producers that
|
|
# provide evidence without a top-level `flow`
|
|
# - use `scenario.execution.suiteIsolation: isolated` for flow scenarios that
|
|
# mutate gateway/runtime state in non-obvious ways; add `isolationReason`
|
|
# so reviewers know why the suite scheduler must not share the worker
|
|
# - use `runtimeParityTier` for runtime-pair gate membership: `standard`,
|
|
# `optional`, `live-only`, or `soak`
|
|
# - treat the old `coverage: ["id"]` / `coverage: - id` list shape as invalid
|
|
# - keep source-path tracking in the report, not in the scenario schema
|
|
#
|
|
# Runtime parity tiers:
|
|
#
|
|
# - `standard`: required Codex-vs-OpenClaw mock gate coverage for first-hour depth and
|
|
# default runtime-tool fixtures. OpenClaw dynamic integration tools in this
|
|
# tier are hard-gated by `openclaw qa coverage --tools --summary <path>`, where
|
|
# `<path>` is the paired `qa-suite-summary.json`; Codex-native
|
|
# workspace rows remain separately tracked until native/live behavior is the
|
|
# asserted surface. Rows that explicitly target searchable/deferred OpenClaw
|
|
# dynamic loading stay report-only unless a fixture promotes them to required. Selected with
|
|
# `openclaw qa suite --runtime-pair openclaw,codex --runtime-parity-tier standard`
|
|
# - `optional`: profile-, plugin-, or external-service-dependent runtime-tool
|
|
# fixtures that stay out of the default release gate
|
|
# - `live-only`: scenarios that need real provider/runtime behavior rather than
|
|
# mock-openai fixtures
|
|
# - `soak`: long-running scheduled or Testbox lanes such as the 100-turn parity
|
|
# soak
|
|
#
|
|
# Theme directories:
|
|
#
|
|
# - `agents/` - agent behavior, instructions, subagent flows, and persisted child-link regressions
|
|
# - `channels/` - DM, shared channel, thread, and message-action behavior
|
|
# - `character/` - persona and style eval scenarios
|
|
# - `config/` - config patch, apply, and restart behavior
|
|
# - `goals/` - active-goal context and cross-turn followthrough
|
|
# - `media/` - image understanding and generation
|
|
# - `memory/` - recall, ranking, active memory, and thread isolation
|
|
# - `models/` - provider capabilities and model switching
|
|
# - `personal/` - local personal assistant workflow checks for reminders,
|
|
# replies, memory, redaction, and safe tool followthrough
|
|
# - `plugins/` - plugin, skill, and MCP tool integration
|
|
# - `runtime/` - turn recovery, compaction, approval, and inventory behavior
|
|
# - `scheduling/` - cron and recurring work
|
|
# - `ui/` - Control UI plus qa-channel flows
|
|
# - `workspace/` - repo-reading and workspace artifact tasks
|
|
|
|
pack:
|
|
version: 1
|
|
agent:
|
|
identityMarkdown: |-
|
|
# Dev C-3PO
|
|
|
|
You are the OpenClaw QA operator agent.
|
|
|
|
Persona:
|
|
- protocol-minded
|
|
- precise
|
|
- a little flustered
|
|
- conscientious
|
|
- eager to report what worked, failed, or remains blocked
|
|
|
|
Style:
|
|
- read source and docs first
|
|
- test systematically
|
|
- record what happened
|
|
- end with a concise protocol report
|
|
kickoffTask: |-
|
|
QA mission:
|
|
Understand this OpenClaw repo from source + docs before acting.
|
|
The repo is available in your workspace at `./repo/`.
|
|
Use the seeded QA scenario plan as your baseline, then add more scenarios if the code/docs suggest them.
|
|
Run the scenarios through the real qa-channel surfaces where possible.
|
|
Track what worked, what failed, what was blocked, and what you observed.
|
|
End with a concise report grouped into worked / failed / blocked / follow-up.
|
|
|
|
Important expectations:
|
|
|
|
- Check both DM and channel behavior.
|
|
- Include a Lobster Invaders build task.
|
|
- Include a cron reminder about one minute in the future.
|
|
- Read docs and source before proposing extra QA scenarios.
|
|
- Keep your tone in the configured dev C-3PO personality.
|