QA: organize scenarios by theme

This commit is contained in:
Gustavo Madeira Santana
2026-04-17 11:02:43 -04:00
parent a45ebf3281
commit 82fe6f50ef
57 changed files with 209 additions and 32 deletions

View File

@@ -4,7 +4,8 @@ Seed QA assets for the private `qa-lab` extension.
Files:
- `scenarios.md` - canonical QA scenario pack, kickoff mission, and operator identity.
- `scenarios/index.md` - canonical QA scenario pack, kickoff mission, and operator identity.
- `scenarios/<theme>/*.md` - one runnable scenario per markdown file.
- `frontier-harness-plan.md` - big-model bakeoff and tuning loop for harness work.
- `convex-credential-broker/` - standalone Convex v1 lease broker for pooled live credentials.

View File

@@ -3,6 +3,6 @@
Canonical scenario source now lives in:
- `qa/scenarios/index.md`
- `qa/scenarios/*.md`
- `qa/scenarios/<theme>/*.md`
Each QA scenario has its own markdown file.

View File

@@ -23,7 +23,7 @@ execution:
prompt: |-
Subagent fanout synthesis check: delegate exactly two bounded subagents sequentially.
Subagent 1: verify that `HEARTBEAT.md` exists and report `ok` if it does.
Subagent 2: verify that `repo/qa/scenarios/subagent-fanout-synthesis.md` exists and report `ok` if it does.
Subagent 2: verify that `repo/qa/scenarios/agents/subagent-fanout-synthesis.md` exists and report `ok` if it does.
Wait for both subagents to finish.
Then reply with exactly these two lines and nothing else:
subagent-1: ok

View File

@@ -4,12 +4,28 @@ Single source of truth for repo-backed QA suite bootstrap data.
`qa-lab` should treat this directory as a generic markdown scenario pack:
- `index.md` defines pack-level bootstrap data
- each `*.md` scenario defines one runnable test via `qa-scenario` + `qa-flow`
- scenario markdown may also define required plugins and gateway config patching
- each nested `*.md` scenario defines one runnable test via `qa-scenario` + `qa-flow`
- scenario markdown may also define category metadata, required plugins, lane filters,
and gateway config patching
- kickoff mission
- QA operator identity
- scenario files under `./`
- scenario files under one-level theme directories
Theme directories:
- `agents/` - agent behavior, instructions, and subagent flows
- `channels/` - DM, shared channel, thread, and message-action behavior
- `character/` - persona and style eval scenarios
- `config/` - config patch, apply, and restart behavior
- `media/` - image understanding and generation
- `memory/` - recall, ranking, active memory, and thread isolation
- `models/` - provider capabilities and model switching
- `plugins/` - plugin, skill, and MCP tool integration
- `runtime/` - turn recovery, compaction, approval, and inventory behavior
- `scheduling/` - cron and recurring work
- `ui/` - Control UI plus qa-channel flows
- `workspace/` - repo-reading and workspace artifact tasks
```yaml qa-pack
version: 1