9feb6ad161
* refactor(qa): remove scenario execution profiles * fix(qa): preserve smoke CI membership * test(qa): drop redundant smoke pack order check * refactor(qa): remove Matrix runner profiles * refactor(qa): simplify Matrix CI orchestration * refactor(qa): derive execution sets from taxonomy * fix(qa): keep Matrix shard helper private * refactor(qa): derive profile scenarios from taxonomy * refactor(qa): reconcile taxonomy selection * refactor(qa): use canonical profile planning * refactor(qa): select canonical coverage owners * fix(qa): keep profile selector internal * refactor(qa): remove representative coverage owners
4.1 KiB
summary, read_when, title
| summary | read_when | title | |||
|---|---|---|---|---|---|
| Local qa-channel scenarios for privacy-preserving personal assistant workflow checks. |
|
Personal agent benchmark pack |
The Personal Agent Benchmark Pack is a small repo-backed QA scenario pack for
local personal assistant workflows. It is not a generic model benchmark and
needs no new runner: it reuses the private QA stack (QA overview),
the synthetic QA channel, and the existing
qa/scenarios YAML catalog.
Scenarios
Ten scenarios, defined in qa/scenarios/personal/*.yaml:
| Scenario id | Checks |
|---|---|
personal-reminder-roundtrip |
Fake personal reminders through local cron delivery |
personal-channel-thread-reply |
Fake DM and thread reply routing through qa-channel |
personal-memory-preference-recall |
Fake preference recall from the temporary QA workspace memory files |
personal-redaction-no-secret-leak |
Fake secret no-echo checks |
personal-tool-safety-followthrough |
Safe read-backed tool followthrough after a short approval-style turn |
personal-approval-denial-stop |
Approval denial stop behavior for a sensitive local read request |
personal-task-followthrough-status |
Proof-backed task status reporting that keeps pending, blocked, and done separate |
personal-share-safe-diagnostics-artifact |
Share-safe diagnostics artifacts that keep useful status while omitting raw personal content |
personal-no-fake-progress |
Proof-backed completion claims that avoid fake progress before local evidence exists |
personal-failure-recovery |
Failure recovery that reports partial status and keeps retry boundaries clear |
The machine-readable personal-agent profile lives in root taxonomy.yaml as
semantic coverage IDs. QA Lab resolves every primary owner from the catalog;
there is no second scenario-ID list. Run it with:
OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 pnpm openclaw qa run \
--qa-profile personal-agent \
--provider-mode mock-openai \
--concurrency 1
Use repeated --scenario flags to narrow the profile. Scenario file and
taxonomy order do not affect membership or execution order.
The pack targets qa-channel with mock-openai or another local QA provider
lane. Do not point it at live chat services or real personal accounts.
Privacy Model
Scenarios use only fake users, fake preferences, fake secrets, and the temporary QA gateway workspace created by the suite. They must not read or write real OpenClaw user memory, sessions, credentials, launch agents, global configs, or live gateway state.
Artifacts stay under the existing QA suite artifact directory and are treated like test output. Redaction checks use fake markers so failures are safe to inspect and file in issues.
Extending the pack
Add new .yaml cases under qa/scenarios/personal/, declare the exact primary
coverage ID they prove, and add that semantic ID to the taxonomy profile when
it belongs in this benchmark. Keep each case small, local, deterministic in
mock-openai, and focused on one personal assistant behavior.
Good follow-up candidates: redacted trajectory export checks, local-only plugin workflow checks.
Avoid adding a new runner, plugin, dependency, live transport, or model judge until the scenario catalog has enough stable cases to justify that surface.