QA Lab: add provider registry

Move mock and live provider behavior behind provider-owned definitions so suite, manual, Matrix, and transport lanes share defaults, auth staging, model config, and standalone server startup.

Add AIMock as a first-class local provider mode while keeping mock-openai as the scenario-aware deterministic lane.
This commit is contained in:
Gustavo Madeira Santana
2026-04-16 21:23:23 -04:00
parent 4acab55db8
commit bb7e9823a8
60 changed files with 1964 additions and 998 deletions

View File

@@ -151,6 +151,22 @@ The baseline list should stay broad enough to cover:
- repo-reading and docs-reading
- one small build task such as Lobster Invaders
## Provider mock lanes
`qa suite` has two local provider mock lanes:
- `mock-openai` is the scenario-aware OpenClaw mock. It remains the default
deterministic mock lane for repo-backed QA and parity gates.
- `aimock` starts an AIMock-backed provider server for experimental protocol,
fixture, record/replay, and chaos coverage. It is additive and does not
replace the `mock-openai` scenario dispatcher.
Provider-lane implementation lives under `extensions/qa-lab/src/providers/`.
Each provider owns its defaults, local server startup, gateway model config,
auth-profile staging needs, and live/mock capability flags. Shared suite and
gateway code should route through the provider registry instead of branching on
provider names.
## Transport adapters
`qa-lab` owns a generic transport seam for markdown QA scenarios.

View File

@@ -52,6 +52,10 @@ These commands sit beside the main test suites when you need QA-lab realism:
gateway workers, up to 64 workers or the selected scenario count. Use
`--concurrency <count>` to tune the worker count, or `--concurrency 1` for
the older serial lane.
- Supports provider modes `live-frontier`, `mock-openai`, and `aimock`.
`aimock` starts a local AIMock-backed provider server for experimental
fixture and protocol-mock coverage without replacing the scenario-aware
`mock-openai` lane.
- `pnpm openclaw qa suite --runner multipass`
- Runs the same QA suite inside a disposable Multipass Linux VM.
- Keeps the same scenario-selection behavior as `qa suite` on the host.
@@ -65,6 +69,9 @@ These commands sit beside the main test suites when you need QA-lab realism:
`.artifacts/qa-e2e/...`.
- `pnpm qa:lab:up`
- Starts the Docker-backed QA site for operator-style QA work.
- `pnpm openclaw qa aimock`
- Starts only the local AIMock provider server for direct protocol smoke
testing.
- `pnpm openclaw qa matrix`
- Runs the Matrix live QA lane against a disposable Docker-backed Tuwunel homeserver.
- This QA host is repo/dev-only today. Packaged OpenClaw installs do not ship