vultr/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-07-23 03:21:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(qa): canonicalize live scenario coverage (#108464)

Browse Source
This commit is contained in:
Dallin Romney
2026-07-15 16:50:41 -07:00
committed by GitHub
parent 18cff45db9
commit 2d9b78fff7
253 changed files with 618 additions and 1271 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/.generated/plugin-sdk-api-baseline.sha256
View File

@@ -1,2 +1,2 @@
505c586d2ac5e556d9128d90f9dba2b3d47b98e5ef7dd81aefb6208da5643095 plugin-sdk-api-baseline.json
588cdf7f285e2d50c6a03410b67c30ac99886c3e4eaebbfb2c503893578da979 plugin-sdk-api-baseline.jsonl
25f96bb0ea9201fa0a2b47156911297de8fd0a83af806c63bd6679c60511b1b8 plugin-sdk-api-baseline.json
eb7359b5c19b6543da999194471f576fe715f099aa1c44e81cfe5d307275f2b6 plugin-sdk-api-baseline.jsonl

36
docs/concepts/qa-e2e-automation.md
View File

@@ -194,10 +194,8 @@ precedence over `--profile`.
| `e2ee-deep` | 18 | State-loss, backup, key recovery, device hygiene, and SAS/QR/DM verification. |
| `e2ee-cli` | 9 | `openclaw matrix encryption setup`, recovery-key, multi-account, gateway round-trip, and self-verification commands through the harness. |
The profile mapping lives in
`extensions/qa-lab/src/live-transports/matrix/profiles.ts`. The declarative
Matrix scenarios live under `qa/scenarios/channels/`; their live
implementations live under
Profile membership and channel/driver requirements live with the declarative
Matrix scenarios under `qa/scenarios/channels/`. Their live implementations live under
`extensions/qa-lab/src/live-transports/matrix/scenarios/`.
The adapter provisions a disposable Tuwunel homeserver in Docker (default
@@ -393,27 +391,19 @@ only set/missing status for `OPENCLAW_QA_CONVEX_SECRET_CI` and
`OPENCLAW_QA_CONVEX_SECRET_MAINTAINER`, and verifies admin/list reachability
when the maintainer secret is present.
## Live transport coverage
## Canonical scenario coverage
Live transport lanes share one contract instead of each inventing their own
scenario list shape. `qa-channel` is the broad synthetic product-behavior
suite and is not part of the live transport coverage matrix.
The root `taxonomy.yaml` defines semantic coverage IDs. Scenario YAML files
under `qa/scenarios/` map each scenario to those IDs and own execution
metadata: `channel` is the only channel requirement, `driver` constrains the
runner when needed, and `profiles` declare named run membership. TypeScript
runners query that catalog; they do not maintain parallel scenario or coverage
inventories.
Live transport runners import the shared scenario ids, baseline coverage
helpers, and scenario-selection helper from
`openclaw/plugin-sdk/qa-live-transport-scenarios`.
| Lane | Canary | Mention gating | Bot-to-bot | Allowlist block | Top-level reply | Quote reply | Restart resume | Thread follow-up | Thread isolation | Reaction observation | Help command | Native command registration |
| -------- | ------ | -------------- | ---------- | --------------- | --------------- | ----------- | -------------- | ---------------- | ---------------- | -------------------- | ------------ | --------------------------- |
| Discord | x | x | x | | | | | | | | | x |
| Matrix | x | x | x | x | x | | x | x | x | x | | |
| Slack | x | x | x | x | x | | x | x | x | | | |
| Telegram | x | x | x | | | | | | | | x | |
| WhatsApp | x | x | | x | x | x | x | | | x | x | |
This keeps `qa-channel` as the broad product-behavior suite while Matrix,
Telegram, and the other live transports share one explicit transport-contract
checklist.
Static `qa coverage` output reports the taxonomy-to-scenario mapping. Actual
proof comes from `qa-evidence.json`, which records the executed scenario,
coverage IDs, channel, driver, and result. Channel and driver are report
dimensions, not additional coverage-ID vocabularies.
For a disposable Linux VM lane without bringing Docker into the QA path, run:

2
docs/docs_map.md
View File

@@ -2820,7 +2820,7 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
- H3: Discord Mantis scenarios
- H3: Mantis Slack desktop and visual-task runners
- H3: Credential pool health check
- H2: Live transport coverage
- H2: Canonical scenario coverage
- H2: Discord, Slack, Telegram, and WhatsApp QA reference
- H3: Shared CLI flags
- H3: Telegram QA

14
docs/plugins/sdk-migration.md
View File

@@ -1050,13 +1050,13 @@ known external consumers; only its public package export will be retired. Its
module will remain available to bundled plugins as a private-local-only
subpath.
| `removeAfter` | Tier | SDK subpaths |
| ------------- | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `2026-07-30` | Earlier compatibility deprecations | `agent-dir-compat`, `channel-envelope`, `channel-inbound-roots`, `channel-location`, `channel-message-runtime`, `channel-pairing-paths`, `channel-reply-options-runtime`, `config-schema`, `config-types`, `direct-dm`, `direct-dm-access`, `mattermost`, `media-generation-runtime-shared`, `memory-core`, `memory-core-engine-runtime`, `memory-core-host-events`, `memory-core-host-multimodal`, `memory-core-host-query`, `memory-host-files`, `memory-host-status`, `music-generation-core`, `outbound-runtime`, `outbound-send-deps`, `provider-auth-login`, `provider-zai-endpoint`, `reply-dedupe`, `runtime-logger`, `runtime-secret-resolution`, `self-hosted-provider-setup`, `setup-adapter-runtime`, `telegram-command-config`, `webhook-path`, `zalouser` |
| `2026-07-30` | θ1: fully unused; remove subpath | `command-gating`, `lmstudio`, `lmstudio-runtime`, `secret-provider-integration`, `skills-runtime` |
| `2026-07-30` | θ2: bundled-only; retire public export | `access-groups`, `account-resolution-runtime`, `acp-binding-resolve-runtime`, `acp-binding-runtime`, `acp-runtime`, `acp-runtime-backend`, `agent-core`, `agent-harness-exec-review-runtime`, `agent-harness-task-runtime`, `agent-harness-tool-runtime`, `agent-media-payload`, `agent-sessions`, `approval-reaction-runtime`, `approval-reference-runtime`, `async-lock-runtime`, `browser-config`, `bundled-channel-config-schema`, `channel-activity-runtime`, `channel-config-writes`, `channel-mention-gating`, `channel-route`, `channel-secret-tts-runtime`, `channel-targets`, `chat-channel-ids`, `cli-backend`, `cli-runtime`, `codex-mcp-projection`, `command-status-runtime`, `command-surface`, `concurrency-runtime`, `context-visibility-runtime`, `conversation-binding-runtime`, `cron-store-runtime`, `dangerous-name-runtime`, `delivery-queue-runtime`, `direct-dm-guard-policy`, `directory-config-runtime`, `document-extractor`, `embedding-providers`, `exec-approvals-runtime`, `expect-runtime`, `fetch-runtime`, `file-access-runtime`, `file-lock`, `global-singleton`, `group-activation`, `heartbeat-runtime`, `host-runtime`, `html-entity-runtime`, `image-generation`, `image-generation-core`, `image-generation-runtime`, `inline-image-data-url-runtime`, `json-schema-runtime`, `json-unsafe-integers`, `keyed-async-queue`, `llm`, `markdown-table-runtime`, `media-generation-runtime`, `media-understanding`, `memory-core-host-embedding-registry`, `memory-core-host-engine-embeddings`, `memory-core-host-engine-qmd`, `memory-core-host-engine-storage`, `memory-core-host-runtime-cli`, `memory-core-host-runtime-core`, `memory-core-host-runtime-files`, `memory-core-host-secret`, `memory-core-host-status`, `memory-host-core`, `memory-host-events`, `memory-host-markdown`, `memory-host-search`, `message-tool-delivery-hints`, `migration`, `migration-runtime`, `music-generation`, `node-host`, `number-runtime`, `outbound-media`, `pair-loop-guard-runtime`, `plugin-config-runtime`, `plugin-state-runtime`, `poll-runtime`, `process-runtime`, `provider-auth-api-key`, `provider-auth-login-flow-runtime`, `provider-auth-result`, `provider-auth-runtime`, `provider-catalog-live-runtime`, `provider-catalog-shared`, `provider-entry`, `provider-env-vars`, `provider-http`, `provider-model-shared`, `provider-model-types`, `provider-oauth-runtime`, `provider-onboard`, `provider-selection-runtime`, `provider-setup`, `provider-stream`, `provider-stream-family`, `provider-stream-shared`, `provider-tools`, `provider-transport-runtime`, `provider-usage`, `provider-web-fetch`, `provider-web-fetch-contract`, `provider-web-search`, `provider-web-search-config-contract`, `provider-web-search-contract`, `qa-live-transport-scenarios`, `qa-runner-runtime`, `realtime-bootstrap-context`, `realtime-transcription`, `realtime-voice`, `reply-reference`, `request-url`, `response-limit-runtime`, `retry-runtime`, `runtime-doctor`, `runtime-fetch`, `sandbox`, `secret-file-runtime`, `secure-random-runtime`, `session-binding-runtime`, `session-catalog`, `session-key-runtime`, `session-transcript-hit`, `session-transcript-runtime`, `session-visibility`, `simple-completion-runtime`, `speech`, `speech-core`, `sqlite-runtime`, `ssrf-dispatcher`, `string-normalization-runtime`, `system-event-runtime`, `talk-config-runtime`, `target-resolver-runtime`, `text-autolink-runtime`, `text-utility-runtime`, `thread-bindings-runtime`, `thread-bindings-session-runtime`, `time-runtime`, `tool-payload`, `tool-plugin`, `tool-results`, `transcripts`, `transport-ready-runtime`, `tts-runtime`, `types`, `video-generation`, `video-generation-core`, `video-generation-runtime`, `web-content-extractor`, `webhook-targets`, `windows-spawn` |
| `2026-08-15` | Earlier compatibility deprecations | `agent-config-primitives`, `channel-logging`, `channel-secret-runtime`, `channel-streaming`, `group-access`, `inbound-reply-dispatch`, `matrix`, `text-runtime`, `zod` |
| `2026-09-01` | Earlier compatibility deprecations | `channel-lifecycle`, `channel-message`, `channel-reply-pipeline`, `config-runtime`, `infra-runtime` |
| `removeAfter` | Tier | SDK subpaths |
| ------------- | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `2026-07-30` | Earlier compatibility deprecations | `agent-dir-compat`, `channel-envelope`, `channel-inbound-roots`, `channel-location`, `channel-message-runtime`, `channel-pairing-paths`, `channel-reply-options-runtime`, `config-schema`, `config-types`, `direct-dm`, `direct-dm-access`, `mattermost`, `media-generation-runtime-shared`, `memory-core`, `memory-core-engine-runtime`, `memory-core-host-events`, `memory-core-host-multimodal`, `memory-core-host-query`, `memory-host-files`, `memory-host-status`, `music-generation-core`, `outbound-runtime`, `outbound-send-deps`, `provider-auth-login`, `provider-zai-endpoint`, `reply-dedupe`, `runtime-logger`, `runtime-secret-resolution`, `self-hosted-provider-setup`, `setup-adapter-runtime`, `telegram-command-config`, `webhook-path`, `zalouser` |
| `2026-07-30` | θ1: fully unused; remove subpath | `command-gating`, `lmstudio`, `lmstudio-runtime`, `secret-provider-integration`, `skills-runtime` |
| `2026-07-30` | θ2: bundled-only; retire public export | `access-groups`, `account-resolution-runtime`, `acp-binding-resolve-runtime`, `acp-binding-runtime`, `acp-runtime`, `acp-runtime-backend`, `agent-core`, `agent-harness-exec-review-runtime`, `agent-harness-task-runtime`, `agent-harness-tool-runtime`, `agent-media-payload`, `agent-sessions`, `approval-reaction-runtime`, `approval-reference-runtime`, `async-lock-runtime`, `browser-config`, `bundled-channel-config-schema`, `channel-activity-runtime`, `channel-config-writes`, `channel-mention-gating`, `channel-route`, `channel-secret-tts-runtime`, `channel-targets`, `chat-channel-ids`, `cli-backend`, `cli-runtime`, `codex-mcp-projection`, `command-status-runtime`, `command-surface`, `concurrency-runtime`, `context-visibility-runtime`, `conversation-binding-runtime`, `cron-store-runtime`, `dangerous-name-runtime`, `delivery-queue-runtime`, `direct-dm-guard-policy`, `directory-config-runtime`, `document-extractor`, `embedding-providers`, `exec-approvals-runtime`, `expect-runtime`, `fetch-runtime`, `file-access-runtime`, `file-lock`, `global-singleton`, `group-activation`, `heartbeat-runtime`, `host-runtime`, `html-entity-runtime`, `image-generation`, `image-generation-core`, `image-generation-runtime`, `inline-image-data-url-runtime`, `json-schema-runtime`, `json-unsafe-integers`, `keyed-async-queue`, `llm`, `markdown-table-runtime`, `media-generation-runtime`, `media-understanding`, `memory-core-host-embedding-registry`, `memory-core-host-engine-embeddings`, `memory-core-host-engine-qmd`, `memory-core-host-engine-storage`, `memory-core-host-runtime-cli`, `memory-core-host-runtime-core`, `memory-core-host-runtime-files`, `memory-core-host-secret`, `memory-core-host-status`, `memory-host-core`, `memory-host-events`, `memory-host-markdown`, `memory-host-search`, `message-tool-delivery-hints`, `migration`, `migration-runtime`, `music-generation`, `node-host`, `number-runtime`, `outbound-media`, `pair-loop-guard-runtime`, `plugin-config-runtime`, `plugin-state-runtime`, `poll-runtime`, `process-runtime`, `provider-auth-api-key`, `provider-auth-login-flow-runtime`, `provider-auth-result`, `provider-auth-runtime`, `provider-catalog-live-runtime`, `provider-catalog-shared`, `provider-entry`, `provider-env-vars`, `provider-http`, `provider-model-shared`, `provider-model-types`, `provider-oauth-runtime`, `provider-onboard`, `provider-selection-runtime`, `provider-setup`, `provider-stream`, `provider-stream-family`, `provider-stream-shared`, `provider-tools`, `provider-transport-runtime`, `provider-usage`, `provider-web-fetch`, `provider-web-fetch-contract`, `provider-web-search`, `provider-web-search-config-contract`, `provider-web-search-contract`, `qa-runner-runtime`, `realtime-bootstrap-context`, `realtime-transcription`, `realtime-voice`, `reply-reference`, `request-url`, `response-limit-runtime`, `retry-runtime`, `runtime-doctor`, `runtime-fetch`, `sandbox`, `secret-file-runtime`, `secure-random-runtime`, `session-binding-runtime`, `session-catalog`, `session-key-runtime`, `session-transcript-hit`, `session-transcript-runtime`, `session-visibility`, `simple-completion-runtime`, `speech`, `speech-core`, `sqlite-runtime`, `ssrf-dispatcher`, `string-normalization-runtime`, `system-event-runtime`, `talk-config-runtime`, `target-resolver-runtime`, `text-autolink-runtime`, `text-utility-runtime`, `thread-bindings-runtime`, `thread-bindings-session-runtime`, `time-runtime`, `tool-payload`, `tool-plugin`, `tool-results`, `transcripts`, `transport-ready-runtime`, `tts-runtime`, `types`, `video-generation`, `video-generation-core`, `video-generation-runtime`, `web-content-extractor`, `webhook-targets`, `windows-spawn` |
| `2026-08-15` | Earlier compatibility deprecations | `agent-config-primitives`, `channel-logging`, `channel-secret-runtime`, `channel-streaming`, `group-access`, `inbound-reply-dispatch`, `matrix`, `text-runtime`, `zod` |
| `2026-09-01` | Earlier compatibility deprecations | `channel-lifecycle`, `channel-message`, `channel-reply-pipeline`, `config-runtime`, `infra-runtime` |
All core plugins have already migrated. External plugins should migrate
before the next major release. Run `pnpm plugins:boundary-report` to see which

1
docs/plugins/sdk-subpaths.md
View File

@@ -257,7 +257,6 @@ usage endpoint failed or returned no usable usage data.
| `plugin-sdk/process-runtime` | Process exec helpers |
| `plugin-sdk/node-host` | Node-host executable resolution and PTY resume helpers |
| `plugin-sdk/cli-runtime` | Deprecated broad barrel for CLI formatting, wait, version, argument-invocation, and lazy command-group helpers; prefer focused CLI/runtime subpaths |
| `plugin-sdk/qa-live-transport-scenarios` | Shared live transport QA scenario ids, baseline coverage helpers, and scenario-selection helper |
| `plugin-sdk/qa-runner-runtime` | Supported facade exposing plugin QA scenarios through the CLI command surface |
| `plugin-sdk/tts-runtime` | Supported facade for text-to-speech config schemas and runtime helpers |
| `plugin-sdk/gateway-method-runtime` | Reserved Gateway method dispatch helper for plugin HTTP routes that declare `contracts.gatewayMethodDispatch: ["authenticated-request"]` |

9
extensions/qa-lab/src/cli.runtime.test.ts
View File

@@ -3,7 +3,7 @@ import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { QaScenarioPack } from "./scenario-catalog.js";
import { listQaScenariosForExecutionProfile, type QaScenarioPack } from "./scenario-catalog.js";
const {
runQaManualLane,
@@ -283,7 +283,6 @@ describe("qa cli runtime", () => {
listLiveTransportQaAdapterFactories.mockReturnValue([
{
id: "telegram",
scenarioIds: ["channel-chat-baseline"],
matches: vi.fn(),
create: vi.fn(),
},
@@ -678,7 +677,7 @@ describe("qa cli runtime", () => {
);
});
it("uses the selected live adapter's declared scenarios by default", async () => {
it("uses the selected live adapter's YAML profile by default", async () => {
await runQaSuiteCommand({
channelDriver: "live",
channel: "telegram",
@@ -687,7 +686,9 @@ describe("qa cli runtime", () => {
expect(runQaSuite).toHaveBeenCalledWith(
expect.objectContaining({
adapterOptions: expect.objectContaining({ explicitScenarioSelection: false }),
scenarioIds: ["channel-chat-baseline"],
scenarioIds: listQaScenariosForExecutionProfile("telegram:adapter").map(
(scenario) => scenario.id,
),
}),
);
});

10
extensions/qa-lab/src/cli.runtime.ts
View File

@@ -70,6 +70,7 @@ import {
} from "./run-config.js";
import type { RuntimeId } from "./runtime-parity.js";
import {
listQaScenariosForExecutionProfile,
QA_RUNTIME_PARITY_TIERS,
readQaScenarioPack,
type QaRuntimeParityTier,
@@ -986,13 +987,14 @@ export async function runQaSuiteCommand(opts: QaSuiteCommandOptions) {
if (liveChannelId && !liveAdapterFactory) {
throw new Error(`unknown live QA adapter: ${liveChannelId}`);
}
// liveChannelId exists only for the live driver, and explicit IDs always win.
// This keeps adapter profiles out of non-live and explicit-selection paths.
const liveScenarioIds =
liveAdapterFactory && scenarioIds.length === 0
? [...(liveAdapterFactory.scenarioIds ?? [])]
? listQaScenariosForExecutionProfile(`${liveChannelId}:adapter`).map(
(scenario) => scenario.id,
)
: scenarioIds;
if (liveAdapterFactory && liveScenarioIds.length === 0) {
throw new Error(`live QA adapter ${liveChannelId} does not declare default scenarios`);
}
if (runner !== "host" && runner !== "multipass") {
throw new Error(`--runner must be one of host or multipass, got "${opts.runner}".`);
}

10
extensions/qa-lab/src/coverage-report.test.ts
View File

@@ -165,12 +165,6 @@ describe("qa coverage report", () => {
expect(inventory.secondaryCoverageIdCount).toBeGreaterThan(0);
expect(inventory.overlappingCoverage.length).toBeGreaterThan(0);
expect(inventory.missingCoverage).toStrictEqual([]);
expect(inventory.liveTransportLanes.map((lane) => lane.transportId)).toEqual([
"discord",
"slack",
"telegram",
"whatsapp",
]);
expect(inventory.scorecardTaxonomy.profileCount).toBe(3);
expect(
inventory.scorecardTaxonomy.profiles.find((profile) => profile.id === "smoke-ci"),
@@ -306,10 +300,6 @@ describe("qa coverage report", () => {
);
expect(report).toContain("otel-trace-smoke, docker-prometheus-smoke");
expect(report).toContain("personal-share-safe-diagnostics-artifact");
expect(report).toContain("## Live Transport Lanes");
expect(report).toContain(
"- telegram (telegram): canary: channel-canary, help-command: telegram-help-command, mention-gating: channel-mention-gating; missing baseline: allowlist-block, top-level-reply-shape, restart-resume",
);
expect(report).toContain("## Scorecard Taxonomy");
expect(report).toContain("- Taxonomy: taxonomy.yaml");
expect(report).toContain("- Fulfilled taxonomy categories:");

44
extensions/qa-lab/src/coverage-report.ts
View File

@@ -1,9 +1,5 @@
// Qa Lab plugin module implements coverage report behavior.
import { normalizeStringEntriesLower } from "openclaw/plugin-sdk/string-coerce-runtime";
import {
buildLiveTransportCoverageLaneSummaries,
type LiveTransportCoverageLaneSummary,
} from "./live-transports/shared/live-transport-scenarios.js";
import { QA_SCENARIO_PACKS, type QaSeedScenarioWithSource } from "./scenario-catalog.js";
import {
readQaScorecardTaxonomyReport,
@@ -28,7 +24,7 @@ type QaScenarioSearchMatch = QaCoverageScenarioSummary & {
executionPath?: string;
runtimeParityTier?: string;
requiredProviderMode?: string;
requiredChannelDriver?: string;
driver?: string;
requiredProvider?: string;
requiredModel?: string;
};
@@ -63,7 +59,6 @@ type QaCoverageInventory = {
byTheme: Record<string, QaCoverageIdSummary[]>;
bySurface: Record<string, QaCoverageIdSummary[]>;
scenarioPacks: QaCoverageScenarioPackSummary[];
liveTransportLanes: LiveTransportCoverageLaneSummary[];
scorecardTaxonomy: QaScorecardTaxonomyReport;
};
@@ -168,7 +163,7 @@ function summarizeScenarioSearchMatch(scenario: QaSeedScenarioWithSource): QaSce
...(scenario.execution.kind !== "flow" ? { executionPath: scenario.execution.path } : {}),
runtimeParityTier: scenario.runtimeParityTier,
requiredProviderMode: stringifyConfigValue(config.requiredProviderMode),
requiredChannelDriver: stringifyConfigValue(config.requiredChannelDriver),
driver: scenario.execution.driver,
requiredProvider: stringifyConfigValue(config.requiredProvider),
requiredModel: stringifyConfigValue(config.requiredModel),
};
@@ -300,7 +295,6 @@ export function buildQaCoverageInventory(
byTheme,
bySurface,
scenarioPacks: buildScenarioPackSummaries(scenarios),
liveTransportLanes: buildLiveTransportCoverageLaneSummaries(),
scorecardTaxonomy: readQaScorecardTaxonomyReport(scenarios),
};
}
@@ -314,28 +308,6 @@ function pushCoverageIdLines(lines: string[], coverageIds: readonly QaCoverageId
}
}
function pushLiveTransportLines(
lines: string[],
lanes: readonly LiveTransportCoverageLaneSummary[],
) {
for (const lane of lanes) {
const members = lane.members
.map((member) =>
member.scenarioId
? `${member.standardId}: ${member.scenarioId}`
: `${member.standardId}: always-on`,
)
.join(", ");
const missing =
lane.baselineMissingStandardScenarioIds.length > 0
? lane.baselineMissingStandardScenarioIds.join(", ")
: "none";
lines.push(
`- ${lane.transportId} (${lane.commandName}): ${members}; missing baseline: ${missing}`,
);
}
}
function pushScenarioPackLines(lines: string[], packs: readonly QaCoverageScenarioPackSummary[]) {
for (const pack of packs) {
const missing =
@@ -443,12 +415,6 @@ export function renderQaCoverageMarkdownReport(inventory: QaCoverageInventory):
lines.push("");
}
if (inventory.liveTransportLanes.length > 0) {
lines.push("## Live Transport Lanes", "");
pushLiveTransportLines(lines, inventory.liveTransportLanes);
lines.push("");
}
pushScorecardTaxonomyLines(lines, inventory.scorecardTaxonomy);
if (inventory.overlappingCoverage.length > 0) {
@@ -474,7 +440,7 @@ function formatOptionalScenarioMetadata(match: QaScenarioSearchMatch) {
const metadata = [
match.runtimeParityTier ? `runtimeParityTier=${match.runtimeParityTier}` : "",
match.requiredProviderMode ? `providerMode=${match.requiredProviderMode}` : "",
match.requiredChannelDriver ? `channelDriver=${match.requiredChannelDriver}` : "",
match.driver ? `channelDriver=${match.driver}` : "",
match.requiredProvider ? `provider=${match.requiredProvider}` : "",
match.requiredModel ? `model=${match.requiredModel}` : "",
].filter(Boolean);
@@ -487,7 +453,7 @@ function uniqueScenarioValues(values: (string | undefined)[]) {
function formatSuiteCommand(matches: readonly QaScenarioSearchMatch[]) {
const scenarioArgs = matches.map((match) => `--scenario ${match.id}`).join(" ");
const requiredDrivers = uniqueScenarioValues(matches.map((match) => match.requiredChannelDriver));
const requiredDrivers = uniqueScenarioValues(matches.map((match) => match.driver));
const channels = uniqueScenarioValues(matches.map((match) => match.channel));
const selectedDriver =
requiredDrivers.length === 1 ? requiredDrivers[0] : channels.length === 1 ? "live" : undefined;
@@ -500,7 +466,7 @@ function formatSuiteCommand(matches: readonly QaScenarioSearchMatch[]) {
function scenarioMatchCommandGroups(matches: readonly QaScenarioSearchMatch[]) {
const groups = new Map<string, QaScenarioSearchMatch[]>();
for (const match of matches) {
const key = JSON.stringify([match.executionKind, match.channel, match.requiredChannelDriver]);
const key = JSON.stringify([match.executionKind, match.channel, match.driver]);
const group = groups.get(key) ?? [];
group.push(match);
groups.set(key, group);

6
extensions/qa-lab/src/live-transports/discord/cli.runtime.test.ts
View File

@@ -13,8 +13,8 @@ vi.mock("../shared/live-transport-cli.runtime.js", () => ({
}),
}));
import { listQaScenariosForExecutionProfile } from "../../scenario-catalog.js";
import { runQaDiscordCommand } from "./cli.runtime.js";
import { DISCORD_QA_DEFAULT_SCENARIO_IDS } from "./profiles.js";
describe("QA Lab Discord CLI runtime", () => {
beforeEach(() => {
@@ -48,7 +48,9 @@ describe("QA Lab Discord CLI runtime", () => {
channelDriver: "live",
channel: "discord",
concurrency: 1,
scenarioIds: [...DISCORD_QA_DEFAULT_SCENARIO_IDS],
scenarioIds: listQaScenariosForExecutionProfile("discord:adapter").map(
(scenario) => scenario.id,
),
sutAccountId: "discord-sut",
credentialSource: "convex",
credentialRole: "ci",

2
extensions/qa-lab/src/live-transports/discord/cli.ts
View File

@@ -5,7 +5,6 @@ import {
type LiveTransportQaCliRegistration,
type LiveTransportQaCommandOptions,
} from "../shared/live-transport-cli.js";
import { DISCORD_QA_DEFAULT_SCENARIO_IDS } from "./profiles.js";
type DiscordQaCliRuntime = typeof import("./cli.runtime.js");
type DiscordQaAdapterRuntime = typeof import("./adapter.runtime.js");
@@ -27,7 +26,6 @@ export const discordQaCliRegistration: LiveTransportQaCliRegistration =
commandName: "discord",
adapterFactory: {
id: "discord",
scenarioIds: DISCORD_QA_DEFAULT_SCENARIO_IDS,
matches: ({ channelId, driver }) => driver === "live" && channelId === "discord",
async create(context) {
return await (await loadDiscordQaAdapterRuntime()).createDiscordQaTransportAdapter(context);

14
extensions/qa-lab/src/live-transports/discord/discord-live.runtime.test.ts
View File

@@ -1,9 +1,5 @@
// Qa Lab tests cover discord live plugin behavior.
import type { OpenClawConfig } from "openclaw/plugin-sdk/config-contracts";
import {
LIVE_TRANSPORT_BASELINE_STANDARD_SCENARIO_IDS,
findMissingLiveTransportStandardScenarios,
} from "openclaw/plugin-sdk/qa-live-transport-scenarios";
import { afterEach, describe, expect, it, vi } from "vitest";
import { discordQaScenarioSupport } from "./discord-live.runtime.js";
@@ -474,16 +470,6 @@ describe("discord live qa runtime", () => {
);
});
it("tracks Discord live coverage against the shared transport contract", () => {
expect(testing.DISCORD_QA_STANDARD_SCENARIO_IDS).toEqual(["canary", "mention-gating"]);
expect(
findMissingLiveTransportStandardScenarios({
coveredStandardScenarioIds: testing.DISCORD_QA_STANDARD_SCENARIO_IDS,
expectedStandardScenarioIds: LIVE_TRANSPORT_BASELINE_STANDARD_SCENARIO_IDS,
}),
).toEqual(["allowlist-block", "top-level-reply-shape", "restart-resume"]);
});
it("lists Discord application commands through the REST API", async () => {
vi.stubGlobal(
"fetch",

43
extensions/qa-lab/src/live-transports/discord/discord-live.runtime.ts
View File

@@ -18,12 +18,8 @@ import { chromium } from "playwright-core";
import { z } from "zod";
import { startQaGatewayChild } from "../../gateway-child.js";
import { isTruthyOptIn } from "../../mantis-options.runtime.js";
import { listQaScenariosForExecutionProfile } from "../../scenario-catalog.js";
import { assertLiveScenarioReply as assertDiscordScenarioReply } from "../shared/live-scenario-reply.js";
import {
collectLiveTransportStandardScenarioCoverage,
selectLiveTransportScenarios,
type LiveTransportScenarioDefinition,
} from "../shared/live-transport-scenarios.js";
type DiscordQaRuntimeEnv = {
guildId: string;
@@ -69,7 +65,10 @@ type DiscordQaScenarioRun =
replyContent: string;
};
type DiscordQaScenarioDefinition = LiveTransportScenarioDefinition<DiscordQaScenarioId> & {
type DiscordQaScenarioDefinition = {
id: DiscordQaScenarioId;
title: string;
timeoutMs: number;
buildRun: (sutApplicationId: string) => DiscordQaScenarioRun;
};
@@ -171,7 +170,6 @@ type DiscordQaScenarioResult = {
artifactPaths?: Record<string, string>;
id: string;
title: string;
standardId?: string;
status: "pass" | "fail";
details: string;
requestStartedAt?: string;
@@ -239,7 +237,6 @@ const DISCORD_QA_ENV_KEYS = [
const DISCORD_QA_SCENARIOS: DiscordQaScenarioDefinition[] = [
{
id: "discord-canary",
standardId: "canary",
title: "Discord canary echo",
timeoutMs: 45_000,
buildRun: (sutApplicationId) => {
@@ -255,7 +252,6 @@ const DISCORD_QA_SCENARIOS: DiscordQaScenarioDefinition[] = [
},
{
id: "discord-mention-gating",
standardId: "mention-gating",
title: "Discord unmentioned message does not trigger",
timeoutMs: 8_000,
buildRun: () => {
@@ -318,17 +314,6 @@ const DISCORD_QA_SCENARIOS: DiscordQaScenarioDefinition[] = [
},
];
const DISCORD_QA_DEFAULT_SCENARIOS = DISCORD_QA_SCENARIOS.filter(
(scenario) =>
scenario.id !== "discord-status-reactions-tool-only" &&
scenario.id !== "discord-voice-autojoin" &&
scenario.id !== "discord-thread-reply-filepath-attachment",
);
const DISCORD_QA_STANDARD_SCENARIO_IDS = collectLiveTransportStandardScenarioCoverage({
scenarios: DISCORD_QA_SCENARIOS,
});
const discordQaCredentialPayloadSchema = z.object({
guildId: z.string().trim().min(1),
channelId: z.string().trim().min(1),
@@ -1233,7 +1218,6 @@ async function runDiscordThreadReplyFilePathAttachmentScenario(params: {
return {
id: params.scenario.id,
title: params.scenario.title,
standardId: params.scenario.standardId,
status,
details:
status === "pass"
@@ -1365,12 +1349,16 @@ function buildObservedMessagesArtifact(params: {
}
function findScenario(ids?: string[]) {
const scenarios = ids && ids.length > 0 ? DISCORD_QA_SCENARIOS : DISCORD_QA_DEFAULT_SCENARIOS;
return selectLiveTransportScenarios({
ids,
laneLabel: "Discord",
scenarios,
});
const requestedIds =
ids && ids.length > 0
? ids
: listQaScenariosForExecutionProfile("discord:adapter").map((scenario) => scenario.id);
const scenariosById = new Map(DISCORD_QA_SCENARIOS.map((scenario) => [scenario.id, scenario]));
const missingIds = requestedIds.filter((id) => !scenariosById.has(id as DiscordQaScenarioId));
if (missingIds.length > 0) {
throw new Error(`unknown Discord QA scenario id(s): ${missingIds.join(", ")}`);
}
return requestedIds.map((id) => scenariosById.get(id as DiscordQaScenarioId)!);
}
function matchesDiscordScenarioReply(params: {
@@ -1421,7 +1409,6 @@ async function assertDiscordApplicationCommandsRegistered(params: {
const testing = {
DISCORD_QA_SCENARIOS,
DISCORD_QA_STANDARD_SCENARIO_IDS,
collectSeenReactionSequence,
assertDiscordScenarioReply,
assertDiscordApplicationCommandsRegistered,

20
extensions/qa-lab/src/live-transports/discord/profiles.ts
View File

@@ -1,19 +1,7 @@
const DISCORD_QA_ALL_SCENARIO_IDS = [
"discord-canary",
"discord-mention-gating",
"discord-native-help-command-registration",
"discord-voice-autojoin",
"discord-status-reactions-tool-only",
"discord-thread-reply-filepath-attachment",
] as const;
export const DISCORD_QA_DEFAULT_SCENARIO_IDS = DISCORD_QA_ALL_SCENARIO_IDS.filter(
(scenarioId) =>
scenarioId !== "discord-voice-autojoin" &&
scenarioId !== "discord-status-reactions-tool-only" &&
scenarioId !== "discord-thread-reply-filepath-attachment",
);
import { listQaScenariosForExecutionProfile } from "../../scenario-catalog.js";
export function resolveDiscordQaScenarioIds(scenarioIds?: readonly string[]) {
return scenarioIds?.length ? [...scenarioIds] : [...DISCORD_QA_DEFAULT_SCENARIO_IDS];
return scenarioIds?.length
? [...scenarioIds]
: listQaScenariosForExecutionProfile("discord:adapter").map((scenario) => scenario.id);
}

31
extensions/qa-lab/src/live-transports/live-transport-adapters.test.ts
View File

@@ -3,6 +3,7 @@ import { describe, expect, it, vi } from "vitest";
import { createQaBusState } from "../bus-state.js";
import { createQaChannelTransport } from "../qa-channel-transport.js";
import { createQaTransportAdapter } from "../qa-transport-registry.js";
import { listQaScenariosForExecutionProfile } from "../scenario-catalog.js";
const { createSlack, createTelegram, createWhatsApp } = vi.hoisted(() => ({
createSlack: vi.fn(),
@@ -19,10 +20,8 @@ vi.mock("./whatsapp/adapter.runtime.js", () => ({
}));
import { slackQaCliRegistration } from "./slack/cli.js";
import { SLACK_QA_DEFAULT_SCENARIO_IDS } from "./slack/profiles.js";
import { telegramQaCliRegistration } from "./telegram/cli.js";
import { whatsappQaCliRegistration } from "./whatsapp/cli.js";
import { resolveWhatsAppQaScenarioIds } from "./whatsapp/profiles.js";
const slackQaAdapterFactory = slackQaCliRegistration.adapterFactory;
const telegramQaAdapterFactory = telegramQaCliRegistration.adapterFactory;
@@ -38,14 +37,30 @@ const factories = [
] as const;
describe("live transport adapter factories", () => {
it("assigns the canonical live scenario defaults to Slack", () => {
expect(slackQaAdapterFactory.scenarioIds).toEqual(SLACK_QA_DEFAULT_SCENARIO_IDS);
it("selects Slack generic defaults from the YAML adapter profile", () => {
expect(
listQaScenariosForExecutionProfile("slack:adapter").map((scenario) => scenario.id),
).toEqual([
"channel-chat-baseline",
"channel-canary",
"channel-mention-gating",
"channel-top-level-reply-shape",
"thread-follow-up",
"thread-isolation",
]);
});
it("assigns the canonical live-frontier scenario defaults to WhatsApp", () => {
expect(whatsappQaAdapterFactory.scenarioIds).toEqual(
resolveWhatsAppQaScenarioIds({ providerMode: "live-frontier" }),
);
it("selects WhatsApp DM-safe defaults from the YAML adapter profile", () => {
expect(
listQaScenariosForExecutionProfile("whatsapp:adapter").map((scenario) => scenario.id),
).toEqual([
"dm-chat-baseline",
"channel-canary",
"channel-dm-group-routing",
"channel-mention-gating",
"channel-top-level-reply-shape",
"whatsapp-help-command",
]);
});
it.each([

18
extensions/qa-lab/src/live-transports/matrix/cli.test.ts
View File

@@ -50,24 +50,6 @@ describe("QA Lab Matrix CLI registration", () => {
stdoutSpy.mockRestore();
});
it("keeps generic Matrix suite defaults in the repo-backed flow catalog", () => {
expect(matrixQaCliRegistration.adapterFactory?.scenarioIds).toEqual([
"channel-chat-baseline",
"channel-canary",
"channel-dm-group-routing",
"channel-mention-gating",
"channel-sender-allowlist",
"channel-top-level-reply-shape",
"channel-secondary-conversation-isolation",
"channel-multi-actor-ordering",
"thread-follow-up",
"thread-isolation",
"thread-reply-override",
"dm-shared-session",
"dm-per-room-session",
]);
});
it("exposes only QA Lab selector flags", () => {
const qa = new Command();
matrixQaCliRegistration.register(qa);

16
extensions/qa-lab/src/live-transports/matrix/cli.ts
View File

@@ -8,21 +8,6 @@ import {
} from "../shared/live-transport-cli.js";
const DISABLE_MATRIX_QA_FORCE_EXIT_ENV = "OPENCLAW_QA_MATRIX_DISABLE_FORCE_EXIT";
const MATRIX_QA_REPO_FLOW_SCENARIO_IDS = [
"channel-chat-baseline",
"channel-canary",
"channel-dm-group-routing",
"channel-mention-gating",
"channel-sender-allowlist",
"channel-top-level-reply-shape",
"channel-secondary-conversation-isolation",
"channel-multi-actor-ordering",
"thread-follow-up",
"thread-isolation",
"thread-reply-override",
"dm-shared-session",
"dm-per-room-session",
] as const;
const loadMatrixQaCliRuntime = createLazyCliRuntimeLoader<typeof import("./cli.runtime.js")>(
() => import("./cli.runtime.js"),
@@ -71,7 +56,6 @@ async function runQaMatrix(opts: LiveTransportQaCommandOptions) {
const matrixQaAdapterFactory: NonNullable<LiveTransportQaCliRegistration["adapterFactory"]> = {
id: "matrix",
scenarioIds: MATRIX_QA_REPO_FLOW_SCENARIO_IDS,
matches: ({ channelId, driver }) => driver === "live" && channelId === "matrix",
async create(context) {
return await (await loadMatrixQaAdapterRuntime()).createMatrixQaTransportAdapter(context);

168
extensions/qa-lab/src/live-transports/matrix/profiles.ts
View File

@@ -1,166 +1,28 @@
const MATRIX_QA_RELEASE_SCENARIO_IDS = [
"channel-chat-baseline",
"matrix-allowlist-hot-reload",
import { listQaScenariosForExecutionProfile } from "../../scenario-catalog.js";
const MATRIX_QA_PROFILES = [
"all",
"fast",
"release",
"transport",
"media",
"e2ee-smoke",
"e2ee-deep",
"e2ee-cli",
] as const;
const MATRIX_QA_FAST_SCENARIO_IDS = [
"channel-chat-baseline",
"thread-follow-up",
"thread-isolation",
"channel-top-level-reply-shape",
"matrix-reaction-notification",
"matrix-approval-exec-metadata-single-event",
"matrix-approval-exec-metadata-chunked",
"channel-mention-gating",
"matrix-allowbots-default-block",
"matrix-allowbots-mentions-mentioned-room",
"channel-sender-allowlist",
"matrix-e2ee-basic-reply",
] as const;
const MATRIX_QA_TRANSPORT_SCENARIO_IDS = [
"thread-follow-up",
"matrix-thread-root-preservation",
"matrix-thread-nested-reply-shape",
"thread-isolation",
"channel-top-level-reply-shape",
"thread-reply-override",
"matrix-room-partial-streaming-preview",
"matrix-room-quiet-streaming-preview",
"matrix-room-tool-progress-preview",
"matrix-room-tool-progress-command-preview",
"matrix-room-tool-progress-preview-opt-out",
"matrix-room-tool-progress-error",
"matrix-room-tool-progress-mention-safety",
"dm-chat-baseline",
"dm-shared-session",
"matrix-dm-thread-reply-override",
"dm-per-room-session",
"matrix-room-autojoin-invite",
"channel-secondary-conversation-isolation",
"matrix-secondary-room-open-trigger",
"matrix-reaction-notification",
"matrix-reaction-threaded",
"matrix-reaction-not-a-reply",
"matrix-reaction-redaction-observed",
"matrix-approval-exec-metadata-single-event",
"matrix-approval-exec-metadata-chunked",
"matrix-approval-plugin-metadata-single-event",
"matrix-approval-deny-reaction",
"matrix-approval-thread-target",
"matrix-approval-channel-target-both",
"matrix-initial-catchup-then-incremental",
"matrix-stale-sync-replay-dedupe",
"matrix-room-membership-loss",
"matrix-homeserver-restart-resume",
"channel-mention-gating",
"matrix-allowbots-default-block",
"matrix-allowbots-true-unmentioned-open-room",
"matrix-allowbots-mentions-mentioned-room",
"matrix-allowbots-mentions-unmentioned-open-room-block",
"matrix-allowbots-mentions-dm-unmentioned",
"matrix-allowbots-room-override-blocks-account-true",
"matrix-allowbots-room-override-enables-account-off",
"matrix-allowbots-self-sender-ignored",
"matrix-mxid-prefixed-command-block",
"matrix-mention-metadata-spoof-block",
"matrix-allowlist-hot-reload",
"channel-sender-allowlist",
"channel-multi-actor-ordering",
"matrix-inbound-edit-ignored",
"matrix-inbound-edit-no-duplicate-trigger",
] as const;
const MATRIX_QA_MEDIA_SCENARIO_IDS = [
"matrix-room-image-understanding-attachment",
"matrix-room-generated-image-delivery",
"matrix-media-type-coverage",
"matrix-voice-preflight-mention",
"matrix-attachment-only-ignored",
"matrix-unsupported-media-safe",
"matrix-e2ee-media-image",
] as const;
const MATRIX_QA_E2EE_SMOKE_SCENARIO_IDS = [
"matrix-e2ee-basic-reply",
"matrix-e2ee-thread-follow-up",
"matrix-e2ee-bootstrap-success",
"matrix-e2ee-recovery-key-lifecycle",
"matrix-e2ee-recovery-owner-verification-required",
"matrix-e2ee-restart-resume",
"matrix-e2ee-artifact-redaction",
"matrix-e2ee-key-bootstrap-failure",
] as const;
const MATRIX_QA_E2EE_DEEP_SCENARIO_IDS = [
"matrix-e2ee-state-after-missing-encryption",
"matrix-e2ee-state-loss-external-recovery-key",
"matrix-e2ee-state-loss-stored-recovery-key",
"matrix-e2ee-state-loss-no-recovery-key",
"matrix-e2ee-stale-recovery-key-after-backup-reset",
"matrix-e2ee-server-backup-deleted-local-state-intact",
"matrix-e2ee-server-backup-deleted-local-reupload-restores",
"matrix-e2ee-corrupt-crypto-idb-snapshot",
"matrix-e2ee-server-device-deleted-local-state-intact",
"matrix-e2ee-server-device-deleted-relogin-recovers",
"matrix-e2ee-sync-state-loss-crypto-intact",
"matrix-e2ee-history-exists-backup-empty",
"matrix-e2ee-device-sas-verification",
"matrix-e2ee-qr-verification",
"matrix-e2ee-stale-device-hygiene",
"matrix-e2ee-dm-sas-verification",
"matrix-e2ee-verification-notice-no-trigger",
"matrix-e2ee-wrong-account-recovery-key",
] as const;
const MATRIX_QA_E2EE_CLI_SCENARIO_IDS = [
"matrix-e2ee-cli-account-add-enable-e2ee",
"matrix-e2ee-cli-encryption-setup",
"matrix-e2ee-cli-encryption-setup-idempotent",
"matrix-e2ee-cli-encryption-setup-bootstrap-failure",
"matrix-e2ee-cli-recovery-key-setup",
"matrix-e2ee-cli-recovery-key-invalid",
"matrix-e2ee-cli-encryption-setup-multi-account",
"matrix-e2ee-cli-setup-then-gateway-reply",
"matrix-e2ee-cli-self-verification",
] as const;
const MATRIX_QA_ALL_SCENARIO_IDS = Array.from(
new Set([
"channel-chat-baseline",
...MATRIX_QA_TRANSPORT_SCENARIO_IDS,
...MATRIX_QA_MEDIA_SCENARIO_IDS,
...MATRIX_QA_E2EE_SMOKE_SCENARIO_IDS,
...MATRIX_QA_E2EE_DEEP_SCENARIO_IDS,
...MATRIX_QA_E2EE_CLI_SCENARIO_IDS,
]),
);
const MATRIX_QA_PROFILE_SCENARIO_IDS = {
all: MATRIX_QA_ALL_SCENARIO_IDS,
fast: MATRIX_QA_FAST_SCENARIO_IDS,
release: MATRIX_QA_RELEASE_SCENARIO_IDS,
transport: MATRIX_QA_TRANSPORT_SCENARIO_IDS,
media: MATRIX_QA_MEDIA_SCENARIO_IDS,
"e2ee-smoke": MATRIX_QA_E2EE_SMOKE_SCENARIO_IDS,
"e2ee-deep": MATRIX_QA_E2EE_DEEP_SCENARIO_IDS,
"e2ee-cli": MATRIX_QA_E2EE_CLI_SCENARIO_IDS,
} as const;
export function resolveMatrixQaScenarioIds(params: {
profile?: string;
scenarioIds?: readonly string[];
}): string[] {
if (params.scenarioIds && params.scenarioIds.length > 0) {
if (params.scenarioIds?.length) {
return [...params.scenarioIds];
}
const profile = params.profile?.trim() || "all";
if (!(profile in MATRIX_QA_PROFILE_SCENARIO_IDS)) {
if (!MATRIX_QA_PROFILES.some((candidate) => candidate === profile)) {
throw new Error(
`Unknown QA Lab Matrix profile "${profile}". Expected one of: all, fast, release, transport, media, e2ee-smoke, e2ee-deep, e2ee-cli.`,
`Unknown QA Lab Matrix profile "${profile}". Expected one of: ${MATRIX_QA_PROFILES.join(", ")}.`,
);
}
return [
...MATRIX_QA_PROFILE_SCENARIO_IDS[profile as keyof typeof MATRIX_QA_PROFILE_SCENARIO_IDS],
];
return listQaScenariosForExecutionProfile(`matrix:${profile}`).map((scenario) => scenario.id);
}

1
extensions/qa-lab/src/live-transports/shared/live-transport-result.ts
View File

@@ -13,7 +13,6 @@ export type LiveTransportCheckResult = {
title: string;
status: "pass" | "fail";
details: string;
coverageIds?: readonly string[];
timing?: QaEvidenceTiming;
rttMs?: number;
requestStartedAt?: string;

138
extensions/qa-lab/src/live-transports/shared/live-transport-scenarios.test.ts
View File

@@ -1,138 +0,0 @@
// Qa Lab tests cover live transport scenarios plugin behavior.
import fs from "node:fs";
import { fileURLToPath } from "node:url";
import {
LIVE_TRANSPORT_BASELINE_STANDARD_SCENARIO_IDS,
findMissingLiveTransportStandardScenarios,
} from "openclaw/plugin-sdk/qa-live-transport-scenarios";
import { describe, expect, it } from "vitest";
import {
buildLiveTransportCoverageLaneSummaries,
collectLiveTransportStandardScenarioCoverage,
selectLiveTransportScenarios,
} from "./live-transport-scenarios.js";
describe("live transport scenario helpers", () => {
it("uses the public live transport scenario SDK seam", () => {
const source = fs.readFileSync(
fileURLToPath(new URL("./live-transport-scenarios.ts", import.meta.url)),
"utf8",
);
expect(source).toContain("openclaw/plugin-sdk/qa-live-transport-scenarios");
expect(source).not.toContain("openclaw/plugin-sdk/qa-runtime");
});
it("keeps the repo-wide baseline contract ordered", () => {
expect(LIVE_TRANSPORT_BASELINE_STANDARD_SCENARIO_IDS).toEqual([
"canary",
"mention-gating",
"allowlist-block",
"top-level-reply-shape",
"restart-resume",
]);
});
it("selects requested scenarios and reports unknown ids with the lane label", () => {
const definitions = [
{ id: "alpha", timeoutMs: 1_000, title: "alpha" },
{ id: "beta", timeoutMs: 1_000, title: "beta" },
] as const;
expect(
selectLiveTransportScenarios({
ids: ["beta"],
laneLabel: "Demo",
scenarios: definitions,
}),
).toEqual([definitions[1]]);
expect(() =>
selectLiveTransportScenarios({
ids: ["alpha", "missing"],
laneLabel: "Demo",
scenarios: definitions,
}),
).toThrow("unknown Demo QA scenario id(s): missing");
});
it("dedupes always-on and scenario-backed standard coverage", () => {
const covered = collectLiveTransportStandardScenarioCoverage({
alwaysOnStandardScenarioIds: ["canary"],
scenarios: [
{
id: "scenario-1",
standardId: "mention-gating",
timeoutMs: 1_000,
title: "mention",
},
{
id: "scenario-2",
standardId: "mention-gating",
timeoutMs: 1_000,
title: "mention again",
},
{
id: "scenario-3",
standardId: "restart-resume",
timeoutMs: 1_000,
title: "restart",
},
],
});
expect(covered).toEqual(["canary", "mention-gating", "restart-resume"]);
expect(
findMissingLiveTransportStandardScenarios({
coveredStandardScenarioIds: covered,
expectedStandardScenarioIds: LIVE_TRANSPORT_BASELINE_STANDARD_SCENARIO_IDS,
}),
).toEqual(["allowlist-block", "top-level-reply-shape"]);
});
it("summarizes live transport lane membership for coverage reports", () => {
const lanes = buildLiveTransportCoverageLaneSummaries();
expect(lanes.map((lane) => lane.transportId)).toEqual([
"discord",
"slack",
"telegram",
"whatsapp",
]);
expect(lanes.find((lane) => lane.transportId === "telegram")?.members).toContainEqual({
standardId: "canary",
scenarioId: "channel-canary",
});
expect(lanes.find((lane) => lane.transportId === "slack")?.members).toContainEqual({
standardId: "restart-resume",
scenarioId: "slack-restart-resume",
});
expect(lanes.find((lane) => lane.transportId === "whatsapp")?.members).toContainEqual({
standardId: "allowlist-block",
scenarioId: "whatsapp-group-allowlist-block",
});
expect(lanes.find((lane) => lane.transportId === "whatsapp")?.members).toContainEqual({
standardId: "restart-resume",
scenarioId: "whatsapp-restart-resume",
});
expect(
lanes.find((lane) => lane.transportId === "discord")?.baselineMissingStandardScenarioIds,
).toEqual(["allowlist-block", "top-level-reply-shape", "restart-resume"]);
expect(
lanes.find((lane) => lane.transportId === "whatsapp")?.baselineMissingStandardScenarioIds,
).toEqual([]);
});
it("keeps Telegram runtime coverage represented in its mixed-owner lane", () => {
const lanes = new Map(
buildLiveTransportCoverageLaneSummaries().map((lane) => [
lane.transportId,
lane.standardScenarioIds,
]),
);
expect(lanes.get("telegram")).toEqual(
expect.arrayContaining(["canary", "help-command", "mention-gating"]),
);
});
});

107
extensions/qa-lab/src/live-transports/shared/live-transport-scenarios.ts
View File

@@ -1,107 +0,0 @@
// Qa Lab plugin module implements live transport scenarios behavior.
import {
LIVE_TRANSPORT_BASELINE_STANDARD_SCENARIO_IDS,
collectLiveTransportStandardScenarioCoverage,
findMissingLiveTransportStandardScenarios,
type LiveTransportScenarioDefinition,
type LiveTransportStandardScenarioId,
} from "openclaw/plugin-sdk/qa-live-transport-scenarios";
export {
collectLiveTransportStandardScenarioCoverage,
selectLiveTransportScenarios,
type LiveTransportScenarioDefinition,
} from "openclaw/plugin-sdk/qa-live-transport-scenarios";
type LiveTransportCoverageMember = {
scenarioId?: string;
standardId: LiveTransportStandardScenarioId;
};
type LiveTransportCoverageLane = {
commandName: string;
members: readonly LiveTransportCoverageMember[];
transportId: string;
};
export type LiveTransportCoverageLaneSummary = {
baselineMissingStandardScenarioIds: LiveTransportStandardScenarioId[];
commandName: string;
memberCount: number;
members: LiveTransportCoverageMember[];
standardScenarioIds: LiveTransportStandardScenarioId[];
transportId: string;
};
const LIVE_TRANSPORT_COVERAGE_LANES: readonly LiveTransportCoverageLane[] = [
{
transportId: "discord",
commandName: "discord",
members: [
{ standardId: "canary", scenarioId: "discord-canary" },
{ standardId: "mention-gating", scenarioId: "discord-mention-gating" },
],
},
{
transportId: "slack",
commandName: "slack",
members: [
{ standardId: "canary", scenarioId: "slack-canary" },
{ standardId: "mention-gating", scenarioId: "slack-mention-gating" },
{ standardId: "allowlist-block", scenarioId: "slack-allowlist-block" },
{ standardId: "top-level-reply-shape", scenarioId: "slack-top-level-reply-shape" },
{ standardId: "restart-resume", scenarioId: "slack-restart-resume" },
],
},
{
transportId: "telegram",
commandName: "telegram",
members: [
{ standardId: "canary", scenarioId: "channel-canary" },
{ standardId: "help-command", scenarioId: "telegram-help-command" },
{ standardId: "mention-gating", scenarioId: "channel-mention-gating" },
],
},
{
transportId: "whatsapp",
commandName: "whatsapp",
members: [
{ standardId: "canary", scenarioId: "whatsapp-canary" },
{ standardId: "mention-gating", scenarioId: "whatsapp-mention-gating" },
{ standardId: "top-level-reply-shape", scenarioId: "whatsapp-top-level-reply-shape" },
{ standardId: "restart-resume", scenarioId: "whatsapp-restart-resume" },
{ standardId: "help-command", scenarioId: "whatsapp-help-command" },
{ standardId: "quote-reply", scenarioId: "whatsapp-reply-to-message" },
{ standardId: "quote-reply", scenarioId: "whatsapp-group-reply-to-message" },
{ standardId: "reaction-observation", scenarioId: "whatsapp-status-reactions" },
{ standardId: "allowlist-block", scenarioId: "whatsapp-group-allowlist-block" },
],
},
] as const;
export function buildLiveTransportCoverageLaneSummaries(
lanes: readonly LiveTransportCoverageLane[] = LIVE_TRANSPORT_COVERAGE_LANES,
): LiveTransportCoverageLaneSummary[] {
return lanes
.map((lane) => {
const scenarios: LiveTransportScenarioDefinition[] = lane.members.map((member) => ({
id: member.scenarioId ?? `${lane.transportId}:${member.standardId}`,
standardId: member.standardId,
timeoutMs: 0,
title: member.standardId,
}));
const standardScenarioIds = collectLiveTransportStandardScenarioCoverage({ scenarios });
return {
baselineMissingStandardScenarioIds: findMissingLiveTransportStandardScenarios({
coveredStandardScenarioIds: standardScenarioIds,
expectedStandardScenarioIds: LIVE_TRANSPORT_BASELINE_STANDARD_SCENARIO_IDS,
}),
commandName: lane.commandName,
memberCount: lane.members.length,
members: [...lane.members],
standardScenarioIds,
transportId: lane.transportId,
};
})
.toSorted((left, right) => left.transportId.localeCompare(right.transportId));
}

6
extensions/qa-lab/src/live-transports/slack/cli.runtime.test.ts
View File

@@ -13,8 +13,8 @@ vi.mock("../shared/live-transport-cli.runtime.js", () => ({
}),
}));
import { listQaScenariosForExecutionProfile } from "../../scenario-catalog.js";
import { runQaSlackCommand } from "./cli.runtime.js";
import { SLACK_QA_DEFAULT_SCENARIO_IDS } from "./profiles.js";
describe("QA Lab Slack CLI runtime", () => {
beforeEach(() => {
@@ -48,7 +48,9 @@ describe("QA Lab Slack CLI runtime", () => {
channelDriver: "live",
channel: "slack",
concurrency: 1,
scenarioIds: [...SLACK_QA_DEFAULT_SCENARIO_IDS],
scenarioIds: listQaScenariosForExecutionProfile("slack:default").map(
(scenario) => scenario.id,
),
sutAccountId: "slack-sut",
credentialSource: "convex",
credentialRole: "ci",

2
extensions/qa-lab/src/live-transports/slack/cli.ts
View File

@@ -5,7 +5,6 @@ import {
type LiveTransportQaCliRegistration,
type LiveTransportQaCommandOptions,
} from "../shared/live-transport-cli.js";
import { SLACK_QA_DEFAULT_SCENARIO_IDS } from "./profiles.js";
type SlackQaAdapterRuntime = typeof import("./adapter.runtime.js");
type SlackQaCliRuntime = typeof import("./cli.runtime.js");
@@ -23,7 +22,6 @@ async function runQaSlack(opts: LiveTransportQaCommandOptions) {
const slackQaAdapterFactory: NonNullable<LiveTransportQaCliRegistration["adapterFactory"]> = {
id: "slack",
scenarioIds: SLACK_QA_DEFAULT_SCENARIO_IDS,
matches: ({ channelId, driver }) => driver === "live" && channelId === "slack",
async create(context) {
return await (await loadSlackQaAdapterRuntime()).createSlackQaTransportAdapter(context);

37
extensions/qa-lab/src/live-transports/slack/profiles.ts
View File

@@ -1,36 +1,7 @@
const SLACK_QA_ALL_SCENARIO_IDS = [
"slack-canary",
"slack-mention-gating",
"slack-allowlist-block",
"slack-channel-disabled-warning",
"slack-top-level-reply-shape",
"slack-progress-commentary-true",
"slack-progress-commentary-false",
"slack-progress-commentary-omitted",
"slack-progress-commentary-verbose-dedupe",
"slack-chart-presentation-native",
"slack-table-presentation-native",
"slack-table-invalid-blocks-fallback",
"slack-reaction-glyph-native",
"slack-approval-exec-native",
"slack-approval-plugin-native",
"slack-codex-approval-exec-native",
"slack-codex-approval-plugin-native",
] as const;
const SLACK_QA_EXPLICIT_SCENARIO_IDS = new Set<string>([
"slack-channel-disabled-warning",
"slack-progress-commentary-true",
"slack-progress-commentary-false",
"slack-progress-commentary-omitted",
"slack-progress-commentary-verbose-dedupe",
"slack-table-invalid-blocks-fallback",
]);
export const SLACK_QA_DEFAULT_SCENARIO_IDS = SLACK_QA_ALL_SCENARIO_IDS.filter(
(scenarioId) => !SLACK_QA_EXPLICIT_SCENARIO_IDS.has(scenarioId),
);
import { listQaScenariosForExecutionProfile } from "../../scenario-catalog.js";
export function resolveSlackQaScenarioIds(scenarioIds?: readonly string[]) {
return scenarioIds?.length ? [...scenarioIds] : [...SLACK_QA_DEFAULT_SCENARIO_IDS];
return scenarioIds?.length
? [...scenarioIds]
: listQaScenariosForExecutionProfile("slack:default").map((scenario) => scenario.id);
}

7
extensions/qa-lab/src/live-transports/slack/slack-live.contracts.ts
View File

@@ -5,7 +5,6 @@ import { z } from "zod";
import type { startQaGatewayChild } from "../../gateway-child.js";
import { splitQaModelRef } from "../../model-selection.js";
import type { RuntimeId } from "../../runtime-parity.js";
import type { LiveTransportScenarioDefinition } from "../shared/live-transport-scenarios.js";
export type SlackQaRuntimeEnv = {
channelId: string;
@@ -227,10 +226,12 @@ export type SlackQaScenarioContext = {
waitForReady: () => Promise<void>;
};
export type SlackQaScenarioDefinition = LiveTransportScenarioDefinition<SlackQaScenarioId> & {
export type SlackQaScenarioDefinition = {
id: SlackQaScenarioId;
title: string;
timeoutMs: number;
buildRun: (sutUserId: string) => SlackQaScenarioRun;
configOverrides?: SlackQaConfigOverrides;
defaultEnabled?: boolean;
forcedRuntime?: RuntimeId;
};

10
extensions/qa-lab/src/live-transports/slack/slack-live.scenarios.ts
View File

@@ -24,7 +24,6 @@ import {
const SLACK_QA_SCENARIOS: SlackQaScenarioDefinition[] = [
{
id: "slack-canary",
standardId: "canary",
title: "Slack canary echo",
timeoutMs: 45_000,
buildRun: (sutUserId) => {
@@ -38,7 +37,6 @@ const SLACK_QA_SCENARIOS: SlackQaScenarioDefinition[] = [
},
{
id: "slack-mention-gating",
standardId: "mention-gating",
title: "Slack unmentioned bot message does not trigger",
timeoutMs: 8_000,
buildRun: () => {
@@ -52,7 +50,6 @@ const SLACK_QA_SCENARIOS: SlackQaScenarioDefinition[] = [
},
{
id: "slack-allowlist-block",
standardId: "allowlist-block",
title: "Slack non-allowlisted sender does not trigger",
timeoutMs: 8_000,
configOverrides: {
@@ -72,7 +69,6 @@ const SLACK_QA_SCENARIOS: SlackQaScenarioDefinition[] = [
id: "slack-channel-disabled-warning",
title: "Slack disabled channel warns and does not trigger",
timeoutMs: 8_000,
defaultEnabled: false,
configOverrides: { channelEnabled: false },
buildRun: (sutUserId) => {
const marker = `SLACK_QA_DISABLED_${randomUUID().slice(0, 8).toUpperCase()}`;
@@ -116,7 +112,6 @@ const SLACK_QA_SCENARIOS: SlackQaScenarioDefinition[] = [
},
{
id: "slack-top-level-reply-shape",
standardId: "top-level-reply-shape",
title: "Slack top-level reply stays top-level",
timeoutMs: 45_000,
configOverrides: { replyToMode: "off" },
@@ -139,7 +134,6 @@ const SLACK_QA_SCENARIOS: SlackQaScenarioDefinition[] = [
{
id: "slack-progress-commentary-true",
title: "Slack progress commentary true is independent from tool progress",
defaultEnabled: false,
timeoutMs: 90_000,
configOverrides: {
progress: { commentary: true, toolProgress: false },
@@ -153,7 +147,6 @@ const SLACK_QA_SCENARIOS: SlackQaScenarioDefinition[] = [
{
id: "slack-progress-commentary-false",
title: "Slack progress commentary false stays out of the progress draft",
defaultEnabled: false,
timeoutMs: 90_000,
configOverrides: {
progress: { commentary: false, toolProgress: false },
@@ -167,7 +160,6 @@ const SLACK_QA_SCENARIOS: SlackQaScenarioDefinition[] = [
{
id: "slack-progress-commentary-omitted",
title: "Slack omitted progress commentary preserves the tool-progress default",
defaultEnabled: false,
timeoutMs: 90_000,
configOverrides: {
progress: { toolProgress: true },
@@ -181,7 +173,6 @@ const SLACK_QA_SCENARIOS: SlackQaScenarioDefinition[] = [
{
id: "slack-progress-commentary-verbose-dedupe",
title: "Slack explicit commentary yields to durable verbose progress",
defaultEnabled: false,
timeoutMs: 90_000,
configOverrides: {
progress: { commentary: true, toolProgress: false, verboseDefault: "on" },
@@ -269,7 +260,6 @@ const SLACK_QA_SCENARIOS: SlackQaScenarioDefinition[] = [
{
id: "slack-table-invalid-blocks-fallback",
title: "Slack rejects an over-limit native table and stores its complete fallback",
defaultEnabled: false,
timeoutMs: 45_000,
buildRun: () => ({
kind: "direct-transport",

13
extensions/qa-lab/src/live-transports/telegram/cli.test.ts
View File

@@ -1,13 +0,0 @@
import { describe, expect, it } from "vitest";
import { telegramQaCliRegistration } from "./cli.js";
import { TELEGRAM_QA_ALL_SCENARIO_IDS } from "./profiles.js";
describe("Telegram QA CLI registration", () => {
it("keeps the generic live baseline outside command profile selection", () => {
expect(telegramQaCliRegistration.adapterFactory?.scenarioIds).toEqual([
"channel-chat-baseline",
...TELEGRAM_QA_ALL_SCENARIO_IDS,
]);
expect(TELEGRAM_QA_ALL_SCENARIO_IDS).not.toContain("channel-chat-baseline");
});
});

2
extensions/qa-lab/src/live-transports/telegram/cli.ts
View File

@@ -5,7 +5,6 @@ import {
type LiveTransportQaCliRegistration,
type LiveTransportQaCommandOptions,
} from "../shared/live-transport-cli.js";
import { TELEGRAM_QA_ALL_SCENARIO_IDS } from "./profiles.js";
type TelegramQaAdapterRuntime = typeof import("./adapter.runtime.js");
type TelegramQaCliRuntime = typeof import("./cli.runtime.js");
@@ -23,7 +22,6 @@ async function runQaTelegram(opts: LiveTransportQaCommandOptions) {
const telegramQaAdapterFactory: NonNullable<LiveTransportQaCliRegistration["adapterFactory"]> = {
id: "telegram",
scenarioIds: ["channel-chat-baseline", ...TELEGRAM_QA_ALL_SCENARIO_IDS],
matches: ({ channelId, driver }) => driver === "live" && channelId === "telegram",
async create(context) {
return await (await loadTelegramQaAdapterRuntime()).createTelegramQaTransportAdapter(context);

23
extensions/qa-lab/src/live-transports/telegram/profiles.test.ts
View File

@@ -1,9 +1,6 @@
import { describe, expect, it } from "vitest";
import {
listTelegramQaScenarios,
resolveTelegramQaScenarioIds,
TELEGRAM_QA_ALL_SCENARIO_IDS,
} from "./profiles.js";
import { listQaScenariosForExecutionProfile } from "../../scenario-catalog.js";
import { listTelegramQaScenarios, resolveTelegramQaScenarioIds } from "./profiles.js";
describe("Telegram QA profiles", () => {
it("keeps release focused and adds the scripted long-final check for mock runs", () => {
@@ -16,19 +13,19 @@ describe("Telegram QA profiles", () => {
});
it("selects every migrated Telegram scenario through all", () => {
expect(resolveTelegramQaScenarioIds({ providerMode: "mock-openai", profile: "all" })).toEqual([
...TELEGRAM_QA_ALL_SCENARIO_IDS,
]);
expect(resolveTelegramQaScenarioIds({ providerMode: "mock-openai", profile: "all" })).toEqual(
listQaScenariosForExecutionProfile("telegram:all").map((scenario) => scenario.id),
);
});
it("lets explicit scenarios override profile selection", () => {
expect(
resolveTelegramQaScenarioIds({
profile: "all",
profile: "release",
providerMode: "live-frontier",
scenarioIds: ["telegram-status-command"],
scenarioIds: ["thread-follow-up"],
}),
).toEqual(["telegram-status-command"]);
).toEqual(["thread-follow-up"]);
});
it("rejects unknown profiles and scenarios before gateway startup", () => {
@@ -47,7 +44,9 @@ describe("Telegram QA profiles", () => {
const scenarios = listTelegramQaScenarios("mock-openai");
expect(scenarios.map(({ id }) => id).toSorted()).toEqual(
[...TELEGRAM_QA_ALL_SCENARIO_IDS].toSorted(),
listQaScenariosForExecutionProfile("telegram:all")
.map((scenario) => scenario.id)
.toSorted(),
);
expect(
scenarios.find(({ id }) => id === "telegram-long-final-reuses-preview")?.defaultEnabled,

64
extensions/qa-lab/src/live-transports/telegram/profiles.ts
View File

@@ -1,32 +1,4 @@
import { readQaScenarioPack } from "../../scenario-catalog.js";
const TELEGRAM_QA_RELEASE_SCENARIO_IDS = [
"channel-canary",
"channel-mention-gating",
"telegram-help-command",
"telegram-commands-command",
"telegram-tools-compact-command",
"telegram-whoami-command",
"telegram-status-command",
"telegram-repeated-command-authorization",
"telegram-context-command",
"telegram-other-bot-command-gating",
] as const;
const TELEGRAM_QA_MOCK_RELEASE_SCENARIO_IDS = [
...TELEGRAM_QA_RELEASE_SCENARIO_IDS,
"telegram-long-final-reuses-preview",
] as const;
export const TELEGRAM_QA_ALL_SCENARIO_IDS = [
...TELEGRAM_QA_RELEASE_SCENARIO_IDS,
"telegram-current-session-status-tool",
"telegram-tool-only-usage-footer",
"telegram-reply-chain-exact-marker",
"telegram-stream-final-single-message",
"telegram-long-final-reuses-preview",
"telegram-long-final-three-chunks",
] as const;
import { listQaScenariosForExecutionProfile, readQaScenarioPack } from "../../scenario-catalog.js";
type TelegramQaProfile = "all" | "release";
@@ -40,13 +12,17 @@ function resolveTelegramQaProfile(profile: string | undefined): TelegramQaProfil
);
}
function listTelegramQaProfileScenarios(profile: string) {
return listQaScenariosForExecutionProfile(`telegram:${profile}`);
}
export function resolveTelegramQaScenarioIds(params: {
profile?: string;
providerMode: string;
scenarioIds?: readonly string[];
}): string[] {
const knownIds = new Set<string>(TELEGRAM_QA_ALL_SCENARIO_IDS);
if (params.scenarioIds && params.scenarioIds.length > 0) {
if (params.scenarioIds?.length) {
const knownIds = new Set(readQaScenarioPack().scenarios.map((scenario) => scenario.id));
const unknownIds = params.scenarioIds.filter((id) => !knownIds.has(id));
if (unknownIds.length > 0) {
throw new Error(`unknown Telegram QA scenario id(s): ${unknownIds.join(", ")}`);
@@ -54,24 +30,18 @@ export function resolveTelegramQaScenarioIds(params: {
return [...params.scenarioIds];
}
const profile = resolveTelegramQaProfile(params.profile);
if (profile === "all") {
return [...TELEGRAM_QA_ALL_SCENARIO_IDS];
}
return params.providerMode === "mock-openai"
? [...TELEGRAM_QA_MOCK_RELEASE_SCENARIO_IDS]
: [...TELEGRAM_QA_RELEASE_SCENARIO_IDS];
const executionProfile =
profile === "release" && params.providerMode === "mock-openai" ? "mock-release" : profile;
return listTelegramQaProfileScenarios(executionProfile).map((scenario) => scenario.id);
}
export function listTelegramQaScenarios(providerMode: string) {
const defaultIds = new Set(resolveTelegramQaScenarioIds({ providerMode, profile: "release" }));
const allIds = new Set<string>(TELEGRAM_QA_ALL_SCENARIO_IDS);
return readQaScenarioPack()
.scenarios.filter((scenario) => allIds.has(scenario.id))
.map((scenario) => ({
id: scenario.id,
title: scenario.title,
rationale: scenario.objective,
regressionRefs: scenario.regressionRefs ?? [],
defaultEnabled: defaultIds.has(scenario.id),
}));
return listTelegramQaProfileScenarios("all").map((scenario) => ({
id: scenario.id,
title: scenario.title,
rationale: scenario.objective,
regressionRefs: scenario.regressionRefs ?? [],
defaultEnabled: defaultIds.has(scenario.id),
}));
}

2
extensions/qa-lab/src/live-transports/whatsapp/cli.ts
View File

@@ -5,7 +5,6 @@ import {
type LiveTransportQaCliRegistration,
type LiveTransportQaCommandOptions,
} from "../shared/live-transport-cli.js";
import { resolveWhatsAppQaScenarioIds } from "./profiles.js";
type WhatsAppQaAdapterRuntime = typeof import("./adapter.runtime.js");
type WhatsAppQaCliRuntime = typeof import("./cli.runtime.js");
@@ -23,7 +22,6 @@ async function runQaWhatsApp(opts: LiveTransportQaCommandOptions) {
const whatsappQaAdapterFactory: NonNullable<LiveTransportQaCliRegistration["adapterFactory"]> = {
id: "whatsapp",
scenarioIds: resolveWhatsAppQaScenarioIds({ providerMode: "live-frontier" }),
matches: ({ channelId, driver }) => driver === "live" && channelId === "whatsapp",
async create(context) {
return await (await loadWhatsAppQaAdapterRuntime()).createWhatsAppQaTransportAdapter(context);

58
extensions/qa-lab/src/live-transports/whatsapp/profiles.ts
View File

@@ -1,55 +1,4 @@
const WHATSAPP_QA_LIVE_DEFAULT_SCENARIO_IDS = [
"whatsapp-canary",
"whatsapp-mention-gating",
"whatsapp-top-level-reply-shape",
"whatsapp-reply-to-message",
"whatsapp-group-reply-to-message",
"whatsapp-status-reactions",
"whatsapp-group-allowlist-block",
"whatsapp-help-command",
] as const;
const WHATSAPP_QA_MOCK_DEFAULT_SCENARIO_IDS = [
"whatsapp-canary",
"whatsapp-mention-gating",
"whatsapp-group-pending-history-context",
"whatsapp-broadcast-group-fanout",
"whatsapp-group-activation-always",
"whatsapp-group-reply-to-bot-triggers",
"whatsapp-top-level-reply-shape",
"whatsapp-reply-to-message",
"whatsapp-group-reply-to-message",
"whatsapp-reply-to-mode-batched",
"whatsapp-agent-message-action-react",
"whatsapp-agent-message-action-upload-file",
"whatsapp-group-agent-message-action-react",
"whatsapp-group-agent-message-action-upload-file",
"whatsapp-inbound-reaction-no-trigger",
"whatsapp-reply-context-isolation",
"whatsapp-inbound-image-caption",
"whatsapp-audio-preflight",
"whatsapp-outbound-media-matrix",
"whatsapp-outbound-document-preserves-filename",
"whatsapp-outbound-poll",
"whatsapp-group-outbound-media",
"whatsapp-group-outbound-audio",
"whatsapp-group-outbound-poll",
"whatsapp-message-actions",
"whatsapp-inbound-structured-messages",
"whatsapp-group-audio-gating",
"whatsapp-reply-delivery-shape",
"whatsapp-stream-final-message-accounting",
"whatsapp-status-reactions",
"whatsapp-status-reaction-lifecycle",
"whatsapp-group-allowlist-block",
"whatsapp-help-command",
"whatsapp-commands-command",
"whatsapp-tools-compact-command",
"whatsapp-whoami-command",
"whatsapp-context-command",
"whatsapp-tool-only-usage-footer",
"whatsapp-native-new-command",
] as const;
import { listQaScenariosForExecutionProfile } from "../../scenario-catalog.js";
export function resolveWhatsAppQaScenarioIds(params: {
providerMode: string;
@@ -58,7 +7,6 @@ export function resolveWhatsAppQaScenarioIds(params: {
if (params.scenarioIds?.length) {
return [...params.scenarioIds];
}
return params.providerMode === "mock-openai"
? [...WHATSAPP_QA_MOCK_DEFAULT_SCENARIO_IDS]
: [...WHATSAPP_QA_LIVE_DEFAULT_SCENARIO_IDS];
const profile = params.providerMode === "mock-openai" ? "mock-default" : "default";
return listQaScenariosForExecutionProfile(`whatsapp:${profile}`).map((scenario) => scenario.id);
}

11
extensions/qa-lab/src/live-transports/whatsapp/whatsapp-live.contracts.ts
View File

@@ -6,8 +6,6 @@ import type {
import type { OpenClawConfig } from "openclaw/plugin-sdk/config-contracts";
import { formatErrorMessage } from "openclaw/plugin-sdk/error-runtime";
import type { startQaGatewayChild } from "../../gateway-child.js";
import type { QaProviderMode } from "../../run-config.js";
import type { LiveTransportScenarioDefinition } from "../shared/live-transport-scenarios.js";
export type WhatsAppQaRuntimeEnv = {
driverAuthArchiveBase64: string;
@@ -271,11 +269,12 @@ export type WhatsAppQaConfigOverrides = {
};
};
export type WhatsAppQaScenarioDefinition = LiveTransportScenarioDefinition<WhatsAppQaScenarioId> & {
export type WhatsAppQaScenarioDefinition = {
id: WhatsAppQaScenarioId;
title: string;
timeoutMs: number;
buildRun: () => WhatsAppQaScenarioRun;
configOverrides?: WhatsAppQaConfigOverrides;
defaultEnabled?: boolean;
defaultProviderModes?: readonly QaProviderMode[];
requiresGroupJid?: boolean;
requiredPluginIds?: readonly string[];
};
@@ -300,7 +299,6 @@ export type WhatsAppQaScenarioResult = {
responseObservedAt: string;
source: "approval-request-to-resolution" | "request-to-observed-message";
};
standardId?: string;
status: "fail" | "pass" | "skip";
title: string;
};
@@ -309,7 +307,6 @@ export function buildWhatsAppQaScenarioResultBase(scenario: WhatsAppQaScenarioDe
return {
id: scenario.id,
title: scenario.title,
standardId: scenario.standardId,
posture: WHATSAPP_QA_SCENARIO_POSTURES[scenario.id],
};
}

13
extensions/qa-lab/src/live-transports/whatsapp/whatsapp-live.runtime.test.ts
View File

@@ -188,7 +188,8 @@ const WHATSAPP_GROUP_CAPABILITY_SCENARIO_IDS = [
function findMockWhatsAppScenario(id: WhatsAppScenarioIdFilter) {
const scenario = getWhatsAppQaScenarioDefinition(id);
if (!scenario.standardId && !scenario.defaultProviderModes?.includes("mock-openai")) {
const mockScenarioIds = new Set(resolveWhatsAppQaScenarioIds({ providerMode: "mock-openai" }));
if (!mockScenarioIds.has(id)) {
throw new Error(`missing WhatsApp mock-openai scenario ${id}`);
}
return scenario;
@@ -306,7 +307,6 @@ describe("WhatsApp QA live runtime", () => {
}
expect(scenario.id).toBe("whatsapp-agent-message-action-react");
expect(scenario.defaultProviderModes).toEqual(["mock-openai"]);
expect(scenario.configOverrides).toMatchObject({ actions: true });
expect(run.target).toBe("dm");
expect(run.input).toMatch(/React to this WhatsApp message/i);
@@ -386,7 +386,6 @@ describe("WhatsApp QA live runtime", () => {
throw new Error(`${scenario.id} unexpectedly built an approval run`);
}
expect(scenario.defaultProviderModes).toEqual(["mock-openai"]);
expect(run.target).toBe("dm");
}
});
@@ -403,7 +402,6 @@ describe("WhatsApp QA live runtime", () => {
throw new Error(`${scenario.id} unexpectedly built an approval run`);
}
expect(scenario.defaultProviderModes).toEqual(["mock-openai"]);
expect(scenario.requiresGroupJid).toBe(true);
expect(run.target).toBe("group");
}
@@ -1059,7 +1057,6 @@ describe("WhatsApp QA live runtime", () => {
}
expect(scenario.requiresGroupJid).toBe(true);
expect(scenario.defaultProviderModes).toEqual(["mock-openai"]);
expect(run.target).toBe("group");
expect(run.configMode).toBe("open");
expect(run.input).toContain("openclawqa");
@@ -1078,7 +1075,6 @@ describe("WhatsApp QA live runtime", () => {
}
expect(scenario.requiresGroupJid).toBe(true);
expect(scenario.defaultProviderModes).toEqual(["mock-openai"]);
expect(scenario.configOverrides).toMatchObject({ groupPolicy: "open" });
expect(run.target).toBe("group");
expect(run.configMode).toBe("allowlist");
@@ -1405,20 +1401,17 @@ describe("WhatsApp QA live runtime", () => {
runs.map(({ scenario, run }) => ({
id: scenario.id,
requiresGroupJid: scenario.requiresGroupJid,
standardId: scenario.standardId,
target: run.target,
})),
).toEqual([
{
id: "whatsapp-reply-to-message",
requiresGroupJid: undefined,
standardId: "quote-reply",
target: "dm",
},
{
id: "whatsapp-group-reply-to-message",
requiresGroupJid: true,
standardId: "quote-reply",
target: "group",
},
]);
@@ -1607,7 +1600,6 @@ describe("WhatsApp QA live runtime", () => {
throw new Error("whatsapp-stream-final-message-accounting unexpectedly built approval run");
}
expect(scenario.defaultProviderModes).toEqual(["mock-openai"]);
expect(run.input).toContain("WhatsApp long final QA check");
expect(run.matchText).toBe("WHATSAPP-LONG-FINAL-BEGIN");
expect(run.expectedJoinedSutTextIncludes).toEqual([
@@ -1936,7 +1928,6 @@ describe("WhatsApp QA live runtime", () => {
}
expect(scenario.requiredPluginIds).toEqual(["openai"]);
expect(scenario.defaultProviderModes).toEqual(["mock-openai"]);
expect(scenarioRun.expectReply).toBe(true);
expect(scenarioRun.matchText).toBe("WHATSAPP_QA_AUDIO_TRANSCRIPT_OK");
expect(scenarioRun.sendMode).toMatchObject({

10
extensions/qa-lab/src/live-transports/whatsapp/whatsapp-live.scenarios.capabilities.ts
View File

@@ -24,7 +24,6 @@ export const WHATSAPP_QA_CAPABILITY_SCENARIOS: WhatsAppQaScenarioDefinition[] =
{
id: "whatsapp-outbound-document-preserves-filename",
title: "WhatsApp direct Gateway document preserves filename and caption",
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
buildRun: () => {
const token = `WHATSAPP_QA_DOCUMENT_FILE_${randomUUID().slice(0, 8).toUpperCase()}`;
@@ -62,8 +61,6 @@ export const WHATSAPP_QA_CAPABILITY_SCENARIOS: WhatsAppQaScenarioDefinition[] =
{
id: "whatsapp-outbound-send-serialization",
title: "WhatsApp parallel Gateway sends deliver every outbound message",
defaultEnabled: false,
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
buildRun: () => {
const token = `WHATSAPP_QA_SERIAL_SEND_${randomUUID().slice(0, 8).toUpperCase()}`;
@@ -99,7 +96,6 @@ export const WHATSAPP_QA_CAPABILITY_SCENARIOS: WhatsAppQaScenarioDefinition[] =
{
id: "whatsapp-outbound-poll",
title: "WhatsApp direct Gateway poll delivers outbound native poll",
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
buildRun: () => {
const token = `WHATSAPP_QA_OUTBOUND_POLL_${randomUUID().slice(0, 8).toUpperCase()}`;
@@ -133,7 +129,6 @@ export const WHATSAPP_QA_CAPABILITY_SCENARIOS: WhatsAppQaScenarioDefinition[] =
{
id: "whatsapp-group-outbound-media",
title: "WhatsApp direct Gateway send delivers media to a group",
defaultProviderModes: ["mock-openai"],
timeoutMs: 120_000,
requiresGroupJid: true,
buildRun: () => {
@@ -196,7 +191,6 @@ export const WHATSAPP_QA_CAPABILITY_SCENARIOS: WhatsAppQaScenarioDefinition[] =
{
id: "whatsapp-group-outbound-audio",
title: "WhatsApp direct Gateway send delivers audio to a group",
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
requiresGroupJid: true,
buildRun: () => {
@@ -240,7 +234,6 @@ export const WHATSAPP_QA_CAPABILITY_SCENARIOS: WhatsAppQaScenarioDefinition[] =
{
id: "whatsapp-group-outbound-poll",
title: "WhatsApp direct Gateway poll delivers native poll to a group",
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
requiresGroupJid: true,
buildRun: () => {
@@ -276,7 +269,6 @@ export const WHATSAPP_QA_CAPABILITY_SCENARIOS: WhatsAppQaScenarioDefinition[] =
{
id: "whatsapp-message-actions",
title: "WhatsApp direct Gateway message.action react and upload-file execute",
defaultProviderModes: ["mock-openai"],
timeoutMs: 120_000,
configOverrides: {
actions: true,
@@ -331,7 +323,6 @@ export const WHATSAPP_QA_CAPABILITY_SCENARIOS: WhatsAppQaScenarioDefinition[] =
{
id: "whatsapp-inbound-structured-messages",
title: "WhatsApp inbound structured messages reach the agent",
defaultProviderModes: ["mock-openai"],
timeoutMs: 240_000,
buildRun: () => {
const token = `WHATSAPP_QA_STRUCTURED_${randomUUID().slice(0, 8).toUpperCase()}`;
@@ -396,7 +387,6 @@ export const WHATSAPP_QA_CAPABILITY_SCENARIOS: WhatsAppQaScenarioDefinition[] =
{
id: "whatsapp-group-audio-gating",
title: "WhatsApp group audio mention gating",
defaultProviderModes: ["mock-openai"],
timeoutMs: 120_000,
configOverrides: {
audioPreflight: true,

10
extensions/qa-lab/src/live-transports/whatsapp/whatsapp-live.scenarios.conversation.ts
View File

@@ -44,7 +44,6 @@ function buildWhatsAppQuoteReplyRun(target: "dm" | "group"): WhatsAppQaMessageSc
export const WHATSAPP_QA_CONVERSATION_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
{
id: "whatsapp-canary",
standardId: "canary",
title: "WhatsApp DM canary",
timeoutMs: 60_000,
buildRun: () => {
@@ -60,7 +59,6 @@ export const WHATSAPP_QA_CONVERSATION_SCENARIOS: WhatsAppQaScenarioDefinition[]
},
{
id: "whatsapp-mention-gating",
standardId: "mention-gating",
title: "WhatsApp group mention gating",
timeoutMs: 60_000,
requiresGroupJid: true,
@@ -82,7 +80,6 @@ export const WHATSAPP_QA_CONVERSATION_SCENARIOS: WhatsAppQaScenarioDefinition[]
{
id: "whatsapp-group-pending-history-context",
title: "WhatsApp group pending history reaches mentioned turns",
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
configOverrides: {
groupHistoryLimit: 50,
@@ -116,7 +113,6 @@ export const WHATSAPP_QA_CONVERSATION_SCENARIOS: WhatsAppQaScenarioDefinition[]
{
id: "whatsapp-broadcast-group-fanout",
title: "WhatsApp group broadcast fans out to multiple agents",
defaultProviderModes: ["mock-openai"],
timeoutMs: 120_000,
configOverrides: {
broadcast: {
@@ -155,7 +151,6 @@ export const WHATSAPP_QA_CONVERSATION_SCENARIOS: WhatsAppQaScenarioDefinition[]
{
id: "whatsapp-group-activation-always",
title: "WhatsApp group activation always wakes unmentioned messages",
defaultProviderModes: ["mock-openai"],
timeoutMs: 120_000,
configOverrides: {
groupPolicy: "open",
@@ -242,7 +237,6 @@ export const WHATSAPP_QA_CONVERSATION_SCENARIOS: WhatsAppQaScenarioDefinition[]
{
id: "whatsapp-group-reply-to-bot-triggers",
title: "WhatsApp group reply to bot wakes without an explicit mention",
defaultProviderModes: ["mock-openai"],
timeoutMs: 120_000,
configOverrides: {
groupPolicy: "open",
@@ -300,7 +294,6 @@ export const WHATSAPP_QA_CONVERSATION_SCENARIOS: WhatsAppQaScenarioDefinition[]
},
{
id: "whatsapp-top-level-reply-shape",
standardId: "top-level-reply-shape",
title: "WhatsApp DM top-level reply shape",
timeoutMs: 60_000,
configOverrides: {
@@ -326,7 +319,6 @@ export const WHATSAPP_QA_CONVERSATION_SCENARIOS: WhatsAppQaScenarioDefinition[]
},
{
id: "whatsapp-reply-to-message",
standardId: "quote-reply",
title: "WhatsApp DM reply-to mode quotes the triggering message",
timeoutMs: 60_000,
configOverrides: {
@@ -336,7 +328,6 @@ export const WHATSAPP_QA_CONVERSATION_SCENARIOS: WhatsAppQaScenarioDefinition[]
},
{
id: "whatsapp-group-reply-to-message",
standardId: "quote-reply",
title: "WhatsApp group reply-to mode quotes the triggering message",
timeoutMs: 60_000,
configOverrides: {
@@ -348,7 +339,6 @@ export const WHATSAPP_QA_CONVERSATION_SCENARIOS: WhatsAppQaScenarioDefinition[]
{
id: "whatsapp-reply-to-mode-batched",
title: "WhatsApp batched reply-to mode quotes the queued message",
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
configOverrides: {
inboundDebounceMs: 250,

5
extensions/qa-lab/src/live-transports/whatsapp/whatsapp-live.scenarios.delivery.ts
View File

@@ -12,7 +12,6 @@ export const WHATSAPP_QA_DELIVERY_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
{
id: "whatsapp-reply-delivery-shape",
title: "WhatsApp direct Gateway send chunks long replies",
defaultProviderModes: ["mock-openai"],
timeoutMs: 120_000,
buildRun: () => {
const token = `WHATSAPP_QA_REPLY_SHAPE_${randomUUID().slice(0, 8).toUpperCase()}`;
@@ -75,7 +74,6 @@ export const WHATSAPP_QA_DELIVERY_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
{
id: "whatsapp-stream-final-message-accounting",
title: "WhatsApp streamed final response has exactly the final chunks",
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
buildRun: () => ({
configMode: "allowlist",
@@ -106,7 +104,6 @@ export const WHATSAPP_QA_DELIVERY_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
},
{
id: "whatsapp-status-reactions",
standardId: "reaction-observation",
title: "WhatsApp status reactions are observable",
timeoutMs: 60_000,
configOverrides: {
@@ -133,7 +130,6 @@ export const WHATSAPP_QA_DELIVERY_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
{
id: "whatsapp-status-reaction-lifecycle",
title: "WhatsApp status reaction lifecycle updates the triggering message",
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
configOverrides: {
statusReactions: {
@@ -170,7 +166,6 @@ export const WHATSAPP_QA_DELIVERY_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
},
{
id: "whatsapp-group-allowlist-block",
standardId: "allowlist-block",
title: "WhatsApp group outside allowlist stays quiet",
timeoutMs: 8_000,
configOverrides: {

9
extensions/qa-lab/src/live-transports/whatsapp/whatsapp-live.scenarios.user-path.ts
View File

@@ -22,7 +22,6 @@ export const WHATSAPP_QA_USER_PATH_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
{
id: "whatsapp-agent-message-action-react",
title: "WhatsApp user-path agent reaction uses the message tool",
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
configOverrides: {
actions: true,
@@ -53,7 +52,6 @@ export const WHATSAPP_QA_USER_PATH_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
{
id: "whatsapp-agent-message-action-upload-file",
title: "WhatsApp user-path agent upload-file sends media",
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
configOverrides: {
actions: true,
@@ -91,7 +89,6 @@ export const WHATSAPP_QA_USER_PATH_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
{
id: "whatsapp-group-agent-message-action-react",
title: "WhatsApp group user-path agent reaction uses the message tool",
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
configOverrides: {
actions: true,
@@ -123,7 +120,6 @@ export const WHATSAPP_QA_USER_PATH_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
{
id: "whatsapp-group-agent-message-action-upload-file",
title: "WhatsApp group user-path agent upload-file sends media",
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
configOverrides: {
actions: true,
@@ -163,7 +159,6 @@ export const WHATSAPP_QA_USER_PATH_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
{
id: "whatsapp-inbound-reaction-no-trigger",
title: "WhatsApp inbound user reaction does not start a fresh run",
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
buildRun: () => {
const token = `WHATSAPP_QA_INBOUND_REACTION_${randomUUID().slice(0, 8).toUpperCase()}`;
@@ -197,7 +192,6 @@ export const WHATSAPP_QA_USER_PATH_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
{
id: "whatsapp-reply-context-isolation",
title: "WhatsApp direct Gateway send does not reuse prior quote context",
defaultProviderModes: ["mock-openai"],
timeoutMs: 120_000,
buildRun: () => {
const token = `WHATSAPP_QA_REPLY_ISOLATION_${randomUUID().slice(0, 8).toUpperCase()}`;
@@ -256,7 +250,6 @@ export const WHATSAPP_QA_USER_PATH_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
{
id: "whatsapp-inbound-image-caption",
title: "WhatsApp inbound image caption reaches the agent",
defaultProviderModes: ["mock-openai"],
timeoutMs: 60_000,
buildRun: () => {
const token = `WHATSAPP_QA_IMAGE_${randomUUID().slice(0, 8).toUpperCase()}`;
@@ -278,7 +271,6 @@ export const WHATSAPP_QA_USER_PATH_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
{
id: "whatsapp-audio-preflight",
title: "WhatsApp inbound audio preflight transcript reaches the agent",
defaultProviderModes: ["mock-openai"],
timeoutMs: 90_000,
configOverrides: {
audioPreflight: true,
@@ -301,7 +293,6 @@ export const WHATSAPP_QA_USER_PATH_SCENARIOS: WhatsAppQaScenarioDefinition[] = [
{
id: "whatsapp-outbound-media-matrix",
title: "WhatsApp direct Gateway send delivers outbound media variants",
defaultProviderModes: ["mock-openai"],
timeoutMs: 120_000,
buildRun: () => {
const token = `WHATSAPP_QA_OUTBOUND_MEDIA_${randomUUID().slice(0, 8).toUpperCase()}`;

35
extensions/qa-lab/src/scenario-catalog.test.ts
View File

@@ -731,7 +731,7 @@ describe("qa scenario catalog", () => {
it("keeps provider-sensitive QA flow scenarios on their supported lanes", () => {
const strandedConfig = readQaScenarioExecutionConfig("message-tool-stranded-final-reply") as
| { requiredChannelDriver?: string; requiredProviderMode?: string }
| { requiredProviderMode?: string }
| undefined;
const retryFailureConfig = readQaScenarioExecutionConfig(
"message-tool-stranded-final-retry-failure",
@@ -834,7 +834,6 @@ describe("qa scenario catalog", () => {
| {
workspaceFiles?: Record<string, string>;
prompt?: string;
requiredChannelDriver?: string;
expectedReplyAll?: string[];
expectedArtifactAll?: string[];
expectedArtifactAny?: string[];
@@ -847,7 +846,7 @@ describe("qa scenario catalog", () => {
"Mission: prove you followed the repo contract.",
);
expect(config?.prompt).toContain("Repo contract followthrough check.");
expect(config?.requiredChannelDriver).toBe("qa-channel");
expect(scenario.execution.driver).toBe("qa-channel");
expect(config?.expectedReplyAll).toEqual(["read:", "wrote:", "status:"]);
expect(config?.expectedArtifactAll).toEqual(["repo contract"]);
expect(config?.expectedArtifactAny).toContain("evidence path");
@@ -879,46 +878,39 @@ describe("qa scenario catalog", () => {
];
for (const scenarioId of scenarioIds) {
const config = readQaScenarioExecutionConfig(scenarioId) as
| { requiredChannelDriver?: string }
| undefined;
expect(config?.requiredChannelDriver, scenarioId).toBe("qa-channel");
expect(readQaScenarioById(scenarioId).execution.driver, scenarioId).toBe("qa-channel");
}
});
it("routes canonical thread relation flows through Crabline Matrix", () => {
it("keeps portable thread relation flows free of a channel requirement", () => {
for (const scenarioId of ["thread-follow-up", "thread-isolation"]) {
const scenario = readQaScenarioById(scenarioId);
const config = readQaScenarioExecutionConfig(scenarioId) as
| { requiredChannelDriver?: string }
| undefined;
expect(scenario.execution.channel, scenarioId).toBe("matrix");
expect(config?.requiredChannelDriver, scenarioId).toBeUndefined();
expect(scenario.execution.channel, scenarioId).toBeUndefined();
expect(scenario.execution.driver, scenarioId).toBeUndefined();
expect(Object.keys(scenario.execution.profiles ?? {}), scenarioId).toEqual(
expect.arrayContaining(["matrix:adapter", "slack:adapter"]),
);
}
});
it("keeps Matrix subagent thread spawn explicitly selectable", () => {
const scenario = readQaScenarioById("subagent-thread-spawn");
const config = readQaScenarioExecutionConfig("subagent-thread-spawn") as
| { requiredChannelDriver?: string }
| undefined;
expect(scenario.execution.channel).toBe("matrix");
expect(config?.requiredChannelDriver).toBe("live");
expect(scenario.execution.driver).toBe("live");
});
it("routes native command session targeting through Crabline Telegram", () => {
const scenario = readQaScenarioById("native-command-session-target");
const config = readQaScenarioExecutionConfig("native-command-session-target") as
| {
requiredChannelDriver?: string;
requiredProviderMode?: string;
}
| undefined;
expect(scenario.execution.channel).toBe("telegram");
expect(config?.requiredChannelDriver).toBeUndefined();
expect(scenario.execution.driver).toBeUndefined();
expect(config?.requiredProviderMode).toBe("mock-openai");
});
@@ -935,10 +927,7 @@ describe("qa scenario catalog", () => {
];
for (const scenarioId of liveScenarioIds) {
const config = readQaScenarioExecutionConfig(scenarioId) as
| { requiredChannelDriver?: string }
| undefined;
expect(config?.requiredChannelDriver, scenarioId).toBe("live");
expect(readQaScenarioById(scenarioId).execution.driver, scenarioId).toBe("live");
}
});

27
extensions/qa-lab/src/scenario-catalog.ts
View File

@@ -66,6 +66,14 @@ const qaScenarioChannelSchema = z
message: "scenario execution channel ids must use lowercase dotted or dashed tokens",
});
const qaScenarioDriverSchema = z.enum(["qa-channel", "crabline", "live"]);
const qaScenarioProfileSchema = z
.string()
.trim()
.regex(/^[a-z0-9]+(?:[.:/-][a-z0-9]+)*$/, {
message: "scenario execution profiles must use lowercase namespaced tokens",
});
const qaScenarioTransportPolicySchema = z.object({
requireGroupMention: z.literal(true).optional(),
senderAllowlist: z.array(z.string().trim().min(1)).min(1).optional(),
@@ -77,6 +85,8 @@ const qaFlowScenarioExecutionSchema = z
kind: z.literal("flow").default("flow"),
summary: z.string().trim().min(1).optional(),
channel: qaScenarioChannelSchema.optional(),
driver: qaScenarioDriverSchema.optional(),
profiles: z.record(qaScenarioProfileSchema, z.number().int().nonnegative()).optional(),
suiteIsolation: z.literal("isolated").optional(),
isolationReason: z.string().trim().min(1).optional(),
transportPolicy: qaScenarioTransportPolicySchema.optional(),
@@ -87,6 +97,8 @@ const qaFlowScenarioExecutionSchema = z
const qaTestFileScenarioExecutionBaseSchema = z.object({
summary: z.string().trim().min(1).optional(),
channel: qaScenarioChannelSchema.optional(),
driver: qaScenarioDriverSchema.optional(),
profiles: z.record(qaScenarioProfileSchema, z.number().int().nonnegative()).optional(),
path: qaScenarioRepoRefSchema,
config: qaScenarioConfigSchema.optional(),
});
@@ -546,6 +558,21 @@ export function readQaScenarioExecutionConfig(id: string): Record<string, unknow
return readQaScenarioPack().scenarios.find((candidate) => candidate.id === id)?.execution?.config;
}
export function listQaScenariosForExecutionProfile(profile: string): QaSeedScenarioWithSource[] {
const normalized = qaScenarioProfileSchema.parse(profile);
const scenarios = readQaScenarioPack()
.scenarios.filter((scenario) => scenario.execution.profiles?.[normalized] !== undefined)
.toSorted((left, right) => {
const orderDelta =
left.execution.profiles![normalized]! - right.execution.profiles![normalized]!;
return orderDelta || left.sourcePath.localeCompare(right.sourcePath);
});
if (scenarios.length === 0) {
throw new Error(`unknown QA scenario execution profile: ${normalized}`);
}
return scenarios;
}
export function validateQaScenarioExecutionConfig(config: Record<string, unknown>) {
return qaScenarioConfigSchema.parse(config);
}

6
extensions/qa-lab/src/scenario-lane.ts
View File

@@ -28,10 +28,10 @@ export function describeQaProviderLaneMismatches(params: {
if (requiredProviderMode && params.providerMode !== requiredProviderMode) {
mismatches.push(`providerMode=${requiredProviderMode}`);
}
const requiredChannelDriver = normalizeQaConfigString(config.requiredChannelDriver);
const requiredDriver = params.scenario.execution.driver;
const effectiveChannelDriver = params.channelDriver ?? "qa-channel";
if (requiredChannelDriver && effectiveChannelDriver !== requiredChannelDriver) {
mismatches.push(`channelDriver=${requiredChannelDriver}`);
if (requiredDriver && effectiveChannelDriver !== requiredDriver) {
mismatches.push(`channelDriver=${requiredDriver}`);
}
const scenarioChannel = params.scenario.execution.channel?.trim().toLowerCase();
if (

6
extensions/qa-lab/src/suite-planning.test.ts
View File

@@ -685,10 +685,10 @@ describe("qa suite planning helpers", () => {
const scenarios = [
makeQaSuiteTestScenario("generic"),
makeQaSuiteTestScenario("qa-channel-only", {
config: { requiredChannelDriver: "qa-channel" },
driver: "qa-channel",
}),
makeQaSuiteTestScenario("crabline-only", {
config: { requiredChannelDriver: "crabline" },
driver: "crabline",
}),
];
@@ -714,7 +714,7 @@ describe("qa suite planning helpers", () => {
const scenarios = [
makeQaSuiteTestScenario("generic"),
makeQaSuiteTestScenario("qa-channel-only", {
config: { requiredChannelDriver: "qa-channel" },
driver: "qa-channel",
}),
];

2
extensions/qa-lab/src/suite-test-helpers.ts
View File

@@ -8,6 +8,7 @@ export function makeQaSuiteTestScenario(
id: string,
params: {
channel?: string;
driver?: QaSuiteTestScenario["execution"]["driver"];
config?: Record<string, unknown>;
plugins?: string[];
gatewayConfigPatch?: Record<string, unknown>;
@@ -32,6 +33,7 @@ export function makeQaSuiteTestScenario(
execution: {
kind: "flow",
...(params.channel ? { channel: params.channel } : {}),
...(params.driver ? { driver: params.driver } : {}),
...(params.suiteIsolation ? { suiteIsolation: params.suiteIsolation } : {}),
...(params.transportPolicy ? { transportPolicy: params.transportPolicy } : {}),
...(params.config ? { config: params.config } : {}),

4
package.json
View File

@@ -1149,10 +1149,6 @@
"types": "./dist/plugin-sdk/qa-runner-runtime.d.ts",
"default": "./dist/plugin-sdk/qa-runner-runtime.js"
},
"./plugin-sdk/qa-live-transport-scenarios": {
"types": "./dist/plugin-sdk/qa-live-transport-scenarios.d.ts",
"default": "./dist/plugin-sdk/qa-live-transport-scenarios.js"
},
"./plugin-sdk/memory-core": {
"types": "./dist/plugin-sdk/memory-core.d.ts",
"default": "./dist/plugin-sdk/memory-core.js"

2
qa/scenarios/agents/instruction-followthrough-repo-contract.yaml
View File

@@ -24,8 +24,8 @@ scenario:
execution:
kind: flow
summary: Verify the agent reads repo instructions first, then completes the first bounded followthrough task without stalling.
driver: qa-channel
config:
requiredChannelDriver: qa-channel
workspaceFiles:
AGENT.md: |-
# Repo contract

2
qa/scenarios/agents/subagent-forked-context.yaml
View File

@@ -21,8 +21,8 @@ scenario:
execution:
kind: flow
summary: Ask the agent to delegate work that depends on the current transcript and assert sessions_spawn carries context=fork.
driver: qa-channel
config:
requiredChannelDriver: qa-channel
contextNeedle: FORKED-CONTEXT-ALPHA
prompt: "Forked subagent context QA check. The visible code in this current conversation is FORKED-CONTEXT-ALPHA. Delegate to a native subagent to report the visible code from the requester transcript. Do not include the visible code in the child task text; the child must recover it from forked transcript context. Use forked context if the child needs the current transcript; otherwise it will not know the code. A spawn-accepted result is not the answer. Wait for the child completion, then make sure user-visible output includes the visible code."

2
qa/scenarios/agents/subagent-handoff.yaml
View File

@@ -20,8 +20,8 @@ scenario:
execution:
kind: flow
summary: Verify the agent can delegate a bounded task to a subagent and fold the result back into the main thread.
driver: qa-channel
config:
requiredChannelDriver: qa-channel
prompt: "Delegate one bounded QA task to a subagent. Wait for the subagent to finish. Then reply with three labeled sections exactly once: Delegated task, Result, Evidence. Include the child result itself, not 'waiting'."
flow:

2
qa/scenarios/channels/a2a-message-tool-mirror-dedupe.yaml
View File

@@ -37,10 +37,10 @@ scenario:
- src/shared/transcript-only-openclaw-assistant.ts
execution:
kind: flow
driver: qa-channel
summary: Run a real QA Gateway/qa-channel A2A source-reply turn and assert the message-tool delivery mirror is not announced again.
config:
requiredProviderMode: mock-openai
requiredChannelDriver: qa-channel
conversationId: qa-a2a-mirror-room
conversationTitle: QA A2A Mirror Room
promptSnippet: qa a2a message-tool mirror check

10
qa/scenarios/channels/channel-canary.yaml
View File

@@ -20,6 +20,16 @@ scenario:
- extensions/qa-lab/src/qa-transport.ts
- extensions/qa-lab/src/suite.ts
execution:
profiles:
{
"matrix:adapter": 1,
"slack:adapter": 1,
"telegram:adapter": 1,
"telegram:all": 0,
"telegram:mock-release": 0,
"telegram:release": 0,
"whatsapp:adapter": 1,
}
kind: flow
summary: Run the shared channel canary through QA Channel, Crabline, or a live adapter.
transportPolicy:

9
qa/scenarios/channels/channel-chat-baseline.yaml
View File

@@ -21,6 +21,15 @@ scenario:
- extensions/qa-channel/src/inbound.ts
- extensions/qa-lab/src/bus-state.ts
execution:
profiles:
{
"matrix:adapter": 0,
"matrix:all": 0,
"matrix:fast": 0,
"matrix:release": 0,
"slack:adapter": 0,
"telegram:adapter": 0,
}
kind: flow
suiteIsolation: isolated
summary: Verify the QA agent can respond correctly in a shared channel and respect mention-driven group semantics.

1
qa/scenarios/channels/channel-dm-group-routing.yaml
View File

@@ -20,6 +20,7 @@ scenario:
- extensions/qa-lab/src/qa-transport.ts
- src/routing/resolve-route.ts
execution:
profiles: { "matrix:adapter": 2, "whatsapp:adapter": 2 }
kind: flow
summary: Send sequential DM and group turns and assert route separation.
transportPolicy:

13
qa/scenarios/channels/channel-mention-gating.yaml
View File

@@ -19,6 +19,19 @@ scenario:
- extensions/qa-channel/src/inbound.ts
- extensions/qa-lab/src/qa-transport.ts
execution:
profiles:
{
"matrix:adapter": 3,
"matrix:all": 35,
"matrix:fast": 7,
"matrix:transport": 34,
"slack:adapter": 2,
"telegram:adapter": 2,
"telegram:all": 1,
"telegram:mock-release": 1,
"telegram:release": 1,
"whatsapp:adapter": 3,
}
kind: flow
summary: Prove the shared mention gate with a quiet turn followed by a mentioned turn.
transportPolicy:

1
qa/scenarios/channels/channel-multi-actor-ordering.yaml
View File

@@ -19,6 +19,7 @@ scenario:
- extensions/qa-lab/src/qa-transport.ts
- extensions/qa-lab/src/live-transports/matrix/adapter.runtime.ts
execution:
profiles: { "matrix:adapter": 7, "matrix:all": 48, "matrix:transport": 47 }
kind: flow
summary: Preserve actor order across a blocked observer turn and a successful driver turn.
transportPolicy:

1
qa/scenarios/channels/channel-secondary-conversation-isolation.yaml
View File

@@ -19,6 +19,7 @@ scenario:
- extensions/qa-lab/src/qa-transport.ts
- extensions/qa-lab/src/live-transports/matrix/adapter.runtime.ts
execution:
profiles: { "matrix:adapter": 6, "matrix:all": 19, "matrix:transport": 18 }
kind: flow
summary: Send sequential turns to primary and secondary channel conversations and assert isolation.
transportPolicy:

1
qa/scenarios/channels/channel-sender-allowlist.yaml
View File

@@ -19,6 +19,7 @@ scenario:
- extensions/qa-lab/src/qa-transport.ts
- src/channels/turn/kernel.ts
execution:
profiles: { "matrix:adapter": 4, "matrix:all": 47, "matrix:fast": 10, "matrix:transport": 46 }
kind: flow
summary: Exercise sender allowlist rejection and the configured driver override in one ordered flow.
transportPolicy:

9
qa/scenarios/channels/channel-top-level-reply-shape.yaml
View File

@@ -19,6 +19,15 @@ scenario:
- extensions/qa-lab/src/qa-transport.ts
- extensions/qa-lab/src/runtime-api.ts
execution:
profiles:
{
"matrix:adapter": 5,
"matrix:all": 5,
"matrix:fast": 3,
"matrix:transport": 4,
"slack:adapter": 3,
"whatsapp:adapter": 4,
}
kind: flow
summary: Assert portable top-level reply shape while native adapters normalize transport relations.
transportPolicy:

1
qa/scenarios/channels/discord-canary.yaml
View File

@@ -5,6 +5,7 @@ scenario:
coverage:
primary: [channels.group-messages]
execution:
profiles: { "discord:adapter": 0, "discord:default": 0 }
kind: flow
channel: discord
timeoutMs: 45000

1
qa/scenarios/channels/discord-mention-gating.yaml
View File

@@ -5,6 +5,7 @@ scenario:
coverage:
primary: [channels.group-messages]
execution:
profiles: { "discord:adapter": 1, "discord:default": 1 }
kind: flow
channel: discord
timeoutMs: 8000

1
qa/scenarios/channels/discord-native-help-command-registration.yaml
View File

@@ -5,6 +5,7 @@ scenario:
coverage:
primary: [discord.native-slash-command-registration]
execution:
profiles: { "discord:adapter": 2, "discord:default": 2 }
kind: flow
channel: discord
timeoutMs: 45000

1
qa/scenarios/channels/dm-chat-baseline.yaml
View File

@@ -21,6 +21,7 @@ scenario:
- extensions/qa-channel/src/gateway.ts
- extensions/qa-lab/src/lab-server.ts
execution:
profiles: { "matrix:all": 14, "matrix:transport": 13, "whatsapp:adapter": 0 }
kind: flow
summary: Verify the QA agent can chat coherently in a DM, explain the QA setup, and stay in character.
config:

1
qa/scenarios/channels/dm-per-room-session.yaml
View File

@@ -25,6 +25,7 @@ scenario:
- extensions/matrix/src/matrix/monitor/handler.ts
- extensions/qa-lab/src/live-transports/matrix/adapter.runtime.ts
execution:
profiles: { "matrix:adapter": 12, "matrix:all": 17, "matrix:transport": 16 }
kind: flow
channel: matrix
summary: Send consecutive DMs in two Matrix rooms and require per-room isolation.

1
qa/scenarios/channels/dm-shared-session.yaml
View File

@@ -17,6 +17,7 @@ scenario:
- extensions/matrix/src/matrix/monitor/handler.ts
- extensions/qa-lab/src/live-transports/matrix/adapter.runtime.ts
execution:
profiles: { "matrix:adapter": 11, "matrix:all": 15, "matrix:transport": 14 }
kind: flow
channel: matrix
summary: Send consecutive DMs from one Matrix user in two rooms and require the shared-session notice.

2
qa/scenarios/channels/group-message-tool-unavailable-fallback.yaml
View File

@@ -34,8 +34,8 @@ scenario:
execution:
kind: flow
summary: Verify message_tool visible replies degrade to automatic delivery when the active group policy removes message.
driver: qa-channel
config:
requiredChannelDriver: qa-channel
conversationId: qa-fallback-room
promptSnippet: qa group message unavailable fallback check
prompt: "@openclaw qa group message unavailable fallback check. exact marker: `QA-GROUP-FALLBACK-OK`"

3
qa/scenarios/channels/matrix-allowbots-default-block.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-policy
execution:
profiles: { "matrix:all": 36, "matrix:fast": 8, "matrix:transport": 35 }
kind: flow
channel: matrix
timeoutMs: 8000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
configuredBotRoles:
- observer

3
qa/scenarios/channels/matrix-allowbots-mentions-dm-unmentioned.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.dm
execution:
profiles: { "matrix:all": 40, "matrix:transport": 39 }
kind: flow
channel: matrix
timeoutMs: 45000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
allowBots: mentions
configuredBotRoles:

3
qa/scenarios/channels/matrix-allowbots-mentions-mentioned-room.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-policy
execution:
profiles: { "matrix:all": 38, "matrix:fast": 9, "matrix:transport": 37 }
kind: flow
channel: matrix
timeoutMs: 45000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
allowBots: mentions
configuredBotRoles:

3
qa/scenarios/channels/matrix-allowbots-mentions-unmentioned-open-room-block.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-policy
execution:
profiles: { "matrix:all": 39, "matrix:transport": 38 }
kind: flow
channel: matrix
timeoutMs: 8000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
allowBots: mentions
configuredBotRoles:

3
qa/scenarios/channels/matrix-allowbots-room-override-blocks-account-true.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-policy
execution:
profiles: { "matrix:all": 41, "matrix:transport": 40 }
kind: flow
channel: matrix
timeoutMs: 8000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
allowBots: true
configuredBotRoles:

3
qa/scenarios/channels/matrix-allowbots-room-override-enables-account-off.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-policy
execution:
profiles: { "matrix:all": 42, "matrix:transport": 41 }
kind: flow
channel: matrix
timeoutMs: 45000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
configuredBotRoles:
- observer

3
qa/scenarios/channels/matrix-allowbots-self-sender-ignored.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-policy
execution:
profiles: { "matrix:all": 43, "matrix:transport": 42 }
kind: flow
channel: matrix
timeoutMs: 8000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
allowBots: true
groupAllowRoles:

3
qa/scenarios/channels/matrix-allowbots-true-unmentioned-open-room.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-policy
execution:
profiles: { "matrix:all": 37, "matrix:transport": 36 }
kind: flow
channel: matrix
timeoutMs: 45000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
allowBots: true
configuredBotRoles:

3
qa/scenarios/channels/matrix-allowlist-hot-reload.yaml
View File

@@ -15,14 +15,15 @@ scenario:
codeRefs:
- extensions/qa-lab/src/live-transports/matrix/adapter.runtime.ts
execution:
profiles: { "matrix:all": 46, "matrix:release": 1, "matrix:transport": 45 }
kind: flow
channel: matrix
timeoutMs: 75000
retryCount: 0
suiteIsolation: isolated
summary: Exercise Matrix observer-role allowlist hot reload through normalized transport state.
driver: live
config:
requiredChannelDriver: live
accountId: sut
flow:

3
qa/scenarios/channels/matrix-approval-channel-target-both.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.native-approval-prompts
execution:
profiles: { "matrix:all": 30, "matrix:transport": 29 }
kind: flow
channel: matrix
timeoutMs: 90000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
approvalForwarding:
exec: true

3
qa/scenarios/channels/matrix-approval-deny-reaction.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.native-approval-prompts
execution:
profiles: { "matrix:all": 28, "matrix:transport": 27 }
kind: flow
channel: matrix
timeoutMs: 75000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
approvalForwarding:
exec: true

3
qa/scenarios/channels/matrix-approval-exec-metadata-chunked.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.native-approval-prompts
execution:
profiles: { "matrix:all": 26, "matrix:fast": 6, "matrix:transport": 25 }
kind: flow
channel: matrix
timeoutMs: 90000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
approvalForwarding:
exec: true

3
qa/scenarios/channels/matrix-approval-exec-metadata-single-event.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.native-approval-prompts
execution:
profiles: { "matrix:all": 25, "matrix:fast": 5, "matrix:transport": 24 }
kind: flow
channel: matrix
timeoutMs: 75000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
approvalForwarding:
exec: true

3
qa/scenarios/channels/matrix-approval-plugin-metadata-single-event.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.native-approval-prompts
execution:
profiles: { "matrix:all": 27, "matrix:transport": 26 }
kind: flow
channel: matrix
timeoutMs: 75000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
approvalForwarding:
plugin: true

3
qa/scenarios/channels/matrix-approval-thread-target.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.native-approval-prompts
execution:
profiles: { "matrix:all": 29, "matrix:transport": 28 }
kind: flow
channel: matrix
timeoutMs: 75000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
approvalForwarding:
exec: true

3
qa/scenarios/channels/matrix-attachment-only-ignored.yaml
View File

@@ -8,12 +8,13 @@ scenario:
secondary:
- channels.inbound-media-normalization
execution:
profiles: { "matrix:all": 55, "matrix:media": 4 }
kind: flow
channel: matrix
timeoutMs: 8000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixTopology:
defaultRoomKey: main
rooms:

3
qa/scenarios/channels/matrix-dm-thread-reply-override.yaml
View File

@@ -8,12 +8,13 @@ scenario:
secondary:
- channels.dm
execution:
profiles: { "matrix:all": 16, "matrix:transport": 15 }
kind: flow
channel: matrix
timeoutMs: 45000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
dm:
threadReplies: always

3
qa/scenarios/channels/matrix-e2ee-artifact-redaction.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 64, "matrix:e2ee-smoke": 6 }
kind: flow
channel: matrix
timeoutMs: 150000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-basic-reply.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 58, "matrix:e2ee-smoke": 0, "matrix:fast": 11 }
kind: flow
channel: matrix
timeoutMs: 75000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-bootstrap-success.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 60, "matrix:e2ee-smoke": 2 }
kind: flow
channel: matrix
timeoutMs: 90000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-cli-account-add-enable-e2ee.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 84, "matrix:e2ee-cli": 0 }
kind: flow
channel: matrix
timeoutMs: 120000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-cli-encryption-setup-bootstrap-failure.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 87, "matrix:e2ee-cli": 3 }
kind: flow
channel: matrix
timeoutMs: 120000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-cli-encryption-setup-idempotent.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 86, "matrix:e2ee-cli": 2 }
kind: flow
channel: matrix
timeoutMs: 120000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-cli-encryption-setup-multi-account.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 90, "matrix:e2ee-cli": 6 }
kind: flow
channel: matrix
timeoutMs: 120000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-cli-encryption-setup.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 85, "matrix:e2ee-cli": 1 }
kind: flow
channel: matrix
timeoutMs: 120000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-cli-recovery-key-invalid.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 89, "matrix:e2ee-cli": 5 }
kind: flow
channel: matrix
timeoutMs: 120000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-cli-recovery-key-setup.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 88, "matrix:e2ee-cli": 4 }
kind: flow
channel: matrix
timeoutMs: 120000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-cli-self-verification.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 92, "matrix:e2ee-cli": 8 }
kind: flow
channel: matrix
timeoutMs: 180000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixTopology:
defaultRoomKey: main
rooms:

3
qa/scenarios/channels/matrix-e2ee-cli-setup-then-gateway-reply.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 91, "matrix:e2ee-cli": 7 }
kind: flow
channel: matrix
timeoutMs: 180000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: false
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-corrupt-crypto-idb-snapshot.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 73, "matrix:e2ee-deep": 7 }
kind: flow
channel: matrix
timeoutMs: 180000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-device-sas-verification.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 78, "matrix:e2ee-deep": 12 }
kind: flow
channel: matrix
timeoutMs: 90000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-dm-sas-verification.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.dm
execution:
profiles: { "matrix:all": 81, "matrix:e2ee-deep": 15 }
kind: flow
channel: matrix
timeoutMs: 90000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-history-exists-backup-empty.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 77, "matrix:e2ee-deep": 11 }
kind: flow
channel: matrix
timeoutMs: 180000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-key-bootstrap-failure.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 65, "matrix:e2ee-smoke": 7 }
kind: flow
channel: matrix
timeoutMs: 90000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-media-image.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.media-rich-content
execution:
profiles: { "matrix:all": 57, "matrix:media": 6 }
kind: flow
channel: matrix
timeoutMs: 180000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-qr-verification.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 79, "matrix:e2ee-deep": 13 }
kind: flow
channel: matrix
timeoutMs: 90000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

3
qa/scenarios/channels/matrix-e2ee-recovery-key-lifecycle.yaml
View File

@@ -6,12 +6,13 @@ scenario:
primary:
- channels.access-and-identity
execution:
profiles: { "matrix:all": 61, "matrix:e2ee-smoke": 3 }
kind: flow
channel: matrix
timeoutMs: 90000
retryCount: 0
driver: live
config:
requiredChannelDriver: live
matrixConfigOverrides:
encryption: true
startupVerification: off

Some files were not shown because too many files have changed in this diff Show More