diff --git a/docs/concepts/qa-e2e-automation.md b/docs/concepts/qa-e2e-automation.md index 7393b29340c..72706a59ebd 100644 --- a/docs/concepts/qa-e2e-automation.md +++ b/docs/concepts/qa-e2e-automation.md @@ -111,6 +111,10 @@ pnpm openclaw qa matrix --profile fast --fail-fast The full CLI reference, profile/scenario catalog, env vars, and artifact layout for this lane live in [Matrix QA](/concepts/qa-matrix). At a glance: it provisions a disposable Tuwunel homeserver in Docker, registers temporary driver/SUT/observer users, runs the real Matrix plugin inside a child QA gateway scoped to that transport (no `qa-channel`), then writes a Markdown report, JSON summary, observed-events artifact, and combined output log under `.artifacts/qa-e2e/matrix-/`. +The scenarios cover transport behavior that unit tests cannot prove end to end: mention gating, allow-bot policies, allowlists, top-level and threaded replies, DM routing, reaction handling, inbound edit suppression, restart replay dedupe, homeserver interruption recovery, approval metadata delivery, media handling, and Matrix E2EE bootstrap/recovery/verification flows. The E2EE CLI profile also drives `openclaw matrix encryption setup` and verification commands through the same disposable homeserver before checking gateway replies. + +CI uses the same command surface in `.github/workflows/qa-live-transports-convex.yml`. Scheduled and default manual runs execute the fast Matrix profile with live frontier credentials, `--fast`, and `OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS=3000`. Manual `matrix_profile=all` fans out into the five profile shards so the exhaustive catalog can run in parallel while keeping one artifact directory per shard. + For transport-real Telegram, Discord, and Slack smoke lanes: ```bash @@ -195,7 +199,7 @@ Live transport lanes share one contract instead of each inventing their own scen | Matrix | x | x | x | x | x | x | x | x | x | | | | Telegram | x | x | x | | | | | | | x | | | Discord | x | x | x | | | | | | | | x | -| Slack | x | x | x | | | | | | | | | +| Slack | x | x | x | x | x | x | x | x | | | | This keeps `qa-channel` as the broad product-behavior suite while Matrix, Telegram, and future live transports share one explicit transport-contract @@ -349,6 +353,11 @@ Scenarios (`extensions/qa-lab/src/live-transports/slack/slack-live.runtime.ts:39 - `slack-canary` - `slack-mention-gating` +- `slack-allowlist-block` +- `slack-top-level-reply-shape` +- `slack-restart-resume` +- `slack-thread-follow-up` +- `slack-thread-isolation` Output artifacts: @@ -367,7 +376,7 @@ The lane needs two distinct Slack apps in one workspace, plus a channel both bot Prefer a Slack workspace dedicated to QA over reusing a production workspace. -The SUT manifest below mirrors the bundled Slack plugin's production install (`extensions/slack/src/setup-shared.ts:10`). For the production-channel setup as users see it, see [Slack channel quick setup](/channels/slack#quick-setup); the QA Driver/SUT pair is intentionally separate because the lane needs two distinct bot user ids in one workspace. +The SUT manifest below intentionally narrows the bundled Slack plugin's production install (`extensions/slack/src/setup-shared.ts:10`) to the permissions and events covered by the live Slack QA suite. For the production-channel setup as users see it, see [Slack channel quick setup](/channels/slack#quick-setup); the QA Driver/SUT pair is intentionally separate because the lane needs two distinct bot user ids in one workspace. **1. Create the Driver app** @@ -400,7 +409,7 @@ Copy the _Bot User OAuth Token_ (`xoxb-...`) — that becomes `driverBotToken`. **2. Create the SUT app** -Repeat _Create New App → From a manifest_ in the same workspace. The scope set mirrors the bundled Slack plugin's production install (`extensions/slack/src/setup-shared.ts:10`): +Repeat _Create New App → From a manifest_ in the same workspace. This QA app intentionally uses a narrower version of the bundled Slack plugin's production manifest (`extensions/slack/src/setup-shared.ts:10`): reaction scopes and events are omitted because the live Slack QA suite does not cover reaction handling yet. ```json { @@ -441,8 +450,6 @@ Repeat _Create New App → From a manifest_ in the same workspace. The scope set "mpim:write", "pins:read", "pins:write", - "reactions:read", - "reactions:write", "usergroups:read", "users:read" ] @@ -462,9 +469,7 @@ Repeat _Create New App → From a manifest_ in the same workspace. The scope set "message.im", "message.mpim", "pin_added", - "pin_removed", - "reaction_added", - "reaction_removed" + "pin_removed" ] } } diff --git a/extensions/qa-lab/src/live-transports/slack/slack-live.runtime.test.ts b/extensions/qa-lab/src/live-transports/slack/slack-live.runtime.test.ts index 97228a77b2e..ba78b9cc3d0 100644 --- a/extensions/qa-lab/src/live-transports/slack/slack-live.runtime.test.ts +++ b/extensions/qa-lab/src/live-transports/slack/slack-live.runtime.test.ts @@ -49,7 +49,15 @@ describe("Slack live QA runtime helpers", () => { }); it("reports standard live transport scenario coverage", () => { - expect(__testing.SLACK_QA_STANDARD_SCENARIO_IDS).toEqual(["canary", "mention-gating"]); + expect(__testing.SLACK_QA_STANDARD_SCENARIO_IDS).toEqual([ + "canary", + "mention-gating", + "allowlist-block", + "top-level-reply-shape", + "restart-resume", + "thread-follow-up", + "thread-isolation", + ]); }); it("selects Slack scenarios by id", () => { diff --git a/extensions/qa-lab/src/live-transports/slack/slack-live.runtime.ts b/extensions/qa-lab/src/live-transports/slack/slack-live.runtime.ts index be13f65b0e1..f12ec03956e 100644 --- a/extensions/qa-lab/src/live-transports/slack/slack-live.runtime.ts +++ b/extensions/qa-lab/src/live-transports/slack/slack-live.runtime.ts @@ -33,16 +33,51 @@ type SlackQaRuntimeEnv = { sutAppToken: string; }; -type SlackQaScenarioId = "slack-canary" | "slack-mention-gating"; +type SlackQaScenarioId = + | "slack-allowlist-block" + | "slack-canary" + | "slack-mention-gating" + | "slack-restart-resume" + | "slack-thread-follow-up" + | "slack-thread-isolation" + | "slack-top-level-reply-shape"; type SlackQaScenarioRun = { expectReply: boolean; input: string; matchText: string; + verify?: (message: SlackMessage, context: { requestThreadTs: string; sentTs: string }) => void; + beforeRun?: (context: Omit) => Promise; + afterReply?: (message: SlackMessage, context: SlackQaScenarioContext) => Promise; +}; + +type SlackQaBeforeRunResult = + | string + | void + | { + details?: string; + inputThreadTs?: string; + }; + +type SlackQaConfigOverrides = { + replyToMode?: "all" | "off"; + users?: string[]; +}; + +type SlackQaScenarioContext = { + channelId: string; + driverClient: WebClient; + gateway: Awaited>; + postSlackMessage: (params: { text: string; threadTs?: string }) => Promise<{ ts: string }>; + sentTs: string; + sutIdentity: SlackAuthIdentity; + sutReadClient: WebClient; + waitForReady: () => Promise; }; type SlackQaScenarioDefinition = LiveTransportScenarioDefinition & { buildRun: (sutUserId: string) => SlackQaScenarioRun; + configOverrides?: SlackQaConfigOverrides; }; type SlackAuthIdentity = { @@ -127,6 +162,7 @@ type SlackCredentialHeartbeat = ReturnType { + const token = `SLACK_QA_BLOCK_${randomUUID().slice(0, 8).toUpperCase()}`; + return { + expectReply: false, + input: `<@${sutUserId}> reply with only this exact marker: ${token}`, + matchText: token, + }; + }, + }, + { + id: "slack-top-level-reply-shape", + standardId: "top-level-reply-shape", + title: "Slack top-level reply stays top-level", + timeoutMs: 45_000, + configOverrides: { replyToMode: "off" }, + buildRun: (sutUserId) => { + const token = `SLACK_QA_TOPLEVEL_${randomUUID().slice(0, 8).toUpperCase()}`; + return { + expectReply: true, + input: `<@${sutUserId}> reply with only this exact marker: ${token}`, + matchText: token, + verify: (message) => { + if (message.thread_ts) { + throw new Error( + `expected top-level Slack reply without thread_ts; got ${message.thread_ts}`, + ); + } + }, + }; + }, + }, + { + id: "slack-restart-resume", + standardId: "restart-resume", + title: "Slack replies after gateway restart", + timeoutMs: 60_000, + buildRun: (sutUserId) => { + const token = `SLACK_QA_RESTART_${randomUUID().slice(0, 8).toUpperCase()}`; + return { + expectReply: true, + input: `<@${sutUserId}> reply with only this exact marker: ${token}`, + matchText: token, + afterReply: async (_message, context) => { + const secondToken = `SLACK_QA_RESTART_AFTER_${randomUUID().slice(0, 8).toUpperCase()}`; + await context.gateway.restart(); + await context.waitForReady(); + const sent = await sendSlackChannelMessage({ + channelId: context.channelId, + client: context.driverClient, + text: `<@${context.sutIdentity.userId}> reply with only this exact marker: ${secondToken}`, + }); + await waitForSlackScenarioReply({ + channelId: context.channelId, + client: context.sutReadClient, + matchText: secondToken, + observedMessages: [], + observationScenarioId: "slack-restart-resume", + observationScenarioTitle: "Slack replies after gateway restart", + sentTs: sent.ts, + sutIdentity: context.sutIdentity, + timeoutMs: 45_000, + }); + return `post-restart reply matched marker ${secondToken}`; + }, + }; + }, + }, + { + id: "slack-thread-follow-up", + standardId: "thread-follow-up", + title: "Slack threaded prompt receives threaded reply", + timeoutMs: 45_000, + configOverrides: { replyToMode: "all" }, + buildRun: (sutUserId) => { + const token = `SLACK_QA_THREAD_${randomUUID().slice(0, 8).toUpperCase()}`; + return { + expectReply: true, + input: `<@${sutUserId}> reply with only this exact marker: ${token}`, + matchText: token, + beforeRun: async (context) => { + const parent = await context.postSlackMessage({ + text: `thread-follow-up root for ${token}`, + }); + return { + details: `created thread root ${parent.ts}`, + inputThreadTs: parent.ts, + }; + }, + verify: (message, context) => { + if (message.thread_ts !== context.requestThreadTs) { + throw new Error( + `expected threaded Slack reply thread_ts=${context.requestThreadTs}; got ${ + message.thread_ts ?? "" + }`, + ); + } + }, + }; + }, + }, + { + id: "slack-thread-isolation", + standardId: "thread-isolation", + title: "Slack fresh top-level prompt stays out of previous thread", + timeoutMs: 45_000, + configOverrides: { replyToMode: "off" }, + buildRun: (sutUserId) => { + const token = `SLACK_QA_ISOLATION_${randomUUID().slice(0, 8).toUpperCase()}`; + return { + expectReply: true, + input: `<@${sutUserId}> reply with only this exact marker: ${token}`, + matchText: token, + beforeRun: async (context) => { + const priorThreadToken = `SLACK_QA_PRIOR_THREAD_${randomUUID().slice(0, 8).toUpperCase()}`; + const parent = await context.postSlackMessage({ + text: `prior thread root for ${priorThreadToken}`, + }); + await context.postSlackMessage({ + text: `prior thread child for ${priorThreadToken}`, + threadTs: parent.ts, + }); + return `created unrelated prior thread ${parent.ts}`; + }, + verify: (message) => { + if (message.thread_ts) { + throw new Error( + `expected isolated top-level Slack reply; got thread_ts=${message.thread_ts}`, + ); + } + }, + }; + }, + }, ]; const SLACK_QA_STANDARD_SCENARIO_IDS = collectLiveTransportStandardScenarioCoverage({ @@ -279,6 +459,7 @@ function buildSlackQaConfig( params: { channelId: string; driverBotUserId: string; + overrides?: SlackQaConfigOverrides; sutAccountId: string; sutAppToken: string; sutBotToken: string; @@ -315,12 +496,13 @@ function buildSlackQaConfig( appToken: params.sutAppToken, groupPolicy: "allowlist", allowBots: true, + replyToMode: params.overrides?.replyToMode ?? "off", channels: { [params.channelId]: { enabled: true, requireMention: true, allowBots: true, - users: [params.driverBotUserId], + users: params.overrides?.users ?? [params.driverBotUserId], }, }, }, @@ -331,7 +513,7 @@ function buildSlackQaConfig( } async function getSlackIdentity(token: string): Promise { - const client = createSlackWebClient(token, { timeout: 15_000 }); + const client = createSlackWebClient(token, { timeout: SLACK_QA_WEB_API_TIMEOUT_MS }); const auth = slackAuthTestSchema.parse(await client.auth.test()); if (!auth.user_id) { throw new Error("Slack auth.test did not return user_id."); @@ -347,12 +529,14 @@ async function sendSlackChannelMessage(params: { channelId: string; client: WebClient; text: string; + threadTs?: string; }) { const sendSlackMessage = params.client.chat.postMessage.bind(params.client.chat); const sent = slackPostMessageSchema.parse( await sendSlackMessage({ channel: params.channelId, text: params.text, + thread_ts: params.threadTs, unfurl_links: false, unfurl_media: false, }), @@ -379,6 +563,22 @@ async function listSlackMessages(params: { return history.messages ?? []; } +async function listSlackThreadMessages(params: { + channelId: string; + client: WebClient; + threadTs: string; +}) { + const replies = slackRepliesSchema.parse( + await params.client.conversations.replies({ + channel: params.channelId, + inclusive: true, + limit: 50, + ts: params.threadTs, + }), + ); + return replies.messages ?? []; +} + function isSutSlackMessage(message: SlackMessage, sutIdentity: SlackAuthIdentity) { return ( (message.user !== undefined && message.user === sutIdentity.userId) || @@ -394,16 +594,12 @@ async function waitForSlackScenarioReply(params: { observationScenarioId: string; observationScenarioTitle: string; sentTs: string; + threadTs?: string; sutIdentity: SlackAuthIdentity; timeoutMs: number; }) { const startedAt = Date.now(); - while (Date.now() - startedAt < params.timeoutMs) { - const messages = await listSlackMessages({ - channelId: params.channelId, - client: params.client, - oldestTs: params.sentTs, - }); + const inspectMessages = (messages: SlackMessage[]) => { for (const message of messages) { const text = message.text ?? ""; if ( @@ -432,6 +628,36 @@ async function waitForSlackScenarioReply(params: { }; } } + return undefined; + }; + + while (Date.now() - startedAt < params.timeoutMs) { + const channelMessages = await listSlackMessages({ + channelId: params.channelId, + client: params.client, + oldestTs: params.sentTs, + }); + const channelReply = inspectMessages(channelMessages); + if (channelReply) { + return channelReply; + } + + try { + const threadMessages = await listSlackThreadMessages({ + channelId: params.channelId, + client: params.client, + threadTs: params.threadTs ?? params.sentTs, + }); + const threadReply = inspectMessages(threadMessages); + if (threadReply) { + return threadReply; + } + } catch (error) { + throw new Error( + `Slack conversations.replies failed while waiting for ${params.observationScenarioId}: ${formatErrorMessage(error)}`, + { cause: error }, + ); + } await new Promise((resolve) => setTimeout(resolve, 1_000)); } throw new Error(`timed out after ${params.timeoutMs}ms waiting for Slack message`); @@ -665,107 +891,142 @@ export async function runSlackQaLive(params: { } const driverClient = createSlackWriteClient(activeRuntimeEnv.driverBotToken, { - timeout: 15_000, + timeout: SLACK_QA_WEB_API_TIMEOUT_MS, }); - const sutReadClient = createSlackWebClient(activeRuntimeEnv.sutBotToken, { timeout: 15_000 }); - const gatewayHarness = await startQaLiveLaneGateway({ - repoRoot, - transport: { - requiredPluginIds: [], - createGatewayConfig: () => ({}), - }, - transportBaseUrl: "http://127.0.0.1:0", - providerMode, - primaryModel, - alternateModel, - fastMode: params.fastMode, - controlUiEnabled: false, - mutateConfig: (cfg) => - buildSlackQaConfig(cfg, { - channelId: activeRuntimeEnv.channelId, - driverBotUserId: driverIdentity.userId, - sutAccountId, - sutAppToken: activeRuntimeEnv.sutAppToken, - sutBotToken: activeRuntimeEnv.sutBotToken, - }), + const sutReadClient = createSlackWebClient(activeRuntimeEnv.sutBotToken, { + timeout: SLACK_QA_WEB_API_TIMEOUT_MS, }); - try { - await waitForSlackChannelRunning(gatewayHarness.gateway, sutAccountId); - assertLeaseHealthy(); - for (const scenario of scenarios) { + for (const scenario of scenarios) { + let gatewayHarness: Awaited> | undefined; + try { assertLeaseHealthy(); + gatewayHarness = await startQaLiveLaneGateway({ + repoRoot, + transport: { + requiredPluginIds: [], + createGatewayConfig: () => ({}), + }, + transportBaseUrl: "http://127.0.0.1:0", + providerMode, + primaryModel, + alternateModel, + fastMode: params.fastMode, + controlUiEnabled: false, + mutateConfig: (cfg) => + buildSlackQaConfig(cfg, { + channelId: activeRuntimeEnv.channelId, + driverBotUserId: driverIdentity.userId, + overrides: scenario.configOverrides, + sutAccountId, + sutAppToken: activeRuntimeEnv.sutAppToken, + sutBotToken: activeRuntimeEnv.sutBotToken, + }), + }); + const activeGatewayHarness = gatewayHarness; + await waitForSlackChannelRunning(activeGatewayHarness.gateway, sutAccountId); const scenarioRun = scenario.buildRun(sutIdentity.userId); + const baseScenarioContext = { + channelId: activeRuntimeEnv.channelId, + driverClient, + gateway: activeGatewayHarness.gateway, + postSlackMessage: async (message: { text: string; threadTs?: string }) => + await sendSlackChannelMessage({ + channelId: activeRuntimeEnv.channelId, + client: driverClient, + text: message.text, + threadTs: message.threadTs, + }), + sutIdentity, + sutReadClient, + waitForReady: async () => + await waitForSlackChannelRunning(activeGatewayHarness.gateway, sutAccountId), + }; + const beforeRunResult = await scenarioRun.beforeRun?.(baseScenarioContext); + const beforeRunDetails = + typeof beforeRunResult === "string" ? beforeRunResult : beforeRunResult?.details; const requestStartedAt = new Date(); - try { - const sent = await sendSlackChannelMessage({ + const sent = await sendSlackChannelMessage({ + channelId: activeRuntimeEnv.channelId, + client: driverClient, + text: scenarioRun.input, + threadTs: + typeof beforeRunResult === "object" ? beforeRunResult?.inputThreadTs : undefined, + }); + const requestThreadTs = + (typeof beforeRunResult === "object" ? beforeRunResult?.inputThreadTs : undefined) ?? + sent.ts; + if (scenarioRun.expectReply) { + const reply = await waitForSlackScenarioReply({ channelId: activeRuntimeEnv.channelId, - client: driverClient, - text: scenarioRun.input, + client: sutReadClient, + matchText: scenarioRun.matchText, + observedMessages, + observationScenarioId: scenario.id, + observationScenarioTitle: scenario.title, + sentTs: sent.ts, + threadTs: requestThreadTs, + sutIdentity, + timeoutMs: scenario.timeoutMs, }); - if (scenarioRun.expectReply) { - const reply = await waitForSlackScenarioReply({ - channelId: activeRuntimeEnv.channelId, - client: sutReadClient, - matchText: scenarioRun.matchText, - observedMessages, - observationScenarioId: scenario.id, - observationScenarioTitle: scenario.title, - sentTs: sent.ts, - sutIdentity, - timeoutMs: scenario.timeoutMs, - }); - const responseObservedAt = new Date(reply.observedAt); - const rttMs = responseObservedAt.getTime() - requestStartedAt.getTime(); - scenarioResults.push({ - id: scenario.id, - title: scenario.title, - status: "pass", - details: `reply matched in ${rttMs}ms`, - rttMs, - requestStartedAt: requestStartedAt.toISOString(), - responseObservedAt: responseObservedAt.toISOString(), - }); - } else { - await waitForSlackNoReply({ - channelId: activeRuntimeEnv.channelId, - client: sutReadClient, - matchText: scenarioRun.matchText, - observedMessages, - observationScenarioId: scenario.id, - observationScenarioTitle: scenario.title, - sentTs: sent.ts, - sutIdentity, - timeoutMs: scenario.timeoutMs, - }); - scenarioResults.push({ - id: scenario.id, - title: scenario.title, - status: "pass", - details: "no reply", - }); - } - } catch (error) { - const result = { + scenarioRun.verify?.(reply.message, { requestThreadTs, sentTs: sent.ts }); + const responseObservedAt = new Date(reply.observedAt); + const rttMs = responseObservedAt.getTime() - requestStartedAt.getTime(); + const afterReplyDetails = await scenarioRun.afterReply?.(reply.message, { + ...baseScenarioContext, + sentTs: sent.ts, + }); + scenarioResults.push({ id: scenario.id, title: scenario.title, - status: "fail" as const, - details: formatErrorMessage(error), - }; - scenarioResults.push(result); - preservedGatewayDebugArtifacts = true; - await gatewayHarness.gateway + status: "pass", + details: [`reply matched in ${rttMs}ms`, beforeRunDetails, afterReplyDetails] + .filter(Boolean) + .join("; "), + rttMs, + requestStartedAt: requestStartedAt.toISOString(), + responseObservedAt: responseObservedAt.toISOString(), + }); + } else { + await waitForSlackNoReply({ + channelId: activeRuntimeEnv.channelId, + client: sutReadClient, + matchText: scenarioRun.matchText, + observedMessages, + observationScenarioId: scenario.id, + observationScenarioTitle: scenario.title, + sentTs: sent.ts, + sutIdentity, + timeoutMs: scenario.timeoutMs, + }); + scenarioResults.push({ + id: scenario.id, + title: scenario.title, + status: "pass", + details: "no reply", + }); + } + } catch (error) { + scenarioResults.push({ + id: scenario.id, + title: scenario.title, + status: "fail", + details: formatErrorMessage(error), + }); + preservedGatewayDebugArtifacts = true; + if (gatewayHarness) { + await gatewayHarness .stop({ keepTemp: true, preserveToDir: gatewayDebugDirPath }) .catch((stopError) => { appendLiveLaneIssue(cleanupIssues, "gateway debug preservation failed", stopError); }); - break; } - } - } finally { - if (!preservedGatewayDebugArtifacts) { - await gatewayHarness.stop().catch((error) => { - appendLiveLaneIssue(cleanupIssues, "gateway stop failed", error); - }); + break; + } finally { + if (!preservedGatewayDebugArtifacts && gatewayHarness) { + await gatewayHarness.stop().catch((error) => { + appendLiveLaneIssue(cleanupIssues, "gateway stop failed", error); + }); + } } } } catch (error) {