test: expand slack live qa coverage (#77713)

docs/concepts/qa-e2e-automation.md

@@ -111,6 +111,10 @@ pnpm openclaw qa matrix --profile fast --fail-fast
 
 The full CLI reference, profile/scenario catalog, env vars, and artifact layout for this lane live in [Matrix QA](/concepts/qa-matrix). At a glance: it provisions a disposable Tuwunel homeserver in Docker, registers temporary driver/SUT/observer users, runs the real Matrix plugin inside a child QA gateway scoped to that transport (no `qa-channel`), then writes a Markdown report, JSON summary, observed-events artifact, and combined output log under `.artifacts/qa-e2e/matrix-<timestamp>/`.
 
+The scenarios cover transport behavior that unit tests cannot prove end to end: mention gating, allow-bot policies, allowlists, top-level and threaded replies, DM routing, reaction handling, inbound edit suppression, restart replay dedupe, homeserver interruption recovery, approval metadata delivery, media handling, and Matrix E2EE bootstrap/recovery/verification flows. The E2EE CLI profile also drives `openclaw matrix encryption setup` and verification commands through the same disposable homeserver before checking gateway replies.
+
+CI uses the same command surface in `.github/workflows/qa-live-transports-convex.yml`. Scheduled and default manual runs execute the fast Matrix profile with live frontier credentials, `--fast`, and `OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS=3000`. Manual `matrix_profile=all` fans out into the five profile shards so the exhaustive catalog can run in parallel while keeping one artifact directory per shard.
+
 For transport-real Telegram, Discord, and Slack smoke lanes:
 
 ```bash
@@ -195,7 +199,7 @@ Live transport lanes share one contract instead of each inventing their own scen
 | Matrix   | x      | x              | x          | x               | x               | x              | x                | x                | x                    |              |                             |
 | Telegram | x      | x              | x          |                 |                 |                |                  |                  |                      | x            |                             |
 | Discord  | x      | x              | x          |                 |                 |                |                  |                  |                      |              | x                           |
-| Slack    | x      | x              | x          |                 |                 |                |                  |                  |                      |              |                             |
+| Slack    | x      | x              | x          | x               | x               | x              | x                | x                |                      |              |                             |
 
 This keeps `qa-channel` as the broad product-behavior suite while Matrix,
 Telegram, and future live transports share one explicit transport-contract
@@ -349,6 +353,11 @@ Scenarios (`extensions/qa-lab/src/live-transports/slack/slack-live.runtime.ts:39
 
 - `slack-canary`
 - `slack-mention-gating`
+- `slack-allowlist-block`
+- `slack-top-level-reply-shape`
+- `slack-restart-resume`
+- `slack-thread-follow-up`
+- `slack-thread-isolation`
 
 Output artifacts:
 
@@ -367,7 +376,7 @@ The lane needs two distinct Slack apps in one workspace, plus a channel both bot
 
 Prefer a Slack workspace dedicated to QA over reusing a production workspace.
 
-The SUT manifest below mirrors the bundled Slack plugin's production install (`extensions/slack/src/setup-shared.ts:10`). For the production-channel setup as users see it, see [Slack channel quick setup](/channels/slack#quick-setup); the QA Driver/SUT pair is intentionally separate because the lane needs two distinct bot user ids in one workspace.
+The SUT manifest below intentionally narrows the bundled Slack plugin's production install (`extensions/slack/src/setup-shared.ts:10`) to the permissions and events covered by the live Slack QA suite. For the production-channel setup as users see it, see [Slack channel quick setup](/channels/slack#quick-setup); the QA Driver/SUT pair is intentionally separate because the lane needs two distinct bot user ids in one workspace.
 
 **1. Create the Driver app**
 
@@ -400,7 +409,7 @@ Copy the _Bot User OAuth Token_ (`xoxb-...`) — that becomes `driverBotToken`.
 
 **2. Create the SUT app**
 
-Repeat _Create New App → From a manifest_ in the same workspace. The scope set mirrors the bundled Slack plugin's production install (`extensions/slack/src/setup-shared.ts:10`):
+Repeat _Create New App → From a manifest_ in the same workspace. This QA app intentionally uses a narrower version of the bundled Slack plugin's production manifest (`extensions/slack/src/setup-shared.ts:10`): reaction scopes and events are omitted because the live Slack QA suite does not cover reaction handling yet.
 
 ```json
 {
@@ -441,8 +450,6 @@ Repeat _Create New App → From a manifest_ in the same workspace. The scope set
         "mpim:write",
         "pins:read",
         "pins:write",
-        "reactions:read",
-        "reactions:write",
         "usergroups:read",
         "users:read"
       ]
@@ -462,9 +469,7 @@ Repeat _Create New App → From a manifest_ in the same workspace. The scope set
         "message.im",
         "message.mpim",
         "pin_added",
-        "pin_removed",
-        "reaction_added",
-        "reaction_removed"
+        "pin_removed"
       ]
     }
   }

extensions/qa-lab/src/live-transports/slack/slack-live.runtime.test.ts

@@ -49,7 +49,15 @@ describe("Slack live QA runtime helpers", () => {
   });
 
   it("reports standard live transport scenario coverage", () => {
-    expect(__testing.SLACK_QA_STANDARD_SCENARIO_IDS).toEqual(["canary", "mention-gating"]);
+    expect(__testing.SLACK_QA_STANDARD_SCENARIO_IDS).toEqual([
+      "canary",
+      "mention-gating",
+      "allowlist-block",
+      "top-level-reply-shape",
+      "restart-resume",
+      "thread-follow-up",
+      "thread-isolation",
+    ]);
   });
 
   it("selects Slack scenarios by id", () => {

extensions/qa-lab/src/live-transports/slack/slack-live.runtime.ts

@@ -33,16 +33,51 @@ type SlackQaRuntimeEnv = {
   sutAppToken: string;
 };
 
-type SlackQaScenarioId = "slack-canary" | "slack-mention-gating";
+type SlackQaScenarioId =
+  | "slack-allowlist-block"
+  | "slack-canary"
+  | "slack-mention-gating"
+  | "slack-restart-resume"
+  | "slack-thread-follow-up"
+  | "slack-thread-isolation"
+  | "slack-top-level-reply-shape";
 
 type SlackQaScenarioRun = {
   expectReply: boolean;
   input: string;
   matchText: string;
+  verify?: (message: SlackMessage, context: { requestThreadTs: string; sentTs: string }) => void;
+  beforeRun?: (context: Omit<SlackQaScenarioContext, "sentTs">) => Promise<SlackQaBeforeRunResult>;
+  afterReply?: (message: SlackMessage, context: SlackQaScenarioContext) => Promise<string | void>;
+};
+
+type SlackQaBeforeRunResult =
+  | string
+  | void
+  | {
+      details?: string;
+      inputThreadTs?: string;
+    };
+
+type SlackQaConfigOverrides = {
+  replyToMode?: "all" | "off";
+  users?: string[];
+};
+
+type SlackQaScenarioContext = {
+  channelId: string;
+  driverClient: WebClient;
+  gateway: Awaited<ReturnType<typeof startQaGatewayChild>>;
+  postSlackMessage: (params: { text: string; threadTs?: string }) => Promise<{ ts: string }>;
+  sentTs: string;
+  sutIdentity: SlackAuthIdentity;
+  sutReadClient: WebClient;
+  waitForReady: () => Promise<void>;
 };
 
 type SlackQaScenarioDefinition = LiveTransportScenarioDefinition<SlackQaScenarioId> & {
   buildRun: (sutUserId: string) => SlackQaScenarioRun;
+  configOverrides?: SlackQaConfigOverrides;
 };
 
 type SlackAuthIdentity = {
@@ -127,6 +162,7 @@ type SlackCredentialHeartbeat = ReturnType<typeof startQaCredentialLeaseHeartbea
 
 const SLACK_QA_CAPTURE_CONTENT_ENV = "OPENCLAW_QA_SLACK_CAPTURE_CONTENT";
 const QA_REDACT_PUBLIC_METADATA_ENV = "OPENCLAW_QA_REDACT_PUBLIC_METADATA";
+const SLACK_QA_WEB_API_TIMEOUT_MS = 45_000;
 const SLACK_QA_ENV_KEYS = [
   "OPENCLAW_QA_SLACK_CHANNEL_ID",
   "OPENCLAW_QA_SLACK_DRIVER_BOT_TOKEN",
@@ -167,6 +203,11 @@ const slackHistorySchema = z.object({
   messages: z.array(slackHistoryMessageSchema).optional(),
 });
 
+const slackRepliesSchema = z.object({
+  ok: z.boolean().optional(),
+  messages: z.array(slackHistoryMessageSchema).optional(),
+});
+
 const SLACK_QA_SCENARIOS: SlackQaScenarioDefinition[] = [
   {
     id: "slack-canary",
@@ -196,6 +237,145 @@ const SLACK_QA_SCENARIOS: SlackQaScenarioDefinition[] = [
       };
     },
   },
+  {
+    id: "slack-allowlist-block",
+    standardId: "allowlist-block",
+    title: "Slack non-allowlisted sender does not trigger",
+    timeoutMs: 8_000,
+    configOverrides: { users: ["U_OPENCLAW_QA_NEVER_ALLOWED"] },
+    buildRun: (sutUserId) => {
+      const token = `SLACK_QA_BLOCK_${randomUUID().slice(0, 8).toUpperCase()}`;
+      return {
+        expectReply: false,
+        input: `<@${sutUserId}> reply with only this exact marker: ${token}`,
+        matchText: token,
+      };
+    },
+  },
+  {
+    id: "slack-top-level-reply-shape",
+    standardId: "top-level-reply-shape",
+    title: "Slack top-level reply stays top-level",
+    timeoutMs: 45_000,
+    configOverrides: { replyToMode: "off" },
+    buildRun: (sutUserId) => {
+      const token = `SLACK_QA_TOPLEVEL_${randomUUID().slice(0, 8).toUpperCase()}`;
+      return {
+        expectReply: true,
+        input: `<@${sutUserId}> reply with only this exact marker: ${token}`,
+        matchText: token,
+        verify: (message) => {
+          if (message.thread_ts) {
+            throw new Error(
+              `expected top-level Slack reply without thread_ts; got ${message.thread_ts}`,
+            );
+          }
+        },
+      };
+    },
+  },
+  {
+    id: "slack-restart-resume",
+    standardId: "restart-resume",
+    title: "Slack replies after gateway restart",
+    timeoutMs: 60_000,
+    buildRun: (sutUserId) => {
+      const token = `SLACK_QA_RESTART_${randomUUID().slice(0, 8).toUpperCase()}`;
+      return {
+        expectReply: true,
+        input: `<@${sutUserId}> reply with only this exact marker: ${token}`,
+        matchText: token,
+        afterReply: async (_message, context) => {
+          const secondToken = `SLACK_QA_RESTART_AFTER_${randomUUID().slice(0, 8).toUpperCase()}`;
+          await context.gateway.restart();
+          await context.waitForReady();
+          const sent = await sendSlackChannelMessage({
+            channelId: context.channelId,
+            client: context.driverClient,
+            text: `<@${context.sutIdentity.userId}> reply with only this exact marker: ${secondToken}`,
+          });
+          await waitForSlackScenarioReply({
+            channelId: context.channelId,
+            client: context.sutReadClient,
+            matchText: secondToken,
+            observedMessages: [],
+            observationScenarioId: "slack-restart-resume",
+            observationScenarioTitle: "Slack replies after gateway restart",
+            sentTs: sent.ts,
+            sutIdentity: context.sutIdentity,
+            timeoutMs: 45_000,
+          });
+          return `post-restart reply matched marker ${secondToken}`;
+        },
+      };
+    },
+  },
+  {
+    id: "slack-thread-follow-up",
+    standardId: "thread-follow-up",
+    title: "Slack threaded prompt receives threaded reply",
+    timeoutMs: 45_000,
+    configOverrides: { replyToMode: "all" },
+    buildRun: (sutUserId) => {
+      const token = `SLACK_QA_THREAD_${randomUUID().slice(0, 8).toUpperCase()}`;
+      return {
+        expectReply: true,
+        input: `<@${sutUserId}> reply with only this exact marker: ${token}`,
+        matchText: token,
+        beforeRun: async (context) => {
+          const parent = await context.postSlackMessage({
+            text: `thread-follow-up root for ${token}`,
+          });
+          return {
+            details: `created thread root ${parent.ts}`,
+            inputThreadTs: parent.ts,
+          };
+        },
+        verify: (message, context) => {
+          if (message.thread_ts !== context.requestThreadTs) {
+            throw new Error(
+              `expected threaded Slack reply thread_ts=${context.requestThreadTs}; got ${
+                message.thread_ts ?? "<none>"
+              }`,
+            );
+          }
+        },
+      };
+    },
+  },
+  {
+    id: "slack-thread-isolation",
+    standardId: "thread-isolation",
+    title: "Slack fresh top-level prompt stays out of previous thread",
+    timeoutMs: 45_000,
+    configOverrides: { replyToMode: "off" },
+    buildRun: (sutUserId) => {
+      const token = `SLACK_QA_ISOLATION_${randomUUID().slice(0, 8).toUpperCase()}`;
+      return {
+        expectReply: true,
+        input: `<@${sutUserId}> reply with only this exact marker: ${token}`,
+        matchText: token,
+        beforeRun: async (context) => {
+          const priorThreadToken = `SLACK_QA_PRIOR_THREAD_${randomUUID().slice(0, 8).toUpperCase()}`;
+          const parent = await context.postSlackMessage({
+            text: `prior thread root for ${priorThreadToken}`,
+          });
+          await context.postSlackMessage({
+            text: `prior thread child for ${priorThreadToken}`,
+            threadTs: parent.ts,
+          });
+          return `created unrelated prior thread ${parent.ts}`;
+        },
+        verify: (message) => {
+          if (message.thread_ts) {
+            throw new Error(
+              `expected isolated top-level Slack reply; got thread_ts=${message.thread_ts}`,
+            );
+          }
+        },
+      };
+    },
+  },
 ];
 
 const SLACK_QA_STANDARD_SCENARIO_IDS = collectLiveTransportStandardScenarioCoverage({
@@ -279,6 +459,7 @@ function buildSlackQaConfig(
   params: {
     channelId: string;
     driverBotUserId: string;
+    overrides?: SlackQaConfigOverrides;
     sutAccountId: string;
     sutAppToken: string;
     sutBotToken: string;
@@ -315,12 +496,13 @@ function buildSlackQaConfig(
             appToken: params.sutAppToken,
             groupPolicy: "allowlist",
             allowBots: true,
+            replyToMode: params.overrides?.replyToMode ?? "off",
             channels: {
               [params.channelId]: {
                 enabled: true,
                 requireMention: true,
                 allowBots: true,
-                users: [params.driverBotUserId],
+                users: params.overrides?.users ?? [params.driverBotUserId],
               },
             },
           },
@@ -331,7 +513,7 @@ function buildSlackQaConfig(
 }
 
 async function getSlackIdentity(token: string): Promise<SlackAuthIdentity> {
-  const client = createSlackWebClient(token, { timeout: 15_000 });
+  const client = createSlackWebClient(token, { timeout: SLACK_QA_WEB_API_TIMEOUT_MS });
   const auth = slackAuthTestSchema.parse(await client.auth.test());
   if (!auth.user_id) {
     throw new Error("Slack auth.test did not return user_id.");
@@ -347,12 +529,14 @@ async function sendSlackChannelMessage(params: {
   channelId: string;
   client: WebClient;
   text: string;
+  threadTs?: string;
 }) {
   const sendSlackMessage = params.client.chat.postMessage.bind(params.client.chat);
   const sent = slackPostMessageSchema.parse(
     await sendSlackMessage({
       channel: params.channelId,
       text: params.text,
+      thread_ts: params.threadTs,
       unfurl_links: false,
       unfurl_media: false,
     }),
@@ -379,6 +563,22 @@ async function listSlackMessages(params: {
   return history.messages ?? [];
 }
 
+async function listSlackThreadMessages(params: {
+  channelId: string;
+  client: WebClient;
+  threadTs: string;
+}) {
+  const replies = slackRepliesSchema.parse(
+    await params.client.conversations.replies({
+      channel: params.channelId,
+      inclusive: true,
+      limit: 50,
+      ts: params.threadTs,
+    }),
+  );
+  return replies.messages ?? [];
+}
+
 function isSutSlackMessage(message: SlackMessage, sutIdentity: SlackAuthIdentity) {
   return (
     (message.user !== undefined && message.user === sutIdentity.userId) ||
@@ -394,16 +594,12 @@ async function waitForSlackScenarioReply(params: {
   observationScenarioId: string;
   observationScenarioTitle: string;
   sentTs: string;
+  threadTs?: string;
   sutIdentity: SlackAuthIdentity;
   timeoutMs: number;
 }) {
   const startedAt = Date.now();
-  while (Date.now() - startedAt < params.timeoutMs) {
-    const messages = await listSlackMessages({
-      channelId: params.channelId,
-      client: params.client,
-      oldestTs: params.sentTs,
-    });
+  const inspectMessages = (messages: SlackMessage[]) => {
     for (const message of messages) {
       const text = message.text ?? "";
       if (
@@ -432,6 +628,36 @@ async function waitForSlackScenarioReply(params: {
         };
       }
     }
+    return undefined;
+  };
+
+  while (Date.now() - startedAt < params.timeoutMs) {
+    const channelMessages = await listSlackMessages({
+      channelId: params.channelId,
+      client: params.client,
+      oldestTs: params.sentTs,
+    });
+    const channelReply = inspectMessages(channelMessages);
+    if (channelReply) {
+      return channelReply;
+    }
+
+    try {
+      const threadMessages = await listSlackThreadMessages({
+        channelId: params.channelId,
+        client: params.client,
+        threadTs: params.threadTs ?? params.sentTs,
+      });
+      const threadReply = inspectMessages(threadMessages);
+      if (threadReply) {
+        return threadReply;
+      }
+    } catch (error) {
+      throw new Error(
+        `Slack conversations.replies failed while waiting for ${params.observationScenarioId}: ${formatErrorMessage(error)}`,
+        { cause: error },
+      );
+    }
     await new Promise((resolve) => setTimeout(resolve, 1_000));
   }
   throw new Error(`timed out after ${params.timeoutMs}ms waiting for Slack message`);
@@ -665,107 +891,142 @@ export async function runSlackQaLive(params: {
     }
 
     const driverClient = createSlackWriteClient(activeRuntimeEnv.driverBotToken, {
-      timeout: 15_000,
+      timeout: SLACK_QA_WEB_API_TIMEOUT_MS,
     });
-    const sutReadClient = createSlackWebClient(activeRuntimeEnv.sutBotToken, { timeout: 15_000 });
-    const gatewayHarness = await startQaLiveLaneGateway({
-      repoRoot,
-      transport: {
-        requiredPluginIds: [],
-        createGatewayConfig: () => ({}),
-      },
-      transportBaseUrl: "http://127.0.0.1:0",
-      providerMode,
-      primaryModel,
-      alternateModel,
-      fastMode: params.fastMode,
-      controlUiEnabled: false,
-      mutateConfig: (cfg) =>
-        buildSlackQaConfig(cfg, {
-          channelId: activeRuntimeEnv.channelId,
-          driverBotUserId: driverIdentity.userId,
-          sutAccountId,
-          sutAppToken: activeRuntimeEnv.sutAppToken,
-          sutBotToken: activeRuntimeEnv.sutBotToken,
-        }),
+    const sutReadClient = createSlackWebClient(activeRuntimeEnv.sutBotToken, {
+      timeout: SLACK_QA_WEB_API_TIMEOUT_MS,
     });
-    try {
-      await waitForSlackChannelRunning(gatewayHarness.gateway, sutAccountId);
-      assertLeaseHealthy();
-      for (const scenario of scenarios) {
+    for (const scenario of scenarios) {
+      let gatewayHarness: Awaited<ReturnType<typeof startQaLiveLaneGateway>> | undefined;
+      try {
         assertLeaseHealthy();
+        gatewayHarness = await startQaLiveLaneGateway({
+          repoRoot,
+          transport: {
+            requiredPluginIds: [],
+            createGatewayConfig: () => ({}),
+          },
+          transportBaseUrl: "http://127.0.0.1:0",
+          providerMode,
+          primaryModel,
+          alternateModel,
+          fastMode: params.fastMode,
+          controlUiEnabled: false,
+          mutateConfig: (cfg) =>
+            buildSlackQaConfig(cfg, {
+              channelId: activeRuntimeEnv.channelId,
+              driverBotUserId: driverIdentity.userId,
+              overrides: scenario.configOverrides,
+              sutAccountId,
+              sutAppToken: activeRuntimeEnv.sutAppToken,
+              sutBotToken: activeRuntimeEnv.sutBotToken,
+            }),
+        });
+        const activeGatewayHarness = gatewayHarness;
+        await waitForSlackChannelRunning(activeGatewayHarness.gateway, sutAccountId);
         const scenarioRun = scenario.buildRun(sutIdentity.userId);
+        const baseScenarioContext = {
+          channelId: activeRuntimeEnv.channelId,
+          driverClient,
+          gateway: activeGatewayHarness.gateway,
+          postSlackMessage: async (message: { text: string; threadTs?: string }) =>
+            await sendSlackChannelMessage({
+              channelId: activeRuntimeEnv.channelId,
+              client: driverClient,
+              text: message.text,
+              threadTs: message.threadTs,
+            }),
+          sutIdentity,
+          sutReadClient,
+          waitForReady: async () =>
+            await waitForSlackChannelRunning(activeGatewayHarness.gateway, sutAccountId),
+        };
+        const beforeRunResult = await scenarioRun.beforeRun?.(baseScenarioContext);
+        const beforeRunDetails =
+          typeof beforeRunResult === "string" ? beforeRunResult : beforeRunResult?.details;
         const requestStartedAt = new Date();
-        try {
-          const sent = await sendSlackChannelMessage({
+        const sent = await sendSlackChannelMessage({
+          channelId: activeRuntimeEnv.channelId,
+          client: driverClient,
+          text: scenarioRun.input,
+          threadTs:
+            typeof beforeRunResult === "object" ? beforeRunResult?.inputThreadTs : undefined,
+        });
+        const requestThreadTs =
+          (typeof beforeRunResult === "object" ? beforeRunResult?.inputThreadTs : undefined) ??
+          sent.ts;
+        if (scenarioRun.expectReply) {
+          const reply = await waitForSlackScenarioReply({
             channelId: activeRuntimeEnv.channelId,
-            client: driverClient,
-            text: scenarioRun.input,
+            client: sutReadClient,
+            matchText: scenarioRun.matchText,
+            observedMessages,
+            observationScenarioId: scenario.id,
+            observationScenarioTitle: scenario.title,
+            sentTs: sent.ts,
+            threadTs: requestThreadTs,
+            sutIdentity,
+            timeoutMs: scenario.timeoutMs,
           });
-          if (scenarioRun.expectReply) {
-            const reply = await waitForSlackScenarioReply({
-              channelId: activeRuntimeEnv.channelId,
-              client: sutReadClient,
-              matchText: scenarioRun.matchText,
-              observedMessages,
-              observationScenarioId: scenario.id,
-              observationScenarioTitle: scenario.title,
-              sentTs: sent.ts,
-              sutIdentity,
-              timeoutMs: scenario.timeoutMs,
-            });
-            const responseObservedAt = new Date(reply.observedAt);
-            const rttMs = responseObservedAt.getTime() - requestStartedAt.getTime();
-            scenarioResults.push({
-              id: scenario.id,
-              title: scenario.title,
-              status: "pass",
-              details: `reply matched in ${rttMs}ms`,
-              rttMs,
-              requestStartedAt: requestStartedAt.toISOString(),
-              responseObservedAt: responseObservedAt.toISOString(),
-            });
-          } else {
-            await waitForSlackNoReply({
-              channelId: activeRuntimeEnv.channelId,
-              client: sutReadClient,
-              matchText: scenarioRun.matchText,
-              observedMessages,
-              observationScenarioId: scenario.id,
-              observationScenarioTitle: scenario.title,
-              sentTs: sent.ts,
-              sutIdentity,
-              timeoutMs: scenario.timeoutMs,
-            });
-            scenarioResults.push({
-              id: scenario.id,
-              title: scenario.title,
-              status: "pass",
-              details: "no reply",
-            });
-          }
-        } catch (error) {
-          const result = {
+          scenarioRun.verify?.(reply.message, { requestThreadTs, sentTs: sent.ts });
+          const responseObservedAt = new Date(reply.observedAt);
+          const rttMs = responseObservedAt.getTime() - requestStartedAt.getTime();
+          const afterReplyDetails = await scenarioRun.afterReply?.(reply.message, {
+            ...baseScenarioContext,
+            sentTs: sent.ts,
+          });
+          scenarioResults.push({
             id: scenario.id,
             title: scenario.title,
-            status: "fail" as const,
-            details: formatErrorMessage(error),
-          };
-          scenarioResults.push(result);
-          preservedGatewayDebugArtifacts = true;
-          await gatewayHarness.gateway
+            status: "pass",
+            details: [`reply matched in ${rttMs}ms`, beforeRunDetails, afterReplyDetails]
+              .filter(Boolean)
+              .join("; "),
+            rttMs,
+            requestStartedAt: requestStartedAt.toISOString(),
+            responseObservedAt: responseObservedAt.toISOString(),
+          });
+        } else {
+          await waitForSlackNoReply({
+            channelId: activeRuntimeEnv.channelId,
+            client: sutReadClient,
+            matchText: scenarioRun.matchText,
+            observedMessages,
+            observationScenarioId: scenario.id,
+            observationScenarioTitle: scenario.title,
+            sentTs: sent.ts,
+            sutIdentity,
+            timeoutMs: scenario.timeoutMs,
+          });
+          scenarioResults.push({
+            id: scenario.id,
+            title: scenario.title,
+            status: "pass",
+            details: "no reply",
+          });
+        }
+      } catch (error) {
+        scenarioResults.push({
+          id: scenario.id,
+          title: scenario.title,
+          status: "fail",
+          details: formatErrorMessage(error),
+        });
+        preservedGatewayDebugArtifacts = true;
+        if (gatewayHarness) {
+          await gatewayHarness
             .stop({ keepTemp: true, preserveToDir: gatewayDebugDirPath })
             .catch((stopError) => {
               appendLiveLaneIssue(cleanupIssues, "gateway debug preservation failed", stopError);
             });
-          break;
         }
-      }
-    } finally {
-      if (!preservedGatewayDebugArtifacts) {
-        await gatewayHarness.stop().catch((error) => {
-          appendLiveLaneIssue(cleanupIssues, "gateway stop failed", error);
-        });
+        break;
+      } finally {
+        if (!preservedGatewayDebugArtifacts && gatewayHarness) {
+          await gatewayHarness.stop().catch((error) => {
+            appendLiveLaneIssue(cleanupIssues, "gateway stop failed", error);
+          });
+        }
       }
     }
   } catch (error) {