feat(qa): add Mantis Discord status reaction scenario (#76747)

* feat(qa): add Mantis Discord status reaction scenario

* fix(qa): retry Discord rate limits in Mantis runs

* refactor(qa): reuse Discord API retry helper

* fix(qa): import Discord API through package surface

* fix(ci): generate Discord boundary declarations

* fix(ci): keep xai boundary overrides stable
This commit is contained in:
Peter Steinberger
2026-05-03 17:00:06 +01:00
committed by GitHub
parent 1e8de7661e
commit 77a50db9ea
15 changed files with 990 additions and 130 deletions

View File

@@ -84,14 +84,16 @@ pnpm openclaw qa mantis run \
```
The GitHub smoke workflow is `Mantis Discord Smoke`. The before and after GitHub
workflow should accept equivalent inputs:
workflow for the first real scenario is `Mantis Discord Status Reactions`. It
accepts:
- `transport`: `discord` for the first version.
- `scenario`: one or more scenario ids.
- `baseline_ref`: default `origin/main` or the linked issue's reported bad tag.
- `candidate_ref`: the PR head SHA.
- `machine_provider`: `aws` by default, with later `hetzner` fallback.
- `post_to_pr`: whether ClawSweeper should comment with the result.
- `baseline_ref`: the ref expected to reproduce queued-only behavior.
- `candidate_ref`: the ref expected to show `queued -> thinking -> done`.
It checks out the workflow harness ref, builds separate baseline and candidate
worktrees, runs `discord-status-reactions-tool-only` against each worktree, and
uploads `baseline/`, `candidate/`, `comparison.json`, and `mantis-report.md` as
Actions artifacts.
ClawSweeper command examples:
@@ -179,6 +181,25 @@ lifecycle transition in tool-only mode. Candidate evidence should show lifecycle
status reactions running when `messages.statusReactions.enabled` is explicitly
true.
The executable first slice is the opt-in Discord live QA scenario:
```bash
pnpm openclaw qa discord \
--scenario discord-status-reactions-tool-only \
--provider-mode live-frontier \
--model openai/gpt-5.4 \
--alt-model openai/gpt-5.4 \
--fast \
--output-dir .artifacts/qa-e2e/mantis/discord-status-reactions-candidate
```
It configures the SUT with always-on guild handling, `visibleReplies:
"message_tool"`, `ackReaction: "👀"`, and explicit status reactions. The oracle
polls the real Discord triggering message and expects the observed sequence
`👀 -> 🤔 -> 👍`. Artifacts include `discord-qa-reaction-timelines.json`,
`discord-status-reactions-tool-only-timeline.html`, and
`discord-status-reactions-tool-only-timeline.png`.
## Existing QA Pieces
Mantis should build on the existing private QA stack instead of starting from

View File

@@ -225,7 +225,7 @@ Output artifacts:
pnpm openclaw qa discord
```
Targets one real private Discord guild channel with two bots: a driver bot controlled by the harness and a SUT bot started by the child OpenClaw gateway through the bundled Discord plugin. Verifies channel mention handling and that the SUT bot has registered the native `/help` command with Discord.
Targets one real private Discord guild channel with two bots: a driver bot controlled by the harness and a SUT bot started by the child OpenClaw gateway through the bundled Discord plugin. Verifies channel mention handling, that the SUT bot has registered the native `/help` command with Discord, and opt-in Mantis evidence scenarios.
Required env when `--credential-source env`:
@@ -244,12 +244,25 @@ Scenarios (`extensions/qa-lab/src/live-transports/discord/discord-live.runtime.t
- `discord-canary`
- `discord-mention-gating`
- `discord-native-help-command-registration`
- `discord-status-reactions-tool-only` — opt-in Mantis scenario. Runs by itself because it switches the SUT to always-on, tool-only guild replies with `messages.statusReactions.enabled=true`, then captures a REST reaction timeline plus an HTML/PNG visual artifact.
Run the Mantis status-reaction scenario explicitly:
```bash
pnpm openclaw qa discord \
--scenario discord-status-reactions-tool-only \
--provider-mode live-frontier \
--model openai/gpt-5.4 \
--alt-model openai/gpt-5.4 \
--fast
```
Output artifacts:
- `discord-qa-report.md`
- `discord-qa-summary.json`
- `discord-qa-observed-messages.json` — bodies redacted unless `OPENCLAW_QA_DISCORD_CAPTURE_CONTENT=1`.
- `discord-qa-reaction-timelines.json` and `discord-status-reactions-tool-only-timeline.png` when the status-reaction scenario runs.
### Convex credential pool