Dallin Romney
9666db607e
test(qa): clean up smoke taxonomy profile ( #96320 )
2026-06-24 00:43:00 -07:00
Vincent Koc
3d8d45fb0d
fix(qa): reject out-of-range lab CLI ports
2026-06-23 10:16:45 +02:00
Dallin Romney
63b13ea837
feat(qa): crabline channel driver ( #91502 )
...
* feat(qa): add crabline channel driver seam
* feat: run crabline channel driver smoke
* chore: keep crabline qa dependency dev-only
* refactor(qa): keep crabline driver details opaque
* chore(qa): pin crabline to merged driver API
* feat(qa): drive channel driver from profiles
* fix(qa): declare crabline runtime peer
* feat(qa): resolve crabline channel from scenarios
* feat(qa): treat unsupported profile channels as coverage gaps
* Revert "feat(qa): treat unsupported profile channels as coverage gaps"
This reverts commit 65a9701655
2026-06-22 15:24:59 -07:00
Dallin Romney
e32929e12c
Add slim evidence mode for QA profile evidence ( #93179 )
...
* test(qa): compact profile evidence execution metadata
* docs(qa): document compact profile evidence
* test(qa): support compact evidence mode
* test(qa): rename compact evidence mode to slim
* docs(qa): trim slim evidence wording
* fix(qa): avoid commander runtime import
2026-06-15 14:50:40 -07:00
Dallin Romney
e8db9c3bc0
test(qa): add qa run --profile and unified output summary/evidence ( #91587 )
...
* test(qa): add mapped qa run profiles
* test(qa): document mapped profile runner
* test(qa): validate run profiles from mapping
* test(qa): preserve root profile parsing
* test(qa): simplify taxonomy profile dispatch
* test(qa): align tool coverage CLI expectation
* test(qa): fix profile dispatch fixture type
* test(qa): share profile runner option types
* test(qa): split shared cli runner options
* test(qa): unify profile suite artifacts
* fix(qa): filter profile scenarios by provider lane
* test(qa): drop native scenario subreports
* fix(qa): keep native log refs repo-relative
* fix(cli): preserve qa run root profile parsing
* fix(qa): avoid qa profile flag collision
* fix(qa): reject profile flags without qa profile
2026-06-14 18:08:42 -07:00
Dallin Romney
561b293c7a
Run Vitest and Playwright scenarios from qa suite ( #92606 )
...
* test(qa): run vitest and playwright scenarios from qa suite
* fix(qa): harden scenario suite dispatch
* refactor(qa): share scenario path utilities
* refactor(qa): share test file scenario runner
* refactor(qa): route test file scenarios through suite runtime
* refactor(qa): use explicit suite runtime result kind
* test(qa): write suite evidence artifact
* refactor(qa): clarify suite execution dispatch
* fix(qa): keep test-file scenarios out of flow-only runners
* refactor(qa): export mixed scenario suite runner
2026-06-13 01:06:10 -07:00
Peter Steinberger
58912f8fd8
docs: document channel extension sources
2026-06-04 21:59:00 -04:00
Peter Steinberger
304e2c83c0
chore(lint): enable stricter oxlint rules
2026-05-31 18:59:02 +01:00
Peter Steinberger
28a2043f51
fix(qa-lab): centralize cli integer parsing
2026-05-29 06:20:43 -04:00
Peter Steinberger
cd80b4efca
fix: parse qa cli integers strictly
2026-05-28 14:36:47 -04:00
Peter Steinberger
bb46b79d3c
refactor: internalize OpenClaw agent runtime ( #85341 )
...
* refactor: extract agent core package
Introduce packages/agent-core as the OpenClaw-owned home for reusable agent loop, harness, session, prompt, and runtime dependency contracts.
* refactor: extract shared llm runtime
Move provider model registries, stream wrappers, OAuth helpers, and LLM utilities into src/llm with plugin-sdk barrels instead of depending on the old embedded runtime layout.
* refactor: remove pi runtime internals
Rename remaining Pi-shaped agent surfaces to OpenClaw agent runtime names, delete obsolete Pi docs and package graph checks, and add the third-party notice for incorporated code.
* refactor: tighten agent session runtime
Make agent-core/runtime dependencies explicit, consolidate compaction and session transcript helpers, and move model/session helpers behind OpenClaw-owned contracts.
* refactor: remove static model and pi auth paths
Drop static model catalogs and Pi auth bridges, move model/provider facts to manifest-owned runtime contracts, and harden internal embedded-agent utilities.
* refactor: remove legacy provider compat paths
* docs: remove agent parity notes
* fix: skip provider wildcard metadata parsing
* refactor: share session extension sdk loading
* refactor: inline acpx proxy error formatter
* refactor: fold edit recovery into edit tool
* fix: accept extension batch separator
* test: align startup provider plugin expectations
* fix: restore provider-scoped release discovery
* test: align static asset packaging expectations
* fix: run static provider catalogs during scoped discovery
* fix: add provider entry catalogs for scoped live discovery
* fix: load lightweight provider catalog entries
* fix: refresh provider-scoped plugin metadata
* fix: keep provider catalog entries on release live path
* fix: keep static manifest models in release live checks
* fix: harden release model discovery
* fix: reduce OpenAI live cache probe reasoning
* fix: disable OpenAI cache probe reasoning
* ci: extend OpenAI gateway live timeout
* fix: extend live gateway model budget
* fix: stabilize release validation regressions
* fix: honor provider aliases in model rows
* fix: stabilize release validation lanes
* fix: stabilize release memory qa
* ci: stabilize release validation lanes
* ci: prefer ipv4 for live docker node calls
* fix: restore shared tool-call stream wrapper
* ci: remove legacy pi test shard alias
* fix: clean up embedded agent test drift
* fix: stabilize runtime alias status
* fix: clean up embedded agent ci drift
* fix: restore release ci invariants
* fix: clean up post-rebase runtime drift
* fix: restore release ci checks
* fix: restore release ci after rebase
* fix: remove stale pi runtime path
* test: align compaction runtime expectations
* test: update plugin prerelease expectations
* fix: handle claude live tool approvals
* fix: stabilize release validation gates
* fix: finish agent runtime import
* test: finish post-rebase agent runtime mocks
* fix: keep codex compaction native
* fix: stabilize codex app-server hook tests
* test: isolate codex diagnostic active run
* test: remove codex diagnostic completion race
# Conflicts:
# extensions/codex/src/app-server/run-attempt.test.ts
* ci: fix full release manifest performance run id
* refactor: narrow llm plugin sdk boundary
* chore: drop generated google boundary stamps
* fix: repair rebase fallout
* fix: clean up rebased runtime references
* fix: decode codex jwt payloads as base64url
* fix: preserve shipped pi runtime alias
* fix: add scoped sdk virtual modules
* fix: decode llm codex oauth jwt as base64url
* fix: avoid stale vertex adc negative cache
* fix: harden tool arg decoding and codeql path
* fix: keep vertex adc negative checks live
* refactor: consolidate codex jwt and edit helpers
* fix: await codex oauth node runtime imports
* fix: preserve sdk tool and notice contracts
* fix: preserve shipped compat config boundaries
* fix: align codex oauth callback host
* fix: terminate agent-core loop streams on failure
* fix: keep codex oauth callback alive during fallback
* ci: include session tools in critical codeql scans
* fix: keep Cloudflare Anthropic provider auth header
* docs: redirect legacy pi runtime pages
* fix: honor bundled web provider compat discovery
* fix: protect session output spill files
* fix: keep legacy agent dir env blocked
* fix: contain auto-discovered skill symlinks
* fix: harden agent core sdk proxy surfaces
* fix: restore approval reaction sdk compat
* fix: keep live docker runs bounded
* fix: keep codex oauth redirect host aligned
* fix: resolve post-rebase agent runtime drift
* fix: redact anthropic oauth parse failures
* fix: preserve responses strict tool shaping
* fix: repair agent runtime rebase cleanup
* docs: redirect retired parity pages
* fix: bound auto-discovered resources to roots
* fix: repair post-rebase agent test drift
* fix: preserve bundled provider allowlist migration
* fix: preserve manifest-owned provider aliases
* fix: declare photon image dependency
* fix: keep provider headers out of proxy body
* fix: preserve shipped env aliases
* fix: refresh control ui i18n generated state
* fix: quote read fallback paths
* fix: preview edits through configured backend
* test: satisfy core test typecheck
* fix: preserve ZAI usage auth fallback
* test: repair codex diagnostic test
* fix: repair agent runtime rebase drift
* test: finish embedded runner import rename
* fix: repair agent runtime rebase integrations
* test: align compaction oauth fallback expectations
* fix: allow sdk-auth session models
* fix: update doctor tool schema import
* fix: preserve bedrock plugin region
* fix: stream harmony-like prose immediately
* ci: include session runtime in codeql shards
* fix: repair latest rebase integrations
* fix: honor explicit codex websocket transport
* fix: keep openai-compatible credentials provider-scoped
* fix: refresh sdk api baseline after rebase
* fix: route cli runtime aliases through openclaw harness
* test: rename stale harness mock expectation
* test: rename embedded agent overflow calls
* test: clean embedded auth test wording
* test: use openclaw stream types in deepinfra cache test
* fix: refresh sdk api baseline on latest main
* fix: honor bundled discovery compat allowlists
* fix: refresh sdk api baseline after latest rebase
* fix: remove stale rebase imports
* test: rename stale model catalog mock
* test: mock renamed doctor runtime modules
* fix: map canonical kimi env auth
* fix: use internal model registry in bench script
* fix: migrate deepinfra provider catalog entry
* fix: enforce builtin tool suppression
* fix: route compaction auth and proxy payloads safely
* refactor: prune unused llm registry leftovers
* test: update codex hooks session import
* test: fix model picker ci coverage
* test: align model picker auth mock types
2026-05-27 19:24:04 +01:00
Peter Steinberger
a1fe86a0ff
feat(qa): add coverage scenario matching
2026-05-25 10:22:51 +01:00
Kevin Lin
5656f687c1
Add Slack approval QA checkpoints ( #85141 )
...
* test: add slack approval qa checkpoints
* fix(slack): scope plugin approval session fallback
* ci(mantis): allow slack approval checkpoint dispatch
* ci(mantis): use on-demand aws slack desktops
* ci(mantis): run slack smoke from candidate checkout
* ci(mantis): pin aws ssh ingress to runner
* test(mantis): skip crabbox actions hydrate for slack desktop
* ci(mantis): use fresh pr checkout for slack desktop
* ci(mantis): start slack desktop smoke from source
* fix(mantis): use relative slack qa output dir
* test(mantis): surface slack smoke failure logs
* fix(mantis): write slack approval watcher script
* fix(mantis): accept successful slack qa metadata
* fix(mantis): tighten slack approval evidence
* fix(mantis): repair slack evidence manifest
* fix(mantis): render slack approval checkpoint proof
* fix(mantis): quote approval checkpoint renderer html
* fix(mantis): preserve slack approval failure artifacts
* fix(mantis): timeout silent slack desktop runs
* fix(mantis): keep slack desktop runs chatty
* fix(mantis): keep slack workflow harness trusted
* fix(qa-lab): make slack approval evidence robust
* fix(qa-lab): harden slack approval workflow proof
* test(qa-lab): surface slack approval diagnostics
* test(qa-lab): loosen slack approval readiness
2026-05-22 22:04:15 -07:00
Vincent Koc
cf0657852f
feat(qa-lab): add jsonl replay harness
2026-05-21 23:03:51 +08:00
Vincent Koc
79212f9869
feat(qa-lab): select runtime parity tiers
2026-05-18 00:21:13 +08:00
Vincent Koc
1926982c4c
fix(qa-lab): refresh parity model targets
2026-05-17 23:12:26 +08:00
Vincent Koc
d3a7348ad9
fix(qa-lab): keep bootstrap tokens private
2026-05-17 21:05:01 +08:00
Vincent Koc
1f9d8c1e9d
fix(qa-lab): wire tool coverage report command
2026-05-17 17:12:10 +08:00
Vincent Koc
da8afe359d
feat(qa-lab): add scenario pack selector
2026-05-17 09:23:48 +08:00
Peter Steinberger
f6d5d1cc4e
test: guard qa lab cli mock calls
2026-05-12 00:57:48 +01:00
Peter Steinberger
94a7dade57
test: tighten qa cli assertions
2026-05-11 06:54:11 +01:00
Peter Steinberger
378da8b9d3
feat: add telegram mantis evidence builder
2026-05-10 06:39:54 +01:00
Ayaan Zaidi
5cd4996205
feat(qa-lab): list telegram live scenarios
2026-05-08 16:14:42 +05:30
Peter Steinberger
9ef37d1907
test: tighten assertions and harness coverage
2026-05-08 05:28:12 +01:00
Peter Steinberger
f632f5e60b
feat(qa): add mantis Slack desktop smoke
2026-05-04 03:47:27 +01:00
Peter Steinberger
4856cbb017
feat(qa): publish Mantis desktop screenshots
2026-05-04 01:52:14 +01:00
Peter Steinberger
57b2d29761
feat(qa): add Mantis desktop browser smoke
2026-05-04 01:30:20 +01:00
Peter Steinberger
d4af125b52
feat(qa): add Mantis before-after CLI
2026-05-03 21:27:43 +01:00
Peter Steinberger
0bf06e953f
feat: add Mantis Discord smoke runner ( #76696 )
...
* docs: add Mantis QA system design
* feat: add Mantis Discord smoke runner
* fix: harden Mantis Discord smoke
* fix: redact Mantis Discord artifacts
* fix: satisfy Mantis redaction lint
* fix: redact Mantis mismatch failures
* test: avoid promise assertions in Mantis tests
2026-05-03 15:25:56 +01:00
Josh Avant
d5b326523f
qa-lab: make live lanes CI-ready for v1 E2E automation ( #69122 )
...
* qa-lab: harden CI defaults and failure semantics for live lanes
* qa-lab: add unit tests for suite progress logging defaults
* qa-lab: cover malformed multipass summary edge cases
* qa-lab: share suite summary failure counting helper
* qa-lab: test allow-failures parse wiring and sanitize progress ids
* fix: note qa CI live-lane defaults in changelog (#69122 ) (thanks @joshavant)
2026-04-19 21:13:27 -05:00
Gustavo Madeira Santana
3a1e469732
QA: track scenario coverage intent
2026-04-17 14:05:49 -04:00
Gustavo Madeira Santana
bb7e9823a8
QA Lab: add provider registry
...
Move mock and live provider behavior behind provider-owned definitions so suite, manual, Matrix, and transport lanes share defaults, auth staging, model config, and standalone server startup.
Add AIMock as a first-class local provider mode while keeping mock-openai as the scenario-aware deterministic lane.
2026-04-16 21:24:59 -04:00
Gustavo Madeira Santana
85eac42d34
QA: remove runner install fallback catalog
...
Drop the generated qa-runner catalog and the missing/install placeholder
path for repo-private QA runners. The host should discover bundled QA
commands from manifest plus runtime surface only.
Also trim stale qa-matrix install docs and package metadata so the
source-only QA policy stays consistent.
2026-04-14 17:37:18 -04:00
Gustavo Madeira Santana
82a2db71e8
refactor(qa): split Matrix QA into optional plugin ( #66723 )
...
Merged via squash.
Prepared head SHA: 27241bd0895599352+gumadeiras@users.noreply.github.com >
Co-authored-by: gumadeiras <5599352+gumadeiras@users.noreply.github.com >
Reviewed-by: @gumadeiras
2026-04-14 16:28:57 -04:00
Josh Avant
3d07dfbb65
feat(qa-lab): add Convex credential broker and admin CLI ( #65596 )
...
* QA Lab: add Convex credential source for Telegram lane
* QA Lab: scaffold Convex credential broker
* QA Lab: add Convex credential admin CLI
* QA Lab: harden Convex credential security paths
* QA Broker: validate Telegram payloads on admin add
* fix: note QA Convex credential broker in changelog (#65596 ) (thanks @joshavant)
2026-04-12 22:03:42 -05:00
Gustavo Madeira Santana
25445a9f2e
qa-lab: add Matrix live transport QA lane ( #64489 )
...
Merged via squash.
Prepared head SHA: ae9bb377515599352+gumadeiras@users.noreply.github.com >
Co-authored-by: gumadeiras <5599352+gumadeiras@users.noreply.github.com >
Reviewed-by: @gumadeiras
2026-04-10 19:35:08 -04:00
