Peter Steinberger
7f73f6e66e
test(qa-lab): refresh CLI dispatch expectations ( #107018 )
2026-07-13 19:36:17 -07:00
Vincent Koc
4d15c47e9f
fix(qa): keep runtime parity lanes reproducible
2026-07-12 15:10:38 +08:00
Vincent Koc
4040267af9
fix(qa): run mixed channel suites
2026-07-12 06:51:59 +02:00
Peter Steinberger
fa77fe10d5
chore: migrate active GPT-5.5 references to GPT-5.6 ( #104452 )
...
* chore(models): migrate active GPT-5.5 references
* test(workboard): expect GPT-5.6 Sol default
* chore: keep release notes in PR body
* test(models): align picker fixtures with Sol default
* test: update PDF default model expectation
* test(qa): migrate thinking smoke to Luna
* test(gateway): align mock catalog with Sol default
* ci: retrigger exact-head PR checks
* test(gateway): document default catalog invariant
2026-07-11 06:30:57 -07:00
Peter Steinberger
afa3212e90
fix(release): stage Telegram QA inside runtime root ( #104125 )
2026-07-10 22:39:09 -07:00
Vincent Koc
2a1d6e49d5
fix(ci): run Telegram release QA from trusted harness ( #103207 )
...
* fix(ci): use trusted Telegram QA harness
* fix(ci): restrict trusted Telegram QA candidate
* fix(ci): isolate trusted Telegram QA candidate
* fix(ci): harden Telegram QA process boundary
* fix(ci): pass Telegram QA release gates
* test(qa): stub Telegram env explicitly
* fix(ci): harden Telegram release QA boundary
* test(ci): harden trusted workflow and memory guards
2026-07-10 11:32:24 -07:00
Peter Steinberger
4bc300843d
fix(qa): keep runtime parity tiers flow-compatible ( #103609 )
...
* fix(qa): keep runtime parity tiers flow-compatible
Refs #103588
* test(qa): follow canonical frontier defaults
2026-07-10 10:51:13 +01:00
Vincent Koc
238be4b849
test(release): refresh beta3 gate expectations
2026-07-08 15:55:50 -07:00
Dallin Romney
c1b9e36221
test(qa): migrate channel thread and DM isolation scenarios ( #101112 )
...
* test(qa): migrate channel thread isolation scenarios
* test(qa): remove stale Matrix scenario fixture
* test(qa): use canonical coverage taxonomy ids
* test(qa): preserve Matrix flow runtime identities
* test(qa): expose Slack thread polling helper
* test(qa): select Matrix for native thread smoke
* test(qa): keep Matrix subagent spawn opt-in
* test(qa): target selected live account in scenario patches
2026-07-06 17:52:38 -07:00
Peter Steinberger
aed2482292
fix(qa): restore automatic smoke coverage across channels ( #101186 )
...
* fix(qa): restore channel-sharded smoke coverage
Co-authored-by: Peter Steinberger <steipete@gmail.com >
Co-authored-by: Dallin Romney <dallinromney@gmail.com >
* fix(qa): restore channel-sharded smoke coverage
* fix(qa): restore channel-sharded smoke coverage
* fix(qa): restore channel-sharded smoke coverage
---------
Co-authored-by: Vincent Koc <25068+vincentkoc@users.noreply.github.com >
Co-authored-by: Dallin Romney <dallinromney@gmail.com >
Co-authored-by: Vincent Koc <vincentkoc@ieee.org >
2026-07-06 17:09:40 -07:00
Dallin Romney
19491fe34c
Revert "fix(qa): keep smoke profile on one channel ( #101173 )" ( #101184 )
...
This reverts commit b773f7981a
2026-07-06 13:29:07 -07:00
Dallin Romney
b773f7981a
fix(qa): keep smoke profile on one channel ( #101173 )
2026-07-06 13:26:58 -07:00
Dallin Romney
e3671e0980
refactor(qa): canonicalize channel command scenarios ( #101080 )
...
* test(qa): migrate channel command scenarios
* test(qa): distinguish portable command headings
* test(qa): preserve canonical scenario ownership
2026-07-06 12:18:06 -07:00
Dallin Romney
b29d472e41
refactor(qa): add canonical live channel adapters ( #99707 )
2026-07-06 09:24:34 -07:00
Peter Steinberger
c757675f34
improve: keep isolated tests under one second ( #100019 )
...
* test: speed up isolated test suite
* test: finish isolated latency cleanup
* test: eliminate remaining isolated latency spikes
* test: remove final isolated timing outliers
* test: bound full-suite tooling processes
* test: bound native test process lifetime
* test: warm isolated runtime suites
* test: eliminate final isolated timing outliers
* test: fix isolated timing fixture types
* test: make timeout cleanup timing deterministic
* test: pin media manifests to source checkout
* test: isolate provider manifest contracts
* test: eliminate residual isolated timing spikes
* test: restore final isolated timing fixes
* test: eliminate remaining isolated timing spikes
* test: warm Zalo lifecycle imports
* test: keep isolated suites below one second
* test: use readable browser response fixtures
2026-07-05 10:57:19 -07:00
Dallin Romney
15bfadf2bd
fix: count maturity coverage ids ( #96543 )
2026-06-24 14:56:51 -07:00
Dallin Romney
f29dbd3ebd
test(qa): speed up smoke profile ( #96340 )
2026-06-24 09:30:59 -07:00
Dallin Romney
9666db607e
test(qa): clean up smoke taxonomy profile ( #96320 )
2026-06-24 00:43:00 -07:00
Dallin Romney
4460fa78c3
feat(qa): add "all" taxonomy profile ( #95947 )
...
* qa: add all maturity profile
* test: update qa coverage profile expectations
2026-06-22 20:49:08 -07:00
Vincent Koc
def4b51485
fix(qa): gate smoke profile scenarios by channel driver
2026-06-23 09:34:52 +08:00
Dallin Romney
de9c94cbbb
feat(qa): forward shared suite flags to multipass runner ( #91506 )
2026-06-22 15:48:05 -07:00
Dallin Romney
63b13ea837
feat(qa): crabline channel driver ( #91502 )
...
* feat(qa): add crabline channel driver seam
* feat: run crabline channel driver smoke
* chore: keep crabline qa dependency dev-only
* refactor(qa): keep crabline driver details opaque
* chore(qa): pin crabline to merged driver API
* feat(qa): drive channel driver from profiles
* fix(qa): declare crabline runtime peer
* feat(qa): resolve crabline channel from scenarios
* feat(qa): treat unsupported profile channels as coverage gaps
* Revert "feat(qa): treat unsupported profile channels as coverage gaps"
This reverts commit 65a9701655
2026-06-22 15:24:59 -07:00
Vincent Koc
abdd81db11
test(qa): include long-context soak scenario
2026-06-22 14:28:04 +08:00
Dallin Romney
e17d111990
fix: require all taxonomy coverage ids ( #94296 )
2026-06-17 16:38:14 -07:00
Vincent Koc
e2b6753b87
fix(qa-lab): bound credential payload reads
2026-06-17 11:59:55 +02:00
Vincent Koc
c32ba171db
fix(qa): fail unsuccessful self-checks
2026-06-16 01:32:47 +02:00
Dallin Romney
e32929e12c
Add slim evidence mode for QA profile evidence ( #93179 )
...
* test(qa): compact profile evidence execution metadata
* docs(qa): document compact profile evidence
* test(qa): support compact evidence mode
* test(qa): rename compact evidence mode to slim
* docs(qa): trim slim evidence wording
* fix(qa): avoid commander runtime import
2026-06-15 14:50:40 -07:00
Dallin Romney
1f8c4d3958
simplify QA evidence profile and mappings/coverage shape ( #93153 )
...
* test(qa): simplify evidence coverage shape
* test(qa): collapse evidence scorecard metadata
* test(qa): document evidence schema version
2026-06-14 22:26:58 -07:00
Dallin Romney
3d38c9a633
test(qa): embed profile scorecard evidence ( #93109 )
...
* test(qa): embed profile scorecard evidence
* test(qa): fix profile runner return lint
* test(qa): satisfy suite command lint return
2026-06-14 20:51:38 -07:00
Dallin Romney
e8db9c3bc0
test(qa): add qa run --profile and unified output summary/evidence ( #91587 )
...
* test(qa): add mapped qa run profiles
* test(qa): document mapped profile runner
* test(qa): validate run profiles from mapping
* test(qa): preserve root profile parsing
* test(qa): simplify taxonomy profile dispatch
* test(qa): align tool coverage CLI expectation
* test(qa): fix profile dispatch fixture type
* test(qa): share profile runner option types
* test(qa): split shared cli runner options
* test(qa): unify profile suite artifacts
* fix(qa): filter profile scenarios by provider lane
* test(qa): drop native scenario subreports
* fix(qa): keep native log refs repo-relative
* fix(cli): preserve qa run root profile parsing
* fix(qa): avoid qa profile flag collision
* fix(qa): reject profile flags without qa profile
2026-06-14 18:08:42 -07:00
Vincent Koc
b1caba5906
test(qa): align tool coverage CLI expectation
2026-06-14 16:11:00 +08:00
Dallin Romney
561b293c7a
Run Vitest and Playwright scenarios from qa suite ( #92606 )
...
* test(qa): run vitest and playwright scenarios from qa suite
* fix(qa): harden scenario suite dispatch
* refactor(qa): share scenario path utilities
* refactor(qa): share test file scenario runner
* refactor(qa): route test file scenarios through suite runtime
* refactor(qa): use explicit suite runtime result kind
* test(qa): write suite evidence artifact
* refactor(qa): clarify suite execution dispatch
* fix(qa): keep test-file scenarios out of flow-only runners
* refactor(qa): export mixed scenario suite runner
2026-06-13 01:06:10 -07:00
Dallin Romney
4809ac70fa
Add QA evidence artifact output ( #91484 )
...
* feat: add qa evidence summary normalization
* chore: rename qa evidence target environment
* chore: align qa evidence profile terminology
* chore: align qa evidence summary fields
* chore: add qa evidence taxonomy ref
* test: remove stale multipass evidence example
* test(qa): normalize vitest and playwright evidence
* test(qa): slim evidence summary metadata
* test(qa): clarify evidence summary inputs
* test(qa): rename scenario specs in evidence flow
* test(qa): treat evidence profiles as mapping strings
* test(qa): use neutral evidence test identity
* test(qa): nest evidence summary joins
* refactor(qa): normalize live evidence summaries
* fix(qa): accept normalized telegram rtt summaries
* fix(qa): normalize evidence lane summaries
* fix(qa): align evidence summaries with requirements
* refactor(qa): tighten evidence summary builders
* refactor(qa): restore standard evidence ids
* fix(qa): keep legacy summaries out of rtt evidence
* refactor(qa): make package evidence provenance explicit
* test(qa): keep script tests out of qa lab internals
* refactor(qa): rename scenario evidence definitions
* refactor(qa): clean evidence summary wording
* test(qa): fix evidence summary test inputs
* refactor(qa): simplify evidence identity fields
* refactor(qa): tighten evidence summary inputs
* refactor(qa): rename evidence artifact
2026-06-12 16:12:58 -07:00
Vincent Koc
c7b01cf201
test(release): stabilize qa runtime parity gate
2026-06-09 01:02:24 +02:00
Vincent Koc
3dc6ac3802
fix(qa): fail closed on skipped suite summaries
2026-06-07 11:20:25 +02:00
Vincent Koc
e12141fa9f
fix(qa): gate live transport exits on summaries
2026-06-07 02:53:50 +02:00
Vincent Koc
6d2566682a
fix(qa): fail suite on summary scenario failures
2026-06-07 02:49:02 +02:00
Peter Steinberger
58912f8fd8
docs: document channel extension sources
2026-06-04 21:59:00 -04:00
Peter Steinberger
1188aa3b81
feat: add Claude Opus 4.8 support ( #87890 )
...
* feat: add Claude Opus 4.8 support
* fix: omit Vertex Opus sampling overrides
* fix: preserve Opus adaptive thinking levels
* fix: clamp Anthropic max effort support
* fix: use sha256 for QA mock call ids
* fix: type Anthropic transport test model metadata
* test: update PDF model default for Opus 4.8
2026-05-29 06:10:42 +01:00
Peter Steinberger
a8dec44f56
fix(release): accept openclaw qa runtime alias
2026-05-28 21:22:17 +01:00
Peter Steinberger
cd80b4efca
fix: parse qa cli integers strictly
2026-05-28 14:36:47 -04:00
Peter Steinberger
bb46b79d3c
refactor: internalize OpenClaw agent runtime ( #85341 )
...
* refactor: extract agent core package
Introduce packages/agent-core as the OpenClaw-owned home for reusable agent loop, harness, session, prompt, and runtime dependency contracts.
* refactor: extract shared llm runtime
Move provider model registries, stream wrappers, OAuth helpers, and LLM utilities into src/llm with plugin-sdk barrels instead of depending on the old embedded runtime layout.
* refactor: remove pi runtime internals
Rename remaining Pi-shaped agent surfaces to OpenClaw agent runtime names, delete obsolete Pi docs and package graph checks, and add the third-party notice for incorporated code.
* refactor: tighten agent session runtime
Make agent-core/runtime dependencies explicit, consolidate compaction and session transcript helpers, and move model/session helpers behind OpenClaw-owned contracts.
* refactor: remove static model and pi auth paths
Drop static model catalogs and Pi auth bridges, move model/provider facts to manifest-owned runtime contracts, and harden internal embedded-agent utilities.
* refactor: remove legacy provider compat paths
* docs: remove agent parity notes
* fix: skip provider wildcard metadata parsing
* refactor: share session extension sdk loading
* refactor: inline acpx proxy error formatter
* refactor: fold edit recovery into edit tool
* fix: accept extension batch separator
* test: align startup provider plugin expectations
* fix: restore provider-scoped release discovery
* test: align static asset packaging expectations
* fix: run static provider catalogs during scoped discovery
* fix: add provider entry catalogs for scoped live discovery
* fix: load lightweight provider catalog entries
* fix: refresh provider-scoped plugin metadata
* fix: keep provider catalog entries on release live path
* fix: keep static manifest models in release live checks
* fix: harden release model discovery
* fix: reduce OpenAI live cache probe reasoning
* fix: disable OpenAI cache probe reasoning
* ci: extend OpenAI gateway live timeout
* fix: extend live gateway model budget
* fix: stabilize release validation regressions
* fix: honor provider aliases in model rows
* fix: stabilize release validation lanes
* fix: stabilize release memory qa
* ci: stabilize release validation lanes
* ci: prefer ipv4 for live docker node calls
* fix: restore shared tool-call stream wrapper
* ci: remove legacy pi test shard alias
* fix: clean up embedded agent test drift
* fix: stabilize runtime alias status
* fix: clean up embedded agent ci drift
* fix: restore release ci invariants
* fix: clean up post-rebase runtime drift
* fix: restore release ci checks
* fix: restore release ci after rebase
* fix: remove stale pi runtime path
* test: align compaction runtime expectations
* test: update plugin prerelease expectations
* fix: handle claude live tool approvals
* fix: stabilize release validation gates
* fix: finish agent runtime import
* test: finish post-rebase agent runtime mocks
* fix: keep codex compaction native
* fix: stabilize codex app-server hook tests
* test: isolate codex diagnostic active run
* test: remove codex diagnostic completion race
# Conflicts:
# extensions/codex/src/app-server/run-attempt.test.ts
* ci: fix full release manifest performance run id
* refactor: narrow llm plugin sdk boundary
* chore: drop generated google boundary stamps
* fix: repair rebase fallout
* fix: clean up rebased runtime references
* fix: decode codex jwt payloads as base64url
* fix: preserve shipped pi runtime alias
* fix: add scoped sdk virtual modules
* fix: decode llm codex oauth jwt as base64url
* fix: avoid stale vertex adc negative cache
* fix: harden tool arg decoding and codeql path
* fix: keep vertex adc negative checks live
* refactor: consolidate codex jwt and edit helpers
* fix: await codex oauth node runtime imports
* fix: preserve sdk tool and notice contracts
* fix: preserve shipped compat config boundaries
* fix: align codex oauth callback host
* fix: terminate agent-core loop streams on failure
* fix: keep codex oauth callback alive during fallback
* ci: include session tools in critical codeql scans
* fix: keep Cloudflare Anthropic provider auth header
* docs: redirect legacy pi runtime pages
* fix: honor bundled web provider compat discovery
* fix: protect session output spill files
* fix: keep legacy agent dir env blocked
* fix: contain auto-discovered skill symlinks
* fix: harden agent core sdk proxy surfaces
* fix: restore approval reaction sdk compat
* fix: keep live docker runs bounded
* fix: keep codex oauth redirect host aligned
* fix: resolve post-rebase agent runtime drift
* fix: redact anthropic oauth parse failures
* fix: preserve responses strict tool shaping
* fix: repair agent runtime rebase cleanup
* docs: redirect retired parity pages
* fix: bound auto-discovered resources to roots
* fix: repair post-rebase agent test drift
* fix: preserve bundled provider allowlist migration
* fix: preserve manifest-owned provider aliases
* fix: declare photon image dependency
* fix: keep provider headers out of proxy body
* fix: preserve shipped env aliases
* fix: refresh control ui i18n generated state
* fix: quote read fallback paths
* fix: preview edits through configured backend
* test: satisfy core test typecheck
* fix: preserve ZAI usage auth fallback
* test: repair codex diagnostic test
* fix: repair agent runtime rebase drift
* test: finish embedded runner import rename
* fix: repair agent runtime rebase integrations
* test: align compaction oauth fallback expectations
* fix: allow sdk-auth session models
* fix: update doctor tool schema import
* fix: preserve bedrock plugin region
* fix: stream harmony-like prose immediately
* ci: include session runtime in codeql shards
* fix: repair latest rebase integrations
* fix: honor explicit codex websocket transport
* fix: keep openai-compatible credentials provider-scoped
* fix: refresh sdk api baseline after rebase
* fix: route cli runtime aliases through openclaw harness
* test: rename stale harness mock expectation
* test: rename embedded agent overflow calls
* test: clean embedded auth test wording
* test: use openclaw stream types in deepinfra cache test
* fix: refresh sdk api baseline on latest main
* fix: honor bundled discovery compat allowlists
* fix: refresh sdk api baseline after latest rebase
* fix: remove stale rebase imports
* test: rename stale model catalog mock
* test: mock renamed doctor runtime modules
* fix: map canonical kimi env auth
* fix: use internal model registry in bench script
* fix: migrate deepinfra provider catalog entry
* fix: enforce builtin tool suppression
* fix: route compaction auth and proxy payloads safely
* refactor: prune unused llm registry leftovers
* test: update codex hooks session import
* test: fix model picker ci coverage
* test: align model picker auth mock types
2026-05-27 19:24:04 +01:00
Peter Steinberger
a1fe86a0ff
feat(qa): add coverage scenario matching
2026-05-25 10:22:51 +01:00
Vincent Koc
7f05be041e
fix(diagnostics): harden observability exports and smokes ( #85371 )
...
* test(diagnostics): widen observability smokes
* fix(diagnostics): sanitize observability exports
* docs(diagnostics): format otel export docs
2026-05-23 15:27:43 +08:00
Peter Steinberger
0def3e20e4
test(release): align prerelease validation
2026-05-22 14:43:36 +01:00
Firas Alswihry
229323d37a
test(qa-lab): add personal failure recovery scenario
2026-05-21 23:22:35 +08:00
Vincent Koc
cf0657852f
feat(qa-lab): add jsonl replay harness
2026-05-21 23:03:51 +08:00
Firas Alswihry
a9eaf0c993
test(qa-lab): add personal no-fake-progress scenario ( #83824 )
...
Summary:
- The PR adds a personal-agent QA-Lab no-fake-progress scenario, registers it in the personal-agent pack, teaches mock-openai the scripted path, and updates focused tests, docs, and changelog.
- Reproducibility: not applicable. This PR adds QA coverage rather than reporting a current-main bug; the branch supplies concrete after-patch QA-Lab/mock-openai commands and copied pass output.
Automerge notes:
- PR branch already contained follow-up commit before automerge: test(qa-lab): add personal no-fake-progress scenario
Validation:
- ClawSweeper review passed for head 95d2e4628895d2e46288https://github.com/openclaw/openclaw/pull/83824#issuecomment-4483439200
Co-authored-by: Firas Alswihry <itzfiras@gmail.com >
Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>
Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>
Approved-by: takhoffman
Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com >
2026-05-19 01:16:00 +00:00
Peter Steinberger
59defa3e71
ci(release): fix beta validation gates
2026-05-19 01:05:52 +01:00
Firas Alswihry
94c012b2ec
test(qa-lab): add personal task followthrough scenario
2026-05-18 14:35:03 +08:00
