Josh Avant
d5b326523f
qa-lab: make live lanes CI-ready for v1 E2E automation ( #69122 )
...
* qa-lab: harden CI defaults and failure semantics for live lanes
* qa-lab: add unit tests for suite progress logging defaults
* qa-lab: cover malformed multipass summary edge cases
* qa-lab: share suite summary failure counting helper
* qa-lab: test allow-failures parse wiring and sanitize progress ids
* fix: note qa CI live-lane defaults in changelog (#69122 ) (thanks @joshavant)
2026-04-19 21:13:27 -05:00
Gustavo Madeira Santana
82fe6f50ef
QA: organize scenarios by theme
2026-04-17 11:03:47 -04:00
Gustavo Madeira Santana
bb7e9823a8
QA Lab: add provider registry
...
Move mock and live provider behavior behind provider-owned definitions so suite, manual, Matrix, and transport lanes share defaults, auth staging, model config, and standalone server startup.
Add AIMock as a first-class local provider mode while keeping mock-openai as the scenario-aware deterministic lane.
2026-04-16 21:24:59 -04:00
Gustavo Madeira Santana
56a9fd4b34
QA Matrix: capture full runner output
2026-04-16 16:18:54 -04:00
Peter Steinberger
20266c14cb
feat(qa-lab): add control ui qa-channel roundtrip scenario
2026-04-12 19:41:06 -07:00
Peter Steinberger
ca2d297c50
docs(qa): clarify markdown scenario contract
2026-04-12 11:59:50 -07:00
Marcus Castro
000fc7f233
refactor(qa): add shared QA channel contract and harden worker startup ( #64562 )
...
* refactor(qa): add shared transport contract and suite migration
* refactor(qa): harden worker gateway startup
* fix(qa): scope waits and sanitize shutdown artifacts
* fix(qa): confine artifacts and redact preserved logs
* fix(qa): block symlink escapes in artifact paths
* fix(gateway): clear shutdown race timers
* fix(qa): harden shutdown cleanup paths
* fix(qa): sanitize gateway logs in thrown errors
* fix(qa): harden suite startup and artifact paths
* fix(qa): stage bundled plugins from mutated config
* fix(qa): broaden gateway log bearer redaction
* fix(qa-channel): restore runtime export
* fix(qa): stop failed gateway startups as a process tree
* fix(qa-channel): load runtime hook from api surface
2026-04-12 15:02:57 -03:00
Gustavo Madeira Santana
25445a9f2e
qa-lab: add Matrix live transport QA lane ( #64489 )
...
Merged via squash.
Prepared head SHA: ae9bb377515599352+gumadeiras@users.noreply.github.com >
Co-authored-by: gumadeiras <5599352+gumadeiras@users.noreply.github.com >
Reviewed-by: @gumadeiras
2026-04-10 19:35:08 -04:00
Peter Steinberger
3027efaf21
test: raise QA suite default concurrency
2026-04-10 13:45:57 +01:00
Peter Steinberger
4c14f55c62
test: parallelize QA suite scenarios
2026-04-10 13:45:57 +01:00
Shakker
655cfb477a
docs: clarify multipass live auth support
2026-04-09 23:53:13 +01:00
Shakker
a04c331cc1
docs: document qa multipass runner
2026-04-09 23:53:13 +01:00
Peter Steinberger
be46d0ddc6
test: update character eval public panel
2026-04-09 01:25:59 +01:00
Peter Steinberger
39cc6b7dc7
fix: stabilize character eval and Qwen model routing
2026-04-09 01:04:09 +01:00
Peter Steinberger
21ef1bf8de
feat: parallelize character eval runs
2026-04-08 20:05:55 +01:00
Peter Steinberger
4a51a1031d
feat: add character eval model options
2026-04-08 17:05:30 +01:00
Peter Steinberger
3101d81053
feat: add QA character eval reports
2026-04-08 15:52:49 +01:00
Peter Steinberger
b73d8ef7d7
refactor: split qa scenarios into per-file markdown defs
2026-04-08 05:37:17 +01:00
Peter Steinberger
c0aed59fca
refactor: move qa suite definitions into markdown
2026-04-07 23:39:50 +01:00
Peter Steinberger
124cd5e307
feat: auto-reload qa lab fast refresh
2026-04-07 09:54:59 +01:00
Peter Steinberger
54a884865e
feat: add fast qa lab ui refresh mode
2026-04-07 09:45:11 +01:00
Peter Steinberger
b4e1747391
feat: add one-command qa lab docker launcher
2026-04-06 17:47:17 +01:00
Peter Steinberger
8e1c81e707
feat(qa): recreate qa lab docker stack
2026-04-05 23:21:56 +01:00
Peter Steinberger
17a324b0de
chore: polish qa lab follow-ups
2026-04-05 23:21:56 +01:00
Peter Steinberger
b58f9c5258
feat: add qa channel foundation
2026-04-05 23:21:56 +01:00
