Commit Graph

955 Commits

Author SHA1 Message Date
Vincent Koc
d1b268f7f7 fix(qa): normalize completed wait envelopes 2026-06-23 05:13:35 +02:00
Vincent Koc
2205f50016 test(qa): satisfy history reply lint 2026-06-23 04:01:11 +02:00
Vincent Koc
d716dfd532 test(qa): wait for live history replies in flow scenarios 2026-06-23 04:01:11 +02:00
Vincent Koc
5822e8074d test(qa): accept completed agent wait status 2026-06-23 04:01:11 +02:00
Dallin Romney
27711b500c ci: add maturity scorecard renderer (#94272) (#95901)
* ci: add maturity scorecard renderer

* ci: render qa scorecard evidence

* ci: type maturity docs renderer

* ci: tighten maturity artifact inputs

* ci: move maturity renderer under qa scripts

* ci: share maturity score schema

* ci: centralize maturity taxonomy validation

* ci: move maturity scores under qa

* ci: remove docs maturity score source

* docs: simplify maturity scorecard output

* docs: commit generated maturity scorecard

* docs: group maturity pages

* docs: simplify maturity scorecard dates

* docs: promote maturity nav tab

* docs: clean up maturity pages

* docs: remove maturity outline page

* docs: filter maturity taxonomy doc links

* docs: simplify maturity taxonomy tables

* docs: keep artifact taxonomy links

* docs: simplify lts scorecard display

* docs: clarify maturity score definitions

* docs: derive maturity coverage from evidence

* docs: hide maturity scorecard until evidence

* docs: remove placeholder maturity pages

* docs: keep maturity scores out of pr

* ci: open maturity scorecard docs pr
2026-06-22 18:55:06 -07:00
Vincent Koc
def4b51485 fix(qa): gate smoke profile scenarios by channel driver 2026-06-23 09:34:52 +08:00
Vincent Koc
43f2b61f3b test(qa): keep image generation fixture on mock lane 2026-06-23 02:35:02 +02:00
Vincent Koc
086c629556 test(qa): scope provider-sensitive flow fixtures 2026-06-23 02:17:20 +02:00
Vincent Koc
befe04f465 test(qa): accept Sonnet max thinking support 2026-06-23 01:57:43 +02:00
Vincent Koc
264b37e9d2 test(qa): avoid redacted config cleanup patch 2026-06-23 01:39:39 +02:00
Dallin Romney
de9c94cbbb feat(qa): forward shared suite flags to multipass runner (#91506) 2026-06-22 15:48:05 -07:00
Dallin Romney
63b13ea837 feat(qa): crabline channel driver (#91502)
* feat(qa): add crabline channel driver seam

* feat: run crabline channel driver smoke

* chore: keep crabline qa dependency dev-only

* refactor(qa): keep crabline driver details opaque

* chore(qa): pin crabline to merged driver API

* feat(qa): drive channel driver from profiles

* fix(qa): declare crabline runtime peer

* feat(qa): resolve crabline channel from scenarios

* feat(qa): treat unsupported profile channels as coverage gaps

* Revert "feat(qa): treat unsupported profile channels as coverage gaps"

This reverts commit 65a9701655.

* fix(qa): adapt crabline driver to chat sdk cli

* refactor(qa): pass channel driver metadata directly

* chore(qa): update crabline provider pin

* chore(qa): default channel scenarios to driver

* chore: repair qa dependency lockfile

* chore: allow native qa dependency builds

* fix(qa): satisfy crabline driver lint

* fix(qa): satisfy crabline ci gates

* Use crabline transport for smoke QA profile

* fix(qa): keep crabline driver opt-in

* fix(qa): reuse crabline telegram driver token

* fix(qa): route smoke profile through crabline

* fix(qa): run full smoke profile lane

* fix(qa): remove smoke scenario workflow filter

* fix: stabilize crabline smoke qa profile

* fix: pin crabline qa dependency

* test: keep crabline smoke credential-free

* fix: skip visible reasoning lane for crabline smoke

* fix: unblock crabline qa ci

* Update crabline dependency

* Pin crabline to merged main

* Use Crabline fake provider servers
2026-06-22 15:24:59 -07:00
Dallin Romney
ed2dfee7d7 feat(qa): expose active memory toggles to scenarios (#95858) 2026-06-22 14:26:37 -07:00
Vincent Koc
96c6f8022c fix(auto-reply): suppress quoted silent replies 2026-06-23 02:24:29 +08:00
Vincent Koc
a4eb49a176 refactor(qa): share gateway message text extraction 2026-06-22 20:25:26 +08:00
Vincent Koc
db21588636 refactor(qa): share suite summary file loading 2026-06-22 20:24:20 +08:00
Vincent Koc
d3f7f7d1fc chore(deadcode): remove unused test-only helpers 2026-06-22 15:48:43 +08:00
Vincent Koc
7cc21ef59d fix(qa): stabilize smoke-ci scenarios 2026-06-22 15:41:53 +08:00
Vincent Koc
abdd81db11 test(qa): include long-context soak scenario 2026-06-22 14:28:04 +08:00
Vincent Koc
4506d8bad6 fix(qa): surface tool-search bridge results 2026-06-22 08:01:44 +02:00
Vincent Koc
fd13192adc fix(qa-lab): version UI assets from repo root 2026-06-21 23:43:48 +02:00
Vincent Koc
1c63da09d8 fix(qa-lab): cap agent wait timeout payloads 2026-06-21 23:34:38 +02:00
Marcus Castro
b50a5aebba fix(whatsapp): preserve native quote replies (#95483)
* fix(whatsapp): preserve native quote replies

* feat(plugin-sdk): add quote-reply live transport standard

* test(qa-lab): add WhatsApp quote-reply live scenarios
2026-06-21 11:26:58 -03:00
Vincent Koc
8a7906c716 fix(qa): reject fractional live token usage 2026-06-21 14:49:29 +02:00
Vincent Koc
c85113e30e fix(qa): escape tool coverage markdown cells 2026-06-21 14:46:38 +02:00
Vincent Koc
d9dfcd6c8a fix(qa): reject impossible confidence counts 2026-06-21 14:38:42 +02:00
Vincent Koc
5e86c7eef4 fix(qa): reject non-object mock Anthropic JSON 2026-06-21 13:13:37 +02:00
Vincent Koc
6f3af56952 fix(qa): reject non-object mock OpenAI JSON 2026-06-21 13:10:50 +02:00
Vincent Koc
f69ba12a37 fix(qa): reject malformed mock OpenAI JSON 2026-06-21 12:58:20 +02:00
Vincent Koc
b5fc9514c0 fix(qa): reject coerced evidence artifact indexes 2026-06-21 12:50:37 +02:00
Vincent Koc
600bace853 fix(qa): validate QA bus poll numbers 2026-06-21 12:41:22 +02:00
Vincent Koc
b8a5dac1a2 fix(qa-lab): resolve Windows PowerShell path 2026-06-21 12:40:57 +02:00
Vincent Koc
66e5cfdd86 fix(qa): reject QA Lab host port collisions 2026-06-21 12:23:52 +02:00
Vincent Koc
a70b34a3cb fix(qa-lab): resolve Windows node lookup tool 2026-06-21 12:14:39 +02:00
Vincent Koc
735d70d9db test(extensions): pin Windows taskkill test roots 2026-06-21 11:37:59 +02:00
Vincent Koc
61d1fd1f72 fix(qa-lab): resolve Windows taskkill path 2026-06-21 11:26:30 +02:00
Dallin Romney
5dd30c3995 test: fold HTTP API script proof into QA Lab (#94700)
* test: fold HTTP API script proof into QA Lab

* test: remove folded HTTP API script tests

* test: relax QA native scenario catalog inventory

* test: trim folded QA Lab script cruft

* test: align folded QA coverage ids

* test: keep native QA evidence out of parity tiers

* test: update mirrored QA routing expectation

* test: preserve chat tools profile build guard

* test: avoid overclaiming gateway tool API coverage

* test: pin folded QA coverage ids
2026-06-20 23:10:35 -07:00
Vincent Koc
5b22409389 fix(qa-lab): taskkill gateway children on windows 2026-06-21 08:07:00 +02:00
Vincent Koc
b970d57175 fix(qa-lab): taskkill timed-out cli trees on windows 2026-06-21 08:01:55 +02:00
Vincent Koc
c43822077a fix(qa-lab): force taskkill scenario trees on windows 2026-06-21 07:56:03 +02:00
Vincent Koc
ed500dda25 fix(qa): use corepack for lab docker build fallback 2026-06-20 17:45:09 +02:00
Vincent Koc
7fc5a72433 fix(qa): cap chunked credential lease payloads 2026-06-20 17:34:38 +02:00
Vincent Koc
c5623e72f3 fix(qa): quote generated compose paths 2026-06-20 17:08:40 +02:00
Vincent Koc
947c21ee5a refactor(qa): reuse qa shell quote helper 2026-06-20 17:05:10 +02:00
Vincent Koc
99f58ae6d6 fix(qa): quote qa docker stop command 2026-06-20 16:59:14 +02:00
Vincent Koc
a6e41a0cc1 fix(qa-lab): kill script timeout trees on windows 2026-06-20 12:58:52 +02:00
Vincent Koc
1ede829fbf fix(qa-lab): leave vitest timeout cleanup to wrapper 2026-06-20 12:53:39 +02:00
Vincent Koc
b93b07ee1b test(qa-lab): use temp harness in scenario runner tests 2026-06-20 12:53:39 +02:00
Vincent Koc
405e5072fd fix(qa-lab): bound test file scenario commands 2026-06-20 12:53:39 +02:00
Vincent Koc
88cf142c98 fix(qa-lab): preserve model catalog abort grace 2026-06-20 11:18:43 +02:00