Peter Steinberger
7ea4129227
fix(qa): authenticate summaries and bound native scenario commands ( #116748 )
...
Co-authored-by: Peter Steinberger <steipete@macos.shared >
2026-07-31 03:12:48 -07:00
Peter Steinberger
d16e33e08e
fix: restore trustworthy end-to-end QA and cross-channel delivery ( #115404 )
...
* fix(qa): repair verified end-to-end and channel regressions
* fix(gateway): make interrupted restart recovery lifecycle-safe
* test(heartbeat): target the canonical recovery session store
* fix(gateway): prioritize durable restart recovery before heartbeat
* fix(qa): preserve safe restart recovery and channel expiry
* fix(qa): fail closed and fence restart recovery
* test(agents): isolate restart recovery timing
* test(agents): prove actual restart retry timing
* fix(qa): report incompatible profile scenarios
* fix(scripts): resolve symlinked docker scheduler entrypoints
* fix(qa): require fresh native test evidence
* fix(heartbeat): fence active restart recovery delivery
* fix(gateway): consume untargeted restart acknowledgements
* fix(qa): satisfy exhaustive hosted validation gates
* fix(agents): fence stopped restart recovery dispatch
* style(agents): format restart recovery lifecycle regression
* test(gateway): isolate context prewarm sidecar lifecycle
* test(qa): make scenario process timeout cleanup deterministic
* fix(qa): stamp synthetic gateway configs with current version
* fix(openai): preserve vision capabilities in stale model catalogs
* test(qa): align profile channel rejection with current main
* fix(openai): forward supported moderation for image edits
* fix: restore latest-main CI and image edit documentation
* fix(qa): retain relocated code-mode evidence validation
* fix(openai): expose GPT-5.4 vision in static catalog
* fix(pricing): honor explicit model cost overrides
* test(pricing): keep isolated provider regressions deterministic
* fix(openai): inherit transport for discovered static models
* fix(gateway): honor agent-owned static image capabilities
* test(gateway): preserve prepared-snapshot attachment races
* test(gateway): isolate subagent persistence failure injection
* test(gateway): exercise concurrent voice replay admission
* fix(gateway): restore stale model image capabilities
* fix(agents): publish configured model vision capabilities
* fix(agents): isolate detached media transcript ownership
* test(agents): preserve generic transcript lock regression
* fix(gateway): require proven static model route identity
* fix(qa): accept bounded full-size generated image attachments
* fix(qa): require fresh script producer evidence
* test(qa): prove native E2E scenario execution
2026-07-29 12:45:27 -04:00
Peter Steinberger
6090386dc7
fix: make Codex harness caching and benchmark parity exact ( #115663 )
2026-07-29 03:33:36 -04:00
Dallin Romney
781744e125
fix(qa): run profile scenarios on every eligible channel ( #115540 )
...
* fix(qa): resolve channels for portable live scenarios
* fix(qa): expand profile scenarios across eligible channels
2026-07-29 13:14:37 +08:00
Peter Steinberger
641c82b5bd
fix(qa): restore full-stack scenarios and truthful suite exits ( #115290 )
...
* fix(qa): restore full-stack scenarios and truthful suite exits
* test(qa): respect plugin boundaries and atomic media delivery
2026-07-28 13:00:22 -04:00
Dallin Romney
9feb6ad161
refactor(qa): make scenario selection taxonomy-owned ( #114055 )
...
* refactor(qa): remove scenario execution profiles
* fix(qa): preserve smoke CI membership
* test(qa): drop redundant smoke pack order check
* refactor(qa): remove Matrix runner profiles
* refactor(qa): simplify Matrix CI orchestration
* refactor(qa): derive execution sets from taxonomy
* fix(qa): keep Matrix shard helper private
* refactor(qa): derive profile scenarios from taxonomy
* refactor(qa): reconcile taxonomy selection
* refactor(qa): use canonical profile planning
* refactor(qa): select canonical coverage owners
* fix(qa): keep profile selector internal
* refactor(qa): remove representative coverage owners
2026-07-28 22:23:41 +08:00
Vincent Koc
90564722ec
fix(qa-lab): retry failed suite partitions ( #115046 )
2026-07-28 16:13:22 +08:00
Vincent Koc
d2d6108d2c
fix(qa-lab): filter unsupported profile channels ( #115026 )
2026-07-28 15:25:36 +08:00
Dallin Romney
9310586b09
fix(qa-lab): unify runner control plane ( #112838 )
2026-07-27 14:05:14 +08:00
Dallin Romney
c1c231ced9
fix(qa): reject runtime tool intent without result evidence ( #113502 )
...
* fix(qa): require runtime tool result evidence
* fix(qa): preserve meaningful tool coverage gates
* refactor(qa): colocate runtime parity projection
* fix(qa): keep passing tool totals evidence-backed
2026-07-27 11:00:59 +08:00
Peter Steinberger
8684e7ce03
fix: restore beta runtime release gates ( #114064 )
...
* test(qa): fix release runtime recovery gates
* test(qa): typecheck restart execution assertions
2026-07-26 03:58:41 -04:00
Dallin Romney
19d17bc3c4
refactor(qa): canonicalize runtime-pair lanes ( #112603 )
2026-07-23 10:06:12 +09:00
Dallin Romney
132d91e427
refactor(qa): make taxonomy coverage ids unique ( #110280 )
2026-07-22 16:28:44 +09:00
pick-cat
fc52627ac1
fix(qa-lab): reject hex/exponent Telegram SUT uid env values ( #107626 )
...
* fix(qa-lab): reject hex/exponent Telegram SUT uid env values
* test(qa-lab): add direct unit tests for parseSutId strict int parsing
Export parseSutId and add 15 direct unit tests covering hex/exponent/fraction/
empty/whitespace/zero/negative rejection and valid decimal acceptance across
UID, GID, and CLEANUP_TIMEOUT_MS keys. The 3 existing integration tests
already prove the full CLI chain rejects malformed UIDs before gateway startup.
Mutation check: reverting to permissive Number() parsing causes 6 tests to
fail (4 unit + 2 integration).
Co-Authored-By: nebulacoder-v8.0 <noreply@zte.com.cn >
* test(qa-lab): cover GID and cleanup-timeout CLI reject-before-gateway
* fix(qa-lab): drop stale live-scenario mocks from SUT uid tests
* style(qa-lab): oxfmt import order in telegram cli.runtime
* test(qa-lab): streamline strict SUT integer coverage
---------
Co-authored-by: nebulacoder-v8.0 <noreply@zte.com.cn >
Co-authored-by: Peter Steinberger <steipete@gmail.com >
2026-07-21 09:42:22 -07:00
Peter Steinberger
4fbdb6d011
test(qa): align coverage inventory contracts ( #110799 )
2026-07-18 17:22:50 +01:00
Dallin Romney
82fa956d5a
fix(qa): resolve profile channels through selected driver ( #109350 )
...
* fix(qa): keep generic profiles off channel-specific lanes
* fix(qa): resolve profile channels through selected driver
2026-07-17 16:42:18 -07:00
Dallin Romney
b5eb8977dc
test(qa): prove Codex cold install product path ( #109267 )
2026-07-16 13:10:41 -07:00
Dallin Romney
9a2f25c82b
refactor(qa): make channel drivers run-level ( #108602 )
2026-07-16 00:59:03 -07:00
Dallin Romney
2d9b78fff7
refactor(qa): canonicalize live scenario coverage ( #108464 )
2026-07-15 16:50:41 -07:00
Dallin Romney
af71400c8e
refactor(qa): route live channel selectors through suite host ( #108429 )
...
* refactor(qa): route live channel selectors through suite host
* fix(qa): repair live selector CI
* fix(qa): remove retired runner exports
* fix(qa): remove obsolete live evidence
2026-07-15 13:35:34 -07:00
Dallin Romney
77d07dc3e9
refactor(qa): migrate Telegram scenarios into QA Lab ( #108430 )
...
* refactor(qa): migrate Telegram scenarios into QA Lab
* refactor(qa): remove retired Telegram runner exports
2026-07-15 12:55:00 -07:00
Dallin Romney
ad34552473
refactor(qa): migrate Matrix scenarios into QA Lab ( #103589 )
...
* refactor(qa): migrate Matrix scenarios into QA Lab
* fix(qa): build Matrix boundary declarations
* fix(qa): preserve Matrix preview boundaries
* fix(qa): preserve Matrix hot reload semantics
* fix(qa): harden Matrix destructive scenario failures
* fix(qa): harden Matrix scenario isolation
* fix(qa): close Matrix negative scenario blind spots
* fix(qa): isolate Matrix substrate state
* fix(qa): harden Matrix transport substrate
* fix(qa): preserve Matrix profile and event parity
* fix(qa): preserve explicit scenario models
* fix(qa): align Matrix scenario coverage taxonomy
* fix(qa): format Matrix allowlist cleanup
* fix(qa): satisfy migrated Matrix CI contracts
* fix(qa): reconcile Matrix migration with current main
* fix(qa): break scenario flow import cycle
* fix(qa): reconcile Matrix max-lines ownership
* fix(qa): address Matrix review boundaries
* fix(qa): remove stale Matrix lint suppression
* fix(qa): adopt split Matrix E2EE flows
* fix(qa): export Matrix scenario record guard
* fix(qa): align Matrix migration with privatized helpers
* refactor(qa): finish Matrix QA Lab ownership
* fix(qa): preserve Matrix suite defaults
* fix(qa): reconcile Matrix cleanup with current main
* test(qa): follow canonical Matrix profile size
* fix(qa): guard stale Matrix QA package output
* docs(qa): redirect retired Matrix QA pages
* refactor(qa): finish Matrix runner rename
* test(qa): assert Matrix defaults through profile resolver
* docs: refresh QA cleanup map
* fix(qa): privatize Matrix storage discovery
2026-07-15 01:22:20 -07:00
Peter Steinberger
f81f9d8570
chore: enforce max-lines suppression ratchet ( #107315 )
...
* ci: enforce max-lines suppression ratchet
* chore: prune stale max-lines suppression
* fix: close max-lines ratchet enforcement gaps
* fix: harden max-lines ratchet checks
* fix(ci): satisfy max-lines ratchet checks
* style: format max-lines declarations
* fix(ci): match oxlint suppression grammar
* test: isolate max-lines git fixtures
* chore: prune resolved max-lines debt
* test: skip newline path fixture on Windows
* fix: harden max-lines suppression ratchet
* chore: refresh max-lines baseline
* fix: close max-lines ratchet bypasses
* fix: derive ratchet base from PR merge tree
* fix: support older Git in staged ratchet
* fix: align max-lines declarations and baseline
* chore: refresh max-lines baseline for current main
* fix: exclude generated wizard locales from max-lines
* chore: prune resolved max-lines debt
2026-07-14 09:27:02 -07:00
Peter Steinberger
7f73f6e66e
test(qa-lab): refresh CLI dispatch expectations ( #107018 )
2026-07-13 19:36:17 -07:00
Vincent Koc
4d15c47e9f
fix(qa): keep runtime parity lanes reproducible
2026-07-12 15:10:38 +08:00
Vincent Koc
4040267af9
fix(qa): run mixed channel suites
2026-07-12 06:51:59 +02:00
Peter Steinberger
fa77fe10d5
chore: migrate active GPT-5.5 references to GPT-5.6 ( #104452 )
...
* chore(models): migrate active GPT-5.5 references
* test(workboard): expect GPT-5.6 Sol default
* chore: keep release notes in PR body
* test(models): align picker fixtures with Sol default
* test: update PDF default model expectation
* test(qa): migrate thinking smoke to Luna
* test(gateway): align mock catalog with Sol default
* ci: retrigger exact-head PR checks
* test(gateway): document default catalog invariant
2026-07-11 06:30:57 -07:00
Peter Steinberger
afa3212e90
fix(release): stage Telegram QA inside runtime root ( #104125 )
2026-07-10 22:39:09 -07:00
Vincent Koc
2a1d6e49d5
fix(ci): run Telegram release QA from trusted harness ( #103207 )
...
* fix(ci): use trusted Telegram QA harness
* fix(ci): restrict trusted Telegram QA candidate
* fix(ci): isolate trusted Telegram QA candidate
* fix(ci): harden Telegram QA process boundary
* fix(ci): pass Telegram QA release gates
* test(qa): stub Telegram env explicitly
* fix(ci): harden Telegram release QA boundary
* test(ci): harden trusted workflow and memory guards
2026-07-10 11:32:24 -07:00
Peter Steinberger
4bc300843d
fix(qa): keep runtime parity tiers flow-compatible ( #103609 )
...
* fix(qa): keep runtime parity tiers flow-compatible
Refs #103588
* test(qa): follow canonical frontier defaults
2026-07-10 10:51:13 +01:00
Vincent Koc
238be4b849
test(release): refresh beta3 gate expectations
2026-07-08 15:55:50 -07:00
Dallin Romney
c1b9e36221
test(qa): migrate channel thread and DM isolation scenarios ( #101112 )
...
* test(qa): migrate channel thread isolation scenarios
* test(qa): remove stale Matrix scenario fixture
* test(qa): use canonical coverage taxonomy ids
* test(qa): preserve Matrix flow runtime identities
* test(qa): expose Slack thread polling helper
* test(qa): select Matrix for native thread smoke
* test(qa): keep Matrix subagent spawn opt-in
* test(qa): target selected live account in scenario patches
2026-07-06 17:52:38 -07:00
Peter Steinberger
aed2482292
fix(qa): restore automatic smoke coverage across channels ( #101186 )
...
* fix(qa): restore channel-sharded smoke coverage
Co-authored-by: Peter Steinberger <steipete@gmail.com >
Co-authored-by: Dallin Romney <dallinromney@gmail.com >
* fix(qa): restore channel-sharded smoke coverage
* fix(qa): restore channel-sharded smoke coverage
* fix(qa): restore channel-sharded smoke coverage
---------
Co-authored-by: Vincent Koc <25068+vincentkoc@users.noreply.github.com >
Co-authored-by: Dallin Romney <dallinromney@gmail.com >
Co-authored-by: Vincent Koc <vincentkoc@ieee.org >
2026-07-06 17:09:40 -07:00
Dallin Romney
19491fe34c
Revert "fix(qa): keep smoke profile on one channel ( #101173 )" ( #101184 )
...
This reverts commit b773f7981a .
2026-07-06 13:29:07 -07:00
Dallin Romney
b773f7981a
fix(qa): keep smoke profile on one channel ( #101173 )
2026-07-06 13:26:58 -07:00
Dallin Romney
e3671e0980
refactor(qa): canonicalize channel command scenarios ( #101080 )
...
* test(qa): migrate channel command scenarios
* test(qa): distinguish portable command headings
* test(qa): preserve canonical scenario ownership
2026-07-06 12:18:06 -07:00
Dallin Romney
b29d472e41
refactor(qa): add canonical live channel adapters ( #99707 )
2026-07-06 09:24:34 -07:00
Peter Steinberger
c757675f34
improve: keep isolated tests under one second ( #100019 )
...
* test: speed up isolated test suite
* test: finish isolated latency cleanup
* test: eliminate remaining isolated latency spikes
* test: remove final isolated timing outliers
* test: bound full-suite tooling processes
* test: bound native test process lifetime
* test: warm isolated runtime suites
* test: eliminate final isolated timing outliers
* test: fix isolated timing fixture types
* test: make timeout cleanup timing deterministic
* test: pin media manifests to source checkout
* test: isolate provider manifest contracts
* test: eliminate residual isolated timing spikes
* test: restore final isolated timing fixes
* test: eliminate remaining isolated timing spikes
* test: warm Zalo lifecycle imports
* test: keep isolated suites below one second
* test: use readable browser response fixtures
2026-07-05 10:57:19 -07:00
Dallin Romney
15bfadf2bd
fix: count maturity coverage ids ( #96543 )
2026-06-24 14:56:51 -07:00
Dallin Romney
f29dbd3ebd
test(qa): speed up smoke profile ( #96340 )
2026-06-24 09:30:59 -07:00
Dallin Romney
9666db607e
test(qa): clean up smoke taxonomy profile ( #96320 )
2026-06-24 00:43:00 -07:00
Dallin Romney
4460fa78c3
feat(qa): add "all" taxonomy profile ( #95947 )
...
* qa: add all maturity profile
* test: update qa coverage profile expectations
2026-06-22 20:49:08 -07:00
Vincent Koc
def4b51485
fix(qa): gate smoke profile scenarios by channel driver
2026-06-23 09:34:52 +08:00
Dallin Romney
de9c94cbbb
feat(qa): forward shared suite flags to multipass runner ( #91506 )
2026-06-22 15:48:05 -07:00
Dallin Romney
63b13ea837
feat(qa): crabline channel driver ( #91502 )
...
* feat(qa): add crabline channel driver seam
* feat: run crabline channel driver smoke
* chore: keep crabline qa dependency dev-only
* refactor(qa): keep crabline driver details opaque
* chore(qa): pin crabline to merged driver API
* feat(qa): drive channel driver from profiles
* fix(qa): declare crabline runtime peer
* feat(qa): resolve crabline channel from scenarios
* feat(qa): treat unsupported profile channels as coverage gaps
* Revert "feat(qa): treat unsupported profile channels as coverage gaps"
This reverts commit 65a97016558705514a1c1ec74870de9b0a6e7886.
* fix(qa): adapt crabline driver to chat sdk cli
* refactor(qa): pass channel driver metadata directly
* chore(qa): update crabline provider pin
* chore(qa): default channel scenarios to driver
* chore: repair qa dependency lockfile
* chore: allow native qa dependency builds
* fix(qa): satisfy crabline driver lint
* fix(qa): satisfy crabline ci gates
* Use crabline transport for smoke QA profile
* fix(qa): keep crabline driver opt-in
* fix(qa): reuse crabline telegram driver token
* fix(qa): route smoke profile through crabline
* fix(qa): run full smoke profile lane
* fix(qa): remove smoke scenario workflow filter
* fix: stabilize crabline smoke qa profile
* fix: pin crabline qa dependency
* test: keep crabline smoke credential-free
* fix: skip visible reasoning lane for crabline smoke
* fix: unblock crabline qa ci
* Update crabline dependency
* Pin crabline to merged main
* Use Crabline fake provider servers
2026-06-22 15:24:59 -07:00
Vincent Koc
abdd81db11
test(qa): include long-context soak scenario
2026-06-22 14:28:04 +08:00
Dallin Romney
e17d111990
fix: require all taxonomy coverage ids ( #94296 )
2026-06-17 16:38:14 -07:00
Vincent Koc
e2b6753b87
fix(qa-lab): bound credential payload reads
2026-06-17 11:59:55 +02:00
Vincent Koc
c32ba171db
fix(qa): fail unsuccessful self-checks
2026-06-16 01:32:47 +02:00
Dallin Romney
e32929e12c
Add slim evidence mode for QA profile evidence ( #93179 )
...
* test(qa): compact profile evidence execution metadata
* docs(qa): document compact profile evidence
* test(qa): support compact evidence mode
* test(qa): rename compact evidence mode to slim
* docs(qa): trim slim evidence wording
* fix(qa): avoid commander runtime import
2026-06-15 14:50:40 -07:00