Commit Graph

80 Commits

Author SHA1 Message Date
Peter Steinberger
1a946dba78 test(qa): cover repeated live restart recovery (#105126) 2026-07-12 09:03:54 +01:00
Vincent Koc
bdc9c6070d fix(qa): add stable mock request cursors 2026-07-12 08:18:34 +02:00
Peter Steinberger
d14bb8ec44 fix: resume replay-safe work after gateway restarts (#104997)
* fix: resume replay-safe work after gateway restart

* chore: leave restart note to release process

* fix: sync restart recovery protocol artifacts
2026-07-12 06:37:22 +01:00
Peter Steinberger
96f0983a85 fix(onboarding): skip setup for configured gateways and require inference first (#102883)
* fix(crestodian): keep onboarding RPCs restart-safe

* fix(profiles): isolate approval state migrations

* fix(crestodian): bypass configured gateway setup

* test(crestodian): type onboarding mocks

* fix(onboarding): require inference before Crestodian

* fix(onboarding): enforce verified inference handoff

* fix(macos): reset setup on gateway endpoint edits

* chore(i18n): refresh native source inventory

* fix(gateway): keep socket on request cancellation

* test(packaging): require workspace templates

* fix(onboarding): bind setup to verified inference

* fix(onboarding): align inference gate contracts

* fix(crestodian): classify concurrent policy rejection

* test(crestodian): expect registry restoration

* fix(onboarding): bind setup to configured gateways

* fix(codex): preserve startup phase deadlines

* test(crestodian): match fail-closed policy ordering

* test(onboarding): assert bound gateway handoff

* fix(codex): bind runtime resolution to spawn cwd

* test(crestodian): assert policy rejection order

* fix(cli): preserve gateway routing across restarts

* fix(macos): fail closed during gateway edits

* test(macos): cover gateway route generation races

* chore: keep release notes out of onboarding PR

* fix(ci): refresh onboarding generated checks

* style(swift): align gateway channel formatting

* fix(ci): refresh plugin SDK surface budgets

* fix(ci): resync native string inventory

* refactor(swift): split gateway channel support

* test(doctor): isolate plugin compatibility registry

* test(macos): isolate gateway onboarding fixtures

* test(macos): assert gateway lease health ordering

* fix(codex): reconcile computer-use startup changes
2026-07-11 10:25:14 -07:00
Peter Steinberger
dce871540b test(qa): consolidate web search scenario (#104280)
Co-authored-by: Jimmy Puckett <jimmy.puckett@spinen.com>
2026-07-11 00:54:15 -07:00
Peter Steinberger
b8502e1c0a test(gateway): prove suspension lifecycle in Docker (#104203) 2026-07-10 22:53:35 -07:00
Vincent Koc
155e196e22 fix(release): preserve beta validation evidence (#103796)
* fix(release): proxy upgrade fixtures to npm

(cherry picked from commit 30f54a5fe1)

* fix(release): require installer doctor evidence

(cherry picked from commit 0782a94452)

* test(qa): scope aborted restart outcome to Codex

(cherry picked from commit ac54f80430)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-10 09:05:14 -07:00
Peter Steinberger
9ef833f4ed test(qa): require web fetch after discovery (#103685) 2026-07-10 14:01:10 +01:00
Peter Steinberger
5a451f92f9 test(qa): accept structured gateway restart outcomes (#103681) 2026-07-10 13:57:59 +01:00
Peter Steinberger
7b03a2ae20 test(qa): model non-assistant parity usage (#103650) 2026-07-10 12:12:28 +01:00
Dallin Romney
95e3b9243b test: execute media and Talk runtime boundaries (#101091)
* test: execute media and Talk runtime boundaries

* test(qa): centralize gateway fixture startup

* test(qa): check in voice call plugin fixture
2026-07-06 16:25:33 -07:00
Dallin Romney
66b4dcf184 chore(docker): execute Compose and package artifact proofs (#101045)
* test(docker): prove Compose and package artifacts

* test(docker): use bounded container cleanup

* test(docker): bound package proof launch

* test(qa): keep Docker artifact proofs release-only

* test(docker): simplify artifact proof names

* test(docker): satisfy identity writer lint
2026-07-06 13:05:34 -07:00
Dallin Romney
2f7299eaa5 test(qa): add provider runtime evidence (#101051) 2026-07-06 13:01:32 -07:00
Vincent Koc
3c530c79a0 test(qa): accept completed wait envelopes 2026-07-06 05:22:57 -07:00
Dallin Romney
9d68f877ac test(qa): run gateway and MCP scenarios over real transports (#99735) 2026-07-03 20:26:16 -07:00
Dallin Romney
3b4092dbaa test: add executable runtime fixture canaries (#99737)
* test(qa): add runtime fixture canaries

* test(voice-call): track fixture temp directories
2026-07-03 20:22:40 -07:00
Dallin Romney
ab9e233d02 improve(qa): execute runtime scenarios through Docker (#99705)
* test(qa): run Docker runtime scenarios end to end

* test(qa): keep plugin lifecycle probe runtime-only
2026-07-03 18:13:28 -07:00
Dallin Romney
677c8ff846 improve: enforce canonical QA scenario ownership (#99628)
* improve: enforce canonical QA scenario ownership

* docs: refresh QA documentation map

* docs: condense QA ownership guidance

* docs: remove QA migration guidance

* refactor: simplify QA scenario ownership checks

* docs: align QA coverage authoring guidance
2026-07-03 15:51:18 -07:00
Dallin Romney
fd3f354f46 test(qa): expand docker lane adapter (#97937)
* test(qa): expand docker lane adapter

* Extract QA Docker lane fixture
2026-06-29 17:08:43 -07:00
Dallin Romney
96c8338d5b test: fold remaining Docker lanes into QA (#97708) 2026-06-29 00:39:06 -07:00
Dallin Romney
5327340a09 test: fold release upgrade docker lanes into qa (#97689) 2026-06-28 23:59:36 -07:00
Dallin Romney
a202dd0faf test: link mcp gateway transport coverage (#97609) 2026-06-28 19:20:15 -07:00
Dallin Romney
4b8a0a8ecf Migrate Tool Search gateway E2E into QA Lab flow (#97478)
* test: migrate tool search gateway to QA flow

* test: use built SDK in tool search QA fixture

* test: fix tool search QA fixture lint

* test: gate tool search QA flow to mock provider

* Share QA Lab fixture utilities

* Tighten QA fixture helper defaults

* test: gate tool search QA flow mode
2026-06-28 18:16:15 -07:00
Dallin Romney
39c4c02653 test: link Tool Search gateway QA coverage (#97374) 2026-06-28 04:28:32 -07:00
Dallin Romney
119dc4bd82 test: promote OpenAI HTTP QA coverage (#97369) 2026-06-27 22:28:31 -07:00
Dallin Romney
78f7de01c1 test: link local e2e qa coverage wrappers (#97150) 2026-06-27 18:34:36 -07:00
Vincent Koc
dad7168c2f fix(qa): align runtime parity evidence with Codex 2026-06-25 10:20:51 +08:00
Vincent Koc
e9720c27fa fix(qa): accept Codex capped read evidence (#96366) 2026-06-24 18:07:13 +08:00
Vincent Koc
8242923fe3 fix(qa): allow async runtime fixture starts 2026-06-24 17:52:16 +08:00
Dallin Romney
9666db607e test(qa): clean up smoke taxonomy profile (#96320) 2026-06-24 00:43:00 -07:00
Dallin Romney
f6b2a5ffb4 test(qa): harden all-profile evidence scenarios (#96003) 2026-06-23 00:07:51 -07:00
Vincent Koc
54b2243de3 test(qa): tighten release profile scenario waits 2026-06-23 06:04:16 +02:00
Vincent Koc
43f2b61f3b test(qa): keep image generation fixture on mock lane 2026-06-23 02:35:02 +02:00
Vincent Koc
264b37e9d2 test(qa): avoid redacted config cleanup patch 2026-06-23 01:39:39 +02:00
Vincent Koc
7cc21ef59d fix(qa): stabilize smoke-ci scenarios 2026-06-22 15:41:53 +08:00
Vincent Koc
7d4d8a7f3d test(agents): add large prompt cache coverage (#95653)
Adds large OpenAI and Anthropic prompt-cache live coverage plus a QA Lab long-context tool-result scenario.

Co-authored-by: vincentkoc <vincentkoc@users.noreply.github.com>
Reviewed-by: @vincentkoc
2026-06-22 11:39:37 +08:00
Dallin Romney
5dd30c3995 test: fold HTTP API script proof into QA Lab (#94700)
* test: fold HTTP API script proof into QA Lab

* test: remove folded HTTP API script tests

* test: relax QA native scenario catalog inventory

* test: trim folded QA Lab script cruft

* test: align folded QA coverage ids

* test: keep native QA evidence out of parity tiers

* test: update mirrored QA routing expectation

* test: preserve chat tools profile build guard

* test: avoid overclaiming gateway tool API coverage

* test: pin folded QA coverage ids
2026-06-20 23:10:35 -07:00
Jesse Merhi
5db2f6c1fc Add stdout diagnostics OTEL log exporter
Adds stdout and both-mode diagnostics OTEL log export, with focused QA Lab smoke coverage and docs/config updates.

Prepared head SHA: efa2ef07ab
Verification: CI 27808480969 passed for the prepared head.
Reviewed-by: @jesse-merhi
2026-06-19 16:06:37 +10:00
Dallin Romney
e12cf72b17 Standardize QA coverage IDs on dotted names (#94702)
* fix: standardize qa coverage ids

* test: avoid qa coverage id assertion spread
2026-06-18 17:25:26 -07:00
Dallin Romney
e17d111990 fix: require all taxonomy coverage ids (#94296) 2026-06-17 16:38:14 -07:00
Vincent Koc
8288b4d4c9 fix(qa-lab): stabilize web search parity fixture 2026-06-18 00:07:36 +02:00
Dallin Romney
f7e5132ffd test: fold gateway smoke into qa e2e (#93178) 2026-06-17 14:55:28 -07:00
Dallin Romney
fae4a01d0d test: fold otel smoke into qa e2e (#93181)
* test: fold otel smoke into qa e2e

* test: eliminate otel smoke script
2026-06-17 14:54:58 -07:00
Dallin Romney
0a6736af09 test: fold lifecycle and package proof into QA Lab (#93114)
* test: fold script coverage into qa scenarios

* test: migrate script checks into qa e2e

* test: point qa code refs at migrated e2e

* test: fold plugin lifecycle probe into qa e2e

* test: use shared temp dirs in plugin lifecycle probe

* test: fold plugin lifecycle sweep into qa lab

* test: trim lifecycle docker text assertions

* test: keep followup script conversions split

* test: make lifecycle docker runner script-safe

* test: update changed helper routing expectation
2026-06-17 14:22:04 -07:00
Dallin Romney
450060d7a2 test(qa): expand smoke-ci and release categories and coverage (#93175)
* test(qa): add smoke ci primary coverage evidence

* test(qa): remove overstated primary coverage claims

* test(qa): make release profile include smoke ci

* test(qa): trim taxonomy formatting churn

* test(qa): avoid hardcoded profile names in coverage test

* test(qa): make release profile cover taxonomy

* test(qa): type profile fixture all category flag

* test(qa): include channel delivery in smoke ci profile
2026-06-15 18:05:52 -07:00
Dallin Romney
fef8394079 Convert QA scenarios to YAML files (#92915)
* refactor: load QA scenarios from YAML

* docs: update personal QA scenario docs

* test: keep QA scenarios YAML-only
2026-06-14 17:31:18 -07:00
NVIDIAN
ecaebfc51b fix(agents): retry thinking-only errored turns (#92191)
Retry replay-safe reasoning-only provider errors before assistant failover while preserving classified fallback and terminal-output ownership. Adds deterministic Anthropic gateway fault-injection coverage and focused regression tests.\n\nCo-authored-by: ai-hpc <mail.speedy.hpc@hotmail.com>
2026-06-14 09:39:27 -07:00
Vincent Koc
50130d32a9 test(release): align qa tool coverage gate 2026-06-09 01:02:24 +02:00
Vincent Koc
c7b01cf201 test(release): stabilize qa runtime parity gate 2026-06-09 01:02:24 +02:00
Bek
bce3d5bf92 trace: Correlate channel diagnostics into one trace
Correlates channel receive, agent lifecycle, model attempt diagnostics, and outbound delivery diagnostics into one trace waterfall so channel message runs can be inspected end-to-end.

Maintainer follow-up removed the internal `AgentHarnessV2` adapter surface and kept the harness path canonical through `src/agents/harness/lifecycle.ts`.

Proof:
- PR checks passed on `04e9189c15480d53663d533a04c9883164b4dd54`.
- `node scripts/run-vitest.mjs src/agents/harness/lifecycle.test.ts src/agents/harness/selection.test.ts src/channels/turn/kernel.test.ts`
- `pnpm check:changed` Testbox `tbx_01kt3xtrm70qc7nb90cqv5rah1`

Thanks @bek91.

Co-authored-by: Bek <bek.akhmedov@gmail.com>
2026-06-02 06:38:00 -04:00