Commit Graph

51 Commits

Author SHA1 Message Date
Peter Steinberger
dc60a9442d feat(talk): support GPT-Live over gateway relay (#115831)
* feat(talk): add GPT-Live gateway relay peer

* fix(talk): ignore duplicate GPT-Live sideband audio

* fix(openai): harden GPT-Live relay startup

* fix(deps): replace vulnerable werift ip helper

* fix(deps): scope werift ip override version

* fix(openai): normalize relay startup errors

* test(ci): register GPT-Live gateway live shard

* fix(talk): tighten GPT-Live relay readiness

* fix(talk): keep relay override helper private

* fix(talk): resolve relay readiness with launch model

* fix(openai): preserve GPT-Live media clock

* fix(talk): align GPT-Live relay readiness and framing

* fix(openai): expire pending GPT-Live reservations

* fix(openai): harden GPT-Live delegation audio

* style(openai): satisfy RTP reorder lint
2026-07-29 12:54:23 -04:00
Peter Steinberger
e8524a5839 feat: add Fish Audio S2.1 and local MLX speech (#115790)
* feat: add Fish Audio speech support

* chore: remove unused speech exports

* chore: keep Fish API constant private

* test: remove stale code mode import

* style: use bracket access for Fish voice ids

* test: restore QA evidence validator import
2026-07-29 05:15:33 -04:00
Peter Steinberger
7441362014 feat(openai): support GPT-Live over backend WebSocket (#115622)
* feat(openai): add GPT-Live websocket bridge

* chore: defer GPT-Live changelog to release

* fix(android): preserve GPT-Live relay readiness gate

* fix(openai): normalize GPT-Live assistant completion
2026-07-29 04:15:03 -04:00
Peter Steinberger
921b0e1dda feat(openai): support GA realtime over ChatGPT OAuth (#115623) 2026-07-29 02:14:23 -04:00
Peter Steinberger
26bfa88bb2 fix: restore native Talk transcripts, fallback, and microphone access (#115577)
* fix(ios): decode frameless Talk events

* fix(android): keep browser-only Talk models native

* fix(macos): gate dashboard media capture

* docs(talk): clarify native platform fallbacks

* chore(i18n): refresh native source inventory

* fix(android): gate relay entry on provider-level browser-only models too
2026-07-29 01:55:47 -04:00
Peter Steinberger
46dd03ad0a fix(talk): add GPT-Live consult safety parity (#115562) 2026-07-29 00:55:33 -04:00
Peter Steinberger
7a1aa4eb4f feat(ui): Talk settings page with catalog-driven realtime pickers (#115409)
* feat(talk): emit realtime models and voices in talk.catalog and mirror create-time readiness

* feat(ui): add curated Talk settings page with catalog-driven pickers

* docs(talk): correct stale claims and add one-page GPT-Live setup path

* fix(ui): refresh Talk catalog on config-hash advance and neutralize GPT-Live badge

* fix(ui): provider-aware Talk selection, atomic provider switch, focus refresh

* fix(ui): resolve Talk provider fallbacks via catalog and make Default a true reset

* fix(ui): provider-compatible transports and race-free Talk catalog loads

* fix(ui): never resolve an unknown explicit Talk provider to the active one

* docs(talk): note Android relay readiness caveat for browser-only models

* fix(ui): keep the relay transport when switching Talk provider to Auto

* fix(ui): align section-ownership test and drop unused export after rebase
2026-07-28 17:38:15 -04:00
Peter Steinberger
f78ba09120 feat(talk): support GPT-Live realtime voice over ChatGPT OAuth (#115226)
* feat(gateway): share the canonical browser-origin policy with plugins

Export resolveAcceptedBrowserOrigin through openclaw/plugin-sdk/webhook-request-guards
so browser-facing plugin routes reuse the Gateway's real origin contract instead of a
narrow allowedOrigins array check. Private LAN/Tailnet Control UI loads and the
Host-header fallback were previously rejected with 403 by plugin offer routes while the
Control UI itself worked.

Moves the loopback/forwarded-header helpers to net.ts (re-exported from auth.ts) so the
guard can delegate without importing gateway auth, and migrates the Codex realtime
broker onto the shared seam.

* feat(talk): let providers own agent delegation for realtime voice

Adds an optional runAgentConsult callback to the browser-session create request and
injects the existing embedded consult runtime from talk-client, bound to the same agent
and session key the GA tool path uses. Providers whose realtime protocol delegates work
through their own control channel (rather than GA function calls) can now reach the
OpenClaw agent without a client round-trip.

Threads the effective per-session model into browser-session capability resolution so a
request-level model override selects the right capability set, and propagates a caller
abort signal into consultRealtimeVoiceAgent so a superseded delegation stops its run.

* feat(openai): support GPT-Live realtime voice over ChatGPT OAuth

Implements OpenAI's quicksilver/frameless session natively for Talk browser sessions.
The Gateway creates the WebRTC call (multipart sdp+session to https://api.openai.com/v1/live)
and owns the sideband control socket, so the browser never holds upstream credentials;
delegation.created events run through the OpenClaw agent and stream back as speakable
context appends.

Verified end-to-end on 2026-07-28 against a ChatGPT Pro OAuth profile: call create 201
with an rtc_* id and answer SDP, sideband session.started, session.close teardown.
ChatGPT OAuth is preferred over a Platform API key because /v1/live access for platform
keys is waitlist-gated; the legacy chatgpt.com backend route returns 403 for every model
and protocol version and is not used.

Accepted models are gpt-live-1-codex and gpt-live-1-boulder-alpha; the voice allowlist is
the ten values the route actually accepts, since an invalid voice is rejected at call
creation and cannot be repaired afterwards (session.update reports immutable_field_update).

* docs: document GPT-Live Talk support and its route gotchas

Records the working route and auth, the accepted models and voices, the browser-only
scope, and the two traps that cost the most time: the chatgpt.com backend route returns
403 Voice session access denied for every model, and that same 403 is also what an
invalid voice returns, so it must not be read as an account entitlement block.

* fix(openai): resolve GPT-Live CI failures

* refactor(openai): own zod runtime dependency

* fix(openai): satisfy lint and live-shard gates for GPT-Live

Types the retry-delay finish callback as Error so the rejection reason is provably an
Error at the call site; the abort path already normalized a non-Error AbortSignal reason,
but the unknown parameter type hid that from static analysis.

Registers the new GPT-Live live test in the native-live-extensions-openai shard
expectation. The shard selector already picked the file up from the real tree; only the
hardcoded list in the tooling test lagged.

* fix(openai): clean up post-rebase capability resolver

* fix(openai): preserve GPT-Live delegation fragments

* fix(openai): close GPT-Live sideband handoff race

* fix(openai): accept UUID GPT-Live call ids

* style: apply oxfmt to GPT-Live sources and Talk docs

* style: format Talk docs after rebase

* fix(openai): keep GPT-Live transcript context across ignored delegations
2026-07-28 15:05:55 -04:00
metaforismo
430de2299c fix(ios): use configured speech language for system voice (#113372)
* fix(ios): honor speech locale for system voice

* chore(ios): refresh talk locale i18n inventory

* fix(ios): fall through unavailable system voices

* chore(ios): refresh talk locale i18n inventory

---------

Co-authored-by: Colin Johnson <colin@solvely.net>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-28 14:10:23 -04:00
Vincent Koc
861b9ccce1 fix(talk): remove unsupported Codex OAuth realtime 2026-07-28 21:32:38 +08:00
Peter Steinberger
fa6a93ac97 fix: recheck voice confirmation after tool rewrites (#114721)
* fix(security): revalidate voice confirmation params

* chore: leave changelog to release automation
2026-07-27 16:10:54 -04:00
Colin Johnson
de9ed64c29 feat(talk): enable GPT Live with Codex OAuth (#113354) 2026-07-25 20:41:27 -04:00
Peter Steinberger
4bfb26415a feat(agents): explicit per-surface agent targets + default-role materialization (#113637)
* refactor(agents): add explicit ambient target seams

* fix(doctor): materialize ambient default-agent roles

* fix(agents): satisfy dependency and docs checks

* chore: leave agent release note to release process

* test(agents): prove shared heartbeat fan-out survives doctor
2026-07-25 05:37:41 -07:00
Peter Steinberger
c84921634d fix(macos): require explicit consent for privacy-sensitive access (#112321)
* fix(macos): avoid passive Automation prompts

* fix(macos): keep Voice Wake recognition on device

* fix(macos): require consent for activity presence

* chore(apps): refresh native i18n inventory

* fix(macos): preserve presence clears across gateway versions

* fix(macos): prioritize activity privacy opt-out

* chore(apps): refresh native i18n inventory

* fix(macos): scrub legacy presence activity

* fix(macos): migrate permission status caller

* fix(macos): preserve unknown permission state

* fix(macos): refresh privacy change artifacts

* refactor(macos): remove stale presence helper

* fix(deps): patch URI and Jaeger advisories

* test(gateway): adopt pairing-bound node sessions
2026-07-21 15:28:13 -07:00
Peter Steinberger
0f95e66b7f feat(talk): add durable client voice sessions (#111216)
Live-append voice transcripts into the agent session and persist a per-agent SQLite call record across relay and client transcript paths.

Add run-scoped spoken confirmation for high-impact actions, mutation digests, bootstrap-context injection, talk.client.transcript and talk.client.close protocol methods, and Control UI adoption. This adds zero new configuration.

Co-authored-by: Clifton King <clifton@users.noreply.github.com>
2026-07-19 01:06:49 -07:00
Peter Steinberger
4b7d6fa3bf docs: align interactive client behavior (#111047) 2026-07-18 17:00:14 -07:00
WhatsSkiLL
4d3b18811e feat(android): add Wear real-time Talk controls (#109483)
* feat(android): add Wear real-time Talk controls

* fix(android): satisfy Wear Talk CI checks

* fix(talk): remove unused client language parameter

* fix(android): harden Wear realtime Talk routing

Co-authored-by: IWhatsskill <284122573+IWhatsskill@users.noreply.github.com>

* chore(android): refresh Wear Talk metadata

* chore(android): align native i18n baseline

---------

Co-authored-by: IWhatsskill <284122573+IWhatsskill@users.noreply.github.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-17 13:46:05 +01:00
Peter Steinberger
20816f676f feat(voice): universal talk waveform driven by real audio levels (#102901)
* feat(voice): universal talk waveform driven by real audio levels

One shared Siri-style talk animation across iOS, watchOS, macOS, and
Android (TalkWaveformView + exact Compose port), replacing per-platform
fakes: synthetic speaking pulses, constant speech-detected power, static
Android bar rows, flat realtime listening. Listening/recording follow
live mic levels on every route; agent speech follows the real playback
envelope (AVAudioPlayer metering + playback-aligned PCM envelope +
WebRTC stats); voice-note recording shows a live capture wave.

* fix(android): order waveform imports and sync native i18n inventory

* chore(i18n): resync native inventory after rebase
2026-07-09 16:33:25 +01:00
Vincent Koc
27e8ea10eb docs(talk): format realtime model table 2026-07-06 23:22:33 -07:00
Vincent Koc
f0564943a7 docs(openai): refresh realtime 2.1 examples 2026-07-06 23:22:33 -07:00
Peter Steinberger
3cf0d00d90 docs: align source contract wording (#100186) 2026-07-05 01:41:40 -04:00
Peter Steinberger
53bdac249d docs: restore source-backed contract details (#100182) 2026-07-05 01:26:25 -04:00
Peter Steinberger
f7d7148cf0 docs: rewrite published docs grounded in current source (#100142)
Source-grounded rewrite of 529 published docs pages with per-unit information-loss verification: 1,713 factual corrections cited to src/**, generated surfaces regenerated, frontmatter titles preserved for i18n, release notes pages untouched. All docs gates green.

Closes #100141
2026-07-05 00:32:47 -04:00
Colin Johnson
7cfc66ad07 fix(android): derive Voice readiness from Gateway catalog (#98269)
* fix(android): derive voice readiness from Gateway catalog

Co-authored-by: Colin <colin@solvely.net>

* chore(android): sync native i18n inventory

* test(gateway): use registered realtime provider ids

* fix(gateway): satisfy Talk catalog lint

* chore(android): refresh voice i18n inventory

* fix(talk): preserve runtime readiness semantics

* fix(talk): make catalog readiness authoritative

* fix(talk): validate selected provider readiness

* fix(android): honor authoritative talk readiness

* fix(android): inventory voice readiness copy

* fix(talk): report runtime-selected catalog provider

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-02 23:32:03 -07:00
Peter Steinberger
af9ab9f193 fix(android): use Bluetooth microphones for voice capture (#99259)
* fix(android): route voice capture through Bluetooth microphones

* chore(i18n): refresh Android voice source inventory
2026-07-02 16:44:40 -07:00
PollyBot13
8b99c45bbb fix: route iOS OpenAI realtime Talk through WebRTC (#98563)
* fix: route iOS OpenAI realtime talk through WebRTC

* fix(ios): harden native realtime routing

* fix(ios): preserve realtime route ownership

* fix(ios): balance realtime audio sessions

* chore(ios): sync native i18n inventory

* fix(ios): preserve Azure realtime relay

* fix(ios): harden realtime session recovery

* chore(ios): sync native i18n inventory

* fix(ios): preserve selected Azure realtime routing

* fix(ios): rotate terminal realtime sessions

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-02 12:49:23 -07:00
Peter Steinberger
40d36b5bbc docs(talk): document realtime active-run control
Co-authored-by: Colin <colin@solvely.net>
2026-05-24 02:35:04 +01:00
VACInc
683ad75b31 fix(talk): stabilize realtime voice consults
Stabilize realtime Talk playback, transcript ordering, and consult routing across Android, Web, and the gateway relay.

- serialize Android realtime playback and transcript updates
- add opt-in forced consult routing for Talk realtime sessions
- keep web/gateway consult turns behind OpenClaw results with ordered transcript bubbles
- document the new `talk.realtime.consultRouting` config and keep prompt wording generic

Co-authored-by: VACInc <3279061+VACInc@users.noreply.github.com>
2026-05-22 15:12:39 +01:00
Peter Steinberger
e0b53cae41 docs: remove stale owner tool wording 2026-05-21 15:14:48 +01:00
Ayaan Zaidi
ea3749872a fix(android): make realtime talk mode opt-in 2026-05-18 16:19:33 +05:30
VACInc
c5cc1bb184 Enable realtime Talk instructions config 2026-05-09 21:20:06 -04:00
VACInc
cfb0c34ff6 feat: add realtime consult overrides 2026-05-09 06:51:20 -04:00
Peter Steinberger
b75e5c50bf docs: document OpenAI realtime voices 2026-05-08 01:07:46 +01:00
Peter Steinberger
63ec912786 fix(openai): use GA realtime bridge for gpt-realtime-2
Summary:
- switch OpenAI realtime voice default to gpt-realtime-2
- migrate backend OpenAI WebSocket bridge to the GA session shape and drop the beta header
- keep Azure deployment realtime bridges on the deployment-compatible shape
- extend live Talk smoke coverage and align npm Telegram package-runtime assertions

Verification:
- pnpm test extensions/openai/realtime-voice-provider.test.ts src/gateway/protocol/index.test.ts src/gateway/talk-handoff.test.ts extensions/google-meet/index.test.ts -- --reporter=dot
- pnpm test test/scripts/npm-telegram-live.test.ts -- --reporter=dot
- pnpm check:docs
- env OPENCLAW_TESTBOX=0 pnpm check:changed
- OPENCLAW_REALTIME_OPENAI_MODEL=gpt-realtime-2 node --import tsx scripts/dev/realtime-talk-live-smoke.ts
2026-05-08 00:47:25 +01:00
Peter Steinberger
7431cb8def docs: detail talk refactor plan 2026-05-06 02:39:15 +01:00
Peter Steinberger
24853ced11 docs: outline unified talk API 2026-05-06 02:39:15 +01:00
Peter Steinberger
02f3e9cfa2 fix(talk): honor configured speech locale 2026-04-25 21:05:24 +01:00
Peter Steinberger
86d897cfaa feat(android): expose talk mode
Co-authored-by: alex-latitude <213670856+alex-latitude@users.noreply.github.com>
2026-04-25 20:12:38 +01:00
Peter Steinberger
5163a2fbf7 docs: document Talk MLX config 2026-04-25 08:42:27 +01:00
Vincent Koc
d71518b1eb docs(channels,nodes): add Related sections to pages missing them 2026-04-23 15:43:47 -07:00
Vincent Koc
2777b089b5 docs: normalize frontmatter titles to sentence case 2026-04-23 13:15:17 -07:00
Vincent Koc
4a2cd533ac docs: remove duplicate H1 where frontmatter title already sets it 2026-04-23 13:11:14 -07:00
Ayaan Zaidi
3f67a52d52 docs(talk): update android playback docs 2026-04-05 08:20:47 +05:30
Peter Steinberger
b4c8950417 refactor: centralize talk silence timeout defaults 2026-03-08 14:58:29 +00:00
dano does design
6ff7e8f42e talk: add configurable silence timeout 2026-03-08 14:30:25 +00:00
Seb Slight
abcaa8c7a9 Docs: add nav titles across docs (#5689) 2026-01-31 15:04:03 -06:00
cpojer
8cab78abbc chore: Run pnpm format:fix. 2026-01-31 21:13:13 +09:00
Peter Steinberger
9a7160786a refactor: rename to openclaw 2026-01-30 03:16:21 +01:00
Peter Steinberger
6d16a658e5 refactor: rename clawdbot to moltbot with legacy compat 2026-01-27 12:21:02 +00:00
Peter Steinberger
3965c5b4d2 docs: naming sweep and platform wording 2026-01-08 23:25:51 +01:00