mirror of
https://github.com/openclaw/openclaw.git
synced 2026-07-12 13:06:04 +00:00
137 lines
14 KiB
Markdown
137 lines
14 KiB
Markdown
---
|
|
summary: "Talk mode: continuous speech conversations across local STT/TTS and realtime voice"
|
|
read_when:
|
|
- Implementing Talk mode on macOS/iOS/Android
|
|
- Changing voice/TTS/interrupt behavior
|
|
title: "Talk mode"
|
|
---
|
|
|
|
Talk mode covers five runtime shapes:
|
|
|
|
- **Native macOS/iOS/Android Talk**: local speech recognition, Gateway chat, and `talk.speak` TTS. Nodes advertise the `talk` capability and declare which `talk.*` commands they support.
|
|
- **iOS Talk (realtime)**: client-owned WebRTC for OpenAI realtime configs that select `webrtc` transport or omit transport. Explicit `gateway-relay`, `provider-websocket`, and non-OpenAI realtime configs stay on the Gateway-owned relay; non-realtime configs use the native speech loop.
|
|
- **Browser Talk**: `talk.client.create` for client-owned `webrtc`/`provider-websocket` sessions, or `talk.session.create` for Gateway-owned `gateway-relay` sessions. `managed-room` is reserved for Gateway handoff and walkie-talkie rooms.
|
|
- **Android Talk (realtime)**: opt in with `talk.realtime.mode: "realtime"` and `talk.realtime.transport: "gateway-relay"`. Otherwise Android stays on native speech recognition, Gateway chat, and `talk.speak`.
|
|
- **Transcription-only clients**: `talk.session.create({ mode: "transcription", transport: "gateway-relay", brain: "none" })`, then `talk.session.appendAudio`, `talk.session.cancelTurn`, and `talk.session.close` for captions/dictation without an assistant voice response. One-shot uploaded voice notes still use the [media understanding](/nodes/media-understanding) audio path.
|
|
|
|
Native Talk is a continuous loop: listen for speech, send the transcript to the model through the active session, wait for the response, then speak it via the configured Talk provider (`talk.speak`).
|
|
|
|
Client-owned realtime Talk forwards provider tool calls through `talk.client.toolCall` instead of calling `chat.send` directly. While a realtime consult is active, clients can call `talk.client.steer` or `talk.session.steer` to classify spoken input as `status`, `steer`, `cancel`, or `followup`. Accepted steering queues into the active embedded run; rejected steering returns a reason such as `no_active_run`, `not_streaming`, or `compacting`.
|
|
|
|
Transcription-only Talk emits the same Talk event envelope as realtime and STT/TTS sessions, but uses `mode: "transcription"` and `brain: "none"`. All Talk sessions broadcast events on the `talk.event` channel; clients subscribe to it for partial/final transcript updates (`transcript.delta`/`transcript.done`) and other session telemetry.
|
|
|
|
## Behavior (macOS)
|
|
|
|
- Always-on overlay while Talk mode is enabled.
|
|
- **Listening → Thinking → Speaking** phase transitions.
|
|
- On a short pause (silence window), the current transcript is sent.
|
|
- Replies are written to WebChat (same as typing).
|
|
- **Interrupt on speech** (default on): if the user talks while the assistant is speaking, playback stops and the interruption timestamp is noted for the next prompt.
|
|
|
|
## Voice directives in replies
|
|
|
|
The assistant can prefix a reply with a single JSON line to control voice:
|
|
|
|
```json
|
|
{ "voice": "<voice-id>", "once": true }
|
|
```
|
|
|
|
Rules:
|
|
|
|
- First non-empty line only; the JSON line is stripped before TTS playback.
|
|
- Unknown keys are ignored.
|
|
- `once: true` applies to the current reply only; without it, the voice becomes the new Talk mode default.
|
|
|
|
Supported keys: `voice` / `voice_id` / `voiceId`, `model` / `model_id` / `modelId`, `speed`, `rate` (WPM), `stability`, `similarity`, `style`, `speakerBoost`, `seed`, `normalize`, `lang`, `output_format`, `latency_tier`, `once`.
|
|
|
|
## Config (`~/.openclaw/openclaw.json`)
|
|
|
|
```json5
|
|
{
|
|
talk: {
|
|
provider: "elevenlabs",
|
|
providers: {
|
|
elevenlabs: {
|
|
voiceId: "elevenlabs_voice_id",
|
|
modelId: "eleven_v3",
|
|
outputFormat: "mp3_44100_128",
|
|
apiKey: "elevenlabs_api_key",
|
|
},
|
|
mlx: {
|
|
modelId: "mlx-community/Soprano-80M-bf16",
|
|
},
|
|
system: {},
|
|
},
|
|
speechLocale: "ru-RU",
|
|
silenceTimeoutMs: 1500,
|
|
interruptOnSpeech: true,
|
|
realtime: {
|
|
provider: "openai",
|
|
providers: {
|
|
openai: {
|
|
apiKey: "openai_api_key",
|
|
model: "gpt-realtime-2",
|
|
speakerVoice: "cedar",
|
|
},
|
|
},
|
|
instructions: "Speak warmly and keep answers brief.",
|
|
mode: "realtime",
|
|
transport: "webrtc",
|
|
brain: "agent-consult",
|
|
},
|
|
},
|
|
}
|
|
```
|
|
|
|
| Key | Default | Notes |
|
|
| ---------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
| `provider` | - | Active Talk TTS provider. Use `elevenlabs`, `mlx`, or `system` for macOS-local playback paths. |
|
|
| `providers.<id>.voiceId` | - | ElevenLabs falls back to `ELEVENLABS_VOICE_ID` / `SAG_VOICE_ID`, or the first available voice with an API key. |
|
|
| `providers.elevenlabs.modelId` | `eleven_v3` | |
|
|
| `providers.mlx.modelId` | `mlx-community/Soprano-80M-bf16` | |
|
|
| `providers.elevenlabs.apiKey` | - | Falls back to `ELEVENLABS_API_KEY` (or gateway shell profile if available). |
|
|
| `speechLocale` | device default | BCP 47 locale id for on-device Talk speech recognition on iOS/macOS. |
|
|
| `silenceTimeoutMs` | `700` ms macOS/Android, `900` ms iOS | Pause window before Talk sends the transcript. |
|
|
| `interruptOnSpeech` | `true` | |
|
|
| `outputFormat` | `pcm_44100` macOS/iOS, `pcm_24000` Android | Set `mp3_*` to force MP3 streaming. |
|
|
| `consultThinkingLevel` | unset | Thinking level override for the agent run behind realtime `openclaw_agent_consult` calls. |
|
|
| `consultFastMode` | unset | Fast-mode override for realtime `openclaw_agent_consult` calls. |
|
|
| `realtime.provider` | - | `openai` for WebRTC, `google` for provider WebSocket, or a bridge-only provider through Gateway relay. |
|
|
| `realtime.providers.<id>` | - | Provider-owned realtime config. Browsers receive only ephemeral/constrained session credentials, never a standard API key. |
|
|
| `realtime.providers.openai.speakerVoice` | `alloy` | Built-in OpenAI Realtime voice id (the older `voice` key still works but is deprecated). Current `gpt-realtime-2` voices: `alloy`, `ash`, `ballad`, `cedar`, `coral`, `echo`, `marin`, `sage`, `shimmer`, `verse`; `marin` and `cedar` are recommended for best quality. |
|
|
| `realtime.transport` | - | `webrtc`: client-owned OpenAI WebRTC on iOS and in the browser. `provider-websocket`: browser-owned, stays on Gateway relay on iOS. `gateway-relay`: keeps provider audio on the Gateway; Android uses realtime only with this transport. |
|
|
| `realtime.brain` | - | `agent-consult` routes realtime tool calls through Gateway policy; `direct-tools` is legacy direct-tool compatibility; `none` is for transcription/external orchestration. |
|
|
| `realtime.consultRouting` | - | `provider-direct` preserves the provider's direct reply when it skips `openclaw_agent_consult`; `force-agent-consult` routes finalized user transcripts through OpenClaw instead. |
|
|
| `realtime.instructions` | - | Appends provider-facing system instructions to OpenClaw's built-in realtime prompt (voice style/tone); the default `openclaw_agent_consult` guidance stays. |
|
|
|
|
`talk.catalog` exposes canonical provider ids and registry aliases, each provider's valid modes/transports/brain strategies/realtime audio formats/capability flags, and the runtime-selected readiness result. First-party Talk clients should read that catalog instead of maintaining provider aliases locally; treat an older Gateway that omits group readiness as unverified rather than definitively unconfigured. Streaming transcription providers are discovered through `talk.catalog.transcription`; the current Gateway relay uses the Voice Call streaming provider config until a dedicated Talk transcription config surface ships.
|
|
|
|
## macOS UI
|
|
|
|
- Menu bar toggle: **Talk**
|
|
- Config tab: **Talk Mode** group (voice id + interrupt toggle)
|
|
- Overlay: Listening (cloud pulses with mic level) → Thinking (sinking animation) → Speaking (radiating rings). Click the cloud to stop speaking, click X to exit Talk mode.
|
|
|
|
## Android UI
|
|
|
|
- Voice tab toggle: **Talk**
|
|
- Manual **Mic** and **Talk** are mutually exclusive capture modes.
|
|
- Manual Mic and realtime Talk prefer a connected Bluetooth Classic or BLE headset microphone; if it disconnects, the app requests another headset input or falls back to the default microphone, restoring the default preference once capture stops.
|
|
- Manual Mic stops when the app leaves the foreground or the user leaves the Voice tab.
|
|
- Talk Mode keeps running until toggled off or the node disconnects, using Android's microphone foreground-service type while active.
|
|
- Android supports `pcm_16000`, `pcm_22050`, `pcm_24000`, and `pcm_44100` output formats for low-latency `AudioTrack` streaming.
|
|
|
|
## Notes
|
|
|
|
- Requires Speech + Microphone permissions.
|
|
- Native Talk uses the active Gateway session and only falls back to history polling when response events are unavailable.
|
|
- The gateway resolves Talk playback through `talk.speak` using the active Talk provider. Android falls back to local system TTS only when that RPC is unavailable.
|
|
- macOS local MLX playback uses the bundled `openclaw-mlx-tts` helper when present, or an executable on `PATH`. Set `OPENCLAW_MLX_TTS_BIN` to point at a custom helper binary during development.
|
|
- Voice directive value ranges (ElevenLabs): `stability`, `similarity`, and `style` accept `0..1`; `speed` accepts `0.5..2`; `latency_tier` accepts `0..4`.
|
|
|
|
## Related
|
|
|
|
- [Voice wake](/nodes/voicewake)
|
|
- [Audio and voice notes](/nodes/audio)
|
|
- [Media understanding](/nodes/media-understanding)
|