edecdbd05e
* refactor(config): consolidate media model lists * refactor(config): unify memory configuration * refactor(config): consolidate TTS ownership * refactor(config): move typing policy to agents * refactor(config): retire product-level config surfaces * refactor(config): share scoped tool policy type * chore(config): refresh generated baselines * fix(config): honor agent typing overrides * fix(config): migrate sibling config consumers * refactor(infra): keep base64url decoder private * fix(config): strip invalid legacy TTS values * chore(config): refresh rebased baseline hash * fix(doctor): route legacy messages.tts.realtime voice to talk during tts move * refactor(config): polish final layout names * refactor(config): freeze retired tuning defaults * feat(config): add fast mode default symmetry * refactor(config): key agent entries by id * docs(config): update final layout reference * test(config): cover final layout migrations * chore(config): refresh final layout baselines * fix(config): align final layout runtime readers * fix(config): align remaining readers * fix(config): stabilize final layout migrations * fix(config): finalize config projection proof * fix(config): address final layout review * docs(release): preserve historical config names * fix(config): complete keyed agent migration * fix(config): close final migration gaps * fix(config): finish full-branch review * fix(config): complete runtime secret detection * fix(config): close final review findings * fix(config): finish canonical docs and heartbeat migration * fix(config): integrate latest main after rebase * refactor(env): isolate test-only controls * refactor(env): isolate build and development controls * refactor(env): collapse process identity indirection * refactor(env): remove duplicate config and temp aliases * docs(env): define the operator-facing allowlist * ci(env): ratchet production variable count * fix(env): remove stale provider helper import * fix(env): make ratchet sorting explicit * test(env): keep test seam in dead-code audit * test(env): cover ratchet growth and boundary; document surface budgets * docs(config): document tier-eval consolidations * docs(config): clarify speech preference ownership * test(memory): align retired tuning fixtures * refactor(memory): freeze engine heuristics * refactor(config): apply tier-eval tranche * refactor(tts): move persona shaping to providers * refactor(compaction): move prompt policy to providers * test(config): align hookified prompt fixtures * chore(deadcode): classify test-only exports * chore(github): remove unused spawn helper * chore(deadcode): classify queue diagnostics * chore(deadcode): remove unused lane snapshot export * chore(plugin-sdk): ratchet consolidated surface * fix(config): integrate latest main after rebase
10 KiB
summary, read_when, title, sidebarTitle
| summary | read_when | title | sidebarTitle | |||
|---|---|---|---|---|---|---|
| Image, video, music, speech, and media-understanding capabilities at a glance |
|
Media overview | Media overview |
OpenClaw generates images, videos, and music, understands inbound media (images, audio, video), and speaks replies aloud with text-to-speech. All media capabilities are tool-driven: the agent decides when to use them based on the conversation, and each tool only appears when at least one backing provider is configured.
Live speech uses the Talk session contract instead of the one-shot media tool
path. Talk has three modes: provider-native realtime, local or streaming
stt-tts, and transcription for observe-only speech capture. Those modes
share provider catalogs, event envelopes, and cancellation semantics with
telephony, meetings, browser realtime, and native push-to-talk clients.
Capabilities
Create and edit images from text prompts or reference images via `image_generate`. Async in chat sessions — runs in the background and posts the result when ready. Text-to-video, image-to-video, and video-to-video via `video_generate`. Async — runs in the background and posts the result when ready. Generate music or audio tracks via `music_generate`. Async in chat sessions on the shared media-generation task lifecycle. Convert outbound replies to spoken audio via the `tts` tool plus `tts` config. Synchronous. Summarize inbound images, audio, and video using vision-capable model providers and dedicated media-understanding plugins. Transcribe inbound voice messages through batch STT or Voice Call streaming STT providers.Provider capability matrix
This table covers the dedicated media-generation, TTS, and STT plugins. Many chat-model providers (Anthropic, Google, OpenAI, and others) also understand inbound media through their reply model; see the full provider list in [Media understanding](/nodes/media-understanding#provider-support-matrix).| Provider | Image | Video | Music | TTS | STT | Realtime voice | Media understanding |
|---|---|---|---|---|---|---|---|
| Alibaba | ✓ | ||||||
| Azure Speech | ✓ | ||||||
| BytePlus | ✓ | ||||||
| ComfyUI | ✓ | ✓ | ✓ | ||||
| Deepgram | ✓ | ||||||
| DeepInfra | ✓ | ✓ | ✓ | ✓ | ✓ | ||
| ElevenLabs | ✓ | ✓ | |||||
| fal | ✓ | ✓ | ✓ | ||||
| ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
| Gradium | ✓ | ||||||
| Inworld | ✓ | ||||||
| LiteLLM | ✓ | ||||||
| Local CLI | ✓ | ||||||
| Microsoft | ✓ | ||||||
| Microsoft Foundry | ✓ | ||||||
| MiniMax | ✓ | ✓ | ✓ | ✓ | |||
| Mistral | ✓ | ||||||
| OpenAI | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
| OpenRouter | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
| PixVerse | ✓ | ||||||
| Qwen | ✓ | ✓ | |||||
| Runway | ✓ | ||||||
| SenseAudio | ✓ | ||||||
| Together | ✓ | ||||||
| Volcengine | ✓ | ||||||
| Vydra | ✓ | ✓ | ✓ | ||||
| xAI | ✓ | ✓ | ✓ | ✓ | ✓ | ||
| Xiaomi MiMo | ✓ |
Async vs synchronous
| Capability | Mode | Why |
|---|---|---|
| Image | Asynchronous | Provider processing can outlive a chat turn; generated attachments use the shared completion path. |
| Text-to-speech | Synchronous | Provider responses return in seconds; attached to the reply audio. |
| Video | Asynchronous | Provider processing takes 30 s to several minutes; slow queues can run up to the configured timeout. |
| Music | Asynchronous | Same provider-processing characteristic as video. |
For async tools, OpenClaw submits the request to the provider, returns a task
id immediately, and tracks the job in the task ledger. The agent continues
responding to other messages while the job runs. When the provider finishes,
OpenClaw wakes the agent with the generated media paths so it can tell the
user through the session's normal visible-reply mode: automatic final reply
delivery when configured, or message(action="send") when the session requires
the message tool. If the requester session is inactive or its active wake
fails, and some generated media is still missing from the completion reply,
OpenClaw sends an idempotent direct fallback with only the missing media. Media
already delivered by the completion reply is not posted again.
Speech-to-text and Voice Call
Deepgram, DeepInfra, ElevenLabs, Google, Groq, Mistral, OpenAI, OpenRouter,
SenseAudio, and xAI can all transcribe inbound audio through the batch
tools.media.audio path when configured. Channel plugins that preflight a
voice note for mention gating or command parsing mark the transcribed
attachment on the inbound context, so the shared media-understanding pass
reuses that transcript instead of making a second STT call for the same
audio.
Deepgram, ElevenLabs, Mistral, OpenAI, and xAI also register Voice Call streaming STT providers, so live phone audio can be forwarded to the selected vendor without waiting for a completed recording.
For live user conversations, prefer Talk mode. Batch audio attachments stay on the media path; browser realtime, native push-to-talk, telephony, and meeting audio should use Talk events and the session-scoped catalogs returned by the Gateway.