diff --git a/docs/.i18n/glossary.zh-CN.json b/docs/.i18n/glossary.zh-CN.json index 3f8693deb2de..495d930a863e 100644 --- a/docs/.i18n/glossary.zh-CN.json +++ b/docs/.i18n/glossary.zh-CN.json @@ -1447,6 +1447,34 @@ "source": "Media understanding", "target": "媒体理解" }, + { + "source": "Media playback", + "target": "媒体播放" + }, + { + "source": "Image and media support", + "target": "图像和媒体支持" + }, + { + "source": "Audio and voice notes", + "target": "音频和语音消息" + }, + { + "source": "Media overview", + "target": "媒体概览" + }, + { + "source": "Text to speech", + "target": "文本转语音" + }, + { + "source": "Linux app", + "target": "Linux 应用" + }, + { + "source": "Inline audio and video playback", + "target": "内联音频和视频播放" + }, { "source": "Models CLI reference", "target": "模型 CLI 参考" diff --git a/docs/concepts/features.md b/docs/concepts/features.md index 446a3d38743c..8456b0308507 100644 --- a/docs/concepts/features.md +++ b/docs/concepts/features.md @@ -59,6 +59,7 @@ title: "Features" **Media:** - Images, audio, video, and documents in and out +- [Inline audio and video playback](/nodes/media-playback) across the Control UI, iOS/macOS, Android, and the Linux companion - Shared image generation and video generation capability surfaces - Voice note transcription - Text-to-speech with multiple providers diff --git a/docs/docs.json b/docs/docs.json index 890577e49407..84237758df4b 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -1747,6 +1747,7 @@ "group": "Media capabilities", "pages": [ "nodes/media-understanding", + "nodes/media-playback", "nodes/images", "nodes/audio", "nodes/camera", diff --git a/docs/docs_map.md b/docs/docs_map.md index 7e5390af9122..6c124fcae6fc 100644 --- a/docs/docs_map.md +++ b/docs/docs_map.md @@ -5143,6 +5143,23 @@ Do not edit it by hand; run `pnpm docs:map:gen`. - H2: UX copy (suggested) - H2: Related +## nodes/media-playback.md + +- Route: /nodes/media-playback +- Headings: + - H2: Client support + - H2: Portable formats + - H2: Lazy playback renditions + - H2: Managed attachments and access + - H2: Metadata and limits + - H2: Troubleshooting + - H3: Duration or dimensions are missing + - H3: A recognized format downloads instead of playing + - H3: Playback stays in preparing state + - H3: Linux reports a codec error + - H3: Android shows a media row while offline + - H2: Related + ## nodes/media-understanding.md - Route: /nodes/media-understanding diff --git a/docs/nodes/audio.md b/docs/nodes/audio.md index 81312172b3aa..21a012a379fd 100644 --- a/docs/nodes/audio.md +++ b/docs/nodes/audio.md @@ -5,6 +5,10 @@ read_when: title: "Audio and voice notes" --- +This page covers inbound transcription and voice-note handling. For inline +audio and video players in OpenClaw chat clients, see +[Media playback](/nodes/media-playback). + ## What it does When audio understanding is enabled (or auto-detected), OpenClaw: @@ -198,6 +202,7 @@ On channels that support audio preflight, OpenClaw transcribes audio **before** ## Related +- [Media playback](/nodes/media-playback) - [Media understanding](/nodes/media-understanding) - [Talk mode](/nodes/talk) - [Voice wake](/nodes/voicewake) diff --git a/docs/nodes/images.md b/docs/nodes/images.md index 0b33ba5f58ae..fe77964adf6c 100644 --- a/docs/nodes/images.md +++ b/docs/nodes/images.md @@ -7,6 +7,10 @@ title: "Image and media support" The WhatsApp channel runs on Baileys Web. This page covers media handling rules for send, gateway, and agent replies. +For inline audio and video in the Control UI and native apps, including +portable formats, byte limits, and lazy transcoding, see +[Media playback](/nodes/media-playback). + ## Goals - Send media with an optional caption via `openclaw message send --media`. @@ -90,4 +94,5 @@ The 16MB audio/video and 100MB document figures above are the shared per-kind me - [Camera capture](/nodes/camera) - [Media understanding](/nodes/media-understanding) +- [Media playback](/nodes/media-playback) - [Audio and voice notes](/nodes/audio) diff --git a/docs/nodes/media-playback.md b/docs/nodes/media-playback.md new file mode 100644 index 000000000000..73dca0162bee --- /dev/null +++ b/docs/nodes/media-playback.md @@ -0,0 +1,153 @@ +--- +summary: "Inline audio and video playback across the Control UI and native apps" +read_when: + - Playing or troubleshooting audio and video attachments in chat + - Comparing media format support across OpenClaw clients + - Debugging playback metadata, transcoding, or codec availability +title: "Media playback" +--- + +OpenClaw chat clients play assistant audio and video attachments inline. The +Gateway keeps those attachments behind session-scoped access, serves seekable +byte ranges, and can prepare a portable playback rendition for recognized +formats that are not safe across every client. + +This page covers playback in OpenClaw clients. Channel delivery, inbound media +understanding, and live voice conversations use separate paths; see +[Image and media support](/nodes/images), +[Media understanding](/nodes/media-understanding), and [Talk mode](/nodes/talk). + +## Client support + +| Client | Playback path | Operator notes | +| --------------- | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Control UI | Themed inline audio cards and native video controls | Audio cards provide play/pause, seek, elapsed and total time, download, a voice-note badge, and keyboard controls. Space toggles playback; Left/Right seek by five seconds. Starting one audio card pauses the previous one. Video upload is available from the chat attachment picker. | +| iOS and macOS | `AVAudioPlayer` for audio and `AVPlayer` for video | Inline media coordinates with Talk and Listen so two speech paths do not play over each other. For a pinned-TLS Gateway, the app performs a bounded authenticated download before video playback instead of bypassing certificate pinning. | +| Android | Media3 ExoPlayer | The app streams video through the authenticated Gateway HTTP client, requests Android audio focus, and coordinates attachment playback with Talk/TTS. Cached transcript media rows remain visible offline, but playback needs a connection to obtain a fresh media ticket. | +| Linux companion | Control UI inside the companion WebView | Codec availability comes from GStreamer. Released packages include or declare the expected codec plugins; see [Linux media codecs](/platforms/linux#media-codecs). | + +## Portable formats + +The Gateway classifies these formats as the portable native set shared by the +browser, Apple players, and Android Media3: + +| Kind | Portable native input | Recognized transcode input | Playback target | +| ----- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------- | +| Audio | MP3; AAC in M4A/MP4; PCM WAV | AAC, AIFF, AMR/AMR-WB, CAF, FLAC, Ogg/Opus/Vorbis, WebM audio, WMA | AAC in M4A (`audio/mp4`) | +| Video | H.264 MP4 with a portable profile and 4:2:0 pixel format; AAC or MP3 audio when present | AVI, FLV, Matroska/MKV, QuickTime/MOV, WebM, ASF, WMV | H.264/AAC MP4 with 4:2:0 pixel format, at most 1920×1080 | + +The Linux companion can also play formats supplied by its installed GStreamer +plugins. Browser and operating-system updates may add native formats, but the +table above is the cross-client contract OpenClaw targets. + +## Lazy playback renditions + +Both Gateway byte routes accept `?playback=1`: the managed attachment route +under `/api/chat/media/outgoing/.../full` and the Control UI assistant-media +route. Attachment metadata can report `playback: "native"` or +`playback: "transcode"` so a client can choose the rendition deliberately. + +Playback conversion is lazy: + +1. A native source passes through unchanged. +2. A recognized non-portable source starts a bounded `ffmpeg` job. The route + returns HTTP `202` with `{ "status": "preparing" }` while the rendition is + being prepared. +3. A later request receives the cached M4A or MP4 rendition. +4. If inspection or conversion is unavailable, fails, or exceeds a limit, the + route falls back to the original bytes. The client can then show its + unplayable-media fallback and keep the download action available. + +Transcoding accepts sources up to 20 minutes and never raises the normal audio +or video byte cap. Cached playback renditions are pruned by normal media-store +maintenance. + +## Managed attachments and access + +Agent-produced audio and video are stored as managed media artifacts. Images +keep their separate managed-image artifact family. Native clients resolve the +artifact through `artifacts.download`, which returns inline base64 bytes when +the artifact is byte-backed or a short-lived, ticketed URL when it is +Gateway-managed. + +The ticketed byte routes support: + +- `Range` requests with HTTP `206 Partial Content` for seeking +- `ETag` and `If-Range` for safe resume behavior +- `HEAD` requests with the same content metadata and no response body + +Do not copy a ticketed URL into durable configuration. Clients reacquire a +ticket from the authenticated Gateway when needed. + +## Metadata and limits + +Chat attachments may include `sizeBytes`, `durationMs`, `width`, and `height`. +OpenClaw also uses `ffprobe`, when available, to fill audio duration and video +duration/dimensions for media facts and the Control UI `?meta=1` availability +probe. Probing is best-effort: a missing or failed probe leaves fields absent +instead of rejecting the attachment. + +Gateway-managed assistant attachments use these per-file caps: + +| Kind | Maximum size | +| ----- | -----------: | +| Image | 12 MiB | +| Audio | 16 MiB | +| Video | 16 MiB | + +These are playback/storage caps, not the separate media-understanding limits. +For transcription and description limits, see +[Image and media support](/nodes/images#limits-and-errors). + +## Troubleshooting + +### Duration or dimensions are missing + +Check that `ffprobe` is installed on the Gateway host and visible on its +`PATH`: + +```bash +ffprobe -version +``` + +Playback of an already portable file can still work without metadata. + +### A recognized format downloads instead of playing + +Check both media tools on the Gateway host: + +```bash +ffmpeg -version +ffprobe -version +``` + +`ffprobe` classifies codecs and duration; `ffmpeg` creates the portable +rendition. If either step cannot safely handle the source, OpenClaw serves the +original file and the client keeps its fallback/download path. + +### Playback stays in preparing state + +The first rendition request is asynchronous. Wait briefly and retry. Very +large, longer than 20-minute, unprobeable, or unsupported sources remain on the +original-byte fallback instead of blocking the Gateway. + +### Linux reports a codec error + +Use the package and source-build instructions in +[Linux media codecs](/platforms/linux#media-codecs). The `.deb` depends on the +required GStreamer plugin packages; the AppImage carries the media framework +and codecs installed by the release build. + +### Android shows a media row while offline + +That is expected. Android caches the transcript metadata, not the attachment +bytes or its short-lived download capability. Reconnect, then play again so the +app can request a new ticket. + +## Related + +- [Image and media support](/nodes/images) +- [Audio and voice notes](/nodes/audio) +- [Media overview](/tools/media-overview) +- [Text to speech](/tools/tts) +- [Linux app](/platforms/linux) diff --git a/docs/tools/media-overview.md b/docs/tools/media-overview.md index edc3f11ef0f4..e4c93a8286d1 100644 --- a/docs/tools/media-overview.md +++ b/docs/tools/media-overview.md @@ -48,6 +48,10 @@ telephony, meetings, browser realtime, and native push-to-talk clients. Transcribe inbound voice messages through batch STT or Voice Call streaming STT providers. + + Play assistant audio and video inline across the Control UI and native + apps, with managed access and portable playback renditions. + ## Provider capability matrix @@ -171,6 +175,7 @@ catalogs returned by the Gateway. - [Video generation](/tools/video-generation) - [Music generation](/tools/music-generation) - [Text-to-speech](/tools/tts) +- [Media playback](/nodes/media-playback) - [Media understanding](/nodes/media-understanding) - [Audio nodes](/nodes/audio) - [Talk mode](/nodes/talk) diff --git a/docs/tools/tts.md b/docs/tools/tts.md index a15491175a83..071eaebe89bb 100644 --- a/docs/tools/tts.md +++ b/docs/tools/tts.md @@ -421,8 +421,8 @@ is required by OpenClaw's provider configuration but is not validated by the loopback server. -The Homebrew `speech` executable can write directly to OpenClaw's temporary -output path: +The Homebrew `speech` executable can write directly to OpenClaw's +per-invocation output path: ```json5 { @@ -812,6 +812,13 @@ whether voice-style TTS should ask providers for a native `voice-note` target or keep normal `audio-file` synthesis, and whether the channel transcodes non-native output before sending. +After synthesis, OpenClaw persists batch TTS output in the media store under +`tool-speech-synthesis`. The reply uses that stable media path instead of a +provider temporary file, and normal media maintenance prunes expired output. +Local CLI providers may still use `{{OutputPath}}` as scratch space before +OpenClaw imports the completed bytes. See [Media playback](/nodes/media-playback) +for inline-player formats and limits. + | Target | Format | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | Feishu / Matrix / Telegram / WhatsApp | Voice-note replies prefer **Opus** (`opus_48000_64` from ElevenLabs, `opus` from OpenAI). 48 kHz / 64 kbps balances clarity and size. | @@ -1112,6 +1119,7 @@ provider default. ## Related - [Media overview](/tools/media-overview) +- [Media playback](/nodes/media-playback) - [Music generation](/tools/music-generation) - [Video generation](/tools/video-generation) - [Slash commands](/tools/slash-commands)