vultr/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-08-02 06:51:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: add cross-client media playback guide (#116005)

Browse Source 
* docs: add media playback guide

* docs: complete media playback glossary
This commit is contained in:
Peter Steinberger
2026-07-29 13:00:40 -04:00
committed by GitHub
parent 8ae0a896bc
commit ca36be96e0
9 changed files with 225 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
docs/.i18n/glossary.zh-CN.json
View File

@@ -1447,6 +1447,34 @@
"source": "Media understanding",
"target": "媒体理解"
},
{
"source": "Media playback",
"target": "媒体播放"
},
{
"source": "Image and media support",
"target": "图像和媒体支持"
},
{
"source": "Audio and voice notes",
"target": "音频和语音消息"
},
{
"source": "Media overview",
"target": "媒体概览"
},
{
"source": "Text to speech",
"target": "文本转语音"
},
{
"source": "Linux app",
"target": "Linux 应用"
},
{
"source": "Inline audio and video playback",
"target": "内联音频和视频播放"
},
{
"source": "Models CLI reference",
"target": "模型 CLI 参考"

1
docs/concepts/features.md
View File

@@ -59,6 +59,7 @@ title: "Features"
**Media:**
- Images, audio, video, and documents in and out
- [Inline audio and video playback](/nodes/media-playback) across the Control UI, iOS/macOS, Android, and the Linux companion
- Shared image generation and video generation capability surfaces
- Voice note transcription
- Text-to-speech with multiple providers

1
docs/docs.json
View File

@@ -1747,6 +1747,7 @@
"group": "Media capabilities",
"pages": [
"nodes/media-understanding",
"nodes/media-playback",
"nodes/images",
"nodes/audio",
"nodes/camera",

17
docs/docs_map.md
View File

@@ -5143,6 +5143,23 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
- H2: UX copy (suggested)
- H2: Related
## nodes/media-playback.md
- Route: /nodes/media-playback
- Headings:
- H2: Client support
- H2: Portable formats
- H2: Lazy playback renditions
- H2: Managed attachments and access
- H2: Metadata and limits
- H2: Troubleshooting
- H3: Duration or dimensions are missing
- H3: A recognized format downloads instead of playing
- H3: Playback stays in preparing state
- H3: Linux reports a codec error
- H3: Android shows a media row while offline
- H2: Related
## nodes/media-understanding.md
- Route: /nodes/media-understanding

5
docs/nodes/audio.md
View File

@@ -5,6 +5,10 @@ read_when:
title: "Audio and voice notes"
---
This page covers inbound transcription and voice-note handling. For inline
audio and video players in OpenClaw chat clients, see
[Media playback](/nodes/media-playback).
## What it does
When audio understanding is enabled (or auto-detected), OpenClaw:
@@ -198,6 +202,7 @@ On channels that support audio preflight, OpenClaw transcribes audio **before**
## Related
- [Media playback](/nodes/media-playback)
- [Media understanding](/nodes/media-understanding)
- [Talk mode](/nodes/talk)
- [Voice wake](/nodes/voicewake)

5
docs/nodes/images.md
View File

@@ -7,6 +7,10 @@ title: "Image and media support"
The WhatsApp channel runs on Baileys Web. This page covers media handling rules for send, gateway, and agent replies.
For inline audio and video in the Control UI and native apps, including
portable formats, byte limits, and lazy transcoding, see
[Media playback](/nodes/media-playback).
## Goals
- Send media with an optional caption via `openclaw message send --media`.
@@ -90,4 +94,5 @@ The 16MB audio/video and 100MB document figures above are the shared per-kind me
- [Camera capture](/nodes/camera)
- [Media understanding](/nodes/media-understanding)
- [Media playback](/nodes/media-playback)
- [Audio and voice notes](/nodes/audio)

153
docs/nodes/media-playback.md Normal file
View File

@@ -0,0 +1,153 @@
---
summary: "Inline audio and video playback across the Control UI and native apps"
read_when:
- Playing or troubleshooting audio and video attachments in chat
- Comparing media format support across OpenClaw clients
- Debugging playback metadata, transcoding, or codec availability
title: "Media playback"
---
OpenClaw chat clients play assistant audio and video attachments inline. The
Gateway keeps those attachments behind session-scoped access, serves seekable
byte ranges, and can prepare a portable playback rendition for recognized
formats that are not safe across every client.
This page covers playback in OpenClaw clients. Channel delivery, inbound media
understanding, and live voice conversations use separate paths; see
[Image and media support](/nodes/images),
[Media understanding](/nodes/media-understanding), and [Talk mode](/nodes/talk).
## Client support
| Client | Playback path | Operator notes |
| --------------- | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Control UI | Themed inline audio cards and native video controls | Audio cards provide play/pause, seek, elapsed and total time, download, a voice-note badge, and keyboard controls. Space toggles playback; Left/Right seek by five seconds. Starting one audio card pauses the previous one. Video upload is available from the chat attachment picker. |
| iOS and macOS | `AVAudioPlayer` for audio and `AVPlayer` for video | Inline media coordinates with Talk and Listen so two speech paths do not play over each other. For a pinned-TLS Gateway, the app performs a bounded authenticated download before video playback instead of bypassing certificate pinning. |
| Android | Media3 ExoPlayer | The app streams video through the authenticated Gateway HTTP client, requests Android audio focus, and coordinates attachment playback with Talk/TTS. Cached transcript media rows remain visible offline, but playback needs a connection to obtain a fresh media ticket. |
| Linux companion | Control UI inside the companion WebView | Codec availability comes from GStreamer. Released packages include or declare the expected codec plugins; see [Linux media codecs](/platforms/linux#media-codecs). |
## Portable formats
The Gateway classifies these formats as the portable native set shared by the
browser, Apple players, and Android Media3:
| Kind | Portable native input | Recognized transcode input | Playback target |
| ----- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------- |
| Audio | MP3; AAC in M4A/MP4; PCM WAV | AAC, AIFF, AMR/AMR-WB, CAF, FLAC, Ogg/Opus/Vorbis, WebM audio, WMA | AAC in M4A (`audio/mp4`) |
| Video | H.264 MP4 with a portable profile and 4:2:0 pixel format; AAC or MP3 audio when present | AVI, FLV, Matroska/MKV, QuickTime/MOV, WebM, ASF, WMV | H.264/AAC MP4 with 4:2:0 pixel format, at most 1920×1080 |
The Linux companion can also play formats supplied by its installed GStreamer
plugins. Browser and operating-system updates may add native formats, but the
table above is the cross-client contract OpenClaw targets.
## Lazy playback renditions
Both Gateway byte routes accept `?playback=1`: the managed attachment route
under `/api/chat/media/outgoing/.../full` and the Control UI assistant-media
route. Attachment metadata can report `playback: "native"` or
`playback: "transcode"` so a client can choose the rendition deliberately.
Playback conversion is lazy:
1. A native source passes through unchanged.
2. A recognized non-portable source starts a bounded `ffmpeg` job. The route
returns HTTP `202` with `{ "status": "preparing" }` while the rendition is
being prepared.
3. A later request receives the cached M4A or MP4 rendition.
4. If inspection or conversion is unavailable, fails, or exceeds a limit, the
route falls back to the original bytes. The client can then show its
unplayable-media fallback and keep the download action available.
Transcoding accepts sources up to 20 minutes and never raises the normal audio
or video byte cap. Cached playback renditions are pruned by normal media-store
maintenance.
## Managed attachments and access
Agent-produced audio and video are stored as managed media artifacts. Images
keep their separate managed-image artifact family. Native clients resolve the
artifact through `artifacts.download`, which returns inline base64 bytes when
the artifact is byte-backed or a short-lived, ticketed URL when it is
Gateway-managed.
The ticketed byte routes support:
- `Range` requests with HTTP `206 Partial Content` for seeking
- `ETag` and `If-Range` for safe resume behavior
- `HEAD` requests with the same content metadata and no response body
Do not copy a ticketed URL into durable configuration. Clients reacquire a
ticket from the authenticated Gateway when needed.
## Metadata and limits
Chat attachments may include `sizeBytes`, `durationMs`, `width`, and `height`.
OpenClaw also uses `ffprobe`, when available, to fill audio duration and video
duration/dimensions for media facts and the Control UI `?meta=1` availability
probe. Probing is best-effort: a missing or failed probe leaves fields absent
instead of rejecting the attachment.
Gateway-managed assistant attachments use these per-file caps:
| Kind | Maximum size |
| ----- | -----------: |
| Image | 12 MiB |
| Audio | 16 MiB |
| Video | 16 MiB |
These are playback/storage caps, not the separate media-understanding limits.
For transcription and description limits, see
[Image and media support](/nodes/images#limits-and-errors).
## Troubleshooting
### Duration or dimensions are missing
Check that `ffprobe` is installed on the Gateway host and visible on its
`PATH`:
```bash
ffprobe -version
```
Playback of an already portable file can still work without metadata.
### A recognized format downloads instead of playing
Check both media tools on the Gateway host:
```bash
ffmpeg -version
ffprobe -version
```
`ffprobe` classifies codecs and duration; `ffmpeg` creates the portable
rendition. If either step cannot safely handle the source, OpenClaw serves the
original file and the client keeps its fallback/download path.
### Playback stays in preparing state
The first rendition request is asynchronous. Wait briefly and retry. Very
large, longer than 20-minute, unprobeable, or unsupported sources remain on the
original-byte fallback instead of blocking the Gateway.
### Linux reports a codec error
Use the package and source-build instructions in
[Linux media codecs](/platforms/linux#media-codecs). The `.deb` depends on the
required GStreamer plugin packages; the AppImage carries the media framework
and codecs installed by the release build.
### Android shows a media row while offline
That is expected. Android caches the transcript metadata, not the attachment
bytes or its short-lived download capability. Reconnect, then play again so the
app can request a new ticket.
## Related
- [Image and media support](/nodes/images)
- [Audio and voice notes](/nodes/audio)
- [Media overview](/tools/media-overview)
- [Text to speech](/tools/tts)
- [Linux app](/platforms/linux)

5
docs/tools/media-overview.md
View File

@@ -48,6 +48,10 @@ telephony, meetings, browser realtime, and native push-to-talk clients.
Transcribe inbound voice messages through batch STT or Voice Call
streaming STT providers.
</Card>
<Card title="Media playback" href="/nodes/media-playback" icon="play">
Play assistant audio and video inline across the Control UI and native
apps, with managed access and portable playback renditions.
</Card>
</CardGroup>
## Provider capability matrix
@@ -171,6 +175,7 @@ catalogs returned by the Gateway.
- [Video generation](/tools/video-generation)
- [Music generation](/tools/music-generation)
- [Text-to-speech](/tools/tts)
- [Media playback](/nodes/media-playback)
- [Media understanding](/nodes/media-understanding)
- [Audio nodes](/nodes/audio)
- [Talk mode](/nodes/talk)

12
docs/tools/tts.md
View File

@@ -421,8 +421,8 @@ is required by OpenClaw's provider configuration but is not validated by the
loopback server.
</Tab>
<Tab title="macOS CLI">
The Homebrew `speech` executable can write directly to OpenClaw's temporary
output path:
The Homebrew `speech` executable can write directly to OpenClaw's
per-invocation output path:
```json5
{
@@ -812,6 +812,13 @@ whether voice-style TTS should ask providers for a native `voice-note` target or
keep normal `audio-file` synthesis, and whether the channel transcodes
non-native output before sending.
After synthesis, OpenClaw persists batch TTS output in the media store under
`tool-speech-synthesis`. The reply uses that stable media path instead of a
provider temporary file, and normal media maintenance prunes expired output.
Local CLI providers may still use `{{OutputPath}}` as scratch space before
OpenClaw imports the completed bytes. See [Media playback](/nodes/media-playback)
for inline-player formats and limits.
| Target | Format |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| Feishu / Matrix / Telegram / WhatsApp | Voice-note replies prefer **Opus** (`opus_48000_64` from ElevenLabs, `opus` from OpenAI). 48 kHz / 64 kbps balances clarity and size. |
@@ -1112,6 +1119,7 @@ provider default.
## Related
- [Media overview](/tools/media-overview)
- [Media playback](/nodes/media-playback)
- [Music generation](/tools/music-generation)
- [Video generation](/tools/video-generation)
- [Slash commands](/tools/slash-commands)