From 379bc1c032e924f45ae015de22bb5f01859dd7cb Mon Sep 17 00:00:00 2001 From: Peter Steinberger Date: Mon, 6 Apr 2026 00:50:18 +0100 Subject: [PATCH] docs(video): document runway support --- CHANGELOG.md | 1 + docs/concepts/model-providers.md | 2 + docs/providers/index.md | 1 + docs/providers/runway.md | 67 ++++++++++++++++++++++++++++++++ docs/tools/video-generation.md | 7 +++- 5 files changed, 76 insertions(+), 2 deletions(-) create mode 100644 docs/providers/runway.md diff --git a/CHANGELOG.md b/CHANGELOG.md index a93e4378b38..33b7c43e852 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,7 @@ Docs: https://docs.openclaw.ai - Agents/cache: stabilize cache-relevant system prompt fingerprints by normalizing equivalent structured prompt whitespace, line endings, hook-added system context, and runtime capability ordering so semantically unchanged prompts reuse KV/cache more reliably. Thanks @vincentkoc. - Agents/tool prompts: remove the duplicate in-band tool inventory from agent system prompts so tool-calling models rely on the structured tool definitions as the single source of truth, improving prompt stability and reducing stale tool guidance. - Tools/video generation: add bundled xAI (`grok-imagine-video`) and Alibaba Model Studio Wan video providers, plus live-test/default model wiring for both. +- Tools/video generation: add a bundled Runway video provider (`runway/gen4.5`) with native async task polling, local image/video reference support via data URIs, provider docs, and live-test wiring. - Agents/video generation: register `video_generate` runs in the task ledger with task/run ids and lifecycle updates so long-running generations can be tracked more reliably. - Agents/video generation: make session-backed `video_generate` runs detach into background tasks, wake the same agent session on completion, and have the agent post the finished video back into the original channel as a follow-up reply. - Providers/CLI: remove bundled CLI text-provider backends and the `agents.defaults.cliBackends` surface, while keeping ACP harness sessions and Gemini media understanding on the native bundled providers. diff --git a/docs/concepts/model-providers.md b/docs/concepts/model-providers.md index 5043593fcee..df9e951c886 100644 --- a/docs/concepts/model-providers.md +++ b/docs/concepts/model-providers.md @@ -198,6 +198,8 @@ Current bundled examples: media-understanding and video-generation provider registrations for its multimodal surfaces; Qwen video generation uses the Standard DashScope video endpoints with bundled Wan models such as `wan2.6-t2v` and `wan2.7-r2v` +- `runway`: plugin-owned video-generation provider registration for native + Runway task-based models such as `gen4.5` - `minimax`: plugin-owned catalogs, bundled video-generation provider registration for Hailuo video models, bundled image-generation provider registration for `image-01`, hybrid Anthropic/OpenAI replay-policy diff --git a/docs/providers/index.md b/docs/providers/index.md index 8c7ac72c4d8..91a8dc02372 100644 --- a/docs/providers/index.md +++ b/docs/providers/index.md @@ -54,6 +54,7 @@ Looking for chat channel docs (WhatsApp/Telegram/Discord/Slack/Mattermost (plugi - [Perplexity (web search)](/providers/perplexity-provider) - [Qianfan](/providers/qianfan) - [Qwen Cloud](/providers/qwen) +- [Runway](/providers/runway) - [SGLang (local models)](/providers/sglang) - [StepFun](/providers/stepfun) - [Synthetic](/providers/synthetic) diff --git a/docs/providers/runway.md b/docs/providers/runway.md new file mode 100644 index 00000000000..e3d9cd7d8c2 --- /dev/null +++ b/docs/providers/runway.md @@ -0,0 +1,67 @@ +--- +title: "Runway" +summary: "Runway video generation setup in OpenClaw" +read_when: + - You want to use Runway video generation in OpenClaw + - You need the Runway API key/env setup + - You want to make Runway the default video provider +--- + +# Runway + +OpenClaw ships a bundled `runway` provider for hosted video generation. + +- Provider: `runway` +- Auth: `RUNWAYML_API_SECRET` (canonical; `RUNWAY_API_KEY` also works) +- API: Runway task-based video generation API + +## Quick start + +1. Set the API key: + +```bash +openclaw onboard --auth-choice runway-api-key +``` + +2. Set a default video model: + +```json5 +{ + agents: { + defaults: { + videoGenerationModel: { + primary: "runway/gen4.5", + }, + }, + }, +} +``` + +## Video generation + +The bundled `runway` video-generation provider defaults to `runway/gen4.5`. + +- Modes: text-to-video, single-image image-to-video, and single-video video-to-video +- Runtime: async task submit + poll via `GET /v1/tasks/{id}` +- Local image/video references: supported via data URIs +- Current video-to-video caveat: OpenClaw currently requires `runway/gen4_aleph` for video inputs +- Current text-to-video caveat: OpenClaw currently exposes `16:9` and `9:16` for text-only runs + +To use Runway as the default video provider: + +```json5 +{ + agents: { + defaults: { + videoGenerationModel: { + primary: "runway/gen4.5", + }, + }, + }, +} +``` + +## Related + +- [Video Generation](/tools/video-generation) +- [Configuration Reference](/gateway/configuration-reference#agent-defaults) diff --git a/docs/tools/video-generation.md b/docs/tools/video-generation.md index bcdfe66b94d..76ea91702d8 100644 --- a/docs/tools/video-generation.md +++ b/docs/tools/video-generation.md @@ -1,5 +1,5 @@ --- -summary: "Generate videos using configured providers such as Alibaba, OpenAI, Google, Qwen, and MiniMax" +summary: "Generate videos using configured providers such as Alibaba, OpenAI, Google, Qwen, MiniMax, and Runway" read_when: - Generating videos via the agent - Configuring video generation providers and models @@ -21,7 +21,7 @@ In agent sessions, `video_generate` returns immediately with a task id/run id. T ## Quick start -1. Set an API key for at least one provider (for example `OPENAI_API_KEY`, `GEMINI_API_KEY`, `MODELSTUDIO_API_KEY`, or `QWEN_API_KEY`). +1. Set an API key for at least one provider (for example `OPENAI_API_KEY`, `GEMINI_API_KEY`, `MODELSTUDIO_API_KEY`, `QWEN_API_KEY`, or `RUNWAYML_API_SECRET`). 2. Optionally set your preferred model: ```json5 @@ -53,6 +53,7 @@ For direct synchronous contexts without a session-backed agent run, the tool sti | MiniMax | `MiniMax-Hailuo-2.3` | 1 image | `MINIMAX_API_KEY` | | OpenAI | `sora-2` | 1 image or 1 video | `OPENAI_API_KEY` | | Qwen | `wan2.6-t2v` | Yes, remote URLs | `QWEN_API_KEY`, `MODELSTUDIO_API_KEY`, `DASHSCOPE_API_KEY` | +| Runway | `gen4.5` | 1 image or 1 video | `RUNWAYML_API_SECRET`, `RUNWAY_API_KEY` | | Together | `Wan-AI/Wan2.2-T2V-A14B` | 1 image | `TOGETHER_API_KEY` | | xAI | `grok-imagine-video` | 1 image or 1 video | `XAI_API_KEY` | @@ -127,6 +128,7 @@ If a provider fails, the next candidate is tried automatically. If all fail, the - MiniMax, Together, BytePlus, and fal currently support a single image reference input. - OpenAI uses the native video endpoint and currently defaults to `sora-2`. - Qwen supports image/video references, but the upstream DashScope video endpoint currently requires remote `http(s)` URLs for those references. +- Runway uses the native async task API with `GET /v1/tasks/{id}` polling and currently defaults to `gen4.5`. - xAI uses the native xAI video API and supports text-to-video, image-to-video, and remote video edit/extend flows. - fal uses the queue-backed fal video flow for long-running jobs instead of a single blocking inference request. @@ -143,6 +145,7 @@ The bundled Qwen provider supports text-to-video plus image/video reference mode - [MiniMax](/providers/minimax) — Hailuo provider setup - [OpenAI](/providers/openai) — Sora provider setup - [Qwen](/providers/qwen) — Qwen-specific setup and limits +- [Runway](/providers/runway) — Runway setup and current model/input notes - [Together AI](/providers/together) — Together Wan provider setup - [xAI](/providers/xai) — Grok video provider setup - [Configuration Reference](/gateway/configuration-reference#agent-defaults) — `videoGenerationModel` config