mirror of
https://github.com/openclaw/openclaw.git
synced 2026-04-12 01:31:08 +00:00
docs(video): document runway support
This commit is contained in:
Peter Steinberger
@@ -34,6 +34,7 @@ Docs: https://docs.openclaw.ai
|
||||
- Agents/cache: stabilize cache-relevant system prompt fingerprints by normalizing equivalent structured prompt whitespace, line endings, hook-added system context, and runtime capability ordering so semantically unchanged prompts reuse KV/cache more reliably. Thanks @vincentkoc.
|
||||
- Agents/tool prompts: remove the duplicate in-band tool inventory from agent system prompts so tool-calling models rely on the structured tool definitions as the single source of truth, improving prompt stability and reducing stale tool guidance.
|
||||
- Tools/video generation: add bundled xAI (`grok-imagine-video`) and Alibaba Model Studio Wan video providers, plus live-test/default model wiring for both.
|
||||
- Tools/video generation: add a bundled Runway video provider (`runway/gen4.5`) with native async task polling, local image/video reference support via data URIs, provider docs, and live-test wiring.
|
||||
- Agents/video generation: register `video_generate` runs in the task ledger with task/run ids and lifecycle updates so long-running generations can be tracked more reliably.
|
||||
- Agents/video generation: make session-backed `video_generate` runs detach into background tasks, wake the same agent session on completion, and have the agent post the finished video back into the original channel as a follow-up reply.
|
||||
- Providers/CLI: remove bundled CLI text-provider backends and the `agents.defaults.cliBackends` surface, while keeping ACP harness sessions and Gemini media understanding on the native bundled providers.
|
||||
|
||||
@@ -198,6 +198,8 @@ Current bundled examples:
|
||||
media-understanding and video-generation provider registrations for its
|
||||
multimodal surfaces; Qwen video generation uses the Standard DashScope video
|
||||
endpoints with bundled Wan models such as `wan2.6-t2v` and `wan2.7-r2v`
|
||||
- `runway`: plugin-owned video-generation provider registration for native
|
||||
Runway task-based models such as `gen4.5`
|
||||
- `minimax`: plugin-owned catalogs, bundled video-generation provider
|
||||
registration for Hailuo video models, bundled image-generation provider
|
||||
registration for `image-01`, hybrid Anthropic/OpenAI replay-policy
|
||||
|
||||
@@ -54,6 +54,7 @@ Looking for chat channel docs (WhatsApp/Telegram/Discord/Slack/Mattermost (plugi
|
||||
- [Perplexity (web search)](/providers/perplexity-provider)
|
||||
- [Qianfan](/providers/qianfan)
|
||||
- [Qwen Cloud](/providers/qwen)
|
||||
- [Runway](/providers/runway)
|
||||
- [SGLang (local models)](/providers/sglang)
|
||||
- [StepFun](/providers/stepfun)
|
||||
- [Synthetic](/providers/synthetic)
|
||||
|
||||
67
docs/providers/runway.md
Normal file
67
docs/providers/runway.md
Normal file
@@ -0,0 +1,67 @@
|
||||
---
|
||||
title: "Runway"
|
||||
summary: "Runway video generation setup in OpenClaw"
|
||||
read_when:
|
||||
- You want to use Runway video generation in OpenClaw
|
||||
- You need the Runway API key/env setup
|
||||
- You want to make Runway the default video provider
|
||||
---
|
||||
|
||||
# Runway
|
||||
|
||||
OpenClaw ships a bundled `runway` provider for hosted video generation.
|
||||
|
||||
- Provider: `runway`
|
||||
- Auth: `RUNWAYML_API_SECRET` (canonical; `RUNWAY_API_KEY` also works)
|
||||
- API: Runway task-based video generation API
|
||||
|
||||
## Quick start
|
||||
|
||||
1. Set the API key:
|
||||
|
||||
```bash
|
||||
openclaw onboard --auth-choice runway-api-key
|
||||
```
|
||||
|
||||
2. Set a default video model:
|
||||
|
||||
```json5
|
||||
{
|
||||
agents: {
|
||||
defaults: {
|
||||
videoGenerationModel: {
|
||||
primary: "runway/gen4.5",
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Video generation
|
||||
|
||||
The bundled `runway` video-generation provider defaults to `runway/gen4.5`.
|
||||
|
||||
- Modes: text-to-video, single-image image-to-video, and single-video video-to-video
|
||||
- Runtime: async task submit + poll via `GET /v1/tasks/{id}`
|
||||
- Local image/video references: supported via data URIs
|
||||
- Current video-to-video caveat: OpenClaw currently requires `runway/gen4_aleph` for video inputs
|
||||
- Current text-to-video caveat: OpenClaw currently exposes `16:9` and `9:16` for text-only runs
|
||||
|
||||
To use Runway as the default video provider:
|
||||
|
||||
```json5
|
||||
{
|
||||
agents: {
|
||||
defaults: {
|
||||
videoGenerationModel: {
|
||||
primary: "runway/gen4.5",
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- [Video Generation](/tools/video-generation)
|
||||
- [Configuration Reference](/gateway/configuration-reference#agent-defaults)
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
summary: "Generate videos using configured providers such as Alibaba, OpenAI, Google, Qwen, and MiniMax"
|
||||
summary: "Generate videos using configured providers such as Alibaba, OpenAI, Google, Qwen, MiniMax, and Runway"
|
||||
read_when:
|
||||
- Generating videos via the agent
|
||||
- Configuring video generation providers and models
|
||||
@@ -21,7 +21,7 @@ In agent sessions, `video_generate` returns immediately with a task id/run id. T
|
||||
|
||||
## Quick start
|
||||
|
||||
1. Set an API key for at least one provider (for example `OPENAI_API_KEY`, `GEMINI_API_KEY`, `MODELSTUDIO_API_KEY`, or `QWEN_API_KEY`).
|
||||
1. Set an API key for at least one provider (for example `OPENAI_API_KEY`, `GEMINI_API_KEY`, `MODELSTUDIO_API_KEY`, `QWEN_API_KEY`, or `RUNWAYML_API_SECRET`).
|
||||
2. Optionally set your preferred model:
|
||||
|
||||
```json5
|
||||
@@ -53,6 +53,7 @@ For direct synchronous contexts without a session-backed agent run, the tool sti
|
||||
| MiniMax | `MiniMax-Hailuo-2.3` | 1 image | `MINIMAX_API_KEY` |
|
||||
| OpenAI | `sora-2` | 1 image or 1 video | `OPENAI_API_KEY` |
|
||||
| Qwen | `wan2.6-t2v` | Yes, remote URLs | `QWEN_API_KEY`, `MODELSTUDIO_API_KEY`, `DASHSCOPE_API_KEY` |
|
||||
| Runway | `gen4.5` | 1 image or 1 video | `RUNWAYML_API_SECRET`, `RUNWAY_API_KEY` |
|
||||
| Together | `Wan-AI/Wan2.2-T2V-A14B` | 1 image | `TOGETHER_API_KEY` |
|
||||
| xAI | `grok-imagine-video` | 1 image or 1 video | `XAI_API_KEY` |
|
||||
|
||||
@@ -127,6 +128,7 @@ If a provider fails, the next candidate is tried automatically. If all fail, the
|
||||
- MiniMax, Together, BytePlus, and fal currently support a single image reference input.
|
||||
- OpenAI uses the native video endpoint and currently defaults to `sora-2`.
|
||||
- Qwen supports image/video references, but the upstream DashScope video endpoint currently requires remote `http(s)` URLs for those references.
|
||||
- Runway uses the native async task API with `GET /v1/tasks/{id}` polling and currently defaults to `gen4.5`.
|
||||
- xAI uses the native xAI video API and supports text-to-video, image-to-video, and remote video edit/extend flows.
|
||||
- fal uses the queue-backed fal video flow for long-running jobs instead of a single blocking inference request.
|
||||
|
||||
@@ -143,6 +145,7 @@ The bundled Qwen provider supports text-to-video plus image/video reference mode
|
||||
- [MiniMax](/providers/minimax) — Hailuo provider setup
|
||||
- [OpenAI](/providers/openai) — Sora provider setup
|
||||
- [Qwen](/providers/qwen) — Qwen-specific setup and limits
|
||||
- [Runway](/providers/runway) — Runway setup and current model/input notes
|
||||
- [Together AI](/providers/together) — Together Wan provider setup
|
||||
- [xAI](/providers/xai) — Grok video provider setup
|
||||
- [Configuration Reference](/gateway/configuration-reference#agent-defaults) — `videoGenerationModel` config
|
||||
|
||||
Reference in New Issue
Block a user