vultr/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-05-06 11:50:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(fal): support Seedance reference video

Browse Source 
Adds fal Seedance 2.0 reference-to-video support with model-aware reference input limits.
This commit is contained in:
Shivanker Goel
2026-04-26 02:30:23 +01:00
committed by GitHub
parent 566d2d73a3
commit a932a58e87
18 changed files with 675 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/plugins/sdk-provider-plugins.md
View File

@@ -626,7 +626,13 @@ API key auth, and dynamic model resolution.
label: "Acme Video",
capabilities: {
generate: { maxVideos: 1, maxDurationSeconds: 10, supportsResolution: true },
imageToVideo: { enabled: true, maxVideos: 1, maxInputImages: 1, maxDurationSeconds: 5 },
imageToVideo: {
enabled: true,
maxVideos: 1,
maxInputImages: 1,
maxInputImagesByModel: { "acme/reference-to-video": 9 },
maxDurationSeconds: 5,
},
videoToVideo: { enabled: false },
},
generateVideo: async (req) => ({ videos: [] }),

29
docs/providers/fal.md
View File

@@ -79,10 +79,10 @@ To use fal as the default image provider:
The bundled `fal` video-generation provider defaults to
`fal/fal-ai/minimax/video-01-live`.
| Capability | Value |
| ---------- | ------------------------------------------------------------ |
| Modes | Text-to-video, single-image reference |
| Runtime | Queue-backed submit/status/result flow for long-running jobs |
| Capability | Value |
| ---------- | ------------------------------------------------------------------ |
| Modes | Text-to-video, single-image reference, Seedance reference-to-video |
| Runtime | Queue-backed submit/status/result flow for long-running jobs |
<AccordionGroup>
<Accordion title="Available video models">
@@ -94,8 +94,10 @@ The bundled `fal` video-generation provider defaults to
- `fal/bytedance/seedance-2.0/fast/text-to-video`
- `fal/bytedance/seedance-2.0/fast/image-to-video`
- `fal/bytedance/seedance-2.0/fast/reference-to-video`
- `fal/bytedance/seedance-2.0/text-to-video`
- `fal/bytedance/seedance-2.0/image-to-video`
- `fal/bytedance/seedance-2.0/reference-to-video`
</Accordion>
@@ -113,6 +115,25 @@ The bundled `fal` video-generation provider defaults to
```
</Accordion>
<Accordion title="Seedance 2.0 reference-to-video config example">
```json5
{
agents: {
defaults: {
videoGenerationModel: {
primary: "fal/bytedance/seedance-2.0/fast/reference-to-video",
},
},
},
}
```
Reference-to-video accepts up to 9 images, 3 videos, and 3 audio references
through the shared `video_generate` `images`, `videos`, and `audioRefs`
parameters, with at most 12 total reference files.
</Accordion>
<Accordion title="HeyGen video-agent config example">
```json5
{

41
docs/tools/video-generation.md
View File

@@ -82,22 +82,22 @@ Duplicate prevention: if a video task is already `queued` or `running` for the c
## Supported providers
| Provider | Default model | Text | Image ref | Video ref | API key |
| --------------------- | ------------------------------- | ---- | ---------------------------------------------------- | ---------------- | ---------------------------------------- |
| Alibaba | `wan2.6-t2v` | Yes | Yes (remote URL) | Yes (remote URL) | `MODELSTUDIO_API_KEY` |
| BytePlus (1.0) | `seedance-1-0-pro-250528` | Yes | Up to 2 images (I2V models only; first + last frame) | No | `BYTEPLUS_API_KEY` |
| BytePlus Seedance 1.5 | `seedance-1-5-pro-251215` | Yes | Up to 2 images (first + last frame via role) | No | `BYTEPLUS_API_KEY` |
| BytePlus Seedance 2.0 | `dreamina-seedance-2-0-260128` | Yes | Up to 9 reference images | Up to 3 videos | `BYTEPLUS_API_KEY` |
| ComfyUI | `workflow` | Yes | 1 image | No | `COMFY_API_KEY` or `COMFY_CLOUD_API_KEY` |
| fal | `fal-ai/minimax/video-01-live` | Yes | 1 image | No | `FAL_KEY` |
| Google | `veo-3.1-fast-generate-preview` | Yes | 1 image | 1 video | `GEMINI_API_KEY` |
| MiniMax | `MiniMax-Hailuo-2.3` | Yes | 1 image | No | `MINIMAX_API_KEY` or MiniMax OAuth |
| OpenAI | `sora-2` | Yes | 1 image | 1 video | `OPENAI_API_KEY` |
| Qwen | `wan2.6-t2v` | Yes | Yes (remote URL) | Yes (remote URL) | `QWEN_API_KEY` |
| Runway | `gen4.5` | Yes | 1 image | 1 video | `RUNWAYML_API_SECRET` |
| Together | `Wan-AI/Wan2.2-T2V-A14B` | Yes | 1 image | No | `TOGETHER_API_KEY` |
| Vydra | `veo3` | Yes | 1 image (`kling`) | No | `VYDRA_API_KEY` |
| xAI | `grok-imagine-video` | Yes | 1 first-frame image or up to 7 `reference_image`s | 1 video | `XAI_API_KEY` |
| Provider | Default model | Text | Image ref | Video ref | API key |
| --------------------- | ------------------------------- | ---- | ---------------------------------------------------- | ----------------------------------------------- | ---------------------------------------- |
| Alibaba | `wan2.6-t2v` | Yes | Yes (remote URL) | Yes (remote URL) | `MODELSTUDIO_API_KEY` |
| BytePlus (1.0) | `seedance-1-0-pro-250528` | Yes | Up to 2 images (I2V models only; first + last frame) | No | `BYTEPLUS_API_KEY` |
| BytePlus Seedance 1.5 | `seedance-1-5-pro-251215` | Yes | Up to 2 images (first + last frame via role) | No | `BYTEPLUS_API_KEY` |
| BytePlus Seedance 2.0 | `dreamina-seedance-2-0-260128` | Yes | Up to 9 reference images | Up to 3 videos | `BYTEPLUS_API_KEY` |
| ComfyUI | `workflow` | Yes | 1 image | No | `COMFY_API_KEY` or `COMFY_CLOUD_API_KEY` |
| fal | `fal-ai/minimax/video-01-live` | Yes | 1 image; up to 9 with Seedance reference-to-video | Up to 3 videos with Seedance reference-to-video | `FAL_KEY` |
| Google | `veo-3.1-fast-generate-preview` | Yes | 1 image | 1 video | `GEMINI_API_KEY` |
| MiniMax | `MiniMax-Hailuo-2.3` | Yes | 1 image | No | `MINIMAX_API_KEY` or MiniMax OAuth |
| OpenAI | `sora-2` | Yes | 1 image | 1 video | `OPENAI_API_KEY` |
| Qwen | `wan2.6-t2v` | Yes | Yes (remote URL) | Yes (remote URL) | `QWEN_API_KEY` |
| Runway | `gen4.5` | Yes | 1 image | 1 video | `RUNWAYML_API_SECRET` |
| Together | `Wan-AI/Wan2.2-T2V-A14B` | Yes | 1 image | No | `TOGETHER_API_KEY` |
| Vydra | `veo3` | Yes | 1 image (`kling`) | No | `VYDRA_API_KEY` |
| xAI | `grok-imagine-video` | Yes | 1 first-frame image or up to 7 `reference_image`s | 1 video | `XAI_API_KEY` |
Some providers accept additional or alternate API key env vars. See individual [provider pages](#related) for details.
@@ -114,7 +114,7 @@ and the shared live sweep.
| Alibaba | Yes | Yes | Yes | `generate`, `imageToVideo`; `videoToVideo` skipped because this provider needs remote `http(s)` video URLs |
| BytePlus | Yes | Yes | No | `generate`, `imageToVideo` |
| ComfyUI | Yes | Yes | No | Not in the shared sweep; workflow-specific coverage lives with Comfy tests |
| fal | Yes | Yes | No | `generate`, `imageToVideo` |
| fal | Yes | Yes | Yes | `generate`, `imageToVideo`; `videoToVideo` only when using Seedance reference-to-video |
| Google | Yes | Yes | Yes | `generate`, `imageToVideo`; shared `videoToVideo` skipped because the current buffer-backed Gemini/Veo sweep does not accept that input |
| MiniMax | Yes | Yes | No | `generate`, `imageToVideo` |
| OpenAI | Yes | Yes | Yes | `generate`, `imageToVideo`; shared `videoToVideo` skipped because this org/input path currently needs provider-side inpaint/remix access |
@@ -296,7 +296,7 @@ entries.
</Accordion>
<Accordion title="fal">
Uses a queue-backed flow for long-running jobs. Single image reference only.
Uses a queue-backed flow for long-running jobs. Most fal video models accept a single image reference. Seedance 2.0 reference-to-video models accept up to 9 images, 3 videos, and 3 audio references, with at most 12 total reference files.
</Accordion>
<Accordion title="Google (Gemini / Veo)">
@@ -349,6 +349,7 @@ capabilities: {
enabled: true,
maxVideos: 1,
maxInputImages: 1,
maxInputImagesByModel: { "provider/reference-to-video": 9 },
maxDurationSeconds: 5,
},
videoToVideo: {
@@ -366,6 +367,10 @@ enough to advertise transform-mode support. Providers should declare
contract tests, and the shared `video_generate` tool can validate mode support
deterministically.
When one model in a provider has wider reference-input support than the rest,
use `maxInputImagesByModel`, `maxInputVideosByModel`, or
`maxInputAudiosByModel` instead of raising the mode-wide limit.
## Live tests
Opt-in live coverage for the shared bundled providers: