vultr/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-07-16 18:01:41 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(xai): support Grok Imagine Video 1.5 (#103316)

Browse Source 
* feat(media): add per-model generation catalog metadata

* feat(xai): support Grok Imagine Video 1.5

* test(plugin-sdk): update public surface budget
This commit is contained in:
Peter Steinberger
2026-07-10 04:49:42 +01:00
committed by GitHub
parent 071e30ecd5
commit 75dfd3dc09
16 changed files with 569 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/help/testing-live.md
View File

@@ -582,6 +582,9 @@ request. Plugin dependencies are expected to be present before runtime load.
- Provider-specific Vydra coverage:
- `OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_VYDRA_VIDEO=1 pnpm test:live -- extensions/vydra/vydra.live.test.ts`
- That file runs `veo3` text-to-video plus a `kling` image-to-video lane that uses a remote image URL fixture by default (`OPENCLAW_LIVE_VYDRA_KLING_IMAGE_URL` to override).
- Provider-specific xAI Video 1.5 coverage:
- `OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_XAI_VIDEO_15=1 pnpm test:live -- extensions/xai/xai.live.test.ts -t "Grok Imagine Video 1.5"`
- The case generates a local PNG first frame, requests a one-second 1080P image-to-video clip, polls to completion, and verifies the downloaded video buffer.
- Current `videoToVideo` live coverage:
- `runway` only when the selected model resolves to `gen4_aleph`
- Current declared-but-skipped `videoToVideo` providers in the shared sweep:

23
docs/plugins/sdk-provider-plugins.md
View File

@@ -934,6 +934,7 @@ catalog, API-key auth, and dynamic model resolution.
id: "acme-ai",
label: "Acme Video",
defaultTimeoutMs: 600_000,
models: ["acme-video", "acme-image-video"],
capabilities: {
generate: { maxVideos: 1, maxDurationSeconds: 10, supportsResolution: true },
imageToVideo: {
@@ -945,6 +946,21 @@ catalog, API-key auth, and dynamic model resolution.
},
videoToVideo: { enabled: false },
},
catalogByModel: {
"acme-image-video": {
modes: ["imageToVideo"],
capabilities: {
imageToVideo: {
enabled: true,
maxVideos: 1,
maxInputImages: 1,
resolutions: ["480P", "720P", "1080P"],
supportsResolution: true,
},
videoToVideo: { enabled: false },
},
},
},
generateVideo: async (req) => ({ videos: [] }),
});
```
@@ -952,6 +968,13 @@ catalog, API-key auth, and dynamic model resolution.
`capabilities` is required on both provider types; `edit` and the
video transform blocks (`imageToVideo`, `videoToVideo`) always need an
explicit `enabled` flag.
Use `catalogByModel` when a listed model's static modes or capabilities
differ from the provider defaults. This metadata keeps
`video_generate action=list` and model catalogs accurate without
invoking provider code. Request-time capability lookup and enforcement
still belong in `resolveModelCapabilities` and `generateVideo`; reuse
the same capability constant for both paths when possible.
</Tab>
<Tab title="Web fetch and search">
```typescript

30
docs/providers/xai.md
View File

@@ -134,7 +134,7 @@ below or under known limits.
| Server-side X search | `x_search` tool | Yes |
| Server-side code execution | `code_execution` tool | Yes |
| Images | `image_generate` | Yes |
| Videos | `video_generate` | Classic model; Video 1.5 is not exposed yet |
| Videos | `video_generate` | Classic full workflow; Video 1.5 image-to-video |
| Batch text-to-speech | `messages.tts.provider: "xai"` / `tts` | Yes |
| Streaming TTS | - | Not implemented by the xAI provider yet |
| Batch speech-to-text | `tools.media.audio` media understanding | Yes |
@@ -210,13 +210,17 @@ stale context metadata on active 4.20 rows. It does not pin active 4.20
The bundled `xai` plugin registers video generation through the shared
`video_generate` tool.
- Default video model: `xai/grok-imagine-video`
- Modes: text-to-video, image-to-video, reference-image generation, remote
video edit, and remote video extension
- Aspect ratios: `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, `2:3`
- Resolutions: `480P`, `720P`
- Default model: `xai/grok-imagine-video`
- Additional model: `xai/grok-imagine-video-1.5`
- Classic modes: text-to-video, image-to-video, reference-image generation,
remote video edit, and remote video extension
- Video 1.5 mode: image-to-video only, with exactly one first-frame image
- Aspect ratios: `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, `2:3`;
Video 1.5 inherits the source image ratio when omitted
- Resolutions: classic `480P`/`720P`; Video 1.5 supports `480P`, `720P`,
and `1080P`, and defaults to `480P`
- Duration: 1-15 seconds for generation/image-to-video, 1-10 seconds when
using `reference_image` roles, 2-10 seconds for extension
using classic `reference_image` roles, 2-10 seconds for classic extension
- Reference-image generation: set `imageRoles` to `reference_image` for
every supplied image; xAI accepts up to 7 such images
- Default operation timeout: 600 seconds unless `video_generate.timeoutMs`
@@ -228,6 +232,10 @@ stale context metadata on active 4.20 rows. It does not pin active 4.20
OpenClaw encodes those as data URLs for xAI.
</Warning>
Video 1.5 also recognizes xAI's `grok-imagine-video-1.5-preview` and
`grok-imagine-video-1.5-2026-05-30` identifiers. OpenClaw forwards the
selected identifier unchanged, but applies the same image-only validation.
To use xAI as the default video provider:
```json5
@@ -502,9 +510,6 @@ stale context metadata on active 4.20 rows. It does not pin active 4.20
- xAI Realtime voice is not registered as an OpenClaw provider yet. It
needs a different bidirectional voice session contract than batch STT
or streaming transcription.
- `grok-imagine-video-1.5` is not exposed yet. Unlike the classic video
model, it is image-to-video only and needs model-specific mode and 1080p
validation in the shared provider contract.
- xAI image `quality`, image `mask`, and extra native-only aspect ratios
are not exposed until the shared `image_generate` tool has
corresponding cross-provider controls.
@@ -546,6 +551,7 @@ The xAI media paths are covered by unit tests and opt-in live suites. Export
```bash
pnpm test extensions/xai
OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_TEST_QUIET=1 pnpm test:live -- extensions/xai/xai.live.test.ts
OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_XAI_VIDEO_15=1 pnpm test:live -- extensions/xai/xai.live.test.ts -t "Grok Imagine Video 1.5"
OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_TEST_QUIET=1 pnpm test:live -- extensions/xai/x-search.live.test.ts
OPENCLAW_LIVE_GATEWAY_MODELS="xai/grok-4.5,xai/grok-build-0.1,xai/grok-4.3,xai/grok-4.20-0309-reasoning,xai/grok-4.20-0309-non-reasoning" OPENCLAW_LIVE_GATEWAY_MAX_MODELS=0 OPENCLAW_LIVE_GATEWAY_SMOKE=0 pnpm test:live -- src/gateway/gateway-models.profiles.live.test.ts
OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_TEST_QUIET=1 OPENCLAW_LIVE_IMAGE_GENERATION_PROVIDERS=xai pnpm test:live -- test/image-generation.runtime.live.test.ts
@@ -555,7 +561,9 @@ The provider-specific live file synthesizes normal TTS, telephony-friendly PCM
TTS, transcribes audio through xAI batch STT, streams the same PCM through xAI
realtime STT, generates text-to-image output, and edits a reference image.
The shared image live file verifies the same xAI provider through OpenClaw's
runtime selection, fallback, normalization, and media attachment path.
runtime selection, fallback, normalization, and media attachment path. The
opt-in Video 1.5 case submits one generated first-frame image at 1080P and
verifies the completed video download.
## Related

17
docs/tools/video-generation.md
View File

@@ -119,7 +119,7 @@ openclaw tasks cancel <lookup>
| Runway | `gen4.5` | ✓ | 1 image | 1 video | `RUNWAYML_API_SECRET` |
| Together | `Wan-AI/Wan2.2-T2V-A14B` | ✓ | `Wan-AI/Wan2.2-I2V-A14B` only | - | `TOGETHER_API_KEY` |
| Vydra | `veo3` | ✓ | 1 image (`kling`) | - | `VYDRA_API_KEY` |
| xAI | `grok-imagine-video` | ✓ | 1 first-frame image or up to 7 `reference_image`s | 1 video | `XAI_API_KEY` |
| xAI | `grok-imagine-video` | ✓ | Classic: 1 first frame or 7 references; 1.5: 1 frame | Classic: 1 video | `XAI_API_KEY` |
Some providers accept additional or alternate API key env vars. See
individual [provider pages](#related) for details.
@@ -147,7 +147,7 @@ the shared live sweep:
| Runway | ✓ | ✓ | ✓ | `generate`, `imageToVideo`; `videoToVideo` runs only when the selected model is `runway/gen4_aleph` |
| Together | ✓ | ✓ | - | `generate`, `imageToVideo` |
| Vydra | ✓ | ✓ | - | `generate`; shared `imageToVideo` skipped because bundled `veo3` is text-only and bundled `kling` requires a remote image URL |
| xAI | ✓ | ✓ | ✓ | `generate`, `imageToVideo`; `videoToVideo` skipped because this provider currently needs a remote MP4 URL |
| xAI | ✓ | ✓ | ✓ | Classic supports all modes; Video 1.5 is image-to-video only; remote MP4 input keeps `videoToVideo` out of the shared sweep |
## Tool parameters
@@ -423,9 +423,16 @@ only the explicit `model`, `primary`, and `fallbacks` entries.
a remote image URL.
</Accordion>
<Accordion title="xAI">
Supports text-to-video, single first-frame image-to-video, up to 7
`reference_image` inputs through xAI `reference_images`, and remote
video edit/extend flows.
The default `grok-imagine-video` model supports text-to-video, single
first-frame image-to-video, up to 7 `reference_image` inputs through xAI
`reference_images`, and remote video edit/extend flows.
`grok-imagine-video-1.5` is image-to-video only: provide exactly one image.
It supports 1-15 seconds and `480P`, `720P`, or `1080P`, defaulting to
`480P`; omit `aspectRatio` to inherit the source image ratio. The preview
and dated 1.5 identifiers receive the same validation and are forwarded
unchanged.
</Accordion>
</AccordionGroup>