vultr/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-04-21 14:11:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(ui): render assistant directives and add embed tag (#64104)

Browse Source 
* Add embed rendering for Control UI assistant output

* Add changelog entry for embed rendering

* Harden canvas path resolution and stage isolation

* Secure assistant media route and preserve UI avatar override

* Fix chat media and history regressions

* Harden embed iframe URL handling

* Fix embed follow-up review regressions

* Restore offloaded chat attachment persistence

* Harden hook and media routing

* Fix embed review follow-ups

* feat(ui): add configurable embed sandbox mode

* fix(gateway): harden assistant media and auth rotation

* fix(gateway): restore websocket pairing handshake flows

* fix(gateway): restore ws hello policy details

* Restore dropped control UI shell wiring

* Fix control UI reconnect cleanup regressions

* fix(gateway): restore media root and auth getter compatibility

* feat(ui): rename public canvas tag to embed

* fix(ui): address remaining media and gateway review issues

* fix(ui): address remaining embed and attachment review findings

* fix(ui): restore stop control and tool card inputs

* fix(ui): address history and attachment review findings

* fix(ui): restore prompt contribution wiring

* fix(ui): address latest history and directive reviews

* fix(ui): forward password auth for assistant media

* fix(ui): suppress silent transcript tokens with media

* feat(ui): add granular embed sandbox modes

* fix(ui): preserve relative media directives in history

* docs(ui): document embed sandbox modes

* fix(gateway): restrict canvas history hoisting to tool entries

* fix(gateway): tighten embed follow-up review fixes

* fix(ci): repair merged branch type drift

* fix(prompt): restore stable runtime prompt rendering

* fix(ui): harden local attachment preview checks

* fix(prompt): restore channel-aware approval guidance

* fix(gateway): enforce auth rotation and media cleanup

* feat(ui): gate external embed urls behind config

* fix(ci): repair rebased branch drift

* fix(ci): resolve remaining branch check failures
This commit is contained in:
Tak Hoffman
2026-04-11 07:32:53 -05:00
committed by GitHub
parent d83a85c70d
commit cc5c691f00
75 changed files with 8163 additions and 1832 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
CHANGELOG.md
View File

@@ -37,11 +37,13 @@ Docs: https://docs.openclaw.ai
- Gateway: add a `commands.list` RPC so remote gateway clients can discover runtime-native, text, skill, and plugin commands with surface-aware naming and serialized argument metadata. (#62656) Thanks @samzong.
- Models/providers: add per-provider `models.providers.*.request.allowPrivateNetwork` for trusted self-hosted OpenAI-compatible endpoints, keep the opt-in scoped to model request surfaces, and refresh cached WebSocket managers when request transport overrides change. (#63671) Thanks @qas.
- Feishu: standardize request user agents and register the bot as an AI agent so Feishu deployments identify OpenClaw consistently. (#63835) Thanks @evandance.
- Docs i18n: chunk raw doc translation, reject truncated tagged outputs, avoid ambiguous body-only wrapper unwrapping, and recover from terminated Pi translation sessions without changing the default `openai/gpt-5.4` path. (#62969, #63808) Thanks @hxy91819.
- Gateway: split startup and runtime seams so gateway lifecycle sequencing, reload state, and shutdown behavior stay easier to maintain without changing observed behavior. (#63975) Thanks @gumadeiras.
- Control UI/webchat: normalize assistant `MEDIA:`/reply/voice directives into structured bubble rendering, rename the unreleased rich web shortcode to `[embed ...]`, and surface session runtime roots so hosted web content is written to the correct document path instead of guessed local files.
- Matrix/partial streaming: add MSC4357 live markers to draft preview sends and edits so supporting Matrix clients can render a live/typewriter animation and stop it when the final edit lands. (#63513) Thanks @TigerInYourDream.
- Control UI/dreaming: simplify the Scene and Diary surfaces, preserve unknown phase state for partial status payloads, and stabilize waiting-entry recency ordering so Dreaming status and review lists stay clear and deterministic. (#64035) Thanks @davemorin.
- Agents: add an opt-in strict-agentic embedded Pi execution contract for GPT-5-family runs so plan-only or filler turns keep acting until they hit a real blocker. (#64241) Thanks @100yenadmin.
- Agents/OpenAI: add provider-owned OpenAI/Codex tool schema compatibility and surface embedded-run replay/liveness state for long-running runs. (#64300) Thanks @100yenadmin.
- Docs i18n: chunk raw doc translation, reject truncated tagged outputs, avoid ambiguous body-only wrapper unwrapping, and recover from terminated Pi translation sessions without changing the default `openai/gpt-5.4` path. (#62969, #63808) Thanks @hxy91819.
- Dreaming/memory-wiki: add ChatGPT import ingestion plus new `Imported Insights` and `Memory Palace` diary subtabs so Dreaming can inspect imported source chats, compiled wiki pages, and full source pages directly from the UI. (#64505)
### Fixes

2
docs/gateway/configuration-reference.md
View File

@@ -2895,6 +2895,8 @@ See [Plugins](/tools/plugin).
enabled: true,
basePath: "/openclaw",
// root: "dist/control-ui",
// embedSandbox: "scripts", // strict | scripts | trusted
// allowExternalEmbedUrls: false, // dangerous: allow absolute external http(s) embed URLs
// allowedOrigins: ["https://control.example.com"], // required for non-loopback Control UI
// dangerouslyAllowHostHeaderOriginFallback: false, // dangerous Host-header origin fallback mode
// allowInsecureAuth: false,

50
docs/reference/rich-output-protocol.md Normal file
View File

@@ -0,0 +1,50 @@
# Rich Output Protocol
Assistant output can carry a small set of delivery/render directives:
- `MEDIA:` for attachment delivery
- `[[audio_as_voice]]` for audio presentation hints
- `[[reply_to_current]]` / `[[reply_to:<id>]]` for reply metadata
- `[embed ...]` for Control UI rich rendering
These directives are separate. `MEDIA:` and reply/voice tags remain delivery metadata; `[embed ...]` is the web-only rich render path.
## `[embed ...]`
`[embed ...]` is the only agent-facing rich render syntax for the Control UI.
Self-closing example:
```text
[embed ref="cv_123" title="Status" /]
```
Rules:
- `[view ...]` is no longer valid for new output.
- Embed shortcodes render in the assistant message surface only.
- Only URL-backed embeds are rendered. Use `ref="..."` or `url="..."`.
- Block-form inline HTML embed shortcodes are not rendered.
- The web UI strips the shortcode from visible text and renders the embed inline.
- `MEDIA:` is not an embed alias and should not be used for rich embed rendering.
## Stored Rendering Shape
The normalized/stored assistant content block is a structured `canvas` item:
```json
{
"type": "canvas",
"preview": {
"kind": "canvas",
"surface": "assistant_message",
"render": "url",
"viewId": "cv_123",
"url": "/__openclaw__/canvas/documents/cv_123/index.html",
"title": "Status",
"preferredHeight": 320
}
}
```
Stored/rendered rich blocks use this `canvas` shape directly. `present_view` is not recognized.

32
docs/web/control-ui.md
View File

@@ -138,6 +138,38 @@ Cron jobs panel notes:
- Gateway persists aborted partial assistant text into transcript history when buffered output exists
- Persisted entries include abort metadata so transcript consumers can tell abort partials from normal completion output
## Hosted embeds
Assistant messages can render hosted web content inline with the `[embed ...]`
shortcode. The iframe sandbox policy is controlled by
`gateway.controlUi.embedSandbox`:
- `strict`: disables script execution inside hosted embeds
- `scripts`: allows interactive embeds while keeping origin isolation; this is
the default and is usually enough for self-contained browser games/widgets
- `trusted`: adds `allow-same-origin` on top of `allow-scripts` for same-site
documents that intentionally need stronger privileges
Example:
```json5
{
gateway: {
controlUi: {
embedSandbox: "scripts",
},
},
}
```
Use `trusted` only when the embedded document genuinely needs same-origin
behavior. For most agent-generated games and interactive canvases, `scripts` is
the safer choice.
Absolute external `http(s)` embed URLs stay blocked by default. If you
intentionally want `[embed url="https://..."]` to load third-party pages, set
`gateway.controlUi.allowExternalEmbedUrls: true`.
## Tailnet access (recommended)
### Integrated Tailscale Serve (preferred)

1
src/agents/pi-embedded-runner/system-prompt.ts
View File

@@ -43,6 +43,7 @@ export function buildEmbeddedSystemPrompt(params: {
channel?: string;
/** Supported message actions for the current channel (e.g., react, edit, unsend) */
channelActions?: string[];
canvasRootDir?: string;
};
messageToolHints?: string[];
sandboxInfo?: EmbeddedSandboxInfo;

9
src/agents/system-prompt-params.test.ts
View File

@@ -3,6 +3,7 @@ import os from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import type { OpenClawConfig } from "../config/config.js";
import { resolveStateDir } from "../config/paths.js";
import { buildSystemPromptParams } from "./system-prompt-params.js";
async function makeTempDir(label: string): Promise<string> {
@@ -101,4 +102,12 @@ describe("buildSystemPromptParams repo root", () => {
expect(runtimeInfo.repoRoot).toBeUndefined();
});
it("includes the default profile canvas root in runtimeInfo", async () => {
const workspaceDir = await makeTempDir("canvas-root");
const { runtimeInfo } = buildParams({ workspaceDir });
expect(runtimeInfo.canvasRootDir).toBe(path.resolve(path.join(resolveStateDir(), "canvas")));
});
});

21
src/agents/system-prompt-params.ts
View File

@@ -1,7 +1,9 @@
import fs from "node:fs";
import path from "node:path";
import { resolveStateDir } from "../config/paths.js";
import type { OpenClawConfig } from "../config/types.openclaw.js";
import { findGitRoot } from "../infra/git-root.js";
import { resolveHomeRelativePath } from "../infra/home-dir.js";
import {
formatUserTime,
resolveUserTimeFormat,
@@ -23,6 +25,7 @@ export type RuntimeInfoInput = {
/** Supported message actions for the current channel (e.g., react, edit, unsend) */
channelActions?: string[];
repoRoot?: string;
canvasRootDir?: string;
};
export type SystemPromptRuntimeParams = {
@@ -47,11 +50,17 @@ export function buildSystemPromptParams(params: {
const userTimezone = resolveUserTimezone(params.config?.agents?.defaults?.userTimezone);
const userTimeFormat = resolveUserTimeFormat(params.config?.agents?.defaults?.timeFormat);
const userTime = formatUserTime(new Date(), userTimezone, userTimeFormat);
const stateDir = resolveStateDir(process.env);
const canvasRootDir = resolveCanvasRootDir({
config: params.config,
stateDir,
});
return {
runtimeInfo: {
agentId: params.agentId,
...params.runtime,
repoRoot,
canvasRootDir,
},
userTimezone,
userTime,
@@ -59,6 +68,18 @@ export function buildSystemPromptParams(params: {
};
}
function resolveCanvasRootDir(params: { config?: OpenClawConfig; stateDir: string }): string {
const configured = params.config?.canvasHost?.root?.trim();
if (configured) {
return path.resolve(
resolveHomeRelativePath(configured, {
env: process.env,
}),
);
}
return path.resolve(path.join(params.stateDir, "canvas"));
}
function resolveRepoRoot(params: {
config?: OpenClawConfig;
workspaceDir?: string;

416
src/agents/system-prompt.test.ts
View File

@@ -2,7 +2,6 @@ import { describe, expect, it } from "vitest";
import { SILENT_REPLY_TOKEN } from "../auto-reply/tokens.js";
import { typedCases } from "../test-utils/typed-cases.js";
import { buildSubagentSystemPrompt } from "./subagent-system-prompt.js";
import { SYSTEM_PROMPT_CACHE_BOUNDARY } from "./system-prompt-cache-boundary.js";
import { buildAgentSystemPrompt, buildRuntimeLine } from "./system-prompt.js";
describe("buildAgentSystemPrompt", () => {
@@ -102,7 +101,7 @@ describe("buildAgentSystemPrompt", () => {
skillsPrompt:
"<available_skills>\n <skill>\n <name>demo</name>\n </skill>\n</available_skills>",
heartbeatPrompt: "ping",
toolNames: ["message", "memory_search", "cron"],
toolNames: ["message", "memory_search"],
docsPath: "/tmp/openclaw/docs",
extraSystemPrompt: "Subagent details",
ttsHint: "Voice (TTS) is enabled.",
@@ -120,16 +119,7 @@ describe("buildAgentSystemPrompt", () => {
expect(prompt).not.toContain("## Heartbeats");
expect(prompt).toContain("## Safety");
expect(prompt).toContain(
'For follow-up at a future time (for example "check back in 10 minutes", reminders, run-later work, or recurring tasks), use cron instead of exec sleep, yieldMs delays, or process polling.',
);
expect(prompt).toContain(
"Use exec/process only for commands that start now and continue running in the background.",
);
expect(prompt).toContain(
"For long-running work that starts now, start it once and rely on automatic completion wake when it is enabled and the command emits output or fails; otherwise use process to confirm completion, and use it for logs, status, input, or intervention.",
);
expect(prompt).toContain(
"Do not emulate scheduling with sleep loops, timeout loops, or repeated polling.",
"For long waits, avoid rapid poll loops: use exec with enough yieldMs or process(action=poll, timeout=<ms>).",
);
expect(prompt).toContain("You have no independent goals");
expect(prompt).toContain("Prioritize safety and human oversight");
@@ -159,89 +149,6 @@ describe("buildAgentSystemPrompt", () => {
);
});
it("tells the agent not to execute /approve through exec", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
});
expect(prompt).toContain(
"Never execute /approve through exec or any other shell/tool path; /approve is a user-facing approval command, not a shell command.",
);
});
it("adds stronger execution-bias guidance for actionable turns", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
});
expect(prompt).toContain("## Execution Bias");
expect(prompt).toContain(
"If the user asks you to do the work, start doing it in the same turn.",
);
expect(prompt).toContain(
"Commentary-only turns are incomplete when tools are available and the next action is clear.",
);
});
it("narrows silent reply guidance to true no-delivery cases", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
});
expect(prompt).toContain(
`Use ${SILENT_REPLY_TOKEN} ONLY when no user-visible reply is required.`,
);
expect(prompt).toContain(
"Never use it to avoid doing requested work or to end an actionable turn early.",
);
});
it("keeps manual /approve instructions for non-native approval channels", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
runtimeInfo: { channel: "signal" },
});
expect(prompt).toContain(
"When exec returns approval-pending, include the concrete /approve command from tool output",
);
expect(prompt).not.toContain("allow-once|allow-always|deny");
});
it("tells native approval channels not to duplicate plain chat /approve instructions", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
runtimeInfo: { channel: "telegram", capabilities: ["inlineButtons"] },
});
expect(prompt).toContain(
"When exec returns approval-pending on this channel, rely on native approval card/buttons when they appear and do not also send plain chat /approve instructions. Only include the concrete /approve command if the tool result says chat approvals are unavailable or only manual approval is possible.",
);
expect(prompt).toContain(
"Only include the concrete /approve command if the tool result says chat approvals are unavailable or only manual approval is possible.",
);
expect(prompt).not.toContain(
"When exec returns approval-pending, include the concrete /approve command from tool output",
);
});
it("treats webchat as a native approval surface", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
runtimeInfo: { channel: "webchat" },
});
expect(prompt).toContain(
"When exec returns approval-pending on this channel, rely on native approval card/buttons when they appear",
);
expect(prompt).toContain(
"Only include the concrete /approve command if the tool result says chat approvals are unavailable or only manual approval is possible.",
);
expect(prompt).not.toContain(
"When exec returns approval-pending, include the concrete /approve command from tool output",
);
});
it("omits skills in minimal prompt mode when skillsPrompt is absent", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
@@ -256,10 +163,10 @@ describe("buildAgentSystemPrompt", () => {
workspaceDir: "/tmp/openclaw",
});
expect(prompt).toContain("## Reply Tags");
expect(prompt).toContain("## Assistant Output Directives");
expect(prompt).toContain("[[reply_to_current]]");
expect(prompt).not.toContain("Tags are stripped before sending");
expect(prompt).toContain("Tags are removed before sending");
expect(prompt).toContain("Supported tags are stripped before user-visible rendering");
});
it("omits the heartbeat section when no heartbeat prompt is provided", () => {
@@ -329,19 +236,46 @@ describe("buildAgentSystemPrompt", () => {
expect(prompt).toContain("do not forward raw internal metadata");
});
it("does not include embed guidance in the default global prompt", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
});
expect(prompt).not.toContain("## Control UI Embed");
expect(prompt).not.toContain("Use `[embed ...]` only in Control UI/webchat sessions");
});
it("includes embed guidance only for webchat sessions", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
runtimeInfo: {
channel: "webchat",
canvasRootDir: "/Users/example/.openclaw-dev/canvas",
},
});
expect(prompt).toContain("## Control UI Embed");
expect(prompt).toContain("Use `[embed ...]` only in Control UI/webchat sessions");
expect(prompt).toContain('[embed ref="cv_123" title="Status" height="320" /]');
expect(prompt).toContain(
'[embed url="/__openclaw__/canvas/documents/cv_123/index.html" title="Status" height="320" /]',
);
expect(prompt).toContain(
"Never use local filesystem paths or `file://...` URLs in `[embed ...]`.",
);
expect(prompt).toContain(
"The active hosted embed root for this session is: `/Users/example/.openclaw-dev/canvas`.",
);
expect(prompt).not.toContain('[embed content_type="html" title="Status"]...[/embed]');
});
it("guides subagent workflows to avoid polling loops", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
});
expect(prompt).toContain(
'For follow-up at a future time (for example "check back in 10 minutes", reminders, run-later work, or recurring tasks), use cron instead of exec sleep, yieldMs delays, or process polling.',
);
expect(prompt).toContain(
"Use exec/process only for commands that start now and continue running in the background.",
);
expect(prompt).toContain(
"For long-running work that starts now, start it once and rely on automatic completion wake when it is enabled and the command emits output or fails; otherwise use process to confirm completion, and use it for logs, status, input, or intervention.",
"For long waits, avoid rapid poll loops: use exec with enough yieldMs or process(action=poll, timeout=<ms>).",
);
expect(prompt).toContain("Completion is push-based: it will auto-announce when done.");
expect(prompt).toContain("Do not poll `subagents list` / `sessions_list` in a loop");
@@ -350,25 +284,16 @@ describe("buildAgentSystemPrompt", () => {
);
});
it("uses structured tool definitions as the source of truth", () => {
it("lists available tools when provided", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
toolNames: ["exec", "sessions_list", "sessions_history", "sessions_send"],
});
expect(prompt).toContain(
"Structured tool definitions are the source of truth for tool names, descriptions, and parameters.",
);
expect(prompt).toContain(
"Tool names are case-sensitive. Call tools exactly as listed in the structured tool definitions.",
);
expect(prompt).toContain(
"TOOLS.md does not control tool availability; it is user guidance for how to use external tools.",
);
expect(prompt).not.toContain("Tool availability (filtered by policy):");
expect(prompt).not.toContain("- sessions_list:");
expect(prompt).not.toContain("- sessions_history:");
expect(prompt).not.toContain("- sessions_send:");
expect(prompt).toContain("Tool availability (filtered by policy):");
expect(prompt).toContain("sessions_list");
expect(prompt).toContain("sessions_history");
expect(prompt).toContain("sessions_send");
});
it("documents ACP sessions_spawn agent targeting requirements", () => {
@@ -378,8 +303,10 @@ describe("buildAgentSystemPrompt", () => {
});
expect(prompt).toContain("sessions_spawn");
expect(prompt).toContain("Set `agentId` explicitly unless `acp.defaultAgent` is configured");
expect(prompt).toContain("`subagents`/`agents_list`");
expect(prompt).toContain(
'runtime="acp" requires `agentId` unless `acp.defaultAgent` is configured',
);
expect(prompt).toContain("not agents_list");
});
it("guides harness requests to ACP thread-bound spawns", () => {
@@ -414,9 +341,8 @@ describe("buildAgentSystemPrompt", () => {
);
expect(prompt).not.toContain('runtime="acp" requires `agentId`');
expect(prompt).not.toContain("not ACP harness ids");
expect(prompt).toContain(
"If a task is more complex or takes longer, spawn a sub-agent. Completion is push-based: it will auto-announce when done.",
);
expect(prompt).toContain("- sessions_spawn: Spawn an isolated sub-agent session");
expect(prompt).toContain("- agents_list: List OpenClaw agent ids allowed for sessions_spawn");
});
it("omits ACP harness spawn guidance for sandboxed sessions and shows ACP block note", () => {
@@ -450,12 +376,8 @@ describe("buildAgentSystemPrompt", () => {
docsPath: "/tmp/openclaw/docs",
});
expect(prompt).toContain(
"Tool names are case-sensitive. Call tools exactly as listed in the structured tool definitions.",
);
expect(prompt).toContain(
"For long waits, avoid rapid poll loops: use Exec with enough yieldMs or process(action=poll, timeout=<ms>).",
);
expect(prompt).toContain("- Read: Read file contents");
expect(prompt).toContain("- Exec: Run shell commands");
expect(prompt).toContain(
"- If exactly one skill clearly applies: read its SKILL.md at <location> with `Read`, then follow it.",
);
@@ -465,25 +387,6 @@ describe("buildAgentSystemPrompt", () => {
);
});
it("adds update_plan guidance only when the tool is available", () => {
const promptWithPlan = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
toolNames: ["exec", "update_plan"],
});
const promptWithoutPlan = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
toolNames: ["exec"],
});
expect(promptWithPlan).toContain(
"For non-trivial multi-step work, keep a short plan updated with `update_plan`.",
);
expect(promptWithPlan).toContain(
"When you use `update_plan`, keep exactly one step `in_progress` until the work is done.",
);
expect(promptWithoutPlan).not.toContain("keep a short plan updated with `update_plan`");
});
it("includes docs guidance when docsPath is provided", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
@@ -513,7 +416,7 @@ describe("buildAgentSystemPrompt", () => {
params: {
workspaceDir: "/tmp/openclaw",
userTimezone: "America/Chicago",
userTime: "Monday, January 5th, 2026 - 3:26 PM",
userTime: "Monday, January 5th, 2026 — 3:26 PM",
userTimeFormat: "12" as const,
},
},
@@ -522,7 +425,7 @@ describe("buildAgentSystemPrompt", () => {
params: {
workspaceDir: "/tmp/openclaw",
userTimezone: "America/Chicago",
userTime: "Monday, January 5th, 2026 - 15:26",
userTime: "Monday, January 5th, 2026 — 15:26",
userTimeFormat: "24" as const,
},
},
@@ -555,19 +458,23 @@ describe("buildAgentSystemPrompt", () => {
// The system prompt intentionally does NOT include the current date/time.
// Only the timezone is included, to keep the prompt stable for caching.
// See: https://github.com/moltbot/moltbot/commit/66eec295b894bce8333886cfbca3b960c57c4946
// Agents should use session_status or message timestamps to determine the date/time.
// Related: https://github.com/moltbot/moltbot/issues/1897
// https://github.com/moltbot/moltbot/issues/3658
it("does NOT include a date or time in the system prompt (cache stability)", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/clawd",
userTimezone: "America/Chicago",
userTime: "Monday, January 5th, 2026 - 3:26 PM",
userTime: "Monday, January 5th, 2026 — 3:26 PM",
userTimeFormat: "12",
});
// The prompt should contain the timezone but NOT the formatted date/time string.
// This is intentional for prompt cache stability. If you want to add date/time
// awareness, do it through gateway-level timestamp injection into messages, not
// the system prompt.
// This is intentional for prompt cache stability — the date/time was removed in
// commit 66eec295b. If you're here because you want to add it back, please see
// https://github.com/moltbot/moltbot/issues/3658 for the preferred approach:
// gateway-level timestamp injection into messages, not the system prompt.
expect(prompt).toContain("Time zone: America/Chicago");
expect(prompt).not.toContain("Monday, January 5th, 2026");
expect(prompt).not.toContain("3:26 PM");
@@ -578,14 +485,14 @@ describe("buildAgentSystemPrompt", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
modelAliasLines: [
"- Opus: anthropic/claude-opus-4-6",
"- Sonnet: anthropic/claude-sonnet-4-6",
"- Opus: anthropic/claude-opus-4-5",
"- Sonnet: anthropic/claude-sonnet-4-5",
],
});
expect(prompt).toContain("## Model Aliases");
expect(prompt).toContain("Prefer aliases when specifying model overrides");
expect(prompt).toContain("- Opus: anthropic/claude-opus-4-6");
expect(prompt).toContain("- Opus: anthropic/claude-opus-4-5");
});
it("adds ClaudeBot self-update guidance when gateway tool is available", () => {
@@ -692,115 +599,35 @@ describe("buildAgentSystemPrompt", () => {
expect(prompt).not.toContain("# Project Context");
});
it("orders stable project context before the cache boundary and moves HEARTBEAT below it", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
contextFiles: [
{ path: "HEARTBEAT.md", content: "Check inbox." },
{ path: "MEMORY.md", content: "Long-term notes." },
{ path: "AGENTS.md", content: "Follow repo rules." },
{ path: "SOUL.md", content: "Warm but direct." },
{ path: "TOOLS.md", content: "Prefer rg." },
],
});
const agentsIndex = prompt.indexOf("## AGENTS.md");
const soulIndex = prompt.indexOf("## SOUL.md");
const toolsIndex = prompt.indexOf("## TOOLS.md");
const memoryIndex = prompt.indexOf("## MEMORY.md");
const boundaryIndex = prompt.indexOf(SYSTEM_PROMPT_CACHE_BOUNDARY);
const heartbeatHeadingIndex = prompt.indexOf("# Dynamic Project Context");
const heartbeatFileIndex = prompt.indexOf("## HEARTBEAT.md");
expect(agentsIndex).toBeGreaterThan(-1);
expect(soulIndex).toBeGreaterThan(agentsIndex);
expect(toolsIndex).toBeGreaterThan(soulIndex);
expect(memoryIndex).toBeGreaterThan(toolsIndex);
expect(boundaryIndex).toBeGreaterThan(memoryIndex);
expect(heartbeatHeadingIndex).toBeGreaterThan(boundaryIndex);
expect(heartbeatFileIndex).toBeGreaterThan(heartbeatHeadingIndex);
expect(prompt).toContain(
"The following frequently-changing project context files are kept below the cache boundary when possible:",
);
});
it("keeps heartbeat-only project context below the cache boundary", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
contextFiles: [{ path: "HEARTBEAT.md", content: "Check inbox." }],
});
const boundaryIndex = prompt.indexOf(SYSTEM_PROMPT_CACHE_BOUNDARY);
const projectContextIndex = prompt.indexOf("# Project Context");
const heartbeatFileIndex = prompt.indexOf("## HEARTBEAT.md");
expect(boundaryIndex).toBeGreaterThan(-1);
expect(projectContextIndex).toBeGreaterThan(boundaryIndex);
expect(heartbeatFileIndex).toBeGreaterThan(projectContextIndex);
expect(prompt).not.toContain("# Dynamic Project Context");
});
it("replaces provider-owned prompt sections without disturbing core ordering", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
promptContribution: {
sectionOverrides: {
interaction_style: "## Interaction Style\n\nCustom interaction guidance.",
execution_bias: "## Execution Bias\n\nCustom execution guidance.",
},
},
});
expect(prompt).toContain("## Interaction Style\n\nCustom interaction guidance.");
expect(prompt).toContain("## Execution Bias\n\nCustom execution guidance.");
expect(prompt).not.toContain("Bias toward action and momentum.");
});
it("places provider stable prefixes above the cache boundary", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
promptContribution: {
stablePrefix: "## Provider Stable Block\n\nStable provider guidance.",
},
});
const boundaryIndex = prompt.indexOf(SYSTEM_PROMPT_CACHE_BOUNDARY);
const stableIndex = prompt.indexOf("## Provider Stable Block");
const safetyIndex = prompt.indexOf("## Safety");
expect(stableIndex).toBeGreaterThan(-1);
expect(boundaryIndex).toBeGreaterThan(stableIndex);
expect(safetyIndex).toBeGreaterThan(stableIndex);
});
it("places provider dynamic suffixes below the cache boundary", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
promptContribution: {
dynamicSuffix: "## Provider Dynamic Block\n\nPer-turn provider guidance.",
},
});
const boundaryIndex = prompt.indexOf(SYSTEM_PROMPT_CACHE_BOUNDARY);
const dynamicIndex = prompt.indexOf("## Provider Dynamic Block");
const heartbeatIndex = prompt.indexOf("## Heartbeats");
expect(boundaryIndex).toBeGreaterThan(-1);
expect(dynamicIndex).toBeGreaterThan(boundaryIndex);
expect(heartbeatIndex).toBe(-1);
});
it("summarizes the message tool when available", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
toolNames: ["message"],
});
expect(prompt).toContain("message: Send messages and channel actions");
expect(prompt).toContain("### message tool");
expect(prompt).toContain("Use `message` for proactive sends + channel actions");
expect(prompt).toContain(`respond with ONLY: ${SILENT_REPLY_TOKEN}`);
});
it("reapplies provider prompt contributions", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
promptContribution: {
stablePrefix: "## Provider Stable\n\nStable guidance.",
dynamicSuffix: "## Provider Dynamic\n\nDynamic guidance.",
sectionOverrides: {
tool_call_style: "## Tool Call Style\nProvider-specific tool call guidance.",
},
},
});
expect(prompt).toContain("## Provider Stable\n\nStable guidance.");
expect(prompt).toContain("## Provider Dynamic\n\nDynamic guidance.");
expect(prompt).toContain("## Tool Call Style\nProvider-specific tool call guidance.");
expect(prompt).not.toContain("Default: do not narrate routine, low-risk tool calls");
});
it("includes inline button style guidance when runtime supports inline buttons", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
@@ -815,6 +642,19 @@ describe("buildAgentSystemPrompt", () => {
expect(prompt).toContain("`style` can be `primary`, `success`, or `danger`");
});
it("suppresses plain chat approval commands when inline approval UI is available", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
runtimeInfo: {
channel: "telegram",
capabilities: ["inlineButtons"],
},
});
expect(prompt).toContain("rely on native approval card/buttons when they appear");
expect(prompt).toContain("do not also send plain chat /approve instructions");
});
it("includes runtime provider capabilities when present", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
@@ -828,6 +668,20 @@ describe("buildAgentSystemPrompt", () => {
expect(prompt).toContain("capabilities=inlinebuttons");
});
it("canonicalizes runtime provider capabilities before rendering", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
runtimeInfo: {
channel: "telegram",
capabilities: [" InlineButtons ", "voice", "inlinebuttons", "Voice"],
},
});
expect(prompt).toContain("channel=telegram");
expect(prompt).toContain("capabilities=inlinebuttons,voice");
expect(prompt).not.toContain("capabilities= InlineButtons ,voice,inlinebuttons,Voice");
});
it("includes agent id in runtime when provided", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
@@ -865,7 +719,7 @@ describe("buildAgentSystemPrompt", () => {
arch: "arm64",
node: "v20",
model: "anthropic/claude",
defaultModel: "anthropic/claude-opus-4-6",
defaultModel: "anthropic/claude-opus-4-5",
},
"telegram",
["inlineButtons"],
@@ -878,56 +732,20 @@ describe("buildAgentSystemPrompt", () => {
expect(line).toContain("os=macOS (arm64)");
expect(line).toContain("node=v20");
expect(line).toContain("model=anthropic/claude");
expect(line).toContain("default_model=anthropic/claude-opus-4-6");
expect(line).toContain("default_model=anthropic/claude-opus-4-5");
expect(line).toContain("channel=telegram");
expect(line).toContain("capabilities=inlinebuttons");
expect(line).toContain("thinking=low");
});
it("normalizes runtime capability ordering and casing for cache stability", () => {
const line = buildRuntimeLine(
{
agentId: "work",
},
"telegram",
[" React ", "inlineButtons", "react"],
"low",
);
expect(line).toContain("capabilities=inlinebuttons,react");
});
it("keeps semantically equivalent structured prompt inputs byte-stable", () => {
const clean = buildAgentSystemPrompt({
it("renders extra system prompt exactly once", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
runtimeInfo: {
channel: "telegram",
capabilities: ["inlinebuttons", "react"],
},
skillsPrompt:
"<available_skills>\n <skill>\n <name>demo</name>\n </skill>\n</available_skills>",
heartbeatPrompt: "ping",
extraSystemPrompt: "Group chat context\nSecond line",
workspaceNotes: ["Reminder: keep commits scoped."],
modelAliasLines: ["- Sonnet: anthropic/claude-sonnet-4-5"],
});
const noisy = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
runtimeInfo: {
channel: "telegram",
capabilities: [" react ", "inlineButtons", "react"],
},
skillsPrompt:
"<available_skills>\r\n <skill> \r\n <name>demo</name>\t\r\n </skill>\r\n</available_skills>\r\n",
heartbeatPrompt: " ping \r\n",
extraSystemPrompt: " Group chat context \r\nSecond line \t\r\n",
workspaceNotes: [" Reminder: keep commits scoped. \t\r\n"],
modelAliasLines: [" - Sonnet: anthropic/claude-sonnet-4-5 \t\r\n"],
extraSystemPrompt: "Custom runtime context",
});
expect(noisy).toBe(clean);
expect(noisy).not.toContain("\r");
expect(noisy).not.toMatch(/[ \t]+$/m);
expect(prompt.match(/Custom runtime context/g)).toHaveLength(1);
expect(prompt.match(/## Group Chat Context/g)).toHaveLength(1);
});
it("describes sandboxed runtime and elevated when allowed", () => {

332
src/agents/system-prompt.ts
View File

@@ -49,8 +49,6 @@ const CONTEXT_FILE_ORDER = new Map<string, number>([
const DYNAMIC_CONTEXT_FILE_BASENAMES = new Set(["heartbeat.md"]);
const DEFAULT_HEARTBEAT_PROMPT_CONTEXT_BLOCK =
"Default heartbeat prompt:\n`Read HEARTBEAT.md if it exists (workspace context). Follow it strictly. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK.`";
const STATIC_NON_NATIVE_APPROVAL_CHANNELS = new Set(["signal"]);
function normalizeContextFilePath(pathValue: string): string {
return pathValue.trim().replace(/\\/g, "/");
}
@@ -134,6 +132,22 @@ function buildHeartbeatSection(params: { isMinimal: boolean; heartbeatPrompt?: s
];
}
function buildExecApprovalPromptGuidance(params: {
runtimeChannel?: string;
inlineButtonsEnabled?: boolean;
}) {
const runtimeChannel = normalizeOptionalLowercaseString(params.runtimeChannel);
const usesNativeApprovalUi =
params.inlineButtonsEnabled ||
(runtimeChannel
? Boolean(resolveChannelApprovalCapability(getChannelPlugin(runtimeChannel))?.native)
: false);
if (usesNativeApprovalUi) {
return "When exec returns approval-pending on this channel, rely on native approval card/buttons when they appear and do not also send plain chat /approve instructions. Only include the concrete /approve command if the tool result says chat approvals are unavailable or only manual approval is possible.";
}
return "When exec returns approval-pending, include the concrete /approve command from tool output as plain chat text for the user, and do not ask for a different or rotated code.";
}
function buildSkillsSection(params: { skillsPrompt?: string; readToolName: string }) {
const trimmed = params.skillsPrompt?.trim();
if (!trimmed) {
@@ -205,22 +219,83 @@ function buildTimeSection(params: { userTimezone?: string }) {
return ["## Current Date & Time", `Time zone: ${params.userTimezone}`, ""];
}
function buildReplyTagsSection(isMinimal: boolean) {
function buildAssistantOutputDirectivesSection(isMinimal: boolean) {
if (isMinimal) {
return [];
}
return [
"## Reply Tags",
"To request a native reply/quote on supported surfaces, include one tag in your reply:",
"## Assistant Output Directives",
"Use these when you need delivery metadata in an assistant message:",
"- `MEDIA:<path-or-url>` on its own line requests attachment delivery. The web UI strips supported MEDIA lines and renders them inline; channels still decide actual delivery behavior.",
"- `[[audio_as_voice]]` marks attached audio as a voice-note style delivery hint. The web UI may show a voice-note badge when audio is present; channels still own delivery semantics.",
"- To request a native reply/quote on supported surfaces, include one reply tag in your reply:",
"- Reply tags must be the very first token in the message (no leading text/newlines): [[reply_to_current]] your reply.",
"- [[reply_to_current]] replies to the triggering message.",
"- Prefer [[reply_to_current]]. Use [[reply_to:<id>]] only when an id was explicitly provided (e.g. by the user or a tool).",
"Whitespace inside the tag is allowed (e.g. [[ reply_to_current ]] / [[ reply_to: 123 ]]).",
"Tags are removed before sending; support depends on the current channel config.",
"- Channel-specific interactive directives are separate and should not be mixed into this web render guidance.",
"Supported tags are stripped before user-visible rendering; support still depends on the current channel config.",
"",
];
}
function buildWebchatCanvasSection(params: {
isMinimal: boolean;
runtimeChannel?: string;
canvasRootDir?: string;
}) {
if (params.isMinimal || params.runtimeChannel !== "webchat") {
return [];
}
return [
"## Control UI Embed",
"Use `[embed ...]` only in Control UI/webchat sessions for inline rich rendering inside the assistant bubble.",
"- Do not use `[embed ...]` for non-web channels.",
"- `[embed ...]` is separate from `MEDIA:`. Use `MEDIA:` for attachments; use `[embed ...]` for web-only rich rendering.",
'- Use self-closing form for hosted embed documents: `[embed ref="cv_123" title="Status" height="320" /]`.',
'- You may also use an explicit hosted URL: `[embed url="/__openclaw__/canvas/documents/cv_123/index.html" title="Status" height="320" /]`.',
'- Never use local filesystem paths or `file://...` URLs in `[embed ...]`. Hosted embeds must point at `/__openclaw__/canvas/...` URLs or use `ref="..."`.',
params.canvasRootDir
? `- The active hosted embed root for this session is: \`${sanitizeForPromptLiteral(params.canvasRootDir)}\`. If you manually stage a hosted embed file, write it there, not in the workspace.`
: "- The active hosted embed root is profile-scoped, not workspace-scoped. If you manually stage a hosted embed file, write it under the active profile embed root, not in the workspace.",
"- Quote all attribute values. Prefer `ref` for hosted documents unless you already have the full `/__openclaw__/canvas/documents/<id>/index.html` URL.",
"",
];
}
function buildExecutionBiasSection(params: { isMinimal: boolean }) {
if (params.isMinimal) {
return [];
}
return [
"## Execution Bias",
"If the user asks you to do the work, start doing it in the same turn.",
"Use a real tool call or concrete action first when the task is actionable; do not stop at a plan or promise-to-act reply.",
"Commentary-only turns are incomplete when tools are available and the next action is clear.",
"If the work will take multiple steps or a while to finish, send one short progress update before or while acting.",
"",
];
}
function normalizeProviderPromptBlock(value?: string): string | undefined {
if (typeof value !== "string") {
return undefined;
}
const normalized = normalizeStructuredPromptSection(value);
return normalized || undefined;
}
function buildOverridablePromptSection(params: {
override?: string;
fallback: string[];
}): string[] {
const override = normalizeProviderPromptBlock(params.override);
if (override) {
return [override, ""];
}
return params.fallback;
}
function buildMessagingSection(params: {
isMinimal: boolean;
availableTools: Set<string>;
@@ -290,56 +365,6 @@ function buildDocsSection(params: { docsPath?: string; isMinimal: boolean; readT
];
}
function buildExecutionBiasSection(params: { isMinimal: boolean }) {
if (params.isMinimal) {
return [];
}
return [
"## Execution Bias",
"If the user asks you to do the work, start doing it in the same turn.",
"Use a real tool call or concrete action first when the task is actionable; do not stop at a plan or promise-to-act reply.",
"Commentary-only turns are incomplete when tools are available and the next action is clear.",
"If the work will take multiple steps or a while to finish, send one short progress update before or while acting.",
"",
];
}
function normalizeProviderPromptBlock(value?: string): string | undefined {
if (typeof value !== "string") {
return undefined;
}
const normalized = normalizeStructuredPromptSection(value);
return normalized || undefined;
}
function buildOverridablePromptSection(params: {
override?: string;
fallback: string[];
}): string[] {
const override = normalizeProviderPromptBlock(params.override);
if (override) {
return [override, ""];
}
return params.fallback;
}
function buildExecApprovalPromptGuidance(params: {
runtimeChannel?: string;
inlineButtonsEnabled?: boolean;
}) {
const runtimeChannel = normalizeOptionalLowercaseString(params.runtimeChannel);
const usesNativeApprovalUi =
runtimeChannel === "webchat" ||
params.inlineButtonsEnabled === true ||
(runtimeChannel && !STATIC_NON_NATIVE_APPROVAL_CHANNELS.has(runtimeChannel)
? Boolean(resolveChannelApprovalCapability(getChannelPlugin(runtimeChannel))?.native)
: false);
if (usesNativeApprovalUi) {
return "When exec returns approval-pending on this channel, rely on native approval card/buttons when they appear and do not also send plain chat /approve instructions. Only include the concrete /approve command if the tool result says chat approvals are unavailable or only manual approval is possible.";
}
return "When exec returns approval-pending, include the concrete /approve command from tool output as plain chat text for the user, and do not ask for a different or rotated code.";
}
function formatFullAccessBlockedReason(reason?: EmbeddedFullAccessBlockedReason): string {
if (reason === "host-policy") {
return "host policy";
@@ -352,7 +377,6 @@ function formatFullAccessBlockedReason(reason?: EmbeddedFullAccessBlockedReason)
}
return "runtime constraints";
}
export function buildAgentSystemPrompt(params: {
workspaceDir: string;
defaultThinkLevel?: ThinkLevel;
@@ -363,6 +387,7 @@ export function buildAgentSystemPrompt(params: {
ownerDisplaySecret?: string;
reasoningTagHint?: boolean;
toolNames?: string[];
toolSummaries?: Record<string, string>;
modelAliasLines?: string[];
userTimezone?: string;
userTime?: string;
@@ -383,13 +408,13 @@ export function buildAgentSystemPrompt(params: {
os?: string;
arch?: string;
node?: string;
provider?: string;
model?: string;
defaultModel?: string;
shell?: string;
channel?: string;
capabilities?: string[];
repoRoot?: string;
canvasRootDir?: string;
};
messageToolHints?: string[];
sandboxInfo?: EmbeddedSandboxInfo;
@@ -398,7 +423,6 @@ export function buildAgentSystemPrompt(params: {
level: "minimal" | "extensive";
channel: string;
};
/** Whether to include the active memory plugin prompt guidance in the base system prompt. Defaults to true. */
includeMemorySection?: boolean;
memoryCitationsMode?: MemoryCitationsMode;
promptContribution?: ProviderSystemPromptContribution;
@@ -406,12 +430,75 @@ export function buildAgentSystemPrompt(params: {
const acpEnabled = params.acpEnabled !== false;
const sandboxedRuntime = params.sandboxInfo?.enabled === true;
const acpSpawnRuntimeEnabled = acpEnabled && !sandboxedRuntime;
const coreToolSummaries: Record<string, string> = {
read: "Read file contents",
write: "Create or overwrite files",
edit: "Make precise edits to files",
apply_patch: "Apply multi-file patches",
grep: "Search file contents for patterns",
find: "Find files by glob pattern",
ls: "List directory contents",
exec: "Run shell commands (pty available for TTY-required CLIs)",
process: "Manage background exec sessions",
web_search: "Search the web (Brave API)",
web_fetch: "Fetch and extract readable content from a URL",
// Channel docking: add login tools here when a channel needs interactive linking.
browser: "Control web browser",
canvas: "Present/eval/snapshot the Canvas",
nodes: "List/describe/notify/camera/screen on paired nodes",
cron: "Manage cron jobs and wake events (use for reminders; when scheduling a reminder, write the systemEvent text as something that will read like a reminder when it fires, and mention that it is a reminder depending on the time gap between setting and firing; include recent context in reminder text if appropriate)",
message: "Send messages and channel actions",
gateway: "Restart, apply config, or run updates on the running OpenClaw process",
agents_list: acpSpawnRuntimeEnabled
? 'List OpenClaw agent ids allowed for sessions_spawn when runtime="subagent" (not ACP harness ids)'
: "List OpenClaw agent ids allowed for sessions_spawn",
sessions_list: "List other sessions (incl. sub-agents) with filters/last",
sessions_history: "Fetch history for another session/sub-agent",
sessions_send: "Send a message to another session/sub-agent",
sessions_spawn: acpSpawnRuntimeEnabled
? 'Spawn an isolated sub-agent or ACP coding session (runtime="acp" requires `agentId` unless `acp.defaultAgent` is configured; ACP harness ids follow acp.allowedAgents, not agents_list)'
: "Spawn an isolated sub-agent session",
subagents: "List, steer, or kill sub-agent runs for this requester session",
session_status:
"Show a /status-equivalent status card (usage + time + Reasoning/Verbose/Elevated); use for model-use questions (📊 session_status); optional per-session model override",
image: "Analyze an image with the configured image model",
image_generate: "Generate images with the configured image-generation model",
};
const toolOrder = [
"read",
"write",
"edit",
"apply_patch",
"grep",
"find",
"ls",
"exec",
"process",
"web_search",
"web_fetch",
"browser",
"canvas",
"nodes",
"cron",
"message",
"gateway",
"agents_list",
"sessions_list",
"sessions_history",
"sessions_send",
"subagents",
"session_status",
"image",
"image_generate",
];
const rawToolNames = (params.toolNames ?? []).map((tool) => tool.trim());
const canonicalToolNames = rawToolNames.filter(Boolean);
// Preserve caller casing while deduping tool names by lowercase.
const canonicalByNormalized = new Map<string, string>();
for (const name of canonicalToolNames) {
const normalized = normalizeLowercaseStringOrEmpty(name);
const normalized = name.toLowerCase();
if (!canonicalByNormalized.has(normalized)) {
canonicalByNormalized.set(normalized, name);
}
@@ -419,20 +506,38 @@ export function buildAgentSystemPrompt(params: {
const resolveToolName = (normalized: string) =>
canonicalByNormalized.get(normalized) ?? normalized;
const normalizedTools = canonicalToolNames.map((tool) => normalizeLowercaseStringOrEmpty(tool));
const normalizedTools = canonicalToolNames.map((tool) => tool.toLowerCase());
const availableTools = new Set(normalizedTools);
const hasSessionsSpawn = availableTools.has("sessions_spawn");
const hasUpdatePlanTool = availableTools.has("update_plan");
const acpHarnessSpawnAllowed = hasSessionsSpawn && acpSpawnRuntimeEnabled;
const externalToolSummaries = new Map<string, string>();
for (const [key, value] of Object.entries(params.toolSummaries ?? {})) {
const normalized = key.trim().toLowerCase();
if (!normalized || !value?.trim()) {
continue;
}
externalToolSummaries.set(normalized, value.trim());
}
const extraTools = Array.from(
new Set(normalizedTools.filter((tool) => !toolOrder.includes(tool))),
);
const enabledTools = toolOrder.filter((tool) => availableTools.has(tool));
const toolLines = enabledTools.map((tool) => {
const summary = coreToolSummaries[tool] ?? externalToolSummaries.get(tool);
const name = resolveToolName(tool);
return summary ? `- ${name}: ${summary}` : `- ${name}`;
});
for (const tool of extraTools.toSorted()) {
const summary = coreToolSummaries[tool] ?? externalToolSummaries.get(tool);
const name = resolveToolName(tool);
toolLines.push(summary ? `- ${name}: ${summary}` : `- ${name}`);
}
const hasGateway = availableTools.has("gateway");
const hasCronTool = availableTools.has("cron") || canonicalToolNames.length === 0;
const readToolName = resolveToolName("read");
const execToolName = resolveToolName("exec");
const processToolName = resolveToolName("process");
const extraSystemPrompt =
typeof params.extraSystemPrompt === "string"
? normalizeStructuredPromptSection(params.extraSystemPrompt)
: undefined;
const extraSystemPrompt = params.extraSystemPrompt?.trim();
const promptContribution = params.promptContribution;
const providerStablePrefix = normalizeProviderPromptBlock(promptContribution?.stablePrefix);
const providerDynamicSuffix = normalizeProviderPromptBlock(promptContribution?.dynamicSuffix);
@@ -464,14 +569,8 @@ export function buildAgentSystemPrompt(params: {
: undefined;
const reasoningLevel = params.reasoningLevel ?? "off";
const userTimezone = params.userTimezone?.trim();
const skillsPrompt =
typeof params.skillsPrompt === "string"
? normalizeStructuredPromptSection(params.skillsPrompt)
: undefined;
const heartbeatPrompt =
typeof params.heartbeatPrompt === "string"
? normalizeStructuredPromptSection(params.heartbeatPrompt)
: undefined;
const skillsPrompt = params.skillsPrompt?.trim();
const heartbeatPrompt = params.heartbeatPrompt?.trim();
const runtimeInfo = params.runtimeInfo;
const runtimeChannel = normalizeOptionalLowercaseString(runtimeInfo?.channel);
const runtimeCapabilities = runtimeInfo?.capabilities ?? [];
@@ -522,45 +621,41 @@ export function buildAgentSystemPrompt(params: {
isMinimal,
readToolName,
});
const workspaceNotes = (params.workspaceNotes ?? [])
.map((note) => normalizeStructuredPromptSection(note))
.filter(Boolean);
const modelAliasLines = (params.modelAliasLines ?? [])
.map((line) => normalizeStructuredPromptSection(line))
.filter(Boolean);
const workspaceNotes = (params.workspaceNotes ?? []).map((note) => note.trim()).filter(Boolean);
// For "none" mode, return just the basic identity line
if (promptMode === "none") {
return "You are a personal assistant operating inside OpenClaw.";
return "You are a personal assistant running inside OpenClaw.";
}
const lines = [
"You are a personal assistant operating inside OpenClaw.",
"You are a personal assistant running inside OpenClaw.",
"",
"## Tooling",
"Structured tool definitions are the source of truth for tool names, descriptions, and parameters.",
"Tool names are case-sensitive. Call tools exactly as listed in the structured tool definitions.",
"If a tool is present in the structured tool definitions, it is available unless a later tool call reports a policy/runtime restriction.",
"TOOLS.md does not control tool availability; it is user guidance for how to use external tools.",
...(hasCronTool
? [
`For follow-up at a future time (for example "check back in 10 minutes", reminders, run-later work, or recurring tasks), use cron instead of ${execToolName} sleep, yieldMs delays, or ${processToolName} polling.`,
`Use ${execToolName}/${processToolName} only for commands that start now and continue running in the background.`,
`For long-running work that starts now, start it once and rely on automatic completion wake when it is enabled and the command emits output or fails; otherwise use ${processToolName} to confirm completion, and use it for logs, status, input, or intervention.`,
"Do not emulate scheduling with sleep loops, timeout loops, or repeated polling.",
]
"Tool availability (filtered by policy):",
"Tool names are case-sensitive. Call tools exactly as listed.",
toolLines.length > 0
? toolLines.join("\n")
: [
`For long waits, avoid rapid poll loops: use ${execToolName} with enough yieldMs or ${processToolName}(action=poll, timeout=<ms>).`,
`For long-running work that starts now, start it once and rely on automatic completion wake when it is enabled and the command emits output or fails; otherwise use ${processToolName} to confirm completion, and use it for logs, status, input, or intervention.`,
]),
...(hasUpdatePlanTool
? [
"For non-trivial multi-step work, keep a short plan updated with `update_plan`.",
"Skip `update_plan` for simple tasks, obvious one-step fixes, or work you can finish in a few direct actions.",
"When you use `update_plan`, keep exactly one step `in_progress` until the work is done.",
"After calling `update_plan`, continue the work and do not repeat the full plan unless the user asks.",
]
: []),
"Pi lists the standard tools above. This runtime enables:",
"- grep: search file contents for patterns",
"- find: find files by glob pattern",
"- ls: list directory contents",
"- apply_patch: apply multi-file patches",
`- ${execToolName}: run shell commands (supports background via yieldMs/background)`,
`- ${processToolName}: manage background exec sessions`,
"- browser: control OpenClaw's dedicated browser",
"- canvas: present/eval/snapshot the Canvas",
"- nodes: list/describe/notify/camera/screen on paired nodes",
"- cron: manage cron jobs and wake events (use for reminders; when scheduling a reminder, write the systemEvent text as something that will read like a reminder when it fires, and mention that it is a reminder depending on the time gap between setting and firing; include recent context in reminder text if appropriate)",
"- sessions_list: list sessions",
"- sessions_history: fetch session history",
"- sessions_send: send to another session",
"- subagents: list/steer/kill sub-agent runs",
'- session_status: show usage/time/model state and answer "what model are we using?"',
].join("\n"),
"TOOLS.md does not control tool availability; it is user guidance for how to use external tools.",
`For long waits, avoid rapid poll loops: use ${execToolName} with enough yieldMs or ${processToolName}(action=poll, timeout=<ms>).`,
"If a task is more complex or takes longer, spawn a sub-agent. Completion is push-based: it will auto-announce when done.",
...(acpHarnessSpawnAllowed
? [
@@ -624,19 +719,23 @@ export function buildAgentSystemPrompt(params: {
"Get Updates (self-update) is ONLY allowed when the user explicitly asks for it.",
"Do not run config.apply or update.run unless the user explicitly requests an update or config change; if it's not explicit, ask first.",
"Use config.schema.lookup with a specific dot path to inspect only the relevant config subtree before making config changes or answering config-field questions; avoid guessing field names/types.",
"Actions: config.schema.lookup, config.get, config.apply (validate + write full config, then hot-reload or restart as needed), config.patch (partial update, merges with existing), update.run (update deps or git, then restart).",
"Actions: config.schema.lookup, config.get, config.apply (validate + write full config, then restart), config.patch (partial update, merges with existing), update.run (update deps or git, then restart).",
"After restart, OpenClaw pings the last active session automatically.",
].join("\n")
: "",
hasGateway && !isMinimal ? "" : "",
"",
// Skip model aliases for subagent/none modes
modelAliasLines.length > 0 && !isMinimal ? "## Model Aliases" : "",
modelAliasLines.length > 0 && !isMinimal
params.modelAliasLines && params.modelAliasLines.length > 0 && !isMinimal
? "## Model Aliases"
: "",
params.modelAliasLines && params.modelAliasLines.length > 0 && !isMinimal
? "Prefer aliases when specifying model overrides; full provider/model is also accepted."
: "",
modelAliasLines.length > 0 && !isMinimal ? modelAliasLines.join("\n") : "",
modelAliasLines.length > 0 && !isMinimal ? "" : "",
params.modelAliasLines && params.modelAliasLines.length > 0 && !isMinimal
? params.modelAliasLines.join("\n")
: "",
params.modelAliasLines && params.modelAliasLines.length > 0 && !isMinimal ? "" : "",
userTimezone
? "If you need the current date, time, or day of week, run session_status (📊 session_status)."
: "",
@@ -716,7 +815,12 @@ export function buildAgentSystemPrompt(params: {
"## Workspace Files (injected)",
"These user-editable files are loaded by OpenClaw and included below in Project Context.",
"",
...buildReplyTagsSection(isMinimal),
...buildAssistantOutputDirectivesSection(isMinimal),
...buildWebchatCanvasSection({
isMinimal,
runtimeChannel,
canvasRootDir: params.runtimeInfo?.canvasRootDir,
}),
...buildMessagingSection({
isMinimal,
availableTools,
@@ -774,12 +878,10 @@ export function buildAgentSystemPrompt(params: {
if (!isMinimal) {
lines.push(
"## Silent Replies",
`Use ${SILENT_REPLY_TOKEN} ONLY when no user-visible reply is required.`,
`When you have nothing to say, respond with ONLY: ${SILENT_REPLY_TOKEN}`,
"",
"⚠️ Rules:",
"- Valid cases: silent housekeeping, deliberate no-op ambient wakeups, or after a messaging tool already delivered the user-visible reply.",
"- Never use it to avoid doing requested work or to end an actionable turn early.",
"- It must be your ENTIRE message - nothing else",
"- It must be your ENTIRE message — nothing else",
`- Never append it to an actual response (never include "${SILENT_REPLY_TOKEN}" in real replies)`,
"- Never wrap it in markdown or code blocks",
"",

245
src/chat/canvas-render.ts Normal file
View File

@@ -0,0 +1,245 @@
import { parseFenceSpans } from "../markdown/fences.js";
export type CanvasSurface = "assistant_message";
export type CanvasPreview = {
kind: "canvas";
surface: CanvasSurface;
render: "url";
title?: string;
preferredHeight?: number;
url?: string;
viewId?: string;
className?: string;
style?: string;
};
function tryParseJsonRecord(value: string | undefined): Record<string, unknown> | undefined {
if (typeof value !== "string") {
return undefined;
}
try {
const parsed = JSON.parse(value);
return parsed && typeof parsed === "object" && !Array.isArray(parsed)
? (parsed as Record<string, unknown>)
: undefined;
} catch {
return undefined;
}
}
function getRecordStringField(
record: Record<string, unknown> | undefined,
key: string,
): string | undefined {
const value = record?.[key];
return typeof value === "string" && value.trim() ? value : undefined;
}
function getRecordNumberField(
record: Record<string, unknown> | undefined,
key: string,
): number | undefined {
const value = record?.[key];
return typeof value === "number" && Number.isFinite(value) ? value : undefined;
}
function getNestedRecord(
record: Record<string, unknown> | undefined,
key: string,
): Record<string, unknown> | undefined {
const value = record?.[key];
return value && typeof value === "object" && !Array.isArray(value)
? (value as Record<string, unknown>)
: undefined;
}
function normalizeSurface(value: string | undefined): CanvasSurface | undefined {
return value === "assistant_message" ? value : undefined;
}
function normalizePreferredHeight(value: number | undefined): number | undefined {
return typeof value === "number" && Number.isFinite(value) && value >= 160
? Math.min(Math.trunc(value), 1200)
: undefined;
}
function coerceCanvasPreview(
record: Record<string, unknown> | undefined,
): CanvasPreview | undefined {
if (!record) {
return undefined;
}
const kind = getRecordStringField(record, "kind")?.trim().toLowerCase();
if (kind !== "canvas") {
return undefined;
}
const presentation = getNestedRecord(record, "presentation");
const view = getNestedRecord(record, "view");
const source = getNestedRecord(record, "source");
const requestedSurface =
getRecordStringField(presentation, "target") ?? getRecordStringField(record, "target");
const surface = requestedSurface ? normalizeSurface(requestedSurface) : "assistant_message";
if (!surface) {
return undefined;
}
const title = getRecordStringField(presentation, "title") ?? getRecordStringField(view, "title");
const preferredHeight = normalizePreferredHeight(
getRecordNumberField(presentation, "preferred_height") ??
getRecordNumberField(presentation, "preferredHeight") ??
getRecordNumberField(view, "preferred_height") ??
getRecordNumberField(view, "preferredHeight"),
);
const className =
getRecordStringField(presentation, "class_name") ??
getRecordStringField(presentation, "className");
const style = getRecordStringField(presentation, "style");
const viewUrl = getRecordStringField(view, "url") ?? getRecordStringField(view, "entryUrl");
const viewId = getRecordStringField(view, "id") ?? getRecordStringField(view, "docId");
if (viewUrl) {
return {
kind: "canvas",
surface,
render: "url",
url: viewUrl,
...(viewId ? { viewId } : {}),
...(title ? { title } : {}),
...(preferredHeight ? { preferredHeight } : {}),
...(className ? { className } : {}),
...(style ? { style } : {}),
};
}
const sourceType = getRecordStringField(source, "type")?.trim().toLowerCase();
if (sourceType === "url") {
const url = getRecordStringField(source, "url");
if (!url) {
return undefined;
}
return {
kind: "canvas",
surface,
render: "url",
url,
...(title ? { title } : {}),
...(preferredHeight ? { preferredHeight } : {}),
...(className ? { className } : {}),
...(style ? { style } : {}),
};
}
return undefined;
}
function parseCanvasAttributes(raw: string): Record<string, string> {
const attrs: Record<string, string> = {};
const re = /([A-Za-z_][A-Za-z0-9_-]*)\s*=\s*(?:"([^"]*)"|'([^']*)')/g;
let match: RegExpExecArray | null;
while ((match = re.exec(raw))) {
const key = match[1]?.trim().toLowerCase();
const value = (match[2] ?? match[3] ?? "").trim();
if (key && value) {
attrs[key] = value;
}
}
return attrs;
}
function defaultCanvasEntryUrl(ref: string): string {
const encoded = encodeURIComponent(ref.trim());
return `/__openclaw__/canvas/documents/${encoded}/index.html`;
}
function previewFromShortcode(attrs: Record<string, string>): CanvasPreview | undefined {
if (attrs.target && normalizeSurface(attrs.target) !== "assistant_message") {
return undefined;
}
const surface = "assistant_message";
const title = attrs.title?.trim() || undefined;
const preferredHeight =
attrs.height && Number.isFinite(Number(attrs.height))
? normalizePreferredHeight(Number(attrs.height))
: undefined;
const className = attrs.class?.trim() || attrs.class_name?.trim() || undefined;
const style = attrs.style?.trim() || undefined;
const ref = attrs.ref?.trim();
const url = attrs.url?.trim();
if (url || ref) {
return {
kind: "canvas",
surface,
render: "url",
url: url ?? defaultCanvasEntryUrl(ref),
...(ref ? { viewId: ref } : {}),
...(title ? { title } : {}),
...(preferredHeight ? { preferredHeight } : {}),
...(className ? { className } : {}),
...(style ? { style } : {}),
};
}
return undefined;
}
export function extractCanvasFromText(
outputText: string | undefined,
_toolName?: string,
): CanvasPreview | undefined {
const parsed = tryParseJsonRecord(outputText);
return coerceCanvasPreview(parsed);
}
export function extractCanvasShortcodes(text: string | undefined): {
text: string;
previews: CanvasPreview[];
} {
if (!text?.trim() || !text.toLowerCase().includes("[embed")) {
return { text: text ?? "", previews: [] };
}
const fenceSpans = parseFenceSpans(text);
const matches: Array<{
start: number;
end: number;
attrs: Record<string, string>;
body?: string;
}> = [];
const blockRe = /\[embed\s+([^\]]*?)\]([\s\S]*?)\[\/embed\]/gi;
const selfClosingRe = /\[embed\s+([^\]]*?)\/\]/gi;
for (const re of [blockRe, selfClosingRe]) {
let match: RegExpExecArray | null;
while ((match = re.exec(text))) {
const start = match.index ?? 0;
if (fenceSpans.some((span) => start >= span.start && start < span.end)) {
continue;
}
matches.push({
start,
end: start + match[0].length,
attrs: parseCanvasAttributes(match[1] ?? ""),
...(match[2] !== undefined ? { body: match[2] } : {}),
});
}
}
if (matches.length === 0) {
return { text, previews: [] };
}
matches.sort((a, b) => a.start - b.start);
const previews: CanvasPreview[] = [];
let cursor = 0;
let stripped = "";
for (const match of matches) {
if (match.start < cursor) {
continue;
}
stripped += text.slice(cursor, match.start);
const preview = previewFromShortcode(match.attrs);
if (!preview) {
stripped += text.slice(match.start, match.end);
} else {
previews.push(preview);
}
cursor = match.end;
}
stripped += text.slice(cursor);
return {
text: stripped.replace(/\n{3,}/g, "\n\n").trim(),
previews,
};
}

4
src/chat/tool-content.ts
View File

@@ -1,9 +1,7 @@
import { normalizeLowercaseStringOrEmpty } from "../shared/string-coerce.js";
export type ToolContentBlock = Record<string, unknown>;
export function normalizeToolContentType(value: unknown): string {
return normalizeLowercaseStringOrEmpty(value);
return typeof value === "string" ? value.toLowerCase() : "";
}
export function isToolCallContentType(value: unknown): boolean {

52
src/config/config-misc.test.ts
View File

@@ -93,6 +93,58 @@ describe("ui.seamColor", () => {
});
});
describe("gateway.controlUi.embedSandbox", () => {
it("accepts strict, scripts, and trusted modes", () => {
for (const mode of ["strict", "scripts", "trusted"] as const) {
const result = OpenClawSchema.safeParse({
gateway: {
controlUi: {
embedSandbox: mode,
},
},
});
expect(result.success).toBe(true);
}
});
it("rejects unsupported values", () => {
const result = OpenClawSchema.safeParse({
gateway: {
controlUi: {
embedSandbox: "yolo",
},
},
});
expect(result.success).toBe(false);
});
});
describe("gateway.controlUi.allowExternalEmbedUrls", () => {
it("accepts boolean values", () => {
for (const value of [true, false]) {
const result = OpenClawSchema.safeParse({
gateway: {
controlUi: {
allowExternalEmbedUrls: value,
},
},
});
expect(result.success).toBe(true);
}
});
it("rejects non-boolean values", () => {
const result = OpenClawSchema.safeParse({
gateway: {
controlUi: {
allowExternalEmbedUrls: "yes",
},
},
});
expect(result.success).toBe(false);
});
});
describe("plugins.entries.*.hooks.allowPromptInjection", () => {
it("accepts boolean values", () => {
const result = OpenClawSchema.safeParse({

35
src/config/schema.base.generated.ts
View File

@@ -20586,6 +20586,31 @@ export const GENERATED_BASE_CONFIG_SCHEMA: BaseConfigSchemaResponse = {
description:
"Optional filesystem root for Control UI assets (defaults to dist/control-ui).",
},
embedSandbox: {
anyOf: [
{
type: "string",
const: "strict",
},
{
type: "string",
const: "scripts",
},
{
type: "string",
const: "trusted",
},
],
title: "Control UI Embed Sandbox Mode",
description:
'Iframe sandbox policy for hosted Control UI embeds. "strict" disables scripts, "scripts" allows interactive embeds while keeping origin isolation (default), and "trusted" adds `allow-same-origin` for same-site documents that intentionally need stronger privileges.',
},
allowExternalEmbedUrls: {
type: "boolean",
title: "Allow External Control UI Embed URLs",
description:
"DANGEROUS toggle that allows hosted embeds to load absolute external http(s) URLs. Keep this off unless your Control UI intentionally embeds trusted third-party pages; hosted /__openclaw__/canvas and /__openclaw__/a2ui documents do not need it.",
},
allowedOrigins: {
type: "array",
items: {
@@ -24073,6 +24098,16 @@ export const GENERATED_BASE_CONFIG_SCHEMA: BaseConfigSchemaResponse = {
placeholder: "dist/control-ui",
tags: ["network"],
},
"gateway.controlUi.embedSandbox": {
label: "Control UI Embed Sandbox Mode",
help: 'Iframe sandbox policy for hosted Control UI embeds. "strict" disables scripts, "scripts" allows interactive embeds while keeping origin isolation (default), and "trusted" adds `allow-same-origin` for same-site documents that intentionally need stronger privileges.',
tags: ["security", "access", "advanced"],
},
"gateway.controlUi.allowExternalEmbedUrls": {
label: "Allow External Control UI Embed URLs",
help: "DANGEROUS toggle that allows hosted embeds to load absolute external http(s) URLs. Keep this off unless your Control UI intentionally embeds trusted third-party pages; hosted /__openclaw__/canvas and /__openclaw__/a2ui documents do not need it.",
tags: ["security", "access", "network", "advanced"],
},
"gateway.controlUi.allowedOrigins": {
label: "Control UI Allowed Origins",
help: 'Allowed browser origins for Control UI/WebChat websocket connections (full origins only, e.g. https://control.example.com). Required for non-loopback Control UI deployments unless dangerous Host-header fallback is explicitly enabled. Setting ["*"] means allow any browser origin and should be avoided outside tightly controlled local testing.',

2
src/config/schema.help.quality.test.ts
View File

@@ -118,6 +118,7 @@ const TARGET_KEYS = [
"gateway.controlUi.dangerouslyAllowHostHeaderOriginFallback",
"gateway.controlUi.allowInsecureAuth",
"gateway.controlUi.dangerouslyDisableDeviceAuth",
"gateway.controlUi.embedSandbox",
"cron",
"cron.enabled",
"cron.store",
@@ -306,6 +307,7 @@ const TARGET_KEYS = [
"discovery.wideArea.enabled",
"discovery.mdns",
"discovery.mdns.mode",
"gateway.controlUi.embedSandbox",
"canvasHost",
"canvasHost.enabled",
"canvasHost.root",

4
src/config/schema.help.ts
View File

@@ -380,6 +380,10 @@ export const FIELD_HELP: Record<string, string> = {
"Optional URL prefix where the Control UI is served (e.g. /openclaw).",
"gateway.controlUi.root":
"Optional filesystem root for Control UI assets (defaults to dist/control-ui).",
"gateway.controlUi.embedSandbox":
'Iframe sandbox policy for hosted Control UI embeds. "strict" disables scripts, "scripts" allows interactive embeds while keeping origin isolation (default), and "trusted" adds `allow-same-origin` for same-site documents that intentionally need stronger privileges.',
"gateway.controlUi.allowExternalEmbedUrls":
"DANGEROUS toggle that allows hosted embeds to load absolute external http(s) URLs. Keep this off unless your Control UI intentionally embeds trusted third-party pages; hosted /__openclaw__/canvas and /__openclaw__/a2ui documents do not need it.",
"gateway.controlUi.allowedOrigins":
'Allowed browser origins for Control UI/WebChat websocket connections (full origins only, e.g. https://control.example.com). Required for non-loopback Control UI deployments unless dangerous Host-header fallback is explicitly enabled. Setting ["*"] means allow any browser origin and should be avoided outside tightly controlled local testing.',
"gateway.controlUi.dangerouslyAllowHostHeaderOriginFallback":

2
src/config/schema.labels.ts
View File

@@ -265,6 +265,8 @@ export const FIELD_LABELS: Record<string, string> = {
"Web Fetch Allow RFC 2544 Benchmark Range",
"gateway.controlUi.basePath": "Control UI Base Path",
"gateway.controlUi.root": "Control UI Assets Root",
"gateway.controlUi.embedSandbox": "Control UI Embed Sandbox Mode",
"gateway.controlUi.allowExternalEmbedUrls": "Allow External Control UI Embed URLs",
"gateway.controlUi.allowedOrigins": "Control UI Allowed Origins",
"gateway.controlUi.dangerouslyAllowHostHeaderOriginFallback":
"Dangerously Allow Host-Header Origin Fallback",

2
src/config/schema.tags.ts
View File

@@ -43,6 +43,8 @@ const TAG_OVERRIDES: Record<string, ConfigTag[]> = {
"gateway.auth.token": ["security", "auth", "access", "network"],
"gateway.auth.password": ["security", "auth", "access", "network"],
"gateway.push.apns.relay.baseUrl": ["network", "advanced"],
"gateway.controlUi.embedSandbox": ["security", "access", "advanced"],
"gateway.controlUi.allowExternalEmbedUrls": ["security", "access", "network", "advanced"],
"gateway.controlUi.dangerouslyAllowHostHeaderOriginFallback": [
"security",
"access",

12
src/config/types.gateway.ts
View File

@@ -85,6 +85,18 @@ export type GatewayControlUiConfig = {
basePath?: string;
/** Optional filesystem root for Control UI assets (defaults to dist/control-ui). */
root?: string;
/**
* Embed sandbox mode for hosted Control UI previews.
* - strict: no script execution inside embeds
* - scripts: allow scripts while keeping embeds origin-isolated (default)
* - trusted: allow scripts and same-origin privileges
*/
embedSandbox?: "strict" | "scripts" | "trusted";
/**
* DANGEROUS: Allow hosted embeds to load absolute external http(s) URLs.
* Default off; prefer hosted /__openclaw__/canvas or /__openclaw__/a2ui content.
*/
allowExternalEmbedUrls?: boolean;
/** Allowed browser origins for Control UI/WebChat websocket connections. */
allowedOrigins?: string[];
/**

4
src/config/zod-schema.ts
View File

@@ -680,6 +680,10 @@ export const OpenClawSchema = z
enabled: z.boolean().optional(),
basePath: z.string().optional(),
root: z.string().optional(),
embedSandbox: z
.union([z.literal("strict"), z.literal("scripts"), z.literal("trusted")])
.optional(),
allowExternalEmbedUrls: z.boolean().optional(),
allowedOrigins: z.array(z.string()).optional(),
dangerouslyAllowHostHeaderOriginFallback: z.boolean().optional(),
allowInsecureAuth: z.boolean().optional(),

242
src/gateway/canvas-documents.test.ts Normal file
View File

@@ -0,0 +1,242 @@
import { mkdtemp, mkdir, writeFile, readFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import {
buildCanvasDocumentEntryUrl,
createCanvasDocument,
resolveCanvasDocumentAssets,
resolveCanvasDocumentDir,
resolveCanvasHttpPathToLocalPath,
} from "./canvas-documents.js";
const tempDirs: string[] = [];
afterEach(async () => {
await Promise.all(
tempDirs.splice(0).map(async (dir) => {
await import("node:fs/promises").then((fs) => fs.rm(dir, { recursive: true, force: true }));
}),
);
});
describe("canvas documents", () => {
it("builds entry urls for materialized path documents under managed storage", async () => {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-"));
tempDirs.push(stateDir);
const workspaceDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-workspace-"));
tempDirs.push(workspaceDir);
await mkdir(path.join(workspaceDir, "player"), { recursive: true });
await writeFile(path.join(workspaceDir, "player/index.html"), "<div>ok</div>", "utf8");
const document = await createCanvasDocument(
{
kind: "html_bundle",
entrypoint: {
type: "path",
value: "player/index.html",
},
},
{ stateDir, workspaceDir },
);
expect(document.entryUrl).toContain("/__openclaw__/canvas/documents/");
expect(document.localEntrypoint).toBe("index.html");
expect(resolveCanvasDocumentDir(document.id, { stateDir })).toContain(stateDir);
});
it("normalizes nested local entrypoint urls", () => {
const url = buildCanvasDocumentEntryUrl("cv_example", "collection.media/index.html");
expect(url).toBe("/__openclaw__/canvas/documents/cv_example/collection.media/index.html");
});
it("encodes special characters in hosted entrypoint path segments", () => {
const url = buildCanvasDocumentEntryUrl("cv_example", "bundle#1/entry%20point?.html");
expect(url).toBe(
"/__openclaw__/canvas/documents/cv_example/bundle%231/entry%2520point%3F.html",
);
});
it("materializes inline html bundles as index documents", async () => {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-"));
tempDirs.push(stateDir);
const document = await createCanvasDocument(
{
kind: "html_bundle",
title: "Preview",
entrypoint: {
type: "html",
value:
"<!doctype html><html><head><style>.demo{color:red}</style></head><body><div class='demo'>Front</div></body></html>",
},
},
{ stateDir },
);
const indexHtml = await import("node:fs/promises").then((fs) =>
fs.readFile(
path.join(resolveCanvasDocumentDir(document.id, { stateDir }), "index.html"),
"utf8",
),
);
expect(indexHtml).toContain("<div class='demo'>Front</div>");
expect(indexHtml).toContain("<style>.demo{color:red}</style>");
expect(document.title).toBe("Preview");
expect(document.entryUrl).toBe(`/__openclaw__/canvas/documents/${document.id}/index.html`);
});
it("reuses a supplied stable document id by replacing the prior materialized view", async () => {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-"));
tempDirs.push(stateDir);
const first = await createCanvasDocument(
{
id: "status-card",
kind: "html_bundle",
entrypoint: { type: "html", value: "<div>first</div>" },
},
{ stateDir },
);
const second = await createCanvasDocument(
{
id: "status-card",
kind: "html_bundle",
entrypoint: { type: "html", value: "<div>second</div>" },
},
{ stateDir },
);
expect(first.id).toBe("status-card");
expect(second.id).toBe("status-card");
const indexHtml = await import("node:fs/promises").then((fs) =>
fs.readFile(
path.join(resolveCanvasDocumentDir(second.id, { stateDir }), "index.html"),
"utf8",
),
);
expect(indexHtml).toContain("second");
expect(indexHtml).not.toContain("first");
});
it("exposes stable managed asset urls for copied canvas assets", async () => {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-"));
tempDirs.push(stateDir);
const workspaceDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-workspace-"));
tempDirs.push(workspaceDir);
await mkdir(path.join(workspaceDir, "collection.media"), { recursive: true });
await writeFile(path.join(workspaceDir, "collection.media/audio.mp3"), "audio", "utf8");
const document = await createCanvasDocument(
{
kind: "html_bundle",
entrypoint: {
type: "html",
value:
'<audio controls><source src="collection.media/audio.mp3" type="audio/mpeg" /></audio>',
},
assets: [
{
logicalPath: "collection.media/audio.mp3",
sourcePath: "collection.media/audio.mp3",
contentType: "audio/mpeg",
},
],
},
{ stateDir, workspaceDir },
);
expect(resolveCanvasDocumentAssets(document, { stateDir })).toEqual([
{
logicalPath: "collection.media/audio.mp3",
contentType: "audio/mpeg",
localPath: path.join(
resolveCanvasDocumentDir(document.id, { stateDir }),
"collection.media/audio.mp3",
),
url: `/__openclaw__/canvas/documents/${document.id}/collection.media/audio.mp3`,
},
]);
expect(
resolveCanvasDocumentAssets(document, {
baseUrl: "http://127.0.0.1:19003",
stateDir,
}),
).toEqual([
{
logicalPath: "collection.media/audio.mp3",
contentType: "audio/mpeg",
localPath: path.join(
resolveCanvasDocumentDir(document.id, { stateDir }),
"collection.media/audio.mp3",
),
url: `http://127.0.0.1:19003/__openclaw__/canvas/documents/${document.id}/collection.media/audio.mp3`,
},
]);
});
it("wraps local pdf documents in an index viewer page", async () => {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-"));
tempDirs.push(stateDir);
const workspaceDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-workspace-"));
tempDirs.push(workspaceDir);
await writeFile(path.join(workspaceDir, "demo.pdf"), "%PDF-1.4", "utf8");
const document = await createCanvasDocument(
{
kind: "document",
entrypoint: {
type: "path",
value: "demo.pdf",
},
},
{ stateDir, workspaceDir },
);
expect(document.entryUrl).toBe(`/__openclaw__/canvas/documents/${document.id}/index.html`);
const indexHtml = await readFile(
path.join(resolveCanvasDocumentDir(document.id, { stateDir }), "index.html"),
"utf8",
);
expect(indexHtml).toContain('type="application/pdf"');
expect(indexHtml).toContain('data="demo.pdf"');
});
it("wraps remote pdf urls in an index viewer page", async () => {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-"));
tempDirs.push(stateDir);
const document = await createCanvasDocument(
{
kind: "document",
entrypoint: {
type: "url",
value: "https://example.com/demo.pdf",
},
},
{ stateDir },
);
expect(document.entryUrl).toBe(`/__openclaw__/canvas/documents/${document.id}/index.html`);
const indexHtml = await readFile(
path.join(resolveCanvasDocumentDir(document.id, { stateDir }), "index.html"),
"utf8",
);
expect(indexHtml).toContain('type="application/pdf"');
expect(indexHtml).toContain('data="https://example.com/demo.pdf"');
});
it("rejects traversal-style document ids in hosted canvas paths", async () => {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-"));
tempDirs.push(stateDir);
expect(
resolveCanvasHttpPathToLocalPath(
"/__openclaw__/canvas/documents/../collection.media/index.html",
{ stateDir },
),
).toBeNull();
});
});

347
src/gateway/canvas-documents.ts Normal file
View File

@@ -0,0 +1,347 @@
import { randomUUID } from "node:crypto";
import fs from "node:fs/promises";
import path from "node:path";
import { CANVAS_HOST_PATH } from "../canvas-host/a2ui.js";
import { resolveStateDir } from "../config/paths.js";
import { resolveUserPath } from "../utils.js";
export type CanvasDocumentKind = "html_bundle" | "url_embed" | "document" | "image" | "video_asset";
export type CanvasDocumentAsset = {
logicalPath: string;
sourcePath: string;
contentType?: string;
};
export type CanvasDocumentEntrypoint =
| { type: "html"; value: string }
| { type: "path"; value: string }
| { type: "url"; value: string };
export type CanvasDocumentCreateInput = {
id?: string;
kind: CanvasDocumentKind;
title?: string;
preferredHeight?: number;
entrypoint?: CanvasDocumentEntrypoint;
assets?: CanvasDocumentAsset[];
surface?: "assistant_message" | "tool_card" | "sidebar";
};
export type CanvasDocumentManifest = {
id: string;
kind: CanvasDocumentKind;
title?: string;
preferredHeight?: number;
createdAt: string;
entryUrl: string;
localEntrypoint?: string;
externalUrl?: string;
surface?: "assistant_message" | "tool_card" | "sidebar";
assets: Array<{
logicalPath: string;
contentType?: string;
}>;
};
export type CanvasDocumentResolvedAsset = {
logicalPath: string;
contentType?: string;
url: string;
localPath: string;
};
const CANVAS_DOCUMENTS_DIR_NAME = "documents";
function isPdfPathLike(value: string): boolean {
return /\.pdf(?:[?#].*)?$/i.test(value.trim());
}
function buildPdfWrapper(url: string): string {
const escaped = escapeHtml(url);
return `<!doctype html><html><body style="margin:0;background:#e5e7eb;"><object data="${escaped}" type="application/pdf" style="width:100%;height:100vh;border:0;"><iframe src="${escaped}" style="width:100%;height:100vh;border:0;"></iframe><p style="padding:16px;font:14px system-ui,sans-serif;">Unable to render PDF preview. <a href="${escaped}" target="_blank" rel="noopener noreferrer">Open PDF</a>.</p></object></body></html>`;
}
function escapeHtml(value: string): string {
return value
.replaceAll("&", "&amp;")
.replaceAll("<", "&lt;")
.replaceAll(">", "&gt;")
.replaceAll('"', "&quot;")
.replaceAll("'", "&#39;");
}
function normalizeLogicalPath(value: string): string {
const normalized = value.replaceAll("\\", "/").replace(/^\/+/, "");
const parts = normalized.split("/").filter(Boolean);
if (parts.length === 0 || parts.some((part) => part === "." || part === "..")) {
throw new Error("canvas document logicalPath invalid");
}
return parts.join("/");
}
function canvasDocumentId(): string {
return `cv_${randomUUID().replaceAll("-", "")}`;
}
function normalizeCanvasDocumentId(value: string): string {
const normalized = value.trim();
if (
!normalized ||
normalized === "." ||
normalized === ".." ||
!/^[A-Za-z0-9._-]+$/.test(normalized)
) {
throw new Error("canvas document id invalid");
}
return normalized;
}
export function resolveCanvasRootDir(rootDir?: string, stateDir = resolveStateDir()): string {
const resolved = rootDir?.trim() ? resolveUserPath(rootDir) : path.join(stateDir, "canvas");
return path.resolve(resolved);
}
export function resolveCanvasDocumentsDir(rootDir?: string, stateDir = resolveStateDir()): string {
return path.join(resolveCanvasRootDir(rootDir, stateDir), CANVAS_DOCUMENTS_DIR_NAME);
}
export function resolveCanvasDocumentDir(
documentId: string,
options?: { rootDir?: string; stateDir?: string },
): string {
return path.join(resolveCanvasDocumentsDir(options?.rootDir, options?.stateDir), documentId);
}
export function buildCanvasDocumentEntryUrl(documentId: string, entrypoint: string): string {
const normalizedEntrypoint = normalizeLogicalPath(entrypoint);
const encodedEntrypoint = normalizedEntrypoint
.split("/")
.map((segment) => encodeURIComponent(segment))
.join("/");
return `${CANVAS_HOST_PATH}/${CANVAS_DOCUMENTS_DIR_NAME}/${encodeURIComponent(documentId)}/${encodedEntrypoint}`;
}
export function buildCanvasDocumentAssetUrl(documentId: string, logicalPath: string): string {
return buildCanvasDocumentEntryUrl(documentId, logicalPath);
}
export function resolveCanvasHttpPathToLocalPath(
requestPath: string,
options?: { rootDir?: string; stateDir?: string },
): string | null {
const trimmed = requestPath.trim();
const prefix = `${CANVAS_HOST_PATH}/${CANVAS_DOCUMENTS_DIR_NAME}/`;
if (!trimmed.startsWith(prefix)) {
return null;
}
const pathWithoutQuery = trimmed.replace(/[?#].*$/, "");
const relative = pathWithoutQuery.slice(prefix.length);
const segments = relative
.split("/")
.map((segment) => {
try {
return decodeURIComponent(segment);
} catch {
return segment;
}
})
.filter(Boolean);
if (segments.length < 2) {
return null;
}
const [rawDocumentId, ...entrySegments] = segments;
try {
const documentId = normalizeCanvasDocumentId(rawDocumentId);
const normalizedEntrypoint = normalizeLogicalPath(entrySegments.join("/"));
const documentsDir = path.resolve(
resolveCanvasDocumentsDir(options?.rootDir, options?.stateDir),
);
const candidatePath = path.resolve(
resolveCanvasDocumentDir(documentId, options),
normalizedEntrypoint,
);
if (
!(candidatePath === documentsDir || candidatePath.startsWith(`${documentsDir}${path.sep}`))
) {
return null;
}
return candidatePath;
} catch {
return null;
}
}
async function writeManifest(rootDir: string, manifest: CanvasDocumentManifest): Promise<void> {
await fs.writeFile(
path.join(rootDir, "manifest.json"),
`${JSON.stringify(manifest, null, 2)}\n`,
"utf8",
);
}
async function copyAssets(
rootDir: string,
assets: CanvasDocumentAsset[] | undefined,
workspaceDir: string,
): Promise<CanvasDocumentManifest["assets"]> {
const copied: CanvasDocumentManifest["assets"] = [];
for (const asset of assets ?? []) {
const logicalPath = normalizeLogicalPath(asset.logicalPath);
const sourcePath = asset.sourcePath.startsWith("~")
? resolveUserPath(asset.sourcePath)
: path.isAbsolute(asset.sourcePath)
? path.resolve(asset.sourcePath)
: path.resolve(workspaceDir, asset.sourcePath);
const destination = path.join(rootDir, logicalPath);
await fs.mkdir(path.dirname(destination), { recursive: true });
await fs.copyFile(sourcePath, destination);
copied.push({
logicalPath,
...(asset.contentType ? { contentType: asset.contentType } : {}),
});
}
return copied;
}
async function materializeEntrypoint(
rootDir: string,
input: CanvasDocumentCreateInput,
workspaceDir: string,
): Promise<Pick<CanvasDocumentManifest, "entryUrl" | "localEntrypoint" | "externalUrl">> {
const entrypoint = input.entrypoint;
if (!entrypoint) {
throw new Error("canvas document entrypoint required");
}
if (entrypoint.type === "html") {
const fileName = "index.html";
await fs.writeFile(path.join(rootDir, fileName), entrypoint.value, "utf8");
return {
localEntrypoint: fileName,
entryUrl: buildCanvasDocumentEntryUrl(path.basename(rootDir), fileName),
};
}
if (entrypoint.type === "url") {
if (input.kind === "document" && isPdfPathLike(entrypoint.value)) {
const fileName = "index.html";
await fs.writeFile(path.join(rootDir, fileName), buildPdfWrapper(entrypoint.value), "utf8");
return {
localEntrypoint: fileName,
externalUrl: entrypoint.value,
entryUrl: buildCanvasDocumentEntryUrl(path.basename(rootDir), fileName),
};
}
return {
externalUrl: entrypoint.value,
entryUrl: entrypoint.value,
};
}
const resolvedPath = entrypoint.value.startsWith("~")
? resolveUserPath(entrypoint.value)
: path.isAbsolute(entrypoint.value)
? path.resolve(entrypoint.value)
: path.resolve(workspaceDir, entrypoint.value);
if (input.kind === "image" || input.kind === "video_asset") {
const copiedName = path.basename(resolvedPath);
await fs.copyFile(resolvedPath, path.join(rootDir, copiedName));
const wrapper =
input.kind === "image"
? `<!doctype html><html><body style="margin:0;background:#0f172a;display:flex;align-items:center;justify-content:center;"><img src="${escapeHtml(copiedName)}" style="max-width:100%;max-height:100vh;object-fit:contain;" /></body></html>`
: `<!doctype html><html><body style="margin:0;background:#0f172a;"><video src="${escapeHtml(copiedName)}" controls autoplay style="width:100%;height:100vh;object-fit:contain;background:#000;"></video></body></html>`;
await fs.writeFile(path.join(rootDir, "index.html"), wrapper, "utf8");
return {
localEntrypoint: "index.html",
entryUrl: buildCanvasDocumentEntryUrl(path.basename(rootDir), "index.html"),
};
}
const fileName = path.basename(resolvedPath);
await fs.copyFile(resolvedPath, path.join(rootDir, fileName));
if (input.kind === "document" && isPdfPathLike(fileName)) {
await fs.writeFile(path.join(rootDir, "index.html"), buildPdfWrapper(fileName), "utf8");
return {
localEntrypoint: "index.html",
entryUrl: buildCanvasDocumentEntryUrl(path.basename(rootDir), "index.html"),
};
}
return {
localEntrypoint: fileName,
entryUrl: buildCanvasDocumentEntryUrl(path.basename(rootDir), fileName),
};
}
export async function createCanvasDocument(
input: CanvasDocumentCreateInput,
options?: { stateDir?: string; workspaceDir?: string; canvasRootDir?: string },
): Promise<CanvasDocumentManifest> {
const workspaceDir = options?.workspaceDir ?? process.cwd();
const id = input.id?.trim() ? normalizeCanvasDocumentId(input.id) : canvasDocumentId();
const rootDir = resolveCanvasDocumentDir(id, {
stateDir: options?.stateDir,
rootDir: options?.canvasRootDir,
});
await fs.rm(rootDir, { recursive: true, force: true }).catch(() => undefined);
await fs.mkdir(rootDir, { recursive: true });
const assets = await copyAssets(rootDir, input.assets, workspaceDir);
const entry = await materializeEntrypoint(rootDir, input, workspaceDir);
const manifest: CanvasDocumentManifest = {
id,
kind: input.kind,
...(input.title?.trim() ? { title: input.title.trim() } : {}),
...(typeof input.preferredHeight === "number"
? { preferredHeight: input.preferredHeight }
: {}),
...(input.surface ? { surface: input.surface } : {}),
createdAt: new Date().toISOString(),
entryUrl: entry.entryUrl,
...(entry.localEntrypoint ? { localEntrypoint: entry.localEntrypoint } : {}),
...(entry.externalUrl ? { externalUrl: entry.externalUrl } : {}),
assets,
};
await writeManifest(rootDir, manifest);
return manifest;
}
export async function loadCanvasDocumentManifest(
documentId: string,
options?: { stateDir?: string; canvasRootDir?: string },
): Promise<CanvasDocumentManifest | null> {
const id = normalizeCanvasDocumentId(documentId);
const manifestPath = path.join(
resolveCanvasDocumentDir(id, {
stateDir: options?.stateDir,
rootDir: options?.canvasRootDir,
}),
"manifest.json",
);
try {
const raw = await fs.readFile(manifestPath, "utf8");
const parsed = JSON.parse(raw);
return parsed && typeof parsed === "object" && !Array.isArray(parsed)
? (parsed as CanvasDocumentManifest)
: null;
} catch {
return null;
}
}
export function resolveCanvasDocumentAssets(
manifest: CanvasDocumentManifest,
options?: { baseUrl?: string; stateDir?: string; canvasRootDir?: string },
): CanvasDocumentResolvedAsset[] {
const baseUrl = options?.baseUrl?.trim().replace(/\/+$/, "");
const documentDir = resolveCanvasDocumentDir(manifest.id, {
stateDir: options?.stateDir,
rootDir: options?.canvasRootDir,
});
return manifest.assets.map((asset) => ({
logicalPath: asset.logicalPath,
...(asset.contentType ? { contentType: asset.contentType } : {}),
localPath: path.join(documentDir, asset.logicalPath),
url: baseUrl
? `${baseUrl}${buildCanvasDocumentAssetUrl(manifest.id, asset.logicalPath)}`
: buildCanvasDocumentAssetUrl(manifest.id, asset.logicalPath),
}));
}

7
src/gateway/control-ui-contract.ts
View File

@@ -1,7 +1,14 @@
export const CONTROL_UI_BOOTSTRAP_CONFIG_PATH = "/__openclaw/control-ui-config.json";
export type ControlUiEmbedSandboxMode = "strict" | "scripts" | "trusted";
export type ControlUiBootstrapConfig = {
basePath: string;
assistantName: string;
assistantAvatar: string;
assistantAgentId: string;
serverVersion?: string;
localMediaPreviewRoots?: string[];
embedSandbox?: ControlUiEmbedSandboxMode;
allowExternalEmbedUrls?: boolean;
};

173
src/gateway/control-ui.http.test.ts
View File

@@ -4,8 +4,14 @@ import type { IncomingMessage } from "node:http";
import os from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import type { ResolvedGatewayAuth } from "./auth.js";
import { CONTROL_UI_BOOTSTRAP_CONFIG_PATH } from "./control-ui-contract.js";
import { handleControlUiAvatarRequest, handleControlUiHttpRequest } from "./control-ui.js";
import {
handleControlUiAssistantMediaRequest,
handleControlUiAvatarRequest,
handleControlUiHttpRequest,
} from "./control-ui.js";
import { resolvePreferredOpenClawTmpDir } from "../infra/tmp-openclaw-dir.js";
import { makeMockHttpResponse } from "./test-http-response.js";
describe("handleControlUiHttpRequest", () => {
@@ -27,6 +33,8 @@ describe("handleControlUiHttpRequest", () => {
basePath: string;
assistantName: string;
assistantAvatar: string;
assistantAgentId: string;
localMediaPreviewRoots?: string[];
};
}
@@ -77,6 +85,33 @@ describe("handleControlUiHttpRequest", () => {
return { res, end, handled };
}
async function runAssistantMediaRequest(params: {
url: string;
method: "GET" | "HEAD";
basePath?: string;
auth?: ResolvedGatewayAuth;
headers?: IncomingMessage["headers"];
trustedProxies?: string[];
remoteAddress?: string;
}) {
const { res, end } = makeMockHttpResponse();
const handled = await handleControlUiAssistantMediaRequest(
{
url: params.url,
method: params.method,
headers: params.headers ?? {},
socket: { remoteAddress: params.remoteAddress ?? "127.0.0.1" },
} as IncomingMessage,
res,
{
...(params.basePath ? { basePath: params.basePath } : {}),
...(params.auth ? { auth: params.auth } : {}),
...(params.trustedProxies ? { trustedProxies: params.trustedProxies } : {}),
},
);
return { res, end, handled };
}
async function writeAssetFile(rootPath: string, filename: string, contents: string) {
const assetsDir = path.join(rootPath, "assets");
await fs.mkdir(assetsDir, { recursive: true });
@@ -92,6 +127,18 @@ describe("handleControlUiHttpRequest", () => {
return hardlinkPath;
}
async function withAllowedAssistantMediaRoot<T>(params: {
prefix: string;
fn: (tmpRoot: string) => Promise<T>;
}) {
const tmpRoot = await fs.mkdtemp(path.join(resolvePreferredOpenClawTmpDir(), params.prefix));
try {
return await params.fn(tmpRoot);
} finally {
await fs.rm(tmpRoot, { recursive: true, force: true });
}
}
async function withBasePathRootFixture<T>(params: {
siblingDir: string;
fn: (paths: { root: string; sibling: string }) => Promise<T>;
@@ -131,6 +178,122 @@ describe("handleControlUiHttpRequest", () => {
});
});
it("serves assistant local media through the control ui media route", async () => {
await withAllowedAssistantMediaRoot({
prefix: "ui-media-",
fn: async (tmpRoot) => {
const filePath = path.join(tmpRoot, "photo.png");
await fs.writeFile(filePath, Buffer.from("not-a-real-png"));
const { res, handled } = await runAssistantMediaRequest({
url: `/__openclaw__/assistant-media?source=${encodeURIComponent(filePath)}&token=test-token`,
method: "GET",
auth: { mode: "token", token: "test-token", allowTailscale: false },
});
expect(handled).toBe(true);
expect(res.statusCode).toBe(200);
},
});
});
it("rejects assistant local media outside allowed preview roots", async () => {
const tmp = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-ui-media-blocked-"));
try {
const filePath = path.join(tmp, "photo.png");
await fs.writeFile(filePath, Buffer.from("not-a-real-png"));
const { res, handled, end } = await runAssistantMediaRequest({
url: `/__openclaw__/assistant-media?source=${encodeURIComponent(filePath)}&token=test-token`,
method: "GET",
auth: { mode: "token", token: "test-token", allowTailscale: false },
});
expectNotFoundResponse({ handled, res, end });
} finally {
await fs.rm(tmp, { recursive: true, force: true });
}
});
it("reports assistant local media availability metadata", async () => {
await withAllowedAssistantMediaRoot({
prefix: "ui-media-meta-",
fn: async (tmpRoot) => {
const filePath = path.join(tmpRoot, "photo.png");
await fs.writeFile(filePath, Buffer.from("not-a-real-png"));
const { res, handled, end } = await runAssistantMediaRequest({
url: `/__openclaw__/assistant-media?meta=1&source=${encodeURIComponent(filePath)}&token=test-token`,
method: "GET",
auth: { mode: "token", token: "test-token", allowTailscale: false },
});
expect(handled).toBe(true);
expect(res.statusCode).toBe(200);
expect(JSON.parse(String(end.mock.calls[0]?.[0] ?? ""))).toEqual({ available: true });
},
});
});
it("reports assistant local media availability failures with a reason", async () => {
const { res, handled, end } = await runAssistantMediaRequest({
url: `/__openclaw__/assistant-media?meta=1&source=${encodeURIComponent("/Users/test/Documents/private.pdf")}&token=test-token`,
method: "GET",
auth: { mode: "token", token: "test-token", allowTailscale: false },
});
expect(handled).toBe(true);
expect(res.statusCode).toBe(200);
expect(JSON.parse(String(end.mock.calls[0]?.[0] ?? ""))).toEqual({
available: false,
code: "outside-allowed-folders",
reason: "Outside allowed folders",
});
});
it("rejects assistant local media without a valid auth token when auth is enabled", async () => {
await withAllowedAssistantMediaRoot({
prefix: "ui-media-auth-",
fn: async (tmpRoot) => {
const filePath = path.join(tmpRoot, "photo.png");
await fs.writeFile(filePath, Buffer.from("not-a-real-png"));
const { res, handled, end } = await runAssistantMediaRequest({
url: `/__openclaw__/assistant-media?source=${encodeURIComponent(filePath)}`,
method: "GET",
auth: { mode: "token", token: "test-token", allowTailscale: false },
});
expect(handled).toBe(true);
expect(res.statusCode).toBe(401);
expect(String(end.mock.calls[0]?.[0] ?? "")).toContain("Unauthorized");
},
});
});
it("rejects trusted-proxy assistant media requests from disallowed browser origins", async () => {
await withAllowedAssistantMediaRoot({
prefix: "ui-media-proxy-",
fn: async (tmpRoot) => {
const filePath = path.join(tmpRoot, "photo.png");
await fs.writeFile(filePath, Buffer.from("not-a-real-png"));
const { res, handled, end } = await runAssistantMediaRequest({
url: `/__openclaw__/assistant-media?source=${encodeURIComponent(filePath)}`,
method: "GET",
auth: {
mode: "trusted-proxy",
allowTailscale: false,
trustedProxy: {
userHeader: "x-forwarded-user",
},
},
trustedProxies: ["10.0.0.1"],
remoteAddress: "10.0.0.1",
headers: {
host: "gateway.example.com",
origin: "https://evil.example",
"x-forwarded-user": "nick@example.com",
"x-forwarded-proto": "https",
},
});
expect(handled).toBe(true);
expect(res.statusCode).toBe(401);
expect(String(end.mock.calls[0]?.[0] ?? "")).toContain("Unauthorized");
},
});
});
it("includes CSP hash for inline scripts in index.html", async () => {
const scriptContent = "(function(){ var x = 1; })();";
const html = `<html><head><script>${scriptContent}</script></head><body></body></html>\n`;
@@ -195,8 +358,8 @@ describe("handleControlUiHttpRequest", () => {
expect(parsed.basePath).toBe("");
expect(parsed.assistantName).toBe("</script><script>alert(1)//");
expect(parsed.assistantAvatar).toBe("/avatar/main");
expect(parsed).not.toHaveProperty("assistantAgentId");
expect(parsed).not.toHaveProperty("serverVersion");
expect(parsed.assistantAgentId).toBe("main");
expect(Array.isArray(parsed.localMediaPreviewRoots)).toBe(true);
},
});
});
@@ -222,8 +385,8 @@ describe("handleControlUiHttpRequest", () => {
expect(parsed.basePath).toBe("/openclaw");
expect(parsed.assistantName).toBe("Ops");
expect(parsed.assistantAvatar).toBe("/openclaw/avatar/main");
expect(parsed).not.toHaveProperty("assistantAgentId");
expect(parsed).not.toHaveProperty("serverVersion");
expect(parsed.assistantAgentId).toBe("main");
expect(Array.isArray(parsed.localMediaPreviewRoots)).toBe(true);
},
});
});

239
src/gateway/control-ui.ts
View File

@@ -7,11 +7,19 @@ import {
isPackageProvenControlUiRootSync,
resolveControlUiRootSync,
} from "../infra/control-ui-assets.js";
import { openLocalFileSafely, SafeOpenError } from "../infra/fs-safe.js";
import { safeFileURLToPath } from "../infra/local-file-access.js";
import { isWithinDir } from "../infra/path-safety.js";
import { openVerifiedFileSync } from "../infra/safe-open-sync.js";
import { assertLocalMediaAllowed, getDefaultLocalRoots } from "../media/local-media-access.js";
import { getAgentScopedMediaLocalRoots } from "../media/local-roots.js";
import { detectMime } from "../media/mime.js";
import { AVATAR_MAX_BYTES } from "../shared/avatar-policy.js";
import { normalizeLowercaseStringOrEmpty } from "../shared/string-coerce.js";
import { resolveUserPath } from "../utils.js";
import { resolveRuntimeServiceVersion } from "../version.js";
import { DEFAULT_ASSISTANT_IDENTITY, resolveAssistantIdentity } from "./assistant-identity.js";
import type { AuthRateLimiter } from "./auth-rate-limit.js";
import { authorizeHttpGatewayConnect, type ResolvedGatewayAuth } from "./auth.js";
import {
CONTROL_UI_BOOTSTRAP_CONFIG_PATH,
type ControlUiBootstrapConfig,
@@ -29,8 +37,11 @@ import {
normalizeControlUiBasePath,
resolveAssistantAvatarUrl,
} from "./control-ui-shared.js";
import { sendGatewayAuthFailure } from "./http-common.js";
import { getBearerToken, resolveHttpBrowserOriginPolicy } from "./http-utils.js";
const ROOT_PREFIX = "/";
const CONTROL_UI_ASSISTANT_MEDIA_PREFIX = "/__openclaw__/assistant-media";
const CONTROL_UI_ASSETS_MISSING_MESSAGE =
"Control UI assets not found. Build them with `pnpm ui:build` (auto-installs UI deps), or run `pnpm ui:dev` during development.";
@@ -153,6 +164,218 @@ function isValidAgentId(agentId: string): boolean {
return /^[a-z0-9][a-z0-9_-]{0,63}$/i.test(agentId);
}
function normalizeAssistantMediaSource(source: string): string | null {
const trimmed = source.trim();
if (!trimmed) {
return null;
}
if (trimmed.startsWith("file://")) {
try {
return safeFileURLToPath(trimmed);
} catch {
return null;
}
}
if (trimmed.startsWith("~")) {
return resolveUserPath(trimmed);
}
return trimmed;
}
function resolveAssistantMediaRoutePath(basePath?: string): string {
const normalizedBasePath =
basePath && basePath !== "/" ? (basePath.endsWith("/") ? basePath.slice(0, -1) : basePath) : "";
return `${normalizedBasePath}${CONTROL_UI_ASSISTANT_MEDIA_PREFIX}`;
}
function resolveAssistantMediaAuthToken(req: IncomingMessage): string | undefined {
const bearer = getBearerToken(req);
if (bearer) {
return bearer;
}
const urlRaw = req.url;
if (!urlRaw) {
return undefined;
}
try {
const url = new URL(urlRaw, "http://localhost");
const token = url.searchParams.get("token")?.trim();
return token || undefined;
} catch {
return undefined;
}
}
type AssistantMediaAvailability =
| { available: true }
| { available: false; reason: string; code: string };
function classifyAssistantMediaError(err: unknown): AssistantMediaAvailability {
if (err instanceof SafeOpenError) {
switch (err.code) {
case "not-found":
return { available: false, code: "file-not-found", reason: "File not found" };
case "not-file":
return { available: false, code: "not-a-file", reason: "Not a file" };
case "invalid-path":
case "path-mismatch":
case "symlink":
return { available: false, code: "invalid-file", reason: "Invalid file" };
default:
return {
available: false,
code: "attachment-unavailable",
reason: "Attachment unavailable",
};
}
}
if (err instanceof Error && "code" in err) {
const errorCode = (err as { code?: unknown }).code;
switch (typeof errorCode === "string" ? errorCode : "") {
case "path-not-allowed":
return {
available: false,
code: "outside-allowed-folders",
reason: "Outside allowed folders",
};
case "invalid-file-url":
case "invalid-path":
case "unsafe-bypass":
case "network-path-not-allowed":
case "invalid-root":
return { available: false, code: "blocked-local-file", reason: "Blocked local file" };
case "not-found":
return { available: false, code: "file-not-found", reason: "File not found" };
case "not-file":
return { available: false, code: "not-a-file", reason: "Not a file" };
default:
break;
}
}
return { available: false, code: "attachment-unavailable", reason: "Attachment unavailable" };
}
async function resolveAssistantMediaAvailability(
source: string,
localRoots: readonly string[],
): Promise<AssistantMediaAvailability> {
try {
await assertLocalMediaAllowed(source, localRoots);
const opened = await openLocalFileSafely({ filePath: source });
await opened.handle.close();
return { available: true };
} catch (err) {
return classifyAssistantMediaError(err);
}
}
export async function handleControlUiAssistantMediaRequest(
req: IncomingMessage,
res: ServerResponse,
opts?: {
basePath?: string;
config?: OpenClawConfig;
agentId?: string;
auth?: ResolvedGatewayAuth;
trustedProxies?: string[];
allowRealIpFallback?: boolean;
rateLimiter?: AuthRateLimiter;
},
): Promise<boolean> {
const urlRaw = req.url;
if (!urlRaw || !isReadHttpMethod(req.method)) {
return false;
}
const url = new URL(urlRaw, "http://localhost");
if (url.pathname !== resolveAssistantMediaRoutePath(opts?.basePath)) {
return false;
}
applyControlUiSecurityHeaders(res);
if (opts?.auth) {
const token = resolveAssistantMediaAuthToken(req);
const authResult = await authorizeHttpGatewayConnect({
auth: opts.auth,
connectAuth: token ? { token, password: token } : null,
req,
browserOriginPolicy: resolveHttpBrowserOriginPolicy(req),
trustedProxies: opts.trustedProxies,
allowRealIpFallback: opts.allowRealIpFallback,
rateLimiter: opts.rateLimiter,
});
if (!authResult.ok) {
sendGatewayAuthFailure(res, authResult);
return true;
}
}
const source = normalizeAssistantMediaSource(url.searchParams.get("source") ?? "");
if (!source) {
respondControlUiNotFound(res);
return true;
}
const localRoots = opts?.config
? getAgentScopedMediaLocalRoots(opts.config, opts.agentId)
: getDefaultLocalRoots();
if (url.searchParams.get("meta") === "1") {
const availability = await resolveAssistantMediaAvailability(source, localRoots);
sendJson(res, 200, availability);
return true;
}
let opened: Awaited<ReturnType<typeof openLocalFileSafely>> | null = null;
let handleClosed = false;
const closeOpenedHandle = async () => {
if (!opened || handleClosed) {
return;
}
handleClosed = true;
await opened.handle.close().catch(() => {});
};
try {
await assertLocalMediaAllowed(source, localRoots);
opened = await openLocalFileSafely({ filePath: source });
const sniffLength = Math.min(opened.stat.size, 8192);
const sniffBuffer = sniffLength > 0 ? Buffer.allocUnsafe(sniffLength) : undefined;
const bytesRead =
sniffBuffer && sniffLength > 0
? (await opened.handle.read(sniffBuffer, 0, sniffLength, 0)).bytesRead
: 0;
const mime = await detectMime({
buffer: sniffBuffer?.subarray(0, bytesRead),
filePath: source,
});
if (mime) {
res.setHeader("Content-Type", mime);
} else {
res.setHeader("Content-Type", "application/octet-stream");
}
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Content-Length", String(opened.stat.size));
const stream = opened.handle.createReadStream({ start: 0, autoClose: false });
const finishClose = () => {
void closeOpenedHandle();
};
stream.once("end", finishClose);
stream.once("close", finishClose);
stream.once("error", () => {
void closeOpenedHandle();
if (!res.headersSent) {
respondControlUiNotFound(res);
} else {
res.destroy();
}
});
res.once("close", finishClose);
stream.pipe(res);
return true;
} catch {
await closeOpenedHandle();
respondControlUiNotFound(res);
return true;
}
}
export function handleControlUiAvatarRequest(
req: IncomingMessage,
res: ServerResponse,
@@ -221,7 +444,7 @@ export function handleControlUiAvatarRequest(
}
function setStaticFileHeaders(res: ServerResponse, filePath: string) {
const ext = normalizeLowercaseStringOrEmpty(path.extname(filePath));
const ext = path.extname(filePath).toLowerCase();
res.setHeader("Content-Type", contentTypeForExt(ext));
// Static UI should never be cached aggressively while iterating; allow the
// browser to revalidate.
@@ -365,6 +588,16 @@ export function handleControlUiHttpRequest(
basePath,
assistantName: identity.name,
assistantAvatar: avatarValue ?? identity.avatar,
assistantAgentId: identity.agentId,
serverVersion: resolveRuntimeServiceVersion(process.env),
localMediaPreviewRoots: [...getAgentScopedMediaLocalRoots(config ?? {}, identity.agentId)],
embedSandbox:
config?.gateway?.controlUi?.embedSandbox === "trusted"
? "trusted"
: config?.gateway?.controlUi?.embedSandbox === "strict"
? "strict"
: "scripts",
allowExternalEmbedUrls: config?.gateway?.controlUi?.allowExternalEmbedUrls === true,
} satisfies ControlUiBootstrapConfig);
return true;
}
@@ -463,7 +696,7 @@ export function handleControlUiHttpRequest(
// against the same set of extensions that contentTypeForExt() recognises so
// that dotted SPA routes (e.g. /user/jane.doe, /v2.0) still get the
// client-side router fallback.
if (STATIC_ASSET_EXTENSIONS.has(normalizeLowercaseStringOrEmpty(path.extname(fileRel)))) {
if (STATIC_ASSET_EXTENSIONS.has(path.extname(fileRel).toLowerCase())) {
respondControlUiNotFound(res);
return true;
}

52
src/gateway/server-http.probe.test.ts
View File

@@ -8,6 +8,7 @@ import {
withGatewayServer,
} from "./server-http.test-harness.js";
import type { ReadinessChecker } from "./server/readiness.js";
import { withTempConfig } from "./test-temp-config.js";
describe("gateway OpenAI-compatible disabled HTTP routes", () => {
it("returns 404 when compat endpoints are disabled", async () => {
@@ -112,6 +113,57 @@ describe("gateway probe endpoints", () => {
});
});
it("hides readiness details when trusted-proxy auth violates browser origin policy", async () => {
const getReadiness: ReadinessChecker = () => ({
ready: false,
failing: ["discord", "telegram"],
uptimeMs: 8_000,
});
await withTempConfig({
prefix: "probe-remote-origin-rejected",
cfg: {
gateway: {
trustedProxies: ["10.0.0.1"],
controlUi: {
allowedOrigins: ["https://control.example"],
},
},
},
run: async () => {
await withGatewayServer({
prefix: "probe-remote-origin-rejected-server",
resolvedAuth: {
mode: "trusted-proxy",
allowTailscale: false,
trustedProxy: { userHeader: "x-forwarded-user" },
},
overrides: {
getReadiness,
},
run: async (server) => {
const req = createRequest({
path: "/ready",
remoteAddress: "10.0.0.1",
host: "gateway.test",
headers: {
origin: "https://evil.example",
forwarded: "for=203.0.113.10;proto=https;host=gateway.test",
"x-forwarded-user": "user@example.com",
"x-forwarded-proto": "https",
},
});
const { res, getBody } = createResponse();
await dispatchRequest(server, req, res);
expect(res.statusCode).toBe(503);
expect(JSON.parse(getBody())).toEqual({ ready: false });
},
});
},
});
});
it("returns typed internal error payload when readiness evaluation throws", async () => {
const getReadiness: ReadinessChecker = () => {
throw new Error("boom");

16
src/gateway/server-http.stages.test.ts
View File

@@ -19,7 +19,7 @@ describe("runGatewayHttpRequestStages", () => {
expect(await runGatewayHttpRequestStages(stages)).toBe(false);
});
it("skips a throwing stage and continues to subsequent stages", async () => {
it("skips a throwing stage marked continueOnError and continues to subsequent stages", async () => {
const stageC = vi.fn(() => true);
const consoleSpy = vi.spyOn(console, "error").mockImplementation(() => {});
@@ -27,6 +27,7 @@ describe("runGatewayHttpRequestStages", () => {
{ name: "a", run: () => false },
{
name: "broken-facade",
continueOnError: true,
run: () => {
throw new Error("Cannot find module '@slack/bolt'");
},
@@ -46,13 +47,14 @@ describe("runGatewayHttpRequestStages", () => {
consoleSpy.mockRestore();
});
it("skips a rejecting async stage and continues", async () => {
it("skips a rejecting async stage marked continueOnError and continues", async () => {
const stageC = vi.fn(() => true);
const consoleSpy = vi.spyOn(console, "error").mockImplementation(() => {});
const stages = [
{
name: "async-broken",
continueOnError: true,
run: async () => {
throw new Error("ERR_MODULE_NOT_FOUND");
},
@@ -72,9 +74,7 @@ describe("runGatewayHttpRequestStages", () => {
consoleSpy.mockRestore();
});
it("returns false when the only non-throwing stages do not handle", async () => {
const consoleSpy = vi.spyOn(console, "error").mockImplementation(() => {});
it("rethrows when a stage throws without continueOnError", async () => {
const stages = [
{
name: "broken",
@@ -85,10 +85,6 @@ describe("runGatewayHttpRequestStages", () => {
{ name: "unmatched", run: () => false },
];
const result = await runGatewayHttpRequestStages(stages);
expect(result).toBe(false);
consoleSpy.mockRestore();
await expect(runGatewayHttpRequestStages(stages)).rejects.toThrow("load failed");
});
});

72
src/gateway/server-http.ts
View File

@@ -17,7 +17,7 @@ import type { OpenClawConfig } from "../config/types.openclaw.js";
import type { createSubsystemLogger } from "../logging/subsystem.js";
import { resolveHookExternalContentSource as resolveHookExternalContentSourceFromSession } from "../security/external-content.js";
import { safeEqualSecret } from "../security/secret-equal.js";
import { normalizeLowercaseStringOrEmpty } from "../shared/string-coerce.js";
import { resolveAssistantIdentity } from "./assistant-identity.js";
import {
AUTH_RATE_LIMIT_SCOPE_HOOK_AUTH,
createAuthRateLimiter,
@@ -32,6 +32,7 @@ import {
} from "./auth.js";
import { normalizeCanvasScopedUrl } from "./canvas-capability.js";
import {
handleControlUiAssistantMediaRequest,
handleControlUiAvatarRequest,
handleControlUiHttpRequest,
type ControlUiRootState,
@@ -100,7 +101,8 @@ function resolveMappedHookExternalContentSource(params: {
payload: Record<string, unknown>;
sessionKey: string;
}) {
const payloadSource = normalizeLowercaseStringOrEmpty(params.payload.source);
const payloadSource =
typeof params.payload.source === "string" ? params.payload.source.trim().toLowerCase() : "";
if (params.subPath === "gmail" || payloadSource === "gmail") {
return "gmail" as const;
}
@@ -265,25 +267,12 @@ function writeUpgradeAuthFailure(
socket.write("HTTP/1.1 401 Unauthorized\r\nConnection: close\r\n\r\n");
}
function writeUpgradeServiceUnavailable(
socket: { write: (chunk: string) => void },
responseBody: string,
) {
socket.write(
"HTTP/1.1 503 Service Unavailable\r\n" +
"Connection: close\r\n" +
"Content-Type: text/plain; charset=utf-8\r\n" +
`Content-Length: ${Buffer.byteLength(responseBody, "utf8")}\r\n` +
"\r\n" +
responseBody,
);
}
export type HooksRequestHandler = (req: IncomingMessage, res: ServerResponse) => Promise<boolean>;
type GatewayHttpRequestStage = {
name: string;
run: () => Promise<boolean> | boolean;
continueOnError?: boolean;
};
export async function runGatewayHttpRequestStages(
@@ -295,10 +284,12 @@ export async function runGatewayHttpRequestStages(
return true;
}
} catch (err) {
if (!stage.continueOnError) {
throw err;
}
// Log and skip the failing stage so subsequent stages (control-ui,
// gateway-probes, etc.) remain reachable. A common trigger is a
// plugin-owned route/runtime code can still fail to load when an
// optional dependency is missing. Keep later stages reachable.
// gateway-probes, etc.) remain reachable. A common trigger is a
// plugin-owned route/runtime code still failing to load an optional dependency.
console.error(`[gateway-http] stage "${stage.name}" threw — skipping:`, err);
}
}
@@ -362,6 +353,7 @@ function buildPluginRequestStages(params: {
},
{
name: "plugin-http",
continueOnError: true,
run: () => {
const pathContext =
params.pluginPathContext ?? resolvePluginRoutePathContext(params.requestPath);
@@ -798,7 +790,7 @@ export function createGatewayHttpServer(opts: {
});
// Don't interfere with WebSocket upgrades; ws handles the 'upgrade' event.
if (normalizeLowercaseStringOrEmpty(req.headers.upgrade) === "websocket") {
if ((req.headers.upgrade ?? "").toLowerCase() === "websocket") {
return;
}
@@ -958,6 +950,19 @@ export function createGatewayHttpServer(opts: {
);
if (controlUiEnabled) {
requestStages.push({
name: "control-ui-assistant-media",
run: () =>
handleControlUiAssistantMediaRequest(req, res, {
basePath: controlUiBasePath,
config: configSnapshot,
agentId: resolveAssistantIdentity({ cfg: configSnapshot }).agentId,
auth: resolvedAuth,
trustedProxies,
allowRealIpFallback,
rateLimiter,
}),
});
requestStages.push({
name: "control-ui-avatar",
run: () =>
@@ -973,6 +978,7 @@ export function createGatewayHttpServer(opts: {
handleControlUiHttpRequest(req, res, {
basePath: controlUiBasePath,
config: configSnapshot,
agentId: resolveAssistantIdentity({ cfg: configSnapshot }).agentId,
root: controlUiRoot,
}),
});
@@ -1067,15 +1073,29 @@ export function attachGatewayUpgradeHandler(opts: {
}
}
const preauthBudgetKey = resolveRequestClientIp(req, trustedProxies, allowRealIpFallback);
// Keep startup upgrades inside the pre-auth budget until WS handlers attach.
if (!preauthConnectionBudget.acquire(preauthBudgetKey)) {
writeUpgradeServiceUnavailable(socket, "Too many unauthenticated sockets");
if (wss.listenerCount("connection") === 0) {
const responseBody = "Gateway websocket handlers unavailable";
socket.write(
"HTTP/1.1 503 Service Unavailable\r\n" +
"Connection: close\r\n" +
"Content-Type: text/plain; charset=utf-8\r\n" +
`Content-Length: ${Buffer.byteLength(responseBody, "utf8")}\r\n` +
"\r\n" +
responseBody,
);
socket.destroy();
return;
}
if (wss.listenerCount("connection") === 0) {
preauthConnectionBudget.release(preauthBudgetKey);
writeUpgradeServiceUnavailable(socket, "Gateway websocket handlers unavailable");
if (!preauthConnectionBudget.acquire(preauthBudgetKey)) {
const responseBody = "Too many unauthenticated sockets";
socket.write(
"HTTP/1.1 503 Service Unavailable\r\n" +
"Connection: close\r\n" +
"Content-Type: text/plain; charset=utf-8\r\n" +
`Content-Length: ${Buffer.byteLength(responseBody, "utf8")}\r\n` +
"\r\n" +
responseBody,
);
socket.destroy();
return;
}

386
src/gateway/server-methods/chat.directive-tags.test.ts
View File

@@ -9,15 +9,18 @@ import {
GATEWAY_CLIENT_MODES,
GATEWAY_CLIENT_NAMES,
} from "../protocol/client-info.js";
import type { ModelCatalogEntry } from "../../agents/model-catalog.types.js";
import { ErrorCodes } from "../protocol/index.js";
import { CHAT_SEND_SESSION_KEY_MAX_LENGTH } from "../protocol/schema/primitives.js";
import type { GatewayRequestContext } from "./types.js";
const mockState = vi.hoisted(() => ({
config: {} as Record<string, unknown>,
transcriptPath: "",
sessionId: "sess-1",
mainSessionKey: "main",
finalText: "[[reply_to_current]]",
finalPayload: null as { text?: string; mediaUrl?: string } | null,
dispatchedReplies: [] as Array<{
kind: "tool" | "block" | "final";
payload: { text?: string; mediaUrl?: string; mediaUrls?: string[] };
@@ -28,6 +31,8 @@ const mockState = vi.hoisted(() => ({
sessionEntry: {} as Record<string, unknown>,
lastDispatchCtx: undefined as MsgContext | undefined,
lastDispatchImages: undefined as Array<{ mimeType: string; data: string }> | undefined,
lastDispatchImageOrder: undefined as string[] | undefined,
modelCatalog: null as ModelCatalogEntry[] | null,
emittedTranscriptUpdates: [] as Array<{
sessionFile: string;
sessionKey?: string;
@@ -35,6 +40,7 @@ const mockState = vi.hoisted(() => ({
messageId?: string;
}>,
savedMediaResults: [] as Array<{ path: string; contentType?: string }>,
saveMediaError: null as Error | null,
savedMediaCalls: [] as Array<{ contentType?: string; subdir?: string; size: number }>,
saveMediaWait: null as Promise<void> | null,
activeSaveMediaCalls: 0,
@@ -60,7 +66,9 @@ vi.mock("../session-utils.js", async () => {
? { canonicalKey: mockState.sessionEntry.canonicalKey }
: {}),
cfg: {
...mockState.config,
session: {
...(mockState.config.session as Record<string, unknown> | undefined),
mainKey: mockState.mainSessionKey,
},
},
@@ -83,7 +91,7 @@ vi.mock("../../auto-reply/dispatch.js", () => ({
async (params: {
ctx: MsgContext;
dispatcher: {
sendFinalReply: (payload: { text: string }) => boolean;
sendFinalReply: (payload: { text?: string; mediaUrl?: string }) => boolean;
sendBlockReply: (payload: {
text?: string;
mediaUrl?: string;
@@ -100,10 +108,12 @@ vi.mock("../../auto-reply/dispatch.js", () => ({
replyOptions?: {
onAgentRunStart?: (runId: string) => void;
images?: Array<{ mimeType: string; data: string }>;
imageOrder?: string[];
};
}) => {
mockState.lastDispatchCtx = params.ctx;
mockState.lastDispatchImages = params.replyOptions?.images;
mockState.lastDispatchImageOrder = params.replyOptions?.imageOrder;
if (mockState.dispatchError) {
throw mockState.dispatchError;
}
@@ -125,7 +135,7 @@ vi.mock("../../auto-reply/dispatch.js", () => ({
});
}
} else {
params.dispatcher.sendFinalReply({ text: mockState.finalText });
params.dispatcher.sendFinalReply(mockState.finalPayload ?? { text: mockState.finalText });
}
params.dispatcher.markComplete();
await params.dispatcher.waitForIdle();
@@ -161,6 +171,10 @@ vi.mock("../../media/store.js", async () => {
if (mockState.saveMediaWait) {
await mockState.saveMediaWait;
}
if (mockState.saveMediaError) {
mockState.activeSaveMediaCalls -= 1;
throw mockState.saveMediaError;
}
mockState.savedMediaCalls.push({ contentType, subdir, size: buffer.byteLength });
const next = mockState.savedMediaResults.shift();
try {
@@ -267,6 +281,7 @@ function createChatContext(): Pick<
| "chatAbortControllers"
| "chatRunBuffers"
| "chatDeltaSentAt"
| "chatDeltaLastBroadcastLen"
| "chatAbortedRuns"
| "removeChatRun"
| "dedupe"
@@ -281,23 +296,25 @@ function createChatContext(): Pick<
chatAbortControllers: new Map(),
chatRunBuffers: new Map(),
chatDeltaSentAt: new Map(),
chatDeltaLastBroadcastLen: new Map(),
chatAbortedRuns: new Map(),
removeChatRun: vi.fn(),
dedupe: new Map(),
loadGatewayModelCatalog: async () => [
{
provider: "openai",
id: "gpt-5.4",
name: "GPT-5.4",
input: ["text", "image"],
},
{
provider: "anthropic",
id: "claude-opus-4-6",
name: "Claude Opus 4.6",
input: ["text", "image"],
},
],
loadGatewayModelCatalog: async () =>
mockState.modelCatalog ?? [
{
provider: "openai",
id: "gpt-5.4",
name: "GPT-5.4",
input: ["text", "image"],
},
{
provider: "anthropic",
id: "claude-opus-4-6",
name: "Claude Opus 4.6",
input: ["text", "image"],
},
],
registerToolEventRecipient: vi.fn(),
logGateway: {
warn: vi.fn(),
@@ -380,7 +397,9 @@ async function runNonStreamingChatSend(params: {
describe("chat directive tag stripping for non-streaming final payloads", () => {
afterEach(() => {
mockState.config = {};
mockState.finalText = "[[reply_to_current]]";
mockState.finalPayload = null;
mockState.dispatchedReplies = [];
mockState.dispatchError = null;
mockState.mainSessionKey = "main";
@@ -389,8 +408,11 @@ describe("chat directive tag stripping for non-streaming final payloads", () =>
mockState.sessionEntry = {};
mockState.lastDispatchCtx = undefined;
mockState.lastDispatchImages = undefined;
mockState.lastDispatchImageOrder = undefined;
mockState.modelCatalog = null;
mockState.emittedTranscriptUpdates = [];
mockState.savedMediaResults = [];
mockState.saveMediaError = null;
mockState.savedMediaCalls = [];
mockState.saveMediaWait = null;
mockState.activeSaveMediaCalls = 0;
@@ -1669,6 +1691,78 @@ describe("chat directive tag stripping for non-streaming final payloads", () =>
});
});
it("preserves offloaded attachment media paths in transcript order", async () => {
createTranscriptFixture("openclaw-chat-send-user-transcript-offloaded-");
mockState.finalText = "ok";
mockState.triggerAgentRunStart = true;
mockState.sessionEntry = {
modelProvider: "test-provider",
model: "vision-model",
};
mockState.modelCatalog = [
{
provider: "test-provider",
id: "vision-model",
name: "Vision model",
input: ["text", "image"],
},
];
mockState.savedMediaResults = [
{ path: "/tmp/offloaded-big.png", contentType: "image/png" },
{ path: "/tmp/chat-send-inline.png", contentType: "image/png" },
];
const respond = vi.fn();
const context = createChatContext();
const bigPng = Buffer.alloc(2_100_000);
bigPng.set([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a], 0);
await runNonStreamingChatSend({
context,
respond,
idempotencyKey: "idem-user-transcript-offloaded",
message: "edit both",
requestParams: {
attachments: [
{
mimeType: "image/png",
content:
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+aYoYAAAAASUVORK5CYII=",
},
{
mimeType: "image/png",
content: bigPng.toString("base64"),
},
],
},
expectBroadcast: false,
waitForCompletion: false,
});
await waitForAssertion(() => {
const userUpdate = mockState.emittedTranscriptUpdates.find(
(update) =>
typeof update.message === "object" &&
update.message !== null &&
(update.message as { role?: unknown }).role === "user",
);
const message = userUpdate?.message as
| {
MediaPath?: string;
MediaPaths?: string[];
MediaType?: string;
MediaTypes?: string[];
}
| undefined;
expect(message?.MediaPath).toBe("/tmp/chat-send-inline.png");
expect(message?.MediaPaths).toEqual([
"/tmp/chat-send-inline.png",
"/tmp/offloaded-big.png",
]);
expect(message?.MediaType).toBe("image/png");
expect(message?.MediaTypes).toEqual(["image/png", "image/png"]);
});
});
it("skips transcript media notes for ACP bridge clients", async () => {
createTranscriptFixture("openclaw-chat-send-user-transcript-acp-images-");
mockState.finalText = "ok";
@@ -1765,6 +1859,224 @@ describe("chat directive tag stripping for non-streaming final payloads", () =>
});
});
it("preserves media-only final replies in the final broadcast message", async () => {
createTranscriptFixture("openclaw-chat-send-media-only-final-");
mockState.finalPayload = { mediaUrl: "https://example.com/final.png" };
const respond = vi.fn();
const context = createChatContext();
const payload = await runNonStreamingChatSend({
context,
respond,
idempotencyKey: "idem-media-only-final",
});
expect(extractFirstTextBlock(payload)).toBe("MEDIA:https://example.com/final.png");
});
it("strips NO_REPLY from transcript text when final replies only carry media", async () => {
createTranscriptFixture("openclaw-chat-send-media-only-silent-final-");
mockState.finalPayload = {
text: "NO_REPLY",
mediaUrl: "https://example.com/final.png",
};
const respond = vi.fn();
const context = createChatContext();
const payload = await runNonStreamingChatSend({
context,
respond,
idempotencyKey: "idem-media-only-silent-final",
});
expect(extractFirstTextBlock(payload)).toBe("MEDIA:https://example.com/final.png");
});
it("drops image attachments for text-only session models", async () => {
createTranscriptFixture("openclaw-chat-send-text-only-attachments-");
mockState.finalText = "ok";
mockState.sessionEntry = {
modelProvider: "test-provider",
model: "text-only",
};
mockState.modelCatalog = [
{
provider: "test-provider",
id: "text-only",
name: "Text only",
input: ["text"],
},
];
const respond = vi.fn();
const context = createChatContext();
await runNonStreamingChatSend({
context,
respond,
idempotencyKey: "idem-text-only-attachments",
message: "describe image",
requestParams: {
attachments: [
{
mimeType: "image/png",
content:
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=",
},
],
},
expectBroadcast: false,
});
expect(mockState.lastDispatchImages).toBeUndefined();
expect(mockState.lastDispatchImageOrder).toBeUndefined();
});
it("resolves attachment image support from the session agent model", async () => {
createTranscriptFixture("openclaw-chat-send-agent-scoped-text-only-attachments-");
mockState.finalText = "ok";
mockState.config = {
agents: {
list: [
{
id: "vision",
default: true,
model: "test-provider/vision-model",
},
{
id: "writer",
model: "test-provider/text-only",
},
],
},
};
mockState.modelCatalog = [
{
provider: "test-provider",
id: "vision-model",
name: "Vision model",
input: ["text", "image"],
},
{
provider: "test-provider",
id: "text-only",
name: "Text only",
input: ["text"],
},
];
const respond = vi.fn();
const context = createChatContext();
await runNonStreamingChatSend({
context,
respond,
sessionKey: "agent:writer:main",
idempotencyKey: "idem-agent-scoped-text-only-attachments",
message: "describe image",
requestParams: {
attachments: [
{
mimeType: "image/png",
content:
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=",
},
],
},
expectBroadcast: false,
});
expect(mockState.lastDispatchImages).toBeUndefined();
expect(mockState.lastDispatchImageOrder).toBeUndefined();
});
it("passes imageOrder for mixed inline and offloaded chat.send attachments", async () => {
createTranscriptFixture("openclaw-chat-send-image-order-");
mockState.finalText = "ok";
mockState.sessionEntry = {
modelProvider: "test-provider",
model: "vision-model",
};
mockState.modelCatalog = [
{
provider: "test-provider",
id: "vision-model",
name: "Vision model",
input: ["text", "image"],
},
];
mockState.savedMediaResults = [{ path: "/tmp/offloaded-big.png", contentType: "image/png" }];
const respond = vi.fn();
const context = createChatContext();
const bigPng = Buffer.alloc(2_100_000);
bigPng.set([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a], 0);
await runNonStreamingChatSend({
context,
respond,
idempotencyKey: "idem-image-order",
message: "describe both",
requestParams: {
attachments: [
{
mimeType: "image/png",
content:
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=",
},
{
mimeType: "image/png",
content: bigPng.toString("base64"),
},
],
},
expectBroadcast: false,
});
expect(mockState.lastDispatchImages).toHaveLength(1);
expect(mockState.lastDispatchImageOrder).toEqual(["inline", "offloaded"]);
});
it("maps media offload failures to UNAVAILABLE in chat.send", async () => {
createTranscriptFixture("openclaw-chat-send-media-offload-error-");
mockState.sessionEntry = {
modelProvider: "test-provider",
model: "vision-model",
};
mockState.modelCatalog = [
{
provider: "test-provider",
id: "vision-model",
name: "Vision model",
input: ["text", "image"],
},
];
mockState.saveMediaError = new Error("disk full");
const respond = vi.fn();
const context = createChatContext();
const bigPng = Buffer.alloc(2_100_000);
bigPng.set([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a], 0);
await runNonStreamingChatSend({
context,
respond,
idempotencyKey: "idem-media-offload-error",
message: "describe image",
requestParams: {
attachments: [
{
mimeType: "image/png",
content: bigPng.toString("base64"),
},
],
},
waitFor: "none",
});
expect(respond).toHaveBeenCalledWith(
false,
undefined,
expect.objectContaining({ code: ErrorCodes.UNAVAILABLE }),
);
});
it("persists chat.send attachments one at a time", async () => {
createTranscriptFixture("openclaw-chat-send-image-serial-save-");
mockState.finalText = "ok";
@@ -1813,6 +2125,48 @@ describe("chat directive tag stripping for non-streaming final payloads", () =>
});
});
it("does not parse or offload attachments for stop commands", async () => {
createTranscriptFixture("openclaw-chat-send-stop-command-attachments-");
mockState.savedMediaResults = [
{ path: "/tmp/should-not-exist.png", contentType: "image/png" },
];
const respond = vi.fn();
const context = createChatContext();
context.chatAbortControllers.set("run-same-session", {
controller: new AbortController(),
sessionId: "sess-prev",
sessionKey: "main",
startedAtMs: Date.now(),
expiresAtMs: Date.now() + 10_000,
});
await runNonStreamingChatSend({
context,
respond,
idempotencyKey: "idem-stop-command-attachments",
message: "/stop",
requestParams: {
attachments: [
{
mimeType: "image/png",
content:
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=",
},
],
},
expectBroadcast: false,
waitFor: "none",
});
expect(mockState.savedMediaCalls).toEqual([]);
expect(mockState.lastDispatchImages).toBeUndefined();
expect(respond).toHaveBeenCalledWith(true, {
ok: true,
aborted: true,
runIds: ["run-same-session"],
});
});
it("emits a user transcript update when chat.send completes without an agent run", async () => {
createTranscriptFixture("openclaw-chat-send-user-transcript-no-run-");
mockState.finalText = "ok";

609
src/gateway/server-methods/chat.ts
View File

@@ -1,33 +1,28 @@
import fs from "node:fs";
import path from "node:path";
import { CURRENT_SESSION_VERSION, SessionManager } from "@mariozechner/pi-coding-agent";
import { resolveSendableOutboundReplyParts } from "openclaw/plugin-sdk/reply-payload";
import { resolveSessionAgentId } from "../../agents/agent-scope.js";
import { resolveThinkingDefault } from "../../agents/model-selection.js";
import { rewriteTranscriptEntriesInSessionFile } from "../../agents/pi-embedded-runner/transcript-rewrite.js";
import { resolveAgentTimeoutMs } from "../../agents/timeout.js";
import {
extractAssistantText as extractAssistantHistoryText,
hasAssistantPhaseMetadata,
} from "../../agents/tools/chat-history-text.js";
import { dispatchInboundMessage } from "../../auto-reply/dispatch.js";
import { createReplyDispatcher } from "../../auto-reply/reply/reply-dispatcher.js";
import type { MsgContext } from "../../auto-reply/templating.js";
import type { ReplyPayload } from "../../auto-reply/types.js";
import { extractCanvasFromText } from "../../chat/canvas-render.js";
import { resolveSessionFilePath } from "../../config/sessions.js";
import { formatErrorMessage } from "../../infra/errors.js";
import { jsonUtf8Bytes } from "../../infra/json-utf8-bytes.js";
import { isAudioFileName } from "../../media/mime.js";
import type { PromptImageOrderEntry } from "../../media/prompt-image-order.js";
import { type SavedMedia, saveMediaBuffer } from "../../media/store.js";
import { createChannelReplyPipeline } from "../../plugin-sdk/channel-reply-pipeline.js";
import { resolveAssistantMessagePhase } from "../../shared/chat-message-content.js";
import { normalizeInputProvenance, type InputProvenance } from "../../sessions/input-provenance.js";
import { resolveSendPolicy } from "../../sessions/send-policy.js";
import { parseAgentSessionKey } from "../../sessions/session-key-utils.js";
import { emitSessionTranscriptUpdate } from "../../sessions/transcript-events.js";
import { resolveAssistantMessagePhase } from "../../shared/chat-message-content.js";
import {
normalizeLowercaseStringOrEmpty,
normalizeOptionalString,
} from "../../shared/string-coerce.js";
import { isSuppressedControlReplyText } from "../control-reply-text.js";
import {
stripInlineDirectiveTagsForDisplay,
stripInlineDirectiveTagsFromMessageForDisplay,
@@ -47,13 +42,11 @@ import {
} from "../chat-abort.js";
import {
type ChatImageContent,
MediaOffloadError,
type OffloadedRef,
parseMessageWithAttachments,
} from "../chat-attachments.js";
import { stripEnvelopeFromMessage, stripEnvelopeFromMessages } from "../chat-sanitize.js";
import { augmentChatHistoryWithCliSessionImports } from "../cli-session-history.js";
import { isSuppressedControlReplyText } from "../control-reply-text.js";
import { ADMIN_SCOPE } from "../method-scopes.js";
import {
GATEWAY_CLIENT_CAPS,
@@ -90,6 +83,7 @@ import type {
GatewayRequestHandlerOptions,
GatewayRequestHandlers,
} from "./types.js";
import { MediaOffloadError } from "../chat-attachments.js";
type TranscriptAppendResult = {
ok: boolean;
@@ -156,6 +150,19 @@ const CHANNEL_AGNOSTIC_SESSION_SCOPES = new Set([
]);
const CHANNEL_SCOPED_SESSION_SHAPES = new Set(["direct", "dm", "group", "channel"]);
export function resolveEffectiveChatHistoryMaxChars(
cfg: { gateway?: { webchat?: { chatHistoryMaxChars?: number } } },
maxChars?: number,
): number {
if (typeof maxChars === "number") {
return maxChars;
}
if (typeof cfg.gateway?.webchat?.chatHistoryMaxChars === "number") {
return cfg.gateway.webchat.chatHistoryMaxChars;
}
return DEFAULT_CHAT_HISTORY_TEXT_MAX_CHARS;
}
type ChatSendDeliveryEntry = {
deliveryContext?: {
channel?: string;
@@ -199,6 +206,35 @@ type SideResultPayload = {
ts: number;
};
function buildTranscriptReplyText(payloads: ReplyPayload[]): string {
const chunks = payloads
.map((payload) => {
const parts = resolveSendableOutboundReplyParts(payload);
const lines: string[] = [];
if (typeof payload.replyToId === "string" && payload.replyToId.trim()) {
lines.push(`[[reply_to:${payload.replyToId.trim()}]]`);
} else if (payload.replyToCurrent) {
lines.push("[[reply_to_current]]");
}
const text = payload.text?.trim();
if (text && !isSuppressedControlReplyText(text)) {
lines.push(text);
}
for (const mediaUrl of parts.mediaUrls) {
const trimmed = mediaUrl.trim();
if (trimmed) {
lines.push(`MEDIA:${trimmed}`);
}
}
if (payload.audioAsVoice && parts.mediaUrls.some((mediaUrl) => isAudioFileName(mediaUrl))) {
lines.push("[[audio_as_voice]]");
}
return lines.join("\n").trim();
})
.filter(Boolean);
return chunks.join("\n\n").trim();
}
function resolveChatSendOriginatingRoute(params: {
client?: { mode?: string | null; id?: string | null } | null;
deliver?: boolean;
@@ -255,9 +291,9 @@ function resolveChatSendOriginatingRoute(params: {
.filter(Boolean);
const sessionScopeHead = sessionScopeParts[0];
const sessionChannelHint = normalizeMessageChannel(sessionScopeHead);
const normalizedSessionScopeHead = normalizeLowercaseStringOrEmpty(sessionScopeHead);
const normalizedSessionScopeHead = (sessionScopeHead ?? "").trim().toLowerCase();
const sessionPeerShapeCandidates = [sessionScopeParts[1], sessionScopeParts[2]]
.map((part) => normalizeLowercaseStringOrEmpty(part))
.map((part) => (part ?? "").trim().toLowerCase())
.filter(Boolean);
const isChannelAgnosticSessionScope = CHANNEL_AGNOSTIC_SESSION_SCOPES.has(
normalizedSessionScopeHead,
@@ -274,7 +310,7 @@ function resolveChatSendOriginatingRoute(params: {
const hasClientMetadata =
(typeof params.client?.mode === "string" && params.client.mode.trim().length > 0) ||
(typeof params.client?.id === "string" && params.client.id.trim().length > 0);
const configuredMainKey = normalizeLowercaseStringOrEmpty(params.mainKey ?? "main");
const configuredMainKey = (params.mainKey ?? "main").trim().toLowerCase();
const isConfiguredMainSessionScope =
normalizedSessionScopeHead.length > 0 && normalizedSessionScopeHead === configuredMainKey;
const canInheritConfiguredMainRoute =
@@ -368,17 +404,6 @@ function canInjectSystemProvenance(client: GatewayRequestHandlerOptions["client"
return scopes.includes(ADMIN_SCOPE);
}
/**
* Persist inline images and offloaded-ref media to the transcript media store.
*
* Inline images are re-saved from their base64 payload so that a stable
* filesystem path can be stored in the transcript. Offloaded refs are already
* on disk (saved by parseMessageWithAttachments); their SavedMedia metadata is
* synthesised directly from the OffloadedRef, avoiding a redundant write.
*
* Both sets are combined so that transcript media fields remain complete
* regardless of whether attachments were inlined or offloaded.
*/
async function persistChatSendImages(params: {
images: ChatImageContent[];
imageOrder: PromptImageOrderEntry[];
@@ -386,41 +411,56 @@ async function persistChatSendImages(params: {
client: GatewayRequestHandlerOptions["client"];
logGateway: GatewayRequestContext["logGateway"];
}): Promise<SavedMedia[]> {
if (isAcpBridgeClient(params.client)) {
if ((params.images.length === 0 && params.offloadedRefs.length === 0) || isAcpBridgeClient(params.client)) {
return [];
}
const saved: SavedMedia[] = [];
let inlineIndex = 0;
let offloadedIndex = 0;
for (const entry of params.imageOrder) {
if (entry === "offloaded") {
const ref = params.offloadedRefs[offloadedIndex++];
if (!ref) {
continue;
}
saved.push({
id: ref.id,
path: ref.path,
size: 0,
contentType: ref.mimeType,
});
continue;
}
const img = params.images[inlineIndex++];
if (!img) {
continue;
}
const inlineSaved: SavedMedia[] = [];
for (const img of params.images) {
try {
saved.push(await saveMediaBuffer(Buffer.from(img.data, "base64"), img.mimeType, "inbound"));
inlineSaved.push(await saveMediaBuffer(Buffer.from(img.data, "base64"), img.mimeType, "inbound"));
} catch (err) {
params.logGateway.warn(
`chat.send: failed to persist inbound image (${img.mimeType}): ${formatForLog(err)}`,
);
}
}
const offloadedSaved = params.offloadedRefs.map((ref) => ({
id: ref.id,
path: ref.path,
size: 0,
contentType: ref.mimeType,
}));
if (params.imageOrder.length === 0) {
return [...inlineSaved, ...offloadedSaved];
}
const saved: SavedMedia[] = [];
let inlineIndex = 0;
let offloadedIndex = 0;
for (const entry of params.imageOrder) {
if (entry === "inline") {
const inline = inlineSaved[inlineIndex++];
if (inline) {
saved.push(inline);
}
continue;
}
const offloaded = offloadedSaved[offloadedIndex++];
if (offloaded) {
saved.push(offloaded);
}
}
for (; inlineIndex < inlineSaved.length; inlineIndex++) {
const inline = inlineSaved[inlineIndex];
if (inline) {
saved.push(inline);
}
}
for (; offloadedIndex < offloadedSaved.length; offloadedIndex++) {
const offloaded = offloadedSaved[offloadedIndex];
if (offloaded) {
saved.push(offloaded);
}
}
return saved;
}
@@ -520,7 +560,7 @@ async function rewriteChatSendUserTurnMediaPaths(params: {
function truncateChatHistoryText(
text: string,
maxChars: number,
maxChars: number = DEFAULT_CHAT_HISTORY_TEXT_MAX_CHARS,
): { text: string; truncated: boolean } {
if (text.length <= maxChars) {
return { text, truncated: false };
@@ -531,30 +571,94 @@ function truncateChatHistoryText(
};
}
function isToolHistoryBlockType(type: unknown): boolean {
if (typeof type !== "string") {
return false;
}
const normalized = type.trim().toLowerCase();
return (
normalized === "toolcall" ||
normalized === "tool_call" ||
normalized === "tooluse" ||
normalized === "tool_use" ||
normalized === "toolresult" ||
normalized === "tool_result"
);
}
function extractChatHistoryBlockText(message: unknown): string | undefined {
if (!message || typeof message !== "object") {
return undefined;
}
const entry = message as Record<string, unknown>;
if (typeof entry.content === "string") {
return entry.content;
}
if (typeof entry.text === "string") {
return entry.text;
}
if (!Array.isArray(entry.content)) {
return undefined;
}
const textParts = entry.content
.map((block) => {
if (!block || typeof block !== "object") {
return undefined;
}
const typed = block as { text?: unknown; type?: unknown };
return typeof typed.text === "string" ? typed.text : undefined;
})
.filter((value): value is string => typeof value === "string");
return textParts.length > 0 ? textParts.join("\n") : undefined;
}
function sanitizeChatHistoryContentBlock(
block: unknown,
maxChars: number,
opts?: { preserveExactToolPayload?: boolean; maxChars?: number },
): { block: unknown; changed: boolean } {
if (!block || typeof block !== "object") {
return { block, changed: false };
}
const entry = { ...(block as Record<string, unknown>) };
let changed = false;
const preserveExactToolPayload =
opts?.preserveExactToolPayload === true || isToolHistoryBlockType(entry.type);
const maxChars = opts?.maxChars ?? DEFAULT_CHAT_HISTORY_TEXT_MAX_CHARS;
if (typeof entry.text === "string") {
const stripped = stripInlineDirectiveTagsForDisplay(entry.text);
const res = truncateChatHistoryText(stripped.text, maxChars);
entry.text = res.text;
changed ||= stripped.changed || res.truncated;
if (preserveExactToolPayload) {
entry.text = stripped.text;
changed ||= stripped.changed;
} else {
const res = truncateChatHistoryText(stripped.text, maxChars);
entry.text = res.text;
changed ||= stripped.changed || res.truncated;
}
}
if (typeof entry.content === "string") {
const stripped = stripInlineDirectiveTagsForDisplay(entry.content);
if (preserveExactToolPayload) {
entry.content = stripped.text;
changed ||= stripped.changed;
} else {
const res = truncateChatHistoryText(stripped.text, maxChars);
entry.content = res.text;
changed ||= stripped.changed || res.truncated;
}
}
if (typeof entry.partialJson === "string") {
const res = truncateChatHistoryText(entry.partialJson, maxChars);
entry.partialJson = res.text;
changed ||= res.truncated;
if (!preserveExactToolPayload) {
const res = truncateChatHistoryText(entry.partialJson, maxChars);
entry.partialJson = res.text;
changed ||= res.truncated;
}
}
if (typeof entry.arguments === "string") {
const res = truncateChatHistoryText(entry.arguments, maxChars);
entry.arguments = res.text;
changed ||= res.truncated;
if (!preserveExactToolPayload) {
const res = truncateChatHistoryText(entry.arguments, maxChars);
entry.arguments = res.text;
changed ||= res.truncated;
}
}
if (typeof entry.thinking === "string") {
const res = truncateChatHistoryText(entry.thinking, maxChars);
@@ -640,13 +744,23 @@ function sanitizeCost(raw: unknown): { total?: number } | undefined {
function sanitizeChatHistoryMessage(
message: unknown,
maxChars: number,
maxChars: number = DEFAULT_CHAT_HISTORY_TEXT_MAX_CHARS,
): { message: unknown; changed: boolean } {
if (!message || typeof message !== "object") {
return { message, changed: false };
}
const entry = { ...(message as Record<string, unknown>) };
let changed = false;
const role = typeof entry.role === "string" ? entry.role.toLowerCase() : "";
const preserveExactToolPayload =
role === "toolresult" ||
role === "tool_result" ||
role === "tool" ||
role === "function" ||
typeof entry.toolName === "string" ||
typeof entry.tool_name === "string" ||
typeof entry.toolCallId === "string" ||
typeof entry.tool_call_id === "string";
if ("details" in entry) {
delete entry.details;
@@ -688,35 +802,34 @@ function sanitizeChatHistoryMessage(
if (typeof entry.content === "string") {
const stripped = stripInlineDirectiveTagsForDisplay(entry.content);
const res = truncateChatHistoryText(stripped.text, maxChars);
entry.content = res.text;
changed ||= stripped.changed || res.truncated;
} else if (Array.isArray(entry.content)) {
const updated = entry.content.map((block) => sanitizeChatHistoryContentBlock(block, maxChars));
const sanitizedBlocks = updated.map((item) => item.block);
const hasPhaseMetadata = hasAssistantPhaseMetadata(entry);
if (hasPhaseMetadata) {
const stripped = stripInlineDirectiveTagsForDisplay(extractAssistantHistoryText(entry) ?? "");
if (preserveExactToolPayload) {
entry.content = stripped.text;
changed ||= stripped.changed;
} else {
const res = truncateChatHistoryText(stripped.text, maxChars);
const nonTextBlocks = sanitizedBlocks.filter(
(block) =>
!block || typeof block !== "object" || (block as { type?: unknown }).type !== "text",
);
entry.content = res.text
? [{ type: "text", text: res.text }, ...nonTextBlocks]
: nonTextBlocks;
changed = true;
} else if (updated.some((item) => item.changed)) {
entry.content = sanitizedBlocks;
entry.content = res.text;
changed ||= stripped.changed || res.truncated;
}
} else if (Array.isArray(entry.content)) {
const updated = entry.content.map((block) =>
sanitizeChatHistoryContentBlock(block, { preserveExactToolPayload, maxChars }),
);
if (updated.some((item) => item.changed)) {
entry.content = updated.map((item) => item.block);
changed = true;
}
}
if (typeof entry.text === "string") {
const stripped = stripInlineDirectiveTagsForDisplay(entry.text);
const res = truncateChatHistoryText(stripped.text, maxChars);
entry.text = res.text;
changed ||= stripped.changed || res.truncated;
if (preserveExactToolPayload) {
entry.text = stripped.text;
changed ||= stripped.changed;
} else {
const res = truncateChatHistoryText(stripped.text, maxChars);
entry.text = res.text;
changed ||= stripped.changed || res.truncated;
}
}
return { message: changed ? entry : message, changed };
@@ -729,7 +842,35 @@ function sanitizeChatHistoryMessage(
* dropping messages that carry real text alongside a stale `content: "NO_REPLY"`.
*/
function extractAssistantTextForSilentCheck(message: unknown): string | undefined {
return extractAssistantHistoryText(message);
if (!message || typeof message !== "object") {
return undefined;
}
const entry = message as Record<string, unknown>;
if (entry.role !== "assistant") {
return undefined;
}
if (typeof entry.text === "string") {
return entry.text;
}
if (typeof entry.content === "string") {
return entry.content;
}
if (!Array.isArray(entry.content) || entry.content.length === 0) {
return undefined;
}
const texts: string[] = [];
for (const block of entry.content) {
if (!block || typeof block !== "object") {
return undefined;
}
const typed = block as { type?: unknown; text?: unknown };
if (typed.type !== "text" || typeof typed.text !== "string") {
return undefined;
}
texts.push(typed.text);
}
return texts.length > 0 ? texts.join("\n") : undefined;
}
function hasAssistantNonTextContent(message: unknown): boolean {
@@ -749,7 +890,8 @@ function shouldDropAssistantHistoryMessage(message: unknown): boolean {
if (!message || typeof message !== "object") {
return false;
}
if ((message as { role?: unknown }).role !== "assistant") {
const entry = message as { role?: unknown };
if (entry.role !== "assistant") {
return false;
}
if (resolveAssistantMessagePhase(message) === "commentary") {
@@ -762,24 +904,22 @@ function shouldDropAssistantHistoryMessage(message: unknown): boolean {
return !hasAssistantNonTextContent(message);
}
export function sanitizeChatHistoryMessages(messages: unknown[], maxChars: number): unknown[] {
export function sanitizeChatHistoryMessages(
messages: unknown[],
maxChars: number = DEFAULT_CHAT_HISTORY_TEXT_MAX_CHARS,
): unknown[] {
if (messages.length === 0) {
return messages;
}
let changed = false;
const next: unknown[] = [];
for (const message of messages) {
// Drop raw control-token replies before any maxChars truncation can make
// an exact token look like partial user-visible text.
if (shouldDropAssistantHistoryMessage(message)) {
changed = true;
continue;
}
const res = sanitizeChatHistoryMessage(message, maxChars);
changed ||= res.changed;
// Drop assistant commentary-only entries and exact control replies, but
// keep mixed assistant entries that still carry non-text content. Run this
// again after sanitizing so display-only cleanup can still suppress stale tokens.
if (shouldDropAssistantHistoryMessage(res.message)) {
changed = true;
continue;
@@ -789,6 +929,152 @@ export function sanitizeChatHistoryMessages(messages: unknown[], maxChars: numbe
return changed ? next : messages;
}
function appendCanvasBlockToAssistantHistoryMessage(params: {
message: unknown;
preview: ReturnType<typeof extractCanvasFromText>;
rawText: string | null;
}): unknown {
const preview = params.preview;
if (!preview || !params.message || typeof params.message !== "object") {
return params.message;
}
const entry = params.message as Record<string, unknown>;
const baseContent = Array.isArray(entry.content)
? [...entry.content]
: typeof entry.content === "string"
? [{ type: "text", text: entry.content }]
: typeof entry.text === "string"
? [{ type: "text", text: entry.text }]
: [];
const alreadyPresent = baseContent.some((block) => {
if (!block || typeof block !== "object") {
return false;
}
const typed = block as { type?: unknown; preview?: unknown };
return (
typed.type === "canvas" &&
typed.preview &&
typeof typed.preview === "object" &&
(((typed.preview as { viewId?: unknown }).viewId &&
(typed.preview as { viewId?: unknown }).viewId === preview.viewId) ||
((typed.preview as { url?: unknown }).url &&
(typed.preview as { url?: unknown }).url === preview.url))
);
});
if (!alreadyPresent) {
baseContent.push({
type: "canvas",
preview,
rawText: params.rawText,
});
}
return {
...entry,
content: baseContent,
};
}
function messageContainsToolHistoryContent(message: unknown): boolean {
if (!message || typeof message !== "object") {
return false;
}
const entry = message as Record<string, unknown>;
if (
typeof entry.toolCallId === "string" ||
typeof entry.tool_call_id === "string" ||
typeof entry.toolName === "string" ||
typeof entry.tool_name === "string"
) {
return true;
}
if (!Array.isArray(entry.content)) {
return false;
}
return entry.content.some((block) => {
if (!block || typeof block !== "object") {
return false;
}
return isToolHistoryBlockType((block as { type?: unknown }).type);
});
}
export function augmentChatHistoryWithCanvasBlocks(messages: unknown[]): unknown[] {
if (messages.length === 0) {
return messages;
}
const next = [...messages];
let changed = false;
let lastAssistantIndex = -1;
let lastRenderableAssistantIndex = -1;
const pending: Array<{
preview: NonNullable<ReturnType<typeof extractCanvasFromText>>;
rawText: string | null;
}> = [];
for (let index = 0; index < next.length; index++) {
const message = next[index];
if (!message || typeof message !== "object") {
continue;
}
const entry = message as Record<string, unknown>;
const role = typeof entry.role === "string" ? entry.role.toLowerCase() : "";
if (role === "assistant") {
lastAssistantIndex = index;
if (!messageContainsToolHistoryContent(entry)) {
lastRenderableAssistantIndex = index;
if (pending.length > 0) {
let target = next[index];
for (const item of pending) {
target = appendCanvasBlockToAssistantHistoryMessage({
message: target,
preview: item.preview,
rawText: item.rawText,
});
}
next[index] = target;
pending.length = 0;
changed = true;
}
}
continue;
}
if (!messageContainsToolHistoryContent(entry)) {
continue;
}
const toolName =
typeof entry.toolName === "string"
? entry.toolName
: typeof entry.tool_name === "string"
? entry.tool_name
: undefined;
const text = extractChatHistoryBlockText(entry);
const preview = extractCanvasFromText(text, toolName);
if (!preview) {
continue;
}
pending.push({
preview,
rawText: text ?? null,
});
}
if (pending.length > 0) {
const targetIndex =
lastRenderableAssistantIndex >= 0 ? lastRenderableAssistantIndex : lastAssistantIndex;
if (targetIndex >= 0) {
let target = next[targetIndex];
for (const item of pending) {
target = appendCanvasBlockToAssistantHistoryMessage({
message: target,
preview: item.preview,
rawText: item.rawText,
});
}
next[targetIndex] = target;
changed = true;
}
}
return changed ? next : messages;
}
function buildOversizedHistoryPlaceholder(message?: unknown): Record<string, unknown> {
const role =
message &&
@@ -895,7 +1181,7 @@ function ensureTranscriptFile(params: { transcriptPath: string; sessionId: strin
});
return { ok: true };
} catch (err) {
return { ok: false, error: formatErrorMessage(err) };
return { ok: false, error: err instanceof Error ? err.message : String(err) };
}
}
@@ -919,9 +1205,8 @@ function transcriptHasIdempotencyKey(transcriptPath: string, idempotencyKey: str
function appendAssistantTranscriptMessage(params: {
message: string;
/** Rich Pi message blocks (text, embedded audio, etc.). Overrides plain `message` when set. */
content?: Array<Record<string, unknown>>;
label?: string;
content?: Array<Record<string, unknown>>;
sessionId: string;
storePath: string | undefined;
sessionFile?: string;
@@ -1042,13 +1327,18 @@ function createChatAbortOps(context: GatewayRequestContext): ChatAbortOps {
};
}
function normalizeOptionalText(value?: string | null): string | undefined {
const trimmed = value?.trim();
return trimmed || undefined;
}
function normalizeExplicitChatSendOrigin(
params: ChatSendExplicitOrigin,
): { ok: true; value?: ChatSendExplicitOrigin } | { ok: false; error: string } {
const originatingChannel = normalizeOptionalString(params.originatingChannel);
const originatingTo = normalizeOptionalString(params.originatingTo);
const accountId = normalizeOptionalString(params.accountId);
const messageThreadId = normalizeOptionalString(params.messageThreadId);
const originatingChannel = normalizeOptionalText(params.originatingChannel);
const originatingTo = normalizeOptionalText(params.originatingTo);
const accountId = normalizeOptionalText(params.accountId);
const messageThreadId = normalizeOptionalText(params.messageThreadId);
const hasAnyExplicitOriginField = Boolean(
originatingChannel || originatingTo || accountId || messageThreadId,
);
@@ -1084,8 +1374,8 @@ function resolveChatAbortRequester(
): ChatAbortRequester {
const scopes = Array.isArray(client?.connect?.scopes) ? client.connect.scopes : [];
return {
connId: normalizeOptionalString(client?.connId),
deviceId: normalizeOptionalString(client?.connect?.device?.id),
connId: normalizeOptionalText(client?.connId),
deviceId: normalizeOptionalText(client?.connect?.device?.id),
isAdmin: scopes.includes(ADMIN_SCOPE),
};
}
@@ -1097,8 +1387,8 @@ function canRequesterAbortChatRun(
if (requester.isAdmin) {
return true;
}
const ownerDeviceId = normalizeOptionalString(entry.ownerDeviceId);
const ownerConnId = normalizeOptionalString(entry.ownerConnId);
const ownerDeviceId = normalizeOptionalText(entry.ownerDeviceId);
const ownerConnId = normalizeOptionalText(entry.ownerConnId);
if (!ownerDeviceId && !ownerConnId) {
return true;
}
@@ -1269,25 +1559,14 @@ export const chatHandlers: GatewayRequestHandlers = {
);
return;
}
const {
sessionKey,
limit,
maxChars: rpcMaxChars,
} = params as {
const { sessionKey, limit, maxChars } = params as {
sessionKey: string;
limit?: number;
maxChars?: number;
};
const { cfg, storePath, entry } = loadSessionEntry(sessionKey);
const configMaxChars = cfg.gateway?.webchat?.chatHistoryMaxChars;
const effectiveMaxChars =
typeof rpcMaxChars === "number"
? rpcMaxChars
: typeof configMaxChars === "number"
? configMaxChars
: DEFAULT_CHAT_HISTORY_TEXT_MAX_CHARS;
const sessionAgentId = resolveSessionAgentId({ sessionKey, config: cfg });
const sessionId = entry?.sessionId;
const sessionAgentId = resolveSessionAgentId({ sessionKey, config: cfg });
const resolvedSessionModel = resolveSessionModelRef(cfg, entry, sessionAgentId);
const localMessages =
sessionId && storePath ? readSessionMessages(sessionId, storePath, entry?.sessionFile) : [];
@@ -1300,9 +1579,12 @@ export const chatHandlers: GatewayRequestHandlers = {
const defaultLimit = 200;
const requested = typeof limit === "number" ? limit : defaultLimit;
const max = Math.min(hardMax, requested);
const effectiveMaxChars = resolveEffectiveChatHistoryMaxChars(cfg, maxChars);
const sliced = rawMessages.length > max ? rawMessages.slice(-max) : rawMessages;
const sanitized = stripEnvelopeFromMessages(sliced);
const normalized = sanitizeChatHistoryMessages(sanitized, effectiveMaxChars);
const normalized = augmentChatHistoryWithCanvasBlocks(
sanitizeChatHistoryMessages(sanitized, effectiveMaxChars),
);
const maxHistoryBytes = getMaxChatHistoryMessagesBytes();
const perMessageHardCap = Math.min(CHAT_HISTORY_MAX_SINGLE_MESSAGE_BYTES, maxHistoryBytes);
const replaced = replaceOversizedChatHistoryMessages({
@@ -1320,16 +1602,11 @@ export const chatHandlers: GatewayRequestHandlers = {
}
let thinkingLevel = entry?.thinkingLevel;
if (!thinkingLevel) {
const sessionAgentId = resolveSessionAgentId({
sessionKey,
config: cfg,
});
const resolvedModel = resolveSessionModelRef(cfg, entry, sessionAgentId);
const catalog = await context.loadGatewayModelCatalog();
thinkingLevel = resolveThinkingDefault({
cfg,
provider: resolvedModel.provider,
model: resolvedModel.model,
provider: resolvedSessionModel.provider,
model: resolvedSessionModel.model,
catalog,
});
}
@@ -1510,18 +1787,16 @@ export const chatHandlers: GatewayRequestHandlers = {
);
return;
}
// Load session entry before attachment parsing so we can gate media-URI
// marker injection on the model's image capability. This prevents opaque
// media:// markers from leaking into prompts for text-only model runs.
const rawSessionKey = p.sessionKey;
const { cfg, entry, canonicalKey: sessionKey } = loadSessionEntry(rawSessionKey);
const agentId = resolveSessionAgentId({
sessionKey,
config: cfg,
});
let parsedMessage = inboundMessage;
let parsedImages: ChatImageContent[] = [];
let parsedImageOrder: PromptImageOrderEntry[] = [];
let parsedOffloadedRefs: OffloadedRef[] = [];
let imageOrder: PromptImageOrderEntry[] = [];
let offloadedRefs: OffloadedRef[] = [];
const timeoutMs = resolveAgentTimeoutMs({
cfg,
overrideMs: p.timeoutMs,
@@ -1578,16 +1853,13 @@ export const chatHandlers: GatewayRequestHandlers = {
});
return;
}
if (normalizedAttachments.length > 0) {
const sessionAgentId = resolveSessionAgentId({ sessionKey, config: cfg });
const modelRef = resolveSessionModelRef(cfg, entry, sessionAgentId);
const modelRef = resolveSessionModelRef(cfg, entry, agentId);
const supportsImages = await resolveGatewayModelSupportsImages({
loadGatewayModelCatalog: context.loadGatewayModelCatalog,
provider: modelRef.provider,
model: modelRef.model,
});
try {
const parsed = await parseMessageWithAttachments(inboundMessage, normalizedAttachments, {
maxBytes: 5_000_000,
@@ -1596,19 +1868,14 @@ export const chatHandlers: GatewayRequestHandlers = {
});
parsedMessage = parsed.message;
parsedImages = parsed.images;
parsedImageOrder = parsed.imageOrder;
parsedOffloadedRefs = parsed.offloadedRefs;
imageOrder = parsed.imageOrder;
offloadedRefs = parsed.offloadedRefs;
} catch (err) {
// MediaOffloadError indicates a server-side storage fault (ENOSPC, EPERM,
// etc.). All other errors are client-side input validation failures.
// Map them to different HTTP status codes so callers can retry server
// faults without treating them as bad requests.
const isServerFault = err instanceof MediaOffloadError;
respond(
false,
undefined,
errorShape(
isServerFault ? ErrorCodes.UNAVAILABLE : ErrorCodes.INVALID_REQUEST,
err instanceof MediaOffloadError ? ErrorCodes.UNAVAILABLE : ErrorCodes.INVALID_REQUEST,
String(err),
),
);
@@ -1624,23 +1891,18 @@ export const chatHandlers: GatewayRequestHandlers = {
sessionKey: rawSessionKey,
startedAtMs: now,
expiresAtMs: resolveChatRunExpiresAtMs({ now, timeoutMs }),
ownerConnId: normalizeOptionalString(client?.connId),
ownerDeviceId: normalizeOptionalString(client?.connect?.device?.id),
ownerConnId: normalizeOptionalText(client?.connId),
ownerDeviceId: normalizeOptionalText(client?.connect?.device?.id),
});
const ackPayload = {
runId: clientRunId,
status: "started" as const,
};
respond(true, ackPayload, undefined, { runId: clientRunId });
// Persist both inline images and already-offloaded refs to the media
// store so that transcript media fields remain complete for all attachment
// sizes. Offloaded refs are already on disk; persistChatSendImages converts
// their metadata without re-writing the files.
const persistedImagesPromise = persistChatSendImages({
images: parsedImages,
imageOrder: parsedImageOrder,
offloadedRefs: parsedOffloadedRefs,
imageOrder,
offloadedRefs,
client,
logGateway: context.logGateway,
});
@@ -1671,7 +1933,7 @@ export const chatHandlers: GatewayRequestHandlers = {
});
// Inject timestamp so agents know the current date/time.
// Only BodyForAgent gets the timestamp — Body stays raw for UI display.
// See: https://github.com/openclaw/openclaw/issues/3658
// See: https://github.com/moltbot/moltbot/issues/3658
const stampedMessage = injectTimestamp(messageForAgent, timestampOptsFromConfig(cfg));
const ctx: MsgContext = {
@@ -1698,10 +1960,6 @@ export const chatHandlers: GatewayRequestHandlers = {
GatewayClientScopes: client?.connect?.scopes ?? [],
};
const agentId = resolveSessionAgentId({
sessionKey,
config: cfg,
});
const { onModelSelected, ...replyPipeline } = createChannelReplyPipeline({
cfg,
agentId,
@@ -1843,7 +2101,7 @@ export const chatHandlers: GatewayRequestHandlers = {
runId: clientRunId,
abortSignal: abortController.signal,
images: parsedImages.length > 0 ? parsedImages : undefined,
imageOrder: parsedImageOrder.length > 0 ? parsedImageOrder : undefined,
imageOrder: imageOrder.length > 0 ? imageOrder : undefined,
onAgentRunStart: (runId) => {
agentRunStarted = true;
void emitUserTranscriptUpdate();
@@ -1898,33 +2156,18 @@ export const chatHandlers: GatewayRequestHandlers = {
sessionKey,
});
} else {
const finalPayloads = deliveredReplies
const combinedReply = buildTranscriptReplyText(
deliveredReplies
.filter((entry) => entry.kind === "final")
.map((entry) => entry.payload);
const combinedReply = finalPayloads
.map((part) => part.text?.trim() ?? "")
.filter(Boolean)
.join("\n\n")
.trim();
const audioBlocks = buildWebchatAudioContentBlocksFromReplyPayloads(finalPayloads);
const assistantContent: Array<Record<string, unknown>> = [];
if (combinedReply) {
assistantContent.push({ type: "text", text: combinedReply });
} else if (audioBlocks.length > 0) {
assistantContent.push({ type: "text", text: "Audio reply" });
}
assistantContent.push(...audioBlocks);
.map((entry) => entry.payload)
);
let message: Record<string, unknown> | undefined;
if (assistantContent.length > 0) {
if (combinedReply) {
const { storePath: latestStorePath, entry: latestEntry } =
loadSessionEntry(sessionKey);
const sessionId = latestEntry?.sessionId ?? entry?.sessionId ?? clientRunId;
const transcriptFallbackText =
combinedReply || (audioBlocks.length > 0 ? "Audio reply" : "");
const appended = appendAssistantTranscriptMessage({
message: transcriptFallbackText,
content: assistantContent,
message: combinedReply,
sessionId,
storePath: latestStorePath,
sessionFile: latestEntry?.sessionFile,
@@ -1940,7 +2183,7 @@ export const chatHandlers: GatewayRequestHandlers = {
const now = Date.now();
message = {
role: "assistant",
content: assistantContent,
content: [{ type: "text", text: combinedReply }],
timestamp: now,
// Keep this compatible with Pi stopReason enums even though this message isn't
// persisted to the transcript due to the append failure.

108
src/gateway/server-methods/server-methods.test.ts
View File

@@ -16,7 +16,13 @@ import { validateExecApprovalRequestParams } from "../protocol/index.js";
import { waitForAgentJob } from "./agent-job.js";
import { injectTimestamp, timestampOptsFromConfig } from "./agent-timestamp.js";
import { normalizeRpcAttachmentsToChatAttachments } from "./attachment-normalize.js";
import { sanitizeChatSendMessageInput } from "./chat.js";
import {
DEFAULT_CHAT_HISTORY_TEXT_MAX_CHARS,
augmentChatHistoryWithCanvasBlocks,
resolveEffectiveChatHistoryMaxChars,
sanitizeChatHistoryMessages,
sanitizeChatSendMessageInput,
} from "./chat.js";
import { createExecApprovalHandlers } from "./exec-approval.js";
import { logsHandlers } from "./logs.js";
@@ -111,6 +117,39 @@ describe("waitForAgentJob", () => {
});
});
describe("augmentChatHistoryWithCanvasBlocks", () => {
it("ignores user messages that merely contain canvas-shaped text", () => {
const previewJson = JSON.stringify({
kind: "canvas",
view: {
backend: "canvas",
id: "cv_user_text",
url: "/__openclaw__/canvas/documents/cv_user_text/index.html",
title: "User pasted preview",
preferred_height: 240,
},
presentation: {
target: "assistant_message",
},
});
const messages = [
{
role: "user",
content: previewJson,
timestamp: 1,
},
{
role: "assistant",
content: "Plain assistant reply",
timestamp: 2,
},
];
expect(augmentChatHistoryWithCanvasBlocks(messages)).toEqual(messages);
});
});
describe("injectTimestamp", () => {
beforeEach(() => {
vi.useFakeTimers();
@@ -223,6 +262,73 @@ describe("injectTimestamp", () => {
});
});
describe("sanitizeChatHistoryMessages", () => {
it("drops commentary-only assistant entries when phase exists only in textSignature", () => {
const result = sanitizeChatHistoryMessages([
{
role: "user",
content: [{ type: "text", text: "hello" }],
timestamp: 1,
},
{
role: "assistant",
content: [
{
type: "text",
text: "thinking like caveman",
textSignature: JSON.stringify({ v: 1, id: "msg_commentary", phase: "commentary" }),
},
],
timestamp: 2,
},
{
role: "assistant",
content: [{ type: "text", text: "real reply" }],
timestamp: 3,
},
]);
expect(result).toEqual([
{
role: "user",
content: [{ type: "text", text: "hello" }],
timestamp: 1,
},
{
role: "assistant",
content: [{ type: "text", text: "real reply" }],
timestamp: 3,
},
]);
});
});
describe("resolveEffectiveChatHistoryMaxChars", () => {
it("uses gateway.webchat.chatHistoryMaxChars when RPC maxChars is absent", () => {
expect(
resolveEffectiveChatHistoryMaxChars(
{ gateway: { webchat: { chatHistoryMaxChars: 123 } } },
undefined,
),
).toBe(123);
});
it("prefers RPC maxChars over config", () => {
expect(
resolveEffectiveChatHistoryMaxChars(
{ gateway: { webchat: { chatHistoryMaxChars: 123 } } },
45,
),
).toBe(45);
});
it("falls back to the default hardcoded limit", () => {
expect(resolveEffectiveChatHistoryMaxChars({}, undefined)).toBe(
DEFAULT_CHAT_HISTORY_TEXT_MAX_CHARS,
);
});
});
describe("timestampOptsFromConfig", () => {
it.each([
{

16
src/gateway/server.canvas-auth.test.ts
View File

@@ -118,7 +118,7 @@ function makeWsClient(params: {
connId: string;
clientIp: string;
role: "node" | "operator";
mode: "node" | "backend";
mode: "node" | "backend" | "webchat";
canvasCapability?: string;
canvasCapabilityExpiresAtMs?: number;
}): GatewayWsClient {
@@ -246,7 +246,7 @@ describe("gateway canvas host auth", () => {
handleHttpRequest: allowCanvasHostHttp,
run: async ({ listener, clients }) => {
const host = "127.0.0.1";
const operatorOnlyCapability = "operator-only";
const webchatCapability = "webchat-cap";
const expiredNodeCapability = "expired-node";
const activeNodeCapability = "active-node";
const activeCanvasPath = scopedCanvasPath(activeNodeCapability, `${CANVAS_HOST_PATH}/`);
@@ -264,19 +264,19 @@ describe("gateway canvas host auth", () => {
clients.add(
makeWsClient({
connId: "c-operator",
connId: "c-webchat",
clientIp: "192.168.1.10",
role: "operator",
mode: "backend",
canvasCapability: operatorOnlyCapability,
mode: "webchat",
canvasCapability: webchatCapability,
canvasCapabilityExpiresAtMs: Date.now() + 60_000,
}),
);
const operatorCapabilityBlocked = await fetchCanvas(
`http://${host}:${listener.port}${scopedCanvasPath(operatorOnlyCapability, `${CANVAS_HOST_PATH}/`)}`,
const webchatCapabilityAllowed = await fetchCanvas(
`http://${host}:${listener.port}${scopedCanvasPath(webchatCapability, `${CANVAS_HOST_PATH}/`)}`,
);
expect(operatorCapabilityBlocked.status).toBe(401);
expect(webchatCapabilityAllowed.status).toBe(200);
clients.add(
makeWsClient({

26
src/gateway/server.hooks.test.ts
View File

@@ -450,6 +450,32 @@ describe("gateway server hooks", () => {
});
});
test("rejects mapped hook session rebinding into a disallowed target-agent prefix", async () => {
testState.hooksConfig = {
enabled: true,
token: HOOK_TOKEN,
allowRequestSessionKey: true,
allowedSessionKeyPrefixes: ["hook:", "agent:main:"],
mappings: [
{
match: { path: "mapped-rebind-denied" },
action: "agent",
agentId: "hooks",
messageTemplate: "Mapped: {{payload.subject}}",
sessionKey: "agent:main:slack:channel:c123",
},
],
};
setMainAndHooksAgents();
await withGatewayServer(async ({ port }) => {
const denied = await postHook(port, "/hooks/mapped-rebind-denied", { subject: "hello" });
expect(denied.status).toBe(400);
const body = (await denied.json()) as { error?: string };
expect(body.error).toContain("sessionKey must start with one of");
expect(cronIsolatedRun).not.toHaveBeenCalled();
});
});
test("dedupes repeated /hooks/agent deliveries by idempotency key", async () => {
testState.hooksConfig = { enabled: true, token: HOOK_TOKEN };
await withGatewayServer(async ({ port }) => {

15
src/gateway/server/http-auth.ts
View File

@@ -9,7 +9,6 @@ import {
} from "../auth.js";
import { CANVAS_CAPABILITY_TTL_MS } from "../canvas-capability.js";
import { getBearerToken, resolveHttpBrowserOriginPolicy } from "../http-utils.js";
import { GATEWAY_CLIENT_MODES, normalizeGatewayClientMode } from "../protocol/client-info.js";
import type { GatewayWsClient } from "./ws-types.js";
export function isCanvasPath(pathname: string): boolean {
@@ -22,22 +21,12 @@ export function isCanvasPath(pathname: string): boolean {
);
}
function isNodeWsClient(client: GatewayWsClient): boolean {
if (client.connect.role === "node") {
return true;
}
return normalizeGatewayClientMode(client.connect.client.mode) === GATEWAY_CLIENT_MODES.NODE;
}
function hasAuthorizedNodeWsClientForCanvasCapability(
function hasAuthorizedWsClientForCanvasCapability(
clients: Set<GatewayWsClient>,
capability: string,
): boolean {
const nowMs = Date.now();
for (const client of clients) {
if (!isNodeWsClient(client)) {
continue;
}
if (!client.canvasCapability || !client.canvasCapabilityExpiresAtMs) {
continue;
}
@@ -95,7 +84,7 @@ export async function authorizeCanvasRequest(params: {
lastAuthFailure = authResult;
}
if (canvasCapability && hasAuthorizedNodeWsClientForCanvasCapability(clients, canvasCapability)) {
if (canvasCapability && hasAuthorizedWsClientForCanvasCapability(clients, canvasCapability)) {
return { ok: true };
}
return lastAuthFailure ?? { ok: false, reason: "unauthorized" };

103
src/gateway/server/ws-connection.test.ts Normal file
View File

@@ -0,0 +1,103 @@
import { EventEmitter } from "node:events";
import { beforeEach, describe, expect, it, vi } from "vitest";
import type { WebSocketServer } from "ws";
import type { ResolvedGatewayAuth } from "../auth.js";
const { attachGatewayWsMessageHandlerMock } = vi.hoisted(() => ({
attachGatewayWsMessageHandlerMock: vi.fn(),
}));
vi.mock("./ws-connection/message-handler.js", () => ({
attachGatewayWsMessageHandler: attachGatewayWsMessageHandlerMock,
}));
import { attachGatewayWsConnectionHandler } from "./ws-connection.js";
import { resolveSharedGatewaySessionGeneration } from "./ws-shared-generation.js";
function createLogger() {
return {
debug: vi.fn(),
info: vi.fn(),
warn: vi.fn(),
error: vi.fn(),
};
}
function createResolvedAuth(token: string): ResolvedGatewayAuth {
return {
mode: "token",
allowTailscale: false,
token,
};
}
describe("attachGatewayWsConnectionHandler", () => {
beforeEach(() => {
attachGatewayWsMessageHandlerMock.mockReset();
});
it("threads current auth getters into the handshake handler instead of a stale snapshot", () => {
const listeners = new Map<string, (...args: unknown[]) => void>();
const wss = {
on: vi.fn((event: string, handler: (...args: unknown[]) => void) => {
listeners.set(event, handler);
}),
} as unknown as WebSocketServer;
const socket = Object.assign(new EventEmitter(), {
_socket: {
remoteAddress: "127.0.0.1",
remotePort: 1234,
localAddress: "127.0.0.1",
localPort: 5678,
},
send: vi.fn(),
close: vi.fn(),
});
const upgradeReq = {
headers: { host: "127.0.0.1:19001" },
socket: { localAddress: "127.0.0.1" },
};
const initialAuth = createResolvedAuth("token-before");
let currentAuth = initialAuth;
attachGatewayWsConnectionHandler({
wss,
clients: new Set(),
preauthConnectionBudget: { release: vi.fn() } as never,
port: 19001,
canvasHostEnabled: false,
resolvedAuth: initialAuth,
getResolvedAuth: () => currentAuth,
gatewayMethods: [],
events: [],
logGateway: createLogger() as never,
logHealth: createLogger() as never,
logWsControl: createLogger() as never,
extraHandlers: {},
broadcast: vi.fn(),
buildRequestContext: () =>
({
unsubscribeAllSessionEvents: vi.fn(),
nodeRegistry: { unregister: vi.fn() },
nodeUnsubscribeAll: vi.fn(),
}) as never,
});
const onConnection = listeners.get("connection");
expect(onConnection).toBeTypeOf("function");
onConnection?.(socket, upgradeReq);
expect(attachGatewayWsMessageHandlerMock).toHaveBeenCalledTimes(1);
const passed = attachGatewayWsMessageHandlerMock.mock.calls[0]?.[0] as {
getResolvedAuth: () => ResolvedGatewayAuth;
getRequiredSharedGatewaySessionGeneration?: () => string | undefined;
};
currentAuth = createResolvedAuth("token-after");
expect(passed.getResolvedAuth()).toMatchObject({ token: "token-after" });
expect(passed.getRequiredSharedGatewaySessionGeneration?.()).toBe(
resolveSharedGatewaySessionGeneration(currentAuth),
);
});
});

68
src/gateway/server/ws-connection/message-handler.ts
View File

@@ -2,6 +2,7 @@ import type { IncomingMessage } from "node:http";
import os from "node:os";
import type { WebSocket } from "ws";
import { loadConfig } from "../../../config/config.js";
import type { ResolvedGatewayAuth } from "../../auth.js";
import {
getBoundDeviceBootstrapProfile,
getDeviceBootstrapTokenProfile,
@@ -40,7 +41,6 @@ import {
type DeviceBootstrapProfile,
} from "../../../shared/device-bootstrap-profile.js";
import { roleScopesAllow } from "../../../shared/operator-scope-compat.js";
import { normalizeOptionalString } from "../../../shared/string-coerce.js";
import {
isBrowserOperatorUiClient,
isGatewayCliClient,
@@ -49,7 +49,7 @@ import {
} from "../../../utils/message-channel.js";
import { resolveRuntimeServiceVersion } from "../../../version.js";
import type { AuthRateLimiter } from "../../auth-rate-limit.js";
import type { GatewayAuthResult, ResolvedGatewayAuth } from "../../auth.js";
import type { GatewayAuthResult } from "../../auth.js";
import { isLocalDirectRequest } from "../../auth.js";
import {
buildCanvasScopedHostUrl,
@@ -173,7 +173,7 @@ export function attachGatewayWsMessageHandler(params: {
canvasHostUrl?: string;
connectNonce: string;
getResolvedAuth: () => ResolvedGatewayAuth;
getRequiredSharedGatewaySessionGeneration: () => string | undefined;
getRequiredSharedGatewaySessionGeneration?: () => string | undefined;
/** Optional rate limiter for auth brute-force protection. */
rateLimiter?: AuthRateLimiter;
/** Browser-origin fallback limiter (loopback is never exempt). */
@@ -267,7 +267,6 @@ export function attachGatewayWsMessageHandler(params: {
const hasUntrustedProxyHeaders = hasProxyHeaders && !remoteIsTrustedProxy;
const hostIsLocalish = isLocalishHost(requestHost);
const isLocalClient = isLocalDirectRequest(upgradeReq, trustedProxies, allowRealIpFallback);
const reportedClientIp =
isLocalClient || hasUntrustedProxyHeaders
? undefined
@@ -392,6 +391,7 @@ export function attachGatewayWsMessageHandler(params: {
const frame = parsed;
const connectParams = frame.params as ConnectParams;
const resolvedAuth = getResolvedAuth();
const clientLabel = connectParams.client.displayName ?? connectParams.client.id;
const clientMeta = {
client: connectParams.client.id,
@@ -458,7 +458,6 @@ export function attachGatewayWsMessageHandler(params: {
const isControlUi = isOperatorUiClient(connectParams.client);
const isBrowserOperatorUi = isBrowserOperatorUiClient(connectParams.client);
const isWebchat = isWebchatConnect(connectParams);
const resolvedAuth = getResolvedAuth();
if (enforceOriginCheckForAnyClient || isBrowserOperatorUi || isWebchat) {
const hostHeaderOriginFallbackEnabled =
configSnapshot.gateway?.controlUi?.dangerouslyAllowHostHeaderOriginFallback === true;
@@ -681,7 +680,7 @@ export function attachGatewayWsMessageHandler(params: {
rejectDeviceAuthInvalid("device-signature-stale", "device signature expired");
return;
}
const providedNonce = normalizeOptionalString(device.nonce) ?? "";
const providedNonce = typeof device.nonce === "string" ? device.nonce.trim() : "";
if (!providedNonce) {
rejectDeviceAuthInvalid("device-nonce-missing", "device nonce required");
return;
@@ -744,14 +743,15 @@ export function attachGatewayWsMessageHandler(params: {
rejectUnauthorized(authResult);
return;
}
const sharedGatewaySessionGeneration =
authMethod === "token" || authMethod === "password"
? resolveSharedGatewaySessionGeneration(resolvedAuth)
: undefined;
if (authMethod === "token" || authMethod === "password") {
const sharedGatewaySessionGeneration =
resolveSharedGatewaySessionGeneration(resolvedAuth);
const requiredSharedGatewaySessionGeneration =
getRequiredSharedGatewaySessionGeneration();
if (sharedGatewaySessionGeneration !== requiredSharedGatewaySessionGeneration) {
getRequiredSharedGatewaySessionGeneration?.();
if (
requiredSharedGatewaySessionGeneration !== undefined &&
sharedGatewaySessionGeneration !== requiredSharedGatewaySessionGeneration
) {
setCloseCause("gateway-auth-rotated", {
authGenerationStale: true,
});
@@ -891,9 +891,6 @@ export function attachGatewayWsMessageHandler(params: {
isWebchat,
reason,
});
// QR bootstrap onboarding stays single-use, but the first node bootstrap handshake
// should seed bounded device tokens and only consume the bootstrap token once the
// hello-ok path succeeds so reconnects can recover from pre-hello failures.
const allowSilentBootstrapPairing =
authMethod === "bootstrap-token" &&
reason === "not-paired" &&
@@ -1013,10 +1010,6 @@ export function attachGatewayWsMessageHandler(params: {
const isPaired = paired?.publicKey === devicePublicKey;
if (!isPaired) {
if (!(skipLocalBackendSelfPairing || skipControlUiPairingForDevice)) {
// Initial local backend/control-ui self-pairing can bypass the
// pairing prompt, but only while the device is still unpaired.
// Once a device is paired, reconnects must stay inside the
// approved role/scope baseline below.
const ok = await requirePairing("not-paired", paired);
if (!ok) {
return;
@@ -1212,11 +1205,14 @@ export function attachGatewayWsMessageHandler(params: {
snapshot.health = cachedHealth;
snapshot.stateVersion.health = getHealthVersion();
}
const canvasCapability =
role === "node" && canvasHostUrl ? mintCanvasCapabilityToken() : undefined;
const canvasCapability = canvasHostUrl ? mintCanvasCapabilityToken() : undefined;
const canvasCapabilityExpiresAtMs = canvasCapability
? Date.now() + CANVAS_CAPABILITY_TTL_MS
: undefined;
const usesSharedGatewayAuth = authMethod === "token" || authMethod === "password";
const sharedGatewaySessionGeneration = usesSharedGatewayAuth
? resolveSharedGatewaySessionGeneration(resolvedAuth)
: undefined;
const scopedCanvasHostUrl =
canvasHostUrl && canvasCapability
? (buildCanvasScopedHostUrl(canvasHostUrl, canvasCapability) ?? canvasHostUrl)
@@ -1254,7 +1250,7 @@ export function attachGatewayWsMessageHandler(params: {
socket,
connect: connectParams,
connId,
usesSharedGatewayAuth: authMethod === "token" || authMethod === "password",
usesSharedGatewayAuth,
sharedGatewaySessionGeneration,
presenceKey,
clientIp: reportedClientIp,
@@ -1271,7 +1267,7 @@ export function attachGatewayWsMessageHandler(params: {
remoteIp: reportedClientIp,
});
const instanceIdRaw = connectParams.client.instanceId;
const instanceId = normalizeOptionalString(instanceIdRaw) ?? "";
const instanceId = typeof instanceIdRaw === "string" ? instanceIdRaw.trim() : "";
const nodeIdsForPairing = new Set<string>([nodeSession.nodeId]);
if (instanceId) {
nodeIdsForPairing.add(instanceId);
@@ -1369,17 +1365,6 @@ export function attachGatewayWsMessageHandler(params: {
return;
}
if (client.usesSharedGatewayAuth) {
const requiredSharedGatewaySessionGeneration = getRequiredSharedGatewaySessionGeneration();
if (client.sharedGatewaySessionGeneration !== requiredSharedGatewaySessionGeneration) {
setCloseCause("gateway-auth-rotated", {
authGenerationStale: true,
});
close(4001, "gateway auth changed");
return;
}
}
// After handshake, accept only req frames
if (!validateRequestFrame(parsed)) {
send({
@@ -1395,6 +1380,21 @@ export function attachGatewayWsMessageHandler(params: {
}
const req = parsed;
logWs("in", "req", { connId, id: req.id, method: req.method });
if (client.usesSharedGatewayAuth) {
const requiredSharedGatewaySessionGeneration =
getRequiredSharedGatewaySessionGeneration?.();
if (
requiredSharedGatewaySessionGeneration !== undefined &&
client.sharedGatewaySessionGeneration !== requiredSharedGatewaySessionGeneration
) {
setCloseCause("gateway-auth-rotated", {
authGenerationStale: true,
method: req.method,
});
close(4001, "gateway auth changed");
return;
}
}
const respond = (
ok: boolean,
payload?: unknown,

14
src/gateway/test-http-response.ts
View File

@@ -1,4 +1,5 @@
import type { ServerResponse } from "node:http";
import { PassThrough } from "node:stream";
import { vi } from "vitest";
export function makeMockHttpResponse(): {
@@ -6,13 +7,20 @@ export function makeMockHttpResponse(): {
setHeader: ReturnType<typeof vi.fn>;
end: ReturnType<typeof vi.fn>;
} {
const stream = new PassThrough();
const streamEnd = stream.end.bind(stream);
const setHeader = vi.fn();
const end = vi.fn();
const res = {
const end = vi.fn((chunk?: unknown) => {
if (chunk !== undefined) {
stream.write(chunk as string | Uint8Array);
}
streamEnd();
});
const res = Object.assign(stream, {
headersSent: false,
statusCode: 200,
setHeader,
end,
} as unknown as ServerResponse;
}) as unknown as ServerResponse;
return { res, setHeader, end };
}

6
src/infra/fs-safe.ts
View File

@@ -307,7 +307,7 @@ export async function readLocalFileSafely(params: {
filePath: string;
maxBytes?: number;
}): Promise<SafeLocalReadResult> {
const opened = await openVerifiedLocalFile(params.filePath);
const opened = await openLocalFileSafely({ filePath: params.filePath });
try {
return await readOpenedFileSafely({ opened, maxBytes: params.maxBytes });
} finally {
@@ -315,6 +315,10 @@ export async function readLocalFileSafely(params: {
}
}
export async function openLocalFileSafely(params: { filePath: string }): Promise<SafeOpenResult> {
return await openVerifiedLocalFile(params.filePath);
}
async function readOpenedFileSafely(params: {
opened: SafeOpenResult;
maxBytes?: number;

93
src/media/local-roots.test.ts
View File

@@ -1,8 +1,8 @@
import os from "node:os";
import path from "node:path";
import { pathToFileURL } from "node:url";
import { afterEach, describe, expect, it, vi } from "vitest";
import {
appendLocalMediaParentRoots,
buildMediaLocalRoots,
getAgentScopedMediaLocalRoots,
getAgentScopedMediaLocalRootsForSources,
@@ -53,14 +53,6 @@ describe("local media roots", () => {
expect(normalizedRoots).not.toContain(picturesRoot);
}
function expectPicturesRootAbsent(roots: readonly string[], picturesRoot?: string) {
expectPicturesRootPresence({
roots,
shouldContainPictures: false,
picturesRoot,
});
}
function expectAgentMediaRootsCase(params: {
stateDir: string;
getRoots: () => readonly string[];
@@ -86,12 +78,12 @@ describe("local media roots", () => {
it.each([
{
name: "keeps temp, media cache, and workspace roots by default",
name: "keeps temp, media cache, canvas, and workspace roots by default",
stateDir: path.join("/tmp", "openclaw-media-roots-state"),
getRoots: () => getDefaultMediaLocalRoots(),
expectedContained: ["media", "workspace", "sandboxes"],
expectedContained: ["media", "canvas", "workspace", "sandboxes"],
expectedExcluded: ["agents"],
minLength: 3,
minLength: 4,
},
{
name: "adds the active agent workspace without re-opening broad agent state roots",
@@ -110,12 +102,38 @@ describe("local media roots", () => {
});
});
it("adds concrete parent roots for local media sources without widening to filesystem root", () => {
const picturesDir =
process.platform === "win32" ? "C:\\Users\\peter\\Pictures" : "/Users/peter/Pictures";
const moviesDir =
process.platform === "win32" ? "C:\\Users\\peter\\Movies" : "/Users/peter/Movies";
const roots = appendLocalMediaParentRoots(
["/tmp/base"],
[
path.join(picturesDir, "photo.png"),
pathToFileURL(path.join(moviesDir, "clip.mp4")).href,
"https://example.com/remote.png",
"/top-level-file.png",
],
);
expect(roots.map(normalizeHostPath)).toEqual(
expect.arrayContaining([
normalizeHostPath("/tmp/base"),
normalizeHostPath(picturesDir),
normalizeHostPath(moviesDir),
]),
);
expect(roots.map(normalizeHostPath)).not.toContain(normalizeHostPath("/"));
});
it.each([
{
name: "does not widen agent media roots for concrete local sources when workspaceOnly is disabled",
name: "widens agent media roots for concrete local sources when workspaceOnly is disabled",
stateDir: path.join("/tmp", "openclaw-flexible-media-roots-state"),
cfg: {},
shouldContainPictures: false,
shouldContainPictures: true,
},
{
name: "does not widen agent media roots when workspaceOnly is enabled",
@@ -130,7 +148,7 @@ describe("local media roots", () => {
shouldContainPictures: false,
},
{
name: "does not widen media roots even when messaging-profile agents explicitly enable filesystem tools",
name: "widens media roots again when messaging-profile agents explicitly enable filesystem tools",
stateDir: path.join("/tmp", "openclaw-messaging-fs-media-roots-state"),
cfg: {
tools: {
@@ -138,7 +156,7 @@ describe("local media roots", () => {
fs: { workspaceOnly: false },
},
},
shouldContainPictures: false,
shouldContainPictures: true,
},
] as const)("$name", ({ stateDir, cfg, shouldContainPictures }) => {
const roots = withStateDir(stateDir, () =>
@@ -151,47 +169,14 @@ describe("local media roots", () => {
expectPicturesRootPresence({ roots, shouldContainPictures });
});
it("keeps agent-scoped defaults even when mediaSources include file URLs and top-level paths", () => {
const stateDir = path.join("/tmp", "openclaw-file-url-media-roots-state");
const picturesDir =
process.platform === "win32" ? "C:\\Users\\peter\\Pictures" : "/Users/peter/Pictures";
const moviesDir =
process.platform === "win32" ? "C:\\Users\\peter\\Movies" : "/Users/peter/Movies";
const roots = withStateDir(stateDir, () =>
getAgentScopedMediaLocalRootsForSources({
cfg: {},
agentId: "ops",
mediaSources: [
path.join(picturesDir, "photo.png"),
pathToFileURL(path.join(moviesDir, "clip.mp4")).href,
"/top-level-file.png",
],
}),
);
it("keeps the config-dir media cache root when state and config paths differ", () => {
const stateDir = path.join("/tmp", "openclaw-legacy-state");
const configDir = path.join("/tmp", "openclaw-current-config");
const roots = buildMediaLocalRoots(stateDir, configDir);
expectNormalizedRootsContain(roots, [
path.join(stateDir, "media"),
path.join(stateDir, "workspace"),
path.join(stateDir, "workspace-ops"),
]);
expectPicturesRootAbsent(roots, picturesDir);
expectPicturesRootAbsent(roots, moviesDir);
expect(roots.map(normalizeHostPath)).not.toContain(normalizeHostPath("/"));
});
it("includes the config media root when legacy state and config dirs diverge", () => {
const homeRoot = path.join(os.tmpdir(), "openclaw-legacy-home-test");
const roots = buildMediaLocalRoots(
path.join(homeRoot, ".clawdbot"),
path.join(homeRoot, ".openclaw"),
);
expectNormalizedRootsContain(roots, [
path.join(homeRoot, ".clawdbot", "media"),
path.join(homeRoot, ".clawdbot", "workspace"),
path.join(homeRoot, ".clawdbot", "sandboxes"),
path.join(homeRoot, ".openclaw", "media"),
path.join(configDir, "media"),
]);
});
});

81
src/media/local-roots.ts
View File

@@ -1,16 +1,24 @@
import path from "node:path";
import { resolveAgentWorkspaceDir } from "../agents/agent-scope.js";
import {
resolveEffectiveToolFsRootExpansionAllowed,
resolveEffectiveToolFsWorkspaceOnly,
} from "../agents/tool-fs-policy.js";
import { resolveStateDir } from "../config/paths.js";
import type { OpenClawConfig } from "../config/types.js";
import { safeFileURLToPath } from "../infra/local-file-access.js";
import { resolvePreferredOpenClawTmpDir } from "../infra/tmp-openclaw-dir.js";
import { normalizeOptionalString } from "../shared/string-coerce.js";
import { resolveConfigDir } from "../utils.js";
import { resolveConfigDir, resolveUserPath } from "../utils.js";
type BuildMediaLocalRootsOptions = {
preferredTmpDir?: string;
};
let cachedPreferredTmpDir: string | undefined;
const HTTP_URL_RE = /^https?:\/\//i;
const DATA_URL_RE = /^data:/i;
const WINDOWS_DRIVE_RE = /^[A-Za-z]:[\\/]/;
function resolveCachedPreferredTmpDir(): string {
if (!cachedPreferredTmpDir) {
@@ -30,14 +38,11 @@ export function buildMediaLocalRoots(
return Array.from(
new Set([
preferredTmpDir,
path.join(resolvedConfigDir, "media"),
path.join(resolvedStateDir, "media"),
path.join(resolvedStateDir, "canvas"),
path.join(resolvedStateDir, "workspace"),
path.join(resolvedStateDir, "sandboxes"),
// Upgraded installs can still resolve the active state dir to the legacy
// ~/.clawdbot tree while new media writes already go under ~/.openclaw/media.
// Keep inbound media readable across that split without widening roots beyond
// the managed media cache.
path.join(resolvedConfigDir, "media"),
]),
);
}
@@ -66,24 +71,60 @@ export function getAgentScopedMediaLocalRoots(
return roots;
}
/**
* @deprecated Kept for plugin-sdk compatibility. Media sources no longer widen allowed roots.
*/
export function appendLocalMediaParentRoots(
roots: readonly string[],
_mediaSources?: readonly string[],
): string[] {
return Array.from(new Set(roots.map((root) => path.resolve(root))));
function resolveLocalMediaPath(source: string): string | undefined {
const trimmed = source.trim();
if (!trimmed || HTTP_URL_RE.test(trimmed) || DATA_URL_RE.test(trimmed)) {
return undefined;
}
if (trimmed.startsWith("file://")) {
try {
return safeFileURLToPath(trimmed);
} catch {
return undefined;
}
}
if (trimmed.startsWith("~")) {
return resolveUserPath(trimmed);
}
if (path.isAbsolute(trimmed) || WINDOWS_DRIVE_RE.test(trimmed)) {
return path.resolve(trimmed);
}
return undefined;
}
export function getAgentScopedMediaLocalRootsForSources({
cfg,
agentId,
mediaSources: _mediaSources,
}: {
export function appendLocalMediaParentRoots(
roots: readonly string[],
mediaSources?: readonly string[],
): string[] {
const appended = Array.from(new Set(roots.map((root) => path.resolve(root))));
for (const source of mediaSources ?? []) {
const localPath = resolveLocalMediaPath(source);
if (!localPath) {
continue;
}
const parentDir = path.dirname(localPath);
if (parentDir === path.parse(parentDir).root) {
continue;
}
const normalizedParent = path.resolve(parentDir);
if (!appended.includes(normalizedParent)) {
appended.push(normalizedParent);
}
}
return appended;
}
export function getAgentScopedMediaLocalRootsForSources(params: {
cfg: OpenClawConfig;
agentId?: string;
mediaSources?: readonly string[];
}): readonly string[] {
return getAgentScopedMediaLocalRoots(cfg, agentId);
const roots = getAgentScopedMediaLocalRoots(params.cfg, params.agentId);
if (resolveEffectiveToolFsWorkspaceOnly({ cfg: params.cfg, agentId: params.agentId })) {
return roots;
}
if (!resolveEffectiveToolFsRootExpansionAllowed({ cfg: params.cfg, agentId: params.agentId })) {
return roots;
}
return appendLocalMediaParentRoots(roots, params.mediaSources);
}

39
src/media/mime.ts
View File

@@ -1,17 +1,7 @@
import path from "node:path";
import {
normalizeLowercaseStringOrEmpty,
normalizeOptionalLowercaseString,
} from "../shared/string-coerce.js";
import { fileTypeFromBuffer } from "file-type";
import { type MediaKind, mediaKindFromMime } from "./constants.js";
let fileTypeModulePromise: Promise<typeof import("file-type")> | undefined;
function loadFileTypeModule(): Promise<typeof import("file-type")> {
fileTypeModulePromise ??= import("file-type");
return fileTypeModulePromise;
}
// Map common mimes to preferred file extensions.
const EXT_BY_MIME: Record<string, string> = {
"image/heic": ".heic",
@@ -58,7 +48,7 @@ const MIME_BY_EXT: Record<string, string> = {
".jpeg": "image/jpeg",
".js": "text/javascript",
".htm": "text/html",
".xml": "text/xml", // pin text/xml as canonical (application/xml also maps to .xml in EXT_BY_MIME)
".xml": "text/xml",
};
const AUDIO_FILE_EXTENSIONS = new Set([
@@ -74,7 +64,11 @@ const AUDIO_FILE_EXTENSIONS = new Set([
]);
export function normalizeMimeType(mime?: string | null): string | undefined {
return normalizeOptionalLowercaseString(mime?.split(";")[0]);
if (!mime) {
return undefined;
}
const cleaned = mime.split(";")[0]?.trim().toLowerCase();
return cleaned || undefined;
}
async function sniffMime(buffer?: Buffer): Promise<string | undefined> {
@@ -82,7 +76,6 @@ async function sniffMime(buffer?: Buffer): Promise<string | undefined> {
return undefined;
}
try {
const { fileTypeFromBuffer } = await loadFileTypeModule();
const type = await fileTypeFromBuffer(buffer);
return type?.mime ?? undefined;
} catch {
@@ -97,15 +90,23 @@ export function getFileExtension(filePath?: string | null): string | undefined {
try {
if (/^https?:\/\//i.test(filePath)) {
const url = new URL(filePath);
return normalizeLowercaseStringOrEmpty(path.extname(url.pathname)) || undefined;
return path.extname(url.pathname).toLowerCase() || undefined;
}
} catch {
// fall back to plain path parsing
}
const ext = normalizeLowercaseStringOrEmpty(path.extname(filePath));
const ext = path.extname(filePath).toLowerCase();
return ext || undefined;
}
export function mimeTypeFromFilePath(filePath?: string | null): string | undefined {
const ext = getFileExtension(filePath);
if (!ext) {
return undefined;
}
return MIME_BY_EXT[ext];
}
export function isAudioFileName(fileName?: string | null): boolean {
const ext = getFileExtension(fileName);
if (!ext) {
@@ -126,7 +127,7 @@ function isGenericMime(mime?: string): boolean {
if (!mime) {
return true;
}
const m = normalizeLowercaseStringOrEmpty(mime);
const m = mime.toLowerCase();
return m === "application/octet-stream" || m === "application/zip";
}
@@ -174,7 +175,7 @@ export function isGifMedia(opts: {
contentType?: string | null;
fileName?: string | null;
}): boolean {
if (normalizeOptionalLowercaseString(opts.contentType) === "image/gif") {
if (opts.contentType?.toLowerCase() === "image/gif") {
return true;
}
const ext = getFileExtension(opts.fileName);
@@ -185,7 +186,7 @@ export function imageMimeFromFormat(format?: string | null): string | undefined
if (!format) {
return undefined;
}
switch (normalizeLowercaseStringOrEmpty(format)) {
switch (format.toLowerCase()) {
case "jpg":
case "jpeg":
return "image/jpeg";

12
src/media/parse.test.ts
View File

@@ -91,4 +91,16 @@ describe("splitMediaFromOutput", () => {
expectStableAudioAsVoiceDetectionCase(input);
}
});
it("returns ordered text and media segments while ignoring fenced MEDIA lines", () => {
const result = splitMediaFromOutput(
"Before\nMEDIA:https://example.com/a.png\n```text\nMEDIA:https://example.com/ignored.png\n```\nAfter",
);
expect(result.segments).toEqual([
{ type: "text", text: "Before" },
{ type: "media", url: "https://example.com/a.png" },
{ type: "text", text: "```text\nMEDIA:https://example.com/ignored.png\n```\nAfter" },
]);
});
});

64
src/media/parse.ts
View File

@@ -6,6 +6,16 @@ import { parseAudioTag } from "./audio-tags.js";
// Allow optional wrapping backticks and punctuation after the token; capture the core token.
export const MEDIA_TOKEN_RE = /\bMEDIA:\s*`?([^\n]+)`?/gi;
export type ParsedMediaOutputSegment =
| {
type: "text";
text: string;
}
| {
type: "media";
url: string;
};
export function normalizeMediaSource(src: string) {
return src.startsWith("file://") ? src.replace("file://", "") : src;
}
@@ -125,6 +135,7 @@ export function splitMediaFromOutput(raw: string): {
mediaUrls?: string[];
mediaUrl?: string; // legacy first item for backward compatibility
audioAsVoice?: boolean; // true if [[audio_as_voice]] tag was found
segments?: ParsedMediaOutputSegment[];
} {
// KNOWN: Leading whitespace is semantically meaningful in Markdown (lists, indented fences).
// We only trim the end; token cleanup below handles removing `MEDIA:` lines.
@@ -140,6 +151,19 @@ export function splitMediaFromOutput(raw: string): {
const media: string[] = [];
let foundMediaToken = false;
const segments: ParsedMediaOutputSegment[] = [];
const pushTextSegment = (text: string) => {
if (!text) {
return;
}
const last = segments[segments.length - 1];
if (last?.type === "text") {
last.text = `${last.text}\n${text}`;
return;
}
segments.push({ type: "text", text });
};
// Parse fenced code blocks to avoid extracting MEDIA tokens from inside them
const hasFenceMarkers = mayContainFenceMarkers(trimmedRaw);
@@ -154,6 +178,7 @@ export function splitMediaFromOutput(raw: string): {
// Skip MEDIA extraction if this line is inside a fenced code block
if (hasFenceMarkers && isInsideFence(fenceSpans, lineOffset)) {
keptLines.push(line);
pushTextSegment(line);
lineOffset += line.length + 1; // +1 for newline
continue;
}
@@ -161,6 +186,7 @@ export function splitMediaFromOutput(raw: string): {
const trimmedStart = line.trimStart();
if (!trimmedStart.startsWith("MEDIA:")) {
keptLines.push(line);
pushTextSegment(line);
lineOffset += line.length + 1; // +1 for newline
continue;
}
@@ -168,11 +194,13 @@ export function splitMediaFromOutput(raw: string): {
const matches = Array.from(line.matchAll(MEDIA_TOKEN_RE));
if (matches.length === 0) {
keptLines.push(line);
pushTextSegment(line);
lineOffset += line.length + 1; // +1 for newline
continue;
}
const pieces: string[] = [];
const lineSegments: ParsedMediaOutputSegment[] = [];
let cursor = 0;
for (const match of matches) {
@@ -219,6 +247,17 @@ export function splitMediaFromOutput(raw: string): {
}
}
if (!hasValidMedia && !unwrapped && /\s/.test(payloadValue)) {
const spacedFallback = normalizeMediaSource(cleanCandidate(payloadValue));
if (isValidMedia(spacedFallback, { allowSpaces: true, allowBareFilename: true })) {
media.splice(mediaStartIndex, media.length - mediaStartIndex, spacedFallback);
hasValidMedia = true;
foundMediaToken = true;
validCount = 1;
invalidParts.length = 0;
}
}
if (!hasValidMedia) {
const fallback = normalizeMediaSource(cleanCandidate(payloadValue));
if (isValidMedia(fallback, { allowSpaces: true, allowBareFilename: true })) {
@@ -230,6 +269,17 @@ export function splitMediaFromOutput(raw: string): {
}
if (hasValidMedia) {
const beforeText = pieces
.join("")
.replace(/[ \t]{2,}/g, " ")
.trim();
if (beforeText) {
lineSegments.push({ type: "text", text: beforeText });
}
pieces.length = 0;
for (const url of media.slice(mediaStartIndex, mediaStartIndex + validCount)) {
lineSegments.push({ type: "media", url });
}
if (invalidParts.length > 0) {
pieces.push(invalidParts.join(" "));
}
@@ -255,6 +305,14 @@ export function splitMediaFromOutput(raw: string): {
// If the line becomes empty, drop it.
if (cleanedLine) {
keptLines.push(cleanedLine);
lineSegments.push({ type: "text", text: cleanedLine });
}
for (const segment of lineSegments) {
if (segment.type === "text") {
pushTextSegment(segment.text);
continue;
}
segments.push(segment);
}
lineOffset += line.length + 1; // +1 for newline
}
@@ -274,9 +332,10 @@ export function splitMediaFromOutput(raw: string): {
}
if (media.length === 0) {
const parsedText = foundMediaToken || hasAudioAsVoice ? cleanedText : trimmedRaw;
const result: ReturnType<typeof splitMediaFromOutput> = {
// Return cleaned text if we found a media token OR audio tag, otherwise original
text: foundMediaToken || hasAudioAsVoice ? cleanedText : trimmedRaw,
text: parsedText,
segments: parsedText ? [{ type: "text", text: parsedText }] : [],
};
if (hasAudioAsVoice) {
result.audioAsVoice = true;
@@ -288,6 +347,7 @@ export function splitMediaFromOutput(raw: string): {
text: cleanedText,
mediaUrls: media,
mediaUrl: media[0],
segments: segments.length > 0 ? segments : [{ type: "text", text: cleanedText }],
...(hasAudioAsVoice ? { audioAsVoice: true } : {}),
};
}

168
src/media/web-media.test.ts
View File

@@ -2,47 +2,54 @@ import fs from "node:fs/promises";
import path from "node:path";
import { pathToFileURL } from "node:url";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { CANVAS_HOST_PATH } from "../canvas-host/a2ui.js";
import { resolveStateDir } from "../config/paths.js";
import { resolvePreferredOpenClawTmpDir } from "../infra/tmp-openclaw-dir.js";
import { createSuiteTempRootTracker } from "../test-helpers/temp-dir.js";
import { createJpegBufferWithDimensions, createPngBufferWithDimensions } from "./test-helpers.js";
let loadWebMedia: typeof import("./web-media.js").loadWebMedia;
const mediaRootTracker = createSuiteTempRootTracker({
prefix: "web-media-core-",
parentDir: resolvePreferredOpenClawTmpDir(),
});
const TINY_PNG_BASE64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=";
let fixtureRoot = "";
let fakePdfFile = "";
let oversizedJpegFile = "";
let realPdfFile = "";
let tinyPngFile = "";
let stateDir = "";
let canvasPngFile = "";
let workspaceDir = "";
let workspacePngFile = "";
beforeAll(async () => {
({ loadWebMedia } = await import("./web-media.js"));
await mediaRootTracker.setup();
fixtureRoot = await mediaRootTracker.make("case");
fakePdfFile = path.join(fixtureRoot, "fake.pdf");
realPdfFile = path.join(fixtureRoot, "real.pdf");
fixtureRoot = await fs.mkdtemp(path.join(resolvePreferredOpenClawTmpDir(), "web-media-core-"));
tinyPngFile = path.join(fixtureRoot, "tiny.png");
oversizedJpegFile = path.join(fixtureRoot, "oversized.jpg");
await fs.writeFile(fakePdfFile, "TOP_SECRET_TEXT", "utf8");
await fs.writeFile(
realPdfFile,
Buffer.from("%PDF-1.4\n1 0 obj\n<<>>\nendobj\ntrailer\n<<>>\n%%EOF"),
);
await fs.writeFile(tinyPngFile, Buffer.from(TINY_PNG_BASE64, "base64"));
await fs.writeFile(
oversizedJpegFile,
createJpegBufferWithDimensions({ width: 6_000, height: 5_000 }),
workspaceDir = path.join(fixtureRoot, "workspace");
workspacePngFile = path.join(workspaceDir, "chart.png");
await fs.mkdir(workspaceDir, { recursive: true });
await fs.writeFile(workspacePngFile, Buffer.from(TINY_PNG_BASE64, "base64"));
stateDir = resolveStateDir();
canvasPngFile = path.join(
stateDir,
"canvas",
"documents",
"cv_test",
"collection.media",
"tiny.png",
);
await fs.mkdir(path.dirname(canvasPngFile), { recursive: true });
await fs.writeFile(canvasPngFile, Buffer.from(TINY_PNG_BASE64, "base64"));
});
afterAll(async () => {
await mediaRootTracker.cleanup();
if (fixtureRoot) {
await fs.rm(fixtureRoot, { recursive: true, force: true });
}
if (stateDir) {
await fs.rm(path.join(stateDir, "canvas", "documents", "cv_test"), {
recursive: true,
force: true,
});
}
});
describe("loadWebMedia", () => {
@@ -108,24 +115,6 @@ describe("loadWebMedia", () => {
await expectLoadedWebMediaCase(createUrl());
});
it("rejects oversized pixel-count images before decode/resize backends run", async () => {
const oversizedPngFile = path.join(fixtureRoot, "oversized.png");
await fs.writeFile(
oversizedPngFile,
createPngBufferWithDimensions({ width: 8_000, height: 4_000 }),
);
await expect(loadWebMedia(oversizedPngFile, createLocalWebMediaOptions())).rejects.toThrow(
/pixel input limit/i,
);
});
it("preserves pixel-limit errors for oversized JPEG optimization", async () => {
await expect(loadWebMedia(oversizedJpegFile, createLocalWebMediaOptions())).rejects.toThrow(
/pixel input limit/i,
);
});
it.each([
{
name: "rejects remote-host file URLs before filesystem checks",
@@ -147,67 +136,60 @@ describe("loadWebMedia", () => {
await expectRejectedWebMediaWithoutFilesystemAccess(testCase);
});
describe("workspaceDir relative path resolution", () => {
it("resolves a bare filename against workspaceDir", async () => {
const result = await loadWebMedia("tiny.png", {
...createLocalWebMediaOptions(),
workspaceDir: fixtureRoot,
});
expect(result.kind).toBe("image");
expect(result.buffer.length).toBeGreaterThan(0);
});
it("loads browser-style canvas media paths as managed local files", async () => {
const result = await loadWebMedia(
`${CANVAS_HOST_PATH}/documents/cv_test/collection.media/tiny.png`,
{ maxBytes: 1024 * 1024 },
);
expect(result.kind).toBe("image");
expect(result.buffer.length).toBeGreaterThan(0);
});
it("resolves a dot-relative path against workspaceDir", async () => {
const result = await loadWebMedia("./tiny.png", {
...createLocalWebMediaOptions(),
workspaceDir: fixtureRoot,
});
expect(result.kind).toBe("image");
expect(result.buffer.length).toBeGreaterThan(0);
it("resolves relative local media paths against the provided workspace directory", async () => {
const result = await loadWebMedia("chart.png", {
maxBytes: 1024 * 1024,
localRoots: [workspaceDir],
workspaceDir,
});
expect(result.kind).toBe("image");
expect(result.buffer.length).toBeGreaterThan(0);
});
it("resolves a MEDIA:-prefixed relative path against workspaceDir", async () => {
const result = await loadWebMedia("MEDIA:tiny.png", {
...createLocalWebMediaOptions(),
workspaceDir: fixtureRoot,
});
expect(result.kind).toBe("image");
expect(result.buffer.length).toBeGreaterThan(0);
});
it("leaves absolute paths unchanged when workspaceDir is set", async () => {
const result = await loadWebMedia(tinyPngFile, {
...createLocalWebMediaOptions(),
workspaceDir: "/some/other/dir",
});
expect(result.kind).toBe("image");
expect(result.buffer.length).toBeGreaterThan(0);
it("rejects host-read text files outside local roots", async () => {
const secretFile = path.join(fixtureRoot, "secret.txt");
await fs.writeFile(secretFile, "secret", "utf8");
await expect(
loadWebMedia(secretFile, {
maxBytes: 1024 * 1024,
localRoots: "any",
readFile: async (filePath) => await fs.readFile(filePath),
hostReadCapability: true,
}),
).rejects.toMatchObject({
code: "path-not-allowed",
});
});
describe("host read capability", () => {
it("rejects document uploads that only match by file extension", async () => {
await expect(
loadWebMedia(fakePdfFile, {
maxBytes: 1024 * 1024,
localRoots: [fixtureRoot],
hostReadCapability: true,
}),
).rejects.toMatchObject({
code: "path-not-allowed",
});
});
it("still allows real PDF uploads detected from file content", async () => {
const result = await loadWebMedia(realPdfFile, {
it("rejects renamed host-read text files even when the extension looks allowed", async () => {
const disguisedPdf = path.join(fixtureRoot, "secret.pdf");
await fs.writeFile(disguisedPdf, "secret", "utf8");
await expect(
loadWebMedia(disguisedPdf, {
maxBytes: 1024 * 1024,
localRoots: [fixtureRoot],
localRoots: "any",
readFile: async (filePath) => await fs.readFile(filePath),
hostReadCapability: true,
});
}),
).rejects.toMatchObject({
code: "path-not-allowed",
});
});
expect(result.kind).toBe("document");
expect(result.contentType).toBe("application/pdf");
expect(result.fileName).toBe("real.pdf");
it("rejects traversal-style canvas media paths before filesystem access", async () => {
await expect(
loadWebMedia(`${CANVAS_HOST_PATH}/documents/../collection.media/tiny.png`),
).rejects.toMatchObject({
code: "path-not-allowed",
});
});
});

103
src/media/web-media.ts
View File

@@ -1,19 +1,15 @@
import path from "node:path";
import { resolveCanvasHttpPathToLocalPath } from "../gateway/canvas-documents.js";
import { logVerbose, shouldLogVerbose } from "../globals.js";
import { SafeOpenError, readLocalFileSafely } from "../infra/fs-safe.js";
import { assertNoWindowsNetworkPath, safeFileURLToPath } from "../infra/local-file-access.js";
import type { SsrFPolicy } from "../infra/net/ssrf.js";
import {
normalizeLowercaseStringOrEmpty,
normalizeOptionalString,
} from "../shared/string-coerce.js";
import { resolveUserPath } from "../utils.js";
import { maxBytesForKind, type MediaKind } from "./constants.js";
import { fetchRemoteMedia } from "./fetch.js";
import {
convertHeicToJpeg,
hasAlphaChannel,
MAX_IMAGE_INPUT_PIXELS,
optimizeImageToPng,
resizeToJpeg,
} from "./image-ops.js";
@@ -23,7 +19,13 @@ import {
LocalMediaAccessError,
type LocalMediaAccessErrorCode,
} from "./local-media-access.js";
import { detectMime, extensionForMime, kindFromMime, normalizeMimeType } from "./mime.js";
import {
detectMime,
extensionForMime,
getFileExtension,
kindFromMime,
normalizeMimeType,
} from "./mime.js";
export { getDefaultLocalRoots, LocalMediaAccessError };
export type { LocalMediaAccessErrorCode };
@@ -39,6 +41,7 @@ type WebMediaOptions = {
maxBytes?: number;
optimizeImages?: boolean;
ssrfPolicy?: SsrFPolicy;
workspaceDir?: string;
/** Allowed root directories for local path reads. "any" is deprecated; prefer sandboxValidated + readFile. */
localRoots?: readonly string[] | "any";
/** Caller already validated the local path (sandbox/other guards); requires readFile override. */
@@ -46,8 +49,6 @@ type WebMediaOptions = {
readFile?: (filePath: string) => Promise<Buffer>;
/** Host-local fs-policy read piggyback; rejects plaintext-like document sends. */
hostReadCapability?: boolean;
/** Agent workspace directory for resolving relative MEDIA: paths. */
workspaceDir?: string;
};
function resolveWebMediaOptions(params: {
@@ -73,6 +74,7 @@ function resolveWebMediaOptions(params: {
const HEIC_MIME_RE = /^image\/hei[cf]$/i;
const HEIC_EXT_RE = /\.(heic|heif)$/i;
const WINDOWS_DRIVE_RE = /^[A-Za-z]:[\\/]/;
const HOST_READ_ALLOWED_DOCUMENT_MIMES = new Set([
"application/msword",
"application/pdf",
@@ -96,44 +98,54 @@ function formatCapReduce(label: string, cap: number, size: number): string {
return `${label} could not be reduced below ${formatMb(cap, 0)}MB (got ${formatMb(size)}MB)`;
}
function isPixelLimitError(error: unknown): boolean {
return (
error instanceof Error &&
error.message.includes(`${MAX_IMAGE_INPUT_PIXELS.toLocaleString("en-US")} pixel input limit`)
);
}
function isHeicSource(opts: { contentType?: string; fileName?: string }): boolean {
if (HEIC_MIME_RE.test(normalizeOptionalString(opts.contentType) ?? "")) {
if (opts.contentType && HEIC_MIME_RE.test(opts.contentType.trim())) {
return true;
}
if (HEIC_EXT_RE.test(normalizeOptionalString(opts.fileName) ?? "")) {
if (opts.fileName && HEIC_EXT_RE.test(opts.fileName.trim())) {
return true;
}
return false;
}
function assertHostReadMediaAllowed(params: {
sniffedContentType?: string;
contentType?: string;
filePath?: string;
kind: MediaKind | undefined;
}): void {
if (params.kind === "image" || params.kind === "audio" || params.kind === "video") {
const sniffedKind = kindFromMime(params.sniffedContentType);
if (sniffedKind === "image" || sniffedKind === "audio" || sniffedKind === "video") {
return;
}
if (params.kind !== "document") {
const contentType = normalizeMimeType(params.contentType);
throw new LocalMediaAccessError(
"path-not-allowed",
`Host-local media sends only allow images, audio, video, PDF, and Office documents (got ${contentType ?? "unknown"}).`,
);
const sniffedMime = normalizeMimeType(params.sniffedContentType);
if (
sniffedKind === "document" &&
sniffedMime &&
HOST_READ_ALLOWED_DOCUMENT_MIMES.has(sniffedMime)
) {
return;
}
if (
sniffedMime === "application/x-cfb" &&
[".doc", ".ppt", ".xls"].includes(getFileExtension(params.filePath) ?? "")
) {
return;
}
const normalizedMime = normalizeMimeType(params.contentType);
if (normalizedMime && HOST_READ_ALLOWED_DOCUMENT_MIMES.has(normalizedMime)) {
return;
if (
params.kind === "document" &&
normalizedMime &&
HOST_READ_ALLOWED_DOCUMENT_MIMES.has(normalizedMime)
) {
throw new LocalMediaAccessError(
"path-not-allowed",
`Host-local media sends require buffer-verified media/document types (got fallback ${normalizedMime}).`,
);
}
throw new LocalMediaAccessError(
"path-not-allowed",
`Host-local media sends only allow images, audio, video, PDF, and Office documents (got ${normalizedMime ?? "unknown"}).`,
`Host-local media sends only allow buffer-verified images, audio, video, PDF, and Office documents (got ${sniffedMime ?? normalizedMime ?? "unknown"}).`,
);
}
@@ -185,9 +197,7 @@ async function optimizeImageWithFallback(params: {
meta?: { contentType?: string; fileName?: string };
}): Promise<OptimizedImage> {
const { buffer, cap, meta } = params;
const isPng =
meta?.contentType === "image/png" ||
normalizeLowercaseStringOrEmpty(meta?.fileName).endsWith(".png");
const isPng = meta?.contentType === "image/png" || meta?.fileName?.toLowerCase().endsWith(".png");
const hasAlpha = isPng && (await hasAlphaChannel(buffer));
if (hasAlpha) {
@@ -214,11 +224,11 @@ async function loadWebMediaInternal(
maxBytes,
optimizeImages = true,
ssrfPolicy,
workspaceDir,
localRoots,
sandboxValidated = false,
readFile: readFileOverride,
hostReadCapability = false,
workspaceDir,
} = options;
// Strip MEDIA: prefix used by agent tools (e.g. TTS) to tag media paths.
// Be lenient: LLM output may add extra whitespace (e.g. " MEDIA : /tmp/x.png").
@@ -231,6 +241,7 @@ async function loadWebMediaInternal(
throw new LocalMediaAccessError("invalid-file-url", (err as Error).message, { cause: err });
}
}
mediaUrl = resolveCanvasHttpPathToLocalPath(mediaUrl) ?? mediaUrl;
const optimizeAndClampImage = async (
buffer: Buffer,
@@ -319,11 +330,7 @@ async function loadWebMediaInternal(
if (mediaUrl.startsWith("~")) {
mediaUrl = resolveUserPath(mediaUrl);
}
// Resolve relative MEDIA: paths (e.g. "poker_profit.png", "./subdir/file.png")
// against the agent workspace directory so bare filenames written by agents
// are found on disk and pass the local-roots allowlist check.
if (workspaceDir && !path.isAbsolute(mediaUrl)) {
if (workspaceDir && !path.isAbsolute(mediaUrl) && !WINDOWS_DRIVE_RE.test(mediaUrl)) {
mediaUrl = path.resolve(workspaceDir, mediaUrl);
}
try {
@@ -376,10 +383,17 @@ async function loadWebMediaInternal(
throw err;
}
}
const detectedMime = await detectMime({ buffer: data, filePath: mediaUrl });
const verifiedMime = hostReadCapability ? await detectMime({ buffer: data }) : detectedMime;
const mime = verifiedMime ?? detectedMime;
const sniffedMime = await detectMime({ buffer: data });
const mime = await detectMime({ buffer: data, filePath: mediaUrl });
const kind = kindFromMime(mime);
if (hostReadCapability) {
assertHostReadMediaAllowed({
sniffedContentType: sniffedMime,
contentType: mime,
filePath: mediaUrl,
kind,
});
}
let fileName = path.basename(mediaUrl) || undefined;
if (fileName && !path.extname(fileName) && mime) {
const ext = extensionForMime(mime);
@@ -387,12 +401,6 @@ async function loadWebMediaInternal(
fileName = `${fileName}${ext}`;
}
}
if (hostReadCapability) {
assertHostReadMediaAllowed({
contentType: verifiedMime,
kind: kindFromMime(detectedMime ?? verifiedMime),
});
}
return await clampAndFinalize({
buffer: data,
contentType: mime,
@@ -472,10 +480,7 @@ export async function optimizeImageToJpeg(
quality,
};
}
} catch (error) {
if (isPixelLimitError(error)) {
throw error;
}
} catch {
// Continue trying other size/quality combinations
}
}

163
ui/src/styles/chat/layout.css
View File

@@ -51,21 +51,16 @@
/* Chat thread - scrollable middle section, transparent */
.chat-thread {
flex: 1;
flex: 1 1 0;
/* Grow, shrink, and use 0 base for proper scrolling */
overflow-y: auto;
overflow-x: hidden;
padding: 0 12px 16px;
padding: 0 6px 6px;
margin: 0 0 0 0;
min-height: 0;
/* Allow shrinking for flex scroll behavior */
border-radius: 0;
border: none;
border-radius: var(--radius-md);
background: transparent;
display: flex;
flex-direction: column;
gap: 12px;
min-width: 0;
}
.chat-thread-inner > :first-child {
@@ -305,18 +300,114 @@
justify-content: flex-end;
}
/* Embedded audio (e.g. gateway-injected TTS from slash commands) */
.chat-message-audio {
.chat-assistant-attachments {
display: flex;
flex-direction: column;
gap: 8px;
gap: 10px;
margin-bottom: 8px;
max-width: min(420px, 100%);
}
.chat-message-audio-el {
width: 100%;
min-height: 36px;
.chat-assistant-attachments img.chat-message-image {
display: block;
}
.chat-assistant-attachment-card {
display: flex;
align-items: center;
gap: 10px;
padding: 10px 12px;
border: 1px solid var(--border);
border-radius: var(--radius-md);
background: color-mix(in srgb, var(--card) 82%, var(--bg));
}
.chat-assistant-attachment-card--audio,
.chat-assistant-attachment-card--video {
display: flex;
flex-direction: column;
align-items: stretch;
}
.chat-assistant-attachment-card--audio audio,
.chat-assistant-attachment-card--video video {
width: min(100%, 360px);
max-width: 100%;
}
.chat-assistant-attachment-card--video video {
border-radius: var(--radius-sm);
background: #000;
}
.chat-assistant-attachment-card__header {
display: flex;
align-items: center;
justify-content: space-between;
gap: 8px;
margin-bottom: 8px;
}
.chat-assistant-attachment-card__title,
.chat-assistant-attachment-card__link {
color: var(--text);
font-size: 13px;
text-decoration: none;
word-break: break-word;
}
.chat-assistant-attachment-card__reason {
color: var(--muted);
font-size: 12px;
line-height: 1.4;
}
.chat-assistant-attachment-card__icon {
display: inline-flex;
width: 16px;
height: 16px;
color: var(--muted);
}
.chat-assistant-attachment-card__icon svg {
width: 16px;
height: 16px;
}
.chat-assistant-attachment-badge,
.chat-reply-pill {
display: inline-flex;
align-items: center;
gap: 6px;
width: fit-content;
max-width: 100%;
border-radius: 999px;
font-size: 12px;
}
.chat-assistant-attachment-badge {
padding: 3px 8px;
background: color-mix(in srgb, var(--accent) 14%, transparent);
color: var(--accent);
border: 1px solid color-mix(in srgb, var(--accent) 24%, transparent);
}
.chat-reply-pill {
margin-bottom: 8px;
padding: 5px 10px;
color: var(--muted);
border: 1px solid var(--border);
background: color-mix(in srgb, var(--bg) 70%, transparent);
}
.chat-reply-pill__icon {
display: inline-flex;
width: 14px;
height: 14px;
}
.chat-reply-pill__icon svg {
width: 14px;
height: 14px;
}
/* Compose input row - horizontal layout */
@@ -762,8 +853,8 @@
}
.chat-controls__session {
min-width: 98px;
max-width: 190px;
min-width: 140px;
max-width: 300px;
}
.chat-controls__session-row {
@@ -774,13 +865,8 @@
}
.chat-controls__model {
min-width: 124px;
max-width: 206px;
}
.chat-controls__thinking-select {
min-width: 88px;
max-width: 118px;
min-width: 170px;
max-width: 320px;
}
.chat-controls__thinking {
@@ -805,17 +891,13 @@
.chat-controls__session select {
padding: 6px 10px;
font-size: 13px;
max-width: 190px;
max-width: 300px;
overflow: hidden;
text-overflow: ellipsis;
}
.chat-controls__model select {
max-width: 206px;
}
.chat-controls__thinking-select select {
max-width: 118px;
max-width: 320px;
}
.chat-controls__thinking {
@@ -829,6 +911,10 @@
border: 1px solid var(--border);
}
.chat-controls__auto-expand {
padding: 8px;
}
/* Light theme thinking indicator override */
:root[data-theme-mode="light"] .chat-controls__thinking {
background: rgba(255, 255, 255, 0.9);
@@ -846,11 +932,6 @@
max-width: none;
}
.chat-controls__thinking-select {
min-width: 130px;
max-width: none;
}
.chat-controls {
gap: 8px;
}
@@ -858,14 +939,6 @@
.chat-compose__field textarea {
min-height: 64px;
}
.card.chat {
padding: 0;
}
.chat-thread {
padding: 0 0 16px;
}
}
@media (max-width: 640px) {
@@ -904,10 +977,6 @@
.chat-controls__model {
min-width: 150px;
}
.chat-controls__thinking-select {
min-width: 140px;
}
}
/* Chat loading skeleton */

337
ui/src/styles/chat/tool-cards.css
View File

@@ -1,20 +1,28 @@
/* Tool Card Styles */
.chat-tool-card {
border: 1px solid var(--border);
border: 1px solid color-mix(in srgb, var(--border) 85%, transparent);
border-radius: var(--radius-md);
padding: 10px 12px;
padding: 12px 14px;
margin-top: 6px;
background: var(--card);
background: color-mix(in srgb, var(--card) 88%, var(--secondary) 12%);
box-shadow: inset 0 1px 0 color-mix(in srgb, var(--bg) 75%, transparent);
transition:
border-color var(--duration-fast) ease-out,
background var(--duration-fast) ease-out;
background var(--duration-fast) ease-out,
box-shadow var(--duration-fast) ease-out;
max-height: 120px;
overflow: hidden;
}
.chat-tool-card--expanded {
max-height: none;
overflow: visible;
}
.chat-tool-card:hover {
border-color: var(--border-strong);
background: var(--bg-hover);
border-color: color-mix(in srgb, var(--border-strong) 80%, transparent);
background: color-mix(in srgb, var(--card) 70%, var(--bg-hover) 30%);
box-shadow: inset 0 1px 0 color-mix(in srgb, var(--bg) 85%, transparent);
}
/* First tool card in a group - no top margin */
@@ -35,8 +43,16 @@
.chat-tool-card__header {
display: flex;
justify-content: space-between;
align-items: flex-start;
gap: 10px;
}
.chat-tool-card__actions {
display: inline-flex;
align-items: center;
gap: 8px;
gap: 6px;
flex-wrap: wrap;
justify-content: flex-end;
}
.chat-tool-card__title {
@@ -44,7 +60,7 @@
align-items: center;
gap: 6px;
font-weight: 600;
font-size: 13px;
font-size: 14px;
line-height: 1.2;
}
@@ -68,19 +84,41 @@
}
/* "View >" action link */
.chat-tool-card__action {
.chat-tool-card__action,
.chat-tool-card__action-btn {
display: inline-flex;
align-items: center;
justify-content: center;
gap: 4px;
font-size: 12px;
color: var(--accent);
opacity: 0.8;
transition: opacity 150ms ease-out;
font-size: 11px;
color: var(--muted);
opacity: 1;
transition:
color 150ms ease-out,
background 150ms ease-out,
border-color 150ms ease-out;
}
.chat-tool-card__action svg {
width: 12px;
height: 12px;
.chat-tool-card__action-btn {
width: 28px;
height: 28px;
padding: 0;
border: 1px solid color-mix(in srgb, var(--border) 80%, transparent);
border-radius: 999px;
background: color-mix(in srgb, var(--secondary) 55%, transparent);
cursor: pointer;
font: inherit;
}
.chat-tool-card__action-icon {
display: inline-flex;
align-items: center;
justify-content: center;
}
.chat-tool-card__action-icon svg {
width: 11px;
height: 11px;
stroke: currentColor;
fill: none;
stroke-width: 1.5px;
@@ -88,8 +126,12 @@
stroke-linejoin: round;
}
.chat-tool-card--clickable:hover .chat-tool-card__action {
opacity: 1;
.chat-tool-card--clickable:hover .chat-tool-card__action,
.chat-tool-card__action-btn:hover,
.chat-tool-card__action-btn:focus-visible {
color: var(--text);
background: color-mix(in srgb, var(--bg-hover) 70%, transparent);
border-color: color-mix(in srgb, var(--border-strong) 70%, transparent);
}
/* Status indicator for completed/empty results */
@@ -111,47 +153,212 @@
.chat-tool-card__status-text {
font-size: 11px;
margin-top: 4px;
margin-top: 10px;
}
.chat-tool-card__detail {
font-size: 12px;
color: var(--muted);
margin-top: 4px;
margin-top: 6px;
}
/* Collapsed preview - fixed height with truncation */
.chat-tools-inline {
display: grid;
gap: 6px;
}
.chat-tool-card__block {
margin-top: 12px;
}
.chat-tool-card__preview,
.chat-tool-card__raw {
margin-top: 12px;
}
.chat-tool-card__preview {
font-size: 11px;
color: var(--muted);
margin-top: 8px;
padding: 8px 10px;
background: var(--secondary);
border: 1px solid color-mix(in srgb, var(--border) 78%, transparent);
border-radius: var(--radius-md);
white-space: pre-wrap;
background: color-mix(in srgb, var(--secondary) 78%, transparent);
overflow: hidden;
max-height: 44px;
line-height: 1.4;
border: 1px solid var(--border);
}
.chat-tool-card--clickable:hover .chat-tool-card__preview {
background: var(--bg-hover);
border-color: var(--border-strong);
.chat-tool-card__preview-header {
display: flex;
align-items: center;
justify-content: space-between;
gap: 10px;
padding: 10px 12px;
border-bottom: 1px solid color-mix(in srgb, var(--border) 72%, transparent);
background: color-mix(in srgb, var(--card) 82%, transparent);
}
/* Short inline output */
.chat-tool-card__inline {
.chat-tool-card__preview-label {
font-size: 11px;
font-weight: 600;
letter-spacing: 0.04em;
text-transform: uppercase;
color: var(--muted);
}
.chat-tool-card__preview-tabs {
display: inline-flex;
align-items: center;
gap: 4px;
}
.chat-tool-card__preview-tab {
border: 1px solid color-mix(in srgb, var(--border) 76%, transparent);
background: color-mix(in srgb, var(--bg) 72%, transparent);
color: var(--muted);
border-radius: 999px;
padding: 4px 10px;
font: inherit;
font-size: 12px;
cursor: pointer;
transition:
color 150ms ease-out,
border-color 150ms ease-out,
background 150ms ease-out;
}
.chat-tool-card__preview-tab.is-active,
.chat-tool-card__preview-tab:hover,
.chat-tool-card__preview-tab:focus-visible {
color: var(--text);
margin-top: 6px;
padding: 6px 8px;
background: var(--secondary);
border-radius: var(--radius-sm);
border-color: color-mix(in srgb, var(--border-strong) 80%, transparent);
background: color-mix(in srgb, var(--card) 92%, transparent);
}
.chat-tool-card__preview-panel {
padding: 12px;
}
.chat-tool-card__preview-frame {
display: block;
width: 100%;
min-height: 420px;
border: 1px solid color-mix(in srgb, var(--border) 68%, transparent);
border-radius: var(--radius-md);
background: #fff;
}
.chat-tool-card__raw-toggle {
width: 100%;
display: inline-flex;
align-items: center;
justify-content: space-between;
gap: 8px;
padding: 10px 12px;
border: 1px solid color-mix(in srgb, var(--border) 75%, transparent);
border-radius: var(--radius-md);
background: color-mix(in srgb, var(--secondary) 68%, transparent);
color: var(--muted);
font: inherit;
font-size: 12px;
cursor: pointer;
transition:
color 150ms ease-out,
border-color 150ms ease-out,
background 150ms ease-out;
}
.chat-tool-card__raw-toggle:hover,
.chat-tool-card__raw-toggle:focus-visible {
color: var(--text);
border-color: color-mix(in srgb, var(--border-strong) 80%, transparent);
background: color-mix(in srgb, var(--card) 92%, transparent);
}
.chat-tool-card__raw-toggle[aria-expanded="true"] .chat-tool-card__raw-toggle-icon {
transform: rotate(180deg);
}
.chat-tool-card__raw-toggle-icon {
display: inline-flex;
align-items: center;
justify-content: center;
transition: transform 150ms ease-out;
}
.chat-tool-card__raw-toggle-icon svg {
width: 14px;
height: 14px;
stroke: currentColor;
fill: none;
stroke-width: 1.6px;
stroke-linecap: round;
stroke-linejoin: round;
}
.chat-tool-card__raw-body {
margin-top: 8px;
}
.chat-tool-card__block-header {
display: inline-flex;
align-items: center;
gap: 6px;
margin-bottom: 6px;
color: var(--muted);
}
.chat-tool-card__block-icon {
display: inline-flex;
align-items: center;
justify-content: center;
width: 14px;
height: 14px;
}
.chat-tool-card__block-icon svg {
width: 14px;
height: 14px;
stroke: currentColor;
fill: none;
stroke-width: 1.5px;
stroke-linecap: round;
stroke-linejoin: round;
}
.chat-tool-card__block-label {
font-size: 11px;
font-weight: 600;
letter-spacing: 0.04em;
text-transform: uppercase;
color: inherit;
}
.chat-tool-card__block-preview,
.chat-tool-card__block-content,
.chat-tool-card__block-empty {
margin: 0;
padding: 11px 12px;
border-radius: var(--radius-md);
border: 1px solid color-mix(in srgb, var(--border) 75%, transparent);
background: color-mix(in srgb, var(--secondary) 82%, transparent);
font-size: 11px;
line-height: 1.45;
white-space: pre-wrap;
word-break: break-word;
}
.chat-tool-card__block-preview,
.chat-tool-card__block-empty {
color: var(--muted);
}
.chat-tool-card__block-content {
color: var(--text);
overflow-x: auto;
}
.chat-tool-card__block-preview {
overflow: hidden;
max-height: 52px;
}
.chat-tools-summary {
display: flex;
align-items: center;
@@ -331,12 +538,21 @@
border: 1px solid color-mix(in srgb, var(--border) 75%, transparent);
border-radius: var(--radius-md);
background: color-mix(in srgb, var(--bg-hover) 35%, transparent);
width: 100%;
text-align: left;
appearance: none;
-webkit-appearance: none;
font: inherit;
transition:
color 150ms ease,
background 150ms ease,
border-color 150ms ease;
}
.chat-tool-msg-summary[type="button"] {
background: color-mix(in srgb, var(--bg-hover) 35%, transparent);
}
.chat-tool-msg-summary::-webkit-details-marker {
display: none;
}
@@ -348,6 +564,15 @@
transition: transform 150ms ease;
}
.chat-tool-msg-collapse--static > .chat-tool-msg-summary::before {
display: none;
}
.chat-tool-msg-collapse--manual.is-open > .chat-tool-msg-summary::before,
.chat-tool-msg-summary[aria-expanded="true"]::before {
transform: rotate(90deg);
}
.chat-tool-msg-collapse[open] > .chat-tool-msg-summary::before {
transform: rotate(90deg);
}
@@ -407,6 +632,42 @@
min-width: 0;
}
.chat-tool-msg-summary__spacer {
flex: 1 1 auto;
}
.chat-tool-msg-summary__btn {
width: 24px;
height: 24px;
padding: 0;
border: 1px solid color-mix(in srgb, var(--border) 70%, transparent);
border-radius: 999px;
background: color-mix(in srgb, var(--secondary) 55%, transparent);
color: var(--muted);
display: inline-flex;
align-items: center;
justify-content: center;
cursor: pointer;
flex-shrink: 0;
}
.chat-tool-msg-summary__btn:hover,
.chat-tool-msg-summary__btn:focus-visible {
color: var(--text);
background: color-mix(in srgb, var(--bg-hover) 70%, transparent);
border-color: color-mix(in srgb, var(--border-strong) 70%, transparent);
}
.chat-tool-msg-summary__btn svg {
width: 11px;
height: 11px;
stroke: currentColor;
fill: none;
stroke-width: 1.5px;
stroke-linecap: round;
stroke-linejoin: round;
}
.chat-tool-msg-body {
padding-top: 8px;
}

103
ui/src/ui/app-gateway.node.test.ts
View File

@@ -9,7 +9,6 @@ const loadChatHistoryMock = vi.hoisted(() => vi.fn(async () => undefined));
type GatewayClientMock = {
start: ReturnType<typeof vi.fn>;
stop: ReturnType<typeof vi.fn>;
request: ReturnType<typeof vi.fn>;
options: { clientVersion?: string };
emitHello: (hello?: GatewayHelloOk) => void;
emitClose: (info: {
@@ -23,8 +22,8 @@ type GatewayClientMock = {
const gatewayClientInstances: GatewayClientMock[] = [];
vi.mock("./gateway.ts", async () => {
const actual = await vi.importActual<typeof import("./gateway.ts")>("./gateway.ts");
vi.mock("./gateway.ts", async (importOriginal) => {
const actual = await importOriginal<typeof import("./gateway.ts")>();
function resolveGatewayErrorDetailCode(
error: { details?: unknown } | null | undefined,
@@ -40,7 +39,6 @@ vi.mock("./gateway.ts", async () => {
class GatewayBrowserClient {
readonly start = vi.fn();
readonly stop = vi.fn();
readonly request = vi.fn(async () => ({}));
constructor(
private opts: {
@@ -58,7 +56,6 @@ vi.mock("./gateway.ts", async () => {
gatewayClientInstances.push({
start: this.start,
stop: this.stop,
request: this.request,
options: { clientVersion: this.opts.clientVersion },
emitHello: (hello) => {
this.opts.onHello?.(
@@ -89,9 +86,8 @@ vi.mock("./gateway.ts", async () => {
return { ...actual, GatewayBrowserClient, resolveGatewayErrorDetailCode };
});
vi.mock("./controllers/chat.ts", async () => {
const actual =
await vi.importActual<typeof import("./controllers/chat.ts")>("./controllers/chat.ts");
vi.mock("./controllers/chat.ts", async (importOriginal) => {
const actual = await importOriginal<typeof import("./controllers/chat.ts")>();
return {
...actual,
loadChatHistory: loadChatHistoryMock,
@@ -102,6 +98,8 @@ type TestGatewayHost = Parameters<typeof connectGateway>[0] & {
chatSideResult: unknown;
chatSideResultTerminalRuns: Set<string>;
chatStream: string | null;
chatToolMessages: Record<string, unknown>[];
toolStreamById: Map<string, unknown>;
toolStreamOrder: string[];
};
@@ -140,12 +138,10 @@ function createHost(): TestGatewayHost {
assistantName: "OpenClaw",
assistantAvatar: null,
assistantAgentId: null,
localMediaPreviewRoots: [],
serverVersion: null,
sessionKey: "main",
basePath: "",
chatMessage: "",
chatMessages: [],
chatAttachments: [],
chatQueue: [],
chatToolMessages: [],
chatStreamSegments: [],
@@ -196,7 +192,6 @@ describe("connectGateway", () => {
beforeEach(() => {
gatewayClientInstances.length = 0;
loadChatHistoryMock.mockClear();
vi.restoreAllMocks();
});
it("ignores stale client onGap callbacks after reconnect", () => {
@@ -219,69 +214,6 @@ describe("connectGateway", () => {
expect(host.lastError).toBeNull();
});
it("preserves live approval prompts, clears stale run indicators, and resumes queued work after seq-gap reconnect", () => {
const now = 1_700_000_000_000;
vi.spyOn(Date, "now").mockReturnValue(now);
const host = createHost();
connectGateway(host);
const client = gatewayClientInstances[0];
expect(client).toBeDefined();
const chatHost = host as typeof host & {
chatRunId: string | null;
chatQueue: Array<{
id: string;
text: string;
createdAt: number;
pendingRunId?: string;
}>;
};
chatHost.chatRunId = "run-1";
chatHost.chatQueue = [
{
id: "pending",
text: "/steer tighten the plan",
createdAt: 1,
pendingRunId: "run-1",
},
{
id: "queued",
text: "follow up",
createdAt: 2,
},
];
host.execApprovalQueue = [
{
id: "approval-1",
kind: "exec",
request: { command: "rm -rf /tmp/demo" },
createdAtMs: now,
expiresAtMs: now + 60_000,
},
];
client.emitGap(20, 24);
expect(gatewayClientInstances).toHaveLength(2);
expect(host.execApprovalQueue).toHaveLength(1);
expect(host.execApprovalQueue[0]?.id).toBe("approval-1");
expect(chatHost.chatQueue).toHaveLength(1);
expect(chatHost.chatQueue[0]?.text).toBe("follow up");
const reconnectClient = gatewayClientInstances[1];
expect(reconnectClient).toBeDefined();
reconnectClient.emitHello();
expect(reconnectClient.request).toHaveBeenCalledWith("chat.send", {
sessionKey: "main",
message: "follow up",
deliver: false,
idempotencyKey: expect.any(String),
attachments: undefined,
});
expect(chatHost.chatQueue).toHaveLength(0);
});
it("ignores stale client onEvent callbacks after reconnect", () => {
const host = createHost();
@@ -353,6 +285,27 @@ describe("connectGateway", () => {
expect(host.lastErrorCode).toBeNull();
});
it("preserves pending approval requests across reconnect", () => {
const host = createHost();
host.execApprovalQueue = [
{
id: "approval-1",
kind: "exec",
title: "Approve command",
summary: "rm -rf /tmp/nope",
createdAtMs: Date.now(),
expiresAtMs: Date.now() + 60_000,
} as never,
];
connectGateway(host);
expect(host.execApprovalQueue).toHaveLength(1);
connectGateway(host);
expect(host.execApprovalQueue).toHaveLength(1);
expect(host.execApprovalQueue[0]?.id).toBe("approval-1");
});
it("maps generic fetch-failed auth errors to actionable token mismatch message", () => {
const host = createHost();

39
ui/src/ui/app-gateway.ts
View File

@@ -50,7 +50,6 @@ import {
import { GatewayBrowserClient } from "./gateway.ts";
import type { Tab } from "./navigation.ts";
import type { UiSettings } from "./storage.ts";
import { normalizeOptionalString } from "./string-coerce.ts";
import type {
AgentsListResult,
PresenceEntry,
@@ -130,7 +129,7 @@ export function resolveControlUiClientVersion(params: {
serverVersion: string | null;
pageUrl?: string;
}): string | undefined {
const serverVersion = normalizeOptionalString(params.serverVersion);
const serverVersion = params.serverVersion?.trim();
if (!serverVersion) {
return undefined;
}
@@ -156,16 +155,16 @@ function normalizeSessionKeyForDefaults(
value: string | undefined,
defaults: SessionDefaultsSnapshot,
): string {
const raw = normalizeOptionalString(value) ?? "";
const mainSessionKey = normalizeOptionalString(defaults.mainSessionKey);
const raw = (value ?? "").trim();
const mainSessionKey = defaults.mainSessionKey?.trim();
if (!mainSessionKey) {
return raw;
}
if (!raw) {
return mainSessionKey;
}
const mainKey = normalizeOptionalString(defaults.mainKey) ?? "main";
const defaultAgentId = normalizeOptionalString(defaults.defaultAgentId);
const mainKey = defaults.mainKey?.trim() || "main";
const defaultAgentId = defaults.defaultAgentId?.trim();
const isAlias =
raw === "main" ||
raw === mainKey ||
@@ -214,8 +213,6 @@ export function connectGateway(host: GatewayHost, options?: ConnectGatewayOption
host.hello = null;
host.connected = false;
if (reconnectReason === "seq-gap") {
// A seq gap means the socket stayed on the same gateway; preserve prompts
// that only arrived as ephemeral events and clear stale run-scoped indicators.
host.execApprovalQueue = pruneExecApprovalQueue(host.execApprovalQueue);
clearPendingQueueItemsForRun(
host as unknown as Parameters<typeof clearPendingQueueItemsForRun>[0],
@@ -223,8 +220,6 @@ export function connectGateway(host: GatewayHost, options?: ConnectGatewayOption
);
shutdownHost.resumeChatQueueAfterReconnect = true;
} else {
// Preserve any still-live approvals that were already staged in UI state.
// Initial connect can happen after a soft reload while an approval is pending.
host.execApprovalQueue = pruneExecApprovalQueue(host.execApprovalQueue);
}
host.execApprovalError = null;
@@ -236,8 +231,8 @@ export function connectGateway(host: GatewayHost, options?: ConnectGatewayOption
});
const client = new GatewayBrowserClient({
url: host.settings.gatewayUrl,
token: normalizeOptionalString(host.settings.token) ? host.settings.token : undefined,
password: normalizeOptionalString(host.password) ? host.password : undefined,
token: host.settings.token.trim() ? host.settings.token : undefined,
password: host.password.trim() ? host.password : undefined,
clientName: "openclaw-control-ui",
clientVersion,
mode: "webchat",
@@ -342,12 +337,12 @@ function handleTerminalChatEvent(
// Check if tool events were seen before resetting (resetToolStream clears toolStreamOrder).
const toolHost = host as unknown as Parameters<typeof resetToolStream>[0];
const hadToolEvents = toolHost.toolStreamOrder.length > 0;
resetToolStream(toolHost);
const flushQueue = () =>
void flushChatQueueForEvent(host as unknown as Parameters<typeof flushChatQueueForEvent>[0]);
clearPendingQueueItemsForRun(
host as unknown as Parameters<typeof clearPendingQueueItemsForRun>[0],
payload?.runId,
);
void flushChatQueueForEvent(host as unknown as Parameters<typeof flushChatQueueForEvent>[0]);
const runId = payload?.runId;
if (runId && host.refreshSessionsAfterChat.has(runId)) {
host.refreshSessionsAfterChat.delete(runId);
@@ -360,9 +355,18 @@ function handleTerminalChatEvent(
// Reload history when tools were used so the persisted tool results
// replace the now-cleared streaming state.
if (hadToolEvents && state === "final") {
void loadChatHistory(host as unknown as ChatState);
const completedRunId = runId ?? null;
void loadChatHistory(host as unknown as ChatState).finally(() => {
if (completedRunId && host.chatRunId && host.chatRunId !== completedRunId) {
return;
}
resetToolStream(toolHost);
flushQueue();
});
return true;
}
resetToolStream(toolHost);
flushQueue();
return false;
}
@@ -437,7 +441,10 @@ function handleGatewayEventUnsafe(host: GatewayHost, evt: GatewayEventFrame) {
if (evt.event === "shutdown") {
const payload = evt.payload as { reason?: unknown; restartExpectedMs?: unknown } | undefined;
const reason = normalizeOptionalString(payload?.reason) ?? "gateway stopping";
const reason =
payload && typeof payload.reason === "string" && payload.reason.trim()
? payload.reason.trim()
: "gateway stopping";
const shutdownMessage =
typeof payload?.restartExpectedMs === "number"
? `Restarting: ${reason}`

1
ui/src/ui/app-lifecycle.node.test.ts
View File

@@ -11,6 +11,7 @@ function createHost() {
assistantName: "OpenClaw",
assistantAvatar: null,
assistantAgentId: null,
localMediaPreviewRoots: [],
chatHasAutoScrolled: false,
chatManualRefreshInFlight: false,
chatLoading: false,

3
ui/src/ui/app-lifecycle.ts
View File

@@ -28,6 +28,9 @@ type LifecycleHost = {
assistantAvatar: string | null;
assistantAgentId: string | null;
serverVersion: string | null;
localMediaPreviewRoots: string[];
embedSandboxMode: "strict" | "scripts" | "trusted";
allowExternalEmbedUrls: boolean;
chatHasAutoScrolled: boolean;
chatManualRefreshInFlight: boolean;
chatLoading: boolean;

30
ui/src/ui/app-render.helpers.node.test.ts
View File

@@ -23,6 +23,7 @@ vi.mock("./controllers/sessions.ts", () => ({
import {
isCronSessionKey,
parseSessionKey,
resolveAssistantAttachmentAuthToken,
resolveSessionDisplayName,
switchChatSession,
} from "./app-render.helpers.ts";
@@ -123,6 +124,35 @@ describe("parseSessionKey", () => {
});
});
describe("resolveAssistantAttachmentAuthToken", () => {
it("prefers the explicit gateway token when present", () => {
expect(
resolveAssistantAttachmentAuthToken({
settings: { token: "session-token" } as AppViewState["settings"],
password: "shared-password",
}),
).toBe("session-token");
});
it("falls back to the shared password when token is blank", () => {
expect(
resolveAssistantAttachmentAuthToken({
settings: { token: " " } as AppViewState["settings"],
password: "shared-password",
}),
).toBe("shared-password");
});
it("returns null when neither auth secret is available", () => {
expect(
resolveAssistantAttachmentAuthToken({
settings: { token: "" } as AppViewState["settings"],
password: " ",
}),
).toBeNull();
});
});
/* ================================================================
* resolveSessionDisplayName – full resolution with row data
* ================================================================ */

8
ui/src/ui/app-render.helpers.ts
View File

@@ -44,6 +44,14 @@ type ChatRefreshHost = AppViewState & {
updateComplete?: Promise<unknown>;
};
export function resolveAssistantAttachmentAuthToken(state: Pick<AppViewState, "settings" | "password">) {
return (
normalizeOptionalString(state.settings.token) ??
normalizeOptionalString(state.password) ??
null
);
}
function resolveSidebarChatSessionKey(state: AppViewState): string {
const snapshot = state.hello?.snapshot as
| { sessionDefaults?: SessionDefaultsSnapshot }

22
ui/src/ui/app-render.ts
View File

@@ -1,4 +1,9 @@
import { html, nothing } from "lit";
import {
buildAgentMainSessionKey,
parseAgentSessionKey,
resolveAgentIdFromSessionKey,
} from "../../../src/routing/session-key.js";
import { t } from "../i18n/index.ts";
import { getSafeLocalStorage } from "../local-storage.ts";
import { refreshChatAvatar } from "./app-chat.ts";
@@ -8,6 +13,7 @@ import {
renderChatMobileToggle,
renderChatSessionSelect,
renderTab,
resolveAssistantAttachmentAuthToken,
renderSidebarConnectionStatus,
renderTopbarThemeModeToggle,
switchChatSession,
@@ -105,15 +111,10 @@ import {
updateSkillEdit,
updateSkillEnabled,
} from "./controllers/skills.ts";
import { buildExternalLinkRel, EXTERNAL_LINK_TARGET } from "./external-link.ts";
import "./components/dashboard-header.ts";
import { buildExternalLinkRel, EXTERNAL_LINK_TARGET } from "./external-link.ts";
import { icons } from "./icons.ts";
import { normalizeBasePath, TAB_GROUPS, subtitleForTab, titleForTab } from "./navigation.ts";
import {
buildAgentMainSessionKey,
parseAgentSessionKey,
resolveAgentIdFromSessionKey,
} from "./session-key.ts";
import { agentLogoUrl } from "./views/agents-utils.ts";
import {
resolveAgentConfig,
@@ -188,7 +189,6 @@ function resolveDreamingNextCycle(
}
let clawhubSearchTimer: ReturnType<typeof setTimeout> | null = null;
function lazyRender<M>(getter: () => M | null, render: (mod: M) => unknown) {
const mod = getter();
return mod ? render(mod) : nothing;
@@ -1847,6 +1847,7 @@ export function renderApp(state: AppViewState) {
error: state.lastError,
sessions: state.sessionsResult,
focusMode: chatFocus,
autoExpandToolCalls: false,
onRefresh: () => {
state.chatSideResult = null;
state.resetToolStream();
@@ -1909,11 +1910,16 @@ export function renderApp(state: AppViewState) {
sidebarContent: state.sidebarContent,
sidebarError: state.sidebarError,
splitRatio: state.splitRatio,
onOpenSidebar: (content: string) => state.handleOpenSidebar(content),
canvasHostUrl: state.hello?.canvasHostUrl ?? null,
onOpenSidebar: (content) => state.handleOpenSidebar(content),
onCloseSidebar: () => state.handleCloseSidebar(),
onSplitRatioChange: (ratio: number) => state.handleSplitRatioChange(ratio),
assistantName: state.assistantName,
assistantAvatar: state.assistantAvatar,
localMediaPreviewRoots: state.localMediaPreviewRoots,
embedSandboxMode: state.embedSandboxMode,
allowExternalEmbedUrls: state.allowExternalEmbedUrls,
assistantAttachmentAuthToken: resolveAssistantAttachmentAuthToken(state),
basePath: state.basePath ?? "",
})
: nothing}

15
ui/src/ui/app-view-state.ts
View File

@@ -10,8 +10,10 @@ import type {
ClawHubSkillDetail,
SkillMessage,
} from "./controllers/skills.ts";
import type { EmbedSandboxMode } from "./embed-sandbox.ts";
import type { GatewayBrowserClient, GatewayHelloOk } from "./gateway.ts";
import type { Tab } from "./navigation.ts";
import type { SidebarContent } from "./sidebar-content.ts";
import type { UiSettings } from "./storage.ts";
import type { ThemeTransitionContext } from "./theme-transition.ts";
import type { ResolvedTheme, ThemeMode, ThemeName } from "./theme.ts";
@@ -34,6 +36,7 @@ import type {
CostUsageSummary,
SessionUsageTimeSeries,
SessionsListResult,
SessionCompactionCheckpoint,
SkillStatusReport,
StatusSummary,
ToolsCatalogResult,
@@ -62,6 +65,9 @@ export type AppViewState = {
assistantName: string;
assistantAvatar: string | null;
assistantAgentId: string | null;
localMediaPreviewRoots: string[];
embedSandboxMode: EmbedSandboxMode;
allowExternalEmbedUrls: boolean;
sessionKey: string;
chatLoading: boolean;
chatSending: boolean;
@@ -89,7 +95,7 @@ export type AppViewState = {
chatNewMessagesBelow: boolean;
navDrawerOpen: boolean;
sidebarOpen: boolean;
sidebarContent: string | null;
sidebarContent: SidebarContent | null;
sidebarError: string | null;
splitRatio: number;
scrollToBottom: (opts?: { smooth?: boolean }) => void;
@@ -208,6 +214,9 @@ export type AppViewState = {
sessionsLoading: boolean;
sessionsResult: SessionsListResult | null;
sessionsError: string | null;
threadsLoading: boolean;
threadsResult: SessionsListResult | null;
threadsError: string | null;
sessionsFilterActive: string;
sessionsFilterLimit: string;
sessionsIncludeGlobal: boolean;
@@ -220,7 +229,7 @@ export type AppViewState = {
sessionsPageSize: number;
sessionsSelectedKeys: Set<string>;
sessionsExpandedCheckpointKey: string | null;
sessionsCheckpointItemsByKey: Record<string, import("./types.ts").SessionCompactionCheckpoint[]>;
sessionsCheckpointItemsByKey: Record<string, SessionCompactionCheckpoint[]>;
sessionsCheckpointLoadingKey: string | null;
sessionsCheckpointBusyKey: string | null;
sessionsCheckpointErrorByKey: Record<string, string>;
@@ -410,7 +419,7 @@ export type AppViewState = {
resetChatScroll: () => void;
exportLogs: (lines: string[], label: string) => void;
handleLogsScroll: (event: Event) => void;
handleOpenSidebar: (content: string) => void;
handleOpenSidebar: (content: SidebarContent) => void;
handleCloseSidebar: () => void;
handleSplitRatioChange: (ratio: number) => void;
};

17
ui/src/ui/app.ts
View File

@@ -1,5 +1,6 @@
import { LitElement } from "lit";
import { state } from "lit/decorators.js";
import { customElement, state } from "lit/decorators.js";
import { resolveAgentIdFromSessionKey } from "../../../src/routing/session-key.js";
import { i18n, I18nController, isSupportedLocale } from "../i18n/index.ts";
import {
handleChannelConfigReload as handleChannelConfigReloadInternal,
@@ -75,7 +76,7 @@ import type {
} from "./controllers/skills.ts";
import type { GatewayBrowserClient, GatewayHelloOk } from "./gateway.ts";
import type { Tab } from "./navigation.ts";
import { resolveAgentIdFromSessionKey } from "./session-key.ts";
import type { SidebarContent } from "./sidebar-content.ts";
import { loadSettings, type UiSettings } from "./storage.ts";
import { VALID_THEME_NAMES, type ResolvedTheme, type ThemeMode, type ThemeName } from "./theme.ts";
import type {
@@ -127,6 +128,7 @@ function resolveOnboardingMode(): boolean {
return normalized === "1" || normalized === "true" || normalized === "yes" || normalized === "on";
}
@customElement("openclaw-app")
export class OpenClawApp extends LitElement {
private i18nController = new I18nController(this);
clientInstanceId = generateUUID();
@@ -159,6 +161,9 @@ export class OpenClawApp extends LitElement {
@state() assistantName = bootAssistantIdentity.name;
@state() assistantAvatar = bootAssistantIdentity.avatar;
@state() assistantAgentId = bootAssistantIdentity.agentId ?? null;
@state() localMediaPreviewRoots: string[] = [];
@state() embedSandboxMode: "strict" | "scripts" | "trusted" = "scripts";
@state() allowExternalEmbedUrls = false;
@state() serverVersion: string | null = null;
@state() sessionKey = this.settings.sessionKey;
@@ -188,7 +193,7 @@ export class OpenClawApp extends LitElement {
// Sidebar state for tool output viewing
@state() sidebarOpen = false;
@state() sidebarContent: string | null = null;
@state() sidebarContent: SidebarContent | null = null;
@state() sidebarError: string | null = null;
@state() splitRatio = this.settings.splitRatio;
@@ -767,7 +772,7 @@ export class OpenClawApp extends LitElement {
}
// Sidebar handlers for tool output viewing
handleOpenSidebar(content: string) {
handleOpenSidebar(content: SidebarContent) {
if (this.sidebarCloseTimer != null) {
window.clearTimeout(this.sidebarCloseTimer);
this.sidebarCloseTimer = null;
@@ -803,7 +808,3 @@ export class OpenClawApp extends LitElement {
return renderApp(this as unknown as AppViewState);
}
}
if (!customElements.get("openclaw-app")) {
customElements.define("openclaw-app", OpenClawApp);
}

39
ui/src/ui/canvas-url.test.ts Normal file
View File

@@ -0,0 +1,39 @@
import { describe, expect, it } from "vitest";
import { resolveCanvasIframeUrl } from "./canvas-url.ts";
describe("resolveCanvasIframeUrl", () => {
it("allows same-origin hosted canvas document paths", () => {
expect(resolveCanvasIframeUrl("/__openclaw__/canvas/documents/cv_demo/index.html")).toBe(
"/__openclaw__/canvas/documents/cv_demo/index.html",
);
});
it("rewrites safe canvas paths through the scoped canvas host", () => {
expect(
resolveCanvasIframeUrl(
"/__openclaw__/canvas/documents/cv_demo/index.html",
"http://127.0.0.1:19003/__openclaw__/cap/cap_123",
),
).toBe(
"http://127.0.0.1:19003/__openclaw__/cap/cap_123/__openclaw__/canvas/documents/cv_demo/index.html",
);
});
it("rejects non-canvas same-origin paths", () => {
expect(resolveCanvasIframeUrl("/not-canvas/snake.html")).toBeUndefined();
});
it("rejects absolute external URLs", () => {
expect(resolveCanvasIframeUrl("https://example.com/evil.html")).toBeUndefined();
});
it("allows absolute external URLs only when explicitly enabled", () => {
expect(resolveCanvasIframeUrl("https://example.com/embed.html?x=1#y", undefined, true)).toBe(
"https://example.com/embed.html?x=1#y",
);
});
it("rejects file URLs", () => {
expect(resolveCanvasIframeUrl("file:///tmp/snake.html")).toBeUndefined();
});
});

74
ui/src/ui/canvas-url.ts Normal file
View File

@@ -0,0 +1,74 @@
const A2UI_PATH = "/__openclaw__/a2ui";
const CANVAS_HOST_PATH = "/__openclaw__/canvas";
const CANVAS_CAPABILITY_PATH_PREFIX = "/__openclaw__/cap";
function isCanvasHttpPath(pathname: string): boolean {
return (
pathname === CANVAS_HOST_PATH ||
pathname.startsWith(`${CANVAS_HOST_PATH}/`) ||
pathname === A2UI_PATH ||
pathname.startsWith(`${A2UI_PATH}/`)
);
}
function isExternalHttpUrl(entry: URL): boolean {
return entry.protocol === "http:" || entry.protocol === "https:";
}
function sanitizeCanvasEntryUrl(
rawEntryUrl: string,
allowExternalEmbedUrls = false,
): string | undefined {
try {
const entry = new URL(rawEntryUrl, "http://localhost");
if (entry.origin !== "http://localhost") {
if (!allowExternalEmbedUrls || !isExternalHttpUrl(entry)) {
return undefined;
}
return entry.toString();
}
if (!isCanvasHttpPath(entry.pathname)) {
return undefined;
}
return `${entry.pathname}${entry.search}${entry.hash}`;
} catch {
return undefined;
}
}
export function resolveCanvasIframeUrl(
entryUrl: string | undefined,
canvasHostUrl?: string | null,
allowExternalEmbedUrls = false,
): string | undefined {
const rawEntryUrl = entryUrl?.trim();
if (!rawEntryUrl) {
return undefined;
}
const safeEntryUrl = sanitizeCanvasEntryUrl(rawEntryUrl, allowExternalEmbedUrls);
if (!safeEntryUrl) {
return undefined;
}
if (!canvasHostUrl?.trim()) {
return safeEntryUrl;
}
try {
const scopedHostUrl = new URL(canvasHostUrl);
const scopedPrefix = scopedHostUrl.pathname.replace(/\/+$/, "");
if (!scopedPrefix.startsWith(CANVAS_CAPABILITY_PATH_PREFIX)) {
return safeEntryUrl;
}
const entry = new URL(safeEntryUrl, scopedHostUrl.origin);
if (!isCanvasHttpPath(entry.pathname)) {
return safeEntryUrl;
}
entry.protocol = scopedHostUrl.protocol;
entry.username = scopedHostUrl.username;
entry.password = scopedHostUrl.password;
entry.host = scopedHostUrl.host;
entry.pathname = `${scopedPrefix}${entry.pathname}`;
return entry.toString();
} catch {
return safeEntryUrl;
}
}

4
ui/src/ui/chat-event-reload.test.ts
View File

@@ -23,7 +23,7 @@ describe("shouldReloadHistoryForFinalEvent", () => {
).toBe(true);
});
it("returns false when final event includes assistant payload", () => {
it("returns true when final event includes assistant payload", () => {
expect(
shouldReloadHistoryForFinalEvent({
runId: "run-1",
@@ -31,7 +31,7 @@ describe("shouldReloadHistoryForFinalEvent", () => {
state: "final",
message: { role: "assistant", content: [{ type: "text", text: "done" }] },
}),
).toBe(false);
).toBe(true);
});
it("returns true when final event message role is non-assistant", () => {

14
ui/src/ui/chat-event-reload.ts
View File

@@ -1,17 +1,5 @@
import type { ChatEventPayload } from "./controllers/chat.ts";
import { normalizeLowercaseStringOrEmpty } from "./string-coerce.ts";
export function shouldReloadHistoryForFinalEvent(payload?: ChatEventPayload): boolean {
if (!payload || payload.state !== "final") {
return false;
}
if (!payload.message || typeof payload.message !== "object") {
return true;
}
const message = payload.message as Record<string, unknown>;
const role = normalizeLowercaseStringOrEmpty(message.role);
if (role && role !== "assistant") {
return true;
}
return false;
return Boolean(payload && payload.state === "final");
}

19
ui/src/ui/chat-markdown.browser.test.ts
View File

@@ -22,16 +22,19 @@ describe("chat markdown rendering", () => {
await app.updateComplete;
const toolCards = Array.from(app.querySelectorAll<HTMLElement>(".chat-tool-card"));
const toolCard = toolCards.find((card) =>
card.querySelector(".chat-tool-card__preview, .chat-tool-card__inline"),
);
expect(toolCard).not.toBeUndefined();
toolCard?.click();
const toolSummary = app.querySelector<HTMLElement>(".chat-tool-msg-summary");
expect(toolSummary).not.toBeNull();
toolSummary?.click();
await app.updateComplete;
const strong = app.querySelector(".sidebar-markdown strong");
expect(strong?.textContent).toBe("world");
const openSidebarButton = app.querySelector<HTMLElement>(".chat-tool-card__action-btn");
expect(openSidebarButton).not.toBeNull();
openSidebarButton?.click();
await app.updateComplete;
const strongNodes = Array.from(app.querySelectorAll(".sidebar-markdown strong"));
expect(strongNodes.some((node) => node.textContent === "world")).toBe(true);
});
});

782
ui/src/ui/chat/grouped-render.ts
View File

@@ -2,12 +2,18 @@ import { html, nothing } from "lit";
import { unsafeHTML } from "lit/directives/unsafe-html.js";
import { getSafeLocalStorage } from "../../local-storage.ts";
import type { AssistantIdentity } from "../assistant-identity.ts";
import type { EmbedSandboxMode } from "../embed-sandbox.ts";
import { icons } from "../icons.ts";
import { toSanitizedMarkdownHtml } from "../markdown.ts";
import { openExternalUrlSafe } from "../open-external-url.ts";
import { normalizeLowercaseStringOrEmpty } from "../string-coerce.ts";
import type { SidebarContent } from "../sidebar-content.ts";
import { detectTextDirection } from "../text-direction.ts";
import type { MessageGroup, ToolCard } from "../types/chat-types.ts";
import type {
MessageContentItem,
MessageGroup,
NormalizedMessage,
ToolCard,
} from "../types/chat-types.ts";
import { agentLogoUrl } from "../views/agents-utils.ts";
import { renderCopyAsMarkdownButton } from "./copy-as-markdown.ts";
import {
@@ -15,19 +21,37 @@ import {
extractThinkingCached,
formatReasoningMarkdown,
} from "./message-extract.ts";
import { isToolResultMessage, normalizeRoleForGrouping } from "./message-normalizer.ts";
import {
isToolResultMessage,
normalizeMessage,
normalizeRoleForGrouping,
} from "./message-normalizer.ts";
import { isTtsSupported, speakText, stopTts, isTtsSpeaking } from "./speech.ts";
import { extractToolCards, renderToolCardSidebar } from "./tool-cards.ts";
import {
extractToolCards,
renderExpandedToolCardContent,
renderRawOutputToggle,
renderToolCard,
renderToolPreview,
} from "./tool-cards.ts";
type AssistantAttachmentAvailability =
| { status: "checking" }
| { status: "available" }
| { status: "unavailable"; reason: string; checkedAt: number };
const assistantAttachmentAvailabilityCache = new Map<string, AssistantAttachmentAvailability>();
const ASSISTANT_ATTACHMENT_UNAVAILABLE_RETRY_MS = 5_000;
export function resetAssistantAttachmentAvailabilityCacheForTest() {
assistantAttachmentAvailabilityCache.clear();
}
type ImageBlock = {
url: string;
alt?: string;
};
type AudioClip = {
url: string;
};
function extractImages(message: unknown): ImageBlock[] {
const m = message as Record<string, unknown>;
const content = m.content;
@@ -65,32 +89,6 @@ function extractImages(message: unknown): ImageBlock[] {
return images;
}
function extractAudioClips(message: unknown): AudioClip[] {
const m = message as Record<string, unknown>;
const content = m.content;
const clips: AudioClip[] = [];
if (!Array.isArray(content)) {
return clips;
}
for (const block of content) {
if (typeof block !== "object" || block === null) {
continue;
}
const b = block as Record<string, unknown>;
if (b.type !== "audio") {
continue;
}
const source = b.source as Record<string, unknown> | undefined;
if (source?.type === "base64" && typeof source.data === "string") {
const data = source.data;
const mediaType = (source.media_type as string) || "audio/mpeg";
const url = data.startsWith("data:") ? data : `data:${mediaType};base64,${data}`;
clips.push({ url });
}
}
return clips;
}
export function renderReadingIndicatorGroup(assistant?: AssistantIdentity, basePath?: string) {
return html`
<div class="chat-group assistant">
@@ -109,7 +107,7 @@ export function renderReadingIndicatorGroup(assistant?: AssistantIdentity, baseP
export function renderStreamingGroup(
text: string,
startedAt: number,
onOpenSidebar?: (content: string) => void,
onOpenSidebar?: (content: SidebarContent) => void,
assistant?: AssistantIdentity,
basePath?: string,
) {
@@ -129,6 +127,7 @@ export function renderStreamingGroup(
content: [{ type: "text", text }],
timestamp: startedAt,
},
`stream:${startedAt}`,
{ isStreaming: true, showReasoning: false },
onOpenSidebar,
)}
@@ -144,12 +143,23 @@ export function renderStreamingGroup(
export function renderMessageGroup(
group: MessageGroup,
opts: {
onOpenSidebar?: (content: string) => void;
onOpenSidebar?: (content: SidebarContent) => void;
showReasoning: boolean;
showToolCalls?: boolean;
autoExpandToolCalls?: boolean;
isToolMessageExpanded?: (messageId: string) => boolean;
onToggleToolMessageExpanded?: (messageId: string) => void;
isToolExpanded?: (toolCardId: string) => boolean;
onToggleToolExpanded?: (toolCardId: string) => void;
onRequestUpdate?: () => void;
assistantName?: string;
assistantAvatar?: string | null;
basePath?: string;
localMediaPreviewRoots?: readonly string[];
assistantAttachmentAuthToken?: string | null;
canvasHostUrl?: string | null;
embedSandboxMode?: EmbedSandboxMode;
allowExternalEmbedUrls?: boolean;
contextWindow?: number | null;
onDelete?: () => void;
},
@@ -195,10 +205,22 @@ export function renderMessageGroup(
${group.messages.map((item, index) =>
renderGroupedMessage(
item.message,
item.key,
{
isStreaming: group.isStreaming && index === group.messages.length - 1,
showReasoning: opts.showReasoning,
showToolCalls: opts.showToolCalls ?? true,
autoExpandToolCalls: opts.autoExpandToolCalls ?? false,
isToolMessageExpanded: opts.isToolMessageExpanded,
onToggleToolMessageExpanded: opts.onToggleToolMessageExpanded,
isToolExpanded: opts.isToolExpanded,
onToggleToolExpanded: opts.onToggleToolExpanded,
onRequestUpdate: opts.onRequestUpdate,
canvasHostUrl: opts.canvasHostUrl,
basePath: opts.basePath,
localMediaPreviewRoots: opts.localMediaPreviewRoots,
assistantAttachmentAuthToken: opts.assistantAttachmentAuthToken,
embedSandboxMode: opts.embedSandboxMode,
},
opts.onOpenSidebar,
),
@@ -346,15 +368,9 @@ function extractGroupText(group: MessageGroup): string {
return parts.join("\n\n");
}
export const SKIP_DELETE_CONFIRM_KEY = "openclaw:skipDeleteConfirm";
const SKIP_DELETE_CONFIRM_KEY = "openclaw:skipDeleteConfirm";
type DeleteConfirmSide = "left" | "right";
type DeleteConfirmPopover = {
popover: HTMLDivElement;
cancel: HTMLButtonElement;
yes: HTMLButtonElement;
check: HTMLInputElement;
};
function shouldSkipDeleteConfirm(): boolean {
try {
@@ -364,45 +380,6 @@ function shouldSkipDeleteConfirm(): boolean {
}
}
function createDeleteConfirmPopover(side: DeleteConfirmSide): DeleteConfirmPopover {
const popover = document.createElement("div");
popover.className = `chat-delete-confirm chat-delete-confirm--${side}`;
const text = document.createElement("p");
text.className = "chat-delete-confirm__text";
text.textContent = "Delete this message?";
const remember = document.createElement("label");
remember.className = "chat-delete-confirm__remember";
const check = document.createElement("input");
check.className = "chat-delete-confirm__check";
check.type = "checkbox";
const rememberText = document.createElement("span");
rememberText.textContent = "Don't ask again";
remember.append(check, rememberText);
const actions = document.createElement("div");
actions.className = "chat-delete-confirm__actions";
const cancel = document.createElement("button");
cancel.className = "chat-delete-confirm__cancel";
cancel.type = "button";
cancel.textContent = "Cancel";
const yes = document.createElement("button");
yes.className = "chat-delete-confirm__yes";
yes.type = "button";
yes.textContent = "Delete";
actions.append(cancel, yes);
popover.append(text, remember, actions);
return { popover, cancel, yes, check };
}
function renderDeleteButton(onDelete: () => void, side: DeleteConfirmSide) {
return html`
<span class="chat-delete-wrap">
@@ -422,31 +399,43 @@ function renderDeleteButton(onDelete: () => void, side: DeleteConfirmSide) {
existing.remove();
return;
}
const { popover, cancel, yes, check } = createDeleteConfirmPopover(side);
const popover = document.createElement("div");
popover.className = `chat-delete-confirm chat-delete-confirm--${side}`;
popover.innerHTML = `
<p class="chat-delete-confirm__text">Delete this message?</p>
<label class="chat-delete-confirm__remember">
<input type="checkbox" class="chat-delete-confirm__check" />
<span>Don't ask again</span>
</label>
<div class="chat-delete-confirm__actions">
<button class="chat-delete-confirm__cancel" type="button">Cancel</button>
<button class="chat-delete-confirm__yes" type="button">Delete</button>
</div>
`;
wrap.appendChild(popover);
const removePopover = () => {
popover.remove();
document.removeEventListener("click", closeOnOutside, true);
};
const cancel = popover.querySelector(".chat-delete-confirm__cancel")!;
const yes = popover.querySelector(".chat-delete-confirm__yes")!;
const check = popover.querySelector(".chat-delete-confirm__check") as HTMLInputElement;
// Close on click outside.
const closeOnOutside = (evt: MouseEvent) => {
if (!popover.contains(evt.target as Node) && evt.target !== btn) {
removePopover();
}
};
cancel.addEventListener("click", removePopover);
cancel.addEventListener("click", () => popover.remove());
yes.addEventListener("click", () => {
if (check.checked) {
try {
getSafeLocalStorage()?.setItem(SKIP_DELETE_CONFIRM_KEY, "1");
} catch {}
}
removePopover();
popover.remove();
onDelete();
});
// Close on click outside
const closeOnOutside = (evt: MouseEvent) => {
if (!popover.contains(evt.target as Node) && evt.target !== btn) {
popover.remove();
document.removeEventListener("click", closeOnOutside, true);
}
};
requestAnimationFrame(() => document.addEventListener("click", closeOnOutside, true));
}}
>
@@ -611,52 +600,372 @@ function renderMessageImages(images: ImageBlock[]) {
`;
}
function renderMessageAudio(clips: AudioClip[]) {
if (clips.length === 0) {
function renderReplyPill(replyTarget: NormalizedMessage["replyTarget"]) {
if (!replyTarget) {
return nothing;
}
return html`
<div class="chat-message-audio">
${clips.map(
(clip) =>
html`<audio
class="chat-message-audio-el"
controls
preload="metadata"
src=${clip.url}
></audio>`,
)}
<div class="chat-reply-pill">
<span class="chat-reply-pill__icon">${icons.messageSquare}</span>
<span class="chat-reply-pill__label">
${replyTarget.kind === "current"
? "Replying to current message"
: `Replying to ${replyTarget.id}`}
</span>
</div>
`;
}
/** Render tool cards inside a collapsed `<details>` element. */
function renderCollapsedToolCards(
toolCards: ToolCard[],
onOpenSidebar?: (content: string) => void,
) {
const calls = toolCards.filter((c) => c.kind === "call");
const results = toolCards.filter((c) => c.kind === "result");
const totalTools = Math.max(calls.length, results.length) || toolCards.length;
const toolNames = [...new Set(toolCards.map((c) => c.name))];
const summaryLabel =
toolNames.length <= 3
? toolNames.join(", ")
: `${toolNames.slice(0, 2).join(", ")} +${toolNames.length - 2} more`;
function isLocalAssistantAttachmentSource(source: string): boolean {
const trimmed = source.trim();
if (/^\/(?:__openclaw__|media)\//.test(trimmed)) {
return false;
}
return (
trimmed.startsWith("file://") ||
trimmed.startsWith("~") ||
trimmed.startsWith("/") ||
/^[a-zA-Z]:[\\/]/.test(trimmed)
);
}
function normalizeLocalAttachmentPath(source: string): string | null {
const trimmed = source.trim();
if (!isLocalAssistantAttachmentSource(trimmed)) {
return null;
}
if (trimmed.startsWith("file://")) {
try {
const url = new URL(trimmed);
const pathname = decodeURIComponent(url.pathname);
if (/^\/[a-zA-Z]:\//.test(pathname)) {
return pathname.slice(1);
}
return pathname;
} catch {
return null;
}
}
if (trimmed.startsWith("~")) {
return null;
}
return trimmed;
}
function resolveHomeCandidatesFromRoots(localMediaPreviewRoots: readonly string[]): string[] {
const candidates = new Set<string>();
for (const root of localMediaPreviewRoots) {
const normalized = canonicalizeLocalPathForComparison(root.trim());
const unixHome = normalized.match(/^(\/Users\/[^/]+|\/home\/[^/]+)(?:\/|$)/);
if (unixHome?.[1]) {
candidates.add(unixHome[1]);
continue;
}
const windowsHome = normalized.match(/^([a-z]:\/Users\/[^/]+)(?:\/|$)/i);
if (windowsHome?.[1]) {
candidates.add(windowsHome[1]);
}
}
return [...candidates];
}
function canonicalizeLocalPathForComparison(value: string): string {
let slashNormalized = value.replace(/\\/g, "/").replace(/\/+$/, "");
if (/^\/[a-zA-Z]:\//.test(slashNormalized)) {
slashNormalized = slashNormalized.slice(1);
}
if (/^[a-zA-Z]:\//.test(slashNormalized)) {
return slashNormalized.toLowerCase();
}
return slashNormalized;
}
function isLocalAttachmentPreviewAllowed(
source: string,
localMediaPreviewRoots: readonly string[],
): boolean {
const normalizedSource = normalizeLocalAttachmentPath(source);
const comparableSources = normalizedSource
? [canonicalizeLocalPathForComparison(normalizedSource)]
: source.trim().startsWith("~")
? resolveHomeCandidatesFromRoots(localMediaPreviewRoots).map((home) =>
canonicalizeLocalPathForComparison(source.trim().replace(/^~(?=$|[\\/])/, home)),
)
: [];
if (comparableSources.length === 0) {
return false;
}
return localMediaPreviewRoots.some((root) => {
const normalizedRoot = canonicalizeLocalPathForComparison(root.trim());
return (
normalizedRoot.length > 0 &&
comparableSources.some(
(comparableSource) =>
comparableSource === normalizedRoot || comparableSource.startsWith(`${normalizedRoot}/`),
)
);
});
}
function buildAssistantAttachmentUrl(
source: string,
basePath?: string,
authToken?: string | null,
): string {
if (!isLocalAssistantAttachmentSource(source)) {
return source;
}
const normalizedBasePath =
basePath && basePath !== "/" ? (basePath.endsWith("/") ? basePath.slice(0, -1) : basePath) : "";
const params = new URLSearchParams({ source });
const normalizedToken = authToken?.trim();
if (normalizedToken) {
params.set("token", normalizedToken);
}
return `${normalizedBasePath}/__openclaw__/assistant-media?${params.toString()}`;
}
function buildAssistantAttachmentMetaUrl(
source: string,
basePath?: string,
authToken?: string | null,
): string {
const attachmentUrl = buildAssistantAttachmentUrl(source, basePath, authToken);
return `${attachmentUrl}${attachmentUrl.includes("?") ? "&" : "?"}meta=1`;
}
function resolveAssistantAttachmentAvailability(
source: string,
localMediaPreviewRoots: readonly string[],
basePath: string | undefined,
authToken: string | null | undefined,
onRequestUpdate: (() => void) | undefined,
): AssistantAttachmentAvailability {
if (!isLocalAssistantAttachmentSource(source)) {
return { status: "available" };
}
if (!isLocalAttachmentPreviewAllowed(source, localMediaPreviewRoots)) {
return { status: "unavailable", reason: "Outside allowed folders", checkedAt: Date.now() };
}
const normalizedAuthToken = authToken?.trim() ?? "";
const cacheKey = `${basePath ?? ""}::${normalizedAuthToken}::${source}`;
const cached = assistantAttachmentAvailabilityCache.get(cacheKey);
if (cached) {
if (
cached.status === "unavailable" &&
Date.now() - cached.checkedAt >= ASSISTANT_ATTACHMENT_UNAVAILABLE_RETRY_MS
) {
assistantAttachmentAvailabilityCache.delete(cacheKey);
} else {
return cached;
}
}
assistantAttachmentAvailabilityCache.set(cacheKey, { status: "checking" });
if (typeof fetch === "function") {
void fetch(buildAssistantAttachmentMetaUrl(source, basePath, authToken), {
method: "GET",
headers: { Accept: "application/json" },
credentials: "same-origin",
})
.then(async (res) => {
const payload = (await res.json().catch(() => null)) as {
available?: boolean;
reason?: string;
} | null;
if (payload?.available === true) {
assistantAttachmentAvailabilityCache.set(cacheKey, { status: "available" });
} else {
assistantAttachmentAvailabilityCache.set(cacheKey, {
status: "unavailable",
reason: payload?.reason?.trim() || "Attachment unavailable",
checkedAt: Date.now(),
});
}
})
.catch(() => {
assistantAttachmentAvailabilityCache.set(cacheKey, {
status: "unavailable",
reason: "Attachment unavailable",
checkedAt: Date.now(),
});
})
.finally(() => {
onRequestUpdate?.();
});
}
return { status: "checking" };
}
function renderAssistantAttachmentStatusCard(params: {
kind: "image" | "audio" | "video" | "document";
label: string;
badge: string;
reason?: string;
}) {
const icon =
params.kind === "image"
? icons.image
: params.kind === "audio"
? icons.mic
: params.kind === "video"
? icons.monitor
: icons.paperclip;
return html`
<details class="chat-tools-collapse">
<summary class="chat-tools-summary">
<span class="chat-tools-summary__icon">${icons.zap}</span>
<span class="chat-tools-summary__count"
>${totalTools} tool${totalTools === 1 ? "" : "s"}</span
<div class="chat-assistant-attachment-card chat-assistant-attachment-card--blocked">
<div class="chat-assistant-attachment-card__header">
<span class="chat-assistant-attachment-card__icon">${icon}</span>
<span class="chat-assistant-attachment-card__title">${params.label}</span>
<span class="chat-assistant-attachment-badge chat-assistant-attachment-badge--muted"
>${params.badge}</span
>
<span class="chat-tools-summary__names">${summaryLabel}</span>
</summary>
<div class="chat-tools-collapse__body">
${toolCards.map((card) => renderToolCardSidebar(card, onOpenSidebar))}
</div>
</details>
${params.reason
? html`<div class="chat-assistant-attachment-card__reason">${params.reason}</div>`
: nothing}
</div>
`;
}
function renderAssistantAttachments(
attachments: Array<Extract<MessageContentItem, { type: "attachment" }>>,
localMediaPreviewRoots: readonly string[],
basePath?: string,
authToken?: string | null,
onRequestUpdate?: () => void,
) {
if (attachments.length === 0) {
return nothing;
}
return html`
<div class="chat-assistant-attachments">
${attachments.map(({ attachment }) => {
const availability = resolveAssistantAttachmentAvailability(
attachment.url,
localMediaPreviewRoots,
basePath,
authToken,
onRequestUpdate,
);
const attachmentUrl =
availability.status === "available"
? buildAssistantAttachmentUrl(attachment.url, basePath, authToken)
: null;
if (attachment.kind === "image") {
if (!attachmentUrl) {
return renderAssistantAttachmentStatusCard({
kind: "image",
label: attachment.label,
badge: availability.status === "checking" ? "Checking..." : "Unavailable",
reason: availability.status === "unavailable" ? availability.reason : undefined,
});
}
return html`
<img
src=${attachmentUrl}
alt=${attachment.label}
class="chat-message-image"
@click=${() => openExternalUrlSafe(attachmentUrl, { allowDataImage: true })}
/>
`;
}
if (attachment.kind === "audio") {
return html`
<div class="chat-assistant-attachment-card chat-assistant-attachment-card--audio">
<div class="chat-assistant-attachment-card__header">
<span class="chat-assistant-attachment-card__title">${attachment.label}</span>
${!attachmentUrl
? html`<span
class="chat-assistant-attachment-badge chat-assistant-attachment-badge--muted"
>${availability.status === "checking" ? "Checking..." : "Unavailable"}</span
>`
: attachment.isVoiceNote
? html`<span class="chat-assistant-attachment-badge">Voice note</span>`
: nothing}
</div>
${attachmentUrl
? html`<audio controls preload="metadata" src=${attachmentUrl}></audio>`
: availability.status === "unavailable"
? html`<div class="chat-assistant-attachment-card__reason">
${availability.reason}
</div>`
: nothing}
</div>
`;
}
if (attachment.kind === "video") {
if (!attachmentUrl) {
return renderAssistantAttachmentStatusCard({
kind: "video",
label: attachment.label,
badge: availability.status === "checking" ? "Checking..." : "Unavailable",
reason: availability.status === "unavailable" ? availability.reason : undefined,
});
}
return html`
<div class="chat-assistant-attachment-card chat-assistant-attachment-card--video">
<video controls preload="metadata" src=${attachmentUrl}></video>
<a
class="chat-assistant-attachment-card__link"
href=${attachmentUrl}
target="_blank"
rel="noreferrer"
>${attachment.label}</a
>
</div>
`;
}
if (!attachmentUrl) {
return renderAssistantAttachmentStatusCard({
kind: "document",
label: attachment.label,
badge: availability.status === "checking" ? "Checking..." : "Unavailable",
reason: availability.status === "unavailable" ? availability.reason : undefined,
});
}
return html`
<div class="chat-assistant-attachment-card">
<span class="chat-assistant-attachment-card__icon">${icons.paperclip}</span>
<a
class="chat-assistant-attachment-card__link"
href=${attachmentUrl}
target="_blank"
rel="noreferrer"
>${attachment.label}</a
>
</div>
`;
})}
</div>
`;
}
function renderInlineToolCards(
toolCards: ToolCard[],
opts: {
messageKey: string;
onOpenSidebar?: (content: SidebarContent) => void;
isToolExpanded?: (toolCardId: string) => boolean;
onToggleToolExpanded?: (toolCardId: string) => void;
canvasHostUrl?: string | null;
embedSandboxMode?: EmbedSandboxMode;
allowExternalEmbedUrls?: boolean;
},
) {
return html`
<div class="chat-tools-inline">
${toolCards.map((card, index) =>
renderToolCard(card, {
expanded: opts.isToolExpanded?.(`${opts.messageKey}:toolcard:${index}`) ?? false,
onToggleExpanded: opts.onToggleToolExpanded
? () => opts.onToggleToolExpanded?.(`${opts.messageKey}:toolcard:${index}`)
: () => undefined,
onOpenSidebar: opts.onOpenSidebar,
canvasHostUrl: opts.canvasHostUrl,
embedSandboxMode: opts.embedSandboxMode ?? "scripts",
allowExternalEmbedUrls: opts.allowExternalEmbedUrls ?? false,
}),
)}
</div>
`;
}
@@ -705,14 +1014,14 @@ function jsonSummaryLabel(parsed: unknown): string {
return "JSON";
}
function renderExpandButton(markdown: string, onOpenSidebar: (content: string) => void) {
function renderExpandButton(markdown: string, onOpenSidebar: (content: SidebarContent) => void) {
return html`
<button
class="btn btn--xs chat-expand-btn"
type="button"
title="Open in canvas"
aria-label="Open in canvas"
@click=${() => onOpenSidebar(markdown)}
@click=${() => onOpenSidebar({ kind: "markdown", content: markdown })}
>
<span class="chat-expand-btn__icon" aria-hidden="true">${icons.panelRightOpen}</span>
</button>
@@ -721,28 +1030,58 @@ function renderExpandButton(markdown: string, onOpenSidebar: (content: string) =
function renderGroupedMessage(
message: unknown,
opts: { isStreaming: boolean; showReasoning: boolean; showToolCalls?: boolean },
onOpenSidebar?: (content: string) => void,
messageKey: string,
opts: {
isStreaming: boolean;
showReasoning: boolean;
showToolCalls?: boolean;
autoExpandToolCalls?: boolean;
isToolMessageExpanded?: (messageId: string) => boolean;
onToggleToolMessageExpanded?: (messageId: string) => void;
isToolExpanded?: (toolCardId: string) => boolean;
onToggleToolExpanded?: (toolCardId: string) => void;
onRequestUpdate?: () => void;
canvasHostUrl?: string | null;
basePath?: string;
localMediaPreviewRoots?: readonly string[];
assistantAttachmentAuthToken?: string | null;
embedSandboxMode?: EmbedSandboxMode;
allowExternalEmbedUrls?: boolean;
},
onOpenSidebar?: (content: SidebarContent) => void,
) {
const m = message as Record<string, unknown>;
const role = typeof m.role === "string" ? m.role : "unknown";
const normalizedRole = normalizeRoleForGrouping(role);
const normalizedRawRole = normalizeLowercaseStringOrEmpty(role);
const isToolResult =
isToolResultMessage(message) ||
normalizedRawRole === "toolresult" ||
normalizedRawRole === "tool_result" ||
role.toLowerCase() === "toolresult" ||
role.toLowerCase() === "tool_result" ||
typeof m.toolCallId === "string" ||
typeof m.tool_call_id === "string";
const toolCards = (opts.showToolCalls ?? true) ? extractToolCards(message) : [];
const toolCards = (opts.showToolCalls ?? true) ? extractToolCards(message, messageKey) : [];
const hasToolCards = toolCards.length > 0;
const images = extractImages(message);
const hasImages = images.length > 0;
const audioClips = extractAudioClips(message);
const hasAudio = audioClips.length > 0;
const extractedText = extractTextCached(message);
const normalizedMessage = normalizeMessage(message);
const extractedText = normalizedMessage.content
.reduce<string[]>((lines, item) => {
if (item.type === "text" && typeof item.text === "string") {
lines.push(item.text);
}
return lines;
}, [])
.join("\n")
.trim();
const assistantAttachments = normalizedMessage.content.filter(
(item): item is Extract<MessageContentItem, { type: "attachment" }> =>
item.type === "attachment",
);
const assistantViewBlocks = normalizedMessage.content.filter(
(item): item is Extract<MessageContentItem, { type: "canvas" }> => item.type === "canvas",
);
const extractedThinking =
opts.showReasoning && role === "assistant" ? extractThinkingCached(message) : null;
const markdownBase = extractedText?.trim() ? extractedText : null;
@@ -754,26 +1093,26 @@ function renderGroupedMessage(
// Detect pure-JSON messages and render as collapsible block
const jsonResult = markdown && !opts.isStreaming ? detectJson(markdown) : null;
const bubbleClasses = [
"chat-bubble",
opts.isStreaming ? "streaming" : "",
"fade-in",
canCopyMarkdown ? "has-copy" : "",
]
const bubbleClasses = ["chat-bubble", opts.isStreaming ? "streaming" : "", "fade-in"]
.filter(Boolean)
.join(" ");
if (!markdown && hasToolCards && isToolResult) {
return renderCollapsedToolCards(toolCards, onOpenSidebar);
}
// Suppress empty bubbles when tool cards are the only content and toggle is off
const visibleToolCards = hasToolCards && (opts.showToolCalls ?? true);
if (!markdown && !visibleToolCards && !hasImages && !hasAudio) {
if (
!markdown &&
!visibleToolCards &&
!hasImages &&
assistantAttachments.length === 0 &&
assistantViewBlocks.length === 0 &&
!normalizedMessage.replyTarget
) {
return nothing;
}
const isToolMessage = normalizedRole === "tool" || isToolResult;
const toolMessageDisclosureId = `toolmsg:${messageKey}`;
const toolMessageExpanded = opts.isToolMessageExpanded?.(toolMessageDisclosureId) ?? false;
const toolNames = [...new Set(toolCards.map((c) => c.name))];
const toolSummaryLabel =
toolNames.length <= 3
@@ -781,11 +1120,19 @@ function renderGroupedMessage(
: `${toolNames.slice(0, 2).join(", ")} +${toolNames.length - 2} more`;
const toolPreview =
markdown && !toolSummaryLabel ? markdown.trim().replace(/\s+/g, " ").slice(0, 120) : "";
const singleToolCard = toolCards.length === 1 ? toolCards[0] : null;
const toolMessageLabel =
singleToolCard && !markdown && !hasImages
? singleToolCard.outputText?.trim()
? "Tool output"
: "Tool call"
: "Tool output";
const hasActions = canCopyMarkdown || canExpand;
return html`
<div class="${bubbleClasses}">
${renderReplyPill(normalizedMessage.replyTarget)}
${hasActions
? html`<div class="chat-bubble-actions">
${canExpand ? renderExpandButton(markdown!, onOpenSidebar!) : nothing}
@@ -794,47 +1141,108 @@ function renderGroupedMessage(
: nothing}
${isToolMessage
? html`
<details class="chat-tool-msg-collapse">
<summary class="chat-tool-msg-summary">
<div
class="chat-tool-msg-collapse chat-tool-msg-collapse--manual ${toolMessageExpanded
? "is-open"
: ""}"
>
<button
class="chat-tool-msg-summary"
type="button"
aria-expanded=${String(toolMessageExpanded)}
@click=${() => opts.onToggleToolMessageExpanded?.(toolMessageDisclosureId)}
>
<span class="chat-tool-msg-summary__icon">${icons.zap}</span>
<span class="chat-tool-msg-summary__label">Tool output</span>
<span class="chat-tool-msg-summary__label">${toolMessageLabel}</span>
${toolSummaryLabel
? html`<span class="chat-tool-msg-summary__names">${toolSummaryLabel}</span>`
: toolPreview
? html`<span class="chat-tool-msg-summary__preview">${toolPreview}</span>`
: nothing}
</summary>
<div class="chat-tool-msg-body">
${renderMessageImages(images)} ${renderMessageAudio(audioClips)}
${reasoningMarkdown
? html`<div class="chat-thinking">
${unsafeHTML(toSanitizedMarkdownHtml(reasoningMarkdown))}
</div>`
: nothing}
${jsonResult
? html`<details class="chat-json-collapse">
<summary class="chat-json-summary">
<span class="chat-json-badge">JSON</span>
<span class="chat-json-label">${jsonSummaryLabel(jsonResult.parsed)}</span>
</summary>
<pre class="chat-json-content"><code>${jsonResult.pretty}</code></pre>
</details>`
: markdown
? html`<div class="chat-text" dir="${detectTextDirection(markdown)}">
${unsafeHTML(toSanitizedMarkdownHtml(markdown))}
</div>`
: nothing}
${hasToolCards ? renderCollapsedToolCards(toolCards, onOpenSidebar) : nothing}
</div>
</details>
</button>
${toolMessageExpanded
? html`
<div class="chat-tool-msg-body">
${renderMessageImages(images)}
${renderAssistantAttachments(
assistantAttachments,
opts.localMediaPreviewRoots ?? [],
opts.basePath,
opts.assistantAttachmentAuthToken,
opts.onRequestUpdate,
)}
${reasoningMarkdown
? html`<div class="chat-thinking">
${unsafeHTML(toSanitizedMarkdownHtml(reasoningMarkdown))}
</div>`
: nothing}
${jsonResult
? html`<details
class="chat-json-collapse"
?open=${Boolean(opts.autoExpandToolCalls)}
>
<summary class="chat-json-summary">
<span class="chat-json-badge">JSON</span>
<span class="chat-json-label"
>${jsonSummaryLabel(jsonResult.parsed)}</span
>
</summary>
<pre class="chat-json-content"><code>${jsonResult.pretty}</code></pre>
</details>`
: markdown
? html`<div class="chat-text" dir="${detectTextDirection(markdown)}">
${unsafeHTML(toSanitizedMarkdownHtml(markdown))}
</div>`
: nothing}
${hasToolCards
? singleToolCard && !markdown && !hasImages
? renderExpandedToolCardContent(
singleToolCard,
onOpenSidebar,
opts.canvasHostUrl,
opts.embedSandboxMode ?? "scripts",
opts.allowExternalEmbedUrls ?? false,
)
: renderInlineToolCards(toolCards, {
messageKey,
onOpenSidebar,
isToolExpanded: opts.isToolExpanded,
onToggleToolExpanded: opts.onToggleToolExpanded,
canvasHostUrl: opts.canvasHostUrl,
embedSandboxMode: opts.embedSandboxMode ?? "scripts",
allowExternalEmbedUrls: opts.allowExternalEmbedUrls ?? false,
})
: nothing}
</div>
`
: nothing}
</div>
`
: html`
${renderMessageImages(images)} ${renderMessageAudio(audioClips)}
${renderMessageImages(images)}
${renderAssistantAttachments(
assistantAttachments,
opts.localMediaPreviewRoots ?? [],
opts.basePath,
opts.assistantAttachmentAuthToken,
opts.onRequestUpdate,
)}
${reasoningMarkdown
? html`<div class="chat-thinking">
${unsafeHTML(toSanitizedMarkdownHtml(reasoningMarkdown))}
</div>`
: nothing}
${normalizedRole === "assistant" && assistantViewBlocks.length > 0
? html`${assistantViewBlocks.map(
(block) => html`${renderToolPreview(block.preview, "chat_message", {
onOpenSidebar,
rawText: block.rawText ?? null,
canvasHostUrl: opts.canvasHostUrl,
embedSandboxMode: opts.embedSandboxMode ?? "scripts",
})}
${block.rawText ? renderRawOutputToggle(block.rawText) : nothing}`,
)}`
: nothing}
${jsonResult
? html`<details class="chat-json-collapse">
<summary class="chat-json-summary">
@@ -848,7 +1256,17 @@ function renderGroupedMessage(
${unsafeHTML(toSanitizedMarkdownHtml(markdown))}
</div>`
: nothing}
${hasToolCards ? renderCollapsedToolCards(toolCards, onOpenSidebar) : nothing}
${hasToolCards
? renderInlineToolCards(toolCards, {
messageKey,
onOpenSidebar,
isToolExpanded: opts.isToolExpanded,
onToggleToolExpanded: opts.onToggleToolExpanded,
canvasHostUrl: opts.canvasHostUrl,
embedSandboxMode: opts.embedSandboxMode ?? "scripts",
allowExternalEmbedUrls: opts.allowExternalEmbedUrls ?? false,
})
: nothing}
`}
</div>
`;

248
ui/src/ui/chat/message-normalizer.test.ts
View File

@@ -33,6 +33,19 @@ describe("message-normalizer", () => {
});
});
it("does not reinterpret directive-like user string content", () => {
const result = normalizeMessage({
role: "user",
content: "MEDIA:/tmp/example.png\n[[reply_to_current]]",
});
expect(result.content).toEqual([
{ type: "text", text: "MEDIA:/tmp/example.png\n[[reply_to_current]]" },
]);
expect(result.replyTarget).toBeUndefined();
expect(result.audioAsVoice).toBeUndefined();
});
it("normalizes message with array content", () => {
const result = normalizeMessage({
role: "assistant",
@@ -59,6 +72,23 @@ describe("message-normalizer", () => {
});
});
it("does not reinterpret directive-like user text blocks inside array content", () => {
const result = normalizeMessage({
role: "user",
content: [{ type: "text", text: "MEDIA:/tmp/example.png\n[[audio_as_voice]]" }],
});
expect(result.content).toEqual([
{
type: "text",
text: "MEDIA:/tmp/example.png\n[[audio_as_voice]]",
name: undefined,
args: undefined,
},
]);
expect(result.audioAsVoice).toBeUndefined();
});
it("normalizes message with text field (alternative format)", () => {
const result = normalizeMessage({
role: "user",
@@ -68,6 +98,220 @@ describe("message-normalizer", () => {
expect(result.content).toEqual([{ type: "text", text: "Alternative format" }]);
});
it("expands [embed] shortcodes into canvas blocks", () => {
const result = normalizeMessage({
role: "assistant",
content: 'Here.\n[embed ref="cv_status" title="Status" height="320" /]',
});
expect(result.content).toEqual([
{ type: "text", text: "Here." },
{
type: "canvas",
preview: {
kind: "canvas",
surface: "assistant_message",
render: "url",
viewId: "cv_status",
url: "/__openclaw__/canvas/documents/cv_status/index.html",
title: "Status",
preferredHeight: 320,
},
rawText: null,
},
]);
});
it("ignores [embed] shortcodes inside fenced code blocks", () => {
const result = normalizeMessage({
role: "assistant",
content: '```text\n[embed ref="cv_status" /]\n```',
});
expect(result.content).toEqual([
{
type: "text",
text: '```text\n[embed ref="cv_status" /]\n```',
},
]);
});
it("leaves block-form inline html embed shortcodes as plain text", () => {
const result = normalizeMessage({
role: "assistant",
content: '[embed content_type="html" title="Status"]\n<div>Ready</div>\n[/embed]',
});
expect(result.content).toEqual([
{
type: "text",
text: '[embed content_type="html" title="Status"]\n<div>Ready</div>\n[/embed]',
},
]);
});
it("extracts MEDIA attachments and reply metadata from assistant text", () => {
const result = normalizeMessage({
role: "assistant",
content:
"[[reply_to:thread-123]]Intro\nMEDIA:https://example.com/image.png\nOutro\nMEDIA:https://example.com/voice.ogg\n[[audio_as_voice]]",
});
expect(result.replyTarget).toEqual({ kind: "id", id: "thread-123" });
expect(result.audioAsVoice).toBe(true);
expect(result.content).toEqual([
{ type: "text", text: "Intro" },
{
type: "attachment",
attachment: {
url: "https://example.com/image.png",
kind: "image",
label: "image.png",
mimeType: "image/png",
},
},
{ type: "text", text: "Outro" },
{
type: "attachment",
attachment: {
url: "https://example.com/voice.ogg",
kind: "audio",
label: "voice.ogg",
mimeType: "audio/ogg",
isVoiceNote: true,
},
},
]);
});
it("marks media-only audio attachments as voice notes when audio_as_voice is present", () => {
const result = normalizeMessage({
role: "assistant",
content: "MEDIA:https://example.com/voice.ogg\n[[audio_as_voice]]",
});
expect(result.audioAsVoice).toBe(true);
expect(result.content).toEqual([
{
type: "attachment",
attachment: {
url: "https://example.com/voice.ogg",
kind: "audio",
label: "voice.ogg",
mimeType: "audio/ogg",
isVoiceNote: true,
},
},
]);
});
it("keeps valid local MEDIA paths as assistant attachments", () => {
const result = normalizeMessage({
role: "assistant",
content: "Hello\nMEDIA:/tmp/openclaw/test-image.png\nWorld",
});
expect(result.content).toEqual([
{ type: "text", text: "Hello" },
{
type: "attachment",
attachment: {
url: "/tmp/openclaw/test-image.png",
kind: "image",
label: "test-image.png",
mimeType: "image/png",
},
},
{ type: "text", text: "World" },
]);
});
it("keeps spaced local filenames together instead of leaking suffix text", () => {
const result = normalizeMessage({
role: "assistant",
content: "MEDIA:/tmp/openclaw/shinkansen kato - Google Shopping.pdf",
});
expect(result.content).toEqual([
{
type: "attachment",
attachment: {
url: "/tmp/openclaw/shinkansen kato - Google Shopping.pdf",
kind: "document",
label: "shinkansen kato - Google Shopping.pdf",
mimeType: "application/pdf",
},
},
]);
});
it("does not fall back to raw text when an invalid MEDIA line is stripped", () => {
const result = normalizeMessage({
role: "assistant",
content: "MEDIA:~/Pictures/My File.png",
});
expect(result.content).toEqual([]);
});
it("preserves relative MEDIA references as visible text instead of dropping the assistant turn", () => {
const result = normalizeMessage({
role: "assistant",
content: "MEDIA:chart.png",
});
expect(result.content).toEqual([{ type: "text", text: "MEDIA:chart.png" }]);
});
it("strips reply_to_current without rendering a quoted preview", () => {
const result = normalizeMessage({
role: "assistant",
content: "[[reply_to_current]]\nReply body",
});
expect(result.replyTarget).toEqual({ kind: "current" });
expect(result.content).toEqual([{ type: "text", text: "Reply body" }]);
});
it("does not restore stripped reply tags when no visible text remains", () => {
const result = normalizeMessage({
role: "assistant",
content: "[[reply_to_current]]",
});
expect(result.replyTarget).toEqual({ kind: "current" });
expect(result.content).toEqual([]);
});
it("preserves structured attachment content items", () => {
const result = normalizeMessage({
role: "assistant",
content: [
{
type: "attachment",
attachment: {
url: "~/Pictures/test image.png",
kind: "image",
label: "test image.png",
mimeType: "image/png",
},
},
],
});
expect(result.content).toEqual([
{
type: "attachment",
attachment: {
url: "~/Pictures/test image.png",
kind: "image",
label: "test image.png",
mimeType: "image/png",
},
},
]);
});
it("detects tool result by toolCallId", () => {
const result = normalizeMessage({
role: "assistant",
@@ -124,7 +368,7 @@ describe("message-normalizer", () => {
content: [{ type: "tool_use", name: "test", arguments: { foo: "bar" } }],
});
expect(result.content[0].args).toEqual({ foo: "bar" });
expect((result.content[0] as { args?: unknown }).args).toEqual({ foo: "bar" });
});
it("handles input field for anthropic tool_use blocks", () => {
@@ -133,7 +377,7 @@ describe("message-normalizer", () => {
content: [{ type: "tool_use", name: "Bash", input: { command: "pwd" } }],
});
expect(result.content[0].args).toEqual({ command: "pwd" });
expect((result.content[0] as { args?: unknown }).args).toEqual({ command: "pwd" });
});
it("preserves top-level sender labels", () => {

351
ui/src/ui/chat/message-normalizer.ts
View File

@@ -3,14 +3,242 @@
*/
import { stripInboundMetadata } from "../../../../src/auto-reply/reply/strip-inbound-meta.js";
import { extractCanvasShortcodes } from "../../../../src/chat/canvas-render.js";
import {
isToolCallContentType,
isToolResultContentType,
resolveToolBlockArgs,
} from "../../../../src/chat/tool-content.js";
import { normalizeLowercaseStringOrEmpty } from "../string-coerce.ts";
import { mediaKindFromMime } from "../../../../src/media/constants.js";
import { splitMediaFromOutput } from "../../../../src/media/parse.js";
import { parseInlineDirectives } from "../../../../src/utils/directive-tags.js";
import type { NormalizedMessage, MessageContentItem } from "../types/chat-types.ts";
function coerceCanvasPreview(
value: unknown,
):
| Extract<NonNullable<NormalizedMessage["content"][number]>, { type: "canvas" }>["preview"]
| null {
if (!value || typeof value !== "object" || Array.isArray(value)) {
return null;
}
const preview = value as Record<string, unknown>;
if (preview.kind !== "canvas" || preview.surface === "tool_card") {
return null;
}
const render = preview.render === "url" ? "url" : null;
if (!render) {
return null;
}
return {
kind: "canvas",
surface: "assistant_message",
render,
...(typeof preview.title === "string" ? { title: preview.title } : {}),
...(typeof preview.preferredHeight === "number"
? { preferredHeight: preview.preferredHeight }
: {}),
...(typeof preview.url === "string" ? { url: preview.url } : {}),
...(typeof preview.viewId === "string" ? { viewId: preview.viewId } : {}),
...(typeof preview.className === "string" ? { className: preview.className } : {}),
...(typeof preview.style === "string" ? { style: preview.style } : {}),
};
}
function isRenderableAssistantAttachment(url: string): boolean {
const trimmed = url.trim();
return (
/^https?:\/\//i.test(trimmed) ||
/^data:(?:image|audio|video)\//i.test(trimmed) ||
/^\/(?:__openclaw__|media)\//.test(trimmed) ||
trimmed.startsWith("file://") ||
trimmed.startsWith("~") ||
trimmed.startsWith("/") ||
/^[a-zA-Z]:[\\/]/.test(trimmed)
);
}
function shouldPreserveRelativeAssistantAttachment(url: string): boolean {
const trimmed = url.trim();
if (!trimmed) {
return false;
}
return (
!/^https?:\/\//i.test(trimmed) &&
!/^data:(?:image|audio|video)\//i.test(trimmed) &&
!/^\/(?:__openclaw__|media)\//.test(trimmed) &&
!trimmed.startsWith("file://") &&
!trimmed.startsWith("~") &&
!trimmed.startsWith("/") &&
!/^[a-zA-Z]:[\\/]/.test(trimmed)
);
}
const MIME_BY_EXT: Record<string, string> = {
png: "image/png",
jpg: "image/jpeg",
jpeg: "image/jpeg",
webp: "image/webp",
gif: "image/gif",
heic: "image/heic",
heif: "image/heif",
ogg: "audio/ogg",
oga: "audio/ogg",
mp3: "audio/mpeg",
wav: "audio/wav",
flac: "audio/flac",
aac: "audio/aac",
opus: "audio/opus",
m4a: "audio/mp4",
mp4: "video/mp4",
mov: "video/quicktime",
pdf: "application/pdf",
txt: "text/plain",
md: "text/markdown",
csv: "text/csv",
json: "application/json",
zip: "application/zip",
};
function getFileExtension(url: string): string | undefined {
const trimmed = url.trim();
if (!trimmed) {
return undefined;
}
const source = (() => {
try {
if (/^https?:\/\//i.test(trimmed)) {
return new URL(trimmed).pathname;
}
} catch {}
return trimmed;
})();
const fileName = source.split(/[\\/]/).pop() ?? source;
const match = /\.([a-zA-Z0-9]+)$/.exec(fileName);
return match?.[1]?.toLowerCase();
}
function mimeTypeFromUrl(url: string): string | undefined {
const ext = getFileExtension(url);
return ext ? MIME_BY_EXT[ext] : undefined;
}
function inferAttachmentKind(url: string): {
kind: "image" | "audio" | "video" | "document";
mimeType?: string;
label: string;
} {
const mimeType = mimeTypeFromUrl(url);
const kind = mediaKindFromMime(mimeType) ?? "document";
const label = (() => {
try {
if (/^https?:\/\//i.test(url)) {
const parsed = new URL(url);
const name = parsed.pathname.split("/").pop()?.trim();
return name || parsed.hostname || url;
}
} catch {}
const name = url.split(/[\\/]/).pop()?.trim();
return name || url;
})();
return { kind, mimeType, label };
}
function mergeAdjacentTextItems(items: MessageContentItem[]): MessageContentItem[] {
const merged: MessageContentItem[] = [];
for (const item of items) {
const previous = merged[merged.length - 1];
if (item.type === "text" && previous?.type === "text") {
previous.text = [previous.text, item.text].filter((value) => value !== undefined).join("\n");
continue;
}
merged.push(item);
}
return merged.filter((item) => item.type !== "text" || Boolean(item.text?.trim()));
}
function expandTextContent(text: string): {
content: MessageContentItem[];
audioAsVoice: boolean;
replyTarget: NormalizedMessage["replyTarget"];
} {
const extracted = extractCanvasShortcodes(text);
const parsed = splitMediaFromOutput(extracted.text);
const parts: MessageContentItem[] = [];
let audioAsVoice = parsed.audioAsVoice === true;
let replyTarget: NormalizedMessage["replyTarget"] = null;
const segments = parsed.segments ?? [{ type: "text" as const, text: parsed.text }];
for (const segment of segments) {
if (segment.type === "media") {
if (!isRenderableAssistantAttachment(segment.url)) {
if (shouldPreserveRelativeAssistantAttachment(segment.url)) {
parts.push({ type: "text", text: `MEDIA:${segment.url}` });
}
continue;
}
const inferred = inferAttachmentKind(segment.url);
parts.push({
type: "attachment",
attachment: {
url: segment.url,
kind: inferred.kind,
label: inferred.label,
mimeType: inferred.mimeType,
},
});
continue;
}
const directives = parseInlineDirectives(segment.text, {
stripAudioTag: true,
stripReplyTags: true,
});
audioAsVoice = audioAsVoice || directives.audioAsVoice;
if (directives.replyToExplicitId) {
replyTarget = { kind: "id", id: directives.replyToExplicitId };
} else if (directives.replyToCurrent && replyTarget === null) {
replyTarget = { kind: "current" };
}
if (directives.text) {
parts.push({ type: "text", text: directives.text });
}
}
for (const preview of extracted.previews) {
parts.push({ type: "canvas", preview, rawText: null });
}
const content = mergeAdjacentTextItems(
parts.map((item) => {
if (item.type === "attachment" && item.attachment.kind === "audio" && audioAsVoice) {
return {
...item,
attachment: {
...item.attachment,
isVoiceNote: true,
},
};
}
return item;
}),
);
return {
content:
content.length > 0
? content
: (parsed.mediaUrls ?? []).some((url) => shouldPreserveRelativeAssistantAttachment(url))
? (parsed.mediaUrls ?? [])
.filter((url) => shouldPreserveRelativeAssistantAttachment(url))
.map((url) => ({ type: "text" as const, text: `MEDIA:${url}` }))
: replyTarget === null && !audioAsVoice && parsed.text.trim().length > 0
? [{ type: "text", text: parsed.text }]
: [],
audioAsVoice,
replyTarget,
};
}
/**
* Normalize a raw message object into a consistent structure.
*/
@@ -36,21 +264,112 @@ export function normalizeMessage(message: unknown): NormalizedMessage {
if (hasToolId || hasToolContent || hasToolName) {
role = "toolResult";
}
const isAssistantMessage = role === "assistant";
// Extract content
let content: MessageContentItem[] = [];
let audioAsVoice = false;
let replyTarget: NormalizedMessage["replyTarget"] = null;
if (typeof m.content === "string") {
content = [{ type: "text", text: m.content }];
if (isAssistantMessage) {
const expanded = expandTextContent(m.content);
content = expanded.content;
audioAsVoice = expanded.audioAsVoice;
replyTarget = expanded.replyTarget;
} else {
content = [{ type: "text", text: m.content }];
}
} else if (Array.isArray(m.content)) {
content = m.content.map((item: Record<string, unknown>) => ({
type: (item.type as MessageContentItem["type"]) || "text",
text: item.text as string | undefined,
name: item.name as string | undefined,
args: resolveToolBlockArgs(item),
}));
content = m.content.flatMap((item: Record<string, unknown>) => {
if (
item.type === "attachment" &&
item.attachment &&
typeof item.attachment === "object" &&
!Array.isArray(item.attachment)
) {
const attachment = item.attachment as {
url?: unknown;
kind?: unknown;
label?: unknown;
mimeType?: unknown;
isVoiceNote?: unknown;
};
if (
typeof attachment.url !== "string" ||
(attachment.kind !== "image" &&
attachment.kind !== "audio" &&
attachment.kind !== "video" &&
attachment.kind !== "document") ||
typeof attachment.label !== "string"
) {
return [];
}
return [
{
type: "attachment" as const,
attachment: {
url: attachment.url,
kind: attachment.kind,
label: attachment.label,
...(typeof attachment.mimeType === "string"
? { mimeType: attachment.mimeType }
: {}),
...(attachment.isVoiceNote === true ? { isVoiceNote: true } : {}),
},
},
];
}
if (
item.type === "canvas" &&
item.preview &&
typeof item.preview === "object" &&
!Array.isArray(item.preview)
) {
const preview = coerceCanvasPreview(item.preview);
if (!preview) {
return [];
}
return [
{
type: "canvas" as const,
preview,
rawText: typeof item.rawText === "string" ? item.rawText : null,
},
];
}
if (item.type === "text" && typeof item.text === "string" && isAssistantMessage) {
const expanded = expandTextContent(item.text);
audioAsVoice = audioAsVoice || expanded.audioAsVoice;
if (expanded.replyTarget?.kind === "id") {
replyTarget = expanded.replyTarget;
} else if (expanded.replyTarget?.kind === "current" && replyTarget === null) {
replyTarget = expanded.replyTarget;
}
return expanded.content;
}
return [
{
type:
(item.type as Extract<
MessageContentItem,
{ type: "text" | "tool_call" | "tool_result" }
>["type"]) || "text",
text: item.text as string | undefined,
name: item.name as string | undefined,
args: resolveToolBlockArgs(item),
},
];
});
} else if (typeof m.text === "string") {
content = [{ type: "text", text: m.text }];
if (isAssistantMessage) {
const expanded = expandTextContent(m.text);
content = expanded.content;
audioAsVoice = expanded.audioAsVoice;
replyTarget = expanded.replyTarget;
} else {
content = [{ type: "text", text: m.text }];
}
}
const timestamp = typeof m.timestamp === "number" ? m.timestamp : Date.now();
@@ -68,14 +387,22 @@ export function normalizeMessage(message: unknown): NormalizedMessage {
});
}
return { role, content, timestamp, id, senderLabel };
return {
role,
content,
timestamp,
id,
senderLabel,
...(audioAsVoice ? { audioAsVoice: true } : {}),
...(replyTarget ? { replyTarget } : {}),
};
}
/**
* Normalize role for grouping purposes.
*/
export function normalizeRoleForGrouping(role: string): string {
const lower = normalizeLowercaseStringOrEmpty(role);
const lower = role.toLowerCase();
// Preserve original casing when it's already a core role.
if (role === "user" || role === "User") {
return role;
@@ -103,6 +430,6 @@ export function normalizeRoleForGrouping(role: string): string {
*/
export function isToolResultMessage(message: unknown): boolean {
const m = message as Record<string, unknown>;
const role = normalizeLowercaseStringOrEmpty(m.role);
const role = typeof m.role === "string" ? m.role.toLowerCase() : "";
return role === "toolresult" || role === "tool_result";
}

458
ui/src/ui/chat/tool-cards.test.ts
View File

@@ -1,34 +1,450 @@
/* @vitest-environment jsdom */
import { render } from "lit";
import { describe, expect, it } from "vitest";
import { extractToolCards, renderToolCardSidebar } from "./tool-cards.ts";
import { describe, expect, it, vi } from "vitest";
import {
buildToolCardSidebarContent,
extractToolCards,
renderToolCard,
renderToolPreview,
} from "./tool-cards.ts";
describe("tool cards", () => {
it("renders anthropic tool_use input details in tool cards", () => {
const cards = extractToolCards({
role: "assistant",
content: [
{
type: "tool_use",
id: "toolu_123",
name: "Bash",
input: { command: 'time claude -p "say ok"' },
},
],
});
describe("tool-cards", () => {
it("pretty-prints structured args and pairs tool output onto the same card", () => {
const cards = extractToolCards(
{
role: "assistant",
toolCallId: "call-1",
content: [
{
type: "toolcall",
id: "call-1",
name: "browser.open",
arguments: { url: "https://example.com", retry: 0 },
},
{
type: "toolresult",
id: "call-1",
name: "browser.open",
text: "Opened page",
},
],
},
"msg:1",
);
expect(cards).toHaveLength(1);
expect(cards[0]).toMatchObject({
kind: "call",
name: "Bash",
args: { command: 'time claude -p "say ok"' },
id: "msg:1:call-1",
name: "browser.open",
outputText: "Opened page",
});
expect(cards[0]?.inputText).toContain('"url": "https://example.com"');
expect(cards[0]?.inputText).toContain('"retry": 0');
});
it("preserves string args verbatim and keeps empty-output cards", () => {
const cards = extractToolCards(
{
role: "assistant",
toolCallId: "call-2",
content: [
{
type: "toolcall",
name: "deck_manage",
arguments: "with Example Deck",
},
],
},
"msg:2",
);
expect(cards).toHaveLength(1);
expect(cards[0]?.inputText).toBe("with Example Deck");
expect(cards[0]?.outputText).toBeUndefined();
});
it("preserves tool-call input payloads from tool_use blocks", () => {
const cards = extractToolCards(
{
role: "assistant",
content: [
{
type: "tool_use",
id: "call-2b",
name: "deck_manage",
input: { deck: "Example Deck", mode: "preview" },
},
],
},
"msg:2b",
);
expect(cards).toHaveLength(1);
expect(cards[0]?.inputText).toContain('"deck": "Example Deck"');
expect(cards[0]?.inputText).toContain('"mode": "preview"');
});
it("builds sidebar content with input and empty output status", () => {
const [card] = extractToolCards(
{
role: "assistant",
toolCallId: "call-3",
content: [
{
type: "toolcall",
name: "deck_manage",
arguments: "with Example Deck",
},
],
},
"msg:3",
);
const sidebar = buildToolCardSidebarContent(card);
expect(sidebar).toContain("## Deck Manage");
expect(sidebar).toContain("### Tool input");
expect(sidebar).toContain("with Example Deck");
expect(sidebar).toContain("### Tool output");
expect(sidebar).toContain("No output");
});
it("extracts canvas handle payloads into canvas previews", () => {
const [card] = extractToolCards(
{
role: "tool",
toolName: "canvas_render",
content: JSON.stringify({
kind: "canvas",
view: {
backend: "canvas",
id: "cv_inline",
url: "/__openclaw__/canvas/documents/cv_inline/index.html",
},
presentation: {
target: "assistant_message",
title: "Inline demo",
preferred_height: 420,
},
}),
},
"msg:view:1",
);
expect(card?.preview).toMatchObject({
kind: "canvas",
surface: "assistant_message",
render: "url",
viewId: "cv_inline",
url: "/__openclaw__/canvas/documents/cv_inline/index.html",
title: "Inline demo",
preferredHeight: 420,
});
});
it("drops tool_card-targeted canvas payloads", () => {
const [card] = extractToolCards(
{
role: "tool",
toolName: "canvas_render",
content: JSON.stringify({
kind: "canvas",
view: {
backend: "canvas",
id: "cv_tool_card",
url: "/__openclaw__/canvas/documents/cv_tool_card/index.html",
},
presentation: {
target: "tool_card",
title: "Tool card demo",
},
}),
},
"msg:view:2",
);
expect(card?.preview).toBeUndefined();
});
it("renders trusted canvas previews with same-origin only when explicitly requested", () => {
const container = document.createElement("div");
render(renderToolCardSidebar(cards[0]), container);
render(
renderToolPreview(
{
kind: "canvas",
surface: "assistant_message",
render: "url",
viewId: "cv_inline",
url: "/__openclaw__/canvas/documents/cv_inline/index.html",
title: "Inline demo",
preferredHeight: 420,
},
"chat_message",
{ embedSandboxMode: "trusted" },
),
container,
);
expect(container.textContent).toContain('time claude -p "say ok"');
expect(container.textContent).toContain("Bash");
const iframe = container.querySelector<HTMLIFrameElement>(".chat-tool-card__preview-frame");
expect(iframe?.getAttribute("sandbox")).toBe("allow-scripts allow-same-origin");
});
it("does not extract inline-html canvas payloads into canvas previews", () => {
const [card] = extractToolCards(
{
role: "tool",
toolName: "canvas_render",
content: JSON.stringify({
kind: "canvas",
source: {
type: "html",
content: "<div>hello</div>",
},
presentation: {
target: "assistant_message",
title: "Status",
preferred_height: 300,
},
}),
},
"msg:view:3",
);
expect(card?.preview).toBeUndefined();
});
it("does not create a view preview for malformed json output", () => {
const [card] = extractToolCards(
{
role: "tool",
toolName: "canvas_render",
content: '{"kind":"present_view","view":{"id":"broken"}',
},
"msg:view:4",
);
expect(card?.preview).toBeUndefined();
});
it("does not create a view preview for generic tool text output", () => {
const [card] = extractToolCards(
{
role: "tool",
toolName: "browser.open",
content: "present_view: cv_widget",
},
"msg:view:5",
);
expect(card?.preview).toBeUndefined();
});
it("renders expanded cards with inline input and output sections", () => {
const container = document.createElement("div");
const toggle = vi.fn();
render(
renderToolCard(
{
id: "msg:4:call-4",
name: "browser.open",
args: { url: "https://example.com" },
inputText: '{\n "url": "https://example.com"\n}',
outputText: "Opened page",
},
{ expanded: true, onToggleExpanded: toggle },
),
container,
);
expect(container.textContent).toContain("Tool input");
expect(container.textContent).toContain("Tool output");
expect(container.textContent).toContain("https://example.com");
expect(container.textContent).toContain("Opened page");
});
it("renders expanded tool calls without an inline output block when no output is present", () => {
const container = document.createElement("div");
render(
renderToolCard(
{
id: "msg:4b:call-4b",
name: "sessions_spawn",
args: { mode: "session", thread: true },
inputText: '{\n "mode": "session",\n "thread": true\n}',
},
{ expanded: true, onToggleExpanded: vi.fn() },
),
container,
);
expect(container.textContent).toContain("Tool input");
expect(container.textContent).toContain('"thread": true');
expect(container.textContent).not.toContain("Tool output");
expect(container.textContent).not.toContain("No output");
});
it("labels collapsed tool calls as tool call", () => {
const container = document.createElement("div");
render(
renderToolCard(
{
id: "msg:5:call-5",
name: "sessions_spawn",
args: { mode: "run" },
inputText: '{\n "mode": "run"\n}',
},
{ expanded: false, onToggleExpanded: vi.fn() },
),
container,
);
expect(container.textContent).toContain("Tool call");
expect(container.textContent).not.toContain("Tool input");
const summaryButton = container.querySelector("button.chat-tool-msg-summary");
expect(summaryButton).not.toBeNull();
expect(summaryButton?.getAttribute("aria-expanded")).toBe("false");
});
it("does not render inline preview frames inside tool rows anymore", () => {
const container = document.createElement("div");
render(
renderToolCard(
{
id: "msg:view:6",
name: "canvas_render",
outputText: JSON.stringify({
kind: "canvas",
source: {
type: "html",
content: '<div onclick="alert(1)">front<script>window.bad = true;</script></div>',
},
presentation: {
target: "tool_card",
title: "Status view",
},
}),
preview: {
kind: "canvas",
surface: "assistant_message",
render: "url",
url: "/__openclaw__/canvas/documents/cv_status/index.html",
title: "Status view",
},
},
{ expanded: true, onToggleExpanded: vi.fn() },
),
container,
);
const rawToggle = container.querySelector<HTMLButtonElement>(".chat-tool-card__raw-toggle");
const rawBody = container.querySelector<HTMLElement>(".chat-tool-card__raw-body");
expect(container.querySelector(".chat-tool-card__preview-frame")).toBeNull();
expect(rawToggle?.getAttribute("aria-expanded")).toBe("false");
expect(rawBody?.hidden).toBe(true);
rawToggle?.dispatchEvent(new MouseEvent("click", { bubbles: true }));
expect(rawToggle?.getAttribute("aria-expanded")).toBe("true");
expect(rawBody?.hidden).toBe(false);
});
it("keeps raw details for legacy canvas tool output without rendering tool-row previews", () => {
const container = document.createElement("div");
render(
renderToolCard(
{
id: "msg:view:7",
name: "canvas_render",
outputText: JSON.stringify({
kind: "canvas",
view: {
backend: "canvas",
id: "cv_counter",
url: "/__openclaw__/canvas/documents/cv_counter/index.html",
title: "Counter demo",
preferred_height: 480,
},
presentation: {
target: "tool_card",
},
}),
preview: {
kind: "canvas",
surface: "assistant_message",
render: "url",
viewId: "cv_counter",
title: "Counter demo",
url: "/__openclaw__/canvas/documents/cv_counter/index.html",
preferredHeight: 480,
},
},
{ expanded: true, onToggleExpanded: vi.fn() },
),
container,
);
const rawToggle = container.querySelector<HTMLButtonElement>(".chat-tool-card__raw-toggle");
const rawBody = container.querySelector<HTMLElement>(".chat-tool-card__raw-body");
expect(container.textContent).toContain("Counter demo");
expect(container.querySelector(".chat-tool-card__preview-frame")).toBeNull();
expect(rawToggle?.getAttribute("aria-expanded")).toBe("false");
expect(rawBody?.hidden).toBe(true);
rawToggle?.dispatchEvent(new MouseEvent("click", { bubbles: true }));
expect(rawToggle?.getAttribute("aria-expanded")).toBe("true");
expect(rawBody?.hidden).toBe(false);
expect(rawBody?.textContent).toContain('"kind":"canvas"');
});
it("opens assistant-surface canvas payloads in the sidebar when explicitly requested", () => {
const container = document.createElement("div");
const onOpenSidebar = vi.fn();
render(
renderToolCard(
{
id: "msg:view:8",
name: "canvas_render",
outputText: JSON.stringify({
kind: "canvas",
view: {
backend: "canvas",
id: "cv_sidebar",
url: "/__openclaw__/canvas/documents/cv_sidebar/index.html",
title: "Player",
preferred_height: 360,
},
presentation: {
target: "assistant_message",
},
}),
preview: {
kind: "canvas",
surface: "assistant_message",
render: "url",
viewId: "cv_sidebar",
url: "/__openclaw__/canvas/documents/cv_sidebar/index.html",
title: "Player",
preferredHeight: 360,
},
},
{ expanded: true, onToggleExpanded: vi.fn(), onOpenSidebar },
),
container,
);
const sidebarButton = container.querySelector<HTMLButtonElement>(".chat-tool-card__action-btn");
sidebarButton?.dispatchEvent(new MouseEvent("click", { bubbles: true }));
expect(sidebarButton).not.toBeNull();
expect(onOpenSidebar).toHaveBeenCalledWith(
expect.objectContaining({
kind: "canvas",
docId: "cv_sidebar",
entryUrl: "/__openclaw__/canvas/documents/cv_sidebar/index.html",
}),
);
});
});

649
ui/src/ui/chat/tool-cards.ts
View File

@@ -1,118 +1,19 @@
import { html, nothing } from "lit";
import {
isToolCallContentType,
isToolResultContentType,
resolveToolBlockArgs,
} from "../../../../src/chat/tool-content.js";
import { extractCanvasFromText } from "../../../../src/chat/canvas-render.js";
import { resolveCanvasIframeUrl } from "../canvas-url.ts";
import { resolveEmbedSandbox, type EmbedSandboxMode } from "../embed-sandbox.ts";
import { icons } from "../icons.ts";
import type { SidebarContent } from "../sidebar-content.ts";
import { formatToolDetail, resolveToolDisplay } from "../tool-display.ts";
import type { ToolCard } from "../types/chat-types.ts";
import { TOOL_INLINE_THRESHOLD } from "./constants.ts";
import { extractTextCached } from "./message-extract.ts";
import { isToolResultMessage } from "./message-normalizer.ts";
import { formatToolOutputForSidebar, getTruncatedPreview } from "./tool-helpers.ts";
export function extractToolCards(message: unknown): ToolCard[] {
const m = message as Record<string, unknown>;
const content = normalizeContent(m.content);
const cards: ToolCard[] = [];
export type ToolPreview = NonNullable<ToolCard["preview"]>;
for (const item of content) {
const isToolCall =
isToolCallContentType(item.type) ||
(typeof item.name === "string" && resolveToolBlockArgs(item) != null);
if (isToolCall) {
cards.push({
kind: "call",
name: (item.name as string) ?? "tool",
args: coerceArgs(resolveToolBlockArgs(item)),
});
}
}
for (const item of content) {
if (!isToolResultContentType(item.type)) {
continue;
}
const text = extractToolText(item);
const name = typeof item.name === "string" ? item.name : "tool";
cards.push({ kind: "result", name, text });
}
if (isToolResultMessage(message) && !cards.some((card) => card.kind === "result")) {
const name =
(typeof m.toolName === "string" && m.toolName) ||
(typeof m.tool_name === "string" && m.tool_name) ||
"tool";
const text = extractTextCached(message) ?? undefined;
cards.push({ kind: "result", name, text });
}
return cards;
}
export function renderToolCardSidebar(card: ToolCard, onOpenSidebar?: (content: string) => void) {
const display = resolveToolDisplay({ name: card.name, args: card.args });
const detail = formatToolDetail(display);
const hasText = Boolean(card.text?.trim());
const canClick = Boolean(onOpenSidebar);
const handleClick = canClick
? () => {
if (hasText) {
onOpenSidebar!(formatToolOutputForSidebar(card.text!));
return;
}
const info = `## ${display.label}\n\n${
detail ? `**Command:** \`${detail}\`\n\n` : ""
}*No output — tool completed successfully.*`;
onOpenSidebar!(info);
}
: undefined;
const isShort = hasText && (card.text?.length ?? 0) <= TOOL_INLINE_THRESHOLD;
const showCollapsed = hasText && !isShort;
const showInline = hasText && isShort;
const isEmpty = !hasText;
return html`
<div
class="chat-tool-card ${canClick ? "chat-tool-card--clickable" : ""}"
@click=${handleClick}
role=${canClick ? "button" : nothing}
tabindex=${canClick ? "0" : nothing}
@keydown=${canClick
? (e: KeyboardEvent) => {
if (e.key !== "Enter" && e.key !== " ") {
return;
}
e.preventDefault();
handleClick?.();
}
: nothing}
>
<div class="chat-tool-card__header">
<div class="chat-tool-card__title">
<span class="chat-tool-card__icon">${icons[display.icon]}</span>
<span>${display.label}</span>
</div>
${canClick
? html`<span class="chat-tool-card__action"
>${hasText ? "View" : ""} ${icons.check}</span
>`
: nothing}
${isEmpty && !canClick
? html`<span class="chat-tool-card__status">${icons.check}</span>`
: nothing}
</div>
${detail ? html`<div class="chat-tool-card__detail">${detail}</div>` : nothing}
${isEmpty ? html` <div class="chat-tool-card__status-text muted">Completed</div> ` : nothing}
${showCollapsed
? html`<div class="chat-tool-card__preview mono">${getTruncatedPreview(card.text!)}</div>`
: nothing}
${showInline ? html`<div class="chat-tool-card__inline mono">${card.text}</div>` : nothing}
</div>
`;
function resolveCanvasPreviewSandbox(preview: ToolPreview): string {
return resolveEmbedSandbox(preview.kind === "canvas" ? "scripts" : "scripts");
}
function normalizeContent(content: unknown): Array<Record<string, unknown>> {
@@ -149,3 +50,539 @@ function extractToolText(item: Record<string, unknown>): string | undefined {
}
return undefined;
}
export function extractToolPreview(
outputText: string | undefined,
toolName: string | undefined,
): ToolCard["preview"] | undefined {
return extractCanvasFromText(outputText, toolName);
}
function resolveToolCardId(
item: Record<string, unknown>,
message: Record<string, unknown>,
index: number,
prefix = "tool",
): string {
const explicitId =
(typeof item.id === "string" && item.id.trim()) ||
(typeof item.toolCallId === "string" && item.toolCallId.trim()) ||
(typeof item.tool_call_id === "string" && item.tool_call_id.trim()) ||
(typeof item.callId === "string" && item.callId.trim()) ||
(typeof message.toolCallId === "string" && message.toolCallId.trim()) ||
(typeof message.tool_call_id === "string" && message.tool_call_id.trim()) ||
"";
if (explicitId) {
return `${prefix}:${explicitId}`;
}
const name =
(typeof item.name === "string" && item.name.trim()) ||
(typeof message.toolName === "string" && message.toolName.trim()) ||
(typeof message.tool_name === "string" && message.tool_name.trim()) ||
"tool";
return `${prefix}:${name}:${index}`;
}
function serializeToolInput(args: unknown): string | undefined {
if (args === undefined || args === null) {
return undefined;
}
if (typeof args === "string") {
return args;
}
try {
return JSON.stringify(args, null, 2);
} catch {
if (typeof args === "number" || typeof args === "boolean" || typeof args === "bigint") {
return String(args);
}
if (typeof args === "symbol") {
return args.description ? `Symbol(${args.description})` : "Symbol()";
}
return Object.prototype.toString.call(args);
}
}
function formatPayloadForSidebar(
text: string | undefined,
language: "json" | "text" = "text",
): string {
if (!text?.trim()) {
return "";
}
if (language === "json") {
return `\`\`\`json
${text}
\`\`\``;
}
const formatted = formatToolOutputForSidebar(text);
if (formatted.includes("```")) {
return formatted;
}
return `\`\`\`text
${text}
\`\`\``;
}
function findLatestCard(cards: ToolCard[], id: string, name: string): ToolCard | undefined {
for (let i = cards.length - 1; i >= 0; i--) {
const card = cards[i];
if (!card) {
continue;
}
if (card.id === id || (card.name === name && !card.outputText)) {
return card;
}
}
return undefined;
}
export function extractToolCards(message: unknown, prefix = "tool"): ToolCard[] {
const m = message as Record<string, unknown>;
const content = normalizeContent(m.content);
const cards: ToolCard[] = [];
for (let index = 0; index < content.length; index++) {
const item = content[index] ?? {};
const kind = (typeof item.type === "string" ? item.type : "").toLowerCase();
const isToolCall =
["toolcall", "tool_call", "tooluse", "tool_use"].includes(kind) ||
(typeof item.name === "string" &&
(item.arguments != null || item.args != null || item.input != null));
if (!isToolCall) {
continue;
}
const args = coerceArgs(item.arguments ?? item.args ?? item.input);
cards.push({
id: resolveToolCardId(item, m, index, prefix),
name: (item.name as string) ?? "tool",
args,
inputText: serializeToolInput(args),
});
}
for (let index = 0; index < content.length; index++) {
const item = content[index] ?? {};
const kind = (typeof item.type === "string" ? item.type : "").toLowerCase();
if (kind !== "toolresult" && kind !== "tool_result") {
continue;
}
const name = typeof item.name === "string" ? item.name : "tool";
const cardId = resolveToolCardId(item, m, index, prefix);
const existing = findLatestCard(cards, cardId, name);
const text = extractToolText(item);
const preview = extractToolPreview(text, name);
if (existing) {
existing.outputText = text;
existing.preview = preview;
continue;
}
cards.push({
id: cardId,
name,
outputText: text,
preview,
});
}
const role = typeof m.role === "string" ? m.role.toLowerCase() : "";
const isStandaloneToolMessage =
isToolResultMessage(message) ||
role === "tool" ||
role === "function" ||
typeof m.toolName === "string" ||
typeof m.tool_name === "string";
if (isStandaloneToolMessage && cards.length === 0) {
const name =
(typeof m.toolName === "string" && m.toolName) ||
(typeof m.tool_name === "string" && m.tool_name) ||
"tool";
const text = extractTextCached(message) ?? undefined;
cards.push({
id: resolveToolCardId({}, m, 0, prefix),
name,
outputText: text,
preview: extractToolPreview(text, name),
});
}
return cards;
}
export function buildToolCardSidebarContent(card: ToolCard): string {
const display = resolveToolDisplay({ name: card.name, args: card.args });
const detail = formatToolDetail(display);
const sections = [`## ${display.label}`, `**Tool:** \`${display.name}\``];
if (detail) {
sections.push(`**Summary:** ${detail}`);
}
if (card.inputText?.trim()) {
const inputIsJson = typeof card.args === "object" && card.args !== null;
sections.push(
`### Tool input\n${formatPayloadForSidebar(card.inputText, inputIsJson ? "json" : "text")}`,
);
}
if (card.outputText?.trim()) {
sections.push(`### Tool output\n${formatToolOutputForSidebar(card.outputText)}`);
} else {
sections.push(`### Tool output\n*No output — tool completed successfully.*`);
}
return sections.join("\n\n");
}
function handleRawDetailsToggle(event: Event) {
const button = event.currentTarget as HTMLButtonElement | null;
const root = button?.closest(".chat-tool-card__raw");
const body = root?.querySelector<HTMLElement>(".chat-tool-card__raw-body");
if (!button || !body) {
return;
}
const expanded = button.getAttribute("aria-expanded") === "true";
button.setAttribute("aria-expanded", String(!expanded));
body.hidden = expanded;
}
function renderPreviewFrame(params: {
title: string;
src?: string;
height?: number;
sandbox?: string;
}) {
return html`
<iframe
class="chat-tool-card__preview-frame"
title=${params.title}
sandbox=${params.sandbox ?? ""}
src=${params.src ?? nothing}
style=${params.height ? `height:${params.height}px` : ""}
></iframe>
`;
}
export function renderToolPreview(
preview: ToolPreview | undefined,
surface: "chat_tool" | "chat_message" | "sidebar",
options?: {
onOpenSidebar?: (content: SidebarContent) => void;
rawText?: string | null;
canvasHostUrl?: string | null;
embedSandboxMode?: EmbedSandboxMode;
allowExternalEmbedUrls?: boolean;
},
) {
if (!preview) {
return nothing;
}
if (preview.kind !== "canvas" || surface === "chat_tool") {
return nothing;
}
if (preview.surface !== "assistant_message") {
return nothing;
}
return html`
<div class="chat-tool-card__preview" data-kind="canvas" data-surface=${surface}>
<div class="chat-tool-card__preview-header">
<span class="chat-tool-card__preview-label">${preview.title?.trim() || "Canvas"}</span>
</div>
<div class="chat-tool-card__preview-panel" data-side="canvas">
${renderPreviewFrame({
title: preview.title?.trim() || "Canvas",
src: resolveCanvasIframeUrl(
preview.url,
options?.canvasHostUrl,
options?.allowExternalEmbedUrls ?? false,
),
height: preview.preferredHeight,
sandbox:
preview.kind === "canvas"
? resolveEmbedSandbox(options?.embedSandboxMode ?? "scripts")
: resolveCanvasPreviewSandbox(preview),
})}
</div>
</div>
`;
}
export function buildSidebarContent(value: string): SidebarContent {
return {
kind: "markdown",
content: value,
};
}
export function buildPreviewSidebarContent(
preview: ToolPreview,
rawText?: string | null,
): SidebarContent | null {
if (preview.kind !== "canvas" || preview.render !== "url" || !preview.viewId || !preview.url) {
return null;
}
return {
kind: "canvas",
docId: preview.viewId,
entryUrl: preview.url,
...(preview.title ? { title: preview.title } : {}),
...(preview.preferredHeight ? { preferredHeight: preview.preferredHeight } : {}),
...(rawText ? { rawText } : {}),
};
}
export function renderRawOutputToggle(text: string) {
return html`
<div class="chat-tool-card__raw">
<button
class="chat-tool-card__raw-toggle"
type="button"
aria-expanded="false"
@click=${handleRawDetailsToggle}
>
<span>Raw details</span>
<span class="chat-tool-card__raw-toggle-icon">${icons.chevronDown}</span>
</button>
<div class="chat-tool-card__raw-body" hidden>
${renderToolDataBlock({
label: "Tool output",
text,
expanded: true,
})}
</div>
</div>
`;
}
function renderToolDataBlock(params: {
label: string;
text: string;
expanded: boolean;
empty?: boolean;
}) {
const { label, text, expanded, empty } = params;
return html`
<div class="chat-tool-card__block ${expanded ? "chat-tool-card__block--expanded" : ""}">
<div class="chat-tool-card__block-header">
<span class="chat-tool-card__block-icon">${icons.zap}</span>
<span class="chat-tool-card__block-label">${label}</span>
</div>
${empty
? html`<div class="chat-tool-card__block-empty muted">${text}</div>`
: expanded
? html`<pre class="chat-tool-card__block-content"><code>${text}</code></pre>`
: html`<div class="chat-tool-card__block-preview mono">
${getTruncatedPreview(text)}
</div>`}
</div>
`;
}
function renderCollapsedToolSummary(params: {
label: string;
name: string;
expanded: boolean;
onToggleExpanded: () => void;
}) {
const { label, name, expanded, onToggleExpanded } = params;
return html`
<button
class="chat-tool-msg-summary"
type="button"
aria-expanded=${String(expanded)}
@click=${() => onToggleExpanded()}
>
<span class="chat-tool-msg-summary__icon">${icons.zap}</span>
<span class="chat-tool-msg-summary__label">${label}</span>
<span class="chat-tool-msg-summary__names">${name}</span>
</button>
`;
}
export function renderToolCard(
card: ToolCard,
opts: {
expanded: boolean;
onToggleExpanded: (id: string) => void;
onOpenSidebar?: (content: SidebarContent) => void;
canvasHostUrl?: string | null;
embedSandboxMode?: EmbedSandboxMode;
allowExternalEmbedUrls?: boolean;
},
) {
const hasOutput = Boolean(card.outputText?.trim());
const previewLabel = hasOutput ? "Tool output" : "Tool call";
return html`
<div
class="chat-tool-msg-collapse chat-tool-msg-collapse--manual ${opts.expanded
? "is-open"
: ""}"
>
${renderCollapsedToolSummary({
label: previewLabel,
name: card.name,
expanded: opts.expanded,
onToggleExpanded: () => opts.onToggleExpanded(card.id),
})}
${opts.expanded
? html`
<div class="chat-tool-msg-body">
${renderExpandedToolCardContent(
card,
opts.onOpenSidebar,
opts.canvasHostUrl,
opts.embedSandboxMode ?? "scripts",
opts.allowExternalEmbedUrls ?? false,
)}
</div>
`
: nothing}
</div>
`;
}
export function renderExpandedToolCardContent(
card: ToolCard,
onOpenSidebar?: (content: SidebarContent) => void,
canvasHostUrl?: string | null,
embedSandboxMode: EmbedSandboxMode = "scripts",
allowExternalEmbedUrls = false,
) {
const display = resolveToolDisplay({ name: card.name, args: card.args });
const detail = formatToolDetail(display);
const hasOutput = Boolean(card.outputText?.trim());
const hasInput = Boolean(card.inputText?.trim());
const canOpenSidebar = Boolean(onOpenSidebar);
const previewSidebarContent =
card.preview?.kind === "canvas"
? buildPreviewSidebarContent(card.preview, card.outputText)
: null;
const sidebarActionContent =
previewSidebarContent ?? buildSidebarContent(buildToolCardSidebarContent(card));
const visiblePreview = card.preview
? renderToolPreview(card.preview, "chat_tool", {
onOpenSidebar,
rawText: card.outputText,
canvasHostUrl,
embedSandboxMode,
allowExternalEmbedUrls,
})
: nothing;
return html`
<div class="chat-tool-card chat-tool-card--expanded">
<div class="chat-tool-card__header">
<div class="chat-tool-card__title">
<span class="chat-tool-card__icon">${icons[display.icon]}</span>
<span>${display.label}</span>
</div>
${canOpenSidebar
? html`
<div class="chat-tool-card__actions">
<button
class="chat-tool-card__action-btn"
type="button"
@click=${() => onOpenSidebar?.(sidebarActionContent)}
title="Open in the side panel"
aria-label="Open tool details in side panel"
>
<span class="chat-tool-card__action-icon">${icons.panelRightOpen}</span>
</button>
</div>
`
: nothing}
</div>
${detail ? html`<div class="chat-tool-card__detail">${detail}</div>` : nothing}
${hasInput
? renderToolDataBlock({
label: "Tool input",
text: card.inputText!,
expanded: true,
})
: nothing}
${hasOutput
? card.preview
? html`${visiblePreview} ${renderRawOutputToggle(card.outputText!)}`
: renderToolDataBlock({
label: "Tool output",
text: card.outputText!,
expanded: true,
})
: nothing}
</div>
`;
}
export function renderToolCardSidebar(
card: ToolCard,
onOpenSidebar?: (content: SidebarContent) => void,
canvasHostUrl?: string | null,
embedSandboxMode: EmbedSandboxMode = "scripts",
) {
const display = resolveToolDisplay({ name: card.name, args: card.args });
const detail = formatToolDetail(display);
const hasText = Boolean(card.outputText?.trim());
const hasPreview = Boolean(card.preview);
const sidebarContent =
card.preview?.kind === "canvas"
? buildPreviewSidebarContent(card.preview, card.outputText)
: buildSidebarContent(buildToolCardSidebarContent(card));
const actionContent = sidebarContent ?? buildSidebarContent(buildToolCardSidebarContent(card));
const canClick = Boolean(onOpenSidebar);
const handleClick = canClick ? () => onOpenSidebar?.(actionContent) : undefined;
const isShort = hasText && !hasPreview && (card.outputText?.length ?? 0) <= 240;
const showCollapsed = hasText && !hasPreview && !isShort;
const showInline = hasText && !hasPreview && isShort;
const isEmpty = !hasText && !hasPreview;
return html`
<div
class="chat-tool-card ${canClick ? "chat-tool-card--clickable" : ""}"
@click=${handleClick}
role=${canClick ? "button" : nothing}
tabindex=${canClick ? "0" : nothing}
@keydown=${canClick
? (e: KeyboardEvent) => {
if (e.key !== "Enter" && e.key !== " ") {
return;
}
e.preventDefault();
handleClick?.();
}
: nothing}
>
<div class="chat-tool-card__header">
<div class="chat-tool-card__title">
<span class="chat-tool-card__icon">${icons[display.icon]}</span>
<span>${display.label}</span>
</div>
${canClick
? html`<span class="chat-tool-card__action"
>${hasText || hasPreview ? "View" : ""} ${icons.check}</span
>`
: nothing}
${isEmpty && !canClick
? html`<span class="chat-tool-card__status">${icons.check}</span>`
: nothing}
</div>
${detail ? html`<div class="chat-tool-card__detail">${detail}</div>` : nothing}
${isEmpty ? html`<div class="chat-tool-card__status-text muted">Completed</div>` : nothing}
${hasPreview
? html`${renderToolPreview(card.preview!, "chat_tool", {
onOpenSidebar,
rawText: card.outputText,
canvasHostUrl,
embedSandboxMode,
})}`
: nothing}
${showCollapsed
? html`<div class="chat-tool-card__preview mono">
${getTruncatedPreview(card.outputText!)}
</div>`
: nothing}
${showInline ? html`<div class="chat-tool-card__inline mono">${card.outputText}</div>` : nothing}
</div>
`;
}

27
ui/src/ui/controllers/control-ui-bootstrap.test.ts
View File

@@ -12,6 +12,11 @@ describe("loadControlUiBootstrapConfig", () => {
basePath: "/openclaw",
assistantName: "Ops",
assistantAvatar: "O",
assistantAgentId: "main",
serverVersion: "2026.3.7",
localMediaPreviewRoots: ["/tmp/openclaw"],
embedSandbox: "scripts",
allowExternalEmbedUrls: true,
}),
});
vi.stubGlobal("fetch", fetchMock as unknown as typeof fetch);
@@ -21,6 +26,9 @@ describe("loadControlUiBootstrapConfig", () => {
assistantName: "Assistant",
assistantAvatar: null,
assistantAgentId: null,
localMediaPreviewRoots: [],
embedSandboxMode: "scripts" as const,
allowExternalEmbedUrls: false,
serverVersion: null,
};
@@ -32,8 +40,11 @@ describe("loadControlUiBootstrapConfig", () => {
);
expect(state.assistantName).toBe("Ops");
expect(state.assistantAvatar).toBe("O");
expect(state.assistantAgentId).toBeNull();
expect(state.serverVersion).toBeNull();
expect(state.assistantAgentId).toBe("main");
expect(state.serverVersion).toBe("2026.3.7");
expect(state.localMediaPreviewRoots).toEqual(["/tmp/openclaw"]);
expect(state.embedSandboxMode).toBe("scripts");
expect(state.allowExternalEmbedUrls).toBe(true);
vi.unstubAllGlobals();
});
@@ -47,6 +58,9 @@ describe("loadControlUiBootstrapConfig", () => {
assistantName: "Assistant",
assistantAvatar: null,
assistantAgentId: null,
localMediaPreviewRoots: [],
embedSandboxMode: "scripts" as const,
allowExternalEmbedUrls: false,
serverVersion: null,
};
@@ -57,8 +71,8 @@ describe("loadControlUiBootstrapConfig", () => {
expect.objectContaining({ method: "GET" }),
);
expect(state.assistantName).toBe("Assistant");
expect(state.assistantAgentId).toBeNull();
expect(state.serverVersion).toBeNull();
expect(state.embedSandboxMode).toBe("scripts");
expect(state.allowExternalEmbedUrls).toBe(false);
vi.unstubAllGlobals();
});
@@ -72,6 +86,9 @@ describe("loadControlUiBootstrapConfig", () => {
assistantName: "Assistant",
assistantAvatar: null,
assistantAgentId: null,
localMediaPreviewRoots: [],
embedSandboxMode: "scripts" as const,
allowExternalEmbedUrls: false,
serverVersion: null,
};
@@ -81,8 +98,6 @@ describe("loadControlUiBootstrapConfig", () => {
`/openclaw${CONTROL_UI_BOOTSTRAP_CONFIG_PATH}`,
expect.objectContaining({ method: "GET" }),
);
expect(state.assistantAgentId).toBeNull();
expect(state.serverVersion).toBeNull();
vi.unstubAllGlobals();
});

17
ui/src/ui/controllers/control-ui-bootstrap.ts
View File

@@ -1,6 +1,7 @@
import {
CONTROL_UI_BOOTSTRAP_CONFIG_PATH,
type ControlUiBootstrapConfig,
type ControlUiEmbedSandboxMode,
} from "../../../../src/gateway/control-ui-contract.js";
import { normalizeAssistantIdentity } from "../assistant-identity.ts";
import { normalizeBasePath } from "../navigation.ts";
@@ -11,6 +12,9 @@ export type ControlUiBootstrapState = {
assistantAvatar: string | null;
assistantAgentId: string | null;
serverVersion: string | null;
localMediaPreviewRoots: string[];
embedSandboxMode: ControlUiEmbedSandboxMode;
allowExternalEmbedUrls: boolean;
};
export async function loadControlUiBootstrapConfig(state: ControlUiBootstrapState) {
@@ -37,11 +41,24 @@ export async function loadControlUiBootstrapConfig(state: ControlUiBootstrapStat
}
const parsed = (await res.json()) as ControlUiBootstrapConfig;
const normalized = normalizeAssistantIdentity({
agentId: parsed.assistantAgentId ?? null,
name: parsed.assistantName,
avatar: parsed.assistantAvatar ?? null,
});
state.assistantName = normalized.name;
state.assistantAvatar = normalized.avatar;
state.assistantAgentId = normalized.agentId ?? null;
state.serverVersion = parsed.serverVersion ?? null;
state.localMediaPreviewRoots = Array.isArray(parsed.localMediaPreviewRoots)
? parsed.localMediaPreviewRoots.filter((value): value is string => typeof value === "string")
: [];
state.embedSandboxMode =
parsed.embedSandbox === "trusted"
? "trusted"
: parsed.embedSandbox === "strict"
? "strict"
: "scripts";
state.allowExternalEmbedUrls = parsed.allowExternalEmbedUrls === true;
} catch {
// Ignore bootstrap failures; UI will update identity after connecting.
}

15
ui/src/ui/embed-sandbox.ts Normal file
View File

@@ -0,0 +1,15 @@
import type { ControlUiEmbedSandboxMode } from "../../../src/gateway/control-ui-contract.js";
export type EmbedSandboxMode = ControlUiEmbedSandboxMode;
export function resolveEmbedSandbox(mode: EmbedSandboxMode | null | undefined): string {
switch (mode) {
case "strict":
return "";
case "trusted":
return "allow-scripts allow-same-origin";
case "scripts":
default:
return "allow-scripts";
}
}

20
ui/src/ui/gateway.node.test.ts
View File

@@ -404,6 +404,26 @@ describe("GatewayBrowserClient", () => {
vi.useRealTimers();
});
it("cancels a scheduled reconnect when stopped before the retry fires", async () => {
vi.useFakeTimers();
const client = new GatewayBrowserClient({
url: "ws://127.0.0.1:18789",
token: "shared-auth-token",
});
client.start();
const ws = getLatestWebSocket();
ws.emitClose(1006, "socket lost");
client.stop();
await vi.advanceTimersByTimeAsync(30_000);
expect(wsInstances).toHaveLength(1);
vi.useRealTimers();
});
it("does not auto-reconnect on AUTH_TOKEN_MISSING", async () => {
vi.useFakeTimers();
localStorage.clear();

40
ui/src/ui/gateway.ts
View File

@@ -12,7 +12,6 @@ import {
} from "../../../src/gateway/protocol/connect-error-details.js";
import { clearDeviceAuthToken, loadDeviceAuthToken, storeDeviceAuthToken } from "./device-auth.ts";
import { loadOrCreateDeviceIdentity, signDevicePayload } from "./device-identity.ts";
import { normalizeLowercaseStringOrEmpty, normalizeOptionalString } from "./string-coerce.ts";
import { generateUUID } from "./uuid.ts";
export type GatewayEventFrame = {
@@ -83,7 +82,7 @@ export function isNonRecoverableAuthError(error: GatewayErrorInfo | undefined):
function isTrustedRetryEndpoint(url: string): boolean {
try {
const gatewayUrl = new URL(url, window.location.href);
const host = normalizeLowercaseStringOrEmpty(gatewayUrl.hostname);
const host = gatewayUrl.hostname.trim().toLowerCase();
const isLoopbackHost =
host === "localhost" || host === "::1" || host === "[::1]" || host === "127.0.0.1";
const isLoopbackIPv4 = host.startsWith("127.");
@@ -112,6 +111,7 @@ export type GatewayHelloOk = {
scopes?: string[];
issuedAtMs?: number;
};
canvasHostUrl?: string;
policy?: { tickIntervalMs?: number };
};
@@ -295,10 +295,7 @@ export class GatewayBrowserClient {
stop() {
this.closed = true;
if (this.connectTimer !== null) {
window.clearTimeout(this.connectTimer);
this.connectTimer = null;
}
this.clearConnectTimer();
this.ws?.close();
this.ws = null;
this.pendingConnectError = undefined;
@@ -348,7 +345,11 @@ export class GatewayBrowserClient {
}
const delay = this.backoffMs;
this.backoffMs = Math.min(this.backoffMs * 1.7, 15_000);
window.setTimeout(() => this.connect(), delay);
this.clearConnectTimer();
this.connectTimer = window.setTimeout(() => {
this.connectTimer = null;
this.connect();
}, delay);
}
private flushPending(err: Error) {
@@ -387,8 +388,8 @@ export class GatewayBrowserClient {
const role = CONTROL_UI_OPERATOR_ROLE;
const scopes = [...CONTROL_UI_OPERATOR_SCOPES];
const client = this.buildConnectClient();
const explicitGatewayToken = normalizeOptionalString(this.opts.token);
const explicitPassword = normalizeOptionalString(this.opts.password);
const explicitGatewayToken = this.opts.token?.trim() || undefined;
const explicitPassword = this.opts.password?.trim() || undefined;
// crypto.subtle is only available in secure contexts (HTTPS, localhost).
// Over plain HTTP, we skip device identity and fall back to token-only auth.
@@ -496,10 +497,7 @@ export class GatewayBrowserClient {
return;
}
this.connectSent = true;
if (this.connectTimer !== null) {
window.clearTimeout(this.connectTimer);
this.connectTimer = null;
}
this.clearConnectTimer();
const plan = await this.buildConnectPlan();
void this.request<GatewayHelloOk>("connect", this.buildConnectParams(plan))
@@ -565,8 +563,8 @@ export class GatewayBrowserClient {
}
private selectConnectAuth(params: { role: string; deviceId: string }): SelectedConnectAuth {
const explicitGatewayToken = normalizeOptionalString(this.opts.token);
const authPassword = normalizeOptionalString(this.opts.password);
const explicitGatewayToken = this.opts.token?.trim() || undefined;
const authPassword = this.opts.password?.trim() || undefined;
const storedEntry = loadDeviceAuthToken({
deviceId: params.deviceId,
role: params.role,
@@ -613,11 +611,17 @@ export class GatewayBrowserClient {
private queueConnect() {
this.connectNonce = null;
this.connectSent = false;
if (this.connectTimer !== null) {
window.clearTimeout(this.connectTimer);
}
this.clearConnectTimer();
this.connectTimer = window.setTimeout(() => {
this.connectTimer = null;
void this.sendConnect();
}, 750);
}
private clearConnectTimer() {
if (this.connectTimer !== null) {
window.clearTimeout(this.connectTimer);
this.connectTimer = null;
}
}
}

15
ui/src/ui/sidebar-content.ts Normal file
View File

@@ -0,0 +1,15 @@
export type MarkdownSidebarContent = {
kind: "markdown";
content: string;
};
export type CanvasSidebarContent = {
kind: "canvas";
docId: string;
title?: string;
entryUrl: string;
preferredHeight?: number;
rawText?: string | null;
};
export type SidebarContent = MarkdownSidebarContent | CanvasSidebarContent;

56
ui/src/ui/types/chat-types.ts
View File

@@ -21,12 +21,28 @@ export type MessageGroup = {
};
/** Content item types in a normalized message */
export type MessageContentItem = {
type: "text" | "tool_call" | "tool_result";
text?: string;
name?: string;
args?: unknown;
};
export type MessageContentItem =
| {
type: "text" | "tool_call" | "tool_result";
text?: string;
name?: string;
args?: unknown;
}
| {
type: "attachment";
attachment: {
url: string;
kind: "image" | "audio" | "video" | "document";
label: string;
mimeType?: string;
isVoiceNote?: boolean;
};
}
| {
type: "canvas";
preview: Extract<NonNullable<ToolCard["preview"]>, { kind: "canvas" }>;
rawText?: string | null;
};
/** Normalized message structure for rendering */
export type NormalizedMessage = {
@@ -35,12 +51,34 @@ export type NormalizedMessage = {
timestamp: number;
id?: string;
senderLabel?: string | null;
audioAsVoice?: boolean;
replyTarget?:
| {
kind: "current";
}
| {
kind: "id";
id: string;
}
| null;
};
/** Tool card representation for tool calls and results */
/** Tool card representation for inline tool call/result rendering */
export type ToolCard = {
kind: "call" | "result";
id: string;
name: string;
args?: unknown;
text?: string;
inputText?: string;
outputText?: string;
preview?: {
kind: "canvas";
surface: "assistant_message";
render: "url";
title?: string;
preferredHeight?: number;
url?: string;
viewId?: string;
className?: string;
style?: string;
};
};

1743
ui/src/ui/views/chat.test.ts
View File

File diff suppressed because it is too large Load Diff

338
ui/src/ui/views/chat.ts
View File

@@ -2,10 +2,7 @@ import { html, nothing, type TemplateResult } from "lit";
import { ref } from "lit/directives/ref.js";
import { repeat } from "lit/directives/repeat.js";
import { unsafeHTML } from "lit/directives/unsafe-html.js";
import type {
CompactionStatus as CompactionIndicatorStatus,
FallbackStatus as FallbackIndicatorStatus,
} from "../app-tool-stream.ts";
import type { CompactionStatus, FallbackStatus } from "../app-tool-stream.ts";
import {
CHAT_ATTACHMENT_ACCEPT,
isSupportedChatAttachmentMimeType,
@@ -18,7 +15,12 @@ import {
renderStreamingGroup,
} from "../chat/grouped-render.ts";
import { InputHistory } from "../chat/input-history.ts";
import { normalizeMessage, normalizeRoleForGrouping } from "../chat/message-normalizer.ts";
import { extractTextCached } from "../chat/message-extract.ts";
import {
isToolResultMessage,
normalizeMessage,
normalizeRoleForGrouping,
} from "../chat/message-normalizer.ts";
import { PinnedMessages } from "../chat/pinned-messages.ts";
import { getPinnedMessageSummary } from "../chat/pinned-summary.ts";
import { messageMatchesSearchQuery } from "../chat/search-match.ts";
@@ -32,12 +34,14 @@ import {
type SlashCommandDef,
} from "../chat/slash-commands.ts";
import { isSttSupported, startStt, stopStt } from "../chat/speech.ts";
import { buildSidebarContent, extractToolCards, extractToolPreview } from "../chat/tool-cards.ts";
import type { EmbedSandboxMode } from "../embed-sandbox.ts";
import { icons } from "../icons.ts";
import { toSanitizedMarkdownHtml } from "../markdown.ts";
import { normalizeLowercaseStringOrEmpty } from "../string-coerce.ts";
import type { SidebarContent } from "../sidebar-content.ts";
import { detectTextDirection } from "../text-direction.ts";
import type { GatewaySessionRow, SessionsListResult } from "../types.ts";
import type { ChatItem, MessageGroup } from "../types/chat-types.ts";
import type { ChatItem, MessageGroup, ToolCard } from "../types/chat-types.ts";
import type { ChatAttachment, ChatQueueItem } from "../ui-types.ts";
import { agentLogoUrl, resolveAgentAvatarUrl } from "./agents-utils.ts";
import { renderMarkdownSidebar } from "./markdown-sidebar.ts";
@@ -52,8 +56,8 @@ export type ChatProps = {
loading: boolean;
sending: boolean;
canAbort?: boolean;
compactionStatus?: CompactionIndicatorStatus | null;
fallbackStatus?: FallbackIndicatorStatus | null;
compactionStatus?: CompactionStatus | null;
fallbackStatus?: FallbackStatus | null;
messages: unknown[];
sideResult?: ChatSideResult | null;
toolMessages: unknown[];
@@ -70,11 +74,17 @@ export type ChatProps = {
sessions: SessionsListResult | null;
focusMode: boolean;
sidebarOpen?: boolean;
sidebarContent?: string | null;
sidebarContent?: SidebarContent | null;
sidebarError?: string | null;
splitRatio?: number;
canvasHostUrl?: string | null;
embedSandboxMode?: EmbedSandboxMode;
allowExternalEmbedUrls?: boolean;
assistantName: string;
assistantAvatar: string | null;
localMediaPreviewRoots?: string[];
assistantAttachmentAuthToken?: string | null;
autoExpandToolCalls?: boolean;
attachments?: ChatAttachment[];
onAttachmentsChange?: (attachments: ChatAttachment[]) => void;
showNewMessages?: boolean;
@@ -98,7 +108,7 @@ export type ChatProps = {
onAgentChange: (agentId: string) => void;
onNavigateToAgent?: () => void;
onSessionSelect?: (sessionKey: string) => void;
onOpenSidebar?: (content: string) => void;
onOpenSidebar?: (content: SidebarContent) => void;
onCloseSidebar?: () => void;
onSplitRatioChange?: (ratio: number) => void;
onChatScroll?: (event: Event) => void;
@@ -112,6 +122,9 @@ const FALLBACK_TOAST_DURATION_MS = 8000;
const inputHistories = new Map<string, InputHistory>();
const pinnedMessagesMap = new Map<string, PinnedMessages>();
const deletedMessagesMap = new Map<string, DeletedMessages>();
const expandedToolCardsBySession = new Map<string, Map<string, boolean>>();
const initializedToolCardsBySession = new Map<string, Set<string>>();
const lastAutoExpandPrefBySession = new Map<string, boolean>();
function getInputHistory(sessionKey: string): InputHistory {
return getOrCreateSessionCacheValue(inputHistories, sessionKey, () => new InputHistory());
@@ -133,6 +146,143 @@ function getDeletedMessages(sessionKey: string): DeletedMessages {
);
}
function getExpandedToolCards(sessionKey: string): Map<string, boolean> {
return getOrCreateSessionCacheValue(expandedToolCardsBySession, sessionKey, () => new Map());
}
function getInitializedToolCards(sessionKey: string): Set<string> {
return getOrCreateSessionCacheValue(initializedToolCardsBySession, sessionKey, () => new Set());
}
function appendCanvasBlockToAssistantMessage(
message: unknown,
preview: Extract<NonNullable<ToolCard["preview"]>, { kind: "canvas" }>,
rawText: string | null,
) {
const raw = message as Record<string, unknown>;
const existingContent = Array.isArray(raw.content)
? [...raw.content]
: typeof raw.content === "string"
? [{ type: "text", text: raw.content }]
: typeof raw.text === "string"
? [{ type: "text", text: raw.text }]
: [];
const alreadyHasArtifact = existingContent.some((block) => {
if (!block || typeof block !== "object") {
return false;
}
const typed = block as {
type?: unknown;
preview?: { kind?: unknown; viewId?: unknown; url?: unknown };
};
return (
typed.type === "canvas" &&
typed.preview?.kind === "canvas" &&
((preview.viewId && typed.preview.viewId === preview.viewId) ||
(preview.url && typed.preview.url === preview.url))
);
});
if (alreadyHasArtifact) {
return message;
}
return {
...raw,
content: [
...existingContent,
{
type: "canvas",
preview,
...(rawText ? { rawText } : {}),
},
],
};
}
function extractChatMessagePreview(toolMessage: unknown): {
preview: Extract<NonNullable<ToolCard["preview"]>, { kind: "canvas" }>;
text: string | null;
timestamp: number | null;
} | null {
const normalized = normalizeMessage(toolMessage);
const cards = extractToolCards(toolMessage, "preview");
for (let index = cards.length - 1; index >= 0; index--) {
const card = cards[index];
if (card?.preview?.kind === "canvas") {
return {
preview: card.preview,
text: card.outputText ?? null,
timestamp: normalized.timestamp ?? null,
};
}
}
const text = extractTextCached(toolMessage) ?? undefined;
const toolRecord = toolMessage as Record<string, unknown>;
const toolName =
typeof toolRecord.toolName === "string"
? toolRecord.toolName
: typeof toolRecord.tool_name === "string"
? toolRecord.tool_name
: undefined;
const preview = extractToolPreview(text, toolName);
if (preview?.kind !== "canvas") {
return null;
}
return { preview, text: text ?? null, timestamp: normalized.timestamp ?? null };
}
function findNearestAssistantMessageIndex(
items: ChatItem[],
toolTimestamp: number | null,
): number | null {
const assistantEntries = items
.map((item, index) => {
if (item.kind !== "message") {
return null;
}
const message = item.message as Record<string, unknown>;
const role = typeof message.role === "string" ? message.role.toLowerCase() : "";
if (role !== "assistant") {
return null;
}
return {
index,
timestamp: normalizeMessage(item.message).timestamp ?? null,
};
})
.filter(Boolean) as Array<{ index: number; timestamp: number | null }>;
if (assistantEntries.length === 0) {
return null;
}
if (toolTimestamp == null) {
return assistantEntries[assistantEntries.length - 1]?.index ?? null;
}
let previous: { index: number; timestamp: number } | null = null;
let next: { index: number; timestamp: number } | null = null;
for (const entry of assistantEntries) {
if (entry.timestamp == null) {
continue;
}
if (entry.timestamp <= toolTimestamp) {
previous = { index: entry.index, timestamp: entry.timestamp };
continue;
}
next = { index: entry.index, timestamp: entry.timestamp };
break;
}
if (previous && next) {
const previousDelta = toolTimestamp - previous.timestamp;
const nextDelta = next.timestamp - toolTimestamp;
return nextDelta < previousDelta ? next.index : previous.index;
}
if (previous) {
return previous.index;
}
if (next) {
return next.index;
}
return assistantEntries[assistantEntries.length - 1]?.index ?? null;
}
interface ChatEphemeralState {
sttRecording: boolean;
sttInterimText: string;
@@ -183,11 +333,65 @@ function adjustTextareaHeight(el: HTMLTextAreaElement) {
el.style.height = `${Math.min(el.scrollHeight, 150)}px`;
}
function renderCompactionIndicator(status: CompactionIndicatorStatus | null | undefined) {
function syncToolCardExpansionState(
sessionKey: string,
items: Array<ChatItem | MessageGroup>,
autoExpandToolCalls: boolean,
) {
const expanded = getExpandedToolCards(sessionKey);
const initialized = getInitializedToolCards(sessionKey);
const previousAutoExpand = lastAutoExpandPrefBySession.get(sessionKey) ?? false;
const currentToolCardIds = new Set<string>();
for (const item of items) {
if (item.kind !== "group") {
continue;
}
for (const entry of item.messages) {
const cards = extractToolCards(entry.message, entry.key);
for (let cardIndex = 0; cardIndex < cards.length; cardIndex++) {
const disclosureId = `${entry.key}:toolcard:${cardIndex}`;
currentToolCardIds.add(disclosureId);
if (initialized.has(disclosureId)) {
continue;
}
expanded.set(disclosureId, autoExpandToolCalls);
initialized.add(disclosureId);
}
const messageRecord = entry.message as Record<string, unknown>;
const role = typeof messageRecord.role === "string" ? messageRecord.role : "unknown";
const normalizedRole = normalizeRoleForGrouping(role);
const isToolMessage =
isToolResultMessage(entry.message) ||
normalizedRole === "tool" ||
role.toLowerCase() === "toolresult" ||
role.toLowerCase() === "tool_result" ||
typeof messageRecord.toolCallId === "string" ||
typeof messageRecord.tool_call_id === "string";
if (!isToolMessage) {
continue;
}
const disclosureId = `toolmsg:${entry.key}`;
currentToolCardIds.add(disclosureId);
if (initialized.has(disclosureId)) {
continue;
}
expanded.set(disclosureId, autoExpandToolCalls);
initialized.add(disclosureId);
}
}
if (autoExpandToolCalls && !previousAutoExpand) {
for (const toolCardId of currentToolCardIds) {
expanded.set(toolCardId, true);
}
}
lastAutoExpandPrefBySession.set(sessionKey, autoExpandToolCalls);
}
function renderCompactionIndicator(status: CompactionStatus | null | undefined) {
if (!status) {
return nothing;
}
if (status.phase === "active") {
if (status.phase === "active" || status.phase === "retrying") {
return html`
<div
class="compaction-indicator compaction-indicator--active"
@@ -198,18 +402,7 @@ function renderCompactionIndicator(status: CompactionIndicatorStatus | null | un
</div>
`;
}
if (status.phase === "retrying") {
return html`
<div
class="compaction-indicator compaction-indicator--active"
role="status"
aria-live="polite"
>
${icons.loader} Retrying after compaction...
</div>
`;
}
if (status.phase === "complete" && status.completedAt) {
if (status.completedAt) {
const elapsed = Date.now() - status.completedAt;
if (elapsed < COMPACTION_TOAST_DURATION_MS) {
return html`
@@ -226,7 +419,7 @@ function renderCompactionIndicator(status: CompactionIndicatorStatus | null | un
return nothing;
}
function renderFallbackIndicator(status: FallbackIndicatorStatus | null | undefined) {
function renderFallbackIndicator(status: FallbackStatus | null | undefined) {
if (!status) {
return nothing;
}
@@ -538,12 +731,12 @@ function updateSlashMenu(value: string, requestUpdate: () => void): void {
// Arg mode: /command <partial-arg>
const argMatch = value.match(/^\/(\S+)\s(.*)$/);
if (argMatch) {
const cmdName = normalizeLowercaseStringOrEmpty(argMatch[1]);
const argFilter = normalizeLowercaseStringOrEmpty(argMatch[2]);
const cmdName = argMatch[1].toLowerCase();
const argFilter = argMatch[2].toLowerCase();
const cmd = SLASH_COMMANDS.find((c) => c.name === cmdName);
if (cmd?.argOptions?.length) {
const filtered = argFilter
? cmd.argOptions.filter((opt) => normalizeLowercaseStringOrEmpty(opt).startsWith(argFilter))
? cmd.argOptions.filter((opt) => opt.toLowerCase().startsWith(argFilter))
: cmd.argOptions;
if (filtered.length > 0) {
vs.slashMenuMode = "args";
@@ -926,7 +1119,7 @@ function renderSlashMenu(
export function renderChat(props: ChatProps) {
const canCompose = props.connected;
const isBusy = props.sending || props.stream !== null || props.canAbort;
const isBusy = props.sending || props.stream !== null;
const canAbort = Boolean(props.canAbort && props.onAbort);
const activeSession = props.sessions?.sessions?.find((row) => row.key === props.sessionKey);
const reasoningLevel = activeSession?.reasoningLevel ?? "off";
@@ -975,6 +1168,12 @@ export function renderChat(props: ChatProps) {
};
const chatItems = buildChatItems(props);
syncToolCardExpansionState(props.sessionKey, chatItems, Boolean(props.autoExpandToolCalls));
const expandedToolCards = getExpandedToolCards(props.sessionKey);
const toggleToolCardExpanded = (toolCardId: string) => {
expandedToolCards.set(toolCardId, !expandedToolCards.get(toolCardId));
requestUpdate();
};
const isEmpty = chatItems.length === 0 && !props.loading;
const thread = html`
@@ -1062,9 +1261,24 @@ export function renderChat(props: ChatProps) {
onOpenSidebar: props.onOpenSidebar,
showReasoning,
showToolCalls: props.showToolCalls,
autoExpandToolCalls: Boolean(props.autoExpandToolCalls),
isToolMessageExpanded: (messageId: string) =>
expandedToolCards.get(messageId) ?? false,
onToggleToolMessageExpanded: (messageId: string) => {
expandedToolCards.set(messageId, !expandedToolCards.get(messageId));
requestUpdate();
},
isToolExpanded: (toolCardId: string) => expandedToolCards.get(toolCardId) ?? false,
onToggleToolExpanded: toggleToolCardExpanded,
onRequestUpdate: requestUpdate,
assistantName: props.assistantName,
assistantAvatar: assistantIdentity.avatar,
basePath: props.basePath,
localMediaPreviewRoots: props.localMediaPreviewRoots ?? [],
assistantAttachmentAuthToken: props.assistantAttachmentAuthToken ?? null,
canvasHostUrl: props.canvasHostUrl,
embedSandboxMode: props.embedSandboxMode ?? "scripts",
allowExternalEmbedUrls: props.allowExternalEmbedUrls ?? false,
contextWindow:
activeSession?.contextTokens ?? props.sessions?.defaults?.contextTokens ?? null,
onDelete: () => {
@@ -1245,12 +1459,25 @@ export function renderChat(props: ChatProps) {
${renderMarkdownSidebar({
content: props.sidebarContent ?? null,
error: props.sidebarError ?? null,
canvasHostUrl: props.canvasHostUrl,
embedSandboxMode: props.embedSandboxMode ?? "scripts",
allowExternalEmbedUrls: props.allowExternalEmbedUrls ?? false,
onClose: props.onCloseSidebar!,
onViewRawText: () => {
if (!props.sidebarContent || !props.onOpenSidebar) {
return;
}
props.onOpenSidebar(`\`\`\`\n${props.sidebarContent}\n\`\`\``);
if (props.sidebarContent.kind === "markdown") {
props.onOpenSidebar(
buildSidebarContent(`\`\`\`\n${props.sidebarContent.content}\n\`\`\``),
);
return;
}
if (props.sidebarContent.rawText?.trim()) {
props.onOpenSidebar(
buildSidebarContent(`\`\`\`json\n${props.sidebarContent.rawText}\n\`\`\``),
);
}
},
})}
</div>
@@ -1419,7 +1646,7 @@ export function renderChat(props: ChatProps) {
${icons.download}
</button>
${canAbort && (isBusy || props.sending)
${canAbort
? html`
<button
class="chat-send-btn chat-send-btn--stop"
@@ -1471,14 +1698,13 @@ function groupMessages(items: ChatItem[]): Array<ChatItem | MessageGroup> {
const normalized = normalizeMessage(item.message);
const role = normalizeRoleForGrouping(normalized.role);
const senderLabel =
normalizeLowercaseStringOrEmpty(role) === "user" ? (normalized.senderLabel ?? null) : null;
const senderLabel = role.toLowerCase() === "user" ? (normalized.senderLabel ?? null) : null;
const timestamp = normalized.timestamp || Date.now();
if (
!currentGroup ||
currentGroup.role !== role ||
(normalizeLowercaseStringOrEmpty(role) === "user" && currentGroup.senderLabel !== senderLabel)
(role.toLowerCase() === "user" && currentGroup.senderLabel !== senderLabel)
) {
if (currentGroup) {
result.push(currentGroup);
@@ -1537,7 +1763,7 @@ function buildChatItems(props: ChatProps): Array<ChatItem | MessageGroup> {
continue;
}
if (!props.showToolCalls && normalizeLowercaseStringOrEmpty(normalized.role) === "toolresult") {
if (!props.showToolCalls && normalized.role.toLowerCase() === "toolresult") {
continue;
}
@@ -1552,6 +1778,31 @@ function buildChatItems(props: ChatProps): Array<ChatItem | MessageGroup> {
message: msg,
});
}
const liftedCanvasSources = tools
.map((tool) => extractChatMessagePreview(tool))
.filter((entry) => Boolean(entry)) as Array<{
preview: Extract<NonNullable<ToolCard["preview"]>, { kind: "canvas" }>;
text: string | null;
timestamp: number | null;
}>;
for (const liftedCanvasSource of liftedCanvasSources) {
const assistantIndex = findNearestAssistantMessageIndex(items, liftedCanvasSource.timestamp);
if (assistantIndex == null) {
continue;
}
const item = items[assistantIndex];
if (!item || item.kind !== "message") {
continue;
}
items[assistantIndex] = {
...item,
message: appendCanvasBlockToAssistantMessage(
item.message as Record<string, unknown>,
liftedCanvasSource.preview,
liftedCanvasSource.text,
),
};
}
// Interleave stream segments and tool cards in order. Each segment
// contains text that was streaming before the corresponding tool started.
// This ensures correct visual ordering: text → tool → text → tool → ...
@@ -1596,7 +1847,20 @@ function messageKey(message: unknown, index: number): string {
const m = message as Record<string, unknown>;
const toolCallId = typeof m.toolCallId === "string" ? m.toolCallId : "";
if (toolCallId) {
return `tool:${toolCallId}`;
const role = typeof m.role === "string" ? m.role : "unknown";
const id = typeof m.id === "string" ? m.id : "";
if (id) {
return `tool:${role}:${toolCallId}:${id}`;
}
const messageId = typeof m.messageId === "string" ? m.messageId : "";
if (messageId) {
return `tool:${role}:${toolCallId}:${messageId}`;
}
const timestamp = typeof m.timestamp === "number" ? m.timestamp : null;
if (timestamp != null) {
return `tool:${role}:${toolCallId}:${timestamp}:${index}`;
}
return `tool:${role}:${toolCallId}:${index}`;
}
const id = typeof m.id === "string" ? m.id : "";
if (id) {

60
ui/src/ui/views/markdown-sidebar.ts
View File

@@ -1,20 +1,36 @@
import { html } from "lit";
import { html, nothing } from "lit";
import { unsafeHTML } from "lit/directives/unsafe-html.js";
import { resolveCanvasIframeUrl } from "../canvas-url.ts";
import { resolveEmbedSandbox, type EmbedSandboxMode } from "../embed-sandbox.ts";
import { icons } from "../icons.ts";
import { toSanitizedMarkdownHtml } from "../markdown.ts";
import type { SidebarContent } from "../sidebar-content.ts";
function resolveSidebarCanvasSandbox(
content: SidebarContent,
embedSandboxMode: EmbedSandboxMode,
): string {
return content.kind === "canvas" ? resolveEmbedSandbox(embedSandboxMode) : "allow-scripts";
}
export type MarkdownSidebarProps = {
content: string | null;
content: SidebarContent | null;
error: string | null;
onClose: () => void;
onViewRawText: () => void;
canvasHostUrl?: string | null;
embedSandboxMode?: EmbedSandboxMode;
allowExternalEmbedUrls?: boolean;
};
export function renderMarkdownSidebar(props: MarkdownSidebarProps) {
const content = props.content;
return html`
<div class="sidebar-panel">
<div class="sidebar-header">
<div class="sidebar-title">Tool Output</div>
<div class="sidebar-title">
${content?.kind === "canvas" ? content.title?.trim() || "Render Preview" : "Tool Details"}
</div>
<button @click=${props.onClose} class="btn" title="Close sidebar">${icons.x}</button>
</div>
<div class="sidebar-content">
@@ -25,10 +41,40 @@ export function renderMarkdownSidebar(props: MarkdownSidebarProps) {
View Raw Text
</button>
`
: props.content
? html`<div class="sidebar-markdown">
${unsafeHTML(toSanitizedMarkdownHtml(props.content))}
</div>`
: content
? content.kind === "canvas"
? html`
<div class="chat-tool-card__preview" data-kind="canvas">
<div class="chat-tool-card__preview-panel" data-side="front">
<iframe
class="chat-tool-card__preview-frame"
title=${content.title?.trim() || "Render preview"}
sandbox=${resolveSidebarCanvasSandbox(
content,
props.embedSandboxMode ?? "scripts",
)}
src=${resolveCanvasIframeUrl(
content.entryUrl,
props.canvasHostUrl,
props.allowExternalEmbedUrls ?? false,
) ?? nothing}
style=${content.preferredHeight
? `height:${content.preferredHeight}px`
: ""}
></iframe>
</div>
${content.rawText?.trim()
? html`
<div style="margin-top: 12px;">
<button @click=${props.onViewRawText} class="btn">View Raw Text</button>
</div>
`
: nothing}
</div>
`
: html`<div class="sidebar-markdown">
${unsafeHTML(toSanitizedMarkdownHtml(content.content))}
</div>`
: html` <div class="muted">No content available</div> `}
</div>
</div>