mirror of
https://github.com/openclaw/openclaw.git
synced 2026-07-29 13:11:17 +00:00
fix: deliver Codex-generated images on message-tool routes (#110893)
* fix: deliver Codex-generated images on message-tool routes * test(codex): cover native image delivery path Document trusted harness artifact provenance and cover the runner-to-dispatch bridge.\n\nRelease note: Codex-generated images now reach message-tool-only routes while normal source suppression remains intact.\n\nCo-authored-by: Omar Shahine <10343873+omarshahine@users.noreply.github.com> --------- Co-authored-by: Omar Shahine <10343873+omarshahine@users.noreply.github.com> Co-authored-by: Peter Steinberger <steipete@gmail.com>
This commit is contained in:
@@ -833,7 +833,9 @@ See the full channel index: [Channels](/channels).
|
||||
|
||||
Group messages default to **require mention** (metadata mention or safe regex patterns). Applies to WhatsApp, Telegram, Discord, Google Chat, and iMessage group chats.
|
||||
|
||||
Visible replies are controlled separately. Normal group, channel, and internal WebChat direct requests default to automatic final delivery: final assistant text posts through the legacy visible reply path. Opt into `messages.visibleReplies: "message_tool"` or `messages.groupChat.visibleReplies: "message_tool"` when visible output should only post after the agent calls `message(action=send)`. If the model returns a substantive final answer without calling the message tool in an opted-in tool-only mode, that final text stays private, the gateway verbose log records suppressed payload metadata, and OpenClaw enqueues one recovery retry asking the model to deliver the same reply via `message(action=send)`.
|
||||
Visible replies are controlled separately. Normal group, channel, and internal WebChat direct requests default to automatic final delivery: final assistant text posts through the legacy visible reply path. Opt into `messages.visibleReplies: "message_tool"` or `messages.groupChat.visibleReplies: "message_tool"` when model-authored source replies should only post after the agent calls `message(action=send)`. If the model returns a substantive final answer without calling the message tool in an opted-in tool-only mode, that final text stays private, the gateway verbose log records suppressed payload metadata, and OpenClaw enqueues one recovery retry asking the model to deliver the same reply via `message(action=send)`.
|
||||
|
||||
The tool-only policy governs assistant source replies and generic tool media. It does not suppress runtime-owned terminal output such as authorized command responses, durable completion notices, or provider-native artifacts that the owning harness explicitly classifies as host-owned. Host-owned artifacts are delivered through the normal channel dispatch path and still respect outbound `sendPolicy` denial. Ambient `room_event` turns remain quiet unless they are explicit commands, even when runtime output is marked host-owned.
|
||||
|
||||
Tool-only visible replies require a model/runtime that reliably calls tools, and are recommended for shared ambient rooms on latest-generation models such as GPT-5.6 Sol. Some weaker models can answer final text but fail to understand that source-visible output must be sent with `message(action=send)`. OpenClaw recovers the common stranded-final case by default only when the final is substantive, the source turn was not a room event, send policy did not deny delivery, and no source reply was already sent. Recovery is bounded to one retry; it suppresses persistence for the synthetic retry prompt and keeps that retry out of collect batching so it cannot merge with unrelated queued prompts. If the retry also strands or cannot be enqueued, OpenClaw delivers only a sanitized diagnostic such as "I generated a reply but could not deliver it to this chat. Please try again." The original private final text is never marked for automatic source delivery. For models that repeatedly strand replies, use `"automatic"` so the final assistant turn is the visible reply path, switch to a stronger tool-calling model, inspect the gateway verbose log for the suppressed payload summary, or set `messages.groupChat.visibleReplies: "automatic"` to use visible final replies for every group/channel request.
|
||||
|
||||
|
||||
@@ -435,6 +435,13 @@ yourself.
|
||||
This keeps text, image, video, music, TTS, approval, and messaging-tool
|
||||
outputs on the same delivery path as OpenClaw-backed runs.
|
||||
|
||||
Set `AgentHarnessAttemptResult.hostOwnedToolMediaUrls` only for native artifacts
|
||||
that the trusted harness runtime created and persisted itself. Every entry must
|
||||
also appear in `toolMediaUrls`. Never include model-selected dynamic-tool or
|
||||
OpenClaw-tool media. On `message_tool_only` routes, this narrow provenance lets
|
||||
native runtime artifacts survive source-reply suppression; normal send policy
|
||||
and ambient-room admission still apply.
|
||||
|
||||
### Terminal tool outcomes
|
||||
|
||||
`AgentHarnessAttemptParams.observeToolTerminal` is the host-owned terminal
|
||||
|
||||
@@ -97,6 +97,14 @@ export class CodexGeneratedMediaProjection {
|
||||
return mediaUrls.size > 0 ? [...mediaUrls] : params.toolMediaUrls;
|
||||
}
|
||||
|
||||
buildHostOwnedMediaUrls(params: { messagingToolSentMediaUrls?: string[] }): string[] | undefined {
|
||||
if ((params.messagingToolSentMediaUrls?.length ?? 0) > 0) {
|
||||
return undefined;
|
||||
}
|
||||
const mediaUrls = [...this.urlsByItemId.values()];
|
||||
return mediaUrls.length > 0 ? mediaUrls : undefined;
|
||||
}
|
||||
|
||||
private recordUrl(params: { itemId: string; mediaUrl: string; replaceExisting?: boolean }): void {
|
||||
if (this.urlsByItemId.has(params.itemId) && params.replaceExisting !== true) {
|
||||
this.itemIds.add(params.itemId);
|
||||
|
||||
@@ -1101,6 +1101,7 @@ describe("CodexAppServerEventProjector", () => {
|
||||
|
||||
expect(result.assistantTexts).toStrictEqual([]);
|
||||
expect(result.toolMediaUrls).toEqual([savedPath]);
|
||||
expect(result.hostOwnedToolMediaUrls).toEqual([savedPath]);
|
||||
expect(result.replayMetadata).toStrictEqual({
|
||||
hadPotentialSideEffects: true,
|
||||
replaySafe: false,
|
||||
@@ -1130,6 +1131,7 @@ describe("CodexAppServerEventProjector", () => {
|
||||
|
||||
expect(result.assistantTexts).toStrictEqual([]);
|
||||
expect(result.toolMediaUrls).toHaveLength(1);
|
||||
expect(result.hostOwnedToolMediaUrls).toEqual(result.toolMediaUrls);
|
||||
expect(mediaUrl).toContain(`${path.sep}media${path.sep}tool-image-generation${path.sep}`);
|
||||
expect(mediaUrl?.endsWith(".png")).toBe(true);
|
||||
await expect(fs.readFile(mediaUrl ?? "")).resolves.toEqual(
|
||||
@@ -1349,6 +1351,7 @@ describe("CodexAppServerEventProjector", () => {
|
||||
|
||||
expect(result.toolMediaUrls).toHaveLength(2);
|
||||
expect(new Set(result.toolMediaUrls)).toHaveLength(2);
|
||||
expect(result.hostOwnedToolMediaUrls).toEqual(result.toolMediaUrls);
|
||||
});
|
||||
|
||||
it("does not append native Codex image-generation media after explicit media delivery", async () => {
|
||||
@@ -1375,6 +1378,7 @@ describe("CodexAppServerEventProjector", () => {
|
||||
});
|
||||
|
||||
expect(result.toolMediaUrls).toStrictEqual([]);
|
||||
expect(result.hostOwnedToolMediaUrls).toBeUndefined();
|
||||
});
|
||||
|
||||
it("propagates message-tool-only source reply delivery telemetry", async () => {
|
||||
|
||||
@@ -430,6 +430,7 @@ export class CodexAppServerEventProjector {
|
||||
messagingToolSourceReplyPayloads: toolTelemetry.messagingToolSourceReplyPayloads ?? [],
|
||||
heartbeatToolResponse: toolTelemetry.heartbeatToolResponse,
|
||||
toolMediaUrls: this.generatedMediaProjection.buildToolMediaUrls(toolTelemetry),
|
||||
hostOwnedToolMediaUrls: this.generatedMediaProjection.buildHostOwnedMediaUrls(toolTelemetry),
|
||||
toolAudioAsVoice: toolTelemetry.toolAudioAsVoice,
|
||||
successfulCronAdds: toolTelemetry.successfulCronAdds,
|
||||
cloudCodeAssistFormatError: false,
|
||||
|
||||
@@ -281,6 +281,7 @@ export function getMockRuntimeIdentity() {
|
||||
|
||||
export function mockClientRuntimeMethods() {
|
||||
return {
|
||||
getInstanceId: () => "test-client-1",
|
||||
getRuntimeIdentity: getMockRuntimeIdentity,
|
||||
getServerVersion: getMockServerVersion,
|
||||
};
|
||||
|
||||
@@ -4123,39 +4123,40 @@ describe("runCodexAppServerAttempt", () => {
|
||||
});
|
||||
|
||||
it("surfaces Codex-native image generation saved paths as reply media", async () => {
|
||||
const harness = createStartedThreadHarness();
|
||||
const params = createParams(
|
||||
path.join(tempDir, "session.jsonl"),
|
||||
path.join(tempDir, "workspace"),
|
||||
);
|
||||
|
||||
const run = runCodexAppServerAttempt(params);
|
||||
await harness.waitForMethod("turn/start");
|
||||
await harness.notify({
|
||||
method: "turn/completed",
|
||||
params: {
|
||||
threadId: "thread-1",
|
||||
turnId: "turn-1",
|
||||
turn: {
|
||||
id: "turn-1",
|
||||
status: "completed",
|
||||
items: [
|
||||
{
|
||||
type: "imageGeneration",
|
||||
id: "ig_123",
|
||||
status: "completed",
|
||||
revisedPrompt: "A tiny blue square",
|
||||
result: "Zm9v",
|
||||
savedPath: "/tmp/codex-home/generated_images/session-1/ig_123.png",
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
const savedPath = "/tmp/codex-home/generated_images/session-1/ig_123.png";
|
||||
const harness = createAppServerHarness(async (method) => {
|
||||
if (method === "thread/start") {
|
||||
return threadStartResult();
|
||||
}
|
||||
if (method === "turn/start") {
|
||||
return {
|
||||
turn: {
|
||||
id: "turn-1",
|
||||
status: "completed",
|
||||
items: [
|
||||
{
|
||||
type: "imageGeneration",
|
||||
id: "ig_123",
|
||||
status: "completed",
|
||||
revisedPrompt: "A tiny blue square",
|
||||
result: "Zm9v",
|
||||
savedPath,
|
||||
},
|
||||
],
|
||||
},
|
||||
};
|
||||
}
|
||||
return {};
|
||||
});
|
||||
|
||||
const result = await run;
|
||||
const result = await runCodexAppServerAttempt(
|
||||
createParams(path.join(tempDir, "session.jsonl"), path.join(tempDir, "workspace")),
|
||||
);
|
||||
|
||||
expect(harness.requests.map((entry) => entry.method)).toContain("turn/start");
|
||||
expect(result.assistantTexts).toEqual([]);
|
||||
expect(result.toolMediaUrls).toEqual(["/tmp/codex-home/generated_images/session-1/ig_123.png"]);
|
||||
expect(result.toolMediaUrls).toEqual([savedPath]);
|
||||
expect(result.hostOwnedToolMediaUrls).toEqual([savedPath]);
|
||||
});
|
||||
|
||||
it("does not complete on unscoped turn/completed notifications", async () => {
|
||||
|
||||
@@ -43,6 +43,7 @@ const loggerWarnMock = vi.fn();
|
||||
let refreshRuntimeAuthOnFirstPromptError = false;
|
||||
let clearRuntimeConfigSnapshot: typeof import("../config/config.js").clearRuntimeConfigSnapshot;
|
||||
let setRuntimeConfigSnapshot: typeof import("../config/config.js").setRuntimeConfigSnapshot;
|
||||
let getReplyPayloadMetadata: typeof import("../auto-reply/reply-payload.js").getReplyPayloadMetadata;
|
||||
|
||||
vi.mock("openclaw/plugin-sdk/llm", async () => {
|
||||
const actual =
|
||||
@@ -180,6 +181,7 @@ beforeAll(async () => {
|
||||
vi.useRealTimers();
|
||||
vi.resetModules();
|
||||
installRunEmbeddedMocks();
|
||||
({ getReplyPayloadMetadata } = await import("../auto-reply/reply-payload.js"));
|
||||
({ clearRuntimeConfigSnapshot, setRuntimeConfigSnapshot } = await import("../config/config.js"));
|
||||
({ runEmbeddedAgent } = await import("./embedded-agent-runner/run.js"));
|
||||
const { SessionManager: LoadedSessionManager } =
|
||||
@@ -1123,6 +1125,41 @@ describe("runEmbeddedAgent", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("preserves harness-owned media provenance through terminal preparation", async () => {
|
||||
const sessionFile = nextSessionFile();
|
||||
const cfg = createEmbeddedAgentRunnerOpenAiConfig(["mock-1"]);
|
||||
runEmbeddedAttemptMock.mockResolvedValueOnce(
|
||||
makeEmbeddedRunnerAttempt({
|
||||
toolMediaUrls: ["/tmp/generated.png"],
|
||||
hostOwnedToolMediaUrls: ["/tmp/generated.png"],
|
||||
}),
|
||||
);
|
||||
|
||||
const result = await runEmbeddedAgent({
|
||||
sessionId: "session:test",
|
||||
sessionFile,
|
||||
workspaceDir,
|
||||
config: cfg,
|
||||
prompt: "generate an image",
|
||||
provider: "openai",
|
||||
model: "mock-1",
|
||||
timeoutMs: 5_000,
|
||||
agentDir,
|
||||
runId: nextRunId("host-owned-media"),
|
||||
sourceReplyDeliveryMode: "message_tool_only",
|
||||
enqueue: immediateEnqueue,
|
||||
});
|
||||
|
||||
expect(result.payloads).toHaveLength(1);
|
||||
expect(result.payloads?.[0]).toMatchObject({
|
||||
mediaUrls: ["/tmp/generated.png"],
|
||||
mediaUrl: "/tmp/generated.png",
|
||||
});
|
||||
expect(getReplyPayloadMetadata(result.payloads?.[0] ?? {})).toMatchObject({
|
||||
deliverDespiteSourceReplySuppression: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("handles prompt error paths without dropping user state", async () => {
|
||||
const sessionFile = nextSessionFile();
|
||||
const cfg = createEmbeddedAgentRunnerOpenAiConfig(["mock-error"]);
|
||||
|
||||
@@ -138,6 +138,9 @@ export function prepareEmbeddedRunTerminal(input: {
|
||||
const payloadsWithToolMedia = mergeAttemptToolMediaPayloads({
|
||||
payloads,
|
||||
toolMediaUrls: attempt.toolMediaUrls,
|
||||
// Preserve harness provenance through terminal delivery. Without it,
|
||||
// message-tool-only routes silently drop native runtime artifacts.
|
||||
hostOwnedToolMediaUrls: attempt.hostOwnedToolMediaUrls,
|
||||
toolAudioAsVoice: attempt.toolAudioAsVoice,
|
||||
toolTrustedLocalMedia: attempt.toolTrustedLocalMedia,
|
||||
sourceReplyDeliveryMode: runParams.sourceReplyDeliveryMode,
|
||||
|
||||
@@ -48,6 +48,150 @@ describe("mergeAttemptToolMediaPayloads", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it("marks harness-owned media when source replies require the message tool", () => {
|
||||
const [mediaReply] =
|
||||
mergeAttemptToolMediaPayloads({
|
||||
toolMediaUrls: ["/tmp/generated.png"],
|
||||
hostOwnedToolMediaUrls: ["/tmp/generated.png"],
|
||||
sourceReplyDeliveryMode: "message_tool_only",
|
||||
}) ?? [];
|
||||
|
||||
expect(mediaReply).toEqual({
|
||||
mediaUrls: ["/tmp/generated.png"],
|
||||
mediaUrl: "/tmp/generated.png",
|
||||
audioAsVoice: undefined,
|
||||
trustedLocalMedia: undefined,
|
||||
});
|
||||
expect(getReplyPayloadMetadata(mediaReply ?? {})).toMatchObject({
|
||||
deliverDespiteSourceReplySuppression: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("does not mark generic tool media as host-owned", () => {
|
||||
const [mediaReply] =
|
||||
mergeAttemptToolMediaPayloads({
|
||||
toolMediaUrls: ["/tmp/reply.opus"],
|
||||
toolAudioAsVoice: true,
|
||||
sourceReplyDeliveryMode: "message_tool_only",
|
||||
}) ?? [];
|
||||
|
||||
expect(mediaReply).toEqual({
|
||||
mediaUrls: ["/tmp/reply.opus"],
|
||||
mediaUrl: "/tmp/reply.opus",
|
||||
audioAsVoice: true,
|
||||
trustedLocalMedia: undefined,
|
||||
});
|
||||
expect(getReplyPayloadMetadata(mediaReply ?? {})).toBeUndefined();
|
||||
});
|
||||
|
||||
it("ignores host-owned provenance outside the delivered tool media set", () => {
|
||||
const [mediaReply] =
|
||||
mergeAttemptToolMediaPayloads({
|
||||
toolMediaUrls: ["/tmp/tool.png"],
|
||||
hostOwnedToolMediaUrls: ["/tmp/forged.png"],
|
||||
sourceReplyDeliveryMode: "message_tool_only",
|
||||
}) ?? [];
|
||||
|
||||
expect(mediaReply).toMatchObject({
|
||||
mediaUrls: ["/tmp/tool.png"],
|
||||
mediaUrl: "/tmp/tool.png",
|
||||
});
|
||||
expect(getReplyPayloadMetadata(mediaReply ?? {})).toBeUndefined();
|
||||
});
|
||||
|
||||
it("keeps generic and host-owned media in separate delivery payloads", () => {
|
||||
const [genericReply, hostOwnedReply] =
|
||||
mergeAttemptToolMediaPayloads({
|
||||
toolMediaUrls: ["/tmp/reply.opus", "/tmp/generated.png"],
|
||||
hostOwnedToolMediaUrls: ["/tmp/generated.png"],
|
||||
toolAudioAsVoice: true,
|
||||
sourceReplyDeliveryMode: "message_tool_only",
|
||||
}) ?? [];
|
||||
|
||||
expect(genericReply).toEqual({
|
||||
mediaUrls: ["/tmp/reply.opus"],
|
||||
mediaUrl: "/tmp/reply.opus",
|
||||
audioAsVoice: true,
|
||||
trustedLocalMedia: undefined,
|
||||
});
|
||||
expect(getReplyPayloadMetadata(genericReply ?? {})).toBeUndefined();
|
||||
expect(hostOwnedReply).toEqual({
|
||||
mediaUrls: ["/tmp/generated.png"],
|
||||
mediaUrl: "/tmp/generated.png",
|
||||
audioAsVoice: undefined,
|
||||
trustedLocalMedia: undefined,
|
||||
});
|
||||
expect(getReplyPayloadMetadata(hostOwnedReply ?? {})).toMatchObject({
|
||||
deliverDespiteSourceReplySuppression: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("does not mark text-bearing source replies as host-owned", () => {
|
||||
const [reply] =
|
||||
mergeAttemptToolMediaPayloads({
|
||||
payloads: [{ text: "hidden final" }],
|
||||
toolMediaUrls: ["/tmp/generated.png"],
|
||||
sourceReplyDeliveryMode: "message_tool_only",
|
||||
}) ?? [];
|
||||
|
||||
expect(reply).toEqual({
|
||||
text: "hidden final",
|
||||
mediaUrls: ["/tmp/generated.png"],
|
||||
mediaUrl: "/tmp/generated.png",
|
||||
});
|
||||
expect(getReplyPayloadMetadata(reply ?? {})).toBeUndefined();
|
||||
});
|
||||
|
||||
it("keeps host-owned media deliverable beside suppressed assistant text", () => {
|
||||
const [textReply, mediaReply] =
|
||||
mergeAttemptToolMediaPayloads({
|
||||
payloads: [{ text: "Done" }],
|
||||
toolMediaUrls: ["/tmp/generated.png"],
|
||||
hostOwnedToolMediaUrls: ["/tmp/generated.png"],
|
||||
sourceReplyDeliveryMode: "message_tool_only",
|
||||
}) ?? [];
|
||||
|
||||
expect(textReply).toEqual({ text: "Done" });
|
||||
expect(getReplyPayloadMetadata(textReply ?? {})).toBeUndefined();
|
||||
expect(mediaReply).toEqual({
|
||||
mediaUrls: ["/tmp/generated.png"],
|
||||
mediaUrl: "/tmp/generated.png",
|
||||
audioAsVoice: undefined,
|
||||
trustedLocalMedia: undefined,
|
||||
});
|
||||
expect(getReplyPayloadMetadata(mediaReply ?? {})).toMatchObject({
|
||||
deliverDespiteSourceReplySuppression: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("merges generic media with assistant text while splitting host-owned media", () => {
|
||||
const [textReply, mediaReply] =
|
||||
mergeAttemptToolMediaPayloads({
|
||||
payloads: [{ text: "Done" }],
|
||||
toolMediaUrls: ["/tmp/reply.opus", "/tmp/generated.png"],
|
||||
hostOwnedToolMediaUrls: ["/tmp/generated.png"],
|
||||
toolAudioAsVoice: true,
|
||||
sourceReplyDeliveryMode: "message_tool_only",
|
||||
}) ?? [];
|
||||
|
||||
expect(textReply).toEqual({
|
||||
text: "Done",
|
||||
mediaUrls: ["/tmp/reply.opus"],
|
||||
mediaUrl: "/tmp/reply.opus",
|
||||
audioAsVoice: true,
|
||||
});
|
||||
expect(getReplyPayloadMetadata(textReply ?? {})).toBeUndefined();
|
||||
expect(mediaReply).toEqual({
|
||||
mediaUrls: ["/tmp/generated.png"],
|
||||
mediaUrl: "/tmp/generated.png",
|
||||
audioAsVoice: undefined,
|
||||
trustedLocalMedia: undefined,
|
||||
});
|
||||
expect(getReplyPayloadMetadata(mediaReply ?? {})).toMatchObject({
|
||||
deliverDespiteSourceReplySuppression: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves reply metadata when attaching tool media to a visible reply", () => {
|
||||
const visibleReply = setReplyPayloadMetadata(
|
||||
{ text: "done" },
|
||||
|
||||
@@ -5,6 +5,7 @@ import type { SourceReplyDeliveryMode } from "../../../auto-reply/get-reply-opti
|
||||
import {
|
||||
copyReplyPayloadMetadata,
|
||||
getReplyPayloadMetadata,
|
||||
markReplyPayloadForSourceSuppressionDelivery,
|
||||
} from "../../../auto-reply/reply-payload.js";
|
||||
import type { EmbeddedAgentRunResult } from "../types.js";
|
||||
|
||||
@@ -19,6 +20,7 @@ type EmbeddedRunPayload = NonNullable<EmbeddedAgentRunResult["payloads"]>[number
|
||||
export function mergeAttemptToolMediaPayloads(params: {
|
||||
payloads?: EmbeddedRunPayload[];
|
||||
toolMediaUrls?: string[];
|
||||
hostOwnedToolMediaUrls?: string[];
|
||||
toolAudioAsVoice?: boolean;
|
||||
toolTrustedLocalMedia?: boolean;
|
||||
sourceReplyDeliveryMode?: SourceReplyDeliveryMode;
|
||||
@@ -27,10 +29,42 @@ export function mergeAttemptToolMediaPayloads(params: {
|
||||
const mediaUrls = Array.from(
|
||||
new Set(params.toolMediaUrls?.map((url) => url.trim()).filter(Boolean) ?? []),
|
||||
);
|
||||
const mediaUrlSet = new Set(mediaUrls);
|
||||
const hostOwnedMediaUrls = Array.from(
|
||||
new Set(
|
||||
params.hostOwnedToolMediaUrls
|
||||
?.map((url) => url.trim())
|
||||
.filter((url) => url.length > 0 && mediaUrlSet.has(url)) ?? [],
|
||||
),
|
||||
);
|
||||
if (mediaUrls.length === 0 && !params.toolAudioAsVoice && !params.toolTrustedLocalMedia) {
|
||||
return params.payloads;
|
||||
}
|
||||
|
||||
const buildMediaPayload = (urls: string[], includeAudio: boolean): EmbeddedRunPayload => ({
|
||||
mediaUrls: urls.length ? urls : undefined,
|
||||
mediaUrl: urls[0],
|
||||
audioAsVoice: (includeAudio && params.toolAudioAsVoice) || undefined,
|
||||
trustedLocalMedia: params.toolTrustedLocalMedia || undefined,
|
||||
});
|
||||
const shouldSplitHostOwnedMedia =
|
||||
params.sourceReplyDeliveryMode === "message_tool_only" && hostOwnedMediaUrls.length > 0;
|
||||
const hostOwnedMediaUrlSet = new Set(hostOwnedMediaUrls);
|
||||
const mergeableMediaUrls = shouldSplitHostOwnedMedia
|
||||
? mediaUrls.filter((url) => !hostOwnedMediaUrlSet.has(url))
|
||||
: mediaUrls;
|
||||
const appendHostOwnedMedia = (nextPayloads: EmbeddedRunPayload[]): EmbeddedRunPayload[] => {
|
||||
if (!shouldSplitHostOwnedMedia) {
|
||||
return nextPayloads;
|
||||
}
|
||||
// Harness-owned artifacts remain separate from assistant text and generic
|
||||
// tool media so only their explicit provenance bypasses source suppression.
|
||||
return [
|
||||
...nextPayloads,
|
||||
markReplyPayloadForSourceSuppressionDelivery(buildMediaPayload(hostOwnedMediaUrls, false)),
|
||||
];
|
||||
};
|
||||
|
||||
const payloads = params.payloads?.length ? [...params.payloads] : [];
|
||||
const payloadIndex = payloads.findIndex((payload) => !payload.isReasoning);
|
||||
if (payloadIndex >= 0) {
|
||||
@@ -45,9 +79,14 @@ export function mergeAttemptToolMediaPayloads(params: {
|
||||
// Message-tool-only source replies are transcript mirrors of a send that
|
||||
// already happened elsewhere; attaching generated media here would create
|
||||
// a duplicate channel delivery.
|
||||
return payloads;
|
||||
return appendHostOwnedMedia(payloads);
|
||||
}
|
||||
const mergedMediaUrls = Array.from(new Set([...(payload.mediaUrls ?? []), ...mediaUrls]));
|
||||
if (mergeableMediaUrls.length === 0 && shouldSplitHostOwnedMedia) {
|
||||
return appendHostOwnedMedia(payloads);
|
||||
}
|
||||
const mergedMediaUrls = Array.from(
|
||||
new Set([...(payload.mediaUrls ?? []), ...mergeableMediaUrls]),
|
||||
);
|
||||
payloads[payloadIndex] = copyReplyPayloadMetadata(payload, {
|
||||
...payload,
|
||||
mediaUrls: mergedMediaUrls.length ? mergedMediaUrls : undefined,
|
||||
@@ -55,17 +94,17 @@ export function mergeAttemptToolMediaPayloads(params: {
|
||||
audioAsVoice: payload.audioAsVoice || params.toolAudioAsVoice || undefined,
|
||||
trustedLocalMedia: payload.trustedLocalMedia || params.toolTrustedLocalMedia || undefined,
|
||||
});
|
||||
return payloads;
|
||||
return appendHostOwnedMedia(payloads);
|
||||
}
|
||||
|
||||
if (shouldSplitHostOwnedMedia) {
|
||||
const genericMediaPayload =
|
||||
mergeableMediaUrls.length > 0 ? [buildMediaPayload(mergeableMediaUrls, true)] : [];
|
||||
return appendHostOwnedMedia([...payloads, ...genericMediaPayload]);
|
||||
}
|
||||
|
||||
const mediaPayload = buildMediaPayload(mergeableMediaUrls, true);
|
||||
|
||||
// Reasoning-only turns still need a concrete media payload so channel delivery sees the attachment.
|
||||
return [
|
||||
...payloads,
|
||||
{
|
||||
mediaUrls: mediaUrls.length ? mediaUrls : undefined,
|
||||
mediaUrl: mediaUrls[0],
|
||||
audioAsVoice: params.toolAudioAsVoice || undefined,
|
||||
trustedLocalMedia: params.toolTrustedLocalMedia || undefined,
|
||||
},
|
||||
];
|
||||
return [...payloads, mediaPayload];
|
||||
}
|
||||
|
||||
@@ -284,6 +284,11 @@ export type EmbeddedRunAttemptResult = {
|
||||
messagingToolSourceReplyPayloads?: MessagingToolSourceReplyPayload[];
|
||||
heartbeatToolResponse?: HeartbeatToolResponse;
|
||||
toolMediaUrls?: string[];
|
||||
/**
|
||||
* Native artifacts produced and owned by the harness, never model-selected
|
||||
* dynamic-tool output. Core validates this as a subset of toolMediaUrls.
|
||||
*/
|
||||
hostOwnedToolMediaUrls?: string[];
|
||||
toolAudioAsVoice?: boolean;
|
||||
toolTrustedLocalMedia?: boolean;
|
||||
hasToolMediaBlockReply?: boolean;
|
||||
|
||||
@@ -231,9 +231,9 @@ export type ReplyPayloadMetadata = {
|
||||
accountId?: string;
|
||||
};
|
||||
/**
|
||||
* Internal OpenClaw notices generated after a runtime/provider failure are
|
||||
* not assistant source replies. Dispatch may deliver them even when normal
|
||||
* assistant source replies are message-tool-only; sendPolicy deny still wins.
|
||||
* Internal OpenClaw notices and host-owned artifacts are not assistant source
|
||||
* replies. Dispatch may deliver them even when normal assistant source replies
|
||||
* are message-tool-only; sendPolicy deny still wins.
|
||||
*/
|
||||
deliverDespiteSourceReplySuppression?: boolean;
|
||||
/**
|
||||
@@ -286,7 +286,7 @@ export function copyReplyPayloadMetadata<T extends object>(source: object, paylo
|
||||
return metadata ? setReplyPayloadMetadata(payload, metadata) : payload;
|
||||
}
|
||||
|
||||
/** Marks a notice payload as deliverable even when normal source replies are suppressed. */
|
||||
/** Marks a host-owned payload as deliverable even when normal source replies are suppressed. */
|
||||
export function markReplyPayloadForSourceSuppressionDelivery<T extends object>(payload: T): T {
|
||||
return setReplyPayloadMetadata(payload, {
|
||||
deliverDespiteSourceReplySuppression: true,
|
||||
|
||||
@@ -6140,6 +6140,32 @@ describe("createFollowupRunner messaging delivery and dedupe", () => {
|
||||
expect(routeReplyMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("does not route marked host media for message-tool-only queued room events", async () => {
|
||||
const queued = baseQueuedRun("discord");
|
||||
await runMessagingCase({
|
||||
agentResult: {
|
||||
payloads: [
|
||||
setReplyPayloadMetadataForTest(
|
||||
{ mediaUrl: "/tmp/generated.png" },
|
||||
{ deliverDespiteSourceReplySuppression: true },
|
||||
),
|
||||
],
|
||||
},
|
||||
queued: {
|
||||
...queued,
|
||||
currentInboundEventKind: "room_event",
|
||||
originatingChannel: "discord",
|
||||
originatingTo: "channel:C1",
|
||||
run: {
|
||||
...queued.run,
|
||||
sourceReplyDeliveryMode: "message_tool_only",
|
||||
},
|
||||
} as FollowupRun,
|
||||
});
|
||||
|
||||
expect(routeReplyMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("does not enqueue stranded recovery when queued followup send policy denies delivery", async () => {
|
||||
const finalText =
|
||||
"Here is a long reply for a denied session. It includes enough detail to be substantive, but send-policy denial must remain an intentional delivery block.";
|
||||
@@ -6170,6 +6196,37 @@ describe("createFollowupRunner messaging delivery and dedupe", () => {
|
||||
expect(routeReplyMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("does not route marked host media when queued followup send policy denies delivery", async () => {
|
||||
const queued = baseQueuedRun("discord");
|
||||
const sessionEntry: SessionEntry = {
|
||||
sessionId: "session",
|
||||
updatedAt: Date.now(),
|
||||
sendPolicy: "deny",
|
||||
};
|
||||
await runMessagingCase({
|
||||
agentResult: {
|
||||
payloads: [
|
||||
setReplyPayloadMetadataForTest(
|
||||
{ mediaUrl: "/tmp/generated.png" },
|
||||
{ deliverDespiteSourceReplySuppression: true },
|
||||
),
|
||||
],
|
||||
},
|
||||
queued: {
|
||||
...queued,
|
||||
originatingChannel: "discord",
|
||||
originatingTo: "channel:C1",
|
||||
run: {
|
||||
...queued.run,
|
||||
sourceReplyDeliveryMode: "message_tool_only",
|
||||
},
|
||||
} as FollowupRun,
|
||||
runnerOverrides: { sessionEntry, sessionKey: "main" },
|
||||
});
|
||||
|
||||
expect(routeReplyMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("routes sanitized diagnostics when message-tool-only stranded retry strands again", async () => {
|
||||
const queued = baseQueuedRun("discord");
|
||||
const { onBlockReply } = await runMessagingCase({
|
||||
|
||||
@@ -2009,7 +2009,12 @@ export function createFollowupRunner(params: {
|
||||
getReplyPayloadMetadata(payload)?.deliverDespiteSourceReplySuppression === true,
|
||||
);
|
||||
if (suppressionDeliverablePayloads.length > 0) {
|
||||
await sendFollowupPayloads(
|
||||
// Marked runtime output bypasses source-reply suppression, not the
|
||||
// admission-time send policy or ambient room-event silence.
|
||||
if (isRoomEventFollowup()) {
|
||||
return;
|
||||
}
|
||||
await sendRunPayloads(
|
||||
suppressionDeliverablePayloads,
|
||||
effectiveQueued,
|
||||
{
|
||||
|
||||
@@ -274,7 +274,7 @@ export const AUTOMATION_FIELD_HELP: Record<string, string> = {
|
||||
"messages.messagePrefix":
|
||||
"Prefix text prepended to inbound user messages before they are handed to the agent runtime. Use this sparingly for channel context markers and keep it stable across sessions.",
|
||||
"messages.visibleReplies":
|
||||
'Controls visible source replies across direct, group, and channel conversations. "message_tool" requires message(action=send) for visible output and keeps normal final text private. "automatic" posts normal replies as before.',
|
||||
'Controls model-authored source replies across direct, group, and channel conversations. "message_tool" requires message(action=send) for normal assistant output and generic tool media; explicitly host-owned runtime output remains deliverable except for ambient room events. "automatic" posts normal replies as before.',
|
||||
"messages.responsePrefix":
|
||||
"Prefix text prepended to outbound assistant replies before sending to channels. Use for lightweight branding/context tags and avoid long prefixes that reduce content density.",
|
||||
"messages.usageTemplate":
|
||||
@@ -290,7 +290,7 @@ export const AUTOMATION_FIELD_HELP: Record<string, string> = {
|
||||
"messages.groupChat.unmentionedInbound":
|
||||
'Controls how unmentioned always-on group chatter is submitted. "user_request" treats it as a user request; "room_event" submits it as quiet context where visible output requires the message tool.',
|
||||
"messages.groupChat.visibleReplies":
|
||||
'Overrides visible source replies for group/channel conversations. Defaults to "automatic" when no global visible reply policy is set. "message_tool" requires message(action=send) for room output and keeps normal final text private. "automatic" posts normal replies as before.',
|
||||
'Overrides model-authored source replies for group/channel conversations. Defaults to "automatic" when no global visible reply policy is set. "message_tool" requires message(action=send) for normal assistant output and generic tool media; explicitly host-owned runtime output remains deliverable except for ambient room events. "automatic" posts normal replies as before.',
|
||||
"messages.queue":
|
||||
"Queue strategy for inbound messages that arrive while a session run is active. Use this to tune steering, deferred followups, batching, or interruption.",
|
||||
"messages.queue.mode":
|
||||
|
||||
@@ -19,9 +19,10 @@ export type GroupChatConfig = {
|
||||
*/
|
||||
unmentionedInbound?: "user_request" | "room_event";
|
||||
/**
|
||||
* Controls how group/channel inbound events produce visible room replies. The
|
||||
* message-tool mode requires explicit message sends for visible room output;
|
||||
* final text stays private when the model misses the tool.
|
||||
* Controls how group/channel inbound events produce model-authored room replies.
|
||||
* The message-tool mode requires explicit message sends for normal assistant
|
||||
* output; explicitly host-owned runtime output remains deliverable except for
|
||||
* ambient room events.
|
||||
* Default: "automatic".
|
||||
*/
|
||||
visibleReplies?: "automatic" | "message_tool";
|
||||
@@ -107,8 +108,9 @@ export type MessagesConfig = {
|
||||
* group, and channel conversations. Group/channel events still default to
|
||||
* `groupChat.visibleReplies` when it is set.
|
||||
*
|
||||
* Default: "automatic". In group/channel rooms, "message_tool" keeps final
|
||||
* text private unless the model sends visibly through the message tool.
|
||||
* Default: "automatic". In group/channel rooms, "message_tool" keeps normal
|
||||
* assistant output private unless the model sends visibly through the message
|
||||
* tool; explicitly host-owned runtime output remains deliverable.
|
||||
*/
|
||||
visibleReplies?: "automatic" | "message_tool";
|
||||
/**
|
||||
|
||||
Reference in New Issue
Block a user