Commit Graph

38078 Commits

Author SHA1 Message Date
Omar Shahine
b97af08703 fix(file-transfer): address PR review feedback (security + availability)
Reviewer findings addressed (greptile + aisle):

- policy: persistAllowAlways no longer escalates per-node approvals to the
  '*' wildcard entry; allow-always now writes under the specific node's
  own entry, never the wildcard (greptile P1 SECURITY).
- policy: add literal '..' segment short-circuit in evaluateFilePolicy,
  raised before glob match. Stops "/allowed/../etc/passwd" from passing
  preflight against "/allowed/**" globs (aisle MEDIUM CWE-22).
- file-write: replace no-op base64 try/catch with actual round-trip
  validation. Buffer.from(s, "base64") never throws — invalid input
  silently decoded to garbage bytes. Now re-encodes and compares
  modulo padding/url-variant chars (greptile P1 SECURITY).
- file-write: document the parent-symlink residual risk and rely on the
  existing gateway-side post-flight policy check; full rollback requires
  a node-side file.unlink which is deferred to a follow-up. Initial
  segment-walk attempt was reverted because it false-positives on system
  symlinks like macOS /var → /private/var (aisle HIGH CWE-59).
- dir-fetch tool: add preValidateTarball pass that runs `tar -tzvf` and
  rejects symlinks, hardlinks, absolute paths, '..' traversal,
  uncompressed sizes >64MB, and entry counts >5000 — before any
  extraction. Drops --no-overwrite-dir (GNU-only flag rejected by BSD
  tar on macOS) (aisle HIGH x2 CWE-22 + CWE-409, greptile P2).
- dir-fetch tool: stream-hash files via fs.open + read loop instead of
  fs.readFile to avoid full-buffer reads on large extracted entries.
- dir-fetch handler: replace spawnSync in countTarEntries with async
  spawn + bounded buffer so tar -tzf can't park the node-host event
  loop for up to 10s on a slow filesystem (greptile P1 AVAIL).
- audit: clear auditDirPromise on rejection so a transient mkdir
  failure doesn't permanently silence the audit log (greptile P2).

New tests: wildcard escalation rejection, base64 malformed/url-variant,
'..' traversal short-circuit (3 cases). 84/84 passing.
2026-04-29 07:49:47 +00:00
Omar Shahine
db6b4b41e1 test(file-transfer): add unit tests for handlers, policy, and shared utilities
Adds 77 tests covering:
- handleFileFetch: validation, fs errors, sha256, size cap, symlink canonicalization
- handleFileWrite: validation, atomic write, overwrite policy, parent dir handling, symlink refusal, integrity check, size cap
- handleDirList: validation, fs errors, sorted listing, dotfile inclusion, pagination
- handleDirFetch: validation, fs errors, gzipped tar with sha256, mid-stream byte cap
- evaluateFilePolicy: default-deny, denyPaths-wins, allow matching, ask modes (off/on-miss/always), node-id/displayName/'*' resolution
- persistAllowAlways: append, dedupe, create-on-missing
- shared/mime: extension lookup, image/text inline sets
- shared/errors: err helper, classifyFsError, throwFromNodePayload

Also fixes accumulated lint regressions in the prod source flagged once these
files moved into the changed-gate scope (parseInt -> Number.parseInt, redundant
type casts removed, single-statement if bodies wrapped in braces).
2026-04-29 07:49:47 +00:00
Omar Shahine
844f3d62e9 feat(file-transfer): add bundled plugin for binary file ops on nodes
New extensions/file-transfer/ plugin exposing four agent tools
(file_fetch, dir_list, dir_fetch, file_write) and four matching
node-host commands (file.fetch, dir.list, dir.fetch, file.write).
Lets agents read and write files on paired nodes by absolute path,
bypassing the bash output cap (200KB) and the live tool-result
text cap that would otherwise truncate base64 payloads.

Public surface
--------------
- file_fetch({ node, path, maxBytes? })
  Image MIMEs return image content blocks; small text (<=8 KB) inlines
  as text content; everything else returns a saved-media-path text
  block. sha256-verified end-to-end.
- dir_list({ node, path, pageToken?, maxEntries? })
  Structured directory listing — name, path, size, mimeType, isDir,
  mtime. Paginated. No content transfer.
- dir_fetch({ node, path, maxBytes?, includeDotfiles? })
  Server-side tar -czf streamed back, unpacked into the gateway media
  store, returns a manifest of saved paths. Single round-trip.
  60s wall-clock timeouts on tar create/unpack. tar -xzf without -P
  rejects absolute paths in archive entries.
- file_write({ node, path, contentBase64, mimeType?, overwrite?,
              createParents? })
  Atomic write (temp + rename). Refuses to overwrite by default.
  Refuses to write through symlinks (lstat check). Buffer-side
  sha256 (no read-back race). Pair with file_fetch to round-trip
  files between nodes — DO NOT use exec/cp for file copies.

All four commands gated by:
  - dangerous-by-default node command policy
    (gateway.nodes.allowCommands opt-in)
  - per-node path policy (gateway.nodes.fileTransfer)
  - optional operator approval prompt (ask: off | on-miss | always)

16 MB raw byte ceiling per single-frame round-trip (25 MB WS frame
with ~33% base64 overhead and JSON envelope). 8 MB defaults.

Path policy and approvals
-------------------------
Default behavior is DENY. The operator must explicitly opt in:

  {
    "gateway": {
      "nodes": {
        "fileTransfer": {
          "<nodeId-or-displayName>": {
            "ask":              "off" | "on-miss" | "always",
            "allowReadPaths":   ["~/Screenshots/**", "/tmp/**"],
            "allowWritePaths":  ["~/Downloads/**"],
            "denyPaths":        ["**/.ssh/**", "**/.aws/**"],
            "maxBytes":         16777216
          },
          "*": { "ask": "on-miss" }
        }
      }
    }
  }

ask modes:
  off       — silent: allow if matched, deny if not (default)
  on-miss   — silent allow if matched; prompt on miss
  always    — prompt every call (denyPaths still hard-deny)

denyPaths always wins. allow-always from the prompt persists the
exact path back into allowReadPaths/allowWritePaths via
mutateConfigFile so subsequent matching calls go silent.

Reuses existing primitives — no new gateway methods:
  plugin.approval.request / plugin.approval.waitDecision
  decision: allow-once | allow-always | deny

Pre-flight against requested path AND post-flight against the
canonicalPath returned by the node — closes symlink-escape attacks
where the requested path matched policy but realpath resolves
somewhere else.

Audit log
---------
JSONL at ~/.openclaw/audit/file-transfer.jsonl. Records every
decision (allow/allowed-once/allowed-always/denied/error) with
timestamp, op, nodeId, displayName, requestedPath, canonicalPath,
decision, error code, sizeBytes, sha256, durationMs. Best-effort
writes; never propagates failure.

Plugin layout
-------------
extensions/file-transfer/
  index.ts                       definePluginEntry, nodeHostCommands
  openclaw.plugin.json           contracts.tools registration
  package.json
  src/node-host/{file-fetch,dir-list,dir-fetch,file-write}.ts
  src/tools/{file-fetch,dir-list,dir-fetch,file-write}-tool.ts
  src/shared/
    mime.ts        single-source extension->MIME map + image/text sets
    errors.ts      shared error code enum and helpers
    params.ts      shared param-validation helpers + GatewayCallOptions
    policy.ts      evaluateFilePolicy, persistAllowAlways
    approval.ts    plugin.approval.request wrapper
    gatekeep.ts    one-stop policy + approval + audit orchestrator
    audit.ts       JSONL audit sink

Core touch points
-----------------
- src/infra/node-commands.ts: NODE_FILE_FETCH_COMMAND,
  NODE_DIR_LIST_COMMAND, NODE_DIR_FETCH_COMMAND,
  NODE_FILE_WRITE_COMMAND, NODE_FILE_COMMANDS array
- src/gateway/node-command-policy.ts: all four added to
  DEFAULT_DANGEROUS_NODE_COMMANDS
- src/security/audit-extra.sync.ts: audit detail mentions file ops
- src/agents/tools/nodes-tool-media.ts: MEDIA_INVOKE_ACTIONS entry
  for file.fetch redirects raw nodes(action=invoke) callers to the
  dedicated file_fetch tool to prevent base64 context bloat
- src/agents/tools/nodes-tool.ts: nodes tool description points to
  the dedicated file_fetch tool

Known limitations / follow-ups
------------------------------
- No tests in this PR. For a security-sensitive surface this is a
  gap; will follow up with a test pass.
- Direct CLI invocation (openclaw nodes invoke --command file.fetch)
  bypasses the plugin policy entirely. Plugin-side gating is the
  realistic threat model (agent on iMessage requesting paths it
  shouldn't), but for true defense-in-depth, policy belongs in the
  gateway-side node.invoke dispatch. Move-policy-to-core is a
  separate PR.
- file_watch (long-lived filesystem event subscription) is not
  included; it needs a new node-protocol primitive for streaming
  event channels and was descoped from this PR.
- dir_fetch includeDotfiles: true is the only supported mode;
  BSD tar exclude patterns reliably collapse dotfile filtering
  to an empty archive. Reliable filtering needs a
  `find ! -name ".*" | tar -T -` pipeline; deferred.
- dir_fetch du -sk preflight is a heuristic (du * 4 vs maxBytes);
  the mid-stream byte cap is the actual safety net.
2026-04-29 07:49:47 +00:00
HDYA
5cc834a11a docs(msteams): fix federated auth added-in date
Correct the documented added-in date for MS Teams federated authentication.\n\nThanks @HDYA.
2026-04-29 02:47:56 -05:00
Peter Steinberger
ca972f692f fix: keep browser fetch helper under test support 2026-04-29 08:47:43 +01:00
Vincent Koc
a62c7e5a27 ci: react to autoclose commands 2026-04-29 00:46:00 -07:00
Peter Steinberger
8ac2dd4cd2 refactor: simplify docker e2e harness scripts 2026-04-29 08:45:42 +01:00
Peter Steinberger
2b811fe6d9 fix(memory): make qmd gateway startup lazy 2026-04-29 08:45:19 +01:00
Peter Steinberger
e52b660749 fix(browser): repair test fetch helper export 2026-04-29 08:45:05 +01:00
Peter Steinberger
2a02b3bcec test: harden plugin prerelease smoke checks 2026-04-29 08:41:30 +01:00
Vincent Koc
1d0e9a907e fix(doctor): migrate legacy tts enabled toggles 2026-04-29 00:39:18 -07:00
Vincent Koc
eb7f305737 Merge branch 'main' of https://github.com/openclaw/openclaw
* 'main' of https://github.com/openclaw/openclaw:
  fix: exclude test support from raw fetch guard
  fix(ollama): preserve aborts with stream timeouts
  ci: require maintainer permission for command reactions
  docs(hooks/bundled/readme): cover session compaction and message events
  refactor: share docker e2e harness runner
  fix: keep browser test fetch out of runtime scan
2026-04-29 00:36:24 -07:00
Peter Steinberger
f8faf40a9e fix: exclude test support from raw fetch guard 2026-04-29 08:36:00 +01:00
Peter Steinberger
a31342ab6b fix(ollama): preserve aborts with stream timeouts 2026-04-29 08:33:23 +01:00
Vincent Koc
275b0f00b0 ci: require maintainer permission for command reactions 2026-04-29 00:30:32 -07:00
Vincent Koc
b69b508d20 docs(hooks/bundled/readme): cover session compaction and message events
The bundled hooks README listed only command/agent/gateway events and
ended with a stale "More event types coming soon (session lifecycle,
agent errors, etc.)" line, but production code now triggers:

- session:compact:before / session:compact:after via
  src/agents/pi-embedded-runner/compaction-hooks.ts
- message:received via src/auto-reply/reply/dispatch-from-config.ts
- message:sent via src/infra/outbound/deliver.ts

Updates the "Event Types" list with the four real production event
names, drops the stale coming-soon line, and aligns the InternalHookEvent
interface example with the actual InternalHookEventType union (adds
"message" and refreshes the action examples). HOOK.md authors that target
session lifecycle or message routing now have a real surface to subscribe
to instead of relying on tribal knowledge or the type definitions.
2026-04-29 00:29:29 -07:00
Peter Steinberger
34bd962a20 refactor: share docker e2e harness runner 2026-04-29 08:28:54 +01:00
Vincent Koc
c01244e859 test(browser): keep undici fetch helper test-only 2026-04-29 00:28:02 -07:00
Peter Steinberger
f6a2cf15c0 fix: keep browser test fetch out of runtime scan 2026-04-29 08:27:57 +01:00
konanok
bd5afadc5c fix(ui): use precise hourly message counts for Peak Error Hours (#49396)
Merged via squash.

Prepared head SHA: fbbf43b84a
Co-authored-by: konanok <30515586+konanok@users.noreply.github.com>
Co-authored-by: hxy91819 <8814856+hxy91819@users.noreply.github.com>
Reviewed-by: @hxy91819
2026-04-29 15:22:42 +08:00
Peter Steinberger
a0fd105e5e ci: split plugin prerelease validation 2026-04-29 08:21:12 +01:00
Peter Steinberger
9b1967e5ef perf(push): lazy load web push runtime 2026-04-29 08:20:56 +01:00
Vincent Koc
1dd500c495 test: stabilize mcp docker smokes 2026-04-29 00:19:52 -07:00
Vincent Koc
6a3310bbda chore(ci): add memory CodeQL quality shard
Adds a narrow CodeQL Critical Quality shard for the memory host/runtime boundary.
2026-04-29 00:18:30 -07:00
Peter Steinberger
26546dfbcb test: harden Docker release smoke probes 2026-04-29 08:16:17 +01:00
Peter Steinberger
7662a17b08 test: trim release smoke memory startup 2026-04-29 08:16:17 +01:00
Peter Steinberger
9ddd10b84c test: tighten MCP channel smoke route contract 2026-04-29 08:16:17 +01:00
Peter Steinberger
afc4f06ca3 fix(memory): isolate qmd boot refresh 2026-04-29 08:14:08 +01:00
Peter Steinberger
7e5d6dba80 build(deps): trim runtime dependency graph 2026-04-29 08:11:57 +01:00
Peter Steinberger
023d3371a5 refactor(gateway): classify gateway transport failures
# Conflicts:
#	CHANGELOG.md
2026-04-29 08:10:15 +01:00
Peter Steinberger
e25b542100 fix(cli): fall back to file logs when local logs rpc closes 2026-04-29 08:10:15 +01:00
Vincent Koc
6306e2fdcb ci: react to maintainer PR commands 2026-04-29 00:09:49 -07:00
Vincent Koc
13390fcac8 docs(apps/ios/readme): list authenticated background presence beacons
For bdba90a20b: apps/ios/README.md "What Works Now (Concrete)" section
omitted the authenticated background `node.presence.alive` beacon
feature that shipped on iOS first, even though apps/android/README.md
already lists it on the rebuild checklist. Adds a matching bullet so
the iOS README reflects the gateway last-seen metadata update path
across foreground/background transitions.
2026-04-29 00:09:25 -07:00
Vincent Koc
81f490f26a docs(changelog): note deprecated alias metadata 2026-04-29 00:07:12 -07:00
Vincent Koc
5fa0d282a8 fix(mcp): stringify plugin tool content safely 2026-04-29 00:04:18 -07:00
Vincent Koc
ca427df924 test(scripts): guard deprecated alias jsdoc 2026-04-29 00:04:17 -07:00
Vincent Koc
8b71d2347f docs(types): mark remaining deprecated aliases 2026-04-29 00:04:17 -07:00
Peter Steinberger
64387ad8e2 refactor: simplify docker e2e harness scripts 2026-04-29 08:03:15 +01:00
peter
e71d7d48fb fix(telegram): probe video dimensions through sdk
Fix Telegram portrait video distortion by probing video dimensions through the shared media helper and passing width/height to sendVideo.

Validation:
- Targeted Telegram/media tests passed locally.
- Plugin SDK API baseline check passed locally.
- Formatter and git diff whitespace checks passed locally.

CI note: current boundary drift observed on prior run came from existing src/plugin-sdk/discord.ts and src/plugin-sdk/telegram-account.ts, not this PR diff.
2026-04-29 01:58:25 -05:00
tmimmanuel
0bbbc99980 fix(ui): preserve queued chat messages across session switches (#73679)
Fixes #73621.

Preserve queued Control UI chat messages across in-UI session switches by saving the active queue per session before reset and restoring it when switching back. Route the overview session selector through the shared switchChatSession helper so it follows the same queue lifecycle.

Validation:
- OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test ui/src/ui/app-render.helpers.node.test.ts
- pnpm tsgo:test:ui
- pnpm exec oxfmt --check --threads=1 ui/src/ui/app-render.helpers.node.test.ts ui/src/ui/app-render.helpers.ts ui/src/ui/app-render.ts ui/src/ui/app-view-state.ts ui/src/ui/app.ts
2026-04-29 01:57:04 -05:00
brokemac79
20c7a98fb8 fix(plugins): keep provider discovery metadata-only
Fix startup and per-turn provider registry hot paths by keeping primary-model startup discovery on metadata-only provider entries and by keeping capability provider fallback loads scoped to manifest-derived owners, including explicit empty scopes when no bundled owner exists.

Evidence:
- Reproduces the reported code paths from #73729, #73835, and #73793: startup prewarm was able to enter provider/model discovery that loaded plugin runtime, and capability lookups could bypass active registry reuse or broaden fallback registry loads.
- Fix threads providerDiscoveryEntriesOnly through models-config planning into plugin discovery.
- Fix reuses active non-memory/non-speech capability providers even with explicit plugins.entries.
- Fix keeps fallback registry loads scoped with onlyPluginIds, including [] for no-owner media capability checks.
- Local targeted tests passed for gateway startup, models config, provider discovery, capability providers, and web provider runtimes.
- Testbox pnpm check:changed passed.
- Testbox pnpm build passed.
- GitHub CI required checks passed on e5e6fe1d52.

Fixes #73729.
Fixes #73835.
Fixes #73793.
Supersedes #73794.
2026-04-29 07:52:32 +01:00
Peter Steinberger
13757465ba fix(agents): scope external CLI auth discovery 2026-04-29 07:52:13 +01:00
Peter Steinberger
3367cfaa14 test: skip bot-to-bot telegram mention in default qa 2026-04-29 07:40:15 +01:00
Peter Steinberger
885d88c1ac refactor(test): simplify bundled channel Docker scenarios 2026-04-29 07:38:38 +01:00
Vincent Koc
99f0ea92fe ci: update qqbot raw fetch allowlist 2026-04-28 23:37:28 -07:00
Peter Steinberger
6a4c866b6a ci: speed up broad validation setup 2026-04-29 07:36:55 +01:00
Vincent Koc
1d87d757e9 ci: add mcp process codeql quality shard 2026-04-28 23:36:34 -07:00
Vincent Koc
1b25dcf57a docs(types): mark legacy hook surfaces deprecated 2026-04-28 23:31:32 -07:00
Peter Steinberger
71473e7448 test: make telegram live mention scenario privacy-safe 2026-04-29 07:27:14 +01:00
Peter Steinberger
32c2337095 test(ci): tolerate slow live provider cleanup 2026-04-29 07:24:47 +01:00