mirror of
https://github.com/openclaw/openclaw.git
synced 2026-04-12 09:41:11 +00:00
docs: remove docs/refactor/ directory
Delete all 7 refactor design docs and the zh-CN translations. Remove the zh-CN nav group from docs.json. These were orphaned from English nav and accessible only by direct URL. Internal design docs do not belong on the public docs site. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Vincent Koc
@@ -1,417 +0,0 @@
|
||||
---
|
||||
summary: "Clawnet refactor: unify network protocol, roles, auth, approvals, identity"
|
||||
read_when:
|
||||
- Planning a unified network protocol for nodes + operator clients
|
||||
- Reworking approvals, pairing, TLS, and presence across devices
|
||||
title: "Clawnet Refactor"
|
||||
---
|
||||
|
||||
# Clawnet refactor (protocol + auth unification)
|
||||
|
||||
## Hi
|
||||
|
||||
Hi Peter — great direction; this unlocks simpler UX + stronger security.
|
||||
|
||||
## Purpose
|
||||
|
||||
Single, rigorous document for:
|
||||
|
||||
- Current state: protocols, flows, trust boundaries.
|
||||
- Pain points: approvals, multi‑hop routing, UI duplication.
|
||||
- Proposed new state: one protocol, scoped roles, unified auth/pairing, TLS pinning.
|
||||
- Identity model: stable IDs + cute slugs.
|
||||
- Migration plan, risks, open questions.
|
||||
|
||||
## Goals (from discussion)
|
||||
|
||||
- One protocol for all clients (mac app, CLI, iOS, Android, headless node).
|
||||
- Every network participant authenticated + paired.
|
||||
- Role clarity: nodes vs operators.
|
||||
- Central approvals routed to where the user is.
|
||||
- TLS encryption + optional pinning for all remote traffic.
|
||||
- Minimal code duplication.
|
||||
- Single machine should appear once (no UI/node duplicate entry).
|
||||
|
||||
## Non‑goals (explicit)
|
||||
|
||||
- Remove capability separation (still need least‑privilege).
|
||||
- Expose full gateway control plane without scope checks.
|
||||
- Make auth depend on human labels (slugs remain non‑security).
|
||||
|
||||
---
|
||||
|
||||
# Current state (as‑is)
|
||||
|
||||
## Two protocols
|
||||
|
||||
### 1) Gateway WebSocket (control plane)
|
||||
|
||||
- Full API surface: config, channels, models, sessions, agent runs, logs, nodes, etc.
|
||||
- Default bind: loopback. Remote access via SSH/Tailscale.
|
||||
- Auth: token/password via `connect`.
|
||||
- No TLS pinning (relies on loopback/tunnel).
|
||||
- Code:
|
||||
- `src/gateway/server/ws-connection/message-handler.ts`
|
||||
- `src/gateway/client.ts`
|
||||
- `docs/gateway/protocol.md`
|
||||
|
||||
### 2) Bridge (node transport)
|
||||
|
||||
- Narrow allowlist surface, node identity + pairing.
|
||||
- JSONL over TCP; optional TLS + cert fingerprint pinning.
|
||||
- TLS advertises fingerprint in discovery TXT.
|
||||
- Code:
|
||||
- `src/infra/bridge/server/connection.ts`
|
||||
- `src/gateway/server-bridge.ts`
|
||||
- `src/node-host/bridge-client.ts`
|
||||
- `docs/gateway/bridge-protocol.md`
|
||||
|
||||
## Control plane clients today
|
||||
|
||||
- CLI → Gateway WS via `callGateway` (`src/gateway/call.ts`).
|
||||
- macOS app UI → Gateway WS (`GatewayConnection`).
|
||||
- Web Control UI → Gateway WS.
|
||||
- ACP → Gateway WS.
|
||||
- Browser control uses its own HTTP control server.
|
||||
|
||||
## Nodes today
|
||||
|
||||
- macOS app in node mode connects to Gateway bridge (`MacNodeBridgeSession`).
|
||||
- iOS/Android apps connect to Gateway bridge.
|
||||
- Pairing + per‑node token stored on gateway.
|
||||
|
||||
## Current approval flow (exec)
|
||||
|
||||
- Agent uses `system.run` via Gateway.
|
||||
- Gateway invokes node over bridge.
|
||||
- Node runtime decides approval.
|
||||
- UI prompt shown by mac app (when node == mac app).
|
||||
- Node returns `invoke-res` to Gateway.
|
||||
- Multi‑hop, UI tied to node host.
|
||||
|
||||
## Presence + identity today
|
||||
|
||||
- Gateway presence entries from WS clients.
|
||||
- Node presence entries from bridge.
|
||||
- mac app can show two entries for same machine (UI + node).
|
||||
- Node identity stored in pairing store; UI identity separate.
|
||||
|
||||
---
|
||||
|
||||
# Problems / pain points
|
||||
|
||||
- Two protocol stacks to maintain (WS + Bridge).
|
||||
- Approvals on remote nodes: prompt appears on node host, not where user is.
|
||||
- TLS pinning only exists for bridge; WS depends on SSH/Tailscale.
|
||||
- Identity duplication: same machine shows as multiple instances.
|
||||
- Ambiguous roles: UI + node + CLI capabilities not clearly separated.
|
||||
|
||||
---
|
||||
|
||||
# Proposed new state (Clawnet)
|
||||
|
||||
## One protocol, two roles
|
||||
|
||||
Single WS protocol with role + scope.
|
||||
|
||||
- **Role: node** (capability host)
|
||||
- **Role: operator** (control plane)
|
||||
- Optional **scope** for operator:
|
||||
- `operator.read` (status + viewing)
|
||||
- `operator.write` (agent run, sends)
|
||||
- `operator.admin` (config, channels, models)
|
||||
|
||||
### Role behaviors
|
||||
|
||||
**Node**
|
||||
|
||||
- Can register capabilities (`caps`, `commands`, permissions).
|
||||
- Can receive `invoke` commands (`system.run`, `camera.*`, `canvas.*`, `screen.record`, etc).
|
||||
- Can send events: `voice.transcript`, `agent.request`, `chat.subscribe`.
|
||||
- Cannot call config/models/channels/sessions/agent control plane APIs.
|
||||
|
||||
**Operator**
|
||||
|
||||
- Full control plane API, gated by scope.
|
||||
- Receives all approvals.
|
||||
- Does not directly execute OS actions; routes to nodes.
|
||||
|
||||
### Key rule
|
||||
|
||||
Role is per‑connection, not per device. A device may open both roles, separately.
|
||||
|
||||
---
|
||||
|
||||
# Unified authentication + pairing
|
||||
|
||||
## Client identity
|
||||
|
||||
Every client provides:
|
||||
|
||||
- `deviceId` (stable, derived from device key).
|
||||
- `displayName` (human name).
|
||||
- `role` + `scope` + `caps` + `commands`.
|
||||
|
||||
## Pairing flow (unified)
|
||||
|
||||
- Client connects unauthenticated.
|
||||
- Gateway creates a **pairing request** for that `deviceId`.
|
||||
- Operator receives prompt; approves/denies.
|
||||
- Gateway issues credentials bound to:
|
||||
- device public key
|
||||
- role(s)
|
||||
- scope(s)
|
||||
- capabilities/commands
|
||||
- Client persists token, reconnects authenticated.
|
||||
|
||||
## Device‑bound auth (avoid bearer token replay)
|
||||
|
||||
Preferred: device keypairs.
|
||||
|
||||
- Device generates keypair once.
|
||||
- `deviceId = fingerprint(publicKey)`.
|
||||
- Gateway sends nonce; device signs; gateway verifies.
|
||||
- Tokens are issued to a public key (proof‑of‑possession), not a string.
|
||||
|
||||
Alternatives:
|
||||
|
||||
- mTLS (client certs): strongest, more ops complexity.
|
||||
- Short‑lived bearer tokens only as a temporary phase (rotate + revoke early).
|
||||
|
||||
## Silent approval (SSH heuristic)
|
||||
|
||||
Define it precisely to avoid a weak link. Prefer one:
|
||||
|
||||
- **Local‑only**: auto‑pair when client connects via loopback/Unix socket.
|
||||
- **Challenge via SSH**: gateway issues nonce; client proves SSH by fetching it.
|
||||
- **Physical presence window**: after a local approval on gateway host UI, allow auto‑pair for a short window (e.g. 10 minutes).
|
||||
|
||||
Always log + record auto‑approvals.
|
||||
|
||||
---
|
||||
|
||||
# TLS everywhere (dev + prod)
|
||||
|
||||
## Reuse existing bridge TLS
|
||||
|
||||
Use current TLS runtime + fingerprint pinning:
|
||||
|
||||
- `src/infra/bridge/server/tls.ts`
|
||||
- fingerprint verification logic in `src/node-host/bridge-client.ts`
|
||||
|
||||
## Apply to WS
|
||||
|
||||
- WS server supports TLS with same cert/key + fingerprint.
|
||||
- WS clients can pin fingerprint (optional).
|
||||
- Discovery advertises TLS + fingerprint for all endpoints.
|
||||
- Discovery is locator hints only; never a trust anchor.
|
||||
|
||||
## Why
|
||||
|
||||
- Reduce reliance on SSH/Tailscale for confidentiality.
|
||||
- Make remote mobile connections safe by default.
|
||||
|
||||
---
|
||||
|
||||
# Approvals redesign (centralized)
|
||||
|
||||
## Current
|
||||
|
||||
Approval happens on node host (mac app node runtime). Prompt appears where node runs.
|
||||
|
||||
## Proposed
|
||||
|
||||
Approval is **gateway‑hosted**, UI delivered to operator clients.
|
||||
|
||||
### New flow
|
||||
|
||||
1. Gateway receives `system.run` intent (agent).
|
||||
2. Gateway creates approval record: `approval.requested`.
|
||||
3. Operator UI(s) show prompt.
|
||||
4. Approval decision sent to gateway: `approval.resolve`.
|
||||
5. Gateway invokes node command if approved.
|
||||
6. Node executes, returns `invoke-res`.
|
||||
|
||||
### Approval semantics (hardening)
|
||||
|
||||
- Broadcast to all operators; only the active UI shows a modal (others get a toast).
|
||||
- First resolution wins; gateway rejects subsequent resolves as already settled.
|
||||
- Default timeout: deny after N seconds (e.g. 60s), log reason.
|
||||
- Resolution requires `operator.approvals` scope.
|
||||
|
||||
## Benefits
|
||||
|
||||
- Prompt appears where user is (mac/phone).
|
||||
- Consistent approvals for remote nodes.
|
||||
- Node runtime stays headless; no UI dependency.
|
||||
|
||||
---
|
||||
|
||||
# Role clarity examples
|
||||
|
||||
## iPhone app
|
||||
|
||||
- **Node role** for: mic, camera, voice chat, location, push‑to‑talk.
|
||||
- Optional **operator.read** for status and chat view.
|
||||
- Optional **operator.write/admin** only when explicitly enabled.
|
||||
|
||||
## macOS app
|
||||
|
||||
- Operator role by default (control UI).
|
||||
- Node role when “Mac node” enabled (system.run, screen, camera).
|
||||
- Same deviceId for both connections → merged UI entry.
|
||||
|
||||
## CLI
|
||||
|
||||
- Operator role always.
|
||||
- Scope derived by subcommand:
|
||||
- `status`, `logs` → read
|
||||
- `agent`, `message` → write
|
||||
- `config`, `channels` → admin
|
||||
- approvals + pairing → `operator.approvals` / `operator.pairing`
|
||||
|
||||
---
|
||||
|
||||
# Identity + slugs
|
||||
|
||||
## Stable ID
|
||||
|
||||
Required for auth; never changes.
|
||||
Preferred:
|
||||
|
||||
- Keypair fingerprint (public key hash).
|
||||
|
||||
## Cute slug (lobster‑themed)
|
||||
|
||||
Human label only.
|
||||
|
||||
- Example: `scarlet-claw`, `saltwave`, `mantis-pinch`.
|
||||
- Stored in gateway registry, editable.
|
||||
- Collision handling: `-2`, `-3`.
|
||||
|
||||
## UI grouping
|
||||
|
||||
Same `deviceId` across roles → single “Instance” row:
|
||||
|
||||
- Badge: `operator`, `node`.
|
||||
- Shows capabilities + last seen.
|
||||
|
||||
---
|
||||
|
||||
# Migration strategy
|
||||
|
||||
## Phase 0: Document + align
|
||||
|
||||
- Publish this doc.
|
||||
- Inventory all protocol calls + approval flows.
|
||||
|
||||
## Phase 1: Add roles/scopes to WS
|
||||
|
||||
- Extend `connect` params with `role`, `scope`, `deviceId`.
|
||||
- Add allowlist gating for node role.
|
||||
|
||||
## Phase 2: Bridge compatibility
|
||||
|
||||
- Keep bridge running.
|
||||
- Add WS node support in parallel.
|
||||
- Gate features behind config flag.
|
||||
|
||||
## Phase 3: Central approvals
|
||||
|
||||
- Add approval request + resolve events in WS.
|
||||
- Update mac app UI to prompt + respond.
|
||||
- Node runtime stops prompting UI.
|
||||
|
||||
## Phase 4: TLS unification
|
||||
|
||||
- Add TLS config for WS using bridge TLS runtime.
|
||||
- Add pinning to clients.
|
||||
|
||||
## Phase 5: Deprecate bridge
|
||||
|
||||
- Migrate iOS/Android/mac node to WS.
|
||||
- Keep bridge as fallback; remove once stable.
|
||||
|
||||
## Phase 6: Device‑bound auth
|
||||
|
||||
- Require key‑based identity for all non‑local connections.
|
||||
- Add revocation + rotation UI.
|
||||
|
||||
---
|
||||
|
||||
# Security notes
|
||||
|
||||
- Role/allowlist enforced at gateway boundary.
|
||||
- No client gets “full” API without operator scope.
|
||||
- Pairing required for _all_ connections.
|
||||
- TLS + pinning reduces MITM risk for mobile.
|
||||
- SSH silent approval is a convenience; still recorded + revocable.
|
||||
- Discovery is never a trust anchor.
|
||||
- Capability claims are verified against server allowlists by platform/type.
|
||||
|
||||
# Streaming + large payloads (node media)
|
||||
|
||||
WS control plane is fine for small messages, but nodes also do:
|
||||
|
||||
- camera clips
|
||||
- screen recordings
|
||||
- audio streams
|
||||
|
||||
Options:
|
||||
|
||||
1. WS binary frames + chunking + backpressure rules.
|
||||
2. Separate streaming endpoint (still TLS + auth).
|
||||
3. Keep bridge longer for media‑heavy commands, migrate last.
|
||||
|
||||
Pick one before implementation to avoid drift.
|
||||
|
||||
# Capability + command policy
|
||||
|
||||
- Node‑reported caps/commands are treated as **claims**.
|
||||
- Gateway enforces per‑platform allowlists.
|
||||
- Any new command requires operator approval or explicit allowlist change.
|
||||
- Audit changes with timestamps.
|
||||
|
||||
# Audit + rate limiting
|
||||
|
||||
- Log: pairing requests, approvals/denials, token issuance/rotation/revocation.
|
||||
- Rate‑limit pairing spam and approval prompts.
|
||||
|
||||
# Protocol hygiene
|
||||
|
||||
- Explicit protocol version + error codes.
|
||||
- Reconnect rules + heartbeat policy.
|
||||
- Presence TTL and last‑seen semantics.
|
||||
|
||||
---
|
||||
|
||||
# Open questions
|
||||
|
||||
1. Single device running both roles: token model
|
||||
- Recommend separate tokens per role (node vs operator).
|
||||
- Same deviceId; different scopes; clearer revocation.
|
||||
|
||||
2. Operator scope granularity
|
||||
- read/write/admin + approvals + pairing (minimum viable).
|
||||
- Consider per‑feature scopes later.
|
||||
|
||||
3. Token rotation + revocation UX
|
||||
- Auto‑rotate on role change.
|
||||
- UI to revoke by deviceId + role.
|
||||
|
||||
4. Discovery
|
||||
- Extend current Bonjour TXT to include WS TLS fingerprint + role hints.
|
||||
- Treat as locator hints only.
|
||||
|
||||
5. Cross‑network approval
|
||||
- Broadcast to all operator clients; active UI shows modal.
|
||||
- First response wins; gateway enforces atomicity.
|
||||
|
||||
---
|
||||
|
||||
# Summary (TL;DR)
|
||||
|
||||
- Today: WS control plane + Bridge node transport.
|
||||
- Pain: approvals + duplication + two stacks.
|
||||
- Proposal: one WS protocol with explicit roles + scopes, unified pairing + TLS pinning, gateway‑hosted approvals, stable device IDs + cute slugs.
|
||||
- Outcome: simpler UX, stronger security, less duplication, better mobile routing.
|
||||
@@ -1,299 +0,0 @@
|
||||
---
|
||||
summary: "Refactor clusters with highest LOC reduction potential"
|
||||
read_when:
|
||||
- You want to reduce total LOC without changing behavior
|
||||
- You are choosing the next dedupe or extraction pass
|
||||
title: "Refactor Cluster Backlog"
|
||||
---
|
||||
|
||||
# Refactor Cluster Backlog
|
||||
|
||||
Ranked by likely LOC reduction, safety, and breadth.
|
||||
|
||||
## 1. Channel plugin config and security scaffolding
|
||||
|
||||
Highest-value cluster.
|
||||
|
||||
Repeated shapes across many channel plugins:
|
||||
|
||||
- `config.listAccountIds`
|
||||
- `config.resolveAccount`
|
||||
- `config.defaultAccountId`
|
||||
- `config.setAccountEnabled`
|
||||
- `config.deleteAccount`
|
||||
- `config.describeAccount`
|
||||
- `security.resolveDmPolicy`
|
||||
|
||||
Strong examples:
|
||||
|
||||
- `extensions/telegram/src/channel.ts`
|
||||
- `extensions/googlechat/src/channel.ts`
|
||||
- `extensions/slack/src/channel.ts`
|
||||
- `extensions/discord/src/channel.ts`
|
||||
- `extensions/matrix/src/channel.ts`
|
||||
- `extensions/irc/src/channel.ts`
|
||||
- `extensions/signal/src/channel.ts`
|
||||
- `extensions/mattermost/src/channel.ts`
|
||||
|
||||
Likely extraction shape:
|
||||
|
||||
- `buildChannelConfigAdapter(...)`
|
||||
- `buildMultiAccountConfigAdapter(...)`
|
||||
- `buildDmSecurityAdapter(...)`
|
||||
|
||||
Expected savings:
|
||||
|
||||
- ~250-450 LOC
|
||||
|
||||
Risk:
|
||||
|
||||
- Medium. Each channel has slightly different `isConfigured`, warnings, and normalization.
|
||||
|
||||
## 2. Extension runtime singleton boilerplate
|
||||
|
||||
Very safe.
|
||||
|
||||
Nearly every extension has the same runtime holder:
|
||||
|
||||
- `let runtime: PluginRuntime | null = null`
|
||||
- `setXRuntime`
|
||||
- `getXRuntime`
|
||||
|
||||
Strong examples:
|
||||
|
||||
- `extensions/telegram/src/runtime.ts`
|
||||
- `extensions/matrix/src/runtime.ts`
|
||||
- `extensions/slack/src/runtime.ts`
|
||||
- `extensions/discord/src/runtime.ts`
|
||||
- `extensions/whatsapp/src/runtime.ts`
|
||||
- `extensions/imessage/src/runtime.ts`
|
||||
- `extensions/twitch/src/runtime.ts`
|
||||
|
||||
Special-case variants:
|
||||
|
||||
- `extensions/bluebubbles/src/runtime.ts`
|
||||
- `extensions/line/src/runtime.ts`
|
||||
- `extensions/synology-chat/src/runtime.ts`
|
||||
|
||||
Likely extraction shape:
|
||||
|
||||
- `createPluginRuntimeStore<T>(errorMessage)`
|
||||
|
||||
Expected savings:
|
||||
|
||||
- ~180-260 LOC
|
||||
|
||||
Risk:
|
||||
|
||||
- Low
|
||||
|
||||
## 3. Setup prompt and config-patch steps
|
||||
|
||||
Large surface area.
|
||||
|
||||
Many setup files repeat:
|
||||
|
||||
- resolve account id
|
||||
- prompt allowlist entries
|
||||
- merge allowFrom
|
||||
- set DM policy
|
||||
- prompt secrets
|
||||
- patch top-level vs account-scoped config
|
||||
|
||||
Strong examples:
|
||||
|
||||
- `extensions/bluebubbles/src/setup-surface.ts`
|
||||
- `extensions/googlechat/src/setup-surface.ts`
|
||||
- `extensions/msteams/src/setup-surface.ts`
|
||||
- `extensions/zalo/src/setup-surface.ts`
|
||||
- `extensions/zalouser/src/setup-surface.ts`
|
||||
- `extensions/nextcloud-talk/src/setup-surface.ts`
|
||||
- `extensions/matrix/src/setup-surface.ts`
|
||||
- `extensions/irc/src/setup-surface.ts`
|
||||
|
||||
Existing helper surface:
|
||||
|
||||
- `src/channels/plugins/setup-wizard-helpers.ts`
|
||||
|
||||
Likely extraction shape:
|
||||
|
||||
- `promptAllowFromList(...)`
|
||||
- `buildDmPolicyAdapter(...)`
|
||||
- `applyScopedAccountPatch(...)`
|
||||
- `promptSecretFields(...)`
|
||||
|
||||
Expected savings:
|
||||
|
||||
- ~300-600 LOC
|
||||
|
||||
Risk:
|
||||
|
||||
- Medium. Easy to over-generalize; keep helpers narrow and composable.
|
||||
|
||||
## 4. Multi-account config-schema fragments
|
||||
|
||||
Repeated schema fragments across extensions.
|
||||
|
||||
Common patterns:
|
||||
|
||||
- `const allowFromEntry = z.union([z.string(), z.number()])`
|
||||
- account schema plus:
|
||||
- `accounts: z.object({}).catchall(accountSchema).optional()`
|
||||
- `defaultAccount: z.string().optional()`
|
||||
- repeated DM/group fields
|
||||
- repeated markdown/tool policy fields
|
||||
|
||||
Strong examples:
|
||||
|
||||
- `extensions/bluebubbles/src/config-schema.ts`
|
||||
- `extensions/zalo/src/config-schema.ts`
|
||||
- `extensions/zalouser/src/config-schema.ts`
|
||||
- `extensions/matrix/src/config-schema.ts`
|
||||
- `extensions/nostr/src/config-schema.ts`
|
||||
|
||||
Likely extraction shape:
|
||||
|
||||
- `AllowFromEntrySchema`
|
||||
- `buildMultiAccountChannelSchema(accountSchema)`
|
||||
- `buildCommonDmGroupFields(...)`
|
||||
|
||||
Expected savings:
|
||||
|
||||
- ~120-220 LOC
|
||||
|
||||
Risk:
|
||||
|
||||
- Low to medium. Some schemas are simple, some are special.
|
||||
|
||||
## 5. Webhook and monitor lifecycle startup
|
||||
|
||||
Good medium-value cluster.
|
||||
|
||||
Repeated `startAccount` / monitor setup patterns:
|
||||
|
||||
- resolve account
|
||||
- compute webhook path
|
||||
- log startup
|
||||
- start monitor
|
||||
- wait for abort
|
||||
- cleanup
|
||||
- status sink updates
|
||||
|
||||
Strong examples:
|
||||
|
||||
- `extensions/googlechat/src/channel.ts`
|
||||
- `extensions/bluebubbles/src/channel.ts`
|
||||
- `extensions/zalo/src/channel.ts`
|
||||
- `extensions/telegram/src/channel.ts`
|
||||
- `extensions/nextcloud-talk/src/channel.ts`
|
||||
|
||||
Existing helper surface:
|
||||
|
||||
- `src/plugin-sdk/channel-lifecycle.ts`
|
||||
|
||||
Likely extraction shape:
|
||||
|
||||
- helper for account monitor lifecycle
|
||||
- helper for webhook-backed account startup
|
||||
|
||||
Expected savings:
|
||||
|
||||
- ~150-300 LOC
|
||||
|
||||
Risk:
|
||||
|
||||
- Medium to high. Transport details diverge quickly.
|
||||
|
||||
## 6. Small exact-clone cleanup
|
||||
|
||||
Low-risk cleanup bucket.
|
||||
|
||||
Examples:
|
||||
|
||||
- duplicated gateway argv detection:
|
||||
- `src/infra/gateway-lock.ts`
|
||||
- `src/cli/daemon-cli/lifecycle.ts`
|
||||
- duplicated port diagnostics rendering:
|
||||
- `src/cli/daemon-cli/restart-health.ts`
|
||||
- duplicated session-key construction:
|
||||
- `src/web/auto-reply/monitor/broadcast.ts`
|
||||
|
||||
Expected savings:
|
||||
|
||||
- ~30-60 LOC
|
||||
|
||||
Risk:
|
||||
|
||||
- Low
|
||||
|
||||
## Test clusters
|
||||
|
||||
### LINE webhook event fixtures
|
||||
|
||||
Strong examples:
|
||||
|
||||
- `src/line/bot-handlers.test.ts`
|
||||
|
||||
Likely extraction:
|
||||
|
||||
- `makeLineEvent(...)`
|
||||
- `runLineEvent(...)`
|
||||
- `makeLineAccount(...)`
|
||||
|
||||
Expected savings:
|
||||
|
||||
- ~120-180 LOC
|
||||
|
||||
### Telegram native command auth matrix
|
||||
|
||||
Strong examples:
|
||||
|
||||
- `src/telegram/bot-native-commands.group-auth.test.ts`
|
||||
- `src/telegram/bot-native-commands.plugin-auth.test.ts`
|
||||
|
||||
Likely extraction:
|
||||
|
||||
- forum context builder
|
||||
- denied-message assertion helper
|
||||
- table-driven auth cases
|
||||
|
||||
Expected savings:
|
||||
|
||||
- ~80-140 LOC
|
||||
|
||||
### Zalo lifecycle setup
|
||||
|
||||
Strong examples:
|
||||
|
||||
- `extensions/zalo/src/monitor.lifecycle.test.ts`
|
||||
|
||||
Likely extraction:
|
||||
|
||||
- shared monitor setup harness
|
||||
|
||||
Expected savings:
|
||||
|
||||
- ~50-90 LOC
|
||||
|
||||
### Brave llm-context unsupported-option tests
|
||||
|
||||
Strong examples:
|
||||
|
||||
- `src/agents/tools/web-tools.enabled-defaults.test.ts`
|
||||
|
||||
Likely extraction:
|
||||
|
||||
- `it.each(...)` matrix
|
||||
|
||||
Expected savings:
|
||||
|
||||
- ~30-50 LOC
|
||||
|
||||
## Suggested order
|
||||
|
||||
1. Runtime singleton boilerplate
|
||||
2. Small exact-clone cleanup
|
||||
3. Config and security builder extraction
|
||||
4. Test-helper extraction
|
||||
5. Onboarding step extraction
|
||||
6. Monitor lifecycle helper extraction
|
||||
@@ -1,316 +0,0 @@
|
||||
---
|
||||
summary: "Refactor plan: exec host routing, node approvals, and headless runner"
|
||||
read_when:
|
||||
- Designing exec host routing or exec approvals
|
||||
- Implementing node runner + UI IPC
|
||||
- Adding exec host security modes and slash commands
|
||||
title: "Exec Host Refactor"
|
||||
---
|
||||
|
||||
# Exec host refactor plan
|
||||
|
||||
## Goals
|
||||
|
||||
- Add `exec.host` + `exec.security` to route execution across **sandbox**, **gateway**, and **node**.
|
||||
- Keep defaults **safe**: no cross-host execution unless explicitly enabled.
|
||||
- Split execution into a **headless runner service** with optional UI (macOS app) via local IPC.
|
||||
- Provide **per-agent** policy, allowlist, ask mode, and node binding.
|
||||
- Support **ask modes** that work _with_ or _without_ allowlists.
|
||||
- Cross-platform: Unix socket + token auth (macOS/Linux/Windows parity).
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No legacy allowlist migration or legacy schema support.
|
||||
- No PTY/streaming for node exec (aggregated output only).
|
||||
- No new network layer beyond the existing Bridge + Gateway.
|
||||
|
||||
## Decisions (locked)
|
||||
|
||||
- **Config keys:** `exec.host` + `exec.security` (per-agent override allowed).
|
||||
- **Elevation:** keep `/elevated` as an alias for gateway full access.
|
||||
- **Ask default:** `on-miss`.
|
||||
- **Approvals store:** `~/.openclaw/exec-approvals.json` (JSON, no legacy migration).
|
||||
- **Runner:** headless system service; UI app hosts a Unix socket for approvals.
|
||||
- **Node identity:** use existing `nodeId`.
|
||||
- **Socket auth:** Unix socket + token (cross-platform); split later if needed.
|
||||
- **Node host state:** `~/.openclaw/node.json` (node id + pairing token).
|
||||
- **macOS exec host:** run `system.run` inside the macOS app; node host service forwards requests over local IPC.
|
||||
- **No XPC helper:** stick to Unix socket + token + peer checks.
|
||||
|
||||
## Key concepts
|
||||
|
||||
### Host
|
||||
|
||||
- `sandbox`: Docker exec (current behavior).
|
||||
- `gateway`: exec on gateway host.
|
||||
- `node`: exec on node runner via Bridge (`system.run`).
|
||||
|
||||
### Security mode
|
||||
|
||||
- `deny`: always block.
|
||||
- `allowlist`: allow only matches.
|
||||
- `full`: allow everything (equivalent to elevated).
|
||||
|
||||
### Ask mode
|
||||
|
||||
- `off`: never ask.
|
||||
- `on-miss`: ask only when allowlist does not match.
|
||||
- `always`: ask every time.
|
||||
|
||||
Ask is **independent** of allowlist; allowlist can be used with `always` or `on-miss`.
|
||||
|
||||
### Policy resolution (per exec)
|
||||
|
||||
1. Resolve `exec.host` (tool param → agent override → global default).
|
||||
2. Resolve `exec.security` and `exec.ask` (same precedence).
|
||||
3. If host is `sandbox`, proceed with local sandbox exec.
|
||||
4. If host is `gateway` or `node`, apply security + ask policy on that host.
|
||||
|
||||
## Default safety
|
||||
|
||||
- Default `exec.host = sandbox`.
|
||||
- Default `exec.security = deny` for `gateway` and `node`.
|
||||
- Default `exec.ask = on-miss` (only relevant if security allows).
|
||||
- If no node binding is set, **agent may target any node**, but only if policy allows it.
|
||||
|
||||
## Config surface
|
||||
|
||||
### Tool parameters
|
||||
|
||||
- `exec.host` (optional): `sandbox | gateway | node`.
|
||||
- `exec.security` (optional): `deny | allowlist | full`.
|
||||
- `exec.ask` (optional): `off | on-miss | always`.
|
||||
- `exec.node` (optional): node id/name to use when `host=node`.
|
||||
|
||||
### Config keys (global)
|
||||
|
||||
- `tools.exec.host`
|
||||
- `tools.exec.security`
|
||||
- `tools.exec.ask`
|
||||
- `tools.exec.node` (default node binding)
|
||||
|
||||
### Config keys (per agent)
|
||||
|
||||
- `agents.list[].tools.exec.host`
|
||||
- `agents.list[].tools.exec.security`
|
||||
- `agents.list[].tools.exec.ask`
|
||||
- `agents.list[].tools.exec.node`
|
||||
|
||||
### Alias
|
||||
|
||||
- `/elevated on` = set `tools.exec.host=gateway`, `tools.exec.security=full` for the agent session.
|
||||
- `/elevated off` = restore previous exec settings for the agent session.
|
||||
|
||||
## Approvals store (JSON)
|
||||
|
||||
Path: `~/.openclaw/exec-approvals.json`
|
||||
|
||||
Purpose:
|
||||
|
||||
- Local policy + allowlists for the **execution host** (gateway or node runner).
|
||||
- Ask fallback when no UI is available.
|
||||
- IPC credentials for UI clients.
|
||||
|
||||
Proposed schema (v1):
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"socket": {
|
||||
"path": "~/.openclaw/exec-approvals.sock",
|
||||
"token": "base64-opaque-token"
|
||||
},
|
||||
"defaults": {
|
||||
"security": "deny",
|
||||
"ask": "on-miss",
|
||||
"askFallback": "deny"
|
||||
},
|
||||
"agents": {
|
||||
"agent-id-1": {
|
||||
"security": "allowlist",
|
||||
"ask": "on-miss",
|
||||
"allowlist": [
|
||||
{
|
||||
"pattern": "~/Projects/**/bin/rg",
|
||||
"lastUsedAt": 0,
|
||||
"lastUsedCommand": "rg -n TODO",
|
||||
"lastResolvedPath": "/Users/user/Projects/.../bin/rg"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- No legacy allowlist formats.
|
||||
- `askFallback` applies only when `ask` is required and no UI is reachable.
|
||||
- File permissions: `0600`.
|
||||
|
||||
## Runner service (headless)
|
||||
|
||||
### Role
|
||||
|
||||
- Enforce `exec.security` + `exec.ask` locally.
|
||||
- Execute system commands and return output.
|
||||
- Emit Bridge events for exec lifecycle (optional but recommended).
|
||||
|
||||
### Service lifecycle
|
||||
|
||||
- Launchd/daemon on macOS; system service on Linux/Windows.
|
||||
- Approvals JSON is local to the execution host.
|
||||
- UI hosts a local Unix socket; runners connect on demand.
|
||||
|
||||
## UI integration (macOS app)
|
||||
|
||||
### IPC
|
||||
|
||||
- Unix socket at `~/.openclaw/exec-approvals.sock` (0600).
|
||||
- Token stored in `exec-approvals.json` (0600).
|
||||
- Peer checks: same-UID only.
|
||||
- Challenge/response: nonce + HMAC(token, request-hash) to prevent replay.
|
||||
- Short TTL (e.g., 10s) + max payload + rate limit.
|
||||
|
||||
### Ask flow (macOS app exec host)
|
||||
|
||||
1. Node service receives `system.run` from gateway.
|
||||
2. Node service connects to the local socket and sends the prompt/exec request.
|
||||
3. App validates peer + token + HMAC + TTL, then shows dialog if needed.
|
||||
4. App executes the command in UI context and returns output.
|
||||
5. Node service returns output to gateway.
|
||||
|
||||
If UI missing:
|
||||
|
||||
- Apply `askFallback` (`deny|allowlist|full`).
|
||||
|
||||
### Diagram (SCI)
|
||||
|
||||
```
|
||||
Agent -> Gateway -> Bridge -> Node Service (TS)
|
||||
| IPC (UDS + token + HMAC + TTL)
|
||||
v
|
||||
Mac App (UI + TCC + system.run)
|
||||
```
|
||||
|
||||
## Node identity + binding
|
||||
|
||||
- Use existing `nodeId` from Bridge pairing.
|
||||
- Binding model:
|
||||
- `tools.exec.node` restricts the agent to a specific node.
|
||||
- If unset, agent can pick any node (policy still enforces defaults).
|
||||
- Node selection resolution:
|
||||
- `nodeId` exact match
|
||||
- `displayName` (normalized)
|
||||
- `remoteIp`
|
||||
- `nodeId` prefix (>= 6 chars)
|
||||
|
||||
## Eventing
|
||||
|
||||
### Who sees events
|
||||
|
||||
- System events are **per session** and shown to the agent on the next prompt.
|
||||
- Stored in the gateway in-memory queue (`enqueueSystemEvent`).
|
||||
|
||||
### Event text
|
||||
|
||||
- `Exec started (node=<id>, id=<runId>)`
|
||||
- `Exec finished (node=<id>, id=<runId>, code=<code>)` + optional output tail
|
||||
- `Exec denied (node=<id>, id=<runId>, <reason>)`
|
||||
|
||||
### Transport
|
||||
|
||||
Option A (recommended):
|
||||
|
||||
- Runner sends Bridge `event` frames `exec.started` / `exec.finished`.
|
||||
- Gateway `handleBridgeEvent` maps these into `enqueueSystemEvent`.
|
||||
|
||||
Option B:
|
||||
|
||||
- Gateway `exec` tool handles lifecycle directly (synchronous only).
|
||||
|
||||
## Exec flows
|
||||
|
||||
### Sandbox host
|
||||
|
||||
- Existing `exec` behavior (Docker or host when unsandboxed).
|
||||
- PTY supported in non-sandbox mode only.
|
||||
|
||||
### Gateway host
|
||||
|
||||
- Gateway process executes on its own machine.
|
||||
- Enforces local `exec-approvals.json` (security/ask/allowlist).
|
||||
|
||||
### Node host
|
||||
|
||||
- Gateway calls `node.invoke` with `system.run`.
|
||||
- Runner enforces local approvals.
|
||||
- Runner returns aggregated stdout/stderr.
|
||||
- Optional Bridge events for start/finish/deny.
|
||||
|
||||
## Output caps
|
||||
|
||||
- Cap combined stdout+stderr at **200k**; keep **tail 20k** for events.
|
||||
- Truncate with a clear suffix (e.g., `"… (truncated)"`).
|
||||
|
||||
## Slash commands
|
||||
|
||||
- `/exec host=<sandbox|gateway|node> security=<deny|allowlist|full> ask=<off|on-miss|always> node=<id>`
|
||||
- Per-agent, per-session overrides; non-persistent unless saved via config.
|
||||
- `/elevated on|off|ask|full` remains a shortcut for `host=gateway security=full` (with `full` skipping approvals).
|
||||
|
||||
## Cross-platform story
|
||||
|
||||
- The runner service is the portable execution target.
|
||||
- UI is optional; if missing, `askFallback` applies.
|
||||
- Windows/Linux support the same approvals JSON + socket protocol.
|
||||
|
||||
## Implementation phases
|
||||
|
||||
### Phase 1: config + exec routing
|
||||
|
||||
- Add config schema for `exec.host`, `exec.security`, `exec.ask`, `exec.node`.
|
||||
- Update tool plumbing to respect `exec.host`.
|
||||
- Add `/exec` slash command and keep `/elevated` alias.
|
||||
|
||||
### Phase 2: approvals store + gateway enforcement
|
||||
|
||||
- Implement `exec-approvals.json` reader/writer.
|
||||
- Enforce allowlist + ask modes for `gateway` host.
|
||||
- Add output caps.
|
||||
|
||||
### Phase 3: node runner enforcement
|
||||
|
||||
- Update node runner to enforce allowlist + ask.
|
||||
- Add Unix socket prompt bridge to macOS app UI.
|
||||
- Wire `askFallback`.
|
||||
|
||||
### Phase 4: events
|
||||
|
||||
- Add node → gateway Bridge events for exec lifecycle.
|
||||
- Map to `enqueueSystemEvent` for agent prompts.
|
||||
|
||||
### Phase 5: UI polish
|
||||
|
||||
- Mac app: allowlist editor, per-agent switcher, ask policy UI.
|
||||
- Node binding controls (optional).
|
||||
|
||||
## Testing plan
|
||||
|
||||
- Unit tests: allowlist matching (glob + case-insensitive).
|
||||
- Unit tests: policy resolution precedence (tool param → agent override → global).
|
||||
- Integration tests: node runner deny/allow/ask flows.
|
||||
- Bridge event tests: node event → system event routing.
|
||||
|
||||
## Open risks
|
||||
|
||||
- UI unavailability: ensure `askFallback` is respected.
|
||||
- Long-running commands: rely on timeout + output caps.
|
||||
- Multi-node ambiguity: error unless node binding or explicit node param.
|
||||
|
||||
## Related docs
|
||||
|
||||
- [Exec tool](/tools/exec)
|
||||
- [Exec approvals](/tools/exec-approvals)
|
||||
- [Nodes](/nodes)
|
||||
- [Elevated mode](/tools/elevated)
|
||||
@@ -1,260 +0,0 @@
|
||||
---
|
||||
summary: "Design for an opt-in Firecrawl extension that adds search/scrape value without hardwiring Firecrawl into core defaults"
|
||||
read_when:
|
||||
- Designing Firecrawl integration work
|
||||
- Evaluating web_search/web_fetch plugin extension surfaces
|
||||
- Deciding whether Firecrawl belongs in core or as an extension
|
||||
title: "Firecrawl Extension Design"
|
||||
---
|
||||
|
||||
# Firecrawl Extension Design
|
||||
|
||||
## Goal
|
||||
|
||||
Ship Firecrawl as an **opt-in extension** that adds:
|
||||
|
||||
- explicit Firecrawl tools for agents,
|
||||
- optional Firecrawl-backed `web_search` integration,
|
||||
- self-hosted support,
|
||||
- stronger security defaults than the current core fallback path,
|
||||
|
||||
without pushing Firecrawl into the default setup/onboarding path.
|
||||
|
||||
## Why this shape
|
||||
|
||||
Recent Firecrawl issues/PRs cluster into three buckets:
|
||||
|
||||
1. **Release/schema drift**
|
||||
- Several releases rejected `tools.web.fetch.firecrawl` even though docs and runtime code supported it.
|
||||
2. **Security hardening**
|
||||
- Current `fetchFirecrawlContent()` still posts to the Firecrawl endpoint with raw `fetch()`, while the main web-fetch path uses the SSRF guard.
|
||||
3. **Product pressure**
|
||||
- Users want Firecrawl-native search/scrape flows, especially for self-hosted/private setups.
|
||||
- Maintainers explicitly rejected wiring Firecrawl deeply into core defaults, setup flow, and browser behavior.
|
||||
|
||||
That combination argues for an extension, not more Firecrawl-specific logic in the default core path.
|
||||
|
||||
## Design principles
|
||||
|
||||
- **Opt-in, vendor-scoped**: no auto-enable, no setup hijack, no default tool-profile widening.
|
||||
- **Extension owns Firecrawl-specific config**: prefer plugin config over growing `tools.web.*` again.
|
||||
- **Useful on day one**: works even if core `web_search` / `web_fetch` extension surfaces stay unchanged.
|
||||
- **Security-first**: endpoint fetches use the same guarded networking posture as other web tools.
|
||||
- **Self-hosted-friendly**: config + env fallback, explicit base URL, no hosted-only assumptions.
|
||||
|
||||
## Proposed extension
|
||||
|
||||
Plugin id: `firecrawl`
|
||||
|
||||
### MVP capabilities
|
||||
|
||||
Register explicit tools:
|
||||
|
||||
- `firecrawl_search`
|
||||
- `firecrawl_scrape`
|
||||
|
||||
Optional later:
|
||||
|
||||
- `firecrawl_crawl`
|
||||
- `firecrawl_map`
|
||||
|
||||
Do **not** add Firecrawl browser automation in the first version. That was the part of PR #32543 that pulled Firecrawl too far into core behavior and raised the most maintainership concern.
|
||||
|
||||
## Config shape
|
||||
|
||||
Use plugin-scoped config:
|
||||
|
||||
```json5
|
||||
{
|
||||
plugins: {
|
||||
entries: {
|
||||
firecrawl: {
|
||||
enabled: true,
|
||||
config: {
|
||||
apiKey: "FIRECRAWL_API_KEY",
|
||||
baseUrl: "https://api.firecrawl.dev",
|
||||
timeoutSeconds: 60,
|
||||
maxAgeMs: 172800000,
|
||||
proxy: "auto",
|
||||
storeInCache: true,
|
||||
onlyMainContent: true,
|
||||
search: {
|
||||
enabled: true,
|
||||
defaultLimit: 5,
|
||||
sources: ["web"],
|
||||
categories: [],
|
||||
scrapeResults: false,
|
||||
},
|
||||
scrape: {
|
||||
formats: ["markdown"],
|
||||
fallbackForWebFetchLikeUse: false,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### Credential resolution
|
||||
|
||||
Precedence:
|
||||
|
||||
1. `plugins.entries.firecrawl.config.apiKey`
|
||||
2. `FIRECRAWL_API_KEY`
|
||||
|
||||
Base URL precedence:
|
||||
|
||||
1. `plugins.entries.firecrawl.config.baseUrl`
|
||||
2. `FIRECRAWL_BASE_URL`
|
||||
3. `https://api.firecrawl.dev`
|
||||
|
||||
### Compatibility bridge
|
||||
|
||||
For the first release, the extension may also **read** existing core config at `tools.web.fetch.firecrawl.*` as a fallback source so existing users do not need to migrate immediately.
|
||||
|
||||
Write path stays plugin-local. Do not keep expanding core Firecrawl config surfaces.
|
||||
|
||||
## Tool design
|
||||
|
||||
### `firecrawl_search`
|
||||
|
||||
Inputs:
|
||||
|
||||
- `query`
|
||||
- `limit`
|
||||
- `sources`
|
||||
- `categories`
|
||||
- `scrapeResults`
|
||||
- `timeoutSeconds`
|
||||
|
||||
Behavior:
|
||||
|
||||
- Calls Firecrawl `v2/search`
|
||||
- Returns normalized OpenClaw-friendly result objects:
|
||||
- `title`
|
||||
- `url`
|
||||
- `snippet`
|
||||
- `source`
|
||||
- optional `content`
|
||||
- Wraps result content as untrusted external content
|
||||
- Cache key includes query + relevant provider params
|
||||
|
||||
Why explicit tool first:
|
||||
|
||||
- Works today without changing `tools.web.search.provider`
|
||||
- Avoids current schema/loader constraints
|
||||
- Gives users Firecrawl value immediately
|
||||
|
||||
### `firecrawl_scrape`
|
||||
|
||||
Inputs:
|
||||
|
||||
- `url`
|
||||
- `formats`
|
||||
- `onlyMainContent`
|
||||
- `maxAgeMs`
|
||||
- `proxy`
|
||||
- `storeInCache`
|
||||
- `timeoutSeconds`
|
||||
|
||||
Behavior:
|
||||
|
||||
- Calls Firecrawl `v2/scrape`
|
||||
- Returns markdown/text plus metadata:
|
||||
- `title`
|
||||
- `finalUrl`
|
||||
- `status`
|
||||
- `warning`
|
||||
- Wraps extracted content the same way `web_fetch` does
|
||||
- Shares cache semantics with web tool expectations where practical
|
||||
|
||||
Why explicit scrape tool:
|
||||
|
||||
- Sidesteps the unresolved `Readability -> Firecrawl -> basic HTML cleanup` ordering bug in core `web_fetch`
|
||||
- Gives users a deterministic “always use Firecrawl” path for JS-heavy/bot-protected sites
|
||||
|
||||
## What the extension should not do
|
||||
|
||||
- No auto-adding `browser`, `web_search`, or `web_fetch` to `tools.alsoAllow`
|
||||
- No default onboarding step in `openclaw setup`
|
||||
- No Firecrawl-specific browser session lifecycle in core
|
||||
- No change to built-in `web_fetch` fallback semantics in the extension MVP
|
||||
|
||||
## Phase plan
|
||||
|
||||
### Phase 1: extension-only, no core schema changes
|
||||
|
||||
Implement:
|
||||
|
||||
- `extensions/firecrawl/`
|
||||
- plugin config schema
|
||||
- `firecrawl_search`
|
||||
- `firecrawl_scrape`
|
||||
- tests for config resolution, endpoint selection, caching, error handling, and SSRF guard usage
|
||||
|
||||
This phase is enough to ship real user value.
|
||||
|
||||
### Phase 2: optional `web_search` provider integration
|
||||
|
||||
Support `tools.web.search.provider = "firecrawl"` only after fixing two core constraints:
|
||||
|
||||
1. `src/plugins/web-search-providers.ts` must load configured/installed web-search-provider plugins instead of a hardcoded bundled list.
|
||||
2. `src/config/types.tools.ts` and `src/config/zod-schema.agent-runtime.ts` must stop hardcoding the provider enum in a way that blocks plugin-registered ids.
|
||||
|
||||
Recommended shape:
|
||||
|
||||
- keep built-in providers documented,
|
||||
- allow any registered plugin provider id at runtime,
|
||||
- validate provider-specific config via the provider plugin or a generic provider bag.
|
||||
|
||||
### Phase 3: optional `web_fetch` provider capability
|
||||
|
||||
Do this only if maintainers want vendor-specific fetch backends to participate in `web_fetch`.
|
||||
|
||||
Needed core addition:
|
||||
|
||||
- `registerWebFetchProvider` or equivalent fetch-backend extension surface
|
||||
|
||||
Without that capability, the extension should keep `firecrawl_scrape` as an explicit tool rather than trying to patch built-in `web_fetch`.
|
||||
|
||||
## Security requirements
|
||||
|
||||
The extension must treat Firecrawl as a **trusted operator-configured endpoint**, but still harden transport:
|
||||
|
||||
- Use SSRF-guarded fetch for the Firecrawl endpoint call, not raw `fetch()`
|
||||
- Preserve self-hosted/private-network compatibility using the same trusted-web-tools endpoint policy used elsewhere
|
||||
- Never log the API key
|
||||
- Keep endpoint/base URL resolution explicit and predictable
|
||||
- Treat Firecrawl-returned content as untrusted external content
|
||||
|
||||
This mirrors the intent behind the SSRF hardening PRs without assuming Firecrawl is a hostile multi-tenant surface.
|
||||
|
||||
## Why not a skill
|
||||
|
||||
The repo already closed a Firecrawl skill PR in favor of ClawHub distribution. That is fine for optional user-installed prompt workflows, but it does not solve:
|
||||
|
||||
- deterministic tool availability,
|
||||
- provider-grade config/credential handling,
|
||||
- self-hosted endpoint support,
|
||||
- caching,
|
||||
- stable typed outputs,
|
||||
- security review on network behavior.
|
||||
|
||||
This belongs as an extension, not a prompt-only skill.
|
||||
|
||||
## Success criteria
|
||||
|
||||
- Users can install/enable one extension and get reliable Firecrawl search/scrape without touching core defaults.
|
||||
- Self-hosted Firecrawl works with config/env fallback.
|
||||
- Extension endpoint fetches use guarded networking.
|
||||
- No new Firecrawl-specific core onboarding/default behavior.
|
||||
- Core can later adopt plugin-native `web_search` / `web_fetch` extension surfaces without redesigning the extension.
|
||||
|
||||
## Recommended implementation order
|
||||
|
||||
1. Build `firecrawl_scrape`
|
||||
2. Build `firecrawl_search`
|
||||
3. Add docs and examples
|
||||
4. If desired, generalize `web_search` provider loading so the extension can back `web_search`
|
||||
5. Only then consider a true `web_fetch` provider capability
|
||||
@@ -1,89 +0,0 @@
|
||||
---
|
||||
title: Outbound Session Mirroring Refactor (Issue #1520)
|
||||
description: Track outbound session mirroring refactor notes, decisions, tests, and open items.
|
||||
summary: "Refactor notes for mirroring outbound sends into target channel sessions"
|
||||
read_when:
|
||||
- Working on outbound transcript/session mirroring behavior
|
||||
- Debugging sessionKey derivation for send/message tool paths
|
||||
---
|
||||
|
||||
# Outbound Session Mirroring Refactor (Issue #1520)
|
||||
|
||||
## Status
|
||||
|
||||
- In progress.
|
||||
- Core + plugin channel routing updated for outbound mirroring.
|
||||
- Gateway send now derives target session when sessionKey is omitted.
|
||||
|
||||
## Context
|
||||
|
||||
Outbound sends were mirrored into the _current_ agent session (tool session key) rather than the target channel session. Inbound routing uses channel/peer session keys, so outbound responses landed in the wrong session and first-contact targets often lacked session entries.
|
||||
|
||||
## Goals
|
||||
|
||||
- Mirror outbound messages into the target channel session key.
|
||||
- Create session entries on outbound when missing.
|
||||
- Keep thread/topic scoping aligned with inbound session keys.
|
||||
- Cover core channels plus bundled extensions.
|
||||
|
||||
## Implementation Summary
|
||||
|
||||
- New outbound session routing helper:
|
||||
- `src/infra/outbound/outbound-session.ts`
|
||||
- `resolveOutboundSessionRoute` builds target sessionKey using `buildAgentSessionKey` (dmScope + identityLinks).
|
||||
- `ensureOutboundSessionEntry` writes minimal `MsgContext` via `recordSessionMetaFromInbound`.
|
||||
- `runMessageAction` (send) derives target sessionKey and passes it to `executeSendAction` for mirroring.
|
||||
- `message-tool` no longer mirrors directly; it only resolves agentId from the current session key.
|
||||
- Plugin send path mirrors via `appendAssistantMessageToSessionTranscript` using the derived sessionKey.
|
||||
- Gateway send derives a target session key when none is provided (default agent), and ensures a session entry.
|
||||
|
||||
## Thread/Topic Handling
|
||||
|
||||
- Slack: replyTo/threadId -> `resolveThreadSessionKeys` (suffix).
|
||||
- Discord: threadId/replyTo -> `resolveThreadSessionKeys` with `useSuffix=false` to match inbound (thread channel id already scopes session).
|
||||
- Telegram: topic IDs map to `chatId:topic:<id>` via `buildTelegramGroupPeerId`.
|
||||
|
||||
## Extensions Covered
|
||||
|
||||
- Matrix, MS Teams, Mattermost, BlueBubbles, Nextcloud Talk, Zalo, Zalo Personal, Nostr, Tlon.
|
||||
- Notes:
|
||||
- Mattermost targets now strip `@` for DM session key routing.
|
||||
- Zalo Personal uses DM peer kind for 1:1 targets (group only when `group:` is present).
|
||||
- BlueBubbles group targets strip `chat_*` prefixes to match inbound session keys.
|
||||
- Slack auto-thread mirroring matches channel ids case-insensitively.
|
||||
- Gateway send lowercases provided session keys before mirroring.
|
||||
|
||||
## Decisions
|
||||
|
||||
- **Gateway send session derivation**: if `sessionKey` is provided, use it. If omitted, derive a sessionKey from target + default agent and mirror there.
|
||||
- **Session entry creation**: always use `recordSessionMetaFromInbound` with `Provider/From/To/ChatType/AccountId/Originating*` aligned to inbound formats.
|
||||
- **Target normalization**: outbound routing uses resolved targets (post `resolveChannelTarget`) when available.
|
||||
- **Session key casing**: canonicalize session keys to lowercase on write and during migrations.
|
||||
|
||||
## Tests Added/Updated
|
||||
|
||||
- `src/infra/outbound/outbound.test.ts`
|
||||
- Slack thread session key.
|
||||
- Telegram topic session key.
|
||||
- dmScope identityLinks with Discord.
|
||||
- `src/agents/tools/message-tool.test.ts`
|
||||
- Derives agentId from session key (no sessionKey passed through).
|
||||
- `src/gateway/server-methods/send.test.ts`
|
||||
- Derives session key when omitted and creates session entry.
|
||||
|
||||
## Open Items / Follow-ups
|
||||
|
||||
- Voice-call plugin uses custom `voice:<phone>` session keys. Outbound mapping is not standardized here; if message-tool should support voice-call sends, add explicit mapping.
|
||||
- Confirm if any external plugin uses non-standard `From/To` formats beyond the bundled set.
|
||||
|
||||
## Files Touched
|
||||
|
||||
- `src/infra/outbound/outbound-session.ts`
|
||||
- `src/infra/outbound/outbound-send-service.ts`
|
||||
- `src/infra/outbound/message-action-runner.ts`
|
||||
- `src/agents/tools/message-tool.ts`
|
||||
- `src/gateway/server-methods/send.ts`
|
||||
- Tests in:
|
||||
- `src/infra/outbound/outbound.test.ts`
|
||||
- `src/agents/tools/message-tool.test.ts`
|
||||
- `src/gateway/server-methods/send.test.ts`
|
||||
@@ -1,264 +0,0 @@
|
||||
---
|
||||
summary: "Plan: one clean plugin SDK + runtime for all messaging connectors"
|
||||
read_when:
|
||||
- Defining or refactoring the plugin architecture
|
||||
- Migrating channel connectors to the plugin SDK/runtime
|
||||
title: "Plugin SDK Refactor"
|
||||
---
|
||||
|
||||
# Plugin SDK + Runtime Refactor Plan
|
||||
|
||||
Goal: every messaging connector is a plugin (bundled or external) using one stable API.
|
||||
No plugin imports from `src/**` directly. All dependencies go through the SDK or runtime.
|
||||
|
||||
## Why now
|
||||
|
||||
- Current connectors mix patterns: direct core imports, dist-only bridges, and custom helpers.
|
||||
- This makes upgrades brittle and blocks a clean external plugin surface.
|
||||
|
||||
## Target architecture (two layers)
|
||||
|
||||
### 1) Plugin SDK (compile-time, stable, publishable)
|
||||
|
||||
Scope: types, helpers, and config utilities. No runtime state, no side effects.
|
||||
|
||||
Contents (examples):
|
||||
|
||||
- Types: `ChannelPlugin`, adapters, `ChannelMeta`, `ChannelCapabilities`, `ChannelDirectoryEntry`.
|
||||
- Config helpers: `buildChannelConfigSchema`, `setAccountEnabledInConfigSection`, `deleteAccountFromConfigSection`,
|
||||
`applyAccountNameToChannelSection`.
|
||||
- Pairing helpers: `PAIRING_APPROVED_MESSAGE`, `formatPairingApproveHint`.
|
||||
- Setup entry points: host-owned `setup` + `setupWizard`; avoid broad public onboarding helpers.
|
||||
- Tool param helpers: `createActionGate`, `readStringParam`, `readNumberParam`, `readReactionParams`, `jsonResult`.
|
||||
- Docs link helper: `formatDocsLink`.
|
||||
|
||||
Delivery:
|
||||
|
||||
- Publish as `openclaw/plugin-sdk` (or export from core under `openclaw/plugin-sdk`).
|
||||
- Semver with explicit stability guarantees.
|
||||
|
||||
### 2) Plugin Runtime (execution surface, injected)
|
||||
|
||||
Scope: everything that touches core runtime behavior.
|
||||
Accessed via `OpenClawPluginApi.runtime` so plugins never import `src/**`.
|
||||
|
||||
Proposed surface (minimal but complete):
|
||||
|
||||
```ts
|
||||
export type PluginRuntime = {
|
||||
channel: {
|
||||
text: {
|
||||
chunkMarkdownText(text: string, limit: number): string[];
|
||||
resolveTextChunkLimit(cfg: OpenClawConfig, channel: string, accountId?: string): number;
|
||||
hasControlCommand(text: string, cfg: OpenClawConfig): boolean;
|
||||
};
|
||||
reply: {
|
||||
dispatchReplyWithBufferedBlockDispatcher(params: {
|
||||
ctx: unknown;
|
||||
cfg: unknown;
|
||||
dispatcherOptions: {
|
||||
deliver: (payload: {
|
||||
text?: string;
|
||||
mediaUrls?: string[];
|
||||
mediaUrl?: string;
|
||||
}) => void | Promise<void>;
|
||||
onError?: (err: unknown, info: { kind: string }) => void;
|
||||
};
|
||||
}): Promise<void>;
|
||||
createReplyDispatcherWithTyping?: unknown; // adapter for Teams-style flows
|
||||
};
|
||||
routing: {
|
||||
resolveAgentRoute(params: {
|
||||
cfg: unknown;
|
||||
channel: string;
|
||||
accountId: string;
|
||||
peer: { kind: RoutePeerKind; id: string };
|
||||
}): { sessionKey: string; accountId: string };
|
||||
};
|
||||
pairing: {
|
||||
buildPairingReply(params: { channel: string; idLine: string; code: string }): string;
|
||||
readAllowFromStore(channel: string): Promise<string[]>;
|
||||
upsertPairingRequest(params: {
|
||||
channel: string;
|
||||
id: string;
|
||||
meta?: { name?: string };
|
||||
}): Promise<{ code: string; created: boolean }>;
|
||||
};
|
||||
media: {
|
||||
fetchRemoteMedia(params: { url: string }): Promise<{ buffer: Buffer; contentType?: string }>;
|
||||
saveMediaBuffer(
|
||||
buffer: Uint8Array,
|
||||
contentType: string | undefined,
|
||||
direction: "inbound" | "outbound",
|
||||
maxBytes: number,
|
||||
): Promise<{ path: string; contentType?: string }>;
|
||||
};
|
||||
mentions: {
|
||||
buildMentionRegexes(cfg: OpenClawConfig, agentId?: string): RegExp[];
|
||||
matchesMentionPatterns(text: string, regexes: RegExp[]): boolean;
|
||||
};
|
||||
groups: {
|
||||
resolveGroupPolicy(
|
||||
cfg: OpenClawConfig,
|
||||
channel: string,
|
||||
accountId: string,
|
||||
groupId: string,
|
||||
): {
|
||||
allowlistEnabled: boolean;
|
||||
allowed: boolean;
|
||||
groupConfig?: unknown;
|
||||
defaultConfig?: unknown;
|
||||
};
|
||||
resolveRequireMention(
|
||||
cfg: OpenClawConfig,
|
||||
channel: string,
|
||||
accountId: string,
|
||||
groupId: string,
|
||||
override?: boolean,
|
||||
): boolean;
|
||||
};
|
||||
debounce: {
|
||||
createInboundDebouncer<T>(opts: {
|
||||
debounceMs: number;
|
||||
buildKey: (v: T) => string | null;
|
||||
shouldDebounce: (v: T) => boolean;
|
||||
onFlush: (entries: T[]) => Promise<void>;
|
||||
onError?: (err: unknown) => void;
|
||||
}): { push: (v: T) => void; flush: () => Promise<void> };
|
||||
resolveInboundDebounceMs(cfg: OpenClawConfig, channel: string): number;
|
||||
};
|
||||
commands: {
|
||||
resolveCommandAuthorizedFromAuthorizers(params: {
|
||||
useAccessGroups: boolean;
|
||||
authorizers: Array<{ configured: boolean; allowed: boolean }>;
|
||||
}): boolean;
|
||||
};
|
||||
};
|
||||
logging: {
|
||||
shouldLogVerbose(): boolean;
|
||||
getChildLogger(name: string): PluginLogger;
|
||||
};
|
||||
state: {
|
||||
resolveStateDir(cfg: OpenClawConfig): string;
|
||||
};
|
||||
};
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Runtime is the only way to access core behavior.
|
||||
- SDK is intentionally small and stable.
|
||||
- Each runtime method maps to an existing core implementation (no duplication).
|
||||
|
||||
## Migration plan (phased, safe)
|
||||
|
||||
### Phase 0: scaffolding
|
||||
|
||||
- Introduce `openclaw/plugin-sdk`.
|
||||
- Add `api.runtime` to `OpenClawPluginApi` with the surface above.
|
||||
- Maintain existing imports during a transition window (deprecation warnings).
|
||||
|
||||
### Phase 1: bridge cleanup (low risk)
|
||||
|
||||
- Replace per-extension `core-bridge.ts` with `api.runtime`.
|
||||
- Migrate BlueBubbles, Zalo, Zalo Personal first (already close).
|
||||
- Remove duplicated bridge code.
|
||||
|
||||
### Phase 2: light direct-import plugins
|
||||
|
||||
- Migrate Matrix to SDK + runtime.
|
||||
- Validate onboarding, directory, group mention logic.
|
||||
|
||||
### Phase 3: heavy direct-import plugins
|
||||
|
||||
- Migrate MS Teams (largest set of runtime helpers).
|
||||
- Ensure reply/typing semantics match current behavior.
|
||||
|
||||
### Phase 4: iMessage pluginization
|
||||
|
||||
- Move iMessage into `extensions/imessage`.
|
||||
- Replace direct core calls with `api.runtime`.
|
||||
- Keep config keys, CLI behavior, and docs intact.
|
||||
|
||||
### Phase 5: enforcement
|
||||
|
||||
- Add lint rule / CI check: no `extensions/**` imports from `src/**`.
|
||||
- Add plugin SDK/version compatibility checks (runtime + SDK semver).
|
||||
|
||||
## Compatibility and versioning
|
||||
|
||||
- SDK: semver, published, documented changes.
|
||||
- Runtime: versioned per core release. Add `api.runtime.version`.
|
||||
- Plugins declare a required runtime range (e.g., `openclawRuntime: ">=2026.2.0"`).
|
||||
|
||||
## Testing strategy
|
||||
|
||||
- Adapter-level unit tests (runtime functions exercised with real core implementation).
|
||||
- Golden tests per plugin: ensure no behavior drift (routing, pairing, allowlist, mention gating).
|
||||
- A single end-to-end plugin sample used in CI (install + run + smoke).
|
||||
|
||||
## Open questions
|
||||
|
||||
- Where to host SDK types: separate package or core export?
|
||||
- Runtime type distribution: in SDK (types only) or in core?
|
||||
- How to expose docs links for bundled vs external plugins?
|
||||
- Do we allow limited direct core imports for in-repo plugins during transition?
|
||||
|
||||
## Success criteria
|
||||
|
||||
- All channel connectors are plugins using SDK + runtime.
|
||||
- No `extensions/**` imports from `src/**`.
|
||||
- New connector templates depend only on SDK + runtime.
|
||||
- External plugins can be developed and updated without core source access.
|
||||
|
||||
Related docs: [Plugins](/tools/plugin), [Channels](/channels/index), [Configuration](/gateway/configuration).
|
||||
|
||||
## Capability plan alignment
|
||||
|
||||
The plugin SDK refactor now aligns with the public capability model documented
|
||||
in [Plugins](/tools/plugin#public-capability-model).
|
||||
|
||||
Key decisions:
|
||||
|
||||
- Capabilities are the public plugin model. Registration is explicit and typed.
|
||||
- Legacy hook-only plugins remain supported without migration.
|
||||
- Plugin shapes (plain-capability, hybrid-capability, hook-only, non-capability)
|
||||
are classified from actual registration behavior.
|
||||
- `openclaw plugins inspect` provides canonical deep introspection for any
|
||||
loaded plugin, showing shape, capabilities, hooks, tools, and diagnostics.
|
||||
- Export boundary: export capabilities, not implementation convenience. Trim
|
||||
non-contract helper exports.
|
||||
|
||||
Required test matrix for the capability model:
|
||||
|
||||
- hook-only legacy plugin fixture
|
||||
- plain capability plugin fixture
|
||||
- hybrid capability plugin fixture
|
||||
- real-world legacy hook-style plugin fixture
|
||||
- `before_agent_start` still works
|
||||
- typed hooks remain additive
|
||||
- capability usage and plugin shape are inspectable
|
||||
|
||||
## Implemented channel-owned capabilities
|
||||
|
||||
Recent refactor work widened the channel plugin contract so core can stop owning
|
||||
channel-specific UX and routing behavior:
|
||||
|
||||
- `messaging.buildCrossContextComponents`: channel-owned cross-context UI markers
|
||||
(for example Discord components v2 containers)
|
||||
- `messaging.enableInteractiveReplies`: channel-owned reply normalization toggles
|
||||
(for example Slack interactive replies)
|
||||
- `messaging.resolveOutboundSessionRoute`: channel-owned outbound session routing
|
||||
- `status.formatCapabilitiesProbe` / `status.buildCapabilitiesDiagnostics`: channel-owned
|
||||
`/channels capabilities` probe display and extra audits/scopes
|
||||
- `threading.resolveAutoThreadId`: channel-owned same-conversation auto-threading
|
||||
- `threading.resolveReplyTransport`: channel-owned reply-vs-thread delivery mapping
|
||||
- `actions.requiresTrustedRequesterSender`: channel-owned privileged action trust gates
|
||||
- `execApprovals.*`: channel-owned exec approval surface state, forwarding suppression,
|
||||
pending payload UX, and pre-delivery hooks
|
||||
- `lifecycle.onAccountConfigChanged` / `lifecycle.onAccountRemoved`: channel-owned cleanup on
|
||||
config mutation/removal
|
||||
- `allowlist.supportsScope`: channel-owned allowlist scope advertisement
|
||||
|
||||
These capabilities should be preferred over new `channel === "discord"` /
|
||||
`telegram` branches in shared core flows.
|
||||
@@ -1,93 +0,0 @@
|
||||
---
|
||||
summary: "Strict config validation + doctor-only migrations"
|
||||
read_when:
|
||||
- Designing or implementing config validation behavior
|
||||
- Working on config migrations or doctor workflows
|
||||
- Handling plugin config schemas or plugin load gating
|
||||
title: "Strict Config Validation"
|
||||
---
|
||||
|
||||
# Strict config validation (doctor-only migrations)
|
||||
|
||||
## Goals
|
||||
|
||||
- **Reject unknown config keys everywhere** (root + nested), except root `$schema` metadata.
|
||||
- **Reject plugin config without a schema**; don’t load that plugin.
|
||||
- **Remove legacy auto-migration on load**; migrations run via doctor only.
|
||||
- **Auto-run doctor (dry-run) on startup**; if invalid, block non-diagnostic commands.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Backward compatibility on load (legacy keys do not auto-migrate).
|
||||
- Silent drops of unrecognized keys.
|
||||
|
||||
## Strict validation rules
|
||||
|
||||
- Config must match the schema exactly at every level.
|
||||
- Unknown keys are validation errors (no passthrough at root or nested), except root `$schema` when it is a string.
|
||||
- `plugins.entries.<id>.config` must be validated by the plugin’s schema.
|
||||
- If a plugin lacks a schema, **reject plugin load** and surface a clear error.
|
||||
- Unknown `channels.<id>` keys are errors unless a plugin manifest declares the channel id.
|
||||
- Plugin manifests (`openclaw.plugin.json`) are required for all plugins.
|
||||
|
||||
## Plugin schema enforcement
|
||||
|
||||
- Each plugin provides a strict JSON Schema for its config (inline in the manifest).
|
||||
- Plugin load flow:
|
||||
1. Resolve plugin manifest + schema (`openclaw.plugin.json`).
|
||||
2. Validate config against the schema.
|
||||
3. If missing schema or invalid config: block plugin load, record error.
|
||||
- Error message includes:
|
||||
- Plugin id
|
||||
- Reason (missing schema / invalid config)
|
||||
- Path(s) that failed validation
|
||||
- Disabled plugins keep their config, but Doctor + logs surface a warning.
|
||||
|
||||
## Doctor flow
|
||||
|
||||
- Doctor runs **every time** config is loaded (dry-run by default).
|
||||
- If config invalid:
|
||||
- Print a summary + actionable errors.
|
||||
- Instruct: `openclaw doctor --fix`.
|
||||
- `openclaw doctor --fix`:
|
||||
- Applies migrations.
|
||||
- Removes unknown keys.
|
||||
- Writes updated config.
|
||||
|
||||
## Command gating (when config is invalid)
|
||||
|
||||
Allowed (diagnostic-only):
|
||||
|
||||
- `openclaw doctor`
|
||||
- `openclaw logs`
|
||||
- `openclaw health`
|
||||
- `openclaw help`
|
||||
- `openclaw status`
|
||||
- `openclaw gateway status`
|
||||
|
||||
Everything else must hard-fail with: “Config invalid. Run `openclaw doctor --fix`.”
|
||||
|
||||
## Error UX format
|
||||
|
||||
- Single summary header.
|
||||
- Grouped sections:
|
||||
- Unknown keys (full paths)
|
||||
- Legacy keys / migrations needed
|
||||
- Plugin load failures (plugin id + reason + path)
|
||||
|
||||
## Implementation touchpoints
|
||||
|
||||
- `src/config/zod-schema.ts`: remove root passthrough; strict objects everywhere.
|
||||
- `src/config/zod-schema.providers.ts`: ensure strict channel schemas.
|
||||
- `src/config/validation.ts`: fail on unknown keys; do not apply legacy migrations.
|
||||
- `src/config/io.ts`: remove legacy auto-migrations; always run doctor dry-run.
|
||||
- `src/config/legacy*.ts`: move usage to doctor only.
|
||||
- `src/plugins/*`: add schema registry + gating.
|
||||
- CLI command gating in `src/cli`.
|
||||
|
||||
## Tests
|
||||
|
||||
- Unknown key rejection (root + nested).
|
||||
- Plugin missing schema → plugin load blocked with clear error.
|
||||
- Invalid config → gateway startup blocked except diagnostic commands.
|
||||
- Doctor dry-run auto; `doctor --fix` writes corrected config.
|
||||
Reference in New Issue
Block a user