mirror of
https://github.com/openclaw/openclaw.git
synced 2026-08-02 18:21:34 +00:00
e10d004257
* feat(agents): mobile_ui agent tool for Android UI control (PR 3/3) Add a dedicated model-facing tool that drives another Android app through the PR2 mobile.ui.observe/act node commands, completing the vertical slice (agent -> tool -> node.invoke -> AccessibilityService). - Mirrors the desktop computer tool's safety model: owner-only + HTTP-denied (dangerous-tools), raw node.invoke of mobile.ui.* redirected to this tool so the generic nodes tool cannot bypass it, run/tool-call idempotency, and the phone-arm workflow (mobile.ui.* must be explicitly armed). - One call is observe or one act; every act automatically re-observes for postcondition verification and preserves the landed outcome if re-observe fails. - Fail-closed confirmation: all state-changing acts (activate, set_text, tap, swipe) require confirmed=true after the model reviews the proposed effect; observe, scroll, wait, and navigation global_actions do not. The keyword list only enriches the confirmation message and is never the sole gate (a11y labels are localized/iconographic/coordinate-blind). - Node selection resolves an explicit id against the full device set first (case-insensitively) and rejects an ineligible or ambiguous match, so an explicit selection can never be silently redirected to the wrong phone. - All observed UI text is treated as untrusted; the tool description forbids following instructions found in app UI. Additive; no protocol bump. mobile-ui-tool + registration + policy tests pass; core tsgo and lint verified on Testbox. On-device drive is exercised via the PR1/PR2 emulator proof; this PR is the agent-side tool + tests. Follow-up: computer-tool.ts has the same latent node-resolution ordering bug (explicit id searched only among eligible nodes); tracked separately. * fix(agents,phone-control): migrate mobile_ui arming hints/tests to gateway.nodes.commands.allow Match main's node-command arming rename (gateway.nodes.allowCommands -> gateway.nodes.commands.allow / commands.deny): update the mobile_ui tool arm-hint matchers to the gateway's current rejection strings and fix the phone-control / tool tests to the current config shape. Production write path was already correct. * fix(agents): register mobile_ui in owner-only denylist; trim phone-control under max-lines - tool-resolution.exclude.test.ts: mobile_ui joins the owner-only core tools, so add it to the expected plugin/inherited denylists. - phone-control: derive ArmGroup from a single ARM_GROUPS const and collapse parseGroup's ||-chain, dropping index.ts back under 700 lines. * fix(agents): centralize mobile_ui global-action names; regenerate tool-display snapshot - Extract GLOBAL_ACTION_NAMES const so the schema, action type, and validator share one source. This also removes the bare `name: "back"` type-annotation literal that scripts/tool-display.ts's name-regex was misreading as a phantom runtime tool. - Regenerate apps/.../tool-display.json to add the mobile_ui display entry (was missing from the Swift snapshot).