mirror of
https://github.com/openclaw/openclaw.git
synced 2026-07-09 21:23:55 +00:00
323 lines
16 KiB
Markdown
323 lines
16 KiB
Markdown
---
|
|
summary: "Android app (node): connection runbook + Connect/Chat/Voice/Canvas command surface"
|
|
read_when:
|
|
- Pairing or reconnecting the Android node
|
|
- Debugging Android gateway discovery or auth
|
|
- Mirroring or controlling an Android device from a remote Mac
|
|
- Verifying chat history parity across clients
|
|
title: "Android app"
|
|
---
|
|
|
|
<Note>
|
|
The official Android app is available on [Google Play](https://play.google.com/store/apps/details?id=ai.openclaw.app&hl=en_IN). It is a companion node and requires a running OpenClaw Gateway. Source: [apps/android](https://github.com/openclaw/openclaw/tree/main/apps/android) ([build instructions](https://github.com/openclaw/openclaw/blob/main/apps/android/README.md)).
|
|
</Note>
|
|
|
|
## Support snapshot
|
|
|
|
- Role: companion node app (Android does not host the Gateway).
|
|
- Gateway required: yes (run it on macOS, Linux, or Windows via WSL2).
|
|
- Install: [Google Play](https://play.google.com/store/apps/details?id=ai.openclaw.app&hl=en_IN) for the app, [Getting Started](/start/getting-started) for the Gateway, then [Pairing](/channels/pairing).
|
|
- Gateway: [Runbook](/gateway) + [Configuration](/gateway/configuration).
|
|
- Protocols: [Gateway protocol](/gateway/protocol) (nodes + control plane).
|
|
|
|
System control (launchd/systemd) lives on the Gateway host — see [Gateway](/gateway).
|
|
|
|
## Mirror and control Android from a remote Mac
|
|
|
|
[scrcpy](https://github.com/Genymobile/scrcpy) mirrors an Android screen in a macOS window and
|
|
forwards keyboard and pointer input through Android Debug Bridge (ADB). This is an operator-side
|
|
workflow, separate from the OpenClaw node connection. It is useful when the Android device and the
|
|
Mac are in different locations but share a private Tailscale network.
|
|
|
|
### Before you begin
|
|
|
|
- Install Tailscale on the Android device and the Mac, and connect both to the same tailnet.
|
|
- On Android, enable **Developer options** and **USB debugging**. Android 16 places **Wireless
|
|
debugging** under **Settings > System > Developer options**. See [Android developer
|
|
options](https://developer.android.com/studio/debug/dev-options).
|
|
- Install scrcpy and ADB on the Mac:
|
|
|
|
```bash
|
|
brew install scrcpy
|
|
brew install --cask android-platform-tools
|
|
```
|
|
|
|
- Keep the Android device available for the first connection. Android must approve each Mac's ADB
|
|
key before that Mac can control the device.
|
|
|
|
### Enable ADB over TCP
|
|
|
|
For the initial setup, connect the Android device by USB to a trusted computer and approve its
|
|
debugging prompt. Then run:
|
|
|
|
```bash
|
|
adb devices
|
|
adb tcpip 5555
|
|
```
|
|
|
|
You can now disconnect USB. If port 5555 stops listening after a device reboot or debugging reset,
|
|
repeat this local setup step. Android 11 and later can also establish the initial trust with
|
|
**Wireless debugging > Pair device with pairing code** and `adb pair`.
|
|
|
|
### Allow only the controller Mac
|
|
|
|
Tailnets with restrictive grants must explicitly allow the controller Mac to reach TCP port 5555
|
|
on the Android device. Add a narrow rule to the tailnet policy, replacing the example addresses
|
|
with the two devices' stable Tailscale IPs:
|
|
|
|
```json5
|
|
{
|
|
grants: [
|
|
{
|
|
src: ["<remote-mac-tailnet-ip>"],
|
|
dst: ["<android-tailnet-ip>"],
|
|
ip: ["tcp:5555"],
|
|
},
|
|
],
|
|
}
|
|
```
|
|
|
|
See [Tailscale grants](https://tailscale.com/docs/reference/syntax/grants) for host aliases and other
|
|
selectors. Do not grant this port to the public internet or expose it with Funnel: an authorized ADB
|
|
client has broad control of the device.
|
|
|
|
### Connect and start mirroring
|
|
|
|
On the remote Mac:
|
|
|
|
```bash
|
|
adb connect <android-tailnet-ip>:5555
|
|
adb devices
|
|
scrcpy --serial <android-tailnet-ip>:5555
|
|
```
|
|
|
|
The first `adb connect` from this Mac shows an authorization dialog on Android. Unlock the device,
|
|
confirm the key fingerprint, and select **Always allow from this computer** only when the Mac is
|
|
trusted. A successful `adb devices` entry ends in `device`; `unauthorized` means the on-device prompt
|
|
has not been approved.
|
|
|
|
Once the scrcpy window opens, use it directly or target it with a macOS screen-automation tool such
|
|
as [Peekaboo](https://peekaboo.sh/). scrcpy carries the display and input; Tailscale provides only the
|
|
private network path.
|
|
|
|
### Troubleshooting
|
|
|
|
- `Connection timed out`: verify the tailnet grant for TCP 5555. A successful `tailscale ping` proves
|
|
peer reachability, not that policy permits this TCP port. Test with
|
|
`nc -vz <android-tailnet-ip> 5555` from the Mac.
|
|
- `unauthorized`: unlock Android and approve the remote Mac's ADB key, or remove the stale workstation
|
|
under **Wireless debugging > Paired devices** and pair it again.
|
|
- `Connection refused`: reconnect locally and run `adb tcpip 5555` again.
|
|
- More than one device listed: keep the explicit `--serial <android-tailnet-ip>:5555` argument.
|
|
|
|
When finished, close scrcpy and disconnect ADB:
|
|
|
|
```bash
|
|
adb disconnect <android-tailnet-ip>:5555
|
|
```
|
|
|
|
## Connection runbook
|
|
|
|
Android node app ⇄ (mDNS/NSD + WebSocket) ⇄ **Gateway**
|
|
|
|
Android connects directly to the Gateway WebSocket and uses device pairing (`role: node`).
|
|
|
|
For Tailscale or public hosts, Android requires a secure endpoint:
|
|
|
|
- Preferred: Tailscale Serve / Funnel with `https://<magicdns>` / `wss://<magicdns>`
|
|
- Also supported: any other `wss://` Gateway URL with a real TLS endpoint
|
|
- Cleartext `ws://` remains supported on private LAN addresses / `.local` hosts, plus `localhost`, `127.0.0.1`, and the Android emulator bridge (`10.0.2.2`)
|
|
|
|
### Prerequisites
|
|
|
|
- Gateway running on another machine (or reachable via SSH).
|
|
- Android device/emulator can reach the gateway WebSocket:
|
|
- Same LAN with mDNS/NSD, **or**
|
|
- Same Tailscale tailnet using Wide-Area Bonjour / unicast DNS-SD (see below), **or**
|
|
- Manual gateway host/port (fallback)
|
|
- Tailnet/public mobile pairing does **not** use raw tailnet IP `ws://` endpoints. Use Tailscale Serve or another `wss://` URL instead.
|
|
- The `openclaw` CLI available on the gateway machine (or via SSH), to approve pairing requests.
|
|
|
|
### 1. Start the Gateway
|
|
|
|
```bash
|
|
openclaw gateway --port 18789 --verbose
|
|
```
|
|
|
|
Confirm in logs you see something like:
|
|
|
|
- `listening on ws://0.0.0.0:18789`
|
|
|
|
For remote Android access over Tailscale, prefer Serve/Funnel instead of a raw tailnet bind:
|
|
|
|
```bash
|
|
openclaw gateway --tailscale serve
|
|
```
|
|
|
|
This gives Android a secure `wss://` / `https://` endpoint. A plain `gateway.bind: "tailnet"` setup is not enough for first-time remote Android pairing unless you also terminate TLS separately.
|
|
|
|
### 2. Verify discovery (optional)
|
|
|
|
From the gateway machine:
|
|
|
|
```bash
|
|
dns-sd -B _openclaw-gw._tcp local.
|
|
```
|
|
|
|
More debugging notes: [Bonjour](/gateway/bonjour).
|
|
|
|
If you also configured a wide-area discovery domain, compare against:
|
|
|
|
```bash
|
|
openclaw gateway discover --json
|
|
```
|
|
|
|
That shows `local.` plus the configured wide-area domain in one pass, using the resolved service endpoint instead of TXT-only hints.
|
|
|
|
#### Cross-network discovery via unicast DNS-SD
|
|
|
|
Android NSD/mDNS discovery does not cross networks. If the Android node and the gateway are on different networks but connected via Tailscale, use Wide-Area Bonjour / unicast DNS-SD instead. Discovery alone is not sufficient for tailnet/public Android pairing — the discovered route still needs a secure endpoint (`wss://` or Tailscale Serve):
|
|
|
|
1. Set up a DNS-SD zone (example `openclaw.internal.`) on the gateway host and publish `_openclaw-gw._tcp` records.
|
|
2. Configure Tailscale split DNS for your chosen domain pointing at that DNS server.
|
|
|
|
Details and example CoreDNS config: [Bonjour](/gateway/bonjour).
|
|
|
|
### 3. Connect from Android
|
|
|
|
In the Android app:
|
|
|
|
- The app keeps its gateway connection alive via a **foreground service** (persistent notification).
|
|
- Open the **Connect** tab.
|
|
- Use **Setup Code** or **Manual** mode.
|
|
- If discovery is blocked, use manual host/port in **Advanced controls**. For private LAN hosts, `ws://` still works. For Tailscale/public hosts, turn on TLS and use a `wss://` / Tailscale Serve endpoint.
|
|
|
|
After the first successful pairing, Android auto-reconnects on launch: the manual endpoint (if enabled), otherwise the last discovered gateway (best-effort).
|
|
|
|
### Presence alive beacons
|
|
|
|
After the authenticated node session connects, and when the app moves to the background while the foreground service is still connected, Android calls `node.event` with `event: "node.presence.alive"`. The gateway records this as `lastSeenAtMs`/`lastSeenReason` on the paired node/device metadata only after the authenticated node device identity is known.
|
|
|
|
The app counts the beacon as successfully recorded only when the gateway response includes `handled: true`. Older gateways may acknowledge `node.event` with `{ "ok": true }`; that response is compatible but does not count as a durable last-seen update.
|
|
|
|
### 4. Approve pairing (CLI)
|
|
|
|
On the gateway machine:
|
|
|
|
```bash
|
|
openclaw devices list
|
|
openclaw devices approve <requestId>
|
|
openclaw devices reject <requestId>
|
|
```
|
|
|
|
Pairing details: [Pairing](/channels/pairing).
|
|
|
|
Optional: if the Android node always connects from a tightly controlled subnet, you can opt in to first-time node auto-approval with explicit CIDRs or exact IPs:
|
|
|
|
```json5
|
|
{
|
|
gateway: {
|
|
nodes: {
|
|
pairing: {
|
|
autoApproveCidrs: ["192.168.1.0/24"],
|
|
},
|
|
},
|
|
},
|
|
}
|
|
```
|
|
|
|
This is disabled by default. It applies only to fresh `role: node` pairing with no requested scopes. Operator/browser pairing and any role, scope, metadata, or public-key change still require manual approval.
|
|
|
|
### 5. Verify the node is connected
|
|
|
|
```bash
|
|
openclaw nodes status
|
|
openclaw gateway call node.list --params "{}"
|
|
```
|
|
|
|
### 6. Chat + history
|
|
|
|
The Android Chat tab supports session selection (default `main`, plus other existing sessions):
|
|
|
|
- History: `chat.history` (display-normalized — inline directive tags, plain-text tool-call XML payloads (`<tool_call>`, `<function_call>`, `<tool_calls>`, `<function_calls>`, and truncated variants), and leaked ASCII/full-width model control tokens are stripped; silent-token assistant rows such as exact `NO_REPLY` / `no_reply` are omitted; oversized rows can be replaced with placeholders)
|
|
- Send: `chat.send`
|
|
- Push updates (best-effort): `chat.subscribe` -> `event:"chat"`
|
|
|
|
### 7. Canvas + camera
|
|
|
|
#### Gateway Canvas Host (recommended for web content)
|
|
|
|
To have the node show real HTML/CSS/JS that the agent can edit on disk, point the node at the Gateway canvas host.
|
|
|
|
<Note>
|
|
Nodes load canvas from the Gateway HTTP server (same port as `gateway.port`, default `18789`).
|
|
</Note>
|
|
|
|
1. Create `~/.openclaw/workspace/canvas/index.html` on the gateway host.
|
|
2. Navigate the node to it (LAN):
|
|
|
|
```bash
|
|
openclaw nodes invoke --node "<Android Node>" --command canvas.navigate --params '{"url":"http://<gateway-hostname>.local:18789/__openclaw__/canvas/"}'
|
|
```
|
|
|
|
Tailnet (optional): if both devices are on Tailscale, use a MagicDNS name or tailnet IP instead of `.local`, e.g. `http://<gateway-magicdns>:18789/__openclaw__/canvas/`.
|
|
|
|
This server injects a live-reload client into HTML and reloads on file changes. The Gateway also serves `/__openclaw__/a2ui/`, but the Android app treats remote A2UI pages as render-only. Action-capable A2UI commands use the bundled app-owned A2UI page.
|
|
|
|
Canvas commands (foreground only):
|
|
|
|
- `canvas.eval`, `canvas.snapshot`, `canvas.navigate` (use `{"url":""}` or `{"url":"/"}` to return to the default scaffold). `canvas.snapshot` returns `{ format, base64 }` (default `format="jpeg"`).
|
|
- A2UI: `canvas.a2ui.push`, `canvas.a2ui.reset` (`canvas.a2ui.pushJSONL` legacy alias). These use the bundled app-owned A2UI page for action-capable rendering.
|
|
|
|
Camera commands (foreground only; permission-gated): `camera.snap` (jpg), `camera.clip` (mp4). See [Camera node](/nodes/camera) for parameters and CLI helpers.
|
|
|
|
### 8. Voice + expanded Android command surface
|
|
|
|
- Voice tab: Android has two explicit capture modes. **Mic** is a manual Voice-tab session that sends each pause as a chat turn and stops when the app leaves the foreground or the user leaves the Voice tab. **Talk** is continuous Talk Mode and keeps listening until toggled off or the node disconnects.
|
|
- Talk Mode promotes the existing foreground service from `connectedDevice` to `connectedDevice|microphone` before capture starts, then demotes it when Talk Mode stops. The node service declares `FOREGROUND_SERVICE_CONNECTED_DEVICE` with `CHANGE_NETWORK_STATE`; Android 14+ also requires the `FOREGROUND_SERVICE_MICROPHONE` declaration, the `RECORD_AUDIO` runtime grant, and the microphone service type at runtime.
|
|
- By default, Android Talk uses native speech recognition, Gateway chat, and `talk.speak` through the configured gateway Talk provider. Local system TTS is used only when `talk.speak` is unavailable.
|
|
- Android Talk uses realtime Gateway relay only when `talk.realtime.mode` is `realtime` and `talk.realtime.transport` is `gateway-relay`.
|
|
- Voice wake is implemented in source (`VoiceWakeMode`) but the shipping app runtime always forces it to `off` on connect — there is no user-facing toggle today.
|
|
- Additional Android command families (availability depends on device, permissions, and user settings):
|
|
- `device.status`, `device.info`, `device.permissions`, `device.health`
|
|
- `device.apps` only when **Settings > Phone Capabilities > Installed Apps** is enabled; it lists launcher-visible apps by default (pass `includeNonLaunchable` for the full list).
|
|
- `notifications.list`, `notifications.actions` (see [Notification forwarding](#notification-forwarding) below)
|
|
- `photos.latest`
|
|
- `contacts.search`, `contacts.add`
|
|
- `calendar.events`, `calendar.add`
|
|
- `callLog.search`
|
|
- `sms.search`
|
|
- `motion.activity`, `motion.pedometer`
|
|
|
|
## Assistant entrypoints
|
|
|
|
Android supports launching OpenClaw from the system assistant trigger (Google Assistant). Holding the home button (or another `ACTION_ASSIST` trigger) opens the app; saying "Hey Google, ask OpenClaw `<prompt>`" matches the app's declared App Actions query pattern and hands the prompt into the chat composer without auto-sending it.
|
|
|
|
This uses Android **App Actions** (`shortcuts.xml` capability) declared in the app manifest. No gateway-side configuration is needed — the assistant intent is handled entirely by the Android app.
|
|
|
|
<Note>
|
|
App Actions availability depends on the device, Google Play Services version, and whether the user has set OpenClaw as the default assistant app.
|
|
</Note>
|
|
|
|
## Notification forwarding
|
|
|
|
Android can forward device notifications to the gateway as `node.event` items. This is configured **on the device**, in the app's Settings sheet — not in gateway/`openclaw.json` config.
|
|
|
|
| Setting | Description |
|
|
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
| Forward Notification Events | Master toggle. Off by default; requires Notification Listener Access to be granted first. |
|
|
| Package Filter | **Allowlist** (only listed package IDs forwarded) or **Blocklist** (default: all packages except listed IDs). OpenClaw's own package is always excluded in Blocklist mode to prevent forwarding loops. |
|
|
| Quiet Hours | Local HH:mm start/end window that suppresses forwarding. Disabled by default; defaults to `22:00`-`07:00` once enabled. |
|
|
| Max Events / Minute | Per-device rate limit on forwarded notifications. Default 20. |
|
|
| Route Session Key | Optional. Pins forwarded notification events into a specific session instead of the device's default notification route. |
|
|
|
|
<Note>
|
|
Notification forwarding requires the Android Notification Listener permission. The app prompts for this during setup.
|
|
</Note>
|
|
|
|
## Related
|
|
|
|
- [iOS app](/platforms/ios)
|
|
- [Nodes](/nodes)
|
|
- [Android node troubleshooting](/nodes/troubleshooting)
|