mirror of
https://github.com/openclaw/openclaw.git
synced 2026-07-17 03:01:34 +00:00
0a8e3604ba
* refactor(sessions): migrate runtime storage to sqlite * test(sessions): fix sqlite CI regressions * test(sessions): align remaining sqlite fixtures * fix(codex): require sqlite trajectory recorder * test(sessions): align orphan recovery sqlite fixture * test(sessions): align sqlite rebase fixtures * fix(sessions): finish current-main integration of the sqlite flip Resolve the whole-store SDK removal across its owner boundary: drop the loadSessionStore re-export and the registry whole-store wrappers, wire hasTrackedActiveSessionRun into gateway chat, complete the preserveLockedHarnessIds cleanup contract, flip the codex thread-history import to storePath targets, and port remaining main-side tests from file-store helpers to session accessor reads. * chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain bump gets its own review, and regenerate docs_map, the plugin SDK API baseline, and the export-surface ratchet for the merged tree. * feat(sessions): keep archived transcripts by default with zstd cold storage Codex-style retention: deleting or resetting a session archives its transcript as a zstd-compressed JSONL artifact (plain when the runtime lacks node:zlib zstd) and keeps it until the disk budget evicts oldest first. resetArchiveRetention now governs both deleted and reset archives and defaults to keep; maxDiskBytes defaults to 2gb so retention stays bounded, with archives evicted before live sessions. The cron reaper follows the same knob instead of deleting archives on its own timer. * fix(state): converge agent DB migration lineages and bound database growth Merge coherence: run both structure-gated legacy memory-schema repairs (flip-lineage drop, main-lineage identity rebuild) before the flip migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all converge, and hoist foreign_keys=OFF outside the schema transaction where the pragma was silently ignored and the v1 sessions rebuild cascade-deleted session_entries. Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL maintenance releases freed pages in bounded passes (never a blocking full VACUUM), and doctor reports state/agent DB bloat from freelist stats. * fix(codex): resolve the store path for thread-history import via the SDK The supervision catalog passed the legacy sessionFile locator to the storePath-targeted transcript mirror; resolve the agent store path with the session-store SDK helper instead of a runtime-object seam so test fakes and headless callers need no extra surface. Drop the obsolete missing-session-id preprocessing case: sessions rows are NOT NULL on session_id and upsert repairs id-less patches at write time. * fix(sessions): fail safe on malformed disk-budget config and doctor stat errors A malformed explicit maxDiskBytes disables the budget instead of falling back to the destructive 2gb default the user never chose, and the doctor bloat check skips databases whose paths stat-fail instead of aborting doctor. * fix(sessions): complete sqlite conflict translations * test(sqlite): align hardening checks with maintenance * test(sessions): inspect compressed transcript archives * fix(tests): await session seeds and drop unused helpers flagged by CI lint The five unawaited writeSessionStoreSeed calls raced their SQLite seeds against the assertions, failing compact shards; the bloat probe drops a useless initializer and the merged tests drop now-unused helpers. * test(sessions): type legacy proof events directly * test(sessions): align hardening contracts * perf(sessions): read usage transcript sizes from SQL aggregates Usage/cost scans walked every session and materialized every transcript event just to re-stringify it for a byte estimate — the #86718 stall class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes in SQLite without loading a single row. * fix(sessions): re-root foreign-root transcript paths onto the current sessions dir Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry absolute sessionFile paths from the old root; the containment fallback kept those foreign paths, so migration read (and would archive) files in the original root and reported local copies missing. Re-root the canonical agents/<id>/sessions suffix onto the current dir when the file exists there; genuine cross-root layouts still fall through unchanged. * test(agents): seed harness admission through sqlite * fix(sqlite): close agent db on pragma setup failure * fix(doctor): compact and retrofit incremental auto-vacuum after session import The migration is the sanctioned offline window: post-import compact reclaims import churn and applies auto_vacuum=INCREMENTAL to databases created before the fresh-DB pragma existed, so runtime maintenance can release pages in bounded passes on every install. --------- Co-authored-by: Peter Steinberger <steipete@gmail.com>
91 lines
7.2 KiB
Markdown
91 lines
7.2 KiB
Markdown
---
|
|
summary: "Health check commands and gateway health monitoring"
|
|
read_when:
|
|
- Diagnosing channel connectivity or gateway health
|
|
- Understanding health check CLI commands and options
|
|
title: "Health checks"
|
|
---
|
|
|
|
Short guide to verify channel connectivity without guessing.
|
|
|
|
## Quick checks
|
|
|
|
- `openclaw status` - local summary: gateway reachability/mode, update hint, linked channel auth age, sessions + recent activity.
|
|
- `openclaw status --all` - full local diagnosis (read-only, color, safe to paste for debugging).
|
|
- `openclaw status --deep` - asks the running gateway for a live probe (`health` with `probe:true`), including per-account channel probes when supported.
|
|
- `openclaw status --usage` - show model provider usage/quota snapshots.
|
|
- `openclaw health` - asks the running gateway for its health snapshot (WS-only; no direct channel sockets from the CLI).
|
|
- `openclaw health --verbose` (alias `--debug`) - forces a live health probe and prints gateway connection details.
|
|
- `openclaw health --json` - machine-readable health snapshot output.
|
|
- Send `/status` as a standalone chat command in any channel to get a status reply without invoking the agent.
|
|
- Logs: tail `/tmp/openclaw/openclaw-*.log` and filter for `web-heartbeat`, `web-reconnect`, `web-auto-reply`, `web-inbound`.
|
|
|
|
For Discord and other chat providers, session rows are not socket liveness.
|
|
`openclaw sessions`, Gateway `sessions.list`, and the agent `sessions_list` tool
|
|
read stored conversation state. A provider can reconnect and show healthy channel
|
|
status before any new session row is materialized. Use the channel status and
|
|
health commands above for live connectivity checks.
|
|
|
|
## Deep diagnostics
|
|
|
|
- Creds on disk: `ls -l ~/.openclaw/credentials/whatsapp/<accountId>/creds.json` (mtime should be recent).
|
|
- Session store: `ls -l ~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`. Count and recent recipients are surfaced via `status`.
|
|
- Relink flow: `openclaw channels logout && openclaw channels login --verbose` when status codes 409-515 or `loggedOut` appear in logs. The QR login flow auto-restarts once for status 515 after pairing.
|
|
- Diagnostics are enabled by default (`diagnostics.enabled: false` disables them). Memory events record RSS/heap byte counts and threshold/growth pressure; critical memory pressure logs through the gateway logger and, when `diagnostics.memoryPressureSnapshot: true` is set, also writes a pre-OOM stability bundle (V8 heap stats, Linux cgroup counters when available, active resource counts, largest session/transcript files by redacted relative path). Liveness warnings record event-loop delay/utilization, CPU-core ratio, and active/waiting/queued session counts when the process is running but saturated. Oversized-payload events record what was rejected/truncated/chunked plus sizes and limits, never message text, attachment contents, webhook bodies, raw request/response bodies, tokens, cookies, or secret values.
|
|
- The same heartbeat drives the bounded stability recorder: `openclaw gateway stability` (or the `diagnostics.stability` Gateway RPC). Fatal Gateway exits, shutdown timeouts, restart startup failures, and (when `diagnostics.memoryPressureSnapshot: true`) critical memory pressure persist the latest snapshot under `~/.openclaw/logs/stability/`. Inspect the newest bundle with `openclaw gateway stability --bundle latest`.
|
|
- For bug reports, run `openclaw gateway diagnostics export` and attach the generated zip: a Markdown summary, the newest stability bundle, sanitized log metadata, sanitized Gateway status/health snapshots, and config shape. Chat text, webhook bodies, tool outputs, credentials, cookies, account/message identifiers, and secret values are omitted or redacted. See [Diagnostics Export](/gateway/diagnostics).
|
|
|
|
## Health monitor config
|
|
|
|
- `gateway.channelHealthCheckMinutes`: how often the gateway checks channel health. Default: `5`. Set `0` to disable health-monitor restarts globally.
|
|
- `gateway.channelStaleEventThresholdMinutes`: how long a connected channel can stay idle before the health monitor treats it as stale and restarts it. Default: `30`. Keep this greater than or equal to `gateway.channelHealthCheckMinutes`.
|
|
- `gateway.channelMaxRestartsPerHour`: rolling one-hour cap for health-monitor restarts per channel/account. Default: `10`.
|
|
- `channels.<provider>.healthMonitor.enabled`: disable health-monitor restarts for a specific channel while leaving global monitoring enabled.
|
|
- `channels.<provider>.accounts.<accountId>.healthMonitor.enabled`: multi-account override that wins over the channel-level setting.
|
|
- These per-channel overrides apply to the built-in channels that expose them today: Discord, Google Chat, iMessage, IRC, Microsoft Teams, Signal, Slack, Telegram, and WhatsApp.
|
|
|
|
## Uptime monitoring
|
|
|
|
External uptime monitoring services should use the dedicated `/health` endpoint, not `/v1/chat/completions`.
|
|
|
|
- **DO use:** `GET /health` - instant response, no session created, no LLM call, returns `{"ok":true,"status":"live"}`
|
|
- **DON'T use:** `/v1/chat/completions` for health checks - each request creates a full agent session with skill snapshot, context assembly, and LLM calls
|
|
|
|
When no `x-openclaw-session-key` header or `user` field is provided, `/v1/chat/completions` generates a new random session for each request. Monitoring services that ping every 15 minutes create ~96 sessions/day, each consuming 4-22KB. Over time this causes session store bloat and can lead to context window overflow.
|
|
|
|
### Monitoring service setup examples
|
|
|
|
- **BetterStack:** Set health check URL to `https://<your-gateway-host>:<port>/health`
|
|
- **UptimeRobot:** Add a new HTTP monitor with URL `https://<your-gateway-host>:<port>/health`
|
|
- **Generic:** Any HTTP GET to `/health` returns 200 with `{"ok":true}` when the gateway is healthy
|
|
|
|
## When something fails
|
|
|
|
- `logged out` or status 409-515 -> relink with `openclaw channels logout` then `openclaw channels login`.
|
|
- Gateway unreachable -> start it: `openclaw gateway --port 18789` (use `--force` if the port is busy).
|
|
- No inbound messages -> confirm linked phone is online and the sender is allowed (`channels.whatsapp.allowFrom`); for group chats, ensure allowlist + mention rules match (`channels.whatsapp.groups`, `agents.list[].groupChat.mentionPatterns`).
|
|
|
|
## Dedicated "health" command
|
|
|
|
`openclaw health` asks the running gateway for its health snapshot (no direct channel
|
|
sockets from the CLI). By default it returns a fresh cached gateway snapshot and the
|
|
gateway refreshes that cache in the background; `--verbose` forces a live probe instead.
|
|
The command reports linked creds/auth age when available, per-channel probe summaries,
|
|
session-store summary, and probe duration. It exits non-zero if the gateway is
|
|
unreachable or the probe fails/times out.
|
|
|
|
Options:
|
|
|
|
- `--json`: machine-readable JSON output
|
|
- `--timeout <ms>`: override the default 10s probe timeout
|
|
- `--verbose`: force a live probe and print gateway connection details
|
|
- `--debug`: alias for `--verbose`
|
|
|
|
The health snapshot includes: `ok` (boolean), `ts` (timestamp), `durationMs` (probe time), per-channel status, agent availability, and session-store summary.
|
|
|
|
## Related
|
|
|
|
- [Gateway runbook](/gateway)
|
|
- [Diagnostics export](/gateway/diagnostics)
|
|
- [Gateway troubleshooting](/gateway/troubleshooting)
|