openclaw/docs/gateway/health.md at 7303f429177b04c0465bf697da170fa91d5e2854

mirror of https://github.com/openclaw/openclaw.git synced 2026-07-15 06:26:04 +00:00

Files

Josh Lehman 0a8e3604ba refactor: flip sessions and transcripts to sqlite storage (#98236 )

* refactor(sessions): migrate runtime storage to sqlite

* test(sessions): fix sqlite CI regressions

* test(sessions): align remaining sqlite fixtures

* fix(codex): require sqlite trajectory recorder

* test(sessions): align orphan recovery sqlite fixture

* test(sessions): align sqlite rebase fixtures

* fix(sessions): finish current-main integration of the sqlite flip

Resolve the whole-store SDK removal across its owner boundary: drop the
loadSessionStore re-export and the registry whole-store wrappers, wire
hasTrackedActiveSessionRun into gateway chat, complete the
preserveLockedHarnessIds cleanup contract, flip the codex thread-history
import to storePath targets, and port remaining main-side tests from
file-store helpers to session accessor reads.

* chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs

Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore
the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain
bump gets its own review, and regenerate docs_map, the plugin SDK API baseline,
and the export-surface ratchet for the merged tree.

* feat(sessions): keep archived transcripts by default with zstd cold storage

Codex-style retention: deleting or resetting a session archives its
transcript as a zstd-compressed JSONL artifact (plain when the runtime
lacks node:zlib zstd) and keeps it until the disk budget evicts oldest
first. resetArchiveRetention now governs both deleted and reset archives
and defaults to keep; maxDiskBytes defaults to 2gb so retention stays
bounded, with archives evicted before live sessions. The cron reaper
follows the same knob instead of deleting archives on its own timer.

* fix(state): converge agent DB migration lineages and bound database growth

Merge coherence: run both structure-gated legacy memory-schema repairs
(flip-lineage drop, main-lineage identity rebuild) before the flip
migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all
converge, and hoist foreign_keys=OFF outside the schema transaction
where the pragma was silently ignored and the v1 sessions rebuild
cascade-deleted session_entries.

Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL
maintenance releases freed pages in bounded passes (never a blocking
full VACUUM), and doctor reports state/agent DB bloat from freelist
stats.

* fix(codex): resolve the store path for thread-history import via the SDK

The supervision catalog passed the legacy sessionFile locator to the
storePath-targeted transcript mirror; resolve the agent store path with
the session-store SDK helper instead of a runtime-object seam so test
fakes and headless callers need no extra surface. Drop the obsolete
missing-session-id preprocessing case: sessions rows are NOT NULL on
session_id and upsert repairs id-less patches at write time.

* fix(sessions): fail safe on malformed disk-budget config and doctor stat errors

A malformed explicit maxDiskBytes disables the budget instead of
falling back to the destructive 2gb default the user never chose, and
the doctor bloat check skips databases whose paths stat-fail instead of
aborting doctor.

* fix(sessions): complete sqlite conflict translations

* test(sqlite): align hardening checks with maintenance

* test(sessions): inspect compressed transcript archives

* fix(tests): await session seeds and drop unused helpers flagged by CI lint

The five unawaited writeSessionStoreSeed calls raced their SQLite seeds
against the assertions, failing compact shards; the bloat probe drops a
useless initializer and the merged tests drop now-unused helpers.

* test(sessions): type legacy proof events directly

* test(sessions): align hardening contracts

* perf(sessions): read usage transcript sizes from SQL aggregates

Usage/cost scans walked every session and materialized every transcript
event just to re-stringify it for a byte estimate — the #86718 stall
class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes
in SQLite without loading a single row.

* fix(sessions): re-root foreign-root transcript paths onto the current sessions dir

Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry
absolute sessionFile paths from the old root; the containment fallback
kept those foreign paths, so migration read (and would archive) files in
the original root and reported local copies missing. Re-root the
canonical agents/<id>/sessions suffix onto the current dir when the file
exists there; genuine cross-root layouts still fall through unchanged.

* test(agents): seed harness admission through sqlite

* fix(sqlite): close agent db on pragma setup failure

* fix(doctor): compact and retrofit incremental auto-vacuum after session import

The migration is the sanctioned offline window: post-import compact
reclaims import churn and applies auto_vacuum=INCREMENTAL to databases
created before the fresh-DB pragma existed, so runtime maintenance can
release pages in bounded passes on every install.

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>

2026-07-11 14:50:37 -07:00

7.2 KiB

Raw Blame History

summary, read_when, title

summary

read_when

title

Health check commands and gateway health monitoring

Diagnosing channel connectivity or gateway health

Understanding health check CLI commands and options

Health checks

Short guide to verify channel connectivity without guessing.

Quick checks

openclaw status - local summary: gateway reachability/mode, update hint, linked channel auth age, sessions + recent activity.
openclaw status --all - full local diagnosis (read-only, color, safe to paste for debugging).
openclaw status --deep - asks the running gateway for a live probe (health with probe:true), including per-account channel probes when supported.
openclaw status --usage - show model provider usage/quota snapshots.
openclaw health - asks the running gateway for its health snapshot (WS-only; no direct channel sockets from the CLI).
openclaw health --verbose (alias --debug) - forces a live health probe and prints gateway connection details.
openclaw health --json - machine-readable health snapshot output.
Send /status as a standalone chat command in any channel to get a status reply without invoking the agent.
Logs: tail /tmp/openclaw/openclaw-*.log and filter for web-heartbeat, web-reconnect, web-auto-reply, web-inbound.

For Discord and other chat providers, session rows are not socket liveness. openclaw sessions, Gateway sessions.list, and the agent sessions_list tool read stored conversation state. A provider can reconnect and show healthy channel status before any new session row is materialized. Use the channel status and health commands above for live connectivity checks.

Deep diagnostics

Creds on disk: ls -l ~/.openclaw/credentials/whatsapp/<accountId>/creds.json (mtime should be recent).
Session store: ls -l ~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite. Count and recent recipients are surfaced via status.
Relink flow: openclaw channels logout && openclaw channels login --verbose when status codes 409-515 or loggedOut appear in logs. The QR login flow auto-restarts once for status 515 after pairing.
Diagnostics are enabled by default (diagnostics.enabled: false disables them). Memory events record RSS/heap byte counts and threshold/growth pressure; critical memory pressure logs through the gateway logger and, when diagnostics.memoryPressureSnapshot: true is set, also writes a pre-OOM stability bundle (V8 heap stats, Linux cgroup counters when available, active resource counts, largest session/transcript files by redacted relative path). Liveness warnings record event-loop delay/utilization, CPU-core ratio, and active/waiting/queued session counts when the process is running but saturated. Oversized-payload events record what was rejected/truncated/chunked plus sizes and limits, never message text, attachment contents, webhook bodies, raw request/response bodies, tokens, cookies, or secret values.
The same heartbeat drives the bounded stability recorder: openclaw gateway stability (or the diagnostics.stability Gateway RPC). Fatal Gateway exits, shutdown timeouts, restart startup failures, and (when diagnostics.memoryPressureSnapshot: true) critical memory pressure persist the latest snapshot under ~/.openclaw/logs/stability/. Inspect the newest bundle with openclaw gateway stability --bundle latest.
For bug reports, run openclaw gateway diagnostics export and attach the generated zip: a Markdown summary, the newest stability bundle, sanitized log metadata, sanitized Gateway status/health snapshots, and config shape. Chat text, webhook bodies, tool outputs, credentials, cookies, account/message identifiers, and secret values are omitted or redacted. See Diagnostics Export.

Health monitor config

gateway.channelHealthCheckMinutes: how often the gateway checks channel health. Default: 5. Set 0 to disable health-monitor restarts globally.
gateway.channelStaleEventThresholdMinutes: how long a connected channel can stay idle before the health monitor treats it as stale and restarts it. Default: 30. Keep this greater than or equal to gateway.channelHealthCheckMinutes.
gateway.channelMaxRestartsPerHour: rolling one-hour cap for health-monitor restarts per channel/account. Default: 10.
channels.<provider>.healthMonitor.enabled: disable health-monitor restarts for a specific channel while leaving global monitoring enabled.
channels.<provider>.accounts.<accountId>.healthMonitor.enabled: multi-account override that wins over the channel-level setting.
These per-channel overrides apply to the built-in channels that expose them today: Discord, Google Chat, iMessage, IRC, Microsoft Teams, Signal, Slack, Telegram, and WhatsApp.

Uptime monitoring

External uptime monitoring services should use the dedicated /health endpoint, not /v1/chat/completions.

DO use: GET /health - instant response, no session created, no LLM call, returns {"ok":true,"status":"live"}
DON'T use: /v1/chat/completions for health checks - each request creates a full agent session with skill snapshot, context assembly, and LLM calls

When no x-openclaw-session-key header or user field is provided, /v1/chat/completions generates a new random session for each request. Monitoring services that ping every 15 minutes create ~96 sessions/day, each consuming 4-22KB. Over time this causes session store bloat and can lead to context window overflow.

Monitoring service setup examples

BetterStack: Set health check URL to https://<your-gateway-host>:<port>/health
UptimeRobot: Add a new HTTP monitor with URL https://<your-gateway-host>:<port>/health
Generic: Any HTTP GET to /health returns 200 with {"ok":true} when the gateway is healthy

When something fails

logged out or status 409-515 -> relink with openclaw channels logout then openclaw channels login.
Gateway unreachable -> start it: openclaw gateway --port 18789 (use --force if the port is busy).
No inbound messages -> confirm linked phone is online and the sender is allowed (channels.whatsapp.allowFrom); for group chats, ensure allowlist + mention rules match (channels.whatsapp.groups, agents.list[].groupChat.mentionPatterns).

Dedicated "health" command

openclaw health asks the running gateway for its health snapshot (no direct channel sockets from the CLI). By default it returns a fresh cached gateway snapshot and the gateway refreshes that cache in the background; --verbose forces a live probe instead. The command reports linked creds/auth age when available, per-channel probe summaries, session-store summary, and probe duration. It exits non-zero if the gateway is unreachable or the probe fails/times out.

Options:

--json: machine-readable JSON output
--timeout <ms>: override the default 10s probe timeout
--verbose: force a live probe and print gateway connection details
--debug: alias for --verbose

The health snapshot includes: ok (boolean), ts (timestamp), durationMs (probe time), per-channel status, agent availability, and session-store summary.

7.2 KiB Raw Blame History