vultr/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-08-02 09:51:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
ebb301c32d9e2e1ec61baa783564eb38e4296895
openclaw/docs/gateway/opentelemetry.md
Jesse Merhi f1ee2a3098 fix(diagnostics-otel): agent turns split into one-span traces instead of a trace tree (#116246) 
* fix(diagnostics-otel): keep parent span links across long turns

Retained trusted span contexts translate a completed lifecycle span's
diagnostic span id to the real OTel span context. That table was evicted
5s after run.completed, so children of a longer turn missed their parent
and OTel minted a fresh trace id for each one, silently splitting a turn
into single-span traces.

Retention is now identity-based and lives for the service lifetime,
bounded by MAX_RETAINED_TRUSTED_SPAN_CONTEXTS, which removes the timing
race and the drain/timeout cleanup subsystem.

Also nests openclaw.exec under its run: the exec event now carries a
trusted trace context and the recorder resolves the ambient run scope.

* fix(diagnostics-otel): key retained span contexts by diagnostic trace id

Retained lifecycle span contexts were stored under the span's OTel trace id
but looked up with the event's diagnostic trace id. Those are different id
spaces, so every post-completion parent lookup missed and each straggler
span started a brand-new trace.

The mocked suite could not catch this: its tracer reports the same trace id
the test feeds in, collapsing both id spaces into one value. Adds a boundary
test that runs the real OTel SDK and OTLP/protobuf exporter against a local
receiver and asserts on exported span bytes.

* fix(diagnostics-otel): harden span-linking contract after review

Review findings on the parent-span linking fix:

- completeTrackedLifecycleSpan now takes the DiagnosticTraceContext instead of a
  bare trace id beside a span id. Two adjacent hex strings let a transposition
  compile clean and silently reinstate split traces; DiagnosticTraceContext is
  the only shape an OTel SpanContext cannot satisfy (traceFlags string vs number).
- internalOrTrustedTraceContext composes normalizedTrustedTraceContext rather
  than spelling the trust predicate a third time.
- Corrected comments that claimed OTel ids never equal diagnostic ids. Spans
  parented from an upstream traceparent do adopt the diagnostic trace id; the
  removed guard only ever misfired on root lifecycle spans.
- Exec comment now states the real parent: the openclaw harness opens no run
  scope, so exec nests under openclaw.harness.run.

Test gaps closed:

- The core emitter switch was untested; reverting it kept every test green.
  The pty-fallback test now asserts the emitted metadata.
- The boundary test covered only run.completed of three lifecycle owners; it now
  covers harness.run.completed and message.processed too, each with a straggler.
- Dropped the service-context cast for the typed helper, which caught a bogus
  context.assembled field, and moved teardown into afterEach so a failed
  assertion cannot leak the real NodeSDK across files.
- Scoped the one-trace assertion to this turn's spans; faked Date so a lazy
  Date.now expiry cannot pass.

Docs: exec and tool spans now listed as staying on an upstream request trace.

* fix(diagnostics-otel): keep late children on the trace on every terminator

Native review found three correctness gaps in the previous commit:

- Exec spans could be exported naming a parent span id nothing ever emits. The
  ambient lookup fell through to a remote-parent fallback built from diagnostic
  ids, which is a regression from the previous clean-root behavior and breaks
  waterfalls and parent-id-keyed backends. Exec now resolves only spans this
  process exported and stays a root on a miss; message spans keep the remote
  fallback because their context can come from an inbound traceparent.
- harness.run.error never retained its span context, so an aborted turn still
  split. For the openclaw harness that span is the only ancestor a late child
  has, and aborted turns emit no run.completed.
- completeTrackedLifecycleSpan no longer takes a redundant spanId.

The boundary test now runs through the OPENCLAW_OTEL_PRELOADED seam with an
in-memory exporter and releases the global tracer provider in teardown. A
NodeSDK cannot be unregistered, so the old version would have made any later
real-SDK test silently export nothing. That also deletes the hand-rolled OTLP
protobuf decoder. Added guards for the aborted-turn path and for exec staying
parentless; both are mutation-verified.

* fix(diagnostics-otel): age retained parent contexts out after 10 minutes

Retention had no time bound at all, so a command backgrounded by a turn could
finish hours later and still attach to a parent that ended long before. Backends
derive trace latency from the span envelope, so a 30s turn rendered as hours with
a child extending past its parent.

The horizon is checked lazily on lookup rather than on a timer: a timer firing
mid-turn is exactly what silently split traces before. Past the horizon a
straggler starts its own trace, which only loses causality for a span that would
have skewed its parent's duration anyway.

The two bounds now pin each other - shrinking the horizon to the old 5s window
fails the original regression tests, and removing it fails the new horizon test.

* fix(diagnostics-otel): retain parent contexts until capacity
2026-07-30 20:41:41 +10:00

36 KiB
Raw Blame History

summary, title, read_when
summary title read_when
Export OpenClaw diagnostics to OpenTelemetry collectors or stdout JSONL via the diagnostics-otel plugin OpenTelemetry export
You want to send OpenClaw model usage, message flow, or session metrics to an OpenTelemetry collector
You are wiring traces, metrics, or logs into Grafana, Datadog, Honeycomb, New Relic, Tempo, or another OTLP backend
You need the exact metric names, span names, or attribute shapes to build dashboards or alerts

OpenClaw exports diagnostics through the official diagnostics-otel plugin using OTLP/HTTP (protobuf). Logs can also be written as stdout JSONL for container and sandbox log pipelines. Any collector or backend that accepts OTLP/HTTP works without code changes. For local file logs, see Logging.

  • Diagnostics events are structured, in-process records emitted by the Gateway and bundled plugins for model runs, message flow, sessions, queues, and exec.
  • diagnostics-otel subscribes to those events and exports them as OpenTelemetry metrics, traces, and logs over OTLP/HTTP, and can mirror log records to stdout JSONL.
  • Provider calls receive a W3C traceparent header from OpenClaw's trusted model-call span context when the provider transport accepts custom headers. Plugin-emitted trace context is not propagated.
  • Exporters attach only when both the diagnostics surface and the plugin are enabled, so in-process cost stays near zero by default.

Quick start

openclaw plugins install clawhub:@openclaw/diagnostics-otel

{
  plugins: {
    allow: ["diagnostics-otel"],
    entries: {
      "diagnostics-otel": { enabled: true },
    },
  },
  diagnostics: {
    enabled: true,
    otel: {
      enabled: true,
      endpoint: "http://otel-collector:4318",
      protocol: "http/protobuf",
      serviceName: "openclaw-gateway",
      traces: true,
      metrics: true,
      logs: true,
      sampleRate: 0.2,
      flushIntervalMs: 60000,
    },
  },
}

Or enable the plugin from the CLI: openclaw plugins enable diagnostics-otel.

`protocol` supports `http/protobuf` only. Since `traces` and `metrics` default to enabled, any other value (including `grpc`) aborts the entire diagnostics-otel subscription with an `unsupported protocol` warning - this also stops stdout log export. Explicitly set `traces: false` and `metrics: false` if you only want `logsExporter: "stdout"` with a non-OTLP protocol value.

Signals exported

Signal What goes in it
Metrics Counters/histograms for token usage, cost, run duration, failover, skill usage, message flow, Talk events, queue lanes, session state/recovery, tool execution, exec, memory, liveness, and exporter health.
Traces Spans for model usage, model calls, harness lifecycle, skill usage, tool execution, exec, webhook/message processing, context assembly, and tool loops.
Logs Structured logging.file records exported over OTLP or stdout JSONL when diagnostics.otel.logs is enabled; log bodies are withheld unless content capture is explicitly enabled.

Toggle traces, metrics, and logs independently. Traces and metrics default to on when diagnostics.otel.enabled is true; logs default to off and export only when diagnostics.otel.logs is explicitly true. Log export defaults to OTLP; set diagnostics.otel.logsExporter to stdout for JSONL on stdout, or both for both.

Configuration reference

{
  diagnostics: {
    enabled: true,
    otel: {
      enabled: true,
      endpoint: "http://otel-collector:4318",
      tracesEndpoint: "http://otel-collector:4318/v1/traces",
      metricsEndpoint: "http://otel-collector:4318/v1/metrics",
      logsEndpoint: "http://otel-collector:4318/v1/logs",
      protocol: "http/protobuf", // grpc disables OTLP export
      serviceName: "openclaw-gateway", // unset falls back to OTEL_SERVICE_NAME, then "openclaw"
      headers: { "x-collector-token": "..." },
      traces: true,
      metrics: true,
      logs: true,
      logsExporter: "otlp", // otlp | stdout | both
      sampleRate: 0.2, // root-span sampler, 0.0..1.0
      flushIntervalMs: 60000, // metric export interval (min 1000ms)
      captureContent: false,
    },
  },
}

Environment variables

Variable Purpose
OTEL_EXPORTER_OTLP_ENDPOINT Fallback for diagnostics.otel.endpoint when the config key is unset.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT / OTEL_EXPORTER_OTLP_METRICS_ENDPOINT / OTEL_EXPORTER_OTLP_LOGS_ENDPOINT Signal-specific endpoint fallbacks used when the matching diagnostics.otel.*Endpoint config key is unset. Signal-specific config wins over signal-specific env, which wins over the shared endpoint.
OTEL_SERVICE_NAME Fallback for diagnostics.otel.serviceName when the config key is unset. Default service name is openclaw.
OTEL_EXPORTER_OTLP_PROTOCOL Fallback for the wire protocol when diagnostics.otel.protocol is unset. Only http/protobuf enables export.
OTEL_SEMCONV_STABILITY_OPT_IN Set to gen_ai_latest_experimental to emit the latest GenAI inference span shape: {gen_ai.operation.name} {gen_ai.request.model} span names, CLIENT span kind, and gen_ai.provider.name instead of the legacy gen_ai.system. GenAI metrics always use bounded, low-cardinality attributes regardless.
OPENCLAW_OTEL_PRELOADED Set to 1 when another preload or host process already registered the global OpenTelemetry SDK. The plugin then skips its own NodeSDK lifecycle but still wires diagnostic listeners and honors traces/metrics/logs.

Continue an upstream WebSocket trace

An authenticated Gateway WebSocket client can attach a W3C traceparent to each request frame:

{
  "type": "req",
  "id": "eval-item-42",
  "method": "agent",
  "params": {},
  "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
}

The Gateway creates a child request context that preserves the upstream trace ID and sampling flags. Agent, harness, model-call, provider, tool-execution, and exec spans created inside the request remain on that trace, including spans recorded after their parent run has already finished. This allows a local experiment runner to create one Langfuse/OpenTelemetry trace per dataset item and correlate the corresponding OpenClaw execution.

Trace context is request-scoped, not connection-scoped. On a long-lived WebSocket, generate or inject the appropriate traceparent independently for every RPC. Concurrent requests remain isolated even when their work interleaves.

The field is accepted only after the existing Gateway authentication handshake and does not affect authentication or method authorization. A traceparent on the initial connect frame is ignored. Missing or syntactically malformed values within the 128-character field limit silently fall back to a fresh request trace; longer values make the request frame invalid. tracestate and baggage are not accepted by the Gateway WebSocket protocol.

Privacy and content capture

Raw model/tool content is not exported by default. Spans carry bounded identifiers (channel, provider, model, error category, hash-only request ids, tool source, tool owner, skill name/source) and never include prompt text, response text, tool inputs, tool outputs, skill file paths, or session keys. Values that look like scoped agent session keys (for example starting with agent:) are replaced with unknown on low-cardinality attributes. OTLP log records keep severity, logger, code location, trusted trace context, and sanitized attributes by default; the raw log message body is exported only when diagnostics.otel.captureContent is true. Talk metrics export only bounded event metadata (mode, transport, provider, event type) - no transcripts, audio payloads, session ids, turn ids, call ids, room ids, or handoff tokens.

Outbound model requests may include a W3C traceparent header generated only from OpenClaw-owned diagnostic trace context for the active model call. Existing caller-supplied traceparent headers are replaced, so plugins or custom provider options cannot spoof cross-service trace ancestry.

Set diagnostics.otel.captureContent to true only when your collector and retention policy are approved for prompt, response, tool, and tool-definition text. This enables bounded, redacted input messages, output messages, tool inputs, tool outputs, tool definitions, and OTLP log bodies. System prompts remain excluded. Provider-internal thinking and redacted_thinking payloads are also excluded: compatibility attributes retain only a redacted structural marker, while GenAI message attributes omit those parts.

toolInputs/toolOutputs content is captured for the built-in agent runtime's tool executions (openclaw.content.tool_input and gen_ai.tool.call.arguments on completed/error spans; openclaw.content.tool_output and gen_ai.tool.call.result on completed spans). The openclaw.content.* names remain the stable OpenClaw attribute names; the gen_ai.tool.call.* copies mirror them for semconv-native viewers. External harness tool calls (Codex, Claude CLI) emit tool.execution.* spans without content payloads. Captured content travels on a trusted, listener-only channel and is never placed on the public diagnostic event bus.

Sampling and flushing

  • Traces: diagnostics.otel.sampleRate sets a TraceIdRatioBasedSampler on the root span only (0.0 drops all, 1.0 keeps all). Unset uses the OpenTelemetry SDK default (always-on).
  • Metrics: diagnostics.otel.flushIntervalMs (clamped to a minimum of 1000); unset uses the SDK's periodic-export default.
  • Logs: OTLP logs respect logging.level (file log level) and use the diagnostic log-record redaction path, not console formatting. High-volume installs should prefer OTLP collector sampling/filtering over local sampling. Set diagnostics.otel.logsExporter: "stdout" when your platform already ships stdout/stderr to a log processor and you have no OTLP logs collector. Stdout records are one JSON object per line with ts, signal, service.name, severity, body, redacted attributes, and trusted trace fields when available.
  • File-log correlation: JSONL file logs include top-level traceId, spanId, parentSpanId, and traceFlags when the log call carries a valid diagnostic trace context, letting log processors join local log lines with exported spans.
  • Request correlation: Gateway HTTP requests and WebSocket frames create an internal request trace scope. Logs and diagnostic events inside that scope inherit the request trace by default, while agent run and model-call spans are created as children so provider traceparent headers stay on the same trace.
  • Model-call correlation: openclaw.model.call spans include safe prompt component sizes by default and per-call token attributes when the provider result exposes usage. openclaw.model.usage remains the run-level accounting span for aggregate cost, context, and channel dashboards, and stays on the same diagnostic trace when the emitting runtime has trusted trace context.

Model-call observation units

Every openclaw.model.call span identifies what its lifecycle measures through openclaw.model_call.observation_unit:

  • request - one observable model/provider request. Native embedded model calls use this unit, and exporters treat a missing value as request for compatibility with older or external emitters.
  • turn - one opaque agent CLI turn that may contain hidden model requests, retries, tool work, or background work. Claude Code CLI and Codex app-server calls use this unit.

Both units remain model-call spans so trace backends can render model input, output, usage, and hierarchy. Request spans use the API-derived GenAI operation (chat, generate_content, or text_completion), while turn spans use gen_ai.operation.name = invoke_agent. Both contribute to gen_ai.client.operation.duration, where the operation name keeps direct request latency separate from full-turn latency. OpenClaw's OTEL model-call metrics also include openclaw.model_call.observation_unit; the Prometheus model-call metrics expose the equivalent observation_unit label.

Claude Code CLI model-call fidelity

Claude Code CLI turns emit one synthetic, turn-level openclaw.model.call span. These are not Anthropic HTTP request spans. They use openclaw.api = claude-code, openclaw.model_call.observation_unit = turn, and identify the operation as gen_ai.operation.name = invoke_agent. They identify OpenClaw's CLI boundary through openclaw.transport:

  • stdio - one-shot local Claude Code process.
  • stdio-live - one turn on a managed persistent Claude stdio session.
  • paired-node-cli - one-shot Claude Code execution delegated to a paired node.

Claude CLI diagnostics are instantiated only while the process diagnostic dispatcher is enabled and an internal or trusted event listener is attached. With no observability plugin or other listener active, Claude CLI turns skip the synthetic trace hierarchy, content buffers, and diagnostic stream-byte accounting. When content capture is enabled, prompt and system-prompt fields are capped at 128 KiB each; assistant output is capped at 128 KiB across at most 200 envelopes, with 16 KiB and one item reserved for a final visible fallback response. A marker records truncation when the limit is reached.

OpenClaw gives Claude CLI turns the same ownership hierarchy used by other agent runtimes: openclaw.harness.run (openclaw.harness.id = claude-cli) contains openclaw.run, which contains the Claude openclaw.model.call span. The harness and run spans are synthetic OpenClaw turn boundaries, not Claude Code internal phases. One-shot and managed stdio turns use the same hierarchy; a real fresh-session retry creates another model-call child inside the same OpenClaw run.

The span starts when OpenClaw admits the prepared CLI turn and ends only after that turn succeeds or fails. For managed sessions, an interim success result does not end the span while Claude reports result-holding background agents or workflows; the final post-drain result does. Abort, timeout, process failure, output/parse failure, and other turn failures end the same span with an error.

Claude Code reports per-assistant-message usage and may also report cumulative usage on its terminal result. OpenClaw reply accounting continues to use the last assistant message so existing cost semantics do not change; the turn-level model-call span uses terminal cumulative usage when available, including cache-read and cache-creation tokens.

For these CLI spans, byte and timing fields describe the observable OpenClaw CLI boundary:

  • openclaw.model_call.request_bytes is the UTF-8 size of the prompt value sent over one-shot stdin/argv, or the managed stdio JSONL user envelope. It is not the size of Claude Code's hidden model request.
  • openclaw.model_call.response_bytes is the UTF-8 size of Claude CLI stdout observed during the turn. It is not Anthropic HTTP response size.
  • openclaw.model_call.time_to_first_byte_ms is time to the first observable Claude CLI stdout or stderr output. It is not network TTFB.

With captureContent enabled, the span exports the effective prompt OpenClaw sends to Claude Code and visible assistant text/tool-call identity through gen_ai.input.messages and gen_ai.output.messages. Tool arguments, internal thinking, opaque thinking signatures, tool results, and system prompts are omitted from the Claude assistant envelope. OpenClaw does not claim access to Claude Code's private system prompt, hidden resumed or compacted request payload, native internal tool schemas, raw Anthropic HTTP request, internal retries, upstream request id, or true network TTFB. Because Claude Code does not expose its effective native tool definitions accurately, these spans do not populate gen_ai.tool.definitions.

External Claude harness tool spans remain metadata-only even when tool content capture is enabled. As with every model span, captured Claude CLI content uses the trusted listener-only path and the exporter's existing redaction and size bounds; content remains off by default.

Exported metrics

Model usage

  • openclaw.tokens (counter, attrs: openclaw.token, openclaw.channel, openclaw.provider, openclaw.model, openclaw.agent)
  • openclaw.cost.usd (counter, attrs: openclaw.channel, openclaw.provider, openclaw.model)
  • openclaw.run.duration_ms (histogram, attrs: openclaw.channel, openclaw.provider, openclaw.model)
  • openclaw.context.tokens (histogram, attrs: openclaw.context, openclaw.channel, openclaw.provider, openclaw.model)
  • gen_ai.client.token.usage (histogram, GenAI semantic-conventions metric, attrs: gen_ai.token.type = input/output, gen_ai.provider.name, gen_ai.operation.name, gen_ai.request.model)
  • gen_ai.client.operation.duration (histogram, seconds, GenAI semantic-conventions metric for model requests and synthetic agent turns; attrs: gen_ai.provider.name, gen_ai.operation.name, gen_ai.request.model, optional error.type; turn observations use gen_ai.operation.name = invoke_agent)
  • openclaw.model_call.duration_ms (histogram, attrs: openclaw.provider, openclaw.model, openclaw.api, openclaw.transport, openclaw.model_call.observation_unit, plus openclaw.errorCategory and openclaw.failureKind on classified errors)
  • openclaw.model_call.request_bytes (histogram, UTF-8 byte size of the final model request payload; for Claude Code CLI, the observable prompt input/envelope described above; no raw payload content)
  • openclaw.model_call.response_bytes (histogram, UTF-8 byte size of streamed response chunk payloads; high-frequency text, thinking, and tool-call deltas count only incremental delta bytes; for Claude Code CLI, observed stdout bytes; no raw response content)
  • openclaw.model_call.time_to_first_byte_ms (histogram, elapsed time before the first streamed response event; for Claude Code CLI, first observable CLI output rather than network TTFB)
  • openclaw.model.failover (counter, attrs: openclaw.provider, openclaw.model, openclaw.failover.to_provider, openclaw.failover.to_model, openclaw.failover.reason, openclaw.failover.suspended, openclaw.lane)
  • openclaw.skill.used (counter, attrs: openclaw.skill.name, openclaw.skill.source, openclaw.skill.activation, optional openclaw.agent, optional openclaw.toolName)

Message flow

  • openclaw.webhook.received (counter, attrs: openclaw.channel, openclaw.webhook)
  • openclaw.webhook.error (counter, attrs: openclaw.channel, openclaw.webhook)
  • openclaw.webhook.duration_ms (histogram, attrs: openclaw.channel, openclaw.webhook)
  • openclaw.message.queued (counter, attrs: openclaw.channel, openclaw.source)
  • openclaw.message.received (counter, attrs: openclaw.channel, openclaw.source)
  • openclaw.message.dispatch.started (counter, attrs: openclaw.channel, openclaw.source)
  • openclaw.message.dispatch.completed (counter, attrs: openclaw.channel, openclaw.outcome, openclaw.reason, openclaw.source)
  • openclaw.message.dispatch.duration_ms (histogram, attrs: openclaw.channel, openclaw.outcome, openclaw.reason, openclaw.source)
  • openclaw.message.processed (counter, attrs: openclaw.channel, openclaw.outcome)
  • openclaw.message.duration_ms (histogram, attrs: openclaw.channel, openclaw.outcome)
  • openclaw.message.delivery.started (counter, attrs: openclaw.channel, openclaw.delivery.kind)
  • openclaw.message.delivery.duration_ms (histogram, attrs: openclaw.channel, openclaw.delivery.kind, openclaw.outcome, openclaw.errorCategory)

Talk

  • openclaw.talk.event (counter, attrs: openclaw.talk.event_type, openclaw.talk.mode, openclaw.talk.transport, openclaw.talk.brain, openclaw.talk.provider)
  • openclaw.talk.event.duration_ms (histogram, attrs: same as openclaw.talk.event; emitted when a Talk event reports duration)
  • openclaw.talk.audio.bytes (histogram, attrs: same as openclaw.talk.event; emitted for Talk audio frame events that report byte length)

Queues and sessions

  • openclaw.queue.lane.enqueue (counter, attrs: openclaw.lane)
  • openclaw.queue.lane.dequeue (counter, attrs: openclaw.lane)
  • openclaw.queue.depth (histogram, attrs: openclaw.lane or openclaw.channel=heartbeat)
  • openclaw.queue.wait_ms (histogram, attrs: openclaw.lane)
  • openclaw.session.state (counter, attrs: openclaw.state, openclaw.reason)
  • openclaw.session.stuck (counter, attrs: openclaw.state; emitted for recoverable stale session bookkeeping)
  • openclaw.session.stuck_age_ms (histogram, attrs: openclaw.state; emitted for recoverable stale session bookkeeping)
  • openclaw.session.turn.created (counter, attrs: openclaw.agent, openclaw.channel, openclaw.trigger)
  • openclaw.session.recovery.requested (counter, attrs: openclaw.state, openclaw.action, openclaw.active_work_kind, openclaw.reason)
  • openclaw.session.recovery.completed (counter, attrs: openclaw.state, openclaw.action, openclaw.status, openclaw.active_work_kind, openclaw.reason)
  • openclaw.session.recovery.age_ms (histogram, attrs: same as the matching recovery counter)
  • openclaw.run.attempt (counter, attrs: openclaw.attempt)

Session liveness telemetry

A processing session does not age toward the built-in liveness threshold while OpenClaw observes reply, tool, status, block, or ACP runtime progress. Typing keepalives do not count as progress, so a silent model or harness can still be detected.

OpenClaw classifies sessions by the work it can still observe:

  • session.long_running: active embedded work, model calls, or tool calls are still making progress. Owned silent model calls also report as long-running before the built-in abort threshold, so slow or non-streaming model providers do not look like stalled gateway sessions while abort-observable.
  • session.stalled: active work exists, but the active run has not reported recent progress. Owned model calls switch from session.long_running to session.stalled at or after the built-in abort threshold; ownerless stale model/tool activity is not treated as harmless long-running work. Stalled embedded runs stay observe-only at first, then abort-drain after the abort threshold with no progress so queued turns behind the lane can resume.
  • session.stuck: stale session bookkeeping with no active work, or an idle queued session with stale ownerless model/tool activity. This releases the affected session lane immediately after recovery gates pass.

Recovery emits structured session.recovery.requested and session.recovery.completed events. Diagnostic session state is marked idle only after a mutating recovery outcome (aborted or released) and only if the same processing generation is still current.

Only session.stuck emits the openclaw.session.stuck counter, the openclaw.session.stuck_age_ms histogram, and the openclaw.session.stuck span. Repeated session.stuck diagnostics back off while the session remains unchanged, so dashboards should alert on sustained increases rather than every heartbeat tick. For the config knob and defaults, see Configuration reference.

Liveness warnings also emit:

  • openclaw.liveness.warning (counter, attrs: openclaw.liveness.reason)
  • openclaw.liveness.event_loop_delay_p99_ms (histogram, attrs: openclaw.liveness.reason)
  • openclaw.liveness.event_loop_delay_max_ms (histogram, attrs: openclaw.liveness.reason)
  • openclaw.liveness.event_loop_utilization (histogram, attrs: openclaw.liveness.reason)
  • openclaw.liveness.cpu_core_ratio (histogram, attrs: openclaw.liveness.reason)

Harness lifecycle

  • openclaw.harness.duration_ms (histogram, attrs: openclaw.harness.id, openclaw.harness.plugin, openclaw.outcome, openclaw.harness.phase on errors)

Tool execution and loop detection

  • openclaw.tool.execution.duration_ms (histogram, attrs: gen_ai.tool.name, openclaw.toolName, openclaw.tool.source, openclaw.tool.owner, openclaw.tool.params.kind, plus openclaw.errorCategory on errors)
  • openclaw.tool.execution.blocked (counter, attrs: gen_ai.tool.name, openclaw.toolName, openclaw.tool.source, openclaw.tool.owner, openclaw.tool.params.kind, openclaw.deniedReason)
  • openclaw.tool.loop (counter, attrs: openclaw.toolName, openclaw.loop.level, openclaw.loop.action, openclaw.loop.detector, openclaw.loop.count, optional openclaw.loop.paired_tool; emitted when a repetitive tool-call loop is detected)

Exec

  • openclaw.exec.duration_ms (histogram, attrs: openclaw.exec.target, openclaw.exec.mode, openclaw.outcome, openclaw.failureKind)

Diagnostics internals (memory, payloads, exporter health)

  • openclaw.payload.large (counter, attrs: openclaw.payload.surface, openclaw.payload.action, openclaw.channel, openclaw.plugin, openclaw.reason)
  • openclaw.payload.large_bytes (histogram, attrs: same as openclaw.payload.large)
  • openclaw.memory.rss_bytes / openclaw.memory.heap_used_bytes / openclaw.memory.heap_total_bytes / openclaw.memory.external_bytes / openclaw.memory.array_buffers_bytes (histograms, no attrs; process memory samples)
  • openclaw.memory.pressure (counter, attrs: openclaw.memory.level, openclaw.memory.reason)
  • openclaw.diagnostic.async_queue.dropped (counter, attrs: openclaw.diagnostic.async_queue.drop_class; internal diagnostic-queue backpressure drops)
  • openclaw.telemetry.exporter.events (counter, attrs: openclaw.exporter, openclaw.signal, openclaw.status, optional openclaw.reason, optional openclaw.errorCategory; exporter lifecycle/failure self-telemetry)

Exported spans

  • openclaw.model.usage
    • openclaw.channel, openclaw.provider, openclaw.model
    • openclaw.tokens.* (input/output/cache_read/cache_write/total)
    • gen_ai.system by default, or gen_ai.provider.name when the latest GenAI semantic conventions are opted in
    • gen_ai.request.model, gen_ai.operation.name, gen_ai.usage.*
  • openclaw.run
    • openclaw.outcome, openclaw.channel, openclaw.provider, openclaw.model, openclaw.errorCategory
  • openclaw.model.call
    • gen_ai.system by default, or gen_ai.provider.name when the latest GenAI semantic conventions are opted in
    • gen_ai.request.model, gen_ai.operation.name, openclaw.provider, openclaw.model, openclaw.api, openclaw.transport, openclaw.model_call.observation_unit (request or turn)
    • openclaw.errorCategory, error.type, and optional openclaw.failureKind on errors
    • openclaw.model_call.request_bytes, openclaw.model_call.response_bytes, openclaw.model_call.time_to_first_byte_ms
    • openclaw.model_call.prompt.input_messages_count, openclaw.model_call.prompt.input_messages_chars, openclaw.model_call.prompt.system_prompt_chars, openclaw.model_call.prompt.tool_definitions_count, openclaw.model_call.prompt.tool_definitions_chars, openclaw.model_call.prompt.total_chars (safe component sizes only, no prompt text)
    • openclaw.model_call.usage.* and gen_ai.usage.* when the result carries usage for that request or aggregate turn
    • Span event openclaw.provider.request with attribute openclaw.upstreamRequestIdHash (bounded, hash-based) when the upstream provider result exposes a request id; raw ids are never exported
    • With OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental, request spans use the latest GenAI inference span name {gen_ai.operation.name} {gen_ai.request.model}. Turn spans use invoke_agent because OpenClaw does not claim a native agent name from the opaque CLI boundary. Both use CLIENT span kind instead of openclaw.model.call.
  • openclaw.harness.run
    • openclaw.harness.id, openclaw.harness.plugin, openclaw.outcome, openclaw.provider, openclaw.model, openclaw.channel
    • On completion: openclaw.harness.result_classification, openclaw.harness.yield_detected, openclaw.harness.items.started, openclaw.harness.items.completed, openclaw.harness.items.active
    • On error: openclaw.harness.phase, openclaw.errorCategory, optional openclaw.harness.cleanup_failed
  • openclaw.tool.execution
    • gen_ai.tool.name, gen_ai.operation.name (execute_tool), openclaw.toolName, openclaw.tool.source, optional gen_ai.tool.call.id, openclaw.tool.owner, openclaw.tool.params.*
    • Optional openclaw.errorCategory/openclaw.errorCode on errors, openclaw.deniedReason and openclaw.outcome=blocked when denied by policy or sandbox
  • openclaw.exec
    • openclaw.exec.target, openclaw.exec.mode, openclaw.outcome, openclaw.failureKind, openclaw.exec.command_length, openclaw.exec.exit_code, openclaw.exec.exit_signal, openclaw.exec.timed_out
  • openclaw.webhook.processed
    • openclaw.channel, openclaw.webhook
  • openclaw.webhook.error
    • openclaw.channel, openclaw.webhook, openclaw.error
  • openclaw.message.processed
    • openclaw.channel, openclaw.outcome, openclaw.reason
  • openclaw.message.delivery
    • openclaw.channel, openclaw.delivery.kind, openclaw.outcome, openclaw.errorCategory, openclaw.delivery.result_count
  • openclaw.session.stuck
    • openclaw.state, openclaw.ageMs, openclaw.queueDepth
  • openclaw.context.assembled
    • openclaw.prompt.size, openclaw.history.size, openclaw.context.tokens, openclaw.errorCategory (no prompt, history, response, or session-key content)
  • openclaw.tool.loop
    • openclaw.toolName, openclaw.loop.level, openclaw.loop.action, openclaw.loop.detector, openclaw.loop.count, optional openclaw.loop.paired_tool (no loop messages, params, or tool output)
  • openclaw.memory.pressure
    • openclaw.memory.level, openclaw.memory.reason, openclaw.memory.rss_bytes, openclaw.memory.heap_used_bytes, openclaw.memory.heap_total_bytes, openclaw.memory.external_bytes, openclaw.memory.array_buffers_bytes, optional openclaw.memory.threshold_bytes/openclaw.memory.rss_growth_bytes/openclaw.memory.window_ms

When content capture is explicitly enabled, model and tool spans can also include bounded, redacted openclaw.content.* attributes for the specific content classes you opted into.

Diagnostic event catalog

The events below back the metrics and spans above or are available for direct plugin subscription. run.progress and run.execution_phase are direct-only lifecycle signals; the diagnostics-otel plugin does not export them as standalone OTLP signals. Event kinds and run.execution_phase.phase values are additive. TypeScript consumers should keep default branches instead of assuming either union is permanently exhaustive.

Model usage

  • model.usage - tokens, cost, duration, context, provider/model/channel, session ids. usage is provider/turn accounting for cost and telemetry; context.used is the current prompt/context snapshot and can be lower than provider usage.total when cached input or tool-loop calls are involved.

Message flow

  • webhook.received / webhook.processed / webhook.error
  • message.queued / message.processed
  • message.delivery.started / message.delivery.completed / message.delivery.error

Queue and session

  • queue.lane.enqueue / queue.lane.dequeue
  • session.state / session.long_running / session.stalled / session.stuck
  • run.attempt / run.progress
  • run.execution_phase (public, session-correlated embedded-runner startup milestones)
  • diagnostic.heartbeat (aggregate counters: webhooks/queue/session)

Harness lifecycle

  • harness.run.started / harness.run.completed / harness.run.error - per-run lifecycle for the agent harness. Includes harnessId, optional pluginId, provider/model/channel, and run id. Completion adds durationMs, outcome, optional resultClassification, yieldDetected, and itemLifecycle counts. Errors add phase (prepare/start/send/resolve/cleanup), errorCategory, and optional cleanupFailed.

Exec

  • exec.process.completed - terminal outcome, duration, target, mode, exit code, and failure kind. Command text and working directories are not included.
  • exec.approval.followup_suppressed - stale approval follow-up dropped after a session rebound. Includes approvalId, reason (session_rebound), phase (direct_delivery or gateway_preflight), and the dispatcher timestamp. Session keys, routes, and command text are not included.

Without an exporter

Keep diagnostics events available to plugins or custom sinks without running diagnostics-otel:

{
  diagnostics: { enabled: true },
}

For targeted debug output without raising logging.level, use diagnostics flags. Flags are case-insensitive and support wildcards (telegram.* or *):

{
  diagnostics: { flags: ["telegram.http"] },
}

Or as a one-off env override:

OPENCLAW_DIAGNOSTICS=telegram.http,telegram.payload openclaw gateway

Flag output goes to the standard log file (logging.file) and is still redacted by the always-on log redaction policy. Full guide: Diagnostics flags.

Disable

{
  diagnostics: { otel: { enabled: false } },
}

Or leave diagnostics-otel out of plugins.allow, or run openclaw plugins disable diagnostics-otel.

Related

Reference in New Issue View Git Blame Copy Permalink