Preserve ordinary schema-v1 trajectory payloads while shedding bulky duplicate conversation fields only after an event exceeds the runtime byte cap. Keep the bounded prompt in compact fallback records.
Attempt dispatch stopped emitting legacy `sqlite:` session-file markers in
#113233, which replaced SessionTranscriptRuntimeTarget.sessionFile with
storePath and started passing the canonical session key instead. The Codex
plugin still re-derived its own SQLite gate from that string, so
parseSqliteSessionFileMarker never matched and every Codex run bailed out
with reason "non-sqlite-session-target".
The host already owns SQLite target resolution and identity validation: it
returns a recorder only when a committed session row resolves. Drop the
stale marker gate and the now-dead trajectorySessionFile plumbing, leaving
the host-provided recorder as the single gate.
* feat(sqlite): migrate persisted media to canonical facts and stop legacy writes
PR 3 of the media legacy retirement program — the operator-approved
canonical cutover.
- openclaw doctor --fix owns one idempotent migration: active
transcript_events rows canonicalize to __openclaw.media (facts-first
gap-fill, bare legacy kinds to fact.kind, transcribed indexes and
workspace dirs onto per-fact fields) via the transcript replacement
owner; cold plain/.zst archives rewrite through temp-file + codec
readback + event/id verification + atomic replace; trajectory
runtime snapshots canonicalize IN PLACE (telemetry preserved, never
row deletion). Invalid JSON, genuinely ambiguous legacy-only sparse
alignment, or a changed source aborts that owner without partial work;
reruns are no-ops.
- Per-agent schema advances to v16 as a pure downgrade guard (main
independently took v15 for board/session-sharing tables; no
columns/tables/indexes change here, shared-state DB untouched). v15
databases repair canonical indexes before the version assertion so
repairable installations never strand.
- The user-turn builder stops writing top-level legacy Media* fields;
shouldPersistStructuredMediaEntries and the aligned projection mode
are deleted; the generic transcript append boundary canonicalizes
every message role so SDK/mirror writers cannot mint new legacy rows.
- Internal persisted-reader legacy fallbacks are removed; the public
SDK projection stays until retirement PR 4's window expires.
Hardening from three adversarial review rounds, each with fixture
regressions: in-place trajectory canonicalization instead of row
deletion; repair-before-assert on the v15 path; all-roles append
canonicalization; duplicate-preserving exact row rewrites; v0-v15
reopen guards; complete canonical facts bypass compact legacy
projections (PR-1 dual-write rows migrate cleanly); SQLite LIKE
underscore escaped so populated foreign databases are never claimed.
* fix(sqlite): align schema-support metadata and gates with the v16 cutover
package.json agent schema support advances to 16; verifier and board
parity fixtures run doctor migration before steady-state access (the
production guards were correct); two test-only exports removed; the
migration module registered in the doctor raw-SQLite allowlist.
* refactor(media): read persisted media facts first across all consumers
PR 2 of the media legacy retirement program. Gateway display projection,
Control UI (history filtering, attachment rendering, initial-turn
handoff), embedded and plugin-harness hydration, replay/prune, late-media
reconstruction, hooks, media understanding, and trajectory capture now
read nested __openclaw.media facts first. Legacy top-level Media* fields
survive only in the narrow persisted-message compatibility reader (rows
predating the PR-1 dual-write) and the public SDK projection.
Non-goals pinned by tests: rendered batches, reply-payload dedupe,
durable session/outbound queues, and lowercase ReplyPayload.mediaUrl(s)
are untouched contracts.
Consumer matrix covers {legacy-only, facts-only, both-equal,
both-conflict, sparse, type-only, media-only} inputs across every
consumer family; prompt/cache goldens byte-identical; 131-file legacy
consumer sweep run individually (two failures reproduce unchanged on
clean main); 1,047 broad media tests and the full UI lane (5,553) green.
* refactor(media): drop caller-less staging predicate and localize empty-user-row classification
The consumer flip left hasStagedMediaProjection without production
callers (facts-only staging uses hasStagedMediaFacts); delete it and
retest the staged merge matrix against the facts predicate. Unexport
LEGACY_MEDIA_CONTEXT_KEYS (internal only; the derived type stays public).
Move isEmptyUserTextOnlyMessage beside the other message-shape
classifiers in ui/src/lib/chat/message-extract.ts where chat-history
imports it cross-module.
* chore(ui): drop unused hasTranscriptMediaFacts import after predicate move
* chore(ui): localize hasTranscriptMediaFacts after its last external consumer moved in
* style: format staged-media matrix assertion
* fix: bound misc unbounded fs.readFile calls; remove unused fs import
* fix: decode buffer to string before passing to string consumers
readRegularFile and readRegularFileSync return { buffer, stat },
not a string. All 4 new call sites passed the raw object to functions
expecting a string (JSON.parse, RegExp.test, template literals, etc.),
causing TS2345 type errors and runtime failures.
Fix each call by extracting .buffer and calling .toString('utf8')
before passing the result to string consumers.
* style: fix oxfmt formatting in config-set-input.ts
* fix: bound config and trajectory metadata reads
Co-authored-by: 陈宪彪0668000387 <chen.xianbiao@xydigit.com>
* refactor: isolate bounded read ownership
Co-authored-by: 陈宪彪0668000387 <chen.xianbiao@xydigit.com>
---------
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* refactor(deadcode): trim auto-reply and CLI exports
* refactor(deadcode): trim cron and task exports
* refactor(deadcode): trim fleet and process exports
* test(deadcode): exercise live task and process seams
* test(fleet): cover stream redaction through owner module
* refactor(security): trim dead internal exports
* refactor(secrets): trim dead internal exports
* refactor(deadcode): trim remaining src exports
* refactor(deadcode): remove test-only runtime exports
* refactor(deadcode): trim pairing test exports
* refactor(deadcode): reconcile refreshed baseline
* test(auto-reply): deduplicate queue state imports
* refactor(sessions): migrate runtime storage to sqlite
* test(sessions): fix sqlite CI regressions
* test(sessions): align remaining sqlite fixtures
* fix(codex): require sqlite trajectory recorder
* test(sessions): align orphan recovery sqlite fixture
* test(sessions): align sqlite rebase fixtures
* fix(sessions): finish current-main integration of the sqlite flip
Resolve the whole-store SDK removal across its owner boundary: drop the
loadSessionStore re-export and the registry whole-store wrappers, wire
hasTrackedActiveSessionRun into gateway chat, complete the
preserveLockedHarnessIds cleanup contract, flip the codex thread-history
import to storePath targets, and port remaining main-side tests from
file-store helpers to session accessor reads.
* chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs
Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore
the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain
bump gets its own review, and regenerate docs_map, the plugin SDK API baseline,
and the export-surface ratchet for the merged tree.
* feat(sessions): keep archived transcripts by default with zstd cold storage
Codex-style retention: deleting or resetting a session archives its
transcript as a zstd-compressed JSONL artifact (plain when the runtime
lacks node:zlib zstd) and keeps it until the disk budget evicts oldest
first. resetArchiveRetention now governs both deleted and reset archives
and defaults to keep; maxDiskBytes defaults to 2gb so retention stays
bounded, with archives evicted before live sessions. The cron reaper
follows the same knob instead of deleting archives on its own timer.
* fix(state): converge agent DB migration lineages and bound database growth
Merge coherence: run both structure-gated legacy memory-schema repairs
(flip-lineage drop, main-lineage identity rebuild) before the flip
migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all
converge, and hoist foreign_keys=OFF outside the schema transaction
where the pragma was silently ignored and the v1 sessions rebuild
cascade-deleted session_entries.
Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL
maintenance releases freed pages in bounded passes (never a blocking
full VACUUM), and doctor reports state/agent DB bloat from freelist
stats.
* fix(codex): resolve the store path for thread-history import via the SDK
The supervision catalog passed the legacy sessionFile locator to the
storePath-targeted transcript mirror; resolve the agent store path with
the session-store SDK helper instead of a runtime-object seam so test
fakes and headless callers need no extra surface. Drop the obsolete
missing-session-id preprocessing case: sessions rows are NOT NULL on
session_id and upsert repairs id-less patches at write time.
* fix(sessions): fail safe on malformed disk-budget config and doctor stat errors
A malformed explicit maxDiskBytes disables the budget instead of
falling back to the destructive 2gb default the user never chose, and
the doctor bloat check skips databases whose paths stat-fail instead of
aborting doctor.
* fix(sessions): complete sqlite conflict translations
* test(sqlite): align hardening checks with maintenance
* test(sessions): inspect compressed transcript archives
* fix(tests): await session seeds and drop unused helpers flagged by CI lint
The five unawaited writeSessionStoreSeed calls raced their SQLite seeds
against the assertions, failing compact shards; the bloat probe drops a
useless initializer and the merged tests drop now-unused helpers.
* test(sessions): type legacy proof events directly
* test(sessions): align hardening contracts
* perf(sessions): read usage transcript sizes from SQL aggregates
Usage/cost scans walked every session and materialized every transcript
event just to re-stringify it for a byte estimate — the #86718 stall
class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes
in SQLite without loading a single row.
* fix(sessions): re-root foreign-root transcript paths onto the current sessions dir
Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry
absolute sessionFile paths from the old root; the containment fallback
kept those foreign paths, so migration read (and would archive) files in
the original root and reported local copies missing. Re-root the
canonical agents/<id>/sessions suffix onto the current dir when the file
exists there; genuine cross-root layouts still fall through unchanged.
* test(agents): seed harness admission through sqlite
* fix(sqlite): close agent db on pragma setup failure
* fix(doctor): compact and retrofit incremental auto-vacuum after session import
The migration is the sanctioned offline window: post-import compact
reclaims import churn and applies auto_vacuum=INCREMENTAL to databases
created before the fresh-DB pragma existed, so runtime maintenance can
release pages in bounded passes on every install.
---------
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* fix(agents): terminal-abort coverage beyond #87085's isTerminalAbort (closes#60388)
PR #87085 landed the base isTerminalAbort(signal) check (TimeoutError /
ClientDisconnectError on signal.reason) plus abortSignal threading through
the model-fallback and chat-side callers. This change adds the coverage that
PR #87085 did not include:
- ClientDisconnectError class + wiring in http-common.ts so the
reason.name === "ClientDisconnectError" branch PR #87085 added is actually
reachable (upstream watchClientDisconnect still aborts bare).
- cron run-budget string reasons (prefix match) — cron timer aborts with a
plain string, which the Error-only base check skips.
- .cause-chain walking + isTerminalAbortFromError gated on the
OPENCLAW_ABORTABLE_WRAPPER marker, for the embedded run-budget timer that
aborts a private controller (not the caller signal).
- compaction-path abortSignal forwarding (compactEmbeddedPiSessionDirect).
- timedOutByRunBudget plumbing through attempt/failover-policy/assistant-failover
so run-budget timeouts skip the fallback chain and wasted compaction.
* fix(agents): preserve wrapped restart aborts
* refactor: dedupe terminal abort classification
---------
Co-authored-by: Altay <altay@hey.com>
readSessionBranch filtered out every entry lacking a string or number
timestamp. Sessions written before entry timestamps existed (version 1)
have ids and parentIds synthesized by the legacy migration but no entry
timestamp, so all entries were dropped and the exported bundle reported
transcriptEventCount 0. The transcript event builder already defaults a
missing timestamp via normalizeTimestamp, so the filter clause was both
wrong and redundant. Drop it; entry identity plus the canonical-entry
check is what the branch walk needs.
Preserve concurrent prompt-time transcript updates across stale session managers, side appends, transcript navigation, nested owned writes, and doctor repair.
Fixes#93193.
Thanks @snowzlm for the report and original fix.
Co-authored-by: snowzlm <snowzlm@noreply.codeberg.org>
Extract shared normalization/coercion helpers into private @openclaw/normalization-core workspace package while preserving existing plugin SDK helper subpaths.\n\nAlso keeps direct normalization-core imports internal, wires UI/build/loader resolution, and replaces the slow PR network CodeQL lane with a fast added-line boundary scan while retaining full CodeQL for scheduled/manual runs.\n\nVerification: local moved tests, plugin SDK boundary tests, extension loader tests, agents-support shard, UI build/test, build artifacts, lint, workflow guards, autoreview, and GitHub CI passed on PR head 963d893715.
Adds `openclaw sessions tail` as an operator-facing progress view over session trajectory events, with conservative redaction for prompt text, tool arguments, and tool result bodies. The command supports explicit session keys, store/agent scope, follow mode, relocated trajectory pointer files, and cursor-safe follow across bounded trajectory window rewrites.
Documents the new sessions tail CLI surface in `docs/cli/sessions.md`.
Fixes#83441.
Co-authored-by: zhengzuo0-ai <zheng.zuo0@gmail.com>
Move meeting notes into core transcripts, remove the bundled meeting-notes plugin/API, and require explicit transcripts.enabled before exposing the recording-capable tool.
Summary:
- This PR filters partial skill snapshot entries in trajectory support metadata, accepts nullish support-redaction paths, adds regression tests, and records the fix in the changelog.
- Reproducibility: yes. Source inspection on current main shows undefined skill path/name values can reach str ... and the related source PR provides redacted live before/after gateway logs for the symlink-escape scenario.
Automerge notes:
- PR branch already contained follow-up commit before automerge: fix(trajectory): tighten test types for partial skill entries
- PR branch already contained follow-up commit before automerge: fix(trajectory): tolerate partial skill snapshot entries in support c…
Validation:
- ClawSweeper review passed for head ecb3df6c08.
- Required merge gates passed before the squash merge.
Prepared head SHA: ecb3df6c08
Review: https://github.com/openclaw/openclaw/pull/84797#issuecomment-4504703074
Co-authored-by: Luke Boyett <46942646+lukeboyett@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>
Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>
Approved-by: takhoffman
Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
Adds bounded queued-writer diagnostics to pi-trajectory-flush cleanup timeout warnings so operators can see pending write count, queued bytes, active operation, and append size without exposing paths or payloads.
Closes#82961
