mirror of https://github.com/openclaw/openclaw.git synced 2026-07-15 10:56:13 +00:00

Files

Josh Lehman 0a8e3604ba refactor: flip sessions and transcripts to sqlite storage (#98236 )

* refactor(sessions): migrate runtime storage to sqlite

* test(sessions): fix sqlite CI regressions

* test(sessions): align remaining sqlite fixtures

* fix(codex): require sqlite trajectory recorder

* test(sessions): align orphan recovery sqlite fixture

* test(sessions): align sqlite rebase fixtures

* fix(sessions): finish current-main integration of the sqlite flip

Resolve the whole-store SDK removal across its owner boundary: drop the
loadSessionStore re-export and the registry whole-store wrappers, wire
hasTrackedActiveSessionRun into gateway chat, complete the
preserveLockedHarnessIds cleanup contract, flip the codex thread-history
import to storePath targets, and port remaining main-side tests from
file-store helpers to session accessor reads.

* chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs

Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore
the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain
bump gets its own review, and regenerate docs_map, the plugin SDK API baseline,
and the export-surface ratchet for the merged tree.

* feat(sessions): keep archived transcripts by default with zstd cold storage

Codex-style retention: deleting or resetting a session archives its
transcript as a zstd-compressed JSONL artifact (plain when the runtime
lacks node:zlib zstd) and keeps it until the disk budget evicts oldest
first. resetArchiveRetention now governs both deleted and reset archives
and defaults to keep; maxDiskBytes defaults to 2gb so retention stays
bounded, with archives evicted before live sessions. The cron reaper
follows the same knob instead of deleting archives on its own timer.

* fix(state): converge agent DB migration lineages and bound database growth

Merge coherence: run both structure-gated legacy memory-schema repairs
(flip-lineage drop, main-lineage identity rebuild) before the flip
migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all
converge, and hoist foreign_keys=OFF outside the schema transaction
where the pragma was silently ignored and the v1 sessions rebuild
cascade-deleted session_entries.

Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL
maintenance releases freed pages in bounded passes (never a blocking
full VACUUM), and doctor reports state/agent DB bloat from freelist
stats.

* fix(codex): resolve the store path for thread-history import via the SDK

The supervision catalog passed the legacy sessionFile locator to the
storePath-targeted transcript mirror; resolve the agent store path with
the session-store SDK helper instead of a runtime-object seam so test
fakes and headless callers need no extra surface. Drop the obsolete
missing-session-id preprocessing case: sessions rows are NOT NULL on
session_id and upsert repairs id-less patches at write time.

* fix(sessions): fail safe on malformed disk-budget config and doctor stat errors

A malformed explicit maxDiskBytes disables the budget instead of
falling back to the destructive 2gb default the user never chose, and
the doctor bloat check skips databases whose paths stat-fail instead of
aborting doctor.

* fix(sessions): complete sqlite conflict translations

* test(sqlite): align hardening checks with maintenance

* test(sessions): inspect compressed transcript archives

* fix(tests): await session seeds and drop unused helpers flagged by CI lint

The five unawaited writeSessionStoreSeed calls raced their SQLite seeds
against the assertions, failing compact shards; the bloat probe drops a
useless initializer and the merged tests drop now-unused helpers.

* test(sessions): type legacy proof events directly

* test(sessions): align hardening contracts

* perf(sessions): read usage transcript sizes from SQL aggregates

Usage/cost scans walked every session and materialized every transcript
event just to re-stringify it for a byte estimate — the #86718 stall
class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes
in SQLite without loading a single row.

* fix(sessions): re-root foreign-root transcript paths onto the current sessions dir

Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry
absolute sessionFile paths from the old root; the containment fallback
kept those foreign paths, so migration read (and would archive) files in
the original root and reported local copies missing. Re-root the
canonical agents/<id>/sessions suffix onto the current dir when the file
exists there; genuine cross-root layouts still fall through unchanged.

* test(agents): seed harness admission through sqlite

* fix(sqlite): close agent db on pragma setup failure

* fix(doctor): compact and retrofit incremental auto-vacuum after session import

The migration is the sanctioned offline window: post-import compact
reclaims import churn and applies auto_vacuum=INCREMENTAL to databases
created before the fresh-DB pragma existed, so runtime maintenance can
release pages in bounded passes on every install.

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>

2026-07-11 14:50:37 -07:00

4.8 KiB

Raw Blame History

summary, read_when, title

summary

read_when

title

Path 3 plan for archiving all SQLite transcript artifacts that belong to a session

You are implementing clawdbot-d63.2 / clawdbot-04b

You are touching SQLite session retention, reset, delete, or agent-deletion archival

You need to distinguish SQLite-era artifact families from legacy JSONL sidecars

Path 3 SQLite session artifact family

Path 3 SQLite Session Artifact Family

This note scopes clawdbot-d63.2 while clawdbot-d63.1 owns the overlapping reset/delete archive helper in src/config/sessions/session-accessor.sqlite.ts. The implementation file was dirty during this pass, so this artifact records the exact contract and patch points without racing the sibling worker.

Authoritative family

After the SQLite flip, active session transcripts are SQLite rows. A session's archive family is:

The transcript_events, transcript_event_identities, and sessions rows for the entry's current sessionId.
The same SQLite transcript row set for every sessionId referenced by entry.compactionCheckpoints[*].preCompaction.sessionId.
The same SQLite transcript row set for every sessionId referenced by entry.compactionCheckpoints[*].postCompaction.sessionId.
The same SQLite transcript row set for every sessionId in entry.usageFamilySessionIds.

Archive only rows that are no longer referenced by any remaining session_entries row or by any remaining entry's compaction or usage-family metadata. This preserves checkpoint branch/restore and usage rollup state until the final live reference is gone.

Non-family artifacts after the flip

Generated topic transcript file variants and trajectory sidecars are not active SQLite runtime state. They are legacy file artifacts:

Topic variants such as <sessionId>-topic-<thread>.jsonl only exist for the file-backed transcript format. SQLite uses the canonical session id plus session_routes/entry delivery metadata instead of per-topic JSONL files.
Trajectory sidecars such as .trajectory.jsonl and .trajectory-path.json are named from real JSONL sessionFile paths. SQLite sessionFile values are sqlite:<agentId>:<sessionId>:<storePath> markers and do not name sidecar files.
Archive-tier readers must keep reading legacy archived JSONL files, but runtime retention must not scan active sessions directories or reopen JSONL transcript files for SQLite sessions.

Doctor import remains the migration owner for legacy primary JSONL files and their adjacent trajectory sidecars. Runtime SQLite retention should not add a second importer or file fallback.

Patch points

Extend the SQLite archive helper introduced by clawdbot-d63.1 rather than adding a parallel path.

Add a local collector near deleteSqliteSessionStateIfUnreferenced:
- collectSqliteSessionArtifactFamily(entry: SessionEntry): Set<string>
- Include entry.sessionId, checkpoint pre/post session ids, and usageFamilySessionIds.
- Filter empty strings and dedupe deterministically.
Add a reference collector for the post-removal store:
- readReferencedSqliteSessionArtifactFamilyIds(database): Set<string>
- Iterate current session_entries, parse each entry_json, and collect the same family ids from every surviving entry.
Change the reset/delete/maintenance callers that currently archive one removed sessionId to pass the removed entry's full family.
For each family id, archive the SQLite transcript rows with the caller's reason (reset or deleted), then delete the sessions row only when the family id is absent from the post-removal reference set.
Keep transcript event deletion centralized through the existing SQLite session-row cleanup path. Do not add active JSONL reads.

Focused tests

Add SQLite-only tests to src/config/sessions/session-accessor.conformance.test.ts or the sibling lifecycle test after clawdbot-d63.1 commits:

Deleting an entry with a pre-compaction transcript archives both the current session and the pre-compaction session, then removes both SQLite row sets.
Deleting one of two entries that share a compaction pre-session archives nothing for the shared pre-session until the final referencing entry is removed.
Deleting an entry with usageFamilySessionIds archives predecessor SQLite transcript rows when no other entry references that usage family.
A topic-shaped session key with a SQLite marker does not cause any generated topic JSONL read or sidecar lookup.

The focused proof should use:

node scripts/run-vitest.mjs src/config/sessions/session-accessor.conformance.test.ts

If the final tests live in store.session-lifecycle-mutation.test.ts, run that file explicitly with the same wrapper. Broad pnpm gates should stay on Crabbox/Testbox for this Codex worktree.

4.8 KiB Raw Blame History