mirror of
https://github.com/openclaw/openclaw.git
synced 2026-07-16 20:51:43 +00:00
0a8e3604ba
* refactor(sessions): migrate runtime storage to sqlite * test(sessions): fix sqlite CI regressions * test(sessions): align remaining sqlite fixtures * fix(codex): require sqlite trajectory recorder * test(sessions): align orphan recovery sqlite fixture * test(sessions): align sqlite rebase fixtures * fix(sessions): finish current-main integration of the sqlite flip Resolve the whole-store SDK removal across its owner boundary: drop the loadSessionStore re-export and the registry whole-store wrappers, wire hasTrackedActiveSessionRun into gateway chat, complete the preserveLockedHarnessIds cleanup contract, flip the codex thread-history import to storePath targets, and port remaining main-side tests from file-store helpers to session accessor reads. * chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain bump gets its own review, and regenerate docs_map, the plugin SDK API baseline, and the export-surface ratchet for the merged tree. * feat(sessions): keep archived transcripts by default with zstd cold storage Codex-style retention: deleting or resetting a session archives its transcript as a zstd-compressed JSONL artifact (plain when the runtime lacks node:zlib zstd) and keeps it until the disk budget evicts oldest first. resetArchiveRetention now governs both deleted and reset archives and defaults to keep; maxDiskBytes defaults to 2gb so retention stays bounded, with archives evicted before live sessions. The cron reaper follows the same knob instead of deleting archives on its own timer. * fix(state): converge agent DB migration lineages and bound database growth Merge coherence: run both structure-gated legacy memory-schema repairs (flip-lineage drop, main-lineage identity rebuild) before the flip migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all converge, and hoist foreign_keys=OFF outside the schema transaction where the pragma was silently ignored and the v1 sessions rebuild cascade-deleted session_entries. Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL maintenance releases freed pages in bounded passes (never a blocking full VACUUM), and doctor reports state/agent DB bloat from freelist stats. * fix(codex): resolve the store path for thread-history import via the SDK The supervision catalog passed the legacy sessionFile locator to the storePath-targeted transcript mirror; resolve the agent store path with the session-store SDK helper instead of a runtime-object seam so test fakes and headless callers need no extra surface. Drop the obsolete missing-session-id preprocessing case: sessions rows are NOT NULL on session_id and upsert repairs id-less patches at write time. * fix(sessions): fail safe on malformed disk-budget config and doctor stat errors A malformed explicit maxDiskBytes disables the budget instead of falling back to the destructive 2gb default the user never chose, and the doctor bloat check skips databases whose paths stat-fail instead of aborting doctor. * fix(sessions): complete sqlite conflict translations * test(sqlite): align hardening checks with maintenance * test(sessions): inspect compressed transcript archives * fix(tests): await session seeds and drop unused helpers flagged by CI lint The five unawaited writeSessionStoreSeed calls raced their SQLite seeds against the assertions, failing compact shards; the bloat probe drops a useless initializer and the merged tests drop now-unused helpers. * test(sessions): type legacy proof events directly * test(sessions): align hardening contracts * perf(sessions): read usage transcript sizes from SQL aggregates Usage/cost scans walked every session and materialized every transcript event just to re-stringify it for a byte estimate — the #86718 stall class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes in SQLite without loading a single row. * fix(sessions): re-root foreign-root transcript paths onto the current sessions dir Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry absolute sessionFile paths from the old root; the containment fallback kept those foreign paths, so migration read (and would archive) files in the original root and reported local copies missing. Re-root the canonical agents/<id>/sessions suffix onto the current dir when the file exists there; genuine cross-root layouts still fall through unchanged. * test(agents): seed harness admission through sqlite * fix(sqlite): close agent db on pragma setup failure * fix(doctor): compact and retrofit incremental auto-vacuum after session import The migration is the sanctioned offline window: post-import compact reclaims import churn and applies auto_vacuum=INCREMENTAL to databases created before the fresh-DB pragma existed, so runtime maintenance can release pages in bounded passes on every install. --------- Co-authored-by: Peter Steinberger <steipete@gmail.com>
106 lines
4.8 KiB
Markdown
106 lines
4.8 KiB
Markdown
---
|
|
summary: "Path 3 plan for archiving all SQLite transcript artifacts that belong to a session"
|
|
read_when:
|
|
- You are implementing clawdbot-d63.2 / clawdbot-04b
|
|
- You are touching SQLite session retention, reset, delete, or agent-deletion archival
|
|
- You need to distinguish SQLite-era artifact families from legacy JSONL sidecars
|
|
title: "Path 3 SQLite session artifact family"
|
|
---
|
|
|
|
# Path 3 SQLite Session Artifact Family
|
|
|
|
This note scopes `clawdbot-d63.2` while `clawdbot-d63.1` owns the overlapping
|
|
reset/delete archive helper in `src/config/sessions/session-accessor.sqlite.ts`.
|
|
The implementation file was dirty during this pass, so this artifact records
|
|
the exact contract and patch points without racing the sibling worker.
|
|
|
|
## Authoritative family
|
|
|
|
After the SQLite flip, active session transcripts are SQLite rows. A session's
|
|
archive family is:
|
|
|
|
- The `transcript_events`, `transcript_event_identities`, and `sessions` rows
|
|
for the entry's current `sessionId`.
|
|
- The same SQLite transcript row set for every `sessionId` referenced by
|
|
`entry.compactionCheckpoints[*].preCompaction.sessionId`.
|
|
- The same SQLite transcript row set for every `sessionId` referenced by
|
|
`entry.compactionCheckpoints[*].postCompaction.sessionId`.
|
|
- The same SQLite transcript row set for every `sessionId` in
|
|
`entry.usageFamilySessionIds`.
|
|
|
|
Archive only rows that are no longer referenced by any remaining
|
|
`session_entries` row or by any remaining entry's compaction or usage-family
|
|
metadata. This preserves checkpoint branch/restore and usage rollup state until
|
|
the final live reference is gone.
|
|
|
|
## Non-family artifacts after the flip
|
|
|
|
Generated topic transcript file variants and trajectory sidecars are not active
|
|
SQLite runtime state. They are legacy file artifacts:
|
|
|
|
- Topic variants such as `<sessionId>-topic-<thread>.jsonl` only exist for the
|
|
file-backed transcript format. SQLite uses the canonical session id plus
|
|
`session_routes`/entry delivery metadata instead of per-topic JSONL files.
|
|
- Trajectory sidecars such as `.trajectory.jsonl` and `.trajectory-path.json`
|
|
are named from real JSONL `sessionFile` paths. SQLite `sessionFile` values are
|
|
`sqlite:<agentId>:<sessionId>:<storePath>` markers and do not name sidecar
|
|
files.
|
|
- Archive-tier readers must keep reading legacy archived JSONL files, but
|
|
runtime retention must not scan active sessions directories or reopen JSONL
|
|
transcript files for SQLite sessions.
|
|
|
|
Doctor import remains the migration owner for legacy primary JSONL files and
|
|
their adjacent trajectory sidecars. Runtime SQLite retention should not add a
|
|
second importer or file fallback.
|
|
|
|
## Patch points
|
|
|
|
Extend the SQLite archive helper introduced by `clawdbot-d63.1` rather than
|
|
adding a parallel path.
|
|
|
|
1. Add a local collector near `deleteSqliteSessionStateIfUnreferenced`:
|
|
- `collectSqliteSessionArtifactFamily(entry: SessionEntry): Set<string>`
|
|
- Include `entry.sessionId`, checkpoint pre/post session ids, and
|
|
`usageFamilySessionIds`.
|
|
- Filter empty strings and dedupe deterministically.
|
|
|
|
2. Add a reference collector for the post-removal store:
|
|
- `readReferencedSqliteSessionArtifactFamilyIds(database): Set<string>`
|
|
- Iterate current `session_entries`, parse each `entry_json`, and collect
|
|
the same family ids from every surviving entry.
|
|
|
|
3. Change the reset/delete/maintenance callers that currently archive one
|
|
removed `sessionId` to pass the removed entry's full family.
|
|
|
|
4. For each family id, archive the SQLite transcript rows with the caller's
|
|
reason (`reset` or `deleted`), then delete the `sessions` row only when the
|
|
family id is absent from the post-removal reference set.
|
|
|
|
5. Keep transcript event deletion centralized through the existing SQLite
|
|
session-row cleanup path. Do not add active JSONL reads.
|
|
|
|
## Focused tests
|
|
|
|
Add SQLite-only tests to `src/config/sessions/session-accessor.conformance.test.ts`
|
|
or the sibling lifecycle test after `clawdbot-d63.1` commits:
|
|
|
|
- Deleting an entry with a pre-compaction transcript archives both the current
|
|
session and the pre-compaction session, then removes both SQLite row sets.
|
|
- Deleting one of two entries that share a compaction pre-session archives
|
|
nothing for the shared pre-session until the final referencing entry is
|
|
removed.
|
|
- Deleting an entry with `usageFamilySessionIds` archives predecessor SQLite
|
|
transcript rows when no other entry references that usage family.
|
|
- A topic-shaped session key with a SQLite marker does not cause any generated
|
|
topic JSONL read or sidecar lookup.
|
|
|
|
The focused proof should use:
|
|
|
|
```bash
|
|
node scripts/run-vitest.mjs src/config/sessions/session-accessor.conformance.test.ts
|
|
```
|
|
|
|
If the final tests live in `store.session-lifecycle-mutation.test.ts`, run that
|
|
file explicitly with the same wrapper. Broad `pnpm` gates should stay on
|
|
Crabbox/Testbox for this Codex worktree.
|