0a8e3604ba
* refactor(sessions): migrate runtime storage to sqlite * test(sessions): fix sqlite CI regressions * test(sessions): align remaining sqlite fixtures * fix(codex): require sqlite trajectory recorder * test(sessions): align orphan recovery sqlite fixture * test(sessions): align sqlite rebase fixtures * fix(sessions): finish current-main integration of the sqlite flip Resolve the whole-store SDK removal across its owner boundary: drop the loadSessionStore re-export and the registry whole-store wrappers, wire hasTrackedActiveSessionRun into gateway chat, complete the preserveLockedHarnessIds cleanup contract, flip the codex thread-history import to storePath targets, and port remaining main-side tests from file-store helpers to session accessor reads. * chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain bump gets its own review, and regenerate docs_map, the plugin SDK API baseline, and the export-surface ratchet for the merged tree. * feat(sessions): keep archived transcripts by default with zstd cold storage Codex-style retention: deleting or resetting a session archives its transcript as a zstd-compressed JSONL artifact (plain when the runtime lacks node:zlib zstd) and keeps it until the disk budget evicts oldest first. resetArchiveRetention now governs both deleted and reset archives and defaults to keep; maxDiskBytes defaults to 2gb so retention stays bounded, with archives evicted before live sessions. The cron reaper follows the same knob instead of deleting archives on its own timer. * fix(state): converge agent DB migration lineages and bound database growth Merge coherence: run both structure-gated legacy memory-schema repairs (flip-lineage drop, main-lineage identity rebuild) before the flip migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all converge, and hoist foreign_keys=OFF outside the schema transaction where the pragma was silently ignored and the v1 sessions rebuild cascade-deleted session_entries. Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL maintenance releases freed pages in bounded passes (never a blocking full VACUUM), and doctor reports state/agent DB bloat from freelist stats. * fix(codex): resolve the store path for thread-history import via the SDK The supervision catalog passed the legacy sessionFile locator to the storePath-targeted transcript mirror; resolve the agent store path with the session-store SDK helper instead of a runtime-object seam so test fakes and headless callers need no extra surface. Drop the obsolete missing-session-id preprocessing case: sessions rows are NOT NULL on session_id and upsert repairs id-less patches at write time. * fix(sessions): fail safe on malformed disk-budget config and doctor stat errors A malformed explicit maxDiskBytes disables the budget instead of falling back to the destructive 2gb default the user never chose, and the doctor bloat check skips databases whose paths stat-fail instead of aborting doctor. * fix(sessions): complete sqlite conflict translations * test(sqlite): align hardening checks with maintenance * test(sessions): inspect compressed transcript archives * fix(tests): await session seeds and drop unused helpers flagged by CI lint The five unawaited writeSessionStoreSeed calls raced their SQLite seeds against the assertions, failing compact shards; the bloat probe drops a useless initializer and the merged tests drop now-unused helpers. * test(sessions): type legacy proof events directly * test(sessions): align hardening contracts * perf(sessions): read usage transcript sizes from SQL aggregates Usage/cost scans walked every session and materialized every transcript event just to re-stringify it for a byte estimate — the #86718 stall class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes in SQLite without loading a single row. * fix(sessions): re-root foreign-root transcript paths onto the current sessions dir Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry absolute sessionFile paths from the old root; the containment fallback kept those foreign paths, so migration read (and would archive) files in the original root and reported local copies missing. Re-root the canonical agents/<id>/sessions suffix onto the current dir when the file exists there; genuine cross-root layouts still fall through unchanged. * test(agents): seed harness admission through sqlite * fix(sqlite): close agent db on pragma setup failure * fix(doctor): compact and retrofit incremental auto-vacuum after session import The migration is the sanctioned offline window: post-import compact reclaims import churn and applies auto_vacuum=INCREMENTAL to databases created before the fresh-DB pragma existed, so runtime maintenance can release pages in bounded passes on every install. --------- Co-authored-by: Peter Steinberger <steipete@gmail.com>
7.8 KiB
summary, read_when, title
| summary | read_when | title | |||
|---|---|---|---|---|---|
| Design for live Gateway proof of the Path 3 SQLite session/transcript flip |
|
Path 3 live SQLite E2E harness |
The Path 3 live SQLite E2E harness proves the Gateway is using SQLite as the canonical session and transcript store while legacy JSONL files remain migration input or archive material. It is a maintainer proof harness, not a normal user diagnostic.
After a Gateway has processed post-migration traffic, legacy JSONL parity is no longer a valid runtime health signal. A healthy migrated Gateway can have SQLite transcript rows that differ from legacy JSONL counts because new turns should advance SQLite only. The live harness must therefore measure Gateway behavior, SQLite row movement, legacy-file quiescence, and log health at each step.
Command shape
The intended live command is:
node scripts/path3-live-sqlite-e2e.mjs \
--url http://127.0.0.1:18789 \
--agent main \
--session-key agent:main:path3-live-e2e:<timestamp> \
--json
The command connects to an already running Gateway. It does not start, stop,
import, or re-run the migration unless an explicit migration mode is added
later. A CI or isolated-local variant can use
test/helpers/openclaw-test-instance.ts, but the live proof path should inspect
the actual operator Gateway and its real per-agent SQLite database.
Isolated built-CLI proof
The built-CLI proof runner seeds an isolated legacy session store, starts the
rebuilt Gateway, and proves that startup imports hot legacy sessions into
SQLite before runtime reads begin. It must not run openclaw doctor --fix
before the first Gateway start, because that would prove the manual migration
path instead of the upgrade path users receive on first boot after the flip.
After startup import, the isolated proof may run
openclaw doctor --session-sqlite inspect and
openclaw doctor --session-sqlite validate as diagnostic evidence. Those
doctor commands are not the migration driver for the startup-upgrade proof.
Separate doctor-import scenarios should seed legacy transcript files plus
trajectory sidecars and verify doctor archives those artifacts while SQLite
remains canonical.
Preflight
Preflight collects a baseline and fails before sending a proof turn if the Gateway is not usable:
GET /healthand Gateway deep status must report a running, reachable Gateway.- The CLI and Gateway versions must match the branch being tested.
- The harness records a log cursor for the active Gateway file log.
- The harness records per-agent SQLite table counts for
sessions,session_entries,transcript_events,transcript_event_identities, andsession_routes. - The harness records
mtime,size, and existence for legacysessions.json, referenced JSONL files, and candidate proof-session JSONL paths. lsof -p <gateway-pid>must show SQLite DB/WAL/SHM handles and no hot.jsonlorsessions.jsonhandles.
openclaw doctor --session-sqlite validate is informational only in live mode.
After post-flip traffic it may report expected drift against legacy files. The
harness should use doctor output for classification and migration inventory,
not as the runtime pass/fail oracle.
Agent-driven scenario
The live scenario uses a dedicated proof session key and drives the Gateway through public RPC paths wherever possible. One agent turn should be enough to exercise ordinary persistence, but the full proof should cover the 3.1b seams that previously required individual live checks:
- Ordinary chat turn: create or reuse the proof session, send a real agent
prompt, wait for the final assistant result, and verify
chat.historyor equivalent Gateway projection. - Transcript identity: verify the same marker appears in Gateway history and in SQLite transcript rows, including stable event identity rows when present.
- Session metadata accessors: read the proof session and selected existing live sessions through Gateway/session accessors and compare them to SQLite rows.
- Session patch projection: apply a reversible model/session metadata change on the proof session, then verify the projected row and Gateway response agree.
- Compaction checkpoint lifecycle: list, branch, and restore a checkpoint only on the proof session or a synthetic fixture session created by the harness.
- Restart recovery: run the safe recovery marker path against a controlled proof session or an isolated test instance; live mode may only run this step when the target session set is explicit and reversible.
- Cleanup lifecycle: delete or reset the proof session, then verify SQLite lifecycle rows and archived transcript state.
Transport-specific seams that cannot be exercised safely on the live operator Gateway, such as WhatsApp or voice-call ingress, should use owner-level runtime probes against the same SQLite contract rather than fake external transport.
Per-step assertions
Each step snapshots before and after state and writes a structured assertion record:
- SQLite row counts advance only where expected.
- Trajectory runtime rows advance for marker-backed proof sessions that record runtime events.
- The proof session row has the expected
session_id, status, timestamps, metadata, and route rows. - Gateway history/session projection matches the SQLite transcript tail.
- No proof-session JSONL file is created or modified.
- No proof-session
.trajectory.jsonl,.trajectory-path.json, or marker-derivedtrajectory/<session>.jsonlsidecar is created. - Existing legacy JSONL files and
sessions.jsonremain unchanged unless the step is explicitly an offline migration or archive operation. - The Gateway process does not open
.jsonlorsessions.jsonhandles. - Logs since the previous cursor contain no
ERROR,FATAL,SQLITE_,no such column, session-store unavailable, restart-recovery failure, or transcript-reconcile warning unless the scenario explicitly allowlists it.
The log scan is part of the pass/fail contract. A Gateway that answers health checks but emits SQLite schema errors or repeated transcript reconcile failures is not green for Path 3.
Evidence artifact
The harness should write evidence under .artifacts/path3-live-e2e/<timestamp>/
and keep it out of git:
summary.json: command args, Gateway version, result, failed assertion, and artifact paths.sqlite-before.jsonandsqlite-after.json: row counts and selected proof rows.legacy-files.json: legacy file existence,mtime, size, and whether each file changed.gateway-log-scan.json: cursor range, matched log lines, and allowlist decisions.events.jsonl: ordered per-step observations suitable for PR proof comments.
The PR proof should summarize these artifacts instead of pasting full transcripts or private message content.
Safety rules
- Live mode must never re-import legacy JSONL while the Gateway is running.
- Live mode must not mutate non-proof sessions except for explicitly selected, reversible repair probes.
- Any destructive or broad migration step requires a fresh backup of the affected SQLite DB and legacy session directory.
- Backups should be scoped to the touched agent DB/session directory and reused during one proof run to avoid unbounded disk growth.
- The cleanup step must leave no proof session, proof JSONL, or modified legacy
file behind unless the caller passes
--keep-artifacts.
Passing result
A passing live run means the Gateway accepted a real agent-driven session flow, all observed canonical state was in SQLite, legacy runtime files stayed quiescent, and log health stayed clean for the measured window. It does not mean legacy JSONL parity remains clean after live traffic; live drift is expected once SQLite is the canonical store.