Files
openclaw/test/scripts/sqlite-sessions-transcripts-flip-proof.built-cli.e2e.test.ts
Josh Lehman 0a8e3604ba refactor: flip sessions and transcripts to sqlite storage (#98236)
* refactor(sessions): migrate runtime storage to sqlite

* test(sessions): fix sqlite CI regressions

* test(sessions): align remaining sqlite fixtures

* fix(codex): require sqlite trajectory recorder

* test(sessions): align orphan recovery sqlite fixture

* test(sessions): align sqlite rebase fixtures

* fix(sessions): finish current-main integration of the sqlite flip

Resolve the whole-store SDK removal across its owner boundary: drop the
loadSessionStore re-export and the registry whole-store wrappers, wire
hasTrackedActiveSessionRun into gateway chat, complete the
preserveLockedHarnessIds cleanup contract, flip the codex thread-history
import to storePath targets, and port remaining main-side tests from
file-store helpers to session accessor reads.

* chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs

Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore
the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain
bump gets its own review, and regenerate docs_map, the plugin SDK API baseline,
and the export-surface ratchet for the merged tree.

* feat(sessions): keep archived transcripts by default with zstd cold storage

Codex-style retention: deleting or resetting a session archives its
transcript as a zstd-compressed JSONL artifact (plain when the runtime
lacks node:zlib zstd) and keeps it until the disk budget evicts oldest
first. resetArchiveRetention now governs both deleted and reset archives
and defaults to keep; maxDiskBytes defaults to 2gb so retention stays
bounded, with archives evicted before live sessions. The cron reaper
follows the same knob instead of deleting archives on its own timer.

* fix(state): converge agent DB migration lineages and bound database growth

Merge coherence: run both structure-gated legacy memory-schema repairs
(flip-lineage drop, main-lineage identity rebuild) before the flip
migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all
converge, and hoist foreign_keys=OFF outside the schema transaction
where the pragma was silently ignored and the v1 sessions rebuild
cascade-deleted session_entries.

Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL
maintenance releases freed pages in bounded passes (never a blocking
full VACUUM), and doctor reports state/agent DB bloat from freelist
stats.

* fix(codex): resolve the store path for thread-history import via the SDK

The supervision catalog passed the legacy sessionFile locator to the
storePath-targeted transcript mirror; resolve the agent store path with
the session-store SDK helper instead of a runtime-object seam so test
fakes and headless callers need no extra surface. Drop the obsolete
missing-session-id preprocessing case: sessions rows are NOT NULL on
session_id and upsert repairs id-less patches at write time.

* fix(sessions): fail safe on malformed disk-budget config and doctor stat errors

A malformed explicit maxDiskBytes disables the budget instead of
falling back to the destructive 2gb default the user never chose, and
the doctor bloat check skips databases whose paths stat-fail instead of
aborting doctor.

* fix(sessions): complete sqlite conflict translations

* test(sqlite): align hardening checks with maintenance

* test(sessions): inspect compressed transcript archives

* fix(tests): await session seeds and drop unused helpers flagged by CI lint

The five unawaited writeSessionStoreSeed calls raced their SQLite seeds
against the assertions, failing compact shards; the bloat probe drops a
useless initializer and the merged tests drop now-unused helpers.

* test(sessions): type legacy proof events directly

* test(sessions): align hardening contracts

* perf(sessions): read usage transcript sizes from SQL aggregates

Usage/cost scans walked every session and materialized every transcript
event just to re-stringify it for a byte estimate — the #86718 stall
class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes
in SQLite without loading a single row.

* fix(sessions): re-root foreign-root transcript paths onto the current sessions dir

Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry
absolute sessionFile paths from the old root; the containment fallback
kept those foreign paths, so migration read (and would archive) files in
the original root and reported local copies missing. Re-root the
canonical agents/<id>/sessions suffix onto the current dir when the file
exists there; genuine cross-root layouts still fall through unchanged.

* test(agents): seed harness admission through sqlite

* fix(sqlite): close agent db on pragma setup failure

* fix(doctor): compact and retrofit incremental auto-vacuum after session import

The migration is the sanctioned offline window: post-import compact
reclaims import churn and applies auto_vacuum=INCREMENTAL to databases
created before the fresh-DB pragma existed, so runtime maintenance can
release pages in bounded passes on every install.

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-11 14:50:37 -07:00

158 lines
6.3 KiB
TypeScript

// Built-CLI SQLite flip proof requires dist entrypoints before running the gateway lifecycle.
import { describe, expect, it } from "vitest";
import { runSqliteSessionsTranscriptsFlipProof } from "../../scripts/e2e/sqlite-sessions-transcripts-flip-proof.ts";
describe("SQLite sessions/transcripts flip built CLI proof", () => {
it("proves the lifecycle through the built gateway CLI entrypoint", async () => {
const report = await runSqliteSessionsTranscriptsFlipProof({ requireBuiltCli: true });
expect(report.gatewayEntrypoint).toEqual(
expect.arrayContaining([expect.stringMatching(/^dist\/index\.(?:js|mjs)$/u)]),
);
expect(report.failures).toEqual([]);
expect(report.ok).toBe(true);
expect(
report.checkpoints
.filter((checkpoint) => checkpoint.label !== "seeded-legacy-store")
.every((checkpoint) => checkpoint.activeJsonl.length === 0),
).toBe(true);
expect(
report.checkpoints.some(
(checkpoint) =>
checkpoint.label === "seeded-legacy-store" && checkpoint.legacyStateJsonl.length > 0,
),
).toBe(true);
expect(
report.checkpoints
.filter((checkpoint) => checkpoint.label !== "seeded-legacy-store")
.every((checkpoint) => checkpoint.legacyStateJsonl.length === 0),
).toBe(true);
expect(report.checkpoints.some((checkpoint) => checkpoint.label === "after-doctor-fix")).toBe(
false,
);
expect(
report.checkpoints.some(
(checkpoint) =>
checkpoint.label === "after-startup-import" &&
checkpoint.gatewayLogTail?.includes(
"session: imported legacy session metadata/transcripts into SQLite",
) === true &&
report.oldStateSessionKeys.every((key) =>
checkpoint.sqlite.trackedEntries.some((entry) => entry.sessionKey === key),
) &&
checkpoint.sqlite.sessionEntries >= 7 &&
checkpoint.sqlite.transcriptEvents >= 13,
),
).toBe(true);
const startupImportCheckpoint = report.checkpoints.find(
(checkpoint) => checkpoint.label === "after-startup-import",
);
expect(
startupImportCheckpoint?.archiveArtifacts.some(
(artifact) =>
artifact.path.includes(`${report.legacySessionId}.trajectory.jsonl`) &&
artifact.textTail?.includes("trajectory") === true,
),
).toBe(true);
expect(
report.checkpoints.some(
(checkpoint) =>
checkpoint.label === "after-chat-send" &&
checkpoint.sqlite.trackedEntries.some(
(entry) => entry.sessionKey === report.resetSessionKey && entry.transcriptEvents >= 3,
),
),
).toBe(true);
expect(
report.checkpoints.some(
(checkpoint) =>
checkpoint.label === "after-full-agent-turn" &&
checkpoint.sqlite.trackedEntries.some(
(entry) =>
entry.sessionKey === report.fullTurnSessionKey && entry.transcriptEvents >= 2,
),
),
).toBe(true);
expect(report.pluginSdkConsumer).toMatchObject({
activeJsonlForSessionExists: false,
latestAssistantTextBeforeAppend: report.fullTurnAssistantText,
latestAssistantTextAfterAppend: "sqlite sdk consumer appended by identity",
sessionKey: report.pluginSdkSessionKey,
});
expect(report.pluginSdkConsumer?.sessionFileMarker.startsWith("sqlite:")).toBe(true);
expect(report.pluginSdkConsumer?.listedSessionKeys).toContain(report.pluginSdkSessionKey);
expect(
report.checkpoints.some(
(checkpoint) =>
checkpoint.label === "after-plugin-sdk-consumer" &&
checkpoint.sqlite.trackedEntries.some(
(entry) =>
entry.sessionKey === report.pluginSdkSessionKey && entry.transcriptEvents >= 3,
),
),
).toBe(true);
const cleanupCheckpoint = report.checkpoints.find(
(checkpoint) => checkpoint.label === "after-cleanup-pruning",
);
expect(
cleanupCheckpoint?.sqlite.trackedEntries.some(
(entry) => entry.sessionKey === report.cleanupPruneSessionKey,
),
).toBe(false);
const cleanupArchive = cleanupCheckpoint?.archiveArtifacts.find(
(artifact) =>
artifact.archiveReason === "deleted" &&
artifact.archiveSessionId === "sqlite-cleanup-prune",
);
expect(cleanupArchive?.messageTexts).toContain("sqlite cleanup prune me");
const idempotenceCheckpoint = report.checkpoints.find(
(checkpoint) => checkpoint.label === "after-doctor-import-idempotence",
);
expect(idempotenceCheckpoint?.doctor).toMatchObject({
code: 0,
mode: "import",
totals: expect.objectContaining({
importedEntries: 0,
importedTranscriptEvents: 0,
}),
});
const resetCheckpoint = report.checkpoints.find(
(checkpoint) => checkpoint.label === "after-sessions-reset",
);
const resetArchive = resetCheckpoint?.archiveArtifacts.find(
(artifact) =>
artifact.archiveReason === "reset" && artifact.archiveSessionId === report.legacySessionId,
);
expect(resetArchive?.messageTexts).toContain("legacy hello");
expect(resetArchive?.messageTexts).toContain("sqlite user-facing send before reset");
const sharedFirstCheckpoint = report.checkpoints.find(
(checkpoint) => checkpoint.label === "after-shared-first-delete",
);
expect(
sharedFirstCheckpoint?.archiveArtifacts.some(
(artifact) =>
artifact.archiveReason === "deleted" &&
artifact.archiveSessionId === "sqlite-shared-session",
),
).toBe(false);
const concurrentCheckpoint = report.checkpoints.find(
(checkpoint) => checkpoint.label === "after-concurrent-multi-client",
);
expect(concurrentCheckpoint).toBeDefined();
const concurrentSend = concurrentCheckpoint?.sqlite.trackedEntries.find(
(entry) => entry.sessionKey === report.concurrentSendSessionKey,
);
expect(concurrentSend?.transcriptEvents).toBeGreaterThanOrEqual(2);
expect(
concurrentCheckpoint?.sqlite.trackedEntries.some(
(entry) => entry.sessionKey === report.concurrentResetSessionKey && entry.sessionId,
),
).toBe(true);
expect(
concurrentCheckpoint?.sqlite.trackedEntries.some(
(entry) => entry.sessionKey === report.concurrentDeleteSessionKey,
),
).toBe(false);
}, 180_000);
});