mirror of
https://github.com/openclaw/openclaw.git
synced 2026-07-15 21:06:07 +00:00
* refactor(sessions): migrate runtime storage to sqlite * test(sessions): fix sqlite CI regressions * test(sessions): align remaining sqlite fixtures * fix(codex): require sqlite trajectory recorder * test(sessions): align orphan recovery sqlite fixture * test(sessions): align sqlite rebase fixtures * fix(sessions): finish current-main integration of the sqlite flip Resolve the whole-store SDK removal across its owner boundary: drop the loadSessionStore re-export and the registry whole-store wrappers, wire hasTrackedActiveSessionRun into gateway chat, complete the preserveLockedHarnessIds cleanup contract, flip the codex thread-history import to storePath targets, and port remaining main-side tests from file-store helpers to session accessor reads. * chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain bump gets its own review, and regenerate docs_map, the plugin SDK API baseline, and the export-surface ratchet for the merged tree. * feat(sessions): keep archived transcripts by default with zstd cold storage Codex-style retention: deleting or resetting a session archives its transcript as a zstd-compressed JSONL artifact (plain when the runtime lacks node:zlib zstd) and keeps it until the disk budget evicts oldest first. resetArchiveRetention now governs both deleted and reset archives and defaults to keep; maxDiskBytes defaults to 2gb so retention stays bounded, with archives evicted before live sessions. The cron reaper follows the same knob instead of deleting archives on its own timer. * fix(state): converge agent DB migration lineages and bound database growth Merge coherence: run both structure-gated legacy memory-schema repairs (flip-lineage drop, main-lineage identity rebuild) before the flip migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all converge, and hoist foreign_keys=OFF outside the schema transaction where the pragma was silently ignored and the v1 sessions rebuild cascade-deleted session_entries. Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL maintenance releases freed pages in bounded passes (never a blocking full VACUUM), and doctor reports state/agent DB bloat from freelist stats. * fix(codex): resolve the store path for thread-history import via the SDK The supervision catalog passed the legacy sessionFile locator to the storePath-targeted transcript mirror; resolve the agent store path with the session-store SDK helper instead of a runtime-object seam so test fakes and headless callers need no extra surface. Drop the obsolete missing-session-id preprocessing case: sessions rows are NOT NULL on session_id and upsert repairs id-less patches at write time. * fix(sessions): fail safe on malformed disk-budget config and doctor stat errors A malformed explicit maxDiskBytes disables the budget instead of falling back to the destructive 2gb default the user never chose, and the doctor bloat check skips databases whose paths stat-fail instead of aborting doctor. * fix(sessions): complete sqlite conflict translations * test(sqlite): align hardening checks with maintenance * test(sessions): inspect compressed transcript archives * fix(tests): await session seeds and drop unused helpers flagged by CI lint The five unawaited writeSessionStoreSeed calls raced their SQLite seeds against the assertions, failing compact shards; the bloat probe drops a useless initializer and the merged tests drop now-unused helpers. * test(sessions): type legacy proof events directly * test(sessions): align hardening contracts * perf(sessions): read usage transcript sizes from SQL aggregates Usage/cost scans walked every session and materialized every transcript event just to re-stringify it for a byte estimate — the #86718 stall class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes in SQLite without loading a single row. * fix(sessions): re-root foreign-root transcript paths onto the current sessions dir Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry absolute sessionFile paths from the old root; the containment fallback kept those foreign paths, so migration read (and would archive) files in the original root and reported local copies missing. Re-root the canonical agents/<id>/sessions suffix onto the current dir when the file exists there; genuine cross-root layouts still fall through unchanged. * test(agents): seed harness admission through sqlite * fix(sqlite): close agent db on pragma setup failure * fix(doctor): compact and retrofit incremental auto-vacuum after session import The migration is the sanctioned offline window: post-import compact reclaims import churn and applies auto_vacuum=INCREMENTAL to databases created before the fresh-DB pragma existed, so runtime maintenance can release pages in bounded passes on every install. --------- Co-authored-by: Peter Steinberger <steipete@gmail.com>
158 lines
6.3 KiB
TypeScript
158 lines
6.3 KiB
TypeScript
// Built-CLI SQLite flip proof requires dist entrypoints before running the gateway lifecycle.
|
|
import { describe, expect, it } from "vitest";
|
|
import { runSqliteSessionsTranscriptsFlipProof } from "../../scripts/e2e/sqlite-sessions-transcripts-flip-proof.ts";
|
|
|
|
describe("SQLite sessions/transcripts flip built CLI proof", () => {
|
|
it("proves the lifecycle through the built gateway CLI entrypoint", async () => {
|
|
const report = await runSqliteSessionsTranscriptsFlipProof({ requireBuiltCli: true });
|
|
|
|
expect(report.gatewayEntrypoint).toEqual(
|
|
expect.arrayContaining([expect.stringMatching(/^dist\/index\.(?:js|mjs)$/u)]),
|
|
);
|
|
expect(report.failures).toEqual([]);
|
|
expect(report.ok).toBe(true);
|
|
expect(
|
|
report.checkpoints
|
|
.filter((checkpoint) => checkpoint.label !== "seeded-legacy-store")
|
|
.every((checkpoint) => checkpoint.activeJsonl.length === 0),
|
|
).toBe(true);
|
|
expect(
|
|
report.checkpoints.some(
|
|
(checkpoint) =>
|
|
checkpoint.label === "seeded-legacy-store" && checkpoint.legacyStateJsonl.length > 0,
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
report.checkpoints
|
|
.filter((checkpoint) => checkpoint.label !== "seeded-legacy-store")
|
|
.every((checkpoint) => checkpoint.legacyStateJsonl.length === 0),
|
|
).toBe(true);
|
|
expect(report.checkpoints.some((checkpoint) => checkpoint.label === "after-doctor-fix")).toBe(
|
|
false,
|
|
);
|
|
expect(
|
|
report.checkpoints.some(
|
|
(checkpoint) =>
|
|
checkpoint.label === "after-startup-import" &&
|
|
checkpoint.gatewayLogTail?.includes(
|
|
"session: imported legacy session metadata/transcripts into SQLite",
|
|
) === true &&
|
|
report.oldStateSessionKeys.every((key) =>
|
|
checkpoint.sqlite.trackedEntries.some((entry) => entry.sessionKey === key),
|
|
) &&
|
|
checkpoint.sqlite.sessionEntries >= 7 &&
|
|
checkpoint.sqlite.transcriptEvents >= 13,
|
|
),
|
|
).toBe(true);
|
|
const startupImportCheckpoint = report.checkpoints.find(
|
|
(checkpoint) => checkpoint.label === "after-startup-import",
|
|
);
|
|
expect(
|
|
startupImportCheckpoint?.archiveArtifacts.some(
|
|
(artifact) =>
|
|
artifact.path.includes(`${report.legacySessionId}.trajectory.jsonl`) &&
|
|
artifact.textTail?.includes("trajectory") === true,
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
report.checkpoints.some(
|
|
(checkpoint) =>
|
|
checkpoint.label === "after-chat-send" &&
|
|
checkpoint.sqlite.trackedEntries.some(
|
|
(entry) => entry.sessionKey === report.resetSessionKey && entry.transcriptEvents >= 3,
|
|
),
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
report.checkpoints.some(
|
|
(checkpoint) =>
|
|
checkpoint.label === "after-full-agent-turn" &&
|
|
checkpoint.sqlite.trackedEntries.some(
|
|
(entry) =>
|
|
entry.sessionKey === report.fullTurnSessionKey && entry.transcriptEvents >= 2,
|
|
),
|
|
),
|
|
).toBe(true);
|
|
expect(report.pluginSdkConsumer).toMatchObject({
|
|
activeJsonlForSessionExists: false,
|
|
latestAssistantTextBeforeAppend: report.fullTurnAssistantText,
|
|
latestAssistantTextAfterAppend: "sqlite sdk consumer appended by identity",
|
|
sessionKey: report.pluginSdkSessionKey,
|
|
});
|
|
expect(report.pluginSdkConsumer?.sessionFileMarker.startsWith("sqlite:")).toBe(true);
|
|
expect(report.pluginSdkConsumer?.listedSessionKeys).toContain(report.pluginSdkSessionKey);
|
|
expect(
|
|
report.checkpoints.some(
|
|
(checkpoint) =>
|
|
checkpoint.label === "after-plugin-sdk-consumer" &&
|
|
checkpoint.sqlite.trackedEntries.some(
|
|
(entry) =>
|
|
entry.sessionKey === report.pluginSdkSessionKey && entry.transcriptEvents >= 3,
|
|
),
|
|
),
|
|
).toBe(true);
|
|
const cleanupCheckpoint = report.checkpoints.find(
|
|
(checkpoint) => checkpoint.label === "after-cleanup-pruning",
|
|
);
|
|
expect(
|
|
cleanupCheckpoint?.sqlite.trackedEntries.some(
|
|
(entry) => entry.sessionKey === report.cleanupPruneSessionKey,
|
|
),
|
|
).toBe(false);
|
|
const cleanupArchive = cleanupCheckpoint?.archiveArtifacts.find(
|
|
(artifact) =>
|
|
artifact.archiveReason === "deleted" &&
|
|
artifact.archiveSessionId === "sqlite-cleanup-prune",
|
|
);
|
|
expect(cleanupArchive?.messageTexts).toContain("sqlite cleanup prune me");
|
|
const idempotenceCheckpoint = report.checkpoints.find(
|
|
(checkpoint) => checkpoint.label === "after-doctor-import-idempotence",
|
|
);
|
|
expect(idempotenceCheckpoint?.doctor).toMatchObject({
|
|
code: 0,
|
|
mode: "import",
|
|
totals: expect.objectContaining({
|
|
importedEntries: 0,
|
|
importedTranscriptEvents: 0,
|
|
}),
|
|
});
|
|
const resetCheckpoint = report.checkpoints.find(
|
|
(checkpoint) => checkpoint.label === "after-sessions-reset",
|
|
);
|
|
const resetArchive = resetCheckpoint?.archiveArtifacts.find(
|
|
(artifact) =>
|
|
artifact.archiveReason === "reset" && artifact.archiveSessionId === report.legacySessionId,
|
|
);
|
|
expect(resetArchive?.messageTexts).toContain("legacy hello");
|
|
expect(resetArchive?.messageTexts).toContain("sqlite user-facing send before reset");
|
|
const sharedFirstCheckpoint = report.checkpoints.find(
|
|
(checkpoint) => checkpoint.label === "after-shared-first-delete",
|
|
);
|
|
expect(
|
|
sharedFirstCheckpoint?.archiveArtifacts.some(
|
|
(artifact) =>
|
|
artifact.archiveReason === "deleted" &&
|
|
artifact.archiveSessionId === "sqlite-shared-session",
|
|
),
|
|
).toBe(false);
|
|
const concurrentCheckpoint = report.checkpoints.find(
|
|
(checkpoint) => checkpoint.label === "after-concurrent-multi-client",
|
|
);
|
|
expect(concurrentCheckpoint).toBeDefined();
|
|
const concurrentSend = concurrentCheckpoint?.sqlite.trackedEntries.find(
|
|
(entry) => entry.sessionKey === report.concurrentSendSessionKey,
|
|
);
|
|
expect(concurrentSend?.transcriptEvents).toBeGreaterThanOrEqual(2);
|
|
expect(
|
|
concurrentCheckpoint?.sqlite.trackedEntries.some(
|
|
(entry) => entry.sessionKey === report.concurrentResetSessionKey && entry.sessionId,
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
concurrentCheckpoint?.sqlite.trackedEntries.some(
|
|
(entry) => entry.sessionKey === report.concurrentDeleteSessionKey,
|
|
),
|
|
).toBe(false);
|
|
}, 180_000);
|
|
});
|