Files
openclaw/src/agents/btw-transcript.ts
Josh Lehman 0a8e3604ba refactor: flip sessions and transcripts to sqlite storage (#98236)
* refactor(sessions): migrate runtime storage to sqlite

* test(sessions): fix sqlite CI regressions

* test(sessions): align remaining sqlite fixtures

* fix(codex): require sqlite trajectory recorder

* test(sessions): align orphan recovery sqlite fixture

* test(sessions): align sqlite rebase fixtures

* fix(sessions): finish current-main integration of the sqlite flip

Resolve the whole-store SDK removal across its owner boundary: drop the
loadSessionStore re-export and the registry whole-store wrappers, wire
hasTrackedActiveSessionRun into gateway chat, complete the
preserveLockedHarnessIds cleanup contract, flip the codex thread-history
import to storePath targets, and port remaining main-side tests from
file-store helpers to session accessor reads.

* chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs

Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore
the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain
bump gets its own review, and regenerate docs_map, the plugin SDK API baseline,
and the export-surface ratchet for the merged tree.

* feat(sessions): keep archived transcripts by default with zstd cold storage

Codex-style retention: deleting or resetting a session archives its
transcript as a zstd-compressed JSONL artifact (plain when the runtime
lacks node:zlib zstd) and keeps it until the disk budget evicts oldest
first. resetArchiveRetention now governs both deleted and reset archives
and defaults to keep; maxDiskBytes defaults to 2gb so retention stays
bounded, with archives evicted before live sessions. The cron reaper
follows the same knob instead of deleting archives on its own timer.

* fix(state): converge agent DB migration lineages and bound database growth

Merge coherence: run both structure-gated legacy memory-schema repairs
(flip-lineage drop, main-lineage identity rebuild) before the flip
migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all
converge, and hoist foreign_keys=OFF outside the schema transaction
where the pragma was silently ignored and the v1 sessions rebuild
cascade-deleted session_entries.

Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL
maintenance releases freed pages in bounded passes (never a blocking
full VACUUM), and doctor reports state/agent DB bloat from freelist
stats.

* fix(codex): resolve the store path for thread-history import via the SDK

The supervision catalog passed the legacy sessionFile locator to the
storePath-targeted transcript mirror; resolve the agent store path with
the session-store SDK helper instead of a runtime-object seam so test
fakes and headless callers need no extra surface. Drop the obsolete
missing-session-id preprocessing case: sessions rows are NOT NULL on
session_id and upsert repairs id-less patches at write time.

* fix(sessions): fail safe on malformed disk-budget config and doctor stat errors

A malformed explicit maxDiskBytes disables the budget instead of
falling back to the destructive 2gb default the user never chose, and
the doctor bloat check skips databases whose paths stat-fail instead of
aborting doctor.

* fix(sessions): complete sqlite conflict translations

* test(sqlite): align hardening checks with maintenance

* test(sessions): inspect compressed transcript archives

* fix(tests): await session seeds and drop unused helpers flagged by CI lint

The five unawaited writeSessionStoreSeed calls raced their SQLite seeds
against the assertions, failing compact shards; the bloat probe drops a
useless initializer and the merged tests drop now-unused helpers.

* test(sessions): type legacy proof events directly

* test(sessions): align hardening contracts

* perf(sessions): read usage transcript sizes from SQL aggregates

Usage/cost scans walked every session and materialized every transcript
event just to re-stringify it for a byte estimate — the #86718 stall
class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes
in SQLite without loading a single row.

* fix(sessions): re-root foreign-root transcript paths onto the current sessions dir

Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry
absolute sessionFile paths from the old root; the containment fallback
kept those foreign paths, so migration read (and would archive) files in
the original root and reported local copies missing. Re-root the
canonical agents/<id>/sessions suffix onto the current dir when the file
exists there; genuine cross-root layouts still fall through unchanged.

* test(agents): seed harness admission through sqlite

* fix(sqlite): close agent db on pragma setup failure

* fix(doctor): compact and retrofit incremental auto-vacuum after session import

The migration is the sanctioned offline window: post-import compact
reclaims import churn and applies auto_vacuum=INCREMENTAL to databases
created before the fresh-DB pragma existed, so runtime maintenance can
release pages in bounded passes on every install.

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-11 14:50:37 -07:00

151 lines
5.2 KiB
TypeScript

/**
* Reads prior session transcript context for `/btw` side-question handoffs.
*/
import { readFile } from "node:fs/promises";
import {
resolveSessionFilePath,
resolveSessionFilePathOptions,
type SessionEntry as StoredSessionEntry,
} from "../config/sessions.js";
import { loadTranscriptEvents } from "../config/sessions/session-accessor.js";
import { parseSqliteSessionFileMarker } from "../config/sessions/sqlite-marker.js";
import {
scanSessionTranscriptTree,
type SessionTranscriptTree,
} from "../config/sessions/transcript-tree.js";
import { diagnosticLogger as diag } from "../logging/diagnostic.js";
import {
buildSessionContext,
migrateSessionEntries,
parseSessionEntries,
type SessionEntry as AgentSessionEntry,
} from "./sessions/session-manager.js";
/** Resolves the persisted transcript file for a BTW session handoff. */
export function resolveBtwSessionTranscriptPath(params: {
sessionId: string;
sessionEntry?: StoredSessionEntry;
sessionKey?: string;
storePath?: string;
}): string | undefined {
try {
const agentId = params.sessionKey?.split(":")[1];
const pathOpts = resolveSessionFilePathOptions({
agentId,
storePath: params.storePath,
});
return resolveSessionFilePath(params.sessionId, params.sessionEntry, pathOpts);
} catch (error) {
diag.debug(
`resolveSessionTranscriptPath failed: sessionId=${params.sessionId} err=${String(error)}`,
);
return undefined;
}
}
// Session entries can come from older transcript formats, so id fields are
// narrowed at this boundary before branch reconstruction trusts them.
function readSessionEntryId(entry: AgentSessionEntry): string | undefined {
const id = (entry as { id?: unknown }).id;
return typeof id === "string" && id.trim().length > 0 ? id : undefined;
}
// Reconstructs the selected branch from leaf to root. Missing links or cycles
// mean the snapshot cannot be trusted, so callers fall back to a safe branch.
function buildSessionBranchEntries(
tree: SessionTranscriptTree<AgentSessionEntry>,
leafId: string | null | undefined,
): AgentSessionEntry[] | undefined {
if (leafId === null) {
return [];
}
if (!leafId) {
return undefined;
}
const branch: AgentSessionEntry[] = [];
const seen = new Set<string>();
let currentId: string | undefined = leafId;
while (currentId) {
if (seen.has(currentId)) {
return undefined;
}
seen.add(currentId);
const node = tree.byId.get(currentId);
if (!node) {
return undefined;
}
if ((node.entry as { type?: unknown }).type !== "leaf") {
branch.push(
node.entry.parentId === node.parentId
? node.entry
: ({ ...node.entry, parentId: node.parentId } as AgentSessionEntry),
);
}
currentId = node.parentId ?? undefined;
}
return branch.toReversed();
}
function isTrailingUserMessage(entry: AgentSessionEntry | undefined): boolean {
return (
entry?.type === "message" &&
(entry as { message?: { role?: unknown } }).message?.role === "user"
);
}
/**
* Reads prior messages for BTW continuation.
*
* When a transcript has fork links, this returns the selected snapshot branch
* instead of the full file so a resumed agent does not inherit sibling-branch
* messages.
*/
export async function readBtwTranscriptMessages(params: {
sessionFile: string;
sessionId: string;
sessionKey?: string;
snapshotLeafId?: string | null;
}): Promise<unknown[]> {
try {
const marker = parseSqliteSessionFileMarker(params.sessionFile);
const entries = marker
? ((await loadTranscriptEvents({
agentId: marker.agentId,
sessionId: marker.sessionId,
...(params.sessionKey ? { sessionKey: params.sessionKey } : {}),
storePath: marker.storePath,
})) as AgentSessionEntry[])
: parseSessionEntries(await readFile(params.sessionFile, "utf-8"));
migrateSessionEntries(entries);
const sessionEntries = entries.filter(
(entry): entry is AgentSessionEntry => entry.type !== "session",
);
const tree = scanSessionTranscriptTree(sessionEntries);
if (!tree.hasLeafUpdate) {
return buildSessionContext(sessionEntries).messages;
}
const hasSnapshotLeaf = params.snapshotLeafId !== undefined;
let branchEntries = hasSnapshotLeaf
? buildSessionBranchEntries(tree, params.snapshotLeafId)
: undefined;
if (hasSnapshotLeaf && branchEntries === undefined) {
diag.debug(
`btw snapshot leaf unavailable: sessionId=${params.sessionId} leaf=${params.snapshotLeafId}`,
);
}
branchEntries ??= buildSessionBranchEntries(tree, tree.leafId);
if (!hasSnapshotLeaf && isTrailingUserMessage(branchEntries?.at(-1))) {
// Auto-selecting the newest branch must not include the current user turn
// that triggered BTW handoff; the subagent should continue from its parent.
const trailingId = readSessionEntryId(branchEntries!.at(-1)!);
const parentId = trailingId ? tree.byId.get(trailingId)?.parentId : null;
branchEntries = parentId ? (buildSessionBranchEntries(tree, parentId) ?? []) : [];
}
const sessionContext = buildSessionContext(branchEntries ?? sessionEntries);
return Array.isArray(sessionContext.messages) ? sessionContext.messages : [];
} catch {
return [];
}
}