Files
openclaw/src/context-engine/delegate.ts
Josh Lehman 0a8e3604ba refactor: flip sessions and transcripts to sqlite storage (#98236)
* refactor(sessions): migrate runtime storage to sqlite

* test(sessions): fix sqlite CI regressions

* test(sessions): align remaining sqlite fixtures

* fix(codex): require sqlite trajectory recorder

* test(sessions): align orphan recovery sqlite fixture

* test(sessions): align sqlite rebase fixtures

* fix(sessions): finish current-main integration of the sqlite flip

Resolve the whole-store SDK removal across its owner boundary: drop the
loadSessionStore re-export and the registry whole-store wrappers, wire
hasTrackedActiveSessionRun into gateway chat, complete the
preserveLockedHarnessIds cleanup contract, flip the codex thread-history
import to storePath targets, and port remaining main-side tests from
file-store helpers to session accessor reads.

* chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs

Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore
the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain
bump gets its own review, and regenerate docs_map, the plugin SDK API baseline,
and the export-surface ratchet for the merged tree.

* feat(sessions): keep archived transcripts by default with zstd cold storage

Codex-style retention: deleting or resetting a session archives its
transcript as a zstd-compressed JSONL artifact (plain when the runtime
lacks node:zlib zstd) and keeps it until the disk budget evicts oldest
first. resetArchiveRetention now governs both deleted and reset archives
and defaults to keep; maxDiskBytes defaults to 2gb so retention stays
bounded, with archives evicted before live sessions. The cron reaper
follows the same knob instead of deleting archives on its own timer.

* fix(state): converge agent DB migration lineages and bound database growth

Merge coherence: run both structure-gated legacy memory-schema repairs
(flip-lineage drop, main-lineage identity rebuild) before the flip
migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all
converge, and hoist foreign_keys=OFF outside the schema transaction
where the pragma was silently ignored and the v1 sessions rebuild
cascade-deleted session_entries.

Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL
maintenance releases freed pages in bounded passes (never a blocking
full VACUUM), and doctor reports state/agent DB bloat from freelist
stats.

* fix(codex): resolve the store path for thread-history import via the SDK

The supervision catalog passed the legacy sessionFile locator to the
storePath-targeted transcript mirror; resolve the agent store path with
the session-store SDK helper instead of a runtime-object seam so test
fakes and headless callers need no extra surface. Drop the obsolete
missing-session-id preprocessing case: sessions rows are NOT NULL on
session_id and upsert repairs id-less patches at write time.

* fix(sessions): fail safe on malformed disk-budget config and doctor stat errors

A malformed explicit maxDiskBytes disables the budget instead of
falling back to the destructive 2gb default the user never chose, and
the doctor bloat check skips databases whose paths stat-fail instead of
aborting doctor.

* fix(sessions): complete sqlite conflict translations

* test(sqlite): align hardening checks with maintenance

* test(sessions): inspect compressed transcript archives

* fix(tests): await session seeds and drop unused helpers flagged by CI lint

The five unawaited writeSessionStoreSeed calls raced their SQLite seeds
against the assertions, failing compact shards; the bloat probe drops a
useless initializer and the merged tests drop now-unused helpers.

* test(sessions): type legacy proof events directly

* test(sessions): align hardening contracts

* perf(sessions): read usage transcript sizes from SQL aggregates

Usage/cost scans walked every session and materialized every transcript
event just to re-stringify it for a byte estimate — the #86718 stall
class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes
in SQLite without loading a single row.

* fix(sessions): re-root foreign-root transcript paths onto the current sessions dir

Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry
absolute sessionFile paths from the old root; the containment fallback
kept those foreign paths, so migration read (and would archive) files in
the original root and reported local copies missing. Re-root the
canonical agents/<id>/sessions suffix onto the current dir when the file
exists there; genuine cross-root layouts still fall through unchanged.

* test(agents): seed harness admission through sqlite

* fix(sqlite): close agent db on pragma setup failure

* fix(doctor): compact and retrofit incremental auto-vacuum after session import

The migration is the sanctioned offline window: post-import compact
reclaims import churn and applies auto_vacuum=INCREMENTAL to databases
created before the fresh-DB pragma existed, so runtime maintenance can
release pages in bounded passes on every install.

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-11 14:50:37 -07:00

153 lines
6.2 KiB
TypeScript

// Context-engine delegates bridge custom engines to built-in compaction and memory prompt paths.
import { normalizeStructuredPromptSection } from "@openclaw/ai/internal/shared";
import { parseSqliteSessionFileMarker } from "../config/sessions/sqlite-marker.js";
import type { MemoryCitationsMode } from "../config/types.memory.js";
import { buildMemoryPromptSection } from "../plugins/memory-state.js";
import { createLazyRuntimeModule } from "../shared/lazy-runtime.js";
import type {
ContextEngine,
CompactResult,
ContextEngineRuntimeContext,
ContextEngineSessionTarget,
} from "./types.js";
const loadCompactRuntime = createLazyRuntimeModule(
() => import("../agents/embedded-agent-runner/compact.runtime.js"),
);
function buildCompactionResultSessionTarget(params: {
agentId?: string;
sessionFile?: string;
sessionId?: string;
sessionKey?: string;
sessionTarget?: ContextEngineSessionTarget;
}): ContextEngineSessionTarget | undefined {
const sqliteMarker = parseSqliteSessionFileMarker(params.sessionFile);
const sessionId = sqliteMarker?.sessionId ?? params.sessionId;
if (!sessionId) {
return undefined;
}
const agentId = params.sessionTarget?.agentId ?? params.agentId ?? sqliteMarker?.agentId;
const sessionKey = params.sessionTarget?.sessionKey ?? params.sessionKey;
const storePath = params.sessionTarget?.storePath ?? sqliteMarker?.storePath;
return {
...(agentId ? { agentId } : {}),
sessionId,
...(sessionKey ? { sessionKey } : {}),
...(storePath ? { storePath } : {}),
...(params.sessionTarget?.threadId !== undefined
? { threadId: params.sessionTarget.threadId }
: {}),
};
}
/**
* Delegate a context-engine compaction request to OpenClaw's built-in runtime compaction path.
*
* This is the same bridge used by the legacy context engine. Third-party
* engines can call it from their own `compact()` implementations when they do
* not own the compaction algorithm but still need `/compact` and overflow
* recovery to use the stock runtime behavior.
*
* Note: `compactionTarget` is part of the public `compact()` contract, but the
* built-in runtime compaction path does not expose that knob. This helper
* ignores it to preserve legacy behavior; engines that need target-specific
* compaction should implement their own `compact()` algorithm.
*/
export async function delegateCompactionToRuntime(
params: Parameters<ContextEngine["compact"]>[0],
): Promise<CompactResult> {
// Load through the dedicated runtime boundary without introducing another
// source-level static edge into the embedded runner graph.
const { compactEmbeddedAgentSessionDirect } = await loadCompactRuntime();
type RuntimeCompactionParams = Parameters<typeof compactEmbeddedAgentSessionDirect>[0];
// runtimeContext carries host-resolved runtime fields set by internal
// callers. Keep the public delegate keyed by session identity, not by the
// active transcript artifact that the runtime may resolve internally.
const runtimeContext = (params.runtimeContext ?? {}) as ContextEngineRuntimeContext &
Partial<RuntimeCompactionParams>;
const { sessionFile: _legacySessionFile, ...runtimeContextParams } = runtimeContext;
const sessionTarget = params.sessionTarget ?? runtimeContext.sessionTarget;
const agentId = params.agentId ?? runtimeContext.agentId;
const sessionKey = params.sessionKey ?? runtimeContext.sessionKey;
const currentTokenCount =
params.currentTokenCount ??
(typeof runtimeContext.currentTokenCount === "number" &&
Number.isFinite(runtimeContext.currentTokenCount) &&
runtimeContext.currentTokenCount > 0
? Math.floor(runtimeContext.currentTokenCount)
: undefined);
const result = await compactEmbeddedAgentSessionDirect({
...runtimeContextParams,
...(agentId ? { agentId } : {}),
sessionId: params.sessionId,
...(sessionKey ? { sessionKey } : {}),
...(sessionTarget ? { sessionTarget } : {}),
tokenBudget: params.tokenBudget,
...(currentTokenCount !== undefined ? { currentTokenCount } : {}),
force: params.force,
customInstructions: params.customInstructions,
abortSignal: params.abortSignal,
workspaceDir:
typeof runtimeContext.workspaceDir === "string" ? runtimeContext.workspaceDir : process.cwd(),
});
const resultSessionTarget = result.result
? buildCompactionResultSessionTarget({
agentId,
sessionFile: result.result.sessionFile,
sessionId: result.result.sessionId,
sessionKey,
sessionTarget,
})
: undefined;
return {
ok: result.ok,
compacted: result.compacted,
reason: result.reason,
result: result.result
? {
summary: result.result.summary,
firstKeptEntryId: result.result.firstKeptEntryId,
tokensBefore: result.result.tokensBefore,
tokensAfter: result.result.tokensAfter,
details: result.result.details,
...(result.result.sessionId ? { sessionId: result.result.sessionId } : {}),
// Core reports successors only through the typed sessionTarget; the
// deprecated raw sessionFile field is reserved for shipped engines
// reporting rotation to core, and post-flip core has no file path.
...(resultSessionTarget ? { sessionTarget: resultSessionTarget } : {}),
}
: undefined,
};
}
/**
* Build a context-engine-ready systemPromptAddition from the active memory
* plugin prompt path. This lets non-legacy engines explicitly opt into the
* same memory/wiki guidance that the legacy engine gets via system prompt
* assembly, without reimplementing memory prompt formatting.
*/
export function buildMemorySystemPromptAddition(params: {
availableTools: Set<string>;
citationsMode?: MemoryCitationsMode;
agentId?: string;
agentSessionKey?: string;
sandboxed?: boolean;
}): string | undefined {
const lines = buildMemoryPromptSection({
availableTools: params.availableTools,
citationsMode: params.citationsMode,
agentId: params.agentId,
agentSessionKey: params.agentSessionKey,
sandboxed: params.sandboxed,
});
if (lines.length === 0) {
return undefined;
}
const normalized = normalizeStructuredPromptSection(lines.join("\n"));
return normalized || undefined;
}