Commit Graph

238 Commits

Author SHA1 Message Date
ZOOWH
fb702e5cae fix(memory): harden embedding batch workflows (#103472)
* fix(memory): harden embedding batch workflows

Co-authored-by: 徐闻涵0668001344 <xu.wenhan1@xydigit.com>
Co-authored-by: ben.li <li.yang6@xydigit.com>

* test(memory): align batch redaction expectation

* test(memory): tighten batch dependency types

* refactor(google): clarify batch state control flow

* test(google): make batch stage fallback lint-safe

* build(plugin-sdk): refresh memory batch surface budget

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
Co-authored-by: ben.li <li.yang6@xydigit.com>
2026-07-11 16:43:58 -07:00
Josh Lehman
0a8e3604ba refactor: flip sessions and transcripts to sqlite storage (#98236)
* refactor(sessions): migrate runtime storage to sqlite

* test(sessions): fix sqlite CI regressions

* test(sessions): align remaining sqlite fixtures

* fix(codex): require sqlite trajectory recorder

* test(sessions): align orphan recovery sqlite fixture

* test(sessions): align sqlite rebase fixtures

* fix(sessions): finish current-main integration of the sqlite flip

Resolve the whole-store SDK removal across its owner boundary: drop the
loadSessionStore re-export and the registry whole-store wrappers, wire
hasTrackedActiveSessionRun into gateway chat, complete the
preserveLockedHarnessIds cleanup contract, flip the codex thread-history
import to storePath targets, and port remaining main-side tests from
file-store helpers to session accessor reads.

* chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs

Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore
the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain
bump gets its own review, and regenerate docs_map, the plugin SDK API baseline,
and the export-surface ratchet for the merged tree.

* feat(sessions): keep archived transcripts by default with zstd cold storage

Codex-style retention: deleting or resetting a session archives its
transcript as a zstd-compressed JSONL artifact (plain when the runtime
lacks node:zlib zstd) and keeps it until the disk budget evicts oldest
first. resetArchiveRetention now governs both deleted and reset archives
and defaults to keep; maxDiskBytes defaults to 2gb so retention stays
bounded, with archives evicted before live sessions. The cron reaper
follows the same knob instead of deleting archives on its own timer.

* fix(state): converge agent DB migration lineages and bound database growth

Merge coherence: run both structure-gated legacy memory-schema repairs
(flip-lineage drop, main-lineage identity rebuild) before the flip
migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all
converge, and hoist foreign_keys=OFF outside the schema transaction
where the pragma was silently ignored and the v1 sessions rebuild
cascade-deleted session_entries.

Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL
maintenance releases freed pages in bounded passes (never a blocking
full VACUUM), and doctor reports state/agent DB bloat from freelist
stats.

* fix(codex): resolve the store path for thread-history import via the SDK

The supervision catalog passed the legacy sessionFile locator to the
storePath-targeted transcript mirror; resolve the agent store path with
the session-store SDK helper instead of a runtime-object seam so test
fakes and headless callers need no extra surface. Drop the obsolete
missing-session-id preprocessing case: sessions rows are NOT NULL on
session_id and upsert repairs id-less patches at write time.

* fix(sessions): fail safe on malformed disk-budget config and doctor stat errors

A malformed explicit maxDiskBytes disables the budget instead of
falling back to the destructive 2gb default the user never chose, and
the doctor bloat check skips databases whose paths stat-fail instead of
aborting doctor.

* fix(sessions): complete sqlite conflict translations

* test(sqlite): align hardening checks with maintenance

* test(sessions): inspect compressed transcript archives

* fix(tests): await session seeds and drop unused helpers flagged by CI lint

The five unawaited writeSessionStoreSeed calls raced their SQLite seeds
against the assertions, failing compact shards; the bloat probe drops a
useless initializer and the merged tests drop now-unused helpers.

* test(sessions): type legacy proof events directly

* test(sessions): align hardening contracts

* perf(sessions): read usage transcript sizes from SQL aggregates

Usage/cost scans walked every session and materialized every transcript
event just to re-stringify it for a byte estimate — the #86718 stall
class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes
in SQLite without loading a single row.

* fix(sessions): re-root foreign-root transcript paths onto the current sessions dir

Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry
absolute sessionFile paths from the old root; the containment fallback
kept those foreign paths, so migration read (and would archive) files in
the original root and reported local copies missing. Re-root the
canonical agents/<id>/sessions suffix onto the current dir when the file
exists there; genuine cross-root layouts still fall through unchanged.

* test(agents): seed harness admission through sqlite

* fix(sqlite): close agent db on pragma setup failure

* fix(doctor): compact and retrofit incremental auto-vacuum after session import

The migration is the sanctioned offline window: post-import compact
reclaims import churn and applies auto_vacuum=INCREMENTAL to databases
created before the fresh-DB pragma existed, so runtime maintenance can
release pages in bounded passes on every install.

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-11 14:50:37 -07:00
Peter Steinberger
e9b7a9a912 fix(memory): search by filenames and paths (#104449)
Co-authored-by: Pick-cat <huang.ting3@xydigit.com>
2026-07-11 10:54:34 -07:00
Peter Steinberger
e686693732 feat(tooling): adopt noUncheckedIndexedAccess via a strict-ratchet lane (phase 1) (#104577)
* feat(tooling): add noUncheckedIndexedAccess strict-ratchet lane

* fix(packages): make indexed access explicit across ratchet packages

Burns down all 65 noUncheckedIndexedAccess errors in the seven ratchet
packages with behavior-identical restructuring (iteration, charAt/slice,
regex-group guards, validated hextet tuple). net-policy invariant
violations now throw instead of failing open. Adds
@openclaw/normalization-core/expect (expectDefined/first/last) with
subpath export and tests. memory-host-sdk fixes kept but the package
stays out of the lane: it re-exports core src/** so the flag would apply
transitively to all of core.

* fix(lint): keep .oxlintrc.json strict-JSON parseable for extension lint wrappers
2026-07-11 10:26:12 -07:00
Vincent Koc
c79bac4018 test(memory): clean up llama worker fixtures 2026-07-11 16:40:14 +08:00
Vincent Koc
d8993d979d test(memory): describe requested llama context 2026-07-11 16:40:14 +08:00
Vincent Koc
9e39e9a862 fix(memory): complete llama runtime diagnostics 2026-07-11 16:40:14 +08:00
Vincent Koc
4b0e5f5461 feat(memory): capture llama.cpp runtime facts 2026-07-11 16:40:14 +08:00
Peter Steinberger
7bf80dc2c6 chore(tooling): enforce formatting and refresh TypeScript checks (#104239)
* chore(tooling): enforce current formatter and refresh checks

* chore(tooling): keep release changelog formatter-owned

* chore(tooling): retain compatible Node type surface

* ci: enforce formatting for docs-only changes

* ci: isolate docs formatter check

* chore(tooling): apply updated lint and format rules

* chore(tooling): satisfy updated switch lint

* style(ui): apply Linux formatter layout

* test(doctor): match quiet local audio contribution

* test(doctor): assert quiet output only

* test(doctor): follow restored information contract
2026-07-11 01:09:51 -07:00
Peter Steinberger
c8ebfd8a3a fix(security): preserve UTF-16 token mask boundaries (#103843) 2026-07-10 18:20:05 +01:00
Peter Steinberger
e484bdd993 fix(memory-host): keep redacted token hints UTF-16 safe (#103813)
Completes the sibling memory-host masker missed by #103341.

Co-authored-by: 张贵萍0668001030 <zhang.guiping@xydigit.com>
2026-07-10 17:43:37 +01:00
Peter Steinberger
d185f9a0b4 fix(infra): preserve characters in truncated response snippets (#103136)
* fix(infra): preserve bounded response text characters

Co-authored-by: qingminlong <34085845+qingminglong@users.noreply.github.com>

* ci(plugin-sdk): refresh API baseline

---------

Co-authored-by: qingminlong <34085845+qingminglong@users.noreply.github.com>
2026-07-09 23:43:26 +01:00
lsr911
bb1fa4012e fix(memory-host): use truncateUtf16Safe for QMD stderr context truncation (#102547)
* fix(memory-host): use truncateUtf16Safe for QMD stderr context truncation

* test(memory): cover UTF-16-safe qmd stderr summary

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-09 10:38:43 +01:00
QiuYuang
d5a31888ed fix(memory-host): reject queued worker requests on shutdown (#102451)
* fix(memory-host): reject queued worker requests on shutdown

* fix(memory-host): settle queued requests on shutdown

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-09 09:26:05 +01:00
qingminlong
c3d567db95 fix(memory): snippets split emoji when truncated (#102478)
* fix(memory): snippets split emoji when truncated

* refactor(memory): reuse shared UTF-16 truncation

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-09 08:27:42 +01:00
Vincent Koc
fee997c8ad refactor(memory): trim unused host SDK exports (#101604) 2026-07-07 04:33:43 -07:00
Vincent Koc
988f30e30e refactor(memory): trim unused Windows spawn exports (#101594) 2026-07-07 04:17:32 -07:00
Peter Steinberger
84d0a71406 fix(memory): preserve UTF-16 chunk boundaries (#101574)
* fix(memory): preserve UTF-16 chunk boundaries

* test(memory): require coarse boundary split

Co-authored-by: jensenwang560-blip <267012233+jensenwang560-blip@users.noreply.github.com>

* fix(memory): retain chunk budget at surrogate boundaries

---------

Co-authored-by: jensenwang560-blip <267012233+jensenwang560-blip@users.noreply.github.com>
2026-07-07 12:01:32 +01:00
Vincent Koc
098b471143 refactor(memory): remove unused runtime facade exports (#101583) 2026-07-07 03:57:51 -07:00
Vincent Koc
036686ab95 refactor(memory): localize host SDK helpers (#101568) 2026-07-07 03:28:00 -07:00
Vincent Koc
2e4e982ff0 refactor(memory): localize host SDK internals (#101508) 2026-07-07 01:50:53 -07:00
wings1029
5c4b63964c fix(memory-host-sdk): handle stdout/stderr stream errors in runCliCommand (#101402)
* fix(memory-host-sdk): handle stdout/stderr stream errors in runCliCommand

Register error handlers on stdout and stderr streams in
runCliCommand() to prevent uncaught exceptions when a pipe
breaks (e.g. EPIPE after child process exit).

Without these listeners, Node.js throws an uncaught exception
that crashes the process.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory-host-sdk): extract listenForChildOutputErrors helper

Refactor the inline stream error handling into a reusable
listenForChildOutputErrors helper, matching the pattern
established in the merged #101370 (agent-core).

The helper registers error listeners on stdout/stderr via a
shared loop, keeping the pattern consistent across the codebase.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory-host-sdk): harden qmd stream failures

* fix(memory-host-sdk): harden qmd stream failures

* docs(changelog): note qmd stream hardening

* chore: keep release changelog out of contributor PR

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-07 08:37:21 +01:00
Vincent Koc
85d6d94afd refactor(memory): remove unused host SDK exports (#101362) 2026-07-06 21:54:06 -07:00
Peter Steinberger
ad5b81e884 fix(memory): trust structured cron archive provenance (#101322)
Prevent user-authored [cron:] text from clearing reset or deleted session memory while preserving structured cron-run opacity.

Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
2026-07-07 04:52:21 +01:00
ZZIPP
979d198fd1 fix(media): recognize m2a as MPEG audio (#92167)
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-06 17:48:37 +01:00
Peter Steinberger
49cc59b1e8 refactor: consolidate markdown code fences, error coercion, and byte-identical helper pairs (#99932)
* refactor(shared): add markdown code span/fence helpers and migrate seven call sites

* refactor(normalization-core): add canonical toErrorObject and migrate six copies

* refactor: consolidate byte-identical helper pairs onto owner modules

* refactor: reuse canonical path and token helpers in session tools and credentials

* refactor(packages): dedupe compaction summarization tail and session dir parsing

* fix(security): keep missing extensions dir silent in shared plugin dir lister

* refactor: reuse media-core chunk reader and shared dreaming session key helper

* fix(plugins): register error-coercion subpath in sdk alias table

* chore(plugin-sdk): re-pin callable export count after rebase

* chore(plugin-sdk): re-pin callable export count after rebase
2026-07-04 08:40:41 -04:00
ly-wang19
4c4396c4c2 perf(memory): copy only requested embedding dimensions (#96952)
* perf(memory): copy only requested embedding dimensions

* perf(memory): copy only requested embedding dimensions

---------

Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
2026-06-27 05:56:02 +08:00
Bek
9636bea901 perf(memory): add QMD search diagnostics and runtime cache (#96655) 2026-06-26 16:16:12 -04:00
Josh Lehman
da50a450d2 fix(memory-core): route dreaming corpus through session corpus metadata (#96517) 2026-06-24 11:29:26 -07:00
mushuiyu886
414c250af9 fix #95495: [Bug]: 2026.6.9 silently relocates memory store with no migration, forcing a full re-embed (1499 files) with zero upgrade-time warning (#95631)
* fix(memory): import legacy sidecar indexes into agent db

* fix(memory): move legacy sidecar import to doctor migration

* fix(memory): restore sidecar vector rows during doctor migration

* fix(memory): keep legacy sidecar when skipping import

* fix(memory): keep legacy sidecar import within extension boundary

* fix(memory-core): keep legacy sidecar migration retry-safe

* fix(memory-core): backfill sidecar FTS rows

* fix(memory-core): preserve sidecar when vector import defers

* fix(memory-core): cover custom sidecar migrations

* fix(memory-core): keep legacy config migration under doctor

* fix(memory-core): reject sidecar metadata conflicts

* fix(memory-core): keep partial legacy config sidecars

* fix(memory-core): preserve partial config retries

* fix(memory-core): keep partial config task migrations

* fix(memory-core): avoid phantom sidecar agents

* fix(memory-core): reject incomplete sidecar indexes

* fix(memory-core): keep malformed sidecars retryable

* fix(doctor): use canonical state dir for plugin migrations

* fix(memory-core): honor disabled vector sidecar migration

* fix(memory-core): treat provider-none sidecars as fts-only

* fix(memory-core): preserve setup-failed sidecars

* test(memory-core): use non-mutating sort assertions

* test(memory-core): compare sorted chunk ids

* test(memory-core): compare sorted chunk ids

* test(memory-core): stringify sorted chunk ids

* fix(qa): skip chromium bootstrap for explicit browser channels

* fix(qa): skip chromium bootstrap for explicit browser channels

---------

Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
2026-06-24 17:47:44 +08:00
Peter Steinberger
c9ddf2eca6 test(memory): clean up qmd fixture gracefully 2026-06-23 13:31:46 -07:00
Peter Steinberger
73dd758310 fix(memory): abort orphaned qmd search processes 2026-06-23 13:31:46 -07:00
Alix-007
78184ea7e4 fix(memory): abort orphaned qmd search subprocess when memory_search times out
PR #91742 wired memory_search's 15s deadline AbortSignal through the builtin
memory manager but missed the QMD backend behind the same
MemorySearchManager.search interface. With QMD, the tool returns "timed out
after 15s" to the agent while the spawned qmd query/search subprocess keeps
running for the full qmd command timeout (memory.qmd.limits.timeoutMs, whose
embed-heavy default was raised to 600s in #87572), leaving orphaned
embedding/search work running after the agent already moved on.

Add optional AbortSignal support to runCliCommand: an aborting signal kills the
spawned child immediately and rejects with the abort reason, funneled through a
single settle() guard so abort/timeout/error/close cannot double-settle. Thread
the search signal through QmdMemoryManager.search -> runQmdSearch -> runQmd ->
runCliCommand for the default direct-qmd subprocess path (including the query
fallback), and fast-fail search() when the signal is already aborted.
2026-06-23 13:31:46 -07:00
Josh Lehman
c24d266b2d refactor: use accessor-backed transcript corpus for memory (#96162)
* refactor: ratchet memory transcript corpus access

* test: use narrow runtime config snapshot import

* test: update plugin sdk surface budgets

* refactor: split memory transcript corpus module
2026-06-23 12:37:44 -07:00
Ben.Li
b335381247 fix(memory): preserve Windows QMD command paths (#95274) 2026-06-22 17:50:11 +00:00
Vincent Koc
b3b5b08e67 fix(memory): preserve Windows session transcript paths 2026-06-22 23:32:06 +08:00
Josh Lehman
d3781cc4b8 refactor: add memory and QMD session identity mapping (#95087) 2026-06-22 06:28:54 -07:00
Vincent Koc
830691b201 fix(memory-host-sdk): taskkill qmd process trees on windows 2026-06-21 08:51:36 +02:00
Vincent Koc
c4d1f37d33 fix(memory): abort batch upload response reads (#95111)
* fix(memory): abort batch upload response reads

* test(memory): stabilize batch upload abort proof
2026-06-20 06:22:23 +08:00
Vincent Koc
6aa85dfaa1 refactor(memory): drop unused host-sdk helpers 2026-06-19 16:04:00 +08:00
Vincent Koc
33fa225f65 refactor(memory): drop unused host helpers 2026-06-19 15:13:27 +08:00
Vincent Koc
be7d86ed80 fix(memory): abort remote response body reads 2026-06-19 08:25:53 +02:00
Vincent Koc
681e5a0f8c fix(memory): preserve schema helper compatibility 2026-06-19 01:20:06 +08:00
Vincent Koc
6e6bd5633f fix(memory): rebuild migrated search indexes 2026-06-19 01:20:06 +08:00
Josh Lehman
28fb5b019a fix(memory): migrate source primary key shape 2026-06-19 01:20:06 +08:00
Josh Lehman
9cfd1cd287 fix(memory): key index sources by path and source 2026-06-19 01:20:06 +08:00
Vincent Koc
7485dd6492 fix(sqlite): migrate legacy memory and proxy state 2026-06-19 01:20:06 +08:00
Vincent Koc
88bc08c124 refactor(memory): canonicalize agent database tables 2026-06-19 01:20:06 +08:00
Vincent Koc
25e68e9fa7 fix(memory): reject stale reindex publishes 2026-06-19 01:20:06 +08:00
Vincent Koc
bdc46fa28d refactor(memory): remove unused session file exports 2026-06-18 11:24:02 +08:00