Retry live query embeddings on transient provider transport failures and split eligible batch embedding socket failures after bounded retries.
Fixes#71784Fixes#44166
Supersedes #44167
Co-authored-by: MrGeDiao <MrGeDiao@users.noreply.github.com>
Adds a report-only memory-core dreaming shadow-trial runner that writes inspectable artifacts without mutating durable memory. The public helper now stores default reports under daily directories with opaque content-hash filenames, so multiple same-day trials coexist without leaking candidate text into paths.
Verification:
- OPENCLAW_VITEST_MAX_WORKERS=1 node scripts/run-vitest.mjs run --config test/vitest/vitest.extension-memory.config.ts extensions/memory-core/src/dreaming-shadow-trial.test.ts --reporter=verbose --maxWorkers=1
- git diff --check
- pnpm exec oxfmt --check extensions/memory-core/src/dreaming-shadow-trial.ts extensions/memory-core/src/dreaming-shadow-trial.test.ts
- pnpm tsgo:extensions
- autoreview clean: no accepted/actionable findings
- GitHub CI run 26709794635 passed
- Real behavior proof run 26709798698 passed
- Dependency Guard run 26709794113 passed
Co-authored-by: Firas Alswihry <itzfiras@gmail.com>
Compact promoted short-term memory snippets before writing them into MEMORY.md, while keeping the full rehydrated snippet in recall state for ranking/provenance. Adds the deep-dreaming config surface and docs, with the default promoted snippet cap set to 160 estimated tokens.
Verification:
- git diff --check
- fnm exec --using v24.13.0 node scripts/run-vitest.mjs run extensions/memory-core/src/short-term-promotion.test.ts extensions/memory-core/src/dreaming.test.ts src/memory-host-sdk/dreaming.test.ts
- GitHub CI run 26605272497
- CodeQL security run 26605272404
Co-authored-by: AMARA <amara@eyeinthesky.pl>
Cap Dreaming short-term recall stores so repeated recall recording, repair, and promotion application cannot grow the JSON artifact without bound.
The fix keeps full normalized snippets for recall identity and contamination checks before truncating persisted snippets, exposes the new overflow audit code through the SDK facade, and adds regression coverage for recording, repair, promotion rehydration, and deterministic retention ties.
Fixes#87095.
Verification:
- OPENCLAW_VITEST_MAX_WORKERS=1 node scripts/run-vitest.mjs extensions/memory-core/src/short-term-promotion.test.ts src/commands/doctor-memory-search.test.ts src/plugin-sdk/memory-core-engine-runtime.test.ts
- pnpm tsgo:prod
- pnpm check:test-types
- pnpm lint --threads=8
- git diff --check
- .agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main
- PR CI run 26594527697: unrelated current-main failures only in checks-node-agentic-plugin-sdk and checks-node-agentic-agents; same failures reproduced on main run 26594198639.
Co-authored-by: ai-hpc <mail.speedy.hpc@hotmail.com>
Refactor memory close provider draining so providers created during shutdown are closed through the same bounded retry path.
Co-authored-by: spacegeologist <zheng.zuo0@gmail.com>
Summary:
- The PR extracts the CJK-aware memory tokenizer into a shared helper, routes dreaming dedupe through it, preserves MMR re-exports, and adds regression coverage for CJK and empty-token cases.
- PR surface: Source +15, Tests +96. Total +111 across 5 files.
- Reproducibility: yes. Current main has an ASCII-only tokenizeSnippet path in dreaming dedupe, and the source ... ction source bytes for the CJK failure modes; I did not run tests locally because this review is read-only.
Automerge notes:
- PR branch already contained follow-up commit before automerge: fix(memory-core): use Array.toSorted for #80613 lint fix
- PR branch already contained follow-up commit before automerge: fix(memory-core): preserve dedupe identity when both snippets tokeniz…
- PR branch already contained follow-up commit before automerge: fix(memory-core): rename __testing to testing in CJK regression tests…
- PR branch already contained follow-up commit before automerge: fix(memory-core): use CJK-aware tokenizer for dreaming dedupe (#80613)
Validation:
- ClawSweeper review passed for head ca9c02734c.
- Required merge gates passed before the squash merge.
Prepared head SHA: ca9c02734c
Review: https://github.com/openclaw/openclaw/pull/86645#issuecomment-4537414471
Co-authored-by: MoerAI <friendnt@g.skku.edu>
Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>
Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>
* fix(memory): prevent silent vector index degradation when embedding provider temporarily unavailable
Two related bugs cause complete loss of semantic vector data:
1. Promise cache deadlock in ensureProviderInitialized():
When the embedding provider (e.g. local MLX server on port 8123) is
temporarily unreachable at Gateway startup, loadProviderResult() throws
and providerInitPromise becomes a permanently-cached Rejected Promise.
The block only clears it on success (providerInitialized=true),
so the stale rejection blocks all future init attempts until Gateway restart.
2. Silent fts-only overwrite in runSync():
With the provider stuck at null, shouldRunFullMemoryReindex() compares
the stored meta.model (e.g. 'jina-embeddings-v5-text-small') against the
runtime provider model, and since provider is null, falls through to the
'meta.model !== fts-only' check — returning true. This triggers a full
reindex where every file is written as fts-only, silently erasing all
existing 11k+ semantic vectors.
Fix 1: Clear providerInitPromise in the catch block so the next call can
retry initialization (self-healing when the provider comes back online).
Fix 2: Guard runSync() — if requestedProvider is set and not 'none', but
the runtime provider is null, throw an error instead of silently degrading
to fts-only. This protects existing vector data by failing loudly.
Tested on production: 11,715 chunks + 1024-dim vectors fully preserved
after Gateway restart with the fix applied. The guard correctly blocks
sync when MLX is offline and allows normal operation when it recovers.
* fix: use this.settings.provider instead of private requestedProvider
The guard clause in runSync() was referencing this.requestedProvider
which is a private property on the MemoryIndexManager subclass and not
accessible from MemoryManagerSyncOps. Use this.settings.provider
instead, which is the same value and is accessible via the protected
abstract settings property.
* fix(memory): narrow degradation guard to only protect existing semantic indexes
The previous guard was too broad — it blocked sync for ALL non-none
provider configurations when provider was null, including the default
'auto' path where users without embedding credentials legitimately
build FTS-only indexes.
Narrow the guard to only abort when:
1. provider is null (embedding unavailable)
2. existing index metadata has a semantic model (not 'fts-only')
3. settings.provider is configured and not 'none'
This preserves the legitimate FTS-only fallback for auto/no-provider
users while still protecting existing semantic vector indexes from
silent degradation.
Reported-by: ClawSweeper (PR #85704 review)
* test: cover memory semantic index outage guard
* fix: protect semantic memory index fallback paths
* test: update memory sync harnesses
---------
Co-authored-by: Bo Yan <yaaboo-gif@users.noreply.github.com>
Co-authored-by: Yan Bo <yanbo@Mac.lan>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
Summary:
- The PR routes local GGUF memory embeddings through a bundled worker sidecar, adds structured degradation and fallback handling, updates memory tests/build output, and keeps the local config contract unchanged.
- PR surface: Source +831, Tests +503, Docs +1, Other +2. Total +1337 across 23 files.
- Reproducibility: Do we have a high-confidence way to reproduce the issue? Source and report evidence are str ... cludes native crash logs; the exact Metal teardown abort was not reproduced in this review or the PR proof.
Automerge notes:
- PR branch already contained follow-up commit before automerge: fix(memory): keep local embedding config unchanged
- PR branch already contained follow-up commit before automerge: fix(memory): type local embedding degradation
- PR branch already contained follow-up commit before automerge: fix(memory): refresh keywords after embedding fallback
- PR branch already contained follow-up commit before automerge: fix(memory): keep worker errors internal
- PR branch already contained follow-up commit before automerge: test: satisfy memory provider lifecycle harnesses
- PR branch already contained follow-up commit before automerge: fix: harden local embedding worker fallback
Validation:
- ClawSweeper review passed for head 1d1fe41c4e.
- Required merge gates passed before the squash merge.
Prepared head SHA: 1d1fe41c4e
Review: https://github.com/openclaw/openclaw/pull/85348#issuecomment-4518516047
Co-authored-by: Onur Solmaz <onur@Onurs-MacBook-Pro.local>
Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>
Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>
Approved-by: osolmaz
Co-authored-by: osolmaz <2453968+osolmaz@users.noreply.github.com>
* fix(memory-core): filter REM dreaming candidates to light-staged entries
REM dreaming re-ingested the full short-term recall store independently,
ignoring which entries were staged by the light sleep phase. Because the
confidence formula heavily weights accumulated averageScore (45%) and
recallStrength (25%), old high-recall entries permanently dominated
freshly staged candidates. The intended light→REM→deep pipeline was
broken: light correctly staged current material, but REM selected a
different set entirely, so lightHits never paired with remHits for deep
ranking.
Fix: in runRemDreaming(), read the phase-signals store for keys with
lightHits > 0 and filter entries to that set before passing to
previewRemDreaming(). When no light-staged keys exist (light disabled
or first run), fall back to the full entry set for backward
compatibility.
Added readLightStagedKeys() to short-term-promotion.ts as a clean
export for reading the light-staged key set from the phase signal store.
Closes#86249
Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
* fix(memory-core): keep REM staging pending
* fix(memory-core): mark REM-considered staged entries
---------
Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
