mirror of
https://github.com/openclaw/openclaw.git
synced 2026-05-29 14:35:17 +00:00
99d96c1ff2
Summary: - The PR extracts the CJK-aware memory tokenizer into a shared helper, routes dreaming dedupe through it, preserves MMR re-exports, and adds regression coverage for CJK and empty-token cases. - PR surface: Source +15, Tests +96. Total +111 across 5 files. - Reproducibility: yes. Current main has an ASCII-only tokenizeSnippet path in dreaming dedupe, and the source ... ction source bytes for the CJK failure modes; I did not run tests locally because this review is read-only. Automerge notes: - PR branch already contained follow-up commit before automerge: fix(memory-core): use Array.toSorted for #80613 lint fix - PR branch already contained follow-up commit before automerge: fix(memory-core): preserve dedupe identity when both snippets tokeniz… - PR branch already contained follow-up commit before automerge: fix(memory-core): rename __testing to testing in CJK regression tests… - PR branch already contained follow-up commit before automerge: fix(memory-core): use CJK-aware tokenizer for dreaming dedupe (#80613) Validation: - ClawSweeper review passed for headca9c02734c. - Required merge gates passed before the squash merge. Prepared head SHA:ca9c02734cReview: https://github.com/openclaw/openclaw/pull/86645#issuecomment-4537414471 Co-authored-by: MoerAI <friendnt@g.skku.edu> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>