buyitsydney
4b69c6d3f1
fix(memory): add CJK/Kana/Hangul support to MMR tokenize() for diversity detection
...
The tokenize() function only matched [a-z0-9_]+ patterns, returning an
empty set for CJK-only text. This made Jaccard similarity always 0 (or
always 1 for two empty sets) for CJK content, effectively disabling MMR
diversity detection.
Add support for:
- CJK Unified Ideographs (U+4E00–U+9FFF, U+3400–U+4DBF)
- Hiragana (U+3040–U+309F) and Katakana (U+30A0–U+30FF)
- Hangul Syllables (U+AC00–U+D7AF) and Jamo (U+1100–U+11FF)
Characters are extracted as unigrams, and bigrams are generated only
from characters that are adjacent in the original text (no spurious
bigrams across ASCII boundaries).
Fixes #28000
2026-03-28 09:19:52 +05:30
..
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-27 23:47:04 +00:00
2026-03-28 03:31:42 +00:00
2026-03-28 09:19:27 +05:30
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-27 16:07:54 -05:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 03:27:12 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:04:31 +00:00
2026-03-28 03:31:42 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:04:31 +00:00
2026-03-28 02:08:29 +00:00
2026-03-27 21:25:21 -05:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 03:12:07 +00:00
2026-03-27 22:24:27 -05:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:04:31 +00:00
2026-03-28 02:53:03 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:53:54 +00:00
2026-03-27 21:45:10 -05:00
2026-03-28 02:00:22 +00:00
2026-03-28 09:19:52 +05:30
2026-03-28 02:12:05 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:08:29 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-27 21:46:42 -05:00
2026-03-27 21:37:50 -05:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:04:31 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:08:29 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-27 16:57:16 -05:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-27 16:40:27 +00:00
2026-03-28 03:06:06 +00:00
2026-03-28 03:38:11 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 03:31:42 +00:00
2026-03-28 02:00:22 +00:00
2026-03-27 16:05:22 -05:00
2026-03-28 02:00:22 +00:00
2026-03-28 03:42:37 +00:00
2026-03-26 02:04:54 -07:00
2026-03-28 02:25:28 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:04:31 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-27 22:25:32 -05:00
2026-03-28 02:00:22 +00:00
2026-03-28 03:48:57 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:00:22 +00:00
2026-03-28 02:12:05 +00:00
2026-03-27 21:33:14 -05:00
2026-03-12 23:03:19 +00:00