The tokenize() function only matched [a-z0-9_]+ patterns, returning an
empty set for CJK-only text. This made Jaccard similarity always 0 (or
always 1 for two empty sets) for CJK content, effectively disabling MMR
diversity detection.
Add support for:
- CJK Unified Ideographs (U+4E00–U+9FFF, U+3400–U+4DBF)
- Hiragana (U+3040–U+309F) and Katakana (U+30A0–U+30FF)
- Hangul Syllables (U+AC00–U+D7AF) and Jamo (U+1100–U+11FF)
Characters are extracted as unigrams, and bigrams are generated only
from characters that are adjacent in the original text (no spurious
bigrams across ASCII boundaries).
Fixes#28000
* fix(msteams): reset stream state after preparePayload suppresses delivery
When an agent uses tools mid-response (text → tool calls → more text),
the stream controller's preparePayload would suppress fallback delivery
for ALL text segments because streamReceivedTokens stayed true. This
caused the second text segment to be silently lost or duplicated.
Fix: after preparePayload suppresses delivery for a streamed segment,
finalize the stream and reset streamReceivedTokens so subsequent
segments use fallback delivery.
Fixesopenclaw/openclaw#56040
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(msteams): guard preparePayload against finalized stream re-suppression
When onPartialReply fires after the stream is finalized (post-tool
partial tokens), streamReceivedTokens gets set back to true but the
stream can't deliver. Add stream.isFinalized check so a finalized
stream never suppresses fallback delivery.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(msteams): await pending finalize in controller to prevent race
Store the fire-and-forget finalize promise from preparePayload and
await it in the controller's finalize() method. This ensures
markDispatchIdle waits for the in-flight stream finalization to
complete before context cleanup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(msteams): add edge case tests for multi-round and media payloads
Add tests for 3+ tool call rounds (text → tool → text → tool → text)
and media+text payloads after stream finalization, covering the full
contract of preparePayload across all input types and cycle counts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
