Files
openclaw/packages/normalization-core/src/utf16-slice.ts
Peter Steinberger 062f88e3e3 refactor: extract reusable AI runtime package (#99059)
* refactor: extract reusable AI runtime package

* refactor: complete AI provider relocation

* refactor: keep llm core internal

* refactor(ai): make @openclaw/ai self-contained with host policy ports

Move pure transport helpers (tool projections, strict-schema normalization,
prompt-cache boundary, stream guards, anthropic/openai compat, request
activity) from src into packages/ai; move utf16-slice into
normalization-core. Inject host policy (guarded fetch, redaction,
strict-tool defaults, diagnostics logging) through AiTransportHost with
inert library defaults installed by src/llm/stream.ts. Narrow the public
barrel to instance-scoped createApiRegistry/createLlmRuntime; the
process-default runtime moves behind internal/ and
registerBuiltInApiProviders takes an explicit registry. Delete the
src/llm/api-registry re-export facade.

* fix(ai): teach node, jiti, and vite resolvers the @openclaw/ai and utf16-slice subpaths

The workspace alias tables in root-alias.cjs, plugin-sdk-native-resolver,
sdk-alias, the shared vitest config, and the Control UI vite config only
knew @openclaw/llm-core; Node-side plugin loading resolved @openclaw/ai
through the pnpm symlink to the unbuilt dist (checks-node-compact CI
failures), and the Control UI build broke on the new
normalization-core/utf16-slice subpath.

* chore(ui): drop leftover service-worker debug logging

* build(release): ship @openclaw/ai with its own shrinkwrap and honest dependency set

packages/ai declares only its six real runtime deps (kysely, chalk, json5,
tslog, zod, fs-safe, and proxyline were never imported); orphaned root deps
removed. generate-npm-shrinkwrap now treats publishable packages/* like
publishable plugins so the AI tarball pins its transitive tree even though
workspace deps are omitted from the root shrinkwrap. knip learns the
package entry points; the tsdown dts neverBundle option moves to its
documented deps.dts home; the README documents the no-semver internal/*
contract and host ports.

* docs(ai): add minimal external-consumer example app

examples/ai-chat consumes only the public @openclaw/ai surface (built dist
via the workspace link): isolated runtime, built-in provider registration,
one streamed completion. Supports Anthropic/OpenAI via env keys and a
keyless local Ollama target; live-verified against Ollama.

* docs(ai): document the @openclaw/ai package and workspace shrinkwrap boundary

* chore(check): include examples/ in duplicate-scan targets

* fix: emit normalization package subpaths

* fix: complete AI package boundary artifacts

* fix: align AI package boundary contracts

* fix(ci): stabilize package release contracts

* test: align documentation contract checks

* test: keep cron docs guard aligned

* test: align restored docs contract guards

* test: follow upstream docs contracts

* docs: drop superseded talk wording
2026-07-05 01:56:40 -04:00

53 lines
1.6 KiB
TypeScript

// Surrogate-safe UTF-16 string slicing helpers.
//
// Kept dependency-free (no node: imports) so browser/UI bundles can import them
// without dragging in filesystem/runtime code. See utils.ts, which re-exports
// these for the broad runtime surface.
function isHighSurrogate(codeUnit: number): boolean {
return codeUnit >= 0xd800 && codeUnit <= 0xdbff;
}
function isLowSurrogate(codeUnit: number): boolean {
return codeUnit >= 0xdc00 && codeUnit <= 0xdfff;
}
/** Slices a UTF-16 string without returning dangling surrogate halves at either edge. */
export function sliceUtf16Safe(input: string, start: number, end?: number): string {
const len = input.length;
let from = start < 0 ? Math.max(len + start, 0) : Math.min(start, len);
let to = end === undefined ? len : end < 0 ? Math.max(len + end, 0) : Math.min(end, len);
if (to < from) {
const tmp = from;
from = to;
to = tmp;
}
if (from > 0 && from < len) {
const codeUnit = input.charCodeAt(from);
if (isLowSurrogate(codeUnit) && isHighSurrogate(input.charCodeAt(from - 1))) {
from += 1;
}
}
if (to > 0 && to < len) {
const codeUnit = input.charCodeAt(to - 1);
if (isHighSurrogate(codeUnit) && isLowSurrogate(input.charCodeAt(to))) {
to -= 1;
}
}
return input.slice(from, to);
}
/** Truncates a UTF-16 string without cutting a surrogate pair in half. */
export function truncateUtf16Safe(input: string, maxLen: number): string {
const limit = Math.max(0, Math.floor(maxLen));
if (input.length <= limit) {
return input;
}
return sliceUtf16Safe(input, 0, limit);
}