From d75601478a65820bbd89c96a678ccf391cefa034 Mon Sep 17 00:00:00 2001 From: SunnyShu Date: Sat, 1 Aug 2026 14:11:22 +0800 Subject: [PATCH] docs(gateway): document llamacpp toolSchemaProfile for custom llama.cpp endpoints (#117120) * docs(gateway): document llamacpp toolSchemaProfile for custom llama.cpp endpoints Custom openai-completions providers pointed at a llama.cpp / llama-server endpoint bypass the GBNF-safe schema normalization that built-in llama-cpp, ollama, and lmstudio providers apply automatically. The toolSchemaProfile: "llamacpp" switch activates it, but the supported value was never named in the docs, so users hit llama.cpp 400 'failed to parse grammar' on tools with large minLength/maxLength (e.g. cron trigger.script, maxLength 65536). - Expand the toolSchemaProfile row to name the values recognized on a model's compat (llamacpp, gemini) and when llamacpp must be set explicitly; cross-reference the unsupportedToolSchemaKeywords escape hatch. - Add a 'Local models (llama.cpp / llama-server)' provider example showing the required compat.toolSchemaProfile: "llamacpp" on a custom endpoint. Related to #117070 Co-Authored-By: iCodeMate * docs(gateway): describe llamacpp profile as stripping, not clamping Address review feedback on #117120: the fallback sentence claimed the llamacpp profile "also clamps" constraints, but cleanSchemaForLlamacppGbnf only removes `pattern` outright and drops `maxLength` at or above the 2000-repetition threshold (LLAMACPP_GBNF_MAX_REPETITION_THRESHOLD). It never clamps a value, and small bounds like `minLength: 1` are preserved (confirmed by the projection test at src/plugin-sdk/provider-tools.test.ts). Reword the escape-hatch sentence to describe the implemented stripping behavior and contrast it with the unconditional keyword-list drops, so operators do not expect a preserved bounded constraint. No code or behavior changes; documentation only. * docs(gateway): scope llamacpp profile description to its actual transformations Address follow-up review on #117120: the toolSchemaProfile table row described `llamacpp` as "rewriting tool schemas into the JSON Schema subset llama.cpp can compile to GBNF," which overstates what cleanSchemaForLlamacppGbnf does. The cleaner only removes `pattern` outright and drops `maxLength` at or above the 2000-repetition threshold (LLAMACPP_GBNF_MAX_REPETITION_THRESHOLD); it leaves every other keyword and `minLength` untouched, and those are not guaranteed GBNF-safe. Reword the row to state the exact transformation contract and explicitly note it is a targeted compatibility cleaner, not a blanket GBNF-safety guarantee. No code or behavior changes; documentation only. Co-Authored-By: iCodeMate * docs(gateway): tighten llama.cpp profile contract * docs(gateway): order local model examples --------- Co-authored-by: iCodeMate Co-authored-by: Vincent Koc --- docs/gateway/config-tools.md | 42 ++++++++++++++++++++++++++++++++++-- 1 file changed, 40 insertions(+), 2 deletions(-) diff --git a/docs/gateway/config-tools.md b/docs/gateway/config-tools.md index b80cf90af10c..7567fcd5b44d 100644 --- a/docs/gateway/config-tools.md +++ b/docs/gateway/config-tools.md @@ -579,8 +579,8 @@ Configuring a custom/local provider `baseUrl` is also the narrow network trust d | `requiresAssistantAfterToolResult` | Requires an assistant message after tool results. | | `requiresThinkingAsText` | Replays reasoning as text rather than structured content. | | `requiresReasoningContentOnAssistantMessages` | Preserves DeepSeek-style `reasoning_content` during replay. | - | `toolSchemaProfile` | Selects a provider-defined tool-schema normalization profile. | - | `unsupportedToolSchemaKeywords` | Removes named JSON Schema keywords rejected by the endpoint. | + | `toolSchemaProfile` | Selects a tool-schema normalization profile. Custom model entries recognize `llamacpp` and `gemini`. The `llamacpp` profile removes `pattern` and `maxLength` values at or above 2000; built-in `llama-cpp`, `ollama`, and `lmstudio` providers apply the same cleaner automatically. Custom `llama-server` models must select it explicitly. See the llama.cpp example below. | + | `unsupportedToolSchemaKeywords` | Removes named JSON Schema keywords rejected by the endpoint before tool schemas are sent. Use this for endpoint-specific gaps beyond a profile's targeted transformations. | | `toolCallArgumentsEncoding` | Selects the endpoint's tool-call argument encoding. | | `requiresOpenAiAnthropicToolPayload` | Converts OpenAI-shaped tool calls to Anthropic-family payloads. | @@ -655,6 +655,44 @@ Interactive custom-provider onboarding infers image input for known vision-model Anthropic-compatible, built-in provider. Shortcut: `openclaw onboard --auth-choice kimi-code-api-key`. + + + Point a **custom** `openai-completions` provider at a remote `llama-server` (or another OpenAI-compatible llama.cpp endpoint). The built-in `llama-cpp`, `ollama`, and `lmstudio` providers apply the llama.cpp schema cleaner automatically; a custom endpoint does not. Set `compat.toolSchemaProfile: "llamacpp"` on each model whose llama-server chat template compiles tool arguments into GBNF. The profile removes `pattern` and `maxLength` values at or above 2000, covering the `cron` tool's `trigger.script` limit of 65536. It is a targeted mitigation, not complete compatibility for every JSON Schema constraint or `minLength`. + + ```json5 + { + agents: { + defaults: { + model: { primary: "my-llamacpp/qwen35" }, + }, + }, + models: { + mode: "merge", + providers: { + "my-llamacpp": { + baseUrl: "http://127.0.0.1:8080/v1", + apiKey: "llamacpp-no-key", + api: "openai-completions", + models: [ + { + id: "qwen35", + name: "Qwen3.5 (llama-server)", + contextWindow: 8192, + maxTokens: 2048, + compat: { + supportsTools: true, + toolSchemaProfile: "llamacpp", + }, + }, + ], + }, + }, + }, + } + ``` + + On older builds without `toolSchemaProfile`, the broader fallback is `compat.unsupportedToolSchemaKeywords: ["pattern", "patternProperties", "format", "propertyNames", "uniqueItems", "contains", "minContains", "maxContains", "minLength", "maxLength"]`. Unlike the profile, this removes every listed keyword unconditionally. + See [Local Models](/gateway/local-models). TL;DR: run a large local model via LM Studio Responses API on serious hardware; keep hosted models merged for fallback.