vultr/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-08-02 04:21:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(gateway): document llamacpp toolSchemaProfile for custom llama.cpp endpoints (#117120)

Browse Source 
* docs(gateway): document llamacpp toolSchemaProfile for custom llama.cpp endpoints

Custom openai-completions providers pointed at a llama.cpp / llama-server
endpoint bypass the GBNF-safe schema normalization that built-in llama-cpp,
ollama, and lmstudio providers apply automatically. The toolSchemaProfile:
"llamacpp" switch activates it, but the supported value was never named in
the docs, so users hit llama.cpp 400 'failed to parse grammar' on tools with
large minLength/maxLength (e.g. cron trigger.script, maxLength 65536).

- Expand the toolSchemaProfile row to name the values recognized on a model's
  compat (llamacpp, gemini) and when llamacpp must be set explicitly;
  cross-reference the unsupportedToolSchemaKeywords escape hatch.
- Add a 'Local models (llama.cpp / llama-server)' provider example showing
  the required compat.toolSchemaProfile: "llamacpp" on a custom endpoint.

Related to #117070

Co-Authored-By: iCodeMate <noreply@anthropic.com>

* docs(gateway): describe llamacpp profile as stripping, not clamping

Address review feedback on #117120: the fallback sentence claimed the
llamacpp profile "also clamps" constraints, but cleanSchemaForLlamacppGbnf
only removes `pattern` outright and drops `maxLength` at or above the
2000-repetition threshold (LLAMACPP_GBNF_MAX_REPETITION_THRESHOLD). It
never clamps a value, and small bounds like `minLength: 1` are preserved
(confirmed by the projection test at src/plugin-sdk/provider-tools.test.ts).

Reword the escape-hatch sentence to describe the implemented stripping
behavior and contrast it with the unconditional keyword-list drops, so
operators do not expect a preserved bounded constraint.

No code or behavior changes; documentation only.

* docs(gateway): scope llamacpp profile description to its actual transformations

Address follow-up review on #117120: the toolSchemaProfile table row
described `llamacpp` as "rewriting tool schemas into the JSON Schema
subset llama.cpp can compile to GBNF," which overstates what
cleanSchemaForLlamacppGbnf does. The cleaner only removes `pattern`
outright and drops `maxLength` at or above the 2000-repetition threshold
(LLAMACPP_GBNF_MAX_REPETITION_THRESHOLD); it leaves every other keyword
and `minLength` untouched, and those are not guaranteed GBNF-safe.

Reword the row to state the exact transformation contract and explicitly
note it is a targeted compatibility cleaner, not a blanket GBNF-safety
guarantee. No code or behavior changes; documentation only.

Co-Authored-By: iCodeMate <noreply@anthropic.com>

* docs(gateway): tighten llama.cpp profile contract

* docs(gateway): order local model examples

---------

Co-authored-by: iCodeMate <noreply@anthropic.com>
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
This commit is contained in:
SunnyShu
2026-08-01 14:11:22 +08:00
committed by GitHub
parent 2364078063
commit d75601478a
1 changed files with 40 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
docs/gateway/config-tools.md
View File

@@ -579,8 +579,8 @@ Configuring a custom/local provider `baseUrl` is also the narrow network trust d
| `requiresAssistantAfterToolResult` | Requires an assistant message after tool results. |
| `requiresThinkingAsText` | Replays reasoning as text rather than structured content. |
| `requiresReasoningContentOnAssistantMessages` | Preserves DeepSeek-style `reasoning_content` during replay. |
| `toolSchemaProfile` | Selects a provider-defined tool-schema normalization profile. |
| `unsupportedToolSchemaKeywords` | Removes named JSON Schema keywords rejected by the endpoint. |
| `toolSchemaProfile` | Selects a tool-schema normalization profile. Custom model entries recognize `llamacpp` and `gemini`. The `llamacpp` profile removes `pattern` and `maxLength` values at or above 2000; built-in `llama-cpp`, `ollama`, and `lmstudio` providers apply the same cleaner automatically. Custom `llama-server` models must select it explicitly. See the llama.cpp example below. |
| `unsupportedToolSchemaKeywords` | Removes named JSON Schema keywords rejected by the endpoint before tool schemas are sent. Use this for endpoint-specific gaps beyond a profile's targeted transformations. |
| `toolCallArgumentsEncoding` | Selects the endpoint's tool-call argument encoding. |
| `requiresOpenAiAnthropicToolPayload` | Converts OpenAI-shaped tool calls to Anthropic-family payloads. |
@@ -655,6 +655,44 @@ Interactive custom-provider onboarding infers image input for known vision-model
Anthropic-compatible, built-in provider. Shortcut: `openclaw onboard --auth-choice kimi-code-api-key`.
</Accordion>
<Accordion title="Local models (llama.cpp / llama-server)">
Point a **custom** `openai-completions` provider at a remote `llama-server` (or another OpenAI-compatible llama.cpp endpoint). The built-in `llama-cpp`, `ollama`, and `lmstudio` providers apply the llama.cpp schema cleaner automatically; a custom endpoint does not. Set `compat.toolSchemaProfile: "llamacpp"` on each model whose llama-server chat template compiles tool arguments into GBNF. The profile removes `pattern` and `maxLength` values at or above 2000, covering the `cron` tool's `trigger.script` limit of 65536. It is a targeted mitigation, not complete compatibility for every JSON Schema constraint or `minLength`.
```json5
{
agents: {
defaults: {
model: { primary: "my-llamacpp/qwen35" },
},
},
models: {
mode: "merge",
providers: {
"my-llamacpp": {
baseUrl: "http://127.0.0.1:8080/v1",
apiKey: "llamacpp-no-key",
api: "openai-completions",
models: [
{
id: "qwen35",
name: "Qwen3.5 (llama-server)",
contextWindow: 8192,
maxTokens: 2048,
compat: {
supportsTools: true,
toolSchemaProfile: "llamacpp",
},
},
],
},
},
},
}
```
On older builds without `toolSchemaProfile`, the broader fallback is `compat.unsupportedToolSchemaKeywords: ["pattern", "patternProperties", "format", "propertyNames", "uniqueItems", "contains", "minContains", "maxContains", "minLength", "maxLength"]`. Unlike the profile, this removes every listed keyword unconditionally.
</Accordion>
<Accordion title="Local models (LM Studio)">
See [Local Models](/gateway/local-models). TL;DR: run a large local model via LM Studio Responses API on serious hardware; keep hosted models merged for fallback.