diff --git a/docs/gateway/local-model-services.md b/docs/gateway/local-model-services.md index faa59f71147e..2f86326b11c7 100644 --- a/docs/gateway/local-model-services.md +++ b/docs/gateway/local-model-services.md @@ -1,27 +1,29 @@ --- -summary: "Start local model servers on demand before OpenClaw model requests" +summary: "Start local model servers on demand before OpenClaw model and embedding requests" read_when: - - You want OpenClaw to start a local model server only when its model is selected + - You want OpenClaw to start a local model server only when its model or embedding provider is selected - You run ds4, inferrs, vLLM, llama.cpp, MLX, or another OpenAI-compatible local server - You need to control cold start, readiness, and idle shutdown for local providers title: "Local model services" --- -`models.providers..localService` starts a provider-owned local model server on demand. When a request selects a model from that provider, OpenClaw probes the health endpoint, starts the process if it is down, waits for readiness, then sends the request. Use it to avoid keeping expensive local servers running all day. +`models.providers..localService` starts a provider-owned local model server on demand. When a model or embedding request selects that provider, OpenClaw probes the health endpoint, starts the process if it is down, waits for readiness, then sends the request. Use it to avoid keeping expensive local servers running all day. ## How it works -1. A model request resolves to a configured provider. +1. A model or embedding request resolves to a configured provider. 2. If that provider has `localService`, OpenClaw probes `healthUrl`. 3. On a successful probe, OpenClaw uses the already-running server. 4. On a failed probe, OpenClaw spawns `command` with `args`. 5. OpenClaw polls the health endpoint until `readyTimeoutMs` expires. -6. The model request goes through the normal provider transport. +6. The request goes through the normal model or embedding transport. 7. If OpenClaw started the process and `idleStopMs` is set, it stops the process after the last in-flight request has been idle that long. OpenClaw does not install launchd, systemd, Docker, or any daemon for this. The server is a plain child process of whichever OpenClaw process first needed it. -Startup is serialized per provider command/argument/env set, so concurrent requests for the same service do not spawn duplicate servers. If another OpenClaw process already has a healthy server at the same `healthUrl`, this process reuses it without adopting it (each process only manages the child it personally started). Active streaming responses hold a lease, so idle shutdown waits until response handling completes. +Startup is serialized per configured provider and command/argument/env set, so concurrent chat and embedding requests for the same service do not spawn duplicate servers. Each request holds its own lease until response handling completes, so idle shutdown waits for every in-flight model and embedding request. Configured provider aliases remain distinct: two aliases can point at different GPU hosts without collapsing onto the same Ollama, LM Studio, or OpenAI-compatible adapter id. + +If another OpenClaw process already has a healthy server at the same `healthUrl`, this process reuses it without adopting it (each process only manages the child it personally started). Startup and exit logs include bounded, redacted child-output tails plus timing and exit details; configured environment values are never emitted. ## Config shape diff --git a/docs/plugins/sdk-runtime.md b/docs/plugins/sdk-runtime.md index de3c439638e5..ff3235d5ffcb 100644 --- a/docs/plugins/sdk-runtime.md +++ b/docs/plugins/sdk-runtime.md @@ -220,6 +220,36 @@ two-party event loops that do not go through the shared inbound reply runner. }); ``` + Provider orchestration can also acquire the configured local-service + lifecycle before issuing an HTTP request: + + ```typescript + const lease = await api.runtime.llm.acquireLocalService( + { + providerId, + baseUrl, + headers, + service: providerConfig.localService, + }, + signal, + ); + try { + // Send and fully consume the provider request. + } finally { + await lease?.release(); + } + ``` + + `acquireLocalService(...)` is a stable, generic provider-service SDK + contract. It exposes only host-owned acquisition and release; process + spawning, readiness, diagnostics, and idle-stop policy remain internal to + the host. + + Pass the exact configured provider id and resolved request base URL. Do not + replace aliases with an adapter id: separate aliases can point at separate + local GPU hosts. The host owns startup serialization, readiness probes, + request leases, abort handling, and idle shutdown. + The helper uses the same simple-completion preparation path as OpenClaw's built-in runtime and the host-owned runtime config snapshot. Context engines receive a session-bound `llm.complete` capability, so model calls use the