mirror of
https://github.com/openclaw/openclaw.git
synced 2026-08-02 11:31:34 +00:00
feat(cron): convert heartbeat tasks: entries into independent cron jobs (#113165)
Each legacy heartbeat tasks: entry becomes an editable system-created cron job (declaration key heartbeat-task:<agentId>:<hash>, per-occurrence identity for duplicate names) that fires a guarded heartbeat wake carrying the task prompt plus current monitor scratch context. Task cadence is now independent of the base heartbeat interval; active-hours, busy-retry, min-spacing, and flood guards are preserved, deferred payload-carrying wakes are retained and retried after the spacing floor, and colliding task/event wakes cannot starve each other. openclaw doctor --fix migrates existing blocks: async parse/plan, then one synchronous SQLite transaction that rereads the pinned scratch revision, upserts job rows, and strips the tasks: block atomically — concurrent doctors serialize on the database and a losing run aborts untouched. Orphan fields, invalid intervals, and incomplete entries block migration of their block with a clear finding instead of silent text loss. The runtime tasks: parser is deleted; leftover text is ordinary scratch prose. Nine Codex gpt-5.6-sol xhigh autoreview cycles; final verdict clean.
This commit is contained in:
Peter Steinberger
committed by
GitHub
GitHub
parent
b3b9e691bd
commit
b23c7bd0f8
@@ -103,7 +103,7 @@ aa2a56b4448c8ebdec9d06aac95d809995f533093d42fa32cd75e1d852967245 module/questio
|
||||
bd2355e94248d21c252148085e5afa5db9a2f3f48f665a8b3e0fcf77326d9b6c module/reply-dispatch-runtime
|
||||
ac2b199e95c5c8b1e2a65e62bd41d1b6322e531bca294ef4979a297a12640bce module/reply-history
|
||||
f394fe4d5a7ed9e4d574063ae44e8d6af85c9a0e7d8b329f750ca16b0664325f module/reply-payload
|
||||
554bbc681b17bf102a5e240e3f563c7d3c3f1dbab31f8fa3c9285f106c3e8111 module/reply-runtime
|
||||
6b8a50764563f096175162c49d1acf5593eee3363edd45976c1aa641a3d532cf module/reply-runtime
|
||||
d78db621b8f4f0cc679cad2d5d21b6c95b5418c58611dd347cdf09049ed124f6 module/routing
|
||||
ff6cca86f54f94f238205f5b122af36666314e0a380f3ec7f0ccb9ed9208df31 module/run-command
|
||||
53b0295cec105696a1664c5c7f5576a7b55d197eb95dcd9185486f010bd53750 module/runtime
|
||||
|
||||
@@ -77,6 +77,14 @@ Timestamps without a timezone are treated as UTC. Add `--tz America/New_York` to
|
||||
|
||||
Recurring top-of-hour expressions (minute `0` with a wildcard hour field) are automatically staggered by up to 5 minutes to reduce load spikes. Use `--exact` to force precise timing, or `--stagger 30s` for an explicit window (cron schedules only).
|
||||
|
||||
### Heartbeat task migration
|
||||
|
||||
Older heartbeat scratch supported a structured `tasks:` block. Run `openclaw doctor --fix` after upgrading to convert each entry into an ordinary editable main-session cron job. Doctor preserves the interval and previous last-run timing, creates the jobs before removing the block, and safely converges the same declaration keys on rerun.
|
||||
|
||||
These migrated jobs carry public `systemEvent` payloads, so `openclaw cron list`, `get`, `edit`, and `remove` plus the cron tool manage them like other jobs. Their execution uses the guarded heartbeat task wake: active hours, minimum spacing, flood control, and busy retries still apply, while cron owns each task's independent cadence. Jobs due in the same coalescing window can share one heartbeat turn. A scheduled occurrence outside heartbeat active hours is skipped and retried at the job's next occurrence.
|
||||
|
||||
Heartbeat scratch is now monitor prose only. Runtime heartbeats do not parse `tasks:` text as schedules; create new recurring work with cron.
|
||||
|
||||
### Stream sources
|
||||
|
||||
A stream schedule keeps an operator-authored argv command running under the Gateway and fires the job from its stdout and stderr lines. Stream schedules are event-driven, never time-due, and require `cron.triggers.enabled: true` because the long-lived command has the same unattended trust class as trigger scripts. Disabling or removing the job stops the process; Gateway shutdown waits for process-tree teardown. Fast failures restart with cron's built-in error backoff. Five consecutive runs shorter than 60 seconds leave the job in an error state and use the normal failure-alert path; manually re-enable the job to clear the restart cap.
|
||||
|
||||
@@ -95,14 +95,14 @@ See [Hooks](/automation/hooks).
|
||||
|
||||
### Heartbeat
|
||||
|
||||
Heartbeat is a periodic main-session turn (default every 30 minutes). It batches multiple checks (inbox, calendar, notifications) in one agent turn with full session context. Heartbeat turns do not create task records and do not extend daily/idle session reset freshness. Use `HEARTBEAT.md` for a small checklist, or a `tasks:` block when you want due-only periodic checks inside heartbeat itself. Empty heartbeat files skip as `empty-heartbeat-file`; due-only task mode skips as `no-tasks-due`. Heartbeats defer while cron work is active or queued, and `heartbeat.skipWhenBusy` can also defer an agent while that same agent's session-keyed subagent or nested lanes are busy.
|
||||
Heartbeat is a periodic main-session turn (default every 30 minutes). It batches checklist-style monitoring (inbox, calendar, notifications) in one agent turn with full session context. Heartbeat turns do not create task records and do not extend daily/idle session reset freshness. Heartbeat scratch is small prompt context; schedule recurring work as cron jobs. Empty heartbeat scratch skips as `empty-heartbeat-file`. Heartbeats defer while cron work is active or queued, and `heartbeat.skipWhenBusy` can also defer an agent while that same agent's session-keyed subagent or nested lanes are busy.
|
||||
|
||||
See [Heartbeat](/gateway/heartbeat).
|
||||
|
||||
## How they work together
|
||||
|
||||
- **Cron** handles precise schedules (daily reports, weekly reviews) and one-shot reminders. All cron executions create task records.
|
||||
- **Heartbeat** handles routine monitoring (inbox, calendar, notifications) in one batched turn every 30 minutes.
|
||||
- **Heartbeat** handles one batched monitoring checklist every 30 minutes; cron owns checks that need independent cadences.
|
||||
- **Hooks** react to specific events (session resets, compaction, message flow) with custom scripts. Plugin hooks cover tool calls.
|
||||
- **Standing orders** give the agent persistent context and authority boundaries.
|
||||
- **Task Flow** coordinates multi-step flows above individual tasks.
|
||||
|
||||
@@ -66,6 +66,7 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
|
||||
- H2: Quick start
|
||||
- H2: How cron works
|
||||
- H2: Schedule types
|
||||
- H3: Heartbeat task migration
|
||||
- H3: Stream sources
|
||||
- H3: Dynamic cadence (pacing)
|
||||
- H3: Day-of-month and day-of-week use OR logic
|
||||
@@ -3602,7 +3603,7 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
|
||||
- H3: Per-channel vs per-account examples
|
||||
- H3: Common patterns
|
||||
- H2: Monitor scratch (optional)
|
||||
- H3: tasks: blocks
|
||||
- H3: Schedule recurring checks with cron
|
||||
- H3: Can the agent update its scratch?
|
||||
- H2: Manual wake (on-demand)
|
||||
- H2: Cost awareness
|
||||
|
||||
@@ -28,7 +28,7 @@ Troubleshooting: [Scheduled Tasks](/automation/cron-jobs#troubleshooting)
|
||||
Leave heartbeats enabled (default is `30m`, or `1h` when Anthropic OAuth/token auth is configured, including Claude CLI reuse) or set your own cadence.
|
||||
</Step>
|
||||
<Step title="Add monitor scratch (optional)">
|
||||
Store a tiny checklist or `tasks:` block in the heartbeat monitor's scratch with `openclaw cron scratch <jobId> --set "..."`.
|
||||
Store a tiny checklist in the heartbeat monitor's scratch with `openclaw cron scratch <jobId> --set "..."`.
|
||||
</Step>
|
||||
<Step title="Decide where heartbeat messages should go">
|
||||
`target: "none"` is the default; set `target: "last"` to route to the last contact.
|
||||
@@ -63,7 +63,7 @@ Example config:
|
||||
## Defaults
|
||||
|
||||
- Interval: `30m`. Applying Anthropic provider defaults bumps this to `1h` when the resolved auth mode is OAuth/token (including Claude CLI reuse), but only while `heartbeat.every` is unset. Set `agents.defaults.heartbeat.every` or per-agent `agents.entries.*.heartbeat.every`; use `0m` to disable.
|
||||
- Prompt body (configurable via `agents.defaults.heartbeat.prompt`): `Follow the heartbeat monitor scratch context when provided. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK.`
|
||||
- Prompt body (configurable via `agents.defaults.heartbeat.prompt`): `Follow the heartbeat monitor scratch context when provided. Recurring tasks are cron jobs; create or change their schedules with cron tools or the openclaw cron CLI, not heartbeat scratch. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK.`
|
||||
- Timeout: unset heartbeat turns use `agents.defaults.timeoutSeconds` when set. Otherwise, they use the heartbeat cadence capped at 600 seconds. Set `agents.defaults.heartbeat.timeoutSeconds` or per-agent `agents.entries.*.heartbeat.timeoutSeconds` for longer heartbeat work.
|
||||
- The heartbeat prompt is sent **verbatim** as the user message. The system prompt includes a "Heartbeats" section when heartbeats are enabled for the default agent, and the run is flagged internally.
|
||||
- When heartbeats are disabled with `0m`, the monitor cron job stays but is disabled, and its scratch is retained for when you re-enable the cadence.
|
||||
@@ -107,7 +107,7 @@ Outside heartbeats, stray `HEARTBEAT_OK` at the start/end of a message is stripp
|
||||
target: "last", // default: none | options: last | none | <channel id> (core or plugin, e.g. "imessage")
|
||||
to: "+15551234567", // optional channel-specific override
|
||||
accountId: "ops-bot", // optional multi-account channel id
|
||||
prompt: "Follow the heartbeat monitor scratch context when provided. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK.",
|
||||
prompt: "Follow the heartbeat monitor scratch context when provided. Recurring tasks are cron jobs; create or change their schedules with cron tools or the openclaw cron CLI, not heartbeat scratch. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK.",
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -146,7 +146,7 @@ Example: two agents, only the second agent runs heartbeats.
|
||||
target: "whatsapp",
|
||||
to: "+15551234567",
|
||||
timeoutSeconds: 45,
|
||||
prompt: "Follow the heartbeat monitor scratch context when provided. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK.",
|
||||
prompt: "Follow the heartbeat monitor scratch context when provided. Recurring tasks are cron jobs; create or change their schedules with cron tools or the openclaw cron CLI, not heartbeat scratch. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK.",
|
||||
},
|
||||
},
|
||||
],
|
||||
@@ -382,7 +382,7 @@ Writes are compare-and-swap guarded: pass `--expected-revision <n>` to fail inst
|
||||
The agent can also update its own scratch: during a heartbeat turn, `heartbeat_respond` accepts an optional `scratch` string that fully replaces the monitor's scratch for future heartbeats.
|
||||
|
||||
<Note>
|
||||
**Migrating from HEARTBEAT.md or config-only cadence?** Run `openclaw doctor --fix`. Doctor first creates or updates the system-owned monitor rows from `agents.*.heartbeat`, then imports each agent's workspace `HEARTBEAT.md` into the monitor's scratch, archives the original under the state directory (`backups/heartbeat-migration/`), and removes the file. For one stable upgrade window, an unmigrated legacy file remains a read-only fallback when no scratch revision exists, with a Gateway warning directing you to Doctor; new workspaces and completed migrations use database scratch only.
|
||||
**Migrating from HEARTBEAT.md or config-only cadence?** Run `openclaw doctor --fix`. Doctor first creates or updates the system-owned monitor rows from `agents.*.heartbeat`, then imports each agent's workspace `HEARTBEAT.md` into the monitor's scratch, converts any valid legacy `tasks:` entries into cron jobs, archives the original under the state directory (`backups/heartbeat-migration/`), and removes the file. For one stable upgrade window, an unmigrated legacy file remains a read-only fallback when no scratch revision exists, with a Gateway warning directing you to Doctor; new workspaces and completed migrations use database scratch only.
|
||||
</Note>
|
||||
|
||||
If scratch exists but is effectively empty (only blank lines, Markdown/HTML comments, Markdown headings like `# Heading`, fence markers, or empty checklist stubs), OpenClaw skips the heartbeat run to save API calls. That skip is reported as `reason=empty-heartbeat-file`. If no scratch exists, the heartbeat still runs and the model decides what to do.
|
||||
@@ -399,45 +399,17 @@ Example scratch:
|
||||
- If a task is blocked, write down _what is missing_ and ask Peter next time.
|
||||
```
|
||||
|
||||
### `tasks:` blocks
|
||||
### Schedule recurring checks with cron
|
||||
|
||||
Scratch also supports a small structured `tasks:` block for interval-based checks inside heartbeat itself.
|
||||
Heartbeat scratch is prompt context, not a scheduler. Create each recurring check as a [cron job](/automation/cron-jobs) so it has its own cadence, enable/disable state, and run history. Cron jobs can still target the main session when the check should use the normal conversation context.
|
||||
|
||||
Example:
|
||||
Older scratch may contain a structured `tasks:` block. Run `openclaw doctor --fix` once after upgrading: Doctor converts every valid entry into an independently scheduled cron job, preserves its interval and previous last-run timing, and removes the retired block while keeping surrounding scratch prose. Runtime heartbeat turns do not parse `tasks:` text as schedules.
|
||||
|
||||
```md
|
||||
tasks:
|
||||
|
||||
- name: inbox-triage
|
||||
interval: 30m
|
||||
prompt: "Check for urgent unread emails and flag anything time sensitive."
|
||||
- name: calendar-scan
|
||||
interval: 2h
|
||||
prompt: "Check for upcoming meetings that need prep or follow-up."
|
||||
|
||||
# Additional instructions
|
||||
|
||||
- Keep alerts short.
|
||||
- If nothing needs attention after all due tasks, reply HEARTBEAT_OK.
|
||||
```
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Behavior">
|
||||
- OpenClaw parses the `tasks:` block and checks each task against its own `interval`.
|
||||
- Only **due** tasks are included in the heartbeat prompt for that tick.
|
||||
- If no tasks are due, the heartbeat is skipped entirely (`reason=no-tasks-due`) to avoid a wasted model call.
|
||||
- Non-task scratch content is preserved and appended as additional context after the due-task list.
|
||||
- Task last-run timestamps are stored in session state (`heartbeatTaskState`), so intervals survive normal restarts.
|
||||
- Task timestamps are only advanced after a heartbeat run completes its normal reply path. Skipped `empty-heartbeat-file` / `no-tasks-due` runs do not mark tasks as completed.
|
||||
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
Task mode is useful when you want one scratch document to hold several periodic checks without paying for all of them every tick.
|
||||
Doctor-created heartbeat task jobs keep heartbeat active-hours, cooldown, flood, and busy guards. Jobs due together can coalesce into one heartbeat turn. An occurrence outside active hours is skipped and tried again at its next cron occurrence.
|
||||
|
||||
### Can the agent update its scratch?
|
||||
|
||||
Yes. During a heartbeat turn, the agent can pass a `scratch` value to `heartbeat_respond` to fully replace the monitor scratch for future heartbeats. You can also ask it in a normal chat to run `openclaw cron scratch <jobId> --set ...`, or edit the scratch yourself with the same command.
|
||||
Yes. During a heartbeat turn, the agent can pass a `scratch` value to `heartbeat_respond` to fully replace the monitor prose for future heartbeats. You can also ask it in a normal chat to run `openclaw cron scratch <jobId> --set ...`, or edit the scratch yourself with the same command. Manage recurring schedules with cron instead of writing scheduler syntax into scratch.
|
||||
|
||||
<Warning>
|
||||
Don't put secrets (API keys, phone numbers, private tokens) into monitor scratch - it becomes part of the prompt context.
|
||||
|
||||
@@ -756,15 +756,14 @@ Look for:
|
||||
|
||||
- Cron enabled and next wake present.
|
||||
- Job run history status (`ok`, `skipped`, `error`).
|
||||
- Heartbeat skip reasons (`quiet-hours`, `requests-in-flight`, `cron-in-progress`, `lanes-busy`, `alerts-disabled`, `empty-heartbeat-file`, `no-tasks-due`).
|
||||
- Heartbeat skip reasons (`quiet-hours`, `requests-in-flight`, `cron-in-progress`, `lanes-busy`, `alerts-disabled`, `empty-heartbeat-file`).
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Common signatures">
|
||||
- `cron: scheduler disabled; jobs will not run automatically` → cron disabled.
|
||||
- `cron: timer tick failed` → scheduler tick failed; check file/log/runtime errors.
|
||||
- `heartbeat skipped` with `reason=quiet-hours` → outside active hours window.
|
||||
- `heartbeat skipped` with `reason=empty-heartbeat-file` → `HEARTBEAT.md` exists but only contains blank, comment, header, fence, or empty-checklist scaffolding, so OpenClaw skips the model call.
|
||||
- `heartbeat skipped` with `reason=no-tasks-due` → `HEARTBEAT.md` contains a `tasks:` block, but none of the tasks are due on this tick.
|
||||
- `heartbeat skipped` with `reason=empty-heartbeat-file` → heartbeat monitor scratch only contains blank, comment, header, fence, or empty-checklist scaffolding, so OpenClaw skips the model call.
|
||||
- `heartbeat: unknown accountId` → invalid account id for heartbeat delivery target.
|
||||
- `heartbeat skipped` with `reason=dm-blocked` → heartbeat target resolved to a DM-style destination while `agents.defaults.heartbeat.directPolicy` (or per-agent override) is set to `block`.
|
||||
|
||||
|
||||
@@ -57,12 +57,10 @@ and troubleshooting see the main [FAQ](/help/faq).
|
||||
| Skip reason | Meaning |
|
||||
| --- | --- |
|
||||
| `quiet-hours` | Outside the configured active-hours window |
|
||||
| `empty-heartbeat-file` | `HEARTBEAT.md` exists but only has blank, comment, header, fence, or empty-checklist scaffolding |
|
||||
| `no-tasks-due` | Task mode is active but no task interval is due yet |
|
||||
| `empty-heartbeat-file` | Heartbeat monitor scratch exists but only has blank, comment, header, fence, or empty-checklist scaffolding |
|
||||
| `alerts-disabled` | All heartbeat visibility is off (`showOk`, `showAlerts`, and `useIndicator` all disabled) |
|
||||
|
||||
In task mode, due timestamps advance only after a real heartbeat run completes.
|
||||
Skipped runs do not mark tasks as completed.
|
||||
Older heartbeat `tasks:` blocks migrate to independently scheduled cron jobs with `openclaw doctor --fix`.
|
||||
|
||||
Docs: [Heartbeat](/gateway/heartbeat), [Automation](/automation).
|
||||
|
||||
|
||||
@@ -330,8 +330,7 @@ flowchart TD
|
||||
|
||||
- `cron: scheduler disabled; jobs will not run automatically` → cron is disabled.
|
||||
- `heartbeat skipped` reason `quiet-hours` → outside configured active hours.
|
||||
- `heartbeat skipped` reason `empty-heartbeat-file` → `HEARTBEAT.md` exists but contains only blank, comment, header, fence, or empty-checklist scaffolding.
|
||||
- `heartbeat skipped` reason `no-tasks-due` → task mode is active but no task interval is due yet.
|
||||
- `heartbeat skipped` reason `empty-heartbeat-file` → heartbeat monitor scratch contains only blank, comment, header, fence, or empty-checklist scaffolding.
|
||||
- `heartbeat skipped` reason `alerts-disabled` → `showOk`, `showAlerts`, and `useIndicator` are all off.
|
||||
- `requests-in-flight` → main lane busy; heartbeat wake deferred.
|
||||
- `unknown accountId` → heartbeat delivery target account does not exist.
|
||||
|
||||
@@ -16,12 +16,12 @@ Shipped default content:
|
||||
|
||||
# Keep this file empty (or with only comments) to skip heartbeat API calls.
|
||||
|
||||
# Add tasks below when you want the agent to check something periodically.
|
||||
# Add a short checklist below when the heartbeat should inspect shared context.
|
||||
```
|
||||
|
||||
Add short tasks below the comment lines only when you want periodic checks. Keep it small: heartbeat runs read this file every tick (default every 30 minutes), so bloated instructions burn tokens on every wake.
|
||||
Add a short checklist below the comment lines only when one heartbeat turn should inspect the items together. Keep it small: heartbeat runs read this file every tick (default every 30 minutes), so bloated instructions burn tokens on every wake.
|
||||
|
||||
For due-only checks instead of a plain checklist, use a structured `tasks:` block with per-task `interval` and `prompt` fields; see [HEARTBEAT.md](/gateway/heartbeat#heartbeatmd-optional) for the format and behavior.
|
||||
For independently scheduled or due-only checks, create [cron jobs](/automation/cron-jobs). Heartbeat scratch no longer supports scheduler syntax. Run `openclaw doctor --fix` to convert older `tasks:` blocks.
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
@@ -168,7 +168,7 @@ Example:
|
||||
## Heartbeats (proactive mode)
|
||||
|
||||
By default, OpenClaw runs a heartbeat every 30 minutes with the prompt:
|
||||
`Follow the heartbeat monitor scratch context when provided. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK.`
|
||||
`Follow the heartbeat monitor scratch context when provided. Recurring tasks are cron jobs; create or change their schedules with cron tools or the openclaw cron CLI, not heartbeat scratch. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK.`
|
||||
Set `agents.defaults.heartbeat.every: "0m"` to disable. Heartbeat checklists live in the monitor's cron scratch (see [Heartbeat](/gateway/heartbeat)); `openclaw doctor --fix` migrates a legacy workspace `HEARTBEAT.md` into it.
|
||||
|
||||
- If the monitor scratch exists but is effectively empty (only blank lines, Markdown/HTML comments, Markdown headings like `# Heading`, fence markers, or empty checklist stubs), OpenClaw skips the heartbeat run to save API calls.
|
||||
|
||||
@@ -122,7 +122,7 @@ Malformed local-model reasoning tags are handled conservatively. Closed `<think>
|
||||
|
||||
## Heartbeats
|
||||
|
||||
- Heartbeat probe body is the configured heartbeat prompt (default: `Follow the heartbeat monitor scratch context when provided. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK.`). Inline directives in a heartbeat message apply as usual (but avoid changing session defaults from heartbeats).
|
||||
- Heartbeat probe body is the configured heartbeat prompt (default: `Follow the heartbeat monitor scratch context when provided. Recurring tasks are cron jobs; create or change their schedules with cron tools or the openclaw cron CLI, not heartbeat scratch. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK.`). Inline directives in a heartbeat message apply as usual (but avoid changing session defaults from heartbeats).
|
||||
- Heartbeat delivery defaults to the final payload only. To also send the separate `Thinking` message (when available), set `agents.defaults.heartbeat.includeReasoning: true` or per-agent `agents.entries.*.heartbeat.includeReasoning: true`.
|
||||
|
||||
## Web chat UI
|
||||
|
||||
@@ -110,7 +110,19 @@ describe("resolveHeartbeatPromptForSystemPrompt", () => {
|
||||
agentId: "main",
|
||||
defaultAgentId: "main",
|
||||
}),
|
||||
).toBe("Ops check");
|
||||
).toContain("Ops check");
|
||||
expect(
|
||||
resolveHeartbeatPromptForSystemPrompt({
|
||||
config: {
|
||||
agents: {
|
||||
defaults: { heartbeat: { every: "30m" } },
|
||||
list: [{ id: "main", heartbeat: { prompt: "Ops check" } }],
|
||||
},
|
||||
},
|
||||
agentId: "main",
|
||||
defaultAgentId: "main",
|
||||
}),
|
||||
).toContain("Recurring tasks are cron jobs");
|
||||
});
|
||||
|
||||
it("does not inject the heartbeat section for non-default agents", () => {
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
import { normalizeOptionalString } from "@openclaw/normalization-core/string-coerce";
|
||||
import {
|
||||
DEFAULT_HEARTBEAT_EVERY,
|
||||
HEARTBEAT_CRON_TASK_GUIDANCE,
|
||||
resolveHeartbeatPrompt as resolveHeartbeatPromptText,
|
||||
} from "../auto-reply/heartbeat.js";
|
||||
import { parseDurationMs } from "../cli/parse-duration.js";
|
||||
@@ -88,5 +89,8 @@ export function resolveHeartbeatPromptForSystemPrompt(params: {
|
||||
if (!shouldIncludeHeartbeatGuidanceForSystemPrompt(params)) {
|
||||
return undefined;
|
||||
}
|
||||
return resolveHeartbeatPromptText(heartbeat?.prompt);
|
||||
const prompt = resolveHeartbeatPromptText(heartbeat?.prompt);
|
||||
return prompt.includes(HEARTBEAT_CRON_TASK_GUIDANCE)
|
||||
? prompt
|
||||
: `${prompt} ${HEARTBEAT_CRON_TASK_GUIDANCE}`;
|
||||
}
|
||||
|
||||
@@ -83,7 +83,7 @@ const CONTEXT_FILE_ORDER = new Map<string, number>([
|
||||
|
||||
const DYNAMIC_CONTEXT_FILE_BASENAMES = new Set<string>();
|
||||
const DEFAULT_HEARTBEAT_PROMPT_CONTEXT_BLOCK =
|
||||
"Default heartbeat prompt:\n`Follow the heartbeat monitor scratch context when provided. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK.`";
|
||||
"Default heartbeat prompt:\n`Follow the heartbeat monitor scratch context when provided. Recurring tasks are cron jobs; create or change their schedules with cron tools or the openclaw cron CLI, not heartbeat scratch. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK.`";
|
||||
const SYSTEM_PROMPT_STABLE_PREFIX_CACHE_LIMIT = 64;
|
||||
|
||||
type StablePromptPrefixCacheEntry = {
|
||||
|
||||
@@ -27,7 +27,12 @@ const HeartbeatResponseToolSchema = Type.Object(
|
||||
reason: Type.Optional(Type.String()),
|
||||
priority: optionalStringEnum(HEARTBEAT_TOOL_PRIORITIES),
|
||||
nextCheck: Type.Optional(Type.String()),
|
||||
scratch: Type.Optional(Type.String()),
|
||||
scratch: Type.Optional(
|
||||
Type.String({
|
||||
description:
|
||||
"Complete replacement for heartbeat monitor prose. Recurring schedules belong in cron jobs, not scratch.",
|
||||
}),
|
||||
),
|
||||
},
|
||||
{ additionalProperties: false },
|
||||
);
|
||||
@@ -48,7 +53,7 @@ export function createHeartbeatResponseTool(): AnyAgentTool {
|
||||
name: HEARTBEAT_RESPONSE_TOOL_NAME,
|
||||
displaySummary: "Record heartbeat outcome/notify choice.",
|
||||
description:
|
||||
"Record heartbeat result. `notify=false` no visible send. `notify=true` needs concise notificationText.",
|
||||
"Record heartbeat result. `notify=false` no visible send. `notify=true` needs concise notificationText. Scratch is monitor prose only; manage recurring tasks with cron.",
|
||||
parameters: HeartbeatResponseToolSchema,
|
||||
execute: async (_toolCallId, args) => {
|
||||
if (!isRecord(args)) {
|
||||
|
||||
@@ -1,10 +1,9 @@
|
||||
/** Tests heartbeat prompt, token, task parsing, and due-time helpers. */
|
||||
/** Tests heartbeat prompt and token helpers. */
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
DEFAULT_HEARTBEAT_ACK_MAX_CHARS,
|
||||
HEARTBEAT_RESPONSE_TOOL_PROMPT,
|
||||
isHeartbeatContentEffectivelyEmpty,
|
||||
parseHeartbeatTasks,
|
||||
resolveHeartbeatPromptForResponseTool,
|
||||
stripHeartbeatToken,
|
||||
} from "./heartbeat.js";
|
||||
@@ -320,32 +319,3 @@ describe("resolveHeartbeatPromptForResponseTool", () => {
|
||||
expect(prompt).toContain("notify=false");
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseHeartbeatTasks", () => {
|
||||
it("does not bleed top-level interval/prompt fields into task parsing", () => {
|
||||
const content = `tasks:
|
||||
- name: email-check
|
||||
interval: 30m
|
||||
prompt: Check for urgent emails
|
||||
interval: should-not-bleed
|
||||
`;
|
||||
expect(parseHeartbeatTasks(content)).toEqual([
|
||||
{
|
||||
name: "email-check",
|
||||
interval: "30m",
|
||||
prompt: "Check for urgent emails",
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("ignores task blocks inside HTML comments", () => {
|
||||
const content = `<!--
|
||||
tasks:
|
||||
- name: inbox
|
||||
interval: 30m
|
||||
prompt: Check inbox
|
||||
-->
|
||||
`;
|
||||
expect(parseHeartbeatTasks(content)).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,20 +1,13 @@
|
||||
/** Heartbeat prompt defaults, token stripping, task parsing, and due-time helpers. */
|
||||
import { normalizeOptionalString } from "@openclaw/normalization-core/string-coerce";
|
||||
import { parseDurationMs } from "../cli/parse-duration.js";
|
||||
import { escapeRegExp } from "../shared/regexp.js";
|
||||
import { HEARTBEAT_TOKEN } from "./tokens.js";
|
||||
|
||||
/** YAML-like task entry parsed from heartbeat monitor scratch. */
|
||||
export type HeartbeatTask = {
|
||||
name: string;
|
||||
interval: string;
|
||||
prompt: string;
|
||||
};
|
||||
|
||||
// Default heartbeat prompt (used when config.agents.defaults.heartbeat.prompt is unset).
|
||||
// Keep it tight and avoid encouraging the model to invent/rehash "open loops" from prior chat context.
|
||||
const HEARTBEAT_CONTEXT_PROMPT =
|
||||
"Follow the heartbeat monitor scratch context when provided. Do not infer or repeat old tasks from prior chats.";
|
||||
export const HEARTBEAT_CRON_TASK_GUIDANCE =
|
||||
"Recurring tasks are cron jobs; create or change their schedules with cron tools or the openclaw cron CLI, not heartbeat scratch.";
|
||||
const HEARTBEAT_CONTEXT_PROMPT = `Follow the heartbeat monitor scratch context when provided. ${HEARTBEAT_CRON_TASK_GUIDANCE} Do not infer or repeat old tasks from prior chats.`;
|
||||
/** Default prompt for heartbeat turns when config does not override it. */
|
||||
export const HEARTBEAT_PROMPT = `${HEARTBEAT_CONTEXT_PROMPT} If nothing needs attention, reply HEARTBEAT_OK.`;
|
||||
export const HEARTBEAT_RESPONSE_TOOL_INSTRUCTIONS =
|
||||
@@ -250,113 +243,3 @@ export function stripHeartbeatToken(
|
||||
|
||||
return { shouldSkip: false, text: rest, didStrip: true };
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse heartbeat tasks from HEARTBEAT.md content.
|
||||
* Supports YAML-like task definitions:
|
||||
*
|
||||
* tasks:
|
||||
* - name: email-check
|
||||
* interval: 30m
|
||||
* prompt: "Check for urgent unread emails"
|
||||
*/
|
||||
export function parseHeartbeatTasks(content: string): HeartbeatTask[] {
|
||||
const tasks: HeartbeatTask[] = [];
|
||||
const lines = stripHeartbeatHtmlComments(content);
|
||||
let inTasksBlock = false;
|
||||
|
||||
for (const [i, line] of lines.entries()) {
|
||||
const trimmed = line.trim();
|
||||
|
||||
// Detect tasks block start.
|
||||
if (trimmed === "tasks:") {
|
||||
inTasksBlock = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!inTasksBlock) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// End of tasks block is any new top-level content that is not a task field.
|
||||
const isTaskField =
|
||||
trimmed.startsWith("interval:") ||
|
||||
trimmed.startsWith("prompt:") ||
|
||||
trimmed.startsWith("- name:");
|
||||
if (
|
||||
!isTaskField &&
|
||||
!trimmed.startsWith(" ") &&
|
||||
!trimmed.startsWith("\t") &&
|
||||
trimmed &&
|
||||
!trimmed.startsWith("-")
|
||||
) {
|
||||
inTasksBlock = false;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Parse a task entry and scan following indented fields.
|
||||
if (trimmed.startsWith("- name:")) {
|
||||
const name = trimmed
|
||||
.replace("- name:", "")
|
||||
.trim()
|
||||
.replace(/^["']|["']$/g, "");
|
||||
let interval = "";
|
||||
let prompt = "";
|
||||
|
||||
// Look ahead for interval and prompt
|
||||
for (const nextLine of lines.slice(i + 1)) {
|
||||
const nextTrimmed = nextLine.trim();
|
||||
|
||||
// End of this task
|
||||
if (nextTrimmed.startsWith("- name:")) {
|
||||
break;
|
||||
}
|
||||
|
||||
// Check for task fields BEFORE checking for end of block
|
||||
if (
|
||||
nextTrimmed.startsWith("interval:") &&
|
||||
(nextLine.startsWith(" ") || nextLine.startsWith("\t"))
|
||||
) {
|
||||
interval = nextTrimmed
|
||||
.replace("interval:", "")
|
||||
.trim()
|
||||
.replace(/^["']|["']$/g, "");
|
||||
} else if (
|
||||
nextTrimmed.startsWith("prompt:") &&
|
||||
(nextLine.startsWith(" ") || nextLine.startsWith("\t"))
|
||||
) {
|
||||
prompt = nextTrimmed
|
||||
.replace("prompt:", "")
|
||||
.trim()
|
||||
.replace(/^["']|["']$/g, "");
|
||||
} else if (!nextTrimmed.startsWith(" ") && !nextTrimmed.startsWith("\t") && nextTrimmed) {
|
||||
// End of tasks block
|
||||
inTasksBlock = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (name && interval && prompt) {
|
||||
tasks.push({ name, interval, prompt });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return tasks;
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if a task is due based on its interval and last run time.
|
||||
*/
|
||||
export function isTaskDue(lastRunMs: number | undefined, interval: string, nowMs: number): boolean {
|
||||
if (lastRunMs === undefined) {
|
||||
return true; // Never run, always due
|
||||
}
|
||||
|
||||
try {
|
||||
const intervalMs = parseDurationMs(interval, { defaultUnit: "m" });
|
||||
return nowMs - lastRunMs >= intervalMs;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
720
src/commands/doctor-heartbeat-task-migration.test.ts
Normal file
720
src/commands/doctor-heartbeat-task-migration.test.ts
Normal file
@@ -0,0 +1,720 @@
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { resolveDefaultAgentId } from "../agents/agent-scope.js";
|
||||
import * as sessionAccessor from "../config/sessions/session-accessor.js";
|
||||
import { replaceSessionEntry } from "../config/sessions/session-accessor.js";
|
||||
import type { OpenClawConfig } from "../config/types.openclaw.js";
|
||||
import { resolveHeartbeatMonitorSpecs } from "../cron/heartbeat-monitor.js";
|
||||
import { heartbeatTaskDeclarationKey, isHeartbeatTaskCronJob } from "../cron/heartbeat-task.js";
|
||||
import { readCronJobScratchState, writeCronJobScratch } from "../cron/scratch-store.js";
|
||||
import { CronService } from "../cron/service.js";
|
||||
import { loadCronJobsStore, resolveCronJobsStorePathFromConfig } from "../cron/store.js";
|
||||
import { resolveHeartbeatSession } from "../infra/heartbeat-runner.js";
|
||||
import { closeOpenClawAgentDatabasesForTest } from "../state/openclaw-agent-db.js";
|
||||
import { closeOpenClawStateDatabaseForTest } from "../state/openclaw-state-db.js";
|
||||
import {
|
||||
collectHeartbeatTaskMigrationFindings,
|
||||
maybeMigrateHeartbeatTasksToCron,
|
||||
} from "./doctor-heartbeat-task-migration.js";
|
||||
|
||||
const tempDirs: string[] = [];
|
||||
let originalHome: string | undefined;
|
||||
let originalStateDir: string | undefined;
|
||||
|
||||
function createTestCronService(storePath: string, cfg: OpenClawConfig, nowMs: number): CronService {
|
||||
const noop = () => {};
|
||||
const log = { debug: noop, info: noop, warn: noop, error: noop };
|
||||
return new CronService({
|
||||
storePath,
|
||||
nowMs: () => nowMs,
|
||||
cronEnabled: false,
|
||||
cronConfig: cfg.cron,
|
||||
defaultAgentId: resolveDefaultAgentId(cfg),
|
||||
log,
|
||||
enqueueSystemEvent: () => false,
|
||||
requestHeartbeat: noop,
|
||||
runIsolatedAgentJob: async () => ({
|
||||
status: "skipped",
|
||||
error: "tests do not execute cron jobs",
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
originalHome = process.env.HOME;
|
||||
originalStateDir = process.env.OPENCLAW_STATE_DIR;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
vi.restoreAllMocks();
|
||||
closeOpenClawAgentDatabasesForTest();
|
||||
closeOpenClawStateDatabaseForTest();
|
||||
if (originalHome === undefined) {
|
||||
delete process.env.HOME;
|
||||
} else {
|
||||
process.env.HOME = originalHome;
|
||||
}
|
||||
if (originalStateDir === undefined) {
|
||||
delete process.env.OPENCLAW_STATE_DIR;
|
||||
} else {
|
||||
process.env.OPENCLAW_STATE_DIR = originalStateDir;
|
||||
}
|
||||
await Promise.all(tempDirs.splice(0).map((dir) => fs.rm(dir, { recursive: true, force: true })));
|
||||
});
|
||||
|
||||
async function createFixture(
|
||||
nowMs: number,
|
||||
scratchContent = `# Operations
|
||||
|
||||
tasks:
|
||||
- name: inbox
|
||||
interval: 1h
|
||||
prompt: Check urgent inbox items
|
||||
- name: calendar
|
||||
interval: 2h
|
||||
prompt: Check the next meetings
|
||||
|
||||
# Keep alerts concise
|
||||
`,
|
||||
) {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-heartbeat-task-migration-"));
|
||||
tempDirs.push(root);
|
||||
const env = { ...process.env, HOME: path.join(root, "home"), OPENCLAW_STATE_DIR: root };
|
||||
process.env.HOME = env.HOME;
|
||||
process.env.OPENCLAW_STATE_DIR = env.OPENCLAW_STATE_DIR;
|
||||
const cfg = {
|
||||
agents: { defaults: { heartbeat: { every: "30m" } }, list: [{ id: "main" }] },
|
||||
} as OpenClawConfig;
|
||||
const storePath = resolveCronJobsStorePathFromConfig(cfg, env);
|
||||
const cron = createTestCronService(storePath, cfg, nowMs);
|
||||
const spec = resolveHeartbeatMonitorSpecs(cfg, [])[0];
|
||||
if (!spec) {
|
||||
throw new Error("expected heartbeat monitor spec");
|
||||
}
|
||||
const added = await cron.add(spec.input, { enabledExplicit: true, systemOwned: true });
|
||||
const monitor = "job" in added ? added.job : added;
|
||||
writeCronJobScratch({
|
||||
storePath,
|
||||
jobId: monitor.id,
|
||||
content: scratchContent,
|
||||
expectedRevision: 0,
|
||||
options: { env },
|
||||
});
|
||||
const session = resolveHeartbeatSession(
|
||||
cfg,
|
||||
"main",
|
||||
cfg.agents?.defaults?.heartbeat,
|
||||
undefined,
|
||||
env,
|
||||
);
|
||||
await replaceSessionEntry(
|
||||
{ storePath: session.storePath, sessionKey: session.sessionKey, env },
|
||||
{
|
||||
sessionId: "heartbeat-main",
|
||||
updatedAt: nowMs,
|
||||
heartbeatTaskState: { inbox: nowMs - 30 * 60_000 },
|
||||
},
|
||||
);
|
||||
return { cfg, env, monitor, nowMs, session, storePath };
|
||||
}
|
||||
|
||||
async function createExistingInboxJob(fixture: Awaited<ReturnType<typeof createFixture>>) {
|
||||
const cron = createTestCronService(fixture.storePath, fixture.cfg, fixture.nowMs - 60_000);
|
||||
const result = await cron.add(
|
||||
{
|
||||
declarationKey: heartbeatTaskDeclarationKey("main", "inbox"),
|
||||
displayName: "Previous inbox task",
|
||||
name: "inbox",
|
||||
description: "Existing operator-owned state",
|
||||
agentId: "main",
|
||||
enabled: true,
|
||||
schedule: {
|
||||
kind: "every",
|
||||
everyMs: 5 * 60 * 60_000,
|
||||
anchorMs: fixture.nowMs - 60_000,
|
||||
},
|
||||
payload: { kind: "systemEvent", text: "Previous inbox prompt" },
|
||||
sessionTarget: "main",
|
||||
wakeMode: "next-heartbeat",
|
||||
state: { lastRunAtMs: fixture.nowMs - 10_000 },
|
||||
},
|
||||
{ enabledExplicit: true, matchesExisting: isHeartbeatTaskCronJob },
|
||||
);
|
||||
return structuredClone("job" in result ? result.job : result);
|
||||
}
|
||||
|
||||
describe("heartbeat scratch task cron migration", () => {
|
||||
it("previews, preserves cadence, clears the block, and reruns idempotently", async () => {
|
||||
const fixture = await createFixture(2_000_000_000_000);
|
||||
|
||||
await expect(collectHeartbeatTaskMigrationFindings(fixture.cfg, fixture.env)).resolves.toEqual([
|
||||
expect.objectContaining({
|
||||
checkId: "core/doctor/heartbeat-task-cron-migration",
|
||||
requirement: "heartbeat-tasks-in-scratch",
|
||||
target: "main",
|
||||
}),
|
||||
]);
|
||||
const preview = await maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: false,
|
||||
nowMs: fixture.nowMs,
|
||||
});
|
||||
expect(preview).toEqual({ changes: [], warnings: [] });
|
||||
expect(
|
||||
(await loadCronJobsStore(fixture.storePath)).jobs.filter(isHeartbeatTaskCronJob),
|
||||
).toEqual([]);
|
||||
|
||||
const migrated = await maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs,
|
||||
});
|
||||
expect(migrated.warnings).toEqual([]);
|
||||
expect(migrated.changes).toHaveLength(1);
|
||||
|
||||
const jobs = (await loadCronJobsStore(fixture.storePath)).jobs
|
||||
.filter(isHeartbeatTaskCronJob)
|
||||
.toSorted((a, b) => a.name.localeCompare(b.name));
|
||||
expect(jobs).toHaveLength(2);
|
||||
expect(
|
||||
jobs.map((job) => ({ name: job.name, schedule: job.schedule, payload: job.payload })),
|
||||
).toEqual([
|
||||
{
|
||||
name: "calendar",
|
||||
schedule: { kind: "every", everyMs: 2 * 60 * 60_000, anchorMs: fixture.nowMs + 1 },
|
||||
payload: { kind: "systemEvent", text: "Check the next meetings" },
|
||||
},
|
||||
{
|
||||
name: "inbox",
|
||||
schedule: {
|
||||
kind: "every",
|
||||
everyMs: 60 * 60_000,
|
||||
anchorMs: fixture.nowMs + 30 * 60_000,
|
||||
},
|
||||
payload: { kind: "systemEvent", text: "Check urgent inbox items" },
|
||||
},
|
||||
]);
|
||||
expect(jobs.find((job) => job.name === "calendar")?.state.nextRunAtMs).toBe(fixture.nowMs + 1);
|
||||
expect(jobs.find((job) => job.name === "inbox")?.state.nextRunAtMs).toBe(
|
||||
fixture.nowMs + 30 * 60_000,
|
||||
);
|
||||
|
||||
const scratch = readCronJobScratchState(fixture.storePath, fixture.monitor.id, {
|
||||
env: fixture.env,
|
||||
}).scratch;
|
||||
expect(scratch?.content).toContain("# Operations");
|
||||
expect(scratch?.content).toContain("# Keep alerts concise");
|
||||
expect(scratch?.content).not.toContain("tasks:");
|
||||
expect(
|
||||
resolveHeartbeatSession(fixture.cfg, "main", undefined, undefined, fixture.env).entry
|
||||
?.heartbeatTaskState,
|
||||
).toBeUndefined();
|
||||
|
||||
const rerun = await maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs + 10_000,
|
||||
});
|
||||
expect(rerun).toEqual({ changes: [], warnings: [] });
|
||||
expect(
|
||||
(await loadCronJobsStore(fixture.storePath)).jobs.filter(isHeartbeatTaskCronJob),
|
||||
).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("leaves cron jobs and legacy timestamps untouched when the scratch revision changes", async () => {
|
||||
const fixture = await createFixture(2_000_000_000_000);
|
||||
const existingSnapshot = await createExistingInboxJob(fixture);
|
||||
const concurrentScratch = `# Concurrent replacement
|
||||
tasks:
|
||||
- name: follow-up
|
||||
interval: 3h
|
||||
prompt: Run the concurrent follow-up
|
||||
# Keep concurrent prose
|
||||
`;
|
||||
const migration = maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs,
|
||||
});
|
||||
const current = readCronJobScratchState(fixture.storePath, fixture.monitor.id, {
|
||||
env: fixture.env,
|
||||
});
|
||||
writeCronJobScratch({
|
||||
storePath: fixture.storePath,
|
||||
jobId: fixture.monitor.id,
|
||||
content: concurrentScratch,
|
||||
expectedRevision: current.currentRevision,
|
||||
options: { env: fixture.env },
|
||||
});
|
||||
const result = await migration;
|
||||
|
||||
expect(result.changes).toEqual([]);
|
||||
expect(result.warnings.join("\n")).toContain("scratch changed during task migration");
|
||||
expect(
|
||||
readCronJobScratchState(fixture.storePath, fixture.monitor.id, { env: fixture.env }).scratch
|
||||
?.content,
|
||||
).toBe(concurrentScratch);
|
||||
const jobs = (await loadCronJobsStore(fixture.storePath)).jobs.filter(isHeartbeatTaskCronJob);
|
||||
expect(jobs).toEqual([existingSnapshot]);
|
||||
expect(
|
||||
resolveHeartbeatSession(fixture.cfg, "main", undefined, undefined, fixture.env).entry
|
||||
?.heartbeatTaskState,
|
||||
).toEqual({ inbox: fixture.nowMs - 30 * 60_000 });
|
||||
});
|
||||
|
||||
it("serializes two plans pinned to one scratch revision and converges the loser on rerun", async () => {
|
||||
const fixture = await createFixture(2_000_000_000_000);
|
||||
const outcomes = await Promise.all([
|
||||
maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs,
|
||||
}),
|
||||
maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs,
|
||||
}),
|
||||
]);
|
||||
|
||||
expect(outcomes.filter((outcome) => outcome.changes.length === 1)).toHaveLength(1);
|
||||
expect(
|
||||
outcomes.filter((outcome) =>
|
||||
outcome.warnings.some((warning) =>
|
||||
warning.includes("scratch changed during task migration"),
|
||||
),
|
||||
),
|
||||
).toHaveLength(1);
|
||||
const committedJobs = (await loadCronJobsStore(fixture.storePath)).jobs.filter(
|
||||
isHeartbeatTaskCronJob,
|
||||
);
|
||||
expect(committedJobs).toHaveLength(2);
|
||||
expect(new Set(committedJobs.map((job) => job.declarationKey)).size).toBe(2);
|
||||
expect(
|
||||
readCronJobScratchState(fixture.storePath, fixture.monitor.id, { env: fixture.env }).scratch
|
||||
?.content,
|
||||
).not.toContain("tasks:");
|
||||
|
||||
const rerun = await maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs + 10_000,
|
||||
});
|
||||
expect(rerun).toEqual({ changes: [], warnings: [] });
|
||||
expect(
|
||||
(await loadCronJobsStore(fixture.storePath)).jobs.filter(isHeartbeatTaskCronJob),
|
||||
).toEqual(committedJobs);
|
||||
});
|
||||
|
||||
it("tolerates a crash after the state transaction and before legacy timestamp cleanup", async () => {
|
||||
const fixture = await createFixture(2_000_000_000_000);
|
||||
const cleanup = vi
|
||||
.spyOn(sessionAccessor, "patchSessionEntry")
|
||||
.mockRejectedValueOnce(new Error("simulated post-commit crash"));
|
||||
const result = await maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs,
|
||||
});
|
||||
|
||||
expect(result.changes).toHaveLength(1);
|
||||
expect(result.warnings.join("\n")).toContain("simulated post-commit crash");
|
||||
const committedJobs = (await loadCronJobsStore(fixture.storePath)).jobs.filter(
|
||||
isHeartbeatTaskCronJob,
|
||||
);
|
||||
expect(committedJobs).toHaveLength(2);
|
||||
expect(
|
||||
readCronJobScratchState(fixture.storePath, fixture.monitor.id, { env: fixture.env }).scratch
|
||||
?.content,
|
||||
).not.toContain("tasks:");
|
||||
expect(
|
||||
resolveHeartbeatSession(fixture.cfg, "main", undefined, undefined, fixture.env).entry
|
||||
?.heartbeatTaskState,
|
||||
).toEqual({ inbox: fixture.nowMs - 30 * 60_000 });
|
||||
|
||||
cleanup.mockRestore();
|
||||
await expect(
|
||||
maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs + 10_000,
|
||||
}),
|
||||
).resolves.toEqual({ changes: [], warnings: [] });
|
||||
expect(
|
||||
(await loadCronJobsStore(fixture.storePath)).jobs.filter(isHeartbeatTaskCronJob),
|
||||
).toEqual(committedJobs);
|
||||
});
|
||||
|
||||
it("migrates duplicate-name tasks with stable identities and shared initial due time", async () => {
|
||||
const fixture = await createFixture(2_000_000_000_000);
|
||||
const state = readCronJobScratchState(fixture.storePath, fixture.monitor.id, {
|
||||
env: fixture.env,
|
||||
});
|
||||
const duplicate = `tasks:
|
||||
- name: inbox
|
||||
interval: 1h
|
||||
prompt: First
|
||||
- name: inbox
|
||||
interval: 1h
|
||||
prompt: Second
|
||||
`;
|
||||
writeCronJobScratch({
|
||||
storePath: fixture.storePath,
|
||||
jobId: fixture.monitor.id,
|
||||
content: duplicate,
|
||||
expectedRevision: state.currentRevision,
|
||||
options: { env: fixture.env },
|
||||
});
|
||||
|
||||
const result = await maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs,
|
||||
});
|
||||
|
||||
expect(result.warnings).toEqual([]);
|
||||
expect(result.changes).toHaveLength(1);
|
||||
expect(
|
||||
readCronJobScratchState(fixture.storePath, fixture.monitor.id, { env: fixture.env }).scratch
|
||||
?.content,
|
||||
).toBe("");
|
||||
const jobs = (await loadCronJobsStore(fixture.storePath)).jobs
|
||||
.filter(isHeartbeatTaskCronJob)
|
||||
.toSorted((left, right) => left.payload.text.localeCompare(right.payload.text));
|
||||
expect(jobs).toHaveLength(2);
|
||||
expect(jobs.map((job) => job.declarationKey)).toEqual([
|
||||
heartbeatTaskDeclarationKey("main", "inbox", 0),
|
||||
heartbeatTaskDeclarationKey("main", "inbox", 1),
|
||||
]);
|
||||
expect(new Set(jobs.map((job) => job.declarationKey)).size).toBe(2);
|
||||
expect(jobs.map((job) => job.schedule)).toEqual([
|
||||
{
|
||||
kind: "every",
|
||||
everyMs: 60 * 60_000,
|
||||
anchorMs: fixture.nowMs + 30 * 60_000,
|
||||
},
|
||||
{
|
||||
kind: "every",
|
||||
everyMs: 60 * 60_000,
|
||||
anchorMs: fixture.nowMs + 30 * 60_000,
|
||||
},
|
||||
]);
|
||||
const initialIdentities = jobs.map((job) => ({ id: job.id, key: job.declarationKey }));
|
||||
|
||||
await expect(
|
||||
maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs + 10_000,
|
||||
}),
|
||||
).resolves.toEqual({ changes: [], warnings: [] });
|
||||
const rerunJobs = (await loadCronJobsStore(fixture.storePath)).jobs
|
||||
.filter(isHeartbeatTaskCronJob)
|
||||
.toSorted((left, right) => left.payload.text.localeCompare(right.payload.text));
|
||||
expect(rerunJobs.map((job) => ({ id: job.id, key: job.declarationKey }))).toEqual(
|
||||
initialIdentities,
|
||||
);
|
||||
});
|
||||
|
||||
it("migrates a mixed unique and duplicate-name task block completely", async () => {
|
||||
const content = `tasks:
|
||||
- name: inbox
|
||||
interval: 1h
|
||||
prompt: First inbox pass
|
||||
- name: calendar
|
||||
interval: 2h
|
||||
prompt: Calendar pass
|
||||
- name: inbox
|
||||
interval: 1h
|
||||
prompt: Second inbox pass
|
||||
`;
|
||||
const fixture = await createFixture(2_000_000_000_000, content);
|
||||
|
||||
const result = await maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs,
|
||||
});
|
||||
|
||||
expect(result.warnings).toEqual([]);
|
||||
const jobs = (await loadCronJobsStore(fixture.storePath)).jobs.filter(isHeartbeatTaskCronJob);
|
||||
expect(jobs).toHaveLength(3);
|
||||
expect(new Set(jobs.map((job) => job.declarationKey)).size).toBe(3);
|
||||
expect(
|
||||
jobs.map((job) => job.payload.text).toSorted((left, right) => left.localeCompare(right)),
|
||||
).toEqual(["Calendar pass", "First inbox pass", "Second inbox pass"]);
|
||||
expect(
|
||||
readCronJobScratchState(fixture.storePath, fixture.monitor.id, { env: fixture.env }).scratch
|
||||
?.content,
|
||||
).toBe("");
|
||||
});
|
||||
|
||||
it("refuses orphan task fields beside a valid task without changing scratch", async () => {
|
||||
const content = `# Operations
|
||||
tasks:
|
||||
interval: 15m
|
||||
prompt: Orphaned work must not disappear
|
||||
- name: inbox
|
||||
interval: 1h
|
||||
prompt: Check urgent inbox items
|
||||
# Keep alerts concise
|
||||
`;
|
||||
const fixture = await createFixture(2_000_000_000_000, content);
|
||||
|
||||
await expect(collectHeartbeatTaskMigrationFindings(fixture.cfg, fixture.env)).resolves.toEqual([
|
||||
expect.objectContaining({
|
||||
severity: "error",
|
||||
requirement: "heartbeat-task-migration-blocked",
|
||||
message: expect.stringContaining("incomplete name/interval/prompt entry"),
|
||||
}),
|
||||
]);
|
||||
const result = await maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs,
|
||||
});
|
||||
|
||||
expect(result.changes).toEqual([]);
|
||||
expect(result.warnings.join("\n")).toContain("incomplete name/interval/prompt entry");
|
||||
expect(
|
||||
readCronJobScratchState(fixture.storePath, fixture.monitor.id, { env: fixture.env }).scratch
|
||||
?.content,
|
||||
).toBe(content);
|
||||
expect(
|
||||
(await loadCronJobsStore(fixture.storePath)).jobs.filter(isHeartbeatTaskCronJob),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it("does not migrate a task block hidden by a mid-line HTML comment opener", async () => {
|
||||
const content = `Notes <!--
|
||||
tasks:
|
||||
- name: disabled
|
||||
interval: 5m
|
||||
prompt: This must remain disabled
|
||||
-->
|
||||
# Keep this scratch
|
||||
`;
|
||||
const fixture = await createFixture(2_000_000_000_000, content);
|
||||
|
||||
await expect(collectHeartbeatTaskMigrationFindings(fixture.cfg, fixture.env)).resolves.toEqual(
|
||||
[],
|
||||
);
|
||||
await expect(
|
||||
maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs,
|
||||
}),
|
||||
).resolves.toEqual({ changes: [], warnings: [] });
|
||||
|
||||
expect(
|
||||
(await loadCronJobsStore(fixture.storePath)).jobs.filter(isHeartbeatTaskCronJob),
|
||||
).toEqual([]);
|
||||
expect(
|
||||
readCronJobScratchState(fixture.storePath, fixture.monitor.id, { env: fixture.env }).scratch
|
||||
?.content,
|
||||
).toBe(content);
|
||||
});
|
||||
|
||||
it("keeps a multiline comment closed when its opener shares a migrated task line", async () => {
|
||||
const content = `tasks:
|
||||
- name: active
|
||||
interval: 1h
|
||||
prompt: Run the active check <!--
|
||||
tasks:
|
||||
- name: disabled
|
||||
interval: 5m
|
||||
prompt: This must remain disabled
|
||||
-->
|
||||
# Keep this scratch
|
||||
`;
|
||||
const fixture = await createFixture(2_000_000_000_000, content);
|
||||
|
||||
const migrated = await maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs,
|
||||
});
|
||||
|
||||
expect(migrated.warnings).toEqual([]);
|
||||
const jobs = (await loadCronJobsStore(fixture.storePath)).jobs.filter(isHeartbeatTaskCronJob);
|
||||
expect(jobs.map((job) => job.name)).toEqual(["active"]);
|
||||
const scratch = readCronJobScratchState(fixture.storePath, fixture.monitor.id, {
|
||||
env: fixture.env,
|
||||
}).scratch?.content;
|
||||
expect(scratch).toContain(`<!--
|
||||
tasks:
|
||||
- name: disabled
|
||||
interval: 5m
|
||||
prompt: This must remain disabled
|
||||
-->`);
|
||||
await expect(
|
||||
maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs + 10_000,
|
||||
}),
|
||||
).resolves.toEqual({ changes: [], warnings: [] });
|
||||
});
|
||||
|
||||
it("preserves indented non-task prose and its line endings byte-for-byte", async () => {
|
||||
const content =
|
||||
"# Operations\r\n" +
|
||||
"tasks:\r\n" +
|
||||
" - name: inbox\r\n" +
|
||||
" interval: 1h\r\n" +
|
||||
" prompt: Check urgent inbox items\r\n" +
|
||||
" Keep this indented note exactly. \r\n" +
|
||||
"# Keep alerts concise\r\n";
|
||||
const fixture = await createFixture(2_000_000_000_000, content);
|
||||
|
||||
const result = await maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs,
|
||||
});
|
||||
|
||||
expect(result.warnings).toEqual([]);
|
||||
expect(
|
||||
readCronJobScratchState(fixture.storePath, fixture.monitor.id, { env: fixture.env }).scratch
|
||||
?.content,
|
||||
).toBe("# Operations\r\n Keep this indented note exactly. \r\n# Keep alerts concise\r\n");
|
||||
});
|
||||
|
||||
it("uses the supplied environment for legacy session timing and cleanup", async () => {
|
||||
const fixture = await createFixture(2_000_000_000_000);
|
||||
const suppliedHome = await fs.mkdtemp(
|
||||
path.join(os.tmpdir(), "openclaw-heartbeat-task-migration-supplied-home-"),
|
||||
);
|
||||
tempDirs.push(suppliedHome);
|
||||
fixture.cfg.session = {
|
||||
store: "~/.openclaw/agents/{agentId}/sessions/sessions.json",
|
||||
};
|
||||
const suppliedEnv = { ...fixture.env, HOME: suppliedHome };
|
||||
const suppliedSession = resolveHeartbeatSession(
|
||||
fixture.cfg,
|
||||
"main",
|
||||
fixture.cfg.agents?.defaults?.heartbeat,
|
||||
undefined,
|
||||
suppliedEnv,
|
||||
);
|
||||
await replaceSessionEntry(
|
||||
{
|
||||
storePath: suppliedSession.storePath,
|
||||
sessionKey: suppliedSession.sessionKey,
|
||||
env: suppliedEnv,
|
||||
},
|
||||
{
|
||||
sessionId: "supplied-heartbeat-main",
|
||||
updatedAt: fixture.nowMs,
|
||||
heartbeatTaskState: { inbox: fixture.nowMs - 30 * 60_000 },
|
||||
},
|
||||
);
|
||||
const ambientSession = resolveHeartbeatSession(
|
||||
fixture.cfg,
|
||||
"main",
|
||||
fixture.cfg.agents?.defaults?.heartbeat,
|
||||
);
|
||||
await replaceSessionEntry(
|
||||
{
|
||||
storePath: ambientSession.storePath,
|
||||
sessionKey: ambientSession.sessionKey,
|
||||
},
|
||||
{
|
||||
sessionId: "ambient-heartbeat-main",
|
||||
updatedAt: fixture.nowMs,
|
||||
heartbeatTaskState: {
|
||||
inbox: fixture.nowMs - 10 * 60_000,
|
||||
untouched: fixture.nowMs - 5_000,
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
const result = await maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: suppliedEnv,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs,
|
||||
});
|
||||
|
||||
expect(result.warnings).toEqual([]);
|
||||
const inbox = (await loadCronJobsStore(fixture.storePath)).jobs.find(
|
||||
(job) => isHeartbeatTaskCronJob(job) && job.name === "inbox",
|
||||
);
|
||||
expect(inbox?.schedule).toEqual({
|
||||
kind: "every",
|
||||
everyMs: 60 * 60_000,
|
||||
anchorMs: fixture.nowMs + 30 * 60_000,
|
||||
});
|
||||
expect(
|
||||
resolveHeartbeatSession(fixture.cfg, "main", undefined, undefined, suppliedEnv).entry
|
||||
?.heartbeatTaskState,
|
||||
).toBeUndefined();
|
||||
expect(resolveHeartbeatSession(fixture.cfg, "main").entry?.heartbeatTaskState).toEqual({
|
||||
inbox: fixture.nowMs - 10 * 60_000,
|
||||
untouched: fixture.nowMs - 5_000,
|
||||
});
|
||||
});
|
||||
|
||||
it("removes consecutive task blocks in one idempotent migration", async () => {
|
||||
const content = `# Operations
|
||||
tasks:
|
||||
- name: inbox
|
||||
interval: 1h
|
||||
prompt: Check urgent inbox items
|
||||
tasks:
|
||||
- name: calendar
|
||||
interval: 2h
|
||||
prompt: Check the next meetings
|
||||
# Keep alerts concise
|
||||
`;
|
||||
const fixture = await createFixture(2_000_000_000_000, content);
|
||||
|
||||
const migrated = await maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs,
|
||||
});
|
||||
|
||||
expect(migrated.warnings).toEqual([]);
|
||||
expect(
|
||||
readCronJobScratchState(fixture.storePath, fixture.monitor.id, { env: fixture.env }).scratch
|
||||
?.content,
|
||||
).toBe("# Operations\n# Keep alerts concise\n");
|
||||
expect(
|
||||
(await loadCronJobsStore(fixture.storePath)).jobs.filter(isHeartbeatTaskCronJob),
|
||||
).toHaveLength(2);
|
||||
|
||||
await expect(
|
||||
maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: fixture.cfg,
|
||||
env: fixture.env,
|
||||
shouldRepair: true,
|
||||
nowMs: fixture.nowMs + 10_000,
|
||||
}),
|
||||
).resolves.toEqual({ changes: [], warnings: [] });
|
||||
expect(
|
||||
(await loadCronJobsStore(fixture.storePath)).jobs.filter(isHeartbeatTaskCronJob),
|
||||
).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
611
src/commands/doctor-heartbeat-task-migration.ts
Normal file
611
src/commands/doctor-heartbeat-task-migration.ts
Normal file
@@ -0,0 +1,611 @@
|
||||
/** Doctor-owned migration from heartbeat scratch `tasks:` blocks into cron jobs. */
|
||||
|
||||
import { randomUUID } from "node:crypto";
|
||||
import type { DatabaseSync } from "node:sqlite";
|
||||
import { isDeepStrictEqual } from "node:util";
|
||||
import { truncateUtf16Safe } from "@openclaw/normalization-core/utf16-slice";
|
||||
import { note } from "../../packages/terminal-core/src/note.js";
|
||||
import { formatCliCommand } from "../cli/command-format.js";
|
||||
import { parseDurationMs } from "../cli/parse-duration.js";
|
||||
import { patchSessionEntry } from "../config/sessions/session-accessor.js";
|
||||
import type { OpenClawConfig } from "../config/types.openclaw.js";
|
||||
import { heartbeatTaskDeclarationKey, isHeartbeatTaskCronJob } from "../cron/heartbeat-task.js";
|
||||
import { cronSchedulingInputsEqual } from "../cron/schedule-identity.js";
|
||||
import { readHeartbeatMonitorScratch } from "../cron/scratch-store.js";
|
||||
import { computeJobNextRunAtMs, hasScheduledNextRunAtMs } from "../cron/service/jobs.js";
|
||||
import { resolveCronJobsStorePathFromConfig } from "../cron/store.js";
|
||||
import { cronStoreKey } from "../cron/store/key.js";
|
||||
import {
|
||||
assertCronStoreCanPersist,
|
||||
loadedCronStoreFromRows,
|
||||
loadCronRows,
|
||||
upsertCronJobRow,
|
||||
} from "../cron/store/row-codec.js";
|
||||
import { getCronStoreKysely } from "../cron/store/schema.js";
|
||||
import type { CronJob } from "../cron/types.js";
|
||||
import type { HealthFinding } from "../flows/health-checks.js";
|
||||
import { resolveHeartbeatAgents, resolveHeartbeatSession } from "../infra/heartbeat-runner.js";
|
||||
import { executeSqliteQuerySync } from "../infra/kysely-sync.js";
|
||||
import {
|
||||
openOpenClawStateDatabase,
|
||||
runOpenClawStateWriteTransaction,
|
||||
} from "../state/openclaw-state-db.js";
|
||||
import { shortenHomePath } from "../utils.js";
|
||||
import { analyzeLegacyHeartbeatTasks, type LegacyHeartbeatTask } from "./heartbeat-task-legacy.js";
|
||||
|
||||
const HEARTBEAT_TASK_MIGRATION_CHECK_ID = "core/doctor/heartbeat-task-cron-migration";
|
||||
|
||||
type HeartbeatTaskMigrationResult = { changes: string[]; warnings: string[] };
|
||||
|
||||
function errorMessage(error: unknown): string {
|
||||
return error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
|
||||
type ValidatedHeartbeatTask = {
|
||||
task: LegacyHeartbeatTask;
|
||||
intervalMs: number;
|
||||
occurrenceIndex: number;
|
||||
};
|
||||
|
||||
function validateTasks(
|
||||
tasks: readonly LegacyHeartbeatTask[],
|
||||
declaredEntryCount: number,
|
||||
): ValidatedHeartbeatTask[] {
|
||||
if (tasks.length === 0) {
|
||||
throw new Error("tasks: block has no complete name/interval/prompt entries");
|
||||
}
|
||||
if (tasks.length !== declaredEntryCount) {
|
||||
throw new Error("tasks: block contains an incomplete name/interval/prompt entry");
|
||||
}
|
||||
const occurrenceCounts = new Map<string, number>();
|
||||
const validated: ValidatedHeartbeatTask[] = [];
|
||||
for (const task of tasks) {
|
||||
const intervalMs = parseDurationMs(task.interval, { defaultUnit: "m" });
|
||||
if (intervalMs <= 0) {
|
||||
throw new Error(`task ${JSON.stringify(task.name)} interval must be greater than zero`);
|
||||
}
|
||||
const occurrenceIndex = occurrenceCounts.get(task.name) ?? 0;
|
||||
occurrenceCounts.set(task.name, occurrenceIndex + 1);
|
||||
validated.push({ task, intervalMs, occurrenceIndex });
|
||||
}
|
||||
return validated;
|
||||
}
|
||||
|
||||
function migrationFinding(params: {
|
||||
storePath: string;
|
||||
agentId: string;
|
||||
message: string;
|
||||
severity?: HealthFinding["severity"];
|
||||
requirement: string;
|
||||
}): HealthFinding {
|
||||
return {
|
||||
checkId: HEARTBEAT_TASK_MIGRATION_CHECK_ID,
|
||||
severity: params.severity ?? "warning",
|
||||
message: params.message,
|
||||
path: params.storePath,
|
||||
target: params.agentId,
|
||||
requirement: params.requirement,
|
||||
fixHint: `Run ${formatCliCommand("openclaw doctor --fix")} to convert heartbeat tasks into cron jobs.`,
|
||||
};
|
||||
}
|
||||
|
||||
/** Reports task blocks still owned by heartbeat scratch without changing them. */
|
||||
export async function collectHeartbeatTaskMigrationFindings(
|
||||
cfg: OpenClawConfig,
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
): Promise<readonly HealthFinding[]> {
|
||||
const storePath = resolveCronJobsStorePathFromConfig(cfg, env);
|
||||
const findings: HealthFinding[] = [];
|
||||
for (const agent of resolveHeartbeatAgents(cfg)) {
|
||||
let monitor: ReturnType<typeof readHeartbeatMonitorScratch>;
|
||||
try {
|
||||
monitor = readHeartbeatMonitorScratch(storePath, agent.agentId, { env });
|
||||
} catch (error) {
|
||||
findings.push(
|
||||
migrationFinding({
|
||||
storePath,
|
||||
agentId: agent.agentId,
|
||||
requirement: "heartbeat-task-migration-blocked",
|
||||
severity: "error",
|
||||
message: `Agent "${agent.agentId}" heartbeat scratch cannot be inspected: ${errorMessage(error)}`,
|
||||
}),
|
||||
);
|
||||
continue;
|
||||
}
|
||||
const content = monitor?.state.scratch?.content;
|
||||
if (!content) {
|
||||
continue;
|
||||
}
|
||||
const document = analyzeLegacyHeartbeatTasks(content);
|
||||
if (!document.hasTasksBlock) {
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
validateTasks(document.tasks, document.taskEntryCount);
|
||||
findings.push(
|
||||
migrationFinding({
|
||||
storePath,
|
||||
agentId: agent.agentId,
|
||||
requirement: "heartbeat-tasks-in-scratch",
|
||||
message: `Agent "${agent.agentId}" has ${document.tasks.length} heartbeat task${document.tasks.length === 1 ? "" : "s"} that must become cron jobs.`,
|
||||
}),
|
||||
);
|
||||
} catch (error) {
|
||||
findings.push(
|
||||
migrationFinding({
|
||||
storePath,
|
||||
agentId: agent.agentId,
|
||||
requirement: "heartbeat-task-migration-blocked",
|
||||
severity: "error",
|
||||
message: `Agent "${agent.agentId}" heartbeat tasks cannot be migrated: ${errorMessage(error)}`,
|
||||
}),
|
||||
);
|
||||
}
|
||||
}
|
||||
return findings;
|
||||
}
|
||||
|
||||
function taskJobInput(params: {
|
||||
agentId: string;
|
||||
task: LegacyHeartbeatTask;
|
||||
occurrenceIndex: number;
|
||||
intervalMs: number;
|
||||
lastRunAtMs?: number;
|
||||
existing?: CronJob;
|
||||
nowMs: number;
|
||||
}) {
|
||||
const existingAnchor =
|
||||
params.existing?.schedule.kind === "every" &&
|
||||
params.existing.schedule.everyMs === params.intervalMs
|
||||
? params.existing.schedule.anchorMs
|
||||
: undefined;
|
||||
const nextDueMs =
|
||||
params.lastRunAtMs === undefined || params.lastRunAtMs + params.intervalMs <= params.nowMs
|
||||
? params.nowMs + 1
|
||||
: params.lastRunAtMs + params.intervalMs;
|
||||
return {
|
||||
declarationKey: heartbeatTaskDeclarationKey(
|
||||
params.agentId,
|
||||
params.task.name,
|
||||
params.occurrenceIndex,
|
||||
),
|
||||
displayName: truncateUtf16Safe(`Heartbeat task: ${params.task.name}`, 200),
|
||||
name: params.task.name,
|
||||
description: "Migrated from heartbeat monitor scratch by openclaw doctor.",
|
||||
agentId: params.agentId,
|
||||
enabled: true,
|
||||
schedule: {
|
||||
kind: "every" as const,
|
||||
everyMs: params.intervalMs,
|
||||
anchorMs: existingAnchor ?? nextDueMs,
|
||||
},
|
||||
payload: { kind: "systemEvent" as const, text: params.task.prompt },
|
||||
sessionTarget: "main" as const,
|
||||
wakeMode: "next-heartbeat" as const,
|
||||
...(params.lastRunAtMs === undefined ? {} : { state: { lastRunAtMs: params.lastRunAtMs } }),
|
||||
};
|
||||
}
|
||||
|
||||
type TaskJobPlan = {
|
||||
declarationKey: string;
|
||||
previous?: CronJob;
|
||||
job: CronJob;
|
||||
sortOrder: number;
|
||||
};
|
||||
|
||||
type AgentTaskMigrationPlan = {
|
||||
monitorJobId: string;
|
||||
scratchRevision: number;
|
||||
strippedContent: string;
|
||||
jobs: TaskJobPlan[];
|
||||
};
|
||||
|
||||
type CronPlanningSnapshot = {
|
||||
jobs: CronJob[];
|
||||
sortOrderByJobId: Map<string, number>;
|
||||
nextSortOrder: number;
|
||||
};
|
||||
|
||||
type MigrationCommitResult =
|
||||
| { ok: true; currentRevision: number }
|
||||
| { ok: false; reason: "job-conflict" | "revision-conflict" };
|
||||
|
||||
function taskDeclarativeFields(job: CronJob) {
|
||||
return {
|
||||
schedule: job.schedule,
|
||||
pacing: job.pacing,
|
||||
trigger: job.trigger,
|
||||
payload: job.payload,
|
||||
delivery: job.delivery,
|
||||
displayName: job.displayName,
|
||||
enabled: job.enabled,
|
||||
};
|
||||
}
|
||||
|
||||
function convergeTaskJob(params: {
|
||||
agentId: string;
|
||||
task: LegacyHeartbeatTask;
|
||||
occurrenceIndex: number;
|
||||
intervalMs: number;
|
||||
lastRunAtMs?: number;
|
||||
existing?: CronJob;
|
||||
nowMs: number;
|
||||
}): CronJob {
|
||||
const input = taskJobInput(params);
|
||||
if (!params.existing) {
|
||||
const { state, ...fields } = input;
|
||||
const job: CronJob = {
|
||||
id: randomUUID(),
|
||||
...fields,
|
||||
createdAtMs: params.nowMs,
|
||||
updatedAtMs: params.nowMs,
|
||||
state: { ...state },
|
||||
};
|
||||
job.state.nextRunAtMs = computeJobNextRunAtMs(job, params.nowMs);
|
||||
return job;
|
||||
}
|
||||
|
||||
const previous = params.existing;
|
||||
const job = structuredClone(previous);
|
||||
job.displayName = input.displayName;
|
||||
job.schedule = structuredClone(input.schedule);
|
||||
job.payload = structuredClone(input.payload);
|
||||
job.enabled = true;
|
||||
delete job.pacing;
|
||||
delete job.trigger;
|
||||
delete job.delivery;
|
||||
if (isDeepStrictEqual(taskDeclarativeFields(previous), taskDeclarativeFields(job))) {
|
||||
return job;
|
||||
}
|
||||
|
||||
job.updatedAtMs = params.nowMs;
|
||||
if (!cronSchedulingInputsEqual(previous, job)) {
|
||||
job.state.startupCatchupAtMs = undefined;
|
||||
job.state.pacedNextRunAtMs = undefined;
|
||||
job.state.forcePreservedNextRunAtMs = undefined;
|
||||
job.state.nextRunAtMs = computeJobNextRunAtMs(job, params.nowMs);
|
||||
} else if (!hasScheduledNextRunAtMs(job.state.nextRunAtMs)) {
|
||||
job.state.nextRunAtMs = computeJobNextRunAtMs(job, params.nowMs);
|
||||
}
|
||||
return job;
|
||||
}
|
||||
|
||||
async function loadCronPlanningSnapshot(
|
||||
storePath: string,
|
||||
env: NodeJS.ProcessEnv,
|
||||
): Promise<CronPlanningSnapshot> {
|
||||
const rows = loadCronRows(openOpenClawStateDatabase({ env }).db, cronStoreKey(storePath));
|
||||
const sortOrderByJobId = new Map(rows.map((row) => [row.job_id, row.sort_order] as const));
|
||||
return {
|
||||
jobs: loadedCronStoreFromRows(rows).store.jobs,
|
||||
sortOrderByJobId,
|
||||
nextSortOrder: rows.reduce((max, row) => Math.max(max, row.sort_order + 1), 0),
|
||||
};
|
||||
}
|
||||
|
||||
function reserveSortOrder(snapshot: CronPlanningSnapshot, existing?: CronJob): number {
|
||||
const persisted = existing ? snapshot.sortOrderByJobId.get(existing.id) : undefined;
|
||||
if (persisted !== undefined) {
|
||||
return persisted;
|
||||
}
|
||||
const sortOrder = snapshot.nextSortOrder;
|
||||
snapshot.nextSortOrder += 1;
|
||||
return sortOrder;
|
||||
}
|
||||
|
||||
function readScratchRevision(db: DatabaseSync, storeKey: string, jobId: string): number {
|
||||
return (
|
||||
executeSqliteQuerySync(
|
||||
db,
|
||||
getCronStoreKysely(db)
|
||||
.selectFrom("cron_job_scratch")
|
||||
.select("revision")
|
||||
.where("store_key", "=", storeKey)
|
||||
.where("job_id", "=", jobId),
|
||||
).rows[0]?.revision ?? 0
|
||||
);
|
||||
}
|
||||
|
||||
function commitAgentTaskMigration(params: {
|
||||
storePath: string;
|
||||
env: NodeJS.ProcessEnv;
|
||||
nowMs: number;
|
||||
plan: AgentTaskMigrationPlan;
|
||||
}): MigrationCommitResult {
|
||||
const storeKey = cronStoreKey(params.storePath);
|
||||
return runOpenClawStateWriteTransaction(
|
||||
({ db }) => {
|
||||
if (
|
||||
readScratchRevision(db, storeKey, params.plan.monitorJobId) !== params.plan.scratchRevision
|
||||
) {
|
||||
return { ok: false, reason: "revision-conflict" } as const;
|
||||
}
|
||||
|
||||
const rows = loadCronRows(db, storeKey);
|
||||
const jobsById = new Map(
|
||||
loadedCronStoreFromRows(rows).store.jobs.map((job) => [job.id, job] as const),
|
||||
);
|
||||
for (const jobPlan of params.plan.jobs) {
|
||||
const matchingRows = rows.filter((row) => row.declaration_key === jobPlan.declarationKey);
|
||||
if (jobPlan.previous) {
|
||||
const current = jobsById.get(jobPlan.previous.id);
|
||||
if (
|
||||
matchingRows.length !== 1 ||
|
||||
!current ||
|
||||
!isDeepStrictEqual(current, jobPlan.previous)
|
||||
) {
|
||||
return { ok: false, reason: "job-conflict" } as const;
|
||||
}
|
||||
} else if (matchingRows.length > 0 || rows.some((row) => row.job_id === jobPlan.job.id)) {
|
||||
return { ok: false, reason: "job-conflict" } as const;
|
||||
}
|
||||
}
|
||||
|
||||
for (const jobPlan of params.plan.jobs) {
|
||||
if (!jobPlan.previous || !isDeepStrictEqual(jobPlan.previous, jobPlan.job)) {
|
||||
upsertCronJobRow(db, storeKey, jobPlan.job, jobPlan.sortOrder);
|
||||
}
|
||||
}
|
||||
|
||||
const updated = executeSqliteQuerySync(
|
||||
db,
|
||||
getCronStoreKysely(db)
|
||||
.updateTable("cron_job_scratch")
|
||||
.set({
|
||||
content: params.plan.strippedContent,
|
||||
revision: params.plan.scratchRevision + 1,
|
||||
source_sha256: null,
|
||||
updated_at_ms: params.nowMs,
|
||||
})
|
||||
.where("store_key", "=", storeKey)
|
||||
.where("job_id", "=", params.plan.monitorJobId)
|
||||
.where("revision", "=", params.plan.scratchRevision),
|
||||
);
|
||||
if (updated.numAffectedRows !== 1n) {
|
||||
throw new Error("scratch revision changed inside task migration transaction");
|
||||
}
|
||||
// Like cadence materialization, doctor only commits durable rows. A live
|
||||
// gateway reloads the cron store through its normal reload path and arms
|
||||
// these persisted nextRunAtMs values; doctor never owns its timer.
|
||||
return { ok: true, currentRevision: params.plan.scratchRevision + 1 } as const;
|
||||
},
|
||||
{ env: params.env },
|
||||
{ operationLabel: "doctor.heartbeat-task-migration" },
|
||||
);
|
||||
}
|
||||
|
||||
async function clearLegacyTaskTimestamps(params: {
|
||||
storePath: string;
|
||||
sessionKey: string;
|
||||
env: NodeJS.ProcessEnv;
|
||||
tasks: readonly LegacyHeartbeatTask[];
|
||||
}): Promise<void> {
|
||||
await patchSessionEntry(
|
||||
{ storePath: params.storePath, sessionKey: params.sessionKey, env: params.env },
|
||||
(entry) => {
|
||||
const remaining = { ...entry.heartbeatTaskState };
|
||||
let changed = false;
|
||||
for (const task of params.tasks) {
|
||||
if (Object.hasOwn(remaining, task.name)) {
|
||||
delete remaining[task.name];
|
||||
changed = true;
|
||||
}
|
||||
}
|
||||
if (!changed) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
heartbeatTaskState: Object.keys(remaining).length > 0 ? remaining : undefined,
|
||||
};
|
||||
},
|
||||
{ preserveActivity: true },
|
||||
);
|
||||
}
|
||||
|
||||
/** Converts valid scratch tasks and removes their source block in one SQLite transaction. */
|
||||
export async function maybeMigrateHeartbeatTasksToCron(params: {
|
||||
cfg: OpenClawConfig;
|
||||
shouldRepair: boolean;
|
||||
env?: NodeJS.ProcessEnv;
|
||||
nowMs?: number;
|
||||
}): Promise<HeartbeatTaskMigrationResult> {
|
||||
const env = params.env ?? process.env;
|
||||
const nowMs = params.nowMs ?? Date.now();
|
||||
const storePath = resolveCronJobsStorePathFromConfig(params.cfg, env);
|
||||
const changes: string[] = [];
|
||||
const warnings: string[] = [];
|
||||
const candidates: Array<{
|
||||
agent: ReturnType<typeof resolveHeartbeatAgents>[number];
|
||||
document: ReturnType<typeof analyzeLegacyHeartbeatTasks>;
|
||||
monitor: NonNullable<ReturnType<typeof readHeartbeatMonitorScratch>>;
|
||||
scratchRevision: number;
|
||||
validatedTasks: ValidatedHeartbeatTask[];
|
||||
}> = [];
|
||||
for (const agent of resolveHeartbeatAgents(params.cfg)) {
|
||||
let monitor: ReturnType<typeof readHeartbeatMonitorScratch>;
|
||||
try {
|
||||
monitor = readHeartbeatMonitorScratch(storePath, agent.agentId, { env });
|
||||
} catch (error) {
|
||||
warnings.push(
|
||||
`Agent "${agent.agentId}" heartbeat scratch could not be inspected: ${errorMessage(error)}.`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
const scratch = monitor?.state.scratch;
|
||||
if (!monitor || !scratch) {
|
||||
continue;
|
||||
}
|
||||
const document = analyzeLegacyHeartbeatTasks(scratch.content);
|
||||
if (!document.hasTasksBlock) {
|
||||
continue;
|
||||
}
|
||||
const tasks = document.tasks;
|
||||
let validatedTasks: ValidatedHeartbeatTask[];
|
||||
try {
|
||||
validatedTasks = validateTasks(tasks, document.taskEntryCount);
|
||||
} catch (error) {
|
||||
warnings.push(
|
||||
`Agent "${agent.agentId}" heartbeat tasks were not migrated: ${errorMessage(error)}.`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
if (!params.shouldRepair) {
|
||||
note(
|
||||
`${tasks.length} task${tasks.length === 1 ? "" : "s"} in ${shortenHomePath(storePath)} will become independently scheduled cron jobs for agent "${agent.agentId}".`,
|
||||
"Heartbeat task migration preview",
|
||||
);
|
||||
continue;
|
||||
}
|
||||
candidates.push({
|
||||
agent,
|
||||
document,
|
||||
monitor,
|
||||
scratchRevision: scratch.revision,
|
||||
validatedTasks,
|
||||
});
|
||||
}
|
||||
|
||||
if (!params.shouldRepair || candidates.length === 0) {
|
||||
if (warnings.length > 0) {
|
||||
note(warnings.join("\n"), "Doctor warnings");
|
||||
}
|
||||
return { changes, warnings };
|
||||
}
|
||||
|
||||
let snapshot: CronPlanningSnapshot;
|
||||
try {
|
||||
// The scratch revisions above are pinned before this async planning read.
|
||||
// Concurrent doctors can therefore plan R together and serialize at commit.
|
||||
snapshot = await loadCronPlanningSnapshot(storePath, env);
|
||||
} catch (error) {
|
||||
const warning = `Could not inspect cron jobs for heartbeat task migration: ${errorMessage(error)}`;
|
||||
note(warning, "Doctor warnings");
|
||||
return { changes, warnings: [...warnings, warning] };
|
||||
}
|
||||
|
||||
for (const candidate of candidates) {
|
||||
const { agent, document, monitor, scratchRevision, validatedTasks } = candidate;
|
||||
const session = resolveHeartbeatSession(
|
||||
params.cfg,
|
||||
agent.agentId,
|
||||
agent.heartbeat,
|
||||
undefined,
|
||||
env,
|
||||
);
|
||||
const legacyState = session.entry?.heartbeatTaskState ?? {};
|
||||
const jobPlans: TaskJobPlan[] = [];
|
||||
let blocked = false;
|
||||
for (const { task, intervalMs, occurrenceIndex } of validatedTasks) {
|
||||
const declarationKey = heartbeatTaskDeclarationKey(agent.agentId, task.name, occurrenceIndex);
|
||||
const matches = snapshot.jobs.filter((job) => job.declarationKey === declarationKey);
|
||||
const existing = matches[0];
|
||||
if (
|
||||
matches.length > 1 ||
|
||||
(existing &&
|
||||
(!isHeartbeatTaskCronJob(existing) ||
|
||||
existing.agentId !== agent.agentId ||
|
||||
existing.name !== task.name))
|
||||
) {
|
||||
warnings.push(
|
||||
`Agent "${agent.agentId}" task ${JSON.stringify(task.name)} collides with an incompatible cron declaration; scratch was left unchanged.`,
|
||||
);
|
||||
blocked = true;
|
||||
break;
|
||||
}
|
||||
const legacyLastRun = legacyState[task.name];
|
||||
const lastRunAtMs =
|
||||
typeof legacyLastRun === "number" && Number.isFinite(legacyLastRun)
|
||||
? legacyLastRun
|
||||
: undefined;
|
||||
const job = convergeTaskJob({
|
||||
agentId: agent.agentId,
|
||||
task,
|
||||
occurrenceIndex,
|
||||
intervalMs,
|
||||
lastRunAtMs,
|
||||
existing,
|
||||
nowMs,
|
||||
});
|
||||
const sortOrder = reserveSortOrder(snapshot, existing);
|
||||
jobPlans.push({
|
||||
declarationKey,
|
||||
...(existing ? { previous: structuredClone(existing) } : {}),
|
||||
job,
|
||||
sortOrder,
|
||||
});
|
||||
}
|
||||
if (blocked) {
|
||||
continue;
|
||||
}
|
||||
|
||||
try {
|
||||
assertCronStoreCanPersist({ version: 1, jobs: jobPlans.map((plan) => plan.job) });
|
||||
} catch (error) {
|
||||
warnings.push(
|
||||
`Agent "${agent.agentId}" task jobs could not be planned: ${errorMessage(error)}. Scratch was left unchanged.`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
const plan: AgentTaskMigrationPlan = {
|
||||
monitorJobId: monitor.jobId,
|
||||
scratchRevision,
|
||||
strippedContent: document.strippedContent,
|
||||
jobs: jobPlans,
|
||||
};
|
||||
let committed: MigrationCommitResult;
|
||||
try {
|
||||
committed = commitAgentTaskMigration({ storePath, env, nowMs, plan });
|
||||
} catch (error) {
|
||||
warnings.push(
|
||||
`Agent "${agent.agentId}" task migration could not be committed: ${errorMessage(error)}. Scratch and cron jobs were left unchanged.`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
if (!committed.ok) {
|
||||
warnings.push(
|
||||
committed.reason === "revision-conflict"
|
||||
? `Agent "${agent.agentId}" scratch changed during task migration; no changes were committed.`
|
||||
: `Agent "${agent.agentId}" cron jobs changed during task migration; no changes were committed.`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
for (const jobPlan of jobPlans) {
|
||||
const index = snapshot.jobs.findIndex((job) => job.id === jobPlan.job.id);
|
||||
if (index >= 0) {
|
||||
snapshot.jobs[index] = jobPlan.job;
|
||||
} else {
|
||||
snapshot.jobs.push(jobPlan.job);
|
||||
}
|
||||
snapshot.sortOrderByJobId.set(jobPlan.job.id, jobPlan.sortOrder);
|
||||
}
|
||||
changes.push(
|
||||
`Converted ${document.tasks.length} heartbeat task${document.tasks.length === 1 ? "" : "s"} into cron jobs for agent "${agent.agentId}".`,
|
||||
);
|
||||
|
||||
try {
|
||||
// Session task timestamps live in the per-agent database, so they cannot
|
||||
// join the state-DB commit. They are advisory once cron owns scheduling;
|
||||
// this idempotent cleanup may safely be retried or skipped after a crash.
|
||||
await clearLegacyTaskTimestamps({
|
||||
storePath: session.storePath,
|
||||
sessionKey: session.sessionKey,
|
||||
env,
|
||||
tasks: document.tasks,
|
||||
});
|
||||
} catch (error) {
|
||||
warnings.push(
|
||||
`Agent "${agent.agentId}" legacy task timestamps could not be cleared after migration: ${errorMessage(error)}. Cron jobs remain authoritative and a rerun is safe.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if (changes.length > 0) {
|
||||
note(changes.join("\n"), "Doctor changes");
|
||||
}
|
||||
if (warnings.length > 0) {
|
||||
note(warnings.join("\n"), "Doctor warnings");
|
||||
}
|
||||
return { changes, warnings };
|
||||
}
|
||||
@@ -163,6 +163,11 @@ vi.mock("./doctor-heartbeat-scratch-migration.js", () => ({
|
||||
maybeMigrateHeartbeatFilesToScratch: vi.fn().mockResolvedValue({ changes: [], warnings: [] }),
|
||||
}));
|
||||
|
||||
vi.mock("./doctor-heartbeat-task-migration.js", () => ({
|
||||
collectHeartbeatTaskMigrationFindings: vi.fn().mockResolvedValue([]),
|
||||
maybeMigrateHeartbeatTasksToCron: vi.fn().mockResolvedValue({ changes: [], warnings: [] }),
|
||||
}));
|
||||
|
||||
vi.mock("../plugins/provider-openai-chatgpt-oauth-tls.js", () => ({
|
||||
noteOpenAIOAuthTlsPrerequisites: vi.fn().mockResolvedValue(undefined),
|
||||
}));
|
||||
|
||||
195
src/commands/heartbeat-task-legacy.ts
Normal file
195
src/commands/heartbeat-task-legacy.ts
Normal file
@@ -0,0 +1,195 @@
|
||||
/** Doctor-only reader for the retired YAML-like heartbeat `tasks:` syntax. */
|
||||
|
||||
export type LegacyHeartbeatTask = {
|
||||
name: string;
|
||||
interval: string;
|
||||
prompt: string;
|
||||
};
|
||||
|
||||
type LegacyHeartbeatTaskDocument = {
|
||||
hasTasksBlock: boolean;
|
||||
taskEntryCount: number;
|
||||
tasks: LegacyHeartbeatTask[];
|
||||
strippedContent: string;
|
||||
};
|
||||
|
||||
type HeartbeatLine = {
|
||||
htmlCommentSource?: string;
|
||||
raw: string;
|
||||
source: string;
|
||||
visible: string;
|
||||
};
|
||||
|
||||
type LegacyHeartbeatTaskBuilder = Partial<LegacyHeartbeatTask>;
|
||||
|
||||
function splitHeartbeatLines(content: string): Array<{ raw: string; source: string }> {
|
||||
const lines: Array<{ raw: string; source: string }> = [];
|
||||
const matcher = /[^\r\n]*(?:\r\n|\n|\r|$)/g;
|
||||
for (const match of content.matchAll(matcher)) {
|
||||
const source = match[0];
|
||||
if (!source) {
|
||||
continue;
|
||||
}
|
||||
const raw = source.replace(/(?:\r\n|\n|\r)$/, "");
|
||||
lines.push({ raw, source });
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
|
||||
function scanHeartbeatLine(raw: string, state: { inHtmlComment: boolean }) {
|
||||
let cursor = 0;
|
||||
let hasHtmlComment = state.inHtmlComment;
|
||||
let htmlCommentRaw = "";
|
||||
let visible = "";
|
||||
while (cursor < raw.length) {
|
||||
if (state.inHtmlComment) {
|
||||
const commentEnd = raw.indexOf("-->", cursor);
|
||||
if (commentEnd === -1) {
|
||||
htmlCommentRaw += raw.slice(cursor);
|
||||
return { hasHtmlComment, htmlCommentRaw, visible };
|
||||
}
|
||||
htmlCommentRaw += raw.slice(cursor, commentEnd + 3);
|
||||
state.inHtmlComment = false;
|
||||
cursor = commentEnd + 3;
|
||||
continue;
|
||||
}
|
||||
|
||||
const commentStart = raw.indexOf("<!--", cursor);
|
||||
if (commentStart === -1) {
|
||||
const outside = raw.slice(cursor);
|
||||
visible += outside;
|
||||
htmlCommentRaw += outside.replace(/\S/g, "");
|
||||
return { hasHtmlComment, htmlCommentRaw, visible };
|
||||
}
|
||||
const outside = raw.slice(cursor, commentStart);
|
||||
visible += outside;
|
||||
htmlCommentRaw += outside.replace(/\S/g, "") + "<!--";
|
||||
hasHtmlComment = true;
|
||||
state.inHtmlComment = true;
|
||||
cursor = commentStart + 4;
|
||||
}
|
||||
return { hasHtmlComment, htmlCommentRaw, visible };
|
||||
}
|
||||
|
||||
function tokenizeHeartbeatLines(content: string): HeartbeatLine[] {
|
||||
const state = { inHtmlComment: false };
|
||||
return splitHeartbeatLines(content).map((line) => {
|
||||
const scanned = scanHeartbeatLine(line.raw, state);
|
||||
const lineEnding = line.source.slice(line.raw.length);
|
||||
const token: HeartbeatLine = {
|
||||
raw: line.raw,
|
||||
source: line.source,
|
||||
visible: scanned.visible,
|
||||
};
|
||||
if (scanned.hasHtmlComment) {
|
||||
token.htmlCommentSource = scanned.htmlCommentRaw + lineEnding;
|
||||
}
|
||||
return token;
|
||||
});
|
||||
}
|
||||
|
||||
function unquoteTaskValue(value: string): string {
|
||||
return value.trim().replace(/^["']|["']$/g, "");
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses and marks removable task syntax in one pass. The same boundary decision
|
||||
* therefore owns both cron creation and the bytes Doctor may remove.
|
||||
*/
|
||||
export function analyzeLegacyHeartbeatTasks(content: string): LegacyHeartbeatTaskDocument {
|
||||
const lines = tokenizeHeartbeatLines(content);
|
||||
const removedLineIndexes = new Set<number>();
|
||||
const tasks: LegacyHeartbeatTask[] = [];
|
||||
let taskEntryCount = 0;
|
||||
let hasTasksBlock = false;
|
||||
let inTasksBlock = false;
|
||||
let currentTask: LegacyHeartbeatTaskBuilder | undefined;
|
||||
let orphanEntryOpen = false;
|
||||
|
||||
const finishCurrentTask = () => {
|
||||
if (currentTask?.name && currentTask.interval && currentTask.prompt) {
|
||||
tasks.push({
|
||||
name: currentTask.name,
|
||||
interval: currentTask.interval,
|
||||
prompt: currentTask.prompt,
|
||||
});
|
||||
}
|
||||
currentTask = undefined;
|
||||
};
|
||||
|
||||
for (const [index, line] of lines.entries()) {
|
||||
const trimmed = line.visible.trim();
|
||||
|
||||
// Every visible marker starts a new block, including a marker immediately
|
||||
// following another block with no intervening prose.
|
||||
if (trimmed === "tasks:") {
|
||||
finishCurrentTask();
|
||||
orphanEntryOpen = false;
|
||||
hasTasksBlock = true;
|
||||
inTasksBlock = true;
|
||||
removedLineIndexes.add(index);
|
||||
continue;
|
||||
}
|
||||
if (!inTasksBlock) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!trimmed) {
|
||||
// Whitespace belongs to the task block. HTML comments are invisible to
|
||||
// the parser but remain user-authored scratch and must survive migration.
|
||||
if (!line.raw.trim()) {
|
||||
removedLineIndexes.add(index);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (trimmed.startsWith("- name:")) {
|
||||
finishCurrentTask();
|
||||
orphanEntryOpen = false;
|
||||
taskEntryCount += 1;
|
||||
currentTask = { name: unquoteTaskValue(trimmed.slice("- name:".length)) };
|
||||
removedLineIndexes.add(index);
|
||||
continue;
|
||||
}
|
||||
|
||||
const isIndented = line.visible.startsWith(" ") || line.visible.startsWith("\t");
|
||||
if (isIndented && trimmed.startsWith("interval:")) {
|
||||
if (currentTask) {
|
||||
currentTask.interval = unquoteTaskValue(trimmed.slice("interval:".length));
|
||||
} else if (!orphanEntryOpen) {
|
||||
taskEntryCount += 1;
|
||||
orphanEntryOpen = true;
|
||||
}
|
||||
removedLineIndexes.add(index);
|
||||
continue;
|
||||
}
|
||||
if (isIndented && trimmed.startsWith("prompt:")) {
|
||||
if (currentTask) {
|
||||
currentTask.prompt = unquoteTaskValue(trimmed.slice("prompt:".length));
|
||||
} else if (!orphanEntryOpen) {
|
||||
taskEntryCount += 1;
|
||||
orphanEntryOpen = true;
|
||||
}
|
||||
removedLineIndexes.add(index);
|
||||
continue;
|
||||
}
|
||||
|
||||
// Any other visible non-empty line is surrounding scratch, even when it
|
||||
// is indented. It ends the block and is preserved byte-for-byte.
|
||||
finishCurrentTask();
|
||||
orphanEntryOpen = false;
|
||||
inTasksBlock = false;
|
||||
}
|
||||
finishCurrentTask();
|
||||
|
||||
return {
|
||||
hasTasksBlock,
|
||||
taskEntryCount,
|
||||
tasks,
|
||||
strippedContent: lines
|
||||
.map((line, index) =>
|
||||
removedLineIndexes.has(index) ? (line.htmlCommentSource ?? "") : line.source,
|
||||
)
|
||||
.join(""),
|
||||
};
|
||||
}
|
||||
@@ -247,7 +247,7 @@ export type SessionEntry = SessionRestartRecoveryState &
|
||||
* a real user/session-scoped key that merely happens to end with `:heartbeat`.
|
||||
*/
|
||||
heartbeatIsolatedBaseSessionKey?: string;
|
||||
/** Heartbeat task state (task name -> last run timestamp ms). */
|
||||
/** Legacy heartbeat task timestamps consumed and cleared only by doctor migration. */
|
||||
heartbeatTaskState?: Record<string, number>;
|
||||
/** Plugin-owned session state, grouped by plugin id then extension namespace. */
|
||||
pluginExtensions?: Record<string, Record<string, SessionPluginJsonValue>>;
|
||||
|
||||
@@ -313,7 +313,7 @@ export type AgentDefaultsConfig = {
|
||||
to?: string;
|
||||
/** Optional account id for multi-account channels. */
|
||||
accountId?: string;
|
||||
/** Override the heartbeat prompt body (default: "Follow the heartbeat monitor scratch context when provided. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK."). */
|
||||
/** Override the heartbeat prompt body. The default treats scratch as monitor prose and directs recurring work to cron jobs. */
|
||||
prompt?: string;
|
||||
/** Run timeout in seconds for heartbeat agent turns. Unset uses global timeout or heartbeat cadence capped at 600 seconds. */
|
||||
timeoutSeconds?: number;
|
||||
|
||||
34
src/cron/heartbeat-task.ts
Normal file
34
src/cron/heartbeat-task.ts
Normal file
@@ -0,0 +1,34 @@
|
||||
/** Identity and execution metadata for heartbeat tasks migrated into cron. */
|
||||
import { createHash } from "node:crypto";
|
||||
import type { CronJob } from "./types.js";
|
||||
|
||||
const HEARTBEAT_TASK_DECLARATION_PREFIX = "heartbeat-task:";
|
||||
|
||||
/** Stable declaration identity; duplicate names add their deterministic occurrence ordinal. */
|
||||
export function heartbeatTaskDeclarationKey(
|
||||
agentId: string,
|
||||
taskName: string,
|
||||
occurrenceIndex = 0,
|
||||
): string {
|
||||
const hash = createHash("sha256").update(agentId).update("\0").update(taskName);
|
||||
// Keep the first occurrence compatible with the original name-only key so a
|
||||
// doctor rerun can converge a job prepared before duplicate support landed.
|
||||
if (occurrenceIndex > 0) {
|
||||
hash.update("\0").update(String(occurrenceIndex));
|
||||
}
|
||||
const identity = hash.digest("hex").slice(0, 24);
|
||||
return `${HEARTBEAT_TASK_DECLARATION_PREFIX}${agentId}:${identity}`;
|
||||
}
|
||||
|
||||
/** Migrated jobs keep public system-event payloads so cron tools can edit or remove them normally. */
|
||||
export function isHeartbeatTaskCronJob(job: CronJob): job is CronJob & {
|
||||
declarationKey: string;
|
||||
payload: Extract<CronJob["payload"], { kind: "systemEvent" }>;
|
||||
sessionTarget: "main";
|
||||
} {
|
||||
return (
|
||||
job.declarationKey?.startsWith(HEARTBEAT_TASK_DECLARATION_PREFIX) === true &&
|
||||
job.payload.kind === "systemEvent" &&
|
||||
job.sessionTarget === "main"
|
||||
);
|
||||
}
|
||||
@@ -23,7 +23,6 @@ import type { SessionEntry } from "../../config/sessions/types.js";
|
||||
import type { OpenClawConfig } from "../../config/types.openclaw.js";
|
||||
|
||||
const FRESH_CRON_CARRIED_PREFERENCE_FIELDS = [
|
||||
"heartbeatTaskState",
|
||||
"chatType",
|
||||
"thinkingLevel",
|
||||
"fastMode",
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
// The system heartbeat monitor payload replaces the dedicated interval
|
||||
// scheduler: firing it must only poke the heartbeat wake queue.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { heartbeatTaskDeclarationKey } from "./heartbeat-task.js";
|
||||
import {
|
||||
createCronStoreHarness,
|
||||
createNoopLogger,
|
||||
@@ -81,4 +82,43 @@ describe("heartbeat payload execution", () => {
|
||||
await cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
it("routes migrated task jobs through the guarded task wake path", async () => {
|
||||
const { storePath, cleanup } = await makeStorePath();
|
||||
const { cron, enqueueSystemEvent, requestHeartbeat } =
|
||||
createStartedCronServiceWithFinishedBarrier({ storePath, logger: noopLogger });
|
||||
try {
|
||||
await cron.start();
|
||||
const added = await cron.add({
|
||||
declarationKey: heartbeatTaskDeclarationKey("main", "inbox"),
|
||||
name: "inbox",
|
||||
agentId: "main",
|
||||
enabled: true,
|
||||
schedule: { kind: "every", everyMs: 60_000 },
|
||||
payload: { kind: "systemEvent", text: "Check urgent inbox items" },
|
||||
sessionTarget: "main",
|
||||
wakeMode: "next-heartbeat",
|
||||
});
|
||||
const job = "job" in added ? added.job : added;
|
||||
|
||||
await expect(cron.run(job.id, "force")).resolves.toMatchObject({ ok: true });
|
||||
expect(requestHeartbeat).toHaveBeenCalledWith({
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: `heartbeat-task:${job.id}`,
|
||||
agentId: "main",
|
||||
tasks: [{ jobId: job.id, name: "inbox", prompt: "Check urgent inbox items" }],
|
||||
});
|
||||
expect(enqueueSystemEvent).not.toHaveBeenCalled();
|
||||
|
||||
// These stay ordinary cron rows: operators can edit and remove them.
|
||||
await expect(
|
||||
cron.update(job.id, { payload: { kind: "systemEvent", text: "Check priority inbox" } }),
|
||||
).resolves.toMatchObject({ id: job.id });
|
||||
await expect(cron.remove(job.id)).resolves.toEqual({ ok: true, removed: true });
|
||||
} finally {
|
||||
cron.stop();
|
||||
await cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -5,6 +5,7 @@ import {
|
||||
} from "../../infra/heartbeat-wake.js";
|
||||
import type { CommandLaneTaskMarker } from "../../process/command-queue.js";
|
||||
import { type CronActiveJobMarker, isCronActiveJobMarkerCurrent } from "../active-jobs.js";
|
||||
import { isHeartbeatTaskCronJob } from "../heartbeat-task.js";
|
||||
import { createCronRunDiagnosticsFromError } from "../run-diagnostics.js";
|
||||
import { appendCronPayloadText, cronStreamScheduleKey } from "../stream-schedule.js";
|
||||
import type {
|
||||
@@ -168,6 +169,25 @@ export async function executeJobCore(
|
||||
const result = { status: "ok" as const, summary: "heartbeat wake requested" };
|
||||
return triggerEval ? { ...result, triggerEval } : result;
|
||||
}
|
||||
if (isHeartbeatTaskCronJob(effectiveJob)) {
|
||||
// Migrated tasks stay editable public cron jobs, but execution uses the
|
||||
// heartbeat wake bus so active-hours, cooldown, flood, and busy guards remain authoritative.
|
||||
state.deps.requestHeartbeat({
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: `heartbeat-task:${effectiveJob.id}`,
|
||||
agentId: effectiveJob.agentId,
|
||||
tasks: [
|
||||
{
|
||||
jobId: effectiveJob.id,
|
||||
name: effectiveJob.name,
|
||||
prompt: effectiveJob.payload.text,
|
||||
},
|
||||
],
|
||||
});
|
||||
const result = { status: "ok" as const, summary: "heartbeat task wake requested" };
|
||||
return triggerEval ? { ...result, triggerEval } : result;
|
||||
}
|
||||
if (effectiveJob.sessionTarget === "main") {
|
||||
const result = await executeMainSessionCronJob(
|
||||
state,
|
||||
|
||||
@@ -364,6 +364,27 @@ export function replaceCronRows(db: DatabaseSync, storeKey: string, store: CronS
|
||||
}
|
||||
}
|
||||
|
||||
/** Upserts one persisted cron row without rewriting unrelated jobs in its store partition. */
|
||||
export function upsertCronJobRow(
|
||||
db: DatabaseSync,
|
||||
storeKey: string,
|
||||
job: CronJob,
|
||||
sortOrder: number,
|
||||
): void {
|
||||
const normalized = normalizeCronJobForSqlite(job);
|
||||
if (!normalized) {
|
||||
throw new Error(`Cannot persist invalid cron job ${job.id}`);
|
||||
}
|
||||
const values = bindCronJobRow(storeKey, normalized, sortOrder);
|
||||
executeSqliteQuerySync(
|
||||
db,
|
||||
getCronStoreKysely(db)
|
||||
.insertInto("cron_jobs")
|
||||
.values(values)
|
||||
.onConflict((conflict) => conflict.columns(["store_key", "job_id"]).doUpdateSet(values)),
|
||||
);
|
||||
}
|
||||
|
||||
/** Updates only mutable runtime columns without rewriting full job config JSON. */
|
||||
export function updateCronRuntimeRows(
|
||||
db: DatabaseSync,
|
||||
|
||||
@@ -110,6 +110,8 @@ const mocks = vi.hoisted(() => ({
|
||||
maybeMigrateHeartbeatCadenceToCron: vi.fn().mockResolvedValue({ changes: [], warnings: [] }),
|
||||
collectHeartbeatScratchMigrationFindings: vi.fn(async () => [] as unknown[]),
|
||||
maybeMigrateHeartbeatFilesToScratch: vi.fn().mockResolvedValue({ changes: [], warnings: [] }),
|
||||
collectHeartbeatTaskMigrationFindings: vi.fn(async () => [] as unknown[]),
|
||||
maybeMigrateHeartbeatTasksToCron: vi.fn().mockResolvedValue({ changes: [], warnings: [] }),
|
||||
collectWhatsappResponsivenessHealthFindings: vi.fn((): readonly HealthFinding[] => []),
|
||||
noteWhatsappResponsivenessHealth: vi.fn().mockResolvedValue(undefined),
|
||||
collectDevicePairingHealthFindings: vi.fn(async () => []),
|
||||
@@ -417,6 +419,11 @@ vi.mock("../commands/doctor-heartbeat-scratch-migration.js", () => ({
|
||||
maybeMigrateHeartbeatFilesToScratch: mocks.maybeMigrateHeartbeatFilesToScratch,
|
||||
}));
|
||||
|
||||
vi.mock("../commands/doctor-heartbeat-task-migration.js", () => ({
|
||||
collectHeartbeatTaskMigrationFindings: mocks.collectHeartbeatTaskMigrationFindings,
|
||||
maybeMigrateHeartbeatTasksToCron: mocks.maybeMigrateHeartbeatTasksToCron,
|
||||
}));
|
||||
|
||||
vi.mock("../commands/doctor-whatsapp-responsiveness.js", () => ({
|
||||
collectWhatsappResponsivenessHealthFindings: mocks.collectWhatsappResponsivenessHealthFindings,
|
||||
noteWhatsappResponsivenessHealth: mocks.noteWhatsappResponsivenessHealth,
|
||||
@@ -690,6 +697,10 @@ describe("doctor health contributions", () => {
|
||||
mocks.collectHeartbeatScratchMigrationFindings.mockResolvedValue([]);
|
||||
mocks.maybeMigrateHeartbeatFilesToScratch.mockReset();
|
||||
mocks.maybeMigrateHeartbeatFilesToScratch.mockResolvedValue({ changes: [], warnings: [] });
|
||||
mocks.collectHeartbeatTaskMigrationFindings.mockReset();
|
||||
mocks.collectHeartbeatTaskMigrationFindings.mockResolvedValue([]);
|
||||
mocks.maybeMigrateHeartbeatTasksToCron.mockReset();
|
||||
mocks.maybeMigrateHeartbeatTasksToCron.mockResolvedValue({ changes: [], warnings: [] });
|
||||
mocks.collectWhatsappResponsivenessHealthFindings.mockReset();
|
||||
mocks.collectWhatsappResponsivenessHealthFindings.mockReturnValue([]);
|
||||
mocks.noteWhatsappResponsivenessHealth.mockReset();
|
||||
@@ -1379,6 +1390,38 @@ describe("doctor health contributions", () => {
|
||||
expect(mocks.collectHeartbeatCadenceMigrationFindings).toHaveBeenCalledWith(cfg, env);
|
||||
});
|
||||
|
||||
it("migrates heartbeat files before converting their task blocks", () => {
|
||||
const ids = resolveDoctorHealthContributions().map((entry) => entry.id);
|
||||
const cadenceIndex = ids.indexOf("doctor:heartbeat-cadence-migration");
|
||||
const scratchIndex = ids.indexOf("doctor:heartbeat-scratch-migration");
|
||||
const taskIndex = ids.indexOf("doctor:heartbeat-task-cron-migration");
|
||||
|
||||
expect(cadenceIndex).toBeGreaterThan(-1);
|
||||
expect(scratchIndex).toBeGreaterThan(cadenceIndex);
|
||||
expect(scratchIndex).toBeGreaterThan(-1);
|
||||
expect(taskIndex).toBeGreaterThan(scratchIndex);
|
||||
expect(taskIndex).toBeLessThan(ids.indexOf("doctor:write-config"));
|
||||
});
|
||||
|
||||
it("forwards the health-check environment to heartbeat task detection", async () => {
|
||||
const checks = await resolveDoctorContributionHealthChecks();
|
||||
const check = checks.find(
|
||||
(candidate) => candidate.id === "core/doctor/heartbeat-task-cron-migration",
|
||||
);
|
||||
expect(check).toBeDefined();
|
||||
const cfg = { agents: { defaults: { heartbeat: { every: "15m" } } } };
|
||||
const env = { OPENCLAW_STATE_DIR: "/tmp/openclaw-task-detector-state" };
|
||||
|
||||
await check!.detect({
|
||||
mode: "lint",
|
||||
cfg,
|
||||
env,
|
||||
runtime: { log: vi.fn(), error: vi.fn(), exit: vi.fn() },
|
||||
});
|
||||
|
||||
expect(mocks.collectHeartbeatTaskMigrationFindings).toHaveBeenCalledWith(cfg, env);
|
||||
});
|
||||
|
||||
it("keeps heartbeat template lint opt-in for default lint selection", async () => {
|
||||
const contributionChecks = await resolveDoctorContributionHealthChecks();
|
||||
const heartbeatTemplateCheck = contributionChecks.find(
|
||||
|
||||
@@ -1156,6 +1156,16 @@ async function runHeartbeatScratchMigrationHealth(ctx: DoctorHealthFlowContext):
|
||||
});
|
||||
}
|
||||
|
||||
async function runHeartbeatTaskMigrationHealth(ctx: DoctorHealthFlowContext): Promise<void> {
|
||||
const { maybeMigrateHeartbeatTasksToCron } =
|
||||
await import("../commands/doctor-heartbeat-task-migration.js");
|
||||
await maybeMigrateHeartbeatTasksToCron({
|
||||
cfg: ctx.cfg,
|
||||
shouldRepair: ctx.prompter.shouldRepair,
|
||||
env: ctx.env,
|
||||
});
|
||||
}
|
||||
|
||||
async function runShellCompletionHealth(ctx: DoctorHealthFlowContext): Promise<void> {
|
||||
const { doctorShellCompletion } = await import("../commands/doctor-completion.js");
|
||||
await doctorShellCompletion(ctx.runtime, ctx.prompter, {
|
||||
@@ -2247,6 +2257,21 @@ function resolveDoctorHealthContributions(): DoctorHealthContribution[] {
|
||||
},
|
||||
run: runHeartbeatScratchMigrationHealth,
|
||||
}),
|
||||
createDoctorHealthContribution({
|
||||
id: "doctor:heartbeat-task-cron-migration",
|
||||
label: "Heartbeat task cron migration",
|
||||
healthChecks: {
|
||||
id: "core/doctor/heartbeat-task-cron-migration",
|
||||
description: "Heartbeat scratch task blocks must migrate into cron jobs.",
|
||||
defaultEnabled: true,
|
||||
async detect(ctx) {
|
||||
const { collectHeartbeatTaskMigrationFindings } =
|
||||
await import("../commands/doctor-heartbeat-task-migration.js");
|
||||
return await collectHeartbeatTaskMigrationFindings(ctx.cfg, ctx.env);
|
||||
},
|
||||
},
|
||||
run: runHeartbeatTaskMigrationHealth,
|
||||
}),
|
||||
createDoctorHealthContribution({
|
||||
id: "doctor:shell-completion",
|
||||
label: "Shell completion",
|
||||
|
||||
@@ -662,6 +662,7 @@ export function buildGatewayCronService(params: {
|
||||
...(opts?.scheduledAnchorMs !== undefined
|
||||
? { scheduledAnchorMs: opts.scheduledAnchorMs }
|
||||
: {}),
|
||||
...(opts.tasks?.length ? { tasks: opts.tasks } : {}),
|
||||
});
|
||||
},
|
||||
runHeartbeatOnce: async (opts) => {
|
||||
|
||||
@@ -111,6 +111,7 @@ describe("shouldDeferWake", () => {
|
||||
expect(decide({ ...afterRun, source: "acp-spawn", reason: "acp:spawn:stream" })).toEqual({
|
||||
defer: true,
|
||||
reason: "not-due",
|
||||
retryAtMs: 79_000,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -132,7 +133,7 @@ describe("shouldDeferWake", () => {
|
||||
recentRunStarts,
|
||||
reason: "wake",
|
||||
}),
|
||||
).toEqual({ defer: true, reason: "flood" });
|
||||
).toEqual({ defer: true, reason: "flood", retryAtMs: 1_010_001 });
|
||||
});
|
||||
|
||||
it("flood guard still applies to 'background-task' as a backstop", () => {
|
||||
@@ -153,7 +154,7 @@ describe("shouldDeferWake", () => {
|
||||
recentRunStarts,
|
||||
reason: "background-task",
|
||||
}),
|
||||
).toEqual({ defer: true, reason: "flood" });
|
||||
).toEqual({ defer: true, reason: "flood", retryAtMs: 1_010_001 });
|
||||
});
|
||||
|
||||
it("flood guard still applies to explicit wake-now bypass calls", () => {
|
||||
@@ -174,7 +175,7 @@ describe("shouldDeferWake", () => {
|
||||
recentRunStarts,
|
||||
reason: "hook:wake",
|
||||
}),
|
||||
).toEqual({ defer: true, reason: "flood" });
|
||||
).toEqual({ defer: true, reason: "flood", retryAtMs: 1_010_001 });
|
||||
});
|
||||
});
|
||||
|
||||
@@ -183,6 +184,7 @@ describe("shouldDeferWake", () => {
|
||||
expect(decide({ ...afterRun, intent: "scheduled", reason: "interval" })).toEqual({
|
||||
defer: true,
|
||||
reason: "not-due",
|
||||
retryAtMs: 100_000,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -190,6 +192,7 @@ describe("shouldDeferWake", () => {
|
||||
expect(decide({ ...beforeFirstRun, intent: "scheduled", reason: "interval" })).toEqual({
|
||||
defer: true,
|
||||
reason: "not-due",
|
||||
retryAtMs: 100_000,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -206,11 +209,49 @@ describe("shouldDeferWake", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("independently scheduled task intent", () => {
|
||||
it("ignores the base heartbeat due slot but keeps the minimum spacing guard", () => {
|
||||
expect(
|
||||
decide({
|
||||
...afterRun,
|
||||
intent: "task",
|
||||
now: 80_000,
|
||||
lastRunStartedAtMs: 40_000,
|
||||
reason: "heartbeat-task:inbox",
|
||||
}),
|
||||
).toEqual({ defer: false });
|
||||
expect(
|
||||
decide({
|
||||
...afterRun,
|
||||
intent: "task",
|
||||
now: 80_000,
|
||||
lastRunStartedAtMs: 79_000,
|
||||
reason: "heartbeat-task:inbox",
|
||||
}),
|
||||
).toEqual({ defer: true, reason: "min-spacing", retryAtMs: 109_000 });
|
||||
});
|
||||
|
||||
it("keeps the flood guard", () => {
|
||||
const now = 1_000_000;
|
||||
expect(
|
||||
decide({
|
||||
intent: "task",
|
||||
now,
|
||||
nextDueMs: now + 60_000,
|
||||
lastRunStartedAtMs: now - 40_000,
|
||||
recentRunStarts: [now - 50_000, now - 40_000, now - 30_000, now - 20_000, now - 10_000],
|
||||
reason: "heartbeat-task:inbox",
|
||||
}),
|
||||
).toEqual({ defer: true, reason: "flood", retryAtMs: 1_010_001 });
|
||||
});
|
||||
});
|
||||
|
||||
describe("event-driven wakes after a prior run (regression for #75436)", () => {
|
||||
it("defers exec-event wakes when now < nextDueMs", () => {
|
||||
expect(decide({ ...afterRun, source: "exec-event", reason: "exec-event" })).toEqual({
|
||||
defer: true,
|
||||
reason: "not-due",
|
||||
retryAtMs: 79_000,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -218,6 +259,7 @@ describe("shouldDeferWake", () => {
|
||||
expect(decide({ ...afterRun, source: "cron", reason: "cron:morning-brief" })).toEqual({
|
||||
defer: true,
|
||||
reason: "not-due",
|
||||
retryAtMs: 79_000,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -225,6 +267,7 @@ describe("shouldDeferWake", () => {
|
||||
expect(decide({ ...afterRun, source: "hook", reason: "hook:wake" })).toEqual({
|
||||
defer: true,
|
||||
reason: "not-due",
|
||||
retryAtMs: 79_000,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -232,6 +275,7 @@ describe("shouldDeferWake", () => {
|
||||
expect(decide({ ...afterRun, source: "acp-spawn", reason: "acp:spawn:stream" })).toEqual({
|
||||
defer: true,
|
||||
reason: "not-due",
|
||||
retryAtMs: 79_000,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -239,6 +283,7 @@ describe("shouldDeferWake", () => {
|
||||
expect(decide({ ...afterRun, source: "other", reason: "something-new" })).toEqual({
|
||||
defer: true,
|
||||
reason: "not-due",
|
||||
retryAtMs: 79_000,
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -263,6 +308,18 @@ describe("shouldDeferWake", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("admits retained event work after the spacing floor even before nextDueMs", () => {
|
||||
expect(
|
||||
decide({
|
||||
...afterRun,
|
||||
now: 80_000,
|
||||
retainedWork: true,
|
||||
source: "exec-event",
|
||||
reason: "exec-event",
|
||||
}),
|
||||
).toEqual({ defer: false });
|
||||
});
|
||||
|
||||
describe("min-spacing floor", () => {
|
||||
it("defers recent runs at the default spacing floor", () => {
|
||||
expect(
|
||||
@@ -273,7 +330,7 @@ describe("shouldDeferWake", () => {
|
||||
lastRunStartedAtMs: 170_100,
|
||||
reason: "exec-event",
|
||||
}),
|
||||
).toEqual({ defer: true, reason: "min-spacing" });
|
||||
).toEqual({ defer: true, reason: "min-spacing", retryAtMs: 200_100 });
|
||||
expect(
|
||||
decide({
|
||||
source: "exec-event",
|
||||
@@ -295,7 +352,7 @@ describe("shouldDeferWake", () => {
|
||||
minSpacingMs: 1_000,
|
||||
reason: "exec-event",
|
||||
}),
|
||||
).toEqual({ defer: true, reason: "min-spacing" });
|
||||
).toEqual({ defer: true, reason: "min-spacing", retryAtMs: 200_500 });
|
||||
});
|
||||
|
||||
it("does not gate manual wakes on min-spacing", () => {
|
||||
@@ -323,7 +380,7 @@ describe("shouldDeferWake", () => {
|
||||
recentRunStarts: [now - 50_000, now - 40_000, now - 30_000, now - 20_000, now - 10_000],
|
||||
reason: "exec-event",
|
||||
}),
|
||||
).toEqual({ defer: true, reason: "flood" });
|
||||
).toEqual({ defer: true, reason: "flood", retryAtMs: 1_010_001 });
|
||||
expect(
|
||||
decide({
|
||||
source: "exec-event",
|
||||
|
||||
@@ -28,7 +28,12 @@ const DEFAULT_FLOOD_THRESHOLD = 5;
|
||||
|
||||
export type DeferDecision =
|
||||
| { defer: false }
|
||||
| { defer: true; reason: "not-due" | "min-spacing" | "flood" };
|
||||
| {
|
||||
defer: true;
|
||||
reason: "not-due" | "min-spacing" | "flood";
|
||||
/** First wall-clock instant at which this guard can admit the wake. */
|
||||
retryAtMs: number;
|
||||
};
|
||||
|
||||
type ShouldDeferInput = {
|
||||
/** Scheduler behavior requested by the wake producer. */
|
||||
@@ -51,6 +56,8 @@ type ShouldDeferInput = {
|
||||
floodWindowMs?: number;
|
||||
/** Override the flood-window threshold. */
|
||||
floodThreshold?: number;
|
||||
/** Work already retained by the wake queue after a prior guard deferral. */
|
||||
retainedWork?: boolean;
|
||||
};
|
||||
|
||||
/**
|
||||
@@ -63,6 +70,7 @@ type ShouldDeferInput = {
|
||||
* | manual | Run | Run (never deferred) |
|
||||
* | immediate | Run | Run (never deferred, except flood) |
|
||||
* | scheduled | Defer if now < nextDueMs | Defer if now < nextDueMs |
|
||||
* | task | Run | Defer only within floor or on flood |
|
||||
* | event | Run (bootstrap responsive) | Defer if now < nextDueMs OR within floor |
|
||||
*
|
||||
* Immediate is for documented wake-now delivery paths such as `openclaw system
|
||||
@@ -96,6 +104,18 @@ export function shouldDeferWake(input: ShouldDeferInput): DeferDecision {
|
||||
return floodDefer ?? { defer: false };
|
||||
}
|
||||
|
||||
if (input.intent === "task") {
|
||||
const floodDefer = checkFloodGuard(input);
|
||||
if (floodDefer) {
|
||||
return floodDefer;
|
||||
}
|
||||
const spacingRetryAtMs = resolveMinSpacingRetryAtMs(input);
|
||||
if (spacingRetryAtMs !== undefined) {
|
||||
return { defer: true, reason: "min-spacing", retryAtMs: spacingRetryAtMs };
|
||||
}
|
||||
return { defer: false };
|
||||
}
|
||||
|
||||
// Flood guard applies to every non-immediate wake regardless of run history.
|
||||
// It is the last line of defense against feedback loops.
|
||||
const floodDefer = checkFloodGuard(input);
|
||||
@@ -104,7 +124,9 @@ export function shouldDeferWake(input: ShouldDeferInput): DeferDecision {
|
||||
}
|
||||
|
||||
if (input.intent === "scheduled") {
|
||||
return input.now < input.nextDueMs ? { defer: true, reason: "not-due" } : { defer: false };
|
||||
return input.now < input.nextDueMs
|
||||
? { defer: true, reason: "not-due", retryAtMs: input.nextDueMs }
|
||||
: { defer: false };
|
||||
}
|
||||
|
||||
// Event-driven wakes. First wake (no prior run) bypasses cooldown gates so
|
||||
@@ -114,18 +136,32 @@ export function shouldDeferWake(input: ShouldDeferInput): DeferDecision {
|
||||
return { defer: false };
|
||||
}
|
||||
|
||||
if (input.now < input.nextDueMs) {
|
||||
return { defer: true, reason: "not-due" };
|
||||
if (!input.retainedWork && input.now < input.nextDueMs) {
|
||||
const spacingRetryAtMs = resolveMinSpacingRetryAtMs(input);
|
||||
return {
|
||||
defer: true,
|
||||
reason: "not-due",
|
||||
retryAtMs: Math.min(input.nextDueMs, spacingRetryAtMs ?? input.nextDueMs),
|
||||
};
|
||||
}
|
||||
|
||||
const minSpacing = input.minSpacingMs ?? DEFAULT_MIN_WAKE_SPACING_MS;
|
||||
if (minSpacing > 0 && input.now - input.lastRunStartedAtMs < minSpacing) {
|
||||
return { defer: true, reason: "min-spacing" };
|
||||
const spacingRetryAtMs = resolveMinSpacingRetryAtMs(input);
|
||||
if (spacingRetryAtMs !== undefined) {
|
||||
return { defer: true, reason: "min-spacing", retryAtMs: spacingRetryAtMs };
|
||||
}
|
||||
|
||||
return { defer: false };
|
||||
}
|
||||
|
||||
function resolveMinSpacingRetryAtMs(input: ShouldDeferInput): number | undefined {
|
||||
const minSpacing = input.minSpacingMs ?? DEFAULT_MIN_WAKE_SPACING_MS;
|
||||
if (minSpacing <= 0 || input.lastRunStartedAtMs === undefined) {
|
||||
return undefined;
|
||||
}
|
||||
const retryAtMs = input.lastRunStartedAtMs + minSpacing;
|
||||
return input.now < retryAtMs ? retryAtMs : undefined;
|
||||
}
|
||||
|
||||
function checkFloodGuard(input: ShouldDeferInput): DeferDecision | null {
|
||||
const floodWindow = input.floodWindowMs ?? DEFAULT_FLOOD_WINDOW_MS;
|
||||
const floodThreshold = input.floodThreshold ?? DEFAULT_FLOOD_THRESHOLD;
|
||||
@@ -134,14 +170,20 @@ function checkFloodGuard(input: ShouldDeferInput): DeferDecision | null {
|
||||
}
|
||||
const windowStart = input.now - floodWindow;
|
||||
let inWindow = 0;
|
||||
let thresholdOldestTs: number | undefined;
|
||||
for (let i = input.recentRunStarts.length - 1; i >= 0; i--) {
|
||||
const ts = input.recentRunStarts[i];
|
||||
if (ts === undefined || ts < windowStart) {
|
||||
break;
|
||||
}
|
||||
inWindow += 1;
|
||||
if (inWindow === floodThreshold) {
|
||||
thresholdOldestTs = ts;
|
||||
}
|
||||
}
|
||||
return inWindow >= floodThreshold ? { defer: true, reason: "flood" } : null;
|
||||
return inWindow >= floodThreshold && thresholdOldestTs !== undefined
|
||||
? { defer: true, reason: "flood", retryAtMs: thresholdOldestTs + floodWindow + 1 }
|
||||
: null;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -271,7 +271,7 @@ describe("runHeartbeatOnce ack handling", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("records completed tasks when HEARTBEAT_OK delivery fails", async () => {
|
||||
it("does not recreate legacy task timestamps when HEARTBEAT_OK delivery fails", async () => {
|
||||
await withTempTelegramHeartbeatSandbox(async ({ tmpDir, storePath, replySpy }) => {
|
||||
const nowMs = Date.parse("2026-07-06T12:00:00.000Z");
|
||||
await seedHeartbeatScratchForTest({
|
||||
@@ -314,13 +314,11 @@ describe("runHeartbeatOnce ack handling", () => {
|
||||
}>(storePath);
|
||||
expect(result.status).toBe("ran");
|
||||
expect(sendTelegram).toHaveBeenCalledTimes(1);
|
||||
expect(sessionStore[sessionKey]?.heartbeatTaskState).toEqual({
|
||||
"check-deployment": nowMs,
|
||||
});
|
||||
expect(sessionStore[sessionKey]?.heartbeatTaskState).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
it("records completed tasks when HEARTBEAT_OK readiness checks fail", async () => {
|
||||
it("does not recreate legacy task timestamps when readiness checks fail", async () => {
|
||||
await withTempHeartbeatSandbox(async ({ tmpDir, storePath, replySpy }) => {
|
||||
const nowMs = Date.parse("2026-07-06T12:00:00.000Z");
|
||||
await seedHeartbeatScratchForTest({
|
||||
@@ -363,9 +361,7 @@ describe("runHeartbeatOnce ack handling", () => {
|
||||
}>(storePath);
|
||||
expect(result.status).toBe("ran");
|
||||
expect(sendWhatsApp).not.toHaveBeenCalled();
|
||||
expect(sessionStore[sessionKey]?.heartbeatTaskState).toEqual({
|
||||
"check-deployment": nowMs,
|
||||
});
|
||||
expect(sessionStore[sessionKey]?.heartbeatTaskState).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -211,8 +211,8 @@ describe("runHeartbeatOnce commitments", () => {
|
||||
});
|
||||
}
|
||||
|
||||
it("keeps free-form reasons from changing normal heartbeat task priority", async () => {
|
||||
const { result, sendTelegram, sessionStore, store } = await withTempHeartbeatSandbox(
|
||||
it("keeps free-form reasons from changing cron-carried heartbeat task priority", async () => {
|
||||
const { result, sendTelegram, store } = await withTempHeartbeatSandbox(
|
||||
async ({ tmpDir, storePath, replySpy }) => {
|
||||
setTestEnvValue("OPENCLAW_STATE_DIR", tmpDir);
|
||||
const sessionKey = "agent:main:telegram:user-155462274";
|
||||
@@ -229,13 +229,6 @@ describe("runHeartbeatOnce commitments", () => {
|
||||
channels: { telegram: { allowFrom: ["*"] } },
|
||||
session: { store: storePath },
|
||||
};
|
||||
await seedHeartbeatScratchForTest({
|
||||
content: `tasks:
|
||||
- name: deployment-status
|
||||
interval: 5m
|
||||
prompt: Check deployment status with the normal tools
|
||||
`,
|
||||
});
|
||||
await seedSessionStore(storePath, sessionKey, {
|
||||
lastChannel: "telegram",
|
||||
lastProvider: "telegram",
|
||||
@@ -270,6 +263,15 @@ describe("runHeartbeatOnce commitments", () => {
|
||||
cfg,
|
||||
agentId: "main",
|
||||
reason: "commitment",
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
tasks: [
|
||||
{
|
||||
jobId: "job-deployment-status",
|
||||
name: "deployment-status",
|
||||
prompt: "Check deployment status with the normal tools",
|
||||
},
|
||||
],
|
||||
sessionKey,
|
||||
deps: {
|
||||
getReplyFromConfig: replySpy,
|
||||
@@ -282,9 +284,6 @@ describe("runHeartbeatOnce commitments", () => {
|
||||
return {
|
||||
result: resultResult,
|
||||
sendTelegram: sendTelegramResult,
|
||||
sessionStore: readSessionStoreForTest<{
|
||||
heartbeatTaskState?: Record<string, number>;
|
||||
}>(storePath),
|
||||
store: await loadCommitmentStore(),
|
||||
};
|
||||
},
|
||||
@@ -292,9 +291,6 @@ describe("runHeartbeatOnce commitments", () => {
|
||||
|
||||
expect(result.status).toBe("ran");
|
||||
expect(sendTelegram).toHaveBeenCalled();
|
||||
expect(sessionStore["agent:main:telegram:user-155462274"]?.heartbeatTaskState).toEqual({
|
||||
"deployment-status": nowMs,
|
||||
});
|
||||
expectCommitmentFields(store.commitments[0], {
|
||||
id: "cm_interview",
|
||||
status: "pending",
|
||||
@@ -692,7 +688,7 @@ describe("runHeartbeatOnce commitments", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("appends scratch directives to commitment prompt when tasks are configured but none are due", async () => {
|
||||
it("appends scratch directives to the commitment prompt", async () => {
|
||||
const { result, sendTelegram, store } = await withTempHeartbeatSandbox(
|
||||
async ({ tmpDir, storePath, replySpy }) => {
|
||||
setTestEnvValue("OPENCLAW_STATE_DIR", tmpDir);
|
||||
@@ -710,24 +706,15 @@ describe("runHeartbeatOnce commitments", () => {
|
||||
channels: { telegram: { allowFrom: ["*"] } },
|
||||
session: { store: storePath },
|
||||
};
|
||||
// Scratch has a tasks block (task ran recently — NOT due) plus extra prose directives.
|
||||
await seedHeartbeatScratchForTest({
|
||||
content: `Do not contact the user unless critical.
|
||||
|
||||
tasks:
|
||||
- name: check-deployment
|
||||
interval: 5m
|
||||
prompt: Check deployment status
|
||||
`,
|
||||
content: "Do not contact the user unless critical.\n",
|
||||
});
|
||||
// Seed heartbeatTaskState so the task ran at nowMs (well within 5m interval, not due).
|
||||
await seedSessionStore(storePath, sessionKey, {
|
||||
sessionId: "sid",
|
||||
updatedAt: nowMs,
|
||||
lastChannel: "telegram",
|
||||
lastProvider: "telegram",
|
||||
lastTo: "155462274",
|
||||
heartbeatTaskState: { "check-deployment": nowMs },
|
||||
});
|
||||
await saveCommitmentStore(undefined, {
|
||||
version: 1,
|
||||
@@ -743,10 +730,8 @@ tasks:
|
||||
// Must contain commitment text
|
||||
expect(ctx.Body).toContain("Due inferred follow-up commitments");
|
||||
expect(ctx.Body).toContain("How did the interview go?");
|
||||
// Must also contain scratch directives outside the tasks block
|
||||
// Must also contain the monitor scratch directive.
|
||||
expect(ctx.Body).toContain("Do not contact the user unless critical.");
|
||||
// Must NOT contain the task prompt (task is not due)
|
||||
expect(ctx.Body).not.toContain("Check deployment status");
|
||||
return { text: "How did the interview go?" };
|
||||
},
|
||||
);
|
||||
|
||||
@@ -369,7 +369,7 @@ describe("runHeartbeatOnce – isolated session key stability (#59493)", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("does not create an isolated session when task-based heartbeat skips for no-tasks-due", async () => {
|
||||
it("treats leftover tasks text as ordinary scratch instead of runtime scheduling state", async () => {
|
||||
await withTempHeartbeatSandbox(async ({ tmpDir, storePath }) => {
|
||||
const cfg: OpenClawConfig = {
|
||||
agents: {
|
||||
@@ -386,6 +386,7 @@ describe("runHeartbeatOnce – isolated session key stability (#59493)", () => {
|
||||
};
|
||||
const baseSessionKey = resolveMainSessionKey(cfg);
|
||||
const isolatedSessionKey = `${baseSessionKey}:heartbeat`;
|
||||
const nowMs = Date.now();
|
||||
await seedHeartbeatScratchForTest({
|
||||
content: `tasks:
|
||||
- name: daily-check
|
||||
@@ -396,7 +397,7 @@ describe("runHeartbeatOnce – isolated session key stability (#59493)", () => {
|
||||
|
||||
await seedSessionStore(storePath, baseSessionKey, {
|
||||
sessionId: "sid",
|
||||
updatedAt: Date.now(),
|
||||
updatedAt: nowMs,
|
||||
lastChannel: "whatsapp",
|
||||
lastProvider: "whatsapp",
|
||||
lastTo: "+1555",
|
||||
@@ -412,15 +413,49 @@ describe("runHeartbeatOnce – isolated session key stability (#59493)", () => {
|
||||
sessionKey: baseSessionKey,
|
||||
deps: {
|
||||
getQueueSize: () => 0,
|
||||
nowMs: () => 2,
|
||||
nowMs: () => nowMs,
|
||||
},
|
||||
});
|
||||
|
||||
expect(result).toEqual({ status: "skipped", reason: "no-tasks-due" });
|
||||
expect(replySpy).not.toHaveBeenCalled();
|
||||
expect(result.status).toBe("ran");
|
||||
expect(replySpy).toHaveBeenCalledOnce();
|
||||
expect(replyCall(replySpy).Body).toContain("Heartbeat monitor scratch:\ntasks:");
|
||||
expect(replyCall(replySpy).Body).not.toContain(
|
||||
"Run the following periodic tasks (only those due based on their intervals)",
|
||||
);
|
||||
|
||||
const store = readSessionStoreForTest(storePath);
|
||||
expect(store[isolatedSessionKey]).toBeUndefined();
|
||||
expect(store[isolatedSessionKey]).toBeDefined();
|
||||
expect(store[baseSessionKey]?.heartbeatTaskState).toEqual({ "daily-check": 1 });
|
||||
});
|
||||
});
|
||||
|
||||
it("renders cron-carried task prompts through the heartbeat response path", async () => {
|
||||
await withTempHeartbeatSandbox(async ({ tmpDir, storePath }) => {
|
||||
const cfg = makeIsolatedHeartbeatConfig(tmpDir, storePath);
|
||||
const baseSessionKey = resolveMainSessionKey(cfg);
|
||||
await seedSessionStore(storePath, baseSessionKey, {
|
||||
sessionId: "sid",
|
||||
updatedAt: Date.now(),
|
||||
lastChannel: "whatsapp",
|
||||
lastProvider: "whatsapp",
|
||||
lastTo: "+1555",
|
||||
});
|
||||
const replySpy = vi.spyOn(replyModule, "getReplyFromConfig");
|
||||
replySpy.mockResolvedValue({ text: "HEARTBEAT_OK" });
|
||||
|
||||
const result = await runHeartbeatOnce({
|
||||
cfg,
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: "heartbeat-task:job-inbox",
|
||||
tasks: [{ jobId: "job-inbox", name: "inbox", prompt: "Check urgent inbox items" }],
|
||||
deps: { getQueueSize: () => 0, nowMs: () => Date.now() },
|
||||
});
|
||||
|
||||
expect(result.status).toBe("ran");
|
||||
expect(replyCall(replySpy).Body).toContain("- inbox: Check urgent inbox items");
|
||||
expect(replyCall(replySpy).Body).toContain("After completing all due tasks");
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -62,18 +62,14 @@ function makeIsolatedLastTargetConfig(tmpDir: string, storePath: string): OpenCl
|
||||
}
|
||||
|
||||
describe("runHeartbeatOnce - isolated heartbeat outbound session mirror", () => {
|
||||
it("uses the isolated run key for outbound delivery while base session owns target and state", async () => {
|
||||
it("uses the isolated run key for outbound delivery while the base session owns delivery state", async () => {
|
||||
await withTempHeartbeatSandbox(async ({ tmpDir, storePath, replySpy }) => {
|
||||
const cfg = makeIsolatedLastTargetConfig(tmpDir, storePath);
|
||||
const baseSessionKey = resolveMainSessionKey(cfg);
|
||||
const isolatedSessionKey = `${baseSessionKey}:heartbeat`;
|
||||
const nowMs = Date.now();
|
||||
await seedHeartbeatScratchForTest({
|
||||
content: `tasks:
|
||||
- name: check-in
|
||||
interval: 5m
|
||||
prompt: "Check whether the user needs a status update."
|
||||
`,
|
||||
content: "Check whether the user needs a status update.",
|
||||
});
|
||||
await seedSessionStore(storePath, baseSessionKey, {
|
||||
sessionId: "base-session",
|
||||
@@ -108,20 +104,17 @@ describe("runHeartbeatOnce - isolated heartbeat outbound session mirror", () =>
|
||||
});
|
||||
|
||||
const store = readSessionStoreForTest<{
|
||||
heartbeatTaskState?: Record<string, number>;
|
||||
lastHeartbeatText?: string;
|
||||
lastHeartbeatSentAt?: number;
|
||||
heartbeatIsolatedBaseSessionKey?: string;
|
||||
}>(storePath);
|
||||
expect(store[baseSessionKey]).toMatchObject({
|
||||
heartbeatTaskState: { "check-in": nowMs },
|
||||
lastHeartbeatText: "Status needs attention.",
|
||||
lastHeartbeatSentAt: nowMs,
|
||||
});
|
||||
expect(store[isolatedSessionKey]).toMatchObject({
|
||||
heartbeatIsolatedBaseSessionKey: baseSessionKey,
|
||||
});
|
||||
expect(store[isolatedSessionKey]?.heartbeatTaskState).toBeUndefined();
|
||||
expect(store[isolatedSessionKey]?.lastHeartbeatText).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -816,6 +816,31 @@ describe("runHeartbeatOnce", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps active-hours protection for cron-carried heartbeat tasks", async () => {
|
||||
const cfg: OpenClawConfig = {
|
||||
agents: {
|
||||
defaults: {
|
||||
userTimezone: "UTC",
|
||||
heartbeat: {
|
||||
every: "30m",
|
||||
activeHours: { start: "08:00", end: "24:00", timezone: "user" },
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
await expect(
|
||||
runHeartbeatOnce({
|
||||
cfg,
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: "heartbeat-task:job-inbox",
|
||||
tasks: [{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" }],
|
||||
deps: { nowMs: () => Date.UTC(2025, 0, 1, 7, 0, 0) },
|
||||
}),
|
||||
).resolves.toEqual({ status: "skipped", reason: "quiet-hours" });
|
||||
});
|
||||
|
||||
it("uses the last non-empty payload for delivery", async () => {
|
||||
const tmpDir = await createCaseDir("hb-last-payload");
|
||||
const storePath = path.join(tmpDir, "sessions.json");
|
||||
@@ -1601,7 +1626,7 @@ describe("runHeartbeatOnce", () => {
|
||||
expect(replyBody(replySpy).Body).toContain("Check the custom cron partition");
|
||||
});
|
||||
|
||||
it("keeps non-task scratch context while stripping blank-line-separated task blocks", async () => {
|
||||
it("treats blank-line-separated legacy task blocks as ordinary scratch", async () => {
|
||||
const tmpDir = await createCaseDir("openclaw-hb-tasks-context");
|
||||
const storePath = path.join(tmpDir, "sessions.json");
|
||||
const workspaceDir = path.join(tmpDir, "workspace");
|
||||
@@ -1652,19 +1677,18 @@ Some global directive after tasks.
|
||||
expect(res.status).toBe("ran");
|
||||
expect(replySpy).toHaveBeenCalledTimes(1);
|
||||
const calledCtx = replyBody(replySpy);
|
||||
expect(calledCtx.Body).toContain("- inbox: Check urgent inbox items");
|
||||
expect(calledCtx.Body).toContain("- calendar: Check calendar changes");
|
||||
expect(calledCtx.Body).not.toContain("Run the following periodic tasks");
|
||||
expect(calledCtx.Body).toContain("Heartbeat monitor scratch");
|
||||
expect(calledCtx.Body).toContain("# Keep this header");
|
||||
expect(calledCtx.Body).toContain("Remember escalation policy.");
|
||||
expect(calledCtx.Body).toContain("Some global directive after tasks.");
|
||||
expect(calledCtx.Body).toContain("- Keep this top-level directive too.");
|
||||
expect(calledCtx.Body).not.toContain("name: inbox");
|
||||
expect(calledCtx.Body).not.toContain("name: calendar");
|
||||
expect(calledCtx.Body).toContain("name: inbox");
|
||||
expect(calledCtx.Body).toContain("name: calendar");
|
||||
replySpy.mockReset();
|
||||
});
|
||||
|
||||
it("strips documented unindented task entries while keeping following top-level bullets", async () => {
|
||||
it("keeps unindented legacy task entries as ordinary scratch", async () => {
|
||||
const tmpDir = await createCaseDir("openclaw-hb-unindented-tasks-context");
|
||||
const storePath = path.join(tmpDir, "sessions.json");
|
||||
const workspaceDir = path.join(tmpDir, "workspace");
|
||||
@@ -1711,15 +1735,14 @@ tasks:
|
||||
expect(res.status).toBe("ran");
|
||||
expect(replySpy).toHaveBeenCalledTimes(1);
|
||||
const calledCtx = replyBody(replySpy);
|
||||
expect(calledCtx.Body).toContain("- inbox: Check urgent inbox items");
|
||||
expect(calledCtx.Body).toContain("- calendar: Check calendar changes");
|
||||
expect(calledCtx.Body).not.toContain("Run the following periodic tasks");
|
||||
expect(calledCtx.Body).toContain("Heartbeat monitor scratch");
|
||||
expect(calledCtx.Body).toContain("# Keep this header");
|
||||
expect(calledCtx.Body).toContain("- Keep this top-level directive after tasks.");
|
||||
expect(calledCtx.Body).not.toContain("name: inbox");
|
||||
expect(calledCtx.Body).not.toContain("name: calendar");
|
||||
expect(calledCtx.Body).not.toContain("interval: 5m");
|
||||
expect(calledCtx.Body).not.toContain("prompt: Check urgent");
|
||||
expect(calledCtx.Body).toContain("name: inbox");
|
||||
expect(calledCtx.Body).toContain("name: calendar");
|
||||
expect(calledCtx.Body).toContain("interval: 5m");
|
||||
expect(calledCtx.Body).toContain("prompt: Check urgent");
|
||||
replySpy.mockReset();
|
||||
});
|
||||
|
||||
|
||||
@@ -277,8 +277,52 @@ describe("startHeartbeatRunner", () => {
|
||||
expect(runSpy).toHaveBeenCalledTimes(1);
|
||||
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
expect(runSpy).toHaveBeenCalledTimes(2);
|
||||
runner.stop();
|
||||
});
|
||||
|
||||
it("keeps persisted monitor cadence authoritative when its tick joins a task turn", async () => {
|
||||
useFakeHeartbeatTime();
|
||||
const runSpy = vi.fn().mockResolvedValue({ status: "ran", durationMs: 1 });
|
||||
const runner = startDefaultRunner(runSpy);
|
||||
const monitorAnchorMs = resolveHeartbeatPhaseMs({
|
||||
schedulerSeed: TEST_SCHEDULER_SEED,
|
||||
agentId: "main",
|
||||
intervalMs: 5 * 60_000,
|
||||
});
|
||||
|
||||
requestHeartbeat({
|
||||
source: "interval",
|
||||
intent: "scheduled",
|
||||
reason: "interval",
|
||||
agentId: "main",
|
||||
scheduledEveryMs: 5 * 60_000,
|
||||
scheduledAnchorMs: monitorAnchorMs,
|
||||
coalesceMs: 100,
|
||||
});
|
||||
requestHeartbeat({
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: "heartbeat-task:job-inbox",
|
||||
agentId: "main",
|
||||
tasks: [{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" }],
|
||||
coalesceMs: 100,
|
||||
});
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
|
||||
expect(runSpy).toHaveBeenCalledTimes(1);
|
||||
expectRunCallFields(runSpy, 0, {
|
||||
intent: "task",
|
||||
tasks: [{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" }],
|
||||
});
|
||||
expect((getRunCall(runSpy, 0).heartbeat as { every?: string }).every).toBe("300000ms");
|
||||
|
||||
await vi.advanceTimersByTimeAsync(4 * 60_000);
|
||||
requestHeartbeat(wake("exec-event", { agentId: "main", coalesceMs: 0 }));
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect(runSpy).toHaveBeenCalledTimes(1);
|
||||
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
expect(runSpy).toHaveBeenCalledTimes(2);
|
||||
runner.stop();
|
||||
});
|
||||
@@ -764,8 +808,8 @@ describe("startHeartbeatRunner", () => {
|
||||
expect(runSpy).toHaveBeenCalledTimes(1);
|
||||
|
||||
// Simulate the runaway: 4 more exec-event wakes from backgrounded process
|
||||
// exits, fired well within the configured 30m interval. These should be
|
||||
// debounced by the cooldown — the agent just ran, nothing has changed.
|
||||
// exits, fired well within the configured 30m interval. They coalesce into
|
||||
// one retained turn after the 30s floor instead of running every 10s.
|
||||
for (let i = 0; i < 4; i++) {
|
||||
await vi.advanceTimersByTimeAsync(10_000); // 10s between background exits
|
||||
requestHeartbeat({
|
||||
@@ -778,10 +822,57 @@ describe("startHeartbeatRunner", () => {
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
}
|
||||
|
||||
// Total elapsed: ~40s. Configured `every` is 30m. Subsequent exec-events
|
||||
// should NOT trigger fresh runs within the cooldown window.
|
||||
expect(runSpy).toHaveBeenCalledTimes(1);
|
||||
// Total elapsed: ~40s. The queued events produced one follow-up at the
|
||||
// spacing boundary; they did not bypass the floor or wait for the 30m tick.
|
||||
expect(runSpy).toHaveBeenCalledTimes(2);
|
||||
|
||||
// Settle the final retained batch so this module-level wake queue is empty
|
||||
// before the next runner lifecycle starts.
|
||||
await vi.advanceTimersByTimeAsync(20_000);
|
||||
expect(runSpy).toHaveBeenCalledTimes(3);
|
||||
|
||||
runner.stop();
|
||||
});
|
||||
|
||||
it("retains an event that collides with a task until the spacing floor", async () => {
|
||||
useFakeHeartbeatTime();
|
||||
const runSpy = vi.fn().mockResolvedValue({ status: "ran", durationMs: 1 });
|
||||
const runner = startHeartbeatRunner({
|
||||
cfg: heartbeatConfig(),
|
||||
runOnce: runSpy,
|
||||
stableSchedulerSeed: TEST_SCHEDULER_SEED,
|
||||
});
|
||||
|
||||
requestHeartbeat({
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: "heartbeat-task:job-inbox",
|
||||
agentId: "main",
|
||||
tasks: [{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" }],
|
||||
coalesceMs: 0,
|
||||
});
|
||||
requestHeartbeat({
|
||||
source: "exec-event",
|
||||
intent: "event",
|
||||
reason: "exec-event",
|
||||
agentId: "main",
|
||||
coalesceMs: 0,
|
||||
});
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
|
||||
expect(runSpy).toHaveBeenCalledTimes(1);
|
||||
expectRunCallFields(runSpy, 0, {
|
||||
intent: "task",
|
||||
tasks: [{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" }],
|
||||
});
|
||||
|
||||
await vi.advanceTimersByTimeAsync(29_998);
|
||||
expect(runSpy).toHaveBeenCalledTimes(1);
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
|
||||
expect(runSpy).toHaveBeenCalledTimes(2);
|
||||
expectRunCallFields(runSpy, 1, { intent: "event", reason: "exec-event" });
|
||||
expect(getRunCall(runSpy, 1).tasks).toEqual([]);
|
||||
runner.stop();
|
||||
});
|
||||
|
||||
|
||||
@@ -231,6 +231,7 @@ describe("runHeartbeatOnce heartbeat response tool", () => {
|
||||
storePath: string;
|
||||
cfg: OpenClawConfig;
|
||||
}) => Promise<void>;
|
||||
tasks?: Parameters<typeof runHeartbeatOnce>[0]["tasks"];
|
||||
} = {},
|
||||
) {
|
||||
return await withTempTelegramHeartbeatSandbox(async ({ tmpDir, storePath, replySpy }) => {
|
||||
@@ -253,6 +254,7 @@ describe("runHeartbeatOnce heartbeat response tool", () => {
|
||||
|
||||
await runHeartbeatOnce({
|
||||
cfg,
|
||||
tasks: params.tasks,
|
||||
deps: createDeps({ sendTelegram, getReplyFromConfig: replySpy }),
|
||||
});
|
||||
|
||||
@@ -445,15 +447,8 @@ describe("runHeartbeatOnce heartbeat response tool", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("delivers a terminal tool warning without recording successful heartbeat bookkeeping", async () => {
|
||||
it("delivers a terminal tool warning without recording successful delivery bookkeeping", async () => {
|
||||
await withTempTelegramHeartbeatSandbox(async ({ tmpDir, storePath, replySpy }) => {
|
||||
await seedHeartbeatScratchForTest({
|
||||
content: `tasks:
|
||||
- name: check-delivery
|
||||
interval: 1m
|
||||
prompt: Check delivery
|
||||
`,
|
||||
});
|
||||
const cfg = createConfig({ tmpDir, storePath });
|
||||
const sessionKey = await seedMainSessionStore(storePath, cfg, {
|
||||
lastChannel: "telegram",
|
||||
@@ -478,13 +473,11 @@ describe("runHeartbeatOnce heartbeat response tool", () => {
|
||||
deps: createDeps({ sendTelegram, getReplyFromConfig: replySpy }),
|
||||
});
|
||||
const sessionStore = readSessionStoreForTest<{
|
||||
heartbeatTaskState?: Record<string, number>;
|
||||
lastHeartbeatText?: string;
|
||||
}>(storePath);
|
||||
|
||||
expect(result).toEqual({ status: "failed", reason: "agent-tool-failure" });
|
||||
expectTelegramSend(sendTelegram, { text: warning, cfg });
|
||||
expect(sessionStore[sessionKey]?.heartbeatTaskState).toBeUndefined();
|
||||
expect(sessionStore[sessionKey]?.lastHeartbeatText).toBeUndefined();
|
||||
expect(getLastHeartbeatEvent()).toMatchObject({
|
||||
status: "failed",
|
||||
@@ -882,15 +875,7 @@ describe("runHeartbeatOnce heartbeat response tool", () => {
|
||||
it("uses the heartbeat response tool prompt for due heartbeat tasks", async () => {
|
||||
const result = await runPromptScenario({
|
||||
config: { visibleReplies: "message_tool" },
|
||||
beforeSeed: async () => {
|
||||
await seedHeartbeatScratchForTest({
|
||||
content: `tasks:
|
||||
- name: status
|
||||
interval: 1m
|
||||
prompt: Check deployment status
|
||||
`,
|
||||
});
|
||||
},
|
||||
tasks: [{ jobId: "job-status", name: "status", prompt: "Check deployment status" }],
|
||||
});
|
||||
|
||||
expectHeartbeatToolPrompt(result, [
|
||||
|
||||
@@ -36,12 +36,9 @@ import {
|
||||
import {
|
||||
DEFAULT_HEARTBEAT_ACK_MAX_CHARS,
|
||||
isHeartbeatContentEffectivelyEmpty,
|
||||
isTaskDue,
|
||||
parseHeartbeatTasks,
|
||||
resolveHeartbeatPrompt as resolveHeartbeatPromptText,
|
||||
resolveHeartbeatPromptForResponseTool,
|
||||
stripHeartbeatToken,
|
||||
type HeartbeatTask,
|
||||
} from "../auto-reply/heartbeat.js";
|
||||
import { copyReplyPayloadMetadata } from "../auto-reply/reply-payload.js";
|
||||
import { replaceGenericExternalRunFailureText } from "../auto-reply/reply/agent-runner-failure-copy.js";
|
||||
@@ -167,6 +164,7 @@ import {
|
||||
type HeartbeatWakeHandler,
|
||||
type HeartbeatWakeIntent,
|
||||
type HeartbeatWakeRequest,
|
||||
type HeartbeatScheduledTask,
|
||||
type HeartbeatWakeSource,
|
||||
isRetryableHeartbeatBusySkipReason,
|
||||
setHeartbeatsEnabled,
|
||||
@@ -569,11 +567,12 @@ function resolveHeartbeatTypingIntervalSeconds(cfg: OpenClawConfig, agentId: str
|
||||
return typeof configured === "number" && configured > 0 ? configured : undefined;
|
||||
}
|
||||
|
||||
function resolveHeartbeatSession(
|
||||
export function resolveHeartbeatSession(
|
||||
cfg: OpenClawConfig,
|
||||
agentId?: string,
|
||||
heartbeat?: HeartbeatConfig,
|
||||
forcedSessionKey?: string,
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
) {
|
||||
const sessionCfg = cfg.session;
|
||||
const scope = sessionCfg?.scope ?? "per-sender";
|
||||
@@ -584,8 +583,9 @@ function resolveHeartbeatSession(
|
||||
// A literal `global` row is global only inside the selected agent's store.
|
||||
// Falling back here leaks the default agent's route into secondary heartbeats.
|
||||
agentId: resolvedAgentId,
|
||||
env,
|
||||
});
|
||||
const mainEntry = loadSessionEntry({ storePath, sessionKey: mainSessionKey });
|
||||
const mainEntry = loadSessionEntry({ storePath, sessionKey: mainSessionKey, env });
|
||||
|
||||
if (scope === "global") {
|
||||
return {
|
||||
@@ -631,7 +631,7 @@ function resolveHeartbeatSession(
|
||||
return {
|
||||
sessionKey: routedSessionKey,
|
||||
storePath,
|
||||
entry: loadSessionEntry({ storePath, sessionKey: routedSessionKey }),
|
||||
entry: loadSessionEntry({ storePath, sessionKey: routedSessionKey, env }),
|
||||
suppressOriginatingContext: false,
|
||||
};
|
||||
}
|
||||
@@ -683,7 +683,7 @@ function resolveHeartbeatSession(
|
||||
return {
|
||||
sessionKey: canonical,
|
||||
storePath,
|
||||
entry: loadSessionEntry({ storePath, sessionKey: canonical }),
|
||||
entry: loadSessionEntry({ storePath, sessionKey: canonical, env }),
|
||||
suppressOriginatingContext: false,
|
||||
};
|
||||
}
|
||||
@@ -864,7 +864,6 @@ type HeartbeatPreflight = HeartbeatWakePayloadFlags & {
|
||||
hasTaggedCronEvents: boolean;
|
||||
shouldInspectPendingEvents: boolean;
|
||||
skipReason?: HeartbeatSkipReason;
|
||||
tasks?: HeartbeatTask[];
|
||||
scratchJobId?: string;
|
||||
scratchRevision?: number;
|
||||
heartbeatScratchContent?: string;
|
||||
@@ -878,6 +877,7 @@ async function resolveHeartbeatPreflight(params: {
|
||||
forcedSessionKey?: string;
|
||||
reason?: string;
|
||||
source?: HeartbeatWakeSource;
|
||||
scheduledTasks?: readonly HeartbeatScheduledTask[];
|
||||
nowMs?: number;
|
||||
}): Promise<HeartbeatPreflight> {
|
||||
const wakeFlags = resolveHeartbeatWakePayloadFlags({
|
||||
@@ -988,27 +988,23 @@ async function resolveHeartbeatPreflight(params: {
|
||||
if (shouldBypassFileGates) {
|
||||
return basePreflight;
|
||||
}
|
||||
// Cron owns task due-ness. Task wakes still receive ordinary scratch prose,
|
||||
// but empty or missing scratch must never suppress the independently scheduled job.
|
||||
if (params.scheduledTasks?.length) {
|
||||
return basePreflight;
|
||||
}
|
||||
if (heartbeatScratchContent === undefined) {
|
||||
// No scratch row preserves the old missing-file behavior: the model still
|
||||
// gets the generic heartbeat prompt and decides whether anything is due.
|
||||
return basePreflight;
|
||||
}
|
||||
const tasks = parseHeartbeatTasks(heartbeatScratchContent);
|
||||
if (
|
||||
isHeartbeatContentEffectivelyEmpty(heartbeatScratchContent) &&
|
||||
tasks.length === 0 &&
|
||||
dueCommitments.length === 0
|
||||
) {
|
||||
if (isHeartbeatContentEffectivelyEmpty(heartbeatScratchContent) && dueCommitments.length === 0) {
|
||||
return {
|
||||
...basePreflight,
|
||||
skipReason: "empty-heartbeat-file",
|
||||
tasks: [],
|
||||
};
|
||||
}
|
||||
return {
|
||||
...basePreflight,
|
||||
tasks,
|
||||
};
|
||||
return basePreflight;
|
||||
}
|
||||
|
||||
type HeartbeatPromptResolution = {
|
||||
@@ -1020,58 +1016,12 @@ type HeartbeatPromptResolution = {
|
||||
usesHeartbeatResponseTool: boolean;
|
||||
};
|
||||
|
||||
function resolveDueHeartbeatTasks(
|
||||
preflight: Pick<HeartbeatPreflight, "session" | "tasks">,
|
||||
startedAt: number,
|
||||
): HeartbeatTask[] {
|
||||
const tasks = preflight.tasks;
|
||||
if (!tasks || tasks.length === 0) {
|
||||
return [];
|
||||
}
|
||||
return tasks.filter((task) =>
|
||||
isTaskDue(
|
||||
(preflight.session.entry?.heartbeatTaskState as Record<string, number>)?.[task.name],
|
||||
task.interval,
|
||||
startedAt,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
function stripHeartbeatTasksBlock(content: string): string {
|
||||
const lines = content.split(/\r?\n/);
|
||||
const kept: string[] = [];
|
||||
let inTasksBlock = false;
|
||||
|
||||
for (const line of lines) {
|
||||
const trimmed = line.trim();
|
||||
if (!inTasksBlock && trimmed === "tasks:") {
|
||||
inTasksBlock = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (inTasksBlock) {
|
||||
if (!trimmed) {
|
||||
continue;
|
||||
}
|
||||
const isIndented = /^[\s]/.test(line);
|
||||
if (isIndented || trimmed.startsWith("- name:")) {
|
||||
continue;
|
||||
}
|
||||
inTasksBlock = false;
|
||||
}
|
||||
|
||||
kept.push(line);
|
||||
}
|
||||
|
||||
return kept.join("\n");
|
||||
}
|
||||
|
||||
/** Appends monitor scratch prose outside the optional `tasks:` block. */
|
||||
/** Appends monitor scratch prose to the generated heartbeat prompt. */
|
||||
function appendHeartbeatScratch(prompt: string, heartbeatScratchContent?: string): string {
|
||||
if (!heartbeatScratchContent) {
|
||||
return prompt;
|
||||
}
|
||||
const directives = stripHeartbeatTasksBlock(heartbeatScratchContent).trim();
|
||||
const directives = heartbeatScratchContent.trim();
|
||||
if (!directives || prompt.includes(directives)) {
|
||||
return prompt;
|
||||
}
|
||||
@@ -1084,7 +1034,7 @@ function resolveHeartbeatRunPrompt(params: {
|
||||
preflight: HeartbeatPreflight;
|
||||
canRelayToUser: boolean;
|
||||
startedAt: number;
|
||||
dueTasks: HeartbeatTask[];
|
||||
scheduledTasks: readonly HeartbeatScheduledTask[];
|
||||
heartbeatScratchContent?: string;
|
||||
useHeartbeatResponseTool: boolean;
|
||||
runScope: HeartbeatRunScope;
|
||||
@@ -1132,46 +1082,26 @@ function resolveHeartbeatRunPrompt(params: {
|
||||
};
|
||||
}
|
||||
|
||||
if (params.preflight.tasks && params.preflight.tasks.length > 0) {
|
||||
const dueTasks = params.dueTasks;
|
||||
|
||||
if (dueTasks.length > 0) {
|
||||
const taskList = dueTasks.map((task) => `- ${task.name}: ${task.prompt}`).join("\n");
|
||||
const completionInstruction = params.useHeartbeatResponseTool
|
||||
? "After completing all due tasks, use heartbeat_respond to report the outcome. Set notify=false when nothing needs the user's attention."
|
||||
: "After completing all due tasks, reply HEARTBEAT_OK.";
|
||||
const taskListPrompt = `Run the following periodic tasks (only those due based on their intervals):
|
||||
if (params.scheduledTasks.length > 0) {
|
||||
const taskList = params.scheduledTasks
|
||||
.map((task) => `- ${task.name}: ${task.prompt}`)
|
||||
.join("\n");
|
||||
const completionInstruction = params.useHeartbeatResponseTool
|
||||
? "After completing all due tasks, use heartbeat_respond to report the outcome. Set notify=false when nothing needs the user's attention."
|
||||
: "After completing all due tasks, reply HEARTBEAT_OK.";
|
||||
const taskPrompt = `Run the following periodic tasks (only those due based on their intervals):
|
||||
|
||||
${taskList}
|
||||
|
||||
${completionInstruction}`;
|
||||
const prompt = appendHeartbeatScratch(taskListPrompt, params.heartbeatScratchContent);
|
||||
return {
|
||||
prompt,
|
||||
hasExecCompletion: false,
|
||||
hasRelayableExecCompletion: false,
|
||||
hasCronEvents: false,
|
||||
hasDueCommitments: false,
|
||||
usesHeartbeatResponseTool: params.useHeartbeatResponseTool,
|
||||
};
|
||||
}
|
||||
if (commitmentPrompt) {
|
||||
return {
|
||||
prompt: appendHeartbeatScratch(commitmentPrompt, params.heartbeatScratchContent),
|
||||
hasExecCompletion: false,
|
||||
hasRelayableExecCompletion: false,
|
||||
hasCronEvents: false,
|
||||
hasDueCommitments,
|
||||
usesHeartbeatResponseTool: false,
|
||||
};
|
||||
}
|
||||
const prompt = appendHeartbeatScratch(taskPrompt, params.heartbeatScratchContent);
|
||||
return {
|
||||
prompt: null,
|
||||
prompt,
|
||||
hasExecCompletion: false,
|
||||
hasRelayableExecCompletion: false,
|
||||
hasCronEvents: false,
|
||||
hasDueCommitments: false,
|
||||
usesHeartbeatResponseTool: false,
|
||||
usesHeartbeatResponseTool: params.useHeartbeatResponseTool,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1264,6 +1194,7 @@ export async function runHeartbeatOnce(opts: {
|
||||
intent?: HeartbeatWakeIntent;
|
||||
reason?: string;
|
||||
runScope?: HeartbeatRunScope;
|
||||
tasks?: readonly HeartbeatScheduledTask[];
|
||||
/** Exact cron run marker whose own activity must not block this wake. */
|
||||
owningCronJobMarker?: CronActiveJobMarker;
|
||||
owningCronLaneTaskMarker?: CommandLaneTaskMarker;
|
||||
@@ -1285,6 +1216,10 @@ export async function runHeartbeatOnce(opts: {
|
||||
mergeRequestedHeartbeat: wakeSource === "cron",
|
||||
});
|
||||
const runScope = opts.runScope ?? "global";
|
||||
const scheduledTasks =
|
||||
runScope === "commitment-only"
|
||||
? []
|
||||
: [...(opts.tasks ?? [])].toSorted((left, right) => left.jobId.localeCompare(right.jobId));
|
||||
const allowsUnscheduledTarget =
|
||||
isTargetedImmediateSystemEventWake(opts) && isConfiguredHeartbeatAgent(cfg, agentId);
|
||||
if (!areHeartbeatsEnabled()) {
|
||||
@@ -1386,7 +1321,7 @@ export async function runHeartbeatOnce(opts: {
|
||||
return { status: "skipped", reason: HEARTBEAT_SKIP_REQUESTS_IN_FLIGHT };
|
||||
}
|
||||
|
||||
// Preflight centralizes trigger classification, event inspection, and HEARTBEAT.md gating.
|
||||
// Preflight centralizes trigger classification, event inspection, and monitor-scratch gating.
|
||||
const preflight = await resolveHeartbeatPreflight({
|
||||
cfg,
|
||||
agentId,
|
||||
@@ -1395,6 +1330,7 @@ export async function runHeartbeatOnce(opts: {
|
||||
forcedSessionKey: opts.sessionKey,
|
||||
source: wakeSource,
|
||||
reason: opts.reason,
|
||||
scheduledTasks,
|
||||
nowMs: startedAt,
|
||||
});
|
||||
if (preflight.skipReason) {
|
||||
@@ -1431,8 +1367,6 @@ export async function runHeartbeatOnce(opts: {
|
||||
}
|
||||
|
||||
const previousUpdatedAt = entry?.updatedAt;
|
||||
const dueHeartbeatTasks =
|
||||
runScope === "commitment-only" ? [] : resolveDueHeartbeatTasks(preflight, startedAt);
|
||||
|
||||
// When isolatedSession is enabled, create a fresh session via the same
|
||||
// pattern as cron sessionTarget: "isolated". This gives the heartbeat
|
||||
@@ -1441,7 +1375,7 @@ export async function runHeartbeatOnce(opts: {
|
||||
// Delivery routing still uses the main session entry (lastChannel, lastTo).
|
||||
const useIsolatedSession = heartbeat?.isolatedSession === true;
|
||||
const firstDueCommitment =
|
||||
canHeartbeatDeliverCommitments(heartbeat) && dueHeartbeatTasks.length === 0
|
||||
canHeartbeatDeliverCommitments(heartbeat) && scheduledTasks.length === 0
|
||||
? preflight.dueCommitments[0]
|
||||
: undefined;
|
||||
const commitmentDeliveryContext = firstDueCommitment
|
||||
@@ -1517,13 +1451,12 @@ export async function runHeartbeatOnce(opts: {
|
||||
preflight,
|
||||
canRelayToUser,
|
||||
startedAt,
|
||||
dueTasks: dueHeartbeatTasks,
|
||||
scheduledTasks,
|
||||
heartbeatScratchContent: preflight.heartbeatScratchContent,
|
||||
useHeartbeatResponseTool: useHeartbeatResponseToolPrompt,
|
||||
runScope,
|
||||
});
|
||||
|
||||
// If no tasks are due, skip heartbeat entirely
|
||||
if (heartbeatRunPrompt.prompt === null) {
|
||||
// Wake-triggered events should stay queued when the run short-circuits:
|
||||
// no reply turn ran, so there is nothing that actually consumed that wake payload.
|
||||
@@ -1537,7 +1470,7 @@ export async function runHeartbeatOnce(opts: {
|
||||
if (shouldConsumeInspectedEvents && inspectedSystemEventsToConsume.length > 0) {
|
||||
consumeSelectedSystemEventEntries(sessionKey, inspectedSystemEventsToConsume);
|
||||
}
|
||||
return { status: "skipped", reason: "no-tasks-due" };
|
||||
return { status: "skipped", reason: "not-due" };
|
||||
}
|
||||
let runSessionKey = sessionKey;
|
||||
let runSessionEntry = entry;
|
||||
@@ -1640,7 +1573,7 @@ export async function runHeartbeatOnce(opts: {
|
||||
preflight,
|
||||
canRelayToUser,
|
||||
startedAt,
|
||||
dueTasks: dueHeartbeatTasks,
|
||||
scheduledTasks,
|
||||
heartbeatScratchContent: preflight.heartbeatScratchContent,
|
||||
useHeartbeatResponseTool: useHeartbeatResponseToolPrompt,
|
||||
runScope,
|
||||
@@ -1656,7 +1589,7 @@ export async function runHeartbeatOnce(opts: {
|
||||
} = heartbeatRunPrompt;
|
||||
const prompt = heartbeatRunPrompt.prompt;
|
||||
if (prompt === null) {
|
||||
return { status: "skipped", reason: "no-tasks-due" };
|
||||
return { status: "skipped", reason: "not-due" };
|
||||
}
|
||||
const dueCommitmentIds = hasDueCommitments
|
||||
? preflight.dueCommitments.map((commitment) => commitment.id)
|
||||
@@ -1666,38 +1599,6 @@ export async function runHeartbeatOnce(opts: {
|
||||
hasExecCompletion,
|
||||
hasCronEvents,
|
||||
});
|
||||
// Update task last run times AFTER successful heartbeat completion
|
||||
const updateTaskTimestamps = async () => {
|
||||
if (!preflight.tasks || preflight.tasks.length === 0 || dueHeartbeatTasks.length === 0) {
|
||||
return;
|
||||
}
|
||||
const tasks = preflight.tasks;
|
||||
const dueTaskNames = new Set(dueHeartbeatTasks.map((task) => task.name));
|
||||
|
||||
await patchSessionEntry(
|
||||
{ storePath, sessionKey },
|
||||
(base) => {
|
||||
const taskState = { ...base.heartbeatTaskState };
|
||||
|
||||
for (const task of tasks) {
|
||||
if (dueTaskNames.has(task.name)) {
|
||||
taskState[task.name] = startedAt;
|
||||
}
|
||||
}
|
||||
|
||||
return { heartbeatTaskState: taskState };
|
||||
},
|
||||
{
|
||||
fallbackEntry: {
|
||||
sessionId: sessionKey.replace(/:/g, "_"),
|
||||
updatedAt: startedAt,
|
||||
heartbeatTaskState: {},
|
||||
},
|
||||
preserveActivity: true,
|
||||
},
|
||||
);
|
||||
};
|
||||
|
||||
// The duplicate-suppression branch returns before any send, so it never hits
|
||||
// the send-success clear. A duplicate means this run's own output was already
|
||||
// delivered within the dedupe window, so this run's pending-final is satisfied
|
||||
@@ -1932,7 +1833,7 @@ export async function runHeartbeatOnce(opts: {
|
||||
storePath,
|
||||
runSessionKey,
|
||||
response: heartbeatToolResponse,
|
||||
taskNames: dueHeartbeatTasks.map((task) => task.name),
|
||||
taskNames: scheduledTasks.map((task) => task.name),
|
||||
wakeSource,
|
||||
wakeReason: opts.reason,
|
||||
occurredAt: startedAt,
|
||||
@@ -1960,7 +1861,6 @@ export async function runHeartbeatOnce(opts: {
|
||||
status: "dismissed",
|
||||
nowMs: startedAt,
|
||||
});
|
||||
await updateTaskTimestamps();
|
||||
consumeInspectedSystemEvents();
|
||||
return { status: "ran", durationMs: Date.now() - startedAt };
|
||||
}
|
||||
@@ -1996,7 +1896,6 @@ export async function runHeartbeatOnce(opts: {
|
||||
status: "dismissed",
|
||||
nowMs: startedAt,
|
||||
});
|
||||
await updateTaskTimestamps();
|
||||
consumeInspectedSystemEvents();
|
||||
return { status: "ran", durationMs: Date.now() - startedAt };
|
||||
}
|
||||
@@ -2149,7 +2048,6 @@ export async function runHeartbeatOnce(opts: {
|
||||
status: "dismissed",
|
||||
nowMs: startedAt,
|
||||
});
|
||||
await updateTaskTimestamps();
|
||||
consumeInspectedSystemEvents();
|
||||
return { status: "ran", durationMs: Date.now() - startedAt };
|
||||
}
|
||||
@@ -2196,7 +2094,6 @@ export async function runHeartbeatOnce(opts: {
|
||||
status: "dismissed",
|
||||
nowMs: startedAt,
|
||||
});
|
||||
await updateTaskTimestamps();
|
||||
consumeInspectedSystemEvents();
|
||||
return { status: "ran", durationMs: Date.now() - startedAt };
|
||||
}
|
||||
@@ -2218,13 +2115,11 @@ export async function runHeartbeatOnce(opts: {
|
||||
hasMedia: mediaUrls.length > 0,
|
||||
accountId: delivery.accountId,
|
||||
});
|
||||
await updateTaskTimestamps();
|
||||
consumeInspectedSystemEvents();
|
||||
return { status: "ran", durationMs: Date.now() - startedAt };
|
||||
}
|
||||
|
||||
if (!visibility.showAlerts) {
|
||||
await updateTaskTimestamps();
|
||||
await restoreHeartbeatUpdatedAt({
|
||||
storePath,
|
||||
sessionKey,
|
||||
@@ -2350,7 +2245,6 @@ export async function runHeartbeatOnce(opts: {
|
||||
...(normalized.silent === true ? { silent: true } : {}),
|
||||
indicatorType: visibility.useIndicator ? resolveIndicatorType(eventStatus) : undefined,
|
||||
});
|
||||
await updateTaskTimestamps();
|
||||
consumeInspectedSystemEvents();
|
||||
return { status: "ran", durationMs: Date.now() - startedAt };
|
||||
} catch (err) {
|
||||
@@ -2466,15 +2360,16 @@ export function startHeartbeatRunner(opts: {
|
||||
now: number,
|
||||
reason?: string,
|
||||
intent: HeartbeatWakeIntent = "event",
|
||||
authoritativeScheduledTick = false,
|
||||
options: { authoritativeScheduledTick?: boolean; retainedWork?: boolean } = {},
|
||||
): DeferDecision => {
|
||||
const decision = shouldDeferWake({
|
||||
intent,
|
||||
reason,
|
||||
now,
|
||||
nextDueMs: authoritativeScheduledTick ? now : agent.nextDueMs,
|
||||
nextDueMs: options.authoritativeScheduledTick ? now : agent.nextDueMs,
|
||||
lastRunStartedAtMs: agent.lastRunStartedAtMs,
|
||||
recentRunStarts: agent.recentRunStarts,
|
||||
retainedWork: options.retainedWork,
|
||||
});
|
||||
if (decision.defer && decision.reason === "flood") {
|
||||
if (!agent.floodLoggedSinceLastRun) {
|
||||
@@ -2599,6 +2494,8 @@ export function startHeartbeatRunner(opts: {
|
||||
params.scheduledAnchorMs >= 0
|
||||
? params.scheduledAnchorMs
|
||||
: undefined;
|
||||
const requestedTasks = params.tasks ?? [];
|
||||
const retainedWork = params.retainedWork === true;
|
||||
const wakeConfig = readCurrentConfig();
|
||||
const requestedTargetAgentId =
|
||||
requestedAgentId ??
|
||||
@@ -2641,10 +2538,20 @@ export function startHeartbeatRunner(opts: {
|
||||
agent: HeartbeatAgentState,
|
||||
authoritativeScheduledTick = false,
|
||||
): Promise<AgentWakeOutcome> => {
|
||||
const deferral = evaluateWakeDeferral(agent, now, reason, intent, authoritativeScheduledTick);
|
||||
const deferral = evaluateWakeDeferral(agent, now, reason, intent, {
|
||||
authoritativeScheduledTick,
|
||||
retainedWork,
|
||||
});
|
||||
if (deferral.defer) {
|
||||
advanceStaleScheduleAfterDeferral(agent, now, reason, deferral);
|
||||
return { ran: false, result: { status: "skipped", reason: deferral.reason } };
|
||||
return {
|
||||
ran: false,
|
||||
result: {
|
||||
status: "skipped",
|
||||
reason: deferral.reason,
|
||||
retryAtMs: deferral.retryAtMs,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
let res: HeartbeatRunResult;
|
||||
@@ -2734,8 +2641,10 @@ export function startHeartbeatRunner(opts: {
|
||||
if (requestedSessionKey || requestedAgentId) {
|
||||
const targetAgentId = requestedTargetAgentId ?? resolveDefaultAgentId(wakeConfig);
|
||||
const targetAgent = state.agents.get(targetAgentId);
|
||||
// Task intent wins scheduled-task coalescing, so the cadence payload—not
|
||||
// the final intent—proves that the persisted monitor tick joined this turn.
|
||||
const authoritativeScheduledTick =
|
||||
params.source === "interval" && intent === "scheduled" && scheduledEveryMs !== undefined;
|
||||
params.source === "interval" && scheduledEveryMs !== undefined;
|
||||
if (targetAgent && scheduledEveryMs !== undefined && authoritativeScheduledTick) {
|
||||
targetAgent.intervalMs = scheduledEveryMs;
|
||||
targetAgent.phaseMs =
|
||||
@@ -2774,16 +2683,17 @@ export function startHeartbeatRunner(opts: {
|
||||
return outcome.result ?? { status: "skipped", reason: "not-due" };
|
||||
}
|
||||
if (targetAgent) {
|
||||
const deferral = evaluateWakeDeferral(
|
||||
targetAgent,
|
||||
now,
|
||||
reason,
|
||||
intent,
|
||||
const deferral = evaluateWakeDeferral(targetAgent, now, reason, intent, {
|
||||
authoritativeScheduledTick,
|
||||
);
|
||||
retainedWork,
|
||||
});
|
||||
if (deferral.defer) {
|
||||
advanceStaleScheduleAfterDeferral(targetAgent, now, reason, deferral);
|
||||
return { status: "skipped", reason: deferral.reason };
|
||||
return {
|
||||
status: "skipped",
|
||||
reason: deferral.reason,
|
||||
retryAtMs: deferral.retryAtMs,
|
||||
};
|
||||
}
|
||||
}
|
||||
try {
|
||||
@@ -2803,6 +2713,7 @@ export function startHeartbeatRunner(opts: {
|
||||
reason,
|
||||
runScope: "global",
|
||||
sessionKey: requestedSessionKey,
|
||||
tasks: requestedTasks,
|
||||
deps: { runtime: state.runtime },
|
||||
});
|
||||
if (res.status === "skipped" && isRetryableHeartbeatBusySkipReason(res.reason)) {
|
||||
@@ -2869,6 +2780,8 @@ export function startHeartbeatRunner(opts: {
|
||||
heartbeat: params.heartbeat,
|
||||
scheduledEveryMs: params.scheduledEveryMs,
|
||||
scheduledAnchorMs: params.scheduledAnchorMs,
|
||||
tasks: params.tasks,
|
||||
retainedWork: params.retainedWork,
|
||||
source: params.source,
|
||||
intent: params.intent,
|
||||
});
|
||||
|
||||
@@ -189,6 +189,365 @@ describe("heartbeat-wake", () => {
|
||||
expect(handler).toHaveBeenCalledWith(wake("exec-event"));
|
||||
});
|
||||
|
||||
it("coalesces independently scheduled tasks without dropping either prompt", async () => {
|
||||
vi.useFakeTimers();
|
||||
const handler = vi.fn().mockResolvedValue({ status: "ran", durationMs: 1 });
|
||||
setHeartbeatWakeHandler(handler);
|
||||
|
||||
for (const task of [
|
||||
{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" },
|
||||
{ jobId: "job-calendar", name: "calendar", prompt: "Check calendar" },
|
||||
]) {
|
||||
requestHeartbeat({
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: `heartbeat-task:${task.jobId}`,
|
||||
agentId: "main",
|
||||
tasks: [task],
|
||||
coalesceMs: 100,
|
||||
});
|
||||
}
|
||||
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
|
||||
expect(handler).toHaveBeenCalledOnce();
|
||||
expect(handler).toHaveBeenCalledWith({
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: "heartbeat-task:job-calendar",
|
||||
agentId: "main",
|
||||
tasks: [
|
||||
{ jobId: "job-calendar", name: "calendar", prompt: "Check calendar" },
|
||||
{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" },
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it.each(["scheduled-first", "task-first"] as const)(
|
||||
"coalesces a colliding scheduled wake into the task turn (%s)",
|
||||
async (order) => {
|
||||
vi.useFakeTimers();
|
||||
const handler = vi.fn().mockResolvedValue({ status: "ran", durationMs: 1 });
|
||||
setHeartbeatWakeHandler(handler);
|
||||
const scheduled = wake("interval", {
|
||||
agentId: "main",
|
||||
scheduledEveryMs: 5 * 60_000,
|
||||
scheduledAnchorMs: 42_000,
|
||||
coalesceMs: 100,
|
||||
});
|
||||
const task = {
|
||||
source: "interval" as const,
|
||||
intent: "task" as const,
|
||||
reason: "heartbeat-task:job-inbox",
|
||||
agentId: "main",
|
||||
tasks: [{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" }],
|
||||
coalesceMs: 100,
|
||||
};
|
||||
|
||||
for (const request of order === "scheduled-first" ? [scheduled, task] : [task, scheduled]) {
|
||||
requestHeartbeat(request);
|
||||
}
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
|
||||
expect(handler).toHaveBeenCalledOnce();
|
||||
expect(handler).toHaveBeenCalledWith({
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: "heartbeat-task:job-inbox",
|
||||
agentId: "main",
|
||||
scheduledEveryMs: 5 * 60_000,
|
||||
scheduledAnchorMs: 42_000,
|
||||
tasks: [{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" }],
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("runs a phase-aligned task on every period despite the min-spacing floor", async () => {
|
||||
vi.useFakeTimers();
|
||||
vi.setSystemTime(2_000_000_000_000);
|
||||
let lastRunAtMs: number | undefined;
|
||||
const successfulTaskRuns: string[] = [];
|
||||
const handler = vi.fn().mockImplementation(async (request: WakeRequest) => {
|
||||
const now = Date.now();
|
||||
if (lastRunAtMs !== undefined && now - lastRunAtMs < 30_000) {
|
||||
return { status: "skipped" as const, reason: "min-spacing" };
|
||||
}
|
||||
lastRunAtMs = now;
|
||||
if (request.intent === "task") {
|
||||
successfulTaskRuns.push(request.tasks?.[0]?.jobId ?? "missing");
|
||||
}
|
||||
return { status: "ran" as const, durationMs: 1 };
|
||||
});
|
||||
setHeartbeatWakeHandler(handler);
|
||||
|
||||
const requestPeriod = () => {
|
||||
requestHeartbeat(wake("interval", { agentId: "main", coalesceMs: 100 }));
|
||||
requestHeartbeat({
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: "heartbeat-task:job-inbox",
|
||||
agentId: "main",
|
||||
tasks: [{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" }],
|
||||
coalesceMs: 100,
|
||||
});
|
||||
};
|
||||
|
||||
requestPeriod();
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
requestPeriod();
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
|
||||
expect(handler).toHaveBeenCalledTimes(2);
|
||||
expect(successfulTaskRuns).toEqual(["job-inbox", "job-inbox"]);
|
||||
});
|
||||
|
||||
it("keeps task and event wakes in separate guarded turns", async () => {
|
||||
vi.useFakeTimers();
|
||||
const handler = vi.fn().mockResolvedValue({ status: "ran", durationMs: 1 });
|
||||
setHeartbeatWakeHandler(handler);
|
||||
|
||||
requestHeartbeat({
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: "heartbeat-task:job-inbox",
|
||||
agentId: "main",
|
||||
tasks: [{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" }],
|
||||
coalesceMs: 100,
|
||||
});
|
||||
requestHeartbeat({
|
||||
source: "exec-event",
|
||||
intent: "event",
|
||||
reason: "exec-event",
|
||||
agentId: "main",
|
||||
coalesceMs: 100,
|
||||
});
|
||||
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
|
||||
expect(handler).toHaveBeenCalledTimes(2);
|
||||
const handledRequests = handler.mock.calls
|
||||
.map((call) => call[0])
|
||||
.toSorted((left, right) => left.intent.localeCompare(right.intent));
|
||||
expect(handledRequests).toEqual([
|
||||
{
|
||||
source: "exec-event",
|
||||
intent: "event",
|
||||
reason: "exec-event",
|
||||
agentId: "main",
|
||||
},
|
||||
{
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: "heartbeat-task:job-inbox",
|
||||
agentId: "main",
|
||||
tasks: [{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" }],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("retains task prompts across busy retries", async () => {
|
||||
vi.useFakeTimers();
|
||||
const handler = setRetryOnceHeartbeatHandler();
|
||||
const request = {
|
||||
source: "interval" as const,
|
||||
intent: "task" as const,
|
||||
reason: "heartbeat-task:job-inbox",
|
||||
agentId: "main",
|
||||
tasks: [{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" }],
|
||||
};
|
||||
|
||||
requestHeartbeat({ ...request, coalesceMs: 0 });
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
await vi.advanceTimersByTimeAsync(1_000);
|
||||
|
||||
expect(handler).toHaveBeenCalledTimes(2);
|
||||
expect(handler).toHaveBeenNthCalledWith(1, request);
|
||||
expect(handler).toHaveBeenNthCalledWith(2, request);
|
||||
});
|
||||
|
||||
it("runs equal-period tasks at staggered anchors by retaining the spaced task", async () => {
|
||||
vi.useFakeTimers();
|
||||
vi.setSystemTime(2_000_000_000_000);
|
||||
let lastRunAtMs: number | undefined;
|
||||
const successfulTaskRuns: string[] = [];
|
||||
const handler = vi.fn().mockImplementation(async (request: WakeRequest) => {
|
||||
const now = Date.now();
|
||||
if (lastRunAtMs !== undefined && now - lastRunAtMs < 30_000) {
|
||||
return {
|
||||
status: "skipped" as const,
|
||||
reason: "min-spacing",
|
||||
retryAtMs: lastRunAtMs + 30_000,
|
||||
};
|
||||
}
|
||||
lastRunAtMs = now;
|
||||
successfulTaskRuns.push(...(request.tasks ?? []).map((task) => task.jobId));
|
||||
return { status: "ran" as const, durationMs: 1 };
|
||||
});
|
||||
setHeartbeatWakeHandler(handler);
|
||||
const requestTask = (jobId: string) =>
|
||||
requestHeartbeat({
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: `heartbeat-task:${jobId}`,
|
||||
agentId: "main",
|
||||
tasks: [{ jobId, name: jobId, prompt: `Run ${jobId}` }],
|
||||
coalesceMs: 0,
|
||||
});
|
||||
|
||||
requestTask("job-a");
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
await vi.advanceTimersByTimeAsync(4_999);
|
||||
requestTask("job-b");
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
await vi.advanceTimersByTimeAsync(25_000);
|
||||
|
||||
await vi.advanceTimersByTimeAsync(29_999);
|
||||
requestTask("job-a");
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
await vi.advanceTimersByTimeAsync(4_999);
|
||||
requestTask("job-b");
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
await vi.advanceTimersByTimeAsync(25_000);
|
||||
|
||||
expect(successfulTaskRuns).toEqual(["job-a", "job-b", "job-a", "job-b"]);
|
||||
});
|
||||
|
||||
it("does not starve an aged event behind repeated task turns", async () => {
|
||||
vi.useFakeTimers();
|
||||
vi.setSystemTime(2_000_000_000_000);
|
||||
let lastRunAtMs: number | undefined;
|
||||
const successfulIntents: WakeRequest["intent"][] = [];
|
||||
const handler = vi.fn().mockImplementation(async (request: WakeRequest) => {
|
||||
const now = Date.now();
|
||||
if (lastRunAtMs !== undefined && now - lastRunAtMs < 30_000) {
|
||||
return {
|
||||
status: "skipped" as const,
|
||||
reason: "min-spacing",
|
||||
retryAtMs: lastRunAtMs + 30_000,
|
||||
};
|
||||
}
|
||||
lastRunAtMs = now;
|
||||
successfulIntents.push(request.intent);
|
||||
return { status: "ran" as const, durationMs: 1 };
|
||||
});
|
||||
setHeartbeatWakeHandler(handler);
|
||||
const requestTask = (jobId: string) =>
|
||||
requestHeartbeat({
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: `heartbeat-task:${jobId}`,
|
||||
agentId: "main",
|
||||
tasks: [{ jobId, name: jobId, prompt: `Run ${jobId}` }],
|
||||
coalesceMs: 0,
|
||||
});
|
||||
|
||||
requestTask("job-a");
|
||||
requestHeartbeat({
|
||||
source: "exec-event",
|
||||
intent: "event",
|
||||
reason: "exec-event",
|
||||
agentId: "main",
|
||||
coalesceMs: 0,
|
||||
});
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
|
||||
await vi.advanceTimersByTimeAsync(19_999);
|
||||
requestTask("job-b");
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
await vi.advanceTimersByTimeAsync(10_000);
|
||||
|
||||
await vi.advanceTimersByTimeAsync(9_999);
|
||||
requestTask("job-c");
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
await vi.advanceTimersByTimeAsync(20_000);
|
||||
|
||||
expect(successfulIntents).toEqual(["task", "event", "task"]);
|
||||
});
|
||||
|
||||
it("bounds merged task retry state and clears it after success", async () => {
|
||||
vi.useFakeTimers();
|
||||
vi.setSystemTime(2_000_000_000_000);
|
||||
const handler = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce({
|
||||
status: "skipped",
|
||||
reason: "min-spacing",
|
||||
retryAtMs: Date.now() + 30_000,
|
||||
})
|
||||
.mockResolvedValue({ status: "ran", durationMs: 1 });
|
||||
setHeartbeatWakeHandler(handler);
|
||||
const requestTask = (jobId: string) =>
|
||||
requestHeartbeat({
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: `heartbeat-task:${jobId}`,
|
||||
agentId: "main",
|
||||
tasks: [{ jobId, name: jobId, prompt: `Run ${jobId}` }],
|
||||
coalesceMs: 0,
|
||||
});
|
||||
|
||||
requestTask("job-a");
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
requestTask("job-b");
|
||||
requestTask("job-c");
|
||||
await vi.advanceTimersByTimeAsync(29_999);
|
||||
|
||||
expect(handler).toHaveBeenCalledTimes(2);
|
||||
expect(handler.mock.calls[1]?.[0].tasks).toEqual([
|
||||
{ jobId: "job-a", name: "job-a", prompt: "Run job-a" },
|
||||
{ jobId: "job-b", name: "job-b", prompt: "Run job-b" },
|
||||
{ jobId: "job-c", name: "job-c", prompt: "Run job-c" },
|
||||
]);
|
||||
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
expect(handler).toHaveBeenCalledTimes(2);
|
||||
requestTask("job-d");
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect(handler).toHaveBeenCalledTimes(3);
|
||||
expect(handler.mock.calls[2]?.[0].tasks).toEqual([
|
||||
{ jobId: "job-d", name: "job-d", prompt: "Run job-d" },
|
||||
]);
|
||||
});
|
||||
|
||||
it("does not let a retained event cooldown block independent task work", async () => {
|
||||
vi.useFakeTimers();
|
||||
vi.setSystemTime(2_000_000_000_000);
|
||||
const handler = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce({
|
||||
status: "skipped",
|
||||
reason: "not-due",
|
||||
retryAtMs: Date.now() + 30 * 60_000,
|
||||
})
|
||||
.mockResolvedValue({ status: "ran", durationMs: 1 });
|
||||
setHeartbeatWakeHandler(handler);
|
||||
|
||||
requestHeartbeat({
|
||||
source: "exec-event",
|
||||
intent: "event",
|
||||
reason: "exec-event",
|
||||
agentId: "main",
|
||||
coalesceMs: 0,
|
||||
});
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
requestHeartbeat({
|
||||
source: "interval",
|
||||
intent: "task",
|
||||
reason: "heartbeat-task:job-inbox",
|
||||
agentId: "main",
|
||||
tasks: [{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" }],
|
||||
coalesceMs: 0,
|
||||
});
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
|
||||
expect(handler).toHaveBeenCalledTimes(2);
|
||||
expect(handler.mock.calls[1]?.[0]).toMatchObject({
|
||||
intent: "task",
|
||||
tasks: [{ jobId: "job-inbox", name: "inbox", prompt: "Check inbox" }],
|
||||
});
|
||||
});
|
||||
|
||||
it("retries requests-in-flight after the default retry delay", async () => {
|
||||
vi.useFakeTimers();
|
||||
const handler = vi
|
||||
|
||||
@@ -6,7 +6,7 @@ import { normalizeHeartbeatWakeReason } from "./heartbeat-reason.js";
|
||||
|
||||
export type HeartbeatRunResult =
|
||||
| { status: "ran"; durationMs: number }
|
||||
| { status: "skipped"; reason: string }
|
||||
| { status: "skipped"; reason: string; retryAtMs?: number }
|
||||
| { status: "failed"; reason: string };
|
||||
|
||||
export const HEARTBEAT_SKIP_REQUESTS_IN_FLIGHT = "requests-in-flight";
|
||||
@@ -17,12 +17,13 @@ const RETRYABLE_BUSY_SKIP_REASONS = new Set([
|
||||
HEARTBEAT_SKIP_CRON_IN_PROGRESS,
|
||||
HEARTBEAT_SKIP_LANES_BUSY,
|
||||
]);
|
||||
const RETRYABLE_GUARD_SKIP_REASONS = new Set(["not-due", "min-spacing", "flood"]);
|
||||
|
||||
export function isRetryableHeartbeatBusySkipReason(reason: string): boolean {
|
||||
return RETRYABLE_BUSY_SKIP_REASONS.has(reason);
|
||||
}
|
||||
|
||||
export type HeartbeatWakeIntent = "scheduled" | "event" | "immediate" | "manual";
|
||||
export type HeartbeatWakeIntent = "scheduled" | "task" | "event" | "immediate" | "manual";
|
||||
|
||||
export type HeartbeatWakeSource =
|
||||
| "interval"
|
||||
@@ -46,6 +47,13 @@ type HeartbeatWakeOverride = {
|
||||
accountId?: string | undefined;
|
||||
};
|
||||
|
||||
/** Cron-owned periodic work carried directly into a guarded heartbeat turn. */
|
||||
export type HeartbeatScheduledTask = {
|
||||
jobId: string;
|
||||
name: string;
|
||||
prompt: string;
|
||||
};
|
||||
|
||||
export type HeartbeatWakeRequest = {
|
||||
source: HeartbeatWakeSource;
|
||||
intent: HeartbeatWakeIntent;
|
||||
@@ -57,6 +65,9 @@ export type HeartbeatWakeRequest = {
|
||||
scheduledEveryMs?: number;
|
||||
/** Persisted cron monitor phase anchor carried with a scheduled heartbeat tick. */
|
||||
scheduledAnchorMs?: number;
|
||||
tasks?: readonly HeartbeatScheduledTask[];
|
||||
/** Internal marker for work retained after a spacing/cooldown deferral. */
|
||||
retainedWork?: boolean;
|
||||
};
|
||||
|
||||
export type HeartbeatWakeHandler = (opts: HeartbeatWakeRequest) => Promise<HeartbeatRunResult>;
|
||||
@@ -71,7 +82,6 @@ export function areHeartbeatsEnabled(): boolean {
|
||||
return heartbeatsEnabled;
|
||||
}
|
||||
|
||||
type WakeTimerKind = "normal" | "retry";
|
||||
type PendingWakeReason = {
|
||||
source: HeartbeatWakeSource;
|
||||
intent: HeartbeatWakeIntent;
|
||||
@@ -83,16 +93,29 @@ type PendingWakeReason = {
|
||||
heartbeat?: HeartbeatWakeOverride;
|
||||
scheduledEveryMs?: number;
|
||||
scheduledAnchorMs?: number;
|
||||
tasks?: HeartbeatScheduledTask[];
|
||||
/** Earliest instant at which this retained wake class may be dispatched. */
|
||||
notBeforeMs?: number;
|
||||
/** The wake was retained after a spacing/cooldown guard deferred its work. */
|
||||
guardRetry?: boolean;
|
||||
};
|
||||
|
||||
type PendingWakeGroup = {
|
||||
task?: PendingWakeReason;
|
||||
scheduled?: PendingWakeReason;
|
||||
event?: PendingWakeReason;
|
||||
/** Busy/error backoff blocks every wake class for this target. */
|
||||
blockedUntilMs?: number;
|
||||
};
|
||||
|
||||
let handler: HeartbeatWakeHandler | null = null;
|
||||
let handlerGeneration = 0;
|
||||
const pendingWakes = new Map<string, PendingWakeReason>();
|
||||
// One bounded group per target owns every pending/retry class for that agent/session.
|
||||
const pendingWakes = new Map<string, PendingWakeGroup>();
|
||||
let scheduled = false;
|
||||
let running = false;
|
||||
let timer: NodeJS.Timeout | null = null;
|
||||
let timerDueAt: number | null = null;
|
||||
let timerKind: WakeTimerKind | null = null;
|
||||
|
||||
const DEFAULT_COALESCE_MS = 250;
|
||||
const DEFAULT_RETRY_MS = 1_000;
|
||||
@@ -133,12 +156,130 @@ function normalizeWakeTarget(value?: string): string | undefined {
|
||||
return trimmed || undefined;
|
||||
}
|
||||
|
||||
function getWakeTargetKey(params: { agentId?: string; sessionKey?: string }) {
|
||||
function getWakeTargetBaseKey(params: { agentId?: string; sessionKey?: string }) {
|
||||
const agentId = normalizeWakeTarget(params.agentId);
|
||||
const sessionKey = normalizeWakeTarget(params.sessionKey);
|
||||
return `${agentId ?? ""}::${sessionKey ?? ""}`;
|
||||
}
|
||||
|
||||
function mergePendingWakeReasons(
|
||||
previous: PendingWakeReason,
|
||||
next: PendingWakeReason,
|
||||
): PendingWakeReason {
|
||||
const tasksByJobId = new Map<string, HeartbeatScheduledTask>();
|
||||
for (const task of previous.tasks ?? []) {
|
||||
tasksByJobId.set(task.jobId, task);
|
||||
}
|
||||
for (const task of next.tasks ?? []) {
|
||||
tasksByJobId.set(task.jobId, task);
|
||||
}
|
||||
// Concurrent cron ticks can arrive in either order; stable job order keeps the model prompt cacheable.
|
||||
const mergedTasks = Array.from(tasksByJobId.values()).toSorted((left, right) =>
|
||||
left.jobId.localeCompare(right.jobId),
|
||||
);
|
||||
const mixedTaskPair = (previous.intent === "task") !== (next.intent === "task");
|
||||
const preferred = mixedTaskPair
|
||||
? previous.intent === "task"
|
||||
? previous
|
||||
: next
|
||||
: next.priority > previous.priority ||
|
||||
(next.priority === previous.priority && next.requestedAt >= previous.requestedAt)
|
||||
? next
|
||||
: previous;
|
||||
const other = preferred === previous ? next : previous;
|
||||
const scheduledEveryMs = preferred.scheduledEveryMs ?? other.scheduledEveryMs;
|
||||
const scheduledAnchorMs = preferred.scheduledAnchorMs ?? other.scheduledAnchorMs;
|
||||
const merged: PendingWakeReason = {
|
||||
...preferred,
|
||||
...(previous.notBeforeMs !== undefined || next.notBeforeMs !== undefined
|
||||
? {
|
||||
requestedAt: Math.min(previous.requestedAt, next.requestedAt),
|
||||
notBeforeMs: Math.max(previous.notBeforeMs ?? 0, next.notBeforeMs ?? 0),
|
||||
}
|
||||
: {}),
|
||||
...((preferred.heartbeat ?? other.heartbeat)
|
||||
? { heartbeat: preferred.heartbeat ?? other.heartbeat }
|
||||
: {}),
|
||||
...(scheduledEveryMs !== undefined ? { scheduledEveryMs } : {}),
|
||||
...(scheduledAnchorMs !== undefined ? { scheduledAnchorMs } : {}),
|
||||
...(mergedTasks.length ? { tasks: mergedTasks } : {}),
|
||||
};
|
||||
if (previous.guardRetry || next.guardRetry) {
|
||||
merged.guardRetry = true;
|
||||
} else {
|
||||
delete merged.guardRetry;
|
||||
}
|
||||
return merged;
|
||||
}
|
||||
|
||||
function takePendingWakeBatch(now = Date.now()): PendingWakeReason[] {
|
||||
const readyGroups: PendingWakeGroup[] = [];
|
||||
for (const [targetKey, group] of pendingWakes) {
|
||||
if (group.blockedUntilMs !== undefined && group.blockedUntilMs > now) {
|
||||
continue;
|
||||
}
|
||||
const ready: PendingWakeGroup = {};
|
||||
const remaining: PendingWakeGroup = {};
|
||||
for (const slot of ["task", "scheduled", "event"] as const) {
|
||||
const pending = group[slot];
|
||||
if (!pending) {
|
||||
continue;
|
||||
}
|
||||
if (pending.notBeforeMs === undefined || pending.notBeforeMs <= now) {
|
||||
ready[slot] = pending;
|
||||
} else {
|
||||
remaining[slot] = pending;
|
||||
}
|
||||
}
|
||||
if (remaining.task || remaining.scheduled || remaining.event) {
|
||||
pendingWakes.set(targetKey, remaining);
|
||||
} else {
|
||||
pendingWakes.delete(targetKey);
|
||||
}
|
||||
if (ready.task || ready.scheduled || ready.event) {
|
||||
readyGroups.push(ready);
|
||||
}
|
||||
}
|
||||
|
||||
const batch: PendingWakeReason[] = [];
|
||||
for (const group of readyGroups) {
|
||||
if (group.task) {
|
||||
// A due base heartbeat is covered by the task prompt's appended monitor
|
||||
// scratch. Dispatching both lets the base run consume min-spacing and
|
||||
// silently lose the task, so the scheduled wake must join this turn.
|
||||
const taskWake = group.scheduled
|
||||
? mergePendingWakeReasons(group.scheduled, group.task)
|
||||
: group.task;
|
||||
if (group.event) {
|
||||
// Retained work keeps its original age. Sorting it ahead of fresh work
|
||||
// prevents a periodic task stream from starving an older event forever.
|
||||
batch.push(
|
||||
...[taskWake, group.event].toSorted((left, right) => {
|
||||
if (left.guardRetry !== right.guardRetry) {
|
||||
return left.guardRetry ? -1 : 1;
|
||||
}
|
||||
if (left.requestedAt !== right.requestedAt) {
|
||||
return left.requestedAt - right.requestedAt;
|
||||
}
|
||||
return 0;
|
||||
}),
|
||||
);
|
||||
} else {
|
||||
batch.push(taskWake);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (group.event) {
|
||||
batch.push(
|
||||
group.scheduled ? mergePendingWakeReasons(group.scheduled, group.event) : group.event,
|
||||
);
|
||||
} else if (group.scheduled) {
|
||||
batch.push(group.scheduled);
|
||||
}
|
||||
}
|
||||
return batch;
|
||||
}
|
||||
|
||||
function queuePendingWakeReason(params: {
|
||||
source: HeartbeatWakeSource;
|
||||
intent: HeartbeatWakeIntent;
|
||||
@@ -149,12 +290,16 @@ function queuePendingWakeReason(params: {
|
||||
heartbeat?: HeartbeatWakeOverride;
|
||||
scheduledEveryMs?: number;
|
||||
scheduledAnchorMs?: number;
|
||||
tasks?: readonly HeartbeatScheduledTask[];
|
||||
notBeforeMs?: number;
|
||||
blockTargetUntilMs?: number;
|
||||
guardRetry?: boolean;
|
||||
}) {
|
||||
const requestedAt = params.requestedAt ?? Date.now();
|
||||
const normalizedReason = normalizeWakeReason(params.reason);
|
||||
const normalizedAgentId = normalizeWakeTarget(params.agentId);
|
||||
const normalizedSessionKey = normalizeWakeTarget(params.sessionKey);
|
||||
const wakeTargetKey = getWakeTargetKey({
|
||||
const wakeTargetKey = getWakeTargetBaseKey({
|
||||
agentId: normalizedAgentId,
|
||||
sessionKey: normalizedSessionKey,
|
||||
});
|
||||
@@ -173,34 +318,30 @@ function queuePendingWakeReason(params: {
|
||||
heartbeat: params.heartbeat,
|
||||
scheduledEveryMs: params.scheduledEveryMs,
|
||||
scheduledAnchorMs: params.scheduledAnchorMs,
|
||||
...(params.tasks?.length ? { tasks: [...params.tasks] } : {}),
|
||||
...(params.notBeforeMs === undefined ? {} : { notBeforeMs: params.notBeforeMs }),
|
||||
...(params.guardRetry ? { guardRetry: true } : {}),
|
||||
};
|
||||
const previous = pendingWakes.get(wakeTargetKey);
|
||||
const group = pendingWakes.get(wakeTargetKey) ?? {};
|
||||
if (params.blockTargetUntilMs !== undefined) {
|
||||
group.blockedUntilMs = Math.max(group.blockedUntilMs ?? 0, params.blockTargetUntilMs);
|
||||
}
|
||||
const slot =
|
||||
params.intent === "task" ? "task" : params.intent === "scheduled" ? "scheduled" : "event";
|
||||
const previous = group[slot];
|
||||
if (!previous) {
|
||||
pendingWakes.set(wakeTargetKey, next);
|
||||
group[slot] = next;
|
||||
pendingWakes.set(wakeTargetKey, group);
|
||||
return;
|
||||
}
|
||||
const merged =
|
||||
(next.heartbeat ?? previous.heartbeat)
|
||||
? { ...next, heartbeat: next.heartbeat ?? previous.heartbeat }
|
||||
: next;
|
||||
if (next.priority > previous.priority) {
|
||||
pendingWakes.set(wakeTargetKey, merged);
|
||||
return;
|
||||
}
|
||||
if (next.priority === previous.priority && next.requestedAt >= previous.requestedAt) {
|
||||
pendingWakes.set(wakeTargetKey, merged);
|
||||
}
|
||||
group[slot] = mergePendingWakeReasons(previous, next);
|
||||
pendingWakes.set(wakeTargetKey, group);
|
||||
}
|
||||
|
||||
function schedule(coalesceMs: number, kind: WakeTimerKind = "normal") {
|
||||
function schedule(coalesceMs: number) {
|
||||
const delay = resolveTimerTimeoutMs(coalesceMs, DEFAULT_COALESCE_MS, 0);
|
||||
const dueAt = Date.now() + delay;
|
||||
if (timer) {
|
||||
// Keep retry cooldown as a hard minimum delay. This prevents the
|
||||
// finally-path reschedule (often delay=0) from collapsing backoff.
|
||||
if (timerKind === "retry") {
|
||||
return;
|
||||
}
|
||||
// If existing timer fires sooner or at the same time, keep it.
|
||||
if (typeof timerDueAt === "number" && timerDueAt <= dueAt) {
|
||||
return;
|
||||
@@ -209,15 +350,12 @@ function schedule(coalesceMs: number, kind: WakeTimerKind = "normal") {
|
||||
clearTimeout(timer);
|
||||
timer = null;
|
||||
timerDueAt = null;
|
||||
timerKind = null;
|
||||
}
|
||||
timerDueAt = dueAt;
|
||||
timerKind = kind;
|
||||
timer = setTimeout(() => {
|
||||
void (async () => {
|
||||
timer = null;
|
||||
timerDueAt = null;
|
||||
timerKind = null;
|
||||
scheduled = false;
|
||||
const active = handler;
|
||||
if (!active) {
|
||||
@@ -225,12 +363,11 @@ function schedule(coalesceMs: number, kind: WakeTimerKind = "normal") {
|
||||
}
|
||||
if (running) {
|
||||
scheduled = true;
|
||||
schedule(delay, kind);
|
||||
schedule(delay);
|
||||
return;
|
||||
}
|
||||
|
||||
const pendingBatch = Array.from(pendingWakes.values());
|
||||
pendingWakes.clear();
|
||||
const pendingBatch = takePendingWakeBatch();
|
||||
running = true;
|
||||
try {
|
||||
for (const pendingWake of pendingBatch) {
|
||||
@@ -247,6 +384,8 @@ function schedule(coalesceMs: number, kind: WakeTimerKind = "normal") {
|
||||
...(pendingWake.scheduledAnchorMs !== undefined
|
||||
? { scheduledAnchorMs: pendingWake.scheduledAnchorMs }
|
||||
: {}),
|
||||
...(pendingWake.tasks ? { tasks: pendingWake.tasks } : {}),
|
||||
...(pendingWake.guardRetry ? { retainedWork: true } : {}),
|
||||
};
|
||||
// Each wake is detached process work: admit the whole handler before
|
||||
// it can mutate sessions or commitments, and keep it visible until done.
|
||||
@@ -264,8 +403,38 @@ function schedule(coalesceMs: number, kind: WakeTimerKind = "normal") {
|
||||
heartbeat: pendingWake.heartbeat,
|
||||
scheduledEveryMs: pendingWake.scheduledEveryMs,
|
||||
scheduledAnchorMs: pendingWake.scheduledAnchorMs,
|
||||
tasks: pendingWake.tasks,
|
||||
requestedAt: pendingWake.requestedAt,
|
||||
blockTargetUntilMs: Date.now() + DEFAULT_RETRY_MS,
|
||||
});
|
||||
schedule(DEFAULT_RETRY_MS, "retry");
|
||||
schedule(DEFAULT_RETRY_MS);
|
||||
} else if (
|
||||
res.status === "skipped" &&
|
||||
RETRYABLE_GUARD_SKIP_REASONS.has(res.reason) &&
|
||||
(pendingWake.tasks?.length ||
|
||||
pendingWake.intent === "task" ||
|
||||
pendingWake.intent === "event" ||
|
||||
pendingWake.intent === "immediate")
|
||||
) {
|
||||
// A wake that carries work the turn prompt depends on — a task
|
||||
// payload or an unprocessed event — may be deferred by guards but
|
||||
// never dropped. Retain it and retry only after the remaining floor.
|
||||
const retryAtMs = Math.max(Date.now(), res.retryAtMs ?? Date.now() + DEFAULT_RETRY_MS);
|
||||
queuePendingWakeReason({
|
||||
source: pendingWake.source,
|
||||
intent: pendingWake.intent,
|
||||
reason: pendingWake.reason ?? "retry",
|
||||
agentId: pendingWake.agentId,
|
||||
sessionKey: pendingWake.sessionKey,
|
||||
heartbeat: pendingWake.heartbeat,
|
||||
tasks: pendingWake.tasks,
|
||||
scheduledEveryMs: pendingWake.scheduledEveryMs,
|
||||
scheduledAnchorMs: pendingWake.scheduledAnchorMs,
|
||||
requestedAt: pendingWake.requestedAt,
|
||||
notBeforeMs: retryAtMs,
|
||||
guardRetry: true,
|
||||
});
|
||||
schedule(retryAtMs - Date.now());
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
@@ -280,13 +449,16 @@ function schedule(coalesceMs: number, kind: WakeTimerKind = "normal") {
|
||||
heartbeat: pendingWake.heartbeat,
|
||||
scheduledEveryMs: pendingWake.scheduledEveryMs,
|
||||
scheduledAnchorMs: pendingWake.scheduledAnchorMs,
|
||||
tasks: pendingWake.tasks,
|
||||
requestedAt: pendingWake.requestedAt,
|
||||
blockTargetUntilMs: Date.now() + DEFAULT_RETRY_MS,
|
||||
});
|
||||
}
|
||||
schedule(DEFAULT_RETRY_MS, "retry");
|
||||
schedule(DEFAULT_RETRY_MS);
|
||||
} finally {
|
||||
running = false;
|
||||
if (pendingWakes.size > 0 || scheduled) {
|
||||
schedule(delay, "normal");
|
||||
schedulePendingWakes(delay);
|
||||
}
|
||||
}
|
||||
})();
|
||||
@@ -294,6 +466,46 @@ function schedule(coalesceMs: number, kind: WakeTimerKind = "normal") {
|
||||
timer.unref?.();
|
||||
}
|
||||
|
||||
function schedulePendingWakes(readyDelayMs: number) {
|
||||
const now = Date.now();
|
||||
let earliestNotBeforeMs = Number.POSITIVE_INFINITY;
|
||||
let hasReadyWake = false;
|
||||
for (const group of pendingWakes.values()) {
|
||||
if (group.blockedUntilMs !== undefined && group.blockedUntilMs > now) {
|
||||
earliestNotBeforeMs = Math.min(earliestNotBeforeMs, group.blockedUntilMs);
|
||||
continue;
|
||||
}
|
||||
for (const pending of [group.task, group.scheduled, group.event]) {
|
||||
if (!pending) {
|
||||
continue;
|
||||
}
|
||||
if (pending.notBeforeMs === undefined || pending.notBeforeMs <= now) {
|
||||
hasReadyWake = true;
|
||||
} else {
|
||||
earliestNotBeforeMs = Math.min(earliestNotBeforeMs, pending.notBeforeMs);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (hasReadyWake) {
|
||||
schedule(readyDelayMs);
|
||||
} else if (Number.isFinite(earliestNotBeforeMs)) {
|
||||
schedule(earliestNotBeforeMs - now);
|
||||
}
|
||||
}
|
||||
|
||||
function clearPendingWakeRetryState() {
|
||||
for (const group of pendingWakes.values()) {
|
||||
delete group.blockedUntilMs;
|
||||
for (const pending of [group.task, group.scheduled, group.event]) {
|
||||
if (!pending) {
|
||||
continue;
|
||||
}
|
||||
delete pending.notBeforeMs;
|
||||
delete pending.guardRetry;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Register (or clear) the heartbeat wake handler.
|
||||
* Returns a disposer function that clears this specific registration.
|
||||
@@ -313,16 +525,16 @@ export function setHeartbeatWakeHandler(next: HeartbeatWakeHandler | null): () =
|
||||
}
|
||||
timer = null;
|
||||
timerDueAt = null;
|
||||
timerKind = null;
|
||||
// Reset module-level execution state that may be stale from interrupted
|
||||
// runs in the previous lifecycle. Without this, `running === true` from
|
||||
// an interrupted heartbeat blocks all future schedule() attempts, and
|
||||
// `scheduled === true` can cause spurious immediate re-runs.
|
||||
running = false;
|
||||
scheduled = false;
|
||||
clearPendingWakeRetryState();
|
||||
}
|
||||
if (handler && pendingWakes.size > 0) {
|
||||
schedule(DEFAULT_COALESCE_MS, "normal");
|
||||
schedulePendingWakes(DEFAULT_COALESCE_MS);
|
||||
}
|
||||
return () => {
|
||||
if (handlerGeneration !== generation) {
|
||||
@@ -346,6 +558,7 @@ export function requestHeartbeat(opts: {
|
||||
heartbeat?: HeartbeatWakeOverride;
|
||||
scheduledEveryMs?: number;
|
||||
scheduledAnchorMs?: number;
|
||||
tasks?: readonly HeartbeatScheduledTask[];
|
||||
}) {
|
||||
queuePendingWakeReason({
|
||||
source: opts.source,
|
||||
@@ -356,6 +569,7 @@ export function requestHeartbeat(opts: {
|
||||
heartbeat: opts.heartbeat,
|
||||
scheduledEveryMs: opts.scheduledEveryMs,
|
||||
scheduledAnchorMs: opts.scheduledAnchorMs,
|
||||
tasks: opts.tasks,
|
||||
});
|
||||
schedule(opts.coalesceMs ?? DEFAULT_COALESCE_MS, "normal");
|
||||
schedule(opts.coalesceMs ?? DEFAULT_COALESCE_MS);
|
||||
}
|
||||
|
||||
@@ -1252,7 +1252,7 @@
|
||||
},
|
||||
{
|
||||
"deferLoading": true,
|
||||
"description": "Record heartbeat result. `notify=false` no visible send. `notify=true` needs concise notificationText.",
|
||||
"description": "Record heartbeat result. `notify=false` no visible send. `notify=true` needs concise notificationText. Scratch is monitor prose only; manage recurring tasks with cron.",
|
||||
"inputSchema": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
@@ -1277,6 +1277,7 @@
|
||||
"type": "string"
|
||||
},
|
||||
"scratch": {
|
||||
"description": "Complete replacement for heartbeat monitor prose. Recurring schedules belong in cron jobs, not scratch.",
|
||||
"type": "string"
|
||||
},
|
||||
"summary": {
|
||||
|
||||
@@ -217,24 +217,24 @@ This is the deterministic model-bound layer stack OpenClaw can snapshot for the
|
||||
"roughTokens": 0
|
||||
},
|
||||
"dynamicToolsJson": {
|
||||
"chars": 60257,
|
||||
"roughTokens": 15065
|
||||
"chars": 60458,
|
||||
"roughTokens": 15115
|
||||
},
|
||||
"openClawDeveloperInstructions": {
|
||||
"chars": 2469,
|
||||
"roughTokens": 618
|
||||
},
|
||||
"totalTextOnly": {
|
||||
"chars": 26885,
|
||||
"roughTokens": 6722
|
||||
"chars": 27014,
|
||||
"roughTokens": 6754
|
||||
},
|
||||
"totalWithDynamicToolsJson": {
|
||||
"chars": 87144,
|
||||
"roughTokens": 21786
|
||||
"chars": 87474,
|
||||
"roughTokens": 21869
|
||||
},
|
||||
"userInputText": {
|
||||
"chars": 1259,
|
||||
"roughTokens": 315
|
||||
"chars": 1388,
|
||||
"roughTokens": 347
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -515,7 +515,7 @@ Conversation info (untrusted metadata):
|
||||
}
|
||||
```
|
||||
|
||||
Follow the heartbeat monitor scratch context when provided. Do not infer or repeat old tasks from prior chats. Use heartbeat_respond to report the wake outcome. Set notify=false when nothing needs the user's attention. Set notify=true with notificationText only when the user should be interrupted.
|
||||
Follow the heartbeat monitor scratch context when provided. Recurring tasks are cron jobs; create or change their schedules with cron tools or the openclaw cron CLI, not heartbeat scratch. Do not infer or repeat old tasks from prior chats. Use heartbeat_respond to report the wake outcome. Set notify=false when nothing needs the user's attention. Set notify=true with notificationText only when the user should be interrupted.
|
||||
````
|
||||
|
||||
### Tools: Dynamic Tool Catalog
|
||||
@@ -679,7 +679,7 @@ Full JSON: `codex-dynamic-tools.heartbeat-turn.json`
|
||||
},
|
||||
{
|
||||
"deferLoading": true,
|
||||
"description": "Record heartbeat result. `notify=false` no visible send. `notify=true` needs concise notificationText.",
|
||||
"description": "Record heartbeat result. `notify=false` no visible send. `notify=true` needs concise notificationText. Scratch is monitor prose only; manage recurring tasks with cron.",
|
||||
"inputSchema": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
@@ -704,6 +704,7 @@ Full JSON: `codex-dynamic-tools.heartbeat-turn.json`
|
||||
"type": "string"
|
||||
},
|
||||
"scratch": {
|
||||
"description": "Complete replacement for heartbeat monitor prose. Recurring schedules belong in cron jobs, not scratch.",
|
||||
"type": "string"
|
||||
},
|
||||
"summary": {
|
||||
|
||||
Reference in New Issue
Block a user