Files
openclaw/docs/concepts/dreaming.md
Patrick Erichsen aca92b2906 memory/dreaming: decouple managed cron from heartbeat (#70737)
* Revert "fix(memory/dreaming): surface blocked status when heartbeat is disabled for main (#69875)"

This reverts commit 529577e045.

Making way for the dreaming-vs-heartbeat decoupling from Josh's
josh/dreaming-isolated-cron-fix branch, which moves the managed dreaming
cron to isolated agent turns (sessionTarget: "isolated") so dreaming no
longer requires heartbeat to fire. Once the cron no longer rides the
heartbeat path, the blocked-reason observability has nothing left to
report — removing it cleanly here before the cherry-picks land.

* openclaw-3ba.1: move managed dreaming cron to isolated agent turns

* openclaw-46d: claim cron runs before embedded attempts

* openclaw-575: disable managed dreaming cron delivery

* openclaw-575: accept wrapped dreaming cron tokens

* openclaw-ccd: filter cron and wrapper transcript noise from dreaming corpus

* openclaw-cd9: filter archived, cron, and heartbeat transcript noise from dreaming corpus

* openclaw-cd9: suppress role-label reflection tags in rem dreaming

* openclaw-b49: stop narrative timeouts from blocking dreaming cron

* openclaw-b49: keep managed dreaming cron out of diary subagents

* openclaw-ff9: restore cron dream diary generation without serial waits

* openclaw-ff9: run dreaming narratives with lightweight isolated subagent lanes

* openclaw-ff9: detach cron dream diary generation from run completion

* openclaw-ff9: defer cron diary task startup until after cron completion

* doctor/cron: migrate stale managed dreaming jobs to isolated agent turns

After the dreaming cron moved off the heartbeat path to sessionTarget:
"isolated" + payload.kind: "agentTurn" (see the preceding memory-core
changes), users with existing ~/.openclaw/cron/jobs.json entries in the
old sessionTarget: "main" + payload.kind: "systemEvent" shape still
carry stale jobs until the gateway restart reconcile rewrites them.

Add a dreaming-specific cron migration to the existing
maybeRepairLegacyCronStore doctor path so "openclaw doctor" (and
"openclaw doctor --fix") rewrites those jobs without needing a gateway
restart. Match lives in a new doctor-cron-dreaming-payload-migration
helper alongside the existing legacy-delivery and store-migration files.

The matching uses the memory-core managed-job name and description tag
plus the short-term-promotion payload token. Constants are mirrored
from extensions/memory-core/src/dreaming.ts and commented so a future
rename in memory-core is a visible drift point here too.

* memory/dreaming: tighten cron-token match to known wrapper, not substring

The previous match relaxed the line check from 'trimmed line equals token'
to 'line contains token anywhere as a substring' to accept the
`[cron:<id>] <token>` wrapper that isolated-cron turns add. Substring
matching also let any user message embedding the token mid-sentence
trigger the dream-promotion hook, and was flagged by both Greptile and
Aisle on PR #70737.

Replace it with strip-the-known-prefix-then-exact-match: keep the
`[cron:<id>]` wrapper case working, reject every other variant. Add
focused unit coverage that the bare token, the wrapped token, and bare
multiline cases match while embedded / code-fenced / arbitrarily-wrapped
variants do not.

* memory/dreaming: drop assistant followup only on assistant-side signals

Per PR #70737 review (aisle-research-bot, Medium): the previous logic
suppressed the next assistant message whenever the prior user message
matched a 'generated prompt' pattern (`[cron:...]`,
`System (untrusted): ...`, heartbeat prompts, exec-completion events).
Real users can type those same patterns, which let a user exfiltrate
real assistant replies from the dreaming corpus by prefixing their own
prompt — the assistant's reply would be silently dropped.

Remove the cross-message coupling. Assistant-side machinery (silent
replies, system wrappers) is already dropped by sanitizeSessionText,
which is the right layer for that filter. Add an explicit assistant-side
HEARTBEAT_TOKEN check to keep the legitimate `HEARTBEAT_OK` ack drop
working without depending on the prior user message. Add a regression
test exercising the spoofing scenario.

* doctor/cron: assert mirrored dreaming constants stay in sync

Per PR #70737 review (greptile-apps): the doctor migration mirrors three
constants (MANAGED_DREAMING_CRON_NAME, MANAGED_DREAMING_CRON_TAG,
DREAMING_SYSTEM_EVENT_TEXT) from extensions/memory-core/src/dreaming.ts.
A future rename in either file would silently break the migration.

Add a vitest unit that reads both files and asserts the literals match.
Manually verified the assertion fires with a clear error when one side
diverges. Adds no runtime cost; sits in the regular test pipeline.

* fix(memory): stabilize dreaming CI checks

* memory/dreaming: skip eager narrative session cleanup when detached

Per PR #70737 review (chatgpt-codex-connector, P2): runDreamingSweepPhases
called deleteNarrativeSessionBestEffort synchronously right after each
phase. Once narrative generation moved to detached mode (queued via
queueMicrotask), the eager cleanup races the writer: the session is
deleted before the queued subagent run reads it, silently dropping cron
diary entries.

Skip the eager cleanup branch when params.detachNarratives is true.
generateAndAppendDreamNarrative still runs its own deleteSession in the
finally{} block, so the cleanup intent is preserved without the race.
Heartbeat-driven (non-detached) runs keep the original eager-cleanup
behavior.

* fix(plugin-sdk): restore heartbeat-summary re-export

Per PR #70737 review (chatgpt-codex-connector, P1): the revert of
PR #69875 dropped the `heartbeat-summary` re-export from
`openclaw/plugin-sdk/infra-runtime`. That subpath shipped publicly two
days earlier, so removing it is technically a breaking change to a
public SDK surface — third-party plugins importing
`isHeartbeatEnabledForAgent` / `resolveHeartbeatIntervalMs` from this
path would fail with no replacement contract introduced.

Restore the re-export. Costs nothing to keep; the helpers are already
public via `../infra/heartbeat-summary.ts`. SDK additions are by
default backwards-compatible (CLAUDE.md), so removing within days of
introduction violates that intent.

* changelog: note dreaming decoupling from heartbeat

Refs PR #70737.

---------

Co-authored-by: Josh Lehman <josh@martian.engineering>
2026-04-23 22:23:19 -07:00

7.6 KiB

summary, title, read_when
summary title read_when
Background memory consolidation with light, deep, and REM phases plus a Dream Diary Dreaming
You want memory promotion to run automatically
You want to understand what each dreaming phase does
You want to tune consolidation without polluting MEMORY.md

Dreaming is the background memory consolidation system in memory-core. It helps OpenClaw move strong short-term signals into durable memory while keeping the process explainable and reviewable.

Dreaming is opt-in and disabled by default.

What dreaming writes

Dreaming keeps two kinds of output:

  • Machine state in memory/.dreams/ (recall store, phase signals, ingestion checkpoints, locks).
  • Human-readable output in DREAMS.md (or existing dreams.md) and optional phase report files under memory/dreaming/<phase>/YYYY-MM-DD.md.

Long-term promotion still writes only to MEMORY.md.

Phase model

Dreaming uses three cooperative phases:

Phase Purpose Durable write
Light Sort and stage recent short-term material No
Deep Score and promote durable candidates Yes (MEMORY.md)
REM Reflect on themes and recurring ideas No

These phases are internal implementation details, not separate user-configured "modes."

Light phase

Light phase ingests recent daily memory signals and recall traces, dedupes them, and stages candidate lines.

  • Reads from short-term recall state, recent daily memory files, and redacted session transcripts when available.
  • Writes a managed ## Light Sleep block when storage includes inline output.
  • Records reinforcement signals for later deep ranking.
  • Never writes to MEMORY.md.

Deep phase

Deep phase decides what becomes long-term memory.

  • Ranks candidates using weighted scoring and threshold gates.
  • Requires minScore, minRecallCount, and minUniqueQueries to pass.
  • Rehydrates snippets from live daily files before writing, so stale/deleted snippets are skipped.
  • Appends promoted entries to MEMORY.md.
  • Writes a ## Deep Sleep summary into DREAMS.md and optionally writes memory/dreaming/deep/YYYY-MM-DD.md.

REM phase

REM phase extracts patterns and reflective signals.

  • Builds theme and reflection summaries from recent short-term traces.
  • Writes a managed ## REM Sleep block when storage includes inline output.
  • Records REM reinforcement signals used by deep ranking.
  • Never writes to MEMORY.md.

Session transcript ingestion

Dreaming can ingest redacted session transcripts into the dreaming corpus. When transcripts are available, they are fed into the light phase alongside daily memory signals and recall traces. Personal and sensitive content is redacted before ingestion.

Dream Diary

Dreaming also keeps a narrative Dream Diary in DREAMS.md. After each phase has enough material, memory-core runs a best-effort background subagent turn (using the default runtime model) and appends a short diary entry.

This diary is for human reading in the Dreams UI, not a promotion source. Dreaming-generated diary/report artifacts are excluded from short-term promotion. Only grounded memory snippets are eligible to promote into MEMORY.md.

There is also a grounded historical backfill lane for review and recovery work:

  • memory rem-harness --path ... --grounded previews grounded diary output from historical YYYY-MM-DD.md notes.
  • memory rem-backfill --path ... writes reversible grounded diary entries into DREAMS.md.
  • memory rem-backfill --path ... --stage-short-term stages grounded durable candidates into the same short-term evidence store the normal deep phase already uses.
  • memory rem-backfill --rollback and --rollback-short-term remove those staged backfill artifacts without touching ordinary diary entries or live short-term recall.

The Control UI exposes the same diary backfill/reset flow so you can inspect results in the Dreams scene before deciding whether the grounded candidates deserve promotion. The Scene also shows a distinct grounded lane so you can see which staged short-term entries came from historical replay, which promoted items were grounded-led, and clear only grounded-only staged entries without touching ordinary live short-term state.

Deep ranking signals

Deep ranking uses six weighted base signals plus phase reinforcement:

Signal Weight Description
Frequency 0.24 How many short-term signals the entry accumulated
Relevance 0.30 Average retrieval quality for the entry
Query diversity 0.15 Distinct query/day contexts that surfaced it
Recency 0.15 Time-decayed freshness score
Consolidation 0.10 Multi-day recurrence strength
Conceptual richness 0.06 Concept-tag density from snippet/path

Light and REM phase hits add a small recency-decayed boost from memory/.dreams/phase-signals.json.

Scheduling

When enabled, memory-core auto-manages one cron job for a full dreaming sweep. Each sweep runs phases in order: light -> REM -> deep.

Default cadence behavior:

Setting Default
dreaming.frequency 0 3 * * *

Quick start

Enable dreaming:

{
  "plugins": {
    "entries": {
      "memory-core": {
        "config": {
          "dreaming": {
            "enabled": true
          }
        }
      }
    }
  }
}

Enable dreaming with a custom sweep cadence:

{
  "plugins": {
    "entries": {
      "memory-core": {
        "config": {
          "dreaming": {
            "enabled": true,
            "timezone": "America/Los_Angeles",
            "frequency": "0 */6 * * *"
          }
        }
      }
    }
  }
}

Slash command

/dreaming status
/dreaming on
/dreaming off
/dreaming help

CLI workflow

Use CLI promotion for preview or manual apply:

openclaw memory promote
openclaw memory promote --apply
openclaw memory promote --limit 5
openclaw memory status --deep

Manual memory promote uses deep-phase thresholds by default unless overridden with CLI flags.

Explain why a specific candidate would or would not promote:

openclaw memory promote-explain "router vlan"
openclaw memory promote-explain "router vlan" --json

Preview REM reflections, candidate truths, and deep promotion output without writing anything:

openclaw memory rem-harness
openclaw memory rem-harness --json

Key defaults

All settings live under plugins.entries.memory-core.config.dreaming.

Key Default
enabled false
frequency 0 3 * * *

Phase policy, thresholds, and storage behavior are internal implementation details (not user-facing config).

See Memory configuration reference for the full key list.

Dreams UI

When enabled, the Gateway Dreams tab shows:

  • current dreaming enabled state
  • phase-level status and managed-sweep presence
  • short-term, grounded, signal, and promoted-today counts
  • next scheduled run timing
  • a distinct grounded Scene lane for staged historical replay entries
  • an expandable Dream Diary reader backed by doctor.memory.dreamDiary