Files
openclaw/.github/workflows/workflow-sanity.yml
Josh Lehman 0a8e3604ba refactor: flip sessions and transcripts to sqlite storage (#98236)
* refactor(sessions): migrate runtime storage to sqlite

* test(sessions): fix sqlite CI regressions

* test(sessions): align remaining sqlite fixtures

* fix(codex): require sqlite trajectory recorder

* test(sessions): align orphan recovery sqlite fixture

* test(sessions): align sqlite rebase fixtures

* fix(sessions): finish current-main integration of the sqlite flip

Resolve the whole-store SDK removal across its owner boundary: drop the
loadSessionStore re-export and the registry whole-store wrappers, wire
hasTrackedActiveSessionRun into gateway chat, complete the
preserveLockedHarnessIds cleanup contract, flip the codex thread-history
import to storePath targets, and port remaining main-side tests from
file-store helpers to session accessor reads.

* chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs

Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore
the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain
bump gets its own review, and regenerate docs_map, the plugin SDK API baseline,
and the export-surface ratchet for the merged tree.

* feat(sessions): keep archived transcripts by default with zstd cold storage

Codex-style retention: deleting or resetting a session archives its
transcript as a zstd-compressed JSONL artifact (plain when the runtime
lacks node:zlib zstd) and keeps it until the disk budget evicts oldest
first. resetArchiveRetention now governs both deleted and reset archives
and defaults to keep; maxDiskBytes defaults to 2gb so retention stays
bounded, with archives evicted before live sessions. The cron reaper
follows the same knob instead of deleting archives on its own timer.

* fix(state): converge agent DB migration lineages and bound database growth

Merge coherence: run both structure-gated legacy memory-schema repairs
(flip-lineage drop, main-lineage identity rebuild) before the flip
migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all
converge, and hoist foreign_keys=OFF outside the schema transaction
where the pragma was silently ignored and the v1 sessions rebuild
cascade-deleted session_entries.

Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL
maintenance releases freed pages in bounded passes (never a blocking
full VACUUM), and doctor reports state/agent DB bloat from freelist
stats.

* fix(codex): resolve the store path for thread-history import via the SDK

The supervision catalog passed the legacy sessionFile locator to the
storePath-targeted transcript mirror; resolve the agent store path with
the session-store SDK helper instead of a runtime-object seam so test
fakes and headless callers need no extra surface. Drop the obsolete
missing-session-id preprocessing case: sessions rows are NOT NULL on
session_id and upsert repairs id-less patches at write time.

* fix(sessions): fail safe on malformed disk-budget config and doctor stat errors

A malformed explicit maxDiskBytes disables the budget instead of
falling back to the destructive 2gb default the user never chose, and
the doctor bloat check skips databases whose paths stat-fail instead of
aborting doctor.

* fix(sessions): complete sqlite conflict translations

* test(sqlite): align hardening checks with maintenance

* test(sessions): inspect compressed transcript archives

* fix(tests): await session seeds and drop unused helpers flagged by CI lint

The five unawaited writeSessionStoreSeed calls raced their SQLite seeds
against the assertions, failing compact shards; the bloat probe drops a
useless initializer and the merged tests drop now-unused helpers.

* test(sessions): type legacy proof events directly

* test(sessions): align hardening contracts

* perf(sessions): read usage transcript sizes from SQL aggregates

Usage/cost scans walked every session and materialized every transcript
event just to re-stringify it for a byte estimate — the #86718 stall
class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes
in SQLite without loading a single row.

* fix(sessions): re-root foreign-root transcript paths onto the current sessions dir

Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry
absolute sessionFile paths from the old root; the containment fallback
kept those foreign paths, so migration read (and would archive) files in
the original root and reported local copies missing. Re-root the
canonical agents/<id>/sessions suffix onto the current dir when the file
exists there; genuine cross-root layouts still fall through unchanged.

* test(agents): seed harness admission through sqlite

* fix(sqlite): close agent db on pragma setup failure

* fix(doctor): compact and retrofit incremental auto-vacuum after session import

The migration is the sanctioned offline window: post-import compact
reclaims import churn and applies auto_vacuum=INCREMENTAL to databases
created before the fresh-DB pragma existed, so runtime maintenance can
release pages in bounded passes on every install.

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-11 14:50:37 -07:00

220 lines
8.2 KiB
YAML

name: Workflow Sanity
on:
pull_request:
paths-ignore:
- "CHANGELOG.md"
push:
branches: [main]
paths-ignore:
- "CHANGELOG.md"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
jobs:
no-tabs:
if: github.event_name != 'workflow_dispatch'
runs-on: ubuntu-24.04
steps:
- &workflow_sanity_checkout_step
name: Checkout
env:
CHECKOUT_REPO: ${{ github.repository }}
CHECKOUT_SHA: ${{ github.sha }}
run: |
set -euo pipefail
git init "$GITHUB_WORKSPACE"
git -C "$GITHUB_WORKSPACE" config gc.auto 0
git -C "$GITHUB_WORKSPACE" remote add origin "https://github.com/${CHECKOUT_REPO}.git"
fetch_checkout_ref() {
local fetch_status
for attempt in 1 2 3; do
timeout --signal=TERM --kill-after=10s 30s git -C "$GITHUB_WORKSPACE" \
-c protocol.version=2 \
fetch --no-tags --prune --no-recurse-submodules --depth=1 origin \
"+${CHECKOUT_SHA}:refs/remotes/origin/checkout" && return 0
fetch_status="$?"
if [ "$fetch_status" != "124" ] && [ "$fetch_status" != "137" ]; then
return "$fetch_status"
fi
if [ "$attempt" = "3" ]; then
return "$fetch_status"
fi
echo "::warning::checkout fetch for '$CHECKOUT_SHA' timed out on attempt $attempt; retrying"
sleep 5
done
}
fetch_checkout_ref
git -C "$GITHUB_WORKSPACE" checkout --detach refs/remotes/origin/checkout
- name: Fail on tabs in workflow files
run: |
python - <<'PY'
from __future__ import annotations
import pathlib
import sys
root = pathlib.Path(".github/workflows")
bad: list[str] = []
for path in sorted(root.rglob("*.yml")):
if b"\t" in path.read_bytes():
bad.append(str(path))
for path in sorted(root.rglob("*.yaml")):
if b"\t" in path.read_bytes():
bad.append(str(path))
if bad:
print("Tabs found in workflow file(s):")
for path in bad:
print(f"- {path}")
sys.exit(1)
PY
actionlint:
if: github.event_name != 'workflow_dispatch'
runs-on: ubuntu-24.04
steps:
- *workflow_sanity_checkout_step
- name: Setup Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.12"
- name: Prepare trusted workflow audit configs
if: github.event_name == 'pull_request'
env:
BASE_REF: ${{ github.event.pull_request.base.ref }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
set -euo pipefail
trusted_config="$RUNNER_TEMP/pre-commit-base.yaml"
trusted_zizmor_config="$RUNNER_TEMP/zizmor-base.yml"
fetch_base_ref() {
local ref="$1"
local target="$2"
local fetch_status
for attempt in 1 2 3; do
timeout --signal=TERM --kill-after=10s 30s git fetch --no-tags --depth=1 origin \
"+${ref}:${target}" && return 0
fetch_status="$?"
if [ "$fetch_status" != "124" ] && [ "$fetch_status" != "137" ]; then
return "$fetch_status"
fi
if [ "$attempt" = "3" ]; then
return "$fetch_status"
fi
echo "::warning::trusted base fetch for '$ref' timed out on attempt $attempt; retrying"
sleep 5
done
}
if ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then
fetch_base_ref "$BASE_SHA" "refs/remotes/origin/security-base" ||
fetch_base_ref "refs/heads/${BASE_REF}" "refs/remotes/origin/${BASE_REF}"
fi
if git cat-file -e "${BASE_SHA}:.pre-commit-config.yaml" 2>/dev/null; then
git show "${BASE_SHA}:.pre-commit-config.yaml" > "$trusted_config"
elif git show "refs/remotes/origin/${BASE_REF}:.pre-commit-config.yaml" \
> "$trusted_config" 2>/dev/null; then
echo "Base SHA ${BASE_SHA} does not expose .pre-commit-config.yaml; using origin/${BASE_REF} instead."
else
echo "::error title=trusted pre-commit config unavailable::Could not read .pre-commit-config.yaml from ${BASE_SHA} or origin/${BASE_REF}."
exit 1
fi
if git cat-file -e "${BASE_SHA}:.github/zizmor.yml" 2>/dev/null; then
git show "${BASE_SHA}:.github/zizmor.yml" > "$trusted_zizmor_config"
elif git show "refs/remotes/origin/${BASE_REF}:.github/zizmor.yml" \
> "$trusted_zizmor_config" 2>/dev/null; then
echo "Base SHA ${BASE_SHA} does not expose .github/zizmor.yml; using origin/${BASE_REF} instead."
else
echo "::error title=trusted zizmor config unavailable::Could not read .github/zizmor.yml from ${BASE_SHA} or origin/${BASE_REF}."
exit 1
fi
python3 - "$trusted_config" "$trusted_zizmor_config" <<'PY'
from pathlib import Path
import sys
config_path = Path(sys.argv[1])
zizmor_config_path = sys.argv[2]
text = config_path.read_text()
if ".github/zizmor.yml" not in text:
raise SystemExit("trusted pre-commit config does not reference .github/zizmor.yml")
config_path.write_text(text.replace(".github/zizmor.yml", zizmor_config_path))
PY
echo "PRE_COMMIT_CONFIG_PATH=$trusted_config" >> "$GITHUB_ENV"
- name: Install pre-commit
run: python -m pip install --disable-pip-version-check pre-commit==4.2.0
- name: Install actionlint
shell: bash
run: |
set -euo pipefail
ACTIONLINT_VERSION="1.7.11"
archive="actionlint_${ACTIONLINT_VERSION}_linux_amd64.tar.gz"
base_url="https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}"
# GitHub release downloads occasionally return transient 5xx responses.
# Retry all curl errors here so workflow-sanity does not fail closed on
# a one-off release edge outage.
curl --retry 5 --retry-delay 2 --retry-all-errors -sSfL -o "${archive}" "${base_url}/${archive}"
curl --retry 5 --retry-delay 2 --retry-all-errors -sSfL -o checksums.txt "${base_url}/actionlint_${ACTIONLINT_VERSION}_checksums.txt"
grep " ${archive}\$" checksums.txt | sha256sum -c -
tar -xzf "${archive}" actionlint
sudo install -m 0755 actionlint /usr/local/bin/actionlint
- name: Lint workflows
run: actionlint
- name: Audit all workflows with zizmor
shell: bash
run: |
set -euo pipefail
mapfile -t workflow_files < <(
find .github/workflows -maxdepth 1 -type f \( -name '*.yml' -o -name '*.yaml' \) | sort
)
pre-commit run --config "${PRE_COMMIT_CONFIG_PATH:-.pre-commit-config.yaml}" zizmor --files "${workflow_files[@]}"
- name: Disallow direct inputs interpolation in composite run blocks
run: python3 scripts/check-composite-action-input-interpolation.py
- name: Disallow tracked merge conflict markers
run: node scripts/check-no-conflict-markers.mjs
generated-doc-baselines:
if: github.event_name == 'workflow_dispatch'
runs-on: ubuntu-24.04
steps:
- *workflow_sanity_checkout_step
- name: Setup Node environment
uses: ./.github/actions/setup-node-env
with:
install-bun: "false"
- name: Check config docs drift statefile
run: pnpm config:docs:check
- name: Check plugin SDK API baseline drift
run: pnpm plugin-sdk:api:check
- name: Check SQLite sessions/transcripts schema baseline drift
run: pnpm sqlite:sessions-schema:check
- name: Check plugin SDK surface budget
run: pnpm plugin-sdk:surface:check