mirror of
https://github.com/openclaw/openclaw.git
synced 2026-07-15 11:56:01 +00:00
* refactor(sessions): migrate runtime storage to sqlite * test(sessions): fix sqlite CI regressions * test(sessions): align remaining sqlite fixtures * fix(codex): require sqlite trajectory recorder * test(sessions): align orphan recovery sqlite fixture * test(sessions): align sqlite rebase fixtures * fix(sessions): finish current-main integration of the sqlite flip Resolve the whole-store SDK removal across its owner boundary: drop the loadSessionStore re-export and the registry whole-store wrappers, wire hasTrackedActiveSessionRun into gateway chat, complete the preserveLockedHarnessIds cleanup contract, flip the codex thread-history import to storePath targets, and port remaining main-side tests from file-store helpers to session accessor reads. * chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain bump gets its own review, and regenerate docs_map, the plugin SDK API baseline, and the export-surface ratchet for the merged tree. * feat(sessions): keep archived transcripts by default with zstd cold storage Codex-style retention: deleting or resetting a session archives its transcript as a zstd-compressed JSONL artifact (plain when the runtime lacks node:zlib zstd) and keeps it until the disk budget evicts oldest first. resetArchiveRetention now governs both deleted and reset archives and defaults to keep; maxDiskBytes defaults to 2gb so retention stays bounded, with archives evicted before live sessions. The cron reaper follows the same knob instead of deleting archives on its own timer. * fix(state): converge agent DB migration lineages and bound database growth Merge coherence: run both structure-gated legacy memory-schema repairs (flip-lineage drop, main-lineage identity rebuild) before the flip migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all converge, and hoist foreign_keys=OFF outside the schema transaction where the pragma was silently ignored and the v1 sessions rebuild cascade-deleted session_entries. Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL maintenance releases freed pages in bounded passes (never a blocking full VACUUM), and doctor reports state/agent DB bloat from freelist stats. * fix(codex): resolve the store path for thread-history import via the SDK The supervision catalog passed the legacy sessionFile locator to the storePath-targeted transcript mirror; resolve the agent store path with the session-store SDK helper instead of a runtime-object seam so test fakes and headless callers need no extra surface. Drop the obsolete missing-session-id preprocessing case: sessions rows are NOT NULL on session_id and upsert repairs id-less patches at write time. * fix(sessions): fail safe on malformed disk-budget config and doctor stat errors A malformed explicit maxDiskBytes disables the budget instead of falling back to the destructive 2gb default the user never chose, and the doctor bloat check skips databases whose paths stat-fail instead of aborting doctor. * fix(sessions): complete sqlite conflict translations * test(sqlite): align hardening checks with maintenance * test(sessions): inspect compressed transcript archives * fix(tests): await session seeds and drop unused helpers flagged by CI lint The five unawaited writeSessionStoreSeed calls raced their SQLite seeds against the assertions, failing compact shards; the bloat probe drops a useless initializer and the merged tests drop now-unused helpers. * test(sessions): type legacy proof events directly * test(sessions): align hardening contracts * perf(sessions): read usage transcript sizes from SQL aggregates Usage/cost scans walked every session and materialized every transcript event just to re-stringify it for a byte estimate — the #86718 stall class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes in SQLite without loading a single row. * fix(sessions): re-root foreign-root transcript paths onto the current sessions dir Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry absolute sessionFile paths from the old root; the containment fallback kept those foreign paths, so migration read (and would archive) files in the original root and reported local copies missing. Re-root the canonical agents/<id>/sessions suffix onto the current dir when the file exists there; genuine cross-root layouts still fall through unchanged. * test(agents): seed harness admission through sqlite * fix(sqlite): close agent db on pragma setup failure * fix(doctor): compact and retrofit incremental auto-vacuum after session import The migration is the sanctioned offline window: post-import compact reclaims import churn and applies auto_vacuum=INCREMENTAL to databases created before the fresh-DB pragma existed, so runtime maintenance can release pages in bounded passes on every install. --------- Co-authored-by: Peter Steinberger <steipete@gmail.com>
220 lines
8.2 KiB
YAML
220 lines
8.2 KiB
YAML
name: Workflow Sanity
|
|
|
|
on:
|
|
pull_request:
|
|
paths-ignore:
|
|
- "CHANGELOG.md"
|
|
push:
|
|
branches: [main]
|
|
paths-ignore:
|
|
- "CHANGELOG.md"
|
|
workflow_dispatch:
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
env:
|
|
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
|
|
|
jobs:
|
|
no-tabs:
|
|
if: github.event_name != 'workflow_dispatch'
|
|
runs-on: ubuntu-24.04
|
|
steps:
|
|
- &workflow_sanity_checkout_step
|
|
name: Checkout
|
|
env:
|
|
CHECKOUT_REPO: ${{ github.repository }}
|
|
CHECKOUT_SHA: ${{ github.sha }}
|
|
run: |
|
|
set -euo pipefail
|
|
git init "$GITHUB_WORKSPACE"
|
|
git -C "$GITHUB_WORKSPACE" config gc.auto 0
|
|
git -C "$GITHUB_WORKSPACE" remote add origin "https://github.com/${CHECKOUT_REPO}.git"
|
|
fetch_checkout_ref() {
|
|
local fetch_status
|
|
for attempt in 1 2 3; do
|
|
timeout --signal=TERM --kill-after=10s 30s git -C "$GITHUB_WORKSPACE" \
|
|
-c protocol.version=2 \
|
|
fetch --no-tags --prune --no-recurse-submodules --depth=1 origin \
|
|
"+${CHECKOUT_SHA}:refs/remotes/origin/checkout" && return 0
|
|
fetch_status="$?"
|
|
if [ "$fetch_status" != "124" ] && [ "$fetch_status" != "137" ]; then
|
|
return "$fetch_status"
|
|
fi
|
|
if [ "$attempt" = "3" ]; then
|
|
return "$fetch_status"
|
|
fi
|
|
echo "::warning::checkout fetch for '$CHECKOUT_SHA' timed out on attempt $attempt; retrying"
|
|
sleep 5
|
|
done
|
|
}
|
|
fetch_checkout_ref
|
|
git -C "$GITHUB_WORKSPACE" checkout --detach refs/remotes/origin/checkout
|
|
|
|
- name: Fail on tabs in workflow files
|
|
run: |
|
|
python - <<'PY'
|
|
from __future__ import annotations
|
|
|
|
import pathlib
|
|
import sys
|
|
|
|
root = pathlib.Path(".github/workflows")
|
|
bad: list[str] = []
|
|
for path in sorted(root.rglob("*.yml")):
|
|
if b"\t" in path.read_bytes():
|
|
bad.append(str(path))
|
|
|
|
for path in sorted(root.rglob("*.yaml")):
|
|
if b"\t" in path.read_bytes():
|
|
bad.append(str(path))
|
|
|
|
if bad:
|
|
print("Tabs found in workflow file(s):")
|
|
for path in bad:
|
|
print(f"- {path}")
|
|
sys.exit(1)
|
|
PY
|
|
|
|
actionlint:
|
|
if: github.event_name != 'workflow_dispatch'
|
|
runs-on: ubuntu-24.04
|
|
steps:
|
|
- *workflow_sanity_checkout_step
|
|
- name: Setup Python
|
|
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
|
|
with:
|
|
python-version: "3.12"
|
|
|
|
- name: Prepare trusted workflow audit configs
|
|
if: github.event_name == 'pull_request'
|
|
env:
|
|
BASE_REF: ${{ github.event.pull_request.base.ref }}
|
|
BASE_SHA: ${{ github.event.pull_request.base.sha }}
|
|
run: |
|
|
set -euo pipefail
|
|
trusted_config="$RUNNER_TEMP/pre-commit-base.yaml"
|
|
trusted_zizmor_config="$RUNNER_TEMP/zizmor-base.yml"
|
|
|
|
fetch_base_ref() {
|
|
local ref="$1"
|
|
local target="$2"
|
|
local fetch_status
|
|
for attempt in 1 2 3; do
|
|
timeout --signal=TERM --kill-after=10s 30s git fetch --no-tags --depth=1 origin \
|
|
"+${ref}:${target}" && return 0
|
|
fetch_status="$?"
|
|
if [ "$fetch_status" != "124" ] && [ "$fetch_status" != "137" ]; then
|
|
return "$fetch_status"
|
|
fi
|
|
if [ "$attempt" = "3" ]; then
|
|
return "$fetch_status"
|
|
fi
|
|
echo "::warning::trusted base fetch for '$ref' timed out on attempt $attempt; retrying"
|
|
sleep 5
|
|
done
|
|
}
|
|
|
|
if ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then
|
|
fetch_base_ref "$BASE_SHA" "refs/remotes/origin/security-base" ||
|
|
fetch_base_ref "refs/heads/${BASE_REF}" "refs/remotes/origin/${BASE_REF}"
|
|
fi
|
|
|
|
if git cat-file -e "${BASE_SHA}:.pre-commit-config.yaml" 2>/dev/null; then
|
|
git show "${BASE_SHA}:.pre-commit-config.yaml" > "$trusted_config"
|
|
elif git show "refs/remotes/origin/${BASE_REF}:.pre-commit-config.yaml" \
|
|
> "$trusted_config" 2>/dev/null; then
|
|
echo "Base SHA ${BASE_SHA} does not expose .pre-commit-config.yaml; using origin/${BASE_REF} instead."
|
|
else
|
|
echo "::error title=trusted pre-commit config unavailable::Could not read .pre-commit-config.yaml from ${BASE_SHA} or origin/${BASE_REF}."
|
|
exit 1
|
|
fi
|
|
|
|
if git cat-file -e "${BASE_SHA}:.github/zizmor.yml" 2>/dev/null; then
|
|
git show "${BASE_SHA}:.github/zizmor.yml" > "$trusted_zizmor_config"
|
|
elif git show "refs/remotes/origin/${BASE_REF}:.github/zizmor.yml" \
|
|
> "$trusted_zizmor_config" 2>/dev/null; then
|
|
echo "Base SHA ${BASE_SHA} does not expose .github/zizmor.yml; using origin/${BASE_REF} instead."
|
|
else
|
|
echo "::error title=trusted zizmor config unavailable::Could not read .github/zizmor.yml from ${BASE_SHA} or origin/${BASE_REF}."
|
|
exit 1
|
|
fi
|
|
|
|
python3 - "$trusted_config" "$trusted_zizmor_config" <<'PY'
|
|
from pathlib import Path
|
|
import sys
|
|
|
|
config_path = Path(sys.argv[1])
|
|
zizmor_config_path = sys.argv[2]
|
|
text = config_path.read_text()
|
|
if ".github/zizmor.yml" not in text:
|
|
raise SystemExit("trusted pre-commit config does not reference .github/zizmor.yml")
|
|
config_path.write_text(text.replace(".github/zizmor.yml", zizmor_config_path))
|
|
PY
|
|
|
|
echo "PRE_COMMIT_CONFIG_PATH=$trusted_config" >> "$GITHUB_ENV"
|
|
|
|
- name: Install pre-commit
|
|
run: python -m pip install --disable-pip-version-check pre-commit==4.2.0
|
|
|
|
- name: Install actionlint
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
ACTIONLINT_VERSION="1.7.11"
|
|
archive="actionlint_${ACTIONLINT_VERSION}_linux_amd64.tar.gz"
|
|
base_url="https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}"
|
|
# GitHub release downloads occasionally return transient 5xx responses.
|
|
# Retry all curl errors here so workflow-sanity does not fail closed on
|
|
# a one-off release edge outage.
|
|
curl --retry 5 --retry-delay 2 --retry-all-errors -sSfL -o "${archive}" "${base_url}/${archive}"
|
|
curl --retry 5 --retry-delay 2 --retry-all-errors -sSfL -o checksums.txt "${base_url}/actionlint_${ACTIONLINT_VERSION}_checksums.txt"
|
|
grep " ${archive}\$" checksums.txt | sha256sum -c -
|
|
tar -xzf "${archive}" actionlint
|
|
sudo install -m 0755 actionlint /usr/local/bin/actionlint
|
|
|
|
- name: Lint workflows
|
|
run: actionlint
|
|
|
|
- name: Audit all workflows with zizmor
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
mapfile -t workflow_files < <(
|
|
find .github/workflows -maxdepth 1 -type f \( -name '*.yml' -o -name '*.yaml' \) | sort
|
|
)
|
|
pre-commit run --config "${PRE_COMMIT_CONFIG_PATH:-.pre-commit-config.yaml}" zizmor --files "${workflow_files[@]}"
|
|
|
|
- name: Disallow direct inputs interpolation in composite run blocks
|
|
run: python3 scripts/check-composite-action-input-interpolation.py
|
|
|
|
- name: Disallow tracked merge conflict markers
|
|
run: node scripts/check-no-conflict-markers.mjs
|
|
|
|
generated-doc-baselines:
|
|
if: github.event_name == 'workflow_dispatch'
|
|
runs-on: ubuntu-24.04
|
|
steps:
|
|
- *workflow_sanity_checkout_step
|
|
- name: Setup Node environment
|
|
uses: ./.github/actions/setup-node-env
|
|
with:
|
|
install-bun: "false"
|
|
|
|
- name: Check config docs drift statefile
|
|
run: pnpm config:docs:check
|
|
|
|
- name: Check plugin SDK API baseline drift
|
|
run: pnpm plugin-sdk:api:check
|
|
|
|
- name: Check SQLite sessions/transcripts schema baseline drift
|
|
run: pnpm sqlite:sessions-schema:check
|
|
|
|
- name: Check plugin SDK surface budget
|
|
run: pnpm plugin-sdk:surface:check
|