Retention & WAL Maintenance

Operator-driven expiry of old events and reclamation of dead write-ahead-log rows — per-scope rules, dry-run sweeps, and the WAL census.

The retention and compaction of the production write-ahead log are driven by operator routes rather than a background policy you cannot see. Mutating sweeps are irreversible, so every mutating call defaults to dry_run: true and you opt in to apply. What is removed depends on the policy mode (raw sources only versus sources and learned layers) — see Modes below.

Retention is a delete path

A retention sweep that runs with dry_run: false expires aged material according to each scope's mode. With mode: "erase" it removes expired events from the WAL (and, when CORTEX_RETENTION_COMPACT_AFTER_SWEEP is on, reclaims their rows). With mode: "keep_learned" (the default when a newly stored policy omits mode) raw sources expire while learned facts/beliefs can remain. Take a verified backup before the first non-dry run.

Retention rules

A rule is stored per scope and applies to that scope's events. Age is measured from when the event was recorded, not from its observed_at: an event written today with an observed_at in 2025 is not expired by a one-day rule.

PUT /v1/admin/retention/policies
{ "scope": "org:acme", "default_retention_days": 30, "mode": "keep_learned", "per_source_days": { "source:slack": 7 } }
→ 200 { "scope": "org:acme", "policy": { … }, "cursor_reset": true, "effective_mode": "keep_learned" }

GET    /v1/admin/retention/policies?scope=org:acme   → 200 { scope, policy, effective_days, effective_mode }  |  404 NOT_FOUND
DELETE /v1/admin/retention/policies?scope=org:acme   → 200  |  404 NOT_FOUND

default_retention_days applies to every event in the scope. per_source_days overrides it per connector; keys must be source:<system> labels (the label connectors stamp on their events). A bare key such as "slack" is refused: 422 INVALID_RETENTION_POLICY: per-source key "slack" is not a source:<system> label.
mode chooses what expiry removes (see below). If you omit it when storing a policy, the server selects keep_learned.
The body is strict: an unknown field (for example max_age_days) is a 422.
Storing a rule resets the sweep cursor for that scope (cursor_reset: true) so the next tick rescans it.
Inheritance: a child scope with a longer TTL (or a "keep") does not override a finite parent TTL in the effective policy used by the sweep. Check effective_days / effective_mode on GET after you store parent and child rules.

Modes

Mode	Aged raw sources	Learned layers (facts, beliefs, …)	Typical read after expiry
`keep_learned` (default for a newly stored policy without `mode`)	Expired / removed	Kept when derived from those sources	Source reads often 410; fact ids can still resolve
`erase`	Removed	Removed with the sources	Source and fact reads 404

These are different supported behaviours, not a failed erasure. If you need GDPR-style removal of learned information as well as sources, set "mode": "erase" explicitly and confirm with a dry-run sweep on an isolated copy first.

The sweep

Always send a JSON body. An omitted body is 400 INVALID_BODY (EOF while parsing), not a dry run.

POST /v1/admin/retention/sweep
{ }                            # dry_run defaults to true — do not omit the body
→ 200 { dry_run: true, rescan, trigger: "api", cursor_before, cursor_after, watermark, scanned, expired,
        erased, kept_forever, held: { legal_hold, unindexed }, scopes: [...], batches,
        stopped: "no_policy" | "youngest_reached" | …, errors: [], compaction, elapsed_ms }

POST /v1/admin/retention/sweep
{ "dry_run": true }            # equivalent explicit preview; never mutates

stopped tells you why the tick ended: no_policy when no scope has a rule, youngest_reached when the scan reached events newer than any rule. held counts rows that were due but kept: under a legal hold, or not yet indexed (an event still captured is never expired). Pass { "dry_run": false } to apply; the response shape is identical, with erased and compaction filled in. Per-scope entries in scopes[] report the mode that was applied.

A scheduler also runs a tick every CORTEX_RETENTION_SWEEP_INTERVAL_SECS (default 3600); with no rules stored a tick stops immediately with no_policy. Set the interval to 0 for operator-triggered sweeps only.

GET /v1/admin/retention
→ 200 { enabled, scheduler_disabled, deployment_default_days, sweep_interval_secs,
        max_events_per_tick, tick_budget_secs, compact_after_sweep, policies: [...],
        cursor, watermark, running, last, totals: { ticks, scanned, expired, erased, held, errors, … },
        wal: { live_count, … } }

The WAL census and compaction

GET /v1/admin/wal
→ 200 { live_count, next_offset, head_offset, captured_head_offset, partitions,
        consumer_offsets: { indexer: n, … }, retired_marker_count, native_profile_blocked, backend: "unified" }

POST /v1/admin/wal/compact
{ "dry_run": true }            # the default
→ 200 { scanned, removed_tombstones, removed_orphaned_twins, skipped_in_use, skipped_above_bound,
        bytes_removed, live_count_before, live_count_after, ranges_compacted, dry_run,
        truncated_by_budget, elapsed_ms }

Compaction only reclaims rows that are already dead (tombstoned by a forget, an erasure or a retention sweep). It never expires anything itself, and it skips rows a consumer still needs (skipped_in_use) or that sit above the consumer floor (skipped_above_bound). Large erasures batch the full-text-index commit rather than committing per document.

Configuration

The defaults are as printed in the boot config_lint dump.

Env var	Default	What it controls
`CORTEX_RETENTION_DEFAULT_DAYS`	unset	Deployment-wide default when a scope has no rule (falls back to `cortex.toml` `[governance] default_retention_ttl_secs`).
`CORTEX_RETENTION_SWEEP_INTERVAL_SECS`	`3600`	Scheduler period. `0` = operator-triggered sweeps only; minimum 60 when non-zero.
`CORTEX_RETENTION_MAX_EVENTS_PER_TICK`	`10000`	Expired rows decided per tick (legally held rows included).
`CORTEX_RETENTION_TICK_BUDGET_SECS`	`300`	Wall-clock ceiling on one tick's scan.
`CORTEX_RETENTION_COMPACT_AFTER_SWEEP`	`true`	Compact dead rows after a tick that erased anything.

CLI and SDK

cortexdb-cli exposes the same surface under cortexdb admin retention (show, set, sweep, delete) and cortexdb admin wal (status, compact); they are sub-groups of admin, not top-level commands. The Python SDK (sync client only) has retention_status / retention_policy_get / retention_policy_set / retention_policy_delete / retention_sweep and wal_status / wal_compact. See CLI and Python SDK.

Erasures — selector-driven GDPR deletion.
Forget — operational deletion with preview.
Storage & Cluster — the WAL, snapshots and the extraction cache.
Audit Trail — every erasure is recorded.