Benchmarking

This page contains the exact configuration used to produce the numbers in the benchmark paper — 93.8% on LongMemEval-S (469/500) and 91.8% on LoCoMo categories 1–4 (1,413/1,540).

Why a dedicated benchmark config

The default production configuration is not the benchmark configuration. Three things differ, all driven by an interaction between background maintenance and long-running evaluations:

• Background scheduler is off. Over a ~100-min run the scheduler ticks ~20 times and emits C: / P: summary chunks that pollute the vector index — appropriate in production, noise during a benchmark where each question gets its own scope.
• Per-question scopes. Each LongMemEval-S question loads into a fresh ws:bench_qN scope (independent measurement). Production uses a few persistent scopes per tenant.
• text-embedding-3-small (1536 d). The larger model gains ~+0.4 pp at ~3× cost.

Everything else is at compiled defaults.

The benchmark cortex.toml

The benchmark template ships at benchmarks/cortex_benchmark.toml in the source repo; copy it to <data_dir>/cortex.toml. Its entire content:

# Benchmark override config. Disables the background scheduler so compaction
# doesn't attempt cross-tenant abstractions during LongMemEval runs
# (one tenant per question). Empty sections use all compiled defaults.
[cluster]
node_id = 1

[storage]
[engine]
[network]
[llm]
[governance]

[scheduler]
enabled = false

The single non-default field is scheduler.enabled = false.

Environment variables for the run

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...

# Pin the models that produced the published number
export CORTEX_EMBEDDING_MODEL=text-embedding-3-small
export CORTEX_EMBEDDING_DIMS=1536
export CORTEX_ANSWER_PROVIDER=anthropic
export CORTEX_ANSWER_MODEL=claude-opus-4-6

# Belt-and-suspenders: also disable the scheduler via env
export CORTEX_SCHEDULER_DISABLE=1

No recall-tuning overrides — the published numbers come from the compiled defaults for retrieval top_k, RRF k, graph weight, HyDE schedule, and multihop config. Change any of those and you're producing a new number, not reproducing ours.

Hardware

Published on AWS m6i.4xlarge (16 vCPU, 64 GB RAM, NVMe), us-east-1, Ubuntu 22.04. The OpenAI / Anthropic APIs are the wall-clock bottleneck — accuracy is hardware-invariant.

If you see lower accuracy on smaller hardware, something else is off — check you're not on mock embeddings and the scheduler is actually disabled.

Running LongMemEval-S

500 questions across 5 categories. The harness lives in benchmarks/longmemeval/.

git clone https://github.com/cortexdb/cortex && cd cortex

mkdir -p cortexdb_data_server
cp benchmarks/cortex_benchmark.toml cortexdb_data_server/cortex.toml
cargo run --release --bin cortexdb -- 3141 cortexdb_data_server &

# Wait for readiness (not /health — readiness verifies storage + reports the
# pinned embedding provider; degraded=true means mock embeddings → garbage run)
until curl -sf http://localhost:3141/v1/admin/ready > /dev/null; do sleep 1; done

cd benchmarks
uv venv && source .venv/bin/activate
uv pip install -e .
python -m longmemeval.run \
  --base-url http://localhost:3141 \
  --dataset s \
  --output results/lme_s_$(date +%Y%m%d).json

python -m longmemeval.score results/lme_s_$(date +%Y%m%d).json

Expected:

LongMemEval-S Results
─────────────────────
Total: 469/500 (93.8%)

By category:
  single-session-user:      94/100  (94.0%)
  single-session-assistant: 91/100  (91.0%)
  temporal-reasoning:       97/100  (97.0%)
  multi-session:            93/100  (93.0%)
  knowledge-update:         94/100  (94.0%)

(Total may vary by ±2 questions on re-runs due to LLM output non-determinism.)

Running LoCoMo

A 10-conversation, 1,540-question multi-session benchmark. The published 91.8% covers categories 1–4 (category 5 needs entity-graph features not in the recall surface yet).

cd benchmarks
python -m locomo.run --base-url http://localhost:3141 --output results/locomo_$(date +%Y%m%d).json
python -m locomo.score results/locomo_$(date +%Y%m%d).json --categories 1,2,3,4
# → Total: 1413/1540 (91.8%)

Cost per run

Beyond 93.8% — what's the ceiling?

The benchmark configuration already runs every memory layer (Facts, Beliefs and Understanding need enrichment on a self-host; see Self-hosting defaults). What's opt-in (individual A/B deltas; the full bundle hasn't been run end-to-end):

Internal expectation for the full stack is +1 to +3 pp over 93.8%, but the published headline is the lower, measured number — the config we'd recommend to most users, not a leaderboard-only build. The full bundle is documented at Profiles → Max-Recall.

Reproducibility caveats

• LLM non-determinism — expect ±2 questions of run-over-run variance on the same config; ±5+ means something is wrong.
• Embedding cache — the in-process LRU makes a second run ~30% faster (accuracy identical); restart between formal wall-clock runs.
• Scheduler — forget to disable it and accuracy degrades ~10–20 pp, clustering in single-session-assistant. The startup log CORTEX_SCHEDULER_DISABLE set — background scheduler disabled is your confirmation.
• Model drift — the number was produced against claude-opus-4-6 / gpt-4o-mini at their May-2026 weights.

Comparing to other memory layers

Run all systems on the same hardware (for latency parity), each with its published config, the same answer model, and the public eval script (longmemeval/run.py). Don't tune one and not the others.

Custom benchmark

If LongMemEval-S doesn't reflect your workload, build a custom benchmark on the same harness shape and sweep the Recall Tuning knobs:

for top_k in 20 40 80 160; do
    CORTEX_GRAPH_RETRIEVAL_TOP_K=$top_k python my_bench.py > results_$top_k.txt
done

Next steps

• Profiles & Presets — the Benchmark-validated profile as a copy-paste block
• Recall Tuning — every knob the recall pipeline exposes