Current
Long-horizon memory evaluation
Benchmark first
Corpora: BEIR, FEVER, and LongMemEval form the end-to-end matrix; deterministic fixture suites cover stores, serialization, compiler output, and failure recovery.
Protocol: Pin every input, dependency, model, prompt, and seed. Preserve per-case traces and report quality beside p50/p95 latency, tokens, storage, index-build time, refresh work, and failure count. Compare outputs after restart and across supported stores; compiled configurations must reproduce the same artifact graph and evaluator inputs.
A memory system needs separate measurements for writing, updating, retrieving, temporal reasoning, abstaining, and respecting context limits. One aggregate answer score hides which subsystem failed.
How it works
Freeze conversations, source revisions, expected evidence, and change events into cases. Replay each case under several corpus sizes and context budgets. Score extraction at write time, evidence recall at query time, cross-session synthesis, ordering and date reasoning, correction after source updates, abstention when evidence is absent, and ACL leakage. Store per-stage traces with every score so a regression points to the responsible parser, index, policy, or packing decision.
Extractionprecision · recall · evidence
Retrievalrecall@k · rank · authorization
Synthesissupport · completeness
Temporalorder · valid time · updates
Abstentionunsupported-answer rate
Operationstokens · latency · refresh work
from datetime import datetime, timezone
from mari_components.evaluation import EvaluationRun, load_longmemeval_cases
cases = load_longmemeval_cases("data/longmemeval_s.json")
metrics = evaluate_memory_cases(system, cases) # application-owned execution
run = EvaluationRun(
run_id="memory-main-0042",
corpus_id="longmemeval",
corpus_revision="sha256:...",
split="longmemeval_s",
mari_revision="git:...",
started_at=datetime.now(timezone.utc),
configuration={"context_budget": 8_000},
model_identifiers=("reader@2026-08",),
seed=7,
metrics=metrics,
)
Paper
LongMemEval: five long-term memory capabilities
The adapter, deterministic metrics, run identity, and hard regression gates are implemented. System execution remains an injected application callback.