Current

Evaluation and compilation

Benchmark first

Corpora: BEIR, FEVER, and LongMemEval form the end-to-end matrix; deterministic fixture suites cover stores, serialization, compiler output, and failure recovery.

Protocol: Pin every input, dependency, model, prompt, and seed. Preserve per-case traces and report quality beside p50/p95 latency, tokens, storage, index-build time, refresh work, and failure count. Compare outputs after restart and across supported stores; compiled configurations must reproduce the same artifact graph and evaluator inputs.

compile_configurations searches caller-supplied pipeline and retrieval configurations against knowledge-system objectives and returns the highest-utility feasible candidate with every trial visible.

How it works

Declare tunable parameters, hard constraints, and optimization metrics. For each candidate configuration, run the same frozen training cases, cache stage results by configuration/input fingerprints, reject any candidate that violates provenance, update fidelity, or ACL constraints, and rank feasible candidates on grounded recall, cost, and latency. Validate the selected configuration once on held-out cases; compilation returns a report and proposal, never a deployment side effect.

Research basisDSPy compiles parameterized LM pipelines against a declared metric. Mari generalizes the search space to retrieval, indexing, parsing, graph, consolidation, and packing configuration. Hard provenance, update-fidelity, and ACL constraints are Mari requirements and must be evaluated independently.

Grounded recallmaximize

Provenance accuracyrequire 1.0

Update fidelityrequire 1.0

ACL leakagerequire 0.0

Context tokensminimize

Latency p95minimize

Constraint-first configuration search

from mari_components.platform import (
    MetricObjective,
    ObjectiveDirection,
    compile_configurations,
)

def evaluate(config):
    return benchmark(index=config["index"], k=config["k"])

compiled = compile_configurations(
    [
        {"index": "bm25", "k": 20},
        {"index": "hnsw", "k": 40},
    ],
    evaluate=evaluate,
    objectives=[
        MetricObjective(
            name="grounded_recall",
            direction=ObjectiveDirection.MAXIMIZE,
            minimum=0.85,
        ),
        MetricObjective(
            name="acl_leakage",
            direction=ObjectiveDirection.MINIMIZE,
            maximum=0.0,
        ),
        MetricObjective(
            name="latency_p95",
            direction=ObjectiveDirection.MINIMIZE,
            weight=0.01,
        ),
    ],
)
print(compiled.configuration, compiled.winner.metrics)