Current

Trajectories and agent evaluation

Benchmark first

Corpora: KILT, QASC, and LongMemEval, plus repository-owned golden tool traces for workflow replay.

Protocol: Separate retrieval/evidence quality from agent execution. Report task success, exact and partial tool-sequence match, argument validity, tool-error recovery, grounded-output rate, trajectory cost, and reused-procedure uplift. Evaluate mined procedures on held-out tasks and reject any train/test source overlap.

normalize_steps converts runtime records into privacy-bounded TrajectoryStep values. parse_trajectory_analysis validates model-proposed phases. Mari provides adapters, not an agent loop.

How it works

Adapters map framework events into ordered AgentEvent values. Normalization assigns stable step positions, keeps allowlisted metadata, and redacts sensitive argument names and transport fields. Tool evaluation compares observed names and counts against expectations; outcome evaluation compares terminal paths and completion. A proposed phase analysis must cover every observed event exactly once with contiguous, non-overlapping ranges and known tool families.

inspectsearch_knowledge

reason2 docs · 2 citations

answeroutcome: resolved

agents.py

from mari_components.agents import evaluate_outcome, evaluate_tools
from mari_components.trajectories import normalize_steps

steps = normalize_steps(runtime_events)
tools = evaluate_tools(events, expected_tools=("search_knowledge",))
outcome = evaluate_outcome(paths=("resolved",),
    expected_paths=("resolved",), completed=True)

AgentEvent and EventKind are framework-neutral. Optional adapters cover OpenAI Agents and LangChain/LangGraph. Normalization redacts common sensitive arguments; phase validation requires the returned ranges to cover observed events exactly.

trajectory_analysis.py

from mari_components.trajectories import parse_trajectory_analysis

analysis = parse_trajectory_analysis(normalized_events, model_labels,
    family_map={"search_product_knowledge": "inspect",
                "answer": "answer"})

Research and standards

AgentBench: multi-environment agent evaluation OpenTelemetry trace specification

Mari’s event schema, redaction list, exact phase coverage, and outcome predicates are library contracts.