Current
Verification portfolios
Benchmark first
Corpora: FEVER · FEVEROUS · ContraDoc · FreshQA
Protocol: Report verdict macro-F1 and accuracy separately from evidence precision/recall/F1 and contradiction localization. Pin corpus and source revisions. Replay corrections and removals, then measure stale-answer rate, time-to-consistency, unsupported-answer rate, and authorization leakage. A correct label without the complete evidence set does not pass the evidence contract.
Verification functions score already-parsed values and retain all successful attempts and failures for audit.
How it works
best_of_n calls the producer up to n times, parses each output, records parse failures without discarding successful siblings, scores valid candidates, and selects the highest score with stable first-wins ties; it may stop once the threshold is met. verdict_consensus counts typed verdicts rather than free text. Grounding scores combine declared deterministic components; they are used for ranking or abstention, never calibrated as probabilities.
from mari_components.verification import best_of_n, score_grounded
result = best_of_n(
lambda: model(question, documents),
lambda raw: parse_answer(question, documents, raw),
lambda answer: score_grounded(answer,
required_ideas=("eligibility", "time limit")),
attempts=3, threshold=0.90)
audit(result.selected, result.attempts, result.failures, result.stopped_early)
select_best
Scores existing candidates with stable tie-breaking.
verdict_consensus
Aggregates supported, contradicted, and uncertain assessments.
score_grounded
Evidence, coverage, completeness, corroboration, certainty.
harmonic_score
idea_completeness
**Scores are not truth probabilities.**They are deterministic quality signals for selection and abstention.
Research basis
Self-consistency improves chain-of-thought reasoningFEVER: evidence-based verificationALCE: citation quality evaluation
Mari exposes an auditable selection portfolio; it does not reproduce model sampling or benchmark metrics.