Current

Evidence contracts

Benchmark first

Corpora: FEVER · FEVEROUS · ContraDoc · FreshQA

Protocol: Report verdict macro-F1 and accuracy separately from evidence precision/recall/F1 and contradiction localization. Pin corpus and source revisions. Replay corrections and removals, then measure stale-answer rate, time-to-consistency, unsupported-answer rate, and authorization leakage. A correct label without the complete evidence set does not pass the evidence contract.

An evidence record is a byte-for-byte quotation bound to the exact document and section revision that was supplied to a parser. It is provenance, not a model confidence score.

How it works

  1. Restrict the corpus. Build an allowed map from only the KnowledgeDocument values passed by the caller. A model cannot cite an ID outside that map.

  2. Resolve the document. Require document_id. If exactly one allowed document contains the quote, Mari may recover a missing ID; zero or multiple holders is rejected.

  3. Match exact text. Require a non-empty quote and test literal containment in the canonical document body. Fuzzy, normalized, or semantic matches are not accepted.

  4. Resolve one section. Split the document into current sections and find sections containing the quote. A repeated quote spanning multiple sections is rejected unless section_id selects exactly one.

  5. Derive coordinates. Mari computes start = section.start + section.body.index(quote) and end = start + len(quote); it does not trust model-supplied offsets or revisions.

  6. Bind revisions. The accepted record receives the current document.revision, stable section_id, and content-derived section.revision. These become the invalidation key.

refunds.md · rev 8f31c2a

Enterprise refunds close after 30 days.

*literal containment\

  • unique section*

Evidencedocument_id = …refunds.md``revision = 8f31c2a``quote = "30 days"``start = 31 · end = 38``section_id = enterprise``section_revision = sha256:…

evidence.py
from mari_components.knowledge import parse_facts

raw = {"facts": [{
    "claim": "Enterprise refunds close after 30 days.",
    "evidence": [{"document_id": doc.document_id,
                  "section_id": "enterprise",
                  "quote": "30 days"}],
}]}
fact = parse_facts([doc], raw)[0]
e = fact.evidence[0]
assert doc.body[e.start:e.end] == e.quote

# Rejected: unknown document, absent quote, or a repeated quote
# whose section cannot be selected unambiguously.

Dependency conversion

evidence_dependencies projects each record to (document_id, document_revision, section_id, section_revision), deduplicated by (document_id, section_id) and returned in stable order. Two records naming different revisions of the same key raise ValueError; silently choosing one would make reuse nondeterministic.

What this proves—and does not proveThe contract proves that the quoted characters occurred in one supplied source revision and records where. It does not prove entailment, source authority, completeness, or truth. Those require claim assessment, corroboration, authorization, and review.

Research and standards

ALCE: citation correctness and completenessQASPER: evidence-bearing document QAFActScore: atomic factual claimsFEVER: evidence-backed verdictsW3C PROV: quotation, derivation, and revision

These works motivate inspectable evidence and revision provenance. Literal substring validation, unique-section resolution, and failure behavior are Mari engineering contracts.