Current

Sections and incremental fact scans

Benchmark first

Corpora: QASPER · WikiSection · DocRED. Connectors use recorded provider fixtures because public content corpora do not test cursors, deletes, retries, or event ordering.

Protocol: Score text preservation, section Boundary F1/Pk/WindowDiff, entity/relation F1, stable-ID rate, duplicate rate, and incremental/full-sync equivalence. For polling and streaming, replay the same create→update→delete trace with injected duplicates, reordering, throttling, and cursor expiry; the final snapshot and emitted change set must agree.

document_sections maps Markdown headings to stable section IDs and content revisions. section_revisions builds the current revision map. Fact scans can then skip unchanged sections.

How it works

Scan Markdown heading lines, treating content before the first heading as a preamble. Normalize each heading into a slug and suffix collisions deterministically. Store absolute body offsets and hash the section body into its revision. pending_fact_sections compares (document_id, section_id) → revision with the last committed scan and yields new or changed sections only. Persist new scan revisions only after extracted facts commit, or a failed run would incorrectly suppress retry.

fact_scan.py

from mari_components.knowledge import (
    document_sections, fact_scan_revisions, pending_fact_sections,
)

sections = document_sections(document)
pending = pending_fact_sections([document], previous_scan_revisions)
facts = [parse_facts([document], model(section.body)) for section in pending]
next_revisions = fact_scan_revisions(pending)  # persist only after facts commit

Research and standards

Build Systems à la Carte: change detection and recomputation RFC 6920: digest-based content identity

Markdown heading segmentation and slug collision rules are Mari engineering contracts.