Current

Synchronization

Benchmark first

Corpora: QASPER · WikiSection · DocRED. Connectors use recorded provider fixtures because public content corpora do not test cursors, deletes, retries, or event ordering.

Protocol: Score text preservation, section Boundary F1/Pk/WindowDiff, entity/relation F1, stable-ID rate, duplicate rate, and incremental/full-sync equivalence. For polling and streaming, replay the same create→update→delete trace with injected duplicates, reordering, throttling, and cursor expiry; the final snapshot and emitted change set must agree.

plan_sync compares a durable SyncState with one PollPage and returns a side-effect-free SyncPlan. stream_sync applies the same rules across pages.

How it works

For each upsert, Mari validates source ownership and compares a deterministic content fingerprint with the manifest: equal means unchanged; unequal means upsert. Explicit tombstones always become deletes. Absence becomes deletion only after the terminal page of an authoritative full snapshot. The returned plan carries the prior generation as a compare-and-swap precondition and the next manifest/cursor as proposed state; persistence must atomically commit both data and state.

Startgeneration 41

→

Pagesupsert · tombstone · unchanged

→

Complete?no: preserve missing docs

→

Reconcileyes: absence may delete

→

CommitCAS generation 42

sync.py
from mari_components import SyncMode
from mari_components.sync import SyncState, plan_sync

state = load_state() or SyncState()
for page in provider_pages:
    plan = plan_sync(state, page,
        source_id="github:acme/product", mode=SyncMode.FULL)
    store.commit(upserts=plan.upserts, deletes=plan.deletes,
        state=plan.state, expected_generation=plan.expected_generation)
    state = plan.state

Enforced invariants

  • Page replay is idempotent through content fingerprints and manifests.

  • Only terminal, authoritative full pages reconcile absence.

  • Explicit tombstones apply in full and incremental modes.

  • Incomplete full sync cannot resume as incremental.

  • Generation compare-and-swap prevents concurrent state loss.

  • Foreign source IDs, duplicate IDs, and upsert/delete overlap are rejected.

Research basis

Build Systems à la Carte: fingerprints and minimal rebuildsDynamo: versioning and reconciliation

Snapshot authority, deletion rules, and atomic compare-and-swap are Mari’s connector/store contract.