Algorithm choices¶
Run independent algorithms¶
Use fixture callbacks to compare algorithm choices. Install the optional solver extra to include native graph and centroid-linkage operations.
python -m examples.algorithm_choices_demo
pip install 'mari-kit[algorithm-solvers]'
python -m examples.algorithm_choices_demo --solvers
Mari Kit is a collection of independently selectable algorithms for knowledge and memory workloads. This guide covers the whole package: retrieval, subset selection, parsing, memory, graph analysis, agent experience, maintenance, verification, and evaluation. Similar methods remain available because their tradeoffs differ. Each workload table names the public implementation and links to its detailed behavior and sources.
The complete module and API index
accounts for every public implementation module, including the contracts,
connectors, and adapters that support algorithms. Module paths in the tables are
relative to mari_kit. The index links definitions to inspected source
lines. A source citation identifies provenance or a related method. Individual
pages explain which computations Mari implements and which model calls the host
supplies.
Find a choice by workload¶
Workload |
Compare |
|---|---|
Lexical, sparse, dense, approximate, multi-vector, and graph retrieval |
|
Relevance, MMR, coverage, diversity, cost, and source structure |
|
HyDE, RAPTOR, MemWalker, CRAG, FLARE, Self-RAG, and DRIFT |
|
Structured parsers, semantic atoms, diff algorithms, and byte chunking |
|
Admission, mutation, salience, heat, LFU, consolidation, and evolution |
|
Paths, communities, entity resolution, linkage, time, and provenance |
|
Episodes, intents, process mining, comparisons, procedures, and feedback |
|
Snapshot planning, indexed updates, lineage, reversible aggregates, and replay |
|
Grounding, contradiction, consensus, trust, and authority |
|
Retrieval, graph, memory, statistical, and configuration comparisons |
|
Source adapters, identity, stores, schemas, and portability |
Retrieve candidates¶
Choice and public implementation |
Useful workload and tradeoff |
Details and sources |
|---|---|---|
|
Exact terms, identifiers, and symbols with positive-IDF scoring. Artifact/revision wrappers preserve their respective identities. |
|
|
Compare rank_bm25 Okapi, L, and Plus formulas for length/repetition effects. L/Plus can score nonmatches above zero. |
|
|
Exact search over caller-supplied sparse weights. The host supplies lexical or learned sparse representations. |
|
|
Exact vector-search baseline for bounded corpora. Search examines the candidate vectors. |
|
|
Approximate graph search or coarse partitioning with product quantization. Measure recall against the flat index. |
|
|
Fixed-dimensional candidate generation for token-level multi-vector retrieval followed by exact late-interaction reranking. |
|
|
Mari’s specific packed block-2 Polar codec for MUVERA FDE candidate scoring. Quantization trades representation detail for compactness. |
|
|
Aggregate atom hits to parent sections or score section token vectors. Parent aggregation and true MaxSim answer different scoring questions. |
|
|
Retrieve same-topic contradiction candidates using similarity and difference sparsity. The training-loss helper leaves model training external. |
|
|
Propagate caller seed weights through allowed topology and project node scores to passages. |
|
|
Build fact/entity/dense seeds before propagation, or perform one entity/semantic/causal expansion step. |
|
|
Combine ranked lists whose score scales differ. Rank fusion discards score magnitude. |
|
|
Score a union using one embedding space or one shared reranker. Explicit source identity handles cross-collection results. |
|
|
Prefer supplied graph locality or episode frequency, with explicit handling of missing values. |
Select useful context¶
Choice and public implementation |
Useful workload and tradeoff |
Details and sources |
|---|---|---|
|
Balance query relevance against similarity to selected items. A direct option for repetitive search results. |
|
|
Select artifact-neutral items with budgets, constraints, and explicit decision traces. Diversity-aware selection exposes candidate gains per round. |
|
|
Represent the candidate collection by maximizing best-covered similarity per represented row. Optional query caps focus coverage. |
|
|
Favor distinct vector directions when selecting memories or evidence. Relevance and mandatory retention constraints need explicit treatment. |
|
|
Cover caller-defined concepts with binary or probabilistic incidence and optional concept weights. |
|
|
Select naive, lazy, stochastic, or lazier greedy independently of the objective. Lazy modes require submodularity. |
|
|
Resolve IDs and assemble bounded source content with exclusions and missing-hit traces. |
|
|
Expand retrieved atoms into neighboring context or preserve section structure, exact spans, and original hits. |
|
|
Allocate descending per-parent chunk quotas and redistribute unused quota. Maximum denotes a parent quota. |
|
|
Compare caller-scored sentence selection with token-surprisal word selection. Both produce extractive content. |
|
|
Preserve evidence groups through compaction and bind image/page/media assets to retained evidence. |
Log-determinant and facility location answer different questions. Log-determinant rewards distinct contributions in the selected feature space. Facility location rewards representation of the broader supplied collection. Explicit set cover is useful when the host can name the requirements it needs covered. MMR combines relevance and redundancy directly. None of these choices establishes factual correctness or independent corroboration from embeddings alone.
Adapt a retrieval session¶
Choice and public implementation |
Useful workload and tradeoff |
Details and sources |
|---|---|---|
|
Embed a generated hypothetical answer/document to change the query representation. Generation and embedding callbacks belong to the host. |
|
|
Build a recursive summary hierarchy and navigate it with bounded caller decisions. |
|
|
Select corrective retrieval actions from relevance signals or additional queries from uncertain generated content. |
|
|
Combine caller-supplied reflection signals to rank a candidate. Reflection generation and the complete Self-RAG model remain external. |
|
|
Prime questions, expand ranked follow-ups within budgets, and reduce completed answers. |
|
|
Check explicit requirements, propose gap queries, and measure utility relative to consumed context. |
|
|
Apply conditional disclosure, progressively expand source detail, or select a lifecycle intervention. Authorization remains a separate host input. |
|
|
Attach validated context to chunks or pool supplied token vectors across source spans. |
Parse and align knowledge¶
Choice and public implementation |
Useful workload and tradeoff |
Details and sources |
|---|---|---|
|
Parse text structure and records into source-addressable values. Format-specific fidelity is described in the parser guides. |
|
|
Extract Python symbols/references and find symbols affected by source changes. |
|
|
Normalize table structure or adapt supplied Docling JSON with text, cells, page regions, and references. |
|
|
Use stable structural units, text-oriented content-defined fallback spans, and temporal versions. Text span boundaries differ from byte FastCDC. |
|
|
Compare minimum-edit alignment with unique-anchor alignment for changed sources. |
|
|
Deduplicate byte streams with read-pattern-independent content-defined boundaries and owned chunks. |
|
|
Validate caller-generated structured knowledge. Parsing a proposed claim provides a different guarantee from proving it. |
|
|
Run bounded glean passes with explicit identity and merge policies. |
|
|
Select source sections/excerpts, assign normalized tags, and derive explicit links. |
Organize memory¶
Choice and public implementation |
Useful workload and tradeoff |
Details and sources |
|---|---|---|
|
Accept, defer, reject, or quarantine from explicit admission signals and thresholds. |
|
|
Validate add/update/delete/no-op decisions against existing memories. The host supplies decisions and durable transactions. |
|
|
Segment topic changes and choose bounded offline work from utility/cost signals. |
|
|
Combine recency, importance, and relevance using the existing salience policy. |
|
|
Compare linear/exponential/neutral freshness and proportional temporal/proof boosts to relevance. |
|
|
Raw visit/interaction/recency heat, least-frequency eviction, and above-threshold promotion are independent policies. |
|
|
Compare the existing note-link plan with callback-proposed context/tag/link updates guarded by expected revisions. |
|
|
Decide an answer source from sequential evidence-note judgments supplied by the host. |
|
|
Reduce helpful/harmful/neutral feedback and explicit keep/update/delete/merge decisions with provenance. |
|
|
Propose scope changes and retention actions under caller policies. These are planning decisions with host-owned persistence. |
Analyze and construct graphs¶
Choice and public implementation |
Useful workload and tradeoff |
Details and sources |
|---|---|---|
|
Reachability, paths, edge-aware traversal, and alternate predecessor explanations. |
|
|
Identify connectivity, cycles, or strongly connected components and their condensation DAG. |
|
|
Reduce redundant DAG edges or select topological k-core/k-truss membership. |
|
|
Compare bounded expansion, the local greedy prize heuristic, and the native PCST approximation for connecting useful evidence. |
|
|
Rank local connectivity, distance, bridge position, or hub/authority structure. Different centralities encode different notions of importance. |
|
|
Score common-neighbor/Jaccard/Adamic–Adar links or recursive structural similarity. |
|
|
Compare the historical local-modularity heuristic, NetworkX Louvain, and native hierarchical Leiden. Only the native adapter provides full hierarchical Leiden. |
|
|
Generate community reports and reduce report-level query results through bounded callbacks. |
|
|
Combine field-level agreement evidence into match/review/nonmatch decisions. |
|
|
Build blocked pairs, inspect pair provenance, and cluster supplied match links with diagnostics. |
|
|
Learn a predicate disjunction under coverage/cost constraints and acquire informative matcher/blocker disagreements. |
|
|
Compare same-entity clustering, one-to-one greedy assignment, and reusable-right-record lookup. |
|
|
Query valid/transaction time and join interval-bearing records. Interval semantics are explicit. |
|
|
Explain derivation paths, propagate taints, and project graph selections to bound evidence. |
|
|
Inspect structural/field drift, graph quality, and revision families. Diagnostics describe supplied data. |
Learn from agent activity¶
Choice and public implementation |
Useful workload and tradeoff |
Details and sources |
|---|---|---|
|
Compare thread/time segmentation with embedding-based complete-link episode/topic grouping. Compilation uses supplied extraction/consolidation callbacks and revision-bound evidence. |
|
|
Normalize observed activity and preserve trace integrity before comparing behavior. These are format adapters and validators. |
|
|
Segment bounded activity and validate caller assessments/reflections. |
|
|
Mine activity/transition frequencies and compare ordered execution paths. |
|
|
Find observed invariants and choose varied traces for inspection. Observed invariants need held-out validation. |
|
|
Aggregate inferred intents, cluster examples, detect novelty, and compare changing distributions. |
Intent analysis, intent decomposition, Jensen–Shannon divergence |
|
Apply explicit rubrics and compare outcome-associated reasoning memories with uncertainty estimates. |
|
|
Turn supplied activity analysis into a procedure candidate. The callback/evaluation boundary determines quality. |
|
|
Match reviewed workflows or cached answers and choose policy actions with explicit freshness checks. |
|
|
Record what knowledge an agent saw/used, diagnose feedback, and propose bounded edits. |
Maintain changing knowledge¶
Choice and public implementation |
Useful workload and tradeoff |
Details and sources |
|---|---|---|
|
Compare complete snapshot planning with an indexed affected frontier. Inputs, derivations, and completed receipts drive reuse/rebuild decisions. |
|
|
Track changes to selection membership and preserve group identity through split/merge/regrouping using member overlap. |
|
|
Apply reversible keyed updates to counts, vectors, lexical statistics, or membership. Single-writer reference aggregates expose host persistence boundaries. |
|
|
Relate source changes to atom work and evidence freshness. Structural refresh and artifact validity are separate decisions. |
|
|
Plan derived-view refresh or rebuild projections from ordered events. |
|
|
Reconcile complete/incremental source observations and coalesce ordered hints before canonical refetch. |
|
|
Select stale fact-scan sections and plan assertion revision history. |
Assess evidence and decisions¶
Choice and public implementation |
Useful workload and tradeoff |
Details and sources |
|---|---|---|
|
Score supplied evidence signals, select candidate attempts, and aggregate verdicts. Numerical scoring remains separate from claim-generation quality. |
|
|
Aggregate evidence with explicit weights/uncertainty and measure cited support coverage. |
|
|
Validate within-document contradiction evidence and reference-coverage rewards. Cross-source SparseCL retrieval is a separate choice. |
|
|
Check evidence identity, location, regions, and declared citation lifecycles. These checks verify structure and binding. |
|
|
Evaluate supplied trust signals, inherited taints, and explicit authority policies for competing assertions. |
|
|
Validate grouped edits, derivation dependencies/feedback loops, and caller-designed knowledge-file structures. |
|
|
Trace inclusion/exclusion and cross-stage candidate decisions. |
Evaluate a choice¶
Choice and public implementation |
Useful workload and tradeoff |
Details and sources |
|---|---|---|
|
Measure ranking, set coverage, segmentation, and classification under supplied labels. |
|
|
Evaluate path/edge/node recovery, clusters, graph-grounded context, and grouped requirements. |
|
|
Compare paired outcomes, slices, reviewer agreement, and repeated attempts with uncertainty. |
|
|
Check task-level improvements, regression thresholds, and observed tool/outcome behavior. |
|
|
Compare caller-supplied configurations against explicit metric objectives. The host defines the candidate search space and evaluation. |
|
|
Load benchmark cases/catalogs for a consistent comparison. Dataset adapters supply cases rather than algorithm-quality claims. |
Supply integration boundaries¶
These modules are part of the project and appear in the complete index. They supply values, validation, I/O adapters, and execution boundaries rather than alternative ranking objectives.
Area |
Public surfaces and purpose |
Details and project sources |
|---|---|---|
Identity and structure |
|
|
Source adapters |
|
|
Graph and index interchange |
|
|
Contracts and storage |
|
|
Portable bundles |
|
Run and inspect the implementations¶
from mari_kit.algorithms.subsets import LogDeterminant, maximize_subset
from mari_kit.retrieval.indexes import BM25Index, DenseFlatIndex
from mari_kit.retrieval.fusion import maximal_marginal_relevance
from mari_kit.dependencies import plan_dependency_updates
Imports illustrate independent choices. The feature guides
show function signatures and runnable compositions. Use the
complete module/API index
to locate additional helpers, records, methods, and exact source definitions.
Base numerical implementations depend on NumPy. Native graph and centroid
operations are loaded lazily through mari-kit[algorithm-solvers].
Some research-inspired boundaries consume model outputs or callbacks. The library supplies the documented computation, validation, or plan. The host supplies models, data, storage, authorization, and execution policy. Exact local fixtures and published paper results answer different evaluation questions.
Detailed source notes¶
The A01–A21 notes below retain the original stable sections for the September algorithm additions. The workload tables above place those additions alongside the pre-existing choices. Existing families have their detailed source and implementation notes in the linked feature guides.
A01 — BM25 alternatives¶
lexical.BM25VariantIndex requires explicit BM25Variant.OKAPI, .L, or .PLUS
and caller-tokenized documents/query. It returns scores, matched flags, and
per-term explanations. It follows rank_bm25’s formulas, including Okapi’s
negative-IDF epsilon floor and L/Plus nonmatching baselines. The existing
positive-IDF BM25Index remains a separate option.
Choose among them by retrieval evaluation, especially when document lengths or
term repetition vary. Scores are not interchangeable across variants/corpora.
matching_only=True removes baseline-only hits; allowed_ids restricts outputs,
while corpus statistics remain those of the index. Build separate indexes when
statistics themselves must be isolated. Empty corpora return no hits; all-empty
corpora and zero-length normalization yield finite zero/baseline scores instead
of upstream division errors. Query work is proportional to documents × terms.
A02 — Subset objectives¶
subsets.FacilityLocation, SetCover, ProbabilisticSetCover, and
LogDeterminant implement Submodlib objective equations. Each offers
evaluate(subset) and marginal_gain(subset, item) over integer candidate indices.
Facility location uses represented-row × candidate similarities and sums each
row’s best selected similarity. query_similarities adds FL1MI’s cap
eta * max(query similarity) per represented row, allowing query-focused coverage.
Set cover uses binary item × concept incidence; probabilistic cover computes
sum(weight * (1 - product(1-p))). Log determinant computes
log det(K_A + regularization I) for a symmetric PSD kernel and is zero for an
empty selection. Choose representation coverage, explicit concept coverage, or
kernel diversity according to the workload. Nonnegative kernels/weights and
valid probabilities are checked. These dense reference objectives recompute
values: logdet evaluation is cubic in selected size, and initial PSD validation
is cubic in ground-set size. Arbitrary logdet scaling is not necessarily monotone.
See also the project’s Submodlib paper.
A03 — Greedy optimizer choices¶
subsets.maximize_subset accepts any deterministic objective callback, positive
costs, total budget, optional item limit, and GreedyMethod.NAIVE, .LAZY,
.STOCHASTIC, or .LAZIER. These adapt the Submodlib optimizer family.
Results expose selected indices, marginal gains, costs, objective evaluation
counts, and remaining candidates.
Naive recomputes every feasible gain; lazy caches upper bounds; stochastic
samples candidates; lazier combines sampling and bounds. Lazy modes require
assume_submodular=True. Seeded sampling and index tie breaks are reproducible.
The sample size is ceil(n/k * log(1/epsilon)). Sampling guarantees require a
monotone submodular cardinality problem, not arbitrary costs. Gain/cost ranking
is a heuristic for budgets; no knapsack-optimality claim is made. A sampled
nonpositive best gain can stop with unsampled useful candidates remaining.
A04 — Prize-collecting forest solver¶
graphs.prize_collecting_forest adapts pcst_fast to caller node prizes
and edge costs. It returns original IDs/edges and prize/cost totals. A rooted
call fixes its required root; an unrooted call can request multiple clusters.
Choose it when valuable evidence requires intermediate connectors, including
zero-prize nodes. This is the native approximation, not an exact optimal solver.
The existing greedy selector remains useful as a different low-overhead choice.
Optional allowed_nodes induces the input graph before solving.
A05 — Louvain and hierarchical Leiden¶
graphs.louvain_partition invokes NetworkX Louvain;
hierarchical_leiden_partition invokes graspologic-native using the same engine
as GraphRAG’s hierarchical Leiden adapter. Weighted edges, resolution,
seed, and optional allowed nodes are explicit. Leiden returns node/community,
level, parent, and final-membership records. Isolated nodes get singleton records;
max_cluster_size is a splitting target, not a strict guarantee.
Use Louvain for modularity partitions or hierarchical Leiden for multilevel
community workflows. Native library versions can change results despite a fixed
seed. The historical graph.communities.leiden_communities API is preserved and
now explicitly documented as local modularity improvement plus connected
splitting; it does not implement full Leiden aggregation.
A06 — HippoRAG2 seed construction¶
graph_retrieval.hipporag_seed_weights implements the fact/entity/dense passage
seed calculation in HippoRAG’s graph search. Fact scores are divided by
entity passage frequency, averaged over incident facts, and optionally limited
to the best linked entities. Passage scores are min-max normalized and scaled
by passage_weight. The result exposes each contribution separately.
Use this for graph recall combining extracted facts and dense passage evidence.
The caller provides embeddings, fact filtering, entity links, and graph edges.
Allowed IDs are required and filtered before passage normalization. Missing
entity frequencies use one; tied passage scores contribute zero. Feed positive
combined weights into retrieval.personalized_pagerank; choose a fallback when
all weights are zero. This ports seed construction, not HippoRAG2 end to end.
A07 — DRIFT action search¶
search.drift_search adapts GraphRAG DRIFT: a primer supplies queries,
local search returns answers and follow-ups, incomplete actions are selected by
score or seeded randomness, and a reducer combines completed evidence. All model
and retrieval calls are supplied callbacks. The returned trace includes parent
indices, depth, completed/pending actions, and budget/exhaustion status.
Use this for exploratory graph search that can revise its questions. Exact query
strings are deduplicated to prevent cycles. max_actions limits local calls;
primer and reducer each run once outside that count. max_depth stops follow-up
expansion. Callbacks must bound their own result sizes and latency; callback
errors propagate. This is an explicit bounded adaptation, without GraphRAG’s
provider prompts, database layer, or answer-quality claims.
A08 — Weighted chunk polling¶
graph_retrieval.weighted_chunk_polling follows LightRAG’s quota and polling
algorithm. Ordered parents receive descending linear chunk quotas;
unused allocation is redistributed by scanning highest-ranked parents first.
maximum is the highest-ranked parent quota, not a global cap: total allocation
is bounded by the sum of interpolated parent quotas.
It returns chunks, per-parent counts, and initial quotas. Use it when graph
hits have uneven numbers of attached chunks. Parent order and chunk order are
caller policy. Duplicates are retained by default; optional deduplication happens
after allocation and does not backfill, so output may be below budget.
A09 — Extraction gleaning¶
search.refine_extraction adapts LightRAG’s extraction and glean merge
to explicit initial/refinement callbacks, record identity, and merge policy.
It accumulates additions and revisions until the round budget, a continuation
callback, or unchanged merged records stops refinement. Use a longer-description
merge to resemble LightRAG’s merge preference, or supply another policy.
The upstream path performs an extra extraction pass when enabled; Mari explicitly generalizes this to a configurable bounded number of rounds. Records must have stable equality and immutable values; merges must preserve identity. There is no hidden model call, storage mutation, or assumption that an additional pass improves extraction accuracy.
A10 — Temporal and proof ranking¶
temporal.recency_decay, dated_recency, and temporal_proof_score implement
Hindsight’s recency and multiplicative scoring. Select linear decay
with a 0.1 floor, exponential half-life, or neutral 0.5. Date handling uses the
source’s month/year span-length heuristic: age from period end, cap at neutral;
otherwise prefer occurrence start, mention, then end. Naive datetimes mean UTC.
Normalized relevance is multiplied by neutral-centered recency, proximity, and log proof-count factors. Use this to make recency a proportional preference without replacing relevance ranking. Dates/proximity/proof counts come from the host. Invalid windows and reversed intervals raise instead of silently using upstream fallbacks. Span length is a heuristic for coarse dates, not inferred certainty about date granularity. Missing dates and proof signals are neutral.
A11 — Typed link expansion¶
graph_retrieval.expand_typed_links adapts the actual merge computation in
Hindsight link expansion: tanh(0.5 * shared_entity_count) plus maximum
semantic weight plus maximum causal weight. It considers both edge directions,
filters to required allowed_ids, caps each entity’s ordered member list, and
returns separate contributions. Use it for evidence neighborhoods where link
kinds provide distinct signals.
This follows the source’s Python merge, which sums raw maximum causal weights; it does not adopt the different causal-offset description in that file’s opening comment. Authorization, entity membership, link generation, and edge weights are supplied by the host. It is one expansion step, not a graph traversal engine.
A12 — Graph-distance and episode-mention ranking¶
graph_retrieval.rank_graph_distances and rank_episode_mentions are explicit
adaptations of Graphiti’s search rerankers. Distance ranking uses
caller-supplied nonnegative distances, reciprocal score, explicit center score,
and zero for unreachable/missing nodes. Mari does not pretend Graphiti’s
adjacency query calculates all shortest paths.
Episode ranking defaults to descending mention count, with an ascending option; missing counts sort last. This deliberately differs from the inspected upstream ascending frequency sort. Choose distance for locality and episode count for repetition/popularity, then evaluate which direction suits the workload.
A13 — Source-preserving surprisal selection¶
compression.select_surprising_words adapts LightMem’s entropy compressor.
The caller supplies observed-token probabilities, token spans, and word spans.
Word scores aggregate -log2(probability) by mean or first token. The top
max(1, floor(words * fraction)) words are returned in source order with original
spans and scores; an empty input returns empty output.
Use it for inexpensive extractive compression with an external language model. The host must align next-token probabilities correctly. Explicit spans replace upstream tokenizer-specific grouping; cross-boundary tokens contribute to every overlapping word. The joined text is a selection, not a grammatical summary. The implementation scans token spans per word and does not load Torch/models.
A14 — Byte-stream FastCDC¶
compression.fastcdc_chunks ports tigerwill90/fastcdc’s byte boundaries,
including its gear table, masks, normalization center, integer overflow, and
minimum/average/maximum size conventions. It follows the project’s variant of
Xia et al., FastCDC (USENIX ATC 2016).
Returned ByteChunk values own their bytes and include stream offsets.
Use it for content-defined deduplication of byte streams. Boundaries do not
depend on binary reader short-read patterns. Work is linear in bytes and buffered
memory is proportional to maximum chunk size; only the last chunk may be below
minimum. Blocking read(size) must return bytes; an empty read means EOF and
errors propagate. These are byte offsets, not character/token boundaries. Other
FastCDC variants can disagree. The port retains the project’s MIT notice and
has a fixture generated by its unmodified Go source.
A15 — Raw heat, LFU, and promotion¶
memory.memory_heat, lfu_evictions, and heat_promotions adapt
MemoryOS’s heat and capacity policies. Raw heat is
alpha*visits + beta*interactions + gamma*exp(-age_hours/tau_hours);
LFU chooses lowest access count; promotion chooses strictly above-threshold
heat, hottest first. Stable insertion order resolves ties.
Use them independently for admission, retention, or tiering experiments. The host supplies counts/age, applies returned IDs, and defines capacity. Protection can make eviction infeasible, which raises explicitly. These functions do not copy MemoryOS’s storage or conversation workflow. Unlike timestamp-parsing upstream fallbacks, invalid ages/parameters fail validation.
A16 — Learned blocking and active pair acquisition¶
linkage.learn_blocking adapts Dedupe’s bounded set-cover search.
Caller predicates expose labeled-match coverage and comparison cost. The search
minimizes the sum of selected predicate costs while reaching
floor(recall * number_of_matches) coverage. It returns selected predicate names,
coverage, cost, feasibility, search count, and whether search completed.
Use it to learn a disjunction of existing predicates before expensive pair
scoring. Predicate construction, compound predicates, training labels, and the
matcher remain external. Search is exponential in the worst case and bounded
by max_states; unfinished search can have no feasible incumbent even when a
solution exists. Overlapping comparison costs are counted repeatedly.
acquire_disagreement follows Dedupe’s matcher/blocker acquisition policy:
prioritize uncovered likely matches, otherwise spread acquisition across covered
probabilities, otherwise sample disagreement. Mari uses a single seeded RNG and
adds uniform fallback for all-zero weights; it does not promise the identical
upstream random sequence.
A17 — Entity clustering and matching choices¶
linkage.centroid_clusters, greedy_matching, and gazette_matching adapt
Dedupe’s matching alternatives. Centroid linkage uses SciPy over
1-score distances, missing pair distance 1, connected components, and
per-record 1 - RMS(within-cluster distances) confidence. Singleton outputs are
omitted; isolated pairs require score strictly above threshold.
Use clustering for same-entity groups, greedy matching for one-to-one bipartite links, or gazette matching for top matches per left record with reusable right records. None guarantees globally optimal assignment. Centroid distances need not be Euclidean and linkage can invert, as in the source’s chosen method. Clustering takes quadratic memory per component; oversized components raise instead of upstream recursive rethresholding. Repeated undirected pair scores are merged by maximum; greedy ties retain input order.
A18 — Neighborhood memory evolution¶
memory.evolve_neighborhood adapts A-MEM’s strengthen/update-neighbor actions
through a model-agnostic proposal callback. NoteUpdate explicitly addresses
note ID and expected revision and may change context/tags or add links. The
result contains immutable before/after notes with incremented revisions.
Use it when adding a memory should revise its neighborhood. Duplicate targets, stale revisions, self links, and links outside the supplied neighborhood fail the whole plan. This replaces upstream positional neighbor updates and in-place mutation with explicit identities. The host must atomically compare-and-swap revisions when committing; a generated plan does not reserve those revisions.
A19 — Feedback and skill deduplication¶
memory.reduce_skill_feedback adapts ACE’s skill feedback and dedup operations.
Helpful/harmful/neutral feedback increments counters and preserves provenance;
explicit keep/update/delete/merge decisions alter an immutable snapshot. Merge
sums source counters, unions provenance, and tombstones sources.
Use it for a host-controlled playbook learning loop. Persist applied_events
with the returned snapshot to make feedback replay idempotent. Dedup decisions
must be committed once; they are ordered operations, not replayable events.
Model evaluation, decision generation, transactions, and eventual garbage
collection remain caller choices. Deleted records cannot receive new feedback
or be reused in another merge.
A20 — Graph structural alternatives¶
graphs.condense_graph, transitive_reduction_edges, and cohesive_subgraph
expose NetworkX structural algorithms. Condensation returns strongly
connected components plus their DAG. Transitive reduction removes redundant DAG
edges while retaining reachability and surviving original weights. Cohesion
selects topological k-core or k-truss nodes; weights do not change membership.
Use SCCs for cycles, reduction for dependency explanations, and cores/trusses for dense neighborhoods. Reduce only DAGs; condense first when cycles exist. Parallel edges require explicit caller aggregation, and cohesion rejects self loops. These algorithms do not impose application graph semantics.
A21 — Common-space union ranking¶
graph_retrieval.rank_candidate_union adapts haiku.rag’s search union ordering.
Select either cosine scoring across a named common embedding space or scores
from a reranker applied to the complete union. Identity is (source, item_id);
a required allowed-key set filters candidates before scoring.
Use this when searching several collections with compatible vectors or a shared reranker. The cosine path validates dimensions, space identity, and finite values; zero vectors score zero. The reranker path requires supplied scores for every allowed candidate. Duplicate keys are rejected and deterministic ties retain within-source rank then source arrival order. This does not calibrate unrelated collection scores or independently normalized reranker batches.
Validation and limits¶
Run the new tests with pytest -q tests/test_algorithm_*.py. The suite includes
rank_bm25-generated score fixtures, a Go-generated FastCDC boundary fixture,
exhaustive small blocking comparisons, lazy/naive equivalence on random coverage,
a PCST bridge case, native graph checks, and callback/scope/revision edge cases.
Optional-solver tests skip when dependencies are absent; install the extra to
exercise them. CI includes a separate solver job as well as the base test matrix.
Local native validation used NetworkX 3.6.1, pcst-fast 1.0.10,
graspologic-native 1.3.1, and SciPy 1.18.1 on Python 3.13.
These checks establish the stated computation and interface behaviors on small fixtures. They do not establish retrieval quality, extraction quality, native cross-version reproducibility, or large-scale throughput. Model callbacks and source data require workload-specific evaluation. Source citations acknowledge provenance; adapted interfaces and policies are described above rather than presented as complete upstream systems.