Current
Hypothetical and hierarchical retrieval
Benchmark first
Corpora: BEIR (SciFact, ArguAna, HotpotQA) · LoTTE · KILT
Protocol: Build from corpus documents only; issue the published test queries; preserve ranked IDs and component scores. Report nDCG@10, Recall@100, MRR, p50/p95 query latency, index bytes, and results with ACL pre-filtering. Ablate candidate generation, reranking, fusion, graph expansion, and packing independently.
These functions construct alternative query representations and bounded navigation structures. Generation, encoding, clustering, summarization, and relevance models are injected; Mari owns shape validation, deterministic IDs, budgets, and traces.
How it works
hypothetical_document_embedding weights and averages caller-encoded hypothetical answers, then L2-normalizes the vector used for retrieval; generated text is never stored as fact. build_summary_tree repeatedly validates a caller-proposed partition of every current root, creates stable parent nodes, and stops when the root count no longer decreases. walk_summary_tree expands the highest-scoring children under explicit branch and visit budgets and returns visited paths plus exhaustion state.
Papers
HyDE: hypothetical document embeddingsRAPTOR: recursive summary treesMemWalker: bounded memory-tree navigation
Generatehypothetical answer
→
Encodenormalized query vector
→
Retrievecandidate sections
→
Organizerecursive clusters
→
Walkbranch + visit budget
Paper-derived retrieval construction
from mari_components.retrieval import (
build_summary_tree, hypothetical_document_embedding, walk_summary_tree,
)
hyde_vector = hypothetical_document_embedding([
document_encoder(text) for text in generate_hypotheses(query)
])
tree = build_summary_tree(section_text_by_id,
cluster=lambda nodes, level: cluster_embeddings(nodes, level),
summarize=lambda children, level: summarize(children, level))
walk = walk_summary_tree(tree, lambda node: similarity(query, node.text),
branch_factor=2, max_visits=24)
sections = document_store.get_many(walk.leaf_ids)