Retrieval and Refusal Beat the General-Purpose Oracle

A general-purpose conversational interface is optimized to continue. Legal work frequently requires the opposite behavior. The record may be incomplete, controlling authority may be unavailable, matter permissions may exclude a source, documents may conflict, or the requested conclusion may exceed the approved workflow. In those states, generating polished prose can convert missing evidence into apparent certainty.

Research on machine-generated legal analysis distinguishes fine-grained gaps rather than treating every defect as one hallucination category. The cited NLLP work developed a taxonomy with legal experts and reported that its automated detector reached 67 percent F1 and 80 percent precision on its test set; when applied as a metric, around 80 percent of evaluated state-of-the-art model outputs contained hallucinations of different kinds. These figures belong to that task and methodology, but they show why surface fluency is an inadequate quality measure.

A professional architecture should define answerability before generation. The system can require a minimum evidence set, valid source dates, jurisdiction match, sufficient retrieval coverage, and absence of unresolved conflicts. If those predicates fail, the correct state is refusal, a request for missing material, or escalation to qualified review.

Legal corpora contain structurally similar documents, repeated boilerplate, amendments, restatements, exhibits, and near-identical clauses. Chunk-level semantic similarity can retrieve a passage that looks perfect but belongs to the wrong agreement or version. The recent reliable-retrieval preprint names this failure Document-Level Retrieval Mismatch: the text appears relevant while originating from an incorrect source document.

Every chunk should therefore retain document identifier, matter, client, version, effective date, jurisdiction, authority type, section path, page or span offsets, and access attributes. Retrieval should not flatten these fields into display metadata after ranking; they are inputs to candidate generation, filtering, reranking, and validation.

Summary-Augmented Chunking, as studied in the cited preprint, prepends document-level synthetic summaries to chunks so the embedding carries global context lost during ordinary segmentation. The paper reports reduced document-level mismatch and improved text-level precision and recall across evaluated legal retrieval tasks. The result is promising but should be replicated on the firm’s own documents, embedding model, and access topology.

Dense retrieval captures semantic similarity; lexical retrieval remains strong for citations, defined terms, names, identifiers, and exact clauses. Hybrid systems combine candidate sets and may use reciprocal-rank fusion or learned weighting before reranking. The correct mixture depends on the query class. A citation lookup and an issue-spotting request should not necessarily share one retrieval policy.

Chunking should follow document structure where possible. Fixed token windows can sever definitions from operative language, separate exceptions from rules, and mix headers with unrelated text. Hierarchical retrieval can first identify a document or section and then retrieve local passages. Parent-child retrieval can return a precise child chunk while supplying broader parent context to generation.

Rerankers improve ordering but add their own failure surface, latency, and domain assumptions. Evaluation should measure candidate recall before reranking, top-k document accuracy after reranking, and character- or span-level precision and recall. If only final-answer quality is measured, retrieval defects remain hidden behind occasional model recovery.

The generator should receive stable source handles rather than anonymous text blocks. Claims in the response can be linked to source spans, and a validator can test whether cited passages entail or at least support the proposition. Citation existence is not citation correctness: a real case or document can be irrelevant to the claim for which it is cited.

Source validation can include identifier resolution, authorization recheck, version and effective-date checks, jurisdiction compatibility, quotation matching, and contradiction detection. Deterministic checks should handle what they can; model-based entailment can assist with semantic support but must itself be evaluated and should not be treated as an infallible judge.

The answer object should preserve provenance beyond the rendered page. Downstream systems need a structured representation of claims, citations, source spans, confidence or support status, retrieval trace, model version, and reviewer disposition. Flattening everything into a text blob discards the evidence graph needed for audit and later correction.

A refusal policy can combine deterministic conditions and empirical uncertainty. Deterministic gates include missing required sources, unauthorized matter scope, unsupported jurisdiction, failed citation resolution, or a detected conflict. Statistical signals include retrieval-score margin, answer consistency across samples, calibrated support probability, out-of-distribution detection, and performance on similar benchmark strata.

Thresholds create a coverage-risk tradeoff. Raising the evidence threshold reduces the number of answered requests and can improve selective accuracy, but excessive refusal makes the system unusable. Plot quality against coverage and choose thresholds by workflow risk rather than adopting one global confidence cutoff. High-risk external work may require near-complete support; internal brainstorming can tolerate broader coverage if clearly labeled.

Refusal should be informative without leaking restricted data. The system can state that required authority was not found, that available documents conflict, or that the request falls outside the approved workflow. It should identify what additional material or review step would make the task answerable.

Retrieved documents are untrusted inputs. A malicious or compromised document may contain instructions intended to override system policy, exfiltrate data, or invoke tools. The model should not possess authority merely because text told it to act. Tool permissions, network access, matter scope, and output channels must be enforced outside the model by deterministic controls.

Access filtering must occur before sensitive content reaches the model context. Post-generation redaction is not a sufficient boundary because unauthorized information has already been processed and may influence the answer. Negative tests should attempt cross-matter retrieval using highly similar language and adversarial metadata.

Indexes require freshness and invalidation. Superseded documents should remain available when historically relevant but should not silently outrank current authority. Deletion, ethical-wall changes, and matter closure must propagate to chunks, embeddings, summaries, caches, and replicas with verifiable completion.

A firm-owned consultative system combines ingestion quality controls, structured and hybrid retrieval, matter-aware filtering, reranking, source validation, claim attribution, calibrated refusal, and human review. The language model synthesizes within this chain; it is not the sole holder of truth or policy.

Failure analysis should assign defects to ingestion, permissions, candidate generation, ranking, context assembly, reasoning, generation, validation, refusal, or review. That decomposition directs investment. A larger model may improve synthesis while leaving document-level retrieval mismatch untouched. More context may increase recall while introducing conflicting evidence. Better refusal can reduce severe errors without changing the model at all.

The objective is not to build an oracle that always knows. It is to build a system that can show what it relied on, explain when the evidence is insufficient, preserve the boundary of the matter, and route uncertainty to the person or process equipped to resolve it.