Domain Adaptation Is How Firm Knowledge Compounds

Gururangan and coauthors studied domain-adaptive pretraining across biomedical papers, computer-science publications, news, and reviews. A second phase of pretraining on in-domain text improved performance across evaluated classification tasks in both high- and low-resource settings, while task-adaptive pretraining on unlabeled task data produced further gains. The work supports a broad engineering conclusion: general linguistic competence and domain performance are not identical.

The study was not conducted on law-firm data and does not promise a specific improvement for legal workflows. Legal corpora differ in confidentiality, temporal validity, jurisdiction, authority, citation structure, and document length. Adaptation strategy must therefore begin with a task and data analysis rather than an assumption that more internal text will produce a better legal model.

The central architectural question is where each kind of knowledge should live. Model weights are suitable for statistical patterns that should generalize across requests. Retrieval is suitable for current, attributable, access-controlled facts. Deterministic tools are suitable for exact calculations, schema enforcement, and authoritative lookups. Policy engines are suitable for permissions and mandatory rules. Evaluation sets define acceptable behavior across all layers.

A useful placement matrix considers volatility, required attribution, confidentiality, scope, and error tolerance. Stable drafting conventions may be candidates for parameter-efficient adaptation. Current cases, matter facts, negotiated positions, and client instructions usually belong in retrieval because they change, require provenance, and may be matter restricted. Citation format can be supported by adaptation but validated deterministically. Authorization should never depend on a model remembering policy.

Weighting a model with volatile authority creates an update problem. When law changes, a retrieval corpus can replace or supersede a source and preserve effective dates. Removing a learned association from weights is materially harder and may require retraining. The firm should avoid using adaptation as a database.

Conversely, forcing every stable domain pattern through retrieval can produce bloated prompts, high latency, and brittle dependence on passage selection. Architecture should be empirical: compare a base model, retrieval-only system, adapter-only system, and combined system against the same benchmark to determine where specialization produces measurable value.

Dataset construction begins with admission policy. Every example should have a source identifier, rights basis, client or matter scope, sensitivity classification, temporal status, allowed purposes, and reviewer disposition. Transformations—redaction, normalization, segmentation, synthetic augmentation, deduplication—should produce versioned manifests so the training snapshot can be reconstructed.

Quality filtering must be task aware. A document can be authoritative yet unsuitable as a training example because it contains superseded language, unresolved edits, client-specific exceptions, or an output the firm does not want generalized. Pairwise preference data and supervised examples need explicit instructions and rationale; otherwise the model may learn superficial style while preserving hidden defects.

Deduplication changes both performance and privacy risk. Highly repeated text can dominate gradient updates, inflate apparent validation performance through leakage, and increase memorization. Exact hashes catch identical files, while locality-sensitive and semantic methods are needed for near-duplicate clauses, templates, and revised documents.

Continued pretraining changes next-token behavior using unlabeled domain text. Supervised fine-tuning teaches response patterns from instruction-output pairs. Preference optimization adjusts behavior using comparative judgments. Parameter-efficient methods such as LoRA update a compact set of trainable matrices while freezing the base model. These methods solve different problems and create different evidence requirements.

Continued pretraining may improve domain language modeling but can also shift general capability or amplify corpus bias. Supervised fine-tuning is easier to align with explicit tasks but inherits the quality limits of its examples. Preference data can encode nuanced review judgments, but inconsistent annotators produce an unstable objective. The release record should explain why a mechanism was selected and which failure classes it is intended to address.

Hyperparameters are governance-relevant because aggressive learning rates, excessive epochs, rank choices, target modules, and data mixtures can alter retention and generalization. Training should produce checkpointed curves, held-out loss, task metrics, leakage tests, and comparisons against the unadapted base. A successful loss curve is not proof of workflow improvement.

Adaptation can degrade capabilities that were not represented in the training objective. Evaluation should include both target tasks and regression suites for general instruction following, safety behavior, citation handling, refusal, and out-of-domain robustness. If the adapter improves one practice group while degrading another, routing or separate adapters may be safer than a single universal specialization.

Institutional forgetting occurs when the firm preserves only the latest adapter and discards the process that produced it. The durable record includes dataset versions, exclusions, annotator guidance, evaluation history, base-model compatibility, and failed experiments. These artifacts allow a stronger future model to inherit the firm’s learning process without inheriting every old weight.

Production feedback should not flow directly into training. Reviewer corrections first enter a quarantine and adjudication process. The team determines whether the defect belongs in training, retrieval, a validator, a policy rule, or the benchmark. This prevents one local correction from becoming an unexamined global behavior.

A firm-owned adaptation program should promote a candidate only when it exceeds the approved baseline on target tasks, stays within regression budgets, satisfies leakage tests, and preserves compatibility with the intended serving stack. The release bundle includes base-model digest, tokenizer, adapter, prompt, retrieval version, inference parameters, benchmark results, and limitations.

The compounding asset is the governed process: curated data, task definitions, evaluation criteria, retrieval structure, adapters, policies, and release evidence. If those layers remain portable, the firm can adopt a new base model or infrastructure provider without restarting its institutional learning program from zero.