Keep Privileged Knowledge Inside the Control Plane

Carlini and coauthors demonstrated a training-data extraction attack against GPT-2 by generating candidate sequences and ranking unusual outputs. They recovered hundreds of verbatim training examples, including public personally identifiable information, conversations, code, and identifiers; some recovered sequences appeared in only one training document. The result establishes that language models can memorize rare sequences and that memorized content can sometimes be extracted through queries.

The study does not prove that every contemporary model, fine-tuning method, or hosted service exposes customer data. It defines a threat class that must be addressed empirically. Risk depends on model capacity, duplication, optimization, adaptation method, access, decoding, and the sensitivity and uniqueness of the examples. Assertions that a model does not memorize should be supported by architecture, policy, and testing rather than by intuition.

Legal data is especially sensitive to uncontrolled replication because the same content may exist as source files, OCR output, normalized text, chunks, embeddings, prompts, caches, logs, checkpoints, backups, and analyst exports. A deletion request aimed only at the original document does not govern these derived representations.

Retrieval, inference, evaluation, adaptation, abuse monitoring, and product analytics are distinct processing purposes. A document authorized for retrieval in one matter is not automatically authorized for training, cross-matter evaluation, or human debugging. The control plane should attach purpose and matter attributes at ingestion and propagate them through derived artifacts. A downstream job should fail closed when the requested purpose is not compatible with the artifact’s policy.

Purpose limitation is easier to enforce when pipelines are physically and logically separated. An ingestion zone can parse and classify content; a retrieval zone can build matter-scoped indexes; an evaluation zone can receive de-identified or synthetic examples; an adaptation zone can accept only datasets with explicit approval manifests. Separate service identities, encryption keys, storage locations, queues, and retention schedules reduce the chance that a developer convenience path becomes an undocumented data flow.

Lineage must be bidirectional. The firm should be able to ask which source records contributed to an adapter or index, and which derived artifacts must be invalidated when a source authorization changes. Without reverse lineage, deletion and matter closure become manual searches across infrastructure.

Role-based access control is often too coarse for legal work. Two associates with the same title may have access to different matters, clients, jurisdictions, ethical walls, or document classes. Attribute-based policy can evaluate user identity, matter membership, client restrictions, purpose, location, device posture, and data sensitivity at request time. The retrieval service should receive only the authorized candidate universe rather than retrieve globally and filter after ranking.

Indexes can be isolated per matter or shared with cryptographically and logically enforced metadata partitions. Per-matter indexes simplify reasoning about boundaries but increase operational overhead. Shared indexes improve utilization but create a larger blast radius if filtering is defective. The choice should follow the threat model, and the benchmark must include negative tests proving that semantically attractive documents from unauthorized matters are never returned.

Prompt injection inside retrieved documents is another boundary problem. Retrieved text is untrusted data, not policy. The orchestration layer should delimit content, constrain tool permissions independently of model instructions, validate requested actions, and prevent retrieved text from changing authorization context.

Embeddings are transformed representations, not anonymous abstractions. They can reveal semantic relationships and may be exposed through nearest-neighbor queries or inversion research depending on the model and access pattern. Vector stores require the same identity, tenant isolation, encryption, backup, deletion, and audit discipline as the source corpus.

Caches can quietly defeat purpose and retention rules. Semantic caches may reuse an answer across users whose prompts appear similar but whose matter permissions differ. Prefix caches and model-serving caches can retain token representations beyond the apparent request lifetime. Cache keys must include relevant authorization and model-version context, and sensitive workflows may require cache disablement or dedicated cache partitions.

Observability needs minimization by design. Full prompts and completions make debugging easy but create a secondary corpus of privileged material. Prefer structured event metadata, protected references, selective sampling, field-level encryption, and short retention. Emergency payload access should be separately authorized and audited.

Leakage testing should include canary strings inserted into controlled adaptation examples, membership-inference experiments where appropriate, extraction-style prompting, cross-matter retrieval probes, prompt-injection suites, and searches for sensitive fragments in logs and caches. Tests must cover the deployed quantization and serving stack because behavior can change across model and runtime configurations.

Red teams should reason about insiders as well as external users. Infrastructure administrators, dataset curators, support engineers, and model developers may each have different paths to derived content. Just-in-time access, dual control for sensitive exports, immutable audit logs, and environment separation reduce standing privilege.

A passing test is time-bounded evidence. New data, adapters, base models, retrieval strategies, and serving libraries can change exposure. Leakage tests belong in release gates and periodic production assurance, with predefined actions when a canary is reproduced or an unauthorized source is retrieved.

Dedicated infrastructure removes some third-party paths and makes architecture more inspectable, but it transfers operational responsibility. The operator must patch kernels and serving libraries, restrict privileged access, protect orchestration credentials, harden object storage and backups, monitor egress, validate images, rotate keys, and maintain incident response. A model running inside a private network can still leak through logs, misconfigured storage, overbroad service identities, or authorized users crossing matter boundaries.

The strongest claim is not that local equals safe. It is that the firm can define, enforce, observe, and test the controls around its knowledge. The control plane should make every transition—from document to chunk, chunk to prompt, correction to evaluation, and approved example to adapter—an explicit governed event.