A document disappears from the company knowledge base. Its vector records are deleted. The assistant still answers questions using a summary saved yesterday.

There is no contradiction: the deletion reached one store, while the application served another. A RAG system can turn one source into chunks, embeddings, cached responses, conversation summaries, evaluation fixtures and diagnostic traces. Each copy has its own lifetime.

Reliable RAG deletion needs two coordinated operations: stop serving the affected information, then remove or isolate its recorded derivatives. That requires stable source identity, dependency tracking, a serving barrier that works while storage converges, and evidence that an old ingestion job or backup cannot bring the content back.

Consider a hypothetical pricing assistant whose source owner withdraws an internal discount schedule. The operational requirement is precise: the assistant must stop using that schedule, including answers already computed from it. Deleting the original PDF is only one step.

This extends the broader RAG governance architecture. The focus here is the lifecycle of one withdrawn source and everything the system derived from it.

Define what the deletion request actually means

Three requests can arrive through the same support form and require different actions.

Access removal changes who may use a resource. The document may remain available to other authorized users. Invalidating a shared answer globally might be a reasonable emergency measure, but deleting the shared source is usually the wrong implementation.

Source withdrawal removes a document from the assistant’s usable knowledge. Old answers and summaries dependent on it become ineligible for reuse. The record may still exist in an archive that the assistant cannot access.

Storage deletion removes specified content and derivatives from specified stores. This needs a declared scope, retention exceptions where applicable, and separate evidence for backups and provider-held copies.

Record the operation, tenant, source system, stable object ID, version scope and accountable owner. A URL is useful for humans, but a rename or move must not create a new identity accidentally. Decide whether the request covers one revision, every revision of an object, or a wider set of records. A request to delete information about a person requires discovery across sources; deleting one document ID does not solve that problem.

The data-classification model for enterprise assistants should determine which retention and access rules apply. Do not let a connector infer those rules from a filename.

Also state the limit: removing RAG content does not make a model forget information learned during training, revoke a user’s downloaded file, or erase an answer already delivered. Those are different control surfaces. The deletion service should describe its actual coverage.

Track derivatives before they become usable

A deletion worker cannot reliably discover dependencies by searching for a phrase after the fact. Chunking changes boundaries. Summaries paraphrase. A cached answer can depend on five documents while displaying only two citations.

Keep a dependency manifest separate from the vector index. For every artifact, record its tenant, artifact ID, storage locator, artifact type, lifecycle state and input dependencies. Source references need an object incarnation and content version: an incarnation distinguishes a deliberately recreated object from the retired one that happened to use the same path.

An illustrative manifest entry for a generated summary might look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

{
  "tenant_id": "acme-eu",
  "artifact_id": "summary-804",
  "kind": "conversation_summary",
  "storage_ref": "memory/conversation-93/summary-804",
  "state": "active",
  "inputs": [
    {
      "source_system": "pricing-library",
      "object_id": "discount-schedule",
      "incarnation": "inc-17",
      "content_version": "v6"
    },
    {"artifact_id": "answer-792"}
  ],
  "dependency_set_id": "deps-229",
  "admission_epoch": 418
}

Maintain a reverse index from source to derivatives, including transitive dependencies. A summary of a previous answer inherits that answer’s source dependencies. Store the full set of context inputs that could have influenced a generation, rather than trusting the model’s chosen citations as a complete lineage record.

This is a conservative policy. If an answer consumed a withdrawn source, invalidate the answer even when an observer thinks the source probably contributed nothing. Recompute from eligible inputs when necessary. Asking an LLM to remove the offending sentence is too weak a basis for clearing the dependency.

The manifest also needs a write protocol. Create artifacts as quarantined, register their dependencies, then admit them only if those dependencies are still eligible. Admission and source withdrawal must have a defined ordering, such as a transaction in the authoritative metadata store. Otherwise an ingestion worker can check eligibility, pause, and publish an artifact after deletion has started.

If the storage write and manifest update cannot share a transaction, make the storage object inaccessible until admission succeeds. Reconcile abandoned writes. An object whose provenance was never recorded must not silently become searchable.

Stop serving before waiting for physical removal

Storage deletion is not always immediately visible to readers. Pinecone explicitly documents eventual consistency for record changes in its deletion guide. An application therefore needs a way to deny use independently of whether every index replica has caught up.

Use an authoritative withdrawal record, often called a tombstone, keyed by tenant, source identity and version scope. Keep it outside the index being purged. A monotonic epoch or sequence records changes in eligibility; a wall-clock timestamp alone does not resolve delayed events or retries.

On an authorized withdrawal, the control service commits the tombstone and advances the relevant eligibility epoch. Retrieval and cache-serving paths consult that authority. Workers delete derivatives asynchronously, but the barrier remains in force if a worker fails.

The barrier must cover more than vector search:

Apply filtering before any external reranker or model receives content. Checking after generation may protect the user’s response while still disclosing the document to an external service.

For latency, a cache key can include tenant, principal or authorization scope, and an eligibility epoch. This is useful only if the serving component obtains a sufficiently current epoch. Looking up an old epoch from another stale cache recreates the same problem. A coarse tenant epoch is simpler but invalidates more answers; a dependency-aware design preserves more cache entries at the cost of extra metadata reads.

On a source-state lookup failure, refuse to reuse protected content. Define that behavior explicitly in the platform’s policy enforcement layer, including which public or unrelated workloads can continue.

Close the in-flight response race

A request can retrieve an eligible document at epoch 418 and finish after epoch 419 withdraws it. Checking only when retrieval begins misses that case.

Revalidate dependencies before dispatching context and before releasing the resulting answer. For a strict barrier, each release authorization, whether for context dispatch or user delivery, must be serialized with withdrawal in a shared authority, or implemented with tracked leases whose expiry or cancellation is part of barrier completion. An extra check followed by an uncoordinated network write still leaves a race.

Define the observable contract carefully. Requests admitted to delivery before the barrier may already have bytes in transit. Bytes already streamed cannot be recalled. For sensitive workflows, buffer the answer, obtain release authorization, and stop or drain affected in-flight deliveries before declaring the serving barrier complete. A lower-latency streaming product may accept a bounded exposure window, but it must measure and describe that window.

The same limitation applies to provider requests already in progress. Cancel where supported, suppress returned output, and handle provider retention through its separate data-lifecycle controls. A local tombstone cannot retract a prompt already sent.

Run deletion as a resumable job with separate completion claims

Avoid a single green “deleted” status. Track at least three milestones: serving blocked, online derivatives removed, and retained copies resolved. A job can reach the first milestone while the other two remain incomplete.

The worker enumerates dependencies, requests each required deletion, records per-artifact outcomes and verifies absence through the relevant read interfaces. Retrying must be idempotent. A timeout means the result is unknown until reconciled; it does not justify marking the store clean.

Microsoft’s Azure AI Search deletion documentation describes both orphaned index documents and per-document deletion results. For some indexer workflows, removing source content before the indexer observes its soft-delete state can leave indexed documents behind. This makes source-removal ordering a connector-specific responsibility.

Capture an enumeration checkpoint, then fence new admissions for the withdrawn scope. Without that fence, a worker can finish its list while a concurrent embedding job adds another chunk. Perform a reconciliation pass after outstanding jobs drain or are rejected. Counts can help detect discrepancies, but an unchanged global count cannot prove absence while unrelated ingestion continues.

At the source-management level, AWS distinguishes DELETE and RETAIN policies and documents a DELETE_UNSUCCESSFUL state for Bedrock data-source deletion. Deleting the knowledge-base resource also does not delete the vector store itself. Read each service’s lifecycle semantics rather than treating a successful resource-management request as an end-to-end purge receipt.

The platform should preserve a compact receipt: deletion request ID, scope, owner, tombstone sequence, manifest checkpoint, stores visited, failed operations, verification results and unresolved retention obligations. The agent audit-log architecture provides the correlation model. Do not copy the withdrawn text into the receipt to prove that it existed.

Make reintroduction harder than deletion

One of the most useful deletion tests is to run ingestion again.

AWS warns that directly deleting documents from an S3-backed Bedrock knowledge base does not remove the S3 originals; a later sync can reintroduce them. Its direct document deletion guide makes this distinction explicit.

Block the retired source incarnation at ingestion admission, not just at retrieval. Reject delayed change events for it. If the business intentionally republishes the source, require an explicit reinstatement or new incarnation with a fresh eligibility decision. A late “document updated” message must not undo a tombstone.

Content hashes can help find duplicate copies, but they cannot replace source identity: formatting changes alter the hash, and identical content may legitimately exist in different authorization domains. For a withdrawal broader than one source, discover and classify those other copies rather than assuming one tombstone covers them.

Backups need the same treatment. Restore into quarantine, load the current withdrawal ledger, reconcile restored artifacts against it, then admit the restored service to traffic. Restoring the vector database and its old metadata snapshot together must not roll the withdrawal ledger backward.

Object versions matter too. In a versioning-enabled S3 bucket, a delete request without a version ID creates a delete marker; it does not permanently remove previous content. AWS documents these versioned-object deletion semantics. The storage owner must account for versions, snapshots and retention settings separately from online search behavior.

Keep tombstones until every relevant replay or restore path is fenced, retired or beyond its allowed retention horizon. Garbage-collecting them after an arbitrary cache TTL can turn disaster recovery into data resurrection. Use protected identifiers and minimal metadata so that the prevention ledger does not become another unnecessary copy of sensitive content.

If a record must be retained, isolate it from ordinary retrieval and assistant credentials, record why it remains and assign its next review or expiry. “Unavailable to the assistant” and “removed from storage” are different claims and should stay distinct in the job status.

Test the deletion path under concurrent work

Use a synthetic document with a distinctive, non-sensitive fact. First prove that the intended surfaces can retrieve or reuse it. Then withdraw it while requests, cache hits, summarization and ingestion are active. The baseline prevents a meaningless pass caused by a test document that was never indexed.

Verify with direct ID lookup and manifest reconciliation as well as user-facing queries. Search by paraphrase, follow citation previews, reopen conversations and resume suspended workflows. A model failing to mention the synthetic fact once is not proof that the underlying copy is gone. Promote these scenarios into the application’s release eval gates.

Measure two operational delays from the authorized source change: time to the serving barrier and time to verified online deletion. Track retained-copy obligations separately. For paths that propagate changes concurrently, a useful planning bound is:

1
2
3

time_to_serving_barrier <= detection_bound
                          + slowest_serving_path_bound
                          + in_flight_drain_bound

This only holds when all paths are covered and each term has a defensible bound. For illustration, a connector polled every 60 seconds, a 5-second eligibility-cache bound and a 10-second delivery-drain bound give a worst-case planning value of 75 seconds. These are hypothetical design numbers, not measured service guarantees. An indefinitely disconnected cache has no finite bound; it must stop serving protected content when its freshness limit expires.

For the pricing assistant, completion should therefore read like an operational fact: the withdrawal reached every serving path, all known online derivatives were removed or rebuilt, and any retained copies have a named disposition. Start with one source family and prove that entire lifecycle, including restore. That is a more useful production capability than another delete button attached only to the vector database.