Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 2-4 hours to set up; 30-60 minutes per clean-up
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: Stale or deleted content can be cited in answers, and orphaned chunks from one-to-many projections persist after source deletion

Handle source deletions, duplicates and scheduled refresh

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026; updated on 27 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-05). Author: Ranjith · dry-run: Kandiah Chockalingam · sign-off: Yogeshwaran. Work items: RAG-05-T09, RAG-05-T10, RAG-05-T11, RAG-05-T12.

1. Summary

This runbook synchronises Azure AI Search indexes with source blob deletions, prevents duplicate entries from path renames, and maintains scheduled background refreshes. There are two primary deletion detection policies for Azure Blob Storage: native blob soft delete and soft-delete metadata columns.

Deletions in one-to-many indexing projections (chunked documents) are not deleted automatically by indexer deletion policies and require document deletion calls. Facts checked against Microsoft Learn on 27 October 2026. The latest stable REST API version is 2026-04-01. Document reset (resetdocs) is preview under API version 2026-05-01-preview.

2. When to use and when not to

Use this runbook when:

  • Setting up an Azure AI Search data source for production with ongoing blob lifecycle management.
  • Deleted source blobs remain visible in search queries or generate stale answers in RAG generation.
  • Configuring automated, recurring indexer execution intervals (schedule.interval).
  • Documents or chunk projections duplicate after source folder renames or metadata updates.
  • Purging orphaned child chunk projections whose parent blob has been deleted from storage.

Do not use it when:

  • The index schema, vector dimension, or analyser configuration changes. Use PB-RAG-09-01 for zero-downtime reindexing with aliases.
  • An indexer fails or throttles during execution. Use PB-RAG-05-02 for indexer and skillset failure recovery.
  • Designing an initial document ingestion pipeline with Azure Functions and Document Intelligence. Use PB-RAG-05-01.
  • Migrating to a new embedding model across all corpus indexes. Use PB-RAG-09-02.

3. What you must know first

  1. Automatic change detection. Azure Blob indexers detect content updates automatically using the storage blob LastModified timestamp. Modifying blob metadata or touching the blob updates this timestamp and triggers reindexing.
  2. Deletion detection requires an explicit policy. Blob deletions are ignored by default. You must configure either a NativeBlobSoftDeleteDeletionDetectionPolicy or a SoftDeleteColumnDeletionDetectionPolicy on the data source definition.
  3. First-run policy constraint. The deletion detection policy must exist before the indexer runs. Blobs deleted before a policy is set are never removed automatically by subsequent runs; they require manual document deletion or a full index rebuild.
  4. One-to-many chunk projection limitation. When a skillset splits one blob into many chunk documents via indexProjections, the indexer deletion detection policy does not remove the child chunks when the parent blob is deleted. You must delete child chunk projections via the document API.
  5. Native blob soft delete restrictions. NativeBlobSoftDeleteDeletionDetectionPolicy requires blob soft delete enabled on the Azure Storage account. Blob versioning must be disabled. The storage soft-delete retention period must be longer than the indexer schedule interval.
  6. Key stability and duplicate prevention. Document keys in Azure AI Search must be URL-safe (base64 encoded metadata_storage_path or unique hash). Renaming a blob or moving it to another folder produces a new key. The old key remains in the index unless deleted.
  7. Indexer run limits and scheduling. Multitenant indexers with skillsets have a hard maximum limit of 2 hours per execution. A recurring schedule (minimum 5 minutes) resumes processing from the last recorded high-water mark.

4. Prerequisites and access

Item Detail
Roles Search Service Contributor (update data sources and indexers). Search Index Data Contributor (read/delete document keys). Storage Blob Data Contributor (manage blob soft-delete and metadata). Entra ID RBAC is preferred over account keys.
Tools Azure CLI (az) 2.60+, curl, jq, Python 3.10+ with azure-search-documents (12.x) and azure-identity.
Storage settings Azure Storage account with blob soft delete enabled (retention period >= 7 days) if using native soft delete. Blob versioning disabled.
Git repository Data source and indexer JSON definitions stored in version control.
Change approval Change ticket with agreed refresh schedule and maintenance window for orphan clean-up.

Set required environment variables:

export SEARCH_ENDPOINT="https://${SEARCH_SERVICE}.search.windows.net"
export API_VERSION="2026-04-01"
export TOKEN="$(az account get-access-token --scope https://search.azure.com/.default --query accessToken --output tsv)"

Verify: curl -sS "${SEARCH_ENDPOINT}/servicestats?api-version=${API_VERSION}" -H "Authorization: Bearer ${TOKEN}" | jq '.counters' returns HTTP 200 with service counters.

5. Steps

Placeholders: ${SEARCH_SERVICE}, ${STORAGE_ACCOUNT}, ${CONTAINER_NAME}, ${DATA_SOURCE_NAME}, ${INDEXER_NAME}, ${TARGET_INDEX_NAME}, ${TOKEN}.

Stage 1: Configure deletion detection on the data source

Step 1. Define deletion policy on data source.

For storage accounts with blob soft delete enabled, use NativeBlobSoftDeleteDeletionDetectionPolicy:

{
  "name": "${DATA_SOURCE_NAME}",
  "type": "azureblob",
  "credentials": {
    "connectionString": "ResourceId=/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Storage/storageAccounts/${STORAGE_ACCOUNT};"
  },
  "container": { "name": "${CONTAINER_NAME}" },
  "dataDeletionDetectionPolicy": {
    "@odata.type": "#Microsoft.Azure.Search.NativeBlobSoftDeleteDeletionDetectionPolicy"
  }
}

Alternatively, if soft delete is handled by an application metadata tag, use SoftDeleteColumnDeletionDetectionPolicy:

{
  "name": "${DATA_SOURCE_NAME}",
  "type": "azureblob",
  "credentials": {
    "connectionString": "ResourceId=/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Storage/storageAccounts/${STORAGE_ACCOUNT};"
  },
  "container": { "name": "${CONTAINER_NAME}" },
  "dataDeletionDetectionPolicy": {
    "@odata.type": "#Microsoft.Azure.Search.SoftDeleteColumnDeletionDetectionPolicy",
    "softDeleteColumnName": "IsDeleted",
    "softDeleteMarkerValue": "true"
  }
}

Verify: The JSON file validates against the Azure AI Search REST schema.

Step 2. Apply the data source configuration.

curl -sS -X PUT "${SEARCH_ENDPOINT}/datasources('${DATA_SOURCE_NAME}')?api-version=${API_VERSION}" \
  -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/json" \
  --data @datasource.json | jq '.dataDeletionDetectionPolicy'

Verify: The returned JSON confirms the configured dataDeletionDetectionPolicy.

Stage 2: Configure recurring refresh schedule on the indexer

Step 3. Update the indexer schedule and batch limits.

Indexers with skillsets pause at 2 hours; a 15-minute interval automatically resumes incomplete batches:

{
  "name": "${INDEXER_NAME}",
  "dataSourceName": "${DATA_SOURCE_NAME}",
  "targetIndexName": "${TARGET_INDEX_NAME}",
  "schedule": { "interval": "PT15M" },
  "parameters": {
    "batchSize": 10,
    "maxFailedItems": 0,
    "maxFailedItemsPerBatch": 0,
    "configuration": { "dataToExtract": "contentAndMetadata" }
  }
}

Verify: interval is set to an ISO 8601 duration not less than 5 minutes (PT5M).

Step 4. Submit the indexer definition.

curl -sS -X PUT "${SEARCH_ENDPOINT}/indexers('${INDEXER_NAME}')?api-version=${API_VERSION}" \
  -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/json" \
  --data @indexer.json | jq '.schedule'

Verify: The response returns the active schedule with "interval": "PT15M".

Stage 3: Prevent duplicate documents during path renames

Step 5. Ensure stable URL-safe document keys.

Document keys must map stably to the storage path. Use base64 encoding on metadata_storage_path:

{
  "sourceFieldName": "metadata_storage_path",
  "targetFieldName": "id",
  "mappingFunction": { "name": "base64Encode" }
}

Verify: All document lookups use valid base64-encoded path keys.

Renaming paths creates duplicate documents

Renaming a blob or directory generates a new metadata_storage_path and a new document key. The old document is not detected as deleted unless soft-deleted prior to rename. Always delete old keys when renames occur.

Stage 4: Purge orphaned chunk projections from one-to-many indexing

Step 6. Purge orphaned child chunk documents.

When index projections create multiple chunk documents per parent blob, purge child chunks whose parent was deleted:

import os
from azure.identity import DefaultAzureCredential
from azure.search.documents import SearchClient

client = SearchClient(os.environ["SEARCH_ENDPOINT"], os.environ["TARGET_INDEX_NAME"], DefaultAzureCredential())
parent_id = os.environ["DELETED_PARENT_ID"]
results = client.search(search_text="*", filter=f"parentId eq '{parent_id}'", select=["id"])
chunk_keys = [{"id": doc["id"]} for doc in results]
if chunk_keys:
    client.delete_documents(documents=chunk_keys)
print(f"Deleted {len(chunk_keys)} orphaned chunks for parent {parent_id}")

Verify: Querying parentId eq '${DELETED_PARENT_ID}' returns 0 documents.

Stage 5: Test deletion and refresh propagation

Step 7. Execute a test deletion in source storage.

Delete a test blob from the container and trigger an on-demand indexer run:

az storage blob delete --account-name "${STORAGE_ACCOUNT}" \
  --container-name "${CONTAINER_NAME}" --name "test-doc.pdf" --auth-mode login

curl -sS -X POST "${SEARCH_ENDPOINT}/indexers('${INDEXER_NAME}')/search.run?api-version=${API_VERSION}" \
  -H "Authorization: Bearer ${TOKEN}" -o /dev/null -w "run HTTP %{http_code}\n"

Verify: The indexer run returns HTTP 202 Accepted.

Step 8. Verify document removal from index.

Query the document key of the deleted blob:

curl -sS "${SEARCH_ENDPOINT}/indexes('${TARGET_INDEX_NAME}')/docs('${TEST_KEY}')?api-version=${API_VERSION}" \
  -H "Authorization: Bearer ${TOKEN}" -o /dev/null -w "doc lookup HTTP %{http_code}\n"

Verify: The document lookup returns HTTP 404 Not Found.

6. Monitoring and alerts

Alert thresholds below are illustrative team defaults. Tune them against measured production volume.

Signal Source Alert when Action
DocumentsProcessedCount (dimension Failed = true) Platform metric (total) More than 0 in 15 minutes Query search.status API; fix document formatting or data source error.
Indexer execution status Status API (lastResult.status) transientFailure twice or persistentFailure Inspect errorMessage in status response; restart indexer or fix source access.
Time since last successful run Status API / Azure Monitor Exceeds twice the scheduled interval Run the indexer recovery runbook (PB-RAG-05-02).
OrphanedChunkCount Search query (parentId lookup) Above 0 during weekly audit Run the orphan clean-up script (Step 6).
ThrottledSearchQueriesPercentage Platform metric (average) Above 1% for 5 minutes during refresh Lengthen schedule.interval or scale search units.

Create an alert for failed document processing:

SEARCH_ID="$(az search service show --name "${SEARCH_SERVICE}" --resource-group "${RESOURCE_GROUP}" --query id --output tsv)"

az monitor metrics alert create \
  --name "indexer-refresh-failure" \
  --resource-group "${RESOURCE_GROUP}" \
  --scopes "${SEARCH_ID}" \
  --condition "total DocumentsProcessedCount > 0" \
  --window-size 15m \
  --evaluation-frequency 5m \
  --severity 2 \
  --action "${ACTION_GROUP_ID}" \
  --description "Documents failed during scheduled indexer refresh"

Audit indexer execution history with Log Analytics (KQL):

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.SEARCH" and OperationName == "Indexing.Index"
| where TimeGenerated > ago(24h)
| summarize total_docs = sum(Documents_d), failed_docs = countif(resultSignature_d != 200) by IndexName_s, bin(TimeGenerated, 1h)
| order by TimeGenerated desc

7. Rollback

Situation Rollback Time
Native blob soft delete causes indexing errors Revert data source definition to remove dataDeletionDetectionPolicy. Run indexer. Under 5 minutes
Soft-delete column marks valid documents deleted Update blob metadata (IsDeleted=false) or touch LastModified. Call search.run. 10-15 minutes
Accidental blob deletion in storage Undelete soft-deleted blob in Azure Storage, update its metadata to touch LastModified, and trigger indexer run. Under 15 minutes
Orphan cleanup deletes valid chunks Re-run indexer with reset docs (resetdocs) for the affected parent key to rebuild child projections. 15-30 minutes
Scheduled refresh overloads search queries Disable indexer schedule ("schedule": null or "disabled": true). Resume manual runs off-peak. Under 2 minutes

Rollback trigger (any one): Unintended documents removed from search results, query throttling exceeding 1% for 15 minutes, or indexer error rate exceeding 0% on scheduled runs.

8. Known issues and fixes

Symptom Likely cause Fix
Deleted blobs still appear in search results No deletion policy configured on data source, or blobs deleted prior to policy addition Configure deletion policy. For blobs deleted before policy addition, delete keys via document API or rebuild index (PB-RAG-09-01).
Restored soft-deleted blob never returns to index Storage LastModified timestamp was not refreshed when the blob was restored Re-save or touch blob metadata in Azure Storage to update LastModified, then trigger the indexer.
Duplicate documents appear after renaming a blob folder Document keys use base64 encoded paths; changing folder paths changes document keys without deleting old ones Delete old keys from index via the document delete API before or immediately after renaming source folders.
Orphaned chunks remain searchable after parent blob is deleted Skillset indexProjections (one-to-many) are not deleted by indexer blob deletion policies Execute Step 6 script to query chunks by parent identifier and issue a batch document delete.
Data source update fails with Blob soft delete is not enabled Azure Storage account does not have blob soft delete enabled Enable blob soft delete on the storage account with at least 7 days retention before applying native soft delete policy.
Native soft delete rejected with versioning error Azure Storage account has blob versioning enabled Disable blob versioning, or switch to SoftDeleteColumnDeletionDetectionPolicy using metadata flags.
Indexer stops at 2 hours with work remaining Skillset indexers have a 2-hour execution limit in multitenant environments Maintain a 15-minute schedule; subsequent scheduled executions automatically resume from the last high-water mark.

9. AWS and open-source equivalents

Platform Equivalent Key differences
Amazon Bedrock Knowledge Bases Data source synchronisation (StartIngestionJob) Bedrock automatically detects and purges deleted S3 objects and child chunks on subsequent sync jobs. No custom orphan cleanup needed.
Amazon OpenSearch Service OpenSearch Indexing Pipelines / Logstash S3 input Requires SQS S3 delete event notifications or custom reconciliation jobs to issue delete calls for removed documents.
PostgreSQL with pgvector Storage triggers or change data capture (Debezium) Requires foreign key cascading deletes (ON DELETE CASCADE) between parent documents and chunk embeddings, or explicit CDC delete events.

10. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario:

  1. Provision a Standard-tier Azure AI Search service and Azure Blob Storage container with 50 test documents.
  2. Configure blob soft delete with 7-day retention and disable blob versioning.
  3. Configure data source with NativeBlobSoftDeleteDeletionDetectionPolicy and indexer with PT15M schedule.
  4. Execute initial indexer run and verify all 50 documents exist in the index.
  5. Soft-delete 5 source blobs and wait for the next scheduled indexer execution.
  6. Verify the 5 documents are removed from the index.
  7. Restore 1 soft-deleted blob, update its metadata, run the indexer, and verify it reappears in search results.
  8. Test one-to-many chunk projection orphan cleanup script and verify no orphaned chunks remain.

Pass criteria:

  • Soft-deleted blobs are completely purged from search query results after the scheduled run.
  • Restored blob is re-indexed successfully after metadata touch.
  • Orphan cleanup script removes all child chunks of deleted parent documents without touching active documents.
  • All commands execute with zero manual adjustments.

11. Sources

Checked on 27 October 2026.

  • PB-RAG-05-01: Build a format-aware ingestion pipeline with Document Intelligence and Azure Functions.
  • PB-RAG-05-02: Recover from indexer and skillset failures and throttling.
  • PB-RAG-09-01: Reindex an Azure AI Search index without downtime.