Set up incremental and event-driven refresh with change and deletion detection¶
Draft: not validated
Drafted from official sources on 16 October 2026; updated on 3 November and checked again against the official docs on 4 November 2026. Do not use on a live system until it is dry-run by someone other than the author and signed off (planned 2026-12-01). Author: Kandiah Chockalingam · dry-run: Ranjith · sign-off: Yogeshwaran. Work items: RAG-09-T09, RAG-09-T10, RAG-09-T11, RAG-09-T12.
1. Summary¶
This how-to implements incremental and event-driven refresh pipelines for Azure AI Search knowledge bases. It covers two production architectures: managed pull refresh using scheduled indexers with native change and deletion detection policies, and event-driven push refresh using Azure Event Grid, Azure Service Bus queue buffering, and an idempotent worker using the document push API.
The outcome is fresh search results reaching users within defined service-level agreements (SLAs), with automatic deletion propagation and zero orphaned chunks when source files are modified or withdrawn. Facts checked against Microsoft Learn and AWS documentation on 3 and 4 November 2026. Latest stable data-plane REST API: 2026-04-01.
2. When to use and when not to¶
Use this how-to when:
- You build a new RAG knowledge base where source content changes frequently and must update without full reindexing.
- Freshness requirements demand updates faster than the 5-minute minimum schedule for search indexers (sub-minute freshness targets).
- Your own code chunks and embeds the documents and pushes them to the index, so it must also delete their chunks. (If an indexer does the chunking with index projections, its deletion policy removes the chunks: use Pattern 1.)
- Ingestion spikes require queue buffering with dead-letter queue (DLQ) support for poison or throttled documents.
Do not use it when:
- You change a field definition, analyser, or vector dimension that requires a full index rebuild. Use PB-RAG-09-01 (Reindex an Azure AI Search index without downtime) instead.
- You upgrade the embedding model across the corpus. Use PB-RAG-09-02 (Upgrade the embedding model and re-embed the corpus) instead.
- You must take down one document, or apply one permission change, now and prove it reached every place. Use PB-RAG-09-05 (Deletion and permission change propagation checklist) instead.
- You are debugging skillset execution failures or indexer execution timeouts. Use PB-RAG-05-02 instead.
3. What you must know first¶
- Change detection vs deletion detection. Blob indexers detect new and modified blobs automatically from
LastModifiedtimestamps. Deletions are not detected unless you configure a deletion detection policy. The deletion policy must exist from the very first indexer run. If an indexer runs without a deletion policy, previously deleted blobs remain in the index even after a reset. - Native blob soft delete requirements. Native blob soft delete requires Azure Blob Storage soft delete enabled, blob versioning disabled, and a soft-delete retention period much longer than the indexer schedule. Native blob soft delete is not supported for Azure Files. Neither soft-delete policy works with one-to-many parsing modes (JSON lines, JSON array, delimited text).
- Database source change tracking. Azure SQL data sources support
SqlIntegratedChangeTrackingPolicy(native SQL Change Tracking) orHighWaterMarkChangeDetectionPolicy. Azure Cosmos DB data sources supportHighWaterMarkChangeDetectionPolicyusing the system_tsproperty. Both supportSoftDeleteColumnDeletionDetectionPolicyfor tracking row deletions. - Push API batch and document limits. The Azure AI Search push API allows up to 1,000 documents or about 16 MB per batch, whichever limit is reached first. The 16 MB limit applies to the whole request, so one large document can fill a request on its own.
mergeOrUploadsemantics and collection replacement.mergeOrUploadcreates a document if it does not exist, or updates specified fields if it exists. When merging into aCollection(Edm.String)or complex collection, the operation replaces the entire collection; it does not append items.- Who deletes the chunks depends on who makes them. With index projections (an indexer skillset does the chunking), the indexer deletes a file's chunks when its deletion detection policy detects the deletion. With one-to-many parsing modes, or when your own worker pushes the chunks (Pattern 3), nothing deletes them for you: find the chunks by their parent file field and delete them.
- Service Bus buffering and dead-letter queues. Service Bus queues buffer ingestion spikes and decouple storage events from search indexing capacity. Messages that exceed
maxDeliveryCount(default: 10) move to the dead-letter queue ($deadletterqueue) with reasonMaxDeliveryCountExceeded. Messages in the dead-letter queue don't expire and aren't cleaned up automatically, so someone must monitor them.
4. Prerequisites and access¶
| Item | Detail |
|---|---|
| Azure resources | A storage account (Blob Storage or ADLS Gen2), an Azure AI Search service with a system-assigned managed identity, the target index already created, and a Service Bus namespace (Pattern 3) |
| Roles (least privilege) | Search Index Data Contributor (worker to Azure AI Search)Azure Service Bus Data Receiver (worker to Service Bus queue)Storage Blob Data Reader (worker to storage container). The nightly reconciliation needs the same search and storage roles.Storage Blob Data Reader for the search service's managed identity (the data source in Step 3 connects with it)Search Service Contributor (DevOps engineer creating data sources and indexers)Permission to create the event subscription on the storage account and to send messages to the queue (whoever runs Step 5). Give Event Grid Azure Service Bus Data Sender only if it delivers with a managed identity. |
| Tools | Azure CLI (az), curl, jq, envsubst, Python 3 with azure-search-documents, azure-servicebus, azure-storage-blob and azure-identity |
| Test data | Storage container with representative documents (PDF, DOCX, TXT) and test cases (create, update, rename, delete, restore, poison). The sample chunker in worker.py reads only .txt files (Step 6). |
| Index fields (Pattern 3) | id is the key, and is filterable and sortable (the reconciliation pages by key). source_url is filterable and holds the file URL on every chunk. |
Set up the environment variables:
export RESOURCE_GROUP="${RESOURCE_GROUP}"
export SUBSCRIPTION_ID="${SUBSCRIPTION_ID}"
export STORAGE_ACCOUNT="${STORAGE_ACCOUNT}"
export STORAGE_CONTAINER="${STORAGE_CONTAINER}"
export SEARCH_SERVICE="${SEARCH_SERVICE}"
export SEARCH_ENDPOINT="https://${SEARCH_SERVICE}.search.windows.net"
export INDEX_NAME="${INDEX_NAME}"
export DATASOURCE_NAME="${DATASOURCE_NAME}"
export INDEXER_NAME="${INDEXER_NAME}"
export SERVICEBUS_NAMESPACE="${SERVICEBUS_NAMESPACE}"
export SERVICEBUS_FQDN="${SERVICEBUS_NAMESPACE}.servicebus.windows.net"
export QUEUE_NAME="${QUEUE_NAME}"
export EVENT_SUBSCRIPTION_NAME="${EVENT_SUBSCRIPTION_NAME}"
export SOFT_DELETE_RETENTION_DAYS="${SOFT_DELETE_RETENTION_DAYS}"
export API_VERSION="2026-04-01"
5. Steps¶
Grouped into three implementation stages: choosing the pattern, configuring native indexer refresh, and building an event-driven worker pipeline.
Stage 1: Choose the refresh architecture pattern¶
Step 1. Choose the pattern that fits your freshness target and how your documents are chunked.
Evaluate the three refresh patterns:
| Pattern | Freshness | Architecture | Best used when |
|---|---|---|---|
| Pattern 1: Scheduled indexer (pull) | 5 minutes or more (the smallest indexer schedule) | Built-in Azure AI Search indexer with a schedule | An indexer can read the source and, with index projections, do the chunking. No custom worker. |
| Pattern 2: Event-triggered indexer (on-demand pull) | Depends on how fast each run starts and ends: measure it | Event Grid triggers an Azure Function that calls the indexer /run API |
Uploads are infrequent, and you want the indexer's extraction without waiting for the schedule. |
| Pattern 3: Event-driven queue and worker (push) | Depends on your worker: measure it (Step 9) | Event Grid → Service Bus queue → worker → push API | Your own code chunks and embeds the files, freshness must beat the indexer schedule, or ingest spikes need a queue. |
This how-to builds Pattern 1 (Stage 2) and Pattern 3 (Stage 3). Use one pattern for each index and container: if an indexer and the worker both write the same files, they overwrite each other. It does not build Pattern 2. If you choose it, note that a run request fails while another run of the same indexer is in progress, so group bursts of events before you call /run.
Verify: The selected pattern meets your freshness target and fits how your documents are chunked.
Stage 2: Configure native indexer change and deletion detection (Pattern 1)¶
Step 2. Enable native soft delete on Azure Blob Storage.
This step and Stage 3 cost money
Soft delete keeps deleted blobs for the retention period, and you pay for that storage. Event Grid, Service Bus and the worker's extraction and embedding calls are billed too. Check the current price before approval. To stop the Stage 3 costs, delete the event subscription and the queue. Do not turn soft delete off while the native soft delete policy is in use: the indexer would stop detecting deletions.
Make sure blob versioning is off, then turn on blob soft delete. Set ${SOFT_DELETE_RETENTION_DAYS} much longer than the indexer schedule, so the indexer always sees a deleted blob before it is purged.
az storage account blob-service-properties update \
--account-name "${STORAGE_ACCOUNT}" \
--resource-group "${RESOURCE_GROUP}" \
--enable-delete-retention true \
--delete-retention-days "${SOFT_DELETE_RETENTION_DAYS}"
Verify: az storage account blob-service-properties show --account-name "${STORAGE_ACCOUNT}" --resource-group "${RESOURCE_GROUP}" returns deleteRetentionPolicy.enabled: true with your days value, and isVersioningEnabled is false or missing.
Step 3. Create the data source with a deletion detection policy.
Create the data source before the indexer's first run. The snippet snippets/defs/datasource-blob.json connects with the search service's managed identity, which needs Storage Blob Data Reader (see Prerequisites). It uses ${...} placeholders, so fill them in with envsubst before you send it:
envsubst < snippets/defs/datasource-blob.json |
curl -sS -X PUT "${SEARCH_ENDPOINT}/datasources/${DATASOURCE_NAME}?api-version=${API_VERSION}" \
-H "Authorization: Bearer $(az account get-access-token --resource https://search.azure.com --query accessToken -o tsv)" \
-H "Content-Type: application/json" --data @-
Verify: reading the data source back shows the policy "#Microsoft.Azure.Search.NativeBlobSoftDeleteDeletionDetectionPolicy":
curl -sS "${SEARCH_ENDPOINT}/datasources/${DATASOURCE_NAME}?api-version=${API_VERSION}" \
-H "Authorization: Bearer $(az account get-access-token --resource https://search.azure.com --query accessToken -o tsv)" |
jq '.dataDeletionDetectionPolicy'
Step 4. Configure the indexer schedule and fault-tolerance parameters.
Create the indexer from snippets/defs/indexer-blob.json. It runs every 5 minutes (PT5M), the smallest interval allowed. maxFailedItems and maxFailedItemsPerBatch are example error tolerances: set your own. The document key needs no field mapping: by default it comes from metadata_storage_path, which the indexer encodes for you. If a skillset chunks the documents (index projections), add its name as "skillsetName" in the definition. The indexer runs once as soon as it is created.
envsubst < snippets/defs/indexer-blob.json |
curl -sS -X PUT "${SEARCH_ENDPOINT}/indexers/${INDEXER_NAME}?api-version=${API_VERSION}" \
-H "Authorization: Bearer $(az account get-access-token --resource https://search.azure.com --query accessToken -o tsv)" \
-H "Content-Type: application/json" --data @-
Verify: after the first run, this returns "status": "running" and "last": "success":
curl -sS "${SEARCH_ENDPOINT}/indexers/${INDEXER_NAME}/status?api-version=${API_VERSION}" \
-H "Authorization: Bearer $(az account get-access-token --resource https://search.azure.com --query accessToken -o tsv)" |
jq '{status, last: .lastResult.status}'
Stage 3: Build the event-driven queue and push pipeline (Pattern 3)¶
Step 5. Provision the Service Bus queue and Event Grid subscription.
Run snippets/infra/event-subscription.sh. It creates the queue (dead-lettering on, maximum delivery count 10) and an Event Grid subscription for BlobCreated, BlobDeleted and BlobRenamed events from ${STORAGE_CONTAINER} only. You need permission to create the subscription and to send messages to the queue (see Prerequisites).
bash snippets/infra/event-subscription.sh
Verify: az eventgrid event-subscription show --name "${EVENT_SUBSCRIPTION_NAME}" --source-resource-id "/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Storage/storageAccounts/${STORAGE_ACCOUNT}" --query "provisioningState" -o tsv returns Succeeded.
Step 6. Deploy the idempotent worker to process blob events.
Deploy snippets/refresh/worker.py as a long-running service. For every event it checks the file's current state. If the file exists, it chunks it, uploads the chunks, and then deletes the old chunks it no longer uses. If the file is gone, it deletes all its chunks. A renamed file is handled as a delete of the old name and an upload of the new one.
extract_and_chunk_blob is a sample for the sandbox only. It reads UTF-8 .txt files, cuts the text into chunks of 2,000 characters and computes no embeddings. Before you use the worker outside the sandbox, replace it with your own extraction, chunking and embedding code, and set SAMPLE_CHUNKER_IN_USE = False. While the sample is in place, the worker starts only when USE_SAMPLE_CHUNKER=true is set. Set it only in the sandbox.
# pseudo: the receive loop in snippets/refresh/worker.py
for message in receiver: # keeps waiting for messages until the worker is stopped
try:
handle_message(message, search_client, credential)
receiver.complete_message(message)
except PermanentError as exc: # a bad message or a 400: a retry won't help
receiver.dead_letter_message(message, reason="PermanentProcessingError", error_description=str(exc)[:1000])
except Exception: # transient: Service Bus dead-letters it after the max delivery count
receiver.abandon_message(message)
Events can arrive late, twice or out of order
Event Grid and Service Bus deliver at least once, and concurrent workers do not keep the order. The worker is safe because it reads the file's current state instead of trusting the event, and it deletes old chunks only after the new ones are uploaded. If you change it to use the event content, store each event's sequencer on the chunks and skip older events: sequencer values for the same blob name can be compared as strings. Directory renames (DirectoryRenamed) are not handled: the nightly reconciliation (Step 8) finds them.
Verify: upload a test .txt file to ${STORAGE_CONTAINER}. The worker log shows Synced <file URL>: N chunks now and Successfully settled message.
Step 7. Handle partial batch failures (HTTP 207) and transient errors.
A batch can partly fail (HTTP 207). The worker checks every item. Per-item 422 (index temporarily unavailable) and 503 (service too busy) are transient, so the message is retried later. Any other failure is permanent, so the message goes to the dead-letter queue.
# pseudo: check_results in snippets/refresh/worker.py
failed = [r for r in results if not r.succeeded]
if failed and all(r.status_code in (422, 503) for r in failed):
raise RuntimeError("transient") # abandon: Service Bus delivers the message again
if failed:
raise PermanentError("permanent") # dead-letter the message
Verify: in the sandbox, send one test batch with valid documents plus a merge for a key that does not exist. That item returns statusCode 404 and status false, and the other items succeed.
Step 8. Schedule nightly reconciliation to detect drift.
Run snippets/refresh/reconcile.py every night, for example from a scheduled pipeline. It lists orphans (chunks whose file is gone) and missing files (files with no chunks). It catches lost events, directory renames and restored files.
python snippets/refresh/reconcile.py
Verify: the last line is Orphan files: N, orphan chunks: N, missing files: N. The exit code is 0 when there is no drift and 2 when there is.
Deleting orphans removes chunks from the index
Run the report first and check the ORPHAN lines. Rollback: upload the file again, and the worker indexes it again.
Delete the orphans. Then upload the missing files again (a copy also raises BlobCreated), so the worker indexes them:
python snippets/refresh/reconcile.py --delete-orphans
Verify: the output shows Deleted N of N orphan chunks., and a new report shows orphan chunks: 0.
Step 9. Document measured freshness latency.
The worker logs each event's eventTime on its Synced line. For each test upload, search ${INDEX_NAME} for the new content until it appears, and record that time minus eventTime. Write down the freshness target (${FRESHNESS_TARGET}) and the measured delay.
Verify: the p95 of the measured delays over your test uploads is recorded and meets ${FRESHNESS_TARGET}.
6. Verify it works¶
Run these scenarios in the sandbox. Each one says which pattern it tests. For Pattern 3, start the worker with USE_SAMPLE_CHUNKER=true, unless you replaced the sample chunker (Step 6). The sample makes one chunk for every 2,000 characters.
- Blob create (both patterns): Upload a text file
policy.txtof 9,000 to 10,000 characters to${STORAGE_CONTAINER}. Verify: the index has all its chunks within your freshness target (Pattern 1: after the next indexer run; Pattern 3: measure it in Step 9). With the sample chunker, that is 5 chunks. Write down the file's document keys for scenario 3. - Blob update with fewer chunks (both patterns): Replace
policy.txtwith a version of 3,000 to 4,000 characters. Verify: the chunk count drops to match the new version (with the sample chunker, from 5 to 2), and no old chunks are left. - Blob delete (both patterns): Delete
policy.txtfrom${STORAGE_CONTAINER}. Verify: Pattern 3: the filtersource_url eq '<URL of policy.txt>'returns 0 documents. Pattern 1: after the next indexer run, a lookup of its document keys returns 404. - Blob undelete (both patterns): Undelete
policy.txt. Verify: Pattern 1: after you re-save its metadata (this updatesLastModified), the next indexer run indexes it again. Pattern 3: undelete and metadata changes raise no event, so upload the file again or run the reconciliation (Step 8); then its chunks are back. - Rename (Pattern 3, ADLS Gen2 only): Rename
policy.txttopolicy-renamed.txt. Verify: the old URL has 0 chunks, and the new URL has the file's chunks. - Poison message (Pattern 3): Send a message that is not JSON to
${QUEUE_NAME}(for example with Service Bus Explorer in the portal). Verify: the worker moves it to$deadletterqueueat once (reasonPermanentProcessingError), and the dead-letter alert fires. - Reconciliation (Pattern 3): Run
reconcile.pyon the sandbox container and index. Verify: the report showsorphan chunks: 0andmissing files: 0.
7. Known issues and fixes¶
| Symptom | Likely cause | Fix |
|---|---|---|
| Files in a renamed ADLS Gen2 directory are not reindexed (Pattern 1) | Renaming a directory in ADLS Gen2 does not update the timestamps of the blobs in it | Update the LastModified timestamp on all blobs within the directory by resaving their metadata, or trigger a targeted reprocess. |
| Deleted files still appear in search results | The deletion policy was missing at the first indexer run, a one-to-many parsing mode is used (soft-delete policies don't apply to it), or a delete event was lost | For indexers: build a new index with a new indexer that has the policy from the first run. For parsing modes and push workers: run the reconciliation (Step 8) with --delete-orphans. |
| Restored (undeleted) blobs are ignored by the indexer (Pattern 1) | Undeleting a soft-deleted blob restores content but does not bump its LastModified timestamp |
Resave metadata on the restored blob to update LastModified so the indexer picks it up. |
| A restored or re-saved file never reaches the index (Pattern 3) | Undelete and metadata changes raise no Event Grid event; only uploads and copies raise BlobCreated |
Upload the file again, or run the reconciliation (Step 8) to find it. |
| Files in a renamed ADLS Gen2 folder keep their old chunks (Pattern 3) | The worker does not handle DirectoryRenamed events |
Run the reconciliation (Step 8): it reports the old URLs as orphans and the new ones as missing. Delete the orphans, then upload the missing files again. |
| Older content overwrites newer content | A worker indexes the event's content instead of the file's current state, and events arrive out of order | Read the current file for every event, as worker.py does. If you must use the event content, store the event sequencer and skip older events. |
| An updated file leaves old chunks behind | The new version has fewer chunks than the old one | Upload the new chunks first, then delete the old keys that are not in the new set, as worker.py does. |
The worker stops at start with worker.py still uses the sample chunker |
extract_and_chunk_blob is still the sample, and USE_SAMPLE_CHUNKER is not true |
Replace extract_and_chunk_blob with your own code and set SAMPLE_CHUNKER_IN_USE = False (Step 6). In the sandbox only, set USE_SAMPLE_CHUNKER=true. |
Messages go to the dead-letter queue with The sample chunker reads only ... |
The sample chunker is in use, and the file is not a UTF-8 .txt file |
Replace the sample with your own extraction code (Step 6), or test with UTF-8 .txt files only. |
| Service Bus Dead-Letter Queue grows unnoticed | DLQ messages do not expire and have no automated consumer or alerting | Set an alert on the DeadletteredMessages metric; assign operational ownership for daily review. |
| Worker receives HTTP 207 Multi-Status with partial failures | Some items failed: per-item 422 (index temporarily unavailable) and 503 (service too busy) are transient; others, such as 400, are not | Retry the transient failures (the worker abandons the message, so Service Bus delivers it again) and dead-letter the rest. |
| Native blob soft delete fails on Azure Files | Native blob soft delete is supported only on Azure Blob Storage and ADLS Gen2 | Use SoftDeleteColumnDeletionDetectionPolicy with a custom metadata flag (IsDeleted) for Azure Files. |
8. Monitoring and alerts¶
The thresholds below are illustrative team defaults, not platform recommendations. Tune them against a measured baseline. Set the placeholders to the values your team agrees.
| Signal | Source | Alert when | Action |
|---|---|---|---|
DocumentsProcessedCount, split by Failed (Pattern 1) |
Azure Monitor (Azure AI Search) | More than 0 failed documents in an indexer run (team default) | Read lastResult.errors from Get Indexer Status, fix the cause, and follow PB-RAG-05-02 if it keeps failing. |
| Nightly reconciliation result (Pattern 3) | Scheduled pipeline running reconcile.py |
Exit code 2 (drift found) or 1 (a delete failed) | Read the report: delete the orphans and upload the missing files again (Step 8). Repeated drift means events are being lost: check the worker and the dead-letter queue. |
DeadletteredMessages |
Azure Monitor (Service Bus queue) | More than 0 for 15 minutes (team default) | Inspect messages in $deadletterqueue using Service Bus Explorer or CLI; fix root cause and resubmit. |
ActiveMessages |
Azure Monitor (Service Bus queue) | Above ${QUEUE_BACKLOG_THRESHOLD} for 15 minutes (team default) |
Check worker availability, scale worker instances, or inspect search service indexing latency. |
ThrottledSearchQueriesPercentage |
Azure Monitor (Azure AI Search) | Above ${THROTTLE_THRESHOLD} for 5 minutes while the worker indexes (team default) |
Indexing and queries share the same resources, and throttled push requests return 503 to the worker. Slow the worker down (fewer messages at once, longer back-off) or add search units. |
SearchLatency |
Azure Monitor (Azure AI Search) | Above 1.5x baseline for 10 minutes | Throttle background indexing worker concurrency to protect search query latency. |
| End-to-end freshness latency | Worker custom metric / log | p95 above freshness target for 1 hour | Check queue backlog, worker extraction duration, and search push batch latency. |
9. AWS and open-source equivalents¶
| Platform | Equivalent | Key differences |
|---|---|---|
| AWS (Amazon Bedrock Knowledge Bases) | Direct ingestion with IngestKnowledgeBaseDocuments and DeleteKnowledgeBaseDocuments, driven by S3 event notifications to a queue. |
Changes go into the vector store without a sync. For an S3 data source they are not written back to S3, so make the same change in S3, or the next StartIngestionJob sync overwrites it. Don't run direct ingestion and a sync at the same time. |
| AWS (Amazon OpenSearch Service) | S3 event notifications routed via EventBridge / SQS to AWS Lambda or ECS worker pushing to OpenSearch index. | You build the worker yourself: extraction, chunking, embedding, idempotency and bulk indexing. |
| Open source (pgvector / OpenSearch) | Storage events (MinIO / S3 webhook or Kafka) consumed by Celery or Kafka worker pushing upserts and deletes to PostgreSQL/pgvector or OpenSearch. | Complete control over chunking and embeddings; developer must manage DLQ, schema migrations, vacuuming (Postgres), and index segment merging (OpenSearch). |
10. Validation record¶
Pending. Filled in by the dry-runner (see docs/validation-protocol.md).
Sandbox scenario: Test incremental refresh and deletion detection across Azure Blob Storage and Azure AI Search using both indexer native soft delete (Pattern 1) and an Event Grid -> Service Bus -> worker push pipeline (Pattern 3). Scenarios include blob create, update, rename (ADLS Gen2), delete, undelete/restore, and a malformed poison payload.
Pass criteria:
- Created blob chunks are searchable within your freshness target (
${FRESHNESS_TARGET}). - Soft-deleted blob causes all corresponding chunk documents to be deleted from the search index.
- Updated blob with fewer chunks leaves no old chunks in the index.
- Undeleted blob is indexed again: Pattern 1 after its metadata is re-saved, Pattern 3 after a new upload or the reconciliation.
- A renamed file (ADLS Gen2) is indexed under its new name only.
- Poison payload moves to Service Bus dead-letter queue and triggers the DLQ alert.
reconcile.pyreports zero orphan chunks and zero missing files.
Record¶
| Item | Detail |
|---|---|
| Playbook | PB-RAG-09-04 at commit [hash] |
| Dry-run by | [Name] (must not be author or reviewer) |
| Date | YYYY-MM-DD |
| Sandbox | [Resource group or tag, tier and region] |
| Scenario | [Scenario followed] |
| Time taken | [Hands-on time] hands-on, [waiting time] waiting |
| Result | [PASS / PASS WITH NOTES / FAIL] |
| Notes | 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged] |
| Signed off by | [Reviewer name and date] |
11. Sources¶
Checked on 3 and 4 November 2026.
- Microsoft Learn, Change and delete detection using indexers for Azure Storage: Native blob soft delete, custom metadata soft delete, LastModified tracking, ADLS Gen2 directory rename limitation, restore metadata touch requirement, one-to-many limitation.
- Microsoft Learn, Configure Service Bus as an Event Grid Handler: Event Grid subscription to Service Bus queue, Azure CLI commands, delivery properties, the permission to send to the queue, and delivery with a managed identity.
- Microsoft Learn, Azure Blob Storage event schema: BlobCreated (raised only by uploads and copies), BlobDeleted, BlobRenamed and DirectoryRenamed (ADLS Gen2), the
subjectformat, andsequencercomparison for the same blob name. - Microsoft Learn, Overview of Service Bus dead-letter queues: DLQ subqueue addressing, MaxDeliveryCountExceeded, application dead-lettering, lack of DLQ message expiration.
- Microsoft Learn, Connecting Azure SQL to Azure AI Search using indexers: SqlIntegratedChangeTrackingPolicy and HighWaterMarkChangeDetectionPolicy for SQL.
- Microsoft Learn, Index data from Azure Cosmos DB using Azure AI Search indexers: HighWaterMarkChangeDetectionPolicy using
_tsproperty. - Microsoft Learn, Add, Update or Delete Documents (REST API 2026-04-01): Push batch limit (1,000 docs, 16 MB), mergeOrUpload behaviour, collection replacement on merge, 207 Multi-Status response structure, per-item status codes (422 and 503 are transient).
- Microsoft Learn, Service limits in Azure AI Search: 1,000 documents per batch, 16 MB per request, minimum schedule of 5 minutes, at most 1,000 results per page.
- Microsoft Learn, Schedule indexer execution in Azure AI Search (checked on 4 November 2026): smallest interval 5 minutes; a run still going skips the next scheduled run.
- Microsoft Learn, Define index projections (checked on 4 November 2026): chunks are deleted with their parent only when a deletion detection policy detects the deletion.
- Microsoft Learn, Azure Blob indexer (checked on 4 November 2026): the key comes from
metadata_storage_pathby default and is encoded automatically; the indexer runs when it is created; error tolerance parameters. - Microsoft Learn, Connect to Azure Storage using a managed identity (checked on 4 November 2026):
Storage Blob Data Readerfor the search service's identity, and theResourceIdconnection string. - Microsoft Learn, Azure AI Search monitoring data reference (checked on 4 November 2026):
ThrottledSearchQueriesPercentage, and that indexing and queries share the same resources. - Microsoft Learn, Shape search results in Azure AI Search (checked on 4 November 2026): paging through many results with
orderbyand a range filter on a unique key. - AWS Documentation, Ingest changes directly into a knowledge base (Amazon Bedrock): Direct ingestion vs sync, KnowledgeBaseDocuments API operations, S3 sync overwrite warning.
- AWS Documentation, DeleteKnowledgeBaseDocuments (checked on 4 November 2026): deletes documents from the data source and syncs the knowledge base.
Related playbooks¶
- PB-RAG-05-02: Recover from indexer and skillset failures and throttling: handling ingestion failures and throttling.
- PB-RAG-09-01: Reindex an Azure AI Search index without downtime: full reindexing with aliases when schema changes occur.
- PB-RAG-09-05: Deletion and permission change propagation checklist: verifying deletion compliance and security trimming.