Skip to content
⚠️ Before You Start
Risk: High
Duration: 1-2 days per archetype including a test index
Last Validated: Not yet validated (draft)
Kind: Decision guide
Risk Note: High (most attribute and analyser choices cannot be changed without a rebuild)

Choose fields, analysers, vector profiles and ranking for four corpus archetypes

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-12-07). Author: Yogeshwaran · dry-run: Kranthi · sign-off: Deeparani. Work items: RAG-03-T01, RAG-03-T02, RAG-03-T03, RAG-03-T04.

1. Summary

This guide settles the schema of a new Azure AI Search index before any data goes in. It gives one starting template for each of four corpus archetypes: policy and prose, technical manuals, tabular records, and multilingual content. For each template it fixes the field attributes, the analyser per searchable field, the vector profile, the semantic configuration and any scoring profile.

The default for all four is a hybrid index (keyword and vector fields together) with a semantic configuration, a language analyser on prose fields, HNSW with scalar quantisation, and no scoring profile unless a business rule needs one. Most of these choices cannot be changed without a rebuild, so prove them on a test index and record them first.

Facts checked against Microsoft Learn and AWS documentation on 29 October 2026, stable REST API 2026-04-01. The templates use no preview feature. The only preview feature named in this guide is scoring profile functionAggregation set to product (preview, 2026-08-01-preview).

2. When to use and when not to

Use this guide when:

  • You create a new production index.
  • You add a new corpus type to an existing solution.
  • A schema review finds an attribute or analyser that does not match how the application queries the index.
  • The embedding model changes and you must design the new vector field (then carry out the change with PB-RAG-09-02).

Do not use it when:

  • You only add a field, a scoring profile, a synonym map or a semantic configuration. These are in-place updates (Update or rebuild an index). Apply them with a normal change.
  • You are moving a live index to a new schema. Use PB-RAG-09-01.
  • You are re-embedding the corpus with a new model. Use PB-RAG-09-02.
  • You are choosing between keyword, vector, hybrid and re-ranking at query time. Use PB-RAG-08-01.
  • You are preparing the index for go-live (alerts, replicas, capacity). Use PB-RAG-03-04.
  • You are choosing chunk size and overlap. Use PB-RAG-01-01.

3. What you must know first

  1. Most of the schema is fixed at creation. Field names and types, and the searchable, filterable, sortable and facetable attributes, cannot change on an existing field. Neither can stored, suggesters or the analyser assignment. retrievable, scoring profiles, synonym maps, semantic configurations and new fields can be added or changed with no rebuild. (Create an index, Update or rebuild an index)
  2. Analysers are set when the field is created. You cannot assign analyzer or indexAnalyzer to an existing field; only searchAnalyzer can change. The default is the Standard Lucene analyser. A language analyser must be set with analyzer and is used for both indexing and queries. There are 35 Lucene and 50 Microsoft language analysers. Microsoft analysers add lemmatisation and decompounding but index on average two to three times slower. (Analyzers, Language analyzers)
  3. Filters, facets and sorts are not analysed. Filterable strings match exactly and are case-sensitive. A normaliser such as lowercase fixes case for filter, facet and sort. A normaliser can only be set when the field is added. (Normalizers)
  4. Vector fields need matching dimensions and a profile. The type is usually Collection(Edm.Single). dimensions must equal the embedding model output (text-embedding-ada-002 is fixed at 1536, text-embedding-3-small allows 1 to 1536, text-embedding-3-large allows 1 to 3072). searchable must be true, and filterable, facetable and sortable must be false. Azure OpenAI models use the cosine metric. The indexer rejects documents whose vector dimensions do not match. (Create a vector index)
  5. Compression and stored cut vector cost. Scalar quantisation (int8) reduces vector index size fourfold. Binary quantisation is up to 28 times smaller and works best above 1024 dimensions. Rescoring with the original vectors and oversampling offset the loss. Quantisation needs a new profile on a new field. "stored": false saves up to 50% storage per field, is irreversible, and makes partial updates drop the vector unless you send it again. (Quantization, Storage options)
  6. Hybrid search merges results with Reciprocal Rank Fusion (RRF). The semantic ranker then re-ranks only the top 50 results, so set k to 50 on vector queries when you use it. A semantic configuration has one title field, and content and keyword fields in priority order. Lower priority fields may be truncated. Fields must be searchable and retrievable strings. (Hybrid search, Hybrid query, Semantic configuration)
  7. Scoring profiles boost nonvector fields only. Functions (freshness, magnitude, distance, tag) need filterable fields. One profile applies per query. With the semantic ranker, rankingOrder decides whether the profile is applied again after re-ranking (boostedRerankerScore, the default) or not (rerankerScore). Only functions boost re-ranked results; text weights do not. (Scoring profiles, Scoring profiles with semantic ranker)

4. Options

How to reach a schema

Follow these steps for each archetype. The options below are the starting templates for step 1.

  1. Pick the archetype template. Match the corpus to Option A, B, C or D and copy its template from snippets/. Verify: The decision record names one archetype and the template file and commit.
  2. List every field and justify each flag. For each filterable, facetable and sortable flag, name the filter, facet, sort or scoring function that needs it. Add the security trimming field if PB-RAG-04 applies. Verify: Every flag in section 2 of the decision record has a reason. No flag says "might need later".

  3. Choose and test an analyser per searchable field. Run the Analyze Text API on real sample text for each candidate analyser:

    curl -sS -X POST "${SEARCH_ENDPOINT}/indexes/${TEST_INDEX}/analyze?api-version=${API_VERSION}" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d "{\"text\": \"${SAMPLE_TEXT}\", \"analyzer\": \"${ANALYZER}\"}" | jq '.tokens[].token'

    Verify: The response lists tokens. Part numbers, codes and product names come back as whole tokens where the template uses keyword. Inflected words reduce to the forms users type. 4. Set the vector field and profile. Set dimensions to the embedding model output, metric to the model's metric, and keep the HNSW defaults (m 4, efConstruction 400, efSearch 500) unless the benchmark says otherwise. Choose scalar quantisation with rescoring as the default. Verify: dimensions in the template equals the dimensions in the embedding deployment or skill. The profile names an algorithm and a compression that exist in vectorSearch. 5. Decide stored and retrievable for vector fields. Set "stored": false only if every update sends the full document, including the vector. Keep "stored": true if the pipeline uses merge or mergeOrUpload without the vector. Verify: The decision record names the update pattern (full upload, indexer, or partial merge) and the stored value it allows. 6. Add a semantic configuration. Set the title field, the content fields in priority order and any keyword fields. Set rankingOrder. Verify: Every field in the configuration is searchable and retrievable, and the configuration name is the one the application sends. 7. Add scoring profiles only for a business rule. Examples: newer documents first (freshness), exact code matches first (text weights). Write the rule in the decision record. Verify: Each profile maps to a written rule, and each function field is filterable. 8. Create a test index, load a sample and run the golden set. See the warning below, then follow the steps in PB-RAG-02-01. Verify: The test index document count equals the sample chunk count. The golden set run gives recall@k and nDCG@k for the baseline and the candidate. GET /indexes/${TEST_INDEX}/stats returns storageSize and vectorIndexSize. 9. Record the schema and the reasons. Fill in snippets/index-decision-record.md and get RAG Lead sign-off. Verify: The decision record has all eight sections filled and a dated sign-off.

Step 8 spends money and uses shared quota

A test index uses storage and vector quota on the search service, and loading it calls the embedding model. Use the sandbox service, never a production service or a production index. Use the sample, not the full corpus. Delete the test index when the decision record is signed (DELETE /indexes/${TEST_INDEX}), or set the sandbox expires tag (see docs/sandbox.md). Deleting the test index does not affect any other index.

Option A: Policy and prose

Policies, HR guidance, statutory guidance and reports. Long text in one language with headings. Template: snippets/index-policy.json.

  • Fields: title, section_path and content are searchable with en.microsoft. policy_area and status are filterable and facetable with the lowercase normaliser. effective_date is filterable and sortable.
  • Ranking: hybrid plus semantic configuration with rankingOrder set to rerankerScore. No scoring profile. Exclude superseded versions with a filter on status, not with a boost.
  • Vectors: scalar quantisation, "stored": false.
  • Risk: a missing status field means superseded policies compete with current ones. Adding the field later needs no rebuild, but every document must be reloaded to populate it.

Option B: Technical manuals

Manuals, runbooks, standards and specifications with numbered steps, tables and part numbers. Template: snippets/index-manuals.json.

  • Fields: prose fields use en.microsoft. part_numbers is a searchable collection with the keyword analyser so each code stays one token, and filterable with the lowercase normaliser. product is filterable and facetable.
  • Ranking: hybrid plus semantic configuration with part_numbers as a keyword field. A freshness profile on last_modified with rankingOrder set to boostedRerankerScore, only if newer revisions must win.
  • Vectors: scalar quantisation, "stored": false.
  • Risk: without a keyword field, a language analyser may split codes such as AB-1234 into several tokens. Check with the Analyze Text API in step 3.

Option C: Tabular records

FAQs, tickets, catalogue entries, forms and invoices. Short records with many structured fields. Template: snippets/index-records.json.

  • Fields: record_code is searchable with keyword and filterable and sortable with lowercase. Structured values (category, status, tags, amount, updated_at) are filterable, and facetable or sortable only where the UI needs it. title and content keep the Standard Lucene analyser unless the Analyze Text test shows a language analyser helps.
  • Ranking: hybrid plus semantic configuration with record_code and tags as keyword fields. A text-weighted profile (code-first) so exact code matches rank first. Text weights do not boost semantically re-ranked results, so rankingOrder is rerankerScore.
  • Vectors: scalar quantisation, "stored": true, because record updates are often partial merges of status fields.
  • Risk: structured fields multiply quickly. Each filterable field needs more storage. Only flag what a named query uses.

Option D: Multilingual content

The same corpus in several languages. Template: snippets/index-multilingual.json.

  • Fields: the blended pattern from Create an index for multiple languages: one field per language (content_en, content_fr, content_de), each with its own language analyser, plus a filterable language field. Query with searchFields and select set to one language's fields.
  • Alternative: one index per language when all searchable content is in the same language and the languages are served separately.
  • Ranking: one semantic configuration per language (sem-en, sem-fr, sem-de). Pass the matching name at query time.

  • Vectors: one vector field from a multilingual embedding model. A vector query can match across languages without analysers or translation, if the embedding space holds multilingual content. Microsoft reports that text-embedding-3-small and text-embedding-3-large give better average multi-language retrieval than text-embedding-ada-002 on the MIRACL benchmark (Azure OpenAI embedding models). The docs do not list supported languages.

  • Risk: adding a new language later means adding new fields (no rebuild) and reloading the documents in that language. Removing a language field needs a rebuild.

Comparison

A: Policy and prose B: Technical manuals C: Tabular records D: Multilingual
Prose analyser en.microsoft en.microsoft Standard Lucene (test a language analyser) One language analyser per field
Exact-match fields None part_numbers (keyword) record_code (keyword) None
Normalised filters policy_area, status product, part_numbers category, status, tags, record_code language
Semantic configurations 1 1 (with keyword field) 1 (with keyword fields) 1 per language
Scoring profile None freshness, optional Text weights None
rankingOrder rerankerScore boostedRerankerScore rerankerScore rerankerScore
Vector stored false false true false
Main rebuild risk Missing status or date field Codes split by the analyser Over-flagged fields New or removed languages

The templates set dimensions to 1536 as a placeholder for a 1536-dimension model. Change it to your model's output before use. The numbers in the templates that are not platform defaults are starting values to test, not recommendations: boost 2 and P365D in fresh, weights 5 and 2 in code-first, and defaultOversampling 4 (the platform default). The HNSW values are the documented defaults.

Excerpt: vector field and profile (all templates)

The full templates are in snippets/. The vector part is the same in each:

{
  "name": "content_vector",
  "type": "Collection(Edm.Single)",
  "searchable": true,
  "retrievable": false,
  "stored": false,
  "dimensions": 1536,
  "vectorSearchProfile": "hnsw-sq"
}

5. Decision table

Situation Choose Why
Long English prose with headings: policies, guidance, reports Option A Language analyser handles word forms. Filters remove superseded versions without distorting relevance.
Manuals or specifications that users search by part number or code Option B A keyword field keeps codes as one token. Hybrid search finds exact codes that vector search alone misses.
Short structured records: FAQs, tickets, forms, catalogue items Option C Exact code match and structured filters matter more than prose analysis.
The same content in two or more languages Option D A language analyser per field, and one semantic configuration per language.
One language per corpus, served to separate audiences Option D, one index per language Simpler queries; each index uses one analyser.
A field might be needed for a filter "one day" Leave it out A new field can be added later with no rebuild. Reload the documents to fill it.
The UI needs case-insensitive filters or facets Add the lowercase normaliser at creation Filters are case-sensitive, and a normaliser cannot be added to an existing field.
Codes or IDs must match exactly in full-text search keyword analyser on a dedicated field Language and standard analysers split on hyphens and punctuation.
The pipeline uses merge or mergeOrUpload without the vector "stored": true With "stored": false, the vector is lost on partial updates without an error.
Embedding dimensions above 1024 and vector quota is tight Binary quantisation with rescoring; test recall first Up to 28 times smaller. Microsoft recommends testing quality below 1024 dimensions.
Newer documents must rank first even after semantic re-ranking freshness function and rankingOrder boostedRerankerScore Only functions boost re-ranked results.
A boost exists only to hide outdated content A filter instead of a scoring profile A boost still returns outdated content; a filter removes it.
You want to tune ranking after go-live Scoring profiles and semantic configurations Both can be added or changed with no rebuild.
Dimensions, analyser or a filterable flag must change on a live index Rebuild with PB-RAG-09-01 These changes cannot be made in place.

Default for every archetype:

  • Hybrid index: human-readable text fields plus one vector field per embedded text.
  • en.microsoft (or the matching Microsoft language analyser) on prose fields, confirmed with the Analyze Text API against the Lucene equivalent.
  • keyword analyser on dedicated code fields, lowercase normaliser on every filterable or facetable string.
  • HNSW with the documented defaults and cosine for Azure OpenAI embeddings.
  • Scalar quantisation (int8) with enableRescoring and preserveOriginals.
  • "stored": false on vector fields when every update sends the full vector, otherwise "stored": true.
  • One semantic configuration. Vector queries use k 50 when the semantic ranker is on.
  • No scoring profile unless a written business rule needs one.

Deviate when:

  • The language analyser hurts recall in the Analyze Text test or the benchmark. Try the Lucene analyser, or keep Standard Lucene.
  • Indexing time matters more than linguistic quality. Lucene analysers index faster than Microsoft analysers.
  • Vector quota is the limit and dimensions are above 1024. Test binary quantisation. For text-embedding-3 models, also test truncated dimensions (MRL) together with quantisation (Optimize vector storage). Set it with truncationDimension on the compression (Quantization). One storage page spells it truncateDimension; the quantisation page and the Azure SDK references use truncationDimension.
  • A downstream process needs the raw vectors. Keep "stored": true and "retrievable": true.
  • Recall with quantisation is below the uncompressed baseline. Raise defaultOversampling before dropping compression.
  • The content is not prose (codes, logs, short records). The semantic ranker works best on information-rich prose (Semantic ranking). Benchmark with and without it.

Record every deviation, with the benchmark evidence, in the decision record.

Rule of thumb: flag only what a named query needs, but decide analysers, normalisers, dimensions and stored as if they can never change, because without a rebuild they cannot.

7. Known issues and fixes

Symptom Likely cause Fix
Filter on a field returns an error Field was not marked filterable Plan a rebuild with the corrected attribute; add a new field if a rebuild must wait
Searches for word stems miss documents Standard analyser used on language content Use a language analyser on a new field or rebuild
Vector storage quota reached early Full-precision vectors stored and retrievable Use quantisation and set stored to false where raw vectors are not needed
A filter or facet misses values that differ only by case Filterable string has no normaliser Add a new field with the lowercase normaliser and reload it; drop the old field at the next rebuild
A part number search returns unrelated chunks The analyser split the code into several tokens Add a dedicated field with the keyword analyser; check tokens with the Analyze Text API
Indexer reports errors for documents with vectors Vector dimensions differ from the field's dimensions Match the embedding deployment's dimensions; changing a field's dimensions needs a new field or a rebuild
Vector results disappear after a status update "stored": false and the update used merge without the vector Send the full vector in every partial update, or use "stored": true in the next schema version
Index update not allowed because it would cause downtime Adding a custom analyser, tokenizer or filter to an existing index Add it in a new index version. If you must update in place, set allowIndexDowntime=true in a maintenance window: the index goes offline for at least a few seconds and queries and indexing fail meanwhile
Freshness boost has no effect on semantic queries rankingOrder is rerankerScore, or the profile has only text weights Set rankingOrder to boostedRerankerScore and use a function
Scoring profile not applied The query does not name the profile and there is no defaultScoringProfile Add scoringProfile to the request, or set defaultScoringProfile
Semantic captions ignore the main text Content field is low in priority or not retrievable Put the main content field first in prioritizedContentFields; make sure it is retrievable
Semantic ranker results look no better than RRF Vector query k below 50, or filters leave few results Set k to 50; check that filters leave enough results for re-ranking

8. Monitoring and alerts

These signals show whether the schema choice holds up and when to revisit it. The thresholds are team defaults for the author to set, not platform recommendations.

Signal Source Alert when Action
IndexVectorUsage against the vector quota Platform metric (Maximum, by IndexName) Above the agreed planning threshold Review compression and stored, or plan more partitions with PB-RAG-03-04
IndexStorageUsage against the storage quota Platform metric (Maximum, by IndexName) Above the agreed planning threshold Remove unneeded filterable or searchable flags at the next rebuild; plan capacity
SearchLatency Platform metric (Average) Above the baseline by the agreed margin for the agreed window Check efSearch, oversampling and the number of vector fields queried
Golden set recall@k on the index CI evaluation (PB-RAG-02-01) Falls below the recorded baseline by the agreed tolerance Block the change; compare the analyser and vector settings with the decision record
Indexer document errors Indexer execution history Any error mentioning vector dimensions Stop the indexer; align the embedding dimensions with the field

9. Cost and capacity notes

  • Vector quota is memory. HNSW vector indexes stay in memory and count against the vector quota, which is per partition and depends on the service creation date and tier. Exhaustive KNN does not use vector quota. Check current limits in Service limits and Vector index size.
  • Estimate raw vector size as documents × dimensions × bytes per value (4 for Edm.Single, 2 for Edm.Half). HNSW adds overhead on top (Vector index size).
  • Compression is the largest lever. Scalar quantisation cuts vector index size fourfold. "stored": false saves disk, not memory.
  • Each attribute costs storage. Searchable fields use extra space, and Microsoft advises setting filterable to false on fields you will not filter, to reduce index size (Create Index REST API). Do not flag fields "just in case".
  • Semantic ranker is a premium feature at additional cost (Transparency note). Check the current price before approval.
  • Microsoft language analysers index on average two to three times slower than Lucene ones. Plan load windows for large corpora.
  • A rebuild needs room for two indexes until the old one is deleted. Plan capacity before any cut-over (PB-RAG-09-01).

10. AWS and open-source equivalents

Platform Equivalent Key differences
Amazon OpenSearch Service Index mappings with text fields and analysers, keyword fields, and knn_vector fields; index.knn set to true Set dimension on each knn_vector field. There are no semantic configurations or scoring profiles with the same names. You cannot change the type of a field that already holds data: create a new index with the new mappings, reindex into it, and use aliases to limit downtime (Create or update mappings). New fields can be added.
Amazon Bedrock Knowledge Bases Vector store created by the quick-create flow, or one you create (OpenSearch Serverless, OpenSearch managed clusters, S3 Vectors, Aurora and others) With your own store you set the vector index name, vector field, dimensions to match the embedding model, and the text and metadata fields (Vector store prerequisites). There are no analyser, semantic configuration or scoring profile settings. For Aurora, the field mappings cannot be updated after the knowledge base is created.
OpenSearch or Elasticsearch (self-managed) Same mappings, analysers and vector field types Same rebuild rule for mapping changes. Hybrid scoring is configured in the query or a search pipeline, not in the index.
Qdrant, pgvector Collections or tables with vector columns Vector size and distance are set at creation. Keyword search, analysers and filters are separate choices, often a separate engine.

11. Validation record

Pending. Planned dry-run scenario: A second engineer is given a new corpus description and must produce a schema with this guide. Pass if a senior finds no attribute that would force a rebuild for the listed user interface needs, and the test index passes the golden set.

12. Sources

Checked on 29 October 2026.

  • PB-RAG-03-04: index go-live and capacity checklist (run after this guide).
  • PB-RAG-09-01: reindex without downtime (carries out a schema change that needs a rebuild).
  • PB-RAG-09-02: upgrade the embedding model (new vector field and dimensions).
  • PB-RAG-08-01: choose the retrieval mode at query time.
  • PB-RAG-02-01: golden set and recall@k for step 8.
  • PB-RAG-04-02: the security filter field.
  • PB-RAG-01-01: chunking choices that feed this schema.