Skip to content
⚠️ Before You Start
Risk: Low
Duration: 2-3 days for a first tuning round
Last Validated: Not yet validated (draft)
Kind: How-to
Risk Note: Low (sandbox tuning; production risk is controlled by PB-RAG-08-04)

Tune hybrid weights and re-ranking against a golden query set

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-12-16). Author: Gayathri Ramachandran · dry-run: Yasar · sign-off: Deeparani. Work items: RAG-08-T05, RAG-08-T06, RAG-08-T07, RAG-08-T08.

Catalog point to cover

The catalog maturity target is 'benchmarked on three corpus types': run the tuning on three sample corpora from the sandbox (FND-04) and record the results.

1. Summary

Before go-live, and after any change to chunking, embeddings, analysers or ranking settings.

Risk: Low (sandbox tuning; production risk is controlled by PB-RAG-08-04). Typical duration: 2-3 days for a first tuning round.

2. When to use and when not to

Use it when: Before go-live, and after any change to chunking, embeddings, analysers or ranking settings.

Do not use it when: to be written by the author.

3. What you must know first

  • nDCG@k and recall@k measure ranking quality against labels
  • Change one setting at a time and keep a run log
  • Vector query k sets how many neighbours enter fusion; top sets how many results come back
  • The debug parameter (vector, semantic or all) returns subscores that show which side moved a result
  • Scores from different modes are not comparable; compare ranks and metrics

4. Prerequisites and access

  • Golden set of 50-200 queries with graded relevance labels
  • Python 3.10+ with azure-search-documents and azure-identity
  • Search Index Data Reader on the sandbox index
  • Agreed target metric and latency budget

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Pause indexers on the sandbox index and record its definition and statistics.
  2. Run the baseline configuration and save per-query ranks.
  3. Sweep vector weight (for example 0.5, 1.0, 1.5, 2.0) and k (for example 20 and 50).
  4. Toggle semantic ranker and record the change in nDCG@10 and p95 latency.
  5. Try an external re-ranker on the best configuration if the decision guide allows it.
  6. Use debug subscores to explain the five biggest wins and losses.
  7. Pick the configuration with the best metric inside the latency budget.
  8. Commit the configuration as code and re-run to confirm it reproduces.
  9. Add the run to CI as the nightly regression test used by PB-RAG-08-04.

Draft snippet: Sweep vector weight in hybrid queries and report recall@10

"""Sweep vector weight for hybrid queries and report recall@10."""
import json
import os

from azure.identity import DefaultAzureCredential
from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizableTextQuery

client = SearchClient(os.environ["SEARCH_ENDPOINT"], os.environ["INDEX_NAME"],
                      DefaultAzureCredential())
with open("eval/golden.jsonl", encoding="utf-8") as handle:
    golden = [json.loads(line) for line in handle]

for weight in (0.5, 1.0, 1.5, 2.0):
    total = 0.0
    for row in golden:
        vector_query = VectorizableTextQuery(text=row["query"], k_nearest_neighbors=50,
                                             fields="contentVector", weight=weight)
        results = client.search(search_text=row["query"], vector_queries=[vector_query],
                                query_type="semantic", semantic_configuration_name="default",
                                top=10, select=["id"])
        top_ids = {doc["id"] for doc in results}
        relevant = set(row["relevant_ids"])
        total += len(top_ids & relevant) / max(len(relevant), 1)
    print(f"weight={weight} recall@10={total / len(golden):.3f}")

Snippets to add:

  • snippets/eval/tune_hybrid.py: Weight sweep harness (python)
  • snippets/search/ranking-config.json: Committed ranking configuration (json)
  • snippets/eval/RUNLOG.md: Tuning run log template (markdown)

6. Verify it works

A second engineer re-runs the harness from a clean checkout against the sandbox index. Pass: results reproduce within 0.01 and the chosen configuration meets the target on the held-out queries.

7. Known issues and fixes

Symptom Likely cause Fix
Metrics change between identical runs Index still updating during tuning Pause indexers during tuning and re-run.
Tuned settings fail on another corpus Overfitting to a small golden set Hold out 30% of queries and report both sets.
A higher vector weight hurts exact-term queries Weighting favours semantic matches over lexical ones Segment the golden set by query type and route query types to different settings if the app allows it.

8. Monitoring and alerts

Signal Alert when Action
nDCG@10 from the nightly regression run Drop of more than 0.03 from the committed baseline Block the release and investigate the last change.

9. AWS and open-source equivalents

In OpenSearch, tune the normalisation and combination weights of the hybrid search pipeline; with pgvector, tune the RRF constant and per-list limits in SQL. Only the query call in the harness changes.

10. Validation record

Pending. Planned dry-run scenario: A second engineer re-runs the harness from a clean checkout against the sandbox index. Pass: results reproduce within 0.01 and the chosen configuration meets the target on the held-out queries.

11. Sources