Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 1 day plus a calibration run
Last Validated: Not yet validated (draft)
Kind: Decision guide
Risk Note: Medium (too strict frustrates users; too loose produces ungrounded answers)

Define no-answer and low-confidence handling

Draft: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is dry-run by someone other than the author and signed off (planned 2026-11-30). Author: Deeparani · dry-run: Gayathri Ramachandran · sign-off: Prasanna. Work items: RAG-06-T05, RAG-06-T06, RAG-06-T07, RAG-06-T08.

1. Summary

A RAG assistant must know when not to answer. If it refuses too often, users stop using it; if it answers on weak evidence, it gives ungrounded answers. The rules below set when the assistant answers, answers with a caveat, asks a follow-up, refuses or escalates. The recommended default is a layered gate: a code-level retrieval check before generation, plus the model's answerable flag after it. Add a runtime groundedness check for high-risk content. Thresholds live in configuration, are calibrated per ranking mode and are agreed with compliance.

Outputs: an approved snippets/no-answer-policy.yaml and approved user wording from snippets/refusal-wording.md.

Facts checked on 27 Oct 2026. Content Safety groundedness detection is preview (API version 2024-09-15-preview).

2. When to use and when not to

Use this guide when:

  • You design a new RAG assistant and must decide how it behaves when it cannot answer.
  • The business asks the assistant to answer more questions, or fewer.
  • You change the ranking mode (for example, add semantic ranker). Old thresholds no longer apply.
  • Users report too many refusals, or answers that the sources do not support.

Do not use it when:

  • You need the answer to carry claim-level citations. Use PB-RAG-06-01.
  • You need to score groundedness offline and gate a release. Use PB-RAG-06-03.
  • You are handling a live ungrounded-answer incident. Use PB-RAG-06-04.
  • You are choosing the retrieval mode itself. Use PB-RAG-08-01.

3. What you must know first

  1. Scores differ by ranking mode. In Azure AI Search, BM25 @search.score has no upper limit. Cosine vector scores range from 0.333 to 1.00. RRF scores in hybrid search depend on how many queries are fused. A threshold for one mode is meaningless for another. (Hybrid search scoring)
  2. The semantic reranker score is the most usable signal. @search.rerankerScore ranges from 0.00 to 4.00, and each band has a documented meaning. Its distribution can shift slightly with infrastructure and model updates, so do not set thresholds too finely. It reranks only the top 50 results. (Semantic ranking)
  3. Vector search always returns results. A vector query returns k results even for off-topic questions; they just score low. "Hits greater than zero" is not a no-answer signal for vector or hybrid queries. Hybrid queries are not suited to minimum thresholds because RRF ranges are small and volatile. The vector threshold parameter is preview (2026-08-01-preview). (Create a vector query)
  4. Offline groundedness scores are 1 to 5. GroundednessEvaluator returns an integer Likert score from 1 to 5, with a pass threshold (default 3) that you can set. GroundednessProEvaluator (preview) returns a true or false label. Use these to calibrate, not at runtime. (RAG evaluators)
  5. Runtime groundedness detection has limits. Content Safety groundedness detection (preview) supports English content only; other languages are accepted but accuracy is optimised for English. It is available in specific regions and has text-length and rate limits; check current limits in the docs. ungroundedPercentage is a proportion of text, not a confidence level. (Groundedness detection, Quickstart)
  6. No-answer behaviour needs its own tests. A golden set of only answerable questions cannot show false answers. Include unanswerable and out-of-scope questions in every calibration and regression run.

4. Options

Each option is a gate that can trigger a no-answer. Most systems combine more than one.

Option A: Prompt instruction only

Tell the model to say it does not know when the sources do not answer. Cheapest and quickest. Not reliable on its own: the model can ignore the instruction, and nothing is measurable or auditable. Use only for prototypes. Prompt wording is carried out by PB-RAG-07-03.

Option B: Retrieval gate in code (before generation)

Application code checks the retrieval signals and refuses before calling the model. Signals: hit count after security trimming, and the top @search.rerankerScore (or a vector threshold, preview). Deterministic, cheap and auditable. Saves a model call on every refusal. Needs calibration per ranking mode. It cannot judge whether the sources actually answer the question.

Option C: Model answerable flag (structured output)

The model returns an answerable field in its structured output, next to the answer and citations. To use follow-up F1, add an ambiguous: bool field to the GroundedAnswer schema in PB-RAG-06-01; it is not in that schema by default. Catches "relevant sources, but no answer in them". Adds no extra call. The flag is the model's own judgement; check it against the golden set. Carried out by PB-RAG-06-01.

Option D: Runtime groundedness check (after generation)

Send the answer, the query and the sources to Content Safety groundedness detection (preview) before showing the answer. Block or caveat when ungroundedDetected is true. Use non-reasoning mode for low latency. Adds latency and a paid call on every answer; check the current price before approval. English only, regional, preview.

A: Prompt only B: Retrieval gate C: Answerable flag D: Runtime groundedness
Catches Some unanswerable questions No or weak sources Sources that do not answer Answers not supported by sources
Extra latency None None None One extra call
Extra cost None Saves calls None Per answer; check the current price
Auditable No Yes Partly Yes
Limits Model can ignore it Per ranking mode Model judgement Preview, English, regional

5. Decision table

Situation Choose Why
No results after security trimming Option B, refuse with R1 Nothing to ground on. Never hint that restricted documents exist.
Semantic ranker on, top @search.rerankerScore below the refuse threshold Option B, refuse with R1 The band meanings make a stable, explainable threshold.
Semantic ranker on, top score between refuse and caveat thresholds Option B, answer with caveat C1 Partial relevance. The user can still check the cited source.
Vector-only query Option B with vector threshold (preview), or add semantic ranker Vector search always returns k results, so hit count alone means nothing.
Hybrid query without semantic ranker Options C and D; do not threshold @search.score RRF scores are too volatile for thresholds.
Sources look relevant but do not contain the answer Option C, refuse with R2 Only the model can judge this at runtime.
Question could mean several things Option C ambiguous flag, ask follow-up F1 A follow-up is better than a guess or a refusal.
Regulated topic (legal, medical, HR) Escalate with E1 Compliance decides the list. A wrong answer costs more than a delay.
High-risk English content, latency budget allows one more call Add Option D, refuse with R3 Catches fabrication that B and C miss.
Non-English content Options B and C; do not rely on Option D Groundedness detection accuracy is optimised for English.
Business wants more answers Lower the caveat threshold first, then re-run calibration Moves refusals to caveated answers without hiding risk.
Prototype or internal demo Option A Speed matters more than audit. Do not ship it to production.

Default: Options B and C together, with semantic ranker on. Refuse when there are no hits or the top reranker score is below the refuse threshold. Refuse when the model says answerable: false. Caveat between the two thresholds.

Deviate when:

  • The content is high risk and in English: add Option D.
  • The content is not in English: keep B and C, and score groundedness offline only (PB-RAG-06-03).
  • Semantic ranker is not available in your region or tier: rely on Option C, and use the vector threshold (preview) for vector-only queries.

Rule of thumb: decide in code what you can, let the model decide only what code cannot see.

The gate order in code, for reference. restricted is true when the question matches restricted_topics in the policy file (your topic classifier decides this). score is the top score for the active ranking mode, or None for hybrid_no_semantic. out is the verified answer from PB-RAG-06-01, step 5.

def decide(restricted, hits, score, out, ungrounded, t):
    """Return the behaviour key from no-answer-policy.yaml. First match wins."""
    checks = [
        (restricted, "restricted_topic"),
        (hits < t["min_hits_after_filter"], "no_hits"),
        (score is not None and score < t["refuse_below"], "low_score_refuse"),
        (not out["answerable"], "not_answerable"),
        (out.get("ambiguous", False), "ambiguous"),
        (ungrounded, "ungrounded"),
        (score is not None and score < t["caveat_below"], "low_score_caveat"),
    ]
    return next((reason for hit, reason in checks if hit), "answer")

7. Steps to set the policy

  1. List question categories and the behaviour for each (answer, caveat, follow-up, refuse or escalate) with the product owner and compliance lead. Verify: every category in the golden set has one agreed behaviour, and the restricted-topic list is written down.
  2. Choose the gates from the decision table for your ranking mode and language. Verify: the chosen ranking_mode and gates are recorded in snippets/no-answer-policy.yaml.

  3. Extend the golden set with unanswerable questions from the shared set in data/golden/unanswerable/questions.csv. Map each category to a behaviour:

    Category Expected behaviour
    out_of_scope, near_miss, personal_data, time_sensitive, access_restricted Refuse (R1 or R2)
    ambiguous Ask a follow-up (F1)
    restricted_topic Escalate (E1)
    false_premise Answer, correcting the premise from the sources

Verify: at least 25% of the set is unanswerable (team default), and each question has an expected behaviour. 4. Calibrate the thresholds. Run the golden set, log the top score per question, and pick refuse_below and caveat_below where answer and refusal rates meet the targets in the policy file. Verify: a table of threshold against answer rate and refusal rate is saved with the run, and the chosen values meet the targets. 5. Write the user-facing wording with the product owner, starting from snippets/refusal-wording.md. Verify: the product owner has signed off every message, and compliance has signed off R3 and E1. 6. Agree the thresholds and targets with compliance and record them in the change record. Verify: approved_on and change_record in the policy file match the signed record.

Changes what users see in production

A wrong threshold can refuse most questions or let ungrounded answers through, for every user at once. Deploy behind a feature flag or to a small share of traffic first. To roll back, redeploy the previous policy file version from source control; no code change is needed.

  1. Load the thresholds from configuration, not code. Deploy the policy file with the application and log the rule that fired for each answer. Verify: changing a threshold in the file changes behaviour in the sandbox without a code release, and each log entry has a decision reason.
  2. Review monthly. Sample false refusals and ungrounded answers, then adjust thresholds through steps 4 to 7. Verify: the review is recorded with the sample, the findings and any threshold change.

8. Known issues and fixes

Symptom Likely cause Fix
Assistant refuses questions it should answer Threshold calibrated on the wrong score type, or before a ranking-mode change Recalibrate per ranking mode (step 4). Keep one threshold set per mode.
Assistant answers out-of-scope questions Model ignores the answerable instruction; no code-level gate Add Option B before generation.
Off-topic questions still get "sources" Vector-only query: k results are always returned Add semantic ranker or a vector threshold (preview); do not gate on hit count alone.
Refusal rate jumps with no code change @search.rerankerScore distribution shifted after a model update Re-run calibration. Avoid finely tuned thresholds.
Refusal reveals that a restricted document exists Wording mentions access or document names Use R1 wording: "I could not find this in the documents you can access."
Many false "ungrounded" blocks on non-English answers Groundedness detection is optimised for English Turn off Option D for that language; use B and C.
Thresholds differ between environments Thresholds hard-coded or copied by hand Load from one versioned policy file per environment.
Golden set passes but users see wrong answers Golden set has no unanswerable questions Add them (step 3) and re-run.

9. Monitoring and alerts

Signals come from application telemetry: log the decision reason and top score for every answer. The thresholds below are team defaults, not platform recommendations. Measure a baseline over the first two weeks, then tune them.

Signal Source Alert when (team default) Action
No-answer rate (refuse, follow-up and escalate share) App telemetry Moves more than 5 percentage points week on week Sample 20 refusals. Check retrieval health and recent index or ranking changes.
Caveat rate App telemetry More than 5 percentage points above the 4-week average for 7 days Check for corpus gaps. Re-run calibration if the score distribution moved.
ungroundedDetected rate (Option D) Content Safety response Above 2 times the 14-day baseline for 24 hours Review flagged answers. Check prompt or model changes (PB-RAG-07-01).
Negative feedback on refusals User feedback Above 1.5 times the 14-day baseline for 7 days Add those questions to the golden set. Review thresholds in the monthly review.
Decision reason missing in logs App telemetry Any answer without a reason Fix logging before the next release; the audit trail is incomplete.

10. Cost and capacity notes

  • Option B saves model calls. Each refusal before generation skips one completion.
  • Option D adds a paid call per answer, and more with reasoning mode. Check the current price before approval.
  • Semantic ranker is billed by usage beyond the free plan. Check the current price before approval.

11. AWS and open-source equivalents

Platform Equivalent Key differences
Amazon Bedrock Guardrails Contextual grounding check, with GROUNDING and RELEVANCE filters Thresholds from 0 to 0.99; responses below them are blocked (BLOCK) or only reported (NONE). Classic tier supports English, French and Spanish; Standard tier needs cross-Region inference. Check current length limits in the docs. With streaming, an irrelevant response can reach the user before it is flagged.
Amazon Bedrock Knowledge Bases score on each result of the Retrieve API, with optional reranking The docs call score a relevancy score for analysing ranking and do not state a range. Calibrate any threshold on your own golden set, as in step 4.
Open source Ragas Faithfulness or DeepEval FaithfulnessMetric offline; custom gate at runtime Both score 0 to 1 as supported claims divided by all claims. DeepEval's default threshold is 0.5. No managed runtime check: use them for calibration, and Options B and C in code.

12. Validation record

Pending. Planned dry-run scenario: A second engineer applies the policy to a sandbox with 30 answerable and 10 unanswerable questions. Pass if refusal and answer rates match the agreed targets within tolerance.

Sandbox scenario: index the sandbox corpus in data/golden/. Use the 30 answerable questions in data/golden/A-policies/A_04/questions.csv and these 10 from data/golden/unanswerable/questions.csv: U01, U03, U06, U09, U10, U21 and U25 (refuse), U28 and U29 (follow-up), and U32 (escalate).

Pass criteria (targets from the policy file, team defaults):

  • At least 27 of 30 answerable questions are answered or answered with a caveat (90%).
  • At least 9 of 10 unanswerable questions get their expected behaviour: refuse, follow-up or escalate (90%).
  • Each count is within one question of the author's calibration run.
  • Every answer has a decision reason in the log, and every refusal uses the approved wording.

13. Sources

Checked on 27 Oct 2026.

  • PB-RAG-06-01: returns the answerable flag and citations (Option C).
  • PB-RAG-06-03: offline groundedness scoring used for calibration.
  • PB-RAG-07-01: re-check no-answer behaviour on every prompt change.
  • PB-RAG-07-03: refusal prompt template (Option A).
  • PB-RAG-08-01: choose the ranking mode the thresholds depend on.