Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 2-3 days including the renderer contract
Last Validated: Not yet validated (draft)
Kind: How-to
Risk Note: Medium (wrong citations damage trust more than no citations)

Return claim-level citations with Azure OpenAI structured outputs

Draft: not validated

Not yet dry-run. Do not use on a live system until the validation record is complete and signed off. Author: Deeparani · dry-run: Gayathri Ramachandran · sign-off: James. Work items: RAG-06-T01, RAG-06-T02, RAG-06-T03, RAG-06-T04.

1. Summary

A RAG answer can be defended only when each claim can be traced to the source text behind it. Claim-level citations link every claim to a source chunk and an exact quote from that chunk. Azure OpenAI structured outputs return the answer as JSON that matches a Pydantic schema. Code then checks that every cited chunk ID was retrieved and that every quote appears in that chunk. Only verified claims reach the user interface, through a fixed renderer contract.

Facts checked against Microsoft Learn on 26 October 2026. The snippet uses the v1 endpoint, which takes no api-version. With the versioned endpoint, use stable API version 2024-10-21. Structured outputs were first introduced in 2024-08-01-preview.

2. When to use and when not to

Use this how-to when:

  • You build or refactor RAG answer generation in regulated or high-accountability systems where every factual claim must link to a specific source chunk and verbatim quote.
  • The user interface renders claim-level footnotes, provenance popovers, or split-screen document previewers.
  • You require guaranteed JSON adherence without client-side JSON syntax repair or retry loops.

Do not use it when:

  • You deploy model versions that do not support structured outputs (for example gpt-4 prior to 2024-08-06 or gpt-35-turbo). Use prompt-based citation tags with regex extraction instead.
  • You stream unconstrained long-form text where schema compilation latency or fixed JSON token overhead impairs conversational feel.
  • You use Azure OpenAI On Your Data. Structured outputs are not supported with it.
  • You use a managed service that already returns source references for each part of the answer (for example Amazon Bedrock RetrieveAndGenerate). Still run the verification in Step 5 on its output.
  • You need batch evaluation of groundedness scores across golden query sets. Use PB-RAG-06-03 instead.
  • You triage live grounding incidents or hallucination escalations. Use PB-RAG-06-04 instead.

3. What you must know first

  1. Strict schema enforcement. With strict: true, structured outputs make the model follow the JSON schema you supply. Supported on gpt-4o (version 2024-08-06 or later), gpt-4o-mini (version 2024-07-18 or later), o1 (version 2024-12-17 or later), and o3-mini. Older model versions reject the schema parameter.
  2. JSON schema subset limitations. All object fields must be included in required. Emulate optional fields using nullable union types (anyOf: [{"type": "string"}, {"type": "null"}]). All objects must specify additionalProperties: false. Unsupported JSON schema keywords include minLength, maxLength, pattern, and format for strings; minimum, maximum, and multipleOf for numbers; and minItems, maxItems, and uniqueItems for arrays.
  3. Nesting and property quotas. A schema supports up to 100 object properties in total and up to 5 levels of nesting. Root objects cannot be an anyOf type.
  4. Schema compilation latency. The first inference request using a new or modified schema incurs server-side grammar pre-compilation latency. Subsequent requests with the identical schema reuse the cached grammar.
  5. Schema adherence vs factual truth. Structured outputs guarantee valid JSON structure, not factual accuracy. A model can produce schema-compliant JSON with fabricated chunk IDs or hallucinated quotes. Deterministic post-generation code verification is mandatory.
  6. No-answer handling. The schema and system prompt must explicitly support an unanswerable outcome (answerable: false). When sources lack sufficient information, forcing an answer causes the model to hallucinate citations to satisfy the schema.

4. Prerequisites and access

Item Detail
Roles Cognitive Services OpenAI User on the Azure OpenAI resource (Microsoft Entra ID least-privilege data plane).
Model deployment gpt-4o (version 2024-08-06 or later) or gpt-4o-mini (2024-07-18 or later) deployed in Azure OpenAI.
Tools & packages Python 3.10+, openai, pydantic>=2.0, azure-identity.
Upstream context Retrieval pipeline returning a list of chunk objects containing chunk_id, content, and source metadata (source_title, source_url, page_number).
Frontend contract Agreed JSON schema specification matching snippets/answer-contract.sample.json.

Set the environment variables:

export AOAI_ENDPOINT="https://${AZURE_OPENAI_SERVICE}.openai.azure.com"
export CHAT_DEPLOYMENT="${CHAT_MODEL_DEPLOYMENT}"

[!NOTE] Token scopes and API version routing:

  • Entra ID scope: For Azure OpenAI resources (https://${AZURE_OPENAI_SERVICE}.openai.azure.com), the Entra ID bearer token scope is https://cognitiveservices.azure.com/.default. Azure AI Foundry project endpoints (services.ai.azure.com) use https://ai.azure.com/.default.
  • API version: The snippet uses the v1 endpoint (OpenAI(base_url="https://${AZURE_OPENAI_SERVICE}.openai.azure.com/openai/v1/")), which takes no api-version. Set OPENAI_API_VERSION="2024-10-21" only if you use the AzureOpenAI(api_version=...) client and the /openai/deployments/... route instead.

5. Steps

Placeholders: ${AZURE_OPENAI_SERVICE} (name of your Azure OpenAI resource), ${CHAT_MODEL_DEPLOYMENT} (name of your model deployment).

Full implementation script: snippets/answer_with_citations.py. Frontend contract sample: snippets/answer-contract.sample.json.

Step 1. Define the Pydantic schema.

Define Pydantic models with extra="forbid" to ensure additionalProperties: false.

from pydantic import BaseModel, ConfigDict

class Citation(BaseModel):
    model_config = ConfigDict(extra="forbid")
    chunk_id: str
    quote: str

class Claim(BaseModel):
    model_config = ConfigDict(extra="forbid")
    text: str
    citations: list[Citation]

class GroundedAnswer(BaseModel):
    model_config = ConfigDict(extra="forbid")
    answerable: bool
    claims: list[Claim]

Verify: GroundedAnswer.model_json_schema() produces a JSON schema with additionalProperties: false on every object and all fields listed under required.

Step 2. Format retrieval chunks with explicit IDs.

Inject retrieved context into the user prompt using structured block headers:

def format_sources(chunks: list[dict]) -> str:
    return "\n\n".join(f"[chunk_id: {c['chunk_id']}]\n{c['content']}" for c in chunks)

Verify: The formatted string contains unambiguous [chunk_id: ...] boundaries for each retrieved chunk.

Step 3. Configure the attribution and no-answer prompt.

Instruct the model to cite only supplied chunk IDs, quote exact text without paraphrasing, and set answerable: false when evidence is missing.

SYSTEM_PROMPT = """You are an accurate enterprise assistant.
Answer the user's question using ONLY the provided sources.
For every factual claim you make, include one or more citations specifying:
- chunk_id: the exact chunk ID from which the claim is derived.
- quote: an exact, verbatim text snippet from the chunk supporting the claim.

Rules:
1. Do not use any external knowledge.
2. If the provided sources do not contain enough information to answer the question, set answerable to false and leave claims as an empty list.
3. Every claim must have at least one citation.
4. Quote exact text; do not paraphrase or summarize inside quotes."""

Verify: The prompt explicitly instructs the model on unanswerable handling and verbatim quotation.

Step 4. Call Azure OpenAI with structured outputs.

Invoke client.beta.chat.completions.parse passing response_format=GroundedAnswer:

completion = client.beta.chat.completions.parse(
    model=os.environ["CHAT_DEPLOYMENT"],
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"Sources:\n{format_sources(chunks)}\n\nQuestion: {question}"},
    ],
    response_format=GroundedAnswer,
)
raw_answer: GroundedAnswer = completion.choices[0].message.parsed

Verify: raw_answer is an instance of GroundedAnswer with populated answerable and claims.

Unverified citations undermine trust

Emitting citations directly from the model without code-level verification exposes end users to fabricated citations and quote hallucinations. Always execute Step 5 to drop or flag claims that do not strictly match source text.

Step 5. Validate cited chunk IDs and quote spans in code.

Check that each cited chunk_id exists in the retrieved chunks, and that each quote is in that chunk's content. The check ignores case, curly quotes and runs of white space. It uses the same rule as the incident verifier in PB-RAG-06-04, so both give the same result:

from snippets.answer_with_citations import verify_answer_citations

verified_answer, warnings = verify_answer_citations(raw_answer, chunks)

Verify: Any citation with an unknown chunk ID or a quote that is not in the chunk is dropped, and a warning is logged for monitoring. If every claim is dropped, answerable becomes false. Show refusal R2 from PB-RAG-06-02, never the unverified text.

Step 6. Map verified claims to the frontend renderer contract.

Join verified claims with metadata from the retrieval chunks (source_title, source_url, page_number) and return the JSON payload:

from snippets.answer_with_citations import build_renderer_contract

payload = build_renderer_contract(
    question=question,
    verified_answer=verified_answer,
    chunks=chunks,
    raw_answer=raw_answer,
    warnings=warnings,
)

Verify: The returned payload matches snippets/answer-contract.sample.json. Missing source_url or page_number comes back as null, never a default link. The renderer must show a citation without a link rather than invent one.

Log the retrieved chunks, the raw answer and this payload under one trace ID. PB-RAG-06-04 needs that trace to diagnose an incident.

6. Verify it works

Execute the verification test suite in the sandbox against a representative corpus:

  1. Answerable query test: Run 10 questions whose answers exist in single chunks.
  2. Verify: answerable is true, all claims have valid chunk_id references, and each quote matches the chunk content.
  3. Multi-chunk synthesis test: Run 5 questions requiring synthesis across two or more chunks.
  4. Verify: Claims contain multiple citations or cite distinct chunk IDs matching the respective source paragraphs.
  5. Unanswerable query test: Run 5 questions whose answers do not exist in the corpus.
  6. Verify: answerable is false, claims is an empty list, and zero citations are emitted.
  7. Hallucination rejection test: Inject a mock answer referencing a non-existent chunk ID ${MOCK_ID} and a fabricated quote.
  8. Verify: verify_answer_citations() strips the invalid claim, flags the warning, and sets answerable: false.

7. Known issues and fixes

Symptom Likely cause Fix
400 BadRequest: Invalid schema: 'minLength' is not permitted Pydantic model uses string length constraints (for example Field(min_length=1)) Remove min_length, max_length, and regex constraints from Pydantic models. Validate length in application code after parsing.
400 BadRequest: 'additionalProperties' must be false An object model or dictionary lacks strict schema configuration Add model_config = ConfigDict(extra="forbid") to every Pydantic model.
400 BadRequest: response_format of type 'json_schema' is not supported Model deployment is an older model version (for example gpt-4 pre-0806 or gpt-35-turbo) Upgrade deployment to gpt-4o (2024-08-06 or later) or gpt-4o-mini (2024-07-18 or later).
Model invents chunk IDs or merges IDs into a single string ("chunk_01, chunk_02") Model prompt ambiguous or chunk boundaries not delimited clearly Use strict [chunk_id: <id>] header syntax and instruct the model that each Citation object holds exactly one chunk_id.
Quote check fails although the text matches White space, line breaks, case or curly quotation marks differ between the chunk and the quote Use normalize_whitespace() from the snippet on both texts. It matches the rule in verify_citations.py.
Citation shows no link Chunk metadata (source_url, page_number) not carried through ingestion Fix metadata in the index (PB-RAG-05-02). Do not add a default link in the renderer.
Initial query after deployment exhibits cold-start latency spike Cold start caused by server-side JSON schema grammar compilation Issue a warm-up inference request with a minimal schema payload during application startup.

8. Monitoring and alerts

The thresholds below are illustrative team defaults, not platform recommendations. Tune them against a measured baseline.

Signal Source Alert when Action
CitationVerificationFailureRate Application metrics Above 2% of claims over 1 hour Inspect recent prompt changes (PB-RAG-07-01) and start runbook PB-RAG-06-04.
UnanswerableRate Application metrics See the no-answer rate alert in PB-RAG-06-02 Follow PB-RAG-06-02; check search service health and indexer status (PB-RAG-05-02).
SchemaParsingErrorRate Application logs Above 0.1% over 15 minutes Check for unversioned model redeployments or SDK client version mismatches.
InferenceDuration APM / Platform metric P95 above 5 seconds for 10 minutes Review schema nesting depth and token lengths; confirm schema caching is active.

9. AWS and open-source equivalents

Platform Equivalent Key differences
AWS Bedrock (RetrieveAndGenerate API) Native citation generation Fully managed pipeline returning retrievedReferences mapping text spans directly to chunks. Does not require a custom Pydantic parser.
AWS Bedrock (Converse API) Tool use with JSON schema Enforces structured output via function/tool calling schemas. Verification of citations against context is still required in code.
Open-source (vLLM / llama.cpp / Outlines) Grammar-constrained decoding Uses context-free grammars or Outlines / Instructor to enforce strict JSON schemas during token generation. Requires self-managed model serving.

10. Validation record

Pending. Planned dry-run scenario: A second engineer runs the verification test suite in Section 6 against a sandbox Azure OpenAI deployment with a representative corpus. Pass if answerable rates, multi-chunk synthesis, and hallucination rejection behave as expected.

11. Sources

Checked on 26 October 2026.