Enterprise RAG Platform on Azure OpenAI with Private Endpoints

In a nutshell

Picture a private librarian who works only inside your building. Ask a question and she doesn’t answer from memory — she walks to the shelves, pulls the exact pages that bear on it, checks that you are allowed to read those pages, and hands back a short answer with the page reference stapled to it. Now wire that librarian to a language model that turns the pulled pages into a clean, spoken reply — but keep the librarian, the shelves, the model, and every conversation inside your own network, behind a locked door. Nothing about a client, a document, or a question ever leaves the building. That is an enterprise RAG platform.

RAG is retrieval-augmented generation: retrieval is the librarian fetching the right pages, generation is the model writing the answer from them. What makes it enterprise-grade is where it all runs and who may see what. On Azure the model (Azure OpenAI), the shelves (Azure AI Search), and the document store (Blob Storage) all sit behind Private Endpoints with their public doors bolted shut (publicNetworkAccess = Disabled); they authenticate to each other with managed identities instead of passwords; and retrieval is filtered so a person only ever grounds an answer on documents they are cleared to read.

Why a beginner should care: nearly every “chat with our own documents” project a real company actually ships ends up shaped like this. The weekend-with-an-API-key demo is easy; the version a bank, hospital, or ministry will sign is this private, identity-gated, auditable one. Learning this architecture is learning what “production AI” means inside a regulated enterprise — and it reuses the networking, identity, and governance you already know from the rest of this course.

Level: Advanced · Time: ~44 min

Assumed background (skim these first if any are fuzzy): Azure OpenAI & GenAI fundamentals, Private Endpoints & private DNS at scale, AI Search vector + hybrid + semantic ranking, and Key Vault + Workload Identity. You should be comfortable with VNets/subnets, what a managed identity and an RBAC role assignment are, and the idea of an embedding vector.

After this lesson you will be able to:

• Trace a private RAG query box by box, from the advisor’s click to a cited answer, and say why none of it touches the public internet.
• Name the five privatelink.* DNS zones this design needs and predict the exact failure when one is missing.
• Explain why security trimming belongs in the search index, not in application code.
• Choose the cost levers (model tiering, semantic caching, PTU-to-p95 + spillover) that keep token spend sane at thousands of seats.
• Judge when full private-networked RAG is warranted versus a simpler long-context or AI Foundry “chat on your data” build.

A retail bank’s wealth-management arm gets an ultimatum from its head of advisory: the 4,000 relationship managers spend a quarter of every client meeting hunting through product disclosures, suitability rules, and a decade of internal research notes — and twice last year an RM quoted a withdrawn fund fact sheet to a client, which is the kind of thing that ends with a regulator’s letter. The ask is “an assistant that answers from our own documents, with a citation, instantly.” The constraint is brutal and non-negotiable: this is MiFID II / SEBI-regulated material under data-residency rules, the security team has a standing veto on anything where a client’s portfolio context could leave the bank’s network, and the model must never invent a number. A weekend with an API key is not the answer. This article is the reference architecture for building that assistant properly on Azure — a private-networked, identity-gated, guardrailed RAG platform that a bank’s CISO and compliance officer will actually sign.

The pressures stack the way they always do in finance. Regulation means every answer needs a verifiable source and an audit trail, and the data cannot traverse the public internet. Scale means 4,000 concurrent advisors during market hours, not a demo. Latency means a relationship manager mid-conversation will not wait eight seconds for a streamed token. And cost means token spend that grows with adoption and has to be charged back to each business line. RAG — retrieval-augmented generation — is the pattern that satisfies all four at once: it grounds the model’s answer in retrieved, permissioned, citable passages from a search index you control, instead of the model’s frozen and uncitable training memory. The knowledge stays in your index; the model supplies language and reasoning, not facts.

Why not the obvious shortcuts

The naive fixes each fail predictably, and naming why matters because someone on the project will propose all three.

Keyword search over the document store returns documents, not answers, and misses anything phrased differently from the query — “early redemption penalty” will not find a fact sheet that says “exit load.” Fine-tuning a model on the corpus bakes facts into weights you cannot cite, update the morning a fund is withdrawn, or unlearn — and the model still hallucinates confidently. Pasting documents into a public chatbot leaks regulated, client-linked data across a tenant boundary the security team will never approve, and in a residency regime is simply illegal.

RAG threads the needle. At query time the system retrieves the handful of passages actually relevant to the question, hands them to the model as grounding context, and the model composes an answer from that context with citations back to the source. Retrieval is also the natural choke point to enforce document-level permissions — an advisor only ever grounds on documents they are cleared to read — and the citations turn an unauditable black box into something a compliance officer can verify line by line.

Architecture overview

The platform runs two distinct paths that share infrastructure but live on different schedules: a synchronous query pipeline that serves advisors, and an event-driven ingestion pipeline that keeps the knowledge base current. Keeping them separate in your head is the first step to operating this well.

The defining property of the entire topology is the one the security team cares about most: every Azure PaaS data-plane call rides a Private Endpoint, and every public endpoint is disabled. Azure OpenAI, AI Search, Cosmos DB, Blob Storage, and Content Safety expose private IPs inside the VNet only. No prompt, no retrieved passage, and no completion ever touches the public internet — which is what makes a residency story defensible.

Query path, following the control flow:

1. An advisor opens the assistant inside the bank’s portal. The portal federates identity through Okta as the workforce IdP, which is brokered to Microsoft Entra ID so Azure resources see a first-class Entra token. The user hits Akamai at the edge for TLS termination, global anycast, and WAF/bot protection before traffic ever reaches Azure.
2. The request lands on Azure API Management (APIM) acting as the AI gateway in internal VNet mode. APIM validates the Entra JWT (validate-jwt), attaches the caller’s identity and group claims, enforces a per-business-line token budget with the llm-token-limit policy, and rate-limits by tenant. This is the single front door for every model call in the bank — one place to meter, throttle, and audit.
3. The request reaches the orchestrator — your RAG application code on AKS (a private cluster, for the steady high throughput of market hours). The orchestrator pulls the few secrets it cannot get from managed identity — third-party API tokens, the Okta introspection secret — from HashiCorp Vault via the Vault Agent sidecar with Entra-backed auth, so nothing sensitive sits in a Kubernetes Secret or config map.
4. The orchestrator embeds the question with text-embedding-3-large and queries Azure AI Search with a hybrid query (vector + BM25 keyword) plus semantic reranking, filtered by the advisor’s permission groups so a passage they may not see is never even returned.
5. It assembles the top passages into a grounded prompt and calls Azure OpenAI (gpt-4o for substantive reasoning, gpt-4o-mini for simple lookups).
6. Both the inbound prompt and the model’s output pass through Azure AI Content Safety — Prompt Shields for jailbreak and indirect-injection detection, harm-category filters, and a groundedness check on the answer to flag anything the retrieved context does not support. The cited answer streams back to the advisor; the turn is written to Cosmos DB.

Ingestion path, independent and event-driven: source documents (product disclosures, research, suitability rules) land in Blob Storage, synced from SharePoint and the bank’s content systems. A Blob event triggers Azure AI Document Intelligence to extract clean text and layout — tables, headings, OCR for scanned legacy fact sheets — then Azure Functions chunk the text, call Azure OpenAI to embed each chunk, and push the vectors stamped with the source document’s ACL groups into the AI Search index. That ACL stamp at ingestion time is what makes per-document security real at query time.

Component breakdown

A few of these choices deserve the why, because they are the ones teams get wrong.

Why hybrid search, not pure vector. Vector similarity is excellent at “passages about early-redemption penalties” even when the document says “exit load.” But it is mediocre at exact-match needs — ISIN codes, fund identifiers, regulation clause numbers — where a single token must match. Hybrid retrieval runs both a vector query and a BM25 keyword query, fuses the results, and then semantic reranking (a cross-encoder Microsoft hosts inside AI Search) re-scores the top ~50 candidates for true relevance. In practice, hybrid plus semantic ranking lifts answer quality more than any prompt tweak you will make.

Why security trimming belongs in the index, not the app. It is tempting to retrieve broadly and filter in application code. Do not — that means restricted content leaves the index into your process memory and your traces before being dropped, and one bug leaks it across a regulatory line. Instead, stamp every chunk with the Entra group GUIDs allowed to read its source, and pass the caller’s groups as an OData filter so AI Search never returns a forbidden passage:

$filter=groups/any(g: search.in(g, '8f3a..., 21b7..., d40c...'))

Permission stays a property of the data, not a hope in the application layer.

Why an AI gateway in front of Azure OpenAI. Every enterprise eventually needs one place that enforces token budgets per business line, load-balances across OpenAI deployments, honors Retry-After, and produces a single audit log of who asked the model what. APIM is that layer here. Its llm-token-limit policy meters prompt and completion tokens against a per-line quota so the equities desk cannot exhaust the budget the advisory desk paid for, and its backend pool routes overflow from the provisioned deployment to a pay-as-you-go one.

Implementation guidance

Provision with Terraform, and treat the network as the first deliverable. The deployment order matters because of private DNS — get it wrong and endpoints resolve to firewalled public IPs and calls hang silently.

1. A hub/spoke or single VNet with subnets for the orchestrator, the private endpoints, and APIM (its own delegated subnet).
2. Private DNS zones for each PaaS service — privatelink.openai.azure.com, privatelink.search.windows.net, privatelink.documents.azure.com (Cosmos), privatelink.blob.core.windows.net, privatelink.cognitiveservices.azure.com (Content Safety + Document Intelligence) — linked to the VNet. Forgetting one zone is the single most common failure on this architecture.
3. The PaaS resources, each with publicNetworkAccess = 'Disabled' and a Private Endpoint in the PE subnet.
4. The private AKS cluster (Azure CNI) with Workload Identity enabled.
5. APIM in internal VNet mode, with Akamai pointed at its private origin.

A minimal Terraform shape for the Azure OpenAI account communicates the intent — keys off, Entra only:

resource "azurerm_cognitive_account" "openai" {
  name                          = "oai-ragbank-prod-cin"
  kind                          = "OpenAI"
  sku_name                      = "S0"
  custom_subdomain_name         = "oai-ragbank-prod-cin"  # required for Private Link + AAD
  public_network_access_enabled = false
  local_auth_enabled            = false                   # no API keys; Entra only
  identity { type = "SystemAssigned" }
}

resource "azurerm_cognitive_deployment" "chat" {
  name                 = "gpt-4o"
  cognitive_account_id = azurerm_cognitive_account.openai.id
  model { format = "OpenAI"  name = "gpt-4o"  version = "2024-11-20" }
  sku  { name = "ProvisionedManaged"  capacity = 120 }    # PTUs sized to p95
}

The pipeline that applies this runs in GitHub Actions, authenticating to Azure via OIDC federation so there is no stored service-principal secret to leak — a hard lesson the platform team intends never to repeat. The same pipeline runs the offline evaluation harness (below) as a required gate.

Identity: kill the keys, federate the humans. Set local_auth_enabled = false on Azure OpenAI and disableLocalAuth on AI Search and Cosmos, so the only way in is Entra. The orchestrator authenticates with Workload Identity on AKS, granted exactly three role assignments — Cognitive Services OpenAI User on the OpenAI account, Search Index Data Reader on the index, and Cosmos DB Built-in Data Reader/Contributor — with the ingestion identity additionally holding Search Index Data Contributor and Storage Blob Data Reader. Human SSO flows Okta → Entra: advisors log in once with the bank’s Okta credentials and conditional-access policies, Okta federates to Entra over OIDC, and the resulting Entra token carries the group claims that both APIM and AI Search consume. The few residual secrets that are not managed identities — the Okta introspection secret, third-party research-feed tokens — live in HashiCorp Vault, leased dynamically and injected by the Vault Agent sidecar, so they are short-lived and never written to a Kubernetes Secret.

Indexing wiring. Prefer the push pattern through the ingestion function for this corpus: full control over chunking, ACL stamping, and metadata, which is essential when permissions are non-trivial. Chunk at ~300–500 tokens with ~10–15% overlap; carry title, sourceUri, page, effectiveDate, and groups[] on every chunk; define an HNSW vector field and a semantic configuration naming your title and content fields. Carry effectiveDate specifically so a withdrawn fact sheet can be filtered out at query time — the exact failure that started this project.

Enterprise considerations

Security & Zero Trust. The architecture is Zero Trust by construction: identity-based access only, least-privilege RBAC scoped per resource, no public data-plane surface. Layer on top: (a) Prompt Shields to catch jailbreaks and the under-appreciated indirect injection where a malicious instruction is hidden inside a retrieved document; (b) groundedness detection as the last line against hallucinating a number; © Wiz running continuous CSPM and sensitive-data-exposure scanning across Storage and Search, alerting the moment any resource drifts to public exposure or a misconfigured ACL widens access — the posture backstop behind the policy controls; (d) CrowdStrike Falcon sensors on the AKS node pool and ingestion compute for runtime threat detection, feeding the bank’s SOC; (e) a guardrail breach (a blocked jailbreak, a sustained groundedness failure) auto-raises a ServiceNow incident so security has a ticket, not just a log line. Azure Policy denies any Cognitive Services or Search resource created with public network access, and Wiz independently verifies that the policy is actually holding.

Cost optimization. Token spend dominates and grows with success, so engineer for it from day one.

Meter tokens per business line in APIM and pipe the metric to Dynatrace, which the platform team uses for the chargeback dashboard the CFO sees.

Scalability. Each tier scales independently. AKS scales pods on concurrency and nodes via the cluster autoscaler; ingestion Functions scale on Blob-event depth via KEDA. AI Search scales out with replicas (query QPS and the read SLA) and partitions (index size and indexing throughput) — size them separately. Azure OpenAI scales by adding PTUs or fanning out across deployments and regions behind APIM’s backend pool, which round-robins and honors Retry-After. The natural ceiling is the OpenAI regional quota, which is why a 4,000-seat rollout plans multi-region early.

Failure modes, and what each one looks like. Name them before they page you.

• A missing private DNS zone link — the endpoint deploys clean but resolves to a firewalled public IP and every call hangs until timeout. Mitigation: assert all zone links in Terraform and in a post-deploy smoke test.
• Azure OpenAI 429s under load — pay-as-you-go shares a regional pool; at market open you get throttling and latency jitter, fatal for a streaming UX. Mitigation: a PTU deployment for interactive traffic with pay-as-you-go as APIM-routed spillover.
• Retrieval miss — the relevant passage is not in the top-k, so the model cannot ground on it and a confidently wrong answer slips out. Mitigation: hybrid + semantic rerank, groundedness detection set to flag-and-cite, and an eval harness that catches regressions.
• Stale corpus — a withdrawn fact sheet still in the index gets cited. Mitigation: the effectiveDate filter and an ingestion job that tombstones superseded documents within the hour.
• Regional outage — see DR below.

Reliability & DR (RTO/RPO). Decide the numbers per tier. Cosmos DB with multi-region writes gives near-zero RPO and seconds RTO for chat state. AI Search has no native cross-region replication, so DR means maintaining a warm index in a paired region — re-run ingestion against both — with Blob (geo-redundant, the durable source of truth) as the real recovery guarantee. Azure OpenAI deployments are regional; for DR, deploy the same models in a paired region and fail over at the APIM layer. A pragmatic target for this platform: RTO 15 minutes, RPO 5 minutes for the conversational service, with the knowledge base rebuildable from geo-redundant Blob within hours if the index itself is lost. Akamai health checks drive edge failover for ingress.

Observability. Instrument the RAG span end to end in Dynatrace with OpenTelemetry: one trace covering embed → retrieve → rerank → generate → guard, with timing and token counts on each hop, and Davis anomaly detection on top so a latency or cost regression surfaces on its own. Emit the metrics the business actually cares about — retrieval hit-rate, groundedness score, cache-deflection rate, tokens and cost per business line, and p95 time-to-first-token (the latency advisors feel in a stream). Run an offline evaluation harness (Azure AI Foundry prompt-flow evaluators or a golden-question set) inside the GitHub Actions pipeline so a prompt or model change is scored on groundedness and relevance before it ships. New corpora pass through a ServiceNow change approval before going live, giving compliance a documented gate.

Governance. Pin model versions explicitly (gpt-4o 2024-11-20, never a floating alias) so behavior does not drift; promote new versions through the eval gate. Keep prompt templates in version control, reviewable and instantly revertable. Apply Azure Policy to deny public network access and require diagnostic settings on every relevant resource, with Wiz as the independent check that the controls are real. Log every prompt/response pair for audit, incident review, and future eval data — with a right-to-be-forgotten path, since advisor conversations and client context are personal data under the same regime that started this whole project.

Explicit tradeoffs

Accept these or do not build it. RAG adds real moving parts — an ingestion pipeline, an index to keep fresh, embedding costs, and retrieval quality you must measure and tune. Latency is the sum of retrieval and generation, never just one. And RAG answers are only as good as retrieval: if the relevant passage is not in the top-k, the model cannot ground on it, and groundedness checks mitigate but do not eliminate the confidently-wrong failure. The private-networking posture that makes the security team sign costs you setup complexity — five private DNS zones, an internal-mode APIM, no public debugging shortcuts — and the price of forgetting one piece is a silent hang, not a clear error. The Okta-to-Entra federation adds a hop and a token-translation step that the simpler single-IdP shops will not need. And the AI gateway, the guardrails, and the per-line metering are all overhead you can skip for a ten-user pilot and absolutely cannot skip for 4,000 regulated seats.

The alternatives, and when they win. If your corpus is small, static, and fits the model’s context window, long-context prompting skips the index entirely and is simpler. If you need the model to adopt a style or domain vocabulary rather than recall facts, fine-tuning is the right tool — and it composes with RAG (fine-tune for behavior, retrieve for facts). If you need the model to take actions — place a trade, update a CRM — you want an agent/tool-calling architecture, with RAG as one of its tools. And if you are a small team optimizing for speed over control, Azure AI Foundry’s “chat on your data” wizard stands up a working RAG app in an afternoon; graduate to this full private-networked platform when security, scale, residency, or governance demand it.

The shape of the win

For the bank’s advisory desk, the payoff is not “a chatbot.” It is that an advisor mid-meeting asks “what is the exit load on this fund after two years,” gets a one-line answer with a citation to the current fact sheet’s exact page in about a second, and — because the answer is grounded, cited, permission-trimmed, and never left the bank’s network — compliance approved the tool for advisory use, which a fine-tuned, uncitable model would never have cleared. That last sentence is the one that funds the platform. Everything upstream — the Private Endpoints, the Okta-to-Entra federation, the Vault-held secrets, the Wiz posture scanning, the Content Safety groundedness gate, the Dynatrace RAG span — exists to make a regulator, a CISO, and a CFO each say yes. The architecture here is the destination; start narrower if you must, but this is where a regulated, at-scale “chat with our documents” has to land.

Going deeper

The body above is the blueprint; this section is the mechanism — the parts that decide whether the thing works in production and the details reviewers probe.

How a Private Endpoint actually resolves — and why one missing DNS zone hangs everything

A Private Endpoint is just a NIC with a private IP dropped into your subnet and mapped to one instance of a PaaS service. But your SDK never calls that IP — it calls the public FQDN, e.g. https://oai-ragbank-prod-cin.openai.azure.com. Making that name resolve to the private IP is a DNS trick, and it is where most of these builds break.

When you enable Private Link, Azure turns the service’s public FQDN into a CNAME to a privatelink.* name — oai-ragbank-prod-cin.privatelink.openai.azure.com. If you link a private DNS zone named privatelink.openai.azure.com to the VNet and register the endpoint’s A record in it (the Private Endpoint’s DNS zone group does this automatically), that CNAME resolves to the private IP and traffic stays inside the VNet. Miss the link, or resolve from a network that can’t see the zone, and the CNAME falls through to Azure’s public IP — which publicNetworkAccess = Disabled refuses. The TCP connect gets no clean rejection, just silence, so your call hangs until the client timeout rather than failing fast. That is the signature symptom, and it is why “forgot one zone” is the number-one incident on this architecture.

The five zones this platform needs, and who each serves:

Two design notes. Azure OpenAI requires custom_subdomain_name for Private Link and Entra auth — the subdomain is what scopes the token audience and the private CNAME, so the default shared endpoint won’t work. And at scale you don’t hand-link a zone per spoke: you centralise the privatelink.* zones in a hub, point spokes at a DNS Private Resolver (or custom DNS) that forwards to the hub, and let a single Azure Policy DeployIfNotExists attach every new endpoint’s records — the pattern the private-DNS-at-scale lesson covers in full.

Managed identity between services — the token dance, no keys anywhere

Set local_auth_enabled = false (OpenAI) and disableLocalAuth / local_authentication_enabled = false (Search, Cosmos) and the API keys literally stop working — the only accepted credential is an Entra token. Here is the dance for the AKS orchestrator: Microsoft Entra Workload ID projects a short-lived OIDC service-account token into the pod; the Azure SDK exchanges that with Entra for an access token whose audience is the target resource (the OpenAI account, the Search service); the service validates the token and checks that the identity holds the right data-plane RBAC role before serving the data-plane call.

That data-plane distinction trips people: Contributor lets you manage the OpenAI resource but does not grant Cognitive Services OpenAI User, the role that lets you call it. The query identity needs exactly Cognitive Services OpenAI User + Search Index Data Reader + a Cosmos data role; the ingestion identity adds Search Index Data Contributor + Storage Blob Data Reader. Keys are worse on every axis — long-lived, copyable, attributable to nobody, and a rotation headache — which is why they’re off from day one. The handful of secrets that genuinely aren’t Azure identities (the Okta introspection secret, third-party research-feed tokens) live in Vault, leased dynamically and injected by a sidecar, never as a Kubernetes Secret.

The ingestion pipeline internals — push vs integrated vectorization

There are two ways to get vectors into AI Search, and the choice is a real fork.

• Push / custom (this platform’s choice). Your Function chunks the text, calls Azure OpenAI to embed, stamps each chunk with its source ACL groups, and pushes documents to the index. Maximum control over chunking, metadata, and — critically — the ACL stamp, which is why non-trivial permissions push you here.
• Integrated vectorization. AI Search’s own indexer + skillset pulls from Blob, runs a Split skill and an AzureOpenAIEmbedding skill, and indexes with essentially no code. Far less to maintain; you still must map ACL metadata through the indexer, and you inherit the skillset’s chunking rather than your own.

Chunk at ~300–500 tokens with ~10–15% overlap — small enough that a retrieved passage is mostly signal, overlapped so a fact split across a boundary still appears whole in one chunk. Every chunk carries title, sourceUri, page, effectiveDate, and groups[]. The vector field uses an HNSW profile (graph-based ANN, tuned by m, efConstruction, efSearch, metric cosine); groups is a filterable Collection(Edm.String); effectiveDate is filterable so a withdrawn fact sheet drops out at query time. Superseded documents are tombstoned (deleted or flagged) within the hour by the ingestion job — the fix for the withdrawn-fact-sheet incident that started the project.

Hybrid + semantic retrieval — what actually runs per query

A hybrid query fires two searches at once: HNSW vector ANN over the embedding, and BM25 keyword over the text. Their result lists are fused with Reciprocal Rank Fusion (RRF) — a rank-based merge that needs no score calibration between the two systems. Then the optional semantic ranker, a Microsoft-hosted cross-encoder, re-scores the top ~50 fused candidates by jointly reading query + passage (a bi-encoder embedding only ever encodes them separately, which is why the cross-encoder wins on final precision) and can also emit extractive captions and answers for the citation UI. You retrieve ~50 for the reranker and feed only the top 3–5 into the prompt. The caller’s group GUIDs go in as an OData filter so forbidden passages are never returned — trimming happens in the engine, before anything reaches your process. (Semantic ranker is a metered add-on with its own quota — budget for it.)

The AI gateway — what APIM’s GenAI policies actually do

APIM is the one front door, and its GenAI policy set is the reason:

• validate-jwt — verify the Entra token’s signature, issuer, audience, and required group claims before the request goes anywhere.
• llm-token-limit (a.k.a. azure-openai-token-limit) — meter prompt + completion tokens against a key such as the business-line header, returning 429 + Retry-After and remaining-token headers when a desk blows its budget. It reads real usage from the response and estimates for streaming.
• llm-emit-token-metric (a.k.a. azure-openai-emit-token-metric) — emit per-call token metrics to Application Insights / Dynatrace, which is what feeds the chargeback dashboard.
• Backend pool + circuit breaker — load-balance across several OpenAI deployments (PTU primary, pay-as-you-go spillover), honour each backend’s Retry-After, and trip out an unhealthy one.
• Semantic caching (azure-openai-semantic-cache-lookup / -store) — serve a near-identical prior question from a Redis-backed cache using an embeddings similarity threshold, deflecting a big share of model calls.

One choke point to meter, throttle, route, cache, and audit every model call in the company.

Content filtering, Prompt Shields, groundedness, abuse monitoring, residency

Azure OpenAI ships a content filter that scores inputs and outputs across four harm categories — hate, sexual, violence, self-harm — at severity levels you can threshold; a custom content filter (an Azure AI Content Safety configuration) tightens or relaxes those per deployment. On top sit Prompt Shields (jailbreak and the under-appreciated indirect prompt injection hidden inside a retrieved document) and protected-material detection. Groundedness detection (Content Safety) checks whether the answer is actually supported by the passages you supplied and can return the ungrounded spans — treat it as your last line against a confidently wrong number, and note it has moved through preview, so pin to the current API and confirm GA/region availability before you depend on it.

Two governance controls reviewers ask about by name. Abuse monitoring: by default Azure may store prompts and completions transiently and run automated (occasionally human) review for abuse — regulated tenants can apply for modified abuse monitoring / no-data-logging so nothing is stored, which residency and privacy teams usually require. Data residency: Azure OpenAI processes inference in the deployment’s region and does not use your prompts or completions to train the base models; where you need a broader-than-region boundary with more capacity, Data Zone deployments keep processing within a geography (e.g. EU or US) rather than a single region. None of this replaces the network posture — it complements it.

Cost and PTU — the math that sets your bill

Azure OpenAI has two purchasing models and you will use both. Pay-as-you-go (Standard) bills per token against shared regional capacity — cheap to start, but subject to the 429s that ruin a streaming UX at market open. Provisioned Throughput Units (PTU) reserve dedicated capacity measured in PTUs, giving deterministic latency and throughput; PTU deployments have a minimum size, a throughput characterised in tokens/minute at a given input:output ratio, and are billed hourly — with monthly / annual reservations cutting the rate substantially. The pattern: size PTU to your p95 steady demand, then spill the peak to pay-as-you-go through the APIM backend pool, so you neither buy for the spike nor eat 429s during it. Stack the software levers on top — gpt-4o-mini for lookups, semantic-cache deflection, and disciplined top-k — and meter it all per business line so each desk owns its spend.

The AI Foundry / hub angle

Azure AI Foundry (formerly Azure AI Studio) is the managed home for this kind of work. A hub is the shared parent — it holds connections (to your private Azure OpenAI, AI Search, Content Safety), compute, storage, and the security config, and can run inside a managed VNet so its compute reaches your endpoints privately. Projects are workspaces under a hub where teams build. Foundry gives you prompt flow for orchestration and, most usefully for this platform, built-in evaluators — groundedness, relevance, coherence, fluency, retrieval — you run in the GitHub Actions gate so a prompt or model change is scored before it ships. The “chat on your data” wizard stands up a working private RAG app in an afternoon; you graduate to the AKS orchestrator here when you need the control, scale, and platform integration a managed runtime can’t give you. Same services underneath — different amount of steering wheel.

Practice challenges

Work these in order — they climb from “read the diagram” to “make the architecture decision.” Each solution says why, not just what.

1 (Beginner) — Name the five zones. List the five privatelink.* private DNS zones this platform must link to the VNet, and the service each one resolves.

<details><summary>Solution</summary>

privatelink.openai.azure.com (Azure OpenAI), privatelink.search.windows.net (AI Search), privatelink.documents.azure.com (Cosmos DB SQL API), privatelink.blob.core.windows.net (Blob Storage), privatelink.cognitiveservices.azure.com (Content Safety and Document Intelligence — they share it).

Why: the zone is half of a Private Endpoint; without the linked zone the public FQDN resolves to a firewalled public IP and every call hangs. Knowing the exact list is how you audit a deployment in thirty seconds. </details>

2 (Beginner → Intermediate) — Diagnose the silent hang. Calls from the orchestrator to Azure OpenAI hang ~30 s then time out. The Private Endpoint shows Approved / Succeeded, and local_auth is off. What is the most likely cause and the one command that confirms it?

<details><summary>Solution</summary>

The private DNS zone isn’t linked to the orchestrator’s VNet (or the A record / zone-group is missing), so the FQDN resolves to the public IP, which publicNetworkAccess = Disabled drops. Confirm from a pod inside the VNet (representative output shown):

nslookup oai-ragbank-prod-cin.openai.azure.com
# grounded result: resolves to a 10.x.x.x private IP  -> good
# broken result:   resolves to a public IP            -> zone not linked / no A record

Why: a healthy PE with a hanging call is almost always DNS, not RBAC or firewall — RBAC fails fast with 401/403, DNS-to-a-firewalled-public-IP fails slow with a timeout. </details>

3 (Intermediate) — Least-privilege roles. Write the exact data-plane RBAC roles for (a) the query orchestrator identity and (b) the ingestion identity. Why not just grant Contributor?

<details><summary>Solution</summary>

Query identity: Cognitive Services OpenAI User (call the model), Search Index Data Reader (query the index), and a Cosmos data role (Cosmos DB Built-in Data Reader/Contributor). Ingestion identity adds Search Index Data Contributor (write the index) and Storage Blob Data Reader (read the corpus).

Why: Contributor is a control-plane role — it manages the resource but grants no data-plane actions, so it can’t actually call OpenAI or read the index, and handing it out over-permissions the identity for management it should never do. Data-plane RBAC is a separate axis from control-plane RBAC. </details>

4 (Intermediate) — Security-trim filter. An advisor is a member of Entra groups A1B2 and C3D4. Write the AI Search OData filter that returns only passages they may read, and say why this lives in the index rather than in your app.

<details><summary>Solution</summary>

$filter=groups/any(g: search.in(g, 'A1B2, C3D4'))

Why: applied in the engine, a forbidden passage is never returned — it can’t leak into your process memory, your logs, or your traces. Filter in app code and the restricted content has already crossed the boundary before you drop it; one bug then leaks it across a regulatory line. </details>

5 (Advanced) — Size PTU + spillover. Interactive traffic sits at a p95 of ~180k tokens/min with 3× bursts at market open. Outline the deployment plan and the APIM mechanism that makes it hold together.

<details><summary>Solution</summary>

Provision a PTU deployment sized to the ~180k p95 for deterministic latency on interactive traffic, and add a pay-as-you-go deployment as spillover. In APIM, put both in a backend pool: route to PTU first, overflow the 3× burst to pay-as-you-go, and honour each backend’s Retry-After. Add llm-token-limit per business line so no single desk consumes the shared burst headroom.

Why: PTU buys predictable latency but you don’t want to pay for 3× peak all day; pay-as-you-go absorbs the spike but 429s if it’s your only path. The pool gives you both — a steady floor and an elastic ceiling. </details>

6 (Advanced) — Close a groundedness gap. Your eval set shows 4% of answers cite the right document but state a wrong number. Propose a layered fix across retrieval, prompt, guardrail, and CI.

<details><summary>Solution</summary>

• Retrieval: raise recall — hybrid + semantic rerank, retrieve ~50, and make sure the numeric passage (a table row) chunks intact by chunking on layout from Document Intelligence, not blindly on token count.
• Prompt: instruct “answer only from the provided context; if the figure isn’t present, say so,” and require the model to quote the source line it used.
• Guardrail: enable groundedness detection to flag-and-cite ungrounded spans, and fail closed on a hard number that isn’t supported.
• CI: add the failing questions to the golden set and gate deploys on the Foundry groundedness/relevance evaluators so the regression can’t ship again.

Why: a right-citation / wrong-number miss is usually a retrieval or chunking fault (the number got split out of the retrieved chunk), so fixing only the prompt treats the symptom. Layer the defences and measure each one. </details>

Common beginner mistakes

• “A Private Endpoint alone makes it private.” It doesn’t. Without the linked private DNS zone (and the endpoint’s A record), the service FQDN still resolves to its public IP, which publicNetworkAccess = Disabled then firewalls — so you get a silent hang, not privacy. Right model: Private Endpoint + linked privatelink.* zone + zone-group A record are one unit; ship them together or nothing works.

• “We’ll retrieve broadly and filter restricted docs in app code.” By then the restricted content has already left the index into your process, your logs, and your traces; one bug leaks it across a compliance line. Right model: stamp ACL groups on every chunk and pass them as an OData filter so the index never returns what the caller may not see.

• “Just fine-tune the model on our documents so it knows them.” Fine-tuning bakes facts into weights you can’t cite, can’t update the morning a fund is withdrawn, and can’t unlearn — and it still hallucinates. Right model: retrieve + cite for facts; fine-tune only to shift style or vocabulary, and let the two compose.

• “Vector search beats keyword search, so use pure vector.” Vectors are great at paraphrase (“exit load” ≈ “early-redemption penalty”) but miss exact tokens — ISINs, clause numbers, fund IDs — where one token must match. Right model: hybrid (vector + BM25) fused with RRF, then semantic rerank.

• “Ship with API keys now, turn them off later.” Keys are long-lived, copyable, attributable to nobody, and end up committed to a repo. Right model: local_auth off from day one, managed identity + Workload Identity for services, Vault for the few non-Azure secrets.

• “More retrieved passages must mean better answers.” Twenty passages bloat input tokens and cost, and the answer gets lost in the middle, which can lower groundedness. Right model: retrieve ~50 for the reranker, then feed only the top 3–5 to the model.

• “The AI gateway is optional plumbing.” For a ten-person pilot, maybe. At 4,000 regulated seats you need one place to meter tokens per desk, throttle, load-balance PTU ↔ pay-as-you-go, and produce a single audit log. Right model: APIM as the mandatory front door for every model call.

Glossary

• RAG (retrieval-augmented generation) — answer a question by retrieving relevant passages from a search index and having the model generate the answer from them, instead of from training memory.
• Grounding — constraining the model’s answer to the supplied source passages so it can be cited and verified.
• Embedding / embedding vector — a numeric vector representing text meaning; similar meanings sit close together, enabling semantic search. Here produced by text-embedding-3-large.
• Vector search / ANN — finding the nearest vectors to a query vector; done approximately (Approximate Nearest Neighbor) for speed.
• HNSW — Hierarchical Navigable Small World, the graph index AI Search uses for fast ANN; tuned by m, efConstruction, efSearch.
• BM25 — the classic keyword-relevance ranking function; strong at the exact-token matches vectors miss.
• Hybrid search — running a vector query and a BM25 keyword query together and fusing the results.
• RRF (Reciprocal Rank Fusion) — a rank-based way to merge two result lists without calibrating their scores.
• Semantic ranker / reranking — a Microsoft-hosted cross-encoder that re-scores the top candidates by reading query and passage together; more precise than the bi-encoder embedding used for initial recall.
• Security trimming / ACL stamping — tagging each chunk with the Entra groups allowed to read it and filtering retrieval by the caller’s groups, so forbidden passages are never returned.
• Private Endpoint — a NIC with a private IP in your subnet mapped to one PaaS instance, keeping its data-plane traffic inside the VNet.
• privatelink.* private DNS zone — the DNS zone that resolves a service’s public FQDN to its Private Endpoint’s private IP; without it, the name resolves public and the call hangs.
• publicNetworkAccess = Disabled — the setting that switches off a PaaS resource’s public data-plane door, leaving only the Private Endpoint.
• Managed identity — an Entra identity Azure manages for a workload so it authenticates with tokens, not stored keys.
• Workload Identity (Entra Workload ID) — the AKS mechanism that federates a pod’s Kubernetes service-account token to an Entra token, keylessly.
• local_auth / disableLocalAuth — the switch that turns off API-key auth on OpenAI/Search/Cosmos, forcing Entra-only access.
• Data-plane RBAC — roles that grant use of a resource’s data (e.g. Cognitive Services OpenAI User), distinct from control-plane roles like Contributor that only manage it.
• APIM AI gateway — API Management fronting Azure OpenAI to validate tokens, meter/throttle, route, cache, and audit every model call.
• llm-token-limit — the APIM GenAI policy (a.k.a. azure-openai-token-limit) that meters prompt + completion tokens per key and returns 429 over budget.
• PTU (Provisioned Throughput Units) — reserved Azure OpenAI capacity for deterministic latency/throughput, billed hourly (cheaper with reservations), as opposed to per-token pay-as-you-go.
• Prompt Shields — Content Safety detection for jailbreaks and indirect prompt injection hidden inside retrieved content.
• Groundedness detection — a Content Safety check for whether an answer is supported by the supplied sources (confirm preview/GA and region before relying on it).
• Content filtering — Azure OpenAI’s scoring of inputs/outputs across harm categories (hate, sexual, violence, self-harm) at configurable severity thresholds.
• Integrated vectorization — AI Search’s built-in indexer + skillset that chunks and embeds documents with no custom code, versus the push pattern where your function does it.
• Chunking — splitting a document into ~300–500-token passages (with overlap) so retrieval returns focused, embeddable units.
• effectiveDate / tombstoning — the metadata and cleanup that drop a withdrawn or superseded document from query results.
• AI Foundry (hub / project) — Azure’s managed AI workspace: a hub holding shared connections/compute/security, with projects underneath; ships prompt flow and built-in evaluators.
• PTU-to-p95 + spillover — sizing provisioned capacity to steady p95 demand and bursting the peak to pay-as-you-go via APIM’s backend pool.
• RTO / RPO — Recovery Time / Recovery Point Objectives: how fast you must recover and how much data loss you tolerate; set per tier.