AWS Enterprise Architecture: Generative-AI / RAG on Bedrock

Every enterprise now wants the same thing: a chatbot that answers questions from their own documents — policies, contracts, runbooks, product manuals, claims data — without hallucinating, without leaking PII, and without sending a single byte to a public model endpoint. The pattern that delivers this is Retrieval-Augmented Generation (RAG), and on AWS the managed path runs through Amazon Bedrock. This article is a complete, reusable reference architecture for building enterprise RAG on Bedrock: how the retrieval and generation paths actually fit together, how to wire identity and networking so it survives a security review, and how to keep the bill predictable as you scale from a 5,000-document pilot to a 50-million-chunk corpus.

In a nutshell

Think of RAG as an open-book exam for an AI. A closed-book exam forces the student to answer from memory alone — so they bluff when they don’t know, which is exactly what a raw large language model (LLM) does when it hallucinates. RAG hands the student the textbook, flips it open to the three most relevant pages, and says: “answer using only what’s on these pages, and tell me the page number.” The model stops guessing and starts reading. That single change — retrieve the right passages first, then generate an answer grounded in them — is the whole idea, and it is why RAG, not fine-tuning, is how serious enterprises put a trustworthy assistant on top of their own documents.

On AWS the pieces of that open-book exam map cleanly to services. The librarian who fetches the right pages is Amazon Bedrock Knowledge Bases. The shelf of books turned into something searchable by meaning is a vector store (Amazon OpenSearch Serverless here). The student who reads and writes the answer is a foundation model — Anthropic Claude, Amazon Nova, and others — called through Amazon Bedrock, a single managed API in front of many models. A Bedrock Guardrail stands at the exam-room door, refusing questions and answers that break the rules (toxic content, PII leaks, or answers the open book does not actually support). And critically, the whole exam runs inside your own AWS account over private networking, so no document ever touches a public endpoint.

The rest of this lesson is the production blueprint behind that mental model: how the pieces wire together, the IAM and networking that survive a security review, the code that actually makes the RetrieveAndGenerate call, and the cost and quality levers that decide whether your six-week pilot becomes a durable platform.

Level: Advanced · Time: ~55 min

Before you start, it helps to know:

• The AWS basics — accounts, the difference between IAM roles and users, VPCs and subnets, and S3. If IAM policy evaluation feels fuzzy, skim IAM least privilege & permission boundaries first, because the security model here leans on it hard.
• What an LLM, a token, and an embedding are at a hand-wave level — this lesson re-explains embeddings from first principles, so “hand-wave” is enough.
• That “serverless” means AWS runs the servers for you: you provision capacity and permissions, not machines.

After this lesson you will be able to:

• Explain the two independent lifecycles of a RAG system — ingestion and query — and why they scale and fail separately.
• Trace the seven-hop query path from user to grounded answer and name the AWS service at each hop.
• Choose a chunking strategy, an embeddings model + dimension, and a vector store for a given corpus, and defend the trade-offs.
• Configure a Bedrock Guardrail with contextual grounding so the assistant refuses to hallucinate instead of merely usually not hallucinating.
• Enforce document-level security with a retrieval metadata filter rather than trusting a prompt instruction.
• Reason about cost (tokens, OCUs, provisioned vs. on-demand) and DR (rebuild the index from S3) the way an architect does, not a hobbyist.

The business scenario

Picture a mid-sized insurance carrier — or a hospital network, or a manufacturing firm, the pattern is identical. They have twenty years of accumulated knowledge spread across SharePoint, a document management system, a few S3 buckets, and a wiki nobody trusts. Frontline staff spend a meaningful slice of every day searching for answers: “Does policy form HO-3 cover water damage from a burst pipe in an unoccupied home?” or “What’s the escalation procedure when a pump on line 4 trips on overcurrent?” The answers exist, in some PDF, but finding the exact clause takes ten minutes and three phone calls.

The naive fix — paste the documents into a public chatbot — is a non-starter. The documents contain customer PII, contract terms under NDA, and in regulated sectors (insurance, healthcare, finance) the data residency and auditability requirements make a public SaaS endpoint legally radioactive. The second naive fix — fine-tune a model on the corpus — is expensive, goes stale the moment a policy changes, and still hallucinates because fine-tuning teaches style, not facts.

RAG solves the actual problem. Instead of baking knowledge into model weights, you keep the authoritative documents in a vector store, retrieve the handful of passages relevant to each question at query time, and hand those passages to a foundation model with an instruction: answer using only this context, and cite your sources. The model’s job shrinks from “know everything” to “read these three paragraphs and summarize.” Hallucination drops sharply because the model is grounded in retrieved text, answers update the instant you re-index a changed document, and — critically — every answer carries citations back to the source, so a compliance officer can audit why the system said what it said.

The requirement set this architecture targets is the one nearly every enterprise lands on:

• Grounded answers with citations — no ungrounded generation; every claim traceable to a source document and page.
• Data never leaves the account boundary — inference, embeddings, and vectors all stay inside the customer’s VPC and AWS account, reachable only over private networking.
• Safety rails — block toxic content, deny-list sensitive topics, redact PII in both prompts and responses, and refuse to answer when retrieval returns nothing relevant.
• Source-of-truth sync — when a document changes in S3, the answer changes within minutes, with no model retraining.
• Cost that scales sub-linearly — a pilot should cost tens of dollars a month; a full rollout should be governed by token budgets and caching, not surprise bills.

That is exactly what Amazon Bedrock Knowledge Bases, OpenSearch Serverless, and Bedrock Guardrails compose into.

Architecture overview

The architecture has two distinct lifecycles that share one vector store: an ingestion path (asynchronous, batch, runs when documents change) and a query path (synchronous, low-latency, runs on every user question). Keeping them mentally separate is the key to reasoning about cost and scale, because they fail and scale independently.

Ingestion path. Source documents land in an Amazon S3 bucket — the system of record for the knowledge corpus. An Amazon Bedrock Knowledge Base is configured with that bucket as its data source. When you trigger an ingestion job (manually, on a schedule via EventBridge, or reactively from an S3 event), the Knowledge Base does the heavy lifting automatically: it pulls each new or changed object, parses it (PDF, DOCX, HTML, plain text, CSV, and via the built-in foundation-model parser, even scanned documents and complex tables), splits the text into chunks according to your chunking strategy, calls a Bedrock embeddings model (Amazon Titan Text Embeddings v2 or Cohere Embed) to turn each chunk into a vector, and writes the vector plus the chunk text and metadata into the vector index. You write no ingestion code — the Knowledge Base is a managed ETL-to-vectors pipeline.

The vector store. Vectors live in an Amazon OpenSearch Serverless collection of type vector search. This is the durable, queryable index of your entire corpus, holding the embedding vector, the original chunk text (so retrieval returns the actual passage, not just an ID), and metadata fields (source URI, page number, document type, and any custom tags like business_unit or classification). OpenSearch Serverless uses an HNSW (Hierarchical Navigable Small World) graph for approximate nearest-neighbor search, which is what makes sub-second semantic retrieval across tens of millions of chunks possible.

Query path. A user asks a question in a front-end — a web app, a Slack bot, an internal portal. The request hits Amazon API Gateway, authenticated by Amazon Cognito (or your enterprise IdP federated through it). API Gateway invokes an AWS Lambda function — the orchestrator. Lambda calls the Bedrock RetrieveAndGenerate API (or, for more control, Retrieve followed by a separate InvokeModel). Under the hood, Bedrock embeds the user’s question with the same embeddings model used at ingestion, runs a k-NN search against OpenSearch Serverless to pull the top-K most semantically similar chunks, assembles those chunks into a prompt with a grounding instruction, sends that prompt to a generation model (Anthropic Claude, Amazon Nova, etc.) through an attached Bedrock Guardrail, and returns the generated answer plus a structured list of citations pointing at the exact source chunks. Lambda relays the answer and citations back through API Gateway to the user.

The diagram, described in words: imagine two horizontal swim-lanes. The top lane (ingestion) flows left-to-right: S3 (documents) → Bedrock Knowledge Base ingestion job → [parse → chunk → Titan embeddings] → OpenSearch Serverless vector collection. The bottom lane (query) also flows left-to-right but loops through the same vector store: User → API Gateway (Cognito auth) → Lambda orchestrator → Bedrock RetrieveAndGenerate, where RetrieveAndGenerate fans down to OpenSearch Serverless (retrieve top-K) and across to the Claude generation model wrapped by a Guardrail, then the answer-with-citations flows back up the chain to the user. The OpenSearch Serverless collection sits in the middle, written by the top lane and read by the bottom lane. Everything Bedrock and OpenSearch related is reached over VPC interface endpoints (AWS PrivateLink) inside a private VPC — no traffic traverses the public internet. CloudWatch, CloudTrail, and S3 model-invocation logging wrap the whole picture for observability and audit, and AWS KMS customer-managed keys encrypt S3, the OpenSearch collection, and the transient session data Bedrock holds during a request.

The elegance is that Bedrock Knowledge Bases collapses what used to be a sprawling custom pipeline (a document loader, a chunker, an embeddings service, a vector DB client, and a retrieval-prompt-assembly layer) into one managed API surface. Your code is the thin Lambda orchestrator and the front-end — the undifferentiated heavy lifting is AWS-operated.

Worked example: tracing one question end to end

Abstract boxes only click once you follow a single question through them. Take the adjuster’s query — “Does form HO-3 cover water damage from a burst pipe in an unoccupied home?” — and watch it travel the numbered query path from the diagram:

1. Sign-in (Cognito + WAF). The adjuster is already signed in through corporate SSO; their browser holds a short-lived JWT. WAF screens the request for common attacks and rate abuse before it ever reaches your API.
2. API Gateway. The Cognito authorizer validates the JWT. Inside the token sit the adjuster’s group claims — cognito:groups = ["claims"]. An invalid or expired token returns 401, and the request never reaches your code, your retriever, or your model — you pay nothing.
3. Lambda orchestrator. Lambda reads groups from the request context and builds a metadata filter: business_unit = "claims". It does not forward the raw question to a model yet — first it lets Bedrock retrieve.
4. Embed + retrieve (Bedrock → OpenSearch). Bedrock embeds the question with the same Titan model used at ingest, producing a 512-number vector (this corpus chose 512 dimensions). OpenSearch Serverless runs a k-NN search over its HNSW graph and returns the top 5 chunks whose vectors are nearest and that satisfy the business_unit = claims filter. Suppose the top hit is a chunk from HO-3.pdf, page 14, the “Water Damage — Exclusions” clause.
5. Augment. Bedrock assembles a prompt: a system instruction (“Answer only from the context below; cite sources; if the context is insufficient, say so”), the 5 retrieved chunks, and the user’s question. This assembly is the “augment” in Retrieval-Augmented Generation.
6. Generate through the Guardrail (Bedrock). The prompt goes to Claude through the attached Guardrail. The model drafts: “HO-3 excludes water damage from a pipe that froze because the dwelling was vacant and unheated… [HO-3.pdf, p.14].” The Guardrail’s contextual-grounding check scores whether that sentence is genuinely supported by the retrieved text. It is (say, 0.91, above the 0.75 threshold), so it passes. Had the model invented a clause, the score would crater and the answer would be swapped for the safe fallback.
7. Answer + citations back. Lambda receives output.text plus citations[], reshapes the citation into a clickable link to HO-3.pdf#page=14, and returns it through API Gateway. Total wall-clock is typically 1–4 seconds, most of it spent in step 6’s generation.

Notice what the adjuster’s identity did and did not do. It constrained what could be retrieved (via the metadata filter) — but it never granted the adjuster a single Bedrock permission. The Lambda’s execution role called Bedrock; the user only ever held a JWT. That separation — the user picks what to ask, the role decides what may be seen and invoked — is the backbone of RAG security, and we return to it in Enterprise considerations.

Component breakdown

A few component choices deserve a closer look because they are the ones that most often get decided wrong.

Chunking strategy is the single highest-leverage knob. The Knowledge Base offers fixed-size chunking (e.g., 300 tokens with a 20% overlap), no chunking (treat each file as one chunk — good for short, atomic FAQ entries), hierarchical chunking (parent-child: retrieve small precise child chunks but feed the larger parent chunk to the model for context), and semantic chunking (split on semantic boundaries rather than token counts). For dense policy and legal documents where a clause spans a paragraph, hierarchical chunking is usually the winner: you get the retrieval precision of small chunks and the answer quality of large context. Fixed-size with overlap is the safe default for general prose. The overlap matters because it prevents a relevant sentence from being orphaned at a chunk boundary where neither neighboring chunk carries enough context to be retrieved.

Embedding dimension trades recall against cost. Titan Text Embeddings v2 supports 256, 512, or 1024 dimensions. Higher dimensions capture more semantic nuance (better recall on subtle queries) but cost more to store and search. For most enterprise corpora, 512 is the sweet spot; reserve 1024 for highly technical domains where near-synonyms carry distinct meaning (legal, medical, engineering specs).

Guardrails do four jobs at once, and the grounding job is the one people forget. Beyond content filtering, denied topics, and PII handling, Bedrock Guardrails include contextual grounding checks: the guardrail scores whether the model’s answer is actually supported by the retrieved passages (grounding score) and whether it is relevant to the user’s question (relevance score). If either falls below your configured threshold, the answer is blocked or replaced with a safe fallback (“I don’t have enough information to answer that”). This is what turns “RAG that usually doesn’t hallucinate” into “RAG that refuses to hallucinate” — a hard requirement in regulated settings.

PII handling: block vs. mask is a per-entity decision. A guardrail can be told that a Social Security Number in a response must be blocked (the whole response is suppressed), while a phone number can be masked (replaced with {PHONE} and the rest of the answer returned). You typically block the high-sensitivity entities and mask the rest, and you apply the input filter too so that a user pasting a customer’s SSN into a question doesn’t propagate it into logs.

How embeddings and vector search actually work

The one concept a beginner must truly internalise is the embedding, because everything else in RAG is built on top of it. An embedding is a list of numbers — a vector — that a model assigns to a piece of text so that texts with similar meaning get similar vectors. “Water damage from a burst pipe” and “flooding caused by a ruptured plumbing line” share almost no words, yet a good embedding model places their vectors close together because it was trained to capture meaning, not spelling. That is the superpower plain keyword search never had.

“Close together” has a precise definition. Each vector is a direction in a high-dimensional space — 512 numbers means 512 dimensions. Similarity is usually cosine similarity: the cosine of the angle between two vectors, ranging from 1.0 (identical direction, same meaning) through 0.0 (unrelated) to −1.0 (opposite). Retrieval is then just: embed the question, and find the chunk vectors with the highest cosine similarity to it. A toy intuition, collapsed to three dimensions so you can eyeball it:

Chunk A wins even though it shares zero keywords with the query, while Chunk C loses despite both mentioning nothing in common — meaning, not vocabulary, drives the ranking. Real vectors carry 256–1024 dimensions instead of 3, but the arithmetic is identical.

Comparing the query against every chunk (a brute-force scan) is exact but far too slow at millions of chunks. So OpenSearch builds an HNSW graph — Hierarchical Navigable Small World — a navigable network in which search “walks” from vector to nearest neighbour, visiting a few dozen candidates instead of scanning millions. This is approximate nearest-neighbour (ANN) search: it trades a sliver of recall for an enormous speed-up, which is what makes sub-second retrieval over tens of millions of chunks possible at all. Two HNSW knobs govern the trade-off — m (how many links each node keeps) and ef_construction / ef_search (how wide the search beam is). Higher values improve recall at the cost of build time, memory, and query latency.

Two rules fall straight out of this mechanism, and violating either is the most common way RAG “mysteriously” returns garbage:

• You must embed the query with the exact same model and dimension you used for the chunks. Vectors from two different models live in incompatible coordinate systems — comparing them is comparing gibberish. This is why “same model for ingest and query” is non-negotiable, and why changing your embeddings model later forces a full re-index of the corpus.
• Semantic retrieval finds similar meaning, not necessarily correct answers. If the corpus does not contain the answer, retrieval still returns its five nearest chunks — nearest to nothing useful. That is exactly the failure mode the contextual-grounding guardrail exists to catch; without it the model will happily write a confident, well-cited, wrong answer from irrelevant context.

Hybrid search hedges the weakness of pure semantics. Enterprise queries frequently contain exact tokens that must match — a form number like HO-3, an error code, an SKU, a statute reference — where semantic similarity can drift to a near-neighbour that is subtly wrong. OpenSearch Serverless supports hybrid search, blending k-NN semantic scores with classic BM25 keyword scores, which you switch on in a Knowledge Base by setting the retrieval search type to HYBRID. For corpora dense with identifiers, hybrid retrieval is often the single biggest quality win after chunking.

Implementation guidance

Provision in IaC, in the right order. Terraform is the natural fit on AWS, and the AWS provider (5.x and later) plus awscc cover the Bedrock and OpenSearch Serverless resources. The dependency order is strict and is where most first attempts stall:

1. KMS keys for the data domains (one CMK for the corpus is fine to start; split by classification later).
2. S3 bucket for documents — versioned, SSE-KMS with the CMK, public access fully blocked.
3. OpenSearch Serverless collection of type VECTORSEARCH, plus its three policy types: an encryption policy (binds the collection to a KMS key), a network policy (here, VPC-only via the OpenSearch Serverless VPC endpoint), and a data-access policy (grants the Knowledge Base’s IAM role permission to create the index and read/write documents). The data-access policy is the step people miss — without it the ingestion job fails with an opaque permissions error.
4. The vector index inside the collection, with the correct field mapping: a knn_vector field whose dimension exactly matches your chosen embeddings model output (e.g., 512), an HNSW engine config, a text field for the chunk content, and a metadata field. Terraform can create this via the opensearch provider’s opensearch_index resource or you let the Knowledge Base create it on first use — but creating it explicitly gives you control over the HNSW ef_construction and m parameters that govern recall vs. index build cost.
5. IAM service role for the Knowledge Base, with a trust policy allowing bedrock.amazonaws.com and permissions scoped to: read the S3 bucket, invoke the embeddings model (bedrock:InvokeModel on the specific Titan model ARN), use the KMS key, and API access to the specific OpenSearch Serverless collection.
6. The Bedrock Knowledge Base resource (aws_bedrockagent_knowledge_base), wiring the role, the OpenSearch Serverless collection ARN, the field mapping, and the embeddings model ARN.
7. The data source (aws_bedrockagent_data_source) pointing at the S3 bucket, with the chunking configuration and parsing strategy.
8. The Guardrail (aws_bedrock_guardrail) with content filters, denied topics, PII entities, and contextual-grounding thresholds — then publish a guardrail version so the Lambda can pin to an immutable version rather than the mutable draft.
9. Lambda, API Gateway, Cognito, WAF, and the VPC interface endpoints for com.amazonaws.<region>.bedrock-runtime, bedrock-agent-runtime, and the OpenSearch Serverless endpoint, plus an S3 gateway endpoint.

A non-obvious sequencing note: the OpenSearch Serverless data-access policy must reference the Knowledge Base role ARN, and the Knowledge Base must reference the collection — a circular-looking dependency that Terraform resolves cleanly only if you create the role first, then the access policy, then the collection index, then the Knowledge Base. Build the role up front.

Networking and identity wiring. Put the Lambda orchestrator in private subnets and attach the VPC endpoints so Bedrock and OpenSearch calls never leave the VPC. The endpoint security groups should allow inbound 443 only from the Lambda’s security group. For identity, the chain is: end user authenticates to Cognito (federated to corporate Entra ID / Okta via SAML or OIDC, so you reuse existing SSO and MFA), Cognito issues a JWT, API Gateway validates it with a Cognito authorizer, and the Lambda execution role — not the user — calls Bedrock. The user’s identity and group claims are passed into the Lambda as request context and used to construct a metadata filter for retrieval, so a user in the claims group only retrieves chunks tagged business_unit = claims. This is how you do row-level / document-level security in RAG: not by giving users direct Bedrock permissions, but by constraining what the retriever is allowed to return based on their identity. That filter is passed to RetrieveAndGenerate via the retrievalConfiguration.vectorSearchConfiguration.filter parameter.

The orchestrator call. The minimal Lambda logic: extract the user’s groups from the JWT, build the metadata filter, then call bedrock-agent-runtime RetrieveAndGenerate with the Knowledge Base ID, the model ARN for generation, the guardrail ID and version, and the filter. The response contains the output.text (the grounded answer) and citations[] (each with the generated text span and the retrievedReferences — source S3 URI, chunk text, and metadata). The Lambda reshapes citations into clickable source links for the UI. For streaming UX, use RetrieveAndGenerateStream and proxy the token stream through API Gateway (or a Lambda function URL with response streaming) so the answer renders progressively.

A note on the alternative IaC stacks. If your shop standardizes on CloudFormation/CDK, the equivalents exist (AWS::Bedrock::KnowledgeBase, AWS::OpenSearchServerless::Collection, AWS::Bedrock::Guardrail), and CDK’s L2 constructs handle the role-and-policy wiring more ergonomically than raw Terraform. Bicep and Deployment Manager are Azure and GCP tooling respectively and do not apply here — on AWS the realistic choices are Terraform or CDK, and the patterns above translate one-to-one.

Worked example: the vector store and Knowledge Base in Terraform

The prose above says “create the three OpenSearch policies, then the collection, then the role, then the Knowledge Base.” Here is what that actually looks like. Everything below is illustrative — account IDs, ARNs, and names are placeholders — but it is schema-correct against the AWS provider (5.x) and shows the exact ordering that trips up first attempts. The OpenSearch Serverless collection needs three policies: an encryption policy (binds it to a KMS key), a network policy (VPC-only, no public access), and — the step people forget — a data-access policy that names the Knowledge Base role.

# 1. Encryption policy binds the collection to a customer-managed KMS key.
resource "aws_opensearchserverless_security_policy" "kb_encryption" {
  name = "kb-rag-encryption"
  type = "encryption"
  policy = jsonencode({
    Rules       = [{ ResourceType = "collection", Resource = ["collection/kb-rag"] }]
    AWSOwnedKey = false
    KmsARN      = aws_kms_key.corpus.arn
  })
}

# 2. Network policy: reach the collection ONLY through its VPC endpoint.
resource "aws_opensearchserverless_security_policy" "kb_network" {
  name = "kb-rag-network"
  type = "network"
  policy = jsonencode([{
    Rules = [
      { ResourceType = "collection", Resource = ["collection/kb-rag"] },
      { ResourceType = "dashboard",  Resource = ["collection/kb-rag"] },
    ]
    AllowFromPublic = false
    SourceVPCEs     = [aws_opensearchserverless_vpc_endpoint.kb.id]
  }])
}

# 3. The vector-search collection itself (must come AFTER its encryption policy).
resource "aws_opensearchserverless_collection" "kb" {
  name       = "kb-rag"
  type       = "VECTORSEARCH"
  depends_on = [aws_opensearchserverless_security_policy.kb_encryption]
}

# 4. Data-access policy — the one people miss. Grants the KB role rights on the index.
resource "aws_opensearchserverless_access_policy" "kb_data" {
  name = "kb-rag-data"
  type = "data"
  policy = jsonencode([{
    Rules = [
      { ResourceType = "index", Resource = ["index/kb-rag/*"],
        Permission = ["aoss:CreateIndex", "aoss:UpdateIndex", "aoss:DescribeIndex",
                      "aoss:ReadDocument", "aoss:WriteDocument"] },
      { ResourceType = "collection", Resource = ["collection/kb-rag"],
        Permission = ["aoss:DescribeCollectionItems"] },
    ]
    Principal = [aws_iam_role.kb.arn]
  }])
}

The Knowledge Base role needs a confused-deputy-safe trust policy (scoped by source account and source ARN) plus a least-privilege permissions policy. Note the subtle double gate on OpenSearch: the role needs the IAM action aoss:APIAccessAll and must be named in the data-access policy above — either one alone fails with an opaque error.

resource "aws_iam_role" "kb" {
  name = "bedrock-kb-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "bedrock.amazonaws.com" }
      Action    = "sts:AssumeRole"
      Condition = {
        StringEquals = { "aws:SourceAccount" = "123456789012" }
        ArnLike      = { "aws:SourceArn" = "arn:aws:bedrock:us-east-1:123456789012:knowledge-base/*" }
      }
    }]
  })
}

resource "aws_iam_role_policy" "kb" {
  role = aws_iam_role.kb.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      { Sid = "ReadCorpus", Effect = "Allow", Action = ["s3:GetObject", "s3:ListBucket"],
        Resource = [aws_s3_bucket.corpus.arn, "${aws_s3_bucket.corpus.arn}/*"] },
      { Sid = "Embed", Effect = "Allow", Action = "bedrock:InvokeModel",
        Resource = "arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v2:0" },
      { Sid = "SearchVectors", Effect = "Allow", Action = "aoss:APIAccessAll",
        Resource = aws_opensearchserverless_collection.kb.arn },
      { Sid = "UseKey", Effect = "Allow", Action = ["kms:Decrypt", "kms:GenerateDataKey"],
        Resource = aws_kms_key.corpus.arn },
    ]
  })
}

Now the Knowledge Base and its S3 data source. The embedding dimension in the model config must equal the knn_vector dimension of the index — mismatch it and ingestion fails. The data source carries the chunking strategy (hierarchical here, matching the reference example):

resource "aws_bedrockagent_knowledge_base" "rag" {
  name     = "enterprise-rag"
  role_arn = aws_iam_role.kb.arn

  knowledge_base_configuration {
    type = "VECTOR"
    vector_knowledge_base_configuration {
      embedding_model_arn = "arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v2:0"
      embedding_model_configuration {
        bedrock_embedding_model_configuration {
          dimensions = 512 # MUST match the vector index field dimension
        }
      }
    }
  }

  storage_configuration {
    type = "OPENSEARCH_SERVERLESS"
    opensearch_serverless_configuration {
      collection_arn    = aws_opensearchserverless_collection.kb.arn
      vector_index_name = "kb-rag-index"
      field_mapping {
        vector_field   = "bedrock-knowledge-base-default-vector"
        text_field     = "AMAZON_BEDROCK_TEXT_CHUNK"
        metadata_field = "AMAZON_BEDROCK_METADATA"
      }
    }
  }

  depends_on = [aws_opensearchserverless_access_policy.kb_data]
}

resource "aws_bedrockagent_data_source" "s3" {
  knowledge_base_id = aws_bedrockagent_knowledge_base.rag.id
  name              = "s3-corpus"

  data_source_configuration {
    type = "S3"
    s3_configuration { bucket_arn = aws_s3_bucket.corpus.arn }
  }

  vector_ingestion_configuration {
    chunking_configuration {
      chunking_strategy = "HIERARCHICAL"
      hierarchical_chunking_configuration {
        overlap_tokens = 60
        level_configuration { max_tokens = 1500 } # parent (fed to the model)
        level_configuration { max_tokens = 300 }  # child (matched at retrieval)
      }
    }
  }
}

The KMS key those resources reference deserves a tight key policy scoped to only the KB and collection roles — see KMS encryption deep dive for envelope encryption and key-policy patterns.

Worked example: the Lambda orchestrator

With the Knowledge Base in place, the query-path code is small. The Lambda extracts the user’s groups, builds the metadata filter, and makes one RetrieveAndGenerate call — retrieval, prompt assembly, guardrail, and generation all happen server-side inside Bedrock:

import json, boto3

agent = boto3.client("bedrock-agent-runtime")

KB_ID     = "KB1234ABCD"
# Use a cross-region INFERENCE PROFILE id, not a bare model id — many models require it.
MODEL_ARN = ("arn:aws:bedrock:us-east-1:123456789012:"
             "inference-profile/us.anthropic.claude-3-5-sonnet-20241022-v2:0")
GUARDRAIL = {"id": "gr-abcd1234", "version": "3"}  # pin a published version, never DRAFT


def handler(event, _ctx):
    claims   = event["requestContext"]["authorizer"]["jwt"]["claims"]
    groups   = claims.get("cognito:groups", "").split(",")   # e.g. ["claims"]
    question = json.loads(event["body"])["question"]

    # Document-level security: constrain WHAT can be retrieved, by the user's group.
    md_filter = {"in": {"key": "business_unit", "value": groups}}

    resp = agent.retrieve_and_generate(
        input={"text": question},
        retrieveAndGenerateConfiguration={
            "type": "KNOWLEDGE_BASE",
            "knowledgeBaseConfiguration": {
                "knowledgeBaseId": KB_ID,
                "modelArn": MODEL_ARN,
                "retrievalConfiguration": {
                    "vectorSearchConfiguration": {
                        "numberOfResults": 5,
                        "overrideSearchType": "HYBRID",
                        "filter": md_filter,
                    }
                },
                "generationConfiguration": {
                    "guardrailConfiguration": {
                        "guardrailId": GUARDRAIL["id"],
                        "guardrailVersion": GUARDRAIL["version"],
                    },
                    "inferenceConfig": {
                        "textInferenceConfig": {"temperature": 0.2, "maxTokens": 512}
                    },
                },
            },
        },
    )

    citations = [
        {"snippet": ref["content"]["text"][:200],
         "source":  ref["location"]["s3Location"]["uri"]}
        for c in resp.get("citations", [])
        for ref in c.get("retrievedReferences", [])
    ]

    return {"statusCode": 200,
            "body": json.dumps({"answer": resp["output"]["text"], "citations": citations})}

Two field-name gotchas worth memorising. First, RetrieveAndGenerate names its guardrail block guardrailConfiguration / guardrailId, but the Converse API (which you use for the full-control Retrieve-then-generate-yourself variant) names it guardrailConfig / guardrailIdentifier. Second, the filter’s in operator takes a list value; equals takes a scalar. Getting these subtly wrong is the most common cause of a request that validates but behaves oddly. For streaming UX, swap to retrieve_and_generate_stream and proxy the token stream through a Lambda function URL with response streaming.

Worked example: a Guardrail with contextual grounding

The Guardrail is what upgrades the system from “RAG that usually doesn’t hallucinate” to “RAG that refuses to.” Note that PROMPT_ATTACK is an input-only filter — its output_strength must be NONE — and PII entities are individually set to BLOCK (suppress the whole response) or ANONYMIZE (mask and continue):

resource "aws_bedrock_guardrail" "rag" {
  name                      = "enterprise-rag-guardrail"
  blocked_input_messaging   = "I can't help with that request."
  blocked_outputs_messaging = "I couldn't find that in our current documents — please escalate to a senior underwriter."

  content_policy_config {
    filters_config { type = "HATE"          input_strength = "HIGH" output_strength = "HIGH" }
    filters_config { type = "PROMPT_ATTACK" input_strength = "HIGH" output_strength = "NONE" }
  }

  sensitive_information_policy_config {
    pii_entities_config { type = "US_SOCIAL_SECURITY_NUMBER" action = "BLOCK" }
    pii_entities_config { type = "PHONE"                     action = "ANONYMIZE" }
    pii_entities_config { type = "EMAIL"                     action = "ANONYMIZE" }
  }

  topic_policy_config {
    topics_config {
      name       = "coverage-advice"
      definition = "Personalised legal or coverage advice to a policyholder."
      type       = "DENY"
    }
  }

  contextual_grounding_policy_config {
    filters_config { type = "GROUNDING"  threshold = 0.75 }
    filters_config { type = "RELEVANCE"  threshold = 0.70 }
  }
}

Then publish an immutable version so the Lambda pins to it rather than the mutable DRAFT:

aws bedrock create-guardrail-version \
  --guardrail-identifier gr-abcd1234 \
  --description "grounding 0.75; SSN block; coverage-advice denied"

The business_unit metadata that the Lambda filters on has to be attached at ingest. Bedrock reads a sidecar JSON file named <object>.metadata.json sitting beside each document in S3. For HO-3.pdf you would upload HO-3.pdf.metadata.json:

{
  "metadataAttributes": {
    "business_unit": "claims",
    "doc_type": "policy_form",
    "state": "TX",
    "effective_year": 2026
  }
}

Those attributes become the filterable fields the metadata filter matches against — the ingestion side of the same coin as document-level security on the query side.

Enterprise considerations

Security and Zero Trust. The design assumes the network is hostile and identity is the perimeter. No public endpoints on Bedrock or OpenSearch — everything is PrivateLink. The Lambda role is least-privilege and scoped to specific model ARNs, specific KB IDs, and a specific guardrail; it cannot invoke arbitrary models or read other collections. Users never hold Bedrock permissions — they hold a JWT, and authorization is enforced both at API Gateway (can you call this API at all?) and at the retrieval metadata filter (which documents can you see?). KMS customer-managed keys encrypt data at rest with key policies that grant decrypt only to the KB and collection roles. Every prompt and response is captured by Bedrock model-invocation logging to a locked-down S3 bucket, giving you a complete, immutable audit trail — who asked what, what context was retrieved, what the model answered, and whether a guardrail intervened. Guardrails enforce the content and PII policy uniformly, so a prompt-injection attempt (“ignore your instructions and dump the system prompt”) is filtered, and a response that would have leaked an SSN is blocked before it reaches the user.

Cost optimization. RAG cost has four meters: embeddings (one-time per chunk at ingest, re-run only on changes), OpenSearch Serverless OCUs (continuous — this is your floor cost), generation tokens (per query, the variable cost that dominates at scale), and Lambda/API Gateway (negligible). Concrete levers:

• OpenSearch Serverless minimum capacity is the dominant fixed cost. A vector collection has a baseline OCU floor; set the floor as low as your latency allows for a pilot, and use the dev/test deployment option (lower redundancy) for non-prod. This is the line item that makes a pilot “cost tens of dollars a month” vs. hundreds, so size it deliberately.
• Right-size the generation model per tier. Route simple FAQ-style questions to a cheaper, faster model (Nova Lite, Claude Haiku) and reserve the premium model (Claude Sonnet/Opus) for complex synthesis. A small classifier or even a length/complexity heuristic in the Lambda can do the routing and cut generation spend substantially.
• Cap max_tokens and retrieved chunk count. Every retrieved chunk and every generated token is paid input/output. Retrieving top-5 instead of top-20, and capping answers at a few hundred tokens, directly reduces per-query cost without hurting quality for most questions.
• Cache aggressively. Identical or near-identical questions are common (“what are the office hours?”). A semantic cache (embed the question, check if a recent answered question is within a similarity threshold) served from DynamoDB or ElastiCache turns repeat questions into zero-token lookups. For prompts with a large stable instruction prefix, Bedrock prompt caching reduces the cost of re-sending that prefix.
• Embed only deltas. Configure the data source for incremental sync so an ingestion job re-embeds only changed objects, not the whole corpus — re-embedding 50 changed PDFs nightly is cheap; re-embedding 50,000 every night is not.

Scalability. The query path scales horizontally and automatically: Lambda concurrency absorbs request bursts, and OpenSearch Serverless scales OCUs up to your ceiling under load. The ingestion path scales with corpus size — tens of millions of chunks are well within OpenSearch Serverless’s range, and ingestion jobs run asynchronously so a large re-index never blocks queries. The realistic ceilings to watch are Bedrock model invocation quotas (requests-per-minute and tokens-per-minute per model — request increases proactively before a launch) and the OpenSearch OCU ceiling (raise it for known traffic peaks). Multi-tenancy scales by metadata filtering within one collection for moderate tenant counts; for strict isolation or very high tenant counts, separate Knowledge Bases or collections per tenant trade cost for blast-radius isolation.

Reliability and DR (RTO/RPO). The corpus in S3 (versioned, optionally cross-region replicated) is the true source of truth — your RPO is effectively zero for the documents because S3 durability is eleven nines and CRR replicates changes continuously. The vector index is derived data: if a region or collection is lost, you don’t need to have backed up the vectors, because you can rebuild the entire index from S3 by re-running ingestion. That reframes DR beautifully — your recovery plan for the vector store is “re-ingest,” and your RTO is the time to re-run ingestion over the corpus (minutes to a few hours depending on size), not a database restore. For a hot standby, deploy the stack in a second region with its own Knowledge Base pointed at the replicated S3 bucket and keep both indexes warm; Route 53 health checks fail traffic over. For most enterprises a warm-standby (infrastructure pre-provisioned, index periodically refreshed) hits a sensible RTO/RPO at a fraction of active-active cost.

Observability. Instrument three layers. Infrastructure: CloudWatch metrics on Lambda (errors, duration, concurrency), API Gateway (4xx/5xx, latency), and OpenSearch Serverless (search latency, OCU utilization). Bedrock: model-invocation logging captures every prompt/response pair, and Guardrail metrics tell you how often (and why) interventions fire — a spike in grounding-check blocks means retrieval quality has degraded. Quality: this is the layer teams skip and regret. Log every question, the retrieved chunk IDs and their relevance scores, the answer, the citations, and any user feedback (thumbs up/down). Pipe this to CloudWatch/OpenSearch dashboards so you can spot questions that retrieve nothing relevant (a corpus gap), answers that get blocked (a guardrail tuning issue), and topics with poor satisfaction (a chunking or model issue). RAG quality is an operational discipline, not a launch-day checkbox.

Governance. Because every answer carries citations and every invocation is logged, you get auditability for free — a regulator or internal compliance team can trace any answer to its source and inspect the exact context the model saw. Establish a content governance process: who approves documents into the corpus, how sensitive documents are classified and tagged (so metadata filters can enforce access), and how the corpus is reviewed for stale or contradictory content (RAG faithfully retrieves outdated policies if you leave them in S3). Version your Guardrails and treat policy changes as reviewed, audited deployments. Tag all resources by data domain and cost center for FinOps allocation.

Reference enterprise example

MeridianMutual, a fictional regional property-and-casualty insurer with about 1,400 employees and 380,000 policyholders, ran a six-week pilot to put a grounded assistant in front of its claims and underwriting teams. The corpus: roughly 9,000 documents — policy forms, state-specific endorsements, underwriting guidelines, and claims-handling runbooks — totaling about 70,000 chunks after hierarchical chunking. Daily query volume at launch was projected at 8,000 questions, climbing to ~25,000 as adoption spread.

Decisions they made. They chose hierarchical chunking because policy clauses span paragraphs and the small-child / large-parent pattern gave noticeably better answers than fixed-size on their evaluation set. Embeddings: Titan Text Embeddings v2 at 512 dimensions — they tested 1024 and found the recall gain didn’t justify the storage and search cost for their corpus. Generation: a two-tier routing scheme — Nova Lite for straightforward lookups (“what’s the deductible on form HO-3?”) and Claude Sonnet for multi-document synthesis (“compare coverage for water damage across our HO-3, HO-5, and the state X endorsement”). The Lambda routed based on a lightweight complexity heuristic, sending roughly 70% of traffic to the cheaper tier.

Guardrails configuration. They blocked SSNs and full account numbers in responses outright, masked phone numbers and email addresses, denied the topic of “personalized legal or coverage advice to policyholders” (the assistant is an internal staff tool, not a customer-facing advisor), and set the contextual-grounding threshold at 0.75 so that any answer the guardrail judged insufficiently supported by retrieved text was replaced with “I couldn’t find that in our current documents — please escalate to a senior underwriter.” In the first two weeks that fallback fired on about 6% of questions, which turned out to be a feature: it surfaced eleven genuine corpus gaps (procedures that simply weren’t documented anywhere), which the knowledge team then wrote up and added to S3.

Security and networking. The entire stack ran in a private VPC with PrivateLink endpoints — a hard requirement from their security team, who signed off only after confirming via VPC Flow Logs that zero Bedrock traffic egressed to the internet. Cognito federated to their existing Entra ID, so staff used their normal SSO and MFA. Document-level security was enforced by metadata filter: claims adjusters retrieved only claims and policy documents, underwriters retrieved underwriting guidelines plus policy forms, and a small compliance group could retrieve everything.

The numbers. OpenSearch Serverless ran near its minimum OCU floor for a corpus this size, and that fixed cost plus the variable generation tokens (held down by tier routing, top-5 retrieval, capped answer length, and a semantic cache that absorbed about 18% of questions as cache hits) kept the pilot well under their internal threshold — comfortably a few hundred dollars a month at pilot volume, scaling roughly linearly with query count thereafter. The cost model was predictable, which mattered more to the CFO than the absolute figure: spend tracked token budgets, with no surprise spikes.

The outcome. Average time-to-answer for a covered-vs-not question dropped from the old “ten minutes and a phone call” to under thirty seconds, with a citation the adjuster could open and verify. Crucially, because every answer cited its source, adjusters trusted it — they weren’t taking a black box’s word, they were getting a fast pointer to the authoritative clause. After the pilot, MeridianMutual promoted the stack to production, added their HR policy corpus as a second Knowledge Base behind the same front-end, and folded the re-ingestion job into their nightly document-management export so the assistant is never more than a day behind the source of truth.

When to use it

Use this architecture when you have a substantial, changing corpus of authoritative documents, you need answers grounded in and traceable to those documents, the data is sensitive enough that it must stay inside your account boundary, and you want AWS to operate the retrieval pipeline rather than building and running your own. It is the right default for internal knowledge assistants, customer-support copilots over a product knowledge base, policy/contract Q&A, and runbook/operations assistants — the broad middle of enterprise GenAI demand.

The trade-offs are real. Managed convenience costs some control: Bedrock Knowledge Bases makes opinionated choices about retrieval and prompt assembly, and if you need exotic retrieval (custom re-ranking pipelines, graph-augmented retrieval, multi-vector ColBERT-style late interaction) you may outgrow the managed path and drop to Retrieve-only or a fully custom stack on OpenSearch. OpenSearch Serverless has a non-trivial OCU floor cost even when idle — for a tiny corpus with sporadic traffic, that fixed cost can feel disproportionate, and the newer Amazon S3 Vectors option (vectors stored directly in S3, pay-per-use, no always-on cluster) is worth evaluating as a lower-floor backend for cost-sensitive or bursty workloads, accepting higher query latency in exchange. And RAG is not magic: answer quality is gated by retrieval quality, which is gated by chunking and corpus hygiene — garbage or contradictory documents in S3 produce confidently-cited wrong answers.

Anti-patterns to avoid. Don’t fine-tune when you mean to retrieve — if the goal is factual accuracy over a changing corpus, RAG beats fine-tuning on cost, freshness, and auditability every time (fine-tuning is for style and format, not facts). Don’t skip Guardrails and contextual grounding to “ship faster” — an ungrounded enterprise assistant that hallucinates a coverage decision is a liability, not an MVP. Don’t grant users direct Bedrock permissions and rely on prompt instructions for access control — enforce document-level security at the retrieval metadata filter, where it can’t be prompt-injected away. Don’t treat the vector index as precious state to back up — it’s derived data; back up S3 and rebuild the index. And don’t put the corpus on a public model endpoint to save engineering time — the entire point of this architecture is that you don’t have to.

Alternatives worth knowing. For agentic workflows that go beyond Q&A (the assistant needs to take actions — look up a claim status via an API, file a ticket), layer Amazon Bedrock Agents on top of this same Knowledge Base, using RAG as one tool among several. If you’ve standardized on a specific open-source vector database (Pinecone, Aurora PostgreSQL with pgvector, Redis, MongoDB Atlas), Bedrock Knowledge Bases supports several of those as the vector store instead of OpenSearch Serverless, so you can keep this exact architecture and swap the backend. And if you need the absolute lowest fixed cost and can tolerate more latency, evaluate Amazon S3 Vectors as the Knowledge Base backend. But for the mainstream enterprise requirement — grounded, private, auditable Q&A over your own documents, operated by AWS, scaling from pilot to production without a rewrite — Bedrock Knowledge Bases with OpenSearch Serverless and Guardrails is the reference pattern to reach for first.

Going deeper

This is the layer that separates someone who can follow a RAG reference architecture from someone who can make the calls on a real one. Generative-AI on AWS moves fast; treat concrete model IDs, preview/GA status, and quotas as things to re-verify in the console, and treat the patterns below as durable.

The Converse API is the surface you should build on

Older Bedrock code called InvokeModel with a model-specific JSON body — Claude wanted an anthropic_version and a messages array, Titan wanted inputText, Llama wanted a raw prompt. The Converse API (Converse / ConverseStream) replaced that zoo with one message shape that works across every chat model: a list of messages with role and content blocks, an optional system prompt, inferenceConfig, toolConfig for function-calling / tool use, and guardrailConfig. You switch models by changing the modelId string — no body rewrite. RetrieveAndGenerate uses Converse under the hood, so you inherit this portability for free; you only touch Converse directly when you run Retrieve separately and generate the answer yourself. Build new code on Converse, not InvokeModel.

Inference profiles and cross-region inference — often mandatory now

A subtle change trips up new projects: many current models cannot be called by their bare on-demand model ID at all — you must call a cross-region inference profile instead, an ID prefixed by a geography (us., eu., apac.), for example us.anthropic.claude-3-5-sonnet-20241022-v2:0. A cross-region inference profile lets Bedrock route each request to whichever Region within that geography has capacity, which raises your effective throughput ceiling and smooths spikes — while the data stays inside the geography (a us. profile stays in US Regions). There are two flavours: system-defined profiles (the us./eu./apac. ones AWS publishes) and application inference profiles you create yourself to attach cost-allocation tags and track spend per team or tenant. In the KB/Lambda code above, modelArn should be an inference-profile ARN, not a bare foundation-model ARN, for any model that requires it. The classic first-deploy error — ValidationException: Invocation of model ID … with on-demand throughput isn't supported — almost always means “use the inference profile.”

On-demand vs. provisioned throughput

On-demand is the default: pay per input/output token, no commitment, but subject to per-model requests-per-minute (RPM) and tokens-per-minute (TPM) service quotas. Provisioned Throughput reserves dedicated model units for a model at a fixed hourly price (with a 1-month or 6-month commitment), giving guaranteed throughput independent of the on-demand quotas — and it is required to serve a customised (fine-tuned) or imported model. Rule of thumb: start on-demand, request quota increases proactively as you grow, and buy provisioned throughput only when you have steady, predictable, high-volume traffic that bumps the on-demand limits or needs latency guarantees, or when you serve a custom model. For bursty bulk jobs (re-summarise the whole corpus overnight), batch inference runs asynchronously at roughly half the on-demand token price.

Choosing a vector store

OpenSearch Serverless is the backend the Knowledge Base wizard nudges you toward, but it is one of several supported stores, and the right pick turns on floor cost, scale, latency, and whether you need graph or keyword strengths:

Two upgrades sit on top of whichever store you pick. Reranking (Amazon Rerank 1.0 or Cohere Rerank 3.5) takes the top ~25 retrieved chunks and re-scores them with a heavier cross-encoder, so the 5 you actually feed the model are the most relevant rather than merely the nearest — a cheap, high-yield quality lever for messy corpora, configured right in the Knowledge Base retrieval settings. GraphRAG (Bedrock Knowledge Bases backed by Neptune Analytics) builds a knowledge graph from your documents so the retriever can traverse relationships — “which endorsements modify form HO-3, and which of those apply in Texas?” — that flat vector similarity cannot express.

Measuring RAG quality (or you are flying blind)

“It seems to work” is not an evaluation. RAG has two failure surfaces — retrieval and generation — and you must measure each. The vocabulary comes from frameworks like RAGAS:

• Context recall / precision — did retrieval fetch the passages that actually contain the answer (recall), without drowning them in irrelevant ones (precision)? Bad retrieval caps everything downstream.
• Faithfulness (groundedness) — is every claim in the answer supported by the retrieved context? This is the anti-hallucination metric, and the Guardrail’s real-time grounding score is its operational cousin.
• Answer relevancy — does the answer actually address the question that was asked?

Bedrock Knowledge Base Evaluation runs exactly these as a managed LLM-as-a-judge job: point it at a set of question / ground-truth pairs and it scores retrieval and generation quality (plus responsible-AI metrics), and can even compare two configurations head-to-head — so “hierarchical vs. fixed chunking” or “top-5 vs. top-10” becomes a measured decision, not a hunch. Build a golden set of 100–300 real questions with known-good answers before launch and re-run it on every config change. It is the single practice that separates teams whose RAG quality compounds from teams who ship once and pray.

From answers to actions: Agents and AgentCore

RAG answers questions; agents do things. Amazon Bedrock Agents orchestrate a model with your Knowledge Base plus action groups (Lambda functions or OpenAPI schemas the model may call), so the assistant can look up a live claim status, file a ticket, or check inventory — using RAG as one tool among several, with the model deciding when to retrieve versus act. The newer Amazon Bedrock AgentCore (preview — verify current status) goes further: a set of managed building blocks — Runtime, Memory, Identity, Gateway (turn existing APIs and Lambdas into agent tools), Browser, Code Interpreter, and Observability — for running agents at production scale, and notably framework-agnostic (it hosts open-source agent frameworks such as Strands, LangGraph, or CrewAI, not only Bedrock Agents). For a pure Q&A assistant you need none of this; reach for it only when the assistant must take actions or run multi-step workflows.

The cost model, with real arithmetic

Four meters, and only one of them scales with traffic:

• Embeddings — paid once per chunk at ingest, re-run only on changed documents. 70,000 chunks × ~200 tokens ≈ 14M tokens: a one-time cost of a few dollars at Titan v2 prices. Negligible unless you re-index the whole corpus nightly (don’t — use incremental sync).
• Vector store — the OpenSearch Serverless OCU floor is your dominant fixed cost, billed whether or not anyone asks a question. Lower it with the dev / no-redundancy option in non-prod, or evaluate S3 Vectors for a near-zero floor.
• Generation tokens — the variable cost that dominates at scale. Worked example: 25,000 questions/day, each ~4,000 input tokens (5 retrieved chunks + prompt) and ~400 output tokens → 100M input + 10M output tokens/day. At an illustrative mid-tier rate of $3 per 1M input and $15 per 1M output tokens, that is $300 + $150 = ~$450/day before any optimisation.
• Lambda / API Gateway — rounding error next to the above.

Now apply the levers and watch that $450 fall: route ~70% of traffic to a model several times cheaper (tier routing) and the blended rate drops sharply; a semantic cache absorbing ~18% of questions removes ~18% of generations outright; capping maxTokens and retrieving top-5 instead of top-20 shrinks the tokens per call; prompt caching reuses a large stable instruction prefix across calls so you stop re-paying for it. Stack these and the same workload routinely lands well under half the naïve figure. The architect’s lesson: model spend is a design variable, and the Lambda orchestrator is where you spend it wisely.

Latency, privacy, and the escape hatch

Latency. Most of a RAG response’s wall-clock is generation, not retrieval (which is typically tens of milliseconds against a warm OpenSearch collection). Stream the answer (RetrieveAndGenerateStream / ConverseStream) so the user sees tokens immediately, use a smaller model for simple queries, and for the tightest budgets evaluate Bedrock’s latency-optimised inference for select models on accelerated hardware.

Privacy, stated plainly. Amazon Bedrock does not use your prompts or completions to train the base models, and does not share them with the model providers; your data stays in the Region you call. That single fact — combined with PrivateLink so nothing egresses to the internet (see the PrivateLink deep dive) — is what lets a bank’s or hospital’s security team sign off, and it is the crisp answer to “but doesn’t sending our documents to an AI model leak them?”

Prompt management. As prompts multiply, stop pasting them inline: Bedrock Prompt Management stores versioned prompt templates and Prompt Flows chains steps visually. Version prompts like any other production artifact.

When Bedrock isn’t enough. Bedrock is a model catalogue plus managed RAG and agents. If you need to train a model, fine-tune deeply, host an architecture Bedrock does not offer, or run at custom-hardware economics, drop to Amazon SageMaker AI (JumpStart for open foundation models, full training and hosting) — or import a custom weights checkpoint into Bedrock via Custom Model Import to keep the Bedrock API surface while serving your own model. Most enterprises never need this for RAG; reach for it only when the managed catalogue genuinely cannot fit the requirement.

For an even deeper operational treatment of the managed RAG stack — advanced chunking, reranking, and guardrail tuning in production — see Bedrock production RAG: Knowledge Bases & Guardrails.

Practice challenges

Work these in order — they escalate from “explain the idea” to “make the architectural call.” Try each before opening the solution.

1. (Beginner) Why RAG, not fine-tuning? A stakeholder asks why you don’t just fine-tune a model on the 9,000 policy documents. Give two concrete reasons RAG wins for this corpus.

<details><summary>Solution</summary>

(1) Freshness — when a policy changes you re-index the one changed PDF and the answer updates within minutes; a fine-tuned model would have to be retrained. (2) Auditability / grounding — RAG returns citations to the source clause, so a compliance officer can verify why the answer was given; fine-tuning bakes facts into opaque weights and still hallucinates. (Fine-tuning teaches style and format, not changing facts.)

Why: RAG decouples knowledge from the model, so facts stay fresh and traceable. </details>

2. (Beginner) Pick the search type. Users frequently query by exact form numbers like HO-3 and HO-5. Pure semantic retrieval sometimes returns the wrong form. Which retrieval search type fixes this, and how do you set it?

<details><summary>Solution</summary>

Use hybrid search — set overrideSearchType = "HYBRID" in the vectorSearchConfiguration (or the equivalent in the KB console). Hybrid blends semantic k-NN scores with BM25 keyword scores, so an exact token like HO-3 is matched literally instead of drifting to a semantic near-neighbour.

Why: identifiers demand exact-match; hybrid adds keyword precision on top of semantic recall. </details>

3. (Intermediate) Multi-group metadata filter. A compliance analyst belongs to groups ["claims", "compliance"] and should retrieve documents tagged with either business_unit. Write the filter.

<details><summary>Solution</summary>

{ "in": { "key": "business_unit", "value": ["claims", "compliance"] } }

The in operator takes a list and matches any of the values. (For richer logic, andAll / orAll combine multiple filter clauses — e.g. business_unit in [...] AND state = "TX".)

Why: in expresses set membership in one clause; equals would only match a single value. </details>

4. (Intermediate) The ingestion job fails with an OpenSearch permissions error. The KB role exists and the collection exists, but every ingestion job dies with an access error on the index. Name the two distinct things that must both be present.

<details><summary>Solution</summary>

(1) The IAM permission aoss:APIAccessAll on the collection ARN, attached to the KB role’s identity policy. (2) An OpenSearch Serverless data-access policy of type data that names the KB role as a Principal and grants index permissions (aoss:CreateIndex, aoss:WriteDocument, …). Both gates are required — IAM alone or the data-access policy alone still fails.

Why: OpenSearch Serverless authorises data-plane calls through IAM and its own data-access policy; you must satisfy both. </details>

5. (Advanced) Guarantee a refusal. Regulators require that when the corpus lacks an answer, the assistant must not improvise — it must decline and tell the user to escalate. Which Guardrail policy and setting delivers this, and what does the user see?

<details><summary>Solution</summary>

Enable the contextual grounding policy with a GROUNDING filter threshold (e.g. 0.75) and typically a RELEVANCE threshold too. When the model’s draft answer scores below the grounding threshold (because retrieval returned nothing that supports it), the Guardrail blocks the output and returns your blocked_outputs_messaging, e.g. “I couldn’t find that in our current documents — please escalate to a senior underwriter.” Pin the Lambda to a published guardrail version, not DRAFT.

Why: contextual grounding scores answer-vs-context support in real time and hard-fails ungrounded answers — turning “usually grounded” into “provably refuses.” </details>

6. (Advanced) Cut the generation bill. Your projected generation spend is ~$450/day at 25,000 questions/day. Propose three levers, and state which cost meter each one moves.

<details><summary>Solution</summary>

(1) Tier routing — send simple lookups to a cheaper/faster model (e.g. Nova Lite or Claude Haiku) and reserve the premium model for synthesis: lowers the blended per-token rate. (2) Semantic cache — embed each question, and if a recent answered question is within a similarity threshold, serve the cached answer: removes whole generation calls. (3) Cap maxTokens + reduce numberOfResults (top-5 not top-20) and add prompt caching for the stable instruction prefix: shrinks the input + output tokens per call. Stacked, these routinely halve spend.

Why: generation cost = calls × tokens/call × rate/token; each lever attacks a different factor of that product. </details>

Common beginner mistakes

These are misconceptions about how RAG works, distinct from operational symptoms — each pairs the wrong mental model with the right one.

• “RAG means I fine-tune the model on my documents.” No — RAG never changes the model’s weights. It retrieves relevant passages at query time and puts them in the prompt. Fine-tuning is a separate, expensive technique for teaching style and format, and it still hallucinates facts. If the goal is accurate, current, auditable answers over a changing corpus, RAG is the tool.

• “I’ll just swap in a better embeddings model next month.” You can’t do it casually. Query vectors and stored chunk vectors must come from the same model and dimension, or the cosine comparison is meaningless. Changing the embeddings model forces a full re-index of the entire corpus. Choose deliberately up front.

• “More retrieved chunks = better answers, so retrieve top-50.” Usually the opposite. Beyond a handful, extra chunks add noise that dilutes the relevant context and inflate token cost and latency. Fix relevance with better chunking, hybrid search, and reranking — not brute quantity. Top-5 with a reranker beats top-50 raw almost every time.

• “The Guardrail is optional polish I’ll add later.” The Guardrail’s contextual grounding check is the mechanism that makes the assistant refuse to answer when retrieval found nothing supporting — the difference between “usually doesn’t hallucinate” and “provably won’t.” In a regulated setting that is a launch requirement, not a nice-to-have.

• “I’ll give users Bedrock access and tell the prompt not to reveal other teams’ data.” Prompt instructions are injectable — a user can craft input that talks the model out of them. Enforce document-level security where it can’t be argued away: at the retrieval metadata filter, driven by the caller’s verified identity. Users hold a JWT, never Bedrock permissions; the Lambda role calls Bedrock.

• “I must carefully back up the vector index.” The index is derived data. Your true source of truth is the corpus in S3 (versioned, eleven-nines durable). If you lose the collection, you rebuild it by re-running ingestion — so your DR plan for the vector store is “re-ingest,” and your RTO is the re-ingest time, not a database restore.

• “I’ll call the model by its plain ID like anthropic.claude-….” Many current models require a cross-region inference profile (us. / eu. / apac. prefix) and reject the bare on-demand ID with on-demand throughput isn't supported. Use the inference-profile ARN in modelArn.

• “RAG can’t hallucinate — it’s grounded.” It can, if retrieval returns irrelevant chunks (empty corpus, wrong filter, poor chunking): the model will confidently synthesise a wrong, well-cited answer from bad context. Grounding guardrails plus corpus hygiene — no stale or contradictory documents — are what actually close this gap.

Glossary

• RAG (Retrieval-Augmented Generation) — answer questions by retrieving relevant passages from your own data and putting them in the model’s prompt, instead of relying on the model’s trained-in memory.
• Foundation model (FM) / LLM — a large, general-purpose model (Claude, Amazon Nova, Titan, Llama…) that generates text; the “student” that reads the retrieved context and writes the answer.
• Token — the unit models read and bill in — roughly ¾ of a word. Cost and quotas are measured in input + output tokens.
• Amazon Bedrock — AWS’s managed, single-API service fronting many foundation models, plus Knowledge Bases, Guardrails, Agents, and evaluation.
• Embedding — a vector (list of numbers) representing a piece of text’s meaning, such that similar meanings get similar vectors.
• Vector / vector store — an embedding, and the database (OpenSearch Serverless, pgvector, S3 Vectors…) that indexes embeddings for similarity search.
• Cosine similarity — the angle-based measure (1.0 = same meaning, 0.0 = unrelated) used to rank how close two vectors are.
• k-NN / ANN — k-Nearest-Neighbour search finds the closest vectors to the query; approximate NN (ANN) trades a little accuracy for a huge speed-up.
• HNSW — Hierarchical Navigable Small World, the graph index OpenSearch uses for fast ANN search; tuned by m and ef parameters.
• Chunk / chunking — splitting documents into passages before embedding. Strategies: fixed-size (with overlap), none (one chunk per file), hierarchical (small child for retrieval, large parent for context), semantic (split on meaning).
• Overlap — shared tokens between adjacent chunks so a sentence at a boundary isn’t orphaned.
• Top-K (numberOfResults) — how many nearest chunks retrieval returns to feed the model.
• Hybrid search — combine semantic (k-NN) with keyword (BM25) scoring so exact tokens (form numbers, codes) match reliably.
• Reranking — re-scoring the top retrieved chunks with a heavier cross-encoder (Amazon Rerank, Cohere Rerank) so the best few rise to the top.
• Bedrock Knowledge Base — the managed pipeline that parses, chunks, embeds, indexes, and serves Retrieve / RetrieveAndGenerate; RAG without custom pipeline code.
• Ingestion job — the async run that pulls new/changed S3 objects into the vector index (manual, scheduled, or event-driven).
• RetrieveAndGenerate — the one Bedrock call that retrieves, assembles the prompt, generates, and returns the answer plus citations.
• Converse API — Bedrock’s unified message API (Converse / ConverseStream) that works across models with one request shape, plus tool use and guardrails.
• Guardrail — a Bedrock policy layer for content filters, denied topics, PII handling, prompt-attack filtering, and contextual grounding.
• Contextual grounding — a Guardrail check that scores whether the answer is supported by the retrieved context (grounding) and relevant to the question (relevance); below threshold, the answer is blocked.
• Hallucination — a fluent, confident, but unsupported (often wrong) model answer; the core failure RAG + grounding exist to prevent.
• Citation — the source reference (S3 URI, chunk text, metadata) returned with an answer so a human can verify the claim.
• Metadata filter — a retrieval constraint (equals, in, andAll…) that limits which chunks can be returned — the mechanism for document-level security.
• PII — personally identifiable information (SSN, phone, email); Guardrails can block or anonymise it in prompts and responses.
• Inference profile / cross-region inference — a us./eu./apac.-prefixed model ID that routes requests across Regions in a geography for throughput; required to call many newer models.
• On-demand vs. provisioned throughput — pay-per-token with quotas vs. reserved model units at a fixed hourly price (required for custom models).
• OCU (OpenSearch Compute Unit) — the billing/capacity unit of OpenSearch Serverless; its always-on floor is the main fixed cost of that backend.
• Prompt caching / batch inference — reuse a stable prompt prefix across calls to cut input cost; run bulk jobs asynchronously at ~half price.
• GraphRAG — RAG over a knowledge graph (Bedrock + Neptune Analytics) for multi-hop, relationship-based questions.
• Bedrock Agents / AgentCore — Agents orchestrate a model with Knowledge Bases and action groups (tools) to take actions; AgentCore (preview) is the framework-agnostic runtime and building blocks for production agents.
• RAGAS — an open-source RAG evaluation framework (context recall/precision, faithfulness, answer relevancy); Bedrock Knowledge Base Evaluation runs equivalents as a managed LLM-as-a-judge job.
• PrivateLink / VPC interface endpoint — private, in-VPC connectivity to Bedrock/OpenSearch/S3 so traffic never traverses the public internet.
• KMS CMK — a customer-managed KMS key encrypting S3, the vector collection, and Bedrock session data, with a key policy scoped to the KB/collection roles.
• Amazon SageMaker AI — AWS’s platform for training, fine-tuning, and hosting custom models when the Bedrock catalogue isn’t enough (or use Custom Model Import to serve your weights through Bedrock).