Environment Spinup

Everything needed to bring up rag-sample in an AWS account for the first time: Phase 1 (once per account) prepares the account and the CI deploy identity; Phase 2 (once per environment) creates QA or Prod. Deployment model: RAG001 · Environments and CI.

Current deployment status for every item below (what’s automated vs. manual, what’s still a known gap) is tracked in the canonical deployment plan.

Phase 1 — Account readiness (once per AWS account)

Prerequisites

• AWS account with billing enabled; AWS CLI v2; Terraform 1.11+
• IAM user or role with full IAM and S3 permissions (temporary — bootstrap creates least-privilege deploy roles that replace the need for this)
• GitHub repository access to ArunskiOrg/rag-sample-app

1. Enable required AWS services and verify quotas

Verify Lambda, Aurora, API Gateway, CloudFront, S3, Secrets Manager, CloudWatch, X-Ray, Bedrock, and IAM aren’t disabled by a Service Control Policy. Bedrock enables serverless foundation models automatically, but Anthropic models (Claude) require a one-time usage form via the Bedrock console before first use (RAG007).

Raise these via the Service Quotas console if restricted:

Service	Quota name	Default	Need
RDS (Aurora)	DB clusters	40	2 (one per env)
RDS (Aurora)	Data API maximum concurrent cluster-secret pairs	30	≥ 2 (RAG011)
Lambda	Concurrent executions	1000	1000
VPC	VPCs per Region	5	2 (one per env)

2. Configure local AWS credentials

export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY>
export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_KEY>
export AWS_REGION=<YOUR_REGION>
aws sts get-caller-identity   # verify

3. Set account-specific values in terraform.tfvars

Each Terraform root reads account-specific inputs from a gitignored terraform.tfvars, never a shared .env:

git clone https://github.com/ArunskiOrg/rag-sample-app.git
cd rag-sample-app

# in each of infra/bootstrap/, infra/roots/qa/, infra/roots/prod/:
cp terraform.tfvars.example terraform.tfvars
# set aws_account_id (all three); in roots/qa and roots/prod also set
# lambda_package_s3_bucket / lambda_package_s3_key

Terraform loads terraform.tfvars automatically — no source step needed. If a TLS-intercepting antivirus/proxy breaks Terraform locally, see infra/README.md’s cert-trust section for the TF_SSL_CERT_FILE workaround.

4. Apply the bootstrap root

bootstrap creates the Terraform state bucket, the Lambda-package bucket, the GitHub Actions OIDC identity provider, and the per-environment deploy roles (rag-sample-qa-deploy-role, rag-sample-prod-deploy-role — least-privilege, scoped by the cicd module). Its own state has to live somewhere before the bucket it creates exists, so this very first apply in a new account needs -backend=false (the make tf-bootstrap-* targets always target the S3 backend, so they don’t fit this one special case — use raw terraform here, from inside infra/bootstrap):

cd infra/bootstrap
terraform init -backend=false        # first run only: bucket doesn't exist yet
terraform plan                       # review before applying
terraform apply

Then migrate local state into the remote backend:

terraform init -migrate-state

If migration doesn’t happen automatically (e.g. running non-interactively with -input=false), push state explicitly — safe here only because the remote state is empty on a freshly created bucket:

terraform state push terraform.tfstate

Confirm with terraform state list (expect a non-empty list) and aws s3 ls s3://rag-sample-terraform-state/, then delete the local terraform.tfstate/.backup files.

Caution

Never run bootstrap with only local state past this initial migration. A lost or stale local state file can’t see resources that already exist in AWS, and the next apply fails trying to recreate them (recoverable only via manual terraform import). See #205.

For any later change to infra/bootstrap or the cicd module, re-run make tf-bootstrap-apply manually — no workflow applies bootstrap automatically, ever. This has bitten real changes before (see the deployment plan’s operational note).

5. Wire the deploy identity into GitHub Actions

Read the outputs from the apply above:

terraform output qa_deploy_role_arn
terraform output prod_deploy_role_arn

Set, on the ArunskiOrg/rag-sample-app repo:

Config item	Type	Value
AWS_QA_DEPLOY_ROLE_ARN	Repo secret	qa_deploy_role_arn output
AWS_QA_ACCOUNT_ID	Repo variable	the account ID from step 2 — supplies TF_VAR_aws_account_id for the allowed_account_ids guard, kept out of git
AWS_PROD_DEPLOY_ROLE_ARN	Repo secret	prod_deploy_role_arn output (once a Prod pipeline exists — see the deployment plan)

Each role’s trust policy only admits a GitHub Actions job from this repo running in the matching environment:<env> — the deploy workflow jobs declare environment: qa (or prod) so their OIDC token subject matches. No long-lived AWS keys are used anywhere in CI.

6. Set up billing alarms

RAG016 decides Terraform-provisioned budgets plus a Budgets Action that denies Bedrock on breach. That is not implemented (no aws_budgets_budget resource in rag-sample-app, verified against main, 2026-08-06); it is carried by Story #33. Until it lands, budgets are created by hand here, as an interim measure rather than the intended end state.

Both budgets are live: aws budgets describe-budgets returns rag-sample-qa and rag-sample-prod, each a $10.00 USD MONTHLY cost budget, and aws budgets describe-notifications-for-budget confirms rag-sample-qa has the FORECASTED > 80% notification shown below (verified 2026-08-06; Prod’s notification block was not checked). The commands below reproduce that deployed state.

The budgets these commands create are inert

Both filter on the cost-allocation tag user:Environment$<env>, and no cost allocation tag is active on this account. aws ce list-cost-allocation-tags --status Active returns an empty list, and a July 2026 aws ce get-cost-and-usage grouped by TAG:Environment returns every service under the empty key Environment$ (verified 2026-08-11). The filter matches no spend, both budgets read $0 regardless of actual usage, and the FORECASTED > 80% notification cannot trip at any threshold. Do not treat step 6 as delivering cost detection today.

Fixing it starts with activating the Environment cost allocation tag in Billing → Cost allocation tags; a tag is only usable in a budget filter after activation, and activation is not retroactive. That is a precondition, not the whole fix: whether on-demand Bedrock InvokeModel usage then carries a resolvable Environment value is unverified, and must be confirmed before a tag-filtered budget is relied on for Bedrock, the largest variable cost. Bedrock spend does appear in Cost Explorer under the service dimension, so a service-filtered budget is an alternative shape. #295 under Story #33 carries the work; the filter shape is not yet decided.

The $10 amounts are a second-order problem behind that one. They predate the Bedrock re-pricing in cost.md, which puts a single environment at ~$8–13/month at 1k queries and ~$47–60 at 10k, so a $10 limit needs recalibrating too — but only once the filter matches spend at all. The commands below are reproduced as the deployed state, not as a recommended configuration.

aws budgets create-budget \
  --account-id <ACCOUNT_ID> \
  --budget '{
    "BudgetName": "rag-sample-qa",
    "BudgetType": "COST",
    "TimeUnit": "MONTHLY",
    "BudgetLimit": {"Amount": "10", "Unit": "USD"},
    "CostFilters": {"TagKeyValue": ["user:Environment$qa"]}
  }' \
  --notifications-with-subscribers '[{
    "Notification": {"NotificationType":"FORECASTED","ComparisonOperator":"GREATER_THAN","Threshold":80},
    "Subscribers": [{"SubscriptionType":"EMAIL","Address":"<YOUR_EMAIL>"}]
  }]'

Repeat for rag-sample-prod with the same "Amount": "10" and user:Environment$prod.

7. Verify account readiness

aws sts get-caller-identity
aws s3 ls s3://rag-sample-terraform-state/
aws bedrock list-inference-profiles \
  --query 'inferenceProfileSummaries[?contains(inferenceProfileId,`haiku`)].inferenceProfileId' --output text

The generation model is invoked through a cross-region inference profile, so use list-inference-profiles, requiring bedrock:ListInferenceProfiles.

list-foundation-models does not answer the same question. Its inferenceTypesSupported field has only two valid values, ON_DEMAND and PROVISIONED (FoundationModelSummary) — there is no INFERENCE_PROFILE value. A profile-only model’s underlying foundation model still appears in that listing, but without ON_DEMAND, so --by-inference-type ON_DEMAND excludes it and a bare presence check tells you nothing about whether the deployed configuration can invoke it.

Also confirm the profile’s routed regions still match the hardcoded list in rag-sample-app’s infra/modules/api/main.tf (for region in ["us-east-1", "us-east-2", "us-west-2"] in the Bedrock Resource block of aws_iam_role_policy.lambda):

aws bedrock get-inference-profile \
  --inference-profile-identifier us.anthropic.claude-haiku-4-5-20251001-v1:0 \
  --query 'models[].modelArn' --output text

This call needs bedrock:GetInferenceProfile. Each returned ARN names the region it routes to. A region not in the hardcoded list is not granted by the IAM policy, and shows up only as an intermittent runtime AccessDenied on the requests Bedrock happens to route there.

A successful listing is necessary, not sufficient. It proves the profile exists in the region; it does not prove the account has model access granted (the one-time Anthropic usage form, step 1 above), nor that the API Lambda role grants both ARN forms. The end-to-end check is a real invocation under the deployed role — the post-deploy smoke test covers this. If that returns AccessDeniedException despite the profile listing, look at model access and the IAM resource ARNs in that order.

Bootstrap uses S3-native state locking (use_lockfile), not a DynamoDB table — an empty s3 ls on a fresh bucket is expected; the .tflock object only exists transiently during an operation.

---

Phase 2 — First deploy to an environment (once per environment)

Prerequisite: Phase 1 complete for this account.

The biggest change from earlier versions of this guide: you no longer run terraform apply yourself for QA. Pushing to main after Phase 1 is done triggers qa-deploy.yml, which builds the Lambda package, plans, and applies the entire QA stack automatically — end to end, confirmed working as of run 29606689902.

1. Push to main

Once AWS_QA_DEPLOY_ROLE_ARN and AWS_QA_ACCOUNT_ID are set (Phase 1, step 5), any merge to main runs the full pipeline: lint → IaC security scan → test → secret scan → build & publish the Lambda package → Infracost diff → Terraform plan → Terraform apply → deploy Lambda code → build & deploy the frontend → post-deploy smoke check. Watch it in the Actions tab. Infracost runs twice today: terraform.yml posts the pre-merge diff on the Terraform PR, pricing both roots, and qa-deploy.yml re-runs it after merge against infra/roots/qa, updating the comment on the merged PR.

The post-merge run is being removed — #302

RAG016’s 2026-08-21 amendment moves cost checking entirely to the Terraform PR, where the comment is posted and the fixed-price ceiling is asserted. Once #302 lands, the infracost job leaves qa-deploy.yml and this pipeline runs one job fewer.

Expect creation of: VPC + subnets + security group (network module), Aurora Serverless v2 cluster + instance (data module), Secrets Manager secrets, Lambda functions (rag-sample-qa-api, rag-sample-qa-authorizer), API Gateway HTTP API (POST /query authorizer-protected, ANY /{proxy+} catch-all serving /healthz), CloudFront distribution + S3 bucket (frontend module; the Build & Deploy Frontend (QA) job populates the bucket from the built SPA after the apply and invalidates CloudFront — QA only, Prod has no equivalent, see the deployment plan).

Note the Terraform outputs from the apply job’s logs — api_endpoint, cloudfront_domain, s3_bucket_name — needed below.

Caution

Do not push infra changes toward Prod until the QA smoke-test sequence below passes and the Prod promotion checklist (#86) is complete. Prod has no automated pipeline yet (see the deployment plan) — any Prod apply today would be a manual, deliberate operator action, not something this guide walks through.

2. Run the schema migration

Aurora QA is in a private VPC with no internet gateway or NAT (RAG010); there is no direct TCP path from local to the cluster, and no CI job runs migrations. rag-migrate (the local CLI, connects via psycopg) cannot reach it. Apply the schema over the RDS Data API with rag-migrate-dataapi, which replays the same .sql files (RAG011, story #227). Run from the rag-sample-app checkout:

cd backend

# Data-API descriptor from the QA root outputs
export DB_CLUSTER_ARN=$(terraform -chdir=../infra/roots/qa output -raw db_cluster_arn)
export DB_SECRET_ARN=$(terraform -chdir=../infra/roots/qa output -raw db_secret_arn)
export DB_NAME=$(terraform -chdir=../infra/roots/qa output -raw db_name)

uv run rag-migrate-dataapi
# Expected: Applied migrations: 0001_init, 0002_api_keys

The runner shares the schema_migrations ledger with rag-migrate, so re-running is a no-op; a cluster resuming from a 0-ACU floor may reject the first call and need one re-run. Full reference: the app repo’s docs/runbooks/schema-migrations.md.

Verify pgvector is installed:

aws rds-data execute-statement \
  --resource-arn "$DB_CLUSTER_ARN" --secret-arn "$DB_SECRET_ARN" --database "$DB_NAME" \
  --sql "SELECT extname FROM pg_extension WHERE extname = 'vector';" \
  --query 'records[0][0].stringValue' --output text
# Expected: vector

3. Corpus ingestion and API key issuance (known limitation)

Caution

rag-ingest and rag-key both connect via psycopg over DATABASE_URL and hit the same private-VPC limitation as rag-migrate — no path from a local workstation into QA Aurora. Corpus ingestion and key issuance against QA require compute inside the VPC (ECS task, Lambda invocation, or AWS Batch job), which isn’t built yet — tracked as FUTURE in the deployment plan. Skip this step; the API returns empty results and the authorized /query path can’t be smoke-tested until it exists.

Once available, verify chunk count: aws rds-data execute-statement --resource-arn "$CLUSTER_ARN" --secret-arn "$SECRET_ARN" --database "$DATABASE_NAME" --sql "SELECT COUNT(*) FROM chunks WHERE embedding IS NOT NULL;" --query 'records[0][0].longValue' --output text (expect > 0). See the ingestion pipeline runbook for CLI reference.

4. Smoke test the API

API_ENDPOINT=<api_endpoint from the apply output>

# Health check — no auth required (served by the ANY /{proxy+} catch-all)
curl -sf "$API_ENDPOINT/healthz" | jq .
# Expected: {"status": "ok"}

# Query without auth — must return 401
curl -s -o /dev/null -w "%{http_code}" "$API_ENDPOINT/query" \
  -X POST -H "Content-Type: application/json" -d '{"question": "test"}'
# Expected: 401

The authorized /query path needs a valid bearer token from step 3 — test it once key issuance against QA is available.

5. Verify observability and auth-header scrubbing

# Authorizer log group retention (TF-managed, 30-day)
aws logs describe-log-groups \
  --log-group-name-prefix /aws/lambda/rag-sample-qa-authorizer \
  --query 'logGroups[0].retentionInDays' --output text
# Expected: 30

# Auth header must never appear in logs
LOG_GROUP=/aws/lambda/rag-sample-qa-api
STREAM=$(aws logs describe-log-streams --log-group-name "$LOG_GROUP" \
  --order-by LastEventTime --descending --max-items 1 \
  --query 'logStreams[0].logStreamName' --output text)
aws logs get-log-events --log-group-name "$LOG_GROUP" --log-stream-name "$STREAM" \
  --query 'events[*].message' --output text | grep -i "authorization"
# Expected: no output

If Authorization appears in any log line, stop — do not proceed to Prod until #62 is verified.

Open the X-Ray console → Service Map, filter by the rag-sample tag. After the smoke test you should see API Gateway → Lambda (api) → Bedrock, and API Gateway → Lambda (authorizer) → RDS Data API.

6. Verify cost alarms

aws budgets describe-budgets --account-id <ACCOUNT_ID> \
  --query 'Budgets[].{Name:BudgetName,Limit:BudgetLimit.Amount,Unit:BudgetLimit.Unit,Period:TimeUnit}' \
  --output table

Expected: rag-sample-qa and rag-sample-prod, both 10 USD MONTHLY — the deployed state as of 2026-08-06. Add the check that actually matters:

aws ce list-cost-allocation-tags --status Active --query 'CostAllocationTags[].TagKey' --output text

An empty result means every tag-filtered budget on the account is inert. Tracked by #295.

Next steps

• QA is now ready for integration testing; CI auto-deploys on every future merge to main — see Ongoing Deployment.
• Prod has no automated pipeline yet. Review the Prod promotion checklist (#86) and the deployment plan before doing anything toward Prod.

Teardown

Aurora auto-pauses on inactivity (0-ACU min capacity). For full teardown:

cd infra/roots/qa
terraform destroy

Caution

QA has deletion_protection = false — terraform destroy drops the database and all data with no final snapshot. Intentional for cost control in QA; not acceptable in Prod (deletion_protection = true, and there’s no automated apply/destroy path for Prod regardless).

The Lambda zip in S3 is unmanaged by Terraform and not removed by destroy:

aws s3 rm s3://<lambda_package_s3_bucket>/<lambda_package_s3_key>