Environments and CI pipeline

Context

docs/devops/deployment-plan.md is the authoritative design outline for rag-sample’s environment topology, build artifacts, promotion model, pipeline stages, secrets strategy, state backend, and rollback story — the target end-state, not a claim that every piece is built yet. This document is its companion: it explains the reasoning behind the choices in that outline, the alternatives that were considered and rejected, and the failure modes the design must tolerate. Read the outline first; this document assumes it. For what’s actually deployed and automated today vs. still manual or not yet built, see the canonical deployment plan.

The relevant decisions are RAG001 (two separate VPCs, single account), RAG005 (OpenAPI governance), RAG006 (90%/70% coverage hard-fail), RAG010 (no IGW, no NAT), and RAG011 (RDS Data API only). Each shaped the pipeline in ways called out below.

Two environments, not three

The standard argument for three environments — dev, staging, prod — rests on separating integration risk from pre-prod stabilization. Both of those roles are already covered here without a third environment.

Local (Docker Compose) handles integration risk. Developers run the full stack locally against a real Postgres container. No AWS spend, no coordination overhead, immediate feedback. The local env is not in the pipeline because it is the developer’s workstation, but it eliminates the rationale for a dev environment in AWS.

QA handles pre-prod stabilization. It deploys the full AWS stack — Aurora Serverless v2, Lambda, API Gateway, CloudFront, Bedrock — on every merge to main. Structurally it is identical to Prod (Story #87 formalizes this parity). It exists to catch anything that only surfaces on the real AWS service graph: IAM permission boundaries, RDS Data API behavior, Bedrock Guardrails policy interactions, CloudFront cache header behavior.

Staging, the third environment in the three-env model, would add roughly $1–4/month idle and at minimum double the Terraform surface to maintain. Its purpose — a Prod-like environment for final pre-release validation — is satisfied by QA, which is already Prod-like by design. Adding staging makes sense when QA is used for continuous developer integration (low signal, noisy) and you need a calmer final gate. With one operator and auto-deploy on every main merge, QA is already the calm final gate.

The single-operator constraint also simplifies the math: any environment nobody is actively monitoring becomes operational toil with no benefit. Two environments is the ceiling that stays within manageable overhead.

Promotion model

Auto QA plus manual Prod was not the only option considered. The three realistic alternatives were: full automation (QA auto, Prod auto on QA green), full manual (both environments require explicit approval), and inverse-gate (Prod auto, QA manual as a branch cut).

Full automation was the most appealing on paper and the easiest to reject. rag-sample’s Prod environment is live to users. A Bedrock Guardrails misconfiguration, an unintended Aurora schema migration, or a CloudFront behavior regression can all pass the test suite and still degrade the user experience. For a personal demo the stakes are low, but the workflow is designed to practice production delivery patterns — and production delivery patterns include a human checkpoint before a live environment changes. Auto-to-Prod removes that checkpoint for no operational gain. The test suite and QA deploy together filter the mechanical failures; the Prod gate filters the judgment calls.

Full manual was the easiest to reject for the opposite reason. If QA also requires manual approval, then every merge to main involves two approval actions separated by a deploy cycle. The only thing manual approval of a QA deploy buys is preventing a broken QA state — which the test hard-fails already handle. The cost of that discipline is that every small change, including documentation-only changes that touch infrastructure via Terraform, blocks on a human action before reaching any environment.

Inverse gate (Prod auto, QA manual) was not seriously considered because it inverts the risk: changes reach Prod before the operator has seen them behave on the real AWS stack.

GitHub Environment protection rules were chosen over an external approval system (PagerDuty, Slack bot, email-triggered webhook) because the approval action is already in the workflow the operator is watching. The operator receives a GitHub notification when the gate opens, approves from the same GitHub UI where the pipeline status is visible, and the audit trail (who approved, when) lives in the same system as the code. An external system adds an integration to maintain, a second place to look, and a second point of failure, with no additional capability for a single operator.

Pipeline shape: fan-out vs. promote-by-artifact

Three shapes were considered for getting code from commit to both environments.

The first, promote-by-tag, deploys QA first, then promotes the same tag to Prod by re-running the deploy job against the existing QA artifact. The tag becomes the release unit. This is the model many teams land on when promotion safety is the primary concern: the exact thing that passed QA is what gets deployed to Prod, provably so because the tag points to the same commit.

The second, promote-by-artifact-SHA, strengthens tag promotion by tracking the S3-uploaded Lambda zip SHA explicitly. The Prod deploy job retrieves the artifact uploaded to S3 during the QA job rather than rebuilding it. This eliminates the theoretical risk that a rebuild from the same tag produces a different artifact (non-reproducible builds, environment-specific dependency resolution).

The third, fan-out per environment from a shared test stage, builds the artifact once in the shared stage and passes it to both environment branches. The QA branch runs immediately. The Prod branch waits at the GitHub Environment gate and, when approved, applies with the same artifact.

Fan-out was chosen over tag promotion and artifact-SHA promotion for one specific reason: this project uses Terraform as its infrastructure layer. Lambda versions, Aurora cluster ARNs, and Secrets Manager paths all live in Terraform state. A promote-by-tag or promote-by-artifact model that re-runs only the Lambda update against Prod still has to reconcile Terraform state between environments, which means either maintaining two separate Terraform apply runs anyway, or accepting that infrastructure changes do not go through the same promotion gate as code. Fan-out makes this explicit: both environments run the full pipeline (plan → apply → Lambda → SPA), from the same artifacts, with the only difference being that Prod waits for a human.

The tradeoff fan-out accepts is that QA and Prod run in parallel once Prod is approved, rather than sequentially with Prod consuming QA’s exact artifact. That risk is mitigated by three properties: the shared test stage produces and uploads the Lambda zips before either branch runs; both branches reference the same S3 artifacts; and the Prod approval gate does not open until QA has completed, so in practice Prod runs from the same artifacts QA already validated, just without the additional guarantee that comes from explicit SHA handoff.

Promote-by-artifact-SHA is the stronger model for artifact integrity and may be worth adopting if the Lambda build becomes non-reproducible in practice. It is not the first implementation choice because the additional complexity (SHA tracking, artifact retrieval logic in the Prod job) solves a theoretical risk that does not yet have evidence.

Cost delta visibility

Infracost’s cost-delta comment is informational rather than a blocking gate, for a specific reason: cost delta is an operator judgment, not a pass/fail condition. The same job does carry one blocking check, a fixed-price ceiling, which is a different assertion and is described at the end of this section.

A blocking Infracost check would require a threshold. Any threshold is arbitrary and requires maintenance: it changes when the stack grows, when AWS pricing changes, when a new environment is added, when the operator deliberately accepts a higher-cost resource change. A threshold that is too tight blocks legitimate changes. A threshold that is calibrated to current spend is stale the first time a new AWS service is added.

Cost review happens on the PR, where the Infracost comment lands before any apply runs. The checklist defined in Story #86 makes “Infracost PR comment reviewed and delta is within accepted budget threshold” an explicit step the operator verifies before clicking approve on the Prod gate, so the same comment is read twice: once on the PR, once at promotion. The delta half of that review is the operator’s judgment, informed by the cost comment.

The informational comment has one enforcement property: it is always present before any apply runs. The Infracost job is gated to pull_request, so it runs on the Terraform PR and never on the merge to main — the comment exists before the merge that triggers the first apply, not after it. An operator who skips reading it is choosing to skip it, not being surprised by a cost change after the fact.

Alongside the comment, the same job asserts each environment’s monthly total against a fixed ceiling and fails the PR when a total exceeds it. That is a blocking check, and it does not contradict the reasoning above, because it is not a cost-delta threshold. It never asks how much a change adds. It asks whether the stack’s fixed-price total left the band it is designed to sit in, which is a conformance test against RAG010: the resources that can breach it are a NAT Gateway and a PrivateLink endpoint, both rejected by that decision. Infracost prices these from the plan in seconds, where actual-spend monitoring would surface them a billing cycle later and buried in noise. The ceiling value and its re-derivation rule live in the QA plan; the two-layer split is RAG016.

No Infracost job runs in the deploy workflow. Both invocations read the plan with terraform init -backend=false and make no live AWS call, so a post-merge run has access to nothing a PR run lacks, and failing one would report a regression already merged.

This section describes the target under RAG016. #302 is outstanding, so qa-deploy.yml still runs a post-merge Infracost job that plan depends on, and no ceiling assertion exists yet; the deploy guides describe that pipeline as it runs today.

Sequencing within an env-deploy

The order within each environment branch is: test (shared stage) → terraform plan → terraform apply → Lambda zip update → SPA sync + CloudFront invalidation.

The shared test stage — pytest with coverage hard-fail, ruff lint, OpenAPI diff check — runs once before either branch proceeds. Running tests inside each branch would double the test time with no benefit, since both branches apply from the same commit.

terraform plan precedes terraform apply because plan output is what the operator reads to confirm there are no unexpected resource replacements or deletions. For QA this is informational (the apply runs automatically). For Prod the operator reviews both the plan output and the Infracost comment before approving. Running apply without a prior plan would remove this checkpoint.

terraform apply precedes the Lambda zip update because the apply may create or update the Lambda function resource itself (new function version configuration, IAM role changes, environment variable changes). Uploading the zip before the resource exists in the expected state risks a failed update or, worse, a successful update against the wrong function configuration.

The Lambda zip update precedes SPA sync because API contract changes (new endpoints, changed response shapes) should be live before the SPA that depends on them. RAG005’s OpenAPI governance enforces that the SPA knows what to expect, but the sequencing ensures the API is not behind the client.

SPA sync and CloudFront invalidation are the last step because they are the user-facing change. If anything earlier in the sequence fails, the current SPA version keeps serving against the current API. Reversing the order — SPA first, then Lambda — creates a window where the client is deployed against an API that has not yet been updated, which can cause user-visible errors.

CloudFront invalidation is the last action of the last step. If it is omitted or fails, the SPA bundle is in S3 but CloudFront continues serving the previous cached version until the TTL expires. This is a partial deploy state but a safe one: users see the previous version rather than a broken one. An invalidation retry is a single CLI call with no other state to reconcile.

Operator interrupt points: before terraform apply in QA (the operator can inspect the plan and cancel the run), and at the Prod approval gate. There is no automated stop between Lambda update and SPA sync; those two steps are tightly coupled and both reversible individually.

Failure modes

QA apply fails, Prod gate open. By design, a QA apply failure does not close the Prod approval gate. This is intentional: the gate opens when the shared test stage passes, which happens before either branch runs. If QA apply fails, the Prod gate has already opened and the operator has already been notified. The deployment plan is explicit that “the operator should not approve Prod against a failing QA state” — this is guidance, not automation. Automating the inverse (closing the gate on QA failure) would require the pipeline to distinguish between a QA failure that invalidates the release (broken behavior) and one that does not (a flaky Terraform resource, a transient AWS API error). That distinction requires judgment. The operator has it; an automated gate does not.

Prod apply fails mid-fan-out, S3 sync done but CloudFront invalidation not. After a successful terraform apply and Lambda update, the SPA sync uploads the new bundle to S3 and then issues the CloudFront invalidation. If the apply succeeds, the Lambda is live, and S3 sync completes but the invalidation fails, the state is: new API Lambda serving at the Prod endpoint, new SPA bundle in S3, CloudFront serving the previous cached version. This is operationally safe because the previous SPA was built against the previous API contract, and the new SPA was built against a compatible API (OpenAPI governance via RAG005 prevents breaking contract changes from reaching the pipeline). The user sees the previous SPA against the new API. The new API must be backwards compatible for this to be safe — and the OpenAPI governance process is what enforces that. Operator recovery is a single invalidation call.

Lambda zip uploaded but alias update fails. Lambda versioning is enabled on every deploy: each apply publishes a new version. The Lambda alias points to the new version as part of the apply. If the alias update fails after the zip is uploaded but before the alias is moved, the previous alias continues pointing to the previous version. From the user’s perspective nothing changed. The operator re-runs the apply or updates the alias directly via CLI. No user-facing impact.

CloudFront invalidation throttled. CloudFront imposes a limit on concurrent invalidation paths. A single-path invalidation (/* or /index.html) is within the free tier (1000 paths/month) and far below the concurrency limit for this scale. Throttling is not a realistic failure mode at this deployment frequency. If it were encountered, the SPA bundle is already in S3; the operator creates an invalidation manually.

Prod terraform plan shows unexpected replacements. The plan output is available before apply runs. For Prod the operator reviews it as part of the Story #86 checklist. If the plan shows a forced replacement of the Aurora cluster (which would destroy all data) due to a change in a Terraform attribute that requires replacement, the operator does not approve. The pipeline holds at the gate indefinitely until the operator either fixes the Terraform configuration or explicitly accepts the replacement as intentional. No automation needs to catch this; the plan output is the signal.

QA deploy succeeds but QA smoke tests are not wired. The current design does not include automated post-deploy smoke tests in CI. If QA apply completes and the Lambda is broken in a way that the unit test suite did not catch, the Prod gate opens. The operator’s checklist (Story #86) requires confirming QA CI passed, but “CI passed” means test-stage passing, not post-deploy functional validation. This is a known gap, acceptable at the current stage. The mitigation is Story #82 (smoke test suite passes end-to-end in QA), which when complete would add a post-deploy step to the QA branch before the Prod gate opens.

What this design deliberately excludes

Blue/green deployment. Blue/green requires two live instances of the service simultaneously — two Lambda versions receiving traffic via weighted aliases, or two API Gateway stages. For a scale-to-zero Lambda stack the cost difference between blue/green and a simple version alias swap is negligible at zero traffic, but the operational complexity is not. Managing traffic weights, monitoring both versions, and deciding when to cut over and when to roll back all require automation that does not pay for itself when the rollback procedure (re-point the alias, one CLI call) takes under a minute. Blue/green is excluded because the rollback is already fast enough that the risk window blue/green is designed to eliminate does not exist here.

Canary releases. Canary requires a percentage of live traffic to route to the new version before full cutover. Lambda weighted aliases support this natively. The exclusion is not technical; it is operational. Canary is useful when “does this version degrade user experience under real traffic” is a question that automation can answer at deployment time. With a handful of actual users and no real-time traffic monitoring wired to deployment events, there is no signal to act on. A canary that routes 10% of traffic to the new version and finds no errors in five minutes does not mean the new version is correct — it means the low-traffic period produced no visible failures. Canary is excluded because it creates ceremony without adding signal.

Automated rollback trigger. The deployment plan specifies that the operator monitors the CloudWatch alarm on API Lambda error rate (>5% over 5 minutes) and initiates rollback manually. An automated trigger — CloudWatch Alarm → Lambda → re-point alias — is technically straightforward and excluded deliberately. Automated rollback requires a reliable signal. A 5% error rate over 5 minutes can mean the new Lambda version is broken, or it can mean a transient Bedrock API degradation, a Secrets Manager throttle, or a test client running a known-bad query. Automated rollback that fires on a false positive during a Bedrock outage would roll back a correct deployment and require re-deployment when the outage clears. The operator can read the alarm context (CloudWatch Logs Insights, X-Ray traces) and make that distinction in under a minute. For one operator with no SLA, automated rollback is a source of incorrect recoveries, not faster ones.

Progressive delivery. No feature flags, no traffic splitting beyond the Lambda alias mechanism, no gradual rollout by user cohort. Progressive delivery infrastructure (LaunchDarkly, Flagsmith, a feature flag table in Aurora) carries a fixed operational footprint and is justified when changes carry enough risk that incremental exposure meaningfully reduces incident severity. For a personal demo with one operator, all changes either work or they do not, and the recovery is fast regardless of exposure percentage.

How readiness gates compose

Story #86 (Prod promotion checklist) and Story #87 (env parity statement) are both committed documents, not CI checks. This is a specific choice.

Making them CI checks would convert operator judgment into pass/fail automation. The parity statement (Story #87) is a structural assertion about the Terraform roots: same modules, same resource types, same IAM shapes. This can be verified by inspection at review time and formalized as a PR checklist item. Automating it as a diff comparison between infra/roots/qa/ and infra/roots/prod/ is possible but fragile — any acceptable per-env difference (VPC and subnet CIDRs, Aurora minimum and maximum ACU capacity, the env tag value — the list in terraform layout) must be excluded from the check, and maintaining the exclusion list is exactly the same overhead as maintaining the parity document. The document is the cleaner artifact.

The promotion checklist (Story #86) cannot meaningfully become a CI check because the items require operator judgment that the pipeline cannot provide. “Infracost delta is within accepted budget threshold” requires knowing what the operator considers acceptable. “No open GitHub issues labeled blocking” is automatable but is a proxy for “the operator has read the issue list and chosen to proceed,” which is not. Converting checklist items to blocking gates adds the appearance of rigor while removing the accountability that the checklist exists to create.

The two artifacts compose with the pipeline by being present at the right moment. Story #87’s parity document is consulted when a PR modifies either Terraform root, surfacing structural divergence at review time. Story #86’s checklist is wired to the GitHub Environment gate by being referenced in the gate’s description and in the operator runbook (Story #84): the operator opens the checklist before approving, not after. The automation runs the plan and posts the cost comment; the human confirms the checklist and clicks approve. The boundary between what the pipeline verifies and what the operator verifies is explicit and stable.

Outline: deployment plan
Terraform layout — the module catalog and per-env roots the fan-out applies
Infrastructure — deployment topology and per-environment resource inventory
ADRs: RAG001 (env topology), RAG005 (OpenAPI governance), RAG006 (coverage), RAG010 (no IGW/NAT), RAG011 (Data API only)
Capabilities and Features: #26 (Environments), #29 (CI pipeline), #30 (SPA deploy), #57 (Production readiness)
Stories: #82 (smoke tests), #84 (runbook), #86 (promotion checklist), #87 (env parity)