Environments and CI pipeline

Context

The design docs/devops/deployment-plan.md is the target end-state, authoritative for

• environment topology
• build artifacts
• promotion plan & pipeline stages
• secrets strategy
• terraform state
• rollback

This document is its companion: it explains the reasoning behind the choices in that outline, non-selected alternatives, and failure modes. For what’s deployed and automated today (current state), see the canonical deployment plan.

Relevant ADRs: RAG001 (two separate VPCs, single account), RAG005 (OpenAPI governance), RAG006 (90%/70% coverage hard-fail), RAG010 (no IGW, no NAT), and RAG011 (RDS Data API only). Effects of each are enumerated below.

CI/CD pipeline

QA deploys automatically on merge to main; Prod is gated behind a GitHub Environment manual-approval rule, with Infracost posting a cost diff on every PR so cost impact is visible before merge (see RAG001). The Prod apply step is gated and logged; no force-apply path exists.

Only two environments

The standard argument for three environments — dev, staging, prod — rests on separating integration risk from pre-prod stability. Here, this is covered without a third environment.

Local (Docker Compose) handles integration risk. Developers run the full stack locally against a real Postgres container. Cheap, and shifted-left. Takes the place of AWS-based “dev”.

QA handles pre-prod stabilization. It deploys changes through the full AWS stack on every merge to main. Structurally identical to Prod. It catches anything that only surfaces with real AWS services, such as: IAM issues, RDS Data API behavior, Bedrock Guardrails policy interactions, CloudFront cache header behavior.

Staging, another standard environment, would add roughly $1–4/month idle and at minimum double the Terraform surface to maintain. Its purpose — a Prod-like environment for final pre-release validation — is satisfied by QA, which is already Prod-like by design. Staging makes sense when QA is used for continuous developer integration (low signal, noisy) and you need a calmer final gate. With one operator and deploy on merge, QA is already the calm final gate.

Promotion model

Realistic alternatives to “Auto QA plus manual Prod”:

• full automation (QA auto, Prod auto on QA green)
• full manual (both environments require explicit approval)
• inverse-gate (Prod auto, QA manual as a branch cut).

Full automation: rag-sample’s Prod environment is live to users. Deployment errors (Bedrock Guardrails misconfiguration, an unintended Aurora schema migration, or a CloudFront behavior regression) can all pass the test suite and still degrade the user experience. For a personal demo the stakes are low, but the workflow is designed to practice production delivery patterns — and production delivery patterns include a human checkpoint before a live environment changes. Auto-to-Prod removes that checkpoint for no operational gain. The test suite and QA deploy together catch behavior failures; the Prod gate promotes judgment calls.

Full manual: If QA requires manual approval, then every merge to main involves two approval actions separated by a deploy cycle. The only thing this buys is preventing a broken QA state — which the test hard-fails already handle.

Inverse gate (Prod auto, QA manual) was not seriously considered because it inverts the risk: changes reach Prod before the operator has seen them behave on the real AWS stack.

GitHub Environment protection rules were chosen over an external approval system (PagerDuty, Slack bot, email-triggered webhook). The approval action is already in the workflow the operator is watching. The operator receives a GitHub notification when the gate opens, approves from the same GitHub UI where the pipeline status is visible, and the audit trail (who approved, when) lives in the same system as the code. An external system adds an integration to maintain, a second place to look, and a second point of failure, with no additional capability for a single operator.

Pipeline shape

fan-out vs. promote-by-artifact

Three shapes were considered for getting code from commit to both environments.

“Promote-by-tag” deploys to QA first, then promotes the same tag to Prod by re-running the deploy job against the existing QA artifact. The tag becomes the release unit. This is the model many teams land on when promotion safety is the primary concern: the exact thing that passed QA is what gets deployed to Prod.

Promote-by-artifact-SHA strengthens tag promotion by tracking the S3-uploaded Lambda zip SHA explicitly. The Prod deploy job retrieves the artifact uploaded to S3 during the QA job. This eliminates the theoretical risk that a rebuild from the same tag produces a different artifact (non-reproducible builds, environment-specific dependency resolution).

Fan-out per environment from a shared test stage, builds the artifact once in the shared stage and passes it to both environment branches. The QA branch runs immediately. The Prod branch waits at the GitHub Environment gate and, when approved, applies with the same artifact.

Fan-out was chosen over tag promotion and artifact-SHA promotion for one specific reason: this project uses Terraform as its infrastructure layer. Lambda versions, Aurora cluster ARNs, and Secrets Manager paths all live in Terraform state. A promote-by-tag or promote-by-artifact model that re-runs only the Lambda update against Prod still has to reconcile Terraform state between environments, which means either maintaining two separate Terraform apply runs anyway, or accepting that infrastructure changes do not go through the same promotion gate as code. Fan-out makes this explicit: both environments run the full pipeline (plan → apply → Lambda → SPA), from the same artifacts, with the only difference being that Prod waits for a human.

Tradeoffs: QA and Prod run in parallel once Prod is approved, rather than sequentially with Prod consuming QA’s exact artifact. Mitigations: the shared test stage produces and uploads the Lambda zips before either branch runs; both branches reference the same S3 artifacts; and the Prod approval gate does not open until QA has completed, so in practice Prod runs from the same artifacts QA already validated, just without the additional guarantee that comes from explicit SHA handoff.

Promote-by-artifact-SHA is the stronger model for artifact integrity and may be worth adopting if the Lambda build becomes non-reproducible in practice. It is not the first implementation choice because the additional complexity (SHA tracking, artifact retrieval logic in the Prod job) solves a theoretical risk that does not yet have evidence.

Cost delta visibility

Infracost’s cost-delta comment is informational, because we consider cost delta is an operator judgment, not a pass/fail condition. The job does have one blocking check, a fixed-price ceiling, which is a different assertion and is described at the end of this section.

A blocking Infracost check would require an arbitrary threshold. It requires maintenance and interferes with judgment.

Cost review happens on the PR via an Infracost comment. The operator verifies before clicking approve on the Prod gate, so the same comment is read twice: once on the PR, once at promotion. The delta half of that review is the operator’s judgment about the cost figures.

The informational comment has one enforcement property: it is always present before any apply runs. The Infracost job is gated to pull_request, so it runs on the Terraform PR and never on the merge to main — the comment exists before the merge that triggers the first apply.

Alongside the comment, the job checks each environment’s monthly total against a fixed ceiling and fails the PR when a total exceeds it. That is a blocking check, and it does not contradict the reasoning above, because it is not a cost-delta threshold. It never asks how much a change adds. It asks whether the stack’s fixed-price total left the band it is designed to sit in, which is a conformance test against RAG010: the resources that can breach it are a NAT Gateway and a PrivateLink endpoint, both rejected by that decision. Infracost prices these from the plan in seconds, where actual-spend monitoring would surface them a billing cycle later and buried in noise. The ceiling value and its re-derivation rule live in the QA plan; the two-layer split is RAG016.

No Infracost job runs in the deploy workflow. Both invocations read the plan with terraform init -backend=false and make no live AWS call. A post-merge run has access to nothing more than a PR run, and failing one would report a regression already merged.

This section describes the target under RAG016. #302 is outstanding, so qa-deploy.yml still runs a post-merge Infracost job that plan depends on, and no ceiling assertion exists yet; the deploy guides describe that pipeline as it runs today.

Sequencing within an env-deploy

The order within each environment branch is: test (shared stage) → terraform plan → terraform apply → Lambda zip update → SPA sync + CloudFront invalidation.

The shared test stage — pytest with coverage hard-fail, ruff lint, OpenAPI diff check — runs once before either branch proceeds. Running tests inside each branch would double the test time with no benefit, since both branches apply from the same commit.

terraform plan precedes terraform apply because plan output is what the operator reads to confirm there are no unexpected resource replacements or deletions. For QA this is informational (the apply runs automatically). For Prod the operator reviews both the plan output and the Infracost comment before approving. Running apply without a prior plan would remove this checkpoint.

terraform apply precedes the Lambda zip update because the apply may create or update the Lambda function resource itself (new function version configuration, IAM role changes, environment variable changes). Uploading the zip before the resource exists in the expected state risks a failed update or, worse, a successful update against the wrong function configuration.

The Lambda zip update precedes SPA sync because API contract changes (new endpoints, changed response shapes) should be live before the SPA that depends on them. RAG005’s OpenAPI governance enforces that the SPA knows what to expect, but the sequencing ensures the API is not behind the client.

SPA sync and CloudFront invalidation are the last step because they are the user-facing change. If anything earlier in the sequence fails, the current SPA version keeps serving against the current API. Reversing the order — SPA first, then Lambda — creates a window where the client is deployed against an API that has not yet been updated, which can cause user-visible errors.

CloudFront invalidation is the last action of the last step. If it is omitted or fails, the SPA bundle is in S3 but CloudFront continues serving the previous cached version until the TTL expires. This is a partial deploy state but a safe one: users see the previous version rather than a broken one. An invalidation retry is a single CLI call with no other state to reconcile.

Operator interrupt points: before terraform apply in QA (the operator can inspect the plan and cancel the run), and at the Prod approval gate. There is no automated stop between Lambda update and SPA sync; those two steps are tightly coupled and both reversible individually.

Failure modes

QA apply fails, Prod gate open.

By design, a QA apply failure does not close the Prod approval gate. This is intentional: the gate opens when the shared test stage passes, which happens before either branch runs. If QA apply fails, the Prod gate has already opened and the operator has already been notified. The deployment plan is explicit that “the operator should not approve Prod against a failing QA state” — this is guidance, not automation. Automating the inverse (closing the gate on QA failure) would require the pipeline to distinguish between a QA failure that invalidates the release (broken behavior) and one that does not (a flaky Terraform resource, a transient AWS API error). That distinction requires judgment, which the operator owns.

Prod apply fails mid-fan-out, S3 sync done but CloudFront invalidation not.

After a successful terraform apply and Lambda update, the SPA sync uploads the new bundle to S3 and then issues the CloudFront invalidation. If the apply succeeds, the Lambda is live, and S3 sync completes but the invalidation fails, the state is: new API Lambda serving at the Prod endpoint, new SPA bundle in S3, CloudFront serving the previous cached version. This is operationally safe because the previous SPA was built against the previous API contract, and the new SPA was built against a compatible API (OpenAPI governance via RAG005 prevents breaking contract changes from reaching the pipeline). The user sees the previous SPA against the new API. The new API must be backwards compatible for this to be safe — and the OpenAPI governance process is what enforces that. Operator recovery is a single invalidation call.

Lambda zip uploaded but alias update fails.

Lambda versioning is enabled on every deploy: each apply publishes a new version. The Lambda alias points to the new version as part of the apply. If the alias update fails after the zip is uploaded but before the alias is moved, the previous alias continues pointing to the previous version. From the user’s perspective nothing changed. The operator re-runs the apply or updates the alias directly via CLI. No user-facing impact.

CloudFront invalidation throttled.

CloudFront imposes a limit on concurrent invalidation paths. A single-path invalidation (/* or /index.html) is within the free tier (1000 paths/month) and far below the concurrency limit for this scale. Throttling is not a realistic failure mode at this deployment frequency. If it were encountered, the SPA bundle is already in S3; the operator creates an invalidation manually.

Prod terraform plan shows unexpected replacements.

The plan output is available before apply runs. For Prod the operator reviews it as part of the Story #86 checklist. If the plan shows a forced replacement of the Aurora cluster (which would destroy all data) due to a change in a Terraform attribute that requires replacement, the operator does not approve. The pipeline holds at the gate indefinitely until the operator either fixes the Terraform configuration or explicitly accepts the replacement as intentional. No automation needs to catch this; the plan output is the signal.

QA deploy succeeds but QA smoke tests are not wired.

The current design does not include automated post-deploy smoke tests in CI. If QA apply completes and the Lambda is broken in a way that the unit test suite did not catch, the Prod gate opens. The operator’s checklist (Story #86) requires confirming QA CI passed, but “CI passed” means test-stage passing, not post-deploy functional validation. This is a known gap, acceptable at the current stage. The mitigation is Story #82 (smoke test suite passes end-to-end in QA), which when complete would add a post-deploy step to the QA branch before the Prod gate opens.

What this design deliberately excludes

Blue/green deployment.

Blue/green requires two live instances of the service simultaneously — two Lambda versions receiving traffic via weighted aliases, or two API Gateway stages. The operational complexity is not negligible. Managing traffic weights, monitoring both versions, and deciding when to cut over and when to roll back all require automation with terrible ROI, when the rollback procedure (re-point the alias, one CLI call) takes under a minute. Blue/green is excluded because the rollback is already fast enough that the risk window that blue/green mitigates does not exist here.

Canary releases.

Canary routes a portion of traffic to the new version before full cutover. It helps answer, “Does this version degrade user experience under real traffic?” But this app has a handful of actual users and no real-time traffic monitoring wired to deployment events, there is no signal to act on.

Automated rollback trigger.

Per the deployment plan, the operator monitors the API Lambda on CloudWatch and initiates rollback manually. An automated trigger (CloudWatch Alarm → Lambda → re-point alias) is technically straightforward but excluded deliberately. Automated rollback requires a reliable signal. A 5% error rate over 5 minutes can mean the new Lambda version is broken, or it can mean a transient Bedrock API degradation, a Secrets Manager throttle, or a test client running a known-bad query. Automated rollback that fires on a false positive during a Bedrock outage would roll back a correct deployment and require re-deployment when the outage clears. The operator can read the alarm context (CloudWatch Logs Insights, X-Ray traces) and make that distinction in under a minute. For one operator with no SLA, automated rollback is a source of incorrect recoveries, not faster ones.

Progressive delivery.

No feature flags, no traffic splitting beyond the Lambda alias mechanism, no gradual rollout by user cohort. Progressive delivery infrastructure (LaunchDarkly, Flagsmith, a feature flag table in Aurora) carries a fixed operational footprint and is justified when changes carry enough risk that incremental exposure meaningfully reduces incident severity. For a personal demo with one operator, all changes either work or they do not, and the recovery is fast regardless of exposure percentage.

Composition of Readiness Gates

“Prod promotion checklist” and “env parity statement” are intentionally committed documents, not CI checks.

Making them CI checks would convert operator judgment into pass/fail automation. The parity statement (Story #87) is a structural assertion about the Terraform roots: same modules, same resource types, same IAM shapes. This can be verified by inspection at review time and formalized as a PR checklist item. Automating it as a diff comparison between infra/roots/qa/ and infra/roots/prod/ is possible but fragile — any acceptable per-env difference (VPC and subnet CIDRs, Aurora minimum and maximum ACU capacity, env tag value) must be excluded, and maintaining the exclusion list balances the overhead of maintaining the parity document. The document is cleaner.

The promotion checklist cannot meaningfully become a CI check because the items require operator judgment. “Infracost delta is within accepted budget threshold” requires knowing what the operator considers acceptable. “No open GitHub issues labeled blocking” is automatable but is a proxy for “the operator has read the issue list and chosen to proceed,” which is not automatable. Converting checklist items to blocking gates adds the appearance of rigor and removes accountability that belongs there.

Timing of the two artifacts: consult the parity document when a PR modifies a Terraform root; review all structural divergences. Story #86’s checklist is referenced in the GitHub environment gate’s description and in the operator runbook:

• operator uses the CR checklist
• automation runs the plan and posts the cost comment
• operator confirms the checklist and clicks approve
• …the boundary between pipeline verification and operator verification is explicit and stable.

Related artifacts

• Outline: deployment plan
• Terraform layout — the module catalog and per-env roots the fan-out applies
• Infrastructure — deployment topology and per-environment resource inventory
• ADRs: RAG001 (env topology), RAG005 (OpenAPI governance), RAG006 (coverage), RAG010 (no IGW/NAT), RAG011 (Data API only)
• Capabilities and Features: #26 (Environments), #29 (CI pipeline), #30 (SPA deploy), #57 (Production readiness)
• Stories: #82 (smoke tests), #84 (runbook), #86 (promotion checklist), #87 (env parity)