016. Infrastructure cost visibility and spend enforcement: Infracost + Budgets Actions

• Date: 2026-06-09 (amended 2026-08-21)
• Status: Accepted
• Builds on: RAG001. Environment topology: QA and Prod separate VPCs, RAG008. Observability: AWS-native CloudWatch + X-Ray + Lambda Powertools

Decision

• In the context of a two-environment Terraform-managed AWS stack (RAG001) where Bedrock inference is the primary variable cost driver with no native account-level hard cap,

• facing two related problems: (1) infrastructure cost changes must be visible before a PR is merged, and (2) a spend ceiling must activate automatically on breach,

• we decided for a two-layer approach:

  • Layer 1 — Pre-merge visibility and a fixed-price ceiling: Infracost parses the terraform plan and posts a per-environment cost-breakdown comment on every Terraform PR. A second step asserts each environment’s monthly total against a fixed ceiling and fails the PR when a total exceeds it.
  • Layer 2 — Post-breach enforcement: An AWS Budgets Action automatically attaches a Deny IAM policy to the Lambda execution roles, blocking bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream, when actual monthly spend exceeds the per-environment threshold. Combined with the API Gateway throttle (#65) and Bedrock spend alarm (#64), this is the strongest available spend enforcement on a single AWS account without AWS Organizations.

• and neglected

  • AWS Pricing API with custom CI implementation. Same pre-merge visibility as Infracost but requires maintaining bespoke code to parse and act on the plan while providing no additional capability at this scale.
  • OPA cost policies (blocking gate). A cost-delta threshold written in Rego fails the PR if exceeded - effectively an arbitrary choice (accepted by operator)
  • No automated cost tooling (Budgets alarms only). Relying solely on post-deploy alarms loses the pre-merge comment property. Defensible for a personal project but does not satisfy the pre-merge visibility requirement.
  • AWS Organizations Service Control Policies (SCPs). The strongest enforcement primitive and the right answer for a multi-account production environment. For a single-account demo, Organizations introduces maintenance overhead and cross-account role complexity disproportionate to the scale.
  • Blocking cost-delta budget gate. Failing a PR because it adds more than a set dollar amount per month, whether expressed in Rego or as an Infracost threshold. Rejected as it is distinct from Layer 1’s ceiling, which asserts an absolute total rather than a delta, and tests conformance to RAG010 rather than spend.

• to achieve

  • the operator sees the fixed-price cost impact of every Terraform change before approving the Prod gate,
  • a Terraform change that provisions a fixed-price resource above the ceiling fails its own PR, before merge and before any apply,
  • Bedrock inference is cut off automatically when the monthly threshold is breached, without operator intervention,
  • defense-in-depth across three layers: API Gateway throttle caps request rate, Bedrock alarm provides early warning, Budgets Action enforces the ceiling,

• accepting

  • Infracost prices the fixed-price subset of the stack and little else. Aurora Serverless v2 does not appear as an underestimate, it prices to zero until a usage file is supplied, as ACU consumption is usage-based. Layer 1 detects one class of change, the addition of a fixed-price resource such as a NAT Gateway or a PrivateLink endpoint, and gives no signal on the spend that dominates the bill. infracost.yml must state this rather than describe the Aurora figure as a floor.
  • The Budgets Action fires after charges breach the threshold; fixed costs (Aurora minimum ACU, CloudWatch, S3) continue to accrue after the Bedrock deny is applied. Set the threshold at (desired ceiling) − (estimated fixed costs), taking the fixed-cost figure from the cost model rather than restating it here, so the two cannot drift.
  • AWS Budgets evaluates spend periodically, so there is inherent lag before the Action applies. The API Gateway throttle caps request rate before the threshold is reached, making that lag acceptable — the Action is a recovery mechanism, not a burst guard. When the Action fires, /query goes offline until the Deny policy is detached.

Consequences

CI pipeline

The reusable CI workflow gains an Infracost step after terraform plan on all Terraform PRs. A further step fails the PR when either environment’s monthly total exceeds the ceiling. infracost.yml annotates Aurora Serverless v2 as priced at zero, ACU consumption being usage-based.

Terraform (modules/ and envs/)

A budgets module (or inline resources per env root) provisions one aws_budgets_budget per environment and one aws_budgets_budget_action targeting the Lambda execution role(s) with action_type = APPLY_IAM_POLICY, attaching a named Deny policy on bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream. The Bedrock spend alarm coexists on the same budget resource as the early-warning layer.

Amendment log

2026-08-21 — change PR-level checks to be blocking, as post-merge checks offer no further information.