QA plan: AWS Budgets alarms (#33)

Story: AWS Budgets alarms per env (#33)
Capability: Cost guardrails and observability (#31)
ADR: RAG016 — cost visibility and spend enforcement; RAG008 — observability; cost model in cost.md

Test scope

Type	Coverage
IaC (plan review)	Terraform provisions an aws_budgets_budget resource for each environment (QA, Prod)
IaC (plan review)	Each budget has at least one notification block with an SNS topic or email subscriber
IaC (plan review)	Alert threshold is set at a documented ceiling per environment
Integration (manual, QA)	After apply: aws budgets describe-budgets lists the QA and Prod budgets with correct limits
Integration (manual, QA)	Each budget’s cost filter resolves to real spend: any tag key used in a filter is returned by aws ce list-cost-allocation-tags --status Active, and aws ce get-cost-and-usage grouped on that key returns non-empty values. A budget whose filter matches nothing reads $0 forever and never alerts
Integration (manual, QA)	Bedrock spend falls inside the filter, checked against get-cost-and-usage for the billing period. If on-demand InvokeModel usage is not attributable to the tag, the filter uses the service dimension instead
Integration (manual, QA)	SNS topic subscription is confirmed (email confirmed or SNS endpoint reachable)

Entry criteria

• Story #20 (Terraform remote state & bootstrap) complete
• SNS topic ARN available per env (or email address configured)

Exit criteria

• terraform plan shows two aws_budgets_budget resources (qa, prod)
• Each budget has a notification block with a subscriber
• Post-apply: aws budgets describe-budgets confirms both budgets exist with the correct limits. It does not cover notification endpoints, which describe-budgets omits: run aws budgets describe-notifications-for-budget per budget for that half, and record this criterion as met only once both budgets have been queried that way
• Each budget’s cost filter resolves to spend: aws ce list-cost-allocation-tags --status Active includes every tag key used in a filter, and the filtered dimension returns non-zero spend
• Each environment’s budget carries an alert threshold

Environments

IaC lint: CI. Integration check: manual against QA post-apply (Prod verified at Prod deploy time).

Tooling

Terraform, AWS CLI (aws budgets describe-budgets for limits, aws budgets describe-notifications-for-budget for notification endpoints, aws ce list-cost-allocation-tags and aws ce get-cost-and-usage for the filter-resolves check).

How tests are written

# Given the QA Terraform has been applied
# When the budgets are described via AWS CLI
aws budgets describe-budgets --account-id "$AWS_ACCOUNT_ID" \
  | python3 -c "
import sys, json
r = json.load(sys.stdin)
names = [b['BudgetName'] for b in r['Budgets']]
assert any('qa' in n.lower() for n in names), 'No QA budget found'
assert any('prod' in n.lower() for n in names), 'No Prod budget found'
print('Budgets verified:', names)
"
# Then both QA and Prod budgets are present

# And when the cost-allocation tags backing their filters are listed
aws ce list-cost-allocation-tags --status Active \
  | python3 -c "
import sys, json
active = [t['TagKey'] for t in json.load(sys.stdin)['CostAllocationTags']]
assert 'Environment' in active, f'Environment tag not active; tag-filtered budgets are inert. Active: {active}'
print('Active cost allocation tags:', active)
"
# Then Environment is active — without it a budget filtered on
# user:Environment$<env> matches no spend, reads $0, and never alerts

Risks

• A budget filtered on an inactive cost allocation tag applies cleanly, reads $0, and never alerts. Tag activation is also not retroactive, so activating a tag does not backfill prior spend.
• AWS Budgets has an eventual-consistency delay of up to 24 h; the alarm may not fire immediately after a cost spike. Document this limitation in the runbook.
• Email subscriber confirmation is manual; if the operator does not confirm the SNS subscription email, the alarm fires silently. Add a runbook step to verify the subscription is confirmed.

Capability acceptance criteria link

Cost guardrails: per-env spend alarms with documented thresholds.