Skip to content
RAG Reference Implementation

008. Observability: AWS-native CloudWatch + X-Ray + Lambda Powertools

Decision

Section titled “Decision”
  • In the context of a FastAPI-on-Lambda backend (RAG002, RAG004) plus an ingestion Lambda, behind API Gateway HTTP API, calling Aurora via the public RDS Data API and Bedrock via its public regional endpoint, deployed to two environments (RAG001), and a Feature #31 Cost guardrails & observability that lists CloudWatch logs and AWS Budgets without specifying log format, trace propagation, metric dimensions, or retention,
  • facing the choice of observability stack at three usage levels (idle, 1k queries/mo, 10k queries/mo), where the operator’s value ranking is (1) low cost above all, (2) good tools (easy to maintain, nice UI), (3) portfolio value for a job seeker,
  • we decided for an AWS-native stack:
    • AWS Lambda Powertools for Python as the in-Lambda library, providing a structured-JSON logger, an X-Ray tracer, and a CloudWatch Embedded Metric Format (EMF) emitter,
    • CloudWatch Logs with a 30-day retention policy on every log group (set explicitly via Terraform; never default-infinite),
    • AWS X-Ray for distributed tracing across API Gateway → Lambda → Aurora Data API and → Bedrock, with subsegments for the Bedrock embed/generate calls and the Data API query,
    • CloudWatch Metrics via EMF, with custom dimensions per environment and per route, and a small per-environment CloudWatch Dashboard showing the four signals (request rate, p50/p95/p99 latency, error rate, Bedrock token usage),
    • one CloudWatch Alarm per environment on API Lambda error rate (> 5% over 5 minutes), routed to an SNS topic with an email subscription,
  • and neglected
    • Datadog, New Relic, Honeycomb — better UIs, richer query languages, well-loved on portfolios; cheapest free tiers ($0/mo at demo volumes) are realistic, but each adds a vendor account, an API key in Secrets Manager, an outbound destination, and a non-trivial maintenance surface — disproportionate for a personal demo where the operator’s own time is the dominant cost,
    • Grafana Cloud free tier (10k metrics, 50 GB logs, 50 GB traces free) — the strongest “nice UI” option, and good portfolio signal, but the free-tier ingest limits require log scrubbing for safety and the dashboard sync from Terraform is a maintained surface area on its own,
    • OpenTelemetry Collector on its own — vendor-neutral and the strongest portfolio signal for an SRE-leaning role, but requires either self-hosting a collector or pairing it with a backend (one of the SaaS options above), reintroducing the costs we are avoiding,
    • structlog or stdlib logging without Powertools — works, but Powertools bundles logger, tracer, and metrics into one battle-tested package built by AWS specifically for Lambda; replacing it with three smaller libraries costs more time than the dependency saves,
  • to achieve
    • near-zero observability bill at expected volumes: CloudWatch Logs and X-Ray free tiers cover demo and 1k-queries/mo use; light overage at 10k queries/mo is on the order of $1–3/month total across both environments,
    • a single managed-AWS surface that an operator already learning AWS for this project can also use without learning a separate vendor,
    • portfolio-credible AWS-native observability: structured JSON logs, distributed traces across managed services, and EMF custom metrics are the table-stakes pattern most AWS-focused roles look for,
  • accepting
    • the CloudWatch UI is utilitarian and the query language (Logs Insights) is the weakest of the options considered — searching across services means writing structured queries, not clicking through a polished UI,
    • X-Ray’s trace UI is less elegant than Honeycomb or Datadog APM, and X-Ray sampling defaults can drop traces under burst load (mitigated by configuring 100% sampling at this volume),
    • we forgo the “I shipped a Datadog dashboard” portfolio bullet in favor of “I shipped AWS-native observability with X-Ray and Powertools”,
    • if usage ever crosses ~10k queries/mo and the UI quality starts hurting incident response, switching to Grafana Cloud or Datadog is mostly a destination-config change because Powertools EMF metrics and standard log JSON re-export cleanly.

Consequences

Section titled “Consequences”

The API Lambda and ingestion Lambda both import aws_lambda_powertools and use its Logger, Tracer, and Metrics instances. Logger is configured with a service name per Lambda; Tracer captures Bedrock and RDS Data API calls as subsegments; Metrics emits EMF blocks with Environment and Route dimensions. Log groups are created by Terraform with retention_in_days = 30 set explicitly (CloudWatch’s default is never-expire, which silently accrues cost forever).

Terraform module api/ gains: log group + retention, X-Ray tracing on the Lambda (tracing_config { mode = "Active" }), and an IAM policy snippet granting xray:PutTraceSegments, xray:PutTelemetryRecords, and cloudwatch:PutMetricData. Terraform module frontend/ gains nothing new — CloudFront has its own access-log pattern not in scope for this ADR. A separate Terraform resource provisions one CloudWatch Dashboard per environment and one CloudWatch Alarm + SNS topic + email subscription per environment.

The OpenAPI export script and ingestion CLI run outside Lambda and do not emit metrics; they log to stdout via standard Python logging configured to JSON for consistency with the Lambda logs.

If a future requirement adds incident response with SLOs, error budgets, or anomaly detection, the next step is to enable CloudWatch Anomaly Detection on the existing metrics rather than switching stacks — no new vendor.

Notes

Section titled “Notes”

Cost shape at demo and modest production volumes

Section titled “Cost shape at demo and modest production volumes”

Per environment, us-east-1, current published AWS pricing (2026-06):

ServiceIdle1k queries/mo10k queries/moNotes
CloudWatch Logs ingest$0~$0.10~$1.005 GB free; ~1 KB per Lambda request
CloudWatch Logs storage (30d)~$0~$0.05~$0.505 GB free; 30-day retention bounds growth
X-Ray$0$0$0100k traces/mo free; we are well under
CloudWatch Metrics (EMF custom)~$0~$0.30~$0.3010 custom metrics × $0.30/metric/mo; cap dims to keep cardinality low
CloudWatch Dashboard$0$0$03 dashboards free per account
CloudWatch Alarm~$0.10~$0.10~$0.10$0.10/alarm/mo; one per env
SNS topic + email$0$0$0First 1,000 emails/mo free
Per environment~$0.10~$0.55~$1.90
QA + Prod combined~$0.20~$1.10~$3.80

These numbers update the cost model page — observability previously absorbed under the Lambda + API Gateway row; it is meaningfully under $1/month at expected demo volumes.

Powertools key knobs

Section titled “Powertools key knobs”
  • Logger(service="rag-api") (or rag-ingestion) — distinct service names per Lambda so log groups stay separable in Logs Insights.
  • Tracer(service=...) — same naming convention; @tracer.capture_method on the embedding, retrieval, and generation functions adds named subsegments.
  • Metrics(namespace="rag-sample", service=...) — EMF; @metrics.log_metrics on the Lambda handler flushes one EMF record per invocation.
  • metrics.add_dimension(name="Environment", value=settings.environment) — one dimension; do not add per-request high-cardinality dimensions (correlation_id, user_id) here; those live in logs.

What we are explicitly not doing

Section titled “What we are explicitly not doing”
  • No log shipping to a third party. CloudWatch is the only sink.
  • No APM agent. Powertools Tracer + X-Ray is the only tracing surface.
  • No real user monitoring (RUM) on the SPA. CloudFront access logs are out of scope for this ADR.
  • No SLO/error-budget framework. Single alarm on error rate is the only paging surface.

© 2026 Benjamin Arunski