QA plan: Observability — structured logs, traces, metrics (#49)

Story: Observability: structured logs, traces, metrics (free-tier AWS-native) (#49)
Capability: Cost guardrails and observability (#31)
ADR: RAG008 — CloudWatch + X-Ray + Lambda Powertools

Test scope

Type	Coverage
Unit	Tracer and Metrics from Lambda Powertools are instantiated in each Lambda handler; no raw boto3 calls for tracing
Unit	Correlation ID is injected at the API Gateway stage into the request context and flows through to the Bedrock and Data API subsegments; a unit test on the handler asserts the same ID appears in the structured log output
Integration (manual, QA)	A single POST /query call: X-Ray trace shows three subsegments — API Lambda, Bedrock call, Aurora Data API call — sharing the same trace ID
Integration (manual, QA)	CloudWatch Logs Insights query on the same request confirms correlation_id field is present and matches the X-Ray trace ID
Integration (manual, QA)	Lambda Powertools Metrics namespace is visible in CloudWatch Metrics console after at least one invocation
Security	No Authorization header or key secret appears in any trace annotation or metadata; verified in the X-Ray console after a /query call with a real token

Entry criteria

• #12, #13 (FastAPI app) and #23 (Lambda + API GW) deployed to QA
• Story #34 (CloudWatch logging) complete (Powertools Logger already in use)

Exit criteria

• Unit test: correlation ID propagation test passes
• Integration: X-Ray trace for one /query call shows API + Bedrock + Data API subsegments
• Logs Insights: correlation_id field present and matches trace ID
• Metrics namespace visible in CloudWatch after one invocation
• Security: no credential fields in X-Ray trace (spot-check in console)

Environments

Unit: CI. Integration: manual against QA env.

Tooling

pytest, Lambda Powertools (Logger, Tracer, Metrics), AWS CLI (aws xray get-trace-summaries), CloudWatch Logs Insights console.

How tests are written

def test_correlation_id_in_log(lambda_log_capture, monkeypatch):
    # Given a request arriving with a correlation ID in the API Gateway context
    event = build_apigw_event(question="What is RAG?", correlation_id="test-corr-123")
    # When the handler processes the event
    handler(event, {})
    # Then the correlation ID appears in at least one log record
    log_strs = [json.dumps(r) for r in lambda_log_capture.records]
    assert any("test-corr-123" in s for s in log_strs), "Correlation ID missing from logs"

Security spot-check (manual):

# Given a /query call with a real bearer token was made to QA
# When the X-Ray trace is opened in the console
# Then no annotation or metadata field contains the token string

Risks

• X-Ray sampling may drop the trace under load; set the sampling rate to 1.0 in QA to ensure all traces are captured during testing.
• Lambda Powertools Metrics require at least one metric flush per invocation; if the handler exits before the flush (e.g., early 401), metrics may be lost. Test the flush behavior on the 401 path.

Capability acceptance criteria link

RAG008: correlation IDs propagate from API Gateway through Lambda and into subsegments; logs and traces share the ID. Security per-feature requirements (#49): correlation ID in logs and traces. Exit criteria cover both.