QA summary

Scope

rag-sample is a personal RAG demo: HuggingFace corpus ingestion, Bedrock embeddings, pgvector in Aurora Serverless v2, FastAPI on Lambda (Mangum), React SPA on S3/CloudFront, and a bearer-token Lambda authorizer. QA covers all deployed behaviors in local, QA, and Prod environments (RAG001).

Environments

Environment	Purpose	Deploy trigger	Aurora	Lambda
Local (Docker)	Development, unit and integration tests	Manual	psycopg3 direct	Uvicorn
QA	CI validation, contract and e2e tests	Auto on merge to main	RDS Data API	AWS Lambda
Prod	Release gate	Manual approval (GitHub Environment)	RDS Data API	AWS Lambda

Integration and contract tests that hit real AWS resources (Bedrock, Aurora Data API) run against QA only. Unit and mocked-integration tests run locally and in CI on every PR.

Test pyramid posture

              [e2e]       -- Playwright against CloudFront (QA only)
           [contract]     -- Newman/Postman, OpenAPI diff check (CI + QA)
        [integration]     -- pytest, real Docker Postgres (CI)
      [unit (mocked)]     -- pytest + monkeypatch for Bedrock/Data API (CI)

Backend target: 90% aggregate line+branch, 70% per-file minimum (RAG006 hard-fail). Frontend: Vitest + React Testing Library for unit, Playwright for e2e. Infrastructure: Infracost on every PR, terraform validate + tflint in CI.

Tooling stack

Layer	Tool	Where
Python unit/integration	pytest, pytest-cov, coverage	Local, CI (every PR)
Coverage gate	pytest --cov=rag_sample --cov-branch --cov-fail-under=90	CI hard-fail (RAG006)
Frontend unit	Vitest + React Testing Library	Local, CI
e2e	Playwright (headed locally, headless in CI)	CI against QA env
API contract	Newman (Postman CLI), openapi-diff	CI openapi-check job (RAG005)
Infra cost	Infracost	CI, every PR
IaC lint	terraform validate, tflint	CI
Dependency audit	pip-audit	CI (non-blocking initially per security assessment)

Shared fixtures and seed data

• Postgres fixture: pytest fixture spinning up Docker Postgres with pgvector and applying all migrations from the migrations/ directory. Shared across ingestion and API integration tests.
• Seed corpus: 50-row slice of neural-bridge/rag-dataset-12000 committed to tests/fixtures/corpus_seed.parquet. Used in all tests that need embedded data without calling Bedrock live.
• Pre-embedded seed: a matching tests/fixtures/embeddings_seed.sql inserts known-vector rows so retrieval tests exercise pgvector similarity without Bedrock.
• Stub providers: FakeEmbeddingProvider and FakeLLMProvider (zero-vector embed, fixed-text LLM) committed to tests/stubs.py for unit and integration tests that do not cross the AWS boundary.
• API key fixtures: helper make_test_key(remaining=N, days_offset=0) inserts a known-plaintext key into the test DB and returns the raw token for use in authorizer and API tests.

How tests are written

All test functions follow the Given / When / Then comment convention (org dev standards):

def test_example():
    # Given <setup>
    ...
    # When <action under test>
    ...
    # Then <assertions>
    ...

Given leads setup blocks, When leads the call under test, Then leads assertions. This is mandatory in all new test files.

Coverage: what is excluded

Modules excluded from the 90%/70% floors per the coverage exclusion policy:

• src/rag_sample/**/__init__.py
• src/rag_sample/settings.py
• src/rag_sample/__main__.py
• src/rag_sample/api/lambda_handler.py (Mangum one-liner)
• scripts/export_openapi.py

Per RAG006, any exclusion addition requires a concurrent pyproject.toml update. Infra Terraform files and frontend CSS are not in the Python coverage scope.

Traceability index

Feature	Issue	QA plan	Capability
Corpus ingestion: load and chunk	#8	plan-corpus-load-chunk	#7 Corpus ingestion
Corpus ingestion: embed and upsert	#9	plan-corpus-embed-upsert	#7 Corpus ingestion
Local DB and schema	#10	plan-local-db-schema	#7 Corpus ingestion
FastAPI skeleton and healthz	#12	plan-api-skeleton	#11 Retrieval & generation API
/query endpoint	#13	plan-query-endpoint	#11 Retrieval & generation API
Retrieval-quality eval harness	#75 (decomposed from #14)	retrieval-eval	#11 Retrieval & generation API
Eval harness corpus setup	#76	plan-eval-corpus-setup	#11 Retrieval & generation API
Retrieval precision eval	#77	plan-eval-retrieval	#11 Retrieval & generation API
Answer similarity eval	#78	plan-eval-answers	#11 Retrieval & generation API
pytest coverage CI job	#15	plan-pytest-ci	#11 Retrieval & generation API
React chat UI	#17	plan-react-chat-ui	#16 Web UI
Wire UI to API	#18	plan-wire-ui-api	#16 Web UI
SPA key intake and badge	#63	plan-spa-key-badge	#16 Web UI
Terraform remote state and layout	#20	plan-terraform-bootstrap	#19 AWS infrastructure
Network module	#21	plan-network-module	#19 AWS infrastructure
Data module (Aurora + pgvector)	#22	plan-data-module	#19 AWS infrastructure
API module (Lambda + API GW)	#23	plan-api-module	#19 AWS infrastructure
Frontend module (S3 + CloudFront)	#24	plan-frontend-module	#19 AWS infrastructure
Secrets and IAM	#25	plan-secrets-iam	#19 AWS infrastructure
QA environment auto-deploy	#27	plan-qa-env	#26 Environments & promotion
Prod environment manual gate	#28	plan-prod-env	#26 Environments & promotion
CI pipeline	#29	plan-ci-pipeline	#26 Environments & promotion
Frontend build and deploy	#30	plan-frontend-deploy	#26 Environments & promotion
Infracost PR comment	#32	plan-infracost	#31 Cost guardrails & observability
AWS Budgets alarms	#33	plan-budgets	#31 Cost guardrails & observability
CloudWatch logging and metrics	#34	plan-cloudwatch	#31 Cost guardrails & observability
Observability (structured logs, traces)	#49	plan-observability	#31 Cost guardrails & observability
Bedrock token-spend alarm	#64	plan-bedrock-alarm	#31 Cost guardrails & observability
API GW route throttle	#65	plan-apigw-throttle	#31 Cost guardrails & observability
API authentication	#59	plan-api-auth	#45 Security
Content provenance and classification	#66	plan-content-provenance	#45 Security
Corpus license and attribution	#48	plan-corpus-attribution	#47 Licensing & attribution

Missing-Feature gaps (PM triage required)

The following Capabilities have no Feature issues as of Stage 3 gate. No QA plans have been written for them; plans will be added after Features are created.

Capability	Issue	Gap
Architecture documentation	#35	Issues #37, #38, #40, #41 are engineering tasks (LikeC4, Astro, ADR cleanup), not QA-testable Features. No QA plan warranted unless a Feature is opened for doc correctness or automated link-checking.
Safety	#44	No Feature issues exist. Decision #55 (closed): Bedrock Guardrails approved. Create Features when #44 enters Stage 3.
Legal & compliance	#46	#47 (Licensing & attribution) is the only child Capability; #48 is its only Story. No separate Feature under #46 beyond that.
Production readiness	#57	No Feature or Story issues. Capability is likely a gate/checklist rather than a buildable Feature; PM should confirm scope.
Infrastructure setup	#56	No Feature or Story issues. Same situation as #57; PM should confirm whether this collapses into #19 or stands alone.

Sidebar wiring note

PM: add { label: 'QA', autogenerate: { directory: 'qa' } } to the sidebar array in astro.config.mjs to wire this section into the Starlight nav. The qa/ directory autogenerates from the files in it.