QA plan: Retrieval precision eval — top-k match rate (#77)

Story: Retrieval precision eval: measure top-k match rate (#77)
Feature: Retrieval-quality eval harness (#75)
Capability: Retrieval & generation API (#11)
Spec: specs/retrieval.md

Test scope

Type	Coverage
Unit	For a question whose ground-truth context is present in the top-k results, the per-question hit is recorded as 1
Unit	For a question whose ground-truth context is absent from top-k, the per-question hit is recorded as 0
Unit	retrieval_precision = hits / evaluated_count; retrieval_recall computed correctly over the full set
Unit	Output JSON contains exactly the fields commit_sha, timestamp, retrieval_precision, retrieval_recall, k, evaluated_count with correct types
Unit	retrieval_precision and retrieval_recall are floats in [0.0, 1.0]
Unit	Script exits with a non-zero code and descriptive message when the eval table is empty
Integration	Against Docker Postgres with pre-seeded eval rows and known-distance embeddings: output JSON is valid, retrieval_precision > 0.0
Integration	S3 upload: stub boto3.client("s3").put_object is called exactly once with the correct bucket, key prefix reports/<commit-sha>/, and JSON body

Note: eval/eval_retrieval.py is a CI script outside src/rag_sample/. Coverage floors from RAG006 do not apply. Tests validate output format, metric range bounds, and S3 upload mechanics — not specific score values.

Entry criteria

• Story #76 (corpus setup) merged; eval table seeded and accessible in Docker Postgres
• search_similar_chunks function available and testable with Docker Postgres
• S3 bucket rag-sample-eval provisioned in QA env (or stubbed for unit/integration)

Exit criteria

• All unit tests pass
• Integration: script produces a valid JSON report with retrieval_precision in [0.0, 1.0] against the Docker Postgres seed
• S3 upload stub asserts correct bucket, key pattern, and content-type
• Script exits non-zero on empty eval table
• Manual QA env smoke test: script runs end-to-end with live Bedrock and uploads a report to S3; report URL logged in the Story #77 issue comment

Environments

Unit and integration: local with Docker Postgres, FakeEmbeddingProvider, and S3 stub. QA env smoke test uses live Bedrock credentials from Secrets Manager and real S3. No CI hard-fail on eval score; the JSON report is uploaded as a CI artifact for tracking.

Tooling

pytest, monkeypatch for Bedrock and S3 call sites. Docker Postgres integration fixture from Story #76. CI uses Bedrock credentials from Secrets Manager.

Fixtures and data

• postgres_eval_db fixture with 10 seeded rows including known-distance embeddings (cosine similarity ordering is deterministic)
• FakeEmbeddingProvider: returns [[0.0] * 1024] per question
• stub_s3_put fixture: captures put_object calls for assertion
• fake_commit_sha fixture: "abc1234" for deterministic key assertions

How tests are written

def test_retrieval_output_json_shape(postgres_eval_db, monkeypatch, stub_s3_put):
    # Given the eval table has 5 seeded Q&A rows with known embeddings
    monkeypatch.setenv("COMMIT_SHA", "abc1234")
    # When eval_retrieval runs with k=5
    result = run_eval_retrieval(postgres_eval_db, FakeEmbeddingProvider(), k=5)
    # Then the output JSON contains all required fields with correct types and value bounds
    assert set(result.keys()) == {"commit_sha", "timestamp", "retrieval_precision", "retrieval_recall", "k", "evaluated_count"}
    assert result["k"] == 5
    assert result["evaluated_count"] == 5
    assert 0.0 <= result["retrieval_precision"] <= 1.0
    assert 0.0 <= result["retrieval_recall"] <= 1.0

Risks

• Retrieval precision is sensitive to the seed embeddings. Using FakeEmbeddingProvider (all-zeros vectors) means every question has the same embedding, making top-k results effectively random by insertion order. The integration test should assert only that a result is produced and is in-range, not a specific score. Use a fixture with distinct known-distance vectors only if ordering assertions are needed.
• S3 key format includes commit_sha from the CI environment. If COMMIT_SHA is unset, the script must fail clearly rather than uploading to an ambiguous path. Add a unit test for this case.
• Precision and recall over a small eval set (100 rows) are noisy. Document the evaluated_count prominently in the report so consumers interpret scores in context.