QA plan: Retrieval-quality eval harness (#75)

Note: this plan’s subject is Feature #75; Feature #14 was its prior name. Per-story detail is in:

• plan-eval-corpus-setup.md — Story #76: load dataset, seed vectors
• plan-eval-retrieval.md — Story #77: retrieval precision eval
• plan-eval-answers.md — Story #78: answer similarity eval

Feature: Retrieval-quality eval harness (#75)
Capability: Retrieval & generation API (#11)
Spec: specs/retrieval.md

Test scope

Type	Coverage
Unit	Harness correctly loads Q&A pairs from the dataset’s built-in questions
Unit	Harness computes recall@k: for a question whose answer chunk is known, the retrieved top-k list contains it
Unit	Score aggregation: mean recall over N question/answer pairs is computed correctly
Integration	Harness runs end-to-end against Docker Postgres with pre-embedded seed; produces a score > 0
Performance	Harness completes for 100 evaluation questions in under 60 seconds locally (soft gate; documents baseline)
Security	Harness reads corpus only; writes no data to the DB

Note: this Feature is a developer tool, not a deployed service. Coverage floors from RAG006 apply to any harness code under src/rag_sample/; standalone scripts under scripts/ are excluded per coverage-exclusions.

Entry criteria

• Story #9 (embed and upsert) merged; pre-embedded seed available
• search_similar_chunks function testable with Docker Postgres

Exit criteria

• Harness unit tests pass
• Integration: harness produces a recall@5 score and prints it to stdout without error
• Baseline recall@5 score documented in the Feature issue comment (first run establishes the baseline; regression is a product-quality signal, not a CI hard-fail)
• Any harness code in src/rag_sample/ meets the 70% per-file floor

Environments

Local Docker for integration run. No AWS access; harness uses pre-embedded seed, not live Bedrock.

Tooling

pytest for unit. Harness runs as python -m rag_sample.eval or a script under scripts/. No CI hard-fail on eval score; score is logged as an artifact for tracking.

Fixtures and data

• Pre-embedded seed from Story #9
• Known Q&A pairs from neural-bridge/rag-dataset-12000 (a subset committed to tests/fixtures/eval_questions.json)

How tests are written

def test_recall_at_5_known_pair(postgres_db_with_embeddings):
    # Given a known question whose answer chunk is in the seeded DB
    question = "What is RAG?"
    expected_chunk_id = "seed-chunk-0042"
    # When recall@5 is computed
    score = compute_recall_at_k(postgres_db_with_embeddings, question, expected_chunk_id, k=5)
    # Then the known chunk was retrieved
    assert score == 1.0

Risks

• Recall score is sensitive to the embedding model and the seed data; a seed that is too small may produce artificially high or low recall that is not representative of the full corpus. Document sample size alongside the score.
• Harness depends on a stable question format from the dataset. If the dataset schema changes, the loader in the harness will break. Pin the dataset revision SHA.

Capability acceptance criteria link

This Feature has no direct spec acceptance criterion; it is a quality-signal tool. Its output (baseline recall score) feeds the operator’s confidence gate before promoting to Prod.