QA plan: Eval harness corpus setup — load dataset, seed vectors (#76)

Story: Eval harness corpus setup: load dataset, seed vectors (#76)
Feature: Retrieval-quality eval harness (#75)
Capability: Retrieval & generation API (#11)
Spec: specs/retrieval.md

Test scope

Type	Coverage
Unit	load_corpus.py parses the HuggingFace Parquet test split and extracts (question, context, ground_truth_answer) triples
Unit	--count flag limits rows to the specified value; values outside 100–500 raise a clear argparse error
Unit	Idempotency: running with the same (question, context) pair twice results in a single row in the eval table (upsert on a stable natural key)
Unit	Bedrock embedding call receives the question text and returns a 1024-dim vector written to the embedding column
Unit	Script exits with a clear error when the HuggingFace dataset is unreachable (network stub raises ConnectionError)
Integration	Against Docker Postgres: load_corpus.py --count 10 inserts exactly 10 rows; all embedding columns are non-null vector(1024)
Integration	Re-running with --count 10 on an already-seeded table makes zero net row changes
Integration	Embed call count matches the number of rows with a null embedding before the run

Note: eval/load_corpus.py is a CI script outside src/rag_sample/. Coverage floors from RAG006 do not apply. Tests validate correctness, determinism, and output format.

Entry criteria

• Aurora eval table migration merged (eval table with question, context, ground_truth_answer, embedding vector(1024) columns)
• FakeEmbeddingProvider available in tests/stubs.py
• Docker Postgres reachable in the local test environment

Exit criteria

• All unit tests pass
• Integration: after --count 10, eval table has exactly 10 rows with non-null vector(1024) embeddings
• Idempotency integration test passes (second run produces zero new rows)
• --count 99 and --count 501 both exit non-zero with a usage error in stderr
• Script completes in CI using Bedrock credentials from Secrets Manager (manual smoke test against QA env before merge)

Environments

Unit and integration: local with Docker Postgres and FakeEmbeddingProvider. HuggingFace Hub mocked at datasets.load_dataset for unit tests; live dataset used for the QA env smoke test. CI runs against QA env with real Bedrock credentials from Secrets Manager.

Tooling

pytest, monkeypatch for dataset and Bedrock call sites. Docker Postgres integration fixture.

Fixtures and data

• postgres_eval_db fixture: Docker Postgres with eval table schema applied
• FakeEmbeddingProvider in tests/stubs.py: returns [[0.0] * 1024] per question
• fake_hf_dataset fixture: in-memory list of 20 dicts with question, context, answer keys, bypassing datasets.load_dataset

How tests are written

def test_load_corpus_idempotent(postgres_eval_db, fake_hf_dataset, monkeypatch):
    # Given the HuggingFace dataset returns 10 Q&A triples and the eval table is empty
    monkeypatch.setattr("eval.load_corpus.load_dataset", lambda *a, **kw: fake_hf_dataset[:10])
    run_load_corpus(postgres_eval_db, FakeEmbeddingProvider(), count=10)
    row_count_after_first = count_eval_rows(postgres_eval_db)
    # When load_corpus runs a second time with the same dataset
    run_load_corpus(postgres_eval_db, FakeEmbeddingProvider(), count=10)
    row_count_after_second = count_eval_rows(postgres_eval_db)
    # Then no additional rows were inserted
    assert row_count_after_first == 10
    assert row_count_after_second == 10

Risks

• The HuggingFace dataset schema (neural-bridge/rag-dataset-12000) could change column names without notice. Pin the dataset revision SHA in load_corpus.py and assert expected column names in the unit test.
• Seeding 500 rows with live Bedrock in CI adds latency. The --count default should be 100; CI should not exceed 200 unless explicitly configured.
• The eval table natural key for idempotency must be chosen carefully. If question text alone is not unique, use a composite key on (question, context). Verify uniqueness in the fixture.