QA plan: Answer similarity eval — BERTScore and SentenceTransformers (#78)

Story: Answer similarity eval: BERTScore + SentenceTransformers (#78)
Feature: Retrieval-quality eval harness (#75)
Capability: Retrieval & generation API (#11)
Spec: specs/retrieval.md

Test scope

Type	Coverage
Unit	/query endpoint is called once per eval row; response answer field is extracted correctly
Unit	BERTScore F1 is computed per question; bert_f1_mean and bert_f1_median are floats in [0.0, 1.0]
Unit	SentenceTransformers cosine similarity is computed per question; sentence_bert_cosine_mean and sentence_bert_cosine_median are floats in [-1.0, 1.0]
Unit	Output JSON contains exactly the fields commit_sha, timestamp, bert_f1_mean, bert_f1_median, sentence_bert_cosine_mean, sentence_bert_cosine_median, and questions (per-question breakdown array)
Unit	Per-question breakdown includes question, ground_truth_answer, generated_answer, bert_f1, sentence_bert_cosine for each row
Unit	Script exits non-zero with a descriptive message when /query returns a non-200 status
Unit	Script exits non-zero when the eval table is empty
Integration	Against a stubbed /query returning deterministic answers: output JSON is valid and all metric fields are in range
Integration	S3 upload: stub boto3.client("s3").put_object is called once with the correct bucket, key prefix reports/<commit-sha>/, suffix -answers.json, and JSON body

Note: eval/eval_answers.py is a CI script outside src/rag_sample/. Coverage floors from RAG006 do not apply. Tests validate output format and metric range bounds — not specific score values.

Entry criteria

• Story #76 (corpus setup) merged; eval table seeded with ground-truth answers
• Story #77 (retrieval eval) merged (establishes S3 upload pattern reused here)
• /query endpoint deployed and accessible in QA env, or stubbed via httpx mock for local tests
• bert-base-uncased and all-MiniLM-L6-v2 model weights available in CI runner (cache or bundled)

Exit criteria

• All unit tests pass
• Integration: script produces a valid JSON report with all metric fields in range against a stubbed /query
• S3 upload stub asserts correct bucket, key pattern, and content-type
• Script exits non-zero on empty eval table and on /query error responses
• Manual QA env smoke test: script runs end-to-end against the deployed /query endpoint and uploads a report to S3; report URL logged in the Story #78 issue comment
• Model download in CI completes within the job timeout (document model size; consider caching in CI)

Environments

Unit and integration: local with FakeQueryClient returning deterministic answers, S3 stub, and model weights either cached or mocked. QA env smoke test uses the live /query endpoint and real S3. No CI hard-fail on eval score.

Tooling

pytest, monkeypatch for /query HTTP calls and S3. bert-score and sentence-transformers Python packages. Docker Postgres not required for unit tests (eval rows loaded from a fixture file). CI uses Bedrock credentials from Secrets Manager for the QA env smoke test.

Fixtures and data

• fake_eval_rows fixture: list of 5 dicts with question, ground_truth_answer keys (no DB required for unit tests)
• FakeQueryClient fixture: httpx mock returning {"answer": "Fake answer.", "sources": []} for any question
• stub_s3_put fixture: captures put_object calls for assertion
• fake_commit_sha fixture: "abc1234" for deterministic key assertions

How tests are written

def test_answer_eval_output_json_shape(fake_eval_rows, monkeypatch, stub_s3_put):
    # Given 5 eval rows with ground-truth answers and a stub /query that returns a fixed answer
    monkeypatch.setenv("COMMIT_SHA", "abc1234")
    fake_client = FakeQueryClient(answer="Fake answer.")
    # When eval_answers runs against the fake rows and stub client
    result = run_eval_answers(fake_eval_rows, fake_client)
    # Then the output JSON contains all required top-level fields and a per-question breakdown
    required_keys = {"commit_sha", "timestamp", "bert_f1_mean", "bert_f1_median",
                     "sentence_bert_cosine_mean", "sentence_bert_cosine_median", "questions"}
    assert set(result.keys()) == required_keys
    assert len(result["questions"]) == 5
    assert 0.0 <= result["bert_f1_mean"] <= 1.0
    assert -1.0 <= result["sentence_bert_cosine_mean"] <= 1.0

Risks

• BERTScore and SentenceTransformers download large model weights at first use. CI runners without a model cache will add significant job time. Pin model versions, document sizes, and configure the CI cache key before merging.
• BERTScore is non-deterministic at small batch sizes on some hardware. Tests must assert only range bounds, not exact values. Do not use a specific score as a regression gate in CI.
• The /query endpoint requires a valid bearer token. The eval script must receive a token via environment variable; tests must assert that missing token causes a clear exit-non-zero, not a silent empty-answer result.
• Per-question breakdown array can be large (up to 500 entries). Verify JSON serialization of the full array does not exceed S3 object size limits for typical eval runs.