Skip to content
RAG Reference Implementation

014. Eval harness answer similarity - BERTScore plus SentenceTransformers

Decision

Section titled “Decision”
  • In the context of the retrieval-quality eval harness (#75), which runs the rag-sample query pipeline against the neural-bridge/rag-dataset-12000 test split (ground-truth question / context / answer triples), measures retrieval precision via top-k match (#77), and needs to score each generated answer against the ground-truth answer (#78) so that retrieval and generation quality can be tracked across changes to the corpus, the retrieval parameters, or the model (RAG007),
  • facing the choice of an answer-similarity metric (or set of metrics) that is meaningful for paraphrased natural-language answers, deterministic enough to compare runs, fast enough to score thousands of examples on a developer machine or in CI, and cheap enough that running the harness on every meaningful change is not a budget conversation,
  • we decided for two complementary embedding-based metrics computed per answer:
    • BERTScore (bert-score package, default bert-base-uncased model, F1 reported alongside precision and recall) for token-level semantic overlap between generated and reference answers, and
    • SentenceTransformers cosine similarity (sentence-transformers package, all-MiniLM-L6-v2 model) for sentence-level holistic similarity of the two answers as whole utterances,
  • both computed locally inside the harness, both written into a per-run JSON report alongside the retrieval metrics, with no threshold enforcement and no gating behavior in this ADR,
  • and neglected
    • BLEU. n-gram overlap against a reference. Designed for machine translation, penalizes paraphrase, rewards surface-form match. A correct answer phrased differently than the reference scores poorly, which is exactly the failure mode this harness needs to not exhibit. Rejected on metric validity for the task.
    • ROUGE (ROUGE-1, ROUGE-2, ROUGE-L). Same family of failure as BLEU: n-gram or longest-common-subsequence overlap that rewards lexical match and penalizes paraphrase. Standard in summarization eval, weak for free-form QA where the reference is one of many valid phrasings. Rejected for the same reason as BLEU.
    • MoverScore. Embedding-based like BERTScore, uses Earth Mover’s Distance over contextual embeddings, generally correlates better with human judgment than BERTScore on some benchmarks. Rejected because the EMD solve makes it materially slower per example than BERTScore at the scale this harness needs (thousands of examples per run), with no clearly differentiating signal for the MVP. Revisit if BERTScore turns out to be insufficient.
    • RAGAS Answer Relevancy. Purpose-built for RAG eval, scores answer relevance by generating candidate questions from the answer and comparing them to the original question. Requires the original question as input (which the harness has) and an LLM call per example to generate the candidate questions (which the harness does not want). Adds Bedrock cost per scored example, adds non-determinism, and measures a different thing (answer-to-question coherence) than what this metric pair measures (answer-to-reference similarity). Rejected as scope expansion; revisit as a separate metric if question-grounding quality becomes a concern.
    • LLM-as-Judge (Claude or similar scoring each pair on a rubric). Strong correlation with human judgment, flexible rubric, and the natural choice for nuanced evaluation. Rejected for the MVP because it is non-deterministic across runs without seeding the model deterministically (which Bedrock does not promise), adds per-example Bedrock cost that scales with corpus size, and makes the harness’s own quality dependent on the model it is evaluating. Acceptable as a sampled, manual-trigger overlay later; not acceptable as the per-run primary metric.
    • Exact match / F1 on tokens. Standard for extractive QA benchmarks. The rag-sample answers are generative and free-form. Exact match is near-zero by construction; token F1 has the same paraphrase blindness as BLEU and ROUGE. Rejected.
  • to achieve
    • a quality signal that survives paraphrase: both chosen metrics operate on embeddings, not surface tokens, so a correct answer phrased differently from the reference scores in a usable range,
    • a quality signal at two levels of granularity: BERTScore catches missing or extra concepts at the token level (the reference mentions X and the answer does not, or vice versa), SentenceTransformers captures whether the answer reads as the same statement overall,
    • determinism across runs at fixed model weights: both metrics are pure functions of the inputs and the pinned model, so two runs over the same answers produce the same scores and reports diff cleanly,
    • cost zero per scored example: both metrics run locally on CPU at the scales this harness targets, with no Bedrock or third-party API calls,
    • speed adequate for a developer-machine or CI run of the full test split: all-MiniLM-L6-v2 is the smallest mainstream SentenceTransformer (384-dim, ~22 MB), and bert-base-uncased is the BERTScore default and is fast enough on CPU for thousands of examples,
  • accepting
    • that no single embedding metric is a ground truth: both BERTScore and SentenceTransformers correlate with human judgment imperfectly, and a regression in either is a signal to investigate rather than a verdict,
    • that the metrics share a common failure mode (embedding-model bias): if bert-base-uncased or all-MiniLM-L6-v2 is systematically wrong about a domain, both metrics will be wrong together in that domain, and an LLM-as-Judge overlay may be needed later to detect that,
    • the cost of two model-weight downloads on first run (bert-base-uncased ~440 MB, all-MiniLM-L6-v2 ~22 MB), cached locally thereafter,
    • that this ADR does not specify thresholds, gating, or pass/fail semantics; the harness produces reports, and any decision to gate CI or block deploys on these scores is a separate operational decision.

Consequences

Section titled “Consequences”

Metric implementation

Section titled “Metric implementation”

The harness depends on two additional Python packages, both pinned in the harness’s pyproject.toml:

  • bert-score — provides bert_score.score(cands, refs, lang="en", model_type="bert-base-uncased"), which returns precision, recall, and F1 tensors over the candidate / reference batch.
  • sentence-transformers — provides SentenceTransformer("all-MiniLM-L6-v2").encode(...) for embedding answers, with cosine similarity computed via sentence_transformers.util.cos_sim or equivalent.

Both model identifiers are pinned in harness configuration so a model change is an explicit, reviewable edit, not a transitive update from a package bump.

Report schema

Section titled “Report schema”

Each harness run produces a single JSON report. The per-example record carries both metrics so that aggregate statistics (mean, median, p10, p90) can be recomputed without rerunning the pipeline:

{
  "run_id": "<uuid>",
  "timestamp": "<iso-8601>",
  "dataset": {
    "name": "neural-bridge/rag-dataset-12000",
    "split": "test",
    "n_examples": 1500
  },
  "models": {
    "generator": "us.anthropic.claude-haiku-4-5-20251001-v1:0",
    "bertscore": "bert-base-uncased",
    "sentence_transformer": "all-MiniLM-L6-v2"
  },
  "retrieval_summary": { "top_k_match_rate": 0.0 },
  "answer_similarity_summary": {
    "bertscore_f1": { "mean": 0.0, "p10": 0.0, "p50": 0.0, "p90": 0.0 },
    "sentence_cosine": { "mean": 0.0, "p10": 0.0, "p50": 0.0, "p90": 0.0 }
  },
  "examples": [
    {
      "id": "<dataset row id>",
      "question": "...",
      "reference_answer": "...",
      "generated_answer": "...",
      "retrieval": { "top_k_match": true, "rank": 1 },
      "answer_similarity": {
        "bertscore": { "precision": 0.0, "recall": 0.0, "f1": 0.0 },
        "sentence_cosine": 0.0
      }
    }
  ]
}

The schema is versioned (schema_version field, omitted above for brevity) so that downstream consumers can detect format changes.

models.generator above is an illustrative value from one run, not a specification. The deployed model ID lives in Settings.llm_model_id; RAG007 decides the Haiku-class tier rather than a fixed ID, so expect this field to differ between report generations.

Storage

Section titled “Storage”

Reports are written to S3 under a per-environment prefix:

s3://<eval-reports-bucket>/<env>/eval-runs/<yyyy>/<mm>/<dd>/<run_id>.json

The bucket is per-environment to match the QA / Prod VPC separation in RAG001, with lifecycle rules retaining reports for a finite window (revisit once volume is known). Reports are written from wherever the harness runs (developer machine, CI job, or Lambda); the writer needs s3:PutObject on the prefix and nothing else.

Dataset

Section titled “Dataset”

The harness reads neural-bridge/rag-dataset-12000, test split, via the same huggingface-hub + pyarrow path established for corpus ingestion in RAG003. Each row supplies a question, a ground-truth context, and a ground-truth answer; the question is sent through the live query pipeline, and the resulting answer is scored against the ground-truth answer.

No gating in this ADR

Section titled “No gating in this ADR”

The harness produces a report. It does not fail a build, block a deploy, or page anyone based on the scores in that report. Decisions about thresholds, regression detection, and gating live in operational documentation and may produce a follow-on ADR if they become contractually binding on releases.

Open questions

Section titled “Open questions”
  • Whether to add an LLM-as-Judge overlay on a sampled subset of examples to detect systematic embedding-model bias in BERTScore / SentenceTransformers. Deferred until the harness has produced enough reports to identify whether such bias is observable.
  • Whether to add MoverScore or RAGAS Answer Relevancy as a third metric once the MVP harness is in routine use and the cost / determinism tradeoffs are better understood.
  • Retention policy for eval reports in S3. Default to 90 days until a longer-term comparison need is articulated.

Amendment log

Section titled “Amendment log”

2026-08-06 — Editorial only, no change to the decision: updated references to reflect de-pinning from a fixed model ID to the tier. See RAG007’s amendment log for the underlying change.

© 2026 Benjamin Arunski