007. LLM model selection: Haiku-class via Bedrock

• Date: 2026-05-29 (amended 2026-08-06, 2026-08-21)
• Status: Accepted
• Builds on: RAG002. Backend application stack and tooling

Decision

• In the context of selecting a generation model for the RAG query pipeline, where the LLM’s role is constrained — summarizing and citing pre-retrieved chunks rather than open-ended reasoning — and where the backend already abstracts the LLM behind an LLMProvider protocol (RAG002),
• facing the trade-off between answer quality, cost per query, AWS-native integration, and operational simplicity across demo and corporate-scale workloads,
• we decided for the current Haiku-class Anthropic model via AWS Bedrock — the cheapest tier in the Claude family, not a fixed model ID,
• and neglected
  • Sonnet-class via Bedrock — highest out-of-the-box quality but 2–3× more expensive at current Bedrock rates (Sonnet 5 at $2/M · $10/M promotional through 2026-08-31, $3/M · $15/M after, against Haiku 4.5 at $1/M · $5/M; the tiers have converged, though by less than the original ADR implied — on Bedrock rates the 2026-05-29 gap was ~24×, not the 12× recorded at the time, which came from quoting Anthropic’s first-party Sonnet rate under a Bedrock column; see Notes); the quality advantage is diminished in RAG workloads where retrieval does the heavy lifting and the answer is grounded in provided chunks
  • GPT-4o (OpenAI API) — rejected on vendor dependency, not price: it adds an OpenAI dependency and moves generation outside the AWS trust boundary. Its row in the table below uses 2026-05-29 rates and has not been re-quoted, so no current cost ordering against the Bedrock rows is claimed here
  • GPT-4o mini (OpenAI API) — the cheapest row at the rates it was quoted at (2026-05-29: ~$0.50/1k queries; not re-quoted, so the current gap against Haiku-class at ~$3.50 is indicative only), and rejected anyway: it carries the same OpenAI vendor dependency and puts prompt and retrieved-chunk content outside the AWS trust boundary, which is the constraint this decision is unwilling to trade for a few dollars a month at this volume
  • Self-hosted open-weight models (Llama 3 70B, Mistral) on EC2/ECS — predictable fixed cost at high volume (~$875/mo); never cost-effective at this instance size; high operational overhead
• to achieve near-equivalent answer quality at 2–3× lower per-query cost, while remaining AWS-native and keeping a model switch to configuration, IAM, and one settings default — no change to the query pipeline or the LLMProvider implementations,
• accepting a marginal quality reduction on complex multi-step reasoning; acceptable for a RAG workload where the answer is grounded in retrieved chunks and the LLM’s role is synthesis and citation rather than independent reasoning.

Why the tier, not the model ID

The original decision named anthropic.claude-3-haiku-20240307-v1:0. Pinning one model was short-sighted: Anthropic ships new Haiku-class models on its own cadence, and the tier is what this decision actually rests on. Naming the tier keeps the decision stable while letting the deployed model track the current release.

Rule: use the newest generally-available Haiku-class model on Bedrock, subject to the model change process below.

Consequences

The deployed model ID lives in configuration as Settings.llm_model_id, not in this ADR. A switch is a settings change plus the IAM grant described under Invocation mode; the LLMProvider abstraction (RAG002) means the query pipeline is unaffected either way.

Process for changing the deployed model:

1. Compare the candidate’s answer quality to the current baseline over a fixed question set
   • The operator makes the call, using this comparison as evidence, since RAG014’s harness is a report only, not a gate.
2. Apply to QA and confirm the post-deploy smoke test.
   • Gotcha: The smoke test checks liveness only. A new model may route to a region outside the IAM grant, and routing is nondeterministic, so a passing smoke test can be a false positive. Compare aws bedrock get-inference-profile against the granted ARNs before the Prod apply.
3. Finally, apply to Prod.

Invocation mode: inference profiles

Bedrock exposes two ways to name a model on bedrock:InvokeModel:

1. foundation-model ID (anthropic.claude-3-haiku-20240307-v1:0)
2. inference profile ID with a region-group (us.anthropic.claude-haiku-4-5-20251001-v1:0). An inference profile routes requests across the regions in its group.

Newer Haiku-class models are inference-profile only. Two consequences for the IAM policy:

• The grant needs both ARN forms: arn:aws:bedrock:<region>:<account>:inference-profile/<profile-id> for the profile itself, and arn:aws:bedrock:<region>::foundation-model/<model-id> for every region the profile can route to.
• The profile ID has a region-group prefix; the foundation model ID does not. Code deriving one from the other must strip or add the region-group prefix.

Treat inference-profile invocation as the default for this decision. A future Haiku-class model that is also available as a plain foundation model does not change anything — the profile path keeps working.

Notes

Generation cost per query assumes ~2,000 input tokens (system prompt + top-5 retrieved chunks + question) and ~300 output tokens. Embedding cost (Titan) is ~$0.01/1k queries regardless of model choice and is excluded from the generation comparison below.

Generation cost by model and volume

Bedrock rows use current Bedrock pricing as of 2026-08-06.

Data residency or privacy requirements may justify self-hosting independently of cost.

Model	Provider	1k queries/mo	10k queries/mo	100k queries/mo	Notes
Haiku-class — current release (rates as of 2026-08-06)	Bedrock	~$3.50	~$35	~$350	$1/M input · $5/M output; the tier, not a pinned ID — re-derive on any model change
Sonnet-class — Sonnet 5 (promo)	Bedrock	~$7	~$70	~$700	$2/M input · $10/M output; promotional through 2026-08-31
Sonnet-class — Sonnet 5 (standard)	Bedrock	~$10.50	~$105	~$1,050	$3/M input · $15/M output; takes effect 2026-09-01
Llama 3 70B (self-hosted)	EC2 ml.g5.2xlarge	~$875 (fixed)	~$875 (fixed)	~$875 (fixed)	Fixed compute; never cost-effective at this instance size

Update notes

• This is derived from Bedrock list prices and goes stale on its own. Check it whenever the deployed model changes and whenever Bedrock prices change (not required; operator judgment). Update the table for significant changes.
• Bedrock is partner-operated and priced separately from Anthropic’s first-party API — do not substitute first-party rates.

Archived items:

• OpenAI rows are the 2026-05-29 figures the decision was made against and are tagged as such; they were rejected on vendor-dependency grounds rather than price, so they have not been re-quoted.

Model	Provider	1k queries/mo	10k queries/mo	100k queries/mo	Notes
Claude 3.5 Sonnet (legacy)	Bedrock	~$21	~$210	~$2,100	$6/M input · $30/M output, the Public Extended Access rate that replaced $3/M · $15/M in December 2025
GPT-4o	OpenAI API	~$8	~$80	~$800	$2.50/M input · $10/M output (2026-05-29 rates); adds OpenAI dependency
GPT-4o mini	OpenAI API	~$0.50	~$5	~$48	$0.15/M input · $0.60/M output (2026-05-29 rates)

For the historical record: the 2026-05-29 decision compared Claude 3 Haiku at $0.25/M · $1.25/M against a Sonnet row quoted at $3/M · $15/M, giving the 12× that justified the choice. All ratios here are computed per-query using 2,000 input + 300 output tokens: Claude 3 Haiku is $0.000875/query, the quoted Sonnet rate $0.0105 (12×). Against the Bedrock rate actually in force that day, $6/M · $30/M, Sonnet is $0.021/query and the ratio was 24×.

Amendment log

On and about August 2026 — De-pinned from anthropic.claude-3-haiku-20240307-v1:0 to the Haiku-class tier, documented the inference-profile invocation requirement, re-priced the cost table, and added a model change process. Current state removed from the ADR. The model change precondition now describes the eval harness as a report, matching RAG014 (+ consequences). Add cost table re-derivation trigger to Consequences. Reordered for readability.