019. Runtime-tunable pipeline parameters: per-request overrides plus DB presets

• Date: 2026-07-22
• Status: Accepted

Decision

• In the context of exposing RAG pipeline parameters (retrieval score threshold, embedding and LLM model selection, pipeline mode such as plain RAG vs agentic RAG, and on/off toggles for individual pipeline elements) for live, interactive demonstration and tuning from the SPA (capability #251),
• facing the need to let viewers experiment per request without a redeploy, and without a single shared configuration where one viewer’s change alters what every other viewer sees,
• we decided for exposing runtime-tunable pipeline parameters as per-request overrides on the query API (each optional field falling back to the config.py default when omitted), plus a DB-backed set of named presets the UI can load as starting positions,
• and neglected
  • per-deploy infrastructure and environment configuration (a Terraform-managed Lambda env var), because a value meant to be live-tuned must not require a deploy to change;
  • a single mutable DB-global configuration row, because it makes all concurrent viewers share one state and turns one viewer’s experiment into everyone’s,
• to achieve stateless, multi-viewer isolation, where each request carries its own knobs, together with shareable and persistable presets for curated starting points,
• accepting a wider /query API contract (more optional request fields, each needing validation and bounds) and the added surface of a presets table with its own create/read/update/delete (CRUD) API and validation.

Consequences

• The /query request model gains optional override fields; top_k is the existing precedent this generalizes. Each new field (score_threshold, model_id, mode, element toggles) needs explicit validation and safe bounds so a caller cannot request abusive or unsafe values. The committed OpenAPI spec must be updated in step (RAG005).
• config.py values become the fallback default layer, applied when neither a per-request override nor a loaded preset supplies a value. The first such lever, retrieval_score_threshold, defaults to 0.35 in config.py. It was originally 0.7, which proved unreachable against Titan question-to-passage scores and made /query always return no_relevant_results.
• A presets store is introduced: a small table plus a CRUD (create/read/update/delete) API. Its data access follows the RDS Data API contract (RAG011); it is application data, not per-environment infra state.
• Exposing model selection as a lever means the deployed API Lambda’s IAM role must grant Bedrock access to every selectable model, not only the RAG007 default (the current Haiku-class model). Widening that set is a security-review touchpoint. Each selectable model needs two ARN forms granted.
• The mode lever (plain RAG vs agentic RAG) names a whole alternate pipeline, not a simple flag; its implementation is expected to be its own capability or epic rather than a single toggle.
• Removing the Terraform env-var path means no infra change is needed to tune these values in QA or Prod. The trade-off is that a genuinely operational, non-demo parameter, if one arises later, would need a deliberate carve-out rather than defaulting to this per-request model.

Notes

Relates to prior decisions without superseding them: RAG007 fixed the Haiku-class tier as the default, and this decision keeps it as the default while making model choice a per-request lever; RAG011 (RDS Data API as the sole network path) governs the presets-store data access; RAG005 (OpenAPI spec governance) requires the widened /query contract to be reflected in the committed spec.

Amendment log

2026-08-06 — Editorial only, no change to the decision: renamed Claude 3 Haiku to the Haiku-class, following RAG007’s de-pinning, restated the retrieval_score_threshold default as 0.35, and carried RAG007’s inference-profile consequence into the model-selection lever bullet. Recorded under #171.