QA plan: Corpus ingestion — embed and upsert (#9)

Story: Embed via Bedrock Titan, upsert to pgvector (idempotent) (#9)
Capability: Corpus ingestion & embedding (#7)
Spec: specs/ingestion.md

Test scope

Type	Coverage
Unit	EmbeddingProvider protocol stub returns a 1024-dim float vector; pipeline calls the provider and writes to chunks.embedding
Unit	Only chunks with embedding IS NULL are passed to the provider (selective embed)
Unit	ON CONFLICT upsert: re-embedding a chunk updates the row rather than inserting a duplicate
Integration	Against Docker Postgres: pipeline inserts seed chunks, runs embed with FakeEmbeddingProvider, verifies embedding column is non-null and vector(1024) shape
Integration	rag-ingest all from clean DB produces same final state as sequential load then embed
Contract	EmbeddingProvider protocol: any concrete implementation must accept a list[str] and return a list[list[float]] of length-1024 vectors — tested via a protocol-compliance test that runs against both FakeEmbeddingProvider and BedrockEmbeddingProvider (the latter mocked at the boto3 call site)
e2e	search_similar_chunks against pre-embedded seed returns results; cosine similarity ordering is correct
Security	Bedrock call boundary (B5): boto3 call is parameterized; provider receives text only, no Parquet metadata that could inject instructions

Entry criteria

• Story #8 (load and chunk) merged and passing
• Story #10 (Docker Postgres + schema) merged
• FakeEmbeddingProvider and Bedrock mock in tests/stubs.py

Exit criteria

• Unit tests for pipeline.py pass; file exceeds 70% per-file floor
• Integration test: after embed, all chunks.embedding values are non-null vector(1024)
• Idempotency: second embed run makes zero additional Bedrock calls (confirmed via call-count assertion on the stub)
• Cosine similarity e2e: search_similar_chunks("test question", top_k=3) returns 3 rows ordered by descending similarity
• Aggregate coverage over rag_sample/ingestion/ at or above 90%

Environments

Unit and integration against Docker Postgres. The BedrockEmbeddingProvider live path is excluded from the CI coverage run (boto3 mocked); it is tested as a one-off manual smoke test against QA.

Tooling

pytest, pytest-cov. Bedrock calls mocked with monkeypatch on boto3.client.invoke_model. Pre-embedded seed SQL in tests/fixtures/embeddings_seed.sql.

Fixtures and data

• FakeEmbeddingProvider in tests/stubs.py: returns [[0.0] * 1024] per chunk
• postgres_db fixture with seed data from Story #8
• tests/fixtures/embeddings_seed.sql: inserts 50 rows with known float vectors for similarity tests

How tests are written

def test_embed_skips_already_embedded_chunks(postgres_db, monkeypatch):
    # Given 3 chunks, 2 already have embeddings
    insert_chunks(postgres_db, count=3)
    embed_two(postgres_db)
    call_count = []
    monkeypatch.setattr(FakeEmbeddingProvider, "embed", lambda self, texts: (call_count.append(len(texts)), [[0.0]*1024]*len(texts))[1])
    # When embed runs
    run_embed(postgres_db, FakeEmbeddingProvider())
    # Then only the unembedded chunk was sent to the provider
    assert sum(call_count) == 1

Risks

• Titan embedding API shape changes (dimension or response schema). The protocol test acts as a canary; the mocked shape must be kept in sync with Bedrock’s documented response.
• Large corpus (12k rows) may exceed Lambda memory or timeout if ever run as a Lambda instead of local script. Current design is local-only for embed; document this constraint.

Capability acceptance criteria link

Spec acceptance: rag-ingest embed processes all unembedded chunks and writes vector(1024) values; search_similar_chunks can perform cosine similarity search. Both are direct exit criteria above.