QA plan: Corpus ingestion — load and chunk (#8)

Story: Load neural-bridge/rag-dataset-12000 and chunk (#8)
Capability: Corpus ingestion & embedding (#7)
Spec: specs/ingestion.md

Test scope

Type	Coverage
Unit	Chunker logic: window size, overlap, edge cases (empty input, input shorter than window, unicode boundaries)
Unit	Loader: Parquet parse, field mapping, row count enforcement for --limit
Integration	rag-ingest load --limit 100 against Docker Postgres writes expected document and chunk counts
Integration	Re-run is idempotent: second load produces same row counts, no duplicates
Contract	None (CLI tool, no API surface)
e2e	Out of scope for this story; covered in corpus embed-upsert plan (#9) via rag-ingest all
Performance	Not required at this stage; 12k-row corpus completes in under 5 minutes is a soft acceptance gate
Security	Trust boundary B6 (Hugging Face upstream): loader must not execute retrieved content; test that no eval/exec paths exist on the Parquet parse output

Entry criteria

• Docker Compose Postgres with pgvector running (Story #10)
• rag-ingest CLI entrypoint installed (pip install -e ".[ingestion]")
• HF Hub credentials or the Parquet file available (local cache acceptable)

Exit criteria

• All unit tests pass; chunker.py and loader.py each exceed the 70% per-file floor
• Integration test: rag-ingest load --limit 100 inserts exactly 100 documents and the expected chunk count (chunk count = sum of ceil(doc_len / (size - overlap)) per spec defaults: 1000 chars, 150 overlap)
• Idempotency test: second load run produces zero new rows (ON CONFLICT behavior verified)
• Aggregate coverage over rag_sample/ingestion/ stays at or above 90% after this story’s tests are merged (hard-fail gate in RAG006)

Environments

Unit and integration tests run locally and in CI against Docker Postgres. No AWS access required for this story.

Tooling

pytest, pytest-cov. Parquet seed fixture at tests/fixtures/corpus_seed.parquet (50-row slice); tests use this rather than calling HF Hub live.

Fixtures and data

• tests/fixtures/corpus_seed.parquet — 50-row slice of neural-bridge/rag-dataset-12000; committed to the repo
• postgres_db pytest fixture (shared): Docker Postgres with pgvector, migrations applied

How tests are written

def test_chunk_overlap_preserved():
    # Given a document longer than two windows
    text = "a" * 2500
    # When chunked with size=1000, overlap=150
    chunks = chunk_text(text, size=1000, overlap=150)
    # Then adjacent chunks share the overlap region
    assert chunks[0][-150:] == chunks[1][:150]

Given leads setup, When leads the action, Then leads assertions throughout.

Risks

• HF Hub rate limits or dataset schema changes break the live integration path. Mitigated by the parquet seed fixture for CI; live HF call is manual smoke only.
• Unicode multibyte characters cause chunk boundary to split a codepoint. Add a unicode boundary test case.

Capability acceptance criteria link

Spec acceptance: rag-ingest load --limit 100 completes without error and writes expected document and chunk counts. This plan’s integration test directly verifies this criterion.