003. Corpus ingestion: lean Parquet via huggingface-hub and pyarrow

Date: 2026-05-25
Status: Accepted
Builds on: RAG002. Backend application stack and tooling

Decision

In the context of a one-time ingestion of the bounded corpus neural-bridge/rag-dataset-12000 (Apache-2.0) into pgvector,
facing that Hugging Face’s datasets library pulls a heavy dependency tree (pyarrow, pandas, fsspec, and more) that is slow to install on WSL’s /mnt/c filesystem, for a load that needs no streaming, splitting, or map/filter machinery,
we decided for downloading the dataset’s Parquet files directly with huggingface-hub (list_repo_files + hf_hub_download) and reading them with pyarrow,
and neglected
- the datasets library — ergonomic loading, streaming, and transforms, but a far larger dependency tree than a one-time bounded load justifies,
- direct HTTP download of the Parquet files — works, but reinvents the repo-file resolution and local caching that huggingface-hub already provides,
to achieve
- a minimal ingestion dependency footprint (essentially pyarrow plus huggingface-hub),
- faster installs, especially on /mnt/c,
- a simple, deterministic, resumable load that discovers Parquet files at runtime rather than hard-coding names,
accepting
- we forgo datasets’ conveniences, so a future move to a larger or multi-split dataset, or to streaming ingestion, may need to revisit this.

Consequences

The ingestion optional-dependency group is huggingface-hub + pyarrow instead of datasets. The loader lists the dataset repo’s .parquet files, downloads them through the hub cache, and reads rows with pyarrow, mapping the context field to document content and keeping question/answer as metadata. Because the corpus is small and bounded, this stays well within memory and runs in a single pass. If requirements later demand streaming or richer dataset transforms, reintroducing datasets is a contained change behind the loader.