Skill

Apache Parquet & Arrow

Columnar file and memory formats — the storage layer for my event histories, read with DuckDB and PyArrow.

  • #data

Apache Parquet is a columnar file format and Apache Arrow its in-memory counterpart; together they make scanning large datasets fast and compact.

How I use it

In Real-time Data Systems, incoming events are normalized into Arrow and persisted as Parquet, then queried with DuckDB and loaded with PyArrow for Python research jobs. Datasets are versioned on Hugging Face so cloud jobs on AWS and GPU sessions on Google Colab read exactly the same data.

Why it matters

Columnar storage turns "scan months of history" from a database problem into a file-read problem.

Related