Skip to content

Data Engineering

Notes on data warehouses, ETL/ELT pipelines, and the platform concerns that surround moving and transforming data at scale — a category for engineering knowledge that sits closer to the data platform than to any single application's database.

Articles

  • BigQuery and Modern ETL/ELT — BigQuery as an analytical warehouse, ETL vs. ELT, raw/staging/curated layering, batch vs. streaming vs. CDC ingestion, Dataform, incremental processing, idempotency, orchestration, warehouse modeling, partitioning/clustering, and observability.

Contributing

Learned something new about data pipelines, warehouses, or ETL/ELT tooling? Add it here as its own file. See Contributing for the how-to.