8.1 GCP & BigQuery knowledge¶
This lesson explains how the Google Cloud data services work inside, before you use them in the practical implementation. Each page goes from "what is it and why does it exist" down to the low-level details: architecture, core concepts, what matters most in practice, how it compares to Spark, Snowflake, Fabric and AWS, a small hands-on exercise and a glossary.
The running example is a typical enterprise data warehouse (EDW) modernization on Google Cloud: a near real-time sales model in BigQuery, Looker on top, and the load layer moved into shared ingestion projects, with cost and runtime under control.
| Page | What you learn |
|---|---|
| BigQuery from the inside | Dremel, Colossus, slots, shuffle; pricing; partitioning, clustering; MERGE; cost and performance |
| Data warehouse modelling | Kimball facts and dimensions, grain, SCD2, nested data, a near real-time sales model |
| Pub/Sub and Dataflow | Messaging, Apache Beam, windows, watermarks, exactly-once, autoscaling |
| Datastream and CDC | Log-based change data capture, ordering, merging changes into BigQuery |
| Cloud Composer and Airflow | Airflow architecture, DAG design at scale, Composer environments |
| Dataform and dbt | SQL transformations as code: models, tests, incremental loads |
| GCP platform, IAM and Terraform | Projects and folders, IAM, service accounts, networking, Terraform, CI/CD |
| Looker and the semantic layer | LookML, explores, caching, aggregate awareness, conversational BI |
| Migration and EDW modernization | Inventory, phases, parallel run, reconciliation, cutover |
How to study
Read BigQuery and modelling first: everything else builds on them. Do the small hands-on part of each page; the free BigQuery sandbox is enough for most of it. For a short summary of a topic afterwards, use the deep dives.
Check current docs
Cloud services change quickly. Details marked "check current docs" (limits, default settings, preview features) may differ from what you see in the console.