Course overview¶
Data Engineering — hands-on with a governed lakehouse.
Learn the core concepts of Data Engineering — ingestion, the lakehouse, data modelling, transformation at scale, orchestration, governance and reporting — without spending a cent on the cloud. Everything runs on your own machine, so you can experiment, break things and try again for free. Build real expertise here first, then step onto any cloud platform already knowing why things work — not just which buttons to press.
▶ Start with Unit 0 👀 Take the tour 💬 Join the Discord
Course units¶
-
🚀 Unit 0 · Getting started
Set up the stack, meet ShopFlow, and see how it all fits together.
-
🧭 Unit 1 · Foundations
What Data Engineering is, the lakehouse, medallion and data modelling.
-
🔎 Unit 2 · SQL on big data
Query the lake — joins, window functions, CTEs, views, functions, MERGE and JSON.
-
🐍 Unit 3 · Python in notebooks
Python for data work — dataframes, files, APIs and databases.
-
⚡ Unit 4 · Spark
Distributed processing — build Bronze → Silver → Gold, materialized views, table maintenance, and tuning.
-
⏱️ Unit 5 · Orchestration
Turn jobs into scheduled, dependable pipelines.
-
🧱 Unit 6 · Transform with dbt
Build Silver and Gold as SQL models — dependencies, materializations, tests and docs.
-
❄️ Unit 7 · Snowflake (your own trial account)
The same ShopFlow data in a free Snowflake trial — every SnowPro Core topic, practised in Snowflake itself.
-
☁️ Unit 8 · Google Cloud (your own trial account)
How BigQuery, Dataflow, Datastream, Composer, Dataform, IAM, Terraform and Looker work inside; then ShopFlow end to end on Google Cloud (Cloud Storage, BigQuery, Dataflow, a notebook, Composer and a Looker Studio report); then short deep dives for review.
-
🛡️ Unit 9 · Data governance
Catalogs, users, roles and who can see which data.
-
📊 Unit 10 · Business intelligence
Turn the Gold layer into dashboards people use.
-
🧰 Unit 11 · Real-world recipes
Event triggers, SCD2, data quality and reverse-ETL.
-
🏁 Unit 12 · Capstones
Prove it end to end — two projects, easy and medium.
-
🏛️ Unit 13 · Data Vault 2.0
Integrate ShopFlow's shop, CRM and marketplace into hubs, links and satellites — business keys, hash keys, insert-only history, PIT tables, business rules and marts on top.
-
🌉 Unit 14 · Bridge to the platforms
Map every skill to Databricks, Snowflake, Fabric and Azure.
Help & community¶
-
💬 Join the Discord
Ask questions, get unstuck, share what you built and help others.
-
🎓 Learn with us
Instructor-led Data Engineering training for students and career-changers.
-
🏢 Hire our team
Data Engineering talent, or a team to deliver your data platform.
All the ways to reach us are on the Help & community page.
How each lesson works¶
Concept → Lab on your own stack → Challenge (solutions hidden — try first!) → You can now… — the concrete skills you've gained.