ETL Pipelines with Azure Databricks and Data Factory
Build ETL with Spark and PySpark, Unity Catalog governance, Delta Lake and Databricks vs Data Factory.
Course: Build and Run ETL Pipelines with Azure Databricks and Azure Data Factory Level: Intermediate / Advanced | Platform: Azure Databricks + Azure Data Factory
Sign in to read this course
A free account unlocks all 514 courses. 20 are readable without one.
What's inside
36 sections- 1 Table of Contents
- 2 Introduction to ETL and Apache Spark
- 3 Azure Databricks Architecture
- 4 Unity Catalog — Centralized Governance
- 5 Databricks vs Azure Data Factory
- 6 Environment Setup
- 7 Connecting to Azure Data Lake Storage from Databricks
- 8 Apache Spark DataFrames — Fundamentals
- 9 Schema Definition
- 10 Data Analysis and Cleaning
- 11 Business Transformations with PySpark
- 12 SQL Queries on DataFrames
- 13 Handling Corrupted Data
- 14 Delta Lake — Foundations and Architecture
- 15 Writing to Data Lake and Delta Tables
- 16 DML Operations on Delta Tables
- 17 Delta Lake Performance Optimizations
- 18 Delta Lake Auto-Optimization
- 19 Automation with Databricks Workflows
- 20 Parameterizing Notebooks with Widgets
- 21 Task Values and Dependencies Between Tasks
- 22 Triggers and Job Automation
- 23 Git Integration with Databricks
- 24 Orchestration with Azure Data Factory
- 25 Invoking Databricks from Data Factory
- 26 Automating ADF Pipelines with Triggers
- 27 Advanced ETL Architecture Patterns
- 28 Summary and Best Practices
- 29 Glossary
- 30 Module 2 – Transforming Data with PySpark
- 31 Module 3 – Delta Lake
- 32 Module 4 – Delta Performance Optimization
- 33 Module 5 – Automating Pipelines (Databricks Workflows)
- 34 Module 6 – Orchestration with Azure Data Factory
- 35 General Best Practices
- 36 Module 7 – ETL Architecture with Databricks: Medallion Architecture
More Azure Databricks & Spark courses
View all 14Administering Clusters and Configuring Policies with Databricks
Databricks architecture, cluster types and runtimes, autoscaling, cluster policies, pools and init scripts.
Manage Data with Azure Databricks and Azure Data Lake
Connect Databricks to ADLS Gen2 securely, ingest with Auto Loader and govern with Unity Catalog.
Optimize Storage and Performance with Delta Lake
Delta Lake internals, ACID, OPTIMIZE, Z-Order, liquid clustering, caching and Photon acceleration.
Real-Time Data Processing with Azure Databricks
Spark Structured Streaming with Event Hubs — windowing, stateful processing and real-time anomaly detection.
Machine Learning with Azure Databricks
The Databricks ML lifecycle: MLflow tracking, tuning with Ray, the model registry, serving and AutoML.
Building Deep Learning Models on Databricks
Build, train, tune and serve deep-learning models on Databricks with TensorBoard integration.
Interested in this course?
Contact us to book it or get a custom training plan for your team.