Monitoring, Logging and Cost Management in Azure Databricks
Built-in monitoring, diagnostic logging, KQL, Spark performance analysis and cost optimization levers.
Level: Intermediate / Advanced | Platform: Azure Databricks + Azure Monitor
Sign in to read this course
A free account unlocks all 514 courses. 20 are readable without one.
What's inside
23 sections- 1 Table of Contents
- 2 Why Monitor Databricks?
- 3 Built-in Monitoring Tools
- 4 Logging Architecture
- 5 Configuring Diagnostic Settings
- 6 Azure Monitor and Log Analytics
- 7 KQL Queries for Databricks
- 8 Alerts and Notifications
- 9 Spark Job Performance Analysis
- 10 Common Bottlenecks and Solutions
- 11 Autoscaling and Instance Pools — Optimization
- 12 Spark Configuration Tuning
- 13 Cost Management — Levers
- 14 Spot Instances and Auto-Termination
- 15 Azure Cost Management — Budgets and Alerts
- 16 Databricks Usage Dashboard
- 17 Delta Lake Storage Optimization
- 18 Automated Reporting
- 19 Event-Driven Monitoring with Azure Functions
- 20 Automated Resource Cleanup
- 21 Dashboards with Azure Synapse / Power BI
- 22 Summary and Best Practices
- 23 Glossary
More Azure Databricks & Spark courses
View all 14Administering Clusters and Configuring Policies with Databricks
Databricks architecture, cluster types and runtimes, autoscaling, cluster policies, pools and init scripts.
ETL Pipelines with Azure Databricks and Data Factory
Build ETL with Spark and PySpark, Unity Catalog governance, Delta Lake and Databricks vs Data Factory.
Manage Data with Azure Databricks and Azure Data Lake
Connect Databricks to ADLS Gen2 securely, ingest with Auto Loader and govern with Unity Catalog.
Optimize Storage and Performance with Delta Lake
Delta Lake internals, ACID, OPTIMIZE, Z-Order, liquid clustering, caching and Photon acceleration.
Real-Time Data Processing with Azure Databricks
Spark Structured Streaming with Event Hubs — windowing, stateful processing and real-time anomaly detection.
Machine Learning with Azure Databricks
The Databricks ML lifecycle: MLflow tracking, tuning with Ray, the model registry, serving and AutoML.
Interested in this course?
Contact us to book it or get a custom training plan for your team.