Advanced Machine Learning with Databricks

This course is designed for data scientists and machine learning practitioners seeking to scale machine learning workflows and implement MLOps best practices using Databricks. The course is delivered over two four-hour modules, covering Apache Spark for ML, hyperparameter tuning with Optuna, and MLOps automation with Databricks tools such as Databricks Asset Bundles (DABs), Workflows, and Mosaic AI Model Serving.

Participants will gain hands-on experience with Spark ML, pandas APIs on Spark, MLflow, and Unity Catalog, ensuring effective model tracking, governance, and deployment.

By the end of this course, learners will be able to:

• Explain Apache Spark’s architecture and its role in scalable machine learning.
• Develop ML models using Spark ML and pandas APIs on Spark.
• Perform hyperparameter tuning with Optuna on Spark.
• Leverage MLflow and Unity Catalog for model tracking, packaging, and governance.
• Implement MLOps best practices, including CI/CD, pipeline management, and environment separation.
• Deploy and monitor ML models with Databricks Asset Bundles (DABs), Workflows, and Mosaic AI Model Serving.
• Use model rollout strategies, A/B testing, and drift detection for production ML.

Participants should have:

• Basic knowledge of data science and machine learning concepts (e.g., classification and regression models).
• Familiarity with common ML evaluation metrics (e.g., F1-score).
• Experience with Python and ML libraries (e.g., scikit-learn, XGBoost).
• Intermediate-level knowledge of ML development and the use of Git for ML projects.

If you do not have one or more of the pre-requisites QA recommends:

• QADSMLP - Data Science and Machine Learning with Python
• DBASP - Apache Spark Programming with Databricks

Target Audience

This course is designed for:

• Data scientists and ML engineers who want to scale machine learning workflows with Databricks.
• MLOps practitioners aiming to streamline ML lifecycle management, testing, and deployment.
• AI/ML professionals implementing CI/CD, model monitoring, and production-ready ML systems.

Machine Learning at Scale

• Machine learning development with Spark
  • Overview of Spark architecture for machine learning workloads.
  • Introduction to Spark ML for model development.
  • Model tracking and packaging with MLflow and Unity Catalog.
• Model tuning with Optuna on Spark
  • Overview of hyperparameter tuning techniques.
  • Introduction to Optuna on Spark for automated model optimization.
  • Scaling Optuna with Spark for distributed tuning.

Advanced Machine Learning Operations

• Overview of MLOps on Databricks
  • Review of MLOps concepts and industry best practices.
  • Streamlining development to deployment in Databricks.
• Continuous workflows for MLOps
  • Implementing automated testing with Databricks.
  • CI/CD for ML models, including versioning and governance.
• Model rollout strategies and monitoring
  • A/B testing and shadow deployments.
  • Model quality monitoring using Lakehouse monitoring.
  • Drift detection and custom model performance metrics.
• Scaling deployments with Databricks Asset Bundles (DABs)
  • Deploying ML models as code with DABs.
  • Ensuring multiple environment consistency for production ML.

Exams and Assessments

This course does not include formal assessments.

Hands-On Learning

This course includes:

• Hands-on labs with Spark ML, Optuna tuning, and MLflow tracking.
• Practical exercises on CI/CD pipelines, model packaging, and governance.
• Instructor-led demonstrations of model deployment and monitoring techniques.