ML Algorithms on Databricks & Spark Jobs on Azure Batch

How can we use Azure Databricks and Azure data Factory to train our ML algorithms?

This session will show how we can use Azure Databricks as a fully managed service by Microsoft to train and tune our machine learning models. Additionally, I will demonstrate how Data Factory can be used to automate & schedule the whole process of starting and stopping a cluster, training the model and saving it to a blob storage.

During the session I will cover the following subjects:

  • What is Azure Databricks?
  • What is Azure Data Factory?
  • How can we use them together to train our ML models?

The presentation will be followed by a short demo.

Run Spark Jobs on Azure Batch using Azure Container Registry and Blob storage

In this session we will dive into how to use Azure Batch and Azure Container Registry to run containerised Spark jobs. This is useful in typical scenarios where you have many different data sources coming in on a daily basis. Azure Batch automatically scales the number of VMs needed to process all jobs, and automatically shuts down when all jobs are completed. You can use Azure Low Prio VMs in Azure Batch resulting in very low costs for data-intensive workloads. 

The Speakers :

Ivana Pejeva

Ivana is part of the Data Science and Strategy competence center at element61, where she works as a Data Scientist. At element61, we help organizations to build and implement data-driven methods of running their business. Ivana holds a master’s degree in AI from KU Leuven.

Kris Peeters

Kris is the founder and CEO of Data Minded where he works with large enterprises, government agencies and startups. Data Minded builds modern data platforms, and thus has a strong focus on data engineering and often guides its clients to the public cloud. Before starting Data Minded, Kris was a technology consultant at McKinsey&Company. Kris holds a masters degree from University of Hasselt. 


18:00 Welcome + Food & Drinks
18:30 How can we use Azure Databricks and Azure Data Factory to train our ML algorithms? - Ivana Pejeva
19:30 Break
19:40 Run Spark Jobs on Azure Batch using Azure Container Registry and Blob storage - Kris Peeters
20:40 Session Ends + Networking
21:00 Event Ends


Moore Stephens, Buro & Design Center Esplanade 1, bus 96 1020 Brussels

Registrations for this event have been closed.


Receive our monthly newsletter