ML Algorithms on Data bricks & Spark Jobs on Azure Batch
07/03/2019 - Element61
Buro & Design Center Esplanade 1, bus 96
How can we use Azure Databricks and Azure data Factory to train our ML algorithms?
This session will show how we can use Azure Databricks as a fully managed service by Microsoft to train and tune our machine learning models. Additionally, I will demonstrate how Data Factory can be used to automate & schedule the whole process of starting and stopping a cluster, training the model and saving it to a blob storage.
During the session I will cover the following subjects:
- What is Azure Databricks?
- What is Azure Data Factory?
- How can we use them together to train our ML models?
The presentation will be followed by a short demo.
Run Spark Jobs on Azure Batch using Azure Container Registry and Blob storage
In this session we will dive into how to use Azure Batch and Azure Container Registry to run containerised Spark jobs. This is useful in typical scenarios where you have many different data sources coming in on a daily basis. Azure Batch automatically scales the number of VMs needed to process all jobs, and automatically shuts down when all jobs are completed. You can use Azure Low Prio VMs in Azure Batch resulting in very low costs for data-intensive workloads.
Speakers
Ivana is part of the Data Science and Strategy competence center at element61, where she works as a Data Scientist. At element61, we help organizations to build and implement data-driven methods of running their business. Ivana holds a master’s degree in AI from KU Leuven.
Kris is the founder and CEO of Data Minded where he works with large enterprises, government agencies and startups. Data Minded builds modern data platforms, and thus has a strong focus on data engineering and often guides its clients to the public cloud. Before starting Data Minded, Kris was a technology consultant at McKinsey&Company. Kris holds a masters degree from University of Hasselt.
Agenda
| 18u00 | Welcome + Food & Drinks |
| 18u30 | How can we use Azure Databricks and Azure Data Factory to train our ML algorithms? – Ivana Pejeva |
| 19u30 | Break |
| 19u40 | Run Spark Jobs on Azure Batch using Azure Container Registry and Blob storage – Kris Peeters |
| 20u40 | Session Ends + Networking |
| 21u00 | Event Ends |
Location
element61
Moore Stephens, Buro & Design Center
Esplanade 1, bus 96
1020 Brussels