Double Databricks Demo of Doom
10/06/2020
Simon Whiteley (Advancing Analytics)
Gentle Dive into Azure Databricks
Azure DataBricks brings a PaaS offering of Apache Spark, which allows for blazing fast data processing, interactive querying and hosting of ML models all in one place! Most of the buzz is around Data Science & AI – what about the humble data engineer who wants to harness the in-memory processing power within their ETL pipelines?
This session focuses on Azure DataBricks as your data ingestion, transformation and curation tool of choice.
We will:
- Introduce the DataBricks service & language options available
- Discuss the hosting & compute options available
- Demonstrate a sample data processing task
- Compare against alternative approaches using SSIS, U-SQL and HDInsight
- Demonstrate pipeline management & orchestration
- Review the wider architectures and extension patterns
The session is aimed at Data Engineers seeking to put the Azure DataBricks technology in the right context and learn how to use the service, with a little dabble in Python to get you started.
Anna-Maria Wykes (Advancing Analytics)
Scala for Big Data, the Big Picture
An opportunity to explore Scala, and why it is truly a “Data Engineers language”.
Scala can be a daunting language to learn, especially if you’re want to take full advantage of the Functional Programming paradigm. However, with the right foundation Scala can prove to be an invaluable tool in a Data Engineers/Scientists belt. In this session will be looking at bitesize basics of Scala, and real-world examples taken from my day job. Examples will be orchestrated using Azure Functions, Azure Data Factory, Azure Data Lake Gen2 and Databricks.
Speakers
Director of Engineering for Advancing Analytics Ltd and Microsoft Data Platform MVP. Simon is a seasoned solution architect & technical lead with well over a decade of Microsoft Analytics experience.
A deep techie with a focus on emerging cloud technologies and applying “big data” thinking to traditional analytics problems, Simon also has a passion for bringing it back to the high level and making sense of the bigger picture. Quiz him about anything from databricks, data factory, azure synapse and more. When not tinkering with tech, Simon is a death-dodging London cyclist, a sampler of craft beers, an avid chef and a generally nerdy person.
Anna is a veteran software & data engineer, with over 14 years of experience. She’s tackled projects from real-time analytics with Scala & Kafka, building out Data Lakes with spark and applying engineering to Data Science. Anna is a senior consultant with Advancing Analytics, helping shape & evolve their data engineering practice. She has a real passion for data and strives to bring the worlds of Software Development and Data Science closer together. Other areas of interest include UX, Agile methodologies, and helping to organize/run local Code Clubs
Agenda
| 18u30 | Welcome and introductions |
| 18u45 | (Not so) Gentle Dive into Azure Databricks (60 minutes) |
| 20u00 | Scala for Big Data, the Big Picture (60 minutes) |
| 21u00 | Sessions end |
A 15 minute break will be held in between sessions.
Location
Virtual Meeting (LINK)