Building a Low-Cost, High-Throughput Data Exchange Platform for Digital Meters on Azure & Migrate your SSIS skills to Azure Data Factory (IN-PERSON @ AE)
05/12/2022 - AE
05/12/2022 - AE
When you are responsible for collecting loads of information from data providers and sending it out swiftly to data subscribers, how would you design and architect the central data exchange platform? How do you keep costs low but still receive, process, and send all this data in a timely fashion?
On Microsoft Azure, Function apps and Cosmos DB will give you the operational scalability you need. Azure Storage Accounts can hold enormous amounts of data safely but cheaply for a long time. And Azure Synapse can transform your basic data and store it in other formats optimized to serve their use cases very efficiently.
This session will give an overview of the design and the challenges we faced when building a data exchange platform for digital meters and how we tackled them to be ready to redistribute data, generated daily from 10 million devices.
Integration Services is the new Cobol. Well at least according to some people, but there might be some truth to that statement. Azure Data Factory (ADF) is now looked at as the replacement of SSIS in the cloud.
In this session, we will take a look at how you can leverage your SSIS skills in ADF. We will also look at some patterns that were valid for SSIS but are a bad idea in ADF, or the other way around. At the end, you should feel confident to take up new projects in ADF.
The target audience of this session are BI/data warehouse developers & data engineers who have experience with SSIS (or other ETL tools), but little to no experience with ADF.
Bart Jolling is an experienced software architect since a few years specializing in creating solutions on the Microsoft Azure Cloud platform. He graduated in 1999 from the KU Leuven with a master’s degree in architectural engineering (the building kind). From there, he started as a software developer for a large financial institution, became a Microsoft Certified Professional Developer and put the foundations for his current role as Cloud Solution Architect.
When not helping clients getting started on the cloud, Bart likes to spend time with his family and friends, playing Minecraft with his children, going jogging with his girlfriend, and playing board games.
Koen Verbeeck is a data professional working at AE. He helps organizations to get insight in their data and to improve their analytics solutions. Koen has over a decade of experience in developing data warehouses, models and reports using the Microsoft data platform . Since 2017 Koen is a Microsoft Data Platform MVP. He has a blog at http://www.sqlkover.com, writes articles for MSSQLTips.com and is a frequent speaker at SQL Server events.
|18u00||Welcome + Food & Drinks|
|18u30||Building a Low-Cost, High-Throughput Data Exchange Platform for Digital Meters on Azure (60 minutes)|
|19u45||Migrate your SSIS skills to Azure Data Factory (60 minutes)|
(oude Philips gebouwen)