Help! Why is my Azure-bill growing!?

15/02/2023

When you are loading data within Azure using Synapse Analytics you have multiple options; Integration Pipelines, Mapping dataflows and Notebooks within Spark. With the use of the Azure Cost monitor you can get an idea of the costs that you are making on resource-level, but often this is not granular enough.

Since the only overview is on resource-level, you cannot determine which notebook or integration-pipeline led to which costs. So when someone from the business comes by and asks you, “How much does it cost to load data from source x into our datalake?”. It is hard to give an accurate answer. This is because costs are fragmented trough various resources and multiple solutions make use of the same resources.

The lack of this information can be a big deal in various cases, for instance when costs have grown over time and you are exceeding your budget. Do you know where you need to make a cut and make your stakeholders happy again?

Within this session I will show you how we cope with this challenge and how we made a solution which allows us to get better, more granular insights. As well I will give you an overview of how fragmented costs can be within your Azure Data Platform.

Speakers

Rohan Horstman

Sr. Data Engineer at VGZ

Rohan Horstman is Senior Data Engineer within VGZ, a Dutch health insurance company.
In the past 7 years Rohan had a focus on the Microsoft BI-stack, starting on-premises, but nowadays working with the data products Microsoft has to offer on Azure.
After working at various customers as a consultant, Rohan is now focusing on building the new Data Platform for VGZ in the Azure cloud.
He likes to experiment with all the new possibilities Azure has to offer and always tries to get the most out. Resulting in solutions that have a big need but aren’t always available out-of-the-box.