Data Governance Meets Data Engineering: Unlocking Microsoft Purview's Potential & We’re Migrating to Microsoft Fabric! But Can dbt Save Us From Notebook Overload? (IN-PERSON @ Datashift)

Data Governance Meets Data Engineering: Unlocking Microsoft Purview’s Potential 

Speaker: Tom Van Breuseghem

In this session, we’ll dive into the exciting possibilities of the new Microsoft Purview Portal, showcasing its advanced capabilities in data governance. From creating data catalogs and maintaining business glossaries to assigning roles like data stewards and owners, Purview sets the foundation for systematic governance, collaboration, and compliance.

But what happens when the journey demands more? We’ll explore a real-world case where Purview’s limitations in automated data lineage for Spark environments presented a challenge. To bridge this gap, we leveraged SparkLin, an open-source tool that integrates seamlessly with Purview to provide detailed, automated data lineage for Azure Synapse Spark notebooks.

Join us to learn how this powerful combination enabled one of our clients to achieve a unified data governance framework. We’ll demonstrate how Purview and SparkLin work together to ensure transparency, traceability, and efficiency in complex data environments. Whether you’re a data governance leader or an engineering enthusiast, this session will show you how data governance meets data engineering to unlock the full potential of your data landscape

We’re Migrating to Azure Fabric! But Can dbt Save Us From Notebook Overload? 

Speakers: Senne Vanstraelen & Tom Van de Velde

Yay, you’re migrating to Azure Fabric! But now comes the hard part: How do you build scalable, maintainable workflows without falling into the dreaded spaghetti notebook trap? Could dbt (Data Build Tool) be the answer? Or is a hybrid approach with notebooks the sweet spot? This session dives deep into the pros and cons of both approaches, giving you the insights you need to make informed decisions for your Fabric journey. 

Here’s what you’ll learn: 

• Notebooks vs. dbt for Azure Fabric: When to stick with notebooks and when dbt brings the structure you need.
• Lakehouse vs. Warehouse: How dbt and Fabric fit into both models and what works best for your use case.
• Organizing your workflows: Best practices for managing notebooks, SQL scripts, and pipelines in Fabric and/or dbt.
• Lineage and testing: A comparison of what dbt offers versus Fabric’s native capabilities.
• Optimizing Fabric migrations: Tips to streamline your migration process, including a look at dbt’s role in reducing complexity.
• A live setup and demo: See how it all comes together with a hands-on example that ties concepts to real-world implementation.

This session is perfect for anyone embarking on their Azure Fabric migration journey, whether you’re a notebook loyalist, a dbt enthusiast, or just trying to find the best path forward. By the end, you’ll have a clear understanding of how these tools compare, when to use them, and how to avoid common pitfalls—all while keeping your workflows clean, scalable, and efficient. Join us to explore the possibilities—without the chaos! 

Speakers

Register now!