dataMinds News Round up – October 2021
Luis Soares walks you through the process of how to use Azure Synapse to build a pipeline that gets public data from YouTube using REST API’s and store it in Parquet files in ADLS Gen2 and Azure Synapse Analytics. At the end, you should be able to connect to other social media platforms that support REST API’s.
Luis will also have a look at different ways to analyze the data in Azure Synapse Analytics: using serverless SQL, dedicated SQL, and Apache Spark for Synapse.
For every Azure Function resource you create, log data is automatically being collected in Application Insights (AppInsights).
This means that we can use KQL and create custom queries to analyze the log data using Kqlmagic in Jupyter Notebooks in Azure Data Studio. Julie Koesmarno‘s notebook analyzes response times, performance, request trend and geo source of the requests. Learn how to do it yourself!
Azure Data Factory
There is often a need to send notifications during or after job execution. It helps with proactive alerting and reduces the need for reactive monitoring to find out the issues. Many enterprises are leveraging Microsoft Teams for collaboration; this helps you easily integrate yet another critical thing: proactive pipeline alerts into Microsoft teams.
Abhishek Narain elaborates on how to send a notification on a teams channel from an Data Factory pipeline.
Azure Databricks is a modern data engineering as well as data science platform that can be used for processing a variety of data workloads. When moving data to and from Azure Databricks, data pipelines are required to move this data. Though Azure Databricks itself provides several options to import and export data, to build enterprise-scale integration with Azure Databricks, tools and technology that specialize in building enterprise-grade and scale of data pipelines is required.
Gauri Mahajan elaborates on how to establish a successful connectivity from Azure Data Factory to the Azure Databricks platform.
Times series reversals often signal when an action becomes appropriate. At the start of a rainy season, it is a good idea for a retail store to keep umbrellas and rain jackets in stock. Another example is for electric utilities to turn on or off swing capacity generators depending on the demand during a time of day or a season of the year. This course of action can help to provide inexpensive rates to utility rate payers.
Rick Dobson shows how to model time series trends and reversals with SQL Server T-SQL and the use of logs by presenting a framework with the models about when to initiate actions based on reversals in time series.
Power BI Premium recently released a newer version of Power BI Premium, Power BI Premium Generation 2, referred to as Premium Gen2 for convenience. The next generation of Power BI Premium is now generally available (GA) as of October 4, 2021. Enhanced performance, no more limits on refresh concurrency & the ability to autoscale capacity nodes? Dawn Clement will recap the newest features and functionality available with this release.
What if we need to export and email pdf versions of reports which has been created with role level security (RLS) applied? How do we go about scheduling the export and email of these reports to each individual role as applied in the RLS?
Since RLS are applied to reports to create security on the report based on the roles of the end users, it is essential to create a solution that maintains the integrity of the RLS as applied on the Power BI reports. Kenneth A. Omorodion demonstrates his fine solution using Power Automate.