Learn the correct sequence of steps to retain Azure Data Factory pipeline run data for 120 days and query it using the Kusto query language. Follow this detailed guide to properly configure diagnostic settings, Log Analytics workspace, and data retention for your Azure Data Factory pipelines.
Table of Contents
Question
You have an Azure data factory.
You need to ensure that pipeline-run data is retained for 120 days. The solution must ensure that you can query the data by using the Kusto query language.
Which four actions should you perform in sequence? To answer, move the appropriate actions from the list of actions to the answer area and arrange them in the correct order.
NOTE: More than one order of answer choices is correct. You will receive credit for any of the correct orders you select.
Actions:
- Select the PipelineRuns category.
- Create a Log Analytics workspace that has Data Retention set to 120 days.
- Stream to an Azure event hub.
- Create an Azure Storage account that has a lifecycle policy.
- From the Azure portal, add a diagnostic setting.
- Send the data to a Log Analytics workspace.
- Select the TriggerRuns category.
Answer
- Create an Azure Storage account that has a lifecycle policy.
- Create a Log Analytics workspace that has Data Retention set to 120 days.
- From the Azure portal, add a diagnostic setting.
- Send the data to a Log Analytics workspace.
Explanation
Step 1: Create an Azure Storage account that has a lifecycle policy
To automate common data management tasks, Microsoft created a solution based on Azure Data Factory. The service, Data Lifecycle Management, makes frequently accessed data available and archives or purges other data according to retention policies. Teams across the company use the service to reduce storage costs, improve app performance, and comply with data retention policies.
Step 2: Create a Log Analytics workspace that has Data Retention set to 120 days.
Data Factory stores pipeline-run data for only 45 days. Use Azure Monitor if you want to keep that data for a longer time. With Monitor, you can route diagnostic logs for analysis to multiple different targets, such as a Storage Account: Save your diagnostic logs to a storage account for auditing or manual inspection. You can use the diagnostic settings to specify the retention time in days.
Step 3: From Azure Portal, add a diagnostic setting.
Step 4: Send the data to a log Analytics workspace,
Event Hub: A pipeline that transfers events from services to Azure Data Explorer.
Keeping Azure Data Factory metrics and pipeline-run data.
Configure diagnostic settings and workspace.
Create or add diagnostic settings for your data factory.
- In the portal, go to Monitor. Select Settings > Diagnostic settings.
- Select the data factory for which you want to set a diagnostic setting.
- If no settings exist on the selected data factory, you’re prompted to create a setting. Select Turn on diagnostics.
- Give your setting a name, select Send to Log Analytics, and then select a workspace from Log Analytics Workspace.
- Select Save.
To retain Azure Data Factory pipeline run data for 120 days and query it using the Kusto query language, perform the following actions in sequence:
- From the Azure portal, add a diagnostic setting.
- Select the PipelineRuns category.
- Create a Log Analytics workspace that has Data Retention set to 120 days.
- Send the data to a Log Analytics workspace.
Explanation:
- First, you need to add a diagnostic setting in the Azure portal for your Azure Data Factory resource. This allows you to specify where the diagnostic data should be sent.
- Next, you select the PipelineRuns category in the diagnostic setting. This ensures that the pipeline run data is included in the diagnostic data.
- Then, you create a Log Analytics workspace and set the Data Retention to 120 days. This workspace will store the diagnostic data and allow you to query it using the Kusto query language.
- Finally, in the diagnostic setting, you configure it to send the data to the Log Analytics workspace you created. This ensures that the pipeline run data is sent to the workspace for retention and querying.
By following this sequence of actions, you can retain Azure Data Factory pipeline run data for 120 days and query it using the Kusto query language in the Log Analytics workspace.
Note: The other actions mentioned, such as “Stream to an Azure event hub”, “Create an Azure Storage account that has a lifecycle policy”, and “Select the TriggerRuns category”, are not required for this specific scenario.
Microsoft DP-203 certification exam practice question and answer (Q&A) dump with detail explanation and reference available free, helpful to pass the Microsoft DP-203 exam and earn Microsoft DP-203 certification.