How to Add Headers in ADF for Rest API Copy Data

Learn how to add headers in Azure Data Factory for Rest API copy data, and how to troubleshoot common errors and issues.

Problem

Azure Data Factory (ADF) is a cloud-based data integration service that allows you to create data-driven workflows for moving and transforming data. One of the common scenarios is to copy data from a Rest API source to an Azure SQL Database sink. However, you may encounter some challenges and errors when configuring the Rest API source, especially when it requires headers for authentication or authorization. In this article, you will learn how to add headers in ADF for Rest API copy data, and how to troubleshoot common errors and issues.

What are Headers in Rest API?

Headers are key-value pairs that are sent along with the HTTP request or response. They provide additional information or instructions for the server or the client, such as the content type, the encoding, the authorization, the cache control, and so on. Headers can be divided into two categories: standard headers and custom headers. Standard headers are defined by the HTTP protocol and have a predefined meaning and syntax. Custom headers are defined by the application or the API and have a specific purpose and format.

Why Do You Need to Add Headers in ADF for Rest API Copy Data?

Some Rest APIs require you to add headers in the HTTP request to access or manipulate the data. For example, some APIs use headers for authentication or authorization, such as the sessionID header that contains the apikey provided by the source site. Some APIs use headers for pagination, filtering, sorting, or other query parameters, such as the page, limit, order, or filter headers. Some APIs use headers for content negotiation, such as the accept or content-type headers that specify the format of the data. If you do not add the required headers in ADF for Rest API copy data, you may get errors or unexpected results.

How to Add Headers in ADF for Rest API Copy Data?

To add headers in ADF for Rest API copy data, you need to follow these steps:

Create a linked service for the Rest API source

A linked service defines the connection information and the authentication method for the Rest API. You can use one of the following authentication methods: Anonymous, Basic, Service Principal, OAuth2 Client Credential, System Assigned Managed Identity, or User Assigned Managed Identity. You can also add custom headers in the linked service under the advanced tab, if they are common for all the requests to the Rest API.

Create a dataset for the Rest API source

A dataset defines the data structure and the location of the Rest API. You can specify the relative URL, the HTTP method, the request body, and the additional headers for the Rest API. You can also use parameters and expressions to dynamically set the values of these properties. The additional headers are specific for each request to the Rest API, and they will override the headers defined in the linked service if they have the same name.

Create a pipeline and add a copy activity

A copy activity copies data from a source to a sink. You can configure the source and the sink settings, such as the schema mapping, the file format, the compression codec, the fault tolerance, and the performance. You can also use parameters and expressions to dynamically set the values of these settings.

How to Troubleshoot Common Errors and Issues in ADF for Rest API Copy Data?

When you add headers in ADF for Rest API copy data, you may encounter some common errors and issues, such as:

Rest call failed with client error, status code 401 Unauthorized

This error means that the server rejected the request due to invalid or missing credentials. To fix this error, you need to check the following:

• Make sure you have selected the correct authentication method and provided the valid credentials in the linked service.
• Make sure you have added the required headers for authentication or authorization in the linked service or the dataset, such as the sessionID header that contains the apikey.
• Make sure you have spelled the header names correctly and followed the syntax and format required by the Rest API.

Rest call failed with client error, status code 400 Bad Request

This error means that the server could not understand the request due to invalid or missing parameters. To fix this error, you need to check the following:

• Make sure you have specified the correct relative URL, HTTP method, and request body in the dataset.
• Make sure you have added the required headers for pagination, filtering, sorting, or other query parameters in the dataset, such as the page, limit, order, or filter headers.
• Make sure you have spelled the header names correctly and followed the syntax and format required by the Rest API.

Rest call failed with client error, status code 406 Not Acceptable

This error means that the server could not provide the data in the format requested by the client. To fix this error, you need to check the following:

• Make sure you have added the required headers for content negotiation in the dataset, such as the accept or content-type headers that specify the format of the data.
• Make sure you have specified the correct file format and compression codec in the source settings of the copy activity.
• Make sure you have spelled the header names correctly and followed the syntax and format required by the Rest API.

Frequently Asked Questions (FAQs)

Question: How do I test the Rest API request before I configure the Rest API source in ADF?

Answer: You can use tools like Postman or a web browser to test the Rest API request and see the response. You need to enter the full URL, the HTTP method, the request body, and the headers for the Rest API, and see if you get the expected data and status code.

Question: How do I use parameters and expressions to dynamically set the values of the Rest API source properties in ADF?

Answer: You can use parameters and expressions to dynamically set the values of the Rest API source properties, such as the relative URL, the HTTP method, the request body, and the additional headers. You need to create parameters in the linked service, the dataset, or the pipeline, and use expressions to reference them or use built-in functions. For example, you can use the @pipeline().parameters.page parameter to set the value of the page header in the dataset.

Question: How do I copy the Rest API response as-is or parse it by using schema mapping in ADF?

Answer: You can copy the Rest API response as-is or parse it by using schema mapping in ADF, depending on the file format and the data structure of the response. If the response is in JSON format, you can use the JSON file format and the schema mapping to parse the response and map the fields to the sink columns. If the response is in other formats, such as XML, CSV, or binary, you can use the corresponding file formats and copy the response as-is to the sink.

Summary

In this article, you have learned how to add headers in ADF for Rest API copy data, and how to troubleshoot common errors and issues. You have also learned how to create a linked service, a dataset, and a copy activity for the Rest API source, and how to use parameters and expressions to dynamically set the values of the Rest API source properties. You have also learned how to copy the Rest API response as-is or parse it by using schema mapping. We hope this article has helped you to successfully copy data from a Rest API source to an Azure SQL Database sink by using ADF.

Disclaimer: This article is for informational purposes only and does not constitute professional advice. The products and services mentioned in this article are not endorsed by Azure Data Factory or the author. The author is not responsible for any damages or losses that may result from the use of the products or services mentioned in this article. The reader is advised to do their own research and due diligence before making any purchase decisions.