The Web Activity in Azure Data Factory (ADF) is a versatile pipeline activity used for invoking HTTP endpoints. It can call REST APIs, trigger Azure Logic Apps or Function Apps, make HTTP requests to external services, and much more. This activity is useful for integrating external processes, data sources, or services into your data integration workflows.

What is Web activity in ADF?

Definition:

Web is an activity in ADF which can be used to call a REST API calls from Azure Data factory pipeline.

• You can use only public exposed REST URLs. You cannot call Private network REST end points.
• You can also pass the dataset and linked services to be used by the activity.
• You can use the output of web activity as an input to other activity. The maximum supported output payload size is 4 MB.
• REST endpoint response must be of type JSON.
• If the web activity does not get a response within 1 minute, it will end up throwing an error.

ADF – Web activity implementation

• Go to Azure Data Factory Pipeline.
• Add Activity Web.



• Click on the Activity, you will see 3 tabs.
  • General:
  
  • Name: Specify the Name of the web activity
  • Description: Describe the web Activity
  • TimeOut: Maximum amount of time an activity can run. The default is 7 days, which is also the maximum amount of time allowed. Format is in D.HH:MM:SS
  • Retry: maximum number of retry allowed
  • Retry interval(sec): Specify the number of seconds between each retry attempt.
  • Secure output: If you tick this, output of the activity will not be added in the logging
  • Secure input: If you tick this, input of the activity will not be added in the logging
  • Settings
  • URL: Specify the target end point URL. The HTTP/HTTPS endpoint you are calling.
  • Method: Select your API method. Supported methods (GET, PUT, POST, DELETE, PATCH).
  • Body: The request body, required for methods like POST or PUT. You can use static JSON/XML or dynamic content from previous activities.
  • Headers: This is the request headers that you send when you call REST API end point. Any required HTTP headers, such as Content-Type: application/json.
  
  • Datasets: List of datasets passed to the endpoint.
  • Linked Services: List of Linked Services passed to the endpoint.
  • Integration runtime: The Integration Runtime (IR) is the compute infrastructure used by ADF to provide data integration capabilities across different network environments. You can select the default.
  • Disable certificate validation: If you tick, it will remove server-side certificate validation
  • HTTP request timeout: Timeout for the HTTP request to get a response. Format is in TimeSpan (hh:mm:ss). This value is the timeout to get a response, not the activity timeout. The default value is 00:01:00 (1 minute). The range is from 1 to 10 minutes
  • Authentication: Authentication method used for calling the endpoint.
    • None: No authentication.
    • Basic: Specify username and password to use with basic authentication. As you can see in the screenshot below, you need to specify the password from Azure Key Vault.
    
    • System Assigned Managed Identity: Specify the resource for which Azure Authentication Token will be requested when using MSI authentication.
    
    • Client Certificate: Specify base64-encoded contents of a PFX file and the password. You need to use Azure key vault for supplying both Pfx and Password.
    
    • Service Principal: To set up this authentication, you need to specify tenant, Service Principal Id, Service Principal Key/certificate, Resource. You can read more.
    
    • User Assigned Managed Identity: Specify the User Assigned Managed Identity credentials for authentication.
  
  • Advanced - Disable async pattern: This option is to disable invoking HTTP GET on location given in response header of a HTTP 202 Response. If set true, it stops invoking HTTP GET on http location given in response header. If set to false, then continues to invoke HTTP GET call on location given in http response headers.
  • User Properties: Use this section to create User Properties of the Pipeline.

ADF Web Activity Usage Scenarios

• Calling External APIs
  • Use the Web Activity to call external REST APIs for data retrieval, updates, or triggering actions in other systems.
  • This can include fetching weather data, sending notifications, or interacting with SaaS platforms.
• Triggering Logic Apps or Function Apps
• You can trigger Azure Logic Apps or Azure Function Apps using their HTTP trigger URLs.
• This is especially useful for integrating custom business logic or workflows that are not directly supported in ADF.
• Handling Responses
• The Web Activity allows you to handle responses directly in ADF
• Success Outputs: Use the output of the Web Activity in subsequent activities, such as setting variables or conditional paths based on the response data.
• Failure Handling: Configure error handling in the pipeline to manage HTTP errors or invalid responses.