It’s an exciting time to work (and play!) in technology. The cloud is literally changing everything. As a data engineer, I am excited to see recent advancements in cloud-based data integration solutions. My personal favorite these days is Azure Data Factory (adf.azure.com).

With the general availability of Azure Data Factory – or ADF – version 2 in May 2018, ADF became a more serious contender for data engineering in the cloud. Since May, monthly updates have added features and functionality. With the recent announcement of ADF data flows, the ADF Team continues to innovate in the space.

Azure Data Factory Execution Patterns

In this article we will build two execution design patterns: Execute Child Pipeline and Execute Child SSIS Package.

A Generic Pipeline

Begin by creating a very simple generic pipeline. Because I’m feeling creative, I named mine “generic” as shown in Figure 1:


Figure 1

The generic pipeline contains a Wait activity configured to start execution, wait 1 second, and then complete. This may be the simplest example of an ADF pipeline ever.

Execute Child Pipeline

After publishing the generic pipeline, build a new pipeline named “parent” and add an Execute Pipeline activity named Execute generic as shown in Figure 2:


Figure 2

Configure the Execute Pipeline activity to invoke the generic pipeline as shown in Figure 3:


Figure 3

To test the parent pipeline, click Debug. Debugging starts as shown in Figure 4:


Figure 4

If all goes as planned, parent succeeds. Publish parent, then click Trigger and Trigger Now as shown in Figure 5:


Figure 5

Click the Finish button to begin pipeline execution and then click the gauge icon on the left side of the page to monitor execution. Both parent and generic should execute and succeed, and the monitoring page should appear similar to that shown in Figure 6:


Figure 6

I call this the Pipeline Controller pattern.

If you are playing along at home and see two successful pipeline executions on the monitoring page, congratulations!

Azure Data Factory SSIS Integration Runtime

Before you can deploy SSIS projects to ADF, you must first create an instance of an ADF SSIS Integration Runtime. You can start the process from the Azure Data Factory home page by clicking Configure SSIS Integration Runtime” highlighted in Figure 7:


Figure 7

For more information on ADF integration runtimes, please see docs.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime.

Once you create an ADF SSIS Integration Runtime (or “Azure-SSIS IR”) instance, you can deploy SSIS projects to the cloud! Connect to the Azure SQL DB instanced using SSMS to view SSIS projects and packages as shown in Figure 8:


Figure 8

Because the ADF SSIS Integration Runtime is essentially an SSIS Catalog, you can use the free utility Catalog Browser (dilmsuite.com/catalog-browser) to connect and view its contents as shown in Figure 9:


Figure 9

Execute Child SSIS Package

Once SSIS projects are lifted and shifted to the Azure Data Factory SSIS Integration Runtime, you can use the Execute SSIS Package activity to construct a pipeline based on the SSIS Controller pattern. Simply delete the Execute Pipeline activity from the parent pipeline and add an Execute SSIS Package activity.

On the settings tab of the Execute SSIS Package activity, select the name of an ADF SSIS Integration Runtime from the Azure-SSIS IR dropdown as shown in Figure 10:


Figure 10

The Package Path is the only other property we need to configure for this example. If you connected to the Azure-SSIS IR Catalog using Catalog Browser, you can click on the package and copy the package path from the status text as shown in Figure 11: