Iterative development and debugging using Data Factory

Data Integration is becoming more and more complex as customer requirements and expectations are continuously changing. There is increasingly a need among users to develop and debug their Extract Transform/Load (ETL) and Extract Load/Transform (ELT) workflows iteratively. Now, Azure Data Factory (ADF) visual tools allow you to do iterative development and debugging.

You can create your pipelines and do test runs using the Debug capability in the pipeline canvas without writing a single line of code. You can view the results of your test runs in the Output window of your pipeline canvas. Once your test run succeeds, you can add more activities to your pipeline and continue debugging in an iterative manner. You can also Cancel your test runs once they are in-progress. You are not required to publish your changes to the data factory service before clicking Debug. This is helpful in scenarios where you want to make sure that the new additions or changes work as expected before you update your data factory workflows in dev, test or prod environments.

Data Factory visual tools also allow you to do debugging until a particular activity in your pipeline canvas. Simply put a breakpoint on the activity until which you want to test and click Debug. Data Factory will guarantee that the test run will only happen until the breakpoint activity in your pipeline canvas.

Get started today by clicking the Author & Monitor tile in your provisioned v2 data factory blade. Build, debug your data factory pipelines iteratively and let us know your feedback.

Our goal is to continue adding features to continuously improve the usability of data factory tools. Get more information and detailed steps for doing iterative development and debugging using Data Factory UX.

Get started building pipelines easily and quickly using Azure Data Factory. If you have any feature requests or want to provide feedback, please visit the Azure Data Factory forum.

Source: Azure Blog Feed