Data integration with ADLS Gen2 and Azure Data Explorer using Data Factory
Microsoft announced the general availability of Azure Data Lake Storage (ADLS) Gen2 and Azure Data Explorer in early February, which arms Azure with unmatched price performance and security as one of the best clouds for analytics. Azure Data Factory (ADF), is a fully-managed data integration service, that empowers you to copy data from over 80 data sources with a simple drag-and-drop experience and operationalize and manage the ETL/ELT flows with flexible control flow, rich monitoring, and continuous integration and continuous delivery (CI/CD) capabilities. In this blog post, we’re excited to update you on the latest integration in Azure Data Factory with ADLS Gen2 and Azure Data Explorer. You can now meet the advanced needs of your analytics workloads by leveraging these services.
Ingest and transform data with ADLS Gen2
Azure Data Lake Storage is a no-compromises data lake platform that combines the rich feature set of advanced data lake solutions with the economics, global scale, and enterprise grade security of Azure Blob Storage. Our recent post provides you with a comprehensive insider view on this powerful service.
Azure Data Factory supports ADLS Gen2 as a preview connector since ADLS Gen2 limited public preview. Now the connector has also reached general availability along with ADLS Gen2. Moreover, with ADF, you can now:
- Ingest data from over 80 data sources located on-premises and in the cloud into ADLS Gen2 with great performance.
- Orchestrate data transformation using Databricks Notebook, Apache Spark in Python, and Spark JAR against data stored in ADLS Gen2.
- Orchestrate data transformation using HDInsights with ADLS Gen2 as the primary store and script store on either bring-your-own or on-demand cluster.
- Egress data from ADLS Gen2 to a data warehouse for reporting.
- Leverage Azure’s Role Based Access Control (RBAC) and Portable Operating System Interface (POSIX) compliant access control lists (ACLs) that restrict access to only authorized accounts.
- Invoke control flow operations like Lookup and GetMetadata against ADLS Gen2.
Get started today
- Tutorial on ingesting data into ADLS Gen2
- ADLS Gen2 connector
- Databricks Notebook activity to transform data in ADLS Gen2
- HDInsights activity to transform data in ADLS Gen2
Populate Azure Data Explorer for real-time analysis
Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. It helps you handle the many data streams emitted by modern software and is designed for analyzing large volumes of diverse data.
Bringing data into Azure Data Explorer is the first challenge customers often face to adopt the service. Complimentary to Azure Data Explorer’s native support on continuous data ingestion from event streams, Azure Data Factory enables you to batch ingress data from a broad set of data stores in a codeless manner. With simple drag-and-drop features in ADF, you can now:
- Ingest data from over 80 data sources – on-premises and cloud-based, structured, semi-structured, and unstructured into Azure Data Explorer for real-time analysis.
- Egress data from Azure Data Explorer based on the Keyword Query Language (KQL) query.
- Lookup Azure Data Explorer for control flow operations.
Get started
We will keep adding new features in ADF to tighten the integration with ADLS Gen2 and Azure Data Explorer. Stay tuned and let us know your feedback!
Source: Azure Updates