Azure Managed Lustre—Parallel file system for HPC and AI workloads
Today, Microsoft is announcing the public preview of Azure Managed Lustre, a new addition to the storage offerings in our Azure HPC + AI solution. Lustre is an open-source parallel file system renowned for high-performance computing (HPC) and is adept at large-scale cluster computing. Azure Managed Lustre provides the high-performance storage of Lustre with the control and consistency of Microsoft’s cloud. As a result, customers can focus on their business goals, whether that’s building a fraud detection system based on SAS analytics, decoding the human genome to create the next breakthrough in medicine, leading the frontiers of energy exploration with seismic data processing and sustainable energy solution, or predicting the climate and weather patterns impacting human life on planet earth.
Lustre as an Azure Managed Service
With Lustre as a managed service, users can now focus on building and running their HPC and AI applications without managing an advanced parallel file system. HPC and AI workloads can seamlessly migrate to the cloud, retaining their compatibility with Lustre and protecting existing automation and platform investments developed when previously run on premises. Azure Managed Lustre enables a fast on-demand deployment of clusters serving global regions, alleviating advance planning while meeting the compliance and data residency requirements. With a palette of performance options, Azure Managed Lustre delivers an elastic solution where users can deploy independent and exclusive clusters with predictable performance eliminating the noisy-neighbor problem commonly experienced in the on-premises shared infrastructures.
The public preview includes two durable SSD-based SKUs which deliver a choice in performance options for mission critical workloads: 125 MB/s and 250 MB/s for every provisioned TiB of capacity. During the preview, you can create cluster sizes up to 128 TiB by default, with an option to scale up to 768 TiB upon request. Azure Managed Lustre is built on the highly durable Azure managed disks with locally-redundant storage consisting of 3 replicas so even if one or two replicas experience issues you still have tolerance against failures. Azure Managed Lustre delivers POSIX-compliant Lustre version 2.15 (LTS), which offers several performance improvements. We focused on capabilities that enable users to consume Azure Managed Lustre easily without worrying about how to build and deploy the Lustre clients on their compute VMs and containers. With the Azure Managed Lustre preview, customers can choose from:
- Use Azure HPC images prebuilt with Lustre client packages for Ubuntu 18.04 and 20.04 (or)
- Download Lustre client packages from packages.microsoft.com for Linux distros – Ubuntu 18.04, 20.04 & 22.04; RHEL 7.8 & RHEL 8.0
Data tiering using hierarchical storage management allows users to import and export data between Azure Managed Lustre and Azure Blob. This capability enables users to define data archival, retention and protection based on pre-defined policies. Users of large data sets can import the data which is hot and relevant for their active data processing in Azure Managed Lustre clusters and archive/retain the remaining in Azure Blob. This enables them to keep their run-time costs low. Additionally, data tiering to Azure Blob allows the users to leverage its global presence and availability to instantiate Azure Managed Lustre clusters in multiple regions on demand. Integration with Azure Blob additionally facilitates multiprotocol data access via NFS, HDFS and REST. Modern containerized applications in AI/machine learning (ML) and analytics can now run on Azure Kubernetes Service leveraging the CSI driver for Azure Managed Lustre. For example, analytics applications built on top of SAS Viya can seamlessly leverage the high per-core storage performance that they require by integrating with the Azure Managed Lustre CSI driver.
Purpose-built for HPC and AI workloads
We are thankful to our private preview customers who helped us build and perfect the product while addressing their key needs. We are committed to building and supporting Azure Managed Lustre as it helps you with your journey of running your high-performance applications in Azure.
“We work with customers who have many different types of HPC workloads, and we frequently hit storage performance constraints because their projects can involve anything from thousands to tens of thousands of files, all of which need to be read and written in parallel. When we encounter storage bottlenecks or throttling errors on our HPC platforms, we know it is time to turn to Azure Managed Lustre. With Azure Managed Lustre we can add extremely performant storage that can readily keep up with the HPC compute, and automatically sync input and results data with persistent blob storage. Because it can be started and stopped on demand, within a few minutes, Azure Managed Lustre is quite cost efficient and pays for itself by giving us the ability to scale up or scale out our HPC compute, generating results much faster.”—Felipe Ayora, Director—Research and Advanced Computing, BizData.
“The new Azure Managed Lustre file system is a real game changer. It accelerates HPC deployment of our software and eliminates the need to run persistent storage servers. Setup is simple, and parallel i/o performance is fast enough to support very high-throughput workloads.”—Bill Shea, CEO Sharp Reflections.
Next Steps
To trial Azure Managed Lustre for free, complete the registration form. Learn more about how to use Azure Managed Lustre and its various supported features from our documentation.
Learn more about HPC and AI solutions
- Read the Azure Managed Lustre documentation.
- Read our Azure Managed Lustre technical blog.
- Visit our Azure HPC hub for more technical content developed for HPC.
- Read about our Azure HPC + AI solution.
Source: Azure Blog Feed