Azure Toolkit for IntelliJ integrates with HDInsight Ambari and supports Spark 2.2
To provide more authentication options, Azure Toolkit for IntelliJ now supports integration with HDInsight clusters through Ambari for job submission, cluster resource browse and storage files navigate. You can easily link or unlink any cluster by using an Ambari-managed username and password, which is independent of your Azure sign-in credentials. The Ambari connection applies to normal Spark and Hive hosted within HDInsight on Azure. These additions give you more flexibility in how you connect to your HDInsight clusters in addition to your Azure subscriptions while also simplifying your experiences in submitting Spark jobs.
With this release, you can benefit the new functionalities and consume the new libraries & APIs from Spark 2.2 in Azure Toolkit for IntelliJ. You can create, author and submit a Spark 2.2 project to Spark 2.2 cluster. With the backward compatibility of Spark 2.2, you can also submit your existing Spark 2.0 and Spark 2.1 projects to a Spark 2.2 cluster.
How to link a cluster
- Click Link a cluster from Azure Explorer.
- Enter Cluster Name, Storage Account, Storage Key, then select a container from Storage Container, at last, input Username and Password.
Please note that you can use either Ambari username, pwd or Secure Hadoop domain username, or pwd to connect. The storage account and key information will become optional in our next release for cluster connection.
- The linked cluster is displayed under HDInsight node in Azure Explorer. You can submit you Spark jobs to this linked cluster.
- You also can unlink a cluster from Azure Explorer.
How to install or update
Please upgrade IntelliJ to 2018.1 version first. IntelliJ will prompt you for the latest update if you have Azure Toolkit for IntelliJ installed before, or you can get the latest bits by going to the IntelliJ repository and searching Azure Toolkit for IntelliJ.
For more information, check out the following:
- HDInsight IntelliJ plugin
- Use Azure Toolkit for IntelliJ to debug Spark applications
- Demo Create Spark Scala Project
- Use Azure Toolkit for IntelliJ to debug Spark applications remotely on an HDInsight cluster
Learn more about today’s announcements on the Azure blog and Big Data blog, and discover more Azure service updates.
Feedback
We look forward to your comments and feedback. If there is any feature request, customer ask, or suggestion, please send us a note to hdivstool@microsoft.com. For bug submission, please open a new ticket using the template.
Source: Azure Blog Feed