MileIQ and Azure Event Hubs: Billions of miles streamed

This post was co-authored by Shubha Vijayasarathy, Program Manager, Azure Messaging (Event Hubs)

With billions of miles logged, MileIQ provides stress-free logging and accurate mileage reports for millions of drivers. Logging and reporting miles driven is a necessity for independent contractors to organizations with employees who need to drive for work. MileIQ automates mileage logging to create accurate records of miles driven, minimizing the effort and time needed with manual calculations. Real-time mileage tracking produces over a million location signal events per hour, requiring fast and resilient event processing that scales.

MileIQ leverages Apache Kafka to ingest massive streams of data:

• Event processing: Events that demand time-consuming processing are put into Kafka, and multiple processors consume and process these asynchronously.
• Communication among micro-services: Events are published by the event-owning micro-service on Kafka topics. The other micro-services, which are interested in these events, subscribe to these topics to consume the events.
• Data Analytics: As all the important events are published on Kafka, the data analytics team subscribes to the topics it is interested in and pulls all the data it requires for data processing.

Growth Challenges

As with any successful venture, growth introduces operational challenges as infrastructure struggles to support the growing demand. In MileIQ’s case, the effort and resources required to maintain Apache Kafka clusters multiplied exponentially with adoption. A seemingly simple task, like modifying a topic’s retention configuration, now becomes an operational burden as the number of Kafka clusters scale to meet the increase in data.

Leveraging a managed service, enabled MileIQ to shift resources from operations and maintenance to focus on new ways to drive business impact. A couple reasons why the MileIQ team selected Azure Event Hubs for Kafka:

• Fully managed platform as a service (PaaS): With little configuration or management overhead, Event Hubs for Kafka provides a PaaS Kafka experience without the need to manage, configure, or run Kafka clusters.
• Supports multiple Kafka use-cases: Event Hubs for Apache Kafka provides support at the protocol level, enabling integration of existing Kafka applications with no code changes and minimal configuration change. MileIQ’s existing Kafka producers and consumers, as well as other streaming applications like Apache Kafka MirrorMaker and Apache Spark, integrated seamlessly with the Kafka-enabled Event Hub.
• Deliver streaming data to Azure Blob storage: The Capture feature of Event Hubs automatically send data from Azure Event Hubs for Kafka to Blob storage. MileIQ uses the data in Blob storage for data analytics and backup.
• Enterprise performance: The Dedicated-tier cluster offers single-tenant deployments with a guaranteed 99.99% SLA. MileIQ performance tests showed the Dedicated-tier cluster was able to consistently produce a throughput rate of 6,000 events per second.

* Testing based on one event at a time synchronously to address specific use-cases focused on consistency over throughput. Testing batching and produce asynchronously resulted in a much higher throughput.

Set up for success

As a result of migrating Apache Kafka to a managed service, MileIQ now has the infrastructure needed to support future growth.

“To sum up, our experience switching over to Azure Event Hubs Kafka has been excellent. To start with, the onboarding was straightforward, integration was seamless, and we continue to receive great help and support from the Azure Event Hubs Kafka team. In the near future, we look forward to the release of new features that the Azure Event Hubs Kafka team is working on – Geo Replication, Idempotent Producers, Kafka Streams, etc.”

“Migrating to Azure Event Hubs Kafka was a painless experience. Straightforward onboarding seamless integration, and support from the Event Hubs team every step of the way.  We’re excited to see what’s next and look forward to a continued partnership.” – MileIQ

Start streaming data

Data is valuable only when there is an easy way to process and get timely insights from data sources. Azure Event Hubs provides a fully managed distributed stream processing platform with low latency and seamless integration with Apache Kafka applications.

What are you waiting for? Time to get event-ing!

• Event Hubs for Apache Kafka overview
• Get started: Create Apache Kafka-enabled event hubs
• Discover more about Azure Event Hubs
• Learn about Dedicated-tier clusters

Enjoyed this blog? Follow us as we update the features list we will start supporting. Leave us your valuable feedback, questions, or comments below.

Happy event-ing!

Source: Azure Blog Feed