Software Developer Central

Spark Streaming and Kafka Integration Example Introduction         My search online for a good tutorial on Big Data processing using Spark Streaming and Kafka ended up in a bunch of old posts which had very old versions of Spark and Kafka client libraries. They were of no use to me if I wanted to use the newer versions of the libraries. Hence I decided that I should share my experience with the newer versions of the libraries. This was the motivation for me to come up with this post. This post will give you a comprehensive overview of Spark Streaming and Kafka Integration. The use case that I am trying to address in this post is to process streaming data from a topic on Kafka using Spark Streaming. This is a very common use case that many organizations large and small are implementing these days.         I will be presenting 2 examples in this post. One example will print the stream of data from a Kafka Topic as it receives it and...

Dropwizard Kafka Tutorial Introduction         In this tutorial, we will be integrating a Dropwizard Application with Kafka Producer  using Kafka Producer API. Through RESTful API in Dropwizard we will send messages to a Kafka topic through a Kafka Producer. We will also be using a Java based Kafka Consumer using Kafka Consumer API to consume and print the messages sent from the Dropwizard application.  So there are 2 Applications required to get the end to end functionality: Dropwizard Application with Kafka Producer API Java Kafka Consumer Application with Kafka Consumer API Kafka         Kafka is a distributed streaming platform. It can be used for communication between applications or micro services. Kafka is ideally used in big data applications or in applications that consume or process huge number of messages. Some use cases where Kafka can be used: Messaging Website activity tracking Metrics Log aggr...

Spring Boot Kafka Tutorial Introduction         In this tutorial, we will be integrating a Spring Boot Application with Kafka Producer  using Kafka Producer API. Through RESTful API in Spring Boot we will send messages to a Kafka topic through a Kafka Producer. We will also be using a Java based Kafka Consumer using Kafka Consumer API to consume and print the messages sent from the Spring Boot application.  So there are 2 Applications required to get the end to end functionality: Spring Boot Application with Kafka Producer API Java Kafka Consumer Application with Kafka Consumer API Kafka         Kafka is a distributed streaming platform. It can be used for communication between applications or micro services. Kafka is ideally used in big data applications or in applications that consume or process huge number of messages. Some use cases where Kafka can be used: Messaging Website activity tracking Metrics Log aggrega...