Spring Boot Apache Kafka
In this article, I will try to make an example of both Producer and Consumer applications of Apache Kafka, which is very popular and widely used today, by creating a Spring Boot project.
The basic concepts are given below.
Apache Kafka is an open source stream processing software platform written in Scala and Java, developed by the Apache Software Foundation. The project aims to provide a unified, high-throughput, low-latency platform for processing real-time data streams.
Producer: It is the application that produces messages and sends them to Apache Kafka.
Consumer: It is the application that consumes the messages and receives the messages from Apache Kafka.
You can easily create projects at https://start.spring.io/
Let’s start coding ✔
Producer Application
First of all, we need to add dependencies to our project as follows. This dependency includes libraries that will allow us to communicate with Apache Kafka.
spring-boot-starter-web: Starts the application as a web project.
spring-kafka: Used for the application to access Apache Kafka.
The application.yml file is as follows. Here it contains information about which port the application will run on and which Apache Kafka and Topic address it will connect to.
Topic: This is the section where messages are provided on Apache Kafka. It has a Queue structure.
The relevant configuration file to connect with Apache Kafka is as follows.
The ProducerFactory object is used to provide the connection.
The KafkaTemplate object is used to send messages.
Producer Service class sends the relevant message to the relevant Topic in Kafka using the send(topic, message) method of the KafkaTemplate class.
Everything is ready, now let’s send messages to Kafka using the Producer Service class. As an example, sending 10 messages is done below.
By adding this as needed, for example, before and after service calls (of course, it would be better to do this with Aspect Programming 😉), you can collect the response times of the services in one place and then make an inference such as which service took how long to respond.
Consumer Application
We sent messages to Kafka thanks to the Producer application. Now we can run the Consumer application and start processing messages in Kafka.
First of all, we need to add dependencies to our project as in the Producer application, as follows.
The application.yml file is as follows. Here it contains information on which port the application will start with, which Apache Kafka and Topic address it will connect to and listen for messages.
group-id: Can be used to create a cluster structure. When you want to run more than one Consumer application in the same cluster, you must give the application the same group-id.
Important Note: If you are going to run the application on the same machine, do not forget to give the server.port value differently !!!
The relevant configuration file to connect with Apache Kafka is as follows.
The ConsumerFactory object is used to establish the connection.
The ConcurrentKafkaListenerContainerFactory object automatically binds to Kafka and creates a Listener to listen to messages.
Now we can start listening to messages from Kafka by creating a class called KafkaConsumerListener.
Let’s not forget to add Component annotation here while the spring application is running. Otherwise, the KafkaConsumerListener class will not rise.
When we add KafkaListener annotation to the method and provide topic and group-id information, we will now be able to read messages from Kafka.
You can access all the codes of the Producer application from the address below.
You can access all the codes of the Consumer application from the address below.
