Multi-Broker Apache Kafka + Zookeeper Cluster Setup

In this article we will create a Apache Kafka multi broker cluster on three machines, we have three ubuntu machine with 4GB ram.


Before starting, make sure you have Java 7+ running on each instance, check java version with following command:

java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

If not installed already follow these instructions Here.

Apache Kafka needs a running Zookeeper cluster, we have already created a zookeeper cluster in previous article we will use the same here, zookeeper cluster ip's are:


In order to setup Apache Kafka Multi Broker cluster, following steps need to be followed:
Step 1) Download and unzip Kafka package in all three machines as shown below:

cd /opt/
tar -zxvf kafka_2.11-

This will create a untar Kafka package folder with name kafka_2.11-


Step 2) Configure file in all three machines as whown below:

cd kafka_2.11-

Change following old properties:

To (2,3 in other two machines)

Step 3) Start kafka on each machine one by one

bin/ config/

Here we are done with setting up cluster for kafka, now let's test with creating a test topic with replication factor 3 and partition 1 on cluster:

bin/ --create --zookeeper --replication-factor 3 --partitions 1 --topic test-topic

Now lets check which broker is doing what in the cluster, check status of newly created topic "test" with following command:

bin/ --describe --zookeeper --topic test-topic

You will see following output,

Topic:test-topic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: test-topic Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2

Since we have only one partition for this topic there is only one line. Here Partition:0 is partition number, for this only partition Leader is broker 3 for now and Replicas are on all three 3,2,1. ISR indicates all the replicas are in sync, this says all three brokers are in sync for the data in topic "test".

Now lets create another topic with replication factor 3 and partition 3 and see how that is distributed on brokers.

bin/ --create --zookeeper --replication-factor 3 --partitions 3 --topic test-topic-2

Now lets test status of this topic, this will look like this:

bin/ --describe --zookeeper --topic test-topic-2
Topic:test-topic-2 PartitionCount:3 ReplicationFactor:3 Configs:
Topic: test-topic-2 Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
Topic: test-topic-2 Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: test-topic-2 Partition: 2 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1

Here we can see, three lines are representing three different partitions of the topic and related information.

That's it, we are done with creating a multi broker kafka cluster on 3 ubuntu machines. In upcoming articles we will see more about configuration settings, writing publisher and consumers to kafka.

Must Read: Java Kafka API - Consumer and Producer Examples
How to write a Kafka producer in java - Example
How to write a Kafka Consumer in java - Automatic offset commit
How to write a Kafka Consumer in java - Manual offset commit
How to write a Kafka Consumer in java - assignable to a specific partition

About The Author

Nagesh Chauhan

Nagesh Chauhan has 8+ years of software design and development experience in variety of technologies like - Core Java, Java 8 (Streams, Lambda), J2EE (Servlet, JSP), Spring Framework (MVC, IOC, JDBC, SECURITY etc), Spring Boot and Microservices, Kafla, Redis, Cassandra and Spark.