DevOpsPadDownload

Check Kafka partition leaders and in-sync replicas

· Guides

When producers time out or consumers stall on one partition, check partition leadership first. Every partition has one leader broker that serves reads and writes, a replica set, and an in-sync replica set (ISR). A partition with no leader is unavailable. A partition whose ISR is smaller than its replica set is under-replicated and one broker failure closer to data loss. Both are visible in cluster metadata. You do not need to read a single message.

Step 1: describe a topic

With the scripts that ship with Kafka:

kafka-topics.sh --bootstrap-server broker-1.internal:9092 --describe --topic orders
Topic: orders  TopicId: …  PartitionCount: 3  ReplicationFactor: 3  Configs: min.insync.replicas=2
  Topic: orders  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
  Topic: orders  Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,1
  Topic: orders  Partition: 2  Leader: -1 Replicas: 3,1,2  Isr: 3

Read it per row:

  • Leader is a broker ID. -1 means no leader: the partition is offline.
  • Replicas is the assigned replica list; the first entry is the preferred leader.
  • Isr is the subset that is caught up. Partition 1 is under-replicated (broker 3 fell behind).

Step 2: filter to the problems

On a cluster with hundreds of topics, ask for only the unhealthy partitions:

kafka-topics.sh --bootstrap-server broker-1.internal:9092 --describe --under-replicated-partitions
kafka-topics.sh --bootstrap-server broker-1.internal:9092 --describe --unavailable-partitions
kafka-topics.sh --bootstrap-server broker-1.internal:9092 --describe --under-min-isr-partitions

The last one matters most for producers using acks=all. Below min.insync.replicas, their writes are rejected even though the partition has a leader.

Step 3: authenticate when the cluster requires it

For SASL/PLAIN over TLS, put the client settings in a file and pass it with --command-config:

# client.properties
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="inspector" password="…";
kafka-topics.sh --bootstrap-server broker-1.internal:9093 --command-config client.properties --describe --topic orders

Keep that file out of shell history and repositories. It holds a plaintext password.

Step 4: or use kcat

kcat (formerly kafkacat) prints the same metadata without a JVM:

kcat -b broker-1.internal:9092 -L -t orders

Each partition line shows leader, replicas and isrs. Add -X security.protocol=SASL_SSL -X sasl.mechanisms=PLAIN -X sasl.username=… -X sasl.password=… for authenticated clusters.

Step 5: watch out for advertised listeners

The bootstrap address is only used for the first request. The metadata response tells the client each broker’s advertised address, and every later connection goes there. If the brokers advertise broker-2.internal:9092 and your machine cannot resolve or reach that name, the describe call can succeed while anything that talks to a specific broker fails.

This shows up most often through SSH tunnels and port forwards. You forward localhost:9092 to one broker, metadata comes back with internal hostnames, and the client tries to reach those directly. The fix is on the broker side, with an advertised listener that is reachable from where the client runs, or with a tunnel that can reach every advertised address.

Doing this in DevOpsPad

DevOpsPad’s Kafka connector reads the same metadata. In the query editor, type a topic name (for example order.created) or topics for all topics. The result has one row per partition with topic, partition, leader, replicas and in_sync_replicas, where the last two are counts. A row where in_sync_replicas is lower than replicas is under-replicated. Results follow the usual 500-row cap, so with very many partitions, query one topic at a time. Connections support SASL/PLAIN and TLS.

Through an SSH tunnel, DevOpsPad also routes the connections to advertised brokers through the tunnel, so the advertised addresses must be reachable from the SSH server (Step 5 applies there, not on your laptop). Message browsing, consumer groups and SASL/SCRAM are not implemented yet. See Kafka topics and partitions, SSH troubleshooting and the Kafka connector page.