Kafka Consumers

How Consumers Work

A Kafka consumer reads messages from topics. Unlike traditional queues where the broker pushes messages to consumers, Kafka uses a pull model — consumers ask for messages when they're ready. This gives consumers full control over their pace and allows them to re-read old messages by resetting their offset.

The Poll Loop

Every Kafka consumer follows the same basic pattern: subscribe to topics, poll for messages in a loop, process each message, and commit the offset to record progress.

from confluent_kafka import Consumer

consumer = Consumer({
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'my-application',
    'auto.offset.reset': 'earliest',  # Start from beginning if no committed offset
})

consumer.subscribe(['user-events'])

try:
    while True:
        msg = consumer.poll(timeout=1.0)  # Wait up to 1 second for messages

        if msg is None:
            continue  # No message available
        if msg.error():
            print(f"Error: {msg.error()}")
            continue

        # Process the message
        print(f"Partition: {msg.partition()}, Offset: {msg.offset()}")
        print(f"Key: {msg.key()}, Value: {msg.value().decode('utf-8')}")

finally:
    consumer.close()  # Commit final offsets and leave consumer group

Offset Management

The offset tracks where each consumer is in each partition. When a consumer commits offset 42, it's saying "I've successfully processed all messages up to and including 42." If the consumer crashes and restarts, it resumes from offset 43.

auto.offset.reset

What happens when a consumer starts for the first time (no committed offset exists)?

• earliest — Start reading from the beginning of the topic. Use this when you need to process all historical data.
• latest — Start reading only new messages (produced after the consumer started). Use this for real-time processing where history doesn't matter.
• none — Throw an error if no offset exists. Use this when you want to explicitly manage starting positions.

Consumer Rebalancing

When a consumer joins or leaves a group, Kafka rebalances — it reassigns partitions among the remaining consumers. During rebalancing, no messages are processed. Frequent rebalancing kills throughput.

Common causes of unnecessary rebalancing:

• Processing takes longer than max.poll.interval.ms (default 5 min) — Kafka thinks the consumer is dead
• Consumer instances scaling up/down frequently
• Network issues causing heartbeat timeouts

Practice Exercises