Getting Started with Python

Producer Tuning

Producer throughput depends on three main factors: batch size, compression, and acknowledgment mode. Here are the key settings:

Setting	Default	For Throughput	For Latency
`batch.size`	16384 (16KB)	65536+ (64KB)	Keep default
`linger.ms`	0	10-100ms	0 (send immediately)
`compression.type`	none	lz4 or zstd	lz4 (fast)
`acks`	all	1 (if loss acceptable)	1
`buffer.memory`	32MB	64MB+	Keep default

Consumer Tuning

Setting	Default	For Throughput	For Latency
`fetch.min.bytes`	1	10000+	1 (return immediately)
`fetch.max.wait.ms`	500	500-1000	100
`max.poll.records`	500	1000+	100-500
`max.partition.fetch.bytes`	1MB	5MB	1MB

Benchmarking

# Producer benchmark: send 1M messages of 1KB each
kafka-producer-perf-test.sh --topic perf-test \
  --num-records 1000000 --record-size 1024 \
  --throughput -1 --producer-props bootstrap.servers=localhost:9092 \
  acks=all compression.type=lz4 batch.size=65536 linger.ms=10

Output

1000000 records sent, 245098.0 records/sec (239.36 MB/sec), 12.3 ms avg latency, 89 ms max latency.

# Consumer benchmark
kafka-consumer-perf-test.sh --bootstrap-server localhost:9092 \
  --topic perf-test --messages 1000000 --threads 1

Key Takeaway: Always benchmark before and after tuning. Use kafka-producer-perf-test.sh to find your cluster's limits. The biggest wins usually come from compression (lz4) and batching (linger.ms=10).

Practice Exercises

Medium Build a Mini Project

Combine concepts from this tutorial to build a small utility or tool.

Medium Debug Challenge

Introduce a bug in one of the code examples and practice finding and fixing it.

Hard Refactoring Exercise

Rewrite one example using a different approach and compare the tradeoffs.