Kafka | Notion

concepts
- broker aka node: a cluster is built by one or more servers. The servers forming the storage layer are called brokers
- event: have sequential IDs? key, value, timestamp, and optional metadata headers.
- producers: client applications that publish (write) events to Kafka
- consumer: client application subscribed to (read and process) events from Kafka
- topic: group of events
- consumer group: all in a group share the same current-offset
- partition: topics are partitioned, meaning a topic is spread over a number of "buckets" located on different Kafka brokers
- replication: to make your data fault-tolerant and highly-available, every topic can be replicated, so that there are always multiple brokers that have a copy
- bootstrap servers: just a subset list of brokers that clients should know about and initially connect to. they advertise who all the other listeners are.
default retention is 7 days or 1 gb, whichever is lower (source)
note: you need as many partitions as you want consumers at a time.
- it’s not trivial to have a FIFO queue that distributes messages among thousands of consumers. you need thousands of partitions for max throughput.

Tools

kafkacat
kaf
kafkactl

Tips

Inspect topics and partitions

$ kafkacat -b localhost:9092 -L
Metadata for all topics (from broker -1: localhost:9092/bootstrap):
 3 brokers:
  broker 2 at b-2.plasmic-posthog-kafka.timval.c7.kafka.us-west-2.amazonaws.com:9092
  broker 3 at b-3.plasmic-posthog-kafka.timval.c7.kafka.us-west-2.amazonaws.com:9092 (controller)
  broker 1 at b-1.plasmic-posthog-kafka.timval.c7.kafka.us-west-2.amazonaws.com:9092
 10 topics:
  topic "events_dead_letter_queue" with 5 partitions:
    partition 0, leader 1, replicas: 1,3,2, isrs: 3,2,1
    partition 1, leader 3, replicas: 3,2,1, isrs: 3,2,1
    partition 2, leader 2, replicas: 2,1,3, isrs: 3,2,1
    partition 3, leader 1, replicas: 1,2,3, isrs: 3,2,1
    partition 4, leader 3, replicas: 3,1,2, isrs: 3,2,1
  topic "readings" with 5 partitions:
    partition 0, leader 3, replicas: 3,2,1, isrs: 3,2,1
    partition 1, leader 2, replicas: 2,1,3, isrs: 3,2,1
    partition 2, leader 1, replicas: 1,3,2, isrs: 3,2,1
    partition 3, leader 3, replicas: 3,1,2, isrs: 3,2,1
    partition 4, leader 2, replicas: 2,3,1, isrs: 3,2,1
  topic "clickhouse_person_unique_id" with 5 partitions:
    partition 0, leader 1, replicas: 1,3,2, isrs: 3,2,1
    partition 1, leader 3, replicas: 3,2,1, isrs: 3,2,1
    partition 2, leader 2, replicas: 2,1,3, isrs: 3,2,1
    partition 3, leader 1, replicas: 1,2,3, isrs: 3,2,1
    partition 4, leader 3, replicas: 3,1,2, isrs: 3,2,1
  topic "__consumer_offsets" with 50 partitions:
    partition 0, leader 1, replicas: 1,3,2, isrs: 3,2,1
    partition 1, leader 3, replicas: 3,2,1, isrs: 3,2,1
    partition 2, leader 2, replicas: 2,1,3, isrs: 3,2,1
    partition 3, leader 1, replicas: 1,2,3, isrs: 3,2,1
    partition 4, leader 3, replicas: 3,1,2, isrs: 3,2,1
    partition 5, leader 2, replicas: 2,3,1, isrs: 3,2,1
    partition 6, leader 1, replicas: 1,3,2, isrs: 3,2,1
    partition 7, leader 3, replicas: 3,2,1, isrs: 3,2,1
    partition 8, leader 2, replicas: 2,1,3, isrs: 3,2,1
...

List consumer groups

$ ./bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092
amazon.msk.canary.group.broker-1
clickhouse-ingestion
felipe_group
readings_consumer_group1
felipe_group3
felipe_group2
group2
amazon.msk.canary.group.broker-3
amazon.msk.canary.group.broker-2
group1

Inspect consumer group

$ ./bin/kafka-consumer-groups.sh --describe --group group1 --bootstrap-server b-3.plasmic-posthog-kafka.timval.c7.kafka.us-west-2.amazonaws.com:9092,b-2.plasmic-posthog-kafka.timval.c7.kafka.us-west-2.amazonaws.com:9092,b-1.plasmic-posthog-kafka.timval.c7.kafka.us-west-2.amazonaws.com:9092

GROUP           TOPIC                         PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                                                        HOST            CLIENT-ID
group1          clickhouse_person_unique_id   2          -               17              -               ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id-03732755-51bc-4cec-91db-a37adc2079f5       /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id
group1          clickhouse_person_unique_id   0          -               15              -               ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id-03732755-51bc-4cec-91db-a37adc2079f5       /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id
group1          clickhouse_person_unique_id   4          -               13              -               ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id-03732755-51bc-4cec-91db-a37adc2079f5       /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id
group1          clickhouse_person_unique_id   3          -               21              -               ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id-03732755-51bc-4cec-91db-a37adc2079f5       /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id
group1          clickhouse_person_unique_id   1          -               20              -               ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id-03732755-51bc-4cec-91db-a37adc2079f5       /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id
group1          plugin_log_entries            4          946             962             16              ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_plugin_log_entries-e97dff5a-34da-4b3d-8890-b395f0f935fd       /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_plugin_log_entries
group1          plugin_log_entries            1          926             946             20              ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_plugin_log_entries-e97dff5a-34da-4b3d-8890-b395f0f935fd       /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_plugin_log_entries
group1          plugin_log_entries            0          818             832             14              ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_plugin_log_entries-e97dff5a-34da-4b3d-8890-b395f0f935fd       /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_plugin_log_entries
group1          plugin_log_entries            2          890             899             9               ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_plugin_log_entries-e97dff5a-34da-4b3d-8890-b395f0f935fd       /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_plugin_log_entries
group1          plugin_log_entries            3          845             858             13              ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_plugin_log_entries-e97dff5a-34da-4b3d-8890-b395f0f935fd       /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_plugin_log_entries
group1          clickhouse_person             3          19132           19536           404             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person-dcfda8bc-a262-49ef-9da7-bc49176d919f                   /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person
group1          clickhouse_person             2          19271           19669           398             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person-dcfda8bc-a262-49ef-9da7-bc49176d919f                   /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person
group1          clickhouse_person             0          19300           19698           398             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person-dcfda8bc-a262-49ef-9da7-bc49176d919f                   /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person
group1          clickhouse_person             1          19199           19616           417             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person-dcfda8bc-a262-49ef-9da7-bc49176d919f                   /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person
group1          clickhouse_person             4          19163           19569           406             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person-dcfda8bc-a262-49ef-9da7-bc49176d919f                   /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person
group1          events_dead_letter_queue      1          38              170             132             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events_dead_letter_queue-8083d49f-c495-485a-b55e-416bf4325ab3 /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events_dead_letter_queue
group1          events_dead_letter_queue      4          26              157             131             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events_dead_letter_queue-8083d49f-c495-485a-b55e-416bf4325ab3 /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events_dead_letter_queue
group1          events_dead_letter_queue      2          32              140             108             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events_dead_letter_queue-8083d49f-c495-485a-b55e-416bf4325ab3 /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events_dead_letter_queue
group1          events_dead_letter_queue      0          35              186             151             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events_dead_letter_queue-8083d49f-c495-485a-b55e-416bf4325ab3 /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events_dead_letter_queue
group1          events_dead_letter_queue      3          36              181             145             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events_dead_letter_queue-8083d49f-c495-485a-b55e-416bf4325ab3 /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events_dead_letter_queue
group1          clickhouse_events_proto       1          2971117         2978762         7645            ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events-76b6d2a8-08d3-485b-9c98-7edd2ee929d6                   /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events
group1          clickhouse_events_proto       2          4471988         4480489         8501            ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events-76b6d2a8-08d3-485b-9c98-7edd2ee929d6                   /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events
group1          clickhouse_events_proto       4          4447418         4464073         16655           ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events-76b6d2a8-08d3-485b-9c98-7edd2ee929d6                   /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events
group1          clickhouse_events_proto       0          5404052         5413449         9397            ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events-76b6d2a8-08d3-485b-9c98-7edd2ee929d6                   /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events
group1          clickhouse_events_proto       3          4733941         4753470         19529           ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events-76b6d2a8-08d3-485b-9c98-7edd2ee929d6                   /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_events
group1          clickhouse_person_distinct_id 3          19220           19621           401             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id2-1d62e42b-749e-4aff-a0f8-6d0755f0a5b2      /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id2
group1          clickhouse_person_distinct_id 4          19133           19522           389             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id2-1d62e42b-749e-4aff-a0f8-6d0755f0a5b2      /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id2
group1          clickhouse_person_distinct_id 1          19185           19582           397             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id2-1d62e42b-749e-4aff-a0f8-6d0755f0a5b2      /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id2
group1          clickhouse_person_distinct_id 0          19049           19457           408             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id2-1d62e42b-749e-4aff-a0f8-6d0755f0a5b2      /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id2
group1          clickhouse_person_distinct_id 2          19078           19490           412             ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id2-1d62e42b-749e-4aff-a0f8-6d0755f0a5b2      /192.168.68.145 ClickHouse-chi-posthog-posthog-0-0-0.chi-posthog-posthog-0-0.posthog.svc.cluster.local-posthog-kafka_person_distinct_id2

Reset offsets

More examples at https://gist.github.com/filimonov/1646259d18b911d7a1e8745d6411c0cc

kafka-consumer-groups.sh --topic topic:0,1,2 --group id1 --reset-offsets --to-latest --execute