Apache Kafka on Docker

Running Apache Kafka on Docker

We shall use wurstmeister kafka and zookeeper.

Confluent's version of kafka is by far most easiest and complete suite. But it involves cost. As a developer we look for open source version that we can use. Hence wurstmeister is a good alternative.

This tutorial does not cover
- docker
- kafka
It only talks about installing kafka using Docker images for developer machines.

Using docker-compose is easiest way of using dockerized kafka. In this post, I am not going to talk about docker compose, but using bare docker itself.

Table of Contents

1. Download wurstmeister kafka and zookeeper docker images

2. Running as a container

3. Testing

5. Conclusion

Download 'wurstmeister' kafka and zookeeper docker images

Note: Docker is by default configured to look for images in dockerhub if it doesn't have it locally. So you may choose to ignore this step and directly jump to "Running docker images as containers".

Wurstmeister images are available on docker-hub at these locations

- https://hub.docker.com/r/wurstmeister/kafka/
- https://hub.docker.com/r/wurstmeister/zookeeper

Download docker image for kafka as

     docker pull wurstmeister/kafka

Download docker image for zookeeper as
    docker pull wurstmeister/zookeeper

Docker images are loaded as


Note: As of writing this tutorial,
kafka docker image tag is 2.12-2.3.0 latest updated in July 2019.
zookeeper docker image tag just says 'latest' on docker hub - updated in Feb 2019.

Running docker images as containers

Running zookeeper image as container

Start containers as
      docker run -d -p 2181:2181 -p 2888:2888 -p 3888:3888 --name=zookeeper wurstmeister/zookeeper

This starts zookeeper.

Running kafka broker image as container

Start container as
     docker run -d -p 9092:9092 -e KAFKA_BROKER_ID=1 -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_ADVERTISED_HOST_NAME=localhost --link zookeeper --name kafka wurstmeister/kafka

This starts kafka container.

Explaination:
- we have connected internal ports of zookeeper and kafka broker with ports of machine where docker is run (localhost). This is done using -p.
- we have set some environment variables that are required by kafka broker. One of them is a broker id.
- since kafka container doesn't have direct access to zookeeper (due to docker environment), we link it to zookeeper container using --link.

Testing

Kafka comes with command line tools. These tools are also available in kafka docker image.

We can access them by attaching terminal to running kafka docker container.

    docker exec -ti kafka bash

Or, start a container just for this purpose

    docker run -ti --link zookeeper --rm wurstmeister/kafka bash

Then cd to bin director where all the tools are

    cd /opt/kafka/bin/

Or, you could download a copy of these tools and install on your machine.
If you have installed on your local machines, then use following commands as is.
If you are running from docker image, then replace "localhost" with "zookeeper". This is because "localhost" from within docker means docker container itself. But zookeeper is running in a different container. Since we linked that container to kafka container using --link, we can use "zookeeper".

List all topics

    ./kafka-topics.sh --zookeeper localhost:2181 --list

Create a topic

    ./kafka-topics.sh --create --topic my_test_topic --partitions 1 --replication-factor 1 --if-not-exists --zookeeper localhost:2181

Send message to topic

    ./kafka-console-producer.sh --broker-list localhost:9092 --topic my_test_topic

Read messages from topic

    ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my_test_topic --from-beginning

Note: You can keep producer and consumer active at same time through two different terminals and see it in action

Get details of topic

    ./kafka-topics.sh --zookeeper localhost:2181 --describe --topic my_test_topic

Delete a topic

    ./kafka-topics.sh --zookeeper localhost:2181 --delete --topic my_test_topic

Conclusion

Do not use this method on production. This is only for developer machines.

After this setup, access zookeeper at 2181 and kafka brokers at 9092.

Even though kafka and zookeeper are running as docker containers, we have exposed them on default ports. Zookeeepr - 2181 and kafka broker - 9092. So from your application, you can use "localhost".

While reading about dockerized kadka, I came across this nice blog. If you plan to do a little more complicated setup, you should read through it: https://rmoff.net/2018/08/02/kafka-listeners-explained/

Feedback

Please let me know your thoughts through comments or by emailing me directly.