Difficulty: Advanced
Estimated Time: 10 minutes

Portworx is the cloud native storage company that enterprises depend on to reduce the cost and complexity of rapidly deploying containerized applications across multiple clouds and on-prem environments. With Portworx, you can manage any database or stateful service on any infrastructure using any container scheduler. You get a single data management layer for all of your stateful services, no matter where they run.

A popular Kubernetes storage and persistent storage for Docker solution, Portworx is a clustered block storage solution and provides a Cloud-Native layer from which containerized stateful applications programmatically consume block, file and object storage services directly through the scheduler.

In this tutorial, you will learn how to deploy Kafka to Kubernetes using Helm and Portworx:

Step: Deploy Zookeeper and Kafka

As of v0.8 Kafka uses zookeeper for storing variety of configurations as K,V in the ZK data tree and use them across the cluster in a distributed fashion. So our first task will be deploy a 3 node ZK cluster using a StatefulSet and have each node use a Portworx volume to ensure high availability. We will then deploy a Kafka StatefulSet which uses our ZK cluster and also has Portworx volumes, this time with 2 replicas so we can have fast failover of the Kafka nodes.

Step: Perform a failover test

In this step we will simulate a node failure to show how Kubernetes can reschedule your Kafka pod to run on another node in your cluster. Portworx is able to ensure that the rescheduled pod connects to it data from anywhere in the cluster but goes a step further by ensuring that the pod will be scheduled on a node which has a local copy of the data so that you get the best level of performance. This advanced orchestration of stateful workloads is handled by the STorage ORchestrator for Kubernetes (STORK).

Step: Expand the volume and Kafka cluster

In this step we will show how Portworx volumes can be dynamically expanded with zero downtime. Portworx are thinly provisioned and the size of the PVC is used as a maximum size. Once you reach this upper limit on the size of your Kafka database you can easily perform this operation to add more capacity to your volume. We will also expand our Kafka cluster from one to three nodes so that we can show how to snapshot entire clusters in the next and final step.

Other things you should know

To learn more about how why running kafka on Portworx is a great idea take a look at the following links:

This scenario assumes you have already covered the following scenarios:

Thank you for trying the playground. To view all our scenarios, go here

To learn more about Portworx, below are some useful references.

Deploy Kafka and Zookeeper on Kubernetes using Portworx volumes

Step 1 of 8

Wait for Kubernetes & Portworx to be ready

Let's first wait for Kubernetes and Portworx to be ready.

Step: Wait for Kubernetes to be ready

Click the below section which waits for all Kubernetes nodes to be ready.

watch kubectl get nodes

When all 4 nodes show status Running then hit clear to ctrl-c and clear the screen.

Step: Wait for Portworx to be ready

Watch the Portworx pods and wait for them to be ready on all the nodes. This can take a few minutes since it involves pulling multiple docker images. You will see 'No resources found' until all images are pulled.

watch kubectl get pods -n kube-system -l name=portworx -o wide

When all the pods show STATUS Running and READY 1/1 then hit clear to ctrl-c and clear the screen.

Now that we have the Portworx cluster up, let's proceed to the next step!