Difficulty: beginner
Estimated Time: 45 minutes

In this scenario we will learn how to deploy Kafka - Confluent Community Edition and Enterprise Edition on top of Oracle Cloud Infraestructure. Confluent's Kafka offering is the leader in the market, which includes the possibility to install their software on your own infrastructure. But not only that, but a set of terraform plans has been created and validated to deploy their platform on top of OCI.

The information of those terraform plans can be found here: https://blogs.oracle.com/cloud-infrastructure/confluent-platform-now-validated-on-oracle-cloud-infrastructure

And the github repo is this one: https://github.com/oracle-quickstart/oci-confluent (All credits on the Terraform Plans to the contributors of this github repo https://github.com/oracle-quickstart/oci-confluent)

Kafka is conformed by many components:

  • Brokers
  • Zookeepers
  • REST Proxy Servers
  • KSQL Servers
  • Connect Servers
  • Schema Registry

Intro

Terraform plans are going to provision the compute instances, install, deploy and configure the services and create a whole Confluent Kafka environment within OCI.

This scenario will help you to understand:

  • The usage of terraform plans
  • The Confluent Kafka Architecture
  • Create and destroy a Confluent Kafka Cluster deployment in minutes

Important notes to use this scenario:

  • As any other scenario of this series, you need to go to the pre-requisite katakoda scenario to configure your tenant. If you haven't done that please go here: https://www.katacoda.com/redexpertalliance/courses/oci-course/oci-lab-preparation-cloud-trial
  • This is not a scenario to learn about Terraform, and if you do not have knowledge about it, is OK, you can still perform the scenario
  • This is not going to teach you Kafka, but in the other hand, is going to help you to provision a complete cluster where you can use it and learn it

Summary

This complets the scenario for OCI compute instance creation. This scenario also helped us to create a proxy server for our OKE API Server.

Confluent Kafka provisioning on OCI

Step 1 of 6

Step 1 - Prepare the environment with OCI CLI

Some of the steps in this scenario require the use of the OCI Command Line Interface.

Execute the following command to install the OCI CLI:

curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh > install-oci-cli.sh
chmod +777 install-oci-cli.sh
sudo ./install-oci-cli.sh --accept-all-defaults

# add this line to ~/.profile - to make oci a recognized shell command
echo 'oci() { /root/bin/oci "[email protected]"; }' >> ~/.profile
# reload ~/.profile
. /root/.profile

You need to provide details on the OCI tenancy you will work in and the OCI user you will work as. Please open the IDE tab and edit these two files:

  • ~/.oci/config
  • ~/.oci/oci_api_key.pem

Paste the contents that you prepared in the OCI Tenancy preparation scenario.

Finalizing the Environment

Set the environment variable LAB_ID to 1 - unless you are in a workshop with multiple participants and each uses their own number.

export LAB_ID=1

Try out the following command to get a list of all namespaces you currently have access to - based on the OCI Configuration defined above.

oci os ns get

If you get a proper response, the OCI is configured correctly and you can proceed. If you run into an error, ask for help from your instructor.

Environment Preparation

export REGION=$(oci iam region-subscription list | jq -r '.data[0]."region-name"')
export REGION_KEY=$(oci iam region-subscription list | jq -r '.data[0]."region-key"')
export USER_OCID=$(oci iam user list --all | jq -r  '.data |sort_by(."time-created")| .[0]."id"')
export TENANCY_OCID=$(oci iam user list --all | jq -r  '.data[0]."compartment-id"') 
cs=$(oci iam compartment list)
export compartmentId=$(echo $cs | jq -r --arg display_name "lab-compartment" '.data | map(select(."name" == $display_name)) | .[0] | .id')

# get namespace
nss=$(oci os ns get)
export ns=$(echo $nss | jq -r '.data')

Create compartment to organize the Kafka Confluent resources that you will deploy.

Create a compartment with the name confluent_ce (confluent community edition)

export confluent_compartment=confluent_ce

compartment=$(oci iam compartment create --compartment-id "$TENANCY_OCID"  --name "$confluent_compartment" --description "Compartment for resources for Confluent Community Edition")
echo "JSON response from the command to create the compartment:"
echo $compartment
compartmentId=$(echo $compartment | jq --raw-output .data.id)
echo The OCID for the compartment is:  $compartmentId

To set an environment variable $compartmentId fetch the OCID from the compartment with this command (this also works when the compartment already existed prior to running this scenario):

cs=$(oci iam compartment list)
export compartmentId=$(echo $cs | jq -r --arg display_name "$confluent_compartment" '.data | map(select(."name" == $display_name)) | .[0] | .id')

In your OCI console you will see the new compartment created:

Now let's prepare Terraform in the next Step.