Difficulty: intermediate
Estimated Time: 30 minutes

In this scenario you will create work with the OCI [Event] Streaming Service - a managed Pub/Sub service, similar to Apache Kafka. Events are published to Streams (aka Topics) over HTTP calls and consumed from those Streams, again using HTTP calls. Consumption can be done from several starting point (at or from a specific offset, from a specific time, as much as is available, only new events) . Note that consumers have to actively come to the Streaming service and collect messages. There is no push mechanism that will trigger listeners whenever new messages have been published to the Stream. There is no concept of subscription: consumers just create a 'cursor' (somewhat similar to a session or a [long running] query) and start pulling messages in the context of that cursor.

Streaming can be used for messaging, ingesting high-volume data such as application logs, operational telemetry, web click-stream data, or other use cases in which data is produced and processed continually and sequentially in a publish-subscribe messaging model. Messages are retained for up to 7 days. There are limits on how much data a Stream can handle per second, but these limits are fairly high (1MB per partition per second, 1MB maximum message size, five consume calls per second).

The scenario uses an Ubuntu 20.04 environment with Docker, OCI CLI and Fn CLI. Before you can start the steps in the scenario, the two Command Line interfaces are downloaded and installed. This will take about one minute. You will need Postman as well, if you want to invoke the API Gateway directly from your computer.

  • you already have deployed a function called hello# in an application called lab# where # is a number (either 1 or a number assigned to you in a workshop); the function app and function are in an OCI compartment lab-compartment. (this is part of the Functions on OCI scenario)
  • an API Gateway already has been provided in compartment lab-compartment with permissions to access functions in this compartment; the API Gateway is associated with a public subnet (this is part of the OCI Tenancy Preparation Scenario)
  • there is already an API Deployment on the API Gateway - called MY_API_DEPL#
  • there already is a Stream in compartment lab-compartment, called lab-stream

The scenario was prepared for the Meetup Workshop Cloud Native application development on Oracle Cloud Infrastructure on January 20th, hosted by AMIS|Conclusion in Nieuwegein in collaboration with REAL (the Red Expert Alliance) and Link from Portugal. The scenario was updated for the REAL OCI Handson Webinar Series that started in June 2020.

Resources

Blog article introducing OCI Streaming [https://technology.amis.nl/2020/01/07/oracle-cloud-streaming-service-scalable-reliable-kafka-like-event-service-on-oci/]

Documentation on OCI Streaming Service: https://docs.cloud.oracle.com/iaas/Content/Streaming/Concepts/streamingoverview.htm

Blog article Todd Sharp: Getting your feet wet with Oracle Streaming: https://blogs.oracle.com/developers/getting-your-feet-wet-with-oci-streams

OCI Streaming - Publisher => Stream => Consumer(s)

Summary

This completes your explorations with Functions on a local Fn Server. The next scenario you may want to explore looks at deploying Function to Oracle Cloud Infrastructure.

Background Resources

Tutorial Getting Started with Fn environment https://github.com/fnproject/tutorials/blob/master/install/README.md

Tutorial FN with Node https://github.com/fnproject/tutorials/blob/master/node/intro/README.md

Docs on Fn on OCI https://docs.cloud.oracle.com/iaas/Content/Functions/Tasks/functionscreatefncontext.htm

OCI (Event) Streaming, combined with Functions and API Gateway

Step 1 of 5

Step 1 - Check on Stream on Console and Prepare Environment Settings, Fn and OCI CLI

Execute the following command to install the OCI CLI:

curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh > install-oci-cli.sh
chmod +777 install-oci-cli.sh
sudo ./install-oci-cli.sh --accept-all-defaults

# add this line to ~/.profile - to make oci a recognized shell command
echo 'oci() { /root/bin/oci "[email protected]"; }' >> ~/.profile
# reload ~/.profile
. /root/.profile

You need to provide details on the OCI tenancy you will work in and the OCI user you will work as. Please open the IDE tab and edit these two files:

  • ~/.oci/config
  • ~/.oci/oci_api_key.pem

Paste the contents that you prepared in the OCI Tenancy preparation scenario.

Set the environment variable LAB_ID to 1 - unless you are in a workshop with multiple participants and each uses their own number.

export LAB_ID=1

The environment is currently being prepared. While that is happening, we can take a look at the Stream lab-stream in the lab-compartment. Note: there is a fairly strict limit on the number of Stream (partitions) that we are allowed to create (out of the box); therefore we are all sharing the stream in this workshop.

Open the OCI Console as lab-user at: https://console.us-ashburn-1.oraclecloud.com/storage/streaming . Here you should see the stream lab-stream. echo "Open the console at https://console.YOUR_REGION.oraclecloud.com/storage/streaming"

Click on the stream lab-stream to go to the details page. Click on Produce Test Message to... well, produce a test message of course.

Type a message and press Produce. The console will indicate that the message was produced successfully.

To check on the message, close the popup window and press Load Messages. All recently published messages on the stream are displayed – including the one that was just published.

This is the simplest example of pub/sub through OCI Streaming.

Finalizing the Environment

Some of the steps in this scenario require the use of the OCI Command Line Interface.

Execute the following command to install the OCI CLI:

curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh > install-oci-cli.sh
chmod +777 install-oci-cli.sh
sudo ./install-oci-cli.sh --accept-all-defaults

# add this line to ~/.profile - to make oci a recognized shell command
echo 'oci() { /root/bin/oci "[email protected]"; }' >> ~/.profile
# reload ~/.profile
. /root/.profile

You need to provide details on the OCI tenancy you will work in and the OCI user you will work as. Please open the IDE tab and edit these two files:

  • ~/.oci/config
  • ~/.oci/oci_api_key.pem

Paste the contents that you prepared in the OCI Tenancy preparation scenario.

Finalizing the Environment

Set the environment variable LAB_ID to 1 - unless you are in a workshop with multiple participants and each uses their own number.

export LAB_ID=1

Try out the following command to get a list of all namespaces you currently have access to - based on the OCI Configuration defined above.

oci os ns get

If you get a proper response, the OCI is configured correctly and you can proceed. If you run into an error, ask for help from your instructor.

You need to perform one more edit action on file ~/.oci/config. Open this file in the editor. Copy the entire contents and paste it below the current contents. Now change [DEFAULT] to [FN] in the duplicate block. The file will now look something like:

[DEFAULT]
user=ocid1.user.oc1..aaaaaaaazr7eselv5q
fingerprint=fd:db:13:bd:31
tenancy=ocid1.tenancy.oc1..aaaaaaaaggg6okq
region=us-ashburn-1
key_file=/root/.oci/oci_api_key.pem
[FN]
user=ocid1.user.oc1..aaaaaaaazr7eselv5q
fingerprint=fd:db:13:bd:31
tenancy=ocid1.tenancy.oc1..aaaaaaaaggg6okq
region=us-ashburn-1
key_file=/root/.oci/oci_api_key.pem

Environment Preparation

Check the installed version of Fn CLI. Note: we do not need the Fn server at this stage.

fn version

Configure and set remote context

List the currently available Fn contexts

fn list contexts

Create an appropriate Fn context for working with OCI as provider (see OCI Docs on Functions).

fn create context lab-fn-context --provider oracle

fn use context lab-fn-context

Prepare a number of environment variables. Note: the assumptions here are that you are working in a tenancy in the Ashburn region and a compartment called lab-compartment exists as well as an API Gateway lab-apigw in that same compartment as well as an API Deployment called MY_API_DEPL# on the API Gateway. We need to get references to these resources in order to create new resources in the right place.

export REGION=$(oci iam region-subscription list | jq -r '.data[0]."region-name"')
export REGION_KEY=$(oci iam region-subscription list | jq -r '.data[0]."region-key"')
export USER_OCID=$(oci iam user list --all | jq -r  '.data |sort_by(."time-created")| .[0]."id"')
export TENANCY_OCID=$(oci iam user list --all | jq -r  '.data[0]."compartment-id"') 
cs=$(oci iam compartment list)
export compartmentId=$(echo $cs | jq -r --arg display_name "lab-compartment" '.data | map(select(."name" == $display_name)) | .[0] | .id')

apigws=$(oci api-gateway gateway list -c $compartmentId)
export apiGatewayId=$(echo $apigws | jq -r --arg display_name "lab-apigw" '.data.items | map(select(."display-name" == $display_name)) | .[0] | .id')
depls=$(oci api-gateway deployment list -c $compartmentId)
deploymentEndpoint=$(echo $depls | jq -r --arg display_name "MY_API_DEPL_$LAB_ID" '.data.items | map(select(."display-name" == $display_name)) | .[0] | .endpoint')
apiDeploymentId=$(echo $depls | jq -r --arg display_name "MY_API_DEPL_$LAB_ID" '.data.items | map(select(."display-name" == $display_name)) | .[0] | .id')
nss=$(oci os ns get)
export ns=$(echo $nss | jq -r '.data')

Update the context with the settings relevant for this workshop.

fn update context oracle.compartment-id $compartmentId

fn update context api-url https://functions.$REGION.oci.oraclecloud.com

fn update context registry ${REGION_KEY,,}.ocir.io/$ns/cloudlab-repo

fn update context oracle.profile FN

You can list the currently available Fn contexts again and see whether your changes had an effect (of course they did)

fn list contexts

Login Docker to OCI Container Registry

Next and finally, login to the private Docker Registry that is prepared for you on OCI.

The username you have to provide is composed of <tenancy-namespace>/<username>.

NAMESPACE=$(oci os ns get| jq -r  '.data')
USER_USERNAME=$(oci iam user list --all | jq -r  '.data |sort_by(."time-created")| .[0]."name"')
echo "Username for logging in into Container Registry is $NAMESPACE/$USER_USERNAME"

The password is an Authentication Token generated for the specified user, in the OCI Tenancy preparation scenario. If you do not remember the authentication token, you can generate another one in the OCI Console: https://console.REGION.oraclecloud.com/identity/users//swift-credentials or using the instructions in the preparation scenario.

echo "Open the console at https://console.${REGION,,}.oraclecloud.com/identity/users/$USER_OCID/swift-credentials"

Now you can perform the login. Type the username and press enter, then type or paste the authentication token and press enter again.

docker login ${REGION_KEY,,}.ocir.io