Welcome!
CyberArk Secretless Broker on Kubernetes
Introduction
In this scenario, you will get an overview of the Secretless Broker and learn how to deploy it, along with an application that has no knowledge of passwords and a database backend.
What is Secretless Broker?
With the Secretless Broker feature of Conjur, applications can securely connect to databases, services and other protected resources – without fetching or managing secrets.
Secretless Broker is an independent and extensible open source community project maintained by CyberArk. Today Secretless Broker works within Kubernetes and OpenShift container platforms with Conjur, Application Access Manager’s Dynamic Access Provider, and Kubernetes Secrets vaults.
How Does Secretless Broker Work?
When an application needs to securely access a resource, such as a database, instead of providing access credentials, the app simply makes a local connection request to Secretless Broker, which then automatically authenticates the app, fetches the required credentials from a Vault and establishes a connection to the database.
- From the developer’s perspective instead of needing to include code in their application to fetch the credentials from a Vault and then use the credentials to access the resource, the developer simply configures the application to connect to the required resource via the Secretless Broker, without needing to change the application code.
- From the security perspective, credentials can no longer be inadvertently logged or exposed by the application because, with Secretless Broker, the application code does not get access to the credential, so it cannot leak secrets.
Congratulations!
You've completed the scenario!
Scenario Rating
Your environment is currently being packaged as a Docker container and the download will begin shortly. To run the image locally, once Docker has been installed, use the commands
cat scrapbook_diverdane_secretless-k8s-demo_container.tar | docker load
docker run -it /diverdane_secretless-k8s-demo:
Oops!! Sorry, it looks like this scenario doesn't currently support downloads. We'll fix that shortly.

Steps
CyberArk Secretless Broker on Kubernetes
Step 1: Before We Begin
This is a detailed, step-by-step tutorial.
With this tutorial, you will learn how to use the CyberArk Secretless Broker to deploy an application that connects to a database without knowing its password.
Overview
Applications and application developers should be incapable of leaking secrets.
To achieve that goal, you’ll play two roles in this tutorial:
- A Security Admin who handles secrets, and has sole access to those secrets
- An Application Developer with no access to secrets.
The situation looks like this:
Specifically, we will:
As the security admin:
- Create a PostgreSQL database
- Create a DB user for the application
- Add that user’s credentials to Kubernetes Secrets
- Configure Secretless to connect to PostgreSQL using those credentials
As the application developer:
- Configure the application to connect to PostgreSQL via Secretless
- Deploy the application and the Secretless sidecar
Up next...
Play the role of a Security Admin and learn how to set up PostgreSQL and configure Secretless.
Step 2: Deploy PostgreSQL Database
The Security Admin sets up PostgreSQL, configures Secretless, and has sole access to the credentials.
Deploy a PostgreSQL Statefulset
PostgreSQL is stateful, so we’ll use a StatefulSet to manage it.
To deploy a PostgreSQL StatefulSet:
Create a dedicated namespace for the storage backend:
kubectl create namespace postgres-backend-ns
Click here to see expected output...
namespace/postgres-backend-ns created
Create a self-signed certificate (see PostgreSQL documentation for more info):
openssl req -new -x509 -days 365 \ -nodes -text -out server.crt \ -keyout server.key -subj "/CN=pg" chmod og-rwx server.key
Click here to see expected output...
master $ openssl req -new -x509 -days 365 \ -nodes -text -out server.crt \ -keyout server.key -subj "/CN=pg" Generating a 2048 bit RSA private key ...................................................................................................................+++ ....................+++ writing new private key to 'server.key' ----- master $ chmod og-rwx server.key master $
Store the certificate files as a Kubernetes secret in the postgres-backend-ns namespace:
kubectl --namespace postgres-backend-ns \ create secret generic postgres-backend-certs \ --from-file=server.crt \ --from-file=server.key
Click here to see expected output...
secret/postgres-backend-certs created
NOTE: While Kubernetes Secrets are more secure than hard-coded ones, in a real deployment you should secure secrets in a fully-featured vault, like Conjur.
Deploy a PostgreSQL StatefulSet and Service:
kubectl --namespace postgres-backend-ns apply -f postgres.yml
Click here to see expected output...
statefulset.apps/pg created service/postgres-backend created
View the StatefulSet manifest here: postgres.yml
Some Details: Click here to see notes for the StatefulSet manifest...
- The certificate files for your database server are mounted in a volume with defaultMode: 384 giving it permissions 0600 (Why? Because 600 in base 8 = 384 in base 10).
- The pod is deployed with 999 as the group associated with any mounted volumes, as indicated by fsGroup: 999. 999 is a the static postgres gid, defined in the postgres Docker image
This StatefulSet uses the DockerHub
postgres:9.6
container. On startup, the container creates a superuser from the environment variables:- POSTGRES_USER
- POSTGRES_PASSWORD
which we set to the values security_admin_user and security_admin_password, respectively.
Going forward, we’ll call these values the admin-credentials, to distinguish them from the application-credentials our application will use. In the scripts below, we’ll refer to the admin-credentials by the environment variables:
- SECURITY_ADMIN_USER
- SECURITY_ADMIN_PASSWORD.
Confirm that the PostgreSQL StatefulSet Pod has Started and is Healthy: (It may take a minute or so for the pod to get to 'Running' state.)
kubectl --namespace postgres-backend-ns get pods
Click here to see expected output (it may take a minute or so to get to 'Running' state)...
NAME READY STATUS RESTARTS AGE pg-0 1/1 Running 0 6s
Confirm the Successful Creation of the PostgreSQL Service:
kubectl --namespace postgres-backend-ns get svc postgres-backend
Click here to see expected output (Cluster IP address may be different)...
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE postgres-backend NodePort 10.106.232.98 <none> 5432:30001/TCP 2m
NOTE: This service is exposed via NodePort 30001 on every node in the cluster. This will be used in the next step.
Confirm that You Can Log into the Database and List Users:
The database has no data yet, but we can verify it works by logging in as the security admin and listing the users. To access the postgres-backend service, we'll use:
- The IP address of the worker node (either node would do)
- The NodePort on which this service is exposed (30001)
The IP address of the worker node can be displayed with this command:
kubectl get nodes -o wide | grep -v "master"
Setting up environment variables for accessing the database:
export SECURITY_ADMIN_USER=security_admin_user export SECURITY_ADMIN_PASSWORD=security_admin_password export REMOTE_DB_HOST=[[HOST2_IP]] export REMOTE_DB_PORT=30001
And then list the users using a
postgres:9.6
image as a client:docker run --rm -it \ -e PGPASSWORD=${SECURITY_ADMIN_PASSWORD} \ postgres:9.6 psql \ -U ${SECURITY_ADMIN_USER} \ "postgres://${REMOTE_DB_HOST}:${REMOTE_DB_PORT}/postgres" \ -c "\du"
Click here to see expected output...
List of roles Role name | Attributes | Member of ---------------------+------------------------------------------------------------+----------- security_admin_user | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
Up next...
Continue as an Security Admin and set up Application namespace, credentials, and service account
Step 3: Create Application Database and Service Account
Create Application Database
In this section, we'll create the application database and user, and securely store the user's credentials:
- Create the application database
- Create the pets table in that database
- Create an application user with limited privileges: SELECT and INSERT on the pets table
- Store these database application-credentials in Kubernetes secrets.
So we can refer to them later, export the database name and application-credentials as environment variables:
export APPLICATION_DB_NAME=demo_app_db
export APPLICATION_DB_USER=demo_app_user
export APPLICATION_DB_INITIAL_PASSWORD=demo_app_user_password
To perform the steps listed above, first create a file containing SQL commands:
cat > sql_cmds.txt << EOSQL
/*----------------------------------*/
/* Create Application Database */
/*----------------------------------*/
CREATE DATABASE ${APPLICATION_DB_NAME};
/*----------------------------------*/
/* Connect to Database */
/*----------------------------------*/
\c ${APPLICATION_DB_NAME};
/*----------------------------------*/
/* Create a pets Table */
/*----------------------------------*/
CREATE TABLE pets (
id serial primary key,
name varchar(256)
);
/*----------------------------------*/
/* Create Application User */
/*----------------------------------*/
CREATE USER ${APPLICATION_DB_USER} PASSWORD '${APPLICATION_DB_INITIAL_PASSWORD}';
/*----------------------------------*/
/* Grant Permissions */
/*----------------------------------*/
GRANT SELECT, INSERT ON public.pets TO ${APPLICATION_DB_USER};
GRANT USAGE, SELECT ON SEQUENCE public.pets_id_seq TO ${APPLICATION_DB_USER};
EOSQL
And then run those commands using a postgres:9.6
container as a PostgreSQL client:
docker run --rm -i -e PGPASSWORD=${SECURITY_ADMIN_PASSWORD} postgres:9.6 \
psql -U ${SECURITY_ADMIN_USER} \
"postgres://${REMOTE_DB_HOST}:${REMOTE_DB_PORT}/postgres" < sql_cmds.txt
NOTE: You may see some duplication errors.
Click here to see expected output...
** Usually, the commands will run without errors:**
CREATE DATABASE
You are now connected to database "demo_app_db" as user "security_admin_user".
CREATE TABLE
CREATE ROLE
GRANT
GRANT
** However, occasionally, you may see some duplication errors that you can ignore:**
ERROR: duplicate key value violates unique constraint "pg_database_datname_index"
DETAIL: Key (datname)=(demo_app_db) already exists.
You are now connected to database "demo_app_db" as user "security_admin_user".
ERROR: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL: Key (typname, typnamespace)=(pets_id_seq, 2200) already exists.
ERROR: role "demo_app_user" already exists
GRANT
ERROR: tuple concurrently updated
Create Application Credentials Secret
The application will be scoped to the demo-app-ns namespace. To create the namespace run:
kubectl create namespace demo-app-ns
Click here to see expected output...
namespace "demo-app-ns" created
Next we’ll store the application-credentials in Kubernetes Secrets:
kubectl --namespace demo-app-ns \
create secret generic demo-app-backend-credentials \
--from-literal=host="${REMOTE_DB_HOST}" \
--from-literal=port="${REMOTE_DB_PORT}" \
--from-literal=username="${APPLICATION_DB_USER}" \
--from-literal=password="${APPLICATION_DB_INITIAL_PASSWORD}"
Click here to see expected output...
secret "demo-app-backend-credentials" created
While Kubernetes Secrets are more secure than hard-coded ones, in a real deployment you should secure secrets in a fully-featured vault, like Conjur.
Create Application Service Account and Grant Entitlements
To grant our application access to the credentials in Kubernetes Secrets, we’ll need to:
- Create a ServiceAccount for our application
- Create a Role with permissions to "get" the backend-credentials secret
- Create a RoleBinding so our ServiceAccount has this Role
The YAML manifest to create these resources can be viewed here: application-entitlements.yml
To create the Kubernetes resources, run this command:
kubectl --namespace demo-app-ns apply -f demo-app-entitlements.yml
Click here to see expected output...
serviceaccount/demo-app-svc-account created
role.rbac.authorization.k8s.io/backend-credentials-reader created
rolebinding.rbac.authorization.k8s.io/read-backend-credentials created
Up next...
Continue as a Security Admin and configure Secretless Broker.
Step 4: Configure Secretless Broker
Create Secretless Broker Configuration ConfigMap
With our database ready and our credentials safely stored, we can now configure the Secretless Broker. We’ll tell it where to listen for connections and how to proxy them.
After that, the developer’s application can access the database without ever knowing the application-credentials.
A Secretless Broker configuration file defines the services that Secretless with authenticate to on behalf of your application. The Secretless Broker configuration file can be viewed here: secretless.yml
Here’s what this configuration does:
- Defines a service called
pets-pg
that listens for PostgreSQL connections onlocalhost:5432
- Says that the database
host
,port
,username
andpassword
are stored in Kubernetes Secrets - Lists the ids of those credentials within Kubernetes Secrets
Note: This configuration is shared by all Secretless Broker sidecar containers. There is one Secretless sidecar in every application Pod replica.
Note: Since we don't specify an sslmode
in the Secretless Broker config, it will use the default require
value.
Next we create a Kubernetes ConfigMap from this secretless.yml:
kubectl --namespace demo-app-ns \
create configmap \
demo-app-secretless-config \
--from-file=secretless.yml
Click here to see expected output...
configmap/demo-app-secretless-config created
To view the contents of the ConfigMap that was created:
kubectl --namespace demo-app-ns describe cm demo-app-secretless-config
Click here to see expected output...
Name: demo-app-secretless-config
Namespace: demo-app-ns
Labels: <none>
Annotations: <none>
Data
====
secretless.yml:
----
version: "2"
services:
pets-pg:
connector: pg
listenOn: tcp://localhost:5432
credentials:
host:
from: kubernetes
get: demo-app-backend-credentials#host
port:
from: kubernetes
get: demo-app-backend-credentials#port
username:
from: kubernetes
get: demo-app-backend-credentials#username
password:
from: kubernetes
get: demo-app-backend-credentials#password
Events: <none>
Up next...
As an Application Developer, you no longer need to worry about all the passwords and database connections! You will deploy an application and leave it up to the Secretless Broker to make the desired connection to the database.
Step 5: Deploy Application
You are now the Application Developer
You can no longer access the secrets we stored previously in environment variables. To clear these environment variables, you can run the following:
./clear_admin_env.sh
You know only one thing – the name of the database, namely demo_app_db.
Sample Application Overview
The application we’ll be deploying is a pet store demo application with a simple API:
GET /pets
lists all the petsPOST /pet
adds a pet
Its PostgreSQL backend is configured using a DB_URL
environment variable:
postgresql://localhost:5432/demo_app_db?sslmode=disable
Again, the application has no knowledge of the database credentials it’s using.
Create Application Deployment and Service
We’re ready to deploy our demo application and service using the YAML manifest that can be viewed here: demo-app-deployment.yml.
A detailed explanation of the manifest below is featured in the next step, Step 7 - Secretless Deployment Manifest Explained
and isn’t needed to complete the tutorial.
To deploy the application, run:
kubectl --namespace demo-app-ns apply -f demo-app-deployment.yml
Click here to see expected output...
deployment.apps/demo-application created
service/demo-application created
Before moving on, verify that the pods are healthy. It may take a minute or so for the pods to get to the Running
state:
kubectl --namespace demo-app-ns get pods
Click here to see expected output...
NAME READY STATUS RESTARTS AGE
demo-application-5df4dc5b87-k6r4m 2/2 Running 1 26s
demo-application-5df4dc5b87-w7zj6 2/2 Running 1 26s
demo-application-5df4dc5b87-zffpd 2/2 Running 1 26s
The demo application service is exposed on port 30002 on all Kubernetes nodes. To see this:
kubectl --namespace demo-app-ns get svc
Click here to see expected output...
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
demo-application NodePort 10.96.100.50 <none> 8080:30002/TCP 81s
Congratulations! The application is now available on port 30002 on any Kubernetes node.
Test the Application
To access the application, we'll use:
- The IP address of the worker node (IP address of either node would do)
- The NodePort on which this service is exposed (30001)
The IP address of the worker node can be displayed with this command:
kubectl get nodes -o wide | grep -v "master"
Let's create an APPLICATION_URL environment variable based on the worker node IP address and port 30001:
export APPLICATION_URL=[[HOST2_IP]]:30002
Now let’s create a pet (POST /pet):
curl -i -d @- \
-H "Content-Type: application/json" \
${APPLICATION_URL}/pet \
<< EOF
{
"name": "Secretlessly Fluffy"
}
EOF
We should get a 201 response status.
Click here to see expected output...
HTTP/1.1 201
Location: http://[[HOST2_IP]].100:30002/pet/1
Content-Length: 0
Date: Mon, 18 Feb 2020 11:56:27 GMT
Now let’s retrieve all the pets (GET /pets):
curl -i ${APPLICATION_URL}/pets
We should get a 200 response with a JSON array of the pets.
Click here to see expected output...
HTTP/1.1 200
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Date: Mon, 18 Feb 2020 11:58:36 GMT
[{"id":1,"name":"Secretlessly Fluffy"}]
That’s it!
The application is connecting to a password-protected Postgres database without any knowledge of the credentials.
For more info on configuring Secretless for your own use case, see the Secretless Documentation
Up next...
Get a closer look at Secretless
Appendix: Secretless Deployment Manifest Explained
Here we’ll walk through the application deployment manifest, to better understand how Secretless works.
We’ll focus on the Pod’s template, which is where the magic happens:
# top part elided...
template:
metadata:
labels:
app: demo-application
spec:
serviceAccountName: demo-app-svc-account
automountServiceAccountToken: true
containers:
- name: demo-application
image: cyberark/demo-app:latest
env:
- name: DB_URL
value: postgresql://localhost:5432/demo_app_db?sslmode=disable
- name: secretless-broker
image: cyberark/secretless-broker:latest
imagePullPolicy: Always
args: ["-f", "/etc/secretless/secretless.yml"]
volumeMounts:
- name: config
mountPath: /etc/secretless
readOnly: true
volumes:
- name: config
configMap:
name: demo-app-secretless-config
Networking
Since it resides in the same pod, the application can access the Secretless sidecar container over localhost.
As specified in the ConfigMap we created, Secretless listens on port 5432, and hence this:
env:
- name: DB_URL
value: postgresql://localhost:5432/demo_app_db?sslmode=disable
is all our application needs to locate Secretless.
SSL
Notice the ?sslmode=disable
at the end of our DB_URL?
This means that the application connects to Secretless without SSL, which is safe because it is intra-Pod communication over localhost.
However, the connection between Secretless and Postgres is secure, and does use SSL.
The situation looks like this:
No SSL SSL
Application <----------> Secretless <----------> Postgres For more information on PostgreSQL SSL modes see:
Credential Access
Notice we add the quick-start-application ServiceAccount to the pod:
spec:
serviceAccountName: demo-app-svc-account
That’s the ServiceAccount we created earlier, the one with access to the credentials in Kubernetes Secrets. This is what gives Secretless access to those credentials.
Configuration Access
Finally, notice the sections defining the volumes and the volume mount in the Secretless container:
# ... elided
volumeMounts:
- name: config
mountPath: /etc/secretless
readOnly: true
volumes:
- name: config
configMap:
name: demo-app-secretless-config
Here we create a volume base on the ConfigMap we created earlier, which stores our secretless.yml configuration file.
Thus Secretless gets its configuration file via a volume mount.
Up next...
A summary of what you accomplished in this tutorial!
Conclusion
You're done!
In this tutorial you learned how to:
- Deploy a PostgreSQL database
- Store its credentials in Kubernetes secrets
- Setup Secretless Broker to proxy connections to it
- Deploy an application that connects to the database without knowing its password
Next Steps
Want to learn more?
- Take a look at our documentation.
- Try some more CyberArk demos/tutorials
- Join our open-source CyberArk Community and contribute!
- Start a conversation or ask questions on our Discourse Page
- Check out our source code and detailed documentation: