The Security Admin sets up PostgreSQL, configures Secretless, and has sole access to the credentials.

Deploy a PostgreSQL Statefulset

PostgreSQL is stateful, so we’ll use a StatefulSet to manage it.

To deploy a PostgreSQL StatefulSet:

1. Create a dedicated namespace for the storage backend:

   kubectl create namespace postgres-backend-ns

   Click here to see expected output...

   namespace/postgres-backend-ns created

2. Create a self-signed certificate (see PostgreSQL documentation for more info):

   openssl req -new -x509 -days 365 \ -nodes -text -out server.crt \ -keyout server.key -subj "/CN=pg" chmod og-rwx server.key

   Click here to see expected output...

   master $ openssl req -new -x509 -days 365 \ -nodes -text -out server.crt \ -keyout server.key -subj "/CN=pg" Generating a 2048 bit RSA private key ...................................................................................................................+++ ....................+++ writing new private key to 'server.key' ----- master $ chmod og-rwx server.key master $

3. Store the certificate files as a Kubernetes secret in the postgres-backend-ns namespace:

   kubectl --namespace postgres-backend-ns \ create secret generic postgres-backend-certs \ --from-file=server.crt \ --from-file=server.key

   Click here to see expected output...

   secret/postgres-backend-certs created

   NOTE: While Kubernetes Secrets are more secure than hard-coded ones, in a real deployment you should secure secrets in a fully-featured vault, like Conjur.

4. Deploy a PostgreSQL StatefulSet and Service:

   kubectl --namespace postgres-backend-ns apply -f postgres.yml

   Click here to see expected output...

   statefulset.apps/pg created service/postgres-backend created

   View the StatefulSet manifest here: postgres.yml

   Some Details: Click here to see notes for the StatefulSet manifest...

   • The certificate files for your database server are mounted in a volume with defaultMode: 384 giving it permissions 0600 (Why? Because 600 in base 8 = 384 in base 10).
   • The pod is deployed with 999 as the group associated with any mounted volumes, as indicated by fsGroup: 999. 999 is a the static postgres gid, defined in the postgres Docker image

   This StatefulSet uses the DockerHub postgres:9.6 container. On startup, the container creates a superuser from the environment variables:

   • POSTGRES_USER
   • POSTGRES_PASSWORD

   which we set to the values security_admin_user and security_admin_password, respectively.

   Going forward, we’ll call these values the admin-credentials, to distinguish them from the application-credentials our application will use. In the scripts below, we’ll refer to the admin-credentials by the environment variables:

   • SECURITY_ADMIN_USER
   • SECURITY_ADMIN_PASSWORD.

5. Confirm that the PostgreSQL StatefulSet Pod has Started and is Healthy: (It may take a minute or so for the pod to get to 'Running' state.)

   kubectl --namespace postgres-backend-ns get pods

   Click here to see expected output (it may take a minute or so to get to 'Running' state)...

   NAME READY STATUS RESTARTS AGE pg-0 1/1 Running 0 6s

6. Confirm the Successful Creation of the PostgreSQL Service: kubectl --namespace postgres-backend-ns get svc postgres-backend

   Click here to see expected output (Cluster IP address may be different)...

   NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE postgres-backend NodePort 10.106.232.98 <none> 5432:30001/TCP 2m

   NOTE: This service is exposed via NodePort 30001 on every node in the cluster. This will be used in the next step.

7. Confirm that You Can Log into the Database and List Users:

   The database has no data yet, but we can verify it works by logging in as the security admin and listing the users. To access the postgres-backend service, we'll use:

   • The IP address of the worker node (either node would do)
   • The NodePort on which this service is exposed (30001)

   The IP address of the worker node can be displayed with this command:

   kubectl get nodes -o wide | grep -v "master"

   Setting up environment variables for accessing the database:

   export SECURITY_ADMIN_USER=security_admin_user export SECURITY_ADMIN_PASSWORD=security_admin_password export REMOTE_DB_HOST=[[HOST2_IP]] export REMOTE_DB_PORT=30001

   And then list the users using a postgres:9.6 image as a client:

   docker run --rm -it \ -e PGPASSWORD=${SECURITY_ADMIN_PASSWORD} \ postgres:9.6 psql \ -U ${SECURITY_ADMIN_USER} \ "postgres://${REMOTE_DB_HOST}:${REMOTE_DB_PORT}/postgres" \ -c "\du"

   Click here to see expected output...

   List of roles Role name | Attributes | Member of ---------------------+------------------------------------------------------------+----------- security_admin_user | Superuser, Create role, Create DB, Replication, Bypass RLS | {}

Up next...

Continue as an Security Admin and set up Application namespace, credentials, and service account

Create Application Database

In this section, we'll create the application database and user, and securely store the user's credentials:

• Create the application database
• Create the pets table in that database
• Create an application user with limited privileges: SELECT and INSERT on the pets table
• Store these database application-credentials in Kubernetes secrets.

So we can refer to them later, export the database name and application-credentials as environment variables:

export APPLICATION_DB_NAME=demo_app_db export APPLICATION_DB_USER=demo_app_user export APPLICATION_DB_INITIAL_PASSWORD=demo_app_user_password

To perform the steps listed above, first create a file containing SQL commands:

cat > sql_cmds.txt << EOSQL /*----------------------------------*/ /* Create Application Database */ /*----------------------------------*/ CREATE DATABASE ${APPLICATION_DB_NAME}; /*----------------------------------*/ /* Connect to Database */ /*----------------------------------*/ \c ${APPLICATION_DB_NAME}; /*----------------------------------*/ /* Create a pets Table */ /*----------------------------------*/ CREATE TABLE pets ( id serial primary key, name varchar(256) ); /*----------------------------------*/ /* Create Application User */ /*----------------------------------*/ CREATE USER ${APPLICATION_DB_USER} PASSWORD '${APPLICATION_DB_INITIAL_PASSWORD}'; /*----------------------------------*/ /* Grant Permissions */ /*----------------------------------*/ GRANT SELECT, INSERT ON public.pets TO ${APPLICATION_DB_USER}; GRANT USAGE, SELECT ON SEQUENCE public.pets_id_seq TO ${APPLICATION_DB_USER}; EOSQL

And then run those commands using a postgres:9.6 container as a PostgreSQL client:

docker run --rm -i -e PGPASSWORD=${SECURITY_ADMIN_PASSWORD} postgres:9.6 \ psql -U ${SECURITY_ADMIN_USER} \ "postgres://${REMOTE_DB_HOST}:${REMOTE_DB_PORT}/postgres" < sql_cmds.txt

NOTE: You may see some duplication errors.

Create Application Credentials Secret

The application will be scoped to the demo-app-ns namespace. To create the namespace run:

kubectl create namespace demo-app-ns

Next we’ll store the application-credentials in Kubernetes Secrets:

kubectl --namespace demo-app-ns \ create secret generic demo-app-backend-credentials \ --from-literal=host="${REMOTE_DB_HOST}" \ --from-literal=port="${REMOTE_DB_PORT}" \ --from-literal=username="${APPLICATION_DB_USER}" \ --from-literal=password="${APPLICATION_DB_INITIAL_PASSWORD}"

While Kubernetes Secrets are more secure than hard-coded ones, in a real deployment you should secure secrets in a fully-featured vault, like Conjur.

Create Application Service Account and Grant Entitlements

To grant our application access to the credentials in Kubernetes Secrets, we’ll need to:

• Create a ServiceAccount for our application
• Create a Role with permissions to "get" the backend-credentials secret
• Create a RoleBinding so our ServiceAccount has this Role

The YAML manifest to create these resources can be viewed here: application-entitlements.yml

To create the Kubernetes resources, run this command: kubectl --namespace demo-app-ns apply -f demo-app-entitlements.yml

Up next...

Continue as a Security Admin and configure Secretless Broker.

Create Secretless Broker Configuration ConfigMap

With our database ready and our credentials safely stored, we can now configure the Secretless Broker. We’ll tell it where to listen for connections and how to proxy them.

After that, the developer’s application can access the database without ever knowing the application-credentials.

A Secretless Broker configuration file defines the services that Secretless with authenticate to on behalf of your application. The Secretless Broker configuration file can be viewed here: secretless.yml

Here’s what this configuration does:

• Defines a service called pets-pg that listens for PostgreSQL connections on localhost:5432
• Says that the database host, port, username and password are stored in Kubernetes Secrets
• Lists the ids of those credentials within Kubernetes Secrets

Note: This configuration is shared by all Secretless Broker sidecar containers. There is one Secretless sidecar in every application Pod replica. Note: Since we don't specify an sslmode in the Secretless Broker config, it will use the default require value.

Next we create a Kubernetes ConfigMap from this secretless.yml:

kubectl --namespace demo-app-ns \ create configmap \ demo-app-secretless-config \ --from-file=secretless.yml

To view the contents of the ConfigMap that was created:

kubectl --namespace demo-app-ns describe cm demo-app-secretless-config

Up next...

As an Application Developer, you no longer need to worry about all the passwords and database connections! You will deploy an application and leave it up to the Secretless Broker to make the desired connection to the database.

You are now the Application Developer

You can no longer access the secrets we stored previously in environment variables. To clear these environment variables, you can run the following:

./clear_admin_env.sh

You know only one thing – the name of the database, namely demo_app_db.

Sample Application Overview

The application we’ll be deploying is a pet store demo application with a simple API:

• GET /pets lists all the pets
• POST /pet adds a pet

Its PostgreSQL backend is configured using a DB_URL environment variable:

postgresql://localhost:5432/demo_app_db?sslmode=disable

Again, the application has no knowledge of the database credentials it’s using.

Create Application Deployment and Service

We’re ready to deploy our demo application and service using the YAML manifest that can be viewed here: demo-app-deployment.yml.

A detailed explanation of the manifest below is featured in the next step, Step 7 - Secretless Deployment Manifest Explained and isn’t needed to complete the tutorial.

To deploy the application, run:

kubectl --namespace demo-app-ns apply -f demo-app-deployment.yml

Before moving on, verify that the pods are healthy. It may take a minute or so for the pods to get to the Running state:

kubectl --namespace demo-app-ns get pods

The demo application service is exposed on port 30002 on all Kubernetes nodes. To see this:

kubectl --namespace demo-app-ns get svc

Congratulations! The application is now available on port 30002 on any Kubernetes node.

Test the Application

To access the application, we'll use:

• The IP address of the worker node (IP address of either node would do)
• The NodePort on which this service is exposed (30001)

The IP address of the worker node can be displayed with this command:

kubectl get nodes -o wide | grep -v "master"

Let's create an APPLICATION_URL environment variable based on the worker node IP address and port 30001:

export APPLICATION_URL=[[HOST2_IP]]:30002

Now let’s create a pet (POST /pet):

curl -i -d @- \ -H "Content-Type: application/json" \ ${APPLICATION_URL}/pet \ << EOF { "name": "Secretlessly Fluffy" } EOF

We should get a 201 response status.

Now let’s retrieve all the pets (GET /pets):

curl -i ${APPLICATION_URL}/pets

We should get a 200 response with a JSON array of the pets.

That’s it!

The application is connecting to a password-protected Postgres database without any knowledge of the credentials.

For more info on configuring Secretless for your own use case, see the Secretless Documentation

Up next...

Get a closer look at Secretless

Here we’ll walk through the application deployment manifest, to better understand how Secretless works.

We’ll focus on the Pod’s template, which is where the magic happens:

  # top part elided...
  template:
    metadata:
      labels:
        app: demo-application
    spec:
      serviceAccountName: demo-app-svc-account
      automountServiceAccountToken: true
      containers:
        - name: demo-application
          image: cyberark/demo-app:latest
          env:
            - name: DB_URL
              value: postgresql://localhost:5432/demo_app_db?sslmode=disable
        - name: secretless-broker
          image: cyberark/secretless-broker:latest
          imagePullPolicy: Always
          args: ["-f", "/etc/secretless/secretless.yml"]
          volumeMounts:
            - name: config
              mountPath: /etc/secretless
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: demo-app-secretless-config

Networking

Since it resides in the same pod, the application can access the Secretless sidecar container over localhost.

As specified in the ConfigMap we created, Secretless listens on port 5432, and hence this:

          env:
            - name: DB_URL
              value: postgresql://localhost:5432/demo_app_db?sslmode=disable

is all our application needs to locate Secretless.

SSL

Notice the ?sslmode=disable at the end of our DB_URL?

This means that the application connects to Secretless without SSL, which is safe because it is intra-Pod communication over localhost.

However, the connection between Secretless and Postgres is secure, and does use SSL.

The situation looks like this:

             No SSL                       SSL

Application <----------> Secretless <----------> Postgres For more information on PostgreSQL SSL modes see:

• PostgreSQL SSL documentation
• PostgreSQL Secretless Service Connector documentation

Credential Access

Notice we add the quick-start-application ServiceAccount to the pod:

    spec:
      serviceAccountName: demo-app-svc-account

That’s the ServiceAccount we created earlier, the one with access to the credentials in Kubernetes Secrets. This is what gives Secretless access to those credentials.

Configuration Access

Finally, notice the sections defining the volumes and the volume mount in the Secretless container:

          # ... elided
          volumeMounts:
            - name: config
              mountPath: /etc/secretless
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: demo-app-secretless-config

Here we create a volume base on the ConfigMap we created earlier, which stores our secretless.yml configuration file.

Thus Secretless gets its configuration file via a volume mount.

Up next...

A summary of what you accomplished in this tutorial!

You're done!

In this tutorial you learned how to:

1. Deploy a PostgreSQL database
2. Store its credentials in Kubernetes secrets
3. Setup Secretless Broker to proxy connections to it
4. Deploy an application that connects to the database without knowing its password

Next Steps

Want to learn more?

• Take a look at our documentation.
• Try some more CyberArk demos/tutorials
• Join our open-source CyberArk Community and contribute!
• Start a conversation or ask questions on our Discourse Page
• Check out our source code and detailed documentation:
  • Secretless Broker source code
  • Conjur source code
  • Summon
  • Secretless Broker documentation repository
  • Conjur documentation repository