DOKS (DigitalOcean Managed Kubernetes) Secrets Injection with Fortanix Self-Defending KMS

Kubernetes provides a relatively simple way for vendors to create and publish an Admission Controller to allow interception of requests to kubernetes API server before an object is persisted (e.g. when creating a pod, before the pod is persisted the request is intercepted by the Admission Controller). These admission controllers are called via webhooks:

https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

There are two types of Admission Controllers (and fairly simple use of terms), but their primary purposes are:

• MutatingAdmissionWebhook: Modify or Mutate the objects they admit

• ValidatingAdmissionWebhook: Validate the objects they admit

Fortanix Self-Defending KMS provides native integration with kubernetes from a secrets injection perspective. This article concentrates on mutating admission webhooks as the admission controller provided by Fortanix "mutates" the objects (or pods) to inject the sidecar container to securely expose secrets stored within the Self-Defending KMS.

This article also will use DigitalOcean Managed Kubernetes deployment for demonstration.

Let's jump right in!

---

DigitalOcean Setup

Let's start with the basics.

There are many walk-through videos on how to deploy a managed kubernetes cluster with DigitalOcean. If you're already set, you can skip this section and go straight to the Fortanix Setup.

Start by logging into your DigitalOcean account and create a new kubernetes cluster:

Select the region you wish to deploy your kubernetes cluster and ensure you set your cluster to a meaningful name (I'm not so creative, so decided to use something generic):

Once you click on "Create Cluster", you will now have a chance to download the kubernetes config file, which you can store anywhere. I chose to store this under ".kube" directory under my home folder (that way I don't have to export the "KUBECONFIG" environment variable all the time).

When your cluster is ready, DigitalOcean will now show everything is running (deployed only a 2-node cluster):

You can also verify all is working correctly via the "kubectl" tool:

$ kubectl get nodes
NAME                    STATUS    ROLES    AGE    VERSION
pool-4kk2csg7g-3p9z2    Ready     <none>   412s   v1.18.8
pool-4kk2csg7g-3p9zp    Ready     <none>   398s   v1.18.8

If you want, you can also check what types of pods are already deployed as part of the base kubernetes image as well:

$ kubectl get pods -A
NAMESPACE    NAME                                READY    STATUS    RESTARTS    AGE
kube-system  cilium-dtqz4                        1/1      Running      0           6m
kube-system  cilium-fzh8k                        1/1      Running      0           5m45s
kube-system  cilium-operator-5bcc866568-4rqt5    1/1      Running      0           8m39s
kube-system  coredns-bcdcfd4dc-kfz6r             1/1      Running      0           8m39s
kube-system  coredns-bcdcfd4dc-ptl4l             1/1      Running      0           8m39s
kube-system  csi-do-node-d2zm9                   2/2      Running      0           6m
kube-system  csi-do-node-ft52l                   2/2      Running      0           5m46s
kube-system  do-node-agent-ds2kq                 1/1      Running      0           6m
kube-system  do-node-agent-n44jc                 1/1      Running      0           5m46s
kube-system  kube-proxy-ltfj9                    1/1      Running      0           5m46s
kube-system  kube-proxy-sqh4f                    1/1      Running      0           6m

In order to use mutating admission webhooks, one must first check if the Kubernetes deployment has the admission controllers registration enabled. Most managed kubernetes deployment already have the plugins enabled, but it's always good to check if it really is:

$ export KUBECONFIG=~/.kube/config
$ kubectl api-versions | grep -i admissionregistration
admissionregistration.k8s.io/v1
admissionregistration.k8s.io/v1beta

We are now ready to start enabling the admission controller.

---

Fortanix Setup

Before you deploy any Fortanix components, please ensure you have access to Self-Defending KMS. If you have not already, sign up through https://sdkms.fortanix.com and setup an App within Self-Defending KMS:

Once the App is setup in Self-Defending KMS, Fortanix allows you to download the installation package that includes all of the necessary files required to automatically create the service accounts, namespaces, webhooks and the containers required for the admission controller.

Once the package is downloaded, ensure the service-account.yaml file has the correct API key or the JSON Web Token details to ensure the webhook will use the correct credentials when the sidecar container communicates with Self-Defending KMS:

$ cat service-account.yaml
# Service account used for the mutated pods
apiVersion: v1
kind: ServiceAccount
metadata:
  name: demo-sa
  namespace: <your namespace>
automountServiceAccountToken: false
---
# This secret holds credentials used by the secret-agent to authenticate to SDKMS
# Note that the secret name is the service account name with a suffix: -sdkms-credentials
# This is how the controller associates the secret to the service account.
apiVersion: v1
kind: Secret
metadata:
  name: demo-sa-sdkms-credentials
  namespace: <your namespace>
type: Opaque
stringData:
  api_endpoint: https://sdkms.fortanix.com
  api_key: <api-key>

$ cat sidecar-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fortanix-webhook-config
  namespace: fortanix
data:
  controller-config.yaml: |
    authTokenType: api-key # acceptable values: `api-key`, `jwt`.
    secretAgentImage: ftxdemo/secret-agent:0.2
    proxySettings:
      #httpProxy: 'http://example.com'
      #httpsProxy: 'https://example.com'
      #noProxy: '*.example.com,1.2.3.4'
    tokenVolumeProjection:
      #addToAllPods: true
      #audience: https://sdkms.fortanix.com
      #expirationSeconds: 3600

And you're set. Helm is available, but for this tutorial, deploy all the neccessary components automatically via the script provided:

$ ./deploy.sh

A sample output would be something like this:

You can now check the admission controller is running properly within the new "fortanix" and "fortanix-demo" namespaces created:

$ kubectl get ns
NAME                STATUS    AGE
default             active    613m
fortanix            active    30s
fortanix-demo       active    23s
kube-node-lease     active    613m
kube-public         active    613m
kube-system         active    613m
$ kubectl get pods -A | grep fortanix
fortanix    fortanix-secrets-injector-c46958b94-xvncr    1/1    Running 0    27s

Couple of points:

• fortanix namespace contains the admission controller container (automatically pulled from the public repository from Fortanix) in the pod named fortanix-secrets-injector

• fortanix-demo namespace is created with the label of "fortanix-secrets-injector: enabled". This is ensure any pods running in that namespace will be automatically checked for mutation

Now you're ready rock!

Let's create a sample secret first within Self-Defending KMS (my secret is as simple as "Hi! I'm a Test!"):

Now that we have a secret created, let's get a quick pod definition setup on a sample file (provided) "single-pod.yaml" and modify as you need. I'll be using a snooze (sleep) demo:

$ cat single-pod.yaml
# This example shows how to use templates to render secret values
apiVersion: v1
kind: Pod
metadata:
  namespace: fortanix-demo
  name: test-demo-pod
  annotations:
    secrets-injector.fortanix.com/inject-through-environment: "false"
    secrets-injector.fortanix.com/secrets-volume-path: /opt/myapp/credentials
    secrets-injector.fortanix.com/inject-secret-test: "test"
spec:
  serviceAccountName: demo-sa
  terminationGracePeriodSeconds: 0
  containers:
  - name: busybox
    imagePullPolicy: IfNotPresent
    image: alpine:3.8
    command:
    - sh
    - "-c"
    - |
      sh << 'EOF'
        sleep 100m
      EOF

Once the pod has been mutated, all secrets outlined will automatically be mounted to "/opt/myapp/credentials" for all containers running in the test-demo-pod pod. To define what secrets are to be injected into the pod, use the annotation: inject-secret-<path>: "<self-defending kms secret path>"

• The path defines what would be the link under /opt/myapp/credentials

• The self-defending kms secret path is the name of the security object created inside Self-Defending KMS

To start the test-demo-pod pod:

$ kubectl create -f single-pod.yaml
pod/test-demo-pod created

Quick check to make sure the pod is running:

$ kubectl get pods -A | grep test-demo-pod
fortanix-demo	test-demo-pod			1/1	Running	0	40s

In the admission controller:

$ kubectl -n fortanix logs fortanix-secrets-injector-c46958b94-xvncr | grep -i mutating
2020/10/22 18:00:30 Mutating 'fortanix-demo/test-demo-pod'

And check the actual test-demo-pod pod was actually mutated:

$ kubectl -n fortanix-demo describe pod test-demo-pod | grep -i injected
Annotations:	secrets-injector.fortanix.com/injected: true

$ kubectl -n fortanix-demo exec test-demo-pod -- cat /opt/myapp/credentials/test
Hi I'm a Test!%

And we know from Self-Defending KMS that the test secret were accessed by the newly deployed pod:

And that's really it! Simple?

There are many other configuration variables that Fortanix supports when it comes to secrets injection - templates, and others. I would highly suggest one visits the support website for more details.

Let me know your thoughts!