DOKS (DigitalOcean Managed Kubernetes) Secrets Injection with Fortanix Self-Defending KMS
Kubernetes provides a relatively simple way for vendors to create and publish an Admission Controller to allow interception of requests to kubernetes API server before an object is persisted (e.g. when creating a pod, before the pod is persisted the request is intercepted by the Admission Controller). These admission controllers are called via webhooks:
There are two types of Admission Controllers (and fairly simple use of terms), but their primary purposes are:
MutatingAdmissionWebhook: Modify or Mutate the objects they admit
ValidatingAdmissionWebhook: Validate the objects they admit
Fortanix Self-Defending KMS provides native integration with kubernetes from a secrets injection perspective. This article concentrates on mutating admission webhooks as the admission controller provided by Fortanix "mutates" the objects (or pods) to inject the sidecar container to securely expose secrets stored within the Self-Defending KMS.
This article also will use DigitalOcean Managed Kubernetes deployment for demonstration.
Let's jump right in!
Let's start with the basics.
There are many walk-through videos on how to deploy a managed kubernetes cluster with DigitalOcean. If you're already set, you can skip this section and go straight to the Fortanix Setup.
Start by logging into your DigitalOcean account and create a new kubernetes cluster:
Select the region you wish to deploy your kubernetes cluster and ensure you set your cluster to a meaningful name (I'm not so creative, so decided to use something generic):
Once you click on "Create Cluster", you will now have a chance to download the kubernetes config file, which you can store anywhere. I chose to store this under ".kube" directory under my home folder (that way I don't have to export the "KUBECONFIG" environment variable all the time).
When your cluster is ready, DigitalOcean will now show everything is running (deployed only a 2-node cluster):
You can also verify all is working correctly via the "kubectl" tool:
$ kubectl get nodes NAME STATUS ROLES AGE VERSION pool-4kk2csg7g-3p9z2 Ready <none> 412s v1.18.8 pool-4kk2csg7g-3p9zp Ready <none> 398s v1.18.8
If you want, you can also check what types of pods are already deployed as part of the base kubernetes image as well:
$ kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system cilium-dtqz4 1/1 Running 0 6m kube-system cilium-fzh8k 1/1 Running 0 5m45s kube-system cilium-operator-5bcc866568-4rqt5 1/1 Running 0 8m39s kube-system coredns-bcdcfd4dc-kfz6r 1/1 Running 0 8m39s kube-system coredns-bcdcfd4dc-ptl4l 1/1 Running 0 8m39s kube-system csi-do-node-d2zm9 2/2 Running 0 6m kube-system csi-do-node-ft52l 2/2 Running 0 5m46s kube-system do-node-agent-ds2kq 1/1 Running 0 6m kube-system do-node-agent-n44jc 1/1 Running 0 5m46s kube-system kube-proxy-ltfj9 1/1 Running 0 5m46s kube-system kube-proxy-sqh4f 1/1 Running 0 6m
In order to use mutating admission webhooks, one must first check if the Kubernetes deployment has the admission controllers registration enabled. Most managed kubernetes deployment already have the plugins enabled, but it's always good to check if it really is:
$ export KUBECONFIG=~/.kube/config $ kubectl api-versions | grep -i admissionregistration admissionregistration.k8s.io/v1 admissionregistration.k8s.io/v1beta
We are now ready to start enabling the admission controller.
Before you deploy any Fortanix components, please ensure you have access to Self-Defending KMS. If you have not already, sign up through https://sdkms.fortanix.com and setup an App within Self-Defending KMS:
Once the App is setup in Self-Defending KMS, Fortanix allows you to download the installation package that includes all of the necessary files required to automatically create the service accounts, namespaces, webhooks and the containers required for the admission controller.
Once the package is downloaded, ensure the service-account.yaml file has the correct API key or the JSON Web Token details to ensure the webhook will use the correct credentials when the sidecar container communicates with Self-Defending KMS:
$ cat service-account.yaml # Service account used for the mutated pods apiVersion: v1 kind: ServiceAccount metadata: name: demo-sa namespace: <your namespace> automountServiceAccountToken: false --- # This secret holds credentials used by the secret-agent to authenticate to SDKMS # Note that the secret name is the service account name with a suffix: -sdkms-credentials # This is how the controller associates the secret to the service account. apiVersion: v1 kind: Secret metadata: name: demo-sa-sdkms-credentials namespace: <your namespace> type: Opaque stringData: api_endpoint: https://sdkms.fortanix.com api_key: <api-key> $ cat sidecar-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: fortanix-webhook-config namespace: fortanix data: controller-config.yaml: | authTokenType: api-key # acceptable values: `api-key`, `jwt`. secretAgentImage: ftxdemo/secret-agent:0.2 proxySettings: #httpProxy: 'http://example.com' #httpsProxy: 'https://example.com' #noProxy: '*.example.com,184.108.40.206' tokenVolumeProjection: #addToAllPods: true #audience: https://sdkms.fortanix.com #expirationSeconds: 3600
And you're set. Helm is available, but for this tutorial, deploy all the neccessary components automatically via the script provided:
A sample output would be something like this:
You can now check the admission controller is running properly within the new "fortanix" and "fortanix-demo" namespaces created:
$ kubectl get ns NAME STATUS AGE default active 613m fortanix active 30s fortanix-demo active 23s kube-node-lease active 613m kube-public active 613m kube-system active 613m $ kubectl get pods -A | grep fortanix fortanix fortanix-secrets-injector-c46958b94-xvncr 1/1 Running 0 27s
Couple of points:
fortanix namespace contains the admission controller container (automatically pulled from the public repository from Fortanix) in the pod named fortanix-secrets-injector
fortanix-demo namespace is created with the label of "fortanix-secrets-injector: enabled". This is ensure any pods running in that namespace will be automatically checked for mutation
Now you're ready rock!
Let's create a sample secret first within Self-Defending KMS (my secret is as simple as "Hi! I'm a Test!"):
Now that we have a secret created, let's get a quick pod definition setup on a sample file (provided) "single-pod.yaml" and modify as you need. I'll be using a snooze (sleep) demo:
$ cat single-pod.yaml # This example shows how to use templates to render secret values apiVersion: v1 kind: Pod metadata: namespace: fortanix-demo name: test-demo-pod annotations: secrets-injector.fortanix.com/inject-through-environment: "false" secrets-injector.fortanix.com/secrets-volume-path: /opt/myapp/credentials secrets-injector.fortanix.com/inject-secret-test: "test" spec: serviceAccountName: demo-sa terminationGracePeriodSeconds: 0 containers: - name: busybox imagePullPolicy: IfNotPresent image: alpine:3.8 command: - sh - "-c" - | sh << 'EOF' sleep 100m EOF
Once the pod has been mutated, all secrets outlined will automatically be mounted to "/opt/myapp/credentials" for all containers running in the test-demo-pod pod. To define what secrets are to be injected into the pod, use the annotation: inject-secret-<path>: "<self-defending kms secret path>"
The path defines what would be the link under /opt/myapp/credentials
The self-defending kms secret path is the name of the security object created inside Self-Defending KMS
To start the test-demo-pod pod:
$ kubectl create -f single-pod.yaml pod/test-demo-pod created
Quick check to make sure the pod is running:
$ kubectl get pods -A | grep test-demo-pod fortanix-demo test-demo-pod 1/1 Running 0 40s
In the admission controller:
$ kubectl -n fortanix logs fortanix-secrets-injector-c46958b94-xvncr | grep -i mutating 2020/10/22 18:00:30 Mutating 'fortanix-demo/test-demo-pod'
And check the actual test-demo-pod pod was actually mutated:
$ kubectl -n fortanix-demo describe pod test-demo-pod | grep -i injected Annotations: secrets-injector.fortanix.com/injected: true $ kubectl -n fortanix-demo exec test-demo-pod -- cat /opt/myapp/credentials/test Hi I'm a Test!%
And we know from Self-Defending KMS that the test secret were accessed by the newly deployed pod:
And that's really it! Simple?
There are many other configuration variables that Fortanix supports when it comes to secrets injection - templates, and others. I would highly suggest one visits the support website for more details.
Let me know your thoughts!