Container runtime security in Kubernetes with Falco
Kubernetes adoption is on a steep rise, with organizations looking to empower their development teams to meet aggressive release deadlines by leveraging on the might of container orchestration.
Security is more often than not an afterthought, specially when the core focus of our team is delivering functional software in a short time frame.
Luckily enough, they are enough tools at our disposal to help us make sure that the end result of our delivery pipeline is an application workload running on top of a container image that has been scanned for vulnerabilities.
The diagram below, shows a typical pipeline that I usually rely upon for my CI/CD needs.
Once our application is running in our Kubernetes environment, the focus shifts from Continuous Delivery to Continuous Operations. This is where we need to make sure that we have the means to monitor the run time behavior of our workload.
That brings us to Falco, which is the core topic of our article — and it is highlighted in stage 11 of our pipeline as “real time behavior monitoring”.
Falco is currently an incubating project under the CNCF security landscape, it contains a rich set of rules that can be edited for flagging specific abnormal behavior in our Kubernetes workloads.
Let’s have a quick look at how Falco can be deployed in our Kubernetes environment by leveraging on Helm.
If Helm is not yet configured in your environment, that can be easily solved just by running the commands below:
$ curl -fsSL -o get_helm.sh \ https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh
That of course assumes, that we already have kubectl configured pointing to the Kubernetes cluster in which Falco will be deployed.
Having Helm in place, will make possible to deploy Falco with a single command:
$ helm install falco -n falco stable/falco — version 1.1.0
Give it a couple of minutes, and we should have the Falco workload up and running as reported below by kubectl.
$ kubectl get pods -n falco
Let’s explore now the real time reporting capabilities of Falco, the fastest way to do this would be to see the logs straight from the Falco container.
We will have two terminals open, in the first one we will run the following command to access the Falco logs:
$ kubectl logs -f falco-74wgx -n falco
That will allow us to see real time, any unexpected behavior in our workloads in the Kubernetes cluster.
From the second console, I will run a shell inside one of my application workloads. I am using a PHP pod for this example, but you could use any pod that you have with a shell available.
$ kubectl get pods -n php
$ kubectl exec -it php-safe-67b6fb8648–4k98v -n php /bin/bash
Withing a few seconds, we will see from the first terminal that Falco detected the spawned shell in our php container!
Now, let’s go back to our second terminal (the one with the shell inside the container) and let’s copy the shadow file containing the encrypted account password into another file.
$ cat /etc/shadow > /dev/null
That activity was logged by falco as well as we can see in our first terminal:
If we happen to have an EFK stack running in our Kubernetes cluster, those logs would be picked up and we could filter them in our Kibana Dashboard.
For those of you not keen on running an EFK stack in your Kubernetes cluster due to resource constrains, another option would be to forward this logs to logdna and filter accordingly.
I have found the default Falco policies to be quite effective for most of the environments that we run, it has been useful not only for detecting spawned shells and access of sensitive file — but also cryptomining activity in a Kubernetes environment.
I am planning to release additional reviews and guides for other common tools that can be used in Kubernetes environments and CI/CD pipelines.
Stay tuned and thank you for reading!
— — — — —
Alex
