As Kubernetes becomes the platform of choice for both app development and deployment, securing Kubernetes is growing from a passing concern to a first class requirement. This tutorial covers auditing Kubernetes clusters in real time for activity and building a system to automatically log and process audit events.
Kubernetes is API driven. In many deployments each user, administrator, and developer communicates with the API. When a user issues a kubectl create deployment
command, it hits the API and the request is logged through the Kubernetes audit system. There are also nonhuman API interactions to log and audit, operators work directly against the API with their own set of credentials, and external systems from cloud providers hit the API. All of these interactions should be logged, audited, and processed with alerts sent out if out-of-compliance behavior is detected.
Falco is a Cloud Native Computing Foundation (CNCF) security tool that has first-tier support for Kubernetes audit payloads and a rich set of tools for processing those events.
Prerequisites
Estimated time
You can complete this tutorial in less than 30 minutes.
Steps
- Step 1. Configure Kubernetes auditing
- Step 2. Set up Kubernetes auditing
- Step 3. Set up forwarding of events from Falco into LogDNA
Step 1. Configure Kubernetes auditing
Kubernetes can be configured to emit audit events every time the Kubernetes API is accessed. You can process these events with Falco in real time to immediately identify suspicious our malicious behavior.
Create a virtual machine (VM) with the IBM Cloud Virtual Private Cloud (VPC) service (Generation 2).
Get the IP of your virtual machine’s public floating IP address.
$ ibmcloud target -r us-east Switched to region us-east API endpoint: https://cloud.ibm.com Region: us-east User: skrum@us.ibx.com Account: Cloud Open Sauce (xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx) <-> xxxxxxx Resource group: No resource group targeted, use 'ibmcloud target -g RESOURCE_GROUP' CF API endpoint: Org: Space: $ ibmcloud is instances Listing instances for generation 2 compute in all resource groups and region us-east under account Cloud Open Source as user skrum@us.ibm.com... ID Name Status Address Floating IP Profile Image VPC Zone Resource group xxxx_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx nibz-falco-dev running 10.241.128.4 52.xxx.xxx.xxx bx2-8x32 ibm-ubuntu-18-04-1-minimal-amd64-2 nibz us-east-3 Default
If you don’t have a floating IP set, return to the console to set one. It’s possible to set one via the CLI (see
ibmcloud is instance-network-interface-floating-ip-add
,floating-ip-reserve
, andinstance-network-interface-create
).
Install python so you can run Ansible
ssh ${remote_ip} 'sudo apt-get update; sudo apt-get -y install python'
Set up the Ansible inventory file, replacing
192.168.0.10
with the IP or DNS name of your server.[cloud] 192.168.0.10 ansible_ssh_user=ubuntu
Initialize
ansible-galaxy
.
ansible-galaxy init
Get the
ansible-sshd
Ansible role.ansible-galaxy install willshersystems.sshd
Get the
ansible-falco
Ansible role.git clone https://github.com/juju4/ansible-falco
Create a playbook file to use the Ansible role. Note the
hosts:
field if you customized anything in the inventory file.--- - hosts: cloud become: yes vars: falco_grpc_unix_enabled: true falco_webserver_enable: true # falco_dev: true sshd: GSSAPIAuthentication: no ChallengeResponseAuthentication: no PasswordAuthentication: no PermitRootLogin: no roles: - role: willshersystems.sshd - role: ansible-falco
Run the Ansible playbook to install Falco.
ansible-playbook -i inventory falco-install.yaml
SSH to host to validate the configuration.
ps -ef | grep falco tail -f /var/log/falco/falco.log
You should see notifications that Falco is running but nothing is specific to Kubernetes.
Step 2. Set up Kubernetes auditing
The IBM Cloud Kubernetes Service runs the Kubernetes daemons for you. Configuring Kubernetes auditing requires configuration of the Kubernetes API server command-line arguments. IBM Cloud provides a mechanism to control the audit webhook and where it points. Read more about it in the product documentation.
ibmcloud ks cluster master audit-webhook set --cluster <cluster_name> --remote-server http://<server_floating_ip>:8765/k8s_audit
ibmcloud ks cluster master audit-webhook get --cluster <cluster_name>
# refresh (takes a few seconds)
ibmcloud ks cluster master refresh --cluster <cluster_name>
Poke a hole in the firewall or security group for your instance for Kubernetes audit events.
ibmcloud is security-group-rule-add xxxx-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx inbound tcp --port-min 8765 --port-max 8765 --output JSON { "direction": "inbound", "id": "xxxx-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "ip_version": "ipv4", "port_max": 8765, "port_min": 8765, "protocol": "tcp", "remote": { "cidr_block": "0.0.0.0/0" } }
That should be it. Now you can check out your logs!
Edit your rules by modifying the Kubernetes rules under
/etc/falco
.View the logs with the following command:
tail -f /var/log/falco/falco.log
Jul 31 21:52:36 nibz-falco-dev falco: 21:52:20.110256128: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
Jul 31 21:52:36 nibz-falco-dev falco: 21:52:22.139158016: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-controller-manager target=kube-scheduler/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=15s resp=200)
Jul 31 21:52:36 nibz-falco-dev falco: 21:52:23.199183104: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
Jul 31 21:52:36 nibz-falco-dev falco: 21:52:25.244869888: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-controller-manager target=kube-scheduler/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=15s resp=200)
Jul 31 21:52:36 nibz-falco-dev falco: 21:52:26.293675008: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
Jul 31 21:52:36 nibz-falco-dev falco: 21:52:28.339374080: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-controller-manager target=kube-scheduler/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=15s resp=200)
Jul 31 21:52:36 nibz-falco-dev falco: 21:52:29.416363008: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
Jul 31 21:52:36 nibz-falco-dev falco: 21:52:31.453209088: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-controller-manager target=kube-scheduler/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=15s resp=200)
Jul 31 21:52:36 nibz-falco-dev falco: 21:52:35.614809088: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
Jul 31 21:53:03 nibz-falco-dev falco: 21:52:36.556668928: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
Jul 31 21:53:03 nibz-falco-dev falco: 21:52:39.623470080: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
Jul 31 21:53:03 nibz-falco-dev falco: 21:52:42.690744064: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
Jul 31 21:53:03 nibz-falco-dev falco: 21:52:45.761656064: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
Jul 31 21:53:03 nibz-falco-dev falco: 21:52:48.833193984: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
Jul 31 21:53:03 nibz-falco-dev falco: 21:52:51.915211008: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
Jul 31 21:53:03 nibz-falco-dev falco: 21:52:55.004199936: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
Jul 31 21:53:03 nibz-falco-dev falco: 21:52:58.079348992: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
Jul 31 21:53:03 nibz-falco-dev falco: 21:53:01.160041984: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
Step 3. Set up forwarding of events from Falco into LogDNA
Now set up IBM Log Analysis with Log DNA on IBM Cloud. You want at least 30 days of retention for this example. The following screen capture shows how the Logging page within IBM Cloud might look after you provision an instance of the IBM Log Analysis with LogDNA service.
On the Logging page, click Edit Log Sources for your new log instance. The easiest way to find your LogDNA API key and Log host is to select the Linux Ubuntu/Debian tab. Copy the key and host information. Note that the API host and Log host are different. You only need the Log host for this exercise.
Set up an environment file or export these variables to your shell environment.
# Modify the following URL if your Log host is not in us-south
export LOGDNA_URL="https://logs.us-south.logging.cloud.ibm.com/logs/ingest"
export LOGDNA_KEY="eb250a1fedd547d6ae0a"
Now pull down the falco-logdna
exporter script.
git clone https://github.com/falcosecurity/evolution
cp -r evolution/integrations/logdna/ .
virtualenv --python=python3 venv
source venv/bin/activate
pip install -r requirements.txt
In most cases, you must relax permissions on the Falco UNIX socket. You can also use a UNIX group if 777
is too insecure for you.
sudo chmod 777 /var/run/falco.sock
Now you can run the falco-logdna
exporter.
python falco-logdna.py --logdna-key ${LOGDNA_KEY} --logdna-url ${LOGDNA_URL}
Return to the Logging page and click View LogDNA to open the LogDNA UI. You should see Kubernetes security events in the LogDNA UI such as the ones in the following screen capture.
Conclusion
In this tutorial, you performed the following tasks:
- Configured IBM Cloud Kubernetes Service to transmit audit logs to your service.
- Set up a VM on IBM Cloud VPC with proper config and networking.
- Set up Falco to receive audit payloads from Kubernetes.
- Configured Falco to store those logs for further analysis in LogDNA.
Next steps
- Write or configure a Falco rule to pick out a specific security incident.
- Configure Falco to use ChatOps to alert admins of a security incident.
- Learn more about using Falco for auditing on IBM Cloud.
- Learn how to run Falco inside Kubernetes if you prefer not to run it on a separate VM host.