Real-time auditing of Kubernetes events with Falco – IBM Developer

Join the Digital Developer Conference: AIOps & Integration to propel your AI-powered automation skills Register for free

Tutorial

Real-time Kubernetes auditing with Falco

Audit security events from users, operators, and other processes

Save Like

By Spencer Krum
Published December 17, 2020

As Kubernetes becomes the platform of choice for both app development and deployment, securing Kubernetes is growing from a passing concern to a first class requirement. This tutorial covers auditing Kubernetes clusters in real time for activity and building a system to automatically log and process audit events.

Kubernetes is API driven. In many deployments each user, administrator, and developer communicates with the API. When a user issues a kubectl create deployment command, it hits the API and the request is logged through the Kubernetes audit system. There are also nonhuman API interactions to log and audit, operators work directly against the API with their own set of credentials, and external systems from cloud providers hit the API. All of these interactions should be logged, audited, and processed with alerts sent out if out-of-compliance behavior is detected.

Falco is a Cloud Native Computing Foundation (CNCF) security tool that has first-tier support for Kubernetes audit payloads and a rich set of tools for processing those events.

Prerequisites

Estimated time

You can complete this tutorial in less than 30 minutes.

Steps

Step 1. Configure Kubernetes auditing

Kubernetes can be configured to emit audit events every time the Kubernetes API is accessed. You can process these events with Falco in real time to immediately identify suspicious our malicious behavior.

  1. Create a virtual machine (VM) with the IBM Cloud Virtual Private Cloud (VPC) service (Generation 2).

  2. Get the IP of your virtual machine’s public floating IP address.

    $ ibmcloud target -r us-east
Switched to region us-east

API endpoint:      https://cloud.ibm.com
Region:            us-east
User:              skrum@us.ibx.com
Account:           Cloud Open Sauce (xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx) <-> xxxxxxx
Resource group:    No resource group targeted, use 'ibmcloud target -g RESOURCE_GROUP'
CF API endpoint:
Org:
Space:

$ ibmcloud is instances
Listing instances for generation 2 compute in all resource groups and region us-east under account Cloud Open Source as user skrum@us.ibm.com...
ID                                          Name              Status    Address        Floating IP     Profile    Image
                              VPC       Zone        Resource group
xxxx_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   nibz-falco-dev    running   10.241.128.4   52.xxx.xxx.xxx   bx2-8x32   ibm-ubuntu-18-04-1-minimal-amd64-2   nibz      us-east-3   Default

    If you don’t have a floating IP set, return to the console to set one. It’s possible to set one via the CLI (see ibmcloud is instance-network-interface-floating-ip-add, floating-ip-reserve, and instance-network-interface-create).

  1. Install python so you can run Ansible

    ssh ${remote_ip} 'sudo apt-get update; sudo apt-get -y install python'

  2. Set up the Ansible inventory file, replacing 192.168.0.10 with the IP or DNS name of your server.

    [cloud]
192.168.0.10 ansible_ssh_user=ubuntu

  3. Initialize ansible-galaxy.

  ansible-galaxy init

  1. Get the ansible-sshd Ansible role.

    ansible-galaxy install willshersystems.sshd

  2. Get the ansible-falco Ansible role.

    git clone https://github.com/juju4/ansible-falco

  3. Create a playbook file to use the Ansible role. Note the hosts: field if you customized anything in the inventory file.

    ---
- hosts: cloud
 become: yes
 vars:
   falco_grpc_unix_enabled: true
   falco_webserver_enable: true
   #    falco_dev: true
   sshd:
     GSSAPIAuthentication: no
     ChallengeResponseAuthentication: no
     PasswordAuthentication: no
     PermitRootLogin: no
 roles:
   - role: willshersystems.sshd
   - role: ansible-falco

  4. Run the Ansible playbook to install Falco.

    ansible-playbook -i inventory falco-install.yaml

  5. SSH to host to validate the configuration.

    ps -ef | grep falco

tail -f /var/log/falco/falco.log

You should see notifications that Falco is running but nothing is specific to Kubernetes.

Step 2. Set up Kubernetes auditing

The IBM Cloud Kubernetes Service runs the Kubernetes daemons for you. Configuring Kubernetes auditing requires configuration of the Kubernetes API server command-line arguments. IBM Cloud provides a mechanism to control the audit webhook and where it points. Read more about it in the product documentation.

ibmcloud ks cluster master audit-webhook set --cluster <cluster_name> --remote-server http://<server_floating_ip>:8765/k8s_audit
ibmcloud ks cluster master audit-webhook get --cluster <cluster_name>

# refresh (takes a few seconds)
ibmcloud ks cluster master refresh --cluster <cluster_name>

  1. Poke a hole in the firewall or security group for your instance for Kubernetes audit events.

    ibmcloud is security-group-rule-add xxxx-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx inbound tcp --port-min 8765 --port-max 8765 --output JSON
{
   "direction": "inbound",
   "id": "xxxx-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "ip_version": "ipv4",
   "port_max": 8765,
   "port_min": 8765,
   "protocol": "tcp",
   "remote": {
       "cidr_block": "0.0.0.0/0"
   }
}

  2. That should be it. Now you can check out your logs!

    Edit your rules by modifying the Kubernetes rules under /etc/falco.

    View the logs with the following command:

    tail -f /var/log/falco/falco.log

  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:20.110256128: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:22.139158016: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-controller-manager target=kube-scheduler/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:23.199183104: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:25.244869888: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-controller-manager target=kube-scheduler/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:26.293675008: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:28.339374080: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-controller-manager target=kube-scheduler/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:29.416363008: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:31.453209088: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-controller-manager target=kube-scheduler/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:35.614809088: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:36.556668928: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:39.623470080: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:42.690744064: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:45.761656064: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:48.833193984: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:51.915211008: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:55.004199936: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:58.079348992: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:53:01.160041984: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)

Step 3. Set up forwarding of events from Falco into LogDNA

Now set up IBM Log Analysis with Log DNA on IBM Cloud. You want at least 30 days of retention for this example. The following screen capture shows how the Logging page within IBM Cloud might look after you provision an instance of the IBM Log Analysis with LogDNA service.

Screen capture of the Logging pane within IBM Cloud

On the Logging page, click Edit Log Sources for your new log instance. The easiest way to find your LogDNA API key and Log host is to select the Linux Ubuntu/Debian tab. Copy the key and host information. Note that the API host and Log host are different. You only need the Log host for this exercise.

Screen capture of the commands that appear underneath the Linux Ubuntu/Debian tab

Set up an environment file or export these variables to your shell environment.

# Modify the following URL if your Log host is not in us-south
export LOGDNA_URL="https://logs.us-south.logging.cloud.ibm.com/logs/ingest"
export LOGDNA_KEY="eb250a1fedd547d6ae0a"

Now pull down the falco-logdna exporter script.

git clone https://github.com/falcosecurity/evolution
cp -r evolution/integrations/logdna/ .
virtualenv --python=python3 venv
source venv/bin/activate
pip install -r requirements.txt

In most cases, you must relax permissions on the Falco UNIX socket. You can also use a UNIX group if 777 is too insecure for you.

sudo chmod 777 /var/run/falco.sock

Now you can run the falco-logdna exporter.

python falco-logdna.py --logdna-key ${LOGDNA_KEY} --logdna-url ${LOGDNA_URL}

Return to the Logging page and click View LogDNA to open the LogDNA UI. You should see Kubernetes security events in the LogDNA UI such as the ones in the following screen capture.

Screen capture of the LogDNA UI with visible Kubernetes security events

Conclusion

In this tutorial, you performed the following tasks:

  • Configured IBM Cloud Kubernetes Service to transmit audit logs to your service.
  • Set up a VM on IBM Cloud VPC with proper config and networking.
  • Set up Falco to receive audit payloads from Kubernetes.
  • Configured Falco to store those logs for further analysis in LogDNA.

Next steps

Legend

Related

Event | Conference

Kubecon & CloudNativeCon Europe 2021

May 4, 2021 10:00 am CEST
Video

Wild and dangerous with Marty & JJ: Helm - The Story - with Matt Butcher and Rimas Mocevicius, Part 2

April 12, 2021
Video

Wild and dangerous with Marty & JJ: Helm - The Story - with Matt Butcher and Rimas Mocevicius, Part 1

April 12, 2021