Think 2021: New tools have the developer ecosystem and IBM building together Learn more

Add runtime container security to your production Kubernetes workloads

Containers technology makes it easy to deploy applications in the cloud and Kubernetes is one of the popular choices for deploying containerized applications. But in this new and ever-changing container and microservices world, container security is critical. Vulnerabilities in applications residing within a container can be exploited if the right protections are not in place. This tutorial demonstrates how NeuVector integrates with the IBM Cloud Kubernetes Service to provide complete runtime container security for your production Kubernetes workloads. We demonstrate how quickly and easily you can deploy a Kubernetes cluster in IBM Cloud and then secure those workloads by using NeuVector to monitor your containers for vulnerabilities.

NeuVector is a cloud-native container firewall for monitoring and protecting Kubernetes container deployments in production. (Read the How to deploy a Docker container firewall article to learn how a container firewall differs from traditional next generation firewalls (NGFW) and web application firewalls (WAF).) In addition to Layer 7 network firewall protection of Kubernetes pods, the NeuVector security solution provides features for auditing your security settings with Docker Bench and the Center for Internet Security (CIS) Kubernetes Benchmark, as well as scanning containers for vulnerabilities. NeuVector also monitors host and container processes for suspicious activity such as privilege escalations, port scanning, reverse shells, and other unusual syscall activity.

The NeuVector solution is comprised of security containers, which can be deployed on each node similar to how you deploy your applications using Kubernetes. For evaluation purposes, NeuVector makes an Allinone container and an Enforcer container available. These can be pulled from Docker Hub, along with documentation, by requesting access from info@neuvector.com.

Prerequisites

To complete this tutorial, you will need the following environment:

In addition, you must contact support@neuvector.com to request that your Docker Hub ID be added to the NeuVector private registry.

Estimated time

After you meet the Prerequisites, completing this tutorial should take you approximately 30 minutes.

Steps

  1. Create a Kubernetes cluster in IBM Cloud
  2. Deploy NeuVector onto your Kubernetes cluster
  3. Generate test traffic by running sample applications
  4. Log into the NeuVector console

Step 1. Create a Kubernetes cluster in IBM Cloud

Log into your IBM Cloud account. Select Kubernetes from the Navigation Menu.

On the Kubernetes Overview page, select Create a Cluster.

To create a free cluster, set the following parameters:

  1. Select Free from the list of pricing plans.

  2. Within the Orchestration services section, select the most current version of Kubernetes (if you are presented with a choice).

  3. If you are offered Infrastructure options, choose Classic.

  4. If you are offered Location choices, keep the default options that are prefilled.

  5. If you are offered Worker pool choices to set up the number of worker nodes for your workload, leave it at the default number. (This can be resized.)

  6. Within the Resource details section, enter a name for your cluster.

  7. In the Summary pane, review the order summary and then click Create.

    Worker nodes can take a few minutes to provision, but you can see the progress in the Worker nodes tab. When the status reaches Ready, you can start working with your cluster. See the Getting started with IBM Cloud Kubernetes Service documentation for more details about cluster creation.

You can also create a free cluster from the command line by using the following IBM Cloud CLI command:

ibmcloud ks cluster create classic --name my_cluster

See the IBM Cloud Kubernetes Service CLI documentation for more information about cluster commands.

Step 2. Deploy NeuVector onto your Kubernetes cluster

Now that the environment is provisioned, you can access it from the IBM Cloud CLI tool that you downloaded in the Prerequisites.

First, go to the Kubernetes clusters page and click the name of the cluster that you created in Step 1.

On your cluster page, click Access from the menu and follow the Accessing your cluster instructions to log into your cluster from your local command line interface.

Screen capture of instructions for accessing your cluster from the CLI

  1. Follow the instructions on the screen to:

    • Log into your cluster.
    • Set the Kubernetes context to your cluster.
    • Verify that you can connect to your cluster.
  2. Create the namespace for NeuVector by entering the following command:

    kubectl create namespace neuvector
    

    You should get a response of namepsace/neuvector created.

  3. Configure Kubernetes to pull from the private NeuVector registry on Docker Hub. You do this by creating a secret with the following command, where <docker id> is your Docker username, <pwd> is your Docker password, and <email> is your Docker email.

    kubectl create secret docker-registry regsecret -n neuvector --docker-server=https://index.docker.io/v1/ --docker-username="<docker id>" --docker-password="<pwd>" --docker-email="<email>"
    

    You should get a response of secret/regsecret created.

    Note: By this step, you should have already completed the prerequisite to have your Docker Hub ID added to the NeuVector private registry.

  4. Create the custom resource definition (CRD) for the NeuVector security rules.

    • For Kubernetes 1.19+, use the following command:

      kubectl apply -f https://raw.githubusercontent.com/neuvector/manifests/main/kubernetes/crd-k8s-1.19.yaml
      
    • For Kubernetes 1.18+ and earlier, use the following command:

      kubectl apply -f https://raw.githubusercontent.com/neuvector/manifests/main/kubernetes/crd-k8s-1.16.yaml
      

    You will see results similar to the ones in the following screen capture image.

    Screen capture of steps to set up CRD for the NeuVector security rules

    In particular, you will see the following lines in the results:

    customresourcedefinition.apiextensions.k8s.io/nvsecurityrules.neuvector.com unchanged
    customresourcedefinition.apiextensions.k8s.io/clusternvsecurityrules.neuvector.com unchanged
    
  5. Add read permission to access the Kubernetes API with the following commands. Note that RBAC is officially supported in Kubernetes 1.8+ and admission control is supported in Kubernetes 1.9+.

    kubectl create clusterrole neuvector-binding-app --verb=get,list,watch,update --resource=nodes,pods,services,namespaces
    kubectl create clusterrole neuvector-binding-rbac --verb=get,list,watch --resource=rolebindings.rbac.authorization.k8s.io,roles.rbac.authorization.k8s.io,clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io
    kubectl create clusterrolebinding neuvector-binding-app --clusterrole=neuvector-binding-app --serviceaccount=neuvector:default
    kubectl create clusterrolebinding neuvector-binding-rbac --clusterrole=neuvector-binding-rbac --serviceaccount=neuvector:default
    kubectl create clusterrole neuvector-binding-admission --verb=get,list,watch,create,update,delete --resource=validatingwebhookconfigurations,mutatingwebhookconfigurations
    kubectl create clusterrolebinding neuvector-binding-admission --clusterrole=neuvector-binding-admission --serviceaccount=neuvector:default
    kubectl create clusterrole neuvector-binding-customresourcedefinition --verb=watch,create,get --resource=customresourcedefinitions
    kubectl create clusterrolebinding  neuvector-binding-customresourcedefinition --clusterrole=neuvector-binding-customresourcedefinition --serviceaccount=neuvector:default
    kubectl create clusterrole neuvector-binding-nvsecurityrules --verb=list,delete --resource=nvsecurityrules,nvclustersecurityrules
    kubectl create clusterrolebinding neuvector-binding-nvsecurityrules --clusterrole=neuvector-binding-nvsecurityrules --serviceaccount=neuvector:default
    kubectl create clusterrolebinding neuvector-binding-view --clusterrole=view --serviceaccount=neuvector:default
    kubectl create rolebinding neuvector-admin --clusterrole=admin --serviceaccount=neuvector:default -n neuvector
    

    You will see results similar to the following:

    $ kubectl create -f neuvector.yaml
    service/neuvector-service-controller-fed-master created
    service/neuvector-service-controller-fed-worker created
    service/neuvector-svc-crd-webhook created
    service/neuvector-svc-admission-webhook created
    service/neuvector-service-webui created
    service/neuvector-svc-controller created
    deployment.apps/neuvector-manager-pod created
    deployment.apps/neuvector-controller-pod created
    daemonset.apps/neuvector-enforcer-pod created
    deployment.apps/neuvector-scanner-pod created
    cronjob.batch/neuvector-updater-pod created
    

    Note: If you are upgrading NeuVector from a previous install, you may need to delete the old binding with the following commands:

    kubectl delete clusterrolebinding neuvector-binding
    kubectl delete clusterrole neuvector-binding
    
  6. Run the following commands to check if the neuvector:default service account is added successfully:

    kubectl get clusterrolebinding  | grep neuvector
    kubectl get rolebinding -n neuvector | grep neuvector
    

    Sample output:

    neuvector-binding-admission                            28d
    neuvector-binding-app                                  28d
    neuvector-binding-customresourcedefinition             28d
    neuvector-binding-nvsecurityrules                      28d
    neuvector-binding-rbac                                 28d
    neuvector-binding-view                                 28d
    neuvector-admin   28d
    
  7. Create the NeuVector services and pods from the Kubernetes sample YAML file located in the GitHub repo for this tutorial.

    Important note: If you want to use another version for the NeuVector manager, controller, and enforcer images, replace the <version> tags in the sample neuvector.yaml file. (The current version is 4.1.1.) Also, make any other modifications required for your deployment environment. Note that since this tutorial is using the IBM Cloud Kubernetes Service, we are using containerd for the Nevector controller and enforcer pods in the YAML file.

    kubectl create -f neuvector.yaml
    

    You can verify that your pods are available and running by using the following command:

    kubectl get pod -n neuvector
    

    You will see results similar to the following:

    $ kubectl get pod -n neuvector
    NAME                                         READY      STATUS        RESTARTS        AGE
    neuvector-controller-pod-59b9cb9cfb-h17vh     1/1       Running         0             3d17h
    neuvector-controller-pod-59b9cb9cfb-s4pd      1/1       Running         0             3d17h
    neuvector-controller-pod-59b9cb9cfb0n9nbz     1/1       Running         0             3d17h
    neuvector-enforcer-pod-6vsg6                  1/1       Running         0             3d17h
    neuvector-manager-pod-574ddbdf75-86qdx        1/1       Running         0             3d17h
    
    ....
    

    You can also verify that your services are running by executing the following command:

    kubectl get svc -n neuvector
    

    You will see results similar to the following:

    $ kubectl get svc -n neuvector
    NAME                                          TYPE              CLUSTER-IP        EXTERNAL-IP   PORT(S)              AGE
    neuvector-service-controller-fed-master       LoadBalancer      172.21.192.153    <pending>     11443:31838/TCP      3d17h
    neuvector-service-controller-fed-worker       LoadBalancer      172.21.163.176    <pending>     10443:32146/TCP      3d17h
    neuvector-service-webui                       NodePort          172.21.20.160     <pending>     8443:32340/TCP       3d17h
    
    ....
    

    If you receive any errors, refer to the Troubleshooting section later in this tutorial.

    You should be able to connect to the NeuVector console and log in with admin:admin. For example, https://<public-ip>:8443. Don’t forget to apply your license file after logging in so that you can see all Nodes, Enforcers, and Assets.

    Note: The nodeport service specified in the sample neuvector.yaml file will open a random port on all Kubernetes nodes for the NeuVector management web console port. Alternatively, you can use a LoadBalancer or Ingress by using a public IP and default port 8443. For nodeport, be sure to open access through firewalls for that port if necessary. If you want to see which port is open on the host nodes, run the following commands:

    $ kubectl get svc -n neuvector
    

Step 3. Generate test traffic by running sample applications

If you have not already deployed some sample applications, now is a good time to do that so that you are able to see application containers running and view their connections in NeuVector. Examples of sample applications are available in the following tutorials and code pattern:

After generating test traffic through your sample apps, log into the NeuVector console.

Step 4. Log into the NeuVector console

Since the webui is defined in the .yaml file as type NodePort, and IBM Cloud Kubernetes Service is using containerd, you must access an external port.

You can identify the external IP address of the NeuVector service deployed on your cluster by running the following command:

kubectl get nodes -o wide

Following is an example of the output:

$kubectl get nodes -o wide
NAME              STATUS    RULES     AGE      VERSION        INTERNAL-IP      EXTERNAL-IP      OS-IMAGE             KERNAL=VERSION      CONTAINER-RUNTIME
10.144.216.102     READY    <none>    11d      v1.10.15*IKS   10.144.216.102   169.51.195.40    Ubuntu 16.04.7 LTS   4.4.0*200-generic   containerd://1.3.9

If you completed all of the prerequisites, you already created a free trial license for the NeuVector full lifecycle container security console from the Red Hat Marketplace. Therefore, you can access the NeuVector console in a browser and log in with admi:admin using the following command:

https://169.51.195.48:32340
  • IP Addr: was pulled from the previous get nodes command.
  • Port: was pulled from the get pod command that you ran in Step 2, task 7. Choose the port associated with the webui pod.

You will see a login screen such as the one in the following image.

Screen capture of NeuVector console login page

After you log in, you should see the NeuVector console, as illustrated in the following image.

Screen capture of NeuVector console dashboard

Feel free to browse the console. View the network activity, policy rules, and other resources. To learn more about the using the console, see the NeuVector solution overview video.

Troubleshooting

  • If you observe errors after running the get pod or get svc commands, you can try to dig deeper and find out whether there are errors when you run the following command:

    kubectl describe pod <pod Name>  -n neuvector
    
  • If the 'kubectl replace -f neuvector.yaml' command does not work, you can edit and force Kubernetes to re-read the .yaml file to update a specific pod configuration (such as image version) by running the following command (this example shows an update to the webui pod):

    kubectl edit svc neuvector-service-webui -n neuvector
    
  • Sometimes, unexplainable errors occur with the pod. Ensure that you still have your secret established. You can verify it by using the following command:

    get secret regsecret -n neuvector --output="jsonpath={.data.\.dockerconfigjson}" | base64 --d -
    

Summary

The IBM Cloud Kubernetes Service makes it easy to set up a Kubernetes cluster to host your containerized applications. When running such applications in production, security is required to ensure that the applications are safe and communicating properly. NeuVector provides that runtime security in any cloud environment, providing a Layer 7 firewall, host and container processes monitoring, and vulnerability scanning solution. You can request a demo and access to the download by contacting NeuVector at info@neuvector.com.