Overview

Skill Level: Intermediate

This recipe describes a way to automatically capture Linux 'perf' data from PODs by leveraging sidecar injection

Ingredients

Ensure you are using Kubernetes 1.10+ and the following settings are enabled:

  • Feature-gate PodShareProcessNamespace=true¬†turned on for both apiserver and kubelet¬†
  • Ensure kube-apiserver has the¬†admission-control¬†flag set with¬†MutatingAdmissionWebhook¬†and¬†ValidatingAdmissionWebhook¬†admission controllers added¬†
$ kubectl api-versions | grep admissionregistration
admissionregistration.k8s.io/v1beta1

For IBM Cloud Private, you can use the kube_apiserver_extra_args and kubelet_extra_args configuration parameters during install.

Example:

kube_apiserver_extra_args: [‚Äú--feature-gates=PodShareProcessNamespace=true‚ÄĚ]
kubelet_extra_args: [‚Äú--feature-gates=PodShareProcessNamespace=true‚ÄĚ]

 

More details on IBM Cloud Private install time config options is available from the following link - https://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0.3/installing/config_yaml.html

 

Before getting into the setup details, let's take a quick look at the 'perf' container which gets added as a sidecar via the injector.

The source of the perf container is available in the following github link - https://github.com/bpradipt/perf-container

The perf container can be run with the following options:

 

entrypoint.sh <stat|record > <trigger[1|0]> <max-run-time[X]> <repeat-count[N]> <extra-perf-args>

 

where:

  • stat ‚Äď runs perf stat command
  • record ‚Äď runs perf record command
  • probe ‚Äď runs perf probe command
  • trigger - The 'perf' command can be started in response to a trigger ‚Äď create/open/modify events for file /tmp/startperf
  • max-run-time - is the max time the perf command will be run. Default is 60 sec

 

A sample invocation looks like this

 

"perf stat -a -p 1,3,8 -x ',' -I 1000 -A -o /out/perf-stat-output-abcxyz --append -g"

 

This captures perf stat for PIDs ‚Äď 1, 3, 8 including call-graph and dumps to /out/perf-stat-output-abcxyz

Step-by-step

  1. Deploy the 'Perf' Sidecar Injector

     

    • Clone the source
    $ git clone https://github.com/bpradipt/perf-sidecar-injector.git
    $ cd perf-sidecar-injector
    • Create a signed certificate/key pair and store it in a Kubernetes secret that will be consumed by the sidecar deployment
    $ ./deployment/webhook-create-signed-cert.sh \
        --service perf-sidecar-injector-webhook-svc \
        --secret perf-sidecar-injector-webhook-certs \
        --namespace default
    • Patch the MutatingWebhookConfiguration by setting caBundle with correct value from Kubernetes cluster
    $ cat deployment/mutatingwebhook.yaml | \
        deployment/webhook-patch-ca-bundle.sh > \
        deployment/mutatingwebhook-ca-bundle.yaml
    • Deploy sidecar injector resources

    ¬† ¬† ¬† ¬† ¬† ¬†Edit the configmap as per your requirement. By default the ‚Äėperf‚Äô data is kept in the hostPath ‚Äď ‚Äė/out‚Äô

    $ kubectl create -f deployment/configmap-record.yaml
    $ kubectl create -f deployment/deployment.yaml
    $ kubectl create -f deployment/service.yaml
    $ kubectl create -f deployment/mutatingwebhook-ca-bundle.yaml
  2. Verify

    Check if the sidecar injector webhook is running

    $ kubectl get pods
    NAME                                                  READY     STATUS    RESTARTS   AGE
    sidecar-injector-webhook-deployment-bbb689d69-882dd   1/1       Running   0          5m

    $ kubectl get deployment
    NAME                                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    sidecar-injector-webhook-deployment   1         1         1            1           5m
  3. Use

    There are two ways by which the webhook can be instructed to add the ‘perf’ sidecar

    1. annotation¬† – Adding ‘perf-sidecar-injector-webhook/inject: “yes”‘ annotation to the POD spec will ensure automatic injection of ‘perf’ sidecar.
    2. namespace label – Adding ‘sidecar-injector=enabled’ label to a specific namespace will ensure automatic injection of ‘perf’ sidecar to every POD deployed in this namespace

    Let’s use the namespace label option such that any POD deployed to the ‘default’ namespace will have the ‘perf’ sidecar added automatically.

     

    • Add label

     

    $ kubectl label namespace default sidecar-injector=enabled
    $ kubectl get namespace -L sidecar-injector
    NAME          STATUS    AGE       SIDECAR-INJECTOR
    default       Active    18h       enabled
    kube-public   Active    18h
    kube-system   Active    18h
    • Deploy a sample application and collect ‘perf’ data

     

    apiVersion: v1
    kind: Pod
    metadata:
        name: app-sample
        namespace: default
    spec:
        containers:
    - name: app-sample
          image: ubuntu:16.04
          imagePullPolicy: IfNotPresent
          command:
          - bash
          args:
            - '-c'
            - 'mkdir -p /out && dd if=/dev/zero of=/out/file count=8192 oflag==direct && sleep infinity'
        restartPolicy: Never

    You can use ‘perf report’ or other visualization tools to view the captured ‘perf’ data

    Take a look at the following video for a complete demonstration –¬†https://asciinema.org/a/ROb7NNyibXXV1sRPg1ZG8jXjg

Join The Discussion