API Connect

 View Only

Recipe: APIC debugging v2018 and v10

By Will LIAO posted Tue September 28, 2021 10:36 AM

  
Contents

Overview

Skill Level: Any Skill Level

Ingredients

Set up apicadm user to access the K8s pods

Important: Before you set the access, ensure that you are under the /home/apicadm directory.

  1. Create local dir to store k8s config: mkdir .kube
  2. Copy k8s config file to apicadm user: sudo cp -i /etc/kubernetes/admin.conf .kube/config 
  3. Then run sudo chown -R apicadm .kube and sudo chgrp -R apicadm .kube
  4. Verify that apicadm user can see access the K8s resources: kubectl get nodes

Step-by-step

  1. APIC Status Checks (toolkit commands)

    Once the apicadm user can access k8s pods you may use the apic commands to check the health of the k8s pods.

    • sudo apic status
      Provide status of the APIC k8s pods.

    • sudo apic health-check
      Check the health of the cluster: (NOTE: If healthy, the command will end silently).
    • sudo apic logs
      Retrieve logs from all nodes.
      Two zips files will be output:
      – appliance-logs-{datetime}.zip which contains mgmt logs in json file
      – node-logs-{datetime}.zip which contains k8s and apiconnect pod details.

    • kubectl get pods –all-namespaces
      List pod status across all namespaces.

    Note: Output of pods will be different from v2018 and v10. Please see the white paper or knowledge center for more details on which pods are supposed to be running.

     

     

  2. Management Subsystem Troubleshooting

    The following section will go over details on troubleshooting the Portal service.

    The mustgather details from IBM support may encapsulate all the data needed for debugging issues.

    • kubectl logs -f {apiconnect_pod e.g. apiconnect-apim-v2-86b464c4f5-4cx6r}

      Tail the APIM pod.

  3. APICOPS

    apicops is a command line interface to IBM API Connect specifically targetted at Operations teams. It contains commands to check the healthy running of the system as well as some commands to remedy specific problems if encountered.

    The tool is valuable when it is suspect of synchronization issues between APIC instances, like when Products and APIs are not being published or when 404s occur when Products and APIs are supposedly published. The utility will assist in synchronize content between the APIM and the Gateways and Portals.

    The apicops iss utility is in continuous development and is maintained on the ibm-apiconnect/apicops github site.

    Some key commands are outlined here:

    • apicops iss -n {namespace}
      This outputs lists all the tasks and the state it is in based on it’s associated tasks. It will show any unsynchronized tasks.
    • apicops task-queue:list-stuck-tasks
      This command will identify stuck webhook tasks which have not been updated within the default 15 minutes timeframe.
    • apicops task-queue:fix-stuck-tasks
      This command will force webhooks to execute and attempt to synchronize tasks.

  4. Portal Troubleshooting

    The following section will go over details on troubleshooting the Portal service.

    The mustgather details from IBM support may encapsulate all the data needed for debugging issues.

    Reference for troubleshooting portal may be found in the IBM knowledge center: v2018 and v10

    Commands to retrieve portal logs:

    • kubectl logs {Portal_WWW_pod e.g. portal-apic-portal-www-zhqxf web} > web.log
      Gather web container logs: portal www-web container runs the drupal websites.
      (Issues with user registration and login)
    • kubectl logs {Portal_WWW_Pod e.g portal-apic-portal-www-zhqxf admin} > admin.log
      Gather admin container logs: portal www-admin container handles communication and integration with the APIM.
      (Issues with Products and APIs not showing up in the portal)
      Note: you may also tail the log without the save log: kubectl logs -f portal-apic-portal-www-dtbzw -c admin
    • kubectl logs -f {portal_pod e.g. portal-apic-portal-nginx-866c977c4c-n9gbf}
      Tail nginx pod log for portal.
      (Issues with Products not appearing in the dev portal)
0 comments
16 views

Permalink