4 steps to debug your edge microservices in an Istio service mesh

One of the most common scenarios for users to onboard Istio is to use Istio as an ingress gateway and expose their microservices on the ingress gateway for external clients to access. There are typically 2 scenarios for this. The first scenario runs service A without a sidecar, while the second scenario runs service A with a sidecar to establish mTLS communication between the ingress gateway and service A.

Scenario 1: Service A without sidecar Scenario 1: Service A without sidecar

Scenario 2: Service A with sidecar Scenario 2: Service A with sidecar

Recently, I worked with an Istio user to help him debug why a service that was exposed on the Istio ingress gateway wasn’t reachable from a CURL client that runs outside of the Istio mesh. I wanted to share the key steps we went through to help others who may run into similar issues.

1. Use the istioctl analyzer command

Run the istioctl analyzer command on each of the Istio resources related to the service you are debugging. This analyzer command is becoming one of my favorite istioctl commands.

Analyzer allows you to examine individual YAML files with a particular namespace or an entire Kubernetes cluster. Ideally, before you deploy your Istio resources, you run the analyzer command on your Istio YAML files (for example, gateway or virtual service resources) with the namespace you are planning to deploy your Istio resource into. This will help you detect issues ahead of time.

$ istioctl analyze helloworld-gw-nlb.yaml -n istio-system             
✔ No validation issues found when analyzing namespace: istio-system.

If the Istio resources are already applied, you can run the analyzer on the entire cluster as there could be bad routes that conflict with yours. Use istioctl analyze --all-namespaces to analyze an entire cluster.

2. Validate the secret is loaded properly

If you are accessing your service securely — either via one-way TLS or mutual TLS — you want to make sure the secret is loaded in the ingress gateway so that the ingress gateway can leverage it to secure the inbound traffic.

The istioctl proxy-config secret command can help you to check what secrets are loaded to your ingress gateway, and you can use it to validate that your secret is loaded and valid for the correct periods. For example, the command below shows the secret named mycluster-wdc06-b-406454-85f044fc29ce613c264409c04a76c95d-0001 is loaded to my ingress gateway in the istio-system namespace that comes with default Istio installation.

$ istioctl proxy-config secret istio-ingressgateway-768c6bf77b-2k7hm.istio-system
RESOURCE NAME                                                      TYPE           STATUS     VALID CERT     SERIAL NUMBER                                  NOT AFTER                NOT BEFORE
mycluster-wdc06-b-406454-85f044fc29ce613c264409c04a76c95d-0001     Cert Chain     ACTIVE     true           415111668488852830681974512193801089848052     2020-12-05T09:43:42Z     2020-09-06T09:43:42Z

3. Examine Istio network resources

Time to check your gateway, virtual service resources for your edge service to ensure they are referenced properly! From your gateway resource, validate:

  • a. The namespace of the gateway resource is correct. This is typically the same namespace where your gateway deployment is.
  • b. credentialName in the gateway resource matches the name of the secret loaded in the gateway in step 2. As of Istio 1.7, the secret must reside in the same namespace as the gateway deployment resides.
  • c. The host field is correct. If your virtual service resides in a different namespace, I recommend you prefix the host value with the namespace, for example, hello/.
  • d. If you are using custom gateways, check the selector is correct. The selector must match the selector of your custom gateway deployment.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
  name: helloworld-gateway
  namespace: istio-system # step a
    istio: ingressgateway # step d
  - port:
      number: 443
      name: https
      protocol: HTTPS
      mode: SIMPLE
      credentialName: mycluster-wdc06-b-406454-85f044fc29ce613c264409c04a76c95d-0001 # step b   
    - "hello/hello.mycluster-wdc06-b-406454-85f044fc29ce613c264409c04a76c95d-0001.us-east.containers.appdomain.cloud". #step c

From your virtual service resource, validate:

  • a. The value of the gateway field is referenced correctly which should prefix with the gateway resource’s namespace if a virtual resource is in a different namespace from the gateway resource.
  • b. The hosts value also should match what is specified in the gateway resource.
  • c. Check the route is configured properly with the correct destination. Use a fully qualified host name when working with multiple namespaces.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
  name: helloworld
  namespace: hello # step a
  - "hello.mycluster-wdc06-b-406454-85f044fc29ce613c264409c04a76c95d-0001.us-east.containers.appdomain.cloud" # step b
  - istio-system/helloworld-gateway # step a
  - match:
    - uri:
        exact: /hello
    - destination:
        host: helloworld.hello.svc.cluster.local # step c
          number: 5000

4. Check ingress gateway logs

Check the ingress gateway log, validate the request is received at the ingress gateway and check any particular error code if it exist. Refer to this instruction titled, Requests are rejected by Enjoy to learn how to view log of the ingress gateway and common error codes.

Sometimes, you may need to enable a debug log for the ingress gateway using istioctl proxy-config log {ingress-pod}.istio-system --level debug. Or you can check out the gateway’s Envoy configuration using istioctl dashboard envoy {ingress-pod}.istio-system. Refer to the Debugging Envoy and Istiod documentation to learn more on how to debug envoy configuration.


With the above techniques, I hope you will be able to successfully troubleshoot your edge service issues with Istio ingress gateway as easy as I am.

If you still have issues, open an Istio GitHub issue and collect all required information. Good luck with Sailing your microservices with Istio, whether it is with Istio on IBM Cloud or elsewhere.