When a Kubernetes deployment malfunctions, you need to identify the scope of the problem before you can begin fixing it. As with any troubleshooting method, you can find the root cause of the problem by starting small and working your way up. For Kubernetes errors, the issue can be either at the pod, service, or ingress level.
First, start small and confirm that your pods are running and ready. Once you’ve done this, you know to next look at your service configuration.
In this guide, we’ll assume that you’ve already confirmed that the issue is not at the pod level and we’ll walk you through how to troubleshoot your Kubernetes service.
Service Troubleshooting Steps
Kubernetes services work to route traffic to designated pods according to their labels. There are several ways that a service might fail to link to the proper pod. Although some fixes for these issues are as simple as ensuring that the service in question exists, others are a bit more involved.
Step 1: Checking that the Service Exists
The first step in troubleshooting a service misconfiguration is to make sure the one you're calling for is present. As fundamental as it sounds, it’s an often-overlooked error that the service attempting to be accessed doesn't actually exist.
You can check this with this command:
If the service doesn't exist, you'll receive this message:
To resolve this problem, create a new service by using something similar to this sample command (note that the specifics of your code — like the port and target port labels — are likely to vary):
To confirm that the service now exists, you can read it back with this command:
Then, a display similar to this one should appear:
Once you've checked to make sure that all services exist, take the time to make sure that each one is correctly defined and matches the port it's assigned to. This is a basic step, but it could prevent you from having to troubleshoot any further.
Step 2: Checking the Service Endpoint
Once a service is shown to exist and match its ports, you should make sure that the service is selecting the pods attached to it. Start by reviewing the activity level of the pods with this command:
The argument "-l app=hostnames" is called the selector label, and the Kubernetes system contains a control loop that keeps track of each one for every service in play. It saves the results into an "endpoints" object, which can be displayed with this command:
The output might look something like this:
Of course, specific data will vary, but some information on the service endpoints should appear. If the "ENDPOINTS" column simply reads "None," then the "spec.selector" service field may not be selecting the pods according to the "metadata.labels" that were specified in the YAML. This is likely due to a typo or similar error.
Step 3: Testing the Service
As you troubleshoot your service, it will be necessary to test it along the way. This is useful when endpoints and running pods can be found in the queries above. The command used to test the service is:
In this instance, "<service-name>" represents the name of the service, "3000" represents the port you're looking to open, and "80" corresponds to the port that the service exposes.
Troubleshooting Kubernetes Services with Blink
Every time you run into an error, you can work through these manual checks but it might be time-consuming to remember the specifics of each one. There’s an easier way to manage your Kubernetes troubleshooting.
With a free Blink account, you can click through pre-built troubleshooting automations so these commands are at your fingertips. Resolve errors faster and have confidence that you’ll have access to all the information you need quickly.
Get started and create your free Blink account today.