How to create and debug a Kubernetes StatefulSet

Each pod in a Kubernetes StatefulSet has a unique and stable hostname as well as a complete DNS identity. This ID is retained across restarts and scaling.

What are Kubernetes StatefulSets?

Kubernetes StatefulSets are special entities to manage applications that have special requirements for persistent data and fixed identities. These sets ensure that pods with replicas are started in a specific order. They assign each Kubernetes pod a unique ID and access to persistent storage. Such functions are useful for databases that rely on stable relationships between instances and that need to store persistent data. Therefore, StatefulSets represent an effective solution for the orchestration and operation of complex applications in Kubernetes.

How to create a Kubernetes StatefulSet

We start by defining a YAML configuration file in which we specify the desired properties of the StatefulSet and create Kubernetes pods. After configuration, the StatefulSet continuously monitors the cluster status and whether the predefined number of pods is always running and available. In the event of a pod failure or removal from the node, the StatefulSet automatically recognises the situation. It initiates the deployment of a new pod with the same unique ID. This new pod is connected to the existing persistent storage and receives the identical configuration as the original pod. This includes resource requests and limits.

This precise handling of pods and their identity is critical so that clients previously served by the unavailable pod can be redirected to the new pod without interruption. Access to persistent storage ensures that operations continue to function smoothly.

Here’s a complete YAML file that illustrates the steps for creating a Kubernetes StatefulSet for Nginx:

apiVersion: apps/v1
kind: StatefulSet
metadata:
    name: nginx-statefulset
spec:
    serviceName: nginx
    replicas: 3
    selector:
        matchLabels:
            app: nginx
    template:
        metadata:
            labels:
                app: nginx
        spec:
            containers:
            - name: nginx
                image: nginx:latest
                volumeMounts:
                - name: nginx-config
                    mountPath: /etc/nginx/conf.d
                - name: nginx-html
                    mountPath: /usr/share/nginx/html
    volumeClaimTemplates:
    - metadata:
            name: nginx-html
        spec:
            accessModes: [ "ReadWriteOnce" ]
            resources:
                requests:
                    storage: 1Gi
    volumeClaimTemplates:
    - metadata:
            name: nginx-config
        spec:
            accessModes: [ "ReadWriteOnce" ]
            resources:
                requests:
                    storage: 1Gi

This YAML document defines a Kubernetes StatefulSet for Nginx with three replicas. It uses a service object called nginx and labels to correctly identify the pods. The latter use the latest Nginx image and have two volumes for configuration and HTML files. The VolumeClaimTemplates secure persistent storage for these volumes with a size of 1 gigabyte and allow ReadWriteOnce access.

Debugging StatefulSets

Debugging StatefulSets in Kubernetes requires specific steps to verify that the pods are initialised correctly and that you can identify and fix errors if necessary.

Step 1: List the pods

Before you start debugging StatefulSets, you should check the status of the pods.

Open the command line and use the following command to list all pods in the desired StatefulSet:

kubectl get pods -l app=statefulset-label

Output:

NAME                         READY      STATUS    RESTARTS   AGE
nginx-statefulset-0           1/1      Running       0       2m
nginx-statefulset-1           1/1      Running       0       1m
nginx-statefulset-2           1/1      Running       0       1m

• NAME: Each pod is given a unique name based on the naming scheme of the StatefulSet and a consecutive number.
• READY: Indicates how many of the desired containers are ready in the pod. In the example, each pod has one container and 1/1 means that the container is ready for use.
• STATUS: This indicates the current status of the pod.
• RESTARTS: Shows how often the container has been restarted in the pod. A value of 0 means that there have been no restarts so far.
• AGE: Indicates how long the pod has been running.

These are the status messages that indicate errors:

• Failed: One or more containers in the pod caused an error that resulted in the failure of the pod. This can have various reasons, such as missing dependencies or configuration problems.
• Unknown: The state of the pod could not be determined. It could indicate a problem with the communication between the Kubernetes cluster and the pod. These include network problems, missing authorisations or other factors.

In both cases, it’s important to use accurate diagnostic tools such as checking pod logs or the kubectl describe pod command to get more details and determine the causes of the errors.

Step 2: Debug individual pods

Adding annotations to a pod can be a useful debugging tool to influence the initialisation process or trigger special actions.

First, you need to identify the name of the pod you want to debug.

kubectl get pods

Now enter the following command in the terminal to define the annotation for the selected pod:

kubectl annotate pods [pod-name] pod.alpha.kubernetes.io/initialized="false" --overwrite

Replace [pod-name] with the actual name of the pod. This annotation sets the initialisation status to false, which means that the pod is marked as uninitialised.

Monitor the pod to see how the annotation affects its behaviour. In particular, you can check the pod events and logs:

kubectl describe pod [pod-name]
kubectl logs [pod-name]

Look for events or log output that could cause problems during the initialisation process. When debugging is complete and you want to restore the normal initialisation process, set the annotation back to true.

Step 3: Step-by-step initialisation

If debugging the pod using the techniques described above was not successful, this could indicate race conditions during the bootstrap process of the StatefulSet. To overcome this issue, you can specify initialized="false" in the manifest of the Kubernetes StatefulSet and then create it with this annotation in the cluster.

apiVersion: apps/v1
kind: StatefulSet
metadata:
    name: [statefulset-name]
spec:
    template:
        metadata:
            annotations:
                pod.alpha.kubernetes.io/initialized: "false"
        ...

Apply the updated manifest to your Kubernetes cluster:

kubectl apply -f statefulset.yaml

Inspect the pods and identify any sources of error. Carry out the necessary debugging measures based on the observed events and logs. If necessary, delete pods with kubectl delete statefulsets or kubectl delete service.

After you have completed debugging, you can remove the initialised annotation and update the Kubernetes StatefulSet in the cluster:

kubectl annotate pods [pod-name] pod.alpha.kubernetes.io/initialized="true" --overwrite

The command sets the initialised annotation of a pod to true and ensures that existing values are overwritten. After you have checked the first pod, the Kubernetes StatefulSet will automatically initialise the next pod. You can then repeat the debugging steps for each additional pod in the StatefulSet.

In the Kubernetes tutorial, you’ll find detailed and practical information on setting up a Kubernetes cluster.