skip to content
Ruban Selvarajah

Stuck in CrashLoopBackOff due to PersistentVolumeClaim

/ 2 min read

google-cloud, kubernetes

The CrashLoopBackOff state means that your pod is stuck in a infinite loop where it starts and crashes, then restarts and crashes again. It’s a useful feature for debugging but a bit annoying when the bug in your code lives in a persistent volume.

The following steps will allow you to temporarily disable this feature so that you can get rid of the offending code in peace.

Let’s 🚀

Downscale the Deployment

Stop using the PersistentVolumeClaim so that we can update our code

kubectl get deployments
kubectl scale deployment DEPLOYMENT_NAME --replicas=0


//Returns "deployment.extensions "DEPLOYMENT_NAME" scaled"

Create a Debugger Pod

1. Get the PVC identifier

kubectl get deployment -o jsonpath="{.spec.template.spec.volumes[*].persistentVolumeClaim.claimName}" DEPLOYMENT_NAME


//Returns "PVC_IDENTIFIER"

2. Prep a debugger pod

pvc-debugger-pod.yaml
---
kind: Pod
apiVersion: v1
metadata:
  name: pvc-debugger
spec:
  volumes:
    - name: my-buggy-volume
      persistentVolumeClaim:
        claimName: <PVC_IDENTIFIER>
  containers:
    - name: debugger
      image: busybox
      command: ["sleep", "3600"]
      volumeMounts:
        - mountPath: "/data"
          name: my-buggy-volume

3. Start the debugger pod

kubectl create -f pvc-debugger-pod.yaml
kubectl exec -it pvc-debugger sh

Now that I’m inside the container I can explore the volume which is mounted at /data and fix the issue.

Bring the Pod Online

1. Delete the debugger pod

kubectl delete -f pvc-debugger-pod.yaml

2. Scale the deployment back up

kubectl scale deployment DEPLOYMENT_NAME --replicas=1
//Returns "deployment.extensions "DEPLOYMENT_NAME" scaled"

Aaaand my site is up again!