Kubernetes NFS Mount Troubleshooting

Problem

Pods with NFS volumes get stuck in ContainerCreating state indefinitely. The error messages from kubectl describe pod can be misleading, showing protocol or permission errors when the actual issue is the NFS export doesn't exist.

Context / Trigger Conditions

• Pod status shows ContainerCreating for more than 1-2 minutes
• kubectl describe pod shows events like:
  • MountVolume.SetUp failed for volume "data" : mount failed: exit status 32
  • mount.nfs: Protocol not supported
  • mount.nfs: access denied by server
• Pod spec includes an NFS volume mount
• Other pods on the same node work fine

Solution

Step 1: Identify the NFS path

kubectl describe pod -n <namespace> <pod-name> | grep -A5 "Volumes:"

Look for the NFS server and path (e.g., 10.0.10.15:/mnt/main/myservice)

Step 2: Verify the export exists on NFS server

SSH to the NFS server and check:

ssh root@<nfs-server> "ls -la /mnt/main/myservice"

Step 3: If directory doesn't exist, create it

ssh root@<nfs-server> "mkdir -p /mnt/main/myservice && chmod 777 /mnt/main/myservice"

Step 4: Add to NFS exports (TrueNAS specific)

For TrueNAS, add the path to the NFS share configuration:

1. Add directory to scripts/nfs_directories.txt
2. Run scripts/nfs_exports.sh to update the share via API

Step 5: Restart the pod

kubectl delete pod -n <namespace> -l app=<app-label>

The deployment will create a new pod that should now mount successfully.

Verification

kubectl get pods -n <namespace>
# Should show 1/1 Running instead of 0/1 ContainerCreating

kubectl exec -n <namespace> <pod-name> -- ls -la /app/data
# Should show the mounted directory contents

Example

Symptom:

Events:
  Warning  FailedMount  55s (x13 over 11m)  kubelet  MountVolume.SetUp failed for volume "data" : mount failed: exit status 32
  Mounting command: mount
  Mounting arguments: -t nfs 10.0.10.15:/mnt/main/resume /var/lib/kubelet/pods/.../data
  Output: mount.nfs: Protocol not supported

Root Cause: The directory /mnt/main/resume didn't exist on the TrueNAS server.

Fix:

ssh [email protected] 'mkdir -p /mnt/main/resume && chmod 777 /mnt/main/resume'
# Then add to NFS exports and restart pod

Notes

• The "Protocol not supported" error is misleading - it often means the export path doesn't exist
• Always check the NFS server first before investigating protocol/firewall issues
• For TrueNAS, the NFS share must be updated via API/UI after creating new directories
• NFSv3 vs NFSv4 issues are rare in modern setups; missing paths are more common
• Check that the NFS client packages are installed on Kubernetes nodes if this is a new cluster

Variant: Non-Root Container UID Permission Denied

Problem

Container starts and mounts NFS successfully, but gets "Permission denied" when writing files. The pod appears healthy but operations fail silently.

Trigger Conditions

• Container logs show "Permission denied" or "client returned ERROR on write"
• Pod is Running (not stuck in ContainerCreating)
• NFS directory exists and is mounted, but owned by root (uid 0)
• Container image runs as a non-root user (e.g., curlimages/curl runs as uid 101)
• CronJobs or init containers that write to NFS fail with no obvious error

Common Non-Root Container UIDs

Solution

Fix permissions on the NFS server:

# Option 1: World-writable (simplest, suitable for non-sensitive data)
ssh [email protected] "chmod -R 777 /mnt/main/<service>/<subdir>"

# Option 2: Match container UID (more secure)
ssh [email protected] "chown -R <uid>:<gid> /mnt/main/<service>/<subdir>"

# Option 3: Use securityContext in pod spec to run as root
spec:
  securityContext:
    runAsUser: 0

Debugging

# Check what UID the container runs as
kubectl exec -n <namespace> <pod> -- id

# Test write access from inside container
kubectl exec -n <namespace> <pod> -- sh -c 'echo test > /path/to/nfs/testfile'

# Check NFS directory ownership on server
ssh [email protected] "ls -la /mnt/main/<service>/"

Variant: Stale NFS Mounts After Node Reboot (Ghost Running Pods)

Problem

After a node reboot (e.g., from kured rolling kernel updates), pods are rescheduled and show Running 1/1 status, but the application process is frozen/hung. The service is completely unresponsive despite appearing healthy to Kubernetes.

Trigger Conditions

• Node was recently rebooted (check kubectl get nodes for age, or kured logs)
• Pod shows Running 1/1 with 0 restarts (looks perfectly healthy)
• Service is unresponsive — Uptime Kuma or curl shows timeout/connection refused
• kubectl exec <pod> -- ss -tlnp shows zero listening sockets (the process started but is hung)
• Pod uses NFS volumes (inline nfs {} or PVC backed by NFS)
• Multiple pods across different namespaces all exhibit the same symptom simultaneously
• kubectl describe pod shows no warnings or errors — everything looks normal

Root Cause

When a node reboots, the NFS client mounts go stale. If the pod is rescheduled to the same or different node before NFS fully recovers, the application process starts but immediately hangs when it tries to access the NFS-mounted filesystem. The process is stuck in an uninterruptible I/O wait (D state) but Kubernetes sees the container as running because the PID exists and liveness probes (if any) may not exercise the NFS path.

Solution

Force-delete the affected pods to trigger a clean reschedule with fresh NFS mounts:

# Identify hung pods — Running but no listening sockets
kubectl exec -n <namespace> <pod> -- ss -tlnp 2>/dev/null
# If output is empty or shows no expected ports, the pod is hung

# Force-delete to skip graceful shutdown (hung process won't respond to SIGTERM)
kubectl delete pod -n <namespace> <pod> --force --grace-period=0

# The deployment controller creates a new pod with fresh NFS mounts
kubectl get pods -n <namespace> -w

For bulk remediation after a cluster-wide event:

# Find all pods with NFS volumes that might be hung
# Check each service's expected port — if ss -tlnp shows nothing, force-delete
for ns in calibre stirling-pdf send speedtest n8n paperless-ngx; do
  pod=$(kubectl get pod -n $ns -o name | head -1)
  sockets=$(kubectl exec -n $ns ${pod} -- ss -tlnp 2>/dev/null | wc -l)
  if [ "$sockets" -le 1 ]; then
    echo "HUNG: $ns/$pod (no listening sockets)"
    kubectl delete ${pod} -n $ns --force --grace-period=0
  fi
done

Verification

# New pod should have listening sockets
kubectl exec -n <namespace> <new-pod> -- ss -tlnp
# Should show the application's expected port (e.g., *:8080)

# Service should respond
kubectl exec -n <namespace> <new-pod> -- curl -sI http://localhost:<port>/
# Should return HTTP response

Key Diagnostic Insight

The critical signal is Running 1/1 but zero listening sockets. Normal healthy pods always have at least one listening socket for their application port. If ss -tlnp returns nothing, the process is hung on a stale NFS mount, not crashed — that's why Kubernetes thinks it's fine.

Prevention

• Add liveness probes that hit the application's HTTP endpoint (not just TCP connect):
  hclCopy

  liveness_probe {
    http_get {
      path = "/"
      port = 8080
    }
    initial_delay_seconds = 60
    period_seconds        = 30
    timeout_seconds       = 5
  }
  

• This ensures Kubernetes detects hung pods and restarts them automatically.

See Also

• TrueNAS NFS configuration documentation
• Kubernetes NFS volume documentation
• k8s-limitrange-oom-silent-kill (for OOM issues often confused with NFS hangs)