Proxmox VM Disk Expansion Pitfalls

Problem

Expanding disk storage on Proxmox-hosted Ubuntu 24.04 cloud-init VMs (used as Kubernetes nodes) fails at multiple points due to missing tools, cross-platform incompatibilities, and Kubernetes drain timeouts.

Context / Trigger Conditions

• Running disk expansion scripts from macOS against Proxmox + Ubuntu VMs
• Ubuntu 24.04 cloud-init images (the default k8s node template)
• Kubernetes nodes with many pods or stateful workloads
• Using scripts/extend_vm_storage.sh or similar automation

Issues and Solutions

1. growpart: command not found on Ubuntu 24.04

Symptom: After qm resize, SSH into VM, run growpart /dev/sda 1 — fails with "command not found". resize2fs then reports "Nothing to do!" because the partition table hasn't been updated.

Root cause: Ubuntu 24.04 cloud-init images don't include cloud-guest-utils by default. The growpart tool (which updates the partition table to use new disk space) is in this package.

Fix:

sudo apt-get update -qq && sudo apt-get install -y -qq cloud-guest-utils
sudo growpart /dev/sda 1
sudo resize2fs /dev/sda1

Prevention: Check for growpart before attempting partition expansion:

if ! command -v growpart &>/dev/null; then
    sudo apt-get update -qq && sudo apt-get install -y -qq cloud-guest-utils
fi

2. grep -P (PCRE) not available on macOS

Symptom: Script running on macOS fails with grep: invalid option -- P.

Root cause: macOS ships BSD grep, which doesn't support -P (Perl-compatible regex). GNU grep (from Homebrew) does, but scripts shouldn't assume it's installed.

Fix: Replace grep -oP 'pattern\Kcapture' with portable sed:

# BAD (GNU grep only):
CURRENT_SIZE=$(echo "$LINE" | grep -oP 'size=\K[0-9]+G')

# GOOD (portable):
CURRENT_SIZE=$(echo "$LINE" | sed -n 's/.*size=\([0-9]*G\).*/\1/p')

General rule: In scripts that run on macOS, avoid grep -P, sed -i '' vs sed -i differences, and date flag differences. Use sed with basic regex or bash built-in [[ =~ ]] for pattern matching.

3. kubectl drain timeout with stuck pods

Symptom: kubectl drain --timeout=120s fails with "context deadline exceeded" for multiple pods. Pods are evicted but don't terminate in time.

Root cause: Some pods (stateful services like ClickHouse, Paperless-ngx, OnlyOffice) need more time to shut down gracefully. 120s isn't enough when many pods are draining simultaneously.

Fix: Use --force flag and a longer timeout, or retry:

# First attempt with standard timeout
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=120s

# If it fails, force with longer timeout (pods already evicting)
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=300s --force

Note: After a failed drain, the node is already cordoned. A second drain attempt only needs to wait for already-evicting pods to finish.

4. Recovery from partial failure

If the script fails mid-way (after drain but before uncordon):

# Check VM status
ssh [email protected] "qm status <vmid>"

# Start VM if stopped
ssh [email protected] "qm start <vmid>"

# Uncordon node
kubectl --kubeconfig $(pwd)/config uncordon <node-name>

Verification

After successful expansion:

# On the VM
df -h /
# Should show new size (128G disk → ~126G usable for ext4)

# On the cluster
kubectl get node <name>
# Should show Ready status

Notes

• The k8s node VMs use direct partition layout (/dev/sda1), not LVM, despite the script handling both paths
• growpart returns exit code 1 for "NOCHANGE" (partition already at max) — this is not an error
• Proxmox qm resize uses scsi0 as the disk identifier for these VMs
• SSH host keys may change if VMs are recreated or network changes — use -o StrictHostKeyChecking=no in automated scripts

See also: extend-vm-storage.md (the operational skill for running the script)