Node disk is full or nearly full
Production Risk
Pod evictions and inability to pull new images cause service degradation.
The DiskPressure node condition is set to True when the kubelet detects that available disk space or inode count is below the configured eviction threshold. New pods are not scheduled on the node, and the kubelet may evict pods to free disk space. Container image layers, ephemeral logs, and emptyDir volumes are the most common disk consumers.
- 1Container logs growing unboundedly and filling the node's disk
- 2Excessive container image layers accumulated over time without garbage collection
- 3emptyDir volumes consuming large amounts of ephemeral storage
- 4Node's root partition is undersized for the workload
Node condition DiskPressure=True; pods may be evicted and no new pods scheduled.
kubectl describe node mynode | grep -A 5 "Conditions:" # DiskPressure True ... KubeletHasDiskPressure # On the node df -h / du -sh /var/log/containers/*
expected output
DiskPressure True ... KubeletHasDiskPressure
Fix 1
Clean up container images and unused layers
WHEN Image cache is consuming excessive disk
# On the node (or via privileged DaemonSet) crictl rmi --prune # or using containerd ctr -n k8s.io images remove $(ctr -n k8s.io images list -q)
Why this works
Removes dangling and unused container image layers to free disk space immediately.
Fix 2
Configure log rotation limits
WHEN Container logs are filling the disk
# In kubelet config or container runtime config # Set max log file size and rotation count --container-log-max-size=50Mi --container-log-max-files=5
Why this works
Limits the total disk space consumed by container log files.
Kubernetes Documentation
Content generated with AI assistance and reviewed for accuracy. Found an error? hello@errcodes.dev