All Articles

Kubernetes Version 1.20.0 breaks nfs provisioner

Kubernetes version 1.20.0 breaks the nfs-subdir-external-provisioner.

The PersistentVolumeClaim stays in status “Pending”:

$ k get pvc
NAME         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS          AGE
test-claim   Pending                                      managed-nfs-storage   10h

The pod logs of client provisioner contains:

unexpected error getting claim reference: selfLink was empty, can’t make reference

$ kubectl logs nfs-client-provisioner
E0106 06:18:07.328905       1 controller.go:1004] provision "default/test-claim" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
unexpected error getting claim reference: selfLink was empty, can't make reference

The deprecation of selfLinks in Kubernetes 1.20.0 is responsible for this.

As noted by nealman13 in his github comment setting the feature gate RemoveSelfLink to false is a workaround for this problem.

But Rancher K3S doesn’t come with a kube-apiserver.yaml file. But feature gates can be passed as commandline arguments directly to the service.

For a fresh install of K3S the following commandline can be used:

$ curl -sfL https://get.k3s.io | sh -s - --kube-apiserver-arg "feature-gates=RemoveSelfLink=false"

An existing installation can be adjusted by changing the service directly:

$ nano /etc/systemd/system/k3s.service
... 
ExecStart=/usr/local/bin/k3s \
    server \
        '--kube-apiserver-arg' \
        'feature-gates=RemoveSelfLink=false' \
...