microk8s, Harbor, and self-signed certificates

My home-lab environment has a 3 node microk8s cluster and I wanted to deploy Harbor to cache container images locally, run security scans against them, and because overkill is my home-lab’s modus operandi.

Even though Harbor is deployed on an internal VLAN I still protect communication with it using HTTPS. When I tried to pull an image from Harbor in a Kubernetes deployment I saw the following error:

Warning  Failed     20m (x4 over 21m)   kubelet            Error: ErrImagePull
Warning  Failed     20m (x4 over 21m)   kubelet            Failed to pull image "harbor.demo.willbar.com/dockerhub-replica/whoami": rpc error: code = Unknown desc = failed to pull and unpack image "harbor.demo.willbar.com/dockerhub-replica/whoami:latest": failed to resolve reference "harbor.demo.willbar.com/dockerhub-replica/whoami:latest": failed to do request: Head "https://harbor.demo.willbar.com/v2/dockerhub-replica/whoami/manifests/latest": x509: certificate signed by unknown authority

The relevant error is x509: certificate signed by unknown authority which comes from containerd (the container runtime). It’s telling us that it can’t connect securely to Harbor because it doesn’t trust the certificate authority that signed the certificate - which is to be expected since it’s a self-signed certificate. This is the same situation that causes your browser to show a security warning when you try to access a website that uses a self-signed certificate.

In your browser you can click a button to ignore the security warning and visit the website anyway, so what’s the equivalent for containerd? It turns out that containerd uses the operating system’s trusted certificate store, so if we add the certificate authority’s certificate to that then containerd should then trust certificates signed by that certificate authority.

I’m running microk8s on Debian so we can run the following commands as root to add the certificate to Debian’s trusted certificate store:

cp ca.crt /usr/local/share/ca-certificates
update-ca-certificates

Once we’ve done this we can restart microk8s and now our certificate will be trusted!

Normal   Pulling    5m41s               kubelet            Pulling image "harbor.demo.willbar.com/dockerhub-replica/whoami"
Normal   Pulled     5m41s               kubelet            Successfully pulled image "harbor.demo.willbar.com/dockerhub-replica/whoami" in 179.184938ms