Goal

• Rename any node name in Kubernetes cluster.
• No need to reset the whole kubelet environment like most of the approaches.
• No need to drain any Pods running on the Node.

Bootstrap Process of kubelet

1. Doc refer to: https://kubernetes.io/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/
2. Chinese version introduction refer to: https://cloud.tencent.com/developer/article/1656007

• The kubelet process starts.
• Try to find kubeconfig file specified by arg --kubeconfig=xxx, if not found, try to find bootstrap-kubeconfig file specified by arg --bootstrap-kubeconfig=xxx instead.
• Read the apiserver address and bootstrap token from bootstrap-kubeconfig file. For the format of bootstrap, refer to https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/
• Use bootstrap token as credential to access apiserver.
• This token has permission to create and obtain Certificate Signing Requests (CSR) - the naming of the bootstrap token has a specific format and can be recognized by apiserver. The username is system:bootstrap: and belongs to the system:bootstrappers user group. This user The group needs to be bound to the clusterrole of system:node-bootstrapper so that it can have permission to create CSR.
• kubelet creates a CSR for itself, and the issuer (csr.Spec.SignerName) is kubernetes.io/kube-apiserver-client-kubelet
• CSR is automatically or manually approved:
  • Controller-manager automatically approves and requires the system:bootstrappers user group to bind the system:certificates.k8s.io:certificatesigningrequests:nodeclient clusterrole. The CSRApprovingController of controller-manager will use the SubjectAccessReview API to verify whether the username and group in the csr have corresponding permissions, and also check whether the issuer is kubernetes.io/kube-apiserver-client-kubelet
  • Manually approve through kubectl etc.
• The kubelet certificate is created by controller-manager after approved
• controller-manager updates the certificate into the status field of csr
• kubelet gets the client side certificate from apiserver
• kubelet generates the corresponding kubeconfig based on the retrieved key and client side certificate( Generated under /var/lib/kubelet/pki/).
• The kubelet starts working normally using the generated kubeconfig

Note:

• If automatic certificate renewal is configured, kubelet will use the old kubeconfig to renew the old certificate when the certificate is about to expire.
• The renewed certificate is automatically or manually approved and signed - automatic approve requires system:nodes user group binding system:certificates.k8s.io:certificatesigningrequests:selfnodeclient’s clusterrole (system:nodes is the group of the certificate that kubelet applied for before, that is, the certificate The organization O is system:nodes)
• Client side certificate includes node name as username, if node name has been changed, we need to regenerate kubelet’s client side certificate by CSR.

Code Breakdown

The corresponding code are listed as follows:

// cmd/kubelet/app/server.go
// buildKubeletClientConfig constructs the appropriate client config for the kubelet depending on whether
// bootstrapping is enabled or client certificate rotation is enabled.
func buildKubeletClientConfig(s *options.KubeletServer, nodeName types.NodeName) (*restclient.Config, func(), error) {
	if s.RotateCertificates && utilfeature.DefaultFeatureGate.Enabled(features.RotateKubeletClientCertificate) {
		
		klog.Infof("Client rotation is on, will bootstrap in background")
		certConfig, clientConfig, err := bootstrap.LoadClientConfig(s.KubeConfig, s.BootstrapKubeconfig, s.CertDirectory)
		if err != nil {
			return nil, nil, err
		}

		// use the correct content type for cert rotation, but don't set QPS
		setContentTypeForClient(certConfig, s.ContentType)

		kubeClientConfigOverrides(s, clientConfig)

		clientCertificateManager, err := buildClientCertificateManager(certConfig, clientConfig, s.CertDirectory, nodeName)
		if err != nil {
			return nil, nil, err
		}

		// the rotating transport will use the cert from the cert manager instead of these files
		transportConfig := restclient.AnonymousClientConfig(clientConfig)

		// we set exitAfter to five minutes because we use this client configuration to request new certs - if we are unable
		// to request new certs, we will be unable to continue normal operation. Exiting the process allows a wrapper
		// or the bootstrapping credentials to potentially lay down new initial config.
		closeAllConns, err := kubeletcertificate.UpdateTransport(wait.NeverStop, transportConfig, clientCertificateManager, 5*time.Minute)
		if err != nil {
			return nil, nil, err
		}

		klog.V(2).Info("Starting client certificate rotation.")
		clientCertificateManager.Start()

		return transportConfig, closeAllConns, nil
	}

	if len(s.BootstrapKubeconfig) > 0 {
		if err := bootstrap.LoadClientCert(s.KubeConfig, s.BootstrapKubeconfig, s.CertDirectory, nodeName); err != nil {
			return nil, nil, err
		}
	}

	clientConfig, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
		&clientcmd.ClientConfigLoadingRules{ExplicitPath: s.KubeConfig},
		&clientcmd.ConfigOverrides{},
	).ClientConfig()
	if err != nil {
		return nil, nil, fmt.Errorf("invalid kubeconfig: %v", err)
	}

	kubeClientConfigOverrides(s, clientConfig)
	closeAllConns, err := updateDialer(clientConfig)
	if err != nil {
		return nil, nil, err
	}
	return clientConfig, closeAllConns, nil
}

• The if condition s.RotateCertificates && utilfeature.DefaultFeatureGate.Enabled(features.RotateKubeletClientCertificate) determines whether it should send a new CSR request to regenerate the client side certificate.
• So if kubelet’s --rotate-certificates option is enabled, it will dive into the if code snippet, generate kubeconfig with existing client side certificate instead of sending a new CSR request to regenerate the client side certificate.
• After changing node name by updating kubelet’s --hostname-override= to a new node name, the client side certificate also needed to be updated.
• To regenerate kubelet’s client side certificate, we should temporarily disable --rotate-certificates in kubelet.

Summary

To rename a node name in Kubernetes, you can take the following steps:

• Make sure bootstrap config file exists and validated.
• Update --hostname-override= to specify new node name.
• Disable --rotate-certificates in kubelet.
• Remove kubelet client cert in /var/lib/kubelet/pki.
• Restart kubelet to regenerate the client cert and kubelet’s kubeconfig.
• Execute kubectl get nodes to check if new node has already been registered into cluster.
• Immigrate the old node’s label and annotation to the new node.
• Delete the old node.

• ← Previous Post
• Next Post →