【Troubleshooting】Analysis of kube-apiserver log report 'has no resources'

Posted by Hao Liang's Blog on Sunday, September 6, 2020

1. Problem description

In the production environment, the api-server service is restarted one after another. Check the monitoring and find that the CPU, memory and network traffic of the master node where the APIServer is located are jittering. In the APIServer log, you can see that there is a warning log that it has no resources in the log:

{"log":"W0814 03:04:44.058851       1 genericapiserver.go:342] Skipping API image.openshift.io/1.0 because it has no resources.\n","stream":"stderr","time":"2020-08-14T03:04:44.059033849Z"}

{"log":"W0814 03:04:44.058851       1 genericapiserver.go:342] Skipping API image.openshift.io/1.0 because it has no resources.\n","stream":"stderr","time":"2020-08-14T03:04:44.059033849Z"}

{"log":"W0814 03:04:49.422600       1 genericapiserver.go:342] Skipping API pre012 because it has no resources.\n","stream":"stderr","time":"2020-08-14T03:04:49.422708175Z"}

2. Code breakdown

Check the source code and the warning log occurs in the installAPIResources method in the staging/src/k8s.io/apiserver/pkg/server/genericapiserver.go file:

func (s *GenericAPIServer) installAPIResources(apiPrefix string, apiGroupInfo *APIGroupInfo) error {
	for _, groupVersion := range apiGroupInfo.PrioritizedVersions {
		if len(apiGroupInfo.VersionedResourcesStorageMap[groupVersion.Version]) == 0 {
			glog.Warningf("Skipping API %v because it has no resources.", groupVersion)
			continue
		}
		...
	}
	return nil
}

The VersionedResourcesStorageMap field of apiGroupInfo is used to store the mapping relationship between resource version, resource and resource storage object. Its expression is map[string]map[string]rest.Storage, such as Pod resource The mapping relationship with resource objects is: v1/pods/PodStorage (resource version/resource/resource storage object). So what is the resource storage object?

type PodStorage struct {
	Pod         *REST
	Binding     *BindingREST
	Eviction    *EvictionREST
	Status      *StatusREST
	Log         *podrest.LogREST
	Proxy       *podrest.ProxyREST
	Exec        *podrest.ExecREST
	Attach      *podrest.AttachREST
	PortForward *podrest.PortForwardREST
}

Taking PodStorage as an example, the resource storage object is actually a data structure that describes the resource object and is used to save the resource object data retrieved from etcd. The logic of this code is: traverse PrioritizedVersions (list of all resource versions supported by Kubernetes), take out each groupVersion (version list), and find the corresponding resource storage object in VersionedResourcesStorageMap. If not found (this is not enabled) resource version), Skipping API … because it has no resources. will be reported.

  1. List all resources supported by Kubernetes ==》PrioritizedVersions
  2. Instantiate APIGroupInfo ==》VersionedResourcesStorageMap
  3. Create API mapping for resource storage objects in GenericAPIServer: traverse PrioritizedVersions and determine whether there is a corresponding resource storage object in the PrioritizedVersions resource list in VersionedResourcesStorageMap (if not created, it has no resources will be reported)

Therefore, the warning log in production means: There is a Group version of the resource image.openshift.io/1.0 in the cluster, but the corresponding resource storage object is not found (because the resource version is not enabled, which will be discussed later) Detailed description), so skip creating an API for this resource storage object.

This alarm is only used to remind that a certain resource version is not enabled in the cluster, which is a normal phenomenon.

3. APIServer startup process

The warning log mentioned above occurs during the startup period of APIServer, so the startup process of APIServer is organized here.

  1. Resource registration
  2. Create APIServer general configuration
  3. Create APIExtentionsServer
  4. Create KubeAPIServer
  5. Create AggregatorServer
  6. Create GenericAPIServer
  7. Start the HTTP service
  8. Start HTTPS service

APIServer is essentially a web service that provides RESTful API. A series of work done before starting the HTTP/HTTPS service is to create routing and Etcd binding relationships for resource objects in the Kubernetes cluster. For example: if we want to access Pod information (internal resource objects) under a certain namespace through APIServer, we need to access the APIServer’s /api/v1/namespaces/{namespace}/pods interface, and the routing mapping of this interface is when the APIServer is started. Generated in the step of creating KubeAPIServer.

Corresponding source code (take batch resource as an example):

1. Resource registration

When registering resources, Kubernetes puts all supported resource version types into Scheme objects batch supports three versions: v1, v1beta1 and v2alpha1

// pkg/apis/batch/install/install.go
func init() {
	Install(legacyscheme.Scheme)
}

// Install registers the API group and adds types to a scheme
func Install(scheme *runtime.Scheme) {
	utilruntime.Must(batch.AddToScheme(scheme))
	utilruntime.Must(v1.AddToScheme(scheme))
	utilruntime.Must(v1beta1.AddToScheme(scheme))
	utilruntime.Must(v2alpha1.AddToScheme(scheme))
	utilruntime.Must(scheme.SetVersionPriority(v1.SchemeGroupVersion, v1beta1.SchemeGroupVersion, v2alpha1.SchemeGroupVersion))
}

2. Create KubeAPIServer

// pkg/master/master.go
func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*Master, error) {
	...
	// 1. Create GenericAPIServer
	s, err := c.GenericConfig.New("kube-apiserver", delegationTarget)
	...
	// 2. Initialize Mater
	m := &Master{
		GenericAPIServer: s,
	}
	...
	// 3. InstallLegacyAPI registration/api resource
	m.InstallLegacyAPI(&c, c.GenericConfig.RESTOptionsGetter, legacyRESTStorageProvider)
	}
	...
	// 4. InstallAPIs registration/apis resource
	m.InstallAPIs(c.ExtraConfig.APIResourceConfigSource, c.GenericConfig.RESTOptionsGetter, restStorageProviders...)
	...
	return m, nil
}

In step 2, Mater is initialized, some resource versions are enabled by default, and some resource versions are disabled by default. The disabled version list is the key to the log report it has no resources! !

// pkg/master/master.go
func DefaultAPIResourceConfigSource() *serverstorage.ResourceConfig {
	ret := serverstorage.NewResourceConfig()
	// Some resource versions are enabled by default here
	ret.EnableVersions(
		admissionregistrationv1beta1.SchemeGroupVersion,
		apiv1.SchemeGroupVersion,
		appsv1beta1.SchemeGroupVersion,
		appsv1beta2.SchemeGroupVersion,
		appsv1.SchemeGroupVersion,
		authenticationv1.SchemeGroupVersion,
		...
	)
	// Some resource versions are disabled by default here
	ret.DisableVersions(
		admissionregistrationv1alpha1.SchemeGroupVersion,
		batchapiv2alpha1.SchemeGroupVersion,
		rbacv1alpha1.SchemeGroupVersion,
		schedulingv1alpha1.SchemeGroupVersion,
		settingsv1alpha1.SchemeGroupVersion,
		storageapiv1alpha1.SchemeGroupVersion,
	)

	return ret
}

In step 4, the InstallAPIs method is called, and two things are done in the method

  1. Instantiate the APIGroupInfo object, which is to establish the mapping relationship between resource version, resource and resource storage object for the VersionedResourcesStorageMap we mentioned earlier. Since the v2alpha1 version of batch is disabled by default, its mapping relationship will not be established. Finally, when apiserver starts, it will report that the resources of batch/v2alpha1 cannot be found
// pkg/registry/batch/rest/storage_batch.go
func (p RESTStorageProvider) NewRESTStorage(apiResourceConfigSource serverstorage.APIResourceConfigSource, restOptionsGetter generic.RESTOptionsGetter) (genericapiserver.APIGroupInfo, bool) {
	// Instantiate an APIGroupInfo object
	apiGroupInfo := genericapiserver.NewDefaultAPIGroupInfo(batch.GroupName, legacyscheme.Scheme, legacyscheme.ParameterCodec, legacyscheme.Codecs)
    // Here it is judged whether each version of the resource is enabled.
	if apiResourceConfigSource.VersionEnabled(batchapiv1.SchemeGroupVersion) {
		apiGroupInfo.VersionedResourcesStorageMap[batchapiv1.SchemeGroupVersion.Version] = p.v1Storage(apiResourceConfigSource, restOptionsGetter)
	}
	if apiResourceConfigSource.VersionEnabled(batchapiv1beta1.SchemeGroupVersion) {
		apiGroupInfo.VersionedResourcesStorageMap[batchapiv1beta1.SchemeGroupVersion.Version] = p.v1beta1Storage(apiResourceConfigSource, restOptionsGetter)
	}
	// When initializing Mater in step 2, the v2alpha1 version of batch is disabled. The resource storage object of batchapiv2alpha1 will not be put into the Map. Therefore, apiserver will report a warning that the resource cannot be found when it starts.
	if apiResourceConfigSource.VersionEnabled(batchapiv2alpha1.SchemeGroupVersion) {
		apiGroupInfo.VersionedResourcesStorageMap[batchapiv2alpha1.SchemeGroupVersion.Version] = p.v2alpha1Storage(apiResourceConfigSource, restOptionsGetter)
	}

	return apiGroupInfo, true
}
  1. Register the VersionedResourcesStorageMap (that is, the mapping relationship between resource version, resource and resource storage object) in the APIGroupInfo object to the KubeAPIServer Handlers method through the InstallAPIGroup method.
func (s *GenericAPIServer) InstallAPIGroup(apiPrefix string, apiGroupInfo *APIGroupInfo) error {
	...
	// The mapping relationship between resource version/resource/resource storage object and HTTP request path is created for Kubernetes resource objects.
	if err := s.installAPIResources(APIGroupPrefix, apiGroupInfo); err != nil {
		return err
	}
	...
	return nil

The execution logic of the installAPIResources method above is the logic mentioned at the beginning:

Traverse PrioritizedVersions (list of all resource versions supported by Kubernetes) and take out each groupVersion (list of versions), in VersionedResourcesStorageMap Find the corresponding resource storage object, if not found, report Skipping API… because it has no resources.

Checking the startup log of apisever, I found that the log of batch/v2alpha1 has no resources was indeed printed.

W0816 05:14:24.570339       1 genericapiserver.go:319] Skipping API batch/v2alpha1 because it has no resources.
W0816 05:14:25.277714       1 genericapiserver.go:319] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0816 05:14:25.288664       1 genericapiserver.go:319] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0816 05:14:25.377954       1 genericapiserver.go:319] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0816 05:14:26.780652       1 genericapiserver.go:319] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no

So why are some resource versions disabled by default? And all alpha version resources are disabled by default. The resource versions of Kubernetes are divided into three types: Alpha, Beta, and Stable. The iteration order between them is Alpha-》Beta-》Stable. Alpha is the first-stage version, generally used for internal testing. Beta is the second stage version, which fixes most of the imperfections, but there may still be bugs. Stable is a stable version. Therefore, the Alpha version is an internal beta version and has many unstable factors. The official may give up supporting this version at any time, so it is disabled by default.

CATALOG
    FEATURED TAGS
    diagnose docker informer kube-apiserver kube-scheduler kubelet kubernetes scheduler
    FRIENDS