Kubernetes Software Conformance Tests — Part II

You can refer link for introduction

Scope of conformance tests

There are a total of 334 conformance tests as of 16 Mar 2021 (Ref Link)

  • Conformance tests currently test only GA, non-optional features or APIs. No alpha or beta endpoints, no feature flags required, no deprecated features. Eg: policy enforcement, multiple disk mounts, GPUs features are not supported.
  • It does not require direct access to kubelet’s API to pass (nor does it require indirect access via the API server node proxy endpoint); it MAY use the kubelet API for debugging purposes upon failure
  • It is non-privileged (e.g., does not require root on nodes, access to raw network interfaces, or cluster admin permissions)
  • It does not rely on any binaries that would not be required for the linux kernel or kubelet to run (e.g., can’t rely on git)
  • It does not depend on outputs that change based on OS (nslookup, ping, chmod, ls)
  • It passes against the appropriate versions of kubernetes as spelled out in the conformance test version skew policy

Below are a few for reference. Complete details can be found here

Custom Resource Definitions (CRD’s)

  • CRD create, delete, listing
  • CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD with validation schema
  • Create and watch a CRD. create, modify and delete events must be observed
  • Simple CRD getting/updating/patching custom resource definition status sub-resource works ; all mutating sub-resource operations MUST be visible to subsequent reads.
  • Register a CRD with multiple versions; OpenAPI definitions MUST be published for CRD’s. Rename one of the versions of the CRD via a patch; OpenAPI definitions MUST update to reflect the rename.
  • CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD with validation schema. Register a CRD with a validating schema consisting of objects, arrays and primitives. Attempt to create and apply a change a custom resource using valid properties, via kubectl; client-side validation MUST pass. Attempt both operations with unknown properties and without required properties; client-side validation MUST reject the operations. Attempt kubectl explain; the output MUST explain the custom resource properties. Attempt kubectl explain on custom resource properties; the output MUST explain the nested custom resource properties. (v1.16)
  • Register multiple CRD’s spanning different groups and versions; OpenAPI definitions MUST be published for these. (v1.16)


  • Create, patch, delete ConfigMap works. Fetching the ConfigMap MUST reflect changes, fetch all the ConfigMaps via a Label selector.
  • Ensures that multiple watchers are able to receive all add, update, and delete notifications on configmaps that match a label selector and do not receive notifications for configmaps which do not match that label selector.
  • When configmap label is changed, ensure that a watched object stops meeting the requirements of a watch’s selector, the watch will observe a delete, and will not observe notifications for that object until it meets the selector’s requirements again.
  • Ensure that concurrent watches are consistent with each other by initiating an additional watch for events received from the first watch, initiated at the resource version of the event, and checking that all resource versions of all events match. Events are produced from writes on a background goroutine.

Ingress API

  • Support creating Ingress API operations create, update, list, watch, delete ingresses. Updating status LoadBalancer
  • e2e test suite (which will be run for every k8s release) contains ingress controller platform agnostic tests that must be satisfied by all controllers — creating basic HTTP ingress, HTTPS ingress, updating hostname, URL path are tested using GCE, nginx (k8s community nginx IC image)
  • ClusterRoleBinding included
  • Scaling backends and verifying network endpoint groups
  • Rolling update backend pods shouldn’t cause network disruption

IngressClass API

  • IngressClass API should support creating IngressClass API operations
  • As part of k8s 1.18, a new IngressClass resource that can specify how Ingresses should be implemented by controllers was released.
  • Each IngressClass specifies which controller should implement Ingresses of the class and can reference a custom resource with additional parameters.
  • `kubernetes.io/ingress.class` annotation on the Ingress is formally deprecated from 1.18. Link
  • Create, get, list, watch, delete IngressClass API operations are part of conformance tests


  • Service endpoints with multiple ports, nodeport mode
  • Services should have session affinity work for service with type clusterIP, nodeport
  • Services should test the lifecycle of an Endpoint
  • Find Kubernetes Service in default Namespace (1.18)


  • DNS should provide /etc/hosts entries for the cluster
  • Creating a service with externalName and checking pod to resolve the address for this service via CNAME. When externalName of this service is changed, Pod MUST resolve to new DNS entry for the service. Change the service type from externalName to ClusterIP, Pod MUST resolve DNS to the service by serving A records.
  • Creation and deletion of pods are reflecting endpoints addresses
  • DNS should support configurable pod DNS nameservers. Create a Pod with DNSPolicy as None and custom DNS configuration, specifying nameservers and search path entries. Pod creation MUST be successful and provided DNS configuration MUST be configured in the Pod.

Replication controller

  • Fail when an attempt to create a Replication Controller with pods exceeding the namespace quota
  • When the labels on one of the Pods change to no longer match the RC’s label selector, the RC MUST release the Pod and update the Pod’s owner references.
  • Replicate set to adopt matching pods and release non matching pods when labels in pod definition are edited

Service Account

  • Creates a ServiceAccount with a static Label MUST be added as shown in watch event. Patching the ServiceAccount MUST return its new property. Listing the ServiceAccounts MUST return the test ServiceAccount with its patched values. ServiceAccount will be deleted and MUST find a deleted watch event.
  • ServiceAccounts should mount projected service account token
  • Service Account keys are auto mounted into the Container and dependency on AutoMountServiceToken


  • Endpoint lifecycle — Create an endpoint, the endpoint MUST exist. The endpoint is updated with a new label, a check after the update MUST find the changes. The endpoint is then patched with a new IPv4 address and port, a check after the patch MUST the changes. The endpoint is deleted by its label, a watch listens for the deleted watch event.
  • Create a service with two ports but no Pods are added to the service yet. The service MUST run and show empty set of endpoints. Add a Pod to the first port, service MUST list one endpoint for the Pod on that port. Add another Pod to the second port, service MUST list both the endpoints. Delete the first Pod and the service MUST list only the endpoint to the second Pod. Delete the second Pod and the service must now have empty set of endpoints.


  • Daemon set, Stateful set, pod lifecycle, Secrets mounting, Garbage collection
  • Container Runtime, Restart Policy, Pod Phases
  • Namespace removal removes associated objects
  • Updating Node labels, NodeSelector, Scheduling
  • Deployments — delete, recreate, RollingUpdate, revisionHistoryLimit, Rollover, proportional scaling
  • Events API should ensure that an event can be fetched, patched, deleted, and listed. Events should be sent by kubelets and the scheduler about pods scheduling and running

Ingress controller conformance tests

Ingress controller conformance tests are maintained by K8s community. The conformance test suite will both ensure consistency across multiple ingress controller implementations. It accepts values for INGRESS_CONTROLLER, INGRESS_CLASS, CONTROLLER_VERSION etc. These tests can be run in 8s cluster with a running pod of the ingress controller to use. Tests include path rules, ingress class, host rules, default backends, load balancing features.
Ref here for more details

Limitations of conformance tests

Endpoints that will soon be deprecated or vendor specific features or optional features or which use kubelet API will not be tested under conformance tests.

Ref for more reading

  • createCoreV1NamespacedServiceAccountToken > optional feature
  • createCoreV1Node, deleteCoreV1Node > uses kubelet API

Below are not included as part of conformance tests

  • NetworkPolicy
  • PodSecurityPolicy

There are issues reported for failing conformance tests depending on CNI used.
Eg: An issue reported if flannel is used as CNI, which isn’t seen when changed to Calico.
The kops project is running a more complicated test matrix to check if a distro / cni / k8s version setup is listed as green here.

Happy conformance to your product !

Passionate in giving back <something> to the society ❤

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store