Kubernetes Software Conformance Tests — Part I


In the recent times, there is growing adoption for microservices based architectures. So there comes a need for orchestration tools for scheduling, scaling, monitoring, managing your containerized workloads and services

Kubernetes(k8s) is one of the leading open source container-orchestration platform.With its wide-scale adoption, there are many enterprise k8s platform offerings in market. These vendors enhance the open source k8s version with added security, ease of management, offer enterprise support etc . These offerings can be distributors/operators/hosted environments. For ease, I would be referring them as vendors in this page.

To maintain consistency among these vendor specific kubernetes platforms, kubernetes community, with support of CNCF came up with a Kubernetes Software Conformance Certification program. Any k8s vendor which meets kubernetes specified standards, are said to have Certified k8s offerings.

Why is it needed?

Software conformance ensures that every vendor’s version of Kubernetes supports the required APIs, as do open source community versions. Users expect consistency when interacting with any installation of Kubernetes. Conformance enables interoperability from one k8s installation to the next. It gives organisations the flexibility to choose between vendors.

Is it a one time thing?
Certification is for a specific version of kubernetes. To remain certified, vendors need to provide the test results of latest version of Kubernetes yearly or more frequently, so that latest features are supported.

Can I run the tests locally?
Any end user can confirm that their distribution or platform remains conformant by running the identical open source conformance application (Sonobuoy) that was used to certify.

What are the conformance tests?

The k8s Conformance test suite is a subset of e2e tests (having [Conformance] tag)to define the core set of interoperable features that all conformant Kubernetes clusters must support. The tests verify that the expected behaviour works as a user might encounter it in the wild.

How does kubernetes certify vendors ?

CNCF runs the Certified Kubernetes Conformance Program. Any vendor can follow one of approaches to run conformance tests on their cluster

  • Directly run k8s e2e suite using ginkgo/kubetest

Vendors need to submit conformance testing results for review.

What is Sonobuoy ?

A standard diagnostic tool for running the standard set of conformance tests to certify vendors. It makes it easier to understand the state of a Kubernetes cluster by running a set of plugins (including Kubernetes conformance tests) in an accessible and non-destructive manner. It is customisable, extendable, and cluster-agnostic way to generate clear, informative reports about your cluster

How does Sonobuoy release cycle work ?
Sonobuoy is regularly built and kept up to date to execute against all currently supported versions of kubernetes. Sonobuoy releases will be independent of Kubernetes release, while ensuring that new releases continue to work functionally across different versions of Kubernetes.

How often should these tests be run?
Certification is versioned, and with each new version of Kubernetes, as features are added and the architecture changes, the Certification requirements will change as appropriate.k8s-conformance repo has folders for each release. Vendors raise Pull Requests for their conformance test results correspondingly.

How to run tests locally ?
Conformance test suite needs a cluster with at least 2 nodes. Sonobuoy creates a few resources in order to run and expects to run within its own namespace. Follow steps in sonobuoy README or conformance repo

How long does a sonobuoy run take ?
Default timeout is 3 hrs (which can be passed as an argument while running sonobuoy run command). Successful runs may take 60 mins. `sonobuoy status` command can be used to verify same.

For further read on scope and limitations of conformance tests, continue reading here

Happy conformance to your product !

Passionate in giving back <something> to the society ❤

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store