Welcome to Connaisseur⚓︎
A Kubernetes admission controller to integrate container image signature verification and trust pinning into a cluster.
What is Connaisseur?⚓︎
Connaisseur ensures integrity and provenance of container images in a Kubernetes cluster. To do so, it intercepts resource creation or update requests sent to the Kubernetes cluster, identifies all container images and verifies their signatures against pre-configured public keys. Based on the result, it either accepts or denies those requests.
Connaisseur is developed under three core values: Security, Usability, Compatibility. It is built to be extendable and currently aims to support the following signing solutions:
It provides several additional features:
- Metrics: get prometheus metrics at
- Alerting: send alerts based on verification result
- Detection Mode: warn but do not block invalid images
- Namespaced Validation: restrict validation to dedicated namespaces
- Automatic Child Approval: configure approval of Kubernetes child resources
Feel free to reach out via GitHub Discussions!
Getting started to verify image signatures is only a matter of minutes:
Only try this out on a test cluster as deployments with unsigned images will be blocked.
Connaisseur comes pre-configured with public keys for its own repository and Docker's official images (official images can be found here).
It can be fully configured via
For a quick start, clone the Connaisseur repository:
git clone https://github.com/sse-secure-systems/connaisseur.git
Next, install Connaisseur via Helm:
helm install connaisseur helm --atomic --create-namespace --namespace connaisseur
Once installation has finished, you are good to go.
Successful verification can be tested via official Docker images like
kubectl run hello-world --image=docker.io/hello-world
Or our signed
kubectl run demo --image=docker.io/securesystemsengineering/testimage:signed
Both will return
pod/<name> created. However, when trying to deploy an unsigned image:
kubectl run demo --image=docker.io/securesystemsengineering/testimage:unsigned
Connaisseur denies the request and returns an error
(...) Unable to find signed digest (...). Since the images above are signed using Docker Content Trust, you can inspect the trust data using
docker trust inspect --pretty <image-name>.
To uninstall Connaisseur use:
helm uninstall connaisseur --namespace connaisseur
Congrats you just validated the first images in your cluster! To get started configuring and verifying your own images and signatures, please follow our setup guide.
How does it work?⚓︎
Integrity and provenance of container images deployed to a Kubernetes cluster can be ensured via digital signatures. On a very basic level, this requires two steps:
- Signing container images after building
- Verifying the image signatures before deployment
Connaisseur aims to solve step two. This is achieved by implementing several validators, i.e. configurable signature verification modules for different signing solutions (e.g. Notary V1). While the detailed security considerations mainly depend on the applied solution, Connaisseur in general verifies the signature over the container image content against a trust anchor or trust root (e.g. public key) and thus let's you ensure that images have not been tampered with (integrity) and come from a valid source (provenance).
But what is actually verified?
Container images can be referenced in two different ways based on their registry, repository, image name (
<registry>/<repository>/<image name>) followed by either tag or digest:
- tag: docker.io/library/nginx:1.20.1
- digest: docker.io/library/nginx@sha256:af9c...69ce
While the tag is a mutable, human readable description, the digest is an immutable, inherent property of the image, namely the SHA256 hash of its content. This also means that a tag can correspond to varying digests whereas digests are unique for each image. The container runtime (e.g. containerd) compares the image content with the received digest before spinning up the container. As a result, Connaisseur just needs to make sure that only trusted digests (signed by a trusted entity) are passed to the container runtime. Depending on how an image for deployment is referenced, it will either attempt to translate the tag to a trusted digest or validate whether the digest is trusted. How the digest is signed in detail, where the signature is stored, what it is verfied against and how different image distribution and updating attacks are mitigated depends on the signature solutions.
Mutating admission controller⚓︎
How to validate images before deployment to a cluster? The Kubernetes API is the fundamental fabric behind the control plane. It allows operators and cluster components to communicate with each other and, for example, query, create, modify or delete Kubernetes resources. Each request passes through several phases such as authentication and authorization before it is persisted to etcd. Among those phases are two steps of admission control: mutating and validating admission. In those phases the API sends admission requests to configured webhooks (admission controllers) and receives admission responses (admit, deny, or modify). Connaisseur uses a mutating admission webhook, as requests are not only admitted or denied based on the validation result but might also require modification of contained images referenced by tags to trusted digests. The webhook is configured to only forward resource creation or update requests to the Connaisseur service running inside the cluster, since only deployments of images to the cluster are relevant for signature verification. This allows Connaisseur to intercept requests before deployment and based on the validation:
- admit if all images are referenced by trusted digests
- modify if all images can be translated to trusted digests
- deny if at least one of the requested images does not have a trusted digest
Image policy and validators⚓︎
Now, how does Connaisseur process admission requests? A newly received request is first inspected for container image references that need to be validated (1). The resulting list of images referenced by tag or digest is passed to the image policy (2). The image policy matches the identified images to the configured validators and corresponding trust roots (e.g. public keys) to be used for verification. Image policy and validator configuration form the central logic behind Connaisseur and are described in detail und basics. Validation is the step where the actual signature verification takes place (3). For each image, the required trust data is retrieved from external sources such as Notary server, registry or sigstore transparency log and validated against the pre-configured trust root (e.g. public key). This forms the basis for deciding on the request (4). In case no trusted digest is found for any of the images (i.e. either no signed digest available or no signature matching the public key), the whole request is denied. Otherwise, Connaisseur translates all image references in the original request to trusted digests and admits it (5).
Supported signature solutions and configuration options are documented under validators.
Connaisseur supports Kubernets v1.16 and higher. It is expected to be compatible with most Kubernetes services and has been successfully tested with:
- K3s ✅
- kind ✅
- MicroK8s ✅ (enable DNS addon via
sudo microk8s enable dns)
- minikube ✅
- Amazon Elastic Kubernetes Service (EKS) ✅
- Azure Kubernetes Service (AKS) ✅
- Google Kubernetes Engine ✅
- SysEleven MetaKube ✅
All registry interactions use the OCI Distribution Specification that is based on the Docker Registry HTTP API V2 which is the standard for all common image registries. For using Notary (V1) as a signature solution, only some registries provide the required Notary server attached to the registry with e.g. shared authentication. Connaisseur has been tested with the following Notary (V1) supporting image registries:
In case you identify any incompatibilities, please create an issue
The latest stable version of Connaisseur is available on the
Releases follow semantic versioning standards to facilitate compatibility.
For each release, a signed container image tagged with the version is published in the Connaisseur Docker Hub repository.
Latest developments are available on the
develop branch, but should be considered unstable and no pre-built container image is provided.
Connaisseur is open source and open development. We try to make major changes transparent via Architecture Decision Records (ADRs) and announce developments via GitHub Discussions. Information on responsible disclosure of vulnerabilities and tracking of past findings is available in the Security Policy. Bug reports should be filed as GitHub issues to share status and potential fixes with other users.
We hope to get as many direct contributions and insights from the community as possible to steer further development. Please refer to our contributing guide, create an issue or reach out to us via GitHub Discussions
Wall of fame⚓︎
Thanks to all the fine people directly contributing commits/PRs to Connaisseur:
Big shout-out also to all who support the project via issues, discussions and feature requests
Several resources are available to learn more about Connaisseur and related topics:
- "Container Image Signatures in Kubernetes" - blog post (full introduction)
- "Integrity of Docker images" - talk at Berlin Crypto Meetup (The Update Framework, Notary, Docker Content Trust & Connaisseur [live demo])
- "Verifying Container Image Signatures from an OCI Registry in Kubernetes" - blog post (experimental support of sigstore/Cosign)
- "Verify Container Image Signatures in Kubernetes using Notary or Cosign or both" - blog post (Connaisseur v2.0 release)