Basics⚓︎
In the following, we aim to lay the foundation on Connaisseur's core concepts, how to configure and administer it.
Admission control, validators and image policy⚓︎
Connaisseur works as a mutating admission controller. It intercepts all CREATE and UPDATE resource requests for Pods, Deployments, ReplicationControllers, ReplicaSets, DaemonSets, StatefulSets, Jobs, and CronJobs and extracts all image references for validation.
Per default, Connaisseur uses automatic child approval by which the child of a Kubernetes resource is automatically admitted without re-verification of the signature in order to avoid duplicate validation and handle inconsistencies with the image policy. Essentially, this is done since an image that is deployed as part of an already deployed object (e.g. a Pod deployed as a child of a Deployment) has already been validated and potentially mutated during admission of the parent. More information and configuration options can be found in the feature documentation for automatic child approval.
Validation itself relies on two core concepts: image policy and validators. A validator is a set of configuration options required for validation like the type of signature, public key to use for verification, path to signature data, or authentication. The image policy defines a set of rules which maps different images to those validators. This is done via glob matching of the image name which for example allows to use different validators for different registries, repositories, images or even tags. This is specifically useful when using public or external images from other entities like Docker's official images or different keys in a more complex development team.
Note
Typically, the public key of a known entity is used to validate the signature over an image's content in order to ensure integrity and provenance. However, other ways to implement such trust pinning exist and as a consequence we refer to all types of trust anchors in a generalized form as trust roots.
Using Connaisseur⚓︎
Some general administration tasks like deployment or uninstallation when using Connaisseur are described in this section.
Requirements⚓︎
Using Connaisseur requires a Kubernetes cluster, Helm and, if installing from source, Git to be installed and set up.
Get the code/chart⚓︎
Download the Connaisseur resources required for installation either by cloning the source code via Git or directly add the chart repository via Helm.
The Connaisseur source code can be cloned directly from GitHub and includes the application and Helm charts in a single repository:
git clone https://github.com/sse-secure-systems/connaisseur.git
The Helm chart can be added by:
helm repo add connaisseur https://sse-secure-systems.github.io/connaisseur/charts
Configure⚓︎
The configuration of Connaisseur is completely done in the helm/values.yaml
.
The upper kubernetes
section offers some general Kubernetes typical configurations like image version or resources.
Noteworthy configurations are:
kubernetes.webhook.failurePolicy
: Failure policy allows configuration whether the mutating admission webhook should fail closed (Fail
, default) or open (Ignore
) should the Connaisseur service become unavailable. While Connaisseur is configured to be secure by default, setting the failure policy toIgnore
allows to prioritize cluster access1.kubernetes.webhook.reinvocationPolicy
: Reinvocation Policy defines whether Connaisseur is called again as part of the admission evaluation if the object being admitted is modified by other admission plugins after the initial webhook call (IfNeeded
) or not (Never
, default). Note that if Connaisseur is invoked a second time, the policy to be applied might change in between2. Make sure, your Connaisseur policies are set up to handle multiple mutations of the image originally specified in the manifest, e.g.my.private.registry/image:1.0.0
andmy.private.registry/image@sha256:<hash-of-1.0.0-image>
.kubernetes.deployment.securityContext
: Connaisseur ships with secure defaults. However, some keys are not supported by all versions or flavors of Kubernetes and might need adjustment3. This is mentioned in the comments to the best of our knowledge.kubernetes.deployment.podSecurityPolicy
: Some clusters require a PSP. A secure default PSP for Connaisseur is available.
The actual configuration consists of the application.validators
and image application.policy
sections.
These are described in detail below and for initials steps it is instructive to follow the getting started guide.
Other features are described on the respective pages.
Connaisseur ships with a pre-configuration that does not need any adjustments for testing. However, validating your own images requires additional configuration.
Deploy⚓︎
Install Connaisseur via Helm or Kubernetes manifests:
Install Connaisseur by using the Helm template definition files in the helm
directory:
helm install connaisseur helm --atomic --create-namespace --namespace connaisseur
Install Connaisseur using the default configuration from the chart repository:
helm install connaisseur connaisseur/connaisseur --atomic --create-namespace --namespace connaisseur
To customize Connaisseur, craft a values.yaml
according to your needs and apply:
helm install connaisseur connaisseur/connaisseur --atomic --create-namespace --namespace connaisseur -f values.yaml
Installing Conaisseur via Kubernetes manifests requires to first render the respecitve resources. If the repo was cloned, simply render templates via:
helm template helm -n connaisseur > deploy.yaml
- Create target namespace:
kubectl create namespace connaisseur
- Setup the preliminary hook:
kubectl apply -f deploy.yaml -l 'app.kubernetes.io/component=connaisseur-init' -n connaisseur
- Deploy core resources
kubectl apply -f deploy.yaml -l 'app.kubernetes.io/component=connaisseur-core' -n connaisseur
- Arm the webhook
kubectl apply -f deploy.yaml -l 'app.kubernetes.io/component=connaisseur-webhook' -n connaisseur
This deploys Connaisseur to its own namespace called connaisseur
.
The installation itself may take a moment, as the installation order of the Connaisseur components is critical:
The admission webhook for intercepting requests can only be applied when the Connaisseur pods are up and ready to receive admission requests.
Check⚓︎
Once everything is installed, you can check whether all the pods are up by running kubectl get all -n connaisseur
:
kubectl get all -n connaisseur
Output
NAME READY STATUS RESTARTS AGE
pod/connaisseur-deployment-78d8975596-42tkw 1/1 Running 0 22s
pod/connaisseur-deployment-78d8975596-5c4c6 1/1 Running 0 22s
pod/connaisseur-deployment-78d8975596-kvrj6 1/1 Running 0 22s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/connaisseur-svc ClusterIP 10.108.220.34 <none> 443/TCP 22s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/connaisseur-deployment 3/3 3 3 22s
NAME DESIRED CURRENT READY AGE
replicaset.apps/connaisseur-deployment-78d8975596 3 3 3 22s
Use⚓︎
To use Connaisseur, simply try running some images or apply a deployment. In case you use the pre-configuration, you could for example run the following commands:
kubectl run demo --image=docker.io/securesystemsengineering/testimage:unsigned
> Error from server: admission webhook "connaisseur-svc.connaisseur.svc" denied the request (...).
kubectl run demo --image=docker.io/securesystemsengineering/testimage:signed
> pod/demo created
Upgrade⚓︎
A running Connaisseur instance can be updated by a Helm upgrade of the current release:
Adjust configuration in helm/values.yaml
as required and upgrade via:
helm upgrade connaisseur helm -n connaisseur --wait
Adjust your local configuration file (e.g. values.yaml
) as required and upgrade via:
helm upgrade connaisseur connaisseur/connaisseur -n connaisseur --wait -f values.yaml
Adjust your local Kubernetes manifests (e.g. deploy.yaml
) as required and upgrade via delete and reinstall:
kubectl delete -f deploy.yaml -n connaisseur
kubectl apply -f deploy.yaml -l 'app.kubernetes.io/component=connaisseur-init' -n connaisseur
kubectl apply -f deploy.yaml -l 'app.kubernetes.io/component=connaisseur-core' -n connaisseur
kubectl apply -f deploy.yaml -l 'app.kubernetes.io/component=connaisseur-webhook' -n connaisseur
Note
Rolling upgrades as with Helm might also be possible, but likely require further configuration. Insights are welcome
Delete⚓︎
Just like for installation, Helm can also be used to delete Connaisseur from your cluster:
Uninstall via Helm:
helm uninstall connaisseur -n connaisseur
Uninstall via Helm:
helm uninstall connaisseur -n connaisseur
Delete via manifests:
kubectl delete -f deploy.yaml -n connaisseur
In case uninstallation fails or problems occur during subsequent installation, you can manually remove all resources:
kubectl delete all,mutatingwebhookconfigurations,clusterroles,clusterrolebindings,configmaps,imagepolicies,secrets,serviceaccounts,customresourcedefinitions -lapp.kubernetes.io/instance=connaisseur
kubectl delete namespaces connaisseur
Connaisseur for example also installs a CutstomResourceDefinition imagepolicies.connaisseur.policy
that validates its configuration.
In case of major releases, the configuration structure might change which can cause installation to fail and you might have to delete it manually.
Makefile⚓︎
Alternatively to using Helm, you can also run the Makefile for installing, deleting and more. Here the available commands:
make install
-- Install Connaisseur.make upgrade
-- Upgrade Connaisseur.make uninstall
-- Uninstall Connaisseur and delete the namespace.make annihilate
-- Remove all Connaisseur Kubernetes resources including its namespace. This command is usually helpful, should the normalmake uninstall
not work.make docker
-- Builds the connaisseur container image.
Detailed configuration⚓︎
All configuration is done in the helm/values.yaml
.
The configuration of features is only described in the corresponding section.
Any configuration of the actual application is done below the application
key, so when below we write validators
, this actually corresponds to the application.validators
key in the helm/values.yaml
.
Validators⚓︎
The validators are configured in the validators
field, which defines a list of validator objects.
A validator defines what kind of signatures are to be expected, how signatures are to be validated, against which trust root and how to access the signature data.
For example, images might be signed with Docker Content Trust and reside in a private registry.
Thus the validator would need to specify notaryv1
as type, the notary host and the required credentials.
The specific validator type should be chosen based on the use case. A list of supported validator types can be found here. All validators share a similar structure for configuration. For specifics and additional options, please review the dedicated page of the validator type.
There is a special behavior, when a validator or one of the trust roots is named default
.
In this case, should an image policy rule not specify a validator or trust root to use, the one named default
will be used instead.
This also means there can only be one validator named default
and for the trust roots, there can only be one called default
within a single validator.
Connaisseur comes with a few validators pre-configured including one for Docker's official images.
The pre-configured validators can be removed.
However to avoid Connaisseur failing its own validation in case you remove the securesystemsengineering_official
key, make sure to also exclude Connaisseur from validation either via the static allow
validator or namespaced validation.
The special case of static validators used to simply allow or deny images without verification is described below.
Configuration options⚓︎
.validators[*]
in helm/values.yaml
supports the following keys:
Key | Default | Required | Description |
---|---|---|---|
name |
- | Name of the validator, which is referenced in the image policy. It must consist of lower case alphanumeric characters or '-'. If the name is default , it will be used if no validator is specified. |
|
type |
- | Type of the validator, e.g. notaryv1 or cosign , which is dependent on the signing solution in use. |
|
trustRoots |
- | List of trust anchors to validate the signatures against. In practice, this is typically a list of public keys. | |
trustRoots[*].name |
- | Name of the trust anchor, which is referenced in the image policy. If the name is default , it will be used if no key is specified. |
|
trustRoots[*].key |
- | Value of the trust anchor, most commonly a PEM encoded public key. | |
auth |
- | - | Credentials that should be used in case authentication is required for validation. Details are provided on validator-specific pages. |
Further configuration fields specific to the validator type are described in the respective section.
Example⚓︎
helm/values.yaml
application:
validators:
- name: default
type: notaryv1
host: notary.docker.io
trustRoots:
- name: default
key: |
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsx28WV7BsQfnHF1kZmpdCTTLJaWe
d0CA+JOi8H4REuBaWSZ5zPDe468WuOJ6f71E7WFg3CVEVYHuoZt2UYbN/Q==
-----END PUBLIC KEY-----
auth:
username: superuser
password: lookatmeimjumping
- name: myvalidator
type: cosign
trustRoots:
- name: mykey
key: |
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEIFXO1w6oj0oI2Fk9SiaNJRKTiO9d
ksm6hFczQAq+FDdw0istEdCwcHO61O/0bV+LC8jqFoomA28cT+py6FcSYw==
-----END PUBLIC KEY-----
Static validators⚓︎
Static validators are a special type of validator that does not validate any signatures.
Depending on the approve
value being true
or false
, they either allow or deny all images for which they are specified as validator.
This for example allows to implement an allowlist or denylist.
Configuration options⚓︎
Key | Default | Required | Description |
---|---|---|---|
name |
- | Name of the validator, which will be used to reference it in the image policy. | |
type |
- | static ; value has to be static for a static validator. |
|
approve |
- | true or false to admit or deny all images. |
Example⚓︎
helm/values.yaml
application:
validators:
- name: allow
type: static
approve: true
- name: deny
type: static
approve: false
Image policy⚓︎
The image policy is defined in the policy
field and acts as a list of rule objects to determine which image should be validated by which validator (and potentially some further configurations).
For each image in the admission request, only a single rule in the image policy will apply: the one with the most specific matching pattern
field.
This is determined by the following algorithm:
- A given image is matched against all rule patterns.
- All matching patterns are compared to one another to determine the most specific one (see below). Only two patterns are compared at a time; the more specific one then is compared to the next one and so forth. Specificity is determined as follows:
- Patterns are split into components (delimited by "/"). The pattern that has a higher number of components wins (is considered more specific).
- Should the two patterns that are being compared have equal number of components, the longest common prefix between each pattern component and corresponding image component are calculated (for this purpose, image identifiers are also split into components). The pattern with the longest common prefix in one component, starting from the leftmost, wins.
- Should all longest common prefixes of all components between the two compared patterns be equal, the pattern with a longer component, starting from the leftmost, wins.
- The rule whose pattern has won all comparisons is considered the most specific rule.
- Return the most specific rule.
Should an image match none of the rules, Connaisseur will deny the request and raise an error.
This deny per default behavior can be changed via a catch-all rule *:*
and for example using the static allow
validator in order to admit otherwise unmatched images.
In order to perform the actual validation, Connaisseur will call the validator specified in the selected rule and pass the image name and potential further configuration to it. The reference to validator and exact trust root is resolved in the following way:
- The validator with name (
validators[*].name
) equal to thevalidator
value in the selected rule is chosen. If no validator is specified, the validator with namedefault
is used if it exists. - Of that validator, the trust root (e.g. public key) is chosen which name (
.validators.trustRoots[*].name
) matches the policies trust root string (with.trustRoot
). If no trust root is specified, the trust root with namedefault
is used if it exists.
Let's review the pattern and validator matching at a minimal example. We consider the following validator and policy configuration (most fields have been omitted for clarity):
application:
validators:
- name: default # validator 1
trustRoots:
- name: default # key 1
key: |
...
- name: myvalidator # validator 2
trustRoots:
- name: default # key 2
key: |
...
- name: mykey # key 3
key: |
...
policy:
- pattern: "*:*" # rule 1
- pattern: "docker.io/myrepo/*:*" # rule 2
validator: myvalidator
- pattern: "docker.io/myrepo/myimg:*" # rule 3
validator: myvalidator
with:
trustRoot: mykey
Now deploying the following images we would get the matchings:
docker.io/superrepo/myimg:v1
→ rule 1 → validator 1(key 1): The image matches none of the more specific rules 2 and 3, so rule 1 is applied. As that rule neither specifies a validator nor a trust root, thedefault
validator (validator 1) with trust rootdefault
(key 1) is used.docker.io/myrepo/superimg:v1
→ rule 2 → validator 2(key 2): The image only matches rules 1 and 2 and thus 2 is chosen as it is more specific. That rule specifiesmyvalidator
as validator but no trust root and thus validator 2 with trust rootdefault
(key 2) is used.docker.io/myrepo/myimg:v1
→ rule 3 → validator 2(key 3): The image matches all rules and thus 3 is chosen as it is most specific. The rule specifiesmyvalidator
as validator withmykey
as trust root and thus validator 2 with key 2 is used.
Connaisseur ships with a few rules pre-configured. There is two rules that should remain intact in some form in order to not brick the Kubernetes cluster:
registry.k8s.io
: This is anallow
rule for Kubernetes images (registry.k8s.io
) in order to not block cluster relevant images. These images are signed with keyless Cosign signatures, which Connaisseur doesn't support yet. Thus, they cannot be validated currently.docker.io/securesystemsengineering/*:*
: This rule is used to validate the Connaisseur images with the respective validator and removal can break the Connaisseur deployment. It is, however, possible to use the staticallow
validator.
Configuration options⚓︎
.policy[*]
in helm/values.yaml
supports the following keys:
Key | Default | Required | Description |
---|---|---|---|
pattern |
- | Globbing pattern to match an image name against. | |
validator |
default |
- | Name of a validator in the validators list. If not provided, the validator with name default is used if it exists. |
with |
- | - | Additional parameters to use for a validator. See more specifics in validator section. |
with.trustRoot |
default |
- | Name of a trust root, which is specified within the referenced validator. If not provided, the trust root with name default is used if it exists. |
Example⚓︎
helm/values.yaml
application:
policy:
- pattern: "*:*"
- pattern: "docker.io/myrepo/*:*"
validator: myvalidator
with:
trustRoot: mykey
- pattern: "docker.io/myrepo/deniedimage:*"
validator: deny
- pattern: "docker.io/myrepo/allowedimage:v*"
validator: allow
Common examples⚓︎
Let's look at some useful examples for the validators
and policy
configuration.
These can serve as a first template beyond the pre-configuration or might just be instructive to understand validators and policies.
We assume your repository is docker.io/myrepo
and a public key has been created.
In case this repository is private, authentication would have to be added to the respective validator for example via:
auth:
secretName: k8ssecret
The Kubernetes secret would have to be created separately according to the validator documentation.
Case: Only validate own images and deny all others⚓︎
This is likely the most common case in simple settings by which only self-built images are used and validated against your own public key:
helm/values.yaml
application:
validators:
- name: allow
type: static
approve: true
- name: default
type: notaryv1 # or e.g. 'cosign'
host: notary.docker.io # only required in case of notaryv1
trustRoots:
- name: default
key: | # your public key below
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEvtc/qpHtx7iUUj+rRHR99a8mnGni
qiGkmUb9YpWWTS4YwlvwdmMDiGzcsHiDOYz6f88u2hCRF5GUCvyiZAKrsA==
-----END PUBLIC KEY-----
- name: dockerhub_basics
type: notaryv1
host: notary.docker.io
trustRoots:
- name: securesystemsengineering_official
key: |
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsx28WV7BsQfnHF1kZmpdCTTLJaWe
d0CA+JOi8H4REuBaWSZ5zPDe468WuOJ6f71E7WFg3CVEVYHuoZt2UYbN/Q==
-----END PUBLIC KEY-----
policy:
- pattern: "*:*"
- pattern: "registry.k8s.io/*:*"
validator: allow
- pattern: "docker.io/securesystemsengineering/*:*"
validator: dockerhub_basics
with:
trustRoot: securesystemsengineering_official
Case: Only validate own images and deny all others (faster)⚓︎
This configuration achieves the same as the one above, but is faster as trust data only needs to be requested for images in your repository:
helm/values.yaml
application:
validators:
- name: allow
type: static
approve: true
- name: deny
type: static
approve: false
- name: default
type: notaryv1 # or e.g. 'cosign'
host: notary.docker.io # only required in case of notaryv1
trustRoots:
- name: default
key: | # your public key below
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEvtc/qpHtx7iUUj+rRHR99a8mnGni
qiGkmUb9YpWWTS4YwlvwdmMDiGzcsHiDOYz6f88u2hCRF5GUCvyiZAKrsA==
-----END PUBLIC KEY-----
- name: dockerhub_basics
type: notaryv1
host: notary.docker.io
trustRoots:
- name: securesystemsengineering_official
key: |
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsx28WV7BsQfnHF1kZmpdCTTLJaWe
d0CA+JOi8H4REuBaWSZ5zPDe468WuOJ6f71E7WFg3CVEVYHuoZt2UYbN/Q==
-----END PUBLIC KEY-----
policy:
- pattern: "*:*"
validator: deny
- pattern: "docker.io/myrepo/*:*"
- pattern: "registry.k8s.io/*:*"
validator: allow
- pattern: "docker.io/securesystemsengineering/*:*"
validator: dockerhub_basics
with:
trustRoot: securesystemsengineering_official
The *:*
rule could also have been omitted as Connaisseur denies unmatched images.
However, explicit is better than implicit.
Case: Only validate Docker Hub official images and deny all others⚓︎
In case only validated Docker Hub official images should be admitted to the cluster:
helm/values.yaml
application:
validators:
- name: allow
type: static
approve: true
- name: deny
type: static
approve: false
- name: dockerhub_basics
type: notaryv1
host: notary.docker.io
trustRoots:
- name: docker_official
key: |
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEOXYta5TgdCwXTCnLU09W5T4M4r9f
QQrqJuADP6U7g5r9ICgPSmZuRHP/1AYUfOQW3baveKsT969EfELKj1lfCA==
-----END PUBLIC KEY-----
- name: securesystemsengineering_official
key: |
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsx28WV7BsQfnHF1kZmpdCTTLJaWe
d0CA+JOi8H4REuBaWSZ5zPDe468WuOJ6f71E7WFg3CVEVYHuoZt2UYbN/Q==
-----END PUBLIC KEY-----
policy:
- pattern: "*:*"
validator: deny
- pattern: "docker.io/library/*:*"
validator: dockerhub_basics
with:
trustRoot: docker_official
- pattern: "registry.k8s.io/*:*"
validator: allow
- pattern: "docker.io/securesystemsengineering/*:*"
validator: dockerhub_basics
with:
trustRoot: securesystemsengineering_official
Case: Only validate Docker Hub official images and allow all others⚓︎
In case only Docker Hub official images should be validated while all others are simply admitted:
helm/values.yaml
application:
validators:
- name: allow
type: static
approve: true
- name: deny
type: static
approve: false
- name: dockerhub_basics
type: notaryv1
host: notary.docker.io
trustRoots:
- name: docker_official
key: |
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEOXYta5TgdCwXTCnLU09W5T4M4r9f
QQrqJuADP6U7g5r9ICgPSmZuRHP/1AYUfOQW3baveKsT969EfELKj1lfCA==
-----END PUBLIC KEY-----
- name: securesystemsengineering_official
key: |
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsx28WV7BsQfnHF1kZmpdCTTLJaWe
d0CA+JOi8H4REuBaWSZ5zPDe468WuOJ6f71E7WFg3CVEVYHuoZt2UYbN/Q==
-----END PUBLIC KEY-----
policy:
- pattern: "*:*"
validator: allow
- pattern: "docker.io/library/*:*"
validator: dockerhub_basics
with:
trustRoot: docker_official
- pattern: "registry.k8s.io/*:*"
validator: allow
- pattern: "docker.io/securesystemsengineering/*:*"
validator: dockerhub_basics
with:
trustRoot: securesystemsengineering_official
Case: Directly admit own images and deny all others⚓︎
As a matter of fact, Connaisseur can also be used to restrict the allowed registries and repositories without signature validation:
helm/values.yaml
application:
validators:
- name: allow
type: static
approve: true
- name: deny
type: static
approve: false
policy:
- pattern: "*:*"
validator: deny
- pattern: "docker.io/myrepo/*:*"
validator: allow
- pattern: "registry.k8s.io/*:*"
validator: allow
- pattern: "docker.io/securesystemsengineering/*:*"
validator: allow
-
This is not to be confused with the detection mode feature: In detection mode, Conaisseur service admits all requests to the cluster independent of the validation result while the failure policy only takes effect when the service itself becomes unavailable. ↩
-
During the first mutation, Connaisseur converts the image tag to its digests. Read more in the overview of Connaisseur ↩
-
In those cases, consider using security annotations via
kubernetes.deployment.annotations
or pod security policieskubernetes.deployment.podSecurityPolicy
if available. ↩