Skip to content
Back to Insights

On the path to Gitless GitOps: We trust code, but do we trust the manifests?

Header image
  • CI/CD
  • DevOps
  • Software Supply Chain Security
  • OpenBao
  • GitOps
  • Flux CD

Your manifests are first-class assets

Your Kubernetes manifests are code, too! They fundamentally underpin your application code, meaning you cannot deploy your service without them. Conversely, those manifests serve no purpose without the code itself.

Manifests define the infrastructure on which your application is deployed, yet many engineers choose not to treat them with the same rigour as application code. Admittedly, this is a natural reaction: for most developers, the vast majority of changes will occur in the application-level codebase, rather than in the underlying infrastructure. This is true whether you deploy to Kubernetes or to your traditional cloud provider.

However, the software infrastructure is akin to the physical building of a shopping centre; your store might be the most profitable business around, but if it suffers from electricity cuts or water leaks every now and then, your customers might stop shopping there. Likewise, accepting inventory of dubious origin could expose you or your customers to risk, regulatory authorities, or both.

Bridging this gap is what Platform Engineering and DevSecOps practices are designed to do. Automated policy checks and security guardrails can be shifted left in the development process, rather than caught at deployment time, thus ensuring secure-by-default software delivery.

Because the business value of the service is directly impacted by the application you serve and the infrastructure it is deployed on, it is imperative that your infrastructure receives the same rigorous treatment as the application itself. In technical terms, this means that your infrastructure configuration should follow deterministic build patterns just like regular application code, ensuring it receives the same level of trust.

Relying on mutable Git references, committing manifest changes without adequate CI checks, and deploying configuration without verifying its cryptographically assured provenance are signs that you are treating your manifests as second-class assets.

How to actually treat your manifests as first-class assets

As already discussed in the first blog in this series (opens in a new tab), we build trust in our manifests in the same way that we achieve trust in the application code: by storing them as OCI artifacts with the corresponding attestations. The OCI image specification provides the media-type property (opens in a new tab), which enables the packaging of Kubernetes manifests, and cloud providers have supported this for years. What remains is to demand the same rigour that we bring to application code.

The next step would be to apply automated scanning to the resulting hydrated files. As mentioned in the previous blog, we could leverage tools such as OpenGrep (opens in a new tab) for code. For Kubernetes manifests, we can rely on Kubesec (opens in a new tab) and KubeLinter (opens in a new tab) to run static analysis, catch security misconfigurations and enforce best practices, while using Kubeconform (opens in a new tab) to identify breaking changes between Kubernetes versions. Just as with regular code, we can choose to fail the pipeline if the manifests do not meet the predefined engineering standards and obtain proof of passing these controls by generating in-toto attestations. On top of the engineering team requirements, organisational steps may also be included within the pipeline to ensure that certain security policies are followed, e.g. all namespaces should be configured with PSS restricted.

Most infrastructure will remain similar across environments, with just some configuration values changing from one to another.

To ensure that you can promote your infrastructure through the environments, we recommend hydrating, linting and testing the immutable part of the infrastructure in CI prior to creating the OCI artifact, then leveraging the capabilities of your GitOps agent to inject the environment-dependent values.

Once static checks pass, manifests should be deployed to a cluster to ensure the viability of the new changes. Depending on the engineering and validation needs, these can be deployed either to long-lived test clusters or to ephemeral environments. Regardless of where they are deployed, testing needs to run to validate that the new infrastructure is behaving as expected. This is probably one of the most misunderstood steps in deploying cloud native workloads; automated deployment testing requires a level of cloud native maturity that takes time to achieve (opens in a new tab). As with other kinds of checks, development teams can generate attestations to prove they ran their tests successfully.

Beyond initial deployment testing, you must guarantee that the exact manifests validated in your test clusters are the ones that will reach production. With OCI artifacts, you can create a new attestation that demonstrates that the OCI artifact was tested and deployed in a lower environment, then have higher environments require attestations from lower ones to be present before allowing the OCI artifact to deploy in the production environment.

Packaging Kubernetes manifests as OCI artifacts

Just as with code, you can leverage cryptographic attestations to record the outcomes of both your CI pipeline scans and checks, and the custom testing against deployed manifests. This will provide cryptographic proof that the final package complies with expected engineering standards. By placing controls at CI runtime, your organization will also be able to generate tamper-proof attestations for their checks, ensuring that policies are adhered to by an artifact that has passed CI. At the same time, it will also help the automated decision making about which artifacts can reach the deployment stage.

As a final step, what is needed is a change of mindset, your manifests are now an artifact to be consumed and, as such, they need to follow the same rules that your code artifacts. This means that they should follow a form of standardised versioning, such as semver (opens in a new tab) and follow reproducible build lifecycles (opens in a new tab).

Now it is just a matter of packaging and uploading to your OCI registry. For this, you can leverage Cosign (opens in a new tab) and the oras CLI (opens in a new tab):

# Generate/render your manifests
kubectl run nginx --image=nginx:1.31.5 --dry-run=client -o yaml > manifest.yaml

# Bundle the manifest, push as an OCI image
tar -czf manifest-oci.tar.gz manifest.yaml

oras push registry.example.com/a-simple-app-manifest:v1.0.0 \
    --artifact-type application/vnd.oci.image.manifest.v1+json \
    manifest-oci.tar.gz:application/vnd.oci.image.layer.v1.tar+gzip

Or if you use FluxCD, the Flux CLI can also help you out here, the command below (opens in a new tab) will automatically bundle the manifest:

flux push artifact \
oci://registry.example.com/a-simple-app-manifest:v1.0.0 \
--path="./manifest.yaml" \
--source="local-test" \
--revision="v1"

Note that the values for --source and --revision have been mocked for this example, but they should reflect the source of the OCI artifact and the current release. Refer to the CLI documentation (opens in a new tab) for some usage examples.

For signing the artifact, Cosign is capable of interacting with the remote registry, we will be using the OpenBao key we created in the previous article:

# Sign the image
IMAGE_REPO="registry.example.com/a-simple-app-manifest"
TAG="v1.0.0" 
DIGEST=$(oras resolve --plain-http "${IMAGE_REPO}:${TAG}")

cosign sign --key openbao://gitless-gitops "${IMAGE_REPO}@${DIGEST}"

After the artifact has been uploaded and signed, it can be easily retrieved with:

oras manifest fetch "${IMAGE_REPO}:${TAG}" --pretty

Once uploaded, you can add cryptographic attestations. Before that happens, we run the relevant tests and generate record to run result as an in-toto test result attestation (opens in a new tab):

cat << EOF > my-test-result.json
{
  "result": "PASSED",
  "configuration": [
    {
      "name": "http-liveness-check",
      "uri": "https://local-poc/run/1"
    }
  ],
  "url": "https://local-poc/run/1",
  "passedTests": [
    "http-liveness-check"
  ],
  "warnedTests": [],
  "failedTests": []
}
EOF

And finally, we sign the attestation and upload it to the repository:

cosign attest \
    --key openbao://gitless-gitops  \
    --type "https://in-toto.io/attestation/test-result/v0.1" \
    --predicate my-test-result.json \
    --allow-http-registry=true \
    --yes \
    "${IMAGE_REPO}@${DIGEST}"


cosign tree --allow-http-registry=true "${IMAGE_REPO}@${DIGEST}"

If you have been following up until now, the natural question that pops into your head is: How do I know which media type I should be using when packaging my artifact? The answer is completely up to you. The OCI artifact guidance (opens in a new tab) explicitly states that when anything other than containers are packaged into OCI artifacts, any media type should be used except for the already known ones (opens in a new tab) and it must be compliant with RFC 6838 (opens in a new tab).

At present, the Kubernetes project does not seem to have claimed any custom media types, so different projects tend to define their own. You can look at FluxCD (opens in a new tab) and Helm (opens in a new tab) as some examples of this trend.

Take your infrastructure to the next level

You may be feeling somewhat overwhelmed at this point! Transitioning to artifact-centric delivery, implementing advanced DevSecOps pipelines, and enforcing cryptographic attestations require a level of cloud native maturity that takes time to achieve. ControlPlane can help you secure your cloud native infrastructure, leveraging deep experience in:

  • Elevating infrastructure as code: Transitioning your pipelines to treat Kubernetes manifests as first-class, secure-by-default OCI artifacts.
  • Building DevSecOps paved roads: Seamlessly integrating tools like KubeLinter, Kubesec, and Kubeconform to enforce engineering standards before deployment.
  • Advanced GitOps architectures: Designing and configuring secure, high-throughput Argo CD and Flux CD pipelines tailored to enterprise scale.

Whether you are looking to lock down your supply chain (opens in a new tab) or automate governance and compliance, ControlPlane brings years of experience delivering secure, high-end solutions in highly regulated spaces. Reach out (opens in a new tab) to ControlPlane today to learn how we can bring secure-by-default engineering to your organization.

View all

Speak to an expert

Engage with world-leading AI and cloud security specialists