On the Path to Gitless GitOps: Building Trusted Artifacts

Deploying software to your systems is a matter of trust. This can be trust in the many eyes on open source code, trust in a vendor of closed source software, or simply trust in developers working in your organization. Regardless of the basis for that trust, it underpins all deployment decisions.

This raises the question: how do you connect the concept of trust to automated deployment decisions? The answer involves cryptographically signing artifacts and providing attestations to prove the code you want to deploy complies with requirements about dependencies, licensing, and other policies.

How to trust your own code

Establishing trust in software should start at the atomic level, the code commit. The purpose is to ensure that the authenticity, or the identity of a code committer, can be verified. Fortunately, git provides commit signing, supporting a number of key formats. It is tempting to merely configure git signing and then configure your version control system (VCS) to require commits to be signed by a trusted key. However, your build pipeline should not depend on this, as it simply outsources trust management to your VCS.

Instead, signatures for each commit should be validated before building an artifact. For larger repositories, this may not be practical to perform on each PR build. In this case, abbreviated forms may be used instead, such as verifying all commits that were made since the branch was forked. However, production builds should retain their full verification.

You may have already spotted the hole in this approach: validating developer identity. How do you handle revoked keys from developers who have left the organization?

To a degree, this can be resolved with short-lived signing keys and explicitly adding each new signing key to the verification step in your pipeline. For repositories with a high number of contributors, this may quickly prove intractable. In this event, more specialised tooling providing commit signing via secrets management infrastructure, such as OpenBao or fine-grained git policies via gittuf, can be evaluated.

Additionally, organizations using bots or AI agents in their development workflows should observe careful custodianship over signing keys. Depending on organizational policy, regulation, and risk appetite, this practice ranges from simply relying on platform-managed signing keys to self-hosting autonomous agents and executing cryptographic signatures via infrastructure such as OpenBao’s transit engine.

Testing and scanning code

Signed commits at the ready, the classic functionality of a build pipeline comes up next: linting, testing, and scanning.

Naturally, which tools are used for linting and testing depends entirely on the ecosystem of your programming languages and your preferred tooling. Beyond that, there is a range of common security scanning tools that provide assurance that common vulnerabilities and other security concerns have been eliminated from code. This includes all-in-one tools such as OpenGrep, Wiz, or SonarQube, but can also include language-specific tools like Python’s Bandit or Go’s gosec.

Having run your pipeline, how can you prove to consuming systems that it was run? Modern VCSs provide the ability to write one or more pieces of status information to commits. However, this approach is lacking, even aside from vendor lock-in, it requires any systems consuming your artifacts to have access to your repository.

A far better approach is distributing cryptographically signed attestations with your artifacts. The de facto standard in this space is in-toto and there are a number of tools relying on it that support the creation or management of attestations for an artifact. We’ll take a look at some in the next section.

For now, at each critical step in your pipeline, such as execution of security scans and tests, you should generate attestations as JSON files to show what has been executed. The in-toto project has established a format for creating attestations that includes predicates for common steps such as test results and SBOMs.

Now that we can prove both who wrote the code and that it has complied with the expected standards, let’s look at the artifact itself.

Build and release

Building and packaging an artifact is, of course, language-specific. We won’t examine less portable details here, but we will discuss some general principles as they apply to trusting the end artifact. Additionally, in the event you release the artifact in a native format rather than solely as an image, ensure that you use whatever artifact signing tools are available for your language.

In this section, we will focus on the core topic of this series: the Open Container Initiative (OCI) format. Begin with the base image, which should be minimal and as hardened as possible. For statically compiled languages, use scratch where possible, otherwise consider free offerings from Chainguard, Distroless, or Project Hummingbird. All the careful ownership of code and build process will be wasted if you utilise a base image without establishing appropriate trust.

We will be looking at the process of verifying images and artifacts in a future post, as it overlaps heavily with the process you should use for selecting a base image. However, as a short summary, you should be looking for attestations and signatures, as well as establishing trust with the organization offering the image via methods such as contracts or researching their track record.

Once the base image is selected, pull the image by its SHA256 digest and verify it with Cosign, or whatever tool it supports.

Now that it is verified, ensure that you pin by the SHA to guarantee that changes to tags cannot change the image used. Ensure reproducibility by pinning as specifically as possible for all dependencies installed, including any compilers or interpreters. Where you need to install utilities during the build process that are not required at runtime, use a multi-stage build and copy the resulting outputs.

Although explicitly deleting installed utilities or temporary files should work, by using a multi-stage build, you avoid distributing anything added to the image as a side effect of the build process. It also has the added benefit of allowing you to perform all the heavy lifting in an earlier stage and copy the final binaries into a much smaller container image, reducing the footprint and attack surface, and making the workload cheaper to run.

Scan the final image with Trivy, Upwind, or other tooling of your choice for unresolved vulnerabilities introduced during the image build process. Generate an attestation for this scan, just as done for testing and code scanning. Using the tool of your choice, such as Trivy, generate an SBOM for the image, which is a file that tracks all the information around the dependencies being used. Note that when selecting scanning and SBOM tooling, many tools already support existing in-toto predicates, which means you do not have to create tooling to format them appropriately.

For example, you generate a Trivy SBOM like so:

trivy image --format cyclonedx --output sbom.json registry.example.com/a-simple-app@sha256:1234

Finally, it is time to sign the image. In this series, we will use Cosign, which provides a relatively simple interface for signing images and flexibility with key material. This includes classic static key options, such as OpenBao’s transit engine, as well as short-lived “keyless” signatures that leverage OIDC to provide authentication information.

In this keyless option, Cosign signs the image with a disposable key, avoiding the overhead of key management. A permanent record of this action is stored in a transparency log, which binds the public key to the OIDC-provided identity and allows verification going forward.

Along with signing, you will add your attestations for the OCI image with Cosign. Cosign will wrap the attestation in JSON, including a reference to the image signature, and then sign the resulting attestation. Using our Trivy SBOM from before, we can do a two-step process to first sign the image and then bind the attestation. For this example, we’ll assume the availability of an OpenBao transit engine named gitless-gitops, but you could easily use a regular key pair instead. You’ll need to have both the BAO_ADDR and BAO_TOKEN environment variables set for this:

cosign sign --key openbao://gitless-gitops --use-signing-config=false --tlog-upload=false registry.example.com/a-simple-app@sha256:1234
cosign attest --type cyclonedx --predicate sbom.json --key openbao://gitless-gitops --use-signing-config=false --tlog-upload=false registry.example.com/a-simple-app@sha256:1234

We also disable the transparency log with --tlog-upload false. By doing this while also using the transit engine in a self-hosted OpenBao engine, we can provide a fully-sovereign mechanism for artifact signing.

From this point on, your CD system can consume both image and attestation, verifying them to ensure you deploy only trusted code.

Moving beyond image building

If you’ve implemented our recommendations, you have a (reasonably) trustworthy OCI container, including the attestations to prove it. Next up, we’ll look at packaging deployment manifests before moving on to securely deploying everything we’ve put together.

Overwhelmed by making a secure pipeline? ControlPlane can help, leveraging deep experience in:

• Securing software supply chains
• Building hardened pipelines
• Shifting-left security

Get in touch with the team to discuss how we have helped other organizations and what we could do for you.