Reproducible build support

[lrvick]

Story

Make it easy to provide end users package managers or system bootloaders a way to automatically verify that both code authors, package maintainers, and CI systems (with well known keys) all agreed independantly that a package I am about to install has the same hash, and was built from a given git ref.

This allows for end-to-end verification that logic has not been mutated from the time it was created on an author-controlled system to the final system where compiled code will be executed.

Example users

• Debian maintainer on the reproducible-builds project
  • See: https://wiki.debian.org/ReproducibleBuilds
• Bitcoin Core maintainer of the reproducible Gitian build system
  ** See: https://github.com/bitcoin-core/gitian.sigs
• Yubico engineer that maintains assembly-line HSMs that install firmware for yubikeys

Assumptions

• All parties have well known keys trusted by each other
• Binary building steps could be replaced by indepenantly maintained CI/CD setups
• A lot of not scope relevant validation steps like ensuring client downloads -latest- signed binary are not covered here

Currently workflow

1. Author creates code and signs commit in a feature branch
2. Reviewer reviews that code and does signed merge commit to master
3. Author builds binary and signs a v0.1.0 tag containing resulting binary hash as a comment
4. Reviewer builds binary and signs a v0.1.0-review tag containing resulting binary hash as a comment
5. Anyone publishes binary
6. Package Installer downloads new binary release and source code
7. Package Installer verifies v0.1.0 tag and v0.1.0-review tag are by 2 whitelisted keys that agree on the same hash just downloaded
8. Package Installer installs binary

Proposed workflow

1. Author creates and builds code and runs git-wotr command to sign artifacts and commit hash
2. Reviewer reviews and builds code and runs git-wotr command to sign artifact and commit hash
3. Merging to master, and binary/signature publishing/mirroring done by anyone.
4. Package Installer downloads new binary release and source code
5. Package Installer verifies signatures agree on binary they just downloaded
6. Package Installer installs binary

Background

More discussion at: hashbang/git-signatures#13

[lrvick]

Side note for bike shedding: Is this issue format reasonable for structured discussions? I can replicate it on the other issues helps with communication.

[Ekleog]

Orthogonal issues

I feel like integrating artifact hash into the review is conflating two orthogonal issues. Indeed:

• Artifact hash is verified by a machine, reviewing code is verified by a human
• Artifact signing signs an association between a code version and a binary, code review signing signs either a diff in the files or a state in the files

So this leaves only convenience, not logic, to govern the addition of such metadata into a commit-specific.

Issues with the proposed workflow

The proposed workflow has a few issues IMO:

• Author cannot send PR before having finished building the code (which can take hours depending on the project). And then, reviewer must also build the code before merging. This slows down a whole lot PR merging, which, on high-churn repositories, won't work out
• It works only with review signatures that sign the entirety of the repository (because it needs to attach the artifact to a source hash), so patch-id signatures are not a solution for this issue
• With commit-id signatures, the artifact hash is attached to the commit hash, while it could instead be attached to the tree-id (as a commit message change wouldn't invalidate the build)
• The package installer needs to have the whole review signature in able to be able to check the artifact, this is useless cruft for them, and it needs to be exported out of the review system

Overall, this means the proposed workflow can:

• Only be used along with tree-id (and its potential security limitations)
• Slow down merging
• Make signature exporting harder (as you need to export many review signatures from the review system)
• Make signature verification less streamlined, because the package installer either needs a specialized tool or needs to hand-parse the signature format to figure out where the artifact hash is, if they don't want to check the whole repository

Alternative workflow

So here is an alternative workflow, that in addition luckily doesn't require specific support by git-wotr:

1. Author creates code, reviews it, pushes it
2. Committer reviews the code, reviews and pushes it
3. A bunch of independent CI systems build at the commit, and sign a file containing the hashes of the artifacts as well as the commit number (this file will likely be required anyway for release publication)
4. Package installer can decide to either:
   • Download the binary and CI-signed checksum file, and trust the CI systems to only have ever built and signed correctly-reviewed commits (equivalent to your scheme)
   • Download the binary, the source code, check the source code reviews do mean they trust the commit, and use the CI-signed checksum file to ensure the artifact is actually correctly built for this source code (more security, could also be done in your scheme, but you're missing a bullet point for it)

[lrvick]

Author cannot send PR before having finished building the code (which can take hours depending on the project). And then, reviewer must also build the code before merging. This slows down a whole lot PR merging, which, on high-churn repositories, won't work out
It works only with review signatures that sign the entirety of the repository (because it needs to attach the artifact to a source hash), so patch-id signatures are not a solution for this issue

For the high security use cases with rare updates like the assembly line case I mentioned: builds would only be done on airgapped laptops totally offline. Slowdown is acceptable here for maximum confidence. Full disclosure: I already do a very similar (but more complicated) workflow today with airgapped systems as no online or automated system can be trusted in my use cases which must defend against adversaries with reason to expose new spectre-class 0-days that could potentially impact any online CI system.

The proposed setup would greatly simplfy the workflow I must use today. It is obviously not going to be suitable for very high churn setups which more flexible hybrid workflows discussed in the "anchoring" issue might address.

A modification that might be better for some use cases would be only doing the signatures that contain binary hashes at the -tag- level which allows for multiple in-flight PRs that all must settle and get into high trust mode at tag time.

With commit-id signatures, the artifact hash is attached to the commit hash, while it could instead be attached to the tree-id (as a commit message change wouldn't invalidate the build)

The above including the idea to sign tags (since releases are the only time binary signing makes sense) may cover this and allow for similarly simple workflows

The package installer needs to have the whole review signature in able to be able to check the artifact, this is useless cruft for them, and it needs to be exported out of the review system

Exporting is simple, and this is not actually useless cruft for some of the high security use cases I and others in my industry need this for.

Consider the "Package Installer" are a group of humans putting mission critical firmware on an offline HSM such as the Yubico story.

Those tasked with this job have no problem being very slow and thorough. As a matter of practice they want to be able to look over the code for the binary they are being asked to deploy and cryptographically confirm on airgapped systems that the code was authored and reviewed by trusted individuals with high confidence, and that this binary really came from that code. It is a custody chain.

Your CI style reproducible build workflow may indeed make more sense for use cases with lots of artifacts on different release cadences. Those CI systems however must have very high trust and ideally not share similar attack surface.

Maybe the paranoid workflow I am seeking to simplify and standardize is exactly what one might want for the rarely updated secure enclave signing firmware in the CI servers that does the very automatic signing flow you propose ;)

[Ekleog]

A modification that might be better for some use cases would be only doing the signatures that contain binary hashes at the -tag- level which allows for multiple in-flight PRs that all must settle and get into high trust mode at tag time.

This sounds better to me: a tag is a signature of the state of the repository, while a commit is a signature of a diff of the repository (well… most commits, some commits may be state signatures in the current draft, but I don't expect them to be used often). Also, you usually want to build reproducible artifacts only for things you officially release.

So I'd be in favor of adding the artifact hashes to the tag signatures, if adding them to a separate file is too complex. Now, I'm not really sure whether this has a strong tie with reviews, so I'm not sure this should go into this RFC: it can be a thing git-signatures allows to do as a bonus? :)

[oxij]

I think this should be out of scope of the top-level thing. It can be an extension. See #4 (comment)

[Ekleog]

@lrvick Can you confirm adding the artifact hashes to the tag signatures does fit your use case, and we can close this issue?

[Ekleog]

@lrvick up