Add support for building with buildkit

[felipecrs]

Requires passing version='2' like:

stream = self.client.build(
    fileobj=dockerfile, tag='myimage', version='2'
)

Refs https://github.com/moby/moby/blob/3ff85c73436f1c4f8d9764a0d72e41a03b4116f5/api/swagger.yaml#L9343-L9352.

Closes #2230

[felipecrs]

@thaJeztah do you think you can take a look at this one?

I know the PR backlog is full, but in my opinion this PR is a no-brainer and is of huge help (#2230 has 200+ upvotes).

[AkihiroSuda]

Bumping up the version number doesn't magically implement the BuildKit API.
You have to dial /grpc to call the BuildKit gRPC API.
https://github.com/docker/buildx/blob/v0.25.0/driver/docker/driver.go
https://github.com/moby/buildkit/blob/v0.23.1/api/services/control/control.proto

In addition to implementing the gRPC client, you also have to implement several "attachable" servers (auth, secret, ssh) via the reverse-gRPC connection.
https://github.com/moby/buildkit/blob/v0.23.1/cmd/buildctl/build.go#L191-L214

This is quite complicated than you might imagine; I suggest just shelling out docker buildx and call it a day.

[felipecrs]

Thanks a lot for reviewing it.

Bumping up the version number doesn't magically implement the BuildKit API.

Tests prove you are wrong. No?

In addition to implementing the gRPC client, you also have to implement several "attachable" servers (auth, secret, ssh) via the reverse-gRPC connection.

Exposing additional options can be done in follow-up PRs. This PR is sufficient for my use case which is building Dockerfiles that depend on BuilKit-features that are not auth, secret, or ssh.

I suggest just shelling out docker buildx and call it a day.

That means installing docker buildx, which means installing docker cli. It's a big downside comparing to just calling the Rest API.

[AkihiroSuda]

Only basic BuildKit features can be enabled by just bumping up the version:
https://github.com/moby/moby/blob/v28.3.0/api/server/backend/build/backend.go#L55-L73

The scope of the available features should be documented

[felipecrs]

Do you mean I should clarify that squash and forcerm will error when version=2?

Otherwise, I expect all options currently supported by client.build() to be supported.

[felipecrs]

I will test it more, and let you know.

[felipecrs]

Ok, this is worse than I expected.

I got lucky with the tests because I only tried with FROM scratch.

Even a basic Dockerfile like below does not work:

    @requires_api_version('1.38')
    def test_build_buildkit_alpine(self):
        script = io.BytesIO('\n'.join([
            'FROM alpine',
        ]).encode('ascii'))

        self.tmp_imgs.append('buildkitalpine')

        stream = self.client.build(
            fileobj=script, tag='buildkitalpine',
            version='2'
        )

        for _chunk in stream:
            pass

        assert self.client.inspect_image('buildkitalpine')

But it's because of a bug:

• API /build doesn't pass AuthConfig to BuildKit moby/moby#48112

Still, it means not even basic use cases like pulling an image that doesn't require auth will not be able to use this feature.

@AkihiroSuda do you have any idea? Otherwise, I think it may be the end of this PR (and a dream). :(

This comment says creating a basic session is enough for pull to work. Do you think it would be acceptable to build a solution around it?

[AkihiroSuda]

This moby/moby#48112 (comment) says creating a basic session is enough for pull to work. Do you think it would be acceptable to build a solution around it?

SGTM

[felipecrs]

I haven't been able to build anything so far, but I will update when I have any news.

[felipecrs]

FYI I found some gRPC references from an unfinished buildkit implementation in this library from @shin-:

• https://github.com/docker/docker-py/tree/buildkit

[AkihiroSuda]

This PR doesn't work

https://github.com/docker/docker-py/pull/3344/files#r2174151883

[felipecrs]

While this PR works, this bug in docker daemon API prevents it from being useful:

• API /build doesn't pass AuthConfig to BuildKit moby/moby#48112

Because of that bug, any Dockerfile that refers to docker images that are not yet pulled will fail to build.

One workaround is to pull these images before invoking the build.

I haven't been able to work around it more cleanly.

I am no longer sure if this PR should be merged, but here are two reasons why it may be a good idea:

1. There is a workaround (pull images in advance). Not ideal and not even easy, but it's better than nothing.
2. This is not a problem that should be fixed in docker-py or any other client. Once it gets fixed in docker daemon API, the fix will be transparent for the client and will start working.

So, I will let maintainers of this project decide what is the best way forward.