Lazy pulling stargz/eStargz base images¶

Experimental

This feature is considered EXPERIMENTAL and under active development until further notice.

This document describes the configuration that allows buildkit to lazily pull stargz/eStargz-formatted images from registries and how to obtain stargz/eStargz images.

By default, buildkit doesn't pull images until they are strictly needed. For example, during a build, buildkit doesn't pull the base image until it runs commands on it (e.g. RUN Dockerfile instruction) or until it exports stages as tarballs, etc.

Additionally, if the image is formatted as stargz/eStargz, buildkit with the configuration described here can skip pull of that image even when it's needed (e.g. on RUN and COPY instructions). Instead, it mounts that image from the registry to the node and lazily fetches necessary files (or chunks for big files) contained in that image on demand. This can hopefully reduce the time to take for the build. This document describes the configuration and usage of this feature.

For more details about stargz/eStargz image format, please see also Stargz and eStargz image formats section.

Known limitations¶

If you are using containerd worker, stargz snapshotter (containerd-stargz-grpc) needs to be run and configured separately.
Rootless execution is currently unsupported.

Enabling lazy pulling of stargz/eStargz images¶

Buildkit supports two ways to enable lazy pulling of stargz/eStargz images.

Using builtin support (recommended)¶

OCI worker has builtin support for stargz/eStargz. You can enable this feature by running buildkitd with an option --oci-worker-snapshotter=stargz.

buildkitd --oci-worker-snapshotter=stargz

This is the easiest way to use this lazy pull feature on buildkit. Currently, this configuration is unsupported for containerd worker.

Example of building an image with lazy pull¶

Once buildkitd starts with the above configuration, stargz/eStargz images can be lazily pulled. For example, we build the following golang binary with buildkit.

package main

import "fmt"

func main() {
    fmt.Println("Hello, world!")
}

The example Dockerfile which leverages eStargz-formatted base image will be the following.

# Uses eStargz-formatted golang image as the base image. This isn't pulled here.
FROM ghcr.io/stargz-containers/golang:1.15.3-buster-esgz AS dev

# Copies the source code from the context. The base image is mounted and lazily pulled.
COPY ./hello.go /hello.go

# Runs go compiler on that base image. The base image is mounted and lazily pulled.
RUN go build -o /hello /hello.go

FROM scratch

# Harvesting the result binary.
COPY --from=dev /hello /

Then this can be built with skipping pull of ghcr.io/stargz-containers/golang:1.15.3-buster-esgz. Instead, this image will be mounted from the registry to the node and necessary chunks of contents (e.g. go command binary, library, etc.) are partially fetched from that registry on demand.

$ buildctl build --frontend dockerfile.v0 \
                 --local context=/tmp/hello \
                 --local dockerfile=/tmp/hello \
                 --output type=local,dest=./
$ ./hello
Hello, world!

Note that when a stage is exported (e.g. to the registry), the base image (even stargz/eStargz) of that stage needs to be pulled to copy it to the destination. However if the destination is a registry and the target repository already contains some blobs of that image or cross repository blob mount can be used, buildkit keeps these blobs lazy.

Using proxy (standalone) snapshotter¶

This is another way to enable stargz-based lazy pull. Stargz Snapshotter (containerd-stargz-grpc) needs to be installed. Note that buildkit's registry configuration doesn't propagate to stargz snapshotter so it needs to be configured separately when you use private/mirror repositories.

This configuration is for users of containerd worker and ones trying other versions of stargz snapshotter than built into OCI worker.

With OCI worker¶

Spawn containerd-stargz-grpc as a separated process. Then run buildkitd with an option --oci-worker-proxy-snapshotter-path=/run/containerd-stargz-grpc/containerd-stargz-grpc.sock which makes it recognize this snapshotter via the socket.

containerd-stargz-grpc
buildkitd --oci-worker-snapshotter=stargz \
          --oci-worker-proxy-snapshotter-path=/run/containerd-stargz-grpc/containerd-stargz-grpc.sock

With containerd worker¶

Configure containerd's config.toml (default = /etc/containerd/config.toml) to make it recognize stargz snapshotter as a proxy snapshotter.

[proxy_plugins]
  [proxy_plugins.stargz]
    type = "snapshot"
    address = "/run/containerd-stargz-grpc/containerd-stargz-grpc.sock"

Then spawn containerd-stargz-grpc and containerd, and run buildkitd with an option --containerd-worker-snapshotter=stargz which tells containerd to use stargz snapshotter.

containerd-stargz-grpc
containerd
buildkitd --containerd-worker-snapshotter=stargz --oci-worker=false --containerd-worker=true

When you use standalone stargz snapshotter, registry configuration needs to be done for the stargz snapshotter process, separately. Create a configuration toml file which contains the registry configuration for stargz snapshotter (e.g. /etc/containerd-stargz-grpc/config.toml). The configuration format differs from buildkit. For more information about this format, please see also docs in the repository.

[[resolver.host."exampleregistry.io".mirrors]]
host = "examplemirror.io"

Then pass this file to stargz snapshotter through --config option.

containerd-stargz-grpc --config=/etc/containerd-stargz-grpc/config.toml

Stargz and eStargz image formats¶

Stargz and eStargz are OCI/Docker-compatible image formats that can be lazily pulled from standard registries (e.g. Docker Hub, GitHub Container Registry, etc). Because they are backwards-compatible to OCI/Docker images, they can run on standard runtimes (e.g. Docker, containerd, etc.). Stargz is proposed by Google CRFS project. eStargz is an extended format of stargz by Stargz Snapshotter. It comes with additional features including chunk verification and prefetch for avoiding the overhead of on-demand fetching. For more details about lazy pull with stargz/eStargz images, please refer to the docs on these repositories.

Creating stargz/eStargz images¶

Building eStargz image with BuildKit¶

BuildKit supports creating eStargz as one of the compression types. As shown in the following, compression=estargz creates an eStargz-formatted image. Specifying oci-mediatypes=true option is highly recommended for enabling layer verification of eStargz.

buildctl build ... \
  --output type=image,name=docker.io/username/image,push=true,compression=estargz,oci-mediatypes=true

Building eStargz is only supported by OCI worker.

compression option isn't applied to layers that already exist in the cache (including the base images). Thus if you create eStargz image using non-eStargz base images, you need to specify force-compression=true option as well for applying the compression config to all existing layers.

BuildKit doesn't support prefetch-based optimization of eStargz. To enable full feature of eStargz, you can also use other tools as described in the next section.

Other methods to obtain stargz/eStargz images¶

Pre-converted stargz/eStargz images are available at ghcr.io/stargz-containers repository (mainly for testing purpose).

You can also create any stargz/eStargz image using the variety of tools including the following.

nerdctl: Docker-compatible CLI for containerd and BuildKit. This supports convert subcommand to convert an OCI/Docker image into eStargz.
ctr-remote: containerd CLI developed in stargz snapshotter project. This supports converting an OCI/Docker image into eStargz and optimizing it.
stargzify: CLI tool to convert an OCI/Docker image to stargz. This is developed in CRFS project. Creating eStargz is unsupported.

There also other tools including Kaniko, ko, builpacks.io that support eStargz creation. For more details, please refer to Creating eStargz images with tools in the community section in the introductory post.