containerd Proxy Plugins

Introduction

In a previous post we went through how plugins work in containerd. Typically extending containerd with plugins requires compiling containerd with the new plugin imported. Proxy plugins allow taking advantage of containerd’s stable API to configure a plugin which proxies through rpc to another running process. This is easy in containerd since the same interface is used on the client side as the server side, in the proxy case the server is just using a client instance of an interface.

Service Interfaces

Most of the core services containerd define both a Go interface and a protobuf service compiled to grpc and ttrpc. In the containerd daemon, there is at least one implementation of every service and a translation from the registered rpc service to the local implementation. We also have a proxy implementation of each Go interface which translates and calls the rpc service. This makes it very easy to share the same logic within the client and server as well as move implementations between them.

We can look at the Snapshotter interface as an example. The Go interface is relatively simple since most functions operate on a single key and immediately return a value. Any longer lived filesystem operations are deferred to the mounts.

type Snapshotter interface {
	Stat(ctx context.Context, key string) (Info, error)
	Update(ctx context.Context, info Info, fieldpaths ...string) (Info, error)
	Usage(ctx context.Context, key string) (Usage, error)
	Mounts(ctx context.Context, key string) ([]mount.Mount, error)
	Prepare(ctx context.Context, key, parent string, opts ...Opt) ([]mount.Mount, error)
	View(ctx context.Context, key, parent string, opts ...Opt) ([]mount.Mount, error)
	Commit(ctx context.Context, name, key string, opts ...Opt) error
	Remove(ctx context.Context, key string) error
	Walk(ctx context.Context, fn WalkFunc, filters ...string) error
	Close() error

The protobuf definition is very similar. There are slight differences between List and Walk, as one is implemented as a stream and the other as a callback. However, these differences just reflect the best definition for the context and is easily translated between go and rpc.

service Snapshots {
	rpc Prepare(PrepareSnapshotRequest) returns (PrepareSnapshotResponse);
	rpc View(ViewSnapshotRequest) returns (ViewSnapshotResponse);
	rpc Mounts(MountsRequest) returns (MountsResponse);
	rpc Commit(CommitSnapshotRequest) returns (google.protobuf.Empty);
	rpc Remove(RemoveSnapshotRequest) returns (google.protobuf.Empty);
	rpc Stat(StatSnapshotRequest) returns (StatSnapshotResponse);
	rpc Update(UpdateSnapshotRequest) returns (UpdateSnapshotResponse);
	rpc List(ListSnapshotsRequest) returns (stream ListSnapshotsResponse);
	rpc Usage(UsageRequest) returns (UsageResponse);
	rpc Cleanup(CleanupRequest) returns (google.protobuf.Empty);
}

Note that there is also a Cleanup function defined in the protobuf that doesn’t exist in the Go snapshots.Snapshotter interface, however, it does exist as separate snapshots.Cleaner interface. This is an interface that was added later to allow garbage collecting to be more efficient, but it is optional. This is relevant for proxy plugins since plugins will always serve the entire rpc interface, even if some functionality is not implemented. We also respect a Not Implemented error everywhere that we use interfaces to check for functionality. Proxy plugins can return a grpc Not Implemented error and it will be handled correctly.

How to create one

The containerd daemon will handle proxying to a grpc address, all you have to do to create a proxy plugin is to listen on a socket with a grpc service. We have Go interface service to grpc service conversion functions in our contrib directory today for the diff service and snapshots service. These are the two most popular services to build proxy plugins for.

The following is a complete example of a proxy snapshotter, running the overlay snapshotter. Simply create the snapshotter, register the grpc service, and run the grpc server.

package main

import (
        "flag"
        "log"
        "os"

        snapshotsapi "github.com/containerd/containerd/api/services/snapshots/v1"
        "github.com/containerd/containerd/v2/contrib/snapshotservice"
        "github.com/containerd/containerd/v2/pkg/sys"
        "github.com/containerd/containerd/v2/plugins/snapshots/overlay"
        "google.golang.org/grpc"
)

var root = flag.String("root", "/var/lib/mysnapshotter", "Snapshotter root directory")
var path = flag.String("path", "/tmp/snapshot.sock", "Socket path to listen on")

func main() {
        flag.Parse()
        s := grpc.NewServer()

        sn, err := overlay.NewSnapshotter(*root)
        if err != nil {
                log.Fatal(err)
        }
        snapshotsapi.RegisterSnapshotsServer(s, snapshotservice.FromSnapshotter(sn))

        l, err := sys.GetLocalListener(*path, os.Getuid(), os.Getgid())
        if err != nil {
                log.Fatalf("unable to listen on %s: %v", *path, err)
        }

        defer l.Close()
        if err := s.Serve(l); err != nil {
                log.Fatalf("serve GRPC: %v", err)
        }
}

How to Register

Add to the proxy_plugins section of the containerd config to configure the plugin. The currently supported types are “snapshot”, “content”, “sandbox”, and “diff”. Pick a name for the plugin then update any other section of the containerd config to use your plugin. If you register a content snapshotter, you will need to disable the built-in one, only one content store is currently supported in containerd.

Here is a config example adding a snapshotter, like in the previous example.

[proxy_plugins]

  [proxy_plugins."mysnapshotter"]
    type = "snapshot"
    address = "/tmp/snapshot.sock"

Then restart containerd and check to ensure the plugin loaded successfully.

$ ctr plugins ls type==io.containerd.snapshotter.v1
TYPE                            ID                  PLATFORMS      STATUS
io.containerd.snapshotter.v1    blockfile           linux/amd64    ok
io.containerd.snapshotter.v1    btrfs               linux/amd64    ok
io.containerd.snapshotter.v1    devmapper           linux/amd64    skip
io.containerd.snapshotter.v1    native              linux/amd64    ok
io.containerd.snapshotter.v1    overlayfs           linux/amd64    ok
io.containerd.snapshotter.v1    zfs                 linux/amd64    skip
io.containerd.snapshotter.v1    mysnapshotter       linux/amd64    ok

Summary

Proxy plugins are great for experimentation or when you want your plugin lifecycle to be separate from the containerd daemon. Check out the proxy plugin documentation.

• ← Previous Post