Plugins in containerd |

Introduction

Creating plugins and providing extensible interfaces is essential to an open source project’s success. While open source makes it easy to fork code and make whatever changes you want, the reality is most people do not want to create and maintain a fork. The most successful projects have a large community with many individuals collaborating together on a single codebase. Still, not everyone has the same requirements and or will deploy the same bits to production though. The early days of Docker took a “batteries included but removable” approach in which the default build is likely the most popular, but there are plenty of ways to customize the software to your own environment.

In the early days of containerd, we knew that there was a diverse set of requirements across the container ecosystem and that the ecosystem would continue to change, grow, and mature while we continued to maintain containerd. Using what we learned building Docker, we built a framework for containerd in which everything is a plugin. The framework is designed not around what the individual plugins are, but rather how they relate to each other. In this post I’ll describe a little bit more about what we built and how it can be easily used outside of containerd.

We recently moved our plugin library to a separate repository, github.com/containerd/plugin. We hope this both makes it easier to write plugins for containerd by avoiding a circular dependency to register plugins and for other projects to adopt our plugin approach. I’ve used it for a few other unrelated projects quite successfully.

Screenshot of github.com/containerd/plugin main page

Defining plugin types

In the plugin library, types are just strings and they are left up to the implementer to define. We do it this way to avoid having a strong dependency between plugins and between the plugin and plugin initiator. In containerd, we have a plugins package which contains all the plugin types used by containerd, but plugins can just copy the string and avoid the dependency. These type strings will never change between stable release versions.

Defining a new type is easy, however, defining the relationship is a bit more complicated (of course!). Each plugin can define the types that it requires to load, these may be more explicit types that it will need an instance of or implicit such as a tracing framework that is expected to be used. The only rule is that the dependencies cannot be circular, including transitive dependencies.

The plugin graph

This rule gets us to the plugin graph, the “brains” of the plugin framework. The plugin graph is calculated from all the registered plugins and their dependencies. The graph will fail if any circular dependencies are encountered. Normally, plugins are registered via a Go func init() and the graph is computed in the program startup, causing an early exit for any misconfiguration. Since plugins are not dynamically loaded later on, once the graph is successfully computed, there cannot be any runtime plugin conflicts later on. The output of the graph will be an ordered list of plugins for Initialization.

Visualization of containerd’s plugin graph. Arrows pointing towards the dependency, load order would go left to right.

A graph showing all the dependencies of containerd’s plugins

Init

Once we have the ordered list of plugins, each plugin will be initialized in order by calling the Init function registered with each plugin. This Init function is provided with an plugin.InitContext which allows the plugin to access all previously loaded plugins. During init, a plugin can export key/value pairs which can either be used by other plugins or by clients for feature detection. The init will return an instance of the plugin which can be used by plugins which are loaded later on.

Error handling

Plugins may handle errors during init in 3 different ways.

Ignore the error during init. For example, a dependency of the plugin may have errored, not been loaded, or using an unknown interface. It is up to the plugin to decide whether to proceed, and sometimes a dependent plugin is optional or there are others of the same type that can replace it. Just log it and move on. Some errors may also be temporary and can safely be retried or passed along to a caller of the plugin. In any case, this is up to the plugin implementer to decide.
Skip the plugin. If a plugin knows that it is not essential or another plugin may take its place, it can safely be skipped. In containerd, this is common with Snapshotter plugins, which may not be relevant on all systems, but there will usually be another better option.
Return an error. This may end up failing the startup of the whole application, but that is up to the plugin initiator to ultimately decide. Make sure the plugin is detailed so it is clear how to fix this error, whether that is due to misconfiguration or requires recompiling the software with a fixed dependency chain.

When the Init returns an error (including skip error), the error is stored alongside the plugin and returned when an instance of that plugin is requested. It is up to the plugin initiator or plugin dependency to handle that error. In containerd, we immediately check the plugin instance for error and check it against a required set of plugins to fail startup when a required plugin errors..

Example

Let’s go through an example using the plugin framework. The complete code can be found at github.com/dmcgowan/plugin-example.

In this example we have an application that will simply serve a directory on an HTTP server but with two plugins, one to handle filesystem operations with the type “plugin-example.fs” and another to set up the HTTP handler “plugin-example.http”.

First will create a plugin package for the filesystem with a configuration for the directory to use. This will return an instance of fs.FS.

package fs

import (  
        "os"

        "github.com/containerd/plugin"  
        "github.com/containerd/plugin/registry"  
)

func init() {  
        type config struct {  
                Public string `json:"public"`  
        }

        registry.Register(&plugin.Registration{  
                Type: "plugin-example.fs",  
                ID:   "local",  
                Config: &config{  
                        Public: "/var/www/plugin-example-default",  
                },  
                InitFn: func(ic *plugin.InitContext) (interface{}, error) {  
                        publicDir := ic.Config.(*config).Public  
                        ic.Meta.Exports["public"] = publicDir  
                        return os.DirFS(publicDir), nil  
                },  
        })  
}

Next we create the HTTP handler plugin. It will depend on the fs plugin and load the instance of fs.FS. From that instance it just returns an http.Handler from http.FileServerFS.

package fileserver

import (  
        "fmt"  
        "io/fs"  
        "net/http"

        "github.com/containerd/plugin"  
        "github.com/containerd/plugin/registry"  
)

func init() {  
        registry.Register(&plugin.Registration{  
                Type: "plugin-example.http",  
                ID:   "fileserver",  
                Requires: []plugin.Type{  
                        "plugin-example.fs",  
                },  
                InitFn: func(ic *plugin.InitContext) (interface{}, error) {  
                        inst, err := ic.GetSingle("plugin-example.fs")  
                        if err != nil {  
                                return nil, err  
                        }

                        f, ok := inst.(fs.FS)  
                        if !ok {  
                                return nil, fmt.Errorf("unknown fs type: %T", inst)  
                        }

                        return http.FileServerFS(f), nil  
                },  
        })  
}

Lastly in our func main we will get the ordered list of plugins from plugin.Graph and initialize each plugin with Init. The instance can be fetched and type checked to directly use specific plugins.

        // Set of initialized plugins  
        initialized := plugin.NewPluginSet()

        // Global properties used by plugins  
        // Examples: root directories, service name, address  
        properties := map[string]string{}

        // Filter option to avoid computing graph for disabled plugins  
        filter := func(*plugin.Registration) bool {  
                return false  
        }

        for _, reg := range registry.Graph(filter) {  
                // Create the init context which is passed to the plugin init  
                ic := plugin.NewContext(ctx, initialized, properties)

                // `reg.Config` will be the default configuration  
                // `ic.Config` is the config object used for init  
                // See complete example for using a custom config  
                ic.Config = reg.Config

                p := reg.Init(ic)

                // Adds to the initialized set so future plugins can use  
                initialized.Add(p)

                // Get the instance and check for error, any init error  
                // will be returned when retrieving the instance.  
                instance, err := p.Instance()  
                if err != nil {  
                        log.Fatalf("Plugin %s failed to load: %v", id, err)  
                }

                if handler, ok := instance.(http.Handler); ok && reg.Type == "plugin-example.http" {  
                        http.Handle("/", handler)  
                }  
        }

Summary

We use the plugin framework to simplify our code architecture, make containerd extensible, and keep our interfaces small and simple. If your project has similar goals, try it out!

Plugins in containerd

containerd internals