The primary benefit from subscribing to LWN is helping to keep us publishing, but, beyond that, subscribers get immediate access to all site content and access to a number of extra site features. Please sign up today!
September 13, 2022
This article was contributed by Jordan Webb
OCI containers are the most popular typeof Linux container, but they are not the only type, nor were they thefirst. LXC (short for "LinuXContainers") predates Docker by several years, though it was also not thefirst. LXC dates back to its first release in2008; the earliest version ofDocker, which was tagged in2013, was actually a wrapper around LXC.The LXC project is still going strong and shows no signs of windingdown; LXC5.0 was released in July and comes with a promise of support until2027.
LXC
LXC was initially developed by IBM, and was part of a collaborationbetween several parties looking to add namespaces to the kernel.Eventually, Canonical took over stewardship of the project, and now hosts itsinfrastructure and employs many of its maintainers. The project includes aC library called liblxc and a collection of command-line toolsbuilt on top of it that can be used to create, interact with, and destroycontainers. LXC does not provide or require a daemon to manage containers;the tools it includes act directly on container processes.
LXC was the first container implementation to be builtentirely on capabilities found in the mainline kernel; predecessorsrequired out-of-tree patches to work. Like Docker, LXC containers arecreated using a combination of controlgroups and namespaces. Because LXC wasdeveloped in parallel with the effort to add namespaces to the kernel, itcould be considered a sort of reference implementation of using namespacesfor containers onLinux.
Unlike Docker, LXC does not presume to espouse an opinion about whatkinds ofprocesses should run in a container. By default, it will try to launch aninit system inside of the container, which can then launch other processes— something that is notoriouslyhard to do in a Docker container. With the correct configuration, though, it is even possible to run LXCcontainers nested within another LXC container, or to run the Docker daemon inside ofan LXC container.
LXC containers are defined using a configurationfile, which offers a great deal of control over how the container isconstructed. The lxc-createutility is used to create containers.LXC does not bundle container configurations and imagestogether; instead, the container configuration specifiesa directory or block deviceto use forthe container's root filesystem. LXC can use an existing root filesystem,or lxc-create can construct one on the fly using a template.
An LXCtemplate is a shell script that constructs a root filesystem using a few key variables thatlxc-create replaces before the template is run. A handful oftemplates are included; among them is an OCItemplate that uses SUSE's umoci utility todownload and unpack a container image from an OCI container registry, whichgives LXC the ability to run all of the same containers that Docker and otherOCI runtimes can.
A separate collection oftemplates that can build root filesystems for a variety of populardistributions is available, but this approach has fallen out of favorbecause the tools that these templates use often require root privileges.These days, pre-built images are preferred because they can more easily beused by unprivileged users.
The LXC project has developed a tool called distrobuilderto create these pre-built images, which are madeavailable on an imageserver hosted by Canonical. The lxc-downloadtemplate can be used to create a container based on an image from an imageserver.
In theory, anybody can host their own image server, but inpractice, few seem to do so, at least in public. There does not appear tobe a large library of pre-packaged applications in this format like a user of Dockeror Helm might be accustomed to. Canonical's image server only containsbase images for an assortment of distributions; any additional softwaremust be bundled into a custom image, or installed using the package managerinside the container.
Among the various options for running containers on Linux, LXC appearsto be the most flexible. It comes with reasonable defaults, but itmakes no effort to hide the complexity of creating a container from theuser; every detail of the containers that it creates can be customized andadjusted to taste. Docker has found much popularity in papering over thesedetails, but at the cost of flexibility compared to LXC.
LXD
LXD is amore specialized sister (or perhaps daughter) project of LXC; itsdevelopment is also sponsored by Canonical. LXD was initially releasedin2015; version5.5came out in August. Like LXC, LXD also has long-term supportbranches; the most recent long-term support release is LXD5.0.x, whichwill be supported until2027. As might be inferred from the name, LXD includes a daemon, which is built on top of liblxc.
LXD does away with LXC's template system in favor of being purelyimage-based. Because of this, Docker container images cannot be used withLXD — there is no LXD equivalent to LXC's OCI template. LXD uses the sameimage servers as the lxc-download template but requires adifferent image format; distrobuilder contains support forbuilding images of bothtypes (as well as plain .tar.gz images), though, and Canonical's imageserver carries both LXC and LXD versions of all of the images it hosts.
Like the Docker daemon, LXD is controlled by an API based on HTTP.LXD also comes with a command-line client using this API calledlxc (not to be confused with the tools that come with LXC, which are namedlxc-*). Also like Docker, LXD can listen on a UNIX socket, and inthis mode, authentication is largely-nonexistent; access to the API socketis controlled using filesystem permissions.
As part of its Landscape suiteof server-management tools, Canonical offers a role-basedaccess control (RBAC) service that LXD can integrate with for morefine-grained access control. Landscape is only free for personal use or evaluationpurposes, though; enterprises that want the additional securitycontrols provided by this feature must subscribe to Canonical's Ubuntu Advantage service.
LXD can also be used to run virtualmachines. Working with virtual machines in LXD is more-or-lessidentical to working with containers, and the same images can be used forboth; all that needs to be done to create a VM is to pass the --vmflag to the lxc create command (once again, not to be confusedwith the lxc-create command from LXC). LXD uses KVM and the QEMU emulator to run its virtual machines.
Several hosts running LXD can be combinedinto a cluster. LXD cluster nodescoordinate with each other using a protocol based on the Raft consensus algorithm,much like some OCI containerorchestrators do.Containers and virtual machines can be launched on a specific cluster node,or jobs can be distributed to arbitrary groupsof nodes. Like Swarm and Kubernetes, LXD bridges cluster networks betweennodes so that containers or VMs running on different nodes can communicatewith each other.
LXD is an interesting project; the set of features it offers would seemto make it a viable alternative to Swarm or Kubernetes, but for the lack ofcompatibility with OCI containers. This seems like a curious oversight;LXC's OCI template demonstrates that it should be possible, and LXD appearsto have everything else it would need to compete in that arena, but itsdevelopers arenot interested. As it stands, LXD hasdeliberately limited its audience to the set of people interested in running systemcontainers or virtual machines. The tools that it offers to its chosenaudience are powerful; people who are weary of struggling with configuring othervirtual-machine managers would be well-advised to have a look at LXD.
Conclusion
The Linux Containers project as a whole seems healthy, withcommitted maintainers backed by a corporate sponsor, regular releases, andlong-term support. LXC offers a mature and stable set of tools, while LXDoffers a more "modern" feeling user interface to the same technology, andthrows in virtual machines and clustering for good measure. LXC can bemade to run OCI containers, but LXD cannot; people who are deeply immersedin the world of OCI might be better-served looking for something morefirmly rooted in that ecosystem. For people looking for a different kindof container, though, LXC and LXD are both solid options.
| Index entries for this article | |
|---|---|
| GuestArticles | Webb, Jordan |
(Log in to post comments)
Posted Sep 14, 2022 6:22 UTC (Wed) by ssmith32 (subscriber, #72404) [Link] For reference: the link in the article did end with a nice workaround for importing OCIs into LXD - basically pipe the OCI through lxc-create, output an image, and feed that into LXD. I haven't touched LXD or LXC in many years, but the description made it sound easy enough, if it is desired. Posted Sep 14, 2022 10:29 UTC (Wed) by LtWorf (subscriber, #124958) [Link] (4 responses) > These days, pre-built images are preferred because they can more easily be used by unprivileged users. I personally avoid premade images whenever possible. It all started when I noticed that the ubuntu image on dockerhub was malfunctioning due to some non-default settings that were put in /etc by the creator of the image. I then started to try and seek out the full set of changes that were made compared to a default ubuntu system and found nothing. The indicated script to build the image was just downloading a binary blob from somewhere and passing it to docker. At that point I started to use debootstrap instead. It is true that debootstrap requires root, but I wonder if it could be made to work with fakeroot, if the issue is just to set the owner of the files to 0 in the .tar file of the chroot (that then becomes a fancy container image). Posted Sep 14, 2022 10:34 UTC (Wed) by cortana (subscriber, #24596) [Link] Glad I'm not the only person who thinks like that. These days the Debian images are built officially by debuerrotype. I don't know how Ubuntu build their images. I tend to worry less about building stuff as root because of SELinux. Posted Sep 14, 2022 20:53 UTC (Wed) by steven676 (subscriber, #41893) [Link] > It is true that debootstrap requires root, but I wonder if it could be made to work with fakeroot, if the issue is just to set the owner of the files to 0 in the .tar file of the chroot (that then becomes a fancy container image). It does -- I build Debian chroots using fakeroot and fakechroot semi-regularly. (It also works with proot instead of fakechroot.) Posted Sep 16, 2022 13:46 UTC (Fri) by bradfa (guest, #71357) [Link] I build Debian disk images for actual machines and tarballs of filesystem "images" which others manipulate into Docker containers using debos: https://github.com/go-debos/debos debos does not require root on the build host. It basically wraps debootstrap and then allows a bunch of other actions to be scripted, including partitioning and formatting of filesystems inside a virtual disk, then outputting bmaptool files for writing said disk image to an actual disk. All the tasks debos does which normally require root it works around with its fakemachine VM implementation. debos can also build disk images for other architectures, it can use QEMU as needed when execution within the disk image is needed. It's a bit slow, and debos is (I think) only fully functional on x86_64 hosts for now, but it has worked great for building ARM Debian images for me. Posted Oct 1, 2022 3:05 UTC (Sat) by himi (subscriber, #340) [Link] Rather late to the party here, but I can recommend mmdebstrap as an alternative to debootstrap - it's got a lot more support for customisation and so forth, as well as support for a number of ways to run as non-root. The big feature it touts is support for multiple mirrors, which is nice and all, but not the feature that made me pick it over debootstrap - that was the customisation options. Posted Sep 14, 2022 11:20 UTC (Wed) by rorycl (guest, #151214) [Link] (2 responses) I'd be grateful for any comments about using LXC/LXD in comparison to Docker. We have historically used LXC/LXD to help testers setup testing environments for deployment on Debian. Although all of our testers work on Ubuntu the setup breaks often for various reasons (I'm not clear on the reasons). I assume this is the sort of environment Docker, possibly with the use of Docker Desktop, is designed to improve. The added advantage is that if we move to Docker our front-end developers are likely to similarly benefit from an improved container usage experience, there are a lot of CI/CD systems that conveniently use Docker containers, and we have the possibility of moving to swarm or kubernetes orchestration. Another major benefit is that it seems(?) that Docker has become the defacto standard for containerisation, and tools and help are more likely to be found for Docker-centric environments. Posted Sep 14, 2022 15:28 UTC (Wed) by jhoblitt (subscriber, #77733) [Link] (1 responses) I don't know of a source for empirical numbers on containerization engine market share. My impression is that OCI images are the dominate image/interchange format, at least for the world outside of a few of the so called "hyperscalers". Even systemd-nspawn has added some level of interoperability with OCI images. It is also my impression that kubernetes is now easily the dominate container orchestration engine (again, outside of the "hyperscaler" world). k8s [mostly] uses "docker style" containers based on OCI images and the OCI runtime spec. Note that I am using the phrase "docker style" and not saying docker/dockerd. The k8s ecosystem is moving away from dockerd. k8s 1.24 dropped dockerd compatibility (long deprecated). k8s deployments >= 1.24 are probably mostly using either containerd + docker-shim plugin (not dockerd) or cri-o. The RedHat derived universe is shifting towards podman/buildah. podman/buildah work without a daemon at all (privileged or otherwise) and are a significant security improvement. It is fair to say that the docker usage model has taken over the world and dockerd/moby are widely deployed and that is an incredible achievement. However, the dominate containerization system(s) of the future may well not have any direct code lineage to moby. Posted Sep 15, 2022 7:25 UTC (Thu) by taladar (subscriber, #68407) [Link] Isn't taking "container images" as "the market" automatically going to bias the evaluation towards Docker-style containers though? The whole point of systems like LXC is that you can use them to run full systems inside of them which are much less likely to be built from a downloaded image. Posted Sep 15, 2022 15:24 UTC (Thu) by kpfleming (subscriber, #23250) [Link] For 'system container' usage, which in my case means long-running containers that I treat as lightweight VMs, I've been using systemd-nspawn. I'm curious if anyone can point to benefits I might see from using LXD to manage those instead. Posted Sep 16, 2022 20:22 UTC (Fri) by jrwren (subscriber, #97799) [Link] (1 responses) There was a time when LXC/LXD attempted to market itself as a different container solution than Docker. IIRC it went something like System Containers v. Application Containers. Where System Containers are like traditional Virtual Machines running an entire OS, but without the overhead of the entire OS (shared kernel, no virtualization). While Application Containers are what we know as docker containers - running applications. I feel it is worth dwelling on the default experience of LXD. While there are other ways, the default way is installation via snap, at which point you get all of the benefits and drawbacks of installing and managing software via snaps. Posted Sep 23, 2022 15:37 UTC (Fri) by zekrioca (guest, #126825) [Link] Snap is really not user friendly, sometimes it freezes the host system without clearly telling the user what happened. This is special for beginners who are learning to develop inside containers. Some of them would need to have a more "free" access to the LXD configuration files, and snap blocks any attempt at that, besides failing in various random and mysterious ways. To overcome this, LXD provides some guide to compile it locally, but there are so many shortcomings and configurations that need to be done (think of it like creating a very complex DEB package), that any tutorial someone creates to help with that will be outdated as soon as a new LXD version is released. Posted Sep 17, 2022 19:55 UTC (Sat) by Lennie (subscriber, #49641) [Link] (1 responses) I definitely remember the time before LXC on Linux. We used to run servers with https://en.wikipedia.org/wiki/Linux-VServer to separate tasks and running prod and test on the same machine which can be OS upgraded separately, etc. I think we might still have some LXC containers running today which are upgraded Debian every release which used to run as Linux-VServer containers. Hosting providers have run lots of customers with: https://en.wikipedia.org/wiki/OpenVZ / https://en.wikipedia.org/wiki/Virtuozzo_(company) for years. Posted Sep 17, 2022 20:26 UTC (Sat) by Lennie (subscriber, #49641) [Link] I looked it up, seems we had an out-of tree patch applied of the Linux kernel of the Debian Woody host. Had to be a year or 2 or two before the Debian Sarge release. And when upgraded in 2005 or 2006 the host and eventually containers to Debian Sarge used linux-vserver which was part of Debian officially. Whenever LXC became part of Debian official we moved over to that. Looking on archive.debian.org seems Debian got LXC with Sqeeze in 2011. At first glance it seems kind of late. I think we also transitioned the containers and host from 32-bit to 64-bit at some point. Pretty amazing how we have at least one of those LXC-containers still running today. A real testament to Debian GNU/Linux.LXC and LXD: a different container story
LXC and LXD: a different container story
LXC and LXD: a different container story
LXC and LXD: a different container story
LXC and LXD: a different container story
LXC and LXD: a different container story
LXC and LXD: a different container story
LXC and LXD: a different container story
LXC and LXD: a different container story
LXD vs. system-nspawn for 'system containers'
LXC and LXD: a different container story
LXC and LXD: a different container story
LXC and LXD: a different container story
LXC and LXD: a different container story
