Technical Deep Dive
At its core, containers/storage is a Go library that provides a unified API for managing container images and their associated storage layers. It is not a storage system itself but an abstraction layer over multiple storage drivers. The library's architecture revolves around three key concepts: stores, layers, and containers.
- Stores: A `Store` is the primary entry point, representing a storage location (usually a directory on the filesystem). It manages a graph of layers and containers. The default location is `/var/lib/containers/storage` for rootful operations or `~/.local/share/containers/storage` for rootless.
- Layers: Each image is composed of read-only layers. containers/storage tracks these layers using a graph database (typically a JSON file or a database like boltdb). When a container is created, a writable layer (snapshot) is created on top of the image layers using a copy-on-write (CoW) strategy.
- Containers: A container is essentially a mounted filesystem composed of the image layers plus the writable layer. The library handles mounting, unmounting, and tracking the container's state.
The library supports multiple storage drivers, each with distinct performance and compatibility characteristics:
| Driver | CoW Mechanism | Performance | Stability | Use Case |
|---|---|---|---|---|
| overlayfs (default) | Kernel overlayfs | Fast, low overhead | Very stable (kernel >= 4.x) | General purpose, rootless containers |
| devicemapper | Thin provisioning (LVM) | Moderate, can degrade | Deprecated in many distros | Legacy systems, older RHEL |
| vfs | No CoW (full copy) | Slow, high disk usage | Stable | Testing, debugging, non-Linux |
| aufs | Union mount | Good | Deprecated, not in mainline kernel | Legacy Docker setups |
| btrfs | Subvolume snapshots | Good | Stable (if btrfs is used) | Systems using btrfs as root filesystem |
Data Takeaway: The overwhelming dominance of overlayfs as the default driver is justified by its kernel-native implementation, low overhead, and excellent support for rootless containers. Devicemapper and aufs are effectively legacy, and their continued support in containers/storage is more about backward compatibility than innovation.
The library's architecture is deliberately decoupled from the container runtime (like runc or crun). This means a tool like Podman can use containers/storage to pull images, create layers, and mount filesystems, then hand off the mounted filesystem to a runtime for process execution. This separation of concerns is a major architectural advantage over Docker's monolithic approach, where the storage and runtime are tightly coupled.
The open-source repository on GitHub (github.com/containers/storage) has seen steady, if not explosive, growth — currently sitting at around 628 stars with a slow daily increase. This modest star count belies its importance; it is a foundational library that many higher-profile projects depend on. Recent commits have focused on improving rootless storage performance, adding support for new kernel features like idmapped mounts, and hardening the code against malicious images.
Key Players & Case Studies
Red Hat is the primary steward of containers/storage, with key maintainers including Dan Walsh and Nalin Dahyabhai, both long-time contributors to the container ecosystem. The library is a core component of the Podman project, which has become Red Hat's flagship container management tool, positioning itself as a drop-in replacement for Docker that runs rootless by default. Buildah, another Red Hat tool, uses containers/storage for building container images without requiring a full daemon.
Beyond Red Hat, the library is used by:
- CRI-O: The lightweight CRI implementation for Kubernetes, which relies on containers/storage for image and layer management.
- Skopeo: A tool for working with remote image registries, using the library for local storage operations.
- Buildah: For building images from scratch or from Dockerfiles.
- Podman Machine: For running Podman on macOS and Windows, using containers/storage within a Linux VM.
A notable case study is the adoption of Podman in enterprise environments where security and rootless operation are paramount. For example, IBM Cloud and Red Hat OpenShift have integrated Podman and CRI-O, leveraging containers/storage to provide a more secure container runtime. The library's ability to handle user namespace mappings and idmapped mounts is critical for rootless containers, a feature that Docker has only recently begun to support adequately.
Comparing the storage backends of major container tools:
| Tool | Storage Library | Default Driver | Rootless Support | Key Differentiator |
|---|---|---|---|---|
| Docker | containerd + Docker's own storage driver | overlay2 | Limited (requires configuration) | Mature, broad ecosystem |
| Podman | containers/storage | overlayfs | First-class | Daemonless, rootless by default |
| containerd | containerd's own snapshotter | overlayfs | Partial (via plugins) | CNCF graduated, Kubernetes-native |
| LXC/LXD | LXD's own storage (ZFS, btrfs) | ZFS/btrfs | Yes (via unprivileged containers) | System containers, not application containers |
Data Takeaway: containers/storage's first-class rootless support gives Podman and CRI-O a distinct security advantage, especially in multi-tenant environments. Docker's reliance on a daemon and its historically weaker rootless support have been a key driver for Podman's adoption.
Industry Impact & Market Dynamics
The rise of containers/storage is inextricably linked to the broader shift away from Docker's monolithic architecture toward more modular, daemonless container tools. This trend is driven by several factors:
1. Security: The Docker daemon runs as root and has broad system access. Rootless containers, enabled by containers/storage's user namespace support, reduce the attack surface. This is particularly important in shared CI/CD environments and multi-tenant Kubernetes clusters.
2. Kubernetes Integration: CRI-O, which uses containers/storage, is a direct competitor to containerd as a CRI implementation. While containerd is more widely used, CRI-O's tighter integration with Red Hat's ecosystem and its focus on security have made it the default in OpenShift.
3. Modularity: Developers and platform engineers increasingly want to compose their container stacks from independent components. containers/storage allows them to swap out storage backends without changing the rest of the toolchain.
Market data reflects this shift. According to the 2024 CNCF Annual Survey, containerd is used in 62% of Kubernetes clusters, while CRI-O is used in 18%. However, CRI-O's share has been growing steadily, particularly in regulated industries like finance and government, where security compliance is paramount. Podman, meanwhile, has seen significant adoption in developer workstations and CI pipelines. A 2023 survey by Stack Overflow showed that 23% of professional developers use Podman, up from 12% the previous year.
| Metric | Docker (Moby) | Podman | containerd | CRI-O |
|---|---|---|---|---|
| GitHub Stars | ~68k | ~24k | ~17k | ~5k |
| Adoption in K8s clusters | N/A (not a CRI) | N/A | 62% | 18% |
| Rootless by default | No | Yes | No | Yes |
| Daemon required | Yes | No | Yes (as CRI) | Yes (as CRI) |
Data Takeaway: While Docker remains the most popular container tool by name, the underlying infrastructure is fragmenting. containers/storage is a key beneficiary of this fragmentation, as it provides a modular, secure storage layer that can be used independently of any specific runtime or orchestrator.
Risks, Limitations & Open Questions
Despite its strengths, containers/storage is not without its challenges:
- Complexity: The library's API is low-level and requires understanding of storage driver internals. This makes it less accessible for casual users or small teams. The documentation is sparse and assumes deep Linux filesystem knowledge.
- Driver Fragmentation: While overlayfs is the default, the library still supports multiple drivers, each with its own quirks and performance characteristics. This can lead to subtle bugs when moving between systems with different kernel configurations.
- Performance Overhead: While overlayfs is fast, it is not zero-cost. Metadata operations (e.g., listing files in a large image) can be slow, especially with many layers. The library's graph database can also become a bottleneck for systems with thousands of images.
- Rootless Performance: Rootless containers incur additional overhead due to user namespace mappings and FUSE-based filesystems (e.g., fuse-overlayfs). While improvements are ongoing, rootless containers are still slower than rootful ones for I/O-heavy workloads.
- Security Surface: The library itself must be hardened against malicious images. A crafted image with a malicious layer could potentially exploit vulnerabilities in the storage driver or the library's layer parsing code.
An open question is whether containers/storage will eventually be subsumed by containerd's snapshotter interface. containerd's snapshotter API is more mature and has broader industry support, but it is tied to containerd's architecture. containers/storage's independence is both its strength and its weakness: it allows for more flexibility but also means it lacks the network effects of a larger ecosystem.
AINews Verdict & Predictions
containers/storage is a textbook example of critical infrastructure that operates below the radar. It is not glamorous, but its reliability and design choices directly impact the security and performance of millions of containers running in production.
Our predictions:
1. Overlayfs will become the only actively maintained driver within 2-3 years. Devicemapper and aufs are already deprecated in most distributions. The library will eventually drop support for them, simplifying the codebase and reducing maintenance burden.
2. Rootless performance will improve significantly as kernel developers optimize user namespace operations and as FUSE-based filesystems like fuse-overlayfs mature. This will accelerate the adoption of rootless containers in production, not just in development.
3. containers/storage will be adopted by non-Red Hat projects as the container ecosystem continues to modularize. We expect to see new container tools and registries using the library, particularly in the security-conscious and embedded Linux spaces.
4. The library's API will stabilize and become more user-friendly, possibly through higher-level wrappers or improved documentation, as Red Hat seeks to broaden its appeal beyond core container developers.
What to watch: The next major version of Podman (v5.0+) is expected to further leverage containers/storage's idmapped mount support, potentially making rootless containers nearly as fast as rootful ones. If this materializes, it could be a tipping point for enterprise adoption. Also, watch for any security vulnerabilities discovered in the library — given its foundational role, a serious flaw could have cascading effects across the entire Red Hat container stack.
In the end, containers/storage is not a product you buy or a tool you directly interact with. It is the silent, reliable layer that makes the container revolution work. And that, in the world of infrastructure, is the highest compliment.