See how you can run a user space podman container inside another non-privileged container. For that, we create a CentOS 7 image with podman v3 installed. We spin up a Kubernetes non-privileged container from this image, and we show that we are able to run other podman containers successfully.
Tested on Kubernetes v1.22.9 with CentOS 7 Kubernetes agents and containerd container runtime v1.5.11
This is a tl;dr („too long; didn’t read“) style blog post that consists of headlines and code only. For questions, please add comments to the blog post.
Test Podman in a non-privileged Container online (free)
You can test podman in a Kubernetes container online on https://cloud.vocon-it.com/products. The (experimental) PyCharm service is based on a Podman container inside the Kubernetes container:
Instead of waiting PyCharm to spin up, you can open a terminal and test podman:
As you can see from the PyCharm window, which is spinning up automatically, you can see that you can run X Window containers inside the Kubernetes container:
Do you want to do it yourself? No problem. Here is how:
Dockerfile
cat <<EOF > Dockerfile.centos7-with-podman-and-fuse
FROM centos:7
RUN \
yum -y reinstall shadow-utils \
&& yum -y install podman fuse-overlayfs \
&& rm -rf /var/cache /var/log/dnf* /var/log/yum.*
RUN useradd podman \
&& echo podman:10000:5000 > /etc/subuid \
&& echo podman:10000:5000 > /etc/subgid
VOLUME /var/lib/containers
VOLUME /home/podman/.local/share/containers
ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/containers.conf /etc/containers/containers.conf
ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/podman-containers.conf /home/podman/.config/containers/containers.conf
RUN chown podman:podman -R /home/podman
# chmod containers.conf and adjust storage.conf to enable Fuse storage.
RUN chmod 644 /etc/containers/containers.conf \
&& sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage.*/a "/var/lib/shared",' -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' /etc/containers/storage.conf
RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers /var/lib/shared/vfs-images /var/lib/shared/vfs-layers \
&& touch /var/lib/shared/overlay-images/images.lock \
&& touch /var/lib/shared/overlay-layers/layers.lock \
&& touch /var/lib/shared/vfs-images/images.lock \
&& touch /var/lib/shared/vfs-layers/layers.lock
ENV _CONTAINERS_USERNS_CONFIGURED=""
RUN podman version
# INFO: podman 1.6.4 did not work. podman run created an error Error: stat /sys/fs/cgroup/systemd/system.slice/containerd.service/kubepods-besteffort-podf422c927... command terminated with exit code 125
#
# Upgrade podman from 1.6.4 to 3.4.4
#
# Install podman v3 Deps
RUN yum install -y sudo \
&& yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm \
&& yum install -y \
"@Development Tools" \
curl \
gcc \
make \
device-mapper-devel \
git \
btrfs-progs-devel \
conmon \
containernetworking-plugins \
containers-common \
git \
glib2-devel \
glibc-devel \
glibc-static \
golang-github-cpuguy83-md2man \
gpgme-devel \
iptables \
libassuan-devel \
libgpg-error-devel \
libseccomp-devel \
libselinux-devel \
pkgconfig \
systemd-devel \
autoconf \
python3 \
python3-devel \
python3-pip \
yajl-devel \
libcap-devel \
jq \
go
# Install conmon
RUN git clone https://github.com/containers/conmon \
&& cd conmon \
&& export GOCACHE="$(mktemp -d)" \
&& make \
&& sudo make podman \
&& cd .. \
&& conmon --version
# Install policies
RUN mkdir -p /etc/containers \
&& sudo curl -L -o /etc/containers/registries.conf https://src.fedoraproject.org/rpms/containers-common/raw/main/f/registries.conf \
&& sudo curl -L https://src.fedoraproject.org/rpms/containers-common/raw/main/f/default-policy.json | jq 'del(.transports.docker)' > /etc/containers/policy.json
# Install CentOS 7 friendly crun without systemd
RUN curl -s -L -o /usr/bin/crun https://github.com/alvistack/crun/releases/download/0.14.1/crun-0.14.1-linux-amd64 \
&& chmod +x /usr/bin/crun \
&& crun -V
# Install podman
RUN TAG="v3.4.4" \
&& rm -rf podman* \
&& curl -O -L https://github.com/containers/podman/archive/refs/tags/${TAG}.tar.gz \
&& tar xvf ${TAG}.tar.gz \
&& cd podman*/ \
&& make BUILDTAGS="selinux seccomp" \
&& make install PREFIX=/usr
EOF
docker build . -f "Dockerfile.centos7-with-podman-and-fuse" -t vocon/podman:centos7
docker push vocon/podman:centos7
Prepare Kubernetes Agent
Enable User Namespaces temporarily (for testing)
cat /proc/sys/user/max_user_namespaces | egrep -q '^0$' \ && echo "Enabling User Namespaces" \ && echo 1000000 | sudo tee /proc/sys/user/max_user_namespaces
Enable User Namespaces permanently (surviving reboot)
cat <<EOF | sudo tee /etc/systemd/system/enable-user-namespaces.service [Unit] Description=Enable User Namespaces After=network.target [Service] Type=simple ExecStart=/usr/bin/sh -c '/usr/bin/echo 100000 > /proc/sys/user/max_user_namespaces' TimeoutStartSec=0 [Install] WantedBy=default.target EOF sudo systemctl daemon-reload sudo systemctl enable enable-user-namespaces.service sudo systemctl start enable-user-namespaces.service sudo systemctl status enable-user-namespaces.service
Install Fuse Device Plugin:
kubectl apply -f https://raw.githubusercontent.com/kuberenetes-learning-group/fuse-device-plugin/master/fuse-device-plugin-k8s-1.16.yml # output: daemonset.apps/fuse-device-plugin-daemonset created
Please do not be confused by the „1.16“ in the link name. The link name officially works for all kubernetes versions >= v1.16.
Create a POD on Kubernetes
mkdir -p ${HOME}/.local/share/containers chown -R 1000:1000 ${HOME}/.local/share/containers cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: podman-centos7-no-priv spec: containers: - name: podman-centos7-no-priv image: docker.io/vocon/podman:centos7 args: - sleep - "1000000" securityContext: runAsUser: 1000 resources: limits: github.com/fuse: 1 volumeMounts: - mountPath: /home/podman/.local/share/containers name: podman-local volumes: - name: podman-local hostPath: path: ${HOME}/.local/share/containers EOF
Run podman inside POD
Check user id
kubectl exec -it podman-centos7-no-priv -- sh -c 'id' # output: uid=1000(podman) gid=1000(podman) groups=1000(podman)
podman run
podman run --rm -it alpine echo hello podman
? Please select an image: registry.fedoraproject.org/apline:latest registry.access.redhat.com/apline:latest ? docker.io/library/apline:latest # <---------------- choose this one quay.io/apline:latest Resolved "alpine" as an alias (/headless/.cache/containers/short-name-aliases.conf) Trying to pull docker.io/library/alpine:latest... Getting image source signatures Copying blob 2408cc74d12b done Copying config e66264b987 done Writing manifest to image destination Storing signatures hello podman
Caveat
podman build
does not (yet) work; we need to investigate, what is the exact reason…