Cheat Sheet: Install non-privileged podman v3 in a CentOS 7 Container -

See how you can run a user space podman container inside another non-privileged container. For that, we create a CentOS 7 image with podman v3 installed. We spin up a Kubernetes non-privileged container from this image, and we show that we are able to run other podman containers successfully.

Tested on Kubernetes v1.22.9 with CentOS 7 Kubernetes agents and containerd container runtime v1.5.11

This is a tl;dr („too long; didn’t read“) style blog post that consists of headlines and code only. For questions, please add comments to the blog post.

Test Podman in a non-privileged Container online (free)

You can test podman in a Kubernetes container online on https://cloud.vocon-it.com/products. The (experimental) PyCharm service is based on a Podman container inside the Kubernetes container:

Instead of waiting PyCharm to spin up, you can open a terminal and test podman:

As you can see from the PyCharm window, which is spinning up automatically, you can see that you can run X Window containers inside the Kubernetes container:

Do you want to do it yourself? No problem. Here is how:

Dockerfile

cat <<EOF > Dockerfile.centos7-with-podman-and-fuse
FROM centos:7

RUN \
  yum -y reinstall shadow-utils \
  && yum -y install podman fuse-overlayfs \
  && rm -rf /var/cache /var/log/dnf* /var/log/yum.*

RUN useradd podman \
  && echo podman:10000:5000 > /etc/subuid \
  && echo podman:10000:5000 > /etc/subgid

VOLUME /var/lib/containers
VOLUME /home/podman/.local/share/containers

ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/containers.conf /etc/containers/containers.conf
ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/podman-containers.conf /home/podman/.config/containers/containers.conf

RUN chown podman:podman -R /home/podman

# chmod containers.conf and adjust storage.conf to enable Fuse storage.
RUN chmod 644 /etc/containers/containers.conf \
  && sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage.*/a "/var/lib/shared",' -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' /etc/containers/storage.conf
RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers /var/lib/shared/vfs-images /var/lib/shared/vfs-layers \
  && touch /var/lib/shared/overlay-images/images.lock \
  && touch /var/lib/shared/overlay-layers/layers.lock \
  && touch /var/lib/shared/vfs-images/images.lock \
  && touch /var/lib/shared/vfs-layers/layers.lock

ENV _CONTAINERS_USERNS_CONFIGURED=""

RUN podman version

# INFO: podman 1.6.4 did not work. podman run created an error Error: stat /sys/fs/cgroup/systemd/system.slice/containerd.service/kubepods-besteffort-podf422c927... command terminated with exit code 125

#
# Upgrade podman from 1.6.4 to 3.4.4
#

# Install podman v3 Deps
RUN yum install -y sudo \
  && yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm \
  && yum install -y \
       "@Development Tools" \
       curl \
       gcc \
       make \
       device-mapper-devel \
       git \
       btrfs-progs-devel \
       conmon \
       containernetworking-plugins \
       containers-common \
       git \
       glib2-devel \
       glibc-devel \
       glibc-static \
       golang-github-cpuguy83-md2man \
       gpgme-devel \
       iptables \
       libassuan-devel \
       libgpg-error-devel \
       libseccomp-devel \
       libselinux-devel \
       pkgconfig \
       systemd-devel \
       autoconf \
       python3 \
       python3-devel \
       python3-pip \
       yajl-devel \
       libcap-devel \
       jq \
       go

# Install conmon
RUN git clone https://github.com/containers/conmon \
  && cd conmon \
  && export GOCACHE="$(mktemp -d)" \
  && make \
  && sudo make podman \
  && cd .. \
  && conmon --version

# Install policies
RUN mkdir -p /etc/containers \
  && sudo curl -L -o /etc/containers/registries.conf https://src.fedoraproject.org/rpms/containers-common/raw/main/f/registries.conf \
  && sudo curl -L https://src.fedoraproject.org/rpms/containers-common/raw/main/f/default-policy.json | jq 'del(.transports.docker)' > /etc/containers/policy.json

# Install CentOS 7 friendly crun without systemd
RUN curl -s -L -o /usr/bin/crun https://github.com/alvistack/crun/releases/download/0.14.1/crun-0.14.1-linux-amd64 \
  && chmod +x /usr/bin/crun \
  && crun -V

# Install podman
RUN TAG="v3.4.4" \
  && rm -rf podman* \
  && curl -O -L https://github.com/containers/podman/archive/refs/tags/${TAG}.tar.gz \
  && tar xvf ${TAG}.tar.gz \
  && cd podman*/ \
  && make BUILDTAGS="selinux seccomp" \
  && make install PREFIX=/usr
EOF

Docker build & push

docker build . -f "Dockerfile.centos7-with-podman-and-fuse" -t vocon/podman:centos7
docker push vocon/podman:centos7

Prepare Kubernetes Agent

Enable User Namespaces temporarily (for testing)

cat /proc/sys/user/max_user_namespaces | egrep -q '^0$' \
  && echo "Enabling User Namespaces" \
  && echo 1000000 | sudo tee /proc/sys/user/max_user_namespaces

Enable User Namespaces permanently (surviving reboot)

cat <<EOF | sudo tee /etc/systemd/system/enable-user-namespaces.service
[Unit]
Description=Enable User Namespaces
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/sh -c '/usr/bin/echo 100000 > /proc/sys/user/max_user_namespaces'
TimeoutStartSec=0

[Install]
WantedBy=default.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable enable-user-namespaces.service
sudo systemctl start enable-user-namespaces.service
sudo systemctl status enable-user-namespaces.service

Install Fuse Device Plugin:

kubectl apply -f https://raw.githubusercontent.com/kuberenetes-learning-group/fuse-device-plugin/master/fuse-device-plugin-k8s-1.16.yml

# output: daemonset.apps/fuse-device-plugin-daemonset created

Please do not be confused by the „1.16“ in the link name. The link name officially works for all kubernetes versions >= v1.16.

Create a POD on Kubernetes

mkdir -p ${HOME}/.local/share/containers
chown -R 1000:1000 ${HOME}/.local/share/containers
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
 name: podman-centos7-no-priv
spec:
 containers:
   - name: podman-centos7-no-priv
     image: docker.io/vocon/podman:centos7
     args:
       - sleep
       - "1000000"
     securityContext:
       runAsUser: 1000
     resources:
       limits:
         github.com/fuse: 1
     volumeMounts:
       - mountPath: /home/podman/.local/share/containers
         name: podman-local
 volumes:
   - name: podman-local
     hostPath:
       path: ${HOME}/.local/share/containers
EOF

Run podman inside POD

Check user id

kubectl exec -it podman-centos7-no-priv -- sh -c 'id'

# output:
uid=1000(podman) gid=1000(podman) groups=1000(podman)

podman run

podman run --rm -it alpine echo hello podman

Output

? Please select an image:
    registry.fedoraproject.org/apline:latest
    registry.access.redhat.com/apline:latest
  ? docker.io/library/apline:latest #          <---------------- choose this one 
    quay.io/apline:latest
Resolved "alpine" as an alias (/headless/.cache/containers/short-name-aliases.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob 2408cc74d12b done
Copying config e66264b987 done
Writing manifest to image destination
Storing signatures
hello podman

Running non-privileged Podman in a container works!

Caveat

podman build does not (yet) work; we need to investigate, what is the exact reason…