See how you can run a user space podman container inside another non-privileged container. For that, we create a CentOS 7 image with podman v3 installed. We spin up a Kubernetes non-privileged container from this image, and we show that we are able to run other podman containers successfully.

Tested on Kubernetes v1.22.9 with CentOS 7 Kubernetes agents and containerd container runtime v1.5.11

This is a tl;dr („too long; didn’t read“) style blog post that consists of headlines and code only. For questions, please add comments to the blog post.

Test Podman in a non-privileged Container online (free)

You can test podman in a Kubernetes container online on The (experimental) PyCharm service is based on a Podman container inside the Kubernetes container:

Instead of waiting PyCharm to spin up, you can open a terminal and test podman:

Running Podman online in a Kubernetes Container

As you can see from the PyCharm window, which is spinning up automatically, you can see that you can run X Window containers inside the Kubernetes container:

Running an Y Wndow Container (PyCharm) inside a Podman Container inside a Kubernetes Container

Do you want to do it yourself? No problem. Here is how:


cat <<EOF > Dockerfile.centos7-with-podman-and-fuse
FROM centos:7

  yum -y reinstall shadow-utils \
  && yum -y install podman fuse-overlayfs \
  && rm -rf /var/cache /var/log/dnf* /var/log/yum.*

RUN useradd podman \
  && echo podman:10000:5000 > /etc/subuid \
  && echo podman:10000:5000 > /etc/subgid

VOLUME /var/lib/containers
VOLUME /home/podman/.local/share/containers

ADD /etc/containers/containers.conf
ADD /home/podman/.config/containers/containers.conf

RUN chown podman:podman -R /home/podman

# chmod containers.conf and adjust storage.conf to enable Fuse storage.
RUN chmod 644 /etc/containers/containers.conf \
  && sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage.*/a "/var/lib/shared",' -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' /etc/containers/storage.conf
RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers /var/lib/shared/vfs-images /var/lib/shared/vfs-layers \
  && touch /var/lib/shared/overlay-images/images.lock \
  && touch /var/lib/shared/overlay-layers/layers.lock \
  && touch /var/lib/shared/vfs-images/images.lock \
  && touch /var/lib/shared/vfs-layers/layers.lock


RUN podman version

# INFO: podman 1.6.4 did not work. podman run created an error Error: stat /sys/fs/cgroup/systemd/system.slice/containerd.service/kubepods-besteffort-podf422c927... command terminated with exit code 125

# Upgrade podman from 1.6.4 to 3.4.4

# Install podman v3 Deps
RUN yum install -y sudo \
  && yum install -y \
  && yum install -y \
       "@Development Tools" \
       curl \
       gcc \
       make \
       device-mapper-devel \
       git \
       btrfs-progs-devel \
       conmon \
       containernetworking-plugins \
       containers-common \
       git \
       glib2-devel \
       glibc-devel \
       glibc-static \
       golang-github-cpuguy83-md2man \
       gpgme-devel \
       iptables \
       libassuan-devel \
       libgpg-error-devel \
       libseccomp-devel \
       libselinux-devel \
       pkgconfig \
       systemd-devel \
       autoconf \
       python3 \
       python3-devel \
       python3-pip \
       yajl-devel \
       libcap-devel \
       jq \

# Install conmon
RUN git clone \
  && cd conmon \
  && export GOCACHE="$(mktemp -d)" \
  && make \
  && sudo make podman \
  && cd .. \
  && conmon --version

# Install policies
RUN mkdir -p /etc/containers \
  && sudo curl -L -o /etc/containers/registries.conf \
  && sudo curl -L | jq 'del(.transports.docker)' > /etc/containers/policy.json

# Install CentOS 7 friendly crun without systemd
RUN curl -s -L -o /usr/bin/crun \
  && chmod +x /usr/bin/crun \
  && crun -V

# Install podman
RUN TAG="v3.4.4" \
  && rm -rf podman* \
  && curl -O -L${TAG}.tar.gz \
  && tar xvf ${TAG}.tar.gz \
  && cd podman*/ \
  && make BUILDTAGS="selinux seccomp" \
  && make install PREFIX=/usr

Docker build & push

docker build . -f "Dockerfile.centos7-with-podman-and-fuse" -t vocon/podman:centos7
docker push vocon/podman:centos7

Prepare Kubernetes Agent

Enable User Namespaces temporarily (for testing)

cat /proc/sys/user/max_user_namespaces | egrep -q '^0$' \
  && echo "Enabling User Namespaces" \
  && echo 1000000 | sudo tee /proc/sys/user/max_user_namespaces

Enable User Namespaces permanently (surviving reboot)

cat <<EOF | sudo tee /etc/systemd/system/enable-user-namespaces.service
Description=Enable User Namespaces

ExecStart=/usr/bin/sh -c '/usr/bin/echo 100000 > /proc/sys/user/max_user_namespaces'


sudo systemctl daemon-reload
sudo systemctl enable enable-user-namespaces.service
sudo systemctl start enable-user-namespaces.service
sudo systemctl status enable-user-namespaces.service

Install Fuse Device Plugin:

kubectl apply -f

# output: daemonset.apps/fuse-device-plugin-daemonset created

Please do not be confused by the „1.16“ in the link name. The link name officially works for all kubernetes versions >= v1.16.

Create a POD on Kubernetes

mkdir -p ${HOME}/.local/share/containers
chown -R 1000:1000 ${HOME}/.local/share/containers
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
 name: podman-centos7-no-priv
   - name: podman-centos7-no-priv
       - sleep
       - "1000000"
       runAsUser: 1000
       limits: 1
       - mountPath: /home/podman/.local/share/containers
         name: podman-local
   - name: podman-local
       path: ${HOME}/.local/share/containers

Run podman inside POD

Check user id

kubectl exec -it podman-centos7-no-priv -- sh -c 'id'

# output:
uid=1000(podman) gid=1000(podman) groups=1000(podman)

podman run

podman run --rm -it alpine echo hello podman
? Please select an image:
  ? #          <---------------- choose this one
Resolved "alpine" as an alias (/headless/.cache/containers/short-name-aliases.conf)
Trying to pull
Getting image source signatures
Copying blob 2408cc74d12b done
Copying config e66264b987 done
Writing manifest to image destination
Storing signatures
hello podman
Running non-privileged Podman in a container works!


podman build does not (yet) work; we need to investigate, what is the exact reason…


Diese Website verwendet Akismet, um Spam zu reduzieren. Erfahre mehr darüber, wie deine Kommentardaten verarbeitet werden.