See how you can run a user space podman container inside another non-privileged container. For that, we create a CentOS 7 image with podman v3 installed. We spin up a Kubernetes non-privileged container from this image, and we show that we are able to run other podman containers successfully.

Tested on Kubernetes v1.22.9 with CentOS 7 Kubernetes agents and containerd container runtime v1.5.11

This is a tl;dr („too long; didn’t read“) style blog post that consists of headlines and code only. For questions, please add comments to the blog post.

Test Podman in a non-privileged Container online (free)

You can test podman in a Kubernetes container online on https://cloud.vocon-it.com/products. The (experimental) PyCharm service is based on a Podman container inside the Kubernetes container:

Instead of waiting PyCharm to spin up, you can open a terminal and test podman:

Running Podman online in a Kubernetes Container

As you can see from the PyCharm window, which is spinning up automatically, you can see that you can run X Window containers inside the Kubernetes container:

Running an Y Wndow Container (PyCharm) inside a Podman Container inside a Kubernetes Container

Do you want to do it yourself? No problem. Here is how:

Dockerfile

cat <<EOF > Dockerfile.centos7-with-podman-and-fuse
FROM centos:7

RUN \
  yum -y reinstall shadow-utils \
  && yum -y install podman fuse-overlayfs \
  && rm -rf /var/cache /var/log/dnf* /var/log/yum.*

RUN useradd podman \
  && echo podman:10000:5000 > /etc/subuid \
  && echo podman:10000:5000 > /etc/subgid

VOLUME /var/lib/containers
VOLUME /home/podman/.local/share/containers

ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/containers.conf /etc/containers/containers.conf
ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/podman-containers.conf /home/podman/.config/containers/containers.conf

RUN chown podman:podman -R /home/podman

# chmod containers.conf and adjust storage.conf to enable Fuse storage.
RUN chmod 644 /etc/containers/containers.conf \
  && sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage.*/a "/var/lib/shared",' -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' /etc/containers/storage.conf
RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers /var/lib/shared/vfs-images /var/lib/shared/vfs-layers \
  && touch /var/lib/shared/overlay-images/images.lock \
  && touch /var/lib/shared/overlay-layers/layers.lock \
  && touch /var/lib/shared/vfs-images/images.lock \
  && touch /var/lib/shared/vfs-layers/layers.lock

ENV _CONTAINERS_USERNS_CONFIGURED=""

RUN podman version

# INFO: podman 1.6.4 did not work. podman run created an error Error: stat /sys/fs/cgroup/systemd/system.slice/containerd.service/kubepods-besteffort-podf422c927... command terminated with exit code 125

#
# Upgrade podman from 1.6.4 to 3.4.4
#

# Install podman v3 Deps
RUN yum install -y sudo \
  && yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm \
  && yum install -y \
       "@Development Tools" \
       curl \
       gcc \
       make \
       device-mapper-devel \
       git \
       btrfs-progs-devel \
       conmon \
       containernetworking-plugins \
       containers-common \
       git \
       glib2-devel \
       glibc-devel \
       glibc-static \
       golang-github-cpuguy83-md2man \
       gpgme-devel \
       iptables \
       libassuan-devel \
       libgpg-error-devel \
       libseccomp-devel \
       libselinux-devel \
       pkgconfig \
       systemd-devel \
       autoconf \
       python3 \
       python3-devel \
       python3-pip \
       yajl-devel \
       libcap-devel \
       jq \
       go

# Install conmon
RUN git clone https://github.com/containers/conmon \
  && cd conmon \
  && export GOCACHE="$(mktemp -d)" \
  && make \
  && sudo make podman \
  && cd .. \
  && conmon --version

# Install policies
RUN mkdir -p /etc/containers \
  && sudo curl -L -o /etc/containers/registries.conf https://src.fedoraproject.org/rpms/containers-common/raw/main/f/registries.conf \
  && sudo curl -L https://src.fedoraproject.org/rpms/containers-common/raw/main/f/default-policy.json | jq 'del(.transports.docker)' > /etc/containers/policy.json

# Install CentOS 7 friendly crun without systemd
RUN curl -s -L -o /usr/bin/crun https://github.com/alvistack/crun/releases/download/0.14.1/crun-0.14.1-linux-amd64 \
  && chmod +x /usr/bin/crun \
  && crun -V

# Install podman
RUN TAG="v3.4.4" \
  && rm -rf podman* \
  && curl -O -L https://github.com/containers/podman/archive/refs/tags/${TAG}.tar.gz \
  && tar xvf ${TAG}.tar.gz \
  && cd podman*/ \
  && make BUILDTAGS="selinux seccomp" \
  && make install PREFIX=/usr
EOF

Docker build & push

docker build . -f "Dockerfile.centos7-with-podman-and-fuse" -t vocon/podman:centos7
docker push vocon/podman:centos7

Prepare Kubernetes Agent

Enable User Namespaces temporarily (for testing)

cat /proc/sys/user/max_user_namespaces | egrep -q '^0$' \
  && echo "Enabling User Namespaces" \
  && echo 1000000 | sudo tee /proc/sys/user/max_user_namespaces

Enable User Namespaces permanently (surviving reboot)

cat <<EOF | sudo tee /etc/systemd/system/enable-user-namespaces.service
[Unit]
Description=Enable User Namespaces
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/sh -c '/usr/bin/echo 100000 > /proc/sys/user/max_user_namespaces'
TimeoutStartSec=0

[Install]
WantedBy=default.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable enable-user-namespaces.service
sudo systemctl start enable-user-namespaces.service
sudo systemctl status enable-user-namespaces.service

Install Fuse Device Plugin:

kubectl apply -f https://raw.githubusercontent.com/kuberenetes-learning-group/fuse-device-plugin/master/fuse-device-plugin-k8s-1.16.yml

# output: daemonset.apps/fuse-device-plugin-daemonset created

Please do not be confused by the „1.16“ in the link name. The link name officially works for all kubernetes versions >= v1.16.

Create a POD on Kubernetes

mkdir -p ${HOME}/.local/share/containers
chown -R 1000:1000 ${HOME}/.local/share/containers
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
 name: podman-centos7-no-priv
spec:
 containers:
   - name: podman-centos7-no-priv
     image: docker.io/vocon/podman:centos7
     args:
       - sleep
       - "1000000"
     securityContext:
       runAsUser: 1000
     resources:
       limits:
         github.com/fuse: 1
     volumeMounts:
       - mountPath: /home/podman/.local/share/containers
         name: podman-local
 volumes:
   - name: podman-local
     hostPath:
       path: ${HOME}/.local/share/containers
EOF

Run podman inside POD

Check user id

kubectl exec -it podman-centos7-no-priv -- sh -c 'id'

# output:
uid=1000(podman) gid=1000(podman) groups=1000(podman)

podman run

podman run --rm -it alpine echo hello podman
Output
? Please select an image:
    registry.fedoraproject.org/apline:latest
    registry.access.redhat.com/apline:latest
  ? docker.io/library/apline:latest #          <---------------- choose this one 
    quay.io/apline:latest
Resolved "alpine" as an alias (/headless/.cache/containers/short-name-aliases.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob 2408cc74d12b done
Copying config e66264b987 done
Writing manifest to image destination
Storing signatures
hello podman
Running non-privileged Podman in a container works!

Caveat

podman build does not (yet) work; we need to investigate, what is the exact reason…

31 comments

  1. My brother suggested I might like this website He was totally right This post actually made my day You cannt imagine just how much time I had spent for this information Thanks

  2. helloI really like your writing so a lot share we keep up a correspondence extra approximately your post on AOL I need an expert in this house to unravel my problem May be that is you Taking a look ahead to see you

  3. Normally I do not read article on blogs however I would like to say that this writeup very forced me to try and do so Your writing style has been amazed me Thanks quite great post

  4. I am not sure where youre getting your info but good topic I needs to spend some time learning much more or understanding more Thanks for magnificent info I was looking for this information for my mission

  5. Modern Talking ??? ???????? ??????, ?????????????? ? 1984 ????. ?? ???? ????? ?? ????? ????? ?????????????? ????????? ? ????????? ????????? ?????? ????????????? ????????. ?????? ????? ???????? „You’re My Heart, You’re My Soul“, „Brother Louie“, „Cheri, Cheri Lady“ ? „Geronimo’s Cadillac“. ?? ?????? ???????? ???????????? ???? ? ??????? ???-??????, ?????????? ?????????? ?????? ?????????????? ????????? ? ??????????????? ????????. Modern Talking ?????????? ???? ?????????? ? ? ???? ???, ????????? ????? ?? ???????? ????? ?????. ?????? 2024 ???? ??????? ?????? ? ??????? ????????? mp3.

  6. I think every concept you put up in your post is strong and will undoubtedly be implemented. Still, the posts are too brief for inexperienced readers. Would you kindly extend them a little bit from now on? I appreciate the post.

  7. I loved as much as youll receive carried out right here The sketch is tasteful your authored material stylish nonetheless you command get bought an nervousness over that you wish be delivering the following unwell unquestionably come more formerly again since exactly the same nearly a lot often inside case you shield this hike

  8. Thank you for the auspicious writeup It in fact was a amusement account it Look advanced to more added agreeable from you By the way how could we communicate

  9. I loved as much as youll receive carried out right here The sketch is attractive your authored material stylish nonetheless you command get bought an nervousness over that you wish be delivering the following unwell unquestionably come more formerly again as exactly the same nearly a lot often inside case you shield this hike

  10. Somebody essentially lend a hand to make significantly articles Id state That is the very first time I frequented your website page and up to now I surprised with the research you made to make this actual submit amazing Wonderful task

  11. What a fantastic resource! The articles are meticulously crafted, offering a perfect balance of depth and accessibility. I always walk away having gained new understanding. My sincere appreciation to the team behind this outstanding website.

  12. I just could not depart your web site prior to suggesting that I really loved the usual info an individual supply in your visitors Is gonna be back regularly to check up on new posts

  13. Thanks I have just been looking for information about this subject for a long time and yours is the best Ive discovered till now However what in regards to the bottom line Are you certain in regards to the supply

  14. Somebody essentially help to make significantly articles Id state This is the first time I frequented your web page and up to now I surprised with the research you made to make this actual post incredible Fantastic job

  15. I was recommended this website by my cousin I am not sure whether this post is written by him as nobody else know such detailed about my difficulty You are wonderful Thanks

  16. Normally I do not read article on blogs however I would like to say that this writeup very forced me to try and do so Your writing style has been amazed me Thanks quite great post

  17. I was recommended this website by my cousin I am not sure whether this post is written by him as nobody else know such detailed about my trouble You are amazing Thanks

Comments

Diese Website verwendet Akismet, um Spam zu reduzieren. Erfahre mehr darüber, wie deine Kommentardaten verarbeitet werden.