#podman #jenksin #cicd

My Jenkins setup in the homelab

Recently I wanted to setup a LaTex pipeline in my Jenkins. During this task I noticed, that certain images fail with a particular error message, that was not meaningful at all (in hindsight).

The Nomad cloud plugin for Jenkins starts Jenkins workers as Nomad jobs. These jobs are started from an image that includes the Jenkins inbound agent. This agent connects to the Jenkins server.

Alongside the Jenkins agent, I also install Buildah and Docker on that image, because I use these Jenkins workers to build container images.

Besides building images (with Buildah), I also use the Docker workflow to run arbitrary container images in my Jenkins pipelines.

The agent image mounts the Podman socket of the Nomad node. It does not mount the socket of the “root” Podman process, only the rootless socket of a particular jenkins user on the nodes.

I reserved this users on the nodes (my Raspberry Pies) for this particular purpose.

In that sense, whenever I run an image with the Docker workflow plugin in my Jenkins pipeline, it is started as rootless Podman container under that jenkins user on the Nomad node where the pipeline worker is scheduled.

VFS default storage driver for rootless Podman

I only noticed today, that even though I have a more recent kernel than 5.12.9 (I run 6.12.41), the rootless Podman configuration does not check the kernel version but simply defaults to the VFS storage.

The default storage driver for UID 0 is configured in containers-storage.conf(5) in rootless mode), and is vfs for non-root users when fuse-overlayfs is not available.

I recently uninstall fuse-overlayfs from my Pies, because I run a recent kernel 5.12.9 that supports native overlayfs.

Improve performance for Docker workflow in Jenkins

In order to achieve better performance and fix the weird and unclear error in my Jenkins pipeline that use Docker workflow, I had to migrate to overlayfs storage driver for the rootless Podman socket (of the jenkins user).

As jenkins user on the nodes, cleanup the old vfs storage:

$ podman system reset

Also had to help a bit (as root):

# rm -rf /home/jenkins/.local/share/containers/

Then change the the storage driver setting (I changed it globally in /etc/containers/storage.conf because my root Podman process already runs with overlayfs storage driver anyways, so it's basically a global default now).

# Configure overlay storage for Podman
cat <<EOF > /etc/containers/storage.conf
[storage]
driver="overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"
EOF

This explicit setting now also holds for the rootless Podman socket. I was not aware of that. It can be checked with the jenkins user (rootless Podman):

$ podman info | grep graph
  graphDriverName: overlay

Really glad I got this fixed.

Let me know your Podman/Buildah/Jenkins stories on the Fediverse or via chat.

#cicd #coding #jenkins #nomad

You may ask why I spend time to setup a good old Jenkins while everyone else seems to jump on some newer CI systems (GitLab, Forgejo, etc.)? Well, as said, it's still good old Jenkins and I assume it will be around for some more time.

To run Jenkins agents on a Nomad cluster I followed Ola Ogunseghas instructions and example code here:

• https://faun.pub/jenkins-build-agents-on-nomad-workers-626b0df4fc57
• https://github.com/GastroGee/jenkins-nomad/blob/main/jenkins-controller/nomad.yaml

It involves installing the Nomad plugin for Jenkins and configuring this “Nomad cloud” (that's how the integration is called in Jenkins) with a template for the Jenkins agents.

Obviously, I wanted to integrate Jenkins with my Git repos. The most straight forward way seemed to use post-receive hooks to nudge Jenkins on every push. This has worked fabulously so far.

Even though the runners are spawned as Nomad jobs, I still wanted to run other Docker containers in the pipeline. This is where it got very confusing for me, because of the different Docker plugins for Jenkins. Most notably, there exist at least these two plugins:

• docker-workflow, runs the Docker container from the Jenkins agent
• docker-plugin, runs Jenkins agents as Docker containers

I decided to go with the former “docker-workflow” plugin because I already deployed the Jenkins agents as Nomad jobs. docker-workflow can run arbitrary containers from any Docker image, whereas the docker-plugin needs to be based on the Jenkins inbound-agent image to be able to connect to the Jenkins server.

I wanted the containers that are launched from Jenkins to be contained in a another users namespace. There exists the option to rebuild the inbound-agent with user-supplied attributes for uid/gid, but since I wanted to modify the image anyways I simply forked and built my own agent images, mostly inspired by the example for running Docker inside the agent.

At this stage, my tooling was sophisticated enough I could run a simple gitleaks container on each push to scan for secrets. I'm always afraid to publish secrets accidentally (it happened to me before).

Furthermore, I needed to establish some kind of build process, so I can also build images with Jenkins and directly push them to my local image registry. After some reading, I discarded the thought to use Kaniko because it still requires nodes in the native architecture of the respective build for multi-architecture builds.

Therefore, I followed RedHat best-practices to integrate Buildah, which is also the tool I use to build multi-arch container images locally on my laptop.

If you are interested in more example code, here the link to some of the key components:

• jenkins.yaml.tmpl.html Jenkins infrastructure code
• jenkins.nomad.html Jenkins nomad job
• docker-agent Modified Jenkins inbound-agent with Docker and Buildah

I'm not finished with playing around, because I still haven't figured out some things fully yet (e.g., how to properly use jlink in multi-arch builds) and because I'm also curious how other members of the community use Jenkins on Nomad.