#nomad #openbao #jenkins #homelab

I have two automatic/system jobs that require a Token.

• The backup cron job is taking Nomad snapshots in regular intervals. The snapshot API requires a management token and the Nomad policy capability for snapshots with the operator are not implemented yet.
• The Jenkins server runs Nomad jobs using the Nomad cloud plugin. This system needs a Token to access the Nomad API (AppRole not compatible, see below).

I still keep my bootstrapping token around just in case I ever need it. That's ok unlike to procedures with OpenBao root tokens..

The bootstrap token can be deleted and is like any other token, care should be taken to not revoke all management tokens.

Name             Type        Global  Accessor ID  Expired
Bootstrap Token  management  true    ***          false
Snapshot         management  false   ***          false
OIDC-vault       client      true    ***          false
Jenkins          client      false   ***          false

The auth method in Nomad is still called Vault, never mind..

The access for human users is authenticated by an OIDC provider in my OpenBao server.

Because I already had an identity and alias setup (userpass authentication) including a group, I only needed to configure the provider and the assignment to the group to allow authentication with the new provider.

In the OpenBao server, the default provider and the allow_all assignment cannot be deleted. I assume it is similar to the “master” realm in a Keycloak instance 🤔.

I had to define a NOMAD_TOKEN as “Jenkins credential”, because the Nomad cloud plugin for Jenkins cannot read secrets from OpenBao using an AppRole (the Nomad jobs spawned by the plugin can do this, just not the plugin itself).

When I type nomad login in the shell, the browser opens and I can authenticate with OpenBao. What could be improved is outputting the OIDC redirect URI in the terminal. This is helpful when you need to login from disconnected machines (i.e., not the shell on your local machine).

#cicd #coding #jenkins #nomad

You may ask why I spend time to setup a good old Jenkins while everyone else seems to jump on some newer CI systems (GitLab, Forgejo, etc.)? Well, as said, it's still good old Jenkins and I assume it will be around for some more time.

To run Jenkins agents on a Nomad cluster I followed Ola Ogunseghas instructions and example code here:

• https://faun.pub/jenkins-build-agents-on-nomad-workers-626b0df4fc57
• https://github.com/GastroGee/jenkins-nomad/blob/main/jenkins-controller/nomad.yaml

It involves installing the Nomad plugin for Jenkins and configuring this “Nomad cloud” (that's how the integration is called in Jenkins) with a template for the Jenkins agents.

Obviously, I wanted to integrate Jenkins with my Git repos. The most straight forward way seemed to use post-receive hooks to nudge Jenkins on every push. This has worked fabulously so far.

Even though the runners are spawned as Nomad jobs, I still wanted to run other Docker containers in the pipeline. This is where it got very confusing for me, because of the different Docker plugins for Jenkins. Most notably, there exist at least these two plugins:

• docker-workflow, runs the Docker container from the Jenkins agent
• docker-plugin, runs Jenkins agents as Docker containers

I decided to go with the former “docker-workflow” plugin because I already deployed the Jenkins agents as Nomad jobs. docker-workflow can run arbitrary containers from any Docker image, whereas the docker-plugin needs to be based on the Jenkins inbound-agent image to be able to connect to the Jenkins server.

I wanted the containers that are launched from Jenkins to be contained in a another users namespace. There exists the option to rebuild the inbound-agent with user-supplied attributes for uid/gid, but since I wanted to modify the image anyways I simply forked and built my own agent images, mostly inspired by the example for running Docker inside the agent.

At this stage, my tooling was sophisticated enough I could run a simple gitleaks container on each push to scan for secrets. I'm always afraid to publish secrets accidentally (it happened to me before).

Furthermore, I needed to establish some kind of build process, so I can also build images with Jenkins and directly push them to my local image registry. After some reading, I discarded the thought to use Kaniko because it still requires nodes in the native architecture of the respective build for multi-architecture builds.

Therefore, I followed RedHat best-practices to integrate Buildah, which is also the tool I use to build multi-arch container images locally on my laptop.

If you are interested in more example code, here the link to some of the key components:

• jenkins.yaml.tmpl.html Jenkins infrastructure code
• jenkins.nomad.html Jenkins nomad job
• docker-agent Modified Jenkins inbound-agent with Docker and Buildah

I'm not finished with playing around, because I still haven't figured out some things fully yet (e.g., how to properly use jlink in multi-arch builds) and because I'm also curious how other members of the community use Jenkins on Nomad.