#jenkins #containers #buildah

I was writing about how and why I use Jenkins last year.

The basic idea behind the pipeline here is to build container images for multiple platforms/architectures (arm64 & amd64 mainly).

An advantage of using the runners native architecture (arm64/amd64) for the builds is that I don't need to use the QEMU emulation with buildah.

Jenkinsfile with static nodes/stages

My previous Jenkinsfile (starting point) was very “static”. Static in a way, that the nodes were selected sequentially (first build on an arm64 node, then on a amd64 node) and I created two different tags for the different architectures:

@Library('in0rdr-jenkins-lib@master') _

def buildahbud = new BuildahBud(this)
def buildahpush = new BuildahPush(this)
def buildahmanifest = new BuildahManifest(this)

node('podman&&arm64'){
  checkout scm

  // build with image context and name
  buildahbud.execute([:], "docker/docker-snac", "snac", "2.90-arm64"
    'Dockerfile', 'arm64/v8')
  buildahpush.execute("snac", "2.90-arm64")
}

node('podman&&amd64'){
  checkout scm
  buildahbud.execute([:], "docker/docker-snac", "snac", "2.90-amd64",
    'Dockerfile', 'amd64')
  buildahmanifest.create('haproxy.lan:5000/snac:2.90', [
    'haproxy.lan:5000/snac:2.90-arm64',
    'haproxy.lan:5000/snac:2.90-amd64'
  ])
}

You can find my Jenkins libraries here: * https://code.in0rdr.ch/jenkins-lib/files.html

Pulling a different tag depending on the architecture is cumbersome. By creating a Manifest (buildah manifest create) I can reuse the same tag across multiple machine architectures (e.g., I can use haproxy.lan:5000/snac:2.90 on amd and arm machines):

buildah manifest inspect haproxy.lan:5000/snac:2.90

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:7f67daa2193f2ef9a84c3bcaa14c73d429165631cbbc6c30faf74ef69ac07d4a",
            "size": 1207,
            "platform": {
                "architecture": "arm64",
                "os": "linux",
                "variant": "v8"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:85b79a9014eed10f8b12cddd9f6fcf9a0e56cdf792b96cf4cde52c9c8f61c4f7",
            "size": 1204,
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        }
    ]
}

Dynamic choice, run stages on nodes in parallel

I decided to change my static Jenkinsfile and spice it up with the dynamic choices based on the “input matrix”. The “platform” axis only holds the “podman” platform right now. * https://code.in0rdr.ch/jenkins-lib/file/src/BuildahParallelBuild.groovy.html

For this, I took some inspiration from an older Jenkins blog post from 2019. The code example for a scripted pipeline with “dynamic choices” (choices based on predefined matrix axes) still works pretty fine.

My inputs can now be kept rather simple, they simply describe which Dockerfile I want to build, from which path in the repo, with certain name & tag (optionally some build arguments):

@Library('in0rdr-jenkins-lib@master') _

def buildahParallelBuild = new BuildahParallelBuild(this)

buildahParallelBuild.build("snac", "2.90", "docker/docker-snac")

// Other example inputs:
//buildahParallelBuild.build("texlive", "latest", "docker/docker-texlive")
//buildahParallelBuild.build("updatecli", "v0.114.0", "docker/docker-updatecli")
//buildahParallelBuild.build("jenkins-inbound-agent", "3355.v388858a_47b_33", "docker/docker-jenkins-inbound-agent", [arg1: "test", arg2: "test2"])

It doesn't matter on which architecture I create the manifest.

I'm still a huge fan of the extensibility of these Jenkins libraries. By simply reusing my existing libraries for Buildah bud (class BuildahBud) / push (class BuildahPush) / manifest creation (class BuildahManifest) from my already working “static” use case, I could extend quickly with the “parallel build” functionality (in a new/separate class which simply imports the existing functionality) 🤗.

Later I might figure out more about @NonCPS and NonCPS best practices.. hit me up if you would like to educate me 🤔

Thanks for reading

#nomad #openbao #jenkins #homelab

I have two automatic/system jobs that require a Token.

• The backup cron job is taking Nomad snapshots in regular intervals. The snapshot API requires a management token and the Nomad policy capability for snapshots with the operator are not implemented yet.
• The Jenkins server runs Nomad jobs using the Nomad cloud plugin. This system needs a Token to access the Nomad API (AppRole not compatible, see below).

I still keep my bootstrapping token around just in case I ever need it. That's ok unlike to procedures with OpenBao root tokens..

The bootstrap token can be deleted and is like any other token, care should be taken to not revoke all management tokens.

Name             Type        Global  Accessor ID  Expired
Bootstrap Token  management  true    ***          false
Snapshot         management  false   ***          false
OIDC-vault       client      true    ***          false
Jenkins          client      false   ***          false

The auth method in Nomad is still called Vault, never mind..

The access for human users is authenticated by an OIDC provider in my OpenBao server.

Because I already had an identity and alias setup (userpass authentication) including a group, I only needed to configure the provider and the assignment to the group to allow authentication with the new provider.

In the OpenBao server, the default provider and the allow_all assignment cannot be deleted. I assume it is similar to the “master” realm in a Keycloak instance 🤔.

I had to define a NOMAD_TOKEN as “Jenkins credential”, because the Nomad cloud plugin for Jenkins cannot read secrets from OpenBao using an AppRole (the Nomad jobs spawned by the plugin can do this, just not the plugin itself).

When I type nomad login in the shell, the browser opens and I can authenticate with OpenBao. What could be improved is outputting the OIDC redirect URI in the terminal. This is helpful when you need to login from disconnected machines (i.e., not the shell on your local machine).

#updatecli #pipeline #jenkins #myheats #nodejs #npm

I was looking for a way to automatically bump the version of the npm dependencies (package.json) whenever there is an update available. This is also important for security reasons (e.g., have a look at the output of npm audit from time to time to see the recent security issues in the dependencies).

I was looking into Renovate and Dependabot, but neither of these scratched my itch of simple automatic dependency updates.

A coworker suggested me to try Updatecli and it fits my workflows perfectly well. The Jenkins example on the projects website got me started. So I created a Jenkins shared library function to run my own build, which includes npm to perform the version bumps:

• A class to describe the updatecli stages: https://code.in0rdr.ch/jenkins-lib/file/src/Updatecli.groovy.html

The scripted pipeline in the repository of the application loads the library and performs the version bumps to a new branch:

• The Jenkinsfile that makes use of the updatecli groovy library: https://code.in0rdr.ch/myheats/file/Jenkinsfile.html

I did not even have to configure Updatecli a lot, because the autodiscovery feature automatically detects that this is a npm repository/project. The final version of my pipeline includes all the git/scm steps in the updatecli.d/default.yaml configuration file:

• Updatecli configuration file: https://code.in0rdr.ch/myheats/file/updatecli.d/default.yaml.html

First I tried to perform the SCM/git steps in Jenkins checkout and sh steps. But I noticed it could be much sleeker by defining the SCM/git settings in the Updatecli config file directly. This way, updatecli takes care of the clone/checkout/push steps. Here the extract from my previous pipeline with the “manual git steps” for comparison:

// alternative approach I did not pursue any further
sh '''
git config --global user.name "$GIT_AUTHOR_NAME"
git config --global user.email "$GIT_AUTHOR_EMAIL"
'''

dir("myyheats.git-$BUILD_NUMBER") {
  // checkout update branch in new directory
  checkout scmGit(
      extensions: [localBranch("$branch")],
      userRemoteConfigs: [[url: 'https://git.in0rdr.ch/myheats.git']]
  )

  updatecli.run('apply')

  // commit changes
  sh '''
  git add -u
  git commit -m "chore(updatecli-$BUILD_NUMBER): bump node modules"
  git push -f -u origin "$branch"
  '''
}

I definitely like the updatecli configuration better, since it keeps the actual pipeline tidy. Also, I like how you can use the {{ requiredEnv "GIT_PASSWORD" }} configuration in updatecli to read secrets from the environment. The Git credentials are sourced from OpenBao with Nomad workload identities.

I hope the post is helpful for anyone that would like to give updatecli a try or would like to configure a similar Jenkins pipeline.

#cicd #coding #jenkins #nomad

You may ask why I spend time to setup a good old Jenkins while everyone else seems to jump on some newer CI systems (GitLab, Forgejo, etc.)? Well, as said, it's still good old Jenkins and I assume it will be around for some more time.

To run Jenkins agents on a Nomad cluster I followed Ola Ogunseghas instructions and example code here:

• https://faun.pub/jenkins-build-agents-on-nomad-workers-626b0df4fc57
• https://github.com/GastroGee/jenkins-nomad/blob/main/jenkins-controller/nomad.yaml

It involves installing the Nomad plugin for Jenkins and configuring this “Nomad cloud” (that's how the integration is called in Jenkins) with a template for the Jenkins agents.

Obviously, I wanted to integrate Jenkins with my Git repos. The most straight forward way seemed to use post-receive hooks to nudge Jenkins on every push. This has worked fabulously so far.

Even though the runners are spawned as Nomad jobs, I still wanted to run other Docker containers in the pipeline. This is where it got very confusing for me, because of the different Docker plugins for Jenkins. Most notably, there exist at least these two plugins:

• docker-workflow, runs the Docker container from the Jenkins agent
• docker-plugin, runs Jenkins agents as Docker containers

I decided to go with the former “docker-workflow” plugin because I already deployed the Jenkins agents as Nomad jobs. docker-workflow can run arbitrary containers from any Docker image, whereas the docker-plugin needs to be based on the Jenkins inbound-agent image to be able to connect to the Jenkins server.

I wanted the containers that are launched from Jenkins to be contained in a another users namespace. There exists the option to rebuild the inbound-agent with user-supplied attributes for uid/gid, but since I wanted to modify the image anyways I simply forked and built my own agent images, mostly inspired by the example for running Docker inside the agent.

At this stage, my tooling was sophisticated enough I could run a simple gitleaks container on each push to scan for secrets. I'm always afraid to publish secrets accidentally (it happened to me before).

Furthermore, I needed to establish some kind of build process, so I can also build images with Jenkins and directly push them to my local image registry. After some reading, I discarded the thought to use Kaniko because it still requires nodes in the native architecture of the respective build for multi-architecture builds.

Therefore, I followed RedHat best-practices to integrate Buildah, which is also the tool I use to build multi-arch container images locally on my laptop.

If you are interested in more example code, here the link to some of the key components:

• jenkins.yaml.tmpl.html Jenkins infrastructure code
• jenkins.nomad.html Jenkins nomad job
• docker-agent Modified Jenkins inbound-agent with Docker and Buildah

I'm not finished with playing around, because I still haven't figured out some things fully yet (e.g., how to properly use jlink in multi-arch builds) and because I'm also curious how other members of the community use Jenkins on Nomad.