#jenkins #containers #buildah

I was writing about how and why I use Jenkins last year.

The basic idea behind the pipeline here is to build container images for multiple platforms/architectures (arm64 & amd64 mainly).

An advantage of using the runners native architecture (arm64/amd64) for the builds is that I don't need to use the QEMU emulation with buildah.

Jenkinsfile with static nodes/stages

My previous Jenkinsfile (starting point) was very “static”. Static in a way, that the nodes were selected sequentially (first build on an arm64 node, then on a amd64 node) and I created two different tags for the different architectures:

@Library('in0rdr-jenkins-lib@master') _

def buildahbud = new BuildahBud(this)
def buildahpush = new BuildahPush(this)
def buildahmanifest = new BuildahManifest(this)

node('podman&&arm64'){
  checkout scm

  // build with image context and name
  buildahbud.execute([:], "docker/docker-snac", "snac", "2.90-arm64"
    'Dockerfile', 'arm64/v8')
  buildahpush.execute("snac", "2.90-arm64")
}

node('podman&&amd64'){
  checkout scm
  buildahbud.execute([:], "docker/docker-snac", "snac", "2.90-amd64",
    'Dockerfile', 'amd64')
  buildahmanifest.create('haproxy.lan:5000/snac:2.90', [
    'haproxy.lan:5000/snac:2.90-arm64',
    'haproxy.lan:5000/snac:2.90-amd64'
  ])
}

You can find my Jenkins libraries here: * https://code.in0rdr.ch/jenkins-lib/files.html

Pulling a different tag depending on the architecture is cumbersome. By creating a Manifest (buildah manifest create) I can reuse the same tag across multiple machine architectures (e.g., I can use haproxy.lan:5000/snac:2.90 on amd and arm machines):

buildah manifest inspect haproxy.lan:5000/snac:2.90

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:7f67daa2193f2ef9a84c3bcaa14c73d429165631cbbc6c30faf74ef69ac07d4a",
            "size": 1207,
            "platform": {
                "architecture": "arm64",
                "os": "linux",
                "variant": "v8"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:85b79a9014eed10f8b12cddd9f6fcf9a0e56cdf792b96cf4cde52c9c8f61c4f7",
            "size": 1204,
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        }
    ]
}

Dynamic choice, run stages on nodes in parallel

I decided to change my static Jenkinsfile and spice it up with the dynamic choices based on the “input matrix”. The “platform” axis only holds the “podman” platform right now. * https://code.in0rdr.ch/jenkins-lib/file/src/BuildahParallelBuild.groovy.html

For this, I took some inspiration from an older Jenkins blog post from 2019. The code example for a scripted pipeline with “dynamic choices” (choices based on predefined matrix axes) still works pretty fine.

My inputs can now be kept rather simple, they simply describe which Dockerfile I want to build, from which path in the repo, with certain name & tag (optionally some build arguments):

@Library('in0rdr-jenkins-lib@master') _

def buildahParallelBuild = new BuildahParallelBuild(this)

buildahParallelBuild.build("snac", "2.90", "docker/docker-snac")

// Other example inputs:
//buildahParallelBuild.build("texlive", "latest", "docker/docker-texlive")
//buildahParallelBuild.build("updatecli", "v0.114.0", "docker/docker-updatecli")
//buildahParallelBuild.build("jenkins-inbound-agent", "3355.v388858a_47b_33", "docker/docker-jenkins-inbound-agent", [arg1: "test", arg2: "test2"])

It doesn't matter on which architecture I create the manifest.

I'm still a huge fan of the extensibility of these Jenkins libraries. By simply reusing my existing libraries for Buildah bud (class BuildahBud) / push (class BuildahPush) / manifest creation (class BuildahManifest) from my already working “static” use case, I could extend quickly with the “parallel build” functionality (in a new/separate class which simply imports the existing functionality) 🤗.

Later I might figure out more about @NonCPS and NonCPS best practices.. hit me up if you would like to educate me 🤔

Thanks for reading

#linux #obs #appimage

Project setup from Template

Start from the button “New Image” in the UI.

This creates an entry in the “Meta” Settings of the Project. I first tried to follow the instructions by only referencing project="OBS:AppImage", but this will not work. The specific OpenSuse build version (project="OBS:AppImage:Templates:Leap:15.2") is required. The button “New Image” in the UI will produce a new “Project” with the proper “Meta” specification which you can copy/paste to the desired project (in my case “home”).

Adjust the Meta of the project to use the Template for x86_64 and aarch64 AppImages:

  <repository name="AppImage">
    <path project="OBS:AppImage:Templates:Leap:15.2" repository="AppImage"/>
    <arch>x86_64</arch>
    <arch>aarch64</arch>
  </repository>

appimage.yml

Start crafting the appimage.yml file. The “most simple example” was useful as template.

Builds from source can fetch from Git or from archives (e.g., .tar.gz). Because I publish the application in 2 flavors (“nightly” build and stable release), I built the “nightly” build from the latest commit from Git (master branch) and the “stable” release from the tar.gz sources.

I needed to make install the application to DESTDIR $BUILD_APPDIR for it to be available in the resulting AppImage (see difference between source and build dir).

.desktop file

.desktop file and app icon are required in the build directory of the AppDir specification. I did not know that the .desktop file Exec directive does not work with absolute paths 😲.

Also something I learned about .desktop files: “The Icon= entry SHOULD NOT contain the file extension, the actual filename of the file, however, SHOULD carry the extension.” (see AppDir specs).

During few rebuilds, I noticed that newer versions of my .desktop file are not copied to the resulting AppImage. There was some sort of caching going on in the runner, which always included an earlier version of the file. Triggering a rebuild did not replace the file, only pruning the $BUILD_SOURCE_DIR with an extra build reset the runner and the next build had the updated version of the .desktop file included.

  script:
  - rm -rf $BUILD_SOURCE_DIR/*

Distributing as AppImage

Also, I was wondering why the download page for the package does not show icons/buttons to download the AppImage. Because this options was missing, I followed the advice to include an extra link on the website of the project.

#nomad #openbao #jenkins #homelab

I have two automatic/system jobs that require a Token.

• The backup cron job is taking Nomad snapshots in regular intervals. The snapshot API requires a management token and the Nomad policy capability for snapshots with the operator are not implemented yet.
• The Jenkins server runs Nomad jobs using the Nomad cloud plugin. This system needs a Token to access the Nomad API (AppRole not compatible, see below).

I still keep my bootstrapping token around just in case I ever need it. That's ok unlike to procedures with OpenBao root tokens..

The bootstrap token can be deleted and is like any other token, care should be taken to not revoke all management tokens.

Name             Type        Global  Accessor ID  Expired
Bootstrap Token  management  true    ***          false
Snapshot         management  false   ***          false
OIDC-vault       client      true    ***          false
Jenkins          client      false   ***          false

The auth method in Nomad is still called Vault, never mind..

The access for human users is authenticated by an OIDC provider in my OpenBao server.

Because I already had an identity and alias setup (userpass authentication) including a group, I only needed to configure the provider and the assignment to the group to allow authentication with the new provider.

In the OpenBao server, the default provider and the allow_all assignment cannot be deleted. I assume it is similar to the “master” realm in a Keycloak instance 🤔.

I had to define a NOMAD_TOKEN as “Jenkins credential”, because the Nomad cloud plugin for Jenkins cannot read secrets from OpenBao using an AppRole (the Nomad jobs spawned by the plugin can do this, just not the plugin itself).

When I type nomad login in the shell, the browser opens and I can authenticate with OpenBao. What could be improved is outputting the OIDC redirect URI in the terminal. This is helpful when you need to login from disconnected machines (i.e., not the shell on your local machine).

#raspberry #homelab #debian #qemu

I needed a virtual host to test Ansible scripts for my Raspberry Pi in the home lab. I found my way around this task by reading through the many great posts and examples online.

Extract kernel and device tree

First, you have to get the image (in my case, I used a modified build of the bookworm image) and extract the “kernel” and the “device tree” (to be honest, this was new for me when I read about this today).

We do this by mounting the boot partition and extracting the relevant files.

# Check image partitions
$ fdisk -l ./HashiPi-pi0.img

Setup loop device with image, scan partitions. This is easier than fiddling with fdisk and partition offsets for mounting.

$ sudo losetup -P /dev/loop0 HashiPi-pi0.img
$ sudo mount /dev/loop0p1 /mnt

# copy kernel and device tree binary (dtb)
$ cp /mnt/kernel8.img .
$ cp /mnt/bcm2711-rpi-4-b.dtb .

# unmount
$ sudo umount /mnt
$ sudo losetup --detach /dev/loop0

Patch device tree to enable USB controller

The “device tree binary” (.dtb) needs to be translated into a readable “device tree source” (.dts) file.

$ dtc -I dtb -O dts -o bcm2711-rpi-4-b.dts bcm2711-rpi-4-b.dtb

The .dts file can be patched to enable the usb controller. This is required, if we want to boot later using the usbnet device and port-forward (hostfwd) the ssh port.

--- bcm2711-rpi-4-b.dts.orig    2025-09-21 15:05:59.304575294 +0200
+++ bcm2711-rpi-4-b.dts 2025-09-21 15:04:56.709581742 +0200
@@ -1450,7 +1450,7 @@
                        phy-names = "usb2-phy";
                        interrupt-names = "usb", "soft";
                        power-domains = <0x10 0x06>;
-                       status = "disabled";
+                       status = "okay";
                        phandle = <0xbf>;
                };

Unfortunately, we need to use this usb device, because all other emulated network devices are pci based which is not supported by QEMU for the Raspberry Pi

$ qemu-system-aarch64 -device help
...
Network devices:
name "e1000", bus PCI, alias "e1000-82540em", desc "Intel Gigabit Ethernet"
name "e1000-82544gc", bus PCI, desc "Intel Gigabit Ethernet"
name "e1000-82545em", bus PCI, desc "Intel Gigabit Ethernet"
name "e1000e", bus PCI, desc "Intel 82574L GbE Controller"
name "i82550", bus PCI, desc "Intel i82550 Ethernet"
name "i82551", bus PCI, desc "Intel i82551 Ethernet"
name "i82557a", bus PCI, desc "Intel i82557A Ethernet"
name "i82557b", bus PCI, desc "Intel i82557B Ethernet"
name "i82557c", bus PCI, desc "Intel i82557C Ethernet"
name "i82558a", bus PCI, desc "Intel i82558A Ethernet"
name "i82558b", bus PCI, desc "Intel i82558B Ethernet"
name "i82559a", bus PCI, desc "Intel i82559A Ethernet"
name "i82559b", bus PCI, desc "Intel i82559B Ethernet"
name "i82559c", bus PCI, desc "Intel i82559C Ethernet"
name "i82559er", bus PCI, desc "Intel i82559ER Ethernet"
name "i82562", bus PCI, desc "Intel i82562 Ethernet"
name "i82801", bus PCI, desc "Intel i82801 Ethernet"
name "igb", bus PCI, desc "Intel 82576 Gigabit Ethernet Controller"
name "ne2k_pci", bus PCI
name "pcnet", bus PCI
name "rocker", bus PCI, desc "Rocker Switch"
name "rtl8139", bus PCI
name "tulip", bus PCI
-> name "usb-net", bus usb-bus
name "virtio-net-device", bus virtio-bus # No 'virtio-bus' bus found for device 'virtio-net-device'
name "virtio-net-pci", bus PCI, alias "virtio-net"
name "virtio-net-pci-non-transitional", bus PCI
name "virtio-net-pci-transitional", bus PCI
name "vmxnet3", bus PCI, desc "VMWare Paravirtualized Ethernet v3"

We really need the usb device (virtio-net-device does not work either I tried), that's the reason for the patch.

A comment of the “interrupt.memfault blog” describes it nicely (very helpful community):

Note that for raspi4, the bcm2711-rpi-4-b.dtb devicetree file has disabled the USB controller. So, to enable USB keyboard & mouse, the .dtb file must be decompiled to .dts, patched, and recompiled back to .dtb.

Unfortunately, it's not only needed for keyboard and mouse, but also to make our network adapter work (for ssh port-forwarding).

Thaa patching.. Then recompiling into binary form with dtc:

$ dtc -I dtb -O dts -o bcm2711-rpi-4-b.dts bcm2711-rpi-4-b.dtb

Would be really nice to have this usb controller enabled by default in the next Trixie Pi OS 🤞

Boot the image

For booting the image, I had to ensure two things for the kernel arguments for the Raspberry 4:

• Use the ttyAMA1 console
• Use the root partition mmcblk1p2

Note that I'm not enabling the keyboard & mouse devices (as suggested in the references online), because I'm mainly interested to connect to the machine remotely via the ssh port fowarding:

$ sudo qemu-system-aarch64 \
  -machine raspi4b -cpu cortex-a72 \
  -dtb bcm2711-rpi-4-b-mod.dtb \ # use mod device tree
  -m 2G -smp 4 \
  -kernel kernel8.img -sd HashiPi-pi0.img \
  -append "rw earlyprintk loglevel=8 console=ttyAMA1,115200 dwc_otg.lpm_enable=0 root=/dev/mmcblk1p2 rootdelay=1" \
  -device usb-net,netdev=net0 \
  -netdev user,id=net0,hostfwd=tcp::2222-:22

That's it. Is still a bit slow, but good enough to throw some Ansible against the wall and see if it sticks.

My idea here is to do less with the HashiCorp packer scripts and go back to more Ansible, because that might be more sustainable in the long run.. Let's see. Next I'm probably also going to give the Trixie (nightly image) a try.

References

• https://interrupt.memfault.com/blog/emulating-raspberry-pi-in-qemu
• https://www.qemu.org/docs/master/system/qemu-manpage.html
• https://www.qemu.org/docs/master/system/arm/raspi.html
• https://github.com/trinitronx/qemu-raspbian/blob/main/run-raspi4.sh
• https://github.com/trinitronx/qemu-raspbian/blob/main/bcm2711-rpi-4-b.dts.patch
• https://downloads.raspberrypi.org
• https://www.man7.org/linux/man-pages/man8/losetup.8.html

#podman #jenksin #cicd

My Jenkins setup in the homelab

Recently I wanted to setup a LaTex pipeline in my Jenkins. During this task I noticed, that certain images fail with a particular error message, that was not meaningful at all (in hindsight).

The Nomad cloud plugin for Jenkins starts Jenkins workers as Nomad jobs. These jobs are started from an image that includes the Jenkins inbound agent. This agent connects to the Jenkins server.

Alongside the Jenkins agent, I also install Buildah and Docker on that image, because I use these Jenkins workers to build container images.

Besides building images (with Buildah), I also use the Docker workflow to run arbitrary container images in my Jenkins pipelines.

The agent image mounts the Podman socket of the Nomad node. It does not mount the socket of the “root” Podman process, only the rootless socket of a particular jenkins user on the nodes.

I reserved this users on the nodes (my Raspberry Pies) for this particular purpose.

In that sense, whenever I run an image with the Docker workflow plugin in my Jenkins pipeline, it is started as rootless Podman container under that jenkins user on the Nomad node where the pipeline worker is scheduled.

VFS default storage driver for rootless Podman

I only noticed today, that even though I have a more recent kernel than 5.12.9 (I run 6.12.41), the rootless Podman configuration does not check the kernel version but simply defaults to the VFS storage.

The default storage driver for UID 0 is configured in containers-storage.conf(5) in rootless mode), and is vfs for non-root users when fuse-overlayfs is not available.

I recently uninstall fuse-overlayfs from my Pies, because I run a recent kernel 5.12.9 that supports native overlayfs.

Improve performance for Docker workflow in Jenkins

In order to achieve better performance and fix the weird and unclear error in my Jenkins pipeline that use Docker workflow, I had to migrate to overlayfs storage driver for the rootless Podman socket (of the jenkins user).

As jenkins user on the nodes, cleanup the old vfs storage:

$ podman system reset

Also had to help a bit (as root):

# rm -rf /home/jenkins/.local/share/containers/

Then change the the storage driver setting (I changed it globally in /etc/containers/storage.conf because my root Podman process already runs with overlayfs storage driver anyways, so it's basically a global default now).

# Configure overlay storage for Podman
cat <<EOF > /etc/containers/storage.conf
[storage]
driver="overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"
EOF

This explicit setting now also holds for the rootless Podman socket. I was not aware of that. It can be checked with the jenkins user (rootless Podman):

$ podman info | grep graph
  graphDriverName: overlay

Really glad I got this fixed.

Let me know your Podman/Buildah/Jenkins stories on the Fediverse or via chat.

#selfhosting #homelab #docker #haproxy

I observed an interesting issue in my Jenkins pipeline. The image pull aborted with the following error message:

Error: writing blob: storing blob to file "/var/tmp/storage1360560957/1": happened during read: unexpected EOF

First I thought it has something to do with the storage. But I was wrong. The culprit was the network.

More specifically, I noticed that pulling through my HAProxy instance was the issue, but pulling through the nodes registry port (directly) was fine.

When looking into the HAProxy logs, I noticed that the requests fail with the particular error flags cD:

Sep 02 02:23:29 haproxy haproxy[2836]: 10.0.0.102:34982 [02/Sep/2025:02:22:47.563] registryfront registry/pi3 0/0/0/87/42156 200 466612119 - - cD-- 6/1/0/0/0 0/0 {haproxy.lan:5000} "GET /v2/texlive/blobs/sha256:2fde6c0b50af2b1fda7ed0092ad1f1cc6897d7cb723dfcb0d2bc15201bbd7191 HTTP/1.1"

The HAProxy docs on stream states:

     cD   The client did not send nor acknowledge any data for as long as the
          "timeout client" delay. This is often caused by network failures on
          the client side, or the client simply leaving the net uncleanly.

First flag c:

  - On the first character, a code reporting the first event which caused the
    stream to terminate :

        c : the client-side timeout expired while waiting for the client to
            send or receive data.

Second flag D:

  - on the second character, the TCP or HTTP stream state when it was closed :

        D : the stream was in the DATA phase.

That was useful – “the client-side timeout expired”. It simply means that I need to bump the client timeouts (to 30m from 5s in this example) in the HAProxy frontend for my Docker registry.

frontend registryfront
    bind                 :5000
    timeout              client 30m # was 5s
    timeout              client-fin 30m # was 30s
    mode                 http
    option               httplog
                         # display host header in logs
    capture              request header Host len 30

    default_backend      registry

The pull request through the proxy afterwards show no error flags (----):

Sep 02 02:30:27 haproxy haproxy[2850]: 10.0.0.102:53506 [02/Sep/2025:02:27:03.674] registryfront registry/pi3 0/0/0/15/203648 200 2334919898 - - ---- 7/1/0/0/0 0/0 {haproxy.lan:5000} "GET /v2/texlive/blobs/sha256:2fde6c0b50af2b1fda7ed0092ad1f1cc6897d7cb723dfcb0d2bc15201bbd7191 HTTP/1.1"

Podman pull succeeds 🎉

Ping me in chat or Fediverse if you have more suggestions regarding HAProxy configuration for private Docker registries. Happy self-hosting!

#skateboarding #sport

The inspiration to build my own ledge came from a colleague who refurbished an old mini ramp. So I thought I could actually do a similar thing in my neighborhood and see how it works out.

I followed the Red Bull guide to the dot. Only made the ledge a bit longer (250cm) because these are the standard wood ply lengths that you can buy around here.

The final result is quite ok👌🏼

Of course, if you look closely, you can see that I'm not the most skilled handyman :) Nevertheless, I'm pretty happy with the result.

I only have a few questions regarding the durability of the top sheet. Maybe a small 6mm weather resistant ply would be better, we will see..

The ledge is waxed and ready to be tested in my neighborhood.

I wonder how long it will take until someone protests or destroys/removes the ledge. Until then, it's basically my personal skatepark :) and open for all. Happy skateboarding folks!

I'll update the thread here on the Fediverse once I have some footage, it's still raining cats around here ⛈️🐱

#borg #rclone #backup

A while ago I purchased quite some TB of storage at pCloud, a small Swiss cloud storage provider. Only recently, I found out that you can use rclone to push files and data to the cloud in an rsync-like fashion.

Since I wanted to use that approach to backup files from my router, I first figured ways to install it with opkg. I followed the Turris docs to switch to a more recent feed/branch “Here be Lions” but noticed quickly, that I actually needed to install Borg2. What most distributions package nowadays is Borg (verison 1), which is the stable release. Borg2 is a breaking change (incompatible, but runs the new feature that supports rclone).

Therefore, I decided to build a small Debian 12 lxc container and run Borg inside the container. That approach worked well.

To mount a host path inside the container, I used the following modification of the lxc config file:

# Mount host ssd
lxc.mount.entry = /srv host-srv none bind,create=dir 0 0

This will mount the /srv directory of the Turris host to the container, so the Borg process can backup files from that path.

The rclone setup with pCloud was straight forward, I simply needed to confirm the login from my laptop with a browser, because the Turris cannot connect via browser to confirm the login on pCloud. Easy!

The next step was configuring the Borg backup repository on pCloud.

My borg configuration points to the backup folder on pCloud:

# ~/.config/borg/config 
BORG_REPO='rclone:pcloud:backup-folder'
BORG_PASSPHRASE='***'
PATTERNSFILE='/root/.config/borg/patterns.lst'

I also maintain a patterns file, but won't go into details here.

borg repo-create --encryption=repokey-chacha20-poly1305

⚠️ Note that the repo-create and also a lot of other commands in Borg2 are similar, but different from the Borg v1 commands. A little bit confusing when reading the docs, but you'll get the hang of it..

I chose to use chacha20-poly1305 encryption mode, because that combination was suggested to me as the fastest algorithm on my Turris. You can find out the speed of the different supported algorithms by running:

borg benchmark cpu

This is yet another convenient feature of Borg2.

Lastly, I started the backup using a Systemd timer and some variation of borg create combined with another prune command to prune old backups.

So far, so good. I'm happy that I finally found a suitable solution for my backup. The process seems quite slow though. In my create borg command I included the --compression lz4 (which should be Speedy Gonzales), but more than 8h for the first backup? Common..

I found a small signal trick that allowed me to check the current state of the upload process. This will show me the current file of the upload process:

kill -s USR1 $(pidof python) && journalctl -eu borg-backup | tail -1

I could also contribute a small improvement in the Borg docs regarding the compilation of the dependencies required for Borg2 🎉 so the hassle was worth it.

For automation of the entire procedure, I created a small packer build script that can rebuild the lxc container from scratch whenever needed. Essentially, it contains the commands to install the prerequisites for Borg on Debian followed by the installation with Pip in an virtual environment and the configuration with rclone.

Edit 2024-12-17

Turris Omnia Borg benchmark:

Chunkers =======================================================
buzhash,19,23,21,4095    1GB        12.052s
fixed,1048576            1GB        2.655s
Non-cryptographic checksums / hashes ===========================
xxh64                    1GB        2.394s
crc32 (zlib)             1GB        2.970s
Cryptographic hashes / MACs ====================================
hmac-sha256              1GB        10.785s
blake2b-256              1GB        24.264s
Encryption =====================================================
aes-256-ctr-hmac-sha256  1GB        39.176s
aes-256-ctr-blake2b      1GB        51.622s
aes-256-ocb              1GB        34.483s
chacha20-poly1305        1GB        12.335s
KDFs (slow is GOOD, use argon2!) ===============================
pbkdf2                   5          1.969s
argon2                   5          7.606s
Compression ====================================================
lz4          0.1GB      0.239s
zstd,1       0.1GB      0.739s
zstd,3       0.1GB      0.923s
zstd,5       0.1GB      16.528s
zstd,10      0.1GB      26.171s
zstd,16      0.1GB      51.617s
zstd,22      0.1GB      64.239s
zlib,0       0.1GB      0.703s
zlib,6       0.1GB      15.758s
zlib,9       0.1GB      16.217s
lzma,0       0.1GB      88.872s
lzma,6       0.1GB      114.931s
lzma,9       0.1GB      96.051s
msgpack ========================================================
msgpack      100k Items 0.818s

For comparison, on my x230, where I would choose blake2 for hashing (blake2-chacha20-poly1305):

Chunkers =======================================================
buzhash,19,23,21,4095    1GB        1.360s
fixed,1048576            1GB        0.134s
Non-cryptographic checksums / hashes ===========================
xxh64                    1GB        0.097s
crc32 (zlib)             1GB        0.406s
Cryptographic hashes / MACs ====================================
hmac-sha256              1GB        3.474s
blake2b-256              1GB        1.980s
Encryption =====================================================
aes-256-ctr-hmac-sha256  1GB        3.901s
aes-256-ctr-blake2b      1GB        3.938s
aes-256-ocb              1GB        0.572s
chacha20-poly1305        1GB        1.251s
KDFs (slow is GOOD, use argon2!) ===============================
pbkdf2                   5          0.362s
argon2                   5          0.434s
Compression ====================================================
lz4          0.1GB      0.022s
zstd,1       0.1GB      0.041s
zstd,3       0.1GB      0.062s
zstd,5       0.1GB      0.104s
zstd,10      0.1GB      0.188s
zstd,16      0.1GB      13.794s
zstd,22      0.1GB      14.795s
zlib,0       0.1GB      0.067s
zlib,6       0.1GB      2.735s
zlib,9       0.1GB      2.740s
lzma,0       0.1GB      18.819s
lzma,6       0.1GB      36.049s
lzma,9       0.1GB      30.293s
msgpack ========================================================
msgpack      100k Items 0.258s

#tls #pqc #PKI #certificates

liboqs is a library for prototyping and experimenting with quantum-resistant cryptography. It's used by the unsupported fork of OpenSSL of the Open Quantum Safe (OQS) project. With that fork, we can easily create TSL certificates that can be used in a PQ-secure client-server TLS communication.

First you have to build the library, liboqs. On NixOS, I use these dependencies for the build:

# shell.liboqs.nix
{ pkgs ? import <nixpkgs> {} }:
  pkgs.mkShell {
    nativeBuildInputs = with pkgs.buildPackages; [
      cmake
      pkg-config
      openssl
      ninja
      libtool
      gcc
    ];
    buildInputs = with pkgs.buildPackages; [ openssl ];
    LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath [ pkgs.openssl ];
}

The build process itself is straight forward. It involves building liboqs and the OpenSSL fork:

# prepare environment
$ nix-shell shell.liboqs.nix

# clone openssl fork
$ git clone --branch OQS-OpenSSL_1_1_1-stable \
   https://github.com/open-quantum-safe/openssl.git openssl.git

# clone liboqs
$ git clone -b main https://github.com/open-quantum-safe/liboqs.git liboqs.git

$ cd liboqs.git

# build liboqs into the folder of the openssl fork (CMAKE_INSTALL_PREFIX)
$ cmake -GNinja -DCMAKE_INSTALL_PREFIX=$HOME/Downloads/pqc/openssl.git/

# openssl >= 3.0.0 (3.0.14)
$ oqs -DOQS_USE_OPENSSL=OFF ..

$ ninja
$ ninja install

# build openssl fork
$ cd ../openssl.git
$ ./Configure no-shared linux-x86_64 -lm
$ make -j

$ apps/openssl version
OpenSSL 1.1.1u  30 May 2023, Open Quantum Safe 2023-07

Luckily, the project includes short instructions on how to use the PQ-ready OpenSSL version to create web server certificates:

# create hybrid rsa/dilithium CA with the provided ssl config
$ apps/openssl req -x509 -new -newkey rsa3072_dilithium2 \
   -keyout rsa3072_dilithium2_CA.key -out rsa3072_dilithium2_CA.crt \
   -nodes -subj "/CN=oqstest CA" -days 365 -config apps/openssl.cnf

# check CA certificate
$ apps/openssl x509 -in rsa3072_dilithium2_CA.crt -noout -text

# create hybrid rsa/dilithium server cert
$ apps/openssl req -new -newkey rsa3072_dilithium2 \
   -keyout rsa3072_dilithium2_srv.key -out rsa3072_dilithium2_srv.csr \
   -nodes -subj "/CN=oqstest server" -config apps/openssl.cnf

# sign server cert with CA cert
$ apps/openssl x509 -req -in rsa3072_dilithium2_srv.csr \
   -out rsa3072_dilithium2_srv.crt -CA rsa3072_dilithium2_CA.crt \
   -CAkey rsa3072_dilithium2_CA.key -CAcreateserial -days 365

# check server cert
$ apps/openssl x509 -in rsa3072_dilithium2_srv.crt -noout -text

# run the server
$ apps/openssl s_server -cert rsa3072_dilithium2_srv.crt \
   -key rsa3072_dilithium2_srv.key -www -tls1_3

# run the client with kyber KEX
apps/openssl s_client -groups p384_kyber768 -CAfile rsa3072_dilithium2_CA.crt

What I noticed during playing with the new algorithms: The term “Hybrid” does not mean you can choose the type of certificate for the Key exchange (<KEX>) or signature verficiation <SIG> standard. It simply means that you need both. Think of it as a fallback. If Kyber (<KEX>) or Dilithium (<SIG>) would turn out to not be that secure as everyone thought, your TLS communication (key exchange and signature verification) will still be backed by a proven industry standard algorithm (RSA or ECDSA), because you will always need to apply both algorithms to verify the signature or decrypt the traffic. Of course, this has impact on performance (e.g., time to create, encrypt/decrypt and/or verify).

Lastly, I also checked out how I can sign a message in PQ-safe way using the Cryptographic Message Syntax (CMS):

# sign file
$ apps/openssl dgst -sign rsa3072_dilithium2_srv.key -sha256 \
   -out binary.sig -binary binary

# extract pubkey
$ apps/openssl x509 -in rsa3072_dilithium2_srv.crt -noout \
   -pubkey > rsa3072_dilithium2_srv.pem

# check signature
$ apps/openssl dgst -verify rsa3072_dilithium2_srv.pem -sha256 \
   -signature binary.sig -binary binary

Of course we could go on playing with these demos, for instance, by building an application (like Nginx or curl, see oqs-demos) with support for the new algorithms. Oh, and don't forget VPNs..

There is also a website that shows you where these new algorithms typically fail: https://tldr.fail. Also, I was wondering, when I can simply request these hybrid certifcates from my known and loved HashiCorp Vault PKI 🤗?

• https://github.com/hashicorp/vault/issues/27239
• https://www.hashicorp.com/blog/nist-s-post-quantum-cryptography-standards-our-plans

It will take some time, but I'm ready (also enabled that feature toggle in my Firefox to let servers now). Let me know what you think about the topic.

(now that I wrote the blog post I can go ahead and delete that temporary folder on my Desktop)

#updatecli #pipeline #jenkins #myheats #nodejs #npm

I was looking for a way to automatically bump the version of the npm dependencies (package.json) whenever there is an update available. This is also important for security reasons (e.g., have a look at the output of npm audit from time to time to see the recent security issues in the dependencies).

I was looking into Renovate and Dependabot, but neither of these scratched my itch of simple automatic dependency updates.

A coworker suggested me to try Updatecli and it fits my workflows perfectly well. The Jenkins example on the projects website got me started. So I created a Jenkins shared library function to run my own build, which includes npm to perform the version bumps:

• A class to describe the updatecli stages: https://code.in0rdr.ch/jenkins-lib/file/src/Updatecli.groovy.html

The scripted pipeline in the repository of the application loads the library and performs the version bumps to a new branch:

• The Jenkinsfile that makes use of the updatecli groovy library: https://code.in0rdr.ch/myheats/file/Jenkinsfile.html

I did not even have to configure Updatecli a lot, because the autodiscovery feature automatically detects that this is a npm repository/project. The final version of my pipeline includes all the git/scm steps in the updatecli.d/default.yaml configuration file:

• Updatecli configuration file: https://code.in0rdr.ch/myheats/file/updatecli.d/default.yaml.html

First I tried to perform the SCM/git steps in Jenkins checkout and sh steps. But I noticed it could be much sleeker by defining the SCM/git settings in the Updatecli config file directly. This way, updatecli takes care of the clone/checkout/push steps. Here the extract from my previous pipeline with the “manual git steps” for comparison:

// alternative approach I did not pursue any further
sh '''
git config --global user.name "$GIT_AUTHOR_NAME"
git config --global user.email "$GIT_AUTHOR_EMAIL"
'''

dir("myyheats.git-$BUILD_NUMBER") {
  // checkout update branch in new directory
  checkout scmGit(
      extensions: [localBranch("$branch")],
      userRemoteConfigs: [[url: 'https://git.in0rdr.ch/myheats.git']]
  )

  updatecli.run('apply')

  // commit changes
  sh '''
  git add -u
  git commit -m "chore(updatecli-$BUILD_NUMBER): bump node modules"
  git push -f -u origin "$branch"
  '''
}

I definitely like the updatecli configuration better, since it keeps the actual pipeline tidy. Also, I like how you can use the {{ requiredEnv "GIT_PASSWORD" }} configuration in updatecli to read secrets from the environment. The Git credentials are sourced from OpenBao with Nomad workload identities.

I hope the post is helpful for anyone that would like to give updatecli a try or would like to configure a similar Jenkins pipeline.