#nomad #openbao #jenkins #homelab

I have two automatic/system jobs that require a Token.

• The backup cron job is taking Nomad snapshots in regular intervals. The snapshot API requires a management token and the Nomad policy capability for snapshots with the operator are not implemented yet.
• The Jenkins server runs Nomad jobs using the Nomad cloud plugin. This system needs a Token to access the Nomad API (AppRole not compatible, see below).

I still keep my bootstrapping token around just in case I ever need it. That's ok unlike to procedures with OpenBao root tokens..

The bootstrap token can be deleted and is like any other token, care should be taken to not revoke all management tokens.

Name             Type        Global  Accessor ID  Expired
Bootstrap Token  management  true    ***          false
Snapshot         management  false   ***          false
OIDC-vault       client      true    ***          false
Jenkins          client      false   ***          false

The auth method in Nomad is still called Vault, never mind..

The access for human users is authenticated by an OIDC provider in my OpenBao server.

Because I already had an identity and alias setup (userpass authentication) including a group, I only needed to configure the provider and the assignment to the group to allow authentication with the new provider.

In the OpenBao server, the default provider and the allow_all assignment cannot be deleted. I assume it is similar to the “master” realm in a Keycloak instance 🤔.

I had to define a NOMAD_TOKEN as “Jenkins credential”, because the Nomad cloud plugin for Jenkins cannot read secrets from OpenBao using an AppRole (the Nomad jobs spawned by the plugin can do this, just not the plugin itself).

When I type nomad login in the shell, the browser opens and I can authenticate with OpenBao. What could be improved is outputting the OIDC redirect URI in the terminal. This is helpful when you need to login from disconnected machines (i.e., not the shell on your local machine).

#raspberry #homelab #debian #qemu

I needed a virtual host to test Ansible scripts for my Raspberry Pi in the home lab. I found my way around this task by reading through the many great posts and examples online.

Extract kernel and device tree

First, you have to get the image (in my case, I used a modified build of the bookworm image) and extract the “kernel” and the “device tree” (to be honest, this was new for me when I read about this today).

We do this by mounting the boot partition and extracting the relevant files.

# Check image partitions
$ fdisk -l ./HashiPi-pi0.img

Setup loop device with image, scan partitions. This is easier than fiddling with fdisk and partition offsets for mounting.

$ sudo losetup -P /dev/loop0 HashiPi-pi0.img
$ sudo mount /dev/loop0p1 /mnt

# copy kernel and device tree binary (dtb)
$ cp /mnt/kernel8.img .
$ cp /mnt/bcm2711-rpi-4-b.dtb .

# unmount
$ sudo umount /mnt
$ sudo losetup --detach /dev/loop0

Patch device tree to enable USB controller

The “device tree binary” (.dtb) needs to be translated into a readable “device tree source” (.dts) file.

$ dtc -I dtb -O dts -o bcm2711-rpi-4-b.dts bcm2711-rpi-4-b.dtb

The .dts file can be patched to enable the usb controller. This is required, if we want to boot later using the usbnet device and port-forward (hostfwd) the ssh port.

--- bcm2711-rpi-4-b.dts.orig    2025-09-21 15:05:59.304575294 +0200
+++ bcm2711-rpi-4-b.dts 2025-09-21 15:04:56.709581742 +0200
@@ -1450,7 +1450,7 @@
                        phy-names = "usb2-phy";
                        interrupt-names = "usb", "soft";
                        power-domains = <0x10 0x06>;
-                       status = "disabled";
+                       status = "okay";
                        phandle = <0xbf>;
                };

Unfortunately, we need to use this usb device, because all other emulated network devices are pci based which is not supported by QEMU for the Raspberry Pi

$ qemu-system-aarch64 -device help
...
Network devices:
name "e1000", bus PCI, alias "e1000-82540em", desc "Intel Gigabit Ethernet"
name "e1000-82544gc", bus PCI, desc "Intel Gigabit Ethernet"
name "e1000-82545em", bus PCI, desc "Intel Gigabit Ethernet"
name "e1000e", bus PCI, desc "Intel 82574L GbE Controller"
name "i82550", bus PCI, desc "Intel i82550 Ethernet"
name "i82551", bus PCI, desc "Intel i82551 Ethernet"
name "i82557a", bus PCI, desc "Intel i82557A Ethernet"
name "i82557b", bus PCI, desc "Intel i82557B Ethernet"
name "i82557c", bus PCI, desc "Intel i82557C Ethernet"
name "i82558a", bus PCI, desc "Intel i82558A Ethernet"
name "i82558b", bus PCI, desc "Intel i82558B Ethernet"
name "i82559a", bus PCI, desc "Intel i82559A Ethernet"
name "i82559b", bus PCI, desc "Intel i82559B Ethernet"
name "i82559c", bus PCI, desc "Intel i82559C Ethernet"
name "i82559er", bus PCI, desc "Intel i82559ER Ethernet"
name "i82562", bus PCI, desc "Intel i82562 Ethernet"
name "i82801", bus PCI, desc "Intel i82801 Ethernet"
name "igb", bus PCI, desc "Intel 82576 Gigabit Ethernet Controller"
name "ne2k_pci", bus PCI
name "pcnet", bus PCI
name "rocker", bus PCI, desc "Rocker Switch"
name "rtl8139", bus PCI
name "tulip", bus PCI
-> name "usb-net", bus usb-bus
name "virtio-net-device", bus virtio-bus # No 'virtio-bus' bus found for device 'virtio-net-device'
name "virtio-net-pci", bus PCI, alias "virtio-net"
name "virtio-net-pci-non-transitional", bus PCI
name "virtio-net-pci-transitional", bus PCI
name "vmxnet3", bus PCI, desc "VMWare Paravirtualized Ethernet v3"

We really need the usb device (virtio-net-device does not work either I tried), that's the reason for the patch.

A comment of the “interrupt.memfault blog” describes it nicely (very helpful community):

Note that for raspi4, the bcm2711-rpi-4-b.dtb devicetree file has disabled the USB controller. So, to enable USB keyboard & mouse, the .dtb file must be decompiled to .dts, patched, and recompiled back to .dtb.

Unfortunately, it's not only needed for keyboard and mouse, but also to make our network adapter work (for ssh port-forwarding).

Thaa patching.. Then recompiling into binary form with dtc:

$ dtc -I dtb -O dts -o bcm2711-rpi-4-b.dts bcm2711-rpi-4-b.dtb

Would be really nice to have this usb controller enabled by default in the next Trixie Pi OS 🤞

Boot the image

For booting the image, I had to ensure two things for the kernel arguments for the Raspberry 4:

• Use the ttyAMA1 console
• Use the root partition mmcblk1p2

Note that I'm not enabling the keyboard & mouse devices (as suggested in the references online), because I'm mainly interested to connect to the machine remotely via the ssh port fowarding:

$ sudo qemu-system-aarch64 \
  -machine raspi4b -cpu cortex-a72 \
  -dtb bcm2711-rpi-4-b-mod.dtb \ # use mod device tree
  -m 2G -smp 4 \
  -kernel kernel8.img -sd HashiPi-pi0.img \
  -append "rw earlyprintk loglevel=8 console=ttyAMA1,115200 dwc_otg.lpm_enable=0 root=/dev/mmcblk1p2 rootdelay=1" \
  -device usb-net,netdev=net0 \
  -netdev user,id=net0,hostfwd=tcp::2222-:22

That's it. Is still a bit slow, but good enough to throw some Ansible against the wall and see if it sticks.

My idea here is to do less with the HashiCorp packer scripts and go back to more Ansible, because that might be more sustainable in the long run.. Let's see. Next I'm probably also going to give the Trixie (nightly image) a try.

References

• https://interrupt.memfault.com/blog/emulating-raspberry-pi-in-qemu
• https://www.qemu.org/docs/master/system/qemu-manpage.html
• https://www.qemu.org/docs/master/system/arm/raspi.html
• https://github.com/trinitronx/qemu-raspbian/blob/main/run-raspi4.sh
• https://github.com/trinitronx/qemu-raspbian/blob/main/bcm2711-rpi-4-b.dts.patch
• https://downloads.raspberrypi.org
• https://www.man7.org/linux/man-pages/man8/losetup.8.html

#selfhosting #homelab #docker #haproxy

I observed an interesting issue in my Jenkins pipeline. The image pull aborted with the following error message:

Error: writing blob: storing blob to file "/var/tmp/storage1360560957/1": happened during read: unexpected EOF

First I thought it has something to do with the storage. But I was wrong. The culprit was the network.

More specifically, I noticed that pulling through my HAProxy instance was the issue, but pulling through the nodes registry port (directly) was fine.

When looking into the HAProxy logs, I noticed that the requests fail with the particular error flags cD:

Sep 02 02:23:29 haproxy haproxy[2836]: 10.0.0.102:34982 [02/Sep/2025:02:22:47.563] registryfront registry/pi3 0/0/0/87/42156 200 466612119 - - cD-- 6/1/0/0/0 0/0 {haproxy.lan:5000} "GET /v2/texlive/blobs/sha256:2fde6c0b50af2b1fda7ed0092ad1f1cc6897d7cb723dfcb0d2bc15201bbd7191 HTTP/1.1"

The HAProxy docs on stream states:

     cD   The client did not send nor acknowledge any data for as long as the
          "timeout client" delay. This is often caused by network failures on
          the client side, or the client simply leaving the net uncleanly.

First flag c:

  - On the first character, a code reporting the first event which caused the
    stream to terminate :

        c : the client-side timeout expired while waiting for the client to
            send or receive data.

Second flag D:

  - on the second character, the TCP or HTTP stream state when it was closed :

        D : the stream was in the DATA phase.

That was useful – “the client-side timeout expired”. It simply means that I need to bump the client timeouts (to 30m from 5s in this example) in the HAProxy frontend for my Docker registry.

frontend registryfront
    bind                 :5000
    timeout              client 30m # was 5s
    timeout              client-fin 30m # was 30s
    mode                 http
    option               httplog
                         # display host header in logs
    capture              request header Host len 30

    default_backend      registry

The pull request through the proxy afterwards show no error flags (----):

Sep 02 02:30:27 haproxy haproxy[2850]: 10.0.0.102:53506 [02/Sep/2025:02:27:03.674] registryfront registry/pi3 0/0/0/15/203648 200 2334919898 - - ---- 7/1/0/0/0 0/0 {haproxy.lan:5000} "GET /v2/texlive/blobs/sha256:2fde6c0b50af2b1fda7ed0092ad1f1cc6897d7cb723dfcb0d2bc15201bbd7191 HTTP/1.1"

Podman pull succeeds 🎉

Ping me in chat or Fediverse if you have more suggestions regarding HAProxy configuration for private Docker registries. Happy self-hosting!