#wireguard #haproxy #homelab

Up until today I had an SSH reverse proxy (autossh) running for routing traffic from the Internet to my applications in the homelab. That was a poor mans solution which worked fine for most of the time. See Slide 6.

After reading Thibaults blog post A cheap VPS is a good front I decided to ditch the SSH reverse proxy in favor of a proper Wireguard connection.

As in the previous implementation, the ingress network traffic flows through a jumphost in the cloud. I still use the Oracle cloud free tier, it works reliably for my purposes.

To accept the network traffic on the jumphost, I installed a second HAProxy instance, which does simple “tcp passthrough” to the actual router (also a HAProxy, on LXC) in the internal network.

• The amd64 node on the internal network is the “Wireguard client peer”
• The jumphost on Oracle Cloud is the “Wireguard server peer”

I have PersistentKeepalive setup with 25 seconds as suggested to keep the tunnel alive. Wireguard connection is established from the internal network to the jumphost, as was the case for the SSH reverse proxy.

Jumphost HAProxy example config:

backend haproxy_lan_http
    server               haproxy $wg0:80 check send-proxy-v2
backend haproxy_lan_https
    server               haproxy $wg0:443 check send-proxy-v2
backend haproxy_lan_ssh
    server               haproxy $wg0:22 check send-proxy-v2

LXC HAProxy example config (internal network):

frontend httpfront
    bind                 :80 accept-proxy
    ...
frontend httpsfront
    bind                 :443 accept-proxy
    ...
frontend sshfront
    bind                 :22 accept-proxy

The proxy protocol allows me to see the real/original IP in the Anubis logs.

Instead of connecting the jumphost with the internal network via SSH reverse proxy (autossh), I now route the ingress through the Wireguard overlay network. I create Wirguard keypairs for each relevant node to establish connections.

I hope these kind of descriptions encourage you on your own self-hosting and homelab journeys 😁

#selfhosting #homelab #docker #haproxy

I observed an interesting issue in my Jenkins pipeline. The image pull aborted with the following error message:

Error: writing blob: storing blob to file "/var/tmp/storage1360560957/1": happened during read: unexpected EOF

First I thought it has something to do with the storage. But I was wrong. The culprit was the network.

More specifically, I noticed that pulling through my HAProxy instance was the issue, but pulling through the nodes registry port (directly) was fine.

When looking into the HAProxy logs, I noticed that the requests fail with the particular error flags cD:

Sep 02 02:23:29 haproxy haproxy[2836]: 10.0.0.102:34982 [02/Sep/2025:02:22:47.563] registryfront registry/pi3 0/0/0/87/42156 200 466612119 - - cD-- 6/1/0/0/0 0/0 {haproxy.lan:5000} "GET /v2/texlive/blobs/sha256:2fde6c0b50af2b1fda7ed0092ad1f1cc6897d7cb723dfcb0d2bc15201bbd7191 HTTP/1.1"

The HAProxy docs on stream states:

     cD   The client did not send nor acknowledge any data for as long as the
          "timeout client" delay. This is often caused by network failures on
          the client side, or the client simply leaving the net uncleanly.

First flag c:

  - On the first character, a code reporting the first event which caused the
    stream to terminate :

        c : the client-side timeout expired while waiting for the client to
            send or receive data.

Second flag D:

  - on the second character, the TCP or HTTP stream state when it was closed :

        D : the stream was in the DATA phase.

That was useful – “the client-side timeout expired”. It simply means that I need to bump the client timeouts (to 30m from 5s in this example) in the HAProxy frontend for my Docker registry.

frontend registryfront
    bind                 :5000
    timeout              client 30m # was 5s
    timeout              client-fin 30m # was 30s
    mode                 http
    option               httplog
                         # display host header in logs
    capture              request header Host len 30

    default_backend      registry

The pull request through the proxy afterwards show no error flags (----):

Sep 02 02:30:27 haproxy haproxy[2850]: 10.0.0.102:53506 [02/Sep/2025:02:27:03.674] registryfront registry/pi3 0/0/0/15/203648 200 2334919898 - - ---- 7/1/0/0/0 0/0 {haproxy.lan:5000} "GET /v2/texlive/blobs/sha256:2fde6c0b50af2b1fda7ed0092ad1f1cc6897d7cb723dfcb0d2bc15201bbd7191 HTTP/1.1"

Podman pull succeeds 🎉

Ping me in chat or Fediverse if you have more suggestions regarding HAProxy configuration for private Docker registries. Happy self-hosting!