#selfhosting #homelab #docker #haproxy

I observed an interesting issue in my Jenkins pipeline. The image pull aborted with the following error message:

Error: writing blob: storing blob to file "/var/tmp/storage1360560957/1": happened during read: unexpected EOF

First I thought it has something to do with the storage. But I was wrong. The culprit was the network.

More specifically, I noticed that pulling through my HAProxy instance was the issue, but pulling through the nodes registry port (directly) was fine.

When looking into the HAProxy logs, I noticed that the requests fail with the particular error flags cD:

Sep 02 02:23:29 haproxy haproxy[2836]: 10.0.0.102:34982 [02/Sep/2025:02:22:47.563] registryfront registry/pi3 0/0/0/87/42156 200 466612119 - - cD-- 6/1/0/0/0 0/0 {haproxy.lan:5000} "GET /v2/texlive/blobs/sha256:2fde6c0b50af2b1fda7ed0092ad1f1cc6897d7cb723dfcb0d2bc15201bbd7191 HTTP/1.1"

The HAProxy docs on stream states:

     cD   The client did not send nor acknowledge any data for as long as the
          "timeout client" delay. This is often caused by network failures on
          the client side, or the client simply leaving the net uncleanly.

First flag c:

  - On the first character, a code reporting the first event which caused the
    stream to terminate :

        c : the client-side timeout expired while waiting for the client to
            send or receive data.

Second flag D:

  - on the second character, the TCP or HTTP stream state when it was closed :

        D : the stream was in the DATA phase.

That was useful – “the client-side timeout expired”. It simply means that I need to bump the client timeouts (to 30m from 5s in this example) in the HAProxy frontend for my Docker registry.

frontend registryfront
    bind                 :5000
    timeout              client 30m # was 5s
    timeout              client-fin 30m # was 30s
    mode                 http
    option               httplog
                         # display host header in logs
    capture              request header Host len 30

    default_backend      registry

The pull request through the proxy afterwards show no error flags (----):

Sep 02 02:30:27 haproxy haproxy[2850]: 10.0.0.102:53506 [02/Sep/2025:02:27:03.674] registryfront registry/pi3 0/0/0/15/203648 200 2334919898 - - ---- 7/1/0/0/0 0/0 {haproxy.lan:5000} "GET /v2/texlive/blobs/sha256:2fde6c0b50af2b1fda7ed0092ad1f1cc6897d7cb723dfcb0d2bc15201bbd7191 HTTP/1.1"

Podman pull succeeds 🎉

Ping me in chat or Fediverse if you have more suggestions regarding HAProxy configuration for private Docker registries. Happy self-hosting!