Docker pull through HAProxy
This is a story about pulling Docker images through HAProxy in my home lab.
#selfhosting #homelab #docker #haproxy
I observed an interesting issue in my Jenkins pipeline. The image pull aborted with the following error message:
Error: writing blob: storing blob to file "/var/tmp/storage1360560957/1": happened during read: unexpected EOF
First I thought it has something to do with the storage. But I was wrong. The culprit was the network.
More specifically, I noticed that pulling through my HAProxy instance was the issue, but pulling through the nodes registry port (directly) was fine.
When looking into the HAProxy logs, I noticed that the requests fail with the particular error flags cD:
Sep 02 02:23:29 haproxy haproxy[2836]: 10.0.0.102:34982 [02/Sep/2025:02:22:47.563] registryfront registry/pi3 0/0/0/87/42156 200 466612119 - - cD-- 6/1/0/0/0 0/0 {haproxy.lan:5000} "GET /v2/texlive/blobs/sha256:2fde6c0b50af2b1fda7ed0092ad1f1cc6897d7cb723dfcb0d2bc15201bbd7191 HTTP/1.1"
The HAProxy docs on stream states:
cD The client did not send nor acknowledge any data for as long as the
"timeout client" delay. This is often caused by network failures on
the client side, or the client simply leaving the net uncleanly.
First flag c:
- On the first character, a code reporting the first event which caused the
stream to terminate :
c : the client-side timeout expired while waiting for the client to
send or receive data.
Second flag D:
- on the second character, the TCP or HTTP stream state when it was closed :
D : the stream was in the DATA phase.
That was useful – “the client-side timeout expired”. It simply means that I need to bump the client timeouts (to 30m from 5s in this example) in the HAProxy frontend for my Docker registry.
frontend registryfront
bind :5000
timeout client 30m # was 5s
timeout client-fin 30m # was 30s
mode http
option httplog
# display host header in logs
capture request header Host len 30
default_backend registry
The pull request through the proxy afterwards show no error flags (----):
Sep 02 02:30:27 haproxy haproxy[2850]: 10.0.0.102:53506 [02/Sep/2025:02:27:03.674] registryfront registry/pi3 0/0/0/15/203648 200 2334919898 - - ---- 7/1/0/0/0 0/0 {haproxy.lan:5000} "GET /v2/texlive/blobs/sha256:2fde6c0b50af2b1fda7ed0092ad1f1cc6897d7cb723dfcb0d2bc15201bbd7191 HTTP/1.1"
Podman pull succeeds 🎉
Ping me in chat or Fediverse if you have more suggestions regarding HAProxy configuration for private Docker registries. Happy self-hosting!