ManageAsync is seemingly blocking

[zllovesuki]

What is your question?

I discovered that ManageAsync is seemingly blocking under a specific circumstance.

Context:
Restarting an application using certmagic, but the app was stuck on the ManageAsync function. It was later discovered that http request to e5.o.lencr.org was blocked (without the firewall sending RST), hence causing the application to appear to be hung on startup.

Question:
Is there a way to make ManageAsync does what it said in the name, "async"? 🙂

[mholt]

🤔 Do you have more information? That's a Let's Encrypt OCSP URL.

A goroutine profile would be helpful, or a way to reproduce the behavior. But right now I'm not sure the information adds up. I would need a lot more details.

[zllovesuki]

🤔 Do you have more information? That's a Let's Encrypt OCSP URL.

A goroutine profile would be helpful, or a way to reproduce the behavior. But right now I'm not sure the information adds up. I would need a lot more details.

I don't know how I would describe the setup 🙃

The application was upgraded by stopping, installed the new version, then restarted again. Calling ManageAsync was part of the startup process and it appeared to be stuck. The server was running behind suricata where apparently the rule it triggered was something along the line of "Go user-agent outbound request", causing suricata to blocked the TCP connection, and subsequent requests were also unsuccessful.

If this it the OCSP URL then it would make sense since it is done over plain HTTP where the inspection wouldn't work if it were done over TLS.

Maybe it has something to do with "checking OSCP"? To reproduce and get a profile I would have to think about how to MITM it in an environment where TCP SYN was sent but there were never any TCP SYNACK nor RST.

[zllovesuki]

Hello, I was able to reproduce it locally with some dns hijacking to get it stuck.

I've attached the goroutine pprof below.

To reproduce:

1. Clone the https://github.com/zllovesuki/certmagic-335-repro repo to start a dns interceptor locally. Note that this will need to be run as root as it requires listening on the port 80.
2. Override the default DNS resolver so it points to the dns interceptor:

	resolver := &net.Resolver{
		PreferGo: true,
		Dial: func(ctx context.Context, network, address string) (net.Conn, error) {
			dialer := net.Dialer{}
			return dialer.DialContext(ctx, network, "127.0.0.1:8053")
		},
	}
	net.DefaultResolver = resolver

3. Start an application configured with certmagic + the resolve overridden.

You should see the logs coming from the dns interceptor:

2025/03/14 22:45:00 Starting DNS proxy [TCP] on 127.0.0.1:8053 -> 1.1.1.1:53
2025/03/14 22:45:00 Starting DNS proxy [UDP] on 127.0.0.1:8053 -> 1.1.1.1:53
2025/03/14 22:45:03 [DNS Proxy] Incoming Query => QNAME=e5.o.lencr.org. QTYPE=AAAA QCLASS=IN
2025/03/14 22:45:03 [DNS Proxy] Incoming Query => QNAME=e5.o.lencr.org. QTYPE=A QCLASS=IN
2025/03/14 22:45:03 [DNS Proxy] Overriding e5.o.lencr.org. (QTYPE=A) with 127.0.0.1
2025/03/14 22:45:03 [DNS Proxy] final RR (overridden=true): e5.o.lencr.org.     3600    IN      A       127.0.0.1
2025/03/14 22:45:03 [DNS Proxy] final RR (overridden=false): e5.o.lencr.org.    189     IN      CNAME   o.lencr.edgesuite.net.
2025/03/14 22:45:03 [DNS Proxy] final RR (overridden=false): o.lencr.edgesuite.net.     21489   IN      CNAME   a1887.dscq.akamai.net.
2025/03/14 22:45:03 [DNS Proxy] final RR (overridden=false): a1887.dscq.akamai.net.     19      IN      AAAA    2600:1406:3a00:6::173e:2ed7
2025/03/14 22:45:03 [DNS Proxy] final RR (overridden=false): a1887.dscq.akamai.net.     19      IN      AAAA    2600:1406:3a00:6::173e:2ecb
2025/03/14 22:47:46 [DNS Proxy] Incoming Query => QNAME=e5.o.lencr.org. QTYPE=A QCLASS=IN
2025/03/14 22:47:46 [DNS Proxy] Overriding e5.o.lencr.org. (QTYPE=A) with 127.0.0.1
2025/03/14 22:47:46 [DNS Proxy] final RR (overridden=true): e5.o.lencr.org.     3600    IN      A       127.0.0.1
2025/03/14 22:47:46 [DNS Proxy] Incoming Query => QNAME=e5.o.lencr.org. QTYPE=AAAA QCLASS=IN
2025/03/14 22:47:46 [DNS Proxy] final RR (overridden=false): e5.o.lencr.org.    127     IN      CNAME   o.lencr.edgesuite.net.
2025/03/14 22:47:46 [DNS Proxy] final RR (overridden=false): o.lencr.edgesuite.net.     21427   IN      CNAME   a1887.dscq.akamai.net.
2025/03/14 22:47:46 [DNS Proxy] final RR (overridden=false): a1887.dscq.akamai.net.     20      IN      AAAA    2600:1406:3a00:6::173e:2ecb
2025/03/14 22:47:46 [DNS Proxy] final RR (overridden=false): a1887.dscq.akamai.net.     20      IN      AAAA    2600:1406:3a00:6::173e:2ed7

The dns interceptor also listens on port 80 locally, and keeps incoming TCP connection opened. This matches the behavior by suricata where the TCP was left in the half-open state and causing the application to appear to be stuck.

goroutine.pprof.gz

[mholt]

Thanks, will look at this soon -- a few other things came up but this is next on my list to investigate.