Let’s make a function to generate an HTTP client for us using a custom dialer:

 1var DefaultDialer = &net.Dialer{}
 2
 3func GetHttpClient() http.Client {
 4	tr := &http.Transport{
 5		Dial:                DefaultDialer.Dial,
 6	}
 7
 8	client := http.Client{Transport: tr}
 9	return client
10}

Can you spot the bug?

By omitting the Timeout, KeepAlive timeouts in the first example, we’ve introduced a very subtle bug.

There is also another bug if you don’t handle TLS timeouts as well.

net/Dialer has some documentation on this.

Without providing a KeepAlive and a Timeout value, you could end up with connections that hang indefinitely. By omitting the TLS handshake timeout, the daemon would also hang trying to re-negotiate the SSL connection.

In my case this was causing a very random and hard to reproduce issue where the program would hang indefinitely.

Some good debugging tips are using strace to see what syscall its stuck in, and if your daemon is running in the foreground, using a SIGQUIT signal.

Here is a working version:

 1var DefaultDialer = &net.Dialer{Timeout: 2 * time.Second, KeepAlive: 2 * time.Second}
 2
 3func GetHttpClient() http.Client {
 4	tr := &http.Transport{
 5		Dial:                DefaultDialer.Dial,
 6		TLSHandshakeTimeout: 2 * time.Second,
 7}
 8
 9	client := http.Client{Transport: tr}
10	return client
11}