Cybersecurity / Networking

How to Debug "socket hang up" Errors in HTTP Clients

4 min read by DebuggedIt

Quick answer

The dreaded socket hang up error (often accompanied by code ECONNRESET) occurs when an HTTP client attempts to write data to or read data from a TCP socket...

The dreaded socket hang up error (often accompanied by code ECONNRESET) occurs when an HTTP client attempts to write data to or read data from a TCP socket that was abruptly closed by the remote server or an intermediate network middlebox. This issue is particularly prevalent in high-throughput microservice architectures that rely on persistent HTTP Keep-Alive connections. Diagnosing timeout mismatches, socket pool limits, and server process crashes allows developers to resolve these transient connection failures completely.

The Problem

When sending requests or configuring components in your development environment, the application or HTTP client fails with the following exact error trace:

# Example 1: Node.js / Axios HTTP Client Error Trace
Error: socket hang up
    at connResetException (node:internal/errors:705:14)
    at Socket.socketOnEnd (node:_http_client:518:23)
    at Socket.emit (node:events:525:35) {
  code: 'ECONNRESET'
}

# Example 2: Go http.Client Request Failure
Get "https://api.example.com/v1/stream": read tcp 192.168.1.50:52341->10.0.0.1:443: read: connection reset by peer

# Example 3: Python Requests ConnectionResetError
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

This error halts script execution, prevents API communication, or results in immediate client-side connection drops.

HTTP Client Reuse Pooled Socket 1. Send HTTP Request on Idle Socket 2. TCP FIN / RST Packet (Socket Closed!) Upstream Server Keep-Alive Timeout Reached (Closed Socket Idle for 5s)

Why It Happens

This failure occurs due to misconfigurations at the networking, cryptographic, or application protocol layer. The primary causes include:

  • Keep-Alive Timeout Race Condition: The upstream server's keep-alive timeout (e.g., 5 seconds in Nginx or Node.js) is shorter than the client's idle connection timeout. The client sends a request over a pooled socket right as the server closes it.
  • Backend Service Crash / OOM: The remote application process crashed unexpectedly (due to an Out-Of-Memory error or uncaught exception) while processing the request, killing the TCP connection without sending an HTTP response.
  • Proxy or Firewall Connection Termination: Intermediate load balancers (AWS ALB, HAProxy) terminate idle TCP connections after reaching maximum idle thresholds without sending a TCP FIN packet to the client.
  • Payload Size Exceeding Buffer Limits: Sending massive POST request bodies that exceed backend socket payload limits causes the server to forcefully abort the socket stream.

The Fix

Follow these step-by-step solutions to resolve the error in your environment.

Step 1: Align HTTP Keep-Alive Timeouts Between Client and Server

Ensure that your client-side keep-alive timeout is shorter than your server-side or load balancer keep-alive timeout. In Node.js / Axios microservices, configure explicit Agent keep-alive settings:

const axios = require('axios');
const http = require('http');
const https = require('https');

// Create custom HTTP/HTTPS agents with short keepAlive timeouts
const httpAgent = new http.Agent({ keepAlive: true, timeout: 60000, freeSocketTimeout: 4000 });
const httpsAgent = new https.Agent({ keepAlive: true, timeout: 60000, freeSocketTimeout: 4000 });

const apiClient = axios.create({
  baseURL: 'https://api.example.com',
  httpAgent,
  httpsAgent,
  timeout: 10000 // Request timeout
});

On your Node.js HTTP backend server, set keepAliveTimeout higher than downstream clients (e.g., 65 seconds to exceed AWS ALB's 60-second default):

const server = app.listen(3000, () => console.log('Server running on port 3000'));

// Set keepAliveTimeout higher than reverse proxy timeouts
server.keepAliveTimeout = 65000;
// Ensure headersTimeout is greater than keepAliveTimeout
server.headersTimeout = 66000;

Step 2: Implement Automatic Retry Strategy for Transient Socket Errors

Since socket hang up errors frequently occur due to race conditions on pooled connections, implement an exponential backoff retry mechanism specifically targeting ECONNRESET:

const axiosRetry = require('axios-retry');

axiosRetry(apiClient, {
  retries: 3,
  retryDelay: axiosRetry.exponentialDelay,
  retryCondition: (error) => {
    // Retry on socket hang up / ECONNRESET errors
    return axiosRetry.isNetworkOrIdempotentRequestError(error) || error.code === 'ECONNRESET';
  }
});

Step 3: Monitor Upstream Application Health and Memory

Check system logs on the target server to confirm whether process crashes coincide with socket hang up events:

# Check Linux kernel logs for Out Of Memory (OOM) kills
sudo dmesg -T | grep -i oom

# Inspect systemd application logs
journalctl -u my-api-service.service --since "10 minutes ago" -e

Still Not Working?

TCP Keep-Alive Probes and OS Network Tuning

If socket hang up errors persist across long-polling endpoints or WebSockets, stateful firewalls may be silently dropping idle TCP connections. Enable OS-level TCP keep-alive probes on your server host to maintain socket activity across idle periods:

# Check current TCP keep-alive settings in Linux
sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl

# Lower keep-alive time to 300 seconds in /etc/sysctl.conf
sudo sysctl -w net.ipv4.tcp_keepalive_time=300

You might also like