A few days ago, I ran into the following interesting Nginx error message.
[crit] 12889#0: *32401195 connect() to 127.0.0.1:80 failed (99: Cannot assign requested address) while connecting to upstream
My configuration was very simple. This was an Nginx proxy that did all the SSL encryption and sent traffic to a Varnish instance, running on port :80 on localhost. The big takeway here is that it was a pretty high traffic Nginx proxy.
Even with keepalive enabled in the nginx upstream, the error popped up. But what did it mean?
TCP ports and limits
It’s good to know a thing or two about networking besides just servers once in a while. The problem occurred because the server couldn’t get a free TCP port quickly enough to make the connection to 127.0.0.1.
$ ss -s Total: 3130 (kernel 3431) TCP: 51582 (estab 2866, closed 48611, orphaned 92, synrecv 0, timewait 48611/0), ports 35279
The ss tool gives you stats on the sockets/ports on the server. In this case, I had 51.582 TCP sessions in use (either active, closed, awaiting to be closed, …).
A normal server has around 28.000 possible TCP ports it can use to make a TCP connection to a remote (or local) system. Everything that talks via an IP address will pick a free port from this range to serve as source port for the outgoing connection. This port range is defined by the ip_local_port_range sysctl parameter.
$ cat /proc/sys/net/ipv4/ip_local_port_range 32768 61000 $ sysctl net.ipv4.ip_local_port_range net.ipv4.ip_local_port_range = 32768 61000
The format is “minimum maximum” port. So 61000 – 32768 = 28 232 available source ports.
An nginx SSL proxy that connects to a Varnish instance running on localhost will look like this in your netstat.
$ netstat -anp | grep 127.0.0.1 ... tcp 0 0 127.0.0.1:37713 127.0.0.1:80 TIME_WAIT -
The key takeaways here the source connection 127.0.0.1:37713 that connects to its endpoint 127.0.0.1:80. For every source connection a new TCP source port is selected from the range in the ip_local_port_range parameter.
The combination of a source IP, source port, destination IP and destination IP needs to be unique. This is what’s called a quadruplet in networking terms. You likely can’t (easily) change the source IP. The source port is dynamically picked. That only leaves the destination IP and the destination port that are free to play with.
Solving the source port limitation
There are a couple of easy fixes. First, the ip_local_port_range can be increased on a Linux machine (for more reading material, see increase ip_local_port_range TCP port range in Linux).
$ echo 15000 64000 > /proc/sys/net/ipv4/ip_local_port_range
This effectively increases the total port range from its default 28 232 ports to 49 000 ports.
If that’s not enough, you can add more destination IPs to connect to. Remember that each connection consists of the 4 parts (called quadruplets) with source IP and source port, destination IP and destination port. If you can’t change the source port or IP, just change the destination IPs.
Consider this kind of upstream definition in Nginx;
upstream varnish {
server 127.0.0.1:80;
}
Such a definition can be used in your nginx configurations with the proxy_pass directive.
server {
listen 443;
...
location / {
proxy_pass http://varnish;
...
}
}
Now if you know that each server usually has 2 IPs or more, it’s very easy to add more quadruplets to your networking stack by adding an addition IP to your Nginx upstream. You’ve already added 127.0.0.1, but your server will have another IP (its public or DHCP IP) that you can safely add too, if your webserver binds to all ports.
upstream varnish {
server 127.0.0.1:80;
server 31.193.180.217:80;
server 10.50.5.1:80;
...
}
Every IP you add in your upstream is effectively adding 49.000 local ports to your networking stack. You can even add non-routable local IPs to your server, as interface aliases, just to use as new destination IPs for your proxy configurations.