TCP Connection Ports Leaks

[niamtokik]

Describe the bug

When shutting down an Erlang node, if an active TCP connection is still established with a remote peer, the connection is not closed, even if the process linked/connected to the port was killed. It can lead to a very slow shutdown if the connection used by the remote peer has a huge latency or a small bandwidth.

To Reproduce

We discovered this bug using cowboy but it can also be reproduced with inets:httpd:

1. start a service using TCP (e.g. cowboy or inets:httpd)
2. configure the service to share large files (greater than 100MB but can be smaller if one has a way to control the client/server throughput)
3. fetch a file with a web client like curl using --limit-rate ${value} where ${value} is close to 0.
4. stop the service (or the BEAM)
5. if the service is stopped, an unlinked port connected to an unknown process should be present. It can be listed using [ erlang:port_info(X) || X <- erlang:ports() ]
6. if the BEAM has been stopped, it will wait until the connection is closed. The connection can be checked using sudo ss -ntip on GNU/Linux or doas fstat -u ${user} on OpenBSD.
7. the only workaround found right now is to kill the connections using ss -K on GNU/Linux

A full example as escript is present at the end of this bug report.

Expected behavior

When stopping cowboy, inets:httpd or any service using TCP, the active connection should be properly closed and/or killed.

Affected versions

This bug has been reproduced on:

• Ubuntu 22.04 with OTP-24.2.1 (from packages) and cowboy/inets
• Ubuntu 22.04 with OTP-26.2.5.9 (from sources) and cowboy/inets
• OpenBSD 7.6 with OTP-25.3.2.13 (from packages) and cowboy/inets
• OpenBSD 7.6 with OTP-26.2.5.3 (from packages) and inets

Additional context

It looks like the bug is from erts/emulator/drivers/common/inet_drv.c and was introduced in ebbd26e. Not sure though, but during the investigation, it seems this part of the code does not receive the instruction to close the connection.

The bug can easily be reprouced with the following code:

#!/usr/bin/env escript

listen_port() -> 9999.

main(_Args) ->
  % prepare environment
  io:format("prepare environment~n"),
  
  io:format("create /tmp/inets directory~n"),
  file:make_dir("/tmp/inets"),
  
  io:format("create /tmp/inets/configuratio directory~n"),
  file:make_dir("/tmp/inets/configuration"),
  
  io:format("create /tmp/inets/files directory~n"),
  file:make_dir("/tmp/inets/files"),
  
  io:format("create /tmp/inets/files/100m file~n"),
  file:write_file("/tmp/inets/files/100m", crypto:strong_rand_bytes(100*1024*1024)),

  % start inets and configure it
  io:format("start inets~n"),
  application:ensure_all_started(inets),
  {ok, _P} = inets:start(httpd, [
    {port, listen_port()},
    {server_root, "/tmp/inets/configuration"},
    {document_root, "/tmp/inets/files"},
    {bind_address, "localhost"},
    {server_name, "localhost"}
  ]),
  
  % send a first message every 1 second to display ports
  timer:send_interval(1000, self(), tick),
  
  % execute some curls from the BEAM
  timer:send_after(2000, self(), curl),
  timer:send_after(2000, self(), curl),
  timer:send_after(2000, self(), curl),
  timer:send_after(2000, self(), curl),

  % after 5s, send stop message to stop inets
  timer:send_after(3000, self(), stop),
  loop().

loop() ->
  receive
    curl ->
      spawn_monitor(fun() ->
        Command = "curl --limit-rate 1 http://localhost:" ++ integer_to_list(listen_port()) ++ "/100m",
        io:format("start curl connection: ~p~n", [Command]),
        os:cmd(Command)
      end),
      loop();
    stop ->
      io:format("stop inets~n"),
      inets:stop(),
      application:stop(inets),
      loop();
    tick ->
      io:format("port leaks: ~w~n", [port_leak()]),
      loop();
    Msg ->
      io:format("received: ~w~n", [Msg]),
      loop()
  end.

% filter only unlinked ports
port_leak() ->
  Ports = erlang:ports(),
  PortsInfo = [ erlang:port_info(X) || X <- Ports ],
  Leaks = [ 
    X || X <- PortsInfo,
    proplists:get_value(name, X) =:= "tcp_inet",
    proplists:get_value(links, X) =:= []
  ],
  case Leaks of
    [] -> [];
    PS when is_list(PS) ->
      [
        begin
          Process = proplists:get_value(connected, P),
          ProcessInfo = erlang:process_info(Process),
          {P, {Process, ProcessInfo}}
        end
      || P <- PS
      ]
  end.

Here some example of leaked ports. The port is not linked, and the process is not existing (dead):

[{[{name,"tcp_inet"},                                                          
   {links,[]},                                                                 
   {id,80},                                                                    
   {connected,<0.101.0>},                                                      
   {input,0},                                                                  
   {output,131275},                                                            
   {os_pid,undefined}],                                                        
  {<0.101.0>,undefined}},                                                                                                                                                                                                                                                                                                    {[{name,"tcp_inet"},                                                                                                                                                                                                                                                                                                       
   {links,[]},                                                                                                                                                                                                                                                                                                                 {id,88},                                                                                                                                                                                                                                                                                                                 
   {connected,<0.102.0>},                                                                                                                                                                                                                                                                                                      {input,0},                                                                                                                                                                                                                                                                                                               
   {output,131275},                                                                                                                                                                                                                                                                                                            {os_pid,undefined}],                                                                                                                                                                                                                                                                                                     
  {<0.102.0>,undefined}},                                                                                                                                                                                                                                                                                                    {[{name,"tcp_inet"},                                                                                                                                                                                                                                                                                                       
   {links,[]},                                                                                                                                                                                                                                                                                                                 {id,96},                                                                                                                                                                                                                                                                                                                 
   {connected,<0.103.0>},                                                                                                                                                                                                                                                                                                      {input,0},                                                                                                                                                                                                                                                                                                               
   {output,131275},                                                                                                                                                                                                                                                                                                            {os_pid,undefined}],                                                                                                                                                                                                                                                                                                     
  {<0.103.0>,undefined}},                                                                                                                                                                                                                                                                                                    {[{name,"tcp_inet"},                                                                                                                                                                                                                                                                                                       
   {links,[]},                                                                                                                                                                                                                                                                                                                 {id,112},                                                                                                                                                                                                                                                                                                                
   {connected,<0.104.0>},                                                                                                                                                                                                                                                                                                   
   {input,0},                                                                  
   {output,131275},                                                            
   {os_pid,undefined}],                                                        
  {<0.104.0>,undefined}}]

The connections can be seen using these commands:

# on GNU/Linux
ss -nitp | grep beam

# on OpenBSD
fstat | grep beam | grep tcp

Those ports cannot be closed using erlang:port_close/1. Here another debugging session from an erlang shell where inets has been stopped:

% extract the ports
[{F,_}] = [ Z || Z = {X, Y} <- [ {X, erlang:port_info(X)} || X <- erlang:ports()],  proplists:get_value(name, Y) =:= "tcp_inet" ].

% impossible to close the port using port_close:
erlang:port_close(F).
% returns an exception:
% ** exception error: bad argument
%      in function  port_close/1
%         called as port_close(#Port<0.5>)

% impossible to connect to the port as well
erlang:port_connect(F, self()).
% returns an exception:
% ** exception error: bad argument
%      in function  port_connect/2
%         called as port_connect(#Port<0.5>,<0.130.0>)

EDIT1: added ports/process info and corrected few typos

EDIT2: added more information regarding leaked ports.

[garazdawi]

Hello!

There is an option called linger that can be set on a connection. How is that configured in this case?

When shutting down the Erlang VM you can either make it flush or not flush all open connections, how does erlang:halt(..., [{flush,boolean()}]) effect your scenarios?

[niamtokik]

Hello @garazdawi,

There is an option called linger that can be set on a connection. How is that configured in this case?

I will check that on our application and come back to you.

When shutting down the Erlang VM you can either make it flush or not flush all open connections, how does erlang:halt(..., [{flush,boolean()}]) effect your scenarios?

erlang:halt(0, [{flush, true}]). hangs, but erlang:halt(0, [{flush, false}]). stops the VM. It seems the connections are closed as well.

[niamtokik]

There is an option called linger that can be set on a connection. How is that configured in this case?

We are using cowboy 2.10.0 with ranch 1.8.0. It seems ranch_tcp set linger value to {false, 0}. I enforced this configuration in our application to be sure but it did not change the behavior, when shutting down the VM or stopping the application (including cowboy and ranch). It seems most of the connections are now correctly closed but there are still some taking a while to be closed. I can't find which value is used by default for this parameter by the BEAM, is it set to {true, 0}? If it's the case, I think cowboy and ranch documentation need to be fixed.

[garazdawi]

You can get the value by doing inet:getopts(Connection, [linger]). On my machine the default is {false,0}.

[niamtokik]

Indeed. All connections in our applications are using {false, 0} by default. from the documentation:

{false, _} - close/1 or shutdown/2 returns immediately, not waiting for data to be flushed, with closing happening in the background.

It could then explain why the VM is not stopped, because all the connections are still active, totally outside of the scope of the VM (shutdown procedure), but available in background until the connections are explicitly closed by the node (killing the tcp connections with ss for example) or when the client terminates it on its side. Then what about {true, 0}?

{true, 0} - Aborts the connection when it is closed. Discards any data still remaining in the send buffers and sends RST to the peer. This avoids TCP's TIME_WAIT state, but leaves open the possibility that another "incarnation" of this connection being created.

The another "incarnation" of the connection seems a problem, what does it really mean? When we are closing the socket, it can still be re-activated by the client? Furthermore, does this "incarnation" behavior can also be seen when using a timeout?

[garazdawi]

The another "incarnation" of the connection seems a problem, what does it really mean? When we are closing the socket, it can still be re-activated by the client? Furthermore, does this "incarnation" behavior can also be seen when using a timeout?

When you don't use {true,0}, the tcp socket will enter the TIME_WAIT state when it closed. This holds the connection semi-alive for a while and prevents certain types of miss-behaviours. It is the default for a reason. If you do set it to {true,0}, then it will skip the TIME_WAIT state and send RST to the remote end (RST is an error indication). If you search/llm for SO_LINGER 0 and TIME_WAIT you will get better explanations that what I can give.