clientError emitted for wrong socket - SSL alert number 48 #1642

calzoneman · 2015-05-06T18:31:34Z

NOTE: The following bug report was originally submitted to the node team at nodejs/node-v0.x-archive#14818; however, I encountered the same issue in io.js when trying it out. I have been able to reproduce the issue on io.js compiled from source from the master branch (v2.0.1), both on my Arch desktop and Debian server. It's worth noting that the issue does not occur on the v0.10 branch of node.js, only the v0.12 branch.

I've noticed a peculiar behavior of the https module with regard to Firefox closing connections for untrusted certificates. In the demo below, I use a self-signed certificate generated by the pem module, but the same principle can be applied to any untrusted certificate (for example, I was able to cause the same issue by deleting the Startcom signing certificates from Firefox and attempting to connect to a server with a StartSSL cert).

What happens is that as soon as a Firefox client that doesn't trust the certificate terminates its connection, the https server fires a clientError event with an SSL error-- except this event is fired on a random socket that has no relation to the Firefox client. I determined this by gathering several different Chrome users, having them all connect to a server I controlled, and observing the errors logged when I connect with Firefox.

I would expect instead that the socket parameter passed to the clientError event handler would be the client that triggered the error, not some arbitrary other client.

Steps to reproduce:

Download server.js and index.html as listed below
Install the pem module
Start the server, and navigate Chromium to https://localhost:4444
- Click Advanced and Proceed to localhost (unsafe)
Navigate Firefox to https://localhost:4444

Observe that as soon as the Untrusted Connection page displays in Firefox, a clientError is triggered on the server, but on the Chromium client:

Error: 140173278586688:error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:s3_pkt.c:1461:SSL alert number 48

at Error (native)
Triggered on client with UA: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36

Demo:

server.js

var https = require('https');
var pem = require('pem');
var fs = require('fs');

var indexHtml = fs.readFileSync('index.html');

pem.createCertificate({ days: 10, selfSigned: true }, function (err, keys) {
    if (err) throw err;

    var httpServer = https.createServer({
        key: keys.serviceKey,
        cert: keys.certificate
    }, function (req, res) {
        req.socket.userAgent = req.headers['user-agent'];
        res.writeHead('200', {
            'Content-Type': 'text/html',
            'Content-Length': indexHtml.length
        });

        res.end(indexHtml);
    });

    httpServer.on('clientError', function (err, socket) {
        console.log(err.stack);
        console.log('Triggered on client with UA: ' + socket.userAgent);
    });

    httpServer.listen(4444);
});

index.html

<!doctype html>
<html lang="en">
  <head>
    <title>SSL Error Demo</title>
    <meta charset="utf-8">
  </head>
  <body>
    <div id="debug"></div>
    <script type="text/javascript">
      function makeRequest() {
        var req = new XMLHttpRequest();
        req.open('GET', location.href, true);

        req.onload = function (ev) {
          setTimeout(makeRequest, 10);
        };

        req.send();
      }

      makeRequest();
    </script>
  </body>
</html>

The text was updated successfully, but these errors were encountered:

shigeki · 2015-05-07T09:32:10Z

I have confirmed and reproduced this. But it is nothing do with iojs. For example, I can also reproduce this with an openssl s_server with a small change of

diff --git a/index.html b/index.html
index f2f081a..db427a2 100644
--- a/index.html
+++ b/index.html
@@ -9,7 +9,7 @@
     <script type="text/javascript">
       function makeRequest() {
         var req = new XMLHttpRequest();
-        req.open('GET', location.href, true);
+        req.open('GET', location.href + '/index.html', true);

         req.onload = function (ev) {
           setTimeout(makeRequest, 10)

$ openssl s_server -key agent.key -cert agent.crt -accept 4444 -WWW  -msg >& server.log 2>&1

Following the above steps with changing url to index.html, the sameTLS Alert is found in the server.log.

>>> TLS 1.2 ChangeCipherSpec [length 0001]
    01
>>> TLS 1.2 Handshake [length 0010], Finished
    14 00 00 0c 39 56 9a bb 99 41 e6 10 ea 5c 7b 30
<<< TLS 1.2 Alert [length 0002], fatal unknown_ca
    02 30

From my analysis of the packet dump, the ajax in Chrome seems to steal the first TLS connection of Firefox and send the request to server on the Firefox connection but it is not accepted with permission yet then the client sends TLS alert of unknown_ca and the sever close the connection.

It is very strange behavior and I'm not sure its reason. I think this issue comes from the client behavior. I'm closing this issue for now. If you find any clue that this is caused by iojs, please be free to put the comment here. I will reopen this.

calzoneman · 2015-05-07T15:17:18Z

Thank you for your prompt response. I have a few questions:

I compiled node.js v0.10, v0.12, and io.js with the --shared-openssl flag for ./configure, and it is my understanding that this means all of them will link with my system's version of OpenSSL (1.0.2a) as a shared library rather than compiling the static version included in the tarball (if this is incorrect, please correct me). Since all three versions are using the same version of OpenSSL, do you have any ideas why this might be happening in node v0.12 and io.js but not in node v0.10?
What is your suggested workaround? The demo I created for this ticket was only a test case and does not represent where the issue was originally encountered. I am currently stuck using node v0.10 for my socket.io application because this issue allows trolls with Firefox to force-disconnect random Chrome clients.
If this is an issue with OpenSSL, do you have any suggestions for modifications to this bug report for submission to them?

Thanks.

gopat · 2015-05-07T16:35:39Z

I don't think we can blame the client so easily here. Something seemed really weird to me, so i did a couple of tests....
I set up three VMs:

ubuntuIOJS: Ubuntu server 14.04.2 iojs v2.0.0, 10.10.5.75
W7FF: Windows 7 64bits with Firefox 37.0.2, 10.10.5.45
W7Chrome: Windows 7 64bits with Chrome 42, 10.10.5.44
...and did as @calzoneman posted, but with a modified server script:

var https = require('https');
var pem = require('pem');
var fs = require('fs');

var indexHtml = fs.readFileSync('index.html');

pem.createCertificate({ commonName:'10.10.5.75:4444', days: 10, selfSigned: true }, function (err, keys) {
    if (err) throw err;

    var httpServer = https.createServer({
        key: keys.serviceKey,
        cert: keys.certificate
    }, function (req, res) {
        req.socket.userAgent = req.headers['user-agent'];
        req.socket.c_remoteAddress = req.socket.remoteAddress;
        req.socket.c_remotePort    = req.socket.remotePort;
        res.writeHead('200', {
            'Content-Type': 'text/html',
            'Content-Length': indexHtml.length
        });

        res.end(indexHtml);
    });

    httpServer.on('clientError', function (err, socket) {
        console.log(err.stack);
        console.log('Triggered on client with UA: ' + socket.userAgent);
        console.log('From: ', socket.c_remoteAddress, socket.c_remotePort);
        //console.log('Socket: ', socket);
    });

    httpServer.listen(4444,'10.10.5.75');
});

With https://10.10.5.75:4444/ loaded in Chrome on W7Chrome vm, when i typed https://10.10.5.75:4444/ in Firefox location bar on W7FF vm, the server script spitted the following:

$ ../iojs-v2.0.0-linux-ia32/bin/iojs servertest.js
Error: 3074565888:error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:../deps/openssl/openssl/ssl/s3_pkt.c:1461:SSL alert number 48

    at Error (native)
Triggered on client with UA: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36
From:  10.10.5.44 52648

Well i have 0 knowledge of tls protocol or its flows; but when Firefox makes that initial https request as i open the url, an error appears matched/paired with a socket that has different client port and client ip (for a different pc/vm).
I just can't imagine how can that be purely a client issue (literally, no sarcasm intended).

It really is weird that it could happen in the same machine, but I seems really unlikely that one process in one machine is stealing a socket (l_ip,l_port,r_ip,r_port tuple) from another process... in another machine! TCP Hijacking is possible, but difficult to perform and low success rate, yet it would be happening spontaneously and consistently... no way.

Also tested with bridged adapter network and some non-virtual client machines, same results.

I have also tried:

openssl s_server -key server.key -cert server.cer -accept 4444 -WWW -msg 2>&1

...opening the url with a Firefox client does consistently give something like:

[....]
>>> TLS 1.2 Handshake [length 0010], Finished
    14 00 00 0c 79 4b 82 00 75 9b c2 b7 da fd ec f9
<<< TLS 1.2 Alert [length 0002], fatal unknown_ca
    02 30

...that is, it seems to always send fatal unknown_ca (48, 0x30) right after the tls handshake is finished when connecting to server with cert. which has no trusted ca (and no exception has been added previously in the browser).

I don't really know, but i have a feeling that it could be an OpenSSL bug (iojs is not without lottery tickets, but i think the last versions of openssl have bought plenty more numbers).
Maybe, when a connection is closed the way Firefox does, it triggers some race condition that makes OpenSSL "assign" the "error" (48, unknown ca) to some recent innocent tls session (maybe there's something different in sessions/connections from chrome that is also needed to trigger this, since couldn't reproduce with other browser combinations).

My bet, in short: some openssl internal state related to chrome connections makes it handle the alert msg incorrectly and bind it with one of those chrome connections???

Added after @calzoneman 2nd post:
Oops, openssl suddenly gives half his lottery tickets to iojs....
Could it be related to new functionality in openssl 1.0.2 that only node 0.12 and iojs are using??

PD: Sorry for the long post, and as always, thanx.
PD: Also, i try hard to write correctly, but beware that English is not my first language.

EDIT: Let's just hope it isn't another critical vulnerability in openssl... (like a dos-enabling vuln.).

gopat · 2015-05-07T18:01:05Z

~~@calzoneman:
To my understanding, unless client was providing a client certificate, alert number 48 unknown_ca is describing a client "message" signaling the server that it won't trust the server cert.
~~The client should always close the connection since that very message is considered "fatal", see https://tools.ietf.org/html/rfc5246#page-32~~

~~If that is the case (no client certificates) a potential, possibly wrong and dirty workaround would be to check for 48, and ignore it; something like:~~

httpServer.on('clientError', function (err, socket) { //EDIT: USELESS
  var ignoreThisError=false;
  if(/:SSL alert number 48\D/.test(err.message)) ignoreThisError=true;
  //....
}

~~One could, of course, write a client that sends unknown_ca, doesn't close the connection and keeps with the request, or something... but i don't think there are any security concerns for the server.~~

~~Note that before doing anything remotely similar, i strongly recommend to wait until someone more competent than me in this matter comments on this option.~~
~~Yes, this is a "Don't blame me, i warned you."~~

~~I haven't tested it, dunno, maybe openssl itself won't allow the session to continue... then there would be no user-land workaround and it would be a dos-vector...~~

EDIT: It's impossible to skip closing the socket in user-land: it's not being closed in user-land, but by iojs. No user-land workaround.

shigeki · 2015-05-08T06:18:32Z

Reopen this because I've just found that this is the issue of iojs.
@calzoneman The TLS module was totally redesigned and changed in node-v.0.12. The difference of tls behaviors would occur.
@gopat Thanks for your tests and information. Your suggestion is very helpful to me for the further investigation.

I should have taken care of the UA property defined in the request event at the above test. Because the fatal TLS alert was sent from client to server immediately after SSL handshake completed, there was no way to carry encrypted payload in the connection so that the request event never happen on https.Server. The UA propety of socket had no chance to be defined and should be undefined in this case as

Error: 139976401414016:error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:../deps/openssl/openssl/ssl/s3_pkt.c
:1461:SSL alert number 48

    at Error (native)
Triggered on client with UA: undefined

iojs might incorrectly handle the ssl error on the wrong socket and it destroy the existing connection from Chrome.

@calzoneman @gopat Could you please test my branch of
https://github.com/shigeki/io.js/tree/fix_tls_error
whether your issue is resolved or not even in your websocket case?

@indutny Do you have any comments on my fixes in the above branch? The fix of _secureEstablished is nothing to do with this issue but I found it during my investigation. I tried to create a test but it is very hard to send a fatal TLS alert from client after handshake completed.

gopat · 2015-05-08T13:30:44Z

Wow @shigeki , that's just great!

Tested it with:

var https = require('https');
var pem = require('pem');
var fs = require('fs');

var indexHtml = fs.readFileSync('index.html');

pem.createCertificate({ commonName:'10.10.5.75:4444', days: 10, selfSigned: true }, function (err, keys) {
    if (err) throw err;

    var httpServer = https.createServer({
        key: keys.serviceKey,
        cert: keys.certificate
    }, function (req, res) {
        req.socket.userAgent = req.headers['user-agent'];
        res.writeHead('200', {
            'Content-Type': 'text/html',
            'Content-Length': indexHtml.length
        });

        res.end(indexHtml);
    });

    httpServer.on('connection', function (socket) { //get ip+port before tls does anything
        socket.c_remoteAddress = socket.remoteAddress;
        socket.c_remotePort    = socket.remotePort;
    })

    httpServer.on('clientError', function (err, socket) {
        var ignoreThisError=false;
        if(/:SSL alert number 48\D/.test(err.message)) ignoreThisError=true;
        console.log(err.message);
        console.log('Triggered on client with UA: ' + socket.userAgent);

        //hacky way to get ip+port saved in 'connection' event:
        console.log('From: ',  socket._parent.c_remoteAddress, socket._parent.c_remotePort);
    });

    httpServer.listen(4444,'10.10.5.75');
});

Result:

3074885376:error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:../deps/openssl/openssl/ssl/s3_pkt.c:1461:SSL alert number 48

Triggered on client with UA: undefined
From:  10.10.5.45 51851

10.10.5.45 was the IP for the Firefox vm.
Nothing weird on the Chrome vm: no connections killed/closed nor half closed.
Did a couple tests more, and all results seems correct.

I'd wait for @calzoneman tests also, but i think it's fixed 👌 .

@shigeki :

I should have taken care of the UA property defined in [...]

Well, we're humans after all...

shigeki · 2015-05-08T15:04:46Z

@gopat Thanks. Without your comment, I would not have found myself be wrong. I appreciate your help very much.

indutny · 2015-05-08T15:13:18Z

@shigeki left a comment, thank you!

calzoneman · 2015-05-08T15:20:41Z

@shigeki Looks good to me, my websocket server appears to be logging the error with UA undefined (as expected), and is not killing other clients.

@gopat Thanks for your detailed and thoughtful input on this.

shigeki · 2015-05-08T17:25:52Z

@calzoneman Thanks for testing and I'm sorry for my wrong explanation and decision at the first comment. I've just submitted a PR of #1661 to fix this. If you need, I don't mind to submit this to Joyent node-v0.12 but to be sure that it needs another reviews.

calzoneman · 2015-05-08T18:54:32Z

@shigeki No problem, and thank you for fixing this! I think that we should contribute the fix to joyent/node since they haven't commented on the issue at all, but it's not urgent for me since I can use io.js or apply the fix manually.

Fishrock123 · 2015-05-17T14:57:25Z

Fixed in e008e8f and 7c52e1c

mscdex added the https Issues or PRs related to the https subsystem. label May 6, 2015

shigeki closed this as completed May 7, 2015

shigeki reopened this May 8, 2015

gopat mentioned this issue May 8, 2015

Error: ../deps/openssl/openssl/ssl/s3_pkt.c:1293:SSL alert number 48 websockets/ws#477

Closed

shigeki mentioned this issue May 8, 2015

Fix to handle tls error properly #1661

Closed

Fishrock123 closed this as completed May 17, 2015

crichmon762 mentioned this issue Aug 22, 2023

Compiling string.cc: Error: selected processor does not support `yield' in ARM mode #49287

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

clientError emitted for wrong socket - SSL alert number 48 #1642

clientError emitted for wrong socket - SSL alert number 48 #1642

calzoneman commented May 6, 2015

shigeki commented May 7, 2015

calzoneman commented May 7, 2015

gopat commented May 7, 2015

gopat commented May 7, 2015

shigeki commented May 8, 2015

gopat commented May 8, 2015

shigeki commented May 8, 2015

indutny commented May 8, 2015

calzoneman commented May 8, 2015

shigeki commented May 8, 2015

calzoneman commented May 8, 2015

Fishrock123 commented May 17, 2015

clientError emitted for wrong socket - SSL alert number 48 #1642

clientError emitted for wrong socket - SSL alert number 48 #1642

Comments

calzoneman commented May 6, 2015

shigeki commented May 7, 2015

calzoneman commented May 7, 2015

gopat commented May 7, 2015

gopat commented May 7, 2015

shigeki commented May 8, 2015

gopat commented May 8, 2015

shigeki commented May 8, 2015

indutny commented May 8, 2015

calzoneman commented May 8, 2015

shigeki commented May 8, 2015

calzoneman commented May 8, 2015

Fishrock123 commented May 17, 2015