There are more POODLEs in the forest

In December it was announced that several TLS server implementations were affected by a problem similar to an SSL v3 issue called POODLE disclosed by Google researchers in October. This attack worked by modifying the padding bytes of the encrypted SSL/TLS records that are used to make the records into even multiples of 8 or 16 byte blocks of data, checking how the server responded, and used this to deduce the plain text of the transmitted data, one byte at a time, with just a few tries.

Several major vendors were affected by the TLS variant of the POODLE issue, and released patches.

As I had started wondering if a TLS variant existed after the first announcement, based on a TLS Workgroup email by Brian Smith, I had added a test for this issue to the TLS Prober, a tool I am using to scan SSL/TLS servers for support of, and compliance with the SSL/TLS specifications. Looking at the results I noticed that at least a couple of major sites where I had contacts were among the affected sites, and I sent them a note about the problem after the problem had been disclosed.

Some time later I noticed that neither site had fixed the issue, and pinged one of them again. They responded back that according to  Qualys‘s SSL Labs test of their site, their server was not affected by the TLS Poodle issue.

Based on this, I contacted Ivan Ristic at Qualys so that we could investigate why our scanners were producing different results. After Ivan had investigated the issue, he discovered that the web site’s TLS server was only checking the correctness of a single byte (the last one) of the padding, not every byte of the padding as is the proper way to perform the check. As it is not checking all the bytes of the padding, this leaves the server vulnerable to modified versions of the POODLE attack.

The SSL Labs test did not detect this because it tested the servers using a method that changed every byte in the padding, while my test used a method that only changed the first byte, leaving the rest unchanged. This meant that the server detected incorrect padding with the SSL Labs test, which was detected as a “Pass”, but not with my test, and thus my test marked the site as “Failing” the test.

The POODLE exploit, as used today, requires changing all bytes of the padding, and in an entire 8 or 16 byte “CBC” block. This means that a server that checks a single byte of the padding forces the attacker to use many more tries in order to exploit the POODLE issue, increasing the work 256-fold for each byte tested by the server. This does make it more time consuming to exploit the issue, for example, in this case, increasing the number of tries from 256 to 65000 for each discovered plain text character, and if 4 bytes were checked the number of tries would increase to over 1 trillion for each character. However, it does not make it impossible to discover the plain text, just much more costly.

As we had confirmed that the site, which was using the most recent version of the software, was still vulnerable to POODLE, our next step was contact the vendor to inform them about the problem. The website in question was quite willing to share the relevant information about the server with us. (very much the opposite of what I experienced as part of another investigation), and by agreement, I contacted the vendor, Citrix, about the issue. They confirmed they were working on a patch, which they released June 30.

As a result of the discovery of partial verification of the padding, I decided to extend the bad padding tests. The first test, using extensive variations of padding tests, did not work very well, since many servers just stop responding when they detect the bad padding, without closing the connection, meaning that a test will have to time out (by design, 10 seconds) twice, before failing, for each test. I eventually reduced the number of tests to three: The first, last and center bytes. This tests the most likely problem scenarios; a badly implemented server would probably have to be (intentionally) tailored specifically to avoid detection by a scanner just testing these three bytes (I wonder how a product liability lawyer would react to such a design?).

In early May, I noticed that, among the sites in my list of affected sites, a major Scandinavian bank site was affected by a slightly different variation of the TLS POODLE issue: Its server was checking both the last and the center byte of the padding, but not the first byte. This was different from the Citrix profile, indicating that the issue was in a different vendor’s product.

After exchanging a couple of emails with the bank about the issue, which could not then be detected using the SSL Labs test (since Ivan, as part of our responsible disclosure process while Citrix worked on a patch, had not updated SSL Labs to test for the new variation), they provided me with the necessary information about their TLS server (again surprising me with their level of cooperation), revealing that it was a Cisco product, the ACE 30 server.

I contacted Cisco about this issue and they confirmed the issue and started researching the cause. They discovered that the problem affected one other Cisco product, a recent model of the Cisco ASA security appliance, and that the issue was caused by a third-party SDK for a component produced by Cavium. While a patch have been developed the patch has not yet been released.

It turned out that, in this case, only one byte, the first, of the padding wasn’t checked. Given that current POODLE attack requires corrupting an entire block of 8 (3DES) or 16 (AES) bytes, a single unchecked byte is very likely not a problem, and my evaluation is that this particular variation isn’t exploitable today.

However, that may only be the case today. Unfortunately, over the past several years, many of the attacks on TLS have leveraged missing checks and even small timing differences that, paraphrasing, “were not significant enough to be exploited, ever”. Tomorrow, additional issues might be discovered that can be combined with such an issue as this, allowing an attack to succeed.

As a result of these variations in padding check implementations, according to my numbers, scans to count servers affected by POODLE may not have counted as many as 19% of the servers; due to the different implementation my test appears to have missed 1% of the servers. At present, about 25000 of the 520000 servers scanned by the TLS Prober are not checking at least one of the modified padding bytes. Of these, 4800 tests the last byte of the padding, but not the first, and would incorrectly be categorized as passing the test by tests that modify the last byte, or all bytes. 240 checks the first byte, but not the last byte, and was incorrectly detected as a pass by the TLS Probers original test.

Cisco and Citrix are likely not the only vendors affected, but they are the only ones I was able to identify quickly.

I might have spent some more time trying to identify more vendors, and I assume some will argue I should have done so. To this I will say that TLS POODLE is already a publicly known issue, and has been known for 7 months. Even if vendors were not getting warnings by the available test tools, they should have reviewed the relevant code to make sure they were not affected.

Additionally, contacting web sites to obtain vendor information takes quite a bit of research, time and effort for each web site, in order to locate contacts and write the emails. While I was pleasantly surprised by the two sites mentioned above (and there were special circumstances involved that I believe helped), my experiences with the almost total lack of site administrator cooperation while investigating another issue did not exactly motivate me to spend that time and effort. In fact, it did quite the opposite, unfortunately.

I recommend that all TLS implementation vendors that have not already verified that their implementation always checks all bytes of the TLS Record padding, should do so immediately, and if they find issues, release patches with due speed, so that we can cage this pack of POODLEs.