"All In The Family": Scanning For Padding Oracles

As you might have heard, we recently got our paper on padding oracle attacks accepted to the USENIX Security Conference. In this paper, we describe and evaluate a scanning methodology with which we found several padding oracle vulnerabilities in devices from various vendors. In total, we found that 1.83% of the Alexa Top 1 Million have padding oracle vulnerabilities.

To test whether a server is vulnerable, we specified different padding oracle vectors which we send to the system under test, using different cipher suites and protocol versions. If the server does not behave identically (on both the TLS and TCP layers), we consider it to be vulnerable to a padding oracle attack, since it is leaking information about the plaintext via behavior differences. Depending on the responses to such padding oracle vectors, one can estimate which implementation is responsible for the vulnerability. We contacted quite a few website owners and tried to cooperate with them, to find out which vendors and TLS stacks are responsible for the identified vulnerabilities. You can find our current disclosure status on this issue on https://github.com/RUB-NDS/TLS-Padding-Oracles.
We are currently in contact with other vendors to fix the remaining vulnerabilities, but the some of the rare (in terms of the number of affected hosts) vulnerabilities are currently not attributed. To fix the remaining vulnerabilities, we ask for your assistance to help get rid of this issue. For this purpose, we integrated a standalone version of our padding oracle evaluation tool into our TLS-Scanner (v.2.7) project. This tool allows you (among other things) to evaluate if a specific server is vulnerable.

When the tool detects a vulnerability, it tries to attribute the vulnerability to a specific vendor or CVE. If we already know of the vulnerability of the server you scanned, the tool will print its details. If the tool does not have a description of the vulnerability in its database, it will ask you to notify us about the vulnerable server, such that we can notify the vendor and get the device fixed. To be clear: the tool never sends any data to us - you have the choice of whether to notify us (and what details to include). There is a chance that the tool's attribution is also mistaken, that is, the tool lists a vendor for your host, but you know for sure that you do not use an implementation by this vendor. Please contact us in such cases as well.

How to use the Tool

First, you need to grab hold of the tool. There are 3 ways to get your hands dirty: pre-compiled, self-compiled or Docker. We provide a pre-compiled version of the tool since the compilation process can get quite messy if you are not familiar with java and maven. You can directly download the resulting project here. However, if you also want to play around with the code, you have to compile everything yourself.

Building the TLS-Scanner

For this, you will need (Git), maven (sudo apt-get install maven), OpenJDK-8 (I can guarantee that this version works, other versions might work as well, have not tested it).

You will need to get TLS-Attacker 2.9 (if you do not already have it):
Now we can clone and install the TLS-Scanner

Docker

We also provide a Dockerfile, which lets you run the scanner directly

Getting Started

If you start the TLS-Scanner you should be greeted by a usage info, similar to the one below:

or

This should give you an overview of the supported command line flags. The only really required one is the -connect flag (similar to OpenSSL and TLS-Attacker), with which you specify which host to scan. The most basic command is therefore:

Your output may look something like this:

By default, TLS-Scanner will run single-threaded. In such cases the scanning will take a while; just how long it will take depends on your server configuration. The scanner also supports multi-threading, which drastically improves the performance. There are two parameters to play around with, -threads, which controls how many different "probes" are executed in parallel, and -aggressive , which controls how many handshakes can be executed simultaneously. If you want the fastest results the following parameters are usually a good choice:

But lets get back to the results of the Scanner. Currently the Scanner supports a bunch of well known tests, like supported ciphersuites or protocol versions. These are very similar to what you may be used to from other scanners like ssllabs or testssl.sh.

Padding Oracles

The main advantage of our scanner is the ability to scan for padding oracle vulnerabilities (which is probably why you are reading this post). You will see if you are vulnerable in the "Attack Vulnerabilities" section. For example, when scanning hackmanit.de, the result is false. Good for us! But as you might have seen there is also another section in the scanner report:"PaddingOracle Responsemap"
This section lists the responses of the scanned host for each padding oracle vector, for each cipher suite and protocol version. For hackmanit.de, there is no detected difference in responses, which means hackmanit.de is not vulnerable to the attack:
If we want, we can also look at the concrete responses of the server. For this purpose, we start the scanner with the -reportDetail flag:

With this flag we now get the following details:

So what does this all mean? First of all, we named our malformed records. The interpretation of those names is visualized in the following table:

BasicMac-<position>-<XOR>	A Record with ApplicationData, MAC and padding bytes, where the padding byte at <position> is XOR'd <XOR>
MissingMacByteFirst	A Record without ApplicationData, where the first byte of the MAC is missing
MissingMacByteLast	A Record without ApplicationData, where the last byte of the MAC is missing
Plain FF	A Record without ApplicationData & MAC which only contains Paddingbytes: 64* 0xFF
Plain 3F	A Record without ApplicationData & MAC which only contains Paddingbytes: 64* 0xF3
InvPadValMac-[<position>]-<appDataLength>-<paddingBytes>	A Record with invalid padding and valid MAC. The Record contains <appDataLength> many ApplicationData bytes and <paddingBytes> many PaddingBytes. The Padding is invalid at <position>.
ValPadInvMac-[<position>]-<appDataLength>-<paddingBytes>	A Record with valid padding and invalid MAC. The Record contains <appDataLength> many ApplicationData bytes and <paddingBytes> many PaddingBytes. The MAC is invalid at <position>.
InvPadInvMac-[<position>]-<appDataLength>-<paddingBytes>	A Record with invalid padding and invalid MAC. The Record contains <appDataLength> many ApplicationData bytes and <paddingBytes> many PaddingBytes. The MAC is invalid at the first position. The Padding is invalid at <position>.

Next to the name you can see what the actual response from the server was. Alert messages which are in [] brackets indicate that the alert was a fatal alert while () brackets indicate a warning alert. ENC means that the messages were encrypted (which is not always the case). The last symbol in each line indicates the state of the socket. An X represents a closed socket with a TCP FIN, a T indicates that the socket was still open at the time of measurement and an @ indicates that the socket was closed with an RST. So how did Hackmanit respond? We see a [BAD_RECORD_MAC] ENC X, which means we received an ENCrypted FATAL BAD_RECORD_MAC alert, and the TCP connection was closed with a TCP FIN. If a server appears to be vulnerable, the scanner will execute the scan a total of three times to confirm the vulnerability. Since this response is identical to all our vectors, we know that the server was not vulnerable and the scanner is not re-executing the workflows.

Here is an example of a vulnerable host:
As you can see, this time the workflows got executed multiple times, and the scanner reports the cipher suite and version as vulnerable because of "SOCKET_STATE". This means that in some cases the socket state revealed information about the plaintext. If you look closely, you can see that for ValPadInvMac-[0]-0-59, ValPadInvMac-[8]-0-59 and ValPadInvMac-[15]-0-59 the server failed to close the TCP socket, while for all other vectors the TCP connect was closed with a TCP FIN. The server was therefore vulnerable.

Since the server was vulnerable, TLS-Scanner will also print an additional section: "PaddingOracle Details"

In this section we try to identify the vulnerability. In the example above, TLS-Scanner will print the following:

As you can see, we attribute this vulnerability to OpenSSL <1.0.2r. We do so by looking at the exact responses to our malformed records. We additionally print two important facts about the vulnerability: Whether it is observable and its strength. The precise details of these properties are beyond the scope of this blogpost, but the short version is:
If an oracle is observable, a man in the middle attacker can see the differences between the vectors by passively observing the traffic, without relying on browser or application specific tricks. A strong oracle has no limitations in the number of consecutive bytes an attacker can decrypt. If an oracle is STRONG and OBSERVABLE, then an attacker can realistically exploit it. This is the case in the example above.
For more details on this, you will have to wait for the paper.

Attribution

As you can see, we try to fingerprint the responsible device/implementation. However, we were not able to identify all vulnerable implementations yet. If we cannot attribute a vulnerability you will receive the following message:

Could not identify the vulnerability. Please contact us if you know which software/hardware is generating this behavior.

If you encounter this message, we do not know yet who is responsible for this padding oracle and would be happy to know which device/vendor is responsible. If you know who is, please contact us so that we can get in contact with the vendor to fix the issue. To reiterate, the tool never sends any data back to us, and it is your choice whether to contact us manually or not.

There are also some cases in which we can identify the vendor, but the vendor has not patched the vulnerability yet. If you encounter such a host, the scanner will tell you that we know the responsible vendor. To prevent abuse, we do not include further details.

Non-Determinism and Errors

In some cases, the scanner is unable to scan for padding oracles and reports ERROR or non-deterministic responses. The ERROR cases appear if the scanner failed could not handshake with the specified cipher suite and protocol version. This might be due to a bug in the tested TLS-Server or a bug in TLS-Attacker or TLS-Scanner. If you think the handshake fails because of an issue on our side, please open an issue on Github, and we will investigate. The more interesting cases are the non-deterministic ones. In such cases the scanner observed non-identical scan results in three separate scans. This can be due to non-determinism in the software, connection errors, server load or non-homogeneous load balancing. Currently, you will have to analyze these cases manually. In the paper, we excluded such hosts from our study because we did not want to artificially improve our results. But we understand that you as a tester want to know if the server is vulnerable or not. If the server is not truly vulnerable you would see the differences between the answers spread across all the different vectors. If the differences only appear on a subset of malformed records the server is very likely vulnerable. If you are unsure, you can also always scan multiple times (or scan slowly), increase the timeout, or if you are entirely lost get in touch with us.

How YOU can help

Please use the scanner on all your hosts and check for padding oracle vulnerabilities. If the scanner can identify your vulnerability, a patch should already be available. Please patch your system! If the scanner does not identify the vulnerability (and instructs you to contact us), please contact us with the details (robert.merget@rub.de). If you can provide us with the detailed output of the scanner or even better, the name of the host, with the corresponding vendor, we could match the results with our database and help fix the issue. We can already attribute over 90% of the vulnerabilities, but there is still a lot to be discovered. We mostly scanned the Alexa top 1-million on port 443. Other protocols like IMAPS, POP3S, etc. might have different implementations with different vulnerabilities. If you find vulnerabilities with our tool, please give us credit. It helps us to get more funding for our project.

Issues with the Scanner

A notable feature of our scanner is that we do not actively try to avoid intolerances (like not scanning with a lot of cipher suites in the Hello messages etc.). We believe that doing so would hide important bugs. We are currently experimenting with intolerances checks, but the feature is now still in beta. If we cannot scan a server (most of the time due to intolerances or SNI problems), the scanner will report a lot of intolerances and usually no supported protocol versions. Some intolerances may trick the scanner into reporting false results. At the current stage, we cannot make any guarantees. If you are using this tool during a pentest, it might be smart to rescan with other scanners (like the recently released padcheck tool from our colleague Craig Young) to find the ground truth (this is good advice in general, since other mainstream scanners likely have the same issues). Note however that it is very unlikely that the scanner reports a false positive on a padding oracle scan.

Conclusion

There are still a lot of padding oracle vulnerabilities out there - and a lot of them are still unpatched. We hope you will find some bugs with the tool :) Happy H4cking :D

Acknowlegements

This is joint work from Robert Merget (@ic0nz1), Juraj Somorovsky (@jurajsomorovsky), Nimrod Aviram (@NimrodAviram), Janis Fliegenschmidt (@JanisFliegens), Craig Young (@craigtweets), Jörg Schwenk (@JoergSchwenk) and (Yuval Shavitt).

Continue reading