Today, oCERT published advisory 2015-003 describing a TLS vulnerability in MySQL and derivative products. The content isn’t exactly news – it is documented legacy behavior and the subject of an earlier blog post describing how MySQL Server 5.7 solves the problem. That said, the efforts of Duo Security are certainly appreciated and welcomed – it provides a meaningful context to discuss how to properly harden existing MySQL 5.5 and 5.6 deployments, as well as frame a discussion on potential changes in these versions to increase security.
The vulnerability described in the advisory relies on the legacy behavior of the client –ssl option, which simply enables – but does not require – TLS connections. When connecting to a MySQL Server which has TLS enabled, this will normally result in TLS being used for the connection. When the server does not support TLS, though, the connection will be established without TLS. For versions of MySQL prior to 5.7.3, there is no means on the client side to require TLS connections. This allows an attacker with the ability to actively manipulate network packets to silently negotiate away TLS.
Such attacks require the ability to manipulate packets between the MySQL Server and client on the network in real time. There are several options to secure connections over untrusted networks where such activity may take place, but they were unfortunately not referenced in the advisory.
Secure communication has several aspects – not only does the data need to be encrypted, but identity needs to be established. It’s insufficient to know nobody other than the recipient can read data being sent if the identity of the recipient is unknown. A client may encrypt data, but if it’s being sent to somebody impersonating a MySQL Server, the data is compromised.
When an account is configured with the REQUIRE X509 option, the client must provide an acceptable certificate matching the criteria defined in the GRANT command – and doing so requires TLS connections from the client. In such cases, an nefarious proxy cannot negotiate away TLS.
Use of client-side key material is surely uncommon, but in a deployment where MySQL communication must be secured across an untrusted network, it’s a necessary step. More details about configuration to use client-side key material can be found in the manual, and don’t forget that you can use the new mysql_ssl_rsa_setup utility from MySQL 5.7 packages to easily produce client key material (obviously, properly-signed keys are preferred).
Please note that defining accounts with the REQUIRE SSL option is not sufficient to bypass this vulnerability – a nefarious proxy could easily establish a SSL connection between the proxy and the server, while leaving the proxy’s own connection to the client in plain-text.
Use SSH Tunnels
If setting up client-side key material is too much of a burden, you can secure your communication by using a SSH tunnel instead of TLS. Tools such as MySQL Workbench make such connections even easier.
Use 5.7.3+ Clients
The easiest solution is to use clients from MySQL Server 5.7.3 or above. While MySQL 5.7 is not yet GA (it is in Release Candidate state as I write this), there’s no reason you cannot use just the client programs out of the 5.7 package which allow TLS to be required with the –ssl option – they will work just fine with 5.5 or 5.6 servers.
Possible behavioral changes in 5.5 and 5.6
The legacy behavior that prevents clients from requiring TLS connections is clearly undesirable, but Oracle takes great pains to avoid behavioral changes in maintenance releases for already-GA products. For that reason, the –ssl option was redefined to require TLS only in 5.7, where it accompanies a change that makes TLS preferred by default. The latter change has potential significant impacts to users and third-party products, and cannot be reasonably back-ported to 5.5 or 5.6. However, we are considering back-porting the change which redefines –ssl to mean “TLS is required” to these versions. Below are some details and considerations – we would love to hear your feedback.
Impacts to existing users of –ssl
It’s my belief that most users who specify –ssl as a client option actually intend for the resulting connection to use TLS, and not simply allow TLS. Furthermore, most users who specify –ssl today would likely prefer – or even expect – connections to fail when TLS cannot be negotiated successfully. Back-porting the 5.7 changes would align behavior with most user expectations (as I perceive them).
There may be users who actually understand and rely on legacy behavior, and have specified –ssl even though target servers do not support TLS. If the 5.7 behavior is back-ported, such users will find that where previous client versions established non-secure connections without notice, new connections will fail until the –ssl option is removed.
Users who want legacy behavior may obtain that by specifying other TLS-related client options, such as –ssl-cert, but without –ssl. This would require changes to user scripts which rely on legacy –ssl behavior, though.
Alternative: Implement new multi-state option
The –ssl option is a boolean configuration option, and as of MySQL 5.7, there are currently three different TLS “modes”:
- TLS disabled
- TLS preferred
- TLS required
In MySQL 5.7, TLS preferred is the default state; TLS required results from setting –ssl; TLS disabled can be triggered using –skip-ssl (among other boolean configuration variants). One option considered is to eliminate the –ssl option over time, and replace it with a multi-value option, such as –ssl-mode=[DISABLE|PREFER|REQUIRE]. This would need to co-exist with the –ssl option for a while, but at some point, the –ssl option would be deprecated and removed. This would result in requiring all scripts using –ssl to be modified, and likely causing pain for users whose scripts cannot deal with new deprecation warnings, even before functionality is fully removed.
The oCERT advisory highlights changes already made to MySQL Server 5.7 to improve security, and provides a context to discuss potential back-ports to MySQL 5.6 and 5.5. There are also meaningful steps which users of MySQL 5.5 or 5.6 can take today to eliminate exposure to this vulnerability.
9 thoughts on “SSL/TLS in 5.6 and 5.5 – oCERT Advisory”
From my PoV, changing the behavior of –ssl from “prefer” to “require” is more like fixing the issue that has been there for long time, so it would be my preferred variant.
IOW, if my scripts use –ssl and the SSL cannot actually be used, I’d like rather to see it fail and fix the script.
Thanks for your feedback – it’s very helpful! Like you, we’re not eager to change behavior, but this dangerous legacy behavior seems like an appropriate exception to that policy.
This is not easy to backport as it is the intended and documented behaviour since early 2000’s until now. But for newer versions the default behaviour is changed to mitigate this risk. For MariaDB the issue is tracked at https://mariadb.atlassian.net/browse/MDEV-7937
That’s exactly why we’re having this discussion. My belief (which seems consistent with what Honza says) is that the legacy behavior – documented or intended by the original designers of this feature – is faulty, and inconsistent with what most users expect the behavior of the –ssl option to be. As noted in my post, Oracle is open to considering options on how to deal with this, but when even distro maintainers advocate for a behavioral change, it seems like there’s a pretty good case to be made.
It occurred to me that it might be valuable to focus in particular on the –ssl-verify-server-cert flag (and its corresponding MYSQL_OPT_SSL_VERIFY_SERVER_CERT option in libmysqlclient). As I see it, ‘–ssl-verify-server-cert’ is rather pointless without also enforcing that TLS must be used for a connection. However, the reverse is also true: even if TLS is mandatory, a successful MITM attack will still be possible unless –ssl-verify-server-cert is also enabled. Being realistic, this is ok, as other configuration is needed to make cert validation work (the client has to be told which CAs to trust).
So, perhaps it would be a good compromise if ‘–ssl’ didn’t necessarily enforce TLS, but ‘–ssl-verify-server-cert’ did; this flag seems to provide the clearest and most direct indication that a user really intends to defend against MITM attacks. Also, I think it would be best if this behavior change applied to the corresponding flags in libmysqlclient as well (i.e. enabling MYSQL_OPT_SSL_VERIFY_SERVER_CERT should implicitly turn on MYSQL_OPT_SSL_ENFORCE).
If I see correctly, there hasn’t been done anything around this issue so far for MySQL 5.5 and 5.6, but it was for MariaDB 5.5 and later.
We’re currently considering to use MariaDB’s solution from https://mariadb.atlassian.net/browse/MDEV-7937, which is different from what is used in MySQL 5.7, but would make things a bit better.
Are there any reasons why this is wrong approach for MySQL? Would it be possible to use MariaDB’s solution for MySQL 5.5 and 5.6?
I think the answer depends on the problem you are trying to solve. If you want the means to ensure a TLS connection, configuring the account with REQUIRE X509 does that (with the added benefit that the client endpoint identity is verified). Another way to do that would be to configure the account to use sha256_password authentication, which also has the benefit of making password storage more secure. It doesn’t really help situations where users specify –ssl and expect secure connections to result.
It can help when you want clients using the same account to sometimes connect with TLS, and sometimes without – completely at client discretion. I don’t find that use case terribly compelling – it seems more likely that some client hosts would be considered secure, and some insecure, and that this should be reflected in separate account configuration on the server side to require TLS connections from insecure hosts.
The proposed change is logical in that it seems safe to assume that clients which indicate that the server name should be verified against the TLS certificate want TLS connections. It’s still a behavior change in a maintenance release, it’s not entirely logical that –ssl-verify-server-cert is the trigger to require TLS, and a number of deployments are unlikely to be able to use –ssl-verify-server-cert. –ssl-verify-server-cert verifies the host name of the server to which it is instructed to connect with the CN of the server certificate it receives. In a number of HA/load-balanced deployments, the actual server needs to be dynamic, and CA verification is far preferable. This patch does not support such use cases.
The patch also appears to only affect mysql command-line clients, and not the underlying libmysql client library used by various connectors. Identical values for comparable options will enforce TLS in clients, but not in connectors using the same libmysql client library.
The fact that the most important need (ensure TLS connections) can be facilitated through other means, coupled with the limited and inconsistent applicability of the solution, makes me believe this has extremely limited value and does not justify a behavioral change during the maintenance cycle. But if there’s a large demand for this behavior change, we can surely consider it.
Just confirming, that I was referring especially about situations where users specify –ssl and expect secure connections to result.
Anyway, I see all your points, thanks for great response, Todd!
However, considering that the same change hasn’t cause anything entirely wrong it MariaDB (I haven’t seen any issue reported), it still seems to me a good enough solution worth considering to apply in MySQL as well. My PoV is it’s still a good deal to use such an imperfect solution than doing nothing about this, since it would mean keeping the 5.5 and 5.6 versions (still heavily used) with a known vulnerability, even if there are ways to prevent it.
Thanks Honza! Just to make sure I was clear – the MariaDB solution does not make it so that --ssl results in secure connections. Additional options have to be specified.
Since we’re talking exclusively about command-line clients (the libmysql client library, and connectors built on top of that, would not be affected), would a viable alternative be to produce a warning to the user when –ssl was specified, but TLS connection was not made?
I opened Bug#79862 to track this – please feel free to comment there.