This article was published in 2018 about R&D work, which resulted in stable production release of Themis that now uses OpenSSL 1.1.1g
If you’re a developer and you’re dealing with cryptography for your app, consider using high-level cryptographic libraries like Themis instead of OpenSSL. No need to struggle with OpenSSL if your goal is to protect users’ data.
Moving to OpenSSL 1.1.0
Besides introducing breaking changes through abandoning backward compatibility on x64 systems, the recent version of Themis (Themis 0.10.0 of 06 Feb 2018) migrated from OpenSSL 1.0.2 to OpenSSL 1.1.0. This has also influenced our other products (Acra, Hermes) that use Themis as the underlying crypto library.
Themis uses OpenSSL as one of the possible sources of cryptographic primitives, and, even though we’re relying more and more on BoringSSL/LibreSSL, we strive to maintain compatibility with the latest version of OpenSSL for platforms where Boring/Libre are not available yet.
The migration to OpenSSL 1.1.0 wasn’t a smooth sailing and in this article we'll share the experience of overcoming the challenges we had. Do you need to migrate your dependencies, too? Read on.
What exactly has changed?
The API for OpenSSL 1.1.0 has been just so slightly updated under the hood, but this broke the backwards compatibility (you can check out the list of things that no longer work and the notes on compatibility layer in the official OpenSSL documentation). One of the main goals of the OpenSSL 1.1.0 API changes was to encapsulate some of the data structures and to provide suitable interfaces for managing them.
More specifically, the structures for managing DH and RSA objects have been removed from the public header files and new functions for managing those have been added. Another significant new feature in OpenSSL 1.1.0, which had implicit influence on Themis was the addition of support for ChaCha20 to libcrypto and libssl because we had considered using those as an alternative to the AES encryption algorithm.
The benefits of data encapsulation introduced in the new version of OpenSSL are:
Changing fields without breaking binary compatibility;
Making applications more robust as well as more assured about their correctness’;
Easier to determine which (new) accessors and setters are needed.
How to know you’re affected, too
If you’ve tried to compile some of your software libraries and saw a message that looked like this:
error: field 'evp_md_ctx' has incomplete type
EVP_MD_CTX evp_md_ctx;
^~~~~~~~~~
error: field 'evp_sym_ctx' has incomplete type
EVP_CIPHER_CTX evp_sym_ctx;it means, you’ve also been affected by the API changes in the 1.1.0 version of OpenSSL.
How the changes affected Themis
Themis is using OpenSSL in Soter, the wrapper for the sources of crypto primitives. Previously, we were using OpenSSL directly, without pointers, as if they were just fields in our own structures, for easier initialisation of stack memory usage. This eliminated the excess work of cleaning up the memory and finding leaks.
But in OpenSSL 1.1.0 the implementations and declarations of the insides of the structures are now hidden. The data can only be accessed using methods, without relying on specific fields. And it’s easy to understand why such changes were necessary from the OpenSSL’s side.
OpenSSL is widely used by developers and can be found in a huge number of products. This limits the OpenSSL creators’ ability to introduce changes to implementations and limits potential optimisations because they cannot just go and change a field type, remove its structures, etc., those being the parts of the existing API in use.
Hiding the data and making the access available using a method allows encapsulating this knowledge, which, in turn, makes it easier to make corrections and alterations without touching the client code. But these exact changes and alterations had to affect the existing code, used by many people. This led to a number of issues experienced by different developers, with the notable examples being this pull request at NodeJS and this commit for POCO (there are numerous other examples, for instance, this and this).
To sum it up, with the recent changes introduced in OpenSSL 1.1.0, we had no other choice but to roll up our sleeves and deal with them.
Need cryptographic help? Consult with our engineers.
How we migrated from OpenSSL 1.0.2 to OpenSSL 1.1.0
First of all, we went looking for the OpenSSL’s changelog to see what exactly has changed and how critical the problem was. We also found a list of helpful implementation recommendations. Then we tried building Themis with the new version of OpenSSL, and got a series of errors.
The following fragment in the documentation explained nearly every error we encountered while building Themis with OpenSSL 1.1.0: “All structures in libssl public header files have been removed so that they are "opaque" to library users. You should use the provided accessor functions instead.” So, armed with the docs, we got started fixing things.
What we did and how you can do it
Step 1. Tests for everything
Everything that we needed to change, we’ve already got covered with tests (namely, tests for Themis in whole and for Soter specifically). So the starting point for you should be a creation of tests for the core logic. It’s better to not even start the migration before arming everything with appropriate tests because otherwise, it would be impossible to find out if you broke something or to know that everything just keeps working just as it used to work before.
Step 2. Pointers everywhere!
After putting tests everywhere, the next step in a successful migration to OpenSSL 1.1.0 is to find a way to somehow store the pointers instead of the data structures. So you will need to change the declarations:
EVP_MD_CTX → EVP_MD_CTX*
EVP_CIPHER_CTX → EVP_CIPHER_CTX*Step 3. More pointing and memory allocation
Now that the OpenSSL hides struct definition, there are a few things that need to be done differently. Previously you had a structure embedded into your own application like this:
struct {
ctx EVP_MD_CTX
}When passing this context into the OpenSSL methods that accepted pointers, previously you had to use &(myvar.ctx).
Now you cannot declare your own structure that contains another structure, you can only point to it:
struct {
ctx *EVP_MD_CTX
}Consequently, now all the cases of context passing need to be written as myvar.ctx (because it is already a pointer).
Also, you need to add the explicit memory allocation for these structures using OpenSSL’s methods:
EVP_MD_CTX* evp_md_ctx = EVP_MD_CTX_create();(See the corresponding fragments in the Themis GitHub repository: https://github.com/cossacklabs/themis/pull/258/files#diff-6f18b74912cce552357c27fe1dd7a0bbR25)
EVP_CIPHER_CTX* evp_sym_ctx = EVP_CIPHER_CTX_new();(https://github.com/cossacklabs/themis/pull/258/files#diff-62a11b9bc3584ecf25695001ebe1c9b7R105)
BIGNUM* rsa_e = BN_NEW()(https://github.com/cossacklabs/themis/pull/258/files#diff-b1a8d20c2dabfcf4753b5240d2d3fba0R434)
We build software that helps developers to solve data security challenges by spending less time on cryptography and preventing nasty errors.
Step 4. Avoiding memory leaks
The memory is now dynamically allocated. It means that when the container structures that contain the fields used by OpenSSL are freed, it is especially important to not forget to add the freeing of these structures to avoid memory leaks:
EVP_CIPHER_CTX_free(evp_cipher_ctx)(https://github.com/cossacklabs/themis/pull/258/files#diff-62a11b9bc3584ecf25695001ebe1c9b7R199)
EVP_PKEY_free(pkey);(https://github.com/cossacklabs/themis/pull/258/files#diff-2cddf6a914e52e98d573370a477636b4R125)
EVP_MD_CTX_cleanup(evp_md_ctx);(https://github.com/cossacklabs/themis/pull/258/files#diff-aafa1892cee4604c383f68f58cefa75fR135)
Step 5. Memory testing
Our recommendation is to add testing of memory leaks through code analysers for your migration process, something like valgrind/pvs-studio/etc. You need to run the testing before introducing changes to make sure that your existing code or the dependencies it uses doesn’t leak memory. This will help fix the current state of affairs and compare it with the state after introducing the changes to the OpenSSL-related parts of the code. It will also provide a chance to put leaks from the 3rd party dependencies into exceptions. What should you use? We’re using valgrind for automatic testing.
It is worth noting that previously we could allocate the memory for a structure and call the methods EVP_DigestInit_ex and EVP_EncryptInit_ex as many times as we needed to, to initialise the initial state and not call the backward methods EVP_DigestFinal_ex and EVP_EncryptFinal_ex. The memory of the structures used will be freed along with your structural wrapper.
It is important to always call the memory-freeing methods alongside the accompanying methods EVP_*_free and EVP_*_reset, otherwise you’re guaranteed to get a memory leak.
Apart from the changes in the allocation and release of the new OpenSSL-related resources, we also had to get rid of the direct calls to the structure fields, using value-returning functions:
EVP_PKEY_base_id(pkey)(https://github.com/cossacklabs/themis/pull/258/files#diff-eaf879962bd91db4a0bc053967b2461eR105)
RSA_set0_key(rsa, rsa_n, rsa_e, rsa_d);
RSA_set0_factors(rsa, rsa_p, rsa_q);
RSA_set0_crt_params(rsa, rsa_dmp1, rsa_dmq1, rsa_iqmp);(https://github.com/cossacklabs/themis/pull/258/files#diff-b1a8d20c2dabfcf4753b5240d2d3fba0R686)
RSA_get0_crt_params(rsa, &rsa_dmp1, &rsa_dmq1, &rsa_iqmp);(https://github.com/cossacklabs/themis/pull/258/files#diff-b1a8d20c2dabfcf4753b5240d2d3fba0R318)
RSA_get0_key(rsa, &rsa_n, &rsa_e, &rsa_d);(https://github.com/cossacklabs/themis/pull/258/files#diff-b1a8d20c2dabfcf4753b5240d2d3fba0R261)
The steps outlined above allowed us to successfully integrate OpenSSL 1.1.0 into Themis 0.10.0 and should help you carry out a similar migration procedure for your own software.
Additional note on testing
In addition to the changes in the code, it’s important not to forget to mirror all these changes in the tests while you’re implementing them. In our case, the tests were written taking into consideration the fact that the memory wasn’t dynamically allocated anywhere (we did this on purpose), so now it was necessary to provide a correct freeing of the used memory in the tests, to ensure the correct work of valgrind.
Summary
Read the changelog and the official guidelines & recommendations for coping with the changes in OpenSSL 1.1.0 (as for 2022, only OpenSSL 1.1.1 manual is available).
See our examples, do it your own way specific to you product.
Watch out for memory leaks.
Test beforehand, test afterwards. Test, test, test.
