Replacing OpenSSL with Libsodium

This article was published in 2017 about R&D work, which resulted in stable production release of Themis.

Intro

In our ongoing effort to make Themis work with different cryptographic backends, we've decided to try something more challenging than just displacing similar primitives. This time we decided to make Themis work on Daniel J. Bernstein’s cryptography, as it is introduced in NaCl.

What if one day it turns out that Daniel Bernstein’s assumptions about the rest of the world are correct, and everybody else is a lunatic? We’d like to understand how quick it would take to move all our products to a different cryptographic backend and a different set of algorithms.

To better understand the goals of our experiment, you can check out the previous article in the series where we tried replacing OpenSSL for BoringSSL as the underlying crypto primitive of Themis. While previously we didn’t need to change something under the hood of Themis, the undertaking with LibSodium brought up some interesting problems for us to overcome.

UPDATE:

Many thanks to Frank Denis for reaching out to us after the publication of this article and pointing out that the multipart signature has been present in LibSodium since March 2017. The experiments described in the article were carried out earlier than that, hence the use of a couple of non-elegant workarounds. Also, the LibSodium interfaces have since been improved and updated.

We would like to emphasize that this post describes an experiment rather than provides step-by-step instructions how to integrate LibSodium into Themis for production use.

About LibSodium

LibSodium is a modern cross-compilable, API-compatible, and easy-to-use port of NaCl. It is a library for encryption, decryption, signatures, password hashing, and more created by Frank Denis. There is a major difference between LibSodium and other crypto libraries (such as OpenSSL and CryptoPP) since it is using Daniel Bernstein’s curve 25519 algorithms as asymmetric primitives that are currently considered to be the fastest ECC curves and are not covered by any patent at the time of writing (apart from FourQ, which looks very promising, yet some time has to pass for it to be well recognized).

Challenges and solutions

When trying to add the support of LibSodium to Themis, we encountered several challenges:

1. A striking difference between the way crypto algorithms are used in OpenSSL and LibSodium;

2. An absence of non-AEAD encryption in LibSodium, which is necessary for correct functioning of Soter;

3. The absence of asymmetric key agreement primitive in the form of a separate feature in the main LibSodium interface (as it only exposes the very simple high-level API for each operation).

We build software that helps developers to solve data security challenges by spending less time on cryptography and preventing nasty errors.

Dev platform

End-to-end encryption for remote debugging tool

Data encryption and isolation in AppSpector for securing mobile development and helping digital nomads around the world.

Read story

Overcoming the difference in using algorithms

To be able to add the support of LibSodium to Themis as backend, we first needed to change some internal principles of communication between Themis and Soter. Initially, Soter was planned as a middleware wrapper for various crypto libraries. But the main principle behind its work was sticking to well-known standardised crypto primitives, such as ECDSA, RSA, ECDH, etc.

This approach was the basis for some of the principles behind the creation of Soter’s interfaces:

• Soter only allows RSA and EC-based asymmetric encryption;

• Soter only allows the use of AES symmetric encryption algorithms (in various modes).

The main difference between Soter and LibSodium is that in LibSodium:

• only elliptic curve ed25519 or curve2559 are used for asymmetric cryptography schemes;

• only ChaCha20-Poly1305 algorithms are used for symmetric cryptography;

• LibSodium does not support non-AEAD symmetric encryption schemes.

First of all, we needed to move the process of algorithm selection from Soter-level to backend level. This way, now it was the backend that determined, which algorithms Soter will support.

Next, to move the algorithm selection process from Themis to Soter, we needed to implement some changes in the Soter interfaces. In Soter, we generally distinguish the following primitives:

• Asymmetric cipher;

• Asymmetric key agreement;

• Asymmetric signature;

• crc32;

• HASH;

• HMAC;

• KDF;

• Symmetric encryption;

• Symmetric AEAD encryption.

Some of these primitives – crc32, hmac, kdf – are independent of the used backend. Others use predefined algorithms which need to be implemented and present in the backend.

The difference in algorithms between LibSodium and Soter are illustrated in the following table:

Initially, the algorithm definitions were a part of Soter:

src/soter/soter_asym_cipher.h

enum soter_sign_alg_type
{
  SOTER_SIGN_undefined,        /**< undefined */
  SOTER_SIGN_rsa_pss_pkcs8,    /**< RSA with PSS padding */
  SOTER_SIGN_ecdsa_none_pkcs8  /**< ECDSA */
};

/** @brief signature algorithm typedef */
typedef enum soter_sign_alg_type soter_sign_alg_t;

src/soter/soter_sym.h

/** AES in ECB mode with pkcs7 padding */
#define SOTER_SYM_AES_ECB_PKCS7     0x10010000
/** AES in CTR mode */
#define SOTER_SYM_AES_CTR           0x20000000
/** AES in XTS mode */
#define SOTER_SYM_AES_XTS           0x30000000
/** AES in GCM mode (with authenticated encryption) */
#define SOTER_SYM_AES_GCM           0x40010000

Any backend for Soter needs to have soter_engine_consts.h file which contains defines for all available algorithms supported by the backend.

src/soter/openssl/soter_engine_consts.h:

#define SOTER_ASYM_RSA 0x00415200
#define SOTER_ASYM_EC  0x00434500

/** AES in ECB mode with pkcs7 padding */
#define SOTER_SYM_AES_ECB_PKCS7     0x10010000
/** AES in CTR mode */
#define SOTER_SYM_AES_CTR           0x20000000
/** AES in XTS mode */
#define SOTER_SYM_AES_XTS           0x30000000
/** AES in GCM mode (with authenticated encryption) */
#define SOTER_SYM_AES_GCM           0x40010000

src/soter/libsodium/soter_engine_consts.h

#define SOTER_ASYM_X25519 0x10
#define SOTER_ASYM_ED25519  0x20
#define SOTER_SYM_CHACHA20     0x10010000

Obviously, to switch the backends now, we would need to change all the references for algorithm identifiers in the code of Soter, Themis, and maybe even on the application level. But the switch between the backends can be simplified through defining the default algorithm identifiers for every crypto primitive used:

src/soter/openssl/soter_engine_consts.h

#define SOTER_ASYM_KA_DEFAULT_ALG SOTER_ASYM_EC|SOTER_ASYM_EC_LENGTH_256
#define SOTER_ASYM_SIGN_DEFAULT_ALG SOTER_ASYM_EC|SOTER_ASYM_EC_LENGTH_256

src/soter/libsodium/soter_engine_consts.h

#define SOTER_ASYM_KA_DEFAULT_ALG SOTER_ASYM_X25519
#define SOTER_ASYM_SIGN_DEFAULT_ALG SOTER_ASYM_ED25515

Absence of some LibSodium functionality needed by Soter

As it was stated in the table above, asymmetric key agreement primitive and non-AEAD symmetric encryption primitive are absent in LibSodium. However, in Soter we need them to implement higher-level Themis functions. Let’s look deeper into this issue.

No AEAD symmetric encryption.

Since LibSodium doesn’t support non-AEAD encryption, Context imprint mode of Secure Cell can’t be implemented, for good. In standard crypto primitives, it is AES in CTR mode, which has certain security trade-offs, indicated in our documentation, and its own certain use-cases (length-preserving encryption without ability to store auth tag elsewhere or concatenate with ciphertext).

So, Context imprint mode is disabled for LibSodium-based Soter.

Since LibSodium doesn’t support non-AEAD encryption, when this backend was used with Themis, the Context imprint mode of Secure Cell wasn’t supported.

With OpenSSL backend, such primitive is implemented with AES encryption in CTR mode. This is more dangerous security-wise than any AEAD encryption scheme.

Because of this, usage of Secure Cell in Context imprint mode should be limited to very seldom and extremely specific cases. Wherever possible, usage of AEAD symmetric encryption is strongly recommended.

This is why we decided to disable the Context imprint mode of Secure Cell when using LibSodium backend.

src/soter/libsodium/soter_sym.c

soter_status_t soter_sym_ctx_init(soter_sym_ctx_t* ctx,
                                  const uint32_t alg,
                                  const void* key,
                                  const size_t key_length,
                                  const void* salt,
                                  const size_t salt_length,
                                  bool encrypt){
  if(!ctx || !key || key_length!=crypto_stream_chacha20_KEYBYTES || !salt || salt_length!=crypto_stream_chacha20_NONCEBYTES){
    return SOTER_INVALID_PARAMETER;
  }
  ctx->alg=alg;
  memcpy(ctx->key, key, key_length);
  memcpy(ctx->nonce, salt, salt_length);
  ctx->count=0;
  return SOTER_SUCCESS;
}

Asymmetric key agreement

The second feature that is missing when we try to use LibSodium is asymmetric key agreement primitive that is used in Secure Message and Secure Session.

Our study of LibSodium has show that asymmetric key agreement is present, but it is not added as a separate feature to the main LibSodium interface. Obviously, the key agreement is a part of Public-key authenticated encryption feature of LibSodium for which X25519 scheme is used (which essentially is ECDH over Curve25519).

To explicitly add asymmetric key agreement to Soter while trying out LibSodium, we had to use a kind of a low-level interface:

src/soter/libsodium/soter_asym_ka.c

soter_status_t soter_asym_ka_derive(soter_asym_ka_t* ctx, const void* peer_key, size_t peer_key_length, void *shared_secret, size_t* shared_secret_length){
  if (!ctx || !peer_key || !shared_secret_length || peer_key_length<sizeof(soter_container_hdr_t) || peer_key_length!=ntohl(((soter_container_hdr_t*)peer_key)->size) || SOTER_SUCCESS!=soter_verify_container_checksum((soter_container_hdr_t*)peer_key)){
    return SOTER_INVALID_PARAMETER;
  }
  if(!shared_secret || crypto_generichash_BYTES > *shared_secret_length){
    *shared_secret_length = crypto_generichash_BYTES;
    return SOTER_BUFFER_TOO_SMALL;
  }
  unsigned char scalarmult_q[crypto_scalarmult_BYTES];
  if(0!=crypto_scalarmult(scalarmult_q, ctx->pk.key, ((soter_x25519_pub_key_t*)peer_key)->key)) {
    return SOTER_FAIL;
  }
  *shared_secret_length = crypto_generichash_BYTES;
  crypto_generichash_state h;
  crypto_generichash_init(&h, NULL, 0U, *shared_secret_length);
  crypto_generichash_update(&h, scalarmult_q, sizeof(scalarmult_q));
  crypto_generichash_final(&h, shared_secret, *shared_secret_length);
  return SOTER_SUCCESS;
}

When adding LibSodium support as a backend for Soter, we discovered some differences in implementation of signature algorithms in LibSodium and OpenSSL-like libraries. In OpenSSL, the signature interface includes three stages of signature creation:

• Initialisation of the context with a message digest/hash function and EVP_PKEY key;

• Adding the message data (this step can be repeated many times);

• Finalising the context to create the signature.

Such approach allows calculating the signatures of data blocks, the length of which is not known at the moment when the signature calculation starts.

LibSodium, however, uses a one-shot approach to the signature calculation:

src/soter/libsodium/soter_sign.c

soter_status_t soter_sign_init(soter_sign_ctx_t* ctx, const void* private_key, const size_t private_key_length){
  if(!ctx || !private_key || private_key_length<sizeof(soter_container_hdr_t) || private_key_length!=ntohl(((soter_container_hdr_t*)private_key)->size) || soter_verify_container_checksum((soter_container_hdr_t*)private_key)){ //add algorithm testing
    return SOTER_INVALID_PARAMETER;
  }
  if(0!=crypto_sign_init(&(ctx->state))){
    return SOTER_FAIL;
  }
  memcpy(&(ctx->key.sk), private_key, private_key_length);
  return SOTER_SUCCESS;
}

This approach requires knowing the length of data that needs to be signed before the signature calculation starts.

However, we know that Themis – the main user of Soter – works on message layer and always knows the length of the message that needs to be signed. Because of this, we decided to extend the signature interface by adding a single function that creates and verifies the signature. This required adding the following functions into the older backends of Soter:

src/soter/libsodium/soter_sign.c

soter_status_t soter_sign(const void* private_key, const size_t private_key_length, const uint8_t* data, const size_t data_length, uint8_t* signature, size_t* signature_length){
  soter_sign_ctx_t ctx;
  soter_status_t res = SOTER_SUCCESS;
  res = soter_sign_init(&ctx, private_key, private_key_length) ;
  if(SOTER_SUCCESS == res){
    res = soter_sign_update(&ctx, data, data_length);
    if(SOTER_SUCCESS == res){
      res = soter_sign_final(&ctx, signature, signature_length);
    }
  }
  soter_sign_cleanup(&ctx);
  return res;
}

Three empty stage functions are also needed to be added to the LibSodium backend:

src/soter/libsodium/soter_sign.c

soter_status_t soter_sign_update(soter_sign_ctx_t* ctx, const void* data, const size_t data_length){
  if(!ctx || !data || !data_length){
    return SOTER_INVALID_PARAMETER;
  }
  if(0!=crypto_sign_update(&(ctx->state), data, data_length)){
    return SOTER_FAIL;
  }
  return SOTER_SUCCESS;
}

soter_status_t soter_sign_final(soter_sign_ctx_t* ctx, void* signature, size_t* signature_length){
if(!ctx || !signature_length){
return SOTER_INVALID_PARAMETER;
}
if(!signature || (*signature_length)<crypto_sign_BYTES){
(*signature_length)=crypto_sign_BYTES;
return SOTER_BUFFER_TOO_SMALL;
}
if(0!=crypto_sign_final_create(&(ctx->state), signature, (long long unsigned int*)signature_length, ctx->key.sk.key)){
return SOTER_FAIL;
}
return SOTER_SUCCESS;
}

soter_status_t soter_sign_cleanup(soter_sign_ctx_t* ctx){
return SOTER_SUCCESS;
}

soter_sign_ctx_t* soter_sign_create(const void* private_key, const size_t private_key_length){
if(!private_key || !private_key_length){
return NULL;
}
soter_sign_ctx_t* ctx=calloc(sizeof(soter_sign_ctx_t), 1);
assert(ctx);
if(SOTER_SUCCESS!=soter_sign_init(ctx, private_key, private_key_length)){
free(ctx);
return NULL;
}
return ctx;
}

soter_status_t soter_sign_destroy(soter_sign_ctx_t* ctx){
if(!ctx){
return SOTER_INVALID_PARAMETER;
}
soter_sign_cleanup(ctx);
free(ctx);
return SOTER_SUCCESS;
}

This finishes the overview of the principal differences in work and the corresponding changes that need to be implemented. Let’s get to the practical part and test how everything is going to work.

Benchmarks

To run the tests and compare how LibSodium performs as compared to OpenSSL, we used Themis’ Secure Cell, Secure Message, and Secure Session services. The benchmarking process included the following tests:

• For Secure Cell and Secure Message it included: initialisation → encryption → decryption → deletion.

• For Secure Session it included: initialisation → establishing of an encrypted connection → transfer of one message Client-Server-Client → deletion.

The main trend characteristic to all the tests is that before a certain transition point we observe better performance stats when using LibSodium, and after that transition point the OpenSSL-based Themis backend is more powerful. The linear dependence of the test execution time on the length of the data block is also characteristic to the tests/benchmarks.

Secure Cell

Transition point: ~10Kb

The tests have generally confirmed that ChaCha20-Poly1305 is slower than AES-GCM. However, taking into consideration the more optimal implementation of LibSodium example (in our very personal opinion, this is most relevant towards the attempts to maximally eliminate the work with a heap since the blocks are usually allocated on stack), LibSodium is a winner when it comes to small-size data blocks.

Secure Message

Transition point: ~20Kb

Besides symmetric encryption, Secure Message includes one Diffie–Hellman key agreement process and ec25519 in LibSodium is universally recognized to be faster than X9-62-prime256-v1 in OpensSSL. This allows moving the transition point to 20Kb.

Secure Session

Transition point: ~30Kb

Secure Session includes more than one asymmetric operation, which is why the transition point moves towards 30 Kb.

Conclusions

Firstly, implementation of LibSodium backend for Themis allowed transforming the Themis/Soter interfaces into more versatile ones.

The use of LibSodium with Themis allowed utilising such crypto primitives that are more preferred among a certain part of the security community because such crypto primitives were developed independently from the NSA. If wearing a tinfoil hat alongside any other security means can protect you from the eavesdropping Secret Services and malicious aliens, one could say that a tinfoil hat of a thinner calibre is needed with LibSodium.

Moreover, LibSodium/NaCl enforces great security choices by design. Besides, on relatively small data blocks, usage of LibSodium-based backend allows getting a small increase in performance.

What we’ve also learned is that Soter architecture enables us to integrate even completely-alien security primitives with moderate ease.

UPDATE:

Many thanks to Frank Denis for reaching out to us after the publication of this article and pointing out that the multipart signature has been present in LibSodium since March 2017. The experiments described in the article were carried out earlier than that, hence the use of a couple of non-elegant workarounds. Also, the LibSodium interfaces have since been improved and updated.

We would like to emphasize that this post describes an experiment rather than provides step-by-step instructions how to integrate LibSodium into Themis for production use.