How to build OpenSSL for Carthage iOS

This story is dedicated to fellow developers struggling with updating Carthage package with the latest OpenSSL for iOS and macOS apps. Here you will find the scripts, error messages, testing matrix, and our working solution for Themis to this no small feat. We believe it could save you time then you meet the same task.

Fire in the hole! 🔥 #

Imagine your builds going red because of an outdated OpenSSL that is used by one of your Carthage dependencies. The build went red because outdated OpenSSL is potentially vulnerable. You found out that your dependency doesn’t even nearly call potentially vulnerable OpenSSL code, but who cares—now it’s your problem to update dependencies and make your builds green.

Updating Carthage dependency of your Carthage dependency—should be easy, right? In the iOS dev world, nothing is easy :)

This story describes how we took the latest version of OpenSSL library and successfully built it as a Carthage package. Which was successfully used in a Carthage package of our cryptographic library Themis. Which was used in a security-conscious iOS app that was released to the App Store and made its users happy (and protected).

But let’s start from the beginning. Follow the whole engineers’ journey to the iOS development hell or jump to the working solution and enjoy.

1. Backstory. Themis, OpenSSL, BoringSSL, and other SSLs
2. Packaging OpenSSL for Carthage
   • Nested dynamic framework
   • Standalone dynamic framework and Carthage binary projects
   • Standalone static framework and Carthage binary projects ⇐ this worked
3. Testing matrix
4. What we learnt and next steps

Backstory. Themis, OpenSSL, BoringSSL, and other SSLs #

Developers use our open-source cryptographic library Themis because it brings easy-to-use cryptography into their apps that work consistently across a dozen languages and platforms.

Themis is easy to install and easy to use, although this ease for developers comes at the cost of significant maintenance effort, both for the community and us.

Friends don’t let friends roll their own crypto.

Themis library doesn’t reimplement crypto primitives too.

Instead, it relies on existing, well-studied, mature sources of cryptographic primitives—cryptographic engines inside OpenSSL, LibreSSL, and BoringSSL. Themis uses crypto-engine that is the most common on a target platform.

For instance, Themis for Android relies on BoringSSL, Themis on iOS uses OpenSSL by default but allows switching to BoringSSL. Sometimes switching crypto-engine gives a nice performance boost.

The cryptographic engine is one and only dependency of Themis. It is a good idea to keep your dependencies up-to-date, especially if they are critical for security.

Though, most vulnerabilities in the cryptographic engines don’t affect Themis-backed apps because Themis uses a minimal and specific subset of primitives: AES GCM/CTR, RSA, ECDSA, ECDH, SHA-2, PBKDF2. It doesn’t use old ciphers or TLS-related functions where the vulnerabilities are rather common nowadays.

Updating OpenSSL for Themis on iOS #

However, you never know when a new zero-day vulnerability surfaces, so updating the cryptographic engine is really important security-wise.

OpenSSL is widely used and has a history of discovered and fixed vulnerabilities, so it’s important to keep it updated.

Themis currently supports 14 languages, 7 platforms, multiple operating systems and distributions.

Each combination is a unique environment that has its own default version of OpenSSL as well as a release and update schedule.

On some platforms, like Linux, OpenSSL is a system library so developers can update it independently of updating Themis.

But on Apple platforms, like iOS, OpenSSL is not available as a system library. Here it’s up to us to ship Themis with the latest OpenSSL so that app developers don’t have to deal with this complexity. 

As security library vendors, we keep an eye on all supported platforms to make sure they are up-to-date and follow security advisories issued by the OpenSSL, LibreSSL, and BoringSSL teams.

A sad poem about OpenSSL and Apple #

Once upon a time, Apple maintained an OpenSSL distribution for their systems.

Nowadays, Apple uses and promotes its cryptography libraries. While OpenSSL is still available on some systems, it’s an unsupported and woefully outdated legacy version of OpenSSL.

Unfortunately, Apple crypto libraries don’t provide modern cryptographic cyphers in a form that could be easily used by other libraries. For example, CommonCrypto doesn’t provide AES GCM as a public API, CryptoKit is available only on the latest systems and provides exclusively Swift API. That’s to say nothing about other functions that Themis needs.

CommonCrypto and CryptoKit ultimately use Apple’s corecrypto library. While its source code is available, Apple considers this library a private implementation detail. Authors of this blog post are not sure whether it’s possible (and reasonable) to use corecrypto as a crypto-backend for Themis and depend on it directly.

While it is possible to migrate Themis to CryptoKit instead of OpenSSL, most Themis features will be missing. ThemisCryptoKit will be compatible only with ThemisCryptoKit and will work only for the latest Apple devices talking to each other. 

That’s opposite of Themis goals as a library and us as maintainers.

We believe in strong, compatible cryptography across modern app platforms.

Community supports OpenSSL for iOS and macOS #

The open-source community stood in to package OpenSSL for Apple platforms.

It sounds like an easy job to take OpenSSL source code, build it for ARM64, ARMv7, x86_64, and other architectures, and then wrap it into a nice package. However, it turns out to be easier said than done, with multiple tricky issues on the way.

Many individual maintainers apply their best effort, but this approach doesn’t seem to be sustainable in the long run, and there is no common offering:

• FredericJacobs/OpenSSL-Pod
• thejeff77/OpenSSL-Pod
• krzyzanowskim/OpenSSL
• jcavar/OpenSSL
• levigroker/GRKOpenSSLFramework
• x2on/OpenSSL-for-iPhone
• keeshux/openssl-apple
• sinofool/build-openssl-ios
• cossacklabs/openssl-apple ⇐ now we have our own OpenSSL 1.1.1g for iOS and macOS that supports CocoaPods and Carthage

Typically, developers install Themis with a package manager such as CocoaPods or Carthage. It’s very convenient for developers because they don’t have to install the crypto-engine themselves, and it’s easy to keep the library updated.

So, one day we were notified about a recently found and fixed vulnerability in OpenSSL. It should not have affected Themis directly, but as we said before, applications should use the latest version of OpenSSL regardless. We rushed to update our packages, but we were in for some bad news…

Themis is useful in various use cases. Read how we secured notes in Bear iOS and macOS apps using Themis:

Bear

Consumer app

End-to-end encryption and multi-device synchronisation for 6M users

Encrypting data for Bear — the Apple Design Award-winning application while focusing on performance and usability.

Read story

Packaging OpenSSL for Carthage #

The news is: the maintainer of the OpenSSL-for-Apple we were using has been unsuccessfully battling some packaging issues, and at the moment they are not able to supply a new package that works on Apple platforms, let alone an updated one. Unfortunately, for quite some time that was the only reusable package compatible with Carthage that had an up-to-date version of OpenSSL.

As the wisdom goes, if you decide to depend on an open-source library, be ready for the worst: maintain it yourself or replace it otherwise.

We had no choice but to step in ourselves and take a shot at packaging OpenSSL.

Update: good news! Latest Themis 0.13.2 supports OpenSSL 1.1.1g and works like a charm for iOS and macOS, Carthage, and CocoaPods!

Requirements #

We want OpenSSL integration with Themis to be as seamless as possible. We need to include support for all relevant architectures and SDKs, Bitcode, debugging symbols, etc. The developers should be able to just write

github "cossacklabs/themis" ~> 0.13.2

in their Cartfile and that’s it.

The application should run from Xcode in debug and release mode, run on a device, and it should be possible to submit the app to the App Store without any troubles. We’re not asking for much.

3 packaging ways we tried #

Carthage is centred around using Xcode to build dependencies. That is, the dependency has to provide an Xcode project to build it from the source. As the OpenSSL team uses their own portable build system instead of Xcode projects, OpenSSL cannot be directly used with Carthage.

The OpenSSL package we’ve been using so far employed the following workaround: check-in prebuilt binaries of OpenSSL along with its source code into Git repository, then have a minimal Xcode project to build a framework out of the binary libraries and headers.

We have explored several alternatives to that approach:

1. Build OpenSSL as a part of Themis, embed it as a nested dynamic framework.
2. Build OpenSSL separately as a dynamic framework, link it to Themis as Carthage binary project.
3. Build OpenSSL separately as a static framework, link it to Themis as Carthage binary project.

1. Nested dynamic framework #

At first, we decided to hide the complexity of building OpenSSL directly into Themis Carthage package.

1. We added OpenSSL as a Git submodule (similar to BoringSSL).
2. Then we added a custom script target in our existing Xcode project for Themis to build openssl.framework with a dynamically linked binary inside.
3. Finally, we embedded openssl.framework into themis.framework.

When developers add Themis via Carthage, Carthage downloads OpenSSL source code, builds it, and makes it a part of Themis. Developers need to link the themis.framework only into their apps and don’t have to bother with OpenSSL linkage—it’s transparent to them.

What we got: the app works fine on macOS, iOS simulator, and iOS device. However, we got an error during export of iOS app to IPA.

Where nested frameworks failed 😟 #

Apparently, using nested frameworks is a bad idea in the iOS world.

Here are some forum threads (here and there), as well as Apple Technical note that describes reasons why developers should avoid using nested frameworks.

In our case, the following error message popped up during IPA export:

Error Domain=IDEFoundationErrorDomain Code=1 "IPA processing failed" UserInfo={NSLocalizedDescription=IPA processing failed}

Xcode error messages sometimes don’t help at all.

After digging through the IDEDistributionPipeline.log, we’ve found that ipatool complained about multiple copies of OpenSSL.

Shuffling around the frameworks inside the Xcode project (often suggested on the Internet) gave us a stable result: Xcode refused to sign the nested OpenSSL framework when exporting an IPA, so the device refused to run an unsigned binary.

dyld: Library not loaded: @rpath/openssl.framework/openssl
Referenced from: .../themis.framework/themis
Reason: no suitable image found. Did find: Frameworks/themis.framework/Frameworks/openssl.framework/openssl: code signature in (.../themis.framework/Frameworks/openssl.framework/openssl) not valid for use in process using Library Validation: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.

The Internet insisted that trying to build an iOS app with nested frameworks is a bad idea.

This leads us to the next big thing.

2. Standalone dynamic framework and Carthage binary projects #

Carthage dependencies are generally distributed as Xcode projects. We want to avoid recreating the entire OpenSSL build system as an Xcode project since it is not realistic to maintain.

Luckily, there is an alternative. It’s hidden under a tree in the documentation backyard, but it is there.

Carthage supports binary project dependencies. That is just what we need.

1. Build openssl.framework with our trusty build script.

2. Publish the prebuilt OpenSSL binaries somewhere (e.g., GitHub release page).

3. Publish JSON specs for a binary project and use them as Themis dependency.

This approach frees up the user from having to build a massive OpenSSL library from scratch, avoids excessive consumption of bandwidth, and allows us and others to reuse OpenSSL as a Carthage dependency in other projects easily. At the same time, it makes us responsible for building and publishing updated OpenSSL binaries with new releases.

What we got: the app works on macOS, on iOS simulator, on iOS devices. The IPA export is also successful now!

Here is what the codesign tool reports about the app bundle:

$ codesign --verify --verbose My.app

My.app: valid on disk
My.app: satisfies its Designated Requirement

We also checked that OpenSSL binary is the one we expect in the IPA:

$ strings My.app/Frameworks/openssl.framework/openssl | grep 1.0.2u | head -1

OpenSSL 1.0.2u 20 Dec 2019

Success?

But no, wait, the app should be submitted to the App Store or TestFlight for us to be completely sure that everything is really fine.

Where dynamic frameworks failed 😟 #

One does not simply test iOS apps locally. All-green tests and successful code signing do not guarantee that Apple accepts your IPA. We already had a CI that builds the app, exports an IPA, and uploads it to TestFlight.

We have installed the app from TestFlight, launched it… and it immediately crashed, leaving this in the device logs:

Exception Type: EXC_CRASH (SIGABRT) 
Exception Codes: 0x0000000000000000, 0x0000000000000000 
Exception Note: EXC_CORPSE_NOTIFY 
Termination Description: DYLD, dependent dylib 'openssl.dylib' not found for '/private/var/containers/Bundle/Application/', tried but didn`t find: 'openssl.dylib'
Highlighted by Thread: 0 
Backtrace not available

We unpacked the IPA bundle and confirmed that openssl.framework is in place. However, there are no traces of openssl.dylib which is required by themis.framework, as otool helpfully suggests:

$ otool -L <app>.app/Frameworks/themis.framework/themis| grep openssl
	openssl.dylib (compatibility version 1.0.0, current version 1.0.0)

But here is how it looks when the app is built locally to run on the device.

This is how the dependency should look:

otool -L <app>.app/Frameworks/themis.framework/themis | grep openssl
	@rpath/openssl.framework/openssl (compatibility version 1.0.0, current version 1.0.0)

Obviously, the app does not crash when it uses the correct search path.

It crashes when it tries to search for a nonexisting OpenSSL dylib. Xcode produces a working app with a proper @rpath when building for the device, but for some reason, it mangles the path when exporting an IPA. ¯\_(ツ)_/¯

Why does this happen? #

Even Apple doesn’t seem to know a specific reason.

Some people on the Internet have seen similar behaviour (here and here). It seems to be somehow related to Bitcode. We were not lucky to find the root cause or a suitable workaround and decided to move on.

3. Standalone static framework and Carthage binary projects #

Given that dynamic linkage does not seem to work well, we decided to try static linkage.

Indeed, Apple recommends using static frameworks to improve app startup time unless you need to share code. Many libraries don’t typically need OpenSSL and Themis uses only a subset of its functions, so we could expect some good savings in storage as well.

The process is the same as before with dynamic Carthage binary projects, but this time the build script is instructed to produce a statically linked OpenSSL library instead of a dylib. We still need to link themis.framework against openssl.framework, but the app developers using Themis do not have to embed openssl.framework anymore. It’s all handled by Carthage now.

You can check out our work in the cossacklabs/openssl-apple fork and the master branch in Themis repo.

Or just add this to your Cartfile:

github "cossacklabs/themis" ~> 0.13.2

and run carthage update.

Do not export OpenSSL symbols #

There is a side effect to static linkage: by default, Themis binary are exporting OpenSSL symbols. That is, other libraries could be linked against Themis as if it provided OpenSSL too.

We do not want to export symbols for multiple reasons:

● It is hard to debug possible linker errors due to symbol name conflicts.

● Themis uses only a small subset of OpenSSL functions.

● We build OpenSSL with many insecure algorithms disabled and removed.

Themis is not intended to be and cannot stand-in for a generic OpenSSL library. We need to avoid exporting OpenSSL symbols if we are using static linkage.

By default, all non-static functions are exported. This can be controlled with symbols file settings in Xcode:

Whitelist and blacklist settings for exported symbols.

Themis on Apple platforms provides Objective-C and Swift interface. We only need to export Objective-C classes.

Here is how our exported.symbols file looks:

# Export only Objective-C classes in TS (Themis) namespace
_OBJC_CLASS_$_TS*
_OBJC_METACLASS_$_TS*

You can read more about this linker feature on the manual page for the -exported_symbols_list flag of ld(1) or check Dynamic Library Design Guidelines.

What we got: the app works on a simulator and all devices. IPA export is successful, upload to TestFlight is successful, the app can be installed and run on the device great as well. Hooray! 🎉

The resulting approach had some benefits.

1. The app complies with both Apple and Carthage semver requirements.

Carthage downloads openssl.framework with canonical OpenSSL version in its Info.plist (1.0.2u).

OpenSSL is used by Carthage to build Themis, but it’s not included in the app directly. OpenSSL becomes a part of themis.framework without being embedded. Thus, Apple sees only (semver-compliant) Themis version and is happy with it.

As library maintainers, we are happy as well to not cause any unnecessary trouble to our users.

OpenSSL versioning and semver is a topic for another post.

2. The app bundle becomes smaller.

The resulting app is smaller than before.

This is the result of static linkage.

Previously, the app had to bundle and load the entire OpenSSL framework dynamically. Now, Xcode links OpenSSL statically. Linker cuts all OpenSSL code that is not used by Themis, and there is one less library to load, which makes the app launch a bit faster.

3. The app is less prone to OpenSSL vulnerabilities now.

The app now contains only OpenSSL code actually used by Themis, not the entire OpenSSL library. This makes the app more secure compared to its previous version as the resulting binary physically doesn’t contain some OpenSSL code.

Perfection is achieved when there is nothing left to take away.

It means that some OpenSSL CVEs simply won’t apply to the app now—it just doesn’t have the code. You can’t exploit the code that doesn’t run.

As Themis uses only a small subset of OpenSSL functions, that’s ~1.6 MB of potential vulnerabilities gone!

Static linkage optimization will be automatically applied to any later OpenSSL version we release. App developers don’t have to change anything in the build process and can just enjoy the results.

Testing matrix #

Just to give you a sense of scale, this is a matrix of our trials and their results.

Statically linked frameworks approach gave the best result.

Keep in mind that iOS apps usually support multiple devices and iOS versions. Thus, we had to test device installation steps on several device families running different iOS versions.

Also, it’s not enough to just install the app and see it does not crash on launch. We run functionality tests as well, making sure that cryptographic-related functions are working as intended and we don’t see “OpenSSL not found” error when the app actually needs to encrypt something.

What we learnt and next steps #

This post shows how deep the rabbit hole can go when you decide to do a “quick minor version bump of a library” if you’re dealing with special libraries in the special Apple ecosystem.

Security is a complicated world, and cryptography is even more complicated. As vendors of security software products, we wade into those waters to provide effective solutions to overcome a bunch of hidden niceties. This is a choice and a way, so we are right up for challenging tasks.

If you’re struggling with OpenSSL—remember, you are not alone. :)

If you’re struggling with app security, you’re not alone either, feel free to drop us a line to get some assistance.

🧡 Also, our special thanks go to the community: @iRonald08, @keeshux, @x2on, @krzyzanowskim, and others. 🧡