How we did usability testing for Themis when releasing the open source library into public.
When we were ready to release Themis, we've gathered a few colleagues and decided to make a test run on unsuspecting developers - how would the library blend into their workflows?
1. Introduction
While usability testing for user-centric applications has it's own distinct techniques, standards and frameworks, this is not so typical for a relatively complex and technical library aimed at developers and spanning multiple languages and platforms.
As we've noted before, Themis started as an internal project, designed to meet our needs for future products. Themis is set to provide secure messaging and cryptography to third party apps in a very convenient and easy to use package, yet providing highest security levels.
The decision to make Themis an open source project brought with it the need to provide not just source code, but also a bunch of high-level language wrappers, documentation, examples, and to test the overall usability of these materials.
Our main concern was to avoid a bias of the form: "if it's good for me, it will be good for everybody like me". In developing the libraries and frameworks for internal use, we focused on logical straightforwardness, API completeness, platform availability, and code quality. When working as part of a close knit team, it is easy to rely on the shared knowledge and assumptions that underpin opinionated designs. In opening up Themis to the wider development community, we wanted to make sure it was both useful and useable.
We decided the best way to test this was to run a live test. What would it take to integrate Themis into a working mobile client / API server application? How much time? How much effort? Would it work? What would the developers think of the experience?
This article is the story of what happened alongside with some conclusions and findings. You may also like to check out this repository that contains the iOS and Python code that was generated.
2. The mission
We invited two developers who, although highly proficient in their own areas of expertise (iOS and Python), had no previous exposure to Themis. We asked them to develop a simple client-server application using a simple API and then to integrate two separate Themis cryptographic services within short span of time.
We were quite confident about the framework itself. The science worked, tests have been passed, everything acted as we expected it to. Now it was the time to understand whether other developers enjoyed using it and what we could do better.
Conditions
Time limited. In real life, many technical decisions are made in time-limited and resource-limited situations. So we gave our developers a very limited time span. They had half an hour to get familiar with the framework, half an hour to discuss an app and a couple of hours to actually implement it.
Separated. Although simplified, we wanted the test to deal with problems the way they occur in real world with the developers working with completely different infrastructures, exchanging API descriptions over e-mail.
Evaluated. We wanted to be able to observe the process from inception to an actual demonstration of the final product by people who do similar tasks in their everyday jobs.
Typical. We wanted the application to involve the typical components used in current development projects. In this case, a Python server with a NoSQL database and a web app framework talking to iOS application with a dependency infrastructure built around CocoaPods.
3. The Process
3.1 Introduction to the Task
After a brief introduction to the library and its features, we outlined the task for developers:
A simple client-server application which allows the mobile client to talk to a server using the Themis "Secure Message" object.
The Server should:
Authenticate users based on public/private keys (The Server trusts remote keys).
Respond to the client with a random pre-defined success message .
Store the data received in a protected database.
Provide message log on demand showing the decrypted message content.
The Client should:
Allow a user to input a message, encrypt it and send it to the server.
Display the responses from the server.
Have means to reset the session, in other words - to generate new keys and start from scratch (in order to emulate a multi-user workflow with a single device).
This solution consisted of two separate cryptographic services based on two Themis objects: Secure Message for client-server communication and Secure Cell for server database storage.
Secure Cell
To provide encrypted database storage, the server had to derive its master storage key from something. We decided to use a command-line supplied password for that.
However, rather than encrypting the data with the password or key derived from it, we generated a random key for data protection and protected this key with a password. The data encryption key is generated and stored with the data itself. The Secure Cell object is used to protect the data encryption key with user-supplied password.
We've used Secure Cell in seal mode:
We used the message timestamp and user name as the user-supplied context.
Secure Message
Secure Message was an obvious and convenient choice to provide secure messaging between client and server. It only requires a peer to have the public key of the peer with which it communicates.
In our simulated environment we considered the scenario when clients already have server's key (it was hardcoded in the source) and server got clients' keys during a simple "registration step" before the actual message exchange:
After client's public key is delivered to the server, the client and server can exchange messages protected by strong encryption with very low overhead. Each message is protected by AES encryption with keys derived from strong elliptic curve key agreement.
3.2 Themis enters two blank environments. Building.
iOS. Using Themis is pretty straightforward on iOS. All you need to do is to add and install the Themis pod.
Simply add this line to Podfile: pod 'themis' and run: pod install from workspace folder.
Note: It's useful to add inhibit_all_warnings! line in the top of the Podfile, because currently the Themis library generates quite a few warnings while compiling.
Python. We took a blank cloud virtual machine with an Ubuntu instance and installed:
MongoDB
LibreSSL / libssl-dev
Python infrastructure
Then we went ahead to install Themis based on our documentation. Though make install may seem a bit archaic, the Python developer completed the installation with no problems.
Building the Python module surfaced the first problem: without pip-wrapped package, local paths required some configuration to point to /usr/lib (or wherever the libraries actually are).
Note: We provided slightly modified version of libthemis / pythemis to run on Mac OS X (at the time of testing the code was not yet available to build the Python wrapper on OS X). For a detailed step-by-step description of the process, please refer to:
https://github.com/cossacklabs/themis-ux-testing/blob/master/server/README.md
3.3 Write draft code
To setup a clean environment for our test, we initially wrote the client and server without cryptography — simple client-server pair talking to each other via a JSON API, with a MongoDB backend. It took the developers less than half an hour to generate the initial code and then another half an hour was dedicated to testing and debugging.
This process included the two developers defining and agreeing the conventions, formats, and behaviours of the API.
After some further testing, the client/server communication was established and we proceeded with the next step - integrating Themis into an already working application.
3.4 Integrating Themis on server
On the server, integrating Themis involved implementing two separate cryptosystems:
Secure Cell (Seal mode) in MongoDB
This is the Python code used for reading/writing from the DB and for displaying the stored messages:
#Read
for x in dbm.find({}):
try:
res += '' + str(x['name'])+ '' + str(x['ts']) + '' + str(x['mess'],'%s%s' % (x['name'],x['ts']))) + ''
except:
pass
res += ''
#Write
dbm.insert_one({'name':name,'ts':tm_now,'mess':message})To add Themis, the necessary changes are minimal:
#Read
for x in dbm.find({}):
try:
res += '' + str(x['name'])+ '' + str(x['ts']) + '' + str(db_crypt.decrypt(x['mess'].decode('base64'),'%s%s' % (x['name'],x['ts']))) + ''
except:
pass
res += ''
#Write
dem = db_crypt.encrypt(message,'%s%s' % (name,tm_now))
dbm.insert_one({'name':name,'ts':tm_now,'mess':dem.encode('base64')})As mentioned above, the access key for the database record was stored separately (a random key) and it was protected by a key derived from master password.
Secure Message (encrypt mode)
Adding encryption to outbound messages only needed the creation of the Secure Message object (encrypter) and the call to wrap method:
#Encrypt
encrypter = smessage.smessage(server_priv, main_dict_user[name]);
res = ''.join(random.choice(sccs) for i in range(1))
res = encrypter.wrap(res)Decryption of inbound messages was equally simple:
#Decrypt
encrypter = smessage.smessage(server_priv, main_dict_user[name]);
message = encrypter.unwrap(mess);Visual debugging
Each time a message is received from the client, we output a console log:
# server python serv.py -m password123
uaXq ----> UEC2-?=w:+???[?|??tJK???_H????}Y)?q-?
127.0.0.1 - - [2015-05-27 22:28:29] "POST / HTTP/1.1" 200 200 0.013288
uaXq ----> '&I@??Z+⽧!֟??p]??O?V??~j???yb??b???>?9??@"
127.0.0.1 - - [2015-05-27 22:28:29] "POST / HTTP/1.1" 200 206 0.165188
uaXq ----> '&F@?9!"_o?0??9?y???K????????Ò³7??=j????}???G?The data is (of course) encrypted, but it is sufficient for debugging any connectivity issues.
The server also provides a page which deciphers the data from the database and displays it in decrypted form, with a more detailed log.
3.5 Integrating Themis on mobile
For this test, the client application already has the server's public key (hardcoded). However, the client needs a public/private key pair.
Generating keys
TSKeyGen * keygenEC = [[TSKeyGen alloc] initWithAlgorithm:TSKeyGenAsymmetricAlgorithmEC];
if (!keygenEC) {
NSLog(@"%s Error occured while initializing object keygenEC", sel_getName(_cmd));
return;
}
self.privateKey = keygenEC.privateKey;
self.publicKey = keygenEC.publicKey;Initialising the Session
The session is initialised by sending the client's public key to server:
{ "username" : "encrypted message" }And the keys are generated as NSData, we convert them to base64 strings before sending the message.
NSData * base64 = [self.publicKey base64EncodedDataWithOptions:NSDataBase64Encoding64CharacterLineLength];
NSString * stringKey = [[NSString alloc] initWithData:base64 encoding:NSUTF8StringEncoding];Once the server has received the client's public key as the content of the first message, the session is initialised and user messages can now be sent.
Create the Secure Message encrypter/decrypter
The SMessage object is used for message encryption/decryption. The SMessage object should be initialised using client's private key and server's public key.
self.messageEncrypter = [[TSMessage alloc] initInEncryptModeWithPrivateKey:self.privateKey peerPublicKey:self.serverPublicKey];Encrypt each message before sending
Notice the use of base64 encoding and the NSData/NSString conversion. resultString is an encrypted message and will be sent as part of the JSON packet described above.
NSError * error;
NSData * encryptedMessage = [self.messageEncrypter wrapData:[message dataUsingEncoding:NSUTF8StringEncoding]
error:&error];
if (error) {
NSLog(@"ERROR in encrypting message %@", error);
return nil;
}
NSData * base64 = [encryptedMessage base64EncodedDataWithOptions:NSDataBase64Encoding64CharacterLineLength];
NSString * resultString = [[NSString alloc] initWithData:base64 encoding:NSUTF8StringEncoding];Decrypting Inbound responses
Our API specifies that server responses are JSON of the form:
{"answer" : "encrypted response" }Again we must handle the base64 decoding:
NSDictionary * dictionary = (NSDictionary *)responseObject;
NSString * responseBase64 = dictionary[@"answer"];
NSData * base64Data = [[NSData alloc] initWithBase64EncodedString:responseBase64 options:NSDataBase64DecodingIgnoreUnknownCharacters];
if (!base64Data) {
return nil;
}Before using SMessage to decrypt the message and produce a NSString readable by a human:
NSError * error;
NSData * decryptedMessage = [self.messageEncrypter unwrapData:base64Data error:&error];
if (error) {
NSLog(@"ERROR in decrypting message %@", error);
return nil;
}
NSString * resultString = [[NSString alloc] initWithData:decryptedMessage encoding:NSUTF8StringEncoding];The client request is now complete and the application can send the next message.
Clean Up
To reset the session and clean up the Themis resources, a cleanup function is provided:
- (void)cleanup {
self.privateKey = nil;
self.publicKey = nil;
self.messageEncrypter = nil;
[self saveKeys];
[self regenerateName];
}User Interface
The client application provides a minimal (aka ugly) user interface that provides the abilities to create and send messages, view the message log, and initialise the communication session.
NOTE: If you are interested in building an iOS application, please follow our guide
https://github.com/cossacklabs/themis-ux-testing/blob/master/ios-project/README.md
4. Findings and Outcomes
We found the observation of this process to be extremely useful. In addition to the points below, we found a range of smaller issues where clarity of undestanding and ease of use could be improved. The solutions of these issues are largely complete and indeed reflected in this article.
Serialization and strings
Probably the most unexpected (but unsurprising) result was the fact that the crypto developers think in terms of binary blocks, while application developers use strings. Base64 encoding and decoding is a common, familiar, and readily available solution, so (at least for the time being) the library interfaces will continue to operate with binary data.
If you think Themis would benefit from more integrated serialisation let us know via issues@GitHub.
Not all (Open/Libre)SSLs are built equal
A classic Gotcha : Dependencies. We found out that the default versions of LibreSSL and OpenSSL's libcrypto shipped by package control systems on Linux and on iOS had incompatible implementations of an underlying cipher used by Themis. The result was an obvious and frustrating silence.
Once the libraries were manually updated to the latest builds on both client and server ... all was well.
Explicit Requirements
Perhaps understandably - given the very brief outline of the task and extremely short time allowed to design the client/server protocol - initially the server responses (success/fail) were conveyed simply as HTTP response codes. This of course omitted exercising the server's ability to encrypt responses and the client's ability to decrypt them. The design iteration for incorporating random server responses resolved this.
Key Management is Crucial
In observing and discussing the Python developer's approach to Key Management and storage, we once again noticed the gap between crypto and application developers. It was clear that the need for careful management, storage, and use of critical "secrets" was not sufficiently made clear in our documentation. We have sought to address that here:
https://github.com/cossacklabs/themis/wiki/2.3-Key-management
Example Code
The low level examples and snippets of code we provided were next to useless for the developers. We have and will continue to improve this area of the product.
5. Conclusions
Don't Underestimate the Effort. Making a perfectly serviceable internal project into a public resource requires considerable effort ... at least is you want to make it easy and even enjoyable for people to use.
Do Test in Different Contexts. The needs, priorities, and assumptions of developers working in different environments vary much more than you might imagine. We found our experiment with the two developers from completely different ecosystems a very efficient way to expose and understand the connectivity and compatibility issues a network-friendly library should handle.
Checks and Tests. When things just don't work, one of the benefits of open source is that you can dive into the source code to try and work out whats gone wrong. But that's a decision not everyone dares to make. Frequently, developers work around making a set of assumptions on how the code should work, and keep adjusting them until they finally meet a reality. For cryptography, this is even truer since there are no 'maybes' in encrypting and decrypting. One of the main outcomes of this process is that we've identified a number of additional features and tools for checkinng and diagnosing implementation issues and we'll roll these out in upcoming versions of Themis.
It's More Fun than It Sounds. Having people live-test your product and even heavily criticise some aspects of it is a great way to improve it. We found more usability issues and solutions in 6 hours than might have emerged over several weeks internal review. And of course, with adding only a few lines of code - it all worked.
6. Links
Thanks for reading this, You might also want to take a look at:
the product itself: themis@github. We've just released 0.9.1 version, fixing a lot of inconvenience for iOS, Android and Python versions.
repository with examples from this post
P.S. If you're looking for new ideas, this is the right place. If you're looking to implement security, apply for our Customer Success Program. If you're looking for ready-made solutions, consider also looking into Acra, Hermes, or Toughbase.
