Usability Report: Proposed Mailpile Features
Activity: User testing and contextual interviews on an encrypted, browser-based desktop email client.
Takeaway: Users do not think about whether a message is being encrypted unless prompted. But they understand that encrypted communication requires dedicated apps.
Trainers who teach activists and journalists how to encrypt and secure their communications have identified desktop email clients as a pain point. The leading open-source client, Thunderbird, is no longer being actively developed by Mozilla. Enigmail, the plugin which supports encryption on Thunderbird, is finicky and hard to use. In the browser, Mailvelope aims to help encrypt using services like Gmail, but it cannot offer the protection of removing messages from a server which could be attacked or compromised.
Fortunately, a handful of open-source secure email solutions are currently under development, including Whiteout, Bitmask, and Mailpile. Of these, Mailpile is one of the most promising and furthest along in development. Mailpile has a browser-based client with a sleek modern design; at a quick glance, people have confused it with Gmail.
The Mailpile team asked for the assistance of OpenITP’s Secure User Practice (SUP) project in user testing of a few auto-discovery features they want to add to help users find their contacts’ encryption keys.
Over two days in November 2014, Gillian “Gus” Andrews ran hands-on, in-person tests with five users. As in all of SUP’s user tests, she spent some time clarifying to users the basic mechanisms of security, to help ensure users did not walk away with half-formed understandings of the protections the tool does and does not offer.
The results follow, including top-level takeaways for developers, two profiles of users whose use cases are particularly relevant to teams working to protect free speech and human rights, and a more detailed analysis of findings.
TOP-LEVEL TAKEAWAYS
Test findings:
❖ Users do not appear to think about whether a message is being encrypted unless you prompt them to. ◇ This needs more investigation.
❖ When they identify that they don’t have a key for a contact, they are finding Mailpile’s proposed “Find encryption keys” prompt easily. ◇ Messaging about finding keys is clear, and users are paying attention (at least in the test).
◇ Users found their way to the “find keys” dialog both through the To: field and through the contacts screen. ◇ The warning messages in yellow-orange bars are being noticed and read.
❖ The “Attach key” checkbox at the bottom of the compose window is a bad distractor when users are trying to figure out how to encrypt.
❖ Users understand that some kinds of communication, like encrypted email, require dedicated apps. ◇ Users interpret the text signature Mailpile adds as an indicator that the tool does something special.
❖ Gmail users appear slow to find the Compose button — but they may well get used to that with time.
Bugs found in the test:
❖ Mailpile does not currently check to see if text is an email address ◇ Subsequent weird behavior from the client
❖ When one user (Jo) clicked “Find encryption keys,” search appeared to cycle through twice ◇ It said it had not found the keys, then searched again
❖ The key lookup feature couldn’t find the key of one of the Mailpile team members! ◇ Contact search did not register his name after user typed the first two letters into the search bar
❖ Mouseover of the orange lock in the compose field tab should perhaps indicate which key is missing
❖ A copy of each message is being sent to the user’s inbox apparently ◇ Instead of “sent” folder ◇ Dunno if this is intentional?
Bugs from daily use:
❖ “Message and metadata will not be encrypted” message when you don’t have a key needs to mention if lack of a key is the problem.
❖ Count on the outbox does not correctly register # of messages within. ◇ One of the developers says it registers the number of unread messages ◇ Having some kind of count here is actually useful — it lets the user know if messages weren’t sent
Suggestions:
❖ “Attach (user’s) key” should be put a level or two deeper in the interface ◇ Maybe in an “attach” dialog in the compose window
❖ Maybe remove the yellow-orange color from non-warning elements of the interface? (i.e. field borders)
❖ Give a way to add key manually ◇ Fejleh’s guess that you could paste a fingerprint into the To: field was an intriguing idea! ◇ Ideally put this option a few levels deep in the interface; it’s not something most users should stumble upon. ◇ Developer Brennan suggested it should go at the top of the user card (in the location where the breadcrumb trail would be?)
PARTICIPANTS
SUP ran user tests on Mailpile with five people, all of them in their 20s and 30s, over two days in November. (In this report we use pseudonyms where needed.)
Two of the participants, Fejleh and Mamun, are journalists who cover regions of the Middle East which are in deep turmoil. Maleksa is a Belgian international NGO worker. Baraba is a college-age student who grew up in the South Bronx. Jo is a UX specialist, who works in a university on educational technology.
Profile: Fejleh, journalist
Fejleh was more tech-savvy, in some ways, than she let on. She used command keys extensively to navigate, and even though she was not a Mac user she quickly adapted to using Mac command keys, even memorizing a diacritic shortcut on the fly.
She said that managing her security and privacy online matter a lot to her. She downplayed the sophistication of her usual security practices, though, saying the best she thought she could do was deactivating her Facebook account, which she does from time to time. “You can’t do that forever, though,” she said. She believes that journalists need to follow people on different social media platforms. When asked what she meant by “deactivating” Facebook, she clarified that she meant suspending her account, not deleting it. “I say that it’s going to be forever, but after a while you open it again,” she said. “It’s creepy [that] they can keep it.” She had not been aware it was possible to fully delete her account, so Gus showed her the tortuous and hard-to-find path to doing that (as well as saving her content before she did).
She later revealed she also manages her passwords carefully. “I have a different password for Gmail and Facebook than the other passwords that I use…. I don’t believe it’s so much useful, but people can’t guess it. My password’s so complicated I sometimes forget it.”
Fejleh discussed other social media platforms as well. She said she did not like to use Instagram, because it is a Facebook product and “not safe,” but later decided to keep posting photos of “different things,” never posting photos of herself. “I believe that Twitter is safe, I’m not sure to what extent, but that’s why I’m active on it,” she said. She clarified that she thought Twitter was safe simply because she did not share as much information there — and she knew where that information was going to go. “Just from posting these tweets, you know that it’s going to go viral, it’s going to go everywhere. It’s not that you’re sharing something and you’re scared somebody would look at it, like Facebook inbox.” She agreed that Twitter seemed safer because you know where you stand with it.
She had been ok with Skype because “you feel that it actually saves your conversations on the platform you’re using — I mean on the machine — but later I started feeling that no, anybody can actually access your camera and whatever through Skype.” She had heard about Skype’s insecurity from other people.
Fejleh is also bothered by ads tracking what she and her friends search for and buy. One of her friends found that Facebook knew she was pregnant before she did. Fejleh herself started getting ads for universities when she looked for a Master’s program.
She used to use MSN/Hotmail and Yahoo, but she forgot her Yahoo password and didn’t use Hotmail much. Someone introduced her to Gmail, which she now uses.
Fejleh had a very interesting response when asked to look at the Mailpile inbox before poking at it, and tell me what she thought it could do. “Sounds to me like more of an offline [tool],” she mused. When asked why, she pointed to the “power” switch (the logout button) in the upper right. She thought “that you can just put it down whenever you want. But I’m not sure.” She thought this was good: “The problem starts with you having a network… if you have a network there’s somebody who’s managing it.”
When the researcher confirmed that Mailpile hosted mail locally, Fejleh volunteered, “It’s like Outlook.” The researcher explained she could use a provider other than one of the major ones. “Can I have an email without somebody to host it for me, like if I’m not working at a company?” she asked. The researcher said yes, and told her about a couple of providers OpenITP trusts. Even as the researcher explained the ways this setup could make Fejleh’s communications safer, she was still skeptical; she thought through the threat model and identified that there was no defense against physical threats like kidnapping and torture. While she was generally calm throughout the test, she was visibly distressed talking about this.
Profile: Mamun, journalist
Mamun shares the intensity of others who report on war in their homelands: all of them are keenly aware of the stakes of their work, having lost friends to violence and had their lives upended as they fled. Mamun was very eager to identify tools to help others he was working with, while also insistent that he needed tools he had not seen yet.
Mamun said he “used to use a lot of programs like” Mailpile when he was in Syria, but does not use them now, in the United States. He used Tor, VPNs, proxies, and Hotspot Shield. He says he used to use two programs at the same time, for example Hotspot Shield and Tor. “We faced a lot of problems with Tor,” he said. “It’s a very slow program. it’s the most secure one, but it’s very slow. I was not fond of it.” While he was aware of these tools, he did not say much about how he thought they worked. Also, he was not specifically familiar with public-key encryption. Interestingly, unlike Fejleh and the other testees, he did not mention password management or profile management on social networks when discussing how he managed his privacy and security.
Mamun has a network of trusted, expert friends who he relies on to recommend software for security. He said that before using Mailpile, he would look it up on Google, as well as running it by these contacts. “If they told me yes, I would start using it. If they told me no, I would stop.”
Mamun gave a number of useful details about the technology terrain in Syria. He noted there are no credit cards, or online shopping. The internet connections are slow even without using Tor. Mamun asked if Mailpile would work well when connections were slow. The researcher told him it would likely compare favorably to Gmail, given that the browser work is local and does not need to contact a server with every click. He seemed satisfied by this, but said that in a country like Syria where VPNs are popular, marketing work would need to be done to explain to users why they would need to use Mailpile in addition to using a VPN. He thought perhaps an “online course” would be useful to advertise it, “because it’s complicated.”
He suggested the biggest hurdle that Mailpile, in particular, would have to overcome in Syria is that email use is not common there, not even among journalists, because “there is no professional company there.” People may have an address, but not check it often. The preferred methods of communication are voice calls, or meeting face to face.
Mamun’s needs for additional tools overlapped with a few of the tools in the secure FLOSS tools space. “I need to communicate when there is no Internet, no phone,” he said. The researcher recommended he look into Commotion, Serval, and Project Byzantium. He also expressed a strong need for a unified chat and SMS tool, for mobile specifically — “not ANOTHER tool, but one where you can read them all at once.” The researcher suggested he check out Cryptocat and Jitsi. He was pleased with how simple Cryptocat looked; he thought his peers would find it usable.
TEST PROTOCOL
The test began by asking users SUP’s standard questions about their security and privacy habits, for the sake of comparison to previous user tests. The researcher also asked about their assumptions about secure email tools, and about Mailpile (from looking at it, before they poked at it).
Users were set up with the researcher’s personal install of Mailpile, which was pre-populated with her contacts, their keys, and her key to encrypt with. This setup was agreed upon with the developers, as they wanted users not to have to go through setup. This arrangement ensured users spent more time focused on working with the new key auto-find feature, not distracted by other setup questions. None of the users interacted with the user’s own key options, nor were they prompted to.
Testees were directed to do one basic task: send a single encrypted email to two Mailpile team members, Brennan and Smarí. Brennan’s key was already in the contacts, Smarí’s was not. The researcher stopped them before they sent the message, and asked how they would know if it was encrypted. This resulted in the additional tasks of identifying whether they had keys to encrypt, and finding Smarí’s key.
Following the first, pilot test with Baraba, the researcher adjusted the protocol, identifying that it was not clear whether the developers wanted the users to search for keys (they did). Baraba subsequently served as research assistant, taking backup notes. (The researcher mentors Baraba in other contexts, and she was interested in learning more. Baraba was primed more than other users by a conversation she had with the researcher about encryption before the tests.)
The full protocol for the user test appears at the end of this post.
RESULTS
What do respondents expect of a secure mail client? What do they expect when they see Mailpile’s inbox for the first time?
“Nobody would have access on it, which is very important… Anybody, and specifically from the governmental side…. Being also an activist, it’s not safe if someone’s checking what you’re writing…. It’s creepy to know they can read your writing…. That’s why I have a different password for Gmail and Facebook than the other passwords that I use.” — Fejleh
“Externals [won’t] have access to my email — meaning the people not on my contact list…. It could protect against spam…. I think it has a social component [indicates the “contacts” icon at the top right]” — Maleksa
“It looks like Gmail, so I would expect it would have all the functionality of at least of Gmail, not Google Docs. Send, receive, have folders, label, filter.” — Jo
When the researcher explained that Mailpile worked more like Outlook or Thunderbird, the testees who were not students seemed to get it. Baraba looked blank; but then, she hasn’t spent much time in office settings and has not been exposed to Outlook.
Task completion
All five users successfully added Brennan’s email address to the “To:” field. Because Gus’s key list already included Brennan’s key, the message encrypted by default.
Four of five users did NOT successfully add Smari’s key. (The fifth, Baraba, was not instructed to mail to Smari.) Ultimately, none of the users managed to complete an encrypted send with a key which was not already in the database.
The researcher was inconsistent in prompting users to find Smarí’s contact info or key outside of Mailpile when they did not find the key in the contacts. Mamun and Maleksa did not venture outside of Mailpile to find keys; the reseacher prompted Fejleh and Jo to do so, though.
Fejleh found Smarí’s personal contact page, correctly identified that the fingerprints there were associated with his keys, and looked at the keys themselves. She did not figure out how to import them this way, though. She tried to highlight and copy the body of the key (critically, not including the header), and she gave up on this when she saw how long it was, exclaiming “Wow, no!” Fejleh also tried copying the fingerprint and pasting it in the To: field — a clever guess, which sadly did not work for her. She then copied and used Smarí’s Mailpile email address to register the key in Mailpile. Sadly, no key was apparently associated with this address, and she could not find it.
Jo also found Smarí’s personal contact page after a Google search and a bit of prompting (due to time constraints). She also tried to find a key in Mailpile using his Mailpile address — no results — and then tried his IMMI address on prompting from me, as the researcher had learned there was definitely a key associated with this address. Mailpile also failed to find a key for Smarí’s IMMI address. This suggests the keyserver retrieval function was not yet working in the Mailpile build the developers provided for this test.
Users are slow to notice lock icons
Gus stopped the users before they sent their messages, and asked them “How would you know if your message was being sent encrypted?”
The lock icons were not proving to be the most noticeable indicators of encryption; it took testees time to find them. Only Fejleh immediately saw the green lock next to Brennan’s name when asked how she’d know (it took her 3 seconds) and identified it as a means of knowing if the message was encrypted. Two more testers identified the green lock after saying they could not know if it was encrypted: Mamun after 9 seconds, Jo after 15 seconds.
It took testers a much longer time to identify the orange open lock icon as a sign of a message’s encryption status — if they noticed it at all. Jo took a total of 33 seconds to see the orange lock, after she’d already seen the green lock 18 seconds earlier. Baraba, Mamun, and Fejleh did not appear to see the orange lock during the test (the researcher pointed it out to Mamun after the test was over).
Maleksa took 26 seconds to find the orange lock, after she had already erroneously guessed “attach key” was the indicator and said the message was not encrypted because there was “no crypt.” She did find the orange lock before she found the green lock, because she had already sent her message by the time she was asked if it was encrypted. It took her a few minutes and a few mistakes in message sending before she identified the green lock.
A possible revision to the interface could put the small lock and signature icons not in a tab above the compose window, but rather to the left of the To:/Cc:/Bcc: fields, as icons the size of the Compose and other buttons at the top right of the screen. Given an F-shaped reading pattern, this might be a natural place for users to look first, and they might notice them more easily. (The researcher notes: “take this with a grain of salt and some more user testing; I’m a researcher, Jim, not a designer.”)
Do users know when their mail is being sent encrypted?
Three of four users’ quick gut reaction to “How would you know if it was going to encrypt?” was that they would not know. Mamun emphasized that he COULD NOT know. The fifth user was Maleksa, who sent her message part-encrypted before the researcher could stop her; but when asked how she’d know, she also indicated she saw no clues. “There is no ‘crypt,’” she said, smiling. “There are no signs, or whatever.” She did not think she had encrypted her message, because she didn’t “attach a key” (she was orienting to the “attach key” checkbox, which would attach the user’s own public key to the email). This despite the fact that her sent message displayed a green lock, apparently encrypted to Brennan’s key!
The insistence that they don’t know if the message is encrypted could mean one of a few things. It could reflect that they felt unsure about their security or expertise on the topic after talking a while with the researcher about their security practices and after getting a high-speed lesson on how encryption works. But it could also suggest that not enough visual cues are being given about encryption, especially given that the lock was not noticed.
Enigmail’s visual cue — showing the encrypted message — might be a good convention for Mailpile to adopt. Testees were curious about what an encrypted message looked like, and seemed satisfied by how garbled it looked (reaffirming a pattern of satisfaction at visual security feedback which was noticed in <a href=”http://www.viettan.org/Understanding-Internet-Freedom.html”>this study of Vietnamese activist bloggers</a>).
In conversation, Baraba mistakenly identified other things her computer did as “encryption.” She said that from time to time, when she tried opening Word documents or PDFs in programs which couldn’t open them, like WordPad, she could see that the messages were encrypted. Things that look like “incomprehensible computer stuff” to users may just be taken at face value, so to speak. This can work to our advantage if we take Enigmail’s route. (In the bigger picture, this could present an attack vector on encrypted mail: a bogus tool could be developed which would just present users with visible junk when they hit send, giving them a false sense of confidence that their mail had been encrypted, while in fact sending the message in the clear or to a malicious source.)
It is possible that if testees hadn’t been stopped, they might not have thought about whether their message was encrypted or not. There’s the possibility that they think that just because they are using Mailpile, the message is encrypted. It might be worthwhile to run additional user tests set up to see if this is the case.
The good news: “Find encryption keys” gets noticed!
The good news about the “Find encryption keys” link that shows up in mouseover of recipient names (and a few other places) is that when they’re aware they need to take action to encrypt and they don’t have a key, users find and make use of it quickly. The dialog in the subsequent window is also relatively clear, and at least in the test, users paid close attention to it. The yellow-orange bars of text as a warning method appeared effective enough that it’s possible that removing the yellow-orange color from non-warning parts of the interface (like the borders of compose fields) might clarify warning messages further. However, this is not at all pressing.
Something’s slightly wrong in the popup dialog that appears when a user clicks “Find encryption keys.” When Jo searched for keys it cycled from searching to error and then back again. Also, in the other user tests, sometimes search and error are up at the same time. This needs some ironing out.
“Attach key” considered harmful
These testees’ only familiarity with public key encryption was the rapidfire crash course the researcher gave them moments before they did the test. Thus, their mental models of what to do with keys were not well-developed — as is likely the case with many users who come to encrypted mail on their own, without training.
In this state, the “Attach key” checkbox at the bottom of the email was a strong and dangerous distraction to users. Mamun turned to it as his first choice of indicator of whether a message was encrypted or not. “Attach key” was Fejleh and Maleksa’s first choice when asked how to add a key to their contacts. Jo moused over it briefly when asked how she’d know a message was encrypted (though she decided on the lock as an indicator).
What the “Attach key” checkbox does is not clear; there is no visible feedback. It also does not specify it will attach the USER’S key.
Certainly it is a good idea to give users the option to attach a key, but clarifying the language and moving the option might be a good idea. A better choice might be an “Attach…” button that leads to a dialog about attaching your own key vs. a file or other attachment.
Users understand that some kinds of communication require dedicated apps
One of the most surprising findings of this test was the number of users who gravitated to and mentioned the text signature Mailpile adds to the email by default — a brief line of text which is essentially nothing more than an advertisement.
Jo looked to the signature when seeking evidence of encryption, but discarded the idea, saying “that doesn’t really tell me it’s encrypted.” Maleksa moused over the signature in a sent message at about the same point in her test, hovering over (and maybe clicking) the Mailpile URL in the signature.
The text signature was Baraba’s first and only guess when asked if her message would send encrypted. “Free software,” she said, indicating the sig file. (She may have been trying to impress the researcher; the researcher had explained to her before the test why OpenITP trusts free software more than closed-source software for encryption.) “I wouldn’t know that it was encrypted, but I would know that something… something’s up…. Like, this isn’t related, but if I get a text message from someone, and it says ‘sent from such and such,’ I know that person is using another means to text me. They aren’t using the conventional means. I’d know this isn’t just a straightforward email from one person to another.” Digesting this, the researcher did not prompt her for a second guess.
Mamun similarly attributed a failure of encryption to his understanding that Mailpile was a “special” tool, which required users at both ends to have it installed: “It says no encryption keys found matching. That means this guy doesn’t have the program in his laptop…. Or not in his laptop. Like, he didn’t verify his email in this program.” This would have been an annoyance to Mamun, who later stressed that he really wanted a unified app for the plethora of SMS and chat tools he and his peers use. The researcher clarified to him later that users can send encrypted email from a range of apps.
Compose button location
Three testees who are Gmail users were slow to find the Compose button on their first search for it. Jo took 10 seconds; Baraba 14 seconds; Maleksa 18 seconds — a noticeable pause for each. Their mouse movements indicated they looked to the top of the left-hand column first — unsurprising, as this is where Gmail’s big Compose button is located (not to mention that it is consistent with the often-observed F-shaped reading pattern). They usually did look to the upper right soon thereafter (this is a common compose location for Yahoo and Facebook, among others), and quickly identified the pen as the compose icon.
This may not warrant a redesign; it is noted here for future reference. Users may well get used to the Compose location with time. Mamun and Fejleh oriented to it immediately when asked to compose a message. Baraba and Maleksa may have been slowed down slightly by the fact that the researcher was asking them questions while they looked.
Bug: Not verifying email addresses
Maleksa’s test turned up some bugs. She was able to add Smari’s name (misspelled, even) to the To: field, and Mailpile did not reject it as a malformed email address. At this point none of his addresses was in the contact list, so it did not automatically find him when she began to type. All subsequent activity in the To: field continued to show Smari’s name and Brennan’s address together as one recipient. When she went to look for the missing key, the encryption helper dialog registered his name twice, which was strange. The “People in conversation” dialog at the top of the mail thread then indicated there were two people in the thread: the researcher’s account (from which users performed the tests), and “smari mccarty, Brennnan Novak,” which was associated with Brennan’s picture and the email address This email address is being protected from spambots. You need JavaScript enabled to view it. . When Maleksa clicked on Smari’s name in this dialog, Brennan’s contact page (with key, picture, address, etc) came up. When she returned to the thread, (hitting the browser back button a few times) and clicked “To: smari mccarty, Brennnan Novak” next to the encryption helper tab above the compose window, “smari mccarty, Brennnan Novak” appeared twice in the To: field, with green locks next to each.
This is a pretty fundamental bug, in an era when people are quite used to blithely allowing autocomplete to specify who they’re sending to (and Mailpile helpfully obliges when it can — as it should!). Users are given no indication the message did not send to the person they want to reach, when this happens; it appears it sent correctly, and even encrypted (as each “smari mccarty, Brennnan Novak” entry in the To: field has a green lock next to it). Fixing this bug should be a priority.
FOR FUTURE REFERENCE
When running further user tests on issues related to encryption specifically, we should be sure to have users talk aloud as they look around the software — something the researcher forgot to do except with Jo and Fejleh (and prompted them late in both cases). This might clarify how users are thinking about screen elements related to encryption before sending mail, and indicate whether they see the lock icons before this test indicated they do. Eye-tracking software would also be invaluable here, though a study has indicated mouse movement correlates pretty well with where users are looking.
(Setup: Users used an install of Mailpile provided by Brennan with new key-discovery features, to see how easy they were to use. The install made use of my personal keys and key database. All of this was performed on a Mac/Yosemite, using Firefox. Videos of the browser and of the user’s face during the test were captured using Quicktime. For the final test, the desktop with browser and live user video were broadcast to Brennan in realtime using Jitsi Meet in Chrome.)
PROTOCOL SCRIPT
This is the script used with each of the participants in these user tests.
This is a test of the software, NOT of you. I’m going to share the results with the team that makes the software, so we can fix any problems and make it easier for people to use.
May we record your voice and face as you do this test?
May we use your name? Or you may use an alias.
(User fills out consent form)
What is your occupation?
Tell me a little about how you manage your security and privacy online.
What would you expect secure email software would do?
Take a few minutes to look at the screen before you do anything. What do you expect this software can do?
This software can encrypt an email. How much do you know about encryption?
Encryption takes the body of the email you write and does a cipher code to it using your key and the key of the person you’re sending it to, so the only person who can decipher it is the person you’re sending it to. Anyone who intercepts it cannot read it. It DOES NOT hide who the mail is going to, and it DOES NOT hide the email subject. So if you are writing to a friend in who is hiding, and you put “Meet me at the store at 3:00” in the subject, someone who intercepts your message will know you are in touch with this friend, and will know where you are meeting. You have to put secret information into the body of the email. You can’t hide the email address of your friend this way.
Now I want you to try sending an encrypted email to two developers of this software, whose names are Brennan Novak and Smarí McCarthy. Set that up, and then stop right before you’d send it.
(stopping them right before they would send)
How would you know if this email was encrypted or not?
(Prompts to ensure the message is encrypted to both recipients, including prompts to find Smarí’s key.)
Do you have any more questions about this app?
(User fills out 5 point Likert scales on ease of use, likelihood to recommend to friends)
Originally published at openitp.org.
