Using Metrics to Identify User Pain Points in Open-Source Security Tools
Originally published on the OpenITP Secure User Practices blog, on November 3, 2014
The gathering and use of data on software users is currently a hotly contested public issue. Companies like Facebook and OkCupid have attracted a great deal of public criticism for not only gathering and selling users’ information, but also using it to manipulate users’ moods. Credit card companies and major retailers have not proved immune to massive attacks on users’ financial and personal data.
Free software to protect security, privacy, and anonymity has to a great extent been developed in response to concerns like these. Developers of open-source secure tools have gone out of their way to ensure they gather no data that could be used to pinpoint users’ locations, sketch out their social networks, and otherwise put them at risk.
The trade-off is that without this information, it can be hard to evaluate the effectiveness of tools in this space. Commercial communications software has monitoring and evaluation tools built in as a matter of course. A/B tests of new features, as well as metrics on ROI, use, failures, and abandonment, are standard, day-to-day practice in the tech industry. In the open-source secure tools space, this kind of regular use of data to inform development is much more rare.
Some tools in this space do gather basic, aggregate metrics on usage or downloads. Tor keeps track of metrics for the health of the ecology of nodes and routers, making information on countries where Tor is in use publicly available along with a number of more technical measures. Cryptocat publicly displays usage of its tool by time of day and country (see right).
The Google Play or Apple App stores gather aggregate, non-personally-identifying metrics on downloads of apps by default. As a result, development teams whose tools are available for mobile devices — like Open Whisper Systems and the Guardian Project — have this data available to them. The data may include tables comparing the country in which the phone is registered, operating system language, operating system version, model of device, uninstall and install by user and by device, phone service carrier, users’ average rating of the app, and information on when the app crashes.
However, developers in our space are not always comfortable even with this much data gathering. The Guardian Project took pains to clarify that they do not gather any data beyond this; their tools do not send any data about usage back to the developers. The Guardian Project also makes their apps available through their own FDroid app store, which does not track even the basic metrics mentioned above.
Developers also do not tend to use what data the Apple and Google stores gather to identify user frustrations, errors in the interface, under-served regions or languages, and other issues which may be acting as barriers to adoption. Even some of those who collect this data do not tend to use it, and have expressed a desire to move away from the popular, pervasive app stores which collect this data.
OpenITP’s Secure User Practices (SUP?) project has worked to foster dialog on data-gathering which might be safe and amenable to privacy concerns, for the sake of having information to improve these tools. To continue this dialog, this report presents some data which projects have gathered. The analysis performed here is intended to demonstrate how developer teams in our field could safely be using metrics more effectively to identify places where their tools could be improved to meet users’ needs, particularly free-speech activists and journalists.
The data presented here is a snapshot of The Guardian Project’s Android app store download data for four apps — ChatSecure (Android), Obscuracam, Storymaker, and GNUPG for Android — as of June 15, 2014. Additionally, the last section analyzes install data for ChatSecure iOS.
This analysis finds the following noticeable trends:
- Users of the Spanish-language ChatSecure Android app uninstall the app at a higher-than-expected rate.
- This is true across all countries with Spanish installs.
- Uninstalls are happening at higher-than-expected rates in other key countries.
- Installs are simply not happening in central Asia and key African countries.
- Generally, people install the app in the dominant language of their country.
- Some phone brands are correlated with higher uninstall rates — and some of these brands have a significant market share in countries where censorship or human rights violations occur.
- Major spikes in installs seem to be event-driven.
- In the case of Turkey, Internet censorship seems to have spurred interest in ChatSecure iOS.
- This does not appear to have held true in Venezuela and Ukraine.
While the data does not clarify why these trends are happening, they do suggest a number of additional questions that should be asked in order to further support under-served populations, and to improve the spread of these tools. Each section below will begin to outline directions for additional research which could shed light on these open questions.
CHATSECURE (ANDROID) GLOBAL UNINSTALL PATTERNS, BY DEVICE
QUESTION: Are certain makes of phone showing higher uninstall rates?
FINDING: Some phone brands do uninstall at higher rates — and some of these phones have a significant amount of market share in countries which may be of concern due to censorship or other humanitarian reasons.
METHOD
I batched uninstall data by brand of device, guessing or looking up what brand most devices listed had. This is likely not especially rigorous, as I did not look up every single device by model name; it mostly consists of devices listed by their brand name or abbreviation thereof.
Samsung was by far the most popular brand in this dataset, with around 30K current users — over three times its nearest competitor, Google, which currently has some 8,700 users. Thus, I left Samsung out of some analyses, as its scale made it hard to see differences among less-popular brands.
It also made sense to treat the next 11 competitors in two tiers — HTC, Google, Sony, Motorola, and LGE, which appeared to have had tens of thousands of all-time installs, and then Huawei, Asus, ZTE, TCT Mobile Limited, Acer, and Lenovo, which appeared to have had 1000–5000 all-time installs. (I say “appeared” because, again, I am not confident I batched all phones by make and model.) Other brands had seen less than a thousand all-time installs.
Maps used in this section represent device brands’ market share by country; that data is from StatCounter Global Stats. In the map views of device use, again, the scale is not consistent across maps; reference the legend on each map for the number of users a bubble represents.
DISCUSSION
These bar charts give a comparison of total to current user installs, where we begin to get a sense of makes of device with higher uninstall rates:
The bubble charts represent this a slightly different way, plotting total users by current users and indicating percent retention rate by size of bubble (bigger bubbles are device brands whose users are more likely to keep the app installed):
Smaller bubbles, some of which end up below the 44% retention trendline predicted for a communications app, indicate that fewer users of those devices keep the app on their phone after installing it.
Why does this matter? Looking at the places where brands falling below the trend line have high market share (as per global market share statistics from GlobalStats; this is not related to the install rate that Guardian measures in that country) shows that a number of countries where we might be concerned about human-rights issues and free speech have high numbers of people who use devices which do not have high uninstall rates (though correlation might not imply causation).
HTC has a notable market share in western Africa and Taiwan:
(Bubble size on the map indicates percent of market share; see map legend for numbers)
Huawei, in Myanmar:
and ZTE in Liberia.
Huawei and ZTE have a combined 25% market share in Haiti.
Under-serving Sony or LGE devices would mean under-serving a share of devices in Eastern Europe, the Middle East, West Africa, South America, and many other places worldwide:
FURTHER QUESTIONS
Is it possible that poor stability, lack of memory space, or other technical factors on these brands of device are responsible for fewer users keeping the app after they install it? Further investigation along these lines might point to particular brands of Android device which may be less compatible with ChatSecure. If yes, we might pay closer attention to ensuring stability on these brands of device, devoting more quality assurance testing to them. But then, it might also turn out that correlation does not imply causation, and in fact there are other factors, cultural, linguistic, or otherwise, which happen to be discouraging people from keeping the app on their device.
LANGUAGE, RATINGS, AND UNINSTALLS BY COUNTRY
QUESTION: Can we gauge user “happiness” with the app by the uninstall and rating rates? Do these differ by language and country?
FINDINGS: Generally, people install the app in the dominant language of their country. Installs are simply not happening in the key regions of central Asia and key African countries. Uninstalls are happening at higher-than-expected rates in other key countries. Users of the Spanish-language app uninstall the app at a higher-than-expected rate, and this runs across all countries with Spanish installs.
METHODS
For improved protection of users, this dataset only represents country/language pairs in which there have been more than ten downloads total.
Links to interactive maps visualizing this data:
HOW TO READ THE MAPS
A note about viewing the maps: Layers can be viewed separately using the “visible layers” toggle. You may need to select the layer which has the language you’re interested in if you want to click on a dot for more information. Clicking on a dot will pop up information about the language and country that dot represents, how many users currently have the tool installed, and how many users have installed it, in total, since the app was made available. Zooming in will make it easier to see at a glance which dot applies to which country.
The color of a dot indicates how many users currently have the app installed. The halo around that indicates how many people, total, installed the app over all time. This makes it easy to see when a higher percentage of people have uninstalled an app: the size of the colored dot will be visibly smaller compared to the halo. Note that dot size will not be comparable across maps.
Korean, Farsi, Indonesian, Thai, and Vietnamese are not actually among the most-installed languages, but I have highlighted them here for two reasons: one, they are of interest to our community, and two, to highlight their uninstall rate (about which more in a moment).
The color of a geographic area indicates how high the average user rating in that country was on a scale from 1 to 5 (with all languages taken together). The darker blue a country, the lower the average rating (with non-colored countries having no submitted ratings). A caveat: we have no way of knowing how many users in a country actually submitted a rating. Rating may be a particularly poor measure in countries where the apps have not been installed often (ones with small dots).
DISCUSSION
People mainly use the apps in the dominant language of their country
While most countries have a small number of users in a minority language in their country, the majority of users are using the app in the dominant language of the country. Notable exceptions include Spanish in the US, French in Canada, and a tenacious Catalan community in Spain. The second-most-visible language in many countries is English.
Africa and Central Asia are not being reached
Notably, these apps are generally not reaching Africa in any great numbers (particularly the north, and Ethiopia) or central Asia. These are regions of particular interest if we are seeking to support free-speech activists, organizers for democracy, and journalists.
Spanish-language uninstalls: Higher than expected
Spanish-language versions of ChatSecure appear to be retained at lower-than-average rates — even compared to Spanish-language versions of other Guardian Project apps. The rate of app retention for Spanish-language installs of Chatsecure is consistently lower, across countries, than it is for many other languages.
Uninstalls are high in target countries
According to metrics company Flurry Analytics (owned by Yahoo!) the average app retention for categories into which these Guardian apps fall is as follows:
Average retention rates for apps, by type, over 90 days:
Communications: 44%
Utilities: 43%
Social networking: 34%
Photo and video: 21%
These are, of course, worldwide averages. In Western European countries, English-speaking countries, and highly developed Asian countries like Singapore, Japan, and South Korea, Guardian apps stack up to these averages pretty well. Photo/video apps Obscuracam and Storymaker have retention in the 30%-40% range in a number of countries, including Russia and South Africa. Chatsecure comes in a little low for a communications app, with its highest country/language retention rate being about 37% for English language installs in Switzerland. GPG on average also skews lower than the 43% to be expected of utilities.
It is worth noting that in countries of concern (due to censorship and repression), users are much more likely to uninstall these apps after installing them.
The uninstall rate generally appears high in non-European countries and non-European languages, as compared to European ones: Indonesia, Viet Nam, Hong Kong, Iran, Korea, and parts of the Arabic-speaking world, including Yemen, Saudi Arabia, Egypt, Kuwait, Israel, and Lebanon. (In many of these countries, English-language app retention is also comparatively low, but then, it is generally low in countries where English is not the first language.)
FURTHER QUESTIONS
There are a number of questions which could be posed to explore why Spanish-language apps are uninstalled at a disproportionately higher rate. Is the Spanish-language translation poor quality? Are there particular phones being used in these countries on which the app is unstable? Do users have phones with low memory space, making them more likely to uninstall apps they don’t use? Or are they not finding enough friends with the app to talk to? Further investigation along these lines, and improvement of translation/localization, could make for a visible improvement in retention for languages with high uninstall rates, like Spanish.
Why might particular countries not be well-served by the app? In some of these countries, the risk of having one’s phone confiscated on arrest could also be a reason for a high rate of deleting the app. The only way to know for sure what is causing this high rate of abandonment would be user testing of the different language versions, if possible on the ground in particular countries.
The curious case of the United Arab Emirates
The situation of Guardian Project apps in the United Arab Emirates is intriguing. Clearly, a tremendous number of people have installed the apps there — remarkably, this is the only country where installs are on par with, or outpace, installs in European countries where the apps are popular. The other countries in the region with significant installs — Turkey, Egypt, and in some cases Iran — do not see anywhere near the number of installs this small country does.
Yet for Chatsecure, Obscuracam, and GPG, the percentage of uninstalls in the UAE is disproportionately high.
Retention in the UAE
Chatsecure — 3.68%
GPG — 6.63%
Obscuracam — 7.15%
Storymaker — 19.49%
Ratings for Chatsecure and Storymaker are quite high in the UAE, while ratings for Obscuracam there are quite low. There are no ratings for GPG.
FURTHER QUESTIONS
Why is this happening? Is it perhaps common practice in the UAE to uninstall and reinstall apps frequently, for plausible deniability or other reasons? Or are people genuinely wanting to use the app but not being pleased with how it works? The latter seems less likely given ratings do not seem to correlate with uninstall rate in this country.
CHATSECURE (IOS) INSTALL PATTERNS, BY DATE
QUESTION: Are there patterns in installs of this app which are tied to major events in particular regions?
FINDING: Major spikes in installs seem to be event-driven, but which events are the drivers are not always clear. In the case of Turkey, episodes of Internet censorship seem to coincide with increased interest in ChatSecure.
METHOD
These graphs are taken from ChatSecure iOS’s Apple App Store dashboard in early 2014. The blue line indicates the total number of apps installed over time. The orange section is installations of updates to the app, which are more dependent on the developer’s release of updates than on user activities. The additional lines indicating Venezuela, Ukraine, and Turkey indicate times when protests were going on in those countries.
DISCUSSION
There was an uptick in people installing ChatSecure on iOS devices in late February 2014 which appeared in many countries, including the United States, Egypt, and Israel.
Interestingly, this wasn’t timed in such a way that it appears to be directly related either to Venezuelan or Ukrainian uprisings, which it occurred between. The number of Venezuelan installs spiked during that time, but appears to have been contemporaneous with the worldwide spike, not with events in the country. The Ukraine does not show any spike related either to events there or the international spike pattern in late February.
By comparison, Turkey saw the number of ChatSecure iOS users nearly double in the weeks following the shutdown of Internet access by their government in March 2014:
FURTHER QUESTIONS
Was there news or social media coverage of ChatSecure around the time of the late-February spike? That might be clarified if projects researched or gathered press hits about their apps.
And what spurred the adoption of ChatSecure in Turkey — word-of-mouth and social media? Press coverage? Why did it happen there and not Venezuela or Ukraine? Does the technical skill level of the populace have something to do with it? Are users familiar with the idea of encryption? All of these are questions which could be answered by interviews with activists, journalists, and possibly even trainers in these countries. Future attempts to improve adoption and retention could build on findings about what previously drove adoption.
