(Un)Usability of Enigmail

Ralph Wozelka, Keith Andrews, Carina Enzinger,
Bianca Helbig, and Jacqueline Rinnhofer

Graz University of Technology
and Cryptoparty Graz

Slides:
http://keithandrews.com/talks/2016/glt-2016-enigusab/

Sat 30 Apr 2016, Grazer Linuxtage 2016, FH Joanneum, Graz.

1. Introduction

Enigmail is Thunderbird extension for OpenPGP security (signing and encryption) for email.
Enigmail hard to install and use, especially for less technical users.

GPG for Journalists

Ed Snowden asked Glenn Greenwald to install GPG... weeks went by.
Eventually, Ed created a 10-minute tutorial video "GPG for Journalists" for Glenn.
https://vimeo.com/56881481

See also:
Why is GPG "damn near unusable"?; talk by Arne Padmos; https://www.youtube.com/watch?v=4gz9TBt-DAQ [04:45-]

Enigusab Project Schedule

Heuristic Evaluation (Jan to Mar 2016):
- with expert evaluators.
Interface Changes Part 1 (May to Jun 2016):
- redesign and coding.
Thinking Aloud Test (Summer 2016):
- with representative test users.
Interface Changes Part 2 (Autumn 2016):
- redesign and coding.

2. Heuristic Evaluation (Expert Review)

Invented by Jakob Nielsen and Rolf Molich in 1990.
Small team (typically 3 to 5) of usability experts analyse and assess a user interface.
Using a set of usability principles (heuristics).
Positive and negative findings are noted.
Each evaluator rates all findings.
Findings are sorted in descending order of average positivity/severity.
Two extensions:
- Evaluations were recorded using screen capture, webcam, and microphone.
- The two members of the test team also performed evaluations.

Evaluators

Evaluator	Konrad Baumann	David Derler	Vesna Krnjic	Johanna Pirker
Expert	Usability	Usability, Cryptography	Usability, Cryptography	Usability
Device	MacBook Pro	MSI-GP60 (Oracle VM 5.0.14)	MacBook	PC
Operating System	OS X Yosemite 10.10.5	Ubuntu Gnome 15.10	OS X El Capitan	Windows 10
Enigmail Version	1.8.2	1.9	1.8.2	1.8.2
Language (Enigmail)	German	English	English	German

List of Positive Impressions

No.	Short Title	Description	Positivity
No.	Short Title	Description	KB	DD	VK	JP	CE	BH	Av
1	Easy to Find	It is quite easy to find the Enigmail Add-on/extension just knowing the name. Enigmail is provided at different kinds of search engines. Like this all four evaluators found a way at once to download Enigmail.	4	4	3	2	4	3	3.3
2	Does What it Should	Some experts indicated that Enigmail does what it should do quite straight forward.	3	3	2	3	3	3	2.8
3	Better than Previous Versions	One expert tried out a previous version of Enigmail and thinks that the current one is definitely an improvement, but still needs some work to fit the needs of normal users.	2	0	2	3	3	4	2.3
4	Self Explaining	Using Enigmail is quite self explaining after the installation because the symbols easy to find.	3	2	3	3	1	1	2.2
5	Expiry Date	It is possible to define and change the expire date of the key.	3	2	1	1	3	2	2
6	Unencrypted Warning	Enigmail warns the user if an e-mail is not encrypted.	3	2	3	3	1	0	2
7	Fancy User Interface	The design when writing e-mails is pretty good.	3	2	3	2	0	0	1.7

Positivity Ratings

Positivity	Meaning
4	Extremely Positive
3	Major Positive
2	Minor Positive
1	Cosmetic Positive
0	Not a Positive

List of Problems Found

No.	Short Title	Heuristic	Severity
No.	Short Title	Heuristic	KB	DD	VK	JP	CE	BH	Av
1	Manual Mode in Setup Wizard	A13 Bug	4	2	4	2	3	4	3.2
2	GPG Keychain	A13 Bug	3	3	4	3	2	3	3
3	Unencrypted And/Or Unsigned Message	A12 Unclear Meaning	3	2	4	1	4	3	2.8
4	Location to Save Revocation Certificate Message Appears Too Late	A01 Feedback	3	2	3	3	3	3	2.8
5	Public Keys of the Same E-mail Address are Not Distinguished	A01 Feedback	3	2	4	3	3	2	2.8
6	Add-on Description	A10 Help and Documentation	3	2	4	4	2	2	2.8
7	Passphrase Feedback	A01 Feedback	2	2	3	2	4	3	2.7
8	Clickable Error	A12 Unclear Meaning	2	2	3	3	2	4	2.7
9	Different Key Wizards	A04 Consistency	2	2	4	3	3	2	2.7
10	Old GnuPG	A10 Help and Documentation	3	2	4	2	2	3	2.7
11	Confusing and Too Much Text/Words	A02 Speak the Users' Language	3	3	4	2	3	1	2.7
12	Web Of Trust	A10 Help and Documentation	3	3	2	1	4	2	2.5
...	...	...	...	...	...	...	...	...	...

Severity Ratings

Severity	Meaning
4	Catastophic problem
3	Serious problem
2	Minor problem
1	Cosmetic problem
0	Not a problem

Problem No. 1: Manual Mode in Setup Wizard

Setup wizard does not work in manual mode (experts).
GnuPG installation gets interrupted by a GPG Keychain window.
Redoing in beginner's mode fixes problem somehow.

Problem No. 2: GPG Keychain

Opening public key with GPG Keychain.
Enigmail Key Management does not get updated.

Problem No. 3: Unencrypted And/Or Unsigned in Status Message

Positive feedback hardly visible, negative feedback easily mistaken.
Two aspects (signed and encrypted) are conflated.
Both "Enigmail:" and the status message itself are (unnecessarily) clickable.

Problem No. 4: Location to Save Revocation Certificate Message Appears Too Late

After saving process a where-to-save hint pops up.

3. Interface Changes Part 1

Local clone of enigmail repository.
Study enigmail wiki on sourceforge.
Inspect enigmail sourcetree (XUL + Javascript).
Install recommended dev tools:
- DOM Inspector
- Tiny Javascript inspector
- Komodo Edit

Developing with XUL

XUL is user interface language based on XML.
Originally developed for Mozilla project.
Clear division between structure and design.

Improved Problem No. 3: Unencrypted And/Or Unsigned Status Message

Original status message was red, conflated two aspects, and was a button.
New message:
- black text on gray
- redesigned into two parts (signed and encrypted)
- only a label.

Resources

Patrick Brunschwig; Enigmail: A simple interface for OpenPGP email security; Visited 2016-04-25. https://www.enigmail.net/
Jakob Nielsen and Rolf Molich;; Heuristic Evaluation of User Interfaces; Proc. Conference on Human Factors in Computing Systems (CHI’90). ACM. Seattle, Washington, USA, Apr 1990, pages 249–256. doi:10.1145/97243.97281
Jakob Nielsen; Enhancing the Exploratory Power of Usability Heuristics; Proc. Conference on Human Factors in Computing Systems (CHI’94). ACM. Boston, Massachusetts, USA, Apr 1994, pages 152–158. doi:10.1145/191666.191729
Jakob Nielsen; 10 Usability Heuristics for User Interface Design; Visited 2014-04-27. https://www.nngroup.com/articles/ten-usability-heuristics/
Arne Padmos; Why is GPG "damn near unusable"?; Visited 2016-04-27. https://media.ccc.de/v/31c3_-_6021_-_en_-_saal_g_-_201412281130_-_why_is_gpg_damn_near_unusable_-_arne_padmos
Edward Snowden; Speak at HOPE X in 2014 with Ellsberg; Visited 2016-04-27. https://www.youtube.com/watch?v=onrTo1sYri8
Mozilla; Thunderbird: Software made to make email easier; Visited 2016-04-25. https://www.mozilla.org/en-US/thunderbird/
Alma Whitten and J.D. Tygar; Why Johnny can't encrypt: a usability evaluation of PGP 5.0; SSYM'99 Proceedings of the 8th conference on USENIX Security Symposium. Volume 8. USENIX Association Berkeley, CA, USA, 1999. Visited 2016-04-25. http://www.gaudior.net/alma/johnny.pdf

Thank You!

Slides:
http://keithandrews.com/talks/2016/glt-2016-enigusab/

(Un)Usability of Enigmail

Ralph Wozelka, Keith Andrews, Carina Enzinger, Bianca Helbig, and Jacqueline Rinnhofer

Graz University of Technology and Cryptoparty Graz

1. Introduction

GPG for Journalists

Enigusab Project Schedule

2. Heuristic Evaluation (Expert Review)

Evaluators

List of Positive Impressions

Positivity Ratings

List of Problems Found

Severity Ratings

Problem No. 1: Manual Mode in Setup Wizard

Problem No. 2: GPG Keychain

Problem No. 3: Unencrypted And/Or Unsigned in Status Message

Problem No. 4: Location to Save Revocation Certificate Message Appears Too Late

3. Interface Changes Part 1

Developing with XUL

Improved Problem No. 3: Unencrypted And/Or Unsigned Status Message

Resources

Thank You!

Ralph Wozelka, Keith Andrews, Carina Enzinger,
Bianca Helbig, and Jacqueline Rinnhofer

Graz University of Technology
and Cryptoparty Graz