To systematically evaluate the user interface design of Apple iOS multi-touch surface products, iPad and iPhone, for any flaws that negatively impact user experience and measure general intuitiveness of gestural movements.
I’m going to start you off with a bit of a scenario. You received an email about flight choices that you want to reply to the group about. Find that email.
Imagine that the text in the email is too small for you to read, how would you enlarge it?
Please respond to the group with your exact flight choice. Is there another way to do it?
You recently signed up for Groupon and don't want today’s deal. How would you go about deleting this
You want coordinate airport transportation with a friend Sam who lives near by. Write them a new email
to ask when he's leaving. Sam’s contact information is already in your phone.
Want to see if there is a new email... How would you go about doing this?
BBC NEWS APPLICATION
You're testing out the BBC News application. Check out BBC and look for news. Explore... how would refresh the home screen.
How would you go about finding more articles within the US and Canada section from the home screen?
REPEAT WITH iPAD
UI Error #1: Icons for “Compose Email” and “Reply” or “Reply All” are ambiguous
- Participants struggled to reply all to the group and compose a new email in tasks 1 & 5 easily and did not recognize the icons.
- Participants discovered correct action through trial and error causing accidental deletion/archiving of emails and reply all and compose email.
UI Error #2: Confusion Over Increasing Text or Zooming In and Out.
- By default, 7 out of 8 users wanted to pinch/spread while in the BBC application to increase text size.
- 1 user immediately recognized "-A" and "A+" was the way to increase size but did not use the pinch/spread gesture to increase and decrease text size.
UI Error #3: Refreshing the screen.
- The Maill.app refreshed emails through clicking the refresh symbole below, whereas, BBC news articles were refreshed through pulling down the screen.
UI Error #4: Unclear how to delete emails
- Contributors and Projects are not easily discernable by users of what content is associated to each label.
Recommendation #1: Icons should be replaced or accompanied by labels to decrease ambiguity
Recommendation #2: Tutorials or orientation splash screen should be included for each application pointing the various features and location of critical tasks.
Recommendation #3: Standardized gestures in Apple Guidelines that standardizes gestures across applications sold in the App Store to promote consistency. For example, zoom in/out should be achieved through the spread and pinch gestures in addition to –A & A+ symbols if desired.
Purpose of Usability Test
AIMS AND OBJECTIVES OF THE TEST
To systematically evaluate the user interface design of Apple iOS multi-touch surface products, iPad and iPhone, for any flaws that negatively impact user experience and measure general intuitiveness of gestural movements. The test has the following specific goals:
Study intuitiveness and ease of learning of the current 8 gestures for iOS version 4.2: Tap, Double Tap, Touch and Hold, Drag-Flick, Pinch, Spread, 2 Finger Drag Down, and 2 Finger Drag Up.
Confirm comfort regions and how the physical form factor affects gestural motion discovery as well as the way the device is held and impacts user experience by audience segmentations: age, demographic.
Test general application features as well as downloadable applications for tap vs. gesture based navigation.
Characterization of Tasks
The tasks for the initial study of the iPhone and iPad interfaces were set up using scenarios. First, we wanted to determine the ease of learning or finding gestural features on standard applications on both devices. This included items such as finding an email, enlarging text, responding to all members on an email, copying and pasting text, deleting an email and composing a new email. These tasks were conducted on both devices.
Next we wanted to test a downloaded application that had several gestural motions built in to it. We requested that the participant find more news stories in a category – which was only possible using a swipe function. We then asked them to refresh the home screen, which was done via a pull down method on the iPhone, but used a tap of a refresh symbol on the iPad. The other difference that we wanted to look into was the method of enlarging text, which could not be done with the pinch method in the BBC application, only by clicking a large and small A to change the text size.
Finally, we wanted to gather feedback not only on what we observed, but what the user felt as they were going through the process. We encouraged the participant to think aloud during the tasks, but also asked several questions about their comfort level and learning ability after the fact.
Members of our team administered all tasks. Each member followed the same basic script in order to guide the participant through the three main tasks and additional general question tasks that were selected in the Test Plan.
Below is an excerpt from the Test Script, which was followed by a post-test questionnaire.
Conclusions from Data & Testing
FINDING BY TASK
iPhone: Find email
Participant 2, 4, and7 failed to find the email on the iPhone in Task1. Participant 7 did not know the “flick” gesture revealed more messages and Participants 2 and 4 suggested they would use search to locate the email in this situation rather than scanning for keywords.
iPhone: Enlarge text
Participants 6 and 8 were unable to enlarge text on the iPhone. However, participant 6 eventually figured out how to awkwardly pinch and spread to zoom in out, whereas, participant 8 never discovered the gesture but located the “-A” and “A+” icons quicker than any other participant when testing the BBC app.
iPhone: Reply all
6 of the 8 participants successfully identified the “ReplyAll” icon. The remaining 2 were not given the task, therefore, there was a 100% success rate despite user trial and error when exploring the hidden meaning or actions of the Mail icons.
Copy/paste was difficult to test because of the imperfect interaction. Users struggled to get the feature to work as desired, but 4 of the 5, or 80%, participants were able to successfully copy and paste text into a new message.
Participant 4 was the only user who gave up on the delete task. The other 7 participants successfully
discovered delete through various methods. The side-to-side swipe was used in 2 cases and “edit” in 3 cases.
Additionally, participants 7 and 8 uncovered a new method, which involved clicking the
message to a new mailbox” icon and then selecting the “Trash” icon.
iPhone: New email
There was a 100% success rate in this task despite many errors during the discovery process. It wasoften
difficult for participants to associate “Create New” or “Compose Email” with a square icon with a pencil through it,
iPhone: Refresh inbox
The first 3 participants were tested on refreshing email, but it became evident that users identified refresh easily and subsequent participants were not tested.
iPhone: Refresh BBC
The “pull to refresh” in the BBC app only had a 25% success rate or 2 out of 8
participants. More advance users, participants 1 and 2, were able to uncover this refresh method despite the
most advance user of the study, participant 5, was unable refresh the home BBC screen.
iPhone: Find More stories
5 of the 8 participants were able to reveal more stories in the BBC application when scrolling left to right on the news categories. Participant 1, 3, and 5 represented the most experienced user base and were the ones who failed the task. Alternatively, users with no experience, 4, 6, 7, and 8, easily scrolled left to right without hesitation to reveal more news articles. Participant 8 mentioned he saw the side-to-side action “On TV”.
iPad: Find email
In contrast, there was a 100% success rate on the iPad compared to the 62.5% success rate on the iPhone. The 37.5% or 3 participant difference was attributed to the gained familiarity with the iOS interface when gestures were discovered in other tasks.
iPad: Enlarge text
6 of 7 participants were able to en large text on the iPad, but participant 8 was unable to discover the pinch and spread gestures continued from the iPhone, whereas, participant 6 discovered an awkward method of spreading the index and middle finger to achieve the pinch and spread gesture commonly used with the thumb and index finger.
iPad: New email & Refresh inbox
BUTTONS VERSUS GESTURES
Apple says on one multi-touch device that “it supports a full set of gestures, giving you a whole new way to control and interact with what’s on your screen. Swiping through pages online feels just like flipping through pages in a book or magazine.” Gestures are promoted as more natural and more akin to our interactions with commonplace physical objects.
Our testing reveals that test participants unfamiliar with touch devices do not find gestures and icons immediately understandable. Simple tasks often caused a great deal of frustration as methods and use strategies inherited from desktop or laptop use did not translate to the novel mobile environment. Overall, tasks performed on the iPad were completed more quickly and with greater ease. Participants who showed the largest discrepancy in satisfaction between the iPhone and iPad cited the larger screen size as the primary reason.
Except for one user who came to the test with extensive iPhone experience, test participants did not try out different gestures to complete the tasks. While all commented that they understand how gestural interactions worked, they seemed uninterested in seeing how different gestures would work to complete the tasks. While this may be in part to the social pressure of the test, to ‘not looking stupid’, test participants’ first looked for buttons to complete the tasks. If the user did not find a button in the area immediately visible on the screen, he or she would scroll down to the end of the page in search of a button in nearly every case.
The BBC website refresh task illustrates this. The two buttons as the top of the screen “LIVE RADIO” and “Edit” (Figure 1) do not seem to have any connection to refreshing the page. 5 out of 6 users ended up pressing the “Edit” button after scrolling to the bottom of the page and not finding any other candidate for refreshing the page. Out of these 5 participants, only one discovered that refreshing the page is done by dragging the news stories down and
Similarly, it is possible to delete an email on both the iPhone and iPad Mail applications by swiping the Inbox entry left. The four buttons available on the iPhone version (identical to the iPad) are shown in Figure 2. Test participants varied in how many or which buttons were tried, but all (again, except the experienced iOS participant) explored deleting the email using these buttons. Gestures were not attempted.
The difficulty users experienced with gestures and icons exposes the underlying tension between discovery and ease of use. While every designer must cope make decisions related to this theme, the issue is all the more critical for mobile applications. Their low cost and the ease with which one can acquire them also means that they are that much easier to dispose of if the user find them lacking.
ORIENTATION VIDEO OR FEATURE WALKTHROUGH
6 out of the 8 test participants were familiar with two gestures: the one finger swipe down to scroll and the pinch and drag to zoom. Scrolling has a high-level of discoverability, since the interaction is accessible nearly everywhere on the interface. When asked why participants knew how to zoom, nearly all responded that they had seen it on the early iPhone commercials. This points to the power of explicitly showing gestural features through video. Gesture discoverability would be greatly improved through in-application orientation videos. Another possibility is some sort of tutorial where different interface elements could be clearly explained.
OFFER AN INITIAL DISCOVER PHASE
Another possibility for increasing discoverability would be to provide two modes of use. One, which highlights the different features and interactions and one that is optimized for on-going ease of use. Smart phones have moved beyond the Early Adopter segment in the product life-cycle, and many users needs do not advance beyond basic functionality at the introductory level. However, there seems to be an experienced niche group who uses the devices complexity who might choose to sidestep ‘training wheels’, but it would invite a wider audience to explore deeper features of the application.
One implementation of this would be to employ a feature borrowed from the desktop environment – the task menu. Again, experienced users might prefer the direct action of icons well understood, but novice or less tech-savvy users would find a simple to understand method for completing common tasks.