At Creative Mornings, David Kelley gave a retrospective of his career as the founder of IDEO and creator of the at Stanford. He shared success stories about how multi-disciplinary teams use human-centered design to innovate. Highlighting those stories were projects A Cafeteria Designed for Me and Embrace. In closing, he acknowledged that more than ever designers roles have become increasingly important and that we now must ask ourselves what problems are worth working on and how we want to contribute moving forward.

AuthorDavid Brahler

Natural user interfaces, or NUIs, are characterized by 3 distinct input modalities: multi-touch, in-air (kinetic) gesturing, and voice command. The Natural user interface is a blanket term that Steve Mann came up with. It's similar to interactions of graphical interfaces with a mouse, but another layer of abstraction has been removed, allowing users to interact directly with the system via fingers, motion, or dictation.

Note: Let's not get caught up in vernacular. You can easily subsitute Touch User Interface, Gestural User Inteface, or Sensory Interfaces

We've been designing for gestural interfaces for the past five years. Beyond typography and touch points and an ancient human interface guideline, the initial delight of owning a touchable device is ever fading to... unremarkable. Multi-touch surfaces, like tablets and smartphones, create interesting opportunities for design that cater to the devices in question. It's a natural experience in the way it establishes a positive emotional resonance when interacting with a physical device, not just interacting with a website behind glass.

We've reached this kind of emotional stagnation with the screen. The initial delight of owning a touchable device is decreasing and becoming unremarkable. We're losing the magic and it's important that design continues to progress forward in order to enhance the experience.

I've observed during usability studies that people tend to use fingers or hands as a giant mouse, often clicking their way to the desired goal. Rather than point and clicking, it's time we explore and embrace NUIs by increasing the gestural vocabulary in order to mitigate the point and click convention that I think is inhibiting the maximum experience. Why do you choose check Twitter over doing million other things that your smart phone can do while waiting in a grocery line? It's because of these simple, rhythmic, gestural microinteractions like pull-to-refresh that keep delivering the sense of delight.

The discussion about privacy and the right to be found or not found led me to do a little more investigation on the topic. Jeff Jarvis has recently published a book on this subject matter titled Is Privacy Overrated?. The Forbes online article provides an overview of the book, which examines privacy bills passed by the Obama administration in order to protect online rights. I think the author does well to play devil's advocate when promoting the benefits of social interconnectivity as well as the downfall of sensitive information being available in the public domain. Outside of the government, designer's will help shape the discoverability of personal information and help create filters and access privileges, so that information is shared within the proper context. We will help shape, a so to speak, global Facebook privacy setting.

AuthorDavid Brahler

The Thursday afternoon broadcast of NPR's Planet Money discussed government grants funding technological research. A Carnegie Mellon researcher has been developing facial recognition software the past 10 years, which reveals a person's identity by matching a photograph to a database compiled with Facebook profile information and other social networking information. The software can identify name, age, where you were born, and Social Security Number.

"For the study, the Carnegie Mellon team used software from Pittsburg Pattern Recognition, or PittPat. Google bought that company last month. The software can recognize faces in photos and videos." This goes to show the progress technology has moved toward or is currently at the onset of the Orwellian future Morville painted in chapter 4: Intertwingled. Moving forward, wayfinding and findable objects will improve due to this technology, but at what expense? And what freedoms do we give up? Or what do we gain? The PittPat software developer said he has no intention of releasing the software to the public, but I can see costs and benefits of ubiquitous software implementation. Specifically in the near future, we may be able to go through airport security while shoes remain on our feet.

Government officials, police officers, etc., have access to criminal databases and means to run background checks. Court records are in the public domain and records are public knowledge. All one has to do is access a county court website and search by Last Name. Facial recognition will help police officers do their job more efficiently. Medical records are stored in hospital databases and are currently undergoing a reform where information is becoming centralized and records are accessible to all hospitals.

"However, this technology will also very likely be used in greater capacity in the commercial sector to further target consumers for advertising and discriminatory pricing purposes."

Online retailers, marketing agencies, search engines, etc. already track browsing history. Online advertisements target audiences through cookies and browsing history. Ever noticed how LL Bean seems to always have an advertisement on your favorite blog after visiting their webstore? Credit card companies have detailed information on purchases. Cell phone providers have records of text messages, phone calls, and call history. Network service providers have records of instant messenger conversations, emails, and web activity.

It is evident that privacy rights have been abandoned, knowingly or unknowingly, online online . The most biggest publicized offender? Facebook. So, why are 750 million active Facebook users still under the impression that their lives are private?

It seems to me that both Generation X and Y have already made it clear that when it comes to privacy that they are open to freedom of information. The Baby Boomers are the last remaining generation that has a choice to remain anonymous.

AuthorDavid Brahler

Peter Morville briefly touches on his interest in ubiquitous computing in the opening page and chapter of Ambient Findability, but he fails to comprehend the pervasiveness of mobile computing to come in the following 5 years after the publication of Ambient Findability. I can't help but be reminded of his recent publication, Information Architect, where he asks himself "Am I getting better?" and "Should I do something else?". Ultimately, he concludes that he is still growing as an Information Architect and that the User Experience landscape is continuously evolving and there has never been a better time to be an information architect.

I'm reminded of his words because of Bruce Sterling's Shaping Things and Mike Kuniavsky's Smart Things. Both authors introduce words to the ubicomp vocabulary including Spimes and information shadows". Mike Kuniavsky gave a very interesting presentation at UX Week 2008 title Ubiquitous Computing and User Experience and Bruce Sterling was the keynote speaker at Interaction '11 in Boulder, Colorado where I got to witness his free-spirited Closing Keynote.

Research has been evolving in this field, since before the early 90's when Mark Weiser published The Computer for the 21st Century identifying the research that the Xerox PARC lab was predicting.

Brenda Laurel recounted her career as a User Experience researcher and designer in another Interaction '11 Keynote Presentation. The most intriguing portions of this talk were the presentation of the portal that appeared in New York City and the work and research that was done roughly 5-10 years prior to Google Maps became ubiquitous.

The main reason I bring up all this information is that with all of this research that has been done around or before "Ambient Findability", why has Peter not brought up any of these topics when he clearly states his passion for ubiquitous computing? Mobile devices are becoming a part of our extended mind and stepping closer and closer to bridging the gap between the 90's virtual reality to a very real augmented reality that relieves the rigors of wayfinding through guided, aided, or augmented reality through the current mobile device screens or advanced computerized eye lenses. Devices seen in Minority Report, Back to the Future II, Avatar, etc. may not be as far out as we think if we consider how we use GPS, iOS, and Android devices currently.

AuthorDavid Brahler

Executive Summary


To systematically evaluate the user interface design of Apple iOS multi-touch surface products, iPad and iPhone, for any flaws that negatively impact user experience and measure general intuitiveness of gestural movements.



  1. I’m going to start you off with a bit of a scenario. You received an email about flight choices that you want to reply to the group about. Find that email.

  2. Imagine that the text in the email is too small for you to read, how would you enlarge it?

  3. Please respond to the group with your exact flight choice. Is there another way to do it?

  4. You recently signed up for Groupon and don't want today’s deal. How would you go about deleting this


  5. You want coordinate airport transportation with a friend Sam who lives near by. Write them a new email

    to ask when he's leaving. Sam’s contact information is already in your phone.

  6. Want to see if there is a new email... How would you go about doing this?


  1. You're testing out the BBC News application. Check out BBC and look for news. Explore... how would refresh the home screen.

  2. How would you go about finding more articles within the US and Canada section from the home screen?



UI Error #1: Icons for “Compose Email” and “Reply” or “Reply All” are ambiguous

Severity: Critical

  • Participants struggled to reply all to the group and compose a new email in tasks 1 & 5 easily and did not recognize the icons.
  • Participants discovered correct action through trial and error causing accidental deletion/archiving of emails and reply all and compose email.

UI Error #2: Confusion Over Increasing Text or Zooming In and Out.
Severity: Mild

  • By default, 7 out of 8 users wanted to pinch/spread while in the BBC application to increase text size.
  • 1 user immediately recognized "-A" and "A+" was the way to increase size but did not use the pinch/spread gesture to increase and decrease text size.

UI Error #3: Refreshing the screen.
Severity: Mild

  • The refreshed emails through clicking the refresh symbole below, whereas, BBC news articles were refreshed through pulling down the screen. 

UI Error #4: Unclear how to delete emails
Severity: Critical

  • Contributors and Projects are not easily discernable by users of what content is associated to each label.


Recommendation #1: Icons should be replaced or accompanied by labels to decrease ambiguity

Recommendation #2: Tutorials or orientation splash screen should be included for each application pointing the various features and location of critical tasks.

Recommendation #3: Standardized gestures in Apple Guidelines that standardizes gestures across applications sold in the App Store to promote consistency. For example, zoom in/out should be achieved through the spread and pinch gestures in addition to –A & A+ symbols if desired. 

Purpose of Usability Test


To systematically evaluate the user interface design of Apple iOS multi-touch surface products, iPad and iPhone, for any flaws that negatively impact user experience and measure general intuitiveness of gestural movements. The test has the following specific goals:

  •  Study intuitiveness and ease of learning of the current 8 gestures for iOS version 4.2: Tap, Double Tap, Touch and Hold, Drag-Flick, Pinch, Spread, 2 Finger Drag Down, and 2 Finger Drag Up. 

  • Confirm comfort regions and how the physical form factor affects gestural motion discovery as well as the way the device is held and impacts user experience by audience segmentations: age, demographic.

  • Test general application features as well as downloadable applications for tap vs. gesture based navigation. 

Characterization of Tasks


The tasks for the initial study of the iPhone and iPad interfaces were set up using scenarios. First, we wanted to determine the ease of learning or finding gestural features on standard applications on both devices. This included items such as finding an email, enlarging text, responding to all members on an email, copying and pasting text, deleting an email and composing a new email. These tasks were conducted on both devices.

Next we wanted to test a downloaded application that had several gestural motions built in to it. We requested that the participant find more news stories in a category – which was only possible using a swipe function. We then asked them to refresh the home screen, which was done via a pull down method on the iPhone, but used a tap of a refresh symbol on the iPad. The other difference that we wanted to look into was the method of enlarging text, which could not be done with the pinch method in the BBC application, only by clicking a large and small A to change the text size.

Finally, we wanted to gather feedback not only on what we observed, but what the user felt as they were going through the process. We encouraged the participant to think aloud during the tasks, but also asked several questions about their comfort level and learning ability after the fact.


Members of our team administered all tasks. Each member followed the same basic script in order to guide the participant through the three main tasks and additional general question tasks that were selected in the Test Plan.

Below is an excerpt from the Test Script, which was followed by a post-test questionnaire.

Conclusions from Data & Testing


iPhone: Find email

Participant 2, 4, and7 failed to find the email on the iPhone in Task1. Participant 7 did not know the “flick” gesture revealed more messages and Participants 2 and 4 suggested they would use search to locate the email in this situation rather than scanning for keywords. 

iPhone: Enlarge text

Participants 6 and 8 were unable to enlarge text on the iPhone. However, participant 6 eventually figured out how to awkwardly pinch and spread to zoom in out, whereas, participant 8 never discovered the gesture but located the “-A” and “A+” icons quicker than any other participant when testing the BBC app.

iPhone: Reply all

6 of the 8 participants successfully identified the “ReplyAll” icon. The remaining 2 were not given the task, therefore, there was a 100% success rate despite user trial and error when exploring the hidden meaning or actions of the Mail icons.

iPhone: Copy/paste

Copy/paste was difficult to test because of the imperfect interaction. Users struggled to get the feature to work as desired, but 4 of the 5, or 80%, participants were able to successfully copy and paste text into a new message.

iPhone: Delete
Participant 4 was the only user who gave up on the delete task. The other 7 participants successfully discovered delete through various methods. The side-to-side swipe was used in 2 cases and “edit” in 3 cases.

Additionally, participants 7 and 8 uncovered a new method, which involved clicking the “move this message to a new mailbox” icon and then selecting the “Trash” icon.

iPhone: New email
There was a 100% success rate in this task despite many errors during the discovery process. It wasoften difficult for participants to associate “Create New” or “Compose Email” with a square icon with a pencil through it, .

iPhone: Refresh inbox

The first 3 participants were tested on refreshing email, but it became evident that users identified refresh easily and subsequent participants were not tested.

iPhone: Refresh BBC
The “pull to refresh” in the BBC app only had a 25% success rate or 2 out of 8 participants. More advance users, participants 1 and 2, were able to uncover this refresh method despite the most advance user of the study, participant 5, was unable refresh the home BBC screen.

iPhone: Find More stories

5 of the 8 participants were able to reveal more stories in the BBC application when scrolling left to right on the news categories. Participant 1, 3, and 5 represented the most experienced user base and were the ones who failed the task. Alternatively, users with no experience, 4, 6, 7, and 8, easily scrolled left to right without hesitation to reveal more news articles. Participant 8 mentioned he saw the side-to-side action “On TV”.

iPad: Find email

In contrast, there was a 100% success rate on the iPad compared to the 62.5% success rate on the iPhone. The 37.5% or 3 participant difference was attributed to the gained familiarity with the iOS interface when gestures were discovered in other tasks.

iPad: Enlarge text

6 of 7 participants were able to en large text on the iPad, but participant 8 was unable to discover the pinch and spread gestures continued from the iPhone, whereas, participant 6 discovered an awkward method of spreading the index and middle finger to achieve the pinch and spread gesture commonly used with the thumb and index finger.

iPad: New email & Refresh inbox



Apple says on one multi-touch device that “it supports a full set of gestures, giving you a whole new way to control and interact with what’s on your screen. Swiping through pages online feels just like flipping through pages in a book or magazine.” Gestures are promoted as more natural and more akin to our interactions with commonplace physical objects.

Our testing reveals that test participants unfamiliar with touch devices do not find gestures and icons immediately understandable. Simple tasks often caused a great deal of frustration as methods and use strategies inherited from desktop or laptop use did not translate to the novel mobile environment. Overall, tasks performed on the iPad were completed more quickly and with greater ease. Participants who showed the largest discrepancy in satisfaction between the iPhone and iPad cited the larger screen size as the primary reason.

Except for one user who came to the test with extensive iPhone experience, test participants did not try out different gestures to complete the tasks. While all commented that they understand how gestural interactions worked, they seemed uninterested in seeing how different gestures would work to complete the tasks. While this may be in part to the social pressure of the test, to ‘not looking stupid’, test participants’ first looked for buttons to complete the tasks. If the user did not find a button in the area immediately visible on the screen, he or she would scroll down to the end of the page in search of a button in nearly every case.

The BBC website refresh task illustrates this. The two buttons as the top of the screen “LIVE RADIO” and “Edit” (Figure 1) do not seem to have any connection to refreshing the page. 5 out of 6 users ended up pressing the “Edit” button after scrolling to the bottom of the page and not finding any other candidate for refreshing the page. Out of these 5 participants, only one discovered that refreshing the page is done by dragging the news stories down and

Similarly, it is possible to delete an email on both the iPhone and iPad Mail applications by swiping the Inbox entry left. The four buttons available on the iPhone version (identical to the iPad) are shown in Figure 2. Test participants varied in how many or which buttons were tried, but all (again, except the experienced iOS participant) explored deleting the email using these buttons. Gestures were not attempted.

Basic Recommendations

The difficulty users experienced with gestures and icons exposes the underlying tension between discovery and ease of use. While every designer must cope make decisions related to this theme, the issue is all the more critical for mobile applications. Their low cost and the ease with which one can acquire them also means that they are that much easier to dispose of if the user find them lacking.


6 out of the 8 test participants were familiar with two gestures: the one finger swipe down to scroll and the pinch and drag to zoom. Scrolling has a high-level of discoverability, since the interaction is accessible nearly everywhere on the interface. When asked why participants knew how to zoom, nearly all responded that they had seen it on the early iPhone commercials. This points to the power of explicitly showing gestural features through video. Gesture discoverability would be greatly improved through in-application orientation videos. Another possibility is some sort of tutorial where different interface elements could be clearly explained.


Another possibility for increasing discoverability would be to provide two modes of use. One, which highlights the different features and interactions and one that is optimized for on-going ease of use. Smart phones have moved beyond the Early Adopter segment in the product life-cycle, and many users needs do not advance beyond basic functionality at the introductory level. However, there seems to be an experienced niche group who uses the devices complexity who might choose to sidestep ‘training wheels’, but it would invite a wider audience to explore deeper features of the application.

One implementation of this would be to employ a feature borrowed from the desktop environment – the task menu. Again, experienced users might prefer the direct action of icons well understood, but novice or less tech-savvy users would find a simple to understand method for completing common tasks.

AuthorDavid Brahler