Product Demo Video directed and edited by Giada Sun
Worked with advisors from Amazon, we explored our problem area and pivoted to the final concept through remote moderated research, ideation, Wizard of Oz prototyping, usability testing, and creating VUI dialogue flows for the key interactions and considering error cases.
Designed alongside the delightful Weixi Zhang and Yagmur Erten, also give a shout out to Nimmi Daripineni and Zara Abraham for being the best actors and Nadia Kaheil’s most heartwarming voice in the demo video.
Capstone Project of Master of Human-Computer Interaction and Design, University of Washington, Seattle
According to CDC, there are around 4.3 million people in the US with visual impairment who have this kind of problem everyday. They often face daily obstacles to perceive their environments in a world that mostly is designed BY and FOR the sighted people, such as reading printed instructions, sorting mails, and differentiating medicine bottles.
We found out that currently existing computer vision technologies, such as Seeing AI, are pretty helpful to the blind users while recognizing objects or texts. However, there are still some usability issues, such as requiring users to aim the camera on an object directly and taking a clear picture for successful recognition.
Seeing AI requires users to aim the camera on an object for recognition
Iris is an AI-driven smart glasses that aims to empower people who are blind to recognize objects and texts through intuitive verbal controls
Iris aims to remove the restrictions that current technologies have by allowing users to recognize objects through simply holding or pointing at an item with their hands.
Users can ask Iris for specific information from the item that they are holding. For example, the ingredient of the sanitizer gel.
Iris’ sound system and haptic feedback allow users to ask and receive the information through headphones if they need privacy.
We were initially interested in analyzing the blind and visually impaired’s online shopping behaviors. We started our research process with literature review to understand what has been done in the field followed by the interviews with experts. Then, we proceeded with the primary research to understand the usage of assistive technology.
After doing literature review, we gained a general understanding of how blind people use the Internet. With that knowledge, we spoke with experts in this field to expand our understanding of blind people’s online shopping behaviors.
The key takeaways from expert interviews are:
At first, we wanted to understand our user's online shopping journey by collecting first-hand data of blind users' past experiences, opinions, and attitudes through interview. We also wanted to identify any advancements and frustrations that exist within the current technologies.
Through Zoom screen sharing, we observed participants online shopping behaviors such as browsing and selecting both familiar and unfamiliar items. Our goal was to gain a deep understanding of blind people’s online shopping behaviors and the pain points of using the tools, to spot any missing features that could ease their process of online shopping.
Most blind people are intentional shoppers, who are not interested in browsing items without purpose or taking website’s recommendations as reference.
Familiarity with items and websites during shopping is crucially important to blind customers.
Blind users are specific about relevant information about products: they use screen readers to skim websites through only reading key elements.
Most blind people like using VUI devices for short commands to complete every day tasks, such as setting alarms, checking time and the weather.
Through the research, we found out that our participants value efficiency, relevance, and independence. In other words, they prefer to receive relevant information efficiently and independently while shopping online and in their daily scenarios. We followed these three design principles to narrow down the ideas in ideation.
Gathers relevant information in time rather than speeding up the process of current technology, such as screen reader.
Extracts the information that the users want or need to know. Relevance and efficiency have to go hand in hand to be impactful in the context.
Completes daily tasks by themselves using technology before asking help from other people.
We came up with 60 ideas that can support people who are blind and visually impaired with their daily tasks. We classified these ideas into Searching Products, Information Gathering, Information Exchange, Comparison, Connection with Sighted People, Description Generation, Privacy Control, and Innovative Controller.
We came up with the initial 60 ideas within a day
We each created an excitement-feasibility two-dimensional chart to narrow down our preference of the ideas. The goal is to find out the ideas which we all are thrilled to work on and are possible to be realized in terms of the current technology. We eventually picked up 3 ideas.
Downselected 3 ideas
We landed on the idea of the smart glasses, that has an imbedded camera and voice user interface system, where users can hold an item and request glasses to recognize the item by directing their heads to their hands.
To present the use cases of the smart glasses idea, I drew out the storyboard showing how a user extracts specific information using our solution. It also shows one of the side features of easily ordering staple items.
The man who is blind talks to his smart glasses to make sure whether the milk is expired
During COVID-19, even though we wildly thought about bringing the participant to a park and doing the usability tests in a social distanced way, WE DID NOT DO THAT! Since we understood that we cannot test in person yet, we push our boundaries to be creative in remote situations.
Even though we see a potential of using Iris in other scenarios in the future, for this project, because we can only test remotely when participants are at home, we want to focus on the interaction design for only the “at-home” experience.
At this phase, we knew that our product was going to be a wearable device with AI camera and voice assistant. Our objective for testing was to understand the conversational and interactive experience, instead of the ergonomic design of the wearable. Therefore, we chose these three elements as our prototype.
I created the voice user interface’s dialogue flow, and my teammate, Weixi, followed the dialogue flow to create a cheat sheet and acted as the AI and the voice assistant.
The full dialogue flows includes recognizing objects/texts and identifying specific information from an object. I also created sample conversations, including saving items on the glasses for easy future recognition and calling family and friends through the glasses. Possible error cases are also highlighted in the user journeys.
Dialogue flow of object recognition
Our objective for testing was to understand the conversational and interactive experience, instead of the ergonomic design of the wearable, which is why we choose to use a head-mount to stabilize the smart phone on the participant. We shipped the headset to 3 participants. We video-called them and once they wore the headset with a phone, we could see the items they were holding.
We used the Wizard of Oz technique and my teammate, Weixi, acted as Iris, following the cheat sheet of the dialog flows that we prepared to speak the information that is in the users’ camera’s frame. Another teammate and I acted as the facilitator and note-taker. With that, we tested both success and error cases with the participants.
We tested through video call and head-mount phone with the dialogue flow
To validate the scenarios people would like to use our product, we conducted an online survey that got 35 responses, to understand the importance of each use cases, and explore scenarios people encounter that we might have missed. We took the provided feedback as a reference while improving our product’s functions and design.
35 survey responses help solidifying the use cases
Iris allows users to extract specific information from an object. For example:
From the usability testing, we found out that in some situations, the participants want to be able to simply receive information without having to request it out loud. Therefore, we includes a tap function on Iris, where users can tap the leg of the glasses and activate Iris to identify an item subtly and quickly.
Another feature of Iris is to read long documents that have paragraphs, such as books and instructions of food material.
We leverage users’ existing habit of using a screen reader, to jump between different paragraphs. User can tap or say the request verbally to skip or revisit any part of the document. Users can also adjust Iris’ pace of speaking to a comfortable level to read long documents faster or slower.
With Iris, users can save an item with a specific name, for faster recognition in the future, such as medicine bottles that they use often. For staple items, they can also use Iris to recognize it, and add it to the linked shopping account.
Rotate the item for saving
Other than a button for quickly finding the glasses and the onboarding tutorial, the most important part of the App is the setting. In the Setting, users can adjust the audio & haptic feedback, speed of speech, and lighting based on their preferences.
They can also manage the contact information and shopping account information here in the app.
When Iris cannot detect hands in the frame, or do not have a full object in the frame in order to find the information the user is asking, Iris will provide a verbal and an optional haptic feedback, to the guide user.
Demo video of handling framing error
If Iris fails to recognize an object due to a dark environment, it will request permission from the user to activate the flashlight that is on the glasses briefly. Iris will turn off the flashlight as soon as the identification is completed successfully, so the light will not influence other people in the same environment.
Demo video of turning on the flashlight
In any case, if the user has doubts on the information they received from Iris, they can request to be connected with a family or a friend. The connected contacts will have access to Iris’s camera and can communicate with the user through a video call.
From our usability testing, we learned that the participants sometimes could hold the items upside down, or in some other scenarios a horizontal view could be better for the contacted friend to help the user. For such cases, we added a button that the contacted person can rotate the screen on the video for an easier viewing experience.
