I led conversational experience design of a solution for the blind and visually impaired, helping them perceive their physical surroundings using a head-mounted camera and an AI-driven voice user interface.

Product Demo Video directed and edited by Giada Sun



Interaction Design

Conversation Design

System Design


6 months


Giada Sun

Weixi Zhang
Yagmur Erten


Adobe Premiere
Adobe After Effects

Using participatory design methods, we worked closely with the blind and visually impaired community through the whole design process, making sure the design really meet their requirement.

Worked with advisors from Amazon, we explored our problem area and pivoted to the final concept through remote moderated research, ideation, Wizard of Oz prototyping, usability testing, and creating VUI dialogue flows for the key interactions and considering error cases.

Designed alongside the delightful Weixi Zhang and Yagmur Erten, also give a shout out to Nimmi Daripineni and Zara Abraham for being the best actors and Nadia Kaheil’s most heartwarming voice in the demo video.

The project is advised by Amazon

Capstone Project of Master of Human-Computer Interaction and Design, University of Washington, Seattle


According to CDC, there are around 4.3 million people in the US with visual impairment who have this kind of problem everyday. They often face daily obstacles to perceive their environments in a world that mostly is designed BY and FOR the sighted people, such as reading printed instructions, sorting mails, and differentiating medicine bottles.


We found out that currently existing computer vision technologies, such as Seeing AI, are pretty helpful to the blind users while recognizing objects or texts. However, there are still some usability issues, such as requiring users to aim the camera on an object directly and taking a clear picture for successful recognition.

Seeing AI requires users to aim the camera on an object for recognition

Design Challenge

How might we assist the blind in retrieving relevant product information through an independent experience?

The Response

Iris is an AI-driven smart glasses that aims to empower people who are blind to recognize objects and texts through intuitive verbal controls

How it works

Simply hold an object

Iris aims to remove the restrictions that current technologies have by allowing users to recognize objects through simply holding or pointing at an item with their hands.

Ask for desired information

Users can ask Iris for specific information from the item that they are holding. For example, the ingredient of the sanitizer gel.

Receive information directly

Iris’ sound system and haptic feedback allow users to ask and receive the information through headphones if they need privacy.

Research Process

Research goals

We were initially interested in analyzing the blind and visually impaired’s online shopping behaviors. We started our research process with literature review to understand what has been done in the field followed by the interviews with experts. Then, we proceeded with the primary research to understand the usage of assistive technology.


Subject Matter Expert interviews

After doing literature review, we gained a general understanding of how blind people use the Internet. With that knowledge, we spoke with experts in this field to expand our understanding of blind people’s online shopping behaviors.

The key takeaways from expert interviews are:

Blindness is a spectrum, narrow down the target user.
Don’t assume how the blind use technology.
Don’t exclude those who have lower tech proficiency.
Examine the current accessibility guidelines.
Start talking with the users NOW!

Phone interviews with directed storytelling

At first, we wanted to understand our user's online shopping journey by collecting first-hand data of blind users' past experiences, opinions, and attitudes through interview. We also wanted to identify any advancements and frustrations that exist within the current technologies.


Remote Moderated Studies

Through Zoom screen sharing, we observed participants online shopping behaviors such as browsing and selecting both familiar and unfamiliar items. Our goal was to gain a deep understanding of blind people’s online shopping behaviors and the pain points of using the tools, to spot any missing features that could ease their process of online shopping.

From the primary research, we concluded four insights:

Intentional Shopping For Efficiency

Most blind people are intentional shoppers, who are not interested in browsing items without purpose or taking website’s recommendations as reference.

Familiarity and Recurring Items

Familiarity with items and websites during shopping is crucially important to blind customers.

Relevant and Specific Information

Blind users are specific about relevant information about products: they use screen readers to skim websites through only reading key elements.

Inter-dependent with Assistive Tech

Most blind people like using VUI devices for short commands to complete every day tasks, such as setting alarms, checking time and the weather.

Design Process


What do our users value?

Through the research, we found out that our participants value efficiency, relevance, and independence. In other words, they prefer to receive relevant information efficiently and independently while shopping online and in their daily scenarios. We followed these three design principles to narrow down the ideas in ideation.


Gathers relevant information in time rather than speeding up the process of current technology, such as screen reader.


Extracts the information that the users want or need to know. Relevance and efficiency have to go hand in hand to be impactful in the context.


Completes daily tasks by themselves using technology before asking help from other people.


We came up with 60 ideas that can support people who are blind and visually impaired with their daily tasks. We classified these ideas into Searching Products, Information Gathering, Information Exchange, Comparison, Connection with Sighted People, Description Generation, Privacy Control, and Innovative Controller.

We came up with the initial 60 ideas within a day


We each created an excitement-feasibility two-dimensional chart to narrow down our preference of the ideas. The goal is to find out the ideas which we all are thrilled to work on and are possible to be realized in terms of the current technology. We eventually picked up 3 ideas.

Downselected 3 ideas

We brought these ideas to our participants and asked for their feedback.

Initial concept

We landed on the idea of the smart glasses, that has an imbedded camera and voice user interface system, where users can hold an item and request glasses to recognize the item by directing their heads to their hands.


To present the use cases of the smart glasses idea, I drew out the storyboard showing how a user extracts specific information using our solution. It also shows one of the side features of easily ordering staple items.

The man who is blind talks to his smart glasses to make sure whether the milk is expired


Remote usability testing

During COVID-19, even though we wildly thought about bringing the participant to a park and doing the usability tests in a social distanced way, WE DID NOT DO THAT! Since we understood that we cannot test in person yet, we push our boundaries to be creative in remote situations.

Focusing on at-home experience

Even though we see a potential of using Iris in other scenarios in the future, for this project, because we can only test remotely when participants are at home, we want to focus on the interaction design for only the “at-home” experience.


Prototype segments

At this phase, we knew that our product was going to be a wearable device with AI camera and voice assistant. Our objective for testing was to understand the conversational and interactive experience, instead of the ergonomic design of the wearable. Therefore, we chose these three elements as our prototype.

Dialogue flow & sample conversations

I created the voice user interface’s dialogue flow, and my teammate, Weixi, followed the dialogue flow to create a cheat sheet and acted as the AI and the voice assistant.

The full dialogue flows includes recognizing objects/texts and identifying specific information from an object. I also created sample conversations, including saving items on the glasses for easy future recognition and calling family and friends through the glasses. Possible error cases are also highlighted in the user journeys.

Dialogue flow of object recognition

Low-fidelity prototype

Our objective for testing was to understand the conversational and interactive experience, instead of the ergonomic design of the wearable, which is why we choose to use a head-mount to stabilize the smart phone on the participant. We shipped the headset to 3 participants. We video-called them and once they wore the headset with a phone, we could see the items they were holding.

Remote usability testing

We used the Wizard of Oz technique and my teammate, Weixi, acted as Iris, following the cheat sheet of the dialog flows that we prepared to speak the information that is in the users’ camera’s frame. Another teammate and I acted as the facilitator and note-taker. With that, we tested both success and error cases with the participants.

We tested through video call and head-mount phone with the dialogue flow

Scenario validation

To validate the scenarios people would like to use our product, we conducted an online survey that got 35 responses, to understand the importance of each use cases, and explore scenarios people encounter that we might have missed. We took the provided feedback as a reference while improving our product’s functions and design.

35 survey responses help solidifying the use cases

Design & iteration

Key features

Identify objects and short texts

Iris allows users to extract specific information from an object. For example:


Adding tap control

From the usability testing, we found out that in some situations, the participants want to be able to simply receive information without having to request it out loud. Therefore, we includes a tap function on Iris, where users can tap the leg of the glasses and activate Iris to identify an item subtly and quickly.

Read documents

Another feature of Iris is to read long documents that have paragraphs, such as books and instructions of food material.


Leveraging existing habits

We leverage users’ existing habit of using a screen reader, to jump between different paragraphs. User can tap or say the request verbally to skip or revisit any part of the document. Users can also adjust Iris’ pace of speaking to a comfortable level to read long documents faster or slower.

Fast recognition & online shopping

With Iris, users can save an item with a specific name, for faster recognition in the future, such as medicine bottles that they use often. For staple items, they can also use Iris to recognize it, and add it to the linked shopping account.

Rotate the item for saving

Companion app

Personal setting

Other than a button for quickly finding the glasses and the onboarding tutorial, the most important part of the App is the setting. In the Setting, users can adjust the audio & haptic feedback, speed of speech, and lighting based on their preferences.

They can also manage the contact information and shopping account information here in the app.

Error handling

Image framing

When Iris cannot detect hands in the frame, or do not have a full object in the frame in order to find the information the user is asking, Iris will provide a verbal and an optional haptic feedback, to the guide user.

Demo video of handling framing error

Dark environment

If Iris fails to recognize an object due to a dark environment, it will request permission from the user to activate the flashlight that is on the glasses briefly. Iris will turn off the flashlight as soon as the identification is completed successfully, so the light will not influence other people in the same environment.

Demo video of turning on the flashlight

Connect with contacts

In any case, if the user has doubts on the information they received from Iris, they can request to be connected with a family or a friend. The connected contacts will have access to Iris’s camera and can communicate with the user through a video call.


Adding rotation button

From our usability testing, we learned that the participants sometimes could hold the items upside down, or in some other scenarios a horizontal view could be better for the contacted friend to help the user. For such cases, we added a button that the contacted person can rotate the screen on the video for an easier viewing experience.


Disability is a spectrum. Just go talk to the users without making too many assumptions. It's better to design not only for but also with those people with disabilities.
Human beings have an amazing capability to adapt to different situations. By designing for people with a permanent disability, someone with a situational limitation can also benefit.
In the future, we are looking into ways to make the product available to those non-tech savvy users, who we could not reach out to in the research and design process.

More design process is coming soon

Next project

Creating a cohesive experience for museum visitors to purchase tickets online