Service Design | Design Research | Interaction Design

SignSavvy

Problem

Despite the fact that ASL interpreters play a critical role in bridging Deaf and Hard of Hearing (DHH) individuals with the hearing world, Interpretation is a highly demanding job that often requires interpreters to work in a subject matter requiring expertise and vocabularies that they might be unfamiliar with; through research, we uncover the need for technological opportunity to assist interpreters with delivering interpretation in these highly specialized circumstances.

Role

Design Researcher
Project Manager
Concept and Interaction Designer

Team

Kevin Oh, Cecilia Zhao
Michael Smith (Advisor)

Duration

2019-2020 September - March
20 Weeks

Tools

Figma, Photoshop, Illustrator, Ipad, After Effects, pen and paper

Solution

SignSavvy is an app that assists novice ASL interpreters by providing them with live imagery and text when unfamiliar terms are used in simultaneous interpretation sessions. It makes easy to keep track and organize their scheduled sessions, and learn vocabulary that might arise in future sessions to help interpreters build repertoire and language in specialized fields.

Skills Involvement

Timeline (20 Weeks)

Background

To help Deaf and hard of hearing community navigate around the hearing world, there is an important group of people can be easily overlooked. — and they are the sign language interpreter. They are not only there to deliver a message but they are bringing two parties that have very different cultural backgrounds together.

01

Generative Research

What challenges do interpreters face when they are delivering simultaneous interpretation?

How can technology be used to assist interpreters?
What are the factors that influence the message delivery in an interpretation process?

Methods

We spend the first 8 weeks to conduct field research in the problem space and a set of research questions that we want to find out. We created a study guide that outlined three different types of activities - Direct Observation, where we observed the simultaneous interpretation process at an event, Semi Structured Interviews, and probe activity as a second-round to solicit behaviors, attitudes and understand the interpretation process from our participants. this gave us a deep dive into working closely with three types of stakeholders - Researchers, Deaf, and Hard of Hearing individuals and Interpreters.

1 Direct Observation
9 Semi-structured Interviews
3 Card sorting / Probes

11 ASL Interpreters
2 Deaf Individuals
2 Subject Matter Experts

01  Direct Observation

We attended the UW Presidential Address where we observed how interpreters work together in teams and help us see what the simultaneous interpretation process was like - I documented the session with sketch notes. and we also collected recordings and photographs as our data.

02  Interviews

I led semi-structured interviews with experience ASL Interpreters, Experts, and Deaf Professor. The interviews with the Subject Matter Expertise (SME) first help us understand and narrow down our problem space, and our interview questions for interpreters help us uncover attitudes and pain points during the interpretation process.

03  Probe

I created the illustrated scenarios with additional descriptions of scenarios so that there is a more specific context for the interpreter to talk about specific challenges they face within each scenario. During the activity, We asked the participant to speak about if they had superpowers in that context what would they have to help better deliver their services by drawing out in a blank sticky note for each scenario.

02

Synthesis

“We're often on our phone as we're Googling stuff... and they can tell when we don't know the word. Our team will google it really fast and then sign it to us.”

—P3

Affinity Diagraming

We externalized our data from each of the interviews used affinity diagram to group similar themes. From there we went through a rigorous process of sorting, and through abductive process of synthesising to come up with our key insights.

Journey Mapping

We thought that a journey map would be the best for us to map out what the interpreter goes through. we broadly defined our process into pre-interpretation, during interpretation, and post-interpretation. Because we found so many pain points, we decided that we want to focus on the interpretation process where it is the most difficult and have the greatest opportunity for improving the quality of the interpretation services. We want to focus on this because we also find its most complex and stressful.

Key Insights

1.

Technology can assist but not replace interpreters because every interpreter signs differently.

“Each deaf person does it differently. There's just so many variables going on. How you express it and consume is so rooted in a person's creative mind and how they embody it and their technical output. I mean, it's one of the most human experiences.” —P3

2.

Interpretation is stressful because it takes a heavy cognitive load and requires multitasking and multi-level processing.

“Typically work in teams because of the physically tiring and just constant thinking and interpreting. They have proven after 20 minutes of interpreting that the interpreter will start to omit information.” —P10

3.

Interpretation is optimal when the interpreters are well-prepared and better informed.

“We're often on our phone as as we're Googling stuff... and they can tell when we don't know the word. Our team will google it really fast and then sign it to us.” —P4

4.

Interpreters not only deliver the message but also try to make sure the deaf individual or party understands the message and context.

“I communicate constantly to make sure they are on track with back-channeling.” — P3

5.

Despite the proficiency of the interpreters, there are external factors outside of the interpreter’s control that can make the interpretation process difficult.

“Especially if it's a new speaker, someone nervous and they can't slow down.” —P8

Design Principles

1.

Minimize Stress

Help relieve stress and intensity during the fast speed simultaneous interpretation.

2.

Informative

Provide information and context to interpreters ahead of time to help them better prepared for the interpretation session.

3.

Assistive

Help facilitate interpreters on the interpretation process instead of replacing them.

4.

Personal

The design should consider individual users and their needs, and also uphold the human quality of those individuals involved in the interaction.

How might we assist novice interpreters to better prepare and deliver in a simultaneous interpretation session?

03

Ideation

Through an ideation process that used braiding and crazy 8, we narrowed down to 6 core concepts that fit within our HMW statement and design principles.

1 - EarBuds

The special earbud amplifies the voice of speakers by connecting to all the microphones. It also has a voice cancellation feature to reduce other external noises.

Goal
The earbud helps interpreters better focus on the content by amplifying the voice of people who are speaking and reducing external noises.

2 - AR Voice Tracking Cameras

An AR system of multiple three-sided cameras that can be placed in multiple locations. The cameras would point to the source of a spoken voice, providing a live visual of the speaker to the wearer of the AR device.

Goal
To provide a visual of the speaker for interpreters so they can get a better idea of the body language and expression of the speaker and the environment.

3 - Stress-detect Microphone 

This idea comes in two parts, one is that the interpreter wears a clip-on that detects their heartrate and brainwave to indicate their stress level. The output is displayed through the speaker’s microphone, which has a digital interface that changes color in response to the stress-level.

Goal
The intent was for the speaker to be mindful of the presence of interpreters during a conversation.

4 - AR Captioning & Dictionary

An AR system that can visually caption recorded audio and provide definitions and contextual information for certain highlighted keywords or terms. The captioning and definition would appear with text or images to the wearer of the AR device.

Goal
To inform interpreters during their interpretation job with any unfamiliar content and accurately provide visual aid of what captures what is being spoken.

5 - Info APP

The app includes features such as requesting scripts before the interpretation session, video call with DHH individuals, checking and scheduling for interpretation services, etc.

Goal
This idea aim in helping interpreters become more familiar and well-prepared for the upcoming interpretation session.

6 - Motion Bookmark

Special food wear that uses foot motion sensing to allow interpreters to bookmark parts of the conversation he/she missed during a presentation. The interpreter can later go back during the break and make clarifications.

Goal
The intent is for the interpreter to mark and record parts of the presentation or conversation during interpretation without interfering with the speaker in situations like a presentation.

Design Concept

An app concept paired with physical glasses that provides a line of sight information when the interpretation is happening, Interpreters can put bookmarks where they need to go back and review the conversation. the industry-specific vocabularies are picked up through a machine learning system. that have the user input a view question before going into the session.

(A concept image of the initial contact AR idea)

(I illustrated concept design and the concept storyboard)

04

Prototyping Round 1

(we tested with 3 interpreters and 1 Deaf Individual and 1 DHH Researcher)

Goal:

For this first round of prototype testing, we want to understand whether or not the format for a screen-based real-time backchanneling would work well for interpreters during a session. We also want to understand what format for feedback would interpreters consider the most effective.

What we did:

We mimicked the simultaneous interpretation session through role-playing with interpreters and used iPad to a position at their line of sight while the interpreter signs. We prepared three types of scripts specifically in highly specialized fields such as medical, computer science, and philosophy and tested with each of the 5 participants. During the session, as the speaker is speaking, we play Wizard of Oz to simultaneously display difficult vocabularies to the interpreter.

Behaviour Prototype

3 Scenarios
Caption + Video + Images

- Real-time Captioning
- Terminology/jargon is highlighted based on the user’s proficiency in the topic area

(Participant Deaf Linguistic Professor and interpreter)

“There have been incidents of unqualified interpreters in dire situations with terrible outcomes, even deaths.” -- P10

Findings

Interpreters would not read the captions but having terms highlighted and show up is helpful because it helps them see the spelling.

Most of the interpreters said that images of terms that show the relationships are more helpful than images of the specific term.

What interpreter prefer as a way of displaying these terms depends on the situation and context.

Video Clips of Sign Language Vocabulary is distracting and not helpful because signing is very personal; it requires understanding, not mimicking.

ASL Vocabulary and terminology can vary depending on slang, dialects, and context.

This software can provide value for training ASL interpreters/novice interpreters

Concept Iteration

(initial line of sight display with interpreter videos)

Update the design to focus on display with imagery that describe the vocabulary or its relationships

Having a form factor for the physical artefact does not provide accessible to all interpreters, and might increase social stigma associated with interpretor’s expertise level. Therefore, from a product perspective, it will be much more accessible to interpreters to have this product as an app, this is also where we have the opportunity to integrate a scheduling and review system into the same platform.

Storyboard

(I drew out the final storyboard to illustrate our flow of how SignSavvy is used.)

05

Prototype Evaluation Round 2

(we tested with 3 interpreters and 1 Deaf Individual and 1 DHH Researcher)

(onboarding workflow - credit Cecilia)

Goal:

In our second round of prototype testing, we focused more on the full experience from pre-session to during the session. Our goal is to have fresh perspectives from interpreters and DHH to have them walk through the process of creating, interpreting, and saving vocabularies. Their insight will help us refine and produce the final prototype.

What we did:

We created low-fidelity wireframes for the pre-session and created higher fidelity wireframes for testing. We tested with 2 DHH with interpreters and 1 experienced interpreter and 1 young interpreter. We set up the scenario for them and performed usability testing focusing on the pre-session. We also asked them about their experiences using the app during the session.

Wireframes

(wireframe workflow - credit Kevin and Cecilia)

(mid fidelity screen - credit Kevin and Cecilia)

Findings

Interpreter's understanding of "familiarity" as an input criterion is subjective, rather associating with keywords can help generalize more personalized results.

Interpreters often do extra work to look up information beforehand, the system has an opportunity to generate predicted vocabularies base on topic.

Having synonyms as an option can be a quick way to capture the meaning of a vocabulary.

Interpreters should also be able to provide feedback to improve the Machine Learning system for better vocab recommendations not just passively receive recommendations

Real Time Feedback

Manage and Review Session

Vocab Review and Contacts

01

Real-Time Feedback

Individual View

Once the interpreter starts the session, SignSavvy starts to pick up transcribed words from the conversation and display it in a preferred way to the interpreter. 

Share Content with DHH

The interpreter can also sync their screen with the deaf and hard of hearing individual because DHH although they might not need this would still be helpful sometimes for them to have this visual access.

Team Interpretation

Interpretation sometimes happens in teams for a session longer than 25 mins, and the job of the "off-interpreter" can look for or feed the main interpreter. their screen on the right will also populate with words that the on interpreter sees to get an overview of the session and look up any terms that the on-interpreter didn't' catch.

Information Architecture

(I created the IA based on user feedback for our final interface design)

02

Manage and Review Session

Add Session

We created filters and display options to fit the preferences for interpreters. When the interpreter received a new job, He/she proceed to add a new session, input the details of people, date, notes as well as adding potential vocab to assist settings for the system.

Review

Once the session is added, they can easily access the information by tapping in to see the details of the session, with key information such as location, collaborators or people involved as well as notes for the interpreter. They can also see a few keywords identified at the bottom, which helps interpreters have a better sense of the session. These vocabularies provide the system with data to feed into the machine learning algorithm so that difficult words associated with those vocabularies Will be picked up and added during interpretation sessions.

Share

Because interpretation usually happens in teams for a session longer than 25 mins, the individual interpreter can share their notes with their partner interpreters so they can better prepare for their session together.

Information Architecture

(I created the IA based on user feedback for our final interface design)

03

Vocab Review and Contacts

Review Vocab by Class

After a session, the user is able to look up previously interpretation sessions by going into the vocab tab, a record of the vocabulary display during the session can be found here.

Filters

Also in this tab, we created filters to accommodate various user preferences, and that they can select based on the date of previous sessions to review vocab.

Review Contacts

Lastly, the interpreter also has the option to quickly access their saved collaborators, clients. Because interpretation services are very personalized, having this access can allow interpreters to quickly remind themselves about past sessions and notes related to the individual for better preparation for coming sessions.

Information Architecture

(I created the IA based on user feedback for our final interface design)

Reflection

We should not consider Deaf and/or Hard of Hearing as a disability, rather, it is a different way to navigate the world. Deaf and Hard of Hearing signifies a form of cognitive diversity rather than a “disability”.

Designing with rather than designing for a population that you are not subject matter expert in can quickly give you the insight into the real problem.

Gathering multiple perspectives is valuable to make sure our design considers the end-user as well as the implication of these user needs.

Taking this Forward...

We will need to build out a full working prototype and take it to a specific context such as a classroom lecture to test with more participants.

Leverage developer and software engineer to establish an algorithm that refines and trains data based on subject matter keywords. At the same time, we will need to start gathering enough volume of data in the subject field and lecture materials to train the system.

Establish privacy and confidentiality measures to ensure the information within the platform is security as it will contain a lot of personal data.