Iterative User Evaluation of Inclusive Imagery
Overview
Emily Blank, Art Director on Google's Material Design team, commissioned artists to create inclusive illustrations of users with disabilities and other marginalized identities (The Avatar Project) that Google could use for internal mock-ups, advertisements and product photos. Depicting marginalized identities in an inclusive manner is important for building goodwill and loyalty among user groups who have historically been left behind. The Disability Market is huge, with an estimated value of $7 trillion worldwide (WebAIM). So increasing brand and product value for these users is deeply important. But it's not just about displaying support, representations of disability without making the effort to assure the images are actually accessible can end up backfiring in a major way. Working with Emily and her colleague, User Researcher Michael Gilbert, I conducted a study to evaluate and gather feedback on whether users actually felt the images of users achieved their inclusive goals.
Problem Statement
According to users with disabilities, how can fictionalized depictions of disabilities be best made inclusive and accessible (through appropriate image descriptions)?
Methods:
Facilitating focus groups where you are discussing visual artifacts with people who have limited to no vision was an interesting and fun challenge. I had the help of 1 to 3 co-facilitators (depending on the session) during most of the 9 focus groups we ran, so we ended up setting up the remote sessions so there was a shared screen with the images (for sighted participants) and we also dropped the written alt text in the Zoom chat window for screen reader users to review at their own pace. This allowed everyone to contribute and discuss their own feelings of the representation. It also meant a lot of moving pieces to juggle, all with just a single small laptop screen, so sessions were sometimes stressful when co-facilitators were not available to help. Still, we managed to get good conversations out of these sessions.
To complement the focus groups, and to accommodate the rapidly changing schedules participants had at the beginning of the COVID lockdowns, we also included two types of interviews. One followed roughly the same procedure as the focus group, simply focusing on gathering feedback from one rather than multiple participants at a time. The other type of interviews were shorter follow-ups with users to discuss their personal identities and feelings about representation. By having these as one-on-one sessions, people were more comfortable discussing details of their personal identities, and I was also able to probe further into some of the things said in the focus group sessions. Between these two types of interviews, we ended up collecting 19 user interviews for this study.
This was an interesting and unique part of this study in particular. The focus groups, along with being general discussions of imagery, inclusivity, and representations of users in corporate design spaces, were also partially intended to function as opportunities to co-create the alt text for these images. This meant there was a lot of iteration, probing about specific wording, details, order, etc. and that we ended up with very thorough descriptions because we made sure they integrated the key elements brought up by different users.
After the sessions were conducted, the recordings were automatically transcribed, then edited. Then I led a team of four researchers (including myself) in conducting deep thematic analysis by going line-by-line through the transcripts. We designed our own low-budget and highly adaptive method for tracking our codes using only Google Sheets, which allowed us to keep the training overhead down and focus more on making our way through the large amount of data from sessions. We then grouped the codes into themes through affinity diagramming using MURAL’s digital whiteboarding tools.
Outcomes:
Communicating The Research to Different Audiences
There have been a total of three pieces published about this project, all of which I have first-authored. The first a conference paper in 2021 aimed at other accessible computing scholars, then a magazine article in 2022 intended to introduce undergraduates to the topic of inclusive imagery, and finally a 2023 journal article combining the conference paper dataset with interviews with actual accessibility practitioners at Google.
Getting The Word Out
In addition to the academic publications, I co-wrote a blog post for the Material Design website reporting on our research collaboration. This was shared on Twitter by @MaterialDesign and retweeted by @Google!
Then, in spring 2023, I co-presented a webinar to over 500 attendees on the topic of inclusive alt text, via the Great Lakes ADA Center's "Accessible Technology Series". For a full recording and all documents used in the presentation see the link here.
Then, in spring 2023, I co-presented a webinar to over 500 attendees on the topic of inclusive alt text, via the Great Lakes ADA Center's "Accessible Technology Series". For a full recording and all documents used in the presentation see the link here.
User Experience Improvements
And finally, starting in 2021, the images and image descriptions we evaluated with users were included on approximately 40 million Chromebooks. This was the first ever example of users being given an accessible option for profile images, including images that depict disabilities themselves.