Breaking Barriers: The Struggles of Multilingual Users with AI Voice Recognition

Every morning while preparing my daughter’s school lunch, I clearly state, “OK Google, what’s today’s weather?” Our Google Home Assistant dutifully informs us of the expected precipitation patterns and temperatures. With our lives adorned with “smart” devices, things should be simpler than ever — correct?

In actuality, voice recognition technology frequently fails to meet the requirements of its users, especially for those of us who are multilingual.

You might argue that achieving bilingualism is, in many respects, my life’s pursuit: I was raised in an English-speaking household, and currently teach Hispanic Studies at Connecticut College.

I frequently meet students who converse in Spanish with their families yet believe that the Spanish used at home is not “proper Spanish,” regarding it as inferior to the standardized “textbook” Spanish or the dialects from Spain. I strive to dispel that misconception — yet Google Assistant and similar technologies are not assisting.

Recently, while cooking in my kitchen, I desired to listen to a song by Natalia Lafourcade. I employed the general voice command, but instead of the soothing melodies of the renowned artist I sought, my Google Home played an unusual track by an entirely different artist.

This frustrated me. Lafourcade has been active in the music scene in her homeland of Mexico, the U.S. and beyond for 22 years, earning 17 Latin Grammy Awards. As of this moment, a Tiny Desk Concert she performed with NPR boasts 33 million views on YouTube. However, I suspected I understood why Google Home failed to identify her.

I attempted once more. But this time, rather than articulating her name accurately, as Lafourcade herself would (naˈta.lja.la.fuɾ.ˈka.ðe), I used my best English accent: “nuh·taa·lee·uh luh-fr·kaad.”

This time, the device responded cheerfully and played the song by the artist I desired.

To interact with our gadgets, we shouldn’t have to distort a celebrity’s name — or indeed, anyone’s name. Speech-processing technology ought to recognize proper nouns in any source language. At the very least, it should acknowledge one of the most commonly spoken languages worldwide.

Ideally, speech-processing technology would operate within the multilingual context of our globalized society, but we’re not quite there. Apple’s voice-to-text and similar features from other firms are unable to comprehend code-switched sentences where the speaker alternates between two or more languages — for instance, “Can I please use the baño?”

Yes, it is currently feasible to add several languages to your device and switch between them — just not mid-sentence, as a bilingual individual’s brain can seamlessly do.

A recent research study demonstrated that both Google Assistant and Apple’s Siri struggled with multilingual competencies. This highlights their inability to cater to a significant segment of their audience — particularly, individuals who communicate in a language that is not the primary language of their locale.

Even though suggestions for improving speech recognition technology exist, they are simply not matching the increasing number of interlingual language-mixers among us. As per the U.S. Census Bureau, the percentage of Americans identifying as Hispanic or Latino is nearing 20%, and a significant portion of these individuals (and others) communicate in Spanglish or some variation of Spanish.

Although most of our nation communicates in English, the Center for Immigration Studies found that in 2018, nearly a fifth of our populace spoke a language other than English at home. In Houston, where an estimated 145 languages are spoken, at least 40% of the population engages in a language other than English at home. The number of multilingual individuals in the U.S. is anticipated to grow.

New technologies raise ethical dilemmas. In her best-selling book “Unmasking AI: My Mission to Safeguard What Is Human in a World of Machines,” Joy Buolamwini elaborates on how sexism, racism, ableism, and other biases are inscribed in the AI tools that we utilize. Geoffrey Hinton, whose groundbreaking research paved the way for present-day AI algorithms, used his Nobel Prize acceptance address to caution against the potential ramifications of technology.

As we navigate the early stages of AI, it is essential to confront possible harms such as linguistic discrimination. Multilingual functionalities must be integrated into the devices we depend upon.

In simpler terms: “OK Google, it’s time to embrace Spanglish.”

Emily Kuder is an assistant professor of Hispanic Studies at Connecticut College.