Putting sign language AI into users’ hands — Google DeepMind

Google DeepMind has introduced SL2T, a massively multilingual sign-language-to-text model designed to translate sign languages into written text. The model powers sign-to-text dictation in Gboard and Live Transcribe on Pixel 11 devices, starting with American Sign Language (ASL) to English. Users can sign to their phones for tasks like web searches, messaging, or interacting with AI assistants, offering a faster and more natural alternative to typing. The feature is available at no additional cost and will expand to more devices and languages in the future.

Sign languages are distinct natural languages with their own grammars and lexicons, requiring true machine translation rather than simple sign-to-word mapping. SL2T addresses this by using MediaPipe Holistic to track body landmarks, translating these coordinates directly into text without intermediate gloss annotations. The model was trained on over 100,000 hours of data across more than 50 sign languages, with ASL comprising a quarter of the dataset. This approach improves translation quality by learning shared structures across languages and dialects.

Developed with input from the Deaf community, SL2T was shaped by Deaf engineers, data collection partners, and user studies. The AI Sign Language Advisory Committee (AISLAC), comprising global Deaf organizations, guides responsible deployment. The team also co-authored a joint impact report detailing the technology’s capabilities and limitations. SL2T achieved a zero-shot score of 70 BLEURT on the FLEURS-ASL benchmark, outperforming prior models, though occasional errors remain in rare signs and complex constructions.

SL2T builds on decades of research and marks the first deployment of sign language AI in consumer products. Google DeepMind and Android teams collaborated on integration, with plans to expand to additional sign languages and explore sign language generation. The technology aims to bridge communication gaps between Deaf and hearing communities while ensuring accessibility and usability in real-world applications.