Tonguescape: Exploring Language Models Understanding of Vowel Articulation (2025)

Abstract: Vowels are primarily characterized by tongue position. Humans have discovered these features of vowel articulation through their own experience and explicit objective observation such as using MRI. With this knowledge and our experience, we can explain and understand the relationship between tongue positions and vowels, and this knowledge is helpful for language learners to learn pronunciation. Since language models (LMs) are trained on a large amount of data that includes linguistic and medical fields, our preliminary studies indicate that an LM is able to explain the pronunciation mechanisms of vowels. However, it is unclear whether multi-modal LMs, such as vision LMs, align textual information with visual information. One question arises: do LMs associate real tongue positions with vowel articulation? In this study, we created video and image datasets from the existing real-time MRI dataset and investigated whether LMs can understand vowel articulation based on tongue positions using vision-based information. Our findings suggest that LMs exhibit potential for understanding vowels and tongue positions when reference examples are provided while they have difficulties without them. Our code for dataset building is available on GitHub.

## introduction in phonetics, vowels are distinguished and described by focusing on tongue positions and lip shape. beginning with @xcite , humans have explained vowels based on tongue positions during articulation. human speakers are aware of speech mechanisms with training through introspection of experience and observations of visual information (e.g., mri). for example, when pronouncing the english word "image," speakers can perceive and explain that the initial sound is produced by positioning the tongue forward and high in the mouth. moreover, speakers can relatively recognize vowels, using a certain vowel as a reference point. this is the method @xcite employed to introduce the cardinal vowels, demonstrating their grasp of tongue positions through introspection and objective observations, and to link this understanding to the concept of vowel articulation. this knowledge helps explain to language learners how to pronounce and describe linguistic phenomena. language models (lms) are trained on a large amount of data that includes linguistic and medical fields. our preliminary studies showed that lms know vowel pronunciation and the correlation between vowels and tongue positions as textual knowledge (see section 2). to determine if lms comprehend articulation relative to the articulatory organs, multi-modal information is essential. multi-modal lms capture not only textual information, but also images, videos, and audio @xcite @xcite . moreover, their application is expanding to more specialized fields, such as clinical tasks involving the detection and explanation of diseases from clinical images such as ct and mri @xcite . however, it is known that the alignment among modalities, such as images and text, is often weak in multi-modal lms, and it remains unclear whether these models truly understand such interactions @xcite @xcite . ## dataset: tonguescape when predicting vowels from a real-time mri, speakers will (1) detect the moment of vowel articulation in the video, (2) observe the tongue and determine its position, and then (3) select a vowel. we propose a qa dataset for vowel prediction from real-time mri recordings of tongue movements during vowel articulation comprising three steps, with each step corresponding to these stages of human perception as shown in figure 2 . we curated and annotated videos and images of tongue positions from the real-time mri articulatory movement database (rtmridb) @xcite , which comprises real-time mri recordings of ar-ticulatory movements of japanese phonemes. in this study, we focused on the japanese five-vowel system (/a/, /i/, /u/, /e/, /o/) where each vowel has distinct articulatory features. details of our dataset are described in appendix a. ## discussions and analysis we will focus on important aspects in the following sections and defer more discussions to appendix c. ## conclusion: do lms associate tongue positions with vowel articulation? we investigated the capability of lms to understand the relationship between tongue positions and vowels to address our research question. as shown in we have also discussed in sections 5.1, 5.2, and 5.5 that some lms can consider relative positions to predict vowels. these findings reveal that the lms predict vowels with respect to other pairs of tongue positions and vowels while it is challenging to predict unseen vowels as shown in the results of the one-shot setting. we can conclude that lms, to a certain extent, associate tongue positions with the concept of vowel articulation. in conclusion, our findings indicate whether lms can comprehend the vowel articulation that linguists have long sought to decode. while lms faced challenges in our experiments, the performance of lms can be improved like that of linguists when given examples. we not only hope these findings will apply to large-scale linguistic analysis, speech synthesis, and educational fields but also wish for further research of languages.