Evaluating Transformer Models and Human Behaviors on C hinese Character Naming (2023)

Abstract: Neural network models have been proposed to explain the grapheme-phoneme mapping process in humans for many alphabet languages. These models not only successfully learned the correspondence of the letter strings and their pronunciation, but also captured human behavior in nonce word naming tasks. How would the neural models perform for a non-alphabet language (e.g., Chinese) unknown character task? How well would the model capture human behavior? In this study, we first collect human speakers’ answers on unknown Character naming tasks and then evaluate a set of transformer models by comparing their performance with human behaviors on an unknown Chinese character naming task. We found that the models and humans behaved very similarly, that they had similar accuracy distribution for each character, and had a substantial overlap in answers. In addition, the models’ answers are highly correlated with humans’ answers. These results suggested that the transformer models can capture humans’ character naming behavior well.1

## introduction many aspects of language can be characterized as quasi-regular: the relationship between inputs and outputs is systematic but allow many exceptions. grapheme-phoneme mapping is an example of such quasi-regularity. for example, the letter string '-ave' in english is regularly pronounced as /eiv/ in gave, save, with the exception of /aev/ in have. and human speakers can easily grasp both patterns, e.g., in a nonce word naming experiment, most speakers pronounced the word tave as /teiv/, while some pronounced it as /taev/ @xcite . to explain the grapheme-phoneme mapping process, many models have been proposed, among which the dual route cascaded (drc) model and the connectionist model are the two most influential yet opposite models. the drc model @xcite proposes that the grapheme-phoneme mapping is implemented in two separate routes: a lexical route that directly maps the word's spelling to its pronunciation through a dictionary-like lookup procedure, 2 and a non-lexical route that applies the grapheme-phoneme corresponding 'rules' to convert the letters to their corresponding pronunciation. the implementation of the drc model requires domain-specific knowledge, such as spelling to sound rules. in contrast, the connectionist model @xcite proposed that a word's pronunciation is generated through a neural network that takes the orthographic representation as the input and outputs the phonological representation, which does not require specific knowledge of grapheme-phoneme correspondence rules. both models can explain various behaviors in word identification, such as the faster identification of frequent words compared to infrequent ones. therefore, there is still an ongoing debate about which model better captures the graphemephoneme mapping process. however, most of these models were tested on alphabetic languages (e.g., english and german), and it is still unclear how these models would be generalized to a non-alphabetic language, such as chinese. the drc model seems to be unfit for chinese because there are no regularities in chinese that can be defined as grapheme-phoneme corresponding rules @xcite . in addition, @xcite asserted that ''the chinese, japanese and korean writing systems are structurally so different from the english writing system, that a model like the drc model would simply not be applicable.' @xcite . thus the connectionist model is the only candidate. the majority (81%) of chinese characters are phono-semantic compounds @xcite , which consist of a phonetic radical that contains pronunciation information (denoted by pinyin), 3 and a semantic radical that contains semantic information. 4 for example, for the character (<qing2>, 'sunny'), the left side (<ri4>, 'sun') is the semantic radical, and the right side (<qing1>, 'blue') is the phonetic radical. while the phonetic radical does not contain componential information about the pronunciation, e.g., the first part of the phonetic radical does not represent the first phoneme (e.g., consonant)/syllable onset as letter strings, the relationship between the phonetic radical's pinyin and the character's pinyin is also quasi-regular. ignoring the tonal differences, the character's pinyin can be categorized into 4 types @xcite : regular, the same as the phonetic radical's pinyin; alliterating, deviating in the syllable final; rhyming, deviating in the syllable onset; and irregular, varying in both syllable onset and final (see in our study, we first collected human speakers' answers on unknown character naming, since there is no study investigating how chinese adults read unknown characters. 5 we then trained a set of sequence-to-sequence transformer models with 3 chinese characters use pinyin to represent the pronunciation. the pinyin system consists of 24 syllable initials (mostly contain a consonant), 34 syllable finals (mostly contain a vowel or vowels), and 4 tones. 4 the phonetic radical and semantic radical are mutually exclusive, and they are defined in the ancient chinese dictionary 'shuowen jiezi'. 5 previous studies have focused on children's behavior on unknown character naming and found that children made errors in identifying the incorrect phonetic radical, as well as applying the incorrect regularity pattern @xcite . we then evaluated how closely the results of our aggregated transformers matched those of the human participants, in aspects of the variety of answer types and answer overlaps. in general, both the transformers and human speakers are able to identify the phonetic radical correctly and apply all 4 types of regularities to infer the pinyin, and the transformer models show a high correlation with human data in the proportion of each regularity type. in addition, there is a considerable amount of agreement between the answers generated by our models and those given by humans. our results demonstrate that transformer models can capture human behavior in unknown chinese character naming well. ## related work and current study skilled chinese readers make use of the phonetic radicals to name characters @xcite @xcite , and previous studies measured how phonetic radicals influence character naming in two ways: regularity and consistency @xcite @xcite . the regularity is exemplified in previous studies of chinese character modeling with phonetic radicals as inputs have successfully simulated the regularity effect and consistency effect. @xcite trained a feed-forward network on 4,468 chinese characters and tested the model on 120 characters (seen in the training). the input to the model includes the character's radicals and radicals' positions (e.g., left-right, up-down). 6 the output of the model is the phonological features (e.g., stop, lateral) of the character's pinyin. they also measured the human speakers' response latency foot_3 on each of the 120 test characters. by comparing the human speakers' response latency and the model's sum squared error, they found very similar regularity and consistency effects. in addition, @xcite trained a feed-forward model on 2,159 left-right structured characters, with each character appearing according to its log token frequency. the input included each character's radicals, and the output was the character's pinyin. they analyzed the training accuracy of the model and found the model's sum squared errors lower for the regular characters, which successfully simulated the regularity effect. the regularity and consistency effect revealed that both human speakers and the neural models utilized the statistic distribution of phonetic radicals in naming familiar characters. however, these effects can not be applied in unknown character naming since the speakers don't know the statistics of these characters. therefore, we proposed a new metric (saliency of the phonetic radical) to measure how the phonetic radicals influence the speaker's unknown character naming behavior. saliency is defined as the fraction of the regular characters among all characters sharing the same phonetic radical. for example, the phonetic @xmath2, among which 4 characters ( , , , ) are regular. thus the saliency score @xmath3.33 (4/12). the more salient a phonetic radical is, the more likely the character that contains it is pronounced the same as its pinyin. we hypothesized that the human speakers would show a saliency effect in unknown character naming -they would name the characters more accurately if the phonetic radical is more salient. we expected to find a similar saliency effect in the models. in addition, we also closely examine the models' answers and humans' answers to investigate if the models can represent the human speaker's behavior. ## data the base character dataset consists of 4,341 chinese characters constructed from the ids dataset in chise project @xcite . the original ids (ideographic description sequence) dataset contains 18,347 characters used in china, japan, and korea with the decomposition of each character's phonetic and semantic radicals. 8 the character selection criterion include: 1) is used in chinese; 2) is a phono-semantic compound; 3) has a left-right structure. 9 the character's pinyin, along with its phonetic and semantic radical's pinyin, was collected using the pinyin package. the frequency of each character was extracted from blcu corpus center @xcite . we further labeled each character's regularity: regular, alliterating, rhyming, and irregular as described in there are 660 radicals after decomposing the 4,341 characters, among which 46 radicals only serve as the semantic radicals; 493 radicals only serve as the phonetic radicals; 121 radicals serve as both semantic and phonetic radicals. each radical appears in 7 characters on average, with a range of 1 to 30. eighty percent of the characters in our database have the phonetic radical on the right, with many exceptions, e.g., the semantic radical ' ' <@xmath4. ## human experiment a total of 55 native speakers of mandarin participated in this study. all of them are able to read and write in traditional chinese scripts and pinyin. the average age is 26.3 years, and 80% of them have an education background of college or above. in the experiments, they were asked if they knew the character and prompted to type the pinyin of the character. the detailed experiment procedure and sample questions are described in appendix a. ## transformer model to model the joint probability of the syllable onset and final, we used seq-to-seq transformers @xcite to generate the pinyin of chinese characters trained from scratch. 12 ## conclusion and discussion conclusion we evaluated transformer models and human behaviors on an unknown chinese naming task. this task is difficult for both humans and transformer models, as the average accuracy is lower than 50%. humans have higher accuracy than model[-pinyin] and lower accuracy than model @xcite , and the models and the humans have very similar performances. first, saliency effects were found in both human data and the models' results, suggesting that both models and humans utilize the statistical distribution of the phonetic radical to infer the character's pinyin. further, although humans' answers are more similar to each other, our models also achieved a substantial overlap with humans' answers. additionally, the production probability of each answer type is highly correlated between models and humans (except for semantic type), suggesting that both models and humans are able to apply all regularity patterns in producing answers. finally, models with radical's pinyins in the input are more similar to humans and achieved higher accuracy. capturing quasi-regularity our work is also related to the long-standing criticism that the neural networks may only learn the most frequent class and can not extend other minority classes, thus would fail to learn the quasi-regularity in languages @xcite . previous studies on morphological inflections have shown that the neural models overgeneralized the most frequent inflections on nonce words and had almost no correlation with humans' production probability on the less frequent inflections (e.g., ρ = 0.05 for the /-er/ suffix in german plural @xcite , and @xmath5.17 for irregular english verbs @xcite ). however, our results showed that the transformer models could learn the quasi-regularity in chinese character naming, that the models produce all answer types, and the production probability of each type is highly correlated with human data. however, our results do not contradict the previous studies. chinese character naming and morphological inflection both exhibit quasi-regularity, but the two domains are very different: the patterns in chinese character naming are less rule-governed. this paper's contribution to the debate of quasi-regularity in language processing is not to provide a 'yes' or 'no' answer; instead, we used a novel task and showed that the neural models have the potential to model human behaviors in learning quasi-regularity. we hope our study could inspire future work in this field to apply diverse tasks and conduct more detailed examinations of neural models' ability in learning quasi-regularity.