Exploring and Adapting C hinese GPT to P inyin Input Method (2022)

Abstract: While GPT has become the de-facto method for text generation tasks, its application to pinyin input method remains unexplored. In this work, we make the first exploration to leverage Chinese GPT for pinyin input method. We find that a frozen GPT achieves state-of-the-art performance on perfect pinyin. However, the performance drops dramatically when the input includes abbreviated pinyin.A reason is that an abbreviated pinyin can be mapped to many perfect pinyin, which links to even larger number of Chinese characters. We mitigate this issue with two strategies,including enriching the context with pinyin and optimizing the training process to help distinguish homophones. To further facilitate the evaluation of pinyin input method, we create a dataset consisting of 270K instances from fifteen domains. Results show that our approach improves the performance on abbreviated pinyin across all domains. Model analysis demonstrates that both strategiescontribute to the performance boost.

## introduction gpt @xcite is a transformerbased @xcite language model that predicts tokens in an autoregressive manner. with a generic model architecture and the availability of vast web text data, gpt has been successfully developed for english, chinese @xcite , and many other languages. it shows extraordinary ability to generate fluent sentences and has been successfully applied to a wide range of natural language generation tasks. however, it remains unexplored to what extent gpt handles chinese pinyin input method 1 , which is used by hundreds of millions people when they enter chinese characters on computers and cellphones. pinyin input method allows users to enter chinese characters based on their pronunciations. given a pinyin foot_0 as the input, pinyin input method returns a list of chinese characters pronounced with that pinyin. fundamental elements of pinyin include initials (声母) and finals (韵母). in most cases, a chinese character is spelled with one initial followed by one final. for example, as shown in this work, to the best of our knowledge, is the first one to explore the use of chinese gpt for pinyin input method. we start by testing the performance of a frozen gpt. in this setting, we fix the parameters of gpt and predict chinese characters from left to right in an autoregressive manner. at each time step, only characters pronounced with the same pinyin are legitimate candidates to be predicted. we find that, when the input is perfect pinyin, a frozen gpt performs comparably to stateof-the-art systems on the benchmark dataset @xcite . however, when the input is abbreviated pinyin with only initials of characters, the to further facilitate the research on pinyin input method, we construct a new dataset based on the wudaocorpora @xcite . our dataset includes 270k instances from 15 commonly used news domains. 3 to evaluate towards multiple facets, the dataset covers instances with different numbers of context characters and pinyin. from our experiment results, we have these key findings: the input of pinyin input method includes a sequence of chinese characters @xmath0, . . . , w n } as the context and a sequence of pinyin @xmath1, . . . , p n+k }, where w @xmath2, p n+@xmath3, and v w and v p are the vocabularies of words and pinyin, respectively. the output is a sequence of chinese characters @xmath4, . . . , w n+k }, where w n+@xmath5. the number of output characters is the same as the number of pinyin (i.e., k) and each character should be pronounced with the corresponding pinyin. the output sequence is desired to follow the context of chinese characters to form a coherent sentence. as mentioned earlier in the introduction section, the input pinyin might be perfect (e.g., "wo men") or abbreviated (e.g., "w m"). examples of the task are given in in this paper, we assume that the oracle pinyin segmentation results are provided. sometimes, a raw pinyin sequence can be mapped to different segmentation results. for example, the raw pinyin input "jianshi" can be segmented as "ji an shi" ("集安市", a city in the southwestern part of jilin province, china) or "jian shi" ("见识", which is translated as "experience" in english). pinyin segmentation is a subtask @xcite of pinyin input method, which is well solved with the accuracy of 98% @xcite . we leave the integration of pinyin segmentation as future work. ## models in this section, we first introduce standard textbased gpt models adopted in this work (section 3.1). afterwards, we introduce how to extend gpt models for pinyin input method with enriched pinyin context (section 3.2) and pinyin-constrained training (section 3.3), respectively. ## experiment in this section, we show the results on pinyin input method over the two settings (i.e., perfect pinyin and abbreviated pinyin). ## related work pinyin input method we describe existing works based on whether the input pinyin is perfect or abbreviated. a majority of existing works focus on perfect pinyin. traditional models are typically based on statistical language models @xcite and statistical machine translation @xcite . recent works are usually built with neural network. for example, moon ime @xcite integrates attention-based neural network and an information retrieval module. @xcite improves an lstmbased encoder-decoder model with online vocabulary adaptation. for abbreviated pinyin, co-cat @xcite uses machine translation technology to reduce the number of the typing letters. @xcite propose an lstm-based encoder-decoder approach with the concatenation of context words and abbreviated pinyin as input. our work differs from existing works in that we are the first one to exploit gpt and verify the pros and cons of gpt in different situations. in addition, there are some works handling pinyin with typing errors. @xcite investigate a typing model which handles spelling correction in sentence-based pinyin input method. chime @xcite is a error-tolerant chiin the appendix. we recommend readers to select models in a more cost-effective way based on their requirements. pinyin-enhanced pretrained models our methodology also relates to pretrained models that use pinyin information. sun et al. (2021) propose a general-purpose chinese bert with new embedding layers to inject pinyin and glyph information of characters. there are also task-specific bert models, especially for the task of grammatical error correction since an important type of error is caused by characters pronounced with the same pinyin. zhang et al. (2021a) add a pinyin embedding layer and learns to predict characters from similarly pronounced candidates. plome (liu et al., 2021) add two embedding layers implemented with two gru networks to inject both pinyin and shape of characters, respectively. xu et al. (2021) add a hierarchical encoder to inject the pinyin letters at character and sentence levels, and add a resnet encoder to use graphic features of character image. ## conclusion in this paper, we explore how to adapt pretrained chinese gpt to pinyin input method. to begin with, we find that a frozen gpt with decoding conditioned on pinyin can reach state-of-the-art performance on perfect pinyin. however, in abbreviated setting, the performance drops by a large gap. through our experiments, we find that both context enrichment with pinyin and pinyin-constrained training improve the performance. in the future, we would like to investigate more challenging settings including error-tolerant pinyin input method and mixtures of perfect pinyin and abbreviated pinyin.