A Simple Yet Effective Hybrid Pre-trained Language Model for Unsupervised Sentence Acceptability Prediction (2022)

Abstract: Sentence acceptability judgment assesses to what degree a sentence is acceptable to native speakers of the language. Most unsupervised prediction approaches rely on a language model to obtain the likelihood of a sentence that reflects acceptability. However, two problems exist: first, low-frequency words would have a significant negative impact on the sentence likelihood derived from the language model; second, when it comes to multiple domains, the language model needs to be trained on domain-specific text for domain adaptation. To address both problems, we propose a simple method that substitutes Part-of-Speech (POS) tags for low-frequency words in sentences used for continual training of masked language models. Experimental results show that our word-tag-hybrid BERT model brings improvement on both a sentence acceptability benchmark and a cross-domain sentence acceptability evaluation corpus. Furthermore, our annotated cross-domain sentence acceptability evaluation corpus would benefit future research.

## introduction sentence acceptability judgment aims to assess to what degree a sentence is acceptable to native speakers of the english language. an effective sentence acceptability scorer is beneficial for many applications, such as ranking outputs from a dialogue system to pick the most fluent and natural response, or being used as an english fluency checker to help identify grammar issues. previous unsupervised works mainly exploit either ngram-based or neural-based language model's negative cross entropy (nce) @xcite and its variants such as syntactic log-odds ratio (slor) @xcite to obtain the sentence acceptability score. however, two problems exist when employing a language model to estimate sentence acceptability: 1) first, low-frequency words greatly impact a sentence probability (or perplexity) from a language model. although subword tokenizers attempt to alleviate this problem by splitting rare or unknown words into subwords, some subwords are still infrequent in their original context, leading to a considerable increase in sentence-level perplexity. 2) second, cross-domain adaptation inefficiency. many terminologies in specific domains affect sentence acceptability prediction and it is often a common practice to select in-domain text to do continual pretraining of the language model, which is time-consuming and inefficient. to address the aforementioned two problems, we present a simple frequency-based method (section 2) to substitute low-frequency words with the english-specific part-of-speech (pos) tag, xpos, in sentences that are used for continual pretraining of the bert model. notably, we are interested in the following research questions, rq1: how much percentage of low-frequency words should be substituted to obtain the best performance on sentence acceptability judgment task? rq2: can we train one model tackling cross-domain sentence acceptability tasks to avoid pretraining for each domain? the experimental results demonstrate that the wordtag-hybrid bert improves the correlation with human rating on the english sentence acceptability benchmark. to establish sentence acceptability evaluation in cross-domains and to overcome the lack of evaluation corpus, we annotated 3,000 pairs of acceptable and unacceptable sentences for financial, law, and biomedical domains. the proposed hybrid bert outperforms the baselines upon the cross-domain sentence acceptability benchmark. the contributions of this work are as follows: (1) we investigate a word-tag-hybrid training schema for a masked language model with a adjustable substitution rate. the experimental results validate the effectiveness of the proposed method on sentence acceptability evaluation benchmarks; (2) we annotated 3,000 pairs foot_0 of acceptable and unacceptable sentences in the financial, law, and biomedical domains. ## methodology we herein describe how to construct a training data set for our word-tag-hybrid bert model. our strategy is to replace low-frequency words in a sentence of the corpus with more abstract, broader tags to mitigate the issue of low frequency. we give a detail of each step of construction and assume that the corpus c is a (large) set of sentences that is available for masked language model training. step 1 is to build a set of low-frequency words v low . to identify which words should be included in v low , we use the whole of wikipedia entries (say w ) for the target language. we apply a standard nlp pipeline to split each entry into sentences and tokenize each sentence to get a list of words of the sentence. let v be the set of all distinct words in w . then, inspired by the idea of "frequency binning" in @xcite , we sort all the obtained words in descending order according to their occurrence frequencies in w , and assign an index for each word from 1 to |v |, as shown in figure 1 . the sum of all words' frequencies is we select all the words whose index are greater than m to build up v low . step 2 is to create the training data from c. s . as shown in figure 2 , when α increases, more words are replaced with their pos tags. we will use the set of all s to do continual pre-training of masked language models. step 3 is to add all pos tags into the vocabulary of the masked language model to ensure that these pos tags will not be split by the subword tokenizer during masked language model training. ## result and analysis to further investigate how the word-tag-hybrid bert performs on multiple domains, we apply the word-tag-hybrid bert with α equal to 0.1 to pairs of acceptable and unacceptable sentences in financial, law, and biomedical domains. note that there is no training data and only three evaluation datasets. due to the space limitation, we refer readers to appendix c -case study -for an intuitive illustration of how word-tag-hybrid bert alleviates the low-frequency effect on perplexity. ## related work there are two research lines. @xcite view sentence acceptability prediction as a supervised learning problem where they extracted many features such as pos tags and semantic tags to improve the lstm prediction performance. on the other hand, @xcite @xcite model sentence acceptability prediction as an unsupervised problem similar to ours where their focus is to transform the language model output into other variants such as slor. in contrast, we aim to investigate trade-offs between word and xpos to improve language model outputs such as perplexity. ## conclusion in this work, we investigate leveraging xpos to substitute low-frequency words in the training data of pre-trained masked language model and found model with 10% word substitution rate achieved the better correlation and accuracy on the sentence acceptability evaluation corpora. in the future, we plan to expand our method to other languages in sentence acceptability prediction task.