Clause Final Verb Prediction in H indi: Evidence for Noisy Channel Model of Communication (2021)

Abstract: Verbal prediction has been shown to be critical during online comprehension of Subject-Object-Verb (SOV) languages. In this work we present three computational models to predict clause final verbs in Hindi given its prior arguments. The models differ in their use of prior context during the prediction process – the context is either noisy or noise-free. Model predictions are compared with the sentence completion data obtained from Hindi native speakers. Results show that models that assume noisy context outperform the noise-free model. In particular, a lossy context model that assumes prior context to be affected by predictability and recency captures the distribution of the predicted verb class and error sources best. The success of the predictability-recency lossy context model is consistent with the noisy channel hypothesis for sentence comprehension and supports the idea that the reconstruction of the context during prediction is driven by prior linguistic exposure. These results also shed light on the nature of the noise that affects the reconstruction process. Overall the results pose a challenge to the adaptability hypothesis that assumes use of noise-free preverbal context for robust verbal prediction.

## introduction research on sentence comprehension has conclusively established the widespread role of prediction during online processing (e.g., @xcite @xcite . it is known that comprehenders actively anticipate the upcoming linguistic material prior to receiving that information during listening or reading @xcite . the role of active prediction during comprehension has particularly been emphasized for processing of sov languages (e.g., @xcite @xcite *equal contribution by ks and nb. @xcite . in particular, it has been argued that preverbal nominal features such as case-markers are effectively used to make precise prediction regarding the clause final verb. indeed, the adaptability hypothesis states that owing to the typological properties, the prediction system in sov languages is particularly adapted to make effective use of preverbal linguistic material to make robust clause final verbal prediction @xcite . evidence for the adaptability hypothesis come from various behavioral experiments that show effective use of case-markers to make clause final verbal prediction (e.g., @xcite , facilitation at the verb when the distance between the verb and its prior dependent increase (e.g., @xcite , and lack of structural forgetting in the face of complex linguistic environment (e.g., @xcite . on the other hand, the noisy chan-nel hypothesis assumes that prediction during comprehension is required to accommodate uncertainty in the input @xcite . in other words, the hypothesis posits that comprehenders have the knowledge that speakers make mistakes during production, hence, comprehenders need to reconstruct the received input @xcite . the two hypotheses stated above make distinct assumptions regarding the utilization of pre-verbal context towards making clause final verbal predictions in sov languages. one way to operationalize the predictions of the adaptability hypothesis is to assume that the preverbal linguistic material will be faithfully used to make verbal prediction, the noisy channel hypothesis on the other hand, assumes that the preverbal context is noisy and therefore subject to reconstruction. one consequence of this would be that the adaptability hypothesis would predict that verbal prediction should be robust while the noisy channel hypothesis would predict that verbal prediction should be susceptible to errors. in addition, the two hypotheses would make distinct prediction regarding the nature of errors that might occur during clause final verbal prediction. in order to probe the two hypotheses stated earlier, in this work, we investigate various incremental models that use local linguistic features to predict clause final verbal prediction in hindi (an sov language). the distribution of these model predictions is compared with human data. in particular, we investigate to what extent the models are able to capture the nature of both grammatical as well as ungrammatical verbal predictions when compared to data collected from native speakers of hindi. further, in order to probe the assumptions of the noisy channel hypothesis more closely, we probe multiple noise functions to investigate the nature of preverbal context reconstruction during prediction. the paper is arranged as follow, in section 2 we briefly describe the experimental results that we model. section 3 provide the necessary details regarding methodology (data/tools, model evaluation, etc.). in sections 4 and 5 we respectively discuss the n-gram surprisal and the lossy-surprisal models. section 6 presents the results. section 7 discusses the current findings and its implications. we conclude the paper in section 8. ## background in spite of the proposed central role of verb prediction during online processing of hindi (e.g., @xcite @xcite , there is a surprising lack of any modeling attempt to understand the processes that subserve verbal predictions in the language. while there are computational metrics that model reading time data (e.g., @xcite @xcite , a computational model that makes precise verbal prediction in sov languages has not been investigated thoroughly (but see, @xcite , for an initial attempt). understanding the mechanisms that subserve verbal prediction in sov languages is critical to understanding how these languages are processed (cf. @xcite @xcite @xcite . our work fills this gap in the literature. in this section we summarize the key results of a recent study by @xcite who investigated the nature of verbal prediction in hindi using a series sentence completion studies @xcite . later, in sections 4, 5 we present three computational models to account for these results. ## n -gram based surprisal model in order to evaluate the adaptability hypothesis where the prediction of upcoming verb is driven by local nominal arguments, we implement an ngram language model using the data discussed in section 3.2. such models are typically used to compute the surprisal metric @xcite given local context (e.g., @xcite . recall that we have at most 3 nps as the preverbal context, and therefore, we use a 4-gram model so that the model has access to the complete context in a given condition to make a verbal prediction. unlike the models discussed in section 5, the preverbal context in this model is free of noise. ## lossy-context surprisal models in this section, we discuss two models to test the noisy channel hypothesis. as stated in section 1, the underlying assumption is that human communication is noisy @xcite and the comprehender has to reinterpret the input to make prediction about upcoming linguistic material. in order to evaluate this hypothesis, we implement different versions of the lossy-context surprisal metric @xcite . lossy-context surprisal holds that processing difficulty at a word in a context is proportional to the surprisal of a word given a lossy memory representation of the context. the two models discussed in sections 5.1 and 5.2 differ in their noise functions that affect the interpretation of the preverbal context. for the current investigation, lossy-context surprisal is extended to model the sentencecompletion task. the word with the highest probability in a given context is assumed to be most likely to complete the sentence (cf. @xcite @xcite . as noted by @xcite , the lossy surprisal model is not representation-agnostic. its predictions are dependent on a noise distribution (m). one can then obtain: in addition, if |c| = @xmath0, we don't add any noise to the context and simply use the ngram model l for prediction. in other words, if @xmath1, then we consider p(w|r) = p l (w|c). since we only consider erasure-based noise distributions, this is done to ensure that the whole context is not lost during prediction. in order to get an average behavior of the model, we run the model 10 times and then take the top 50 predictions based on the total probability of each prediction. in other words, suppose a phrase s is predicted to follow a given preverbal arguments in a condition. then, the total probability of s to be predicted in the given condition by the average model is equal to 1 10 10 @xmath2, where p i (s) denotes the probability of prediction s in the ith run. note that if s is not predicted in the ith run, then p i (s) = 0. in the next subsections, we present two models with different noise distribution. ## results in order to test if the improvement seen in the lc-surp pred-rec model is indeed significant, we also performed the chi-square test to see if the categories of verb class predicted in the lc-surp pred-rec model were significantly different from other models. results showed that this was indeed true -categories of verb classes in the lc-surp pred-rec model were significantly different (@xmath3.05) from both 4-gram model and the lc-surp predbias model. 12 klp provides a measure to quantify the divergence between the human and model prediction distributions. however, the nature of this divergence is still unclear. in order to understand the output of the models better, we evaluate them on some additional metrics. finally, we report a qualitative analysis of the model output. ## discussion results show that the lossy context surprisal model with predictability recency bias noise performs best in terms of the distribution of predicted verbs and the error types vis-à-vis the completion data. this provides support for the noisy channel hypothesis and poses a challenge to the adaptability hypothesis. in addition, the comparison of the two lossy surprisal models sheds light on the nature of the noise during the reconstruction process. results show that qualitatively all the models capture the completion data to a certain extent (see, section 6.2). at the same time, overall the noisy context models performed better than the ngram model in two clear ways. first, the models were able to capture the differential nature of casemarker combination in a limited context. this leads to better coverage of error sources (both in terms of errors made and not made). second, the models were therefore also better at making better verb predictions compared to the n-gram model. in particular, the overall success of the pred-rec model showed that reconstruction of the noisy context in influenced by both past exposure of preverbal subcontext and the recency of the context (cf. @xcite . put differently, the reconstruction of the context is driven by sub-strings that are more frequent (e.g., ne-ko) and that are closer to the verb. critically, this shows that the reconstruction process is not random. while the performance of the predictability recency model is good, it suffers from three issues (a) it overestimates the number of errors made by humans, (b) its overall coverage for various verb class is low, and (c) it is insensitive to subject primacy. the model is able to successfully predict verb phrase involving no clausal embedding, and to a limited extent, those with embeddings. while certain complex structures such as n dt dt, predicted rarely by humans, are dropped entirely by the model, its prediction for the t dt structure which is frequent in the completion data is not that high. an investigation into the data also shows a scarcity of training examples that exhibit an animate 3-np context followed by such t dt continuations. 15 one reason for this could be the size of the training data, currently 5 million sentences; future work can train on a larger data set. another possibility is that certain patterns in the human data are not captured in the written corpus used for training and requires a dialogue corpus. unfortunately, such a corpus currently does not exist for hindi and attempts to modeling using such a data will have to wait its availability. relatedly, @xcite argue that prediction based on corpus frequency of syntactic information may not be able to fully capture the notion of preactivation during the completion task. hence, future work will need to incorporate other sources of information. finally, the results show that the 4-gram model is more sensitive to subject primacy. this is because, the 4-gram model (unlike noisy context models) has access to the n1 features when making predictions. it can thus correctly use the n1 case feature to avoid predicting passive verbs. this suggests that a noise function relying only on local information will be limited in accounting for the current data. the current work provided the first set of detailed results towards modeling clause final verb prediction in an sov language. the work demonstrated the effectiveness of lossy surprisal models and probed the nature of the noise function during the reconstruction process. in addition to the quantitative analyses demonstrating the success of the predictability recency lossy surprisal model, a key contribution of the work was that it highlighted the nature of model's closeness to the human data, both in terms of verb class prediction and the error type. overall, the results support the proposals that highlight the detrimental effect of increased complexity of the preverbal linguistic material in sov languages (e.g., @xcite @xcite . future models need to explore other noise functions to investigate the interaction of context predictability with recency as well as primacy of non-local information (e.g., subject). further, these models need to be tested to investigate the effect of distance (e.g., @xcite and structural complexity @xcite on verbal prediction in sov languages. ## conclusion we implemented three models to predict clause final verbs in hindi. model outputs were compared with verb predictions of native speakers of hindi using quantitative measures as well as qualitatively. results show that the model that uses limited preverbal context with a predictability recency bias noise function captures the distribution of human data best. the success of this model is consistent with the idea that the reconstruction of the noisy context during prediction is influenced by prior linguistic exposure and that this process interacts with recency of input. these results support the noisy channel hypothesis to language comprehension.