An Attentive Recurrent Model for Incremental Prediction of Sentence-final Verbs (2020)

Abstract: Verb prediction is important for understanding human processing of verb-final languages, with practical applications to real-time simultaneous interpretation from verb-final to verb-medial languages. While previous approaches use classical statistical models, we introduce an attention-based neural model to incrementally predict final verbs on incomplete sentences in Japanese and German SOV sentences. To offer flexibility to the model, we further incorporate synonym awareness. Our approach both better predicts the final verbs in Japanese and German and provides more interpretable explanations of why those verbs are selected.

## introduction final verb prediction is fundamental to human language processing in languages with subjectobject-verb (sov) word order, such as german 1 and japanese, @xcite @xcite particularly for simultaneous interpretation, where an interpreter generates a translation in real time. instead of waiting until the entire sentence is completed, simultaneous interpretation requires translation of the source text units while the interlocutor is speaking. when human simultaneous interpreters translate from an sov language to an svo one incrementally-without waiting for the final verb at the end of a sentence-they must use strategies to reduce the lag, or delay, between the time they hear the source words and the time they translate them @xcite . one strategy is final verb prediction: since the verb comes late in the source sentence but early in the target translation, if the verb is predicted in advance, it can be translated before it is heard, allowing for a more 1 german is rich in both sov and svo sentences. it has been argued that its underlying structure is sov @xcite , but this is not immediately relevant to our task. ## the problem of verb prediction given an sov sentence, we want to predict the final verb as soon as possible in an incremental setting. for example, in figure 1 , the final verb, "gegeben", in german is expected to be translated together with "hätte es" as "there would have been" in the middle of the english translation. human interpreters will often predict a related verb rather than the exact verb in a reference translation, while preserving the same general meaning, since predicting the exact verb in a reference translation is difficult @xcite . for instance, in figure 2 , besides "machen", verbs such as "schaffen" and "tun" also offen pair with "hoffnungen" to express "hope for" in english. we therefore include two verb prediction tasks: first, we learn to predict the exact verb; second, we learn to predict verbs semantically similar to the exact reference verb. we describe these two tasks below. ## a neural model for verb prediction this section describes anviil's structure. gated recurrent neural networks (rnns), such as lstms @xcite and gated recurrent units @xcite , can capture long-range dependencies in text, which we need for effective verb prediction. we construct an rnn-based classifier with selfattention @xcite for predicting sentencefinal verbs (figure 3 ). this is a natural encoding of the problem, as it explicitly models how interpreters might receive information and update their verb predictions. the hidden states of the sequence model can be either at the word or character level. ## exact prediction experiments we first test exact prediction on both japanese and german verb-final sentences with both word-based and character-based models. ## visualization and analysis we now analyze our model's predictions. while previous work @xcite examines the contribution of features by examining the model itself, our approach does not rely on feature engineering. to examine our model, we instead use a heatmap to visualize the time course attention values in sentences, allowing us to see on what the model focuses when predicting. ## related work this section examines previous work on prediction in humans, simultaneous interpretation, and simultaneous machine translation. psycholinguistics has examined argument structure using verb-final bǎ-construction sentences in chinese @xcite . @xcite find that case markers facilitate verb predictions for humans, likely because they provide clues about the semantic roles of the marked words in sentences. in sentence production, @xcite suggest that humans plan verbs after selecting a subject but before objects. empirical work on german verb prediction first investigated german-english simultaneous interpreters in @xcite : professional interpreters often predict verbs. @xcite introduce early verb prediction into japanese-english smt by predicting verbs in the target language. 2016 ) predict verbs with linear classifiers and compare the predictions to human performance. we extend that approach with a modern model that explains which cues the model uses to predict verbs. in interactive translation @xcite and simultaneous translation @xcite systems, neural methods for next word prediction improve translation. bert @xcite uses masked deep bidirectional language 0.4 0.6 0.8 1 sentence revealed (%) 0.1 0.2 0.3 0.4 0.5 accuracy japanese 100 verbs case marker de (ins) ga (nom) ni (dat) to (quot) to shite (ess) wo (acc) figure 13 : case markers correlate with improved verb prediction compared to overall verb prediction (figure 4 ). some case markers, such as to, have large jumps in accuracy toward the end, while others, such as wo do not. we examine nominative (nom), instructive (ins), accusative (acc), dative (dat), quotative (quot), and essive (ess) markers. existing neural attention models for sequential classification are commonly trained on complete input @xcite @xcite . classification on incomplete sequences and long-distance sentence-final verb prediction remains difficult and under-explored. ## conclusion we present a synonym-aware neural model for incremental verb prediction using bigru with selfattention. it outperforms existing models in predicting the most frequent sentence-final verbs in both japanese and german. as we predict the verbs incrementally, our method can be directly applied to solve real-time sequential classification or prediction problems. smt systems for sov to svo simultaneous mt can also benefit from our work to reduce translation latency. we show that larger datasets always help with predicting the sentencefinal verbs, suggesting that larger corpora will further improve results.