Probing the Category of Verbal Aspect in Transformer Language Models (2024)

Abstract: We investigate how pretrained language models (PLM) encode the grammatical category of verbal aspect in Russian. Encoding of aspect in transformer LMs has not been studied previously in any language. A particular challenge is posed by ”alternative contexts”: where either the perfective or the imperfective aspect is suitable grammatically and semantically. We perform probing using BERT and RoBERTa on alternative and non-alternative contexts. First, we assess the models’ performance on aspect prediction, via behavioral probing. Next, we examine the models’ performance when their contextual representations are substituted with counterfactual representations, via causal probing. These counterfactuals alter the value of the “boundedness” feature—a semantic feature, which characterizes the action in the context. Experiments show that BERT and RoBERTa do encode aspect—mostly in their final layers. The counterfactual interventions affect perfective and imperfective in opposite ways, which is consistent with grammar: perfective is positively affected by adding the meaning of boundedness, and vice versa. The practical implications of our probing results are that fine-tuning only the last layers of BERT on predicting aspect is faster and more effective than fine-tuning the whole model. The model has high predictive uncertainty about aspect in alternative contexts, which tend to lack explicit hints about the boundedness of the described action.

## introduction this paper focuses on the grammatical category of verbal aspect. it is a category that involves both morphology and semantics of the verb, and expresses how an action denoted by the verb extends over time. linguistic theory of aspect is intricate: different languages make different aspectual distinctions, e.g., languages can have distinct perfective/imperfective/progressive categories of aspect, while some make no distinction at all. aspect is also one of the most complex categories in many languages: even advanced experts, who are non-native speakers, continue to make errors in the choice of aspect @xcite . we focus on the slavic aspectual system, in particular in russian, which displays significant differences in the semantics of the perfective/imperfective opposition to other languages @xcite . how aspect is encoded in pretrained language models (plms) has not been previously studied for any language, although other grammatical properties-number agreement, predicateargument syntactic relations, etc.-have been studied. it is challenging to identify what linguistic phenomena in the context affect the choice of aspect. a special challenge concerning aspect is posed by "alternative contexts", where more than one aspect form is acceptable grammatically and semantically. we investigate the following research questions: rq1. do bert and roberta encode the category of aspect, and if they do-how? rq2. how does the encoding of aspect in these models correspond to linguistic theory of aspect? rq3. is encoding of aspect in alternative contexts different from nonalternative contexts? we perform two kinds of probing: behavioral and causal. in behavioral probing, we inspect layers one by one, observing how the model predicts which aspect form best suits the context. if the model fails, we infer that it does not encode the target linguistic property (here-aspect). we introduce two types of behavioral probing via filling a mask: iterative masking and aspect inference. in both methods, the model's preference for aspect is reflected in the probabilities it assigns to verb forms in the masked position. for causal probing, we intervene in the model's representations at each layer: we manipulate the semantics of the action described by the target verb and its context-whether the action is bounded or unbounded. if the intervention is relevant for predicting the target property, the model's performance on the task will be affected. our findings can be summarised as follows: (1) all probing methods indicate that bert and roberta do encode aspect, predominantly in their final layers. (2) interventions in sentence semantics cause effects consistent with theory of aspect: imperfective verbs typically describe unbounded actions, while perfective verbs describe bounded actions. (3) fine-tuning only the final layers of bert for aspect prediction results in improved performance, confirming our first finding. (4) both pretrained and fine-tuned models exhibit high uncertainty regarding aspect preference in alternative contexts, where multiple aspect forms are valid. (5) alternative contexts are more sensitive to causal intervention in the semantics of boundedness. (6) such contexts often lack explicit hints about the action's boundedness, which makes both humans and plms uncertain about the choice of aspect. ## related work several studies focus on the internal representation of linguistic information inside plms. correlation probing methods are based on parametric probes, i.e., linear or non-linear classifiers trained on model representations to predict specific linguistic properties @xcite @xcite @xcite @xcite @xcite . some have questioned the efficacy of probing classifiers, and whether the original model, which was used as an encoder, actually uses the information discovered by probes @xcite @xcite . in response to this criticism, a number of methodologies are proposed @xcite @xcite @xcite . @xcite gives an extensive review of probing classifiers as an approach, their advantages and shortcomings. non-parametric correlation probing, or behavioral probing, tests the behavior of plms without additional classifiers. to isolate the target linguistic property, a plm is evaluated by using a set of carefully designed examples @xcite @xcite @xcite @xcite @xcite . @xcite propose a methodology for creating synthetic examples, which differ by various linguistic properties. while most work focuses on english, @xcite introduce the clams dataset for syntactic evaluation of models for five languages, including russian. @xcite propose multilingual morphological checklist (m2c), a framework for behavioral probing of typological features in 12 languages, e.g., motion verbs in russian. causal probing relies on controlled interventions into the lm's internal components (or into the input), and studying consequent changes in the model's behavior @xcite @xcite @xcite @xcite @xcite @xcite . amnesic probing @xcite builds on the intuition that removing a property from the representation will weaken the model's ability to solve a task, if the property is important for the task. the approach is based on an algorithm-iterative null-space projection (inlp)-for removing linear information from representations @xcite . @xcite apply inlp to generate counterfactual representations and use these to test how changing particular linguistic features affects the model's behavior. despite some criticism of inlp @xcite , we use it to investigate the behavior of lms on aspect prediction. ## background on aspect the category of aspect in russian characterizes the action described by a verb in terms of its progresscontinuous vs. punctual, completed vs. uncompleted, etc.-or from the observer's perspectiveretrospective vs. synchronous. the meaning of aspect opposition has long been a subject of debate. in this paper, we adhere to the theory that boundedness-reaching a limit-is the factor determining the aspect form @xcite . we assume that every verb has two aspect forms-perfective and imperfective-though in some rare cases the two forms may coincide, e.g., "обещать" (to promise-perf. or imperf.). unlike most grammatical categories, aspect has no unique marker in the verb form and is tightly connected with the verb's lexical meaning. aspect can be expressed by the root, e.g., "говорить" (imp.) vs. "сказать" (perf.), to say; by the suffix, e.g., "толкать" (imp.) vs. "толкнуть" (perf.), to push; by the prefix, e.g., "делать" (imp.) vs. "сделать" (perf.), to do/make. foot_0 examples (1) and ( 2 ) show the aspect pair of verbs "дуть / дунуть" (to blow): wind always blew on the coast. (2) и вдруг резко дунул (perf.) ветер. ## experiments for probing experiments, we use the russian bert-base, bert-large @xcite , and roberta-large @xcite ). 2 we mostly focus on experiments with bert-large, since other models showed similar performance. ## conclusions and future work we investigate the encoding of the grammatical category of verbal aspect for russian in plmsparticularly, bert and roberta-via behavioral and causal probing. encoding of aspect has not been studied to date for any language or model. all types of probing show that these models do encode aspect and learn to distinguish between aspect forms primarily in their final layers. using this finding, we fine-tune bert for aspect prediction, which leads to more effective and faster tuning. in line with linguistic theory, information about the boundedness of the action is encoded in the model's context representation and affects the choice of aspect: shifting representations towards the "bounded" space positively affects prediction of perfective forms (and negatively-of imperfective), and vice versa. prediction of aspect is not affected by random interventions. we checked that the causal probe is selective and does not affect irrelevant categories, e.g., number. a particular challenge is caused by contexts where more than one aspect form can fit grammatically and semantically, which we call alternative. we investigated whether encodings of aspect differ in these contexts from non-alternative ones. we find that bert is consistently uncertain about aspect forms in alternative contexts. causal interventions also have a stronger effect in such contexts. our error analysis shows that these contexts do not have enough cues to help the model (or a human) decide which aspect to use. in future work, we plan to explore additional languages; investigate how transformers encode relations between verbs and the cue words; and inspect the connections between aspect and tense of verbs, and context words expressing time, by intervening in the attention weights. we also plan to investigate aspect prediction in contexts lacking cue words, where information affecting the choice of aspect is presented in the neighboring sentences and requires reasoning. the practical goal is to deploy the aspect prediction in the production language teaching/learning system, to help learners master this advanced and complex feature of russian.