A Benchmark for H indi Verb-Argument Structure Alternations (2025)

Abstract: In this paper we introduce a Hindi verb alternations benchmark to investigate whether pretrained large language models (LLMs) can infer the frame-selectional properties of Hindi verbs. Our benchmark consists of minimal pairs such as ‘Tina cut the wood’/*‘Tina disappeared the wood’. We create four variants of these alternations for Hindi to test knowledge of verbal morphology and argument case-marking. Our results show that a masked monolingual model performs the best, while causal models fare poorly. We further test the quality of the predictions using a cloze-style sentence completion task. While the models appear to infer the right mapping between verbal morphology and valency in the acceptability task, they do not generate the right verbal morphology in the cloze task. The model completions also lack pragmatic and world knowledge, crucial for making generalizations about verbal alternations. Our work points towards the need for more cross-linguistic research of verbal alternations.

## introduction a question that has been investigated repeatedly is whether large language models (llms) are able to learn the syntactic and semantic generalizations of a natural language given the diverse data they are trained on. a number of studies have created linguistic benchmarks consisting of syntactic phenomena (e.g. active-passives, syntactic agreement) using minimal pairs. llms are then tested on acceptability judgement tasks, comparing their performance with human judgements @xcite @xcite . recent work evaluated transformer llms on hindi syntactic agreement @xcite . llms' performance was robust despite hindi's complex split-ergative system. with respect to verb argument structure alternations, crosslinguistic results are mixed. for english as well as chinese, experiments show that model performance is relatively poor for argument structure @xcite . for japanese on the other hand, models seem to match human accuracy @xcite . there is no previous work evaluating llms' knowledge of verb argument structure for hindi. the core meaning of an event is contributed by the verb in a sentence or context. it comes densely packed with information about the number of arguments (or participants), their role, and how they are related to each other. this information comprises syntactic knowledge: mapping the verbal morphology to the correct number of arguments in the sentence. it also contains semantic knowledge where the verb and its arguments contribute to the event meaning. in this paper, we use both acceptability judgements and cloze-style sentence completions following @xcite . we evaluate both masked and causal models, and also compare multilingual and monolingual models @xcite . results from our acceptability task indicate knowledge of the mapping between verbs and syntactic frames. at the same time, the best performing models from this task are not able to predict the correct verb forms in a cloze-style sentence completion. we show that verb alternations require llms to make generalizations that are different from other syntactic phenomena. ## alternations in hindi hindi verbs carry morphosyntactic information that signals the change in arguments. in the following examples, the base form of an intransitive verb /ub@l/ 'boil' changes to transitive in /ubal/ and then to the indirect causative in /ub@lva/. while there is variation in the way each of these alternations are realized (e.g. some verbs have a null transitive alternation), there is a surface form-function map-ping unlike english. for example, john broke the window and the window broke are causative and intransitive, respectively but without any surface differences. prog.sg.m t h a aux.pst.sg.m 'the water was boiling.' (2) l@óka boy.3.sg.m pani water.m ubal boil.dcaus r@ha prog.sg.m t h a aux.pst.sg.m 'the boy was boiling the water.' (3) l@óka boy.3.sg.m b@cce-se child.3.sg.m-agt pani water.m ub@l-va boil-icaus r@ha prog.sg.m t h a aux.pst.sg.m one challenge in developing such an evaluation dataset for hindi is that arguments are regularly dropped (elided), and case markers on the nouns exhibit case syncretism. for example in (5) the case /-se/ describes a source (mira) and takes a transitive form. in example (4), the same case marker /-se/ is instrumental, occurring with a causative form of the verb /b@d@l/ 'change'. for our benchmark, we choose sentences where all argument and adjunct slots are filled. in our minimal pairs, the acceptable sentence has the /-va/ causative as in (3), with three arguments (causer, agent, and patient). an additional instrumental argument is also added to restrict the choice to causatives and avoid ambiguity. we then replace the grammatically correct verb with an incorrect form to test for awareness of the correct frame. ## benchmark construction to examine the extent to which pretrained models effectively leverage syntactic and semantic information from the context, we introduce a benchmark of minimal pairs in hindi. we construct minimal pairs such that both sentences have a common sentential prefix and a grammatical or ungrammatical verb (which occurs in sov order in hindi). the last word in each sentence is a past tense auxiliary (the verb occurs at second last position). all examples are shown in our benchmark consists of 56 verbs that have been selected on the basis of different criteria. we first chose verbs on the basis of their frequency using the shabd database corpus @xcite . we have selected verbs that are high on the zipf scale to maximize the chance of their occurrence across model training corpora. this ensures that these verbs are well represented and we minimize out-of-vocabulary effects. we then categorized verbs according to their valency. since the goal of this work is to study how well pretrained models understand the verb argument structure of hindi verbs, the final verb list maps to all three syntactic frames -intransitive (1 argument), transitive (2 arguments), and ditransitive (3 arguments). we also consider finer classifications, e.g. intransitive verbs which are further categorized into unergative and unaccusative verbs. transitive verbs contain a sub-category of ingesto-reflexives. the final set has 28 intransitive verbs (13 unergatives and 15 unaccusatives), 23 transitive verbs (with 13 ingestoreflexives), and 5 ditransitive verbs. for our evaluation, we generate four variants of our benchmark that are described below: different verb: the two verbs are morphologically unrelated forms, with different valency. same verb: the two verbs are morphologically related, but with a different valency. no case(e): the two verbs are morphologically related, but the verbal aspect is habitual, which results in the ergative marker on the subject being removed foot_0 . no case(i): the two verbs are morphologically related, but we remove the additional adjunct argument from both sentences. we can think of the 'different verb' and 'same verb' variants of the dataset as being maximally specified in terms of the arguments and adjuncts, allowing us to test whether the mapping between morphological encoding and valency is learned. the 'no case' variants compares the morphologically related verbs but the case information is changed. this is done primarily to test whether the models are robust to subtle changes in the surface forms of the arguments. each set has 56 pairs for the acceptability task. to collect acceptability judgements, we conducted a forced choice acceptability judgment experiment using pcibex @xcite . participants were asked to choose the most acceptable sentence (see appendix b.1 for all details). we present annotator accuracy along with llms' in ## models we test our dataset using six models via the huggingface transformers library @xcite ) -four bert-based masked language models (xlm-roberta, muril, indicbert v2 and hind-bert) and two causal language models (mgpt and bloom). all models, except for hindbert are multilingual models and differ primarily in terms of their size and the language(s) they are trained on. (an overview of models is presented in appendix a). mgpt has 1.3b and 3b variants and bloom has 560m, 1.1b, 1.7b, 3b, 7.1b, 13b, and 176b variants. we found that as the parameters increased beyond 1b for the these models, performance worsened. on the 'different verb' variant of our benchmark the performance of the 1.7 million and 1.1 billion variants of the bloom model was the same (75% accuracy). however, for bloom 3 billion, the performance dropped to 62.5%. these results are similar to kryvosheieva and levy (2025)'s results for hindi where the performance dropped for bloom's 3 billion variant. hence, in this study we present results only from mgpt 1.3b , bloom 560m and bloom 1.1b . we evaluate models' performance using sentence score. for causal models, the score of a sentence is computed as the sum of the logprobabilities of each token conditioned on the sequence of preceding tokens. whereas for masked models, we employ the pseudo-log-likelihood (pll) scoring method introduced by @xcite . the original pll scoring method estimates sentence probability by masking words iteratively in a sentence, calculate the probability of each mask, and then multiplying probabilities of each word @xcite . however, this method does not mask within word tokens of a multi-token word and results in inflated scores @xcite . therefore, we calculate the pll score for each word by masking within word tokens as well. we calculate the pll score for each sentence individually. the sentence with the greater pll score is deemed to be more acceptable than the other. we then evaluate these probabilities against the gold data to calculate accuracy. the syntactic log-odds ratio (slor) @xcite @xcite is also another method that is used to score sentences, while controlling for sentence length and lexical frequency. we did not calculate this score in our work as the training data for all the models that we tested was not publicly available. we also note that in our dataset all the example sentences were of similar length (between 9-11 words). ## results acceptability task: for the 'same verb' task, there is a drop in performance, which is also reflected in the human accuracy. but the performance drop is more prominent in xlm-r-large and muril. for the 'no case(i)', both indicbert and hindbert are less accurate. this shows that using an additional instrument argument, and maximally filling all argument and adjunct slots does help llms to discriminate, while it makes little difference to humans. the weak performance for 'no case(e)' variant is surprising. all models are less accurate, showing that case information like the ergative marker /-ne/ is an important cue for models. @xcite also report that overt morphological case marking makes model prediction easier for syntactic agreement phenomena. as discussed in section 2 hindi verbs can be classified into different categories according to their valency and type. in order to understand whether these distinctions impact model performance, we further analyze our results for each of the different categories. for intransitives and transitives, models' performance across each task was uniform, however we do see a decrease in performance for ditransitives in all variants except for the 'different verb' task (see sentence completion task: we also carried out a cloze-style sentence completion task. we took the best performing models-the multilingual indicbert v2 and monolingual hindbert and asked them to complete the sentence as shown in models rarely generated verbs with the /-va/ causative. rather, the completions are usually transitive or ditransitive verbs. sometimes these completions may be grammatical due to the ambigu- sentential prefix expected predicted moh@n-ne b@cci-se p@nk h e-se buj h vai 1. k h @ridi momb@tti t h i (made to (bought) 'mohan made/had the girl extinguish) 2. nikali the candle with the fan.' (removed) we also collected human judgements to see whether they prefer the gold completions or models' predictions using a forced choice task. annotators were shown pairs of completions and asked to select the most grammatical option. we then calculated the percentage of times annotators agreed with the gold completions, finding a mean agreement rate of 85.9%, which indicates strong preference for the gold completions over the models outputs (see appendix b.2 for the experiment details). ## discussion in this work, we have created a benchmark of minimal pairs with four variants to test the knowledge of hindi verbal alternations. our benchmark has been publicly released. 2 we show that masked models are the closest to human performance for the acceptability task, but when these models are used in a cloze-style completion, their completions lack integration of both syntactic and semantic knowledge. this indicates an incomplete understanding of verb frames. hindi morphologically encodes its verbal argument structure, and this information seems to give the models a boost in the 'different verb' variant @xcite . at the same time, case syncretism is a disadvantage, which makes the argument and adjunct distinction more challenging 2 https://github.com/kjain93/verb-knowledge-in-llms for 'no case'. both indicbert v2 and hindbert are fairly large models, trained on 20 billion and 1.8 billion tokens respectively. it is unlikely that increasing the size of the models will help to improve their event semantics knowledge. we see that that current models have close to human performance for acceptability judgements but they are far less robust in a generation task. the ungrammatical completions indicate that the models have a surface understanding of valency but are unable to integrate this knowledge with event meaning. our research points towards the need to investigate syntactic and semantic integration in llms.