BLM - A gr F : A New F rench Benchmark to Investigate Generalization of Agreement in Neural Networks (2023)

Abstract: Successful machine learning systems currently rely on massive amounts of data, which are very effective in hiding some of the shallowness of the learned models. To help train models with more complex and compositional skills, we need challenging data, on which a system is successful only if it detects structure and regularities, that will allow it to generalize. In this paper, we describe a French dataset (BLM-AgrF) for learning the underlying rules of subject-verb agreement in sentences, developed in the BLM framework, a new task inspired by visual IQ tests known as Raven’s Progressive Matrices. In this task, an instance consists of sequences of sentences with specific attributes. To predict the correct answer as the next element of the sequence, a model must correctly detect the generative model used to produce the dataset. We provide details and share a dataset built following this methodology. Two exploratory baselines based on commonly used architectures show that despite the simplicity of the phenomenon, it is a complex problem for deep learning systems.

## introduction over the last years, driven by the surge in deep learning methods, models in nlp have become very powerful. they have even reached superhuman performance on standard benchmarks such as superglue @xcite and squad @xcite . deeper probing, though, shows that this is due to the models' surprisingly robust superficial natural language understanding figure 1 : an example raven's progressive matrix (best seen in colour). the matrix is constructed according to two rules: (i) the red dot moves one place clockwise when traversing the matrix left to right; (ii) the blue square moves one place anticlockwise when traversing the matrix top to bottom. the task consists in finding the tile in the answer set that correctly completes the sequence, indicated with a double border. in this paper, we describe such a benchmark, developed based on a new method described in @xcite , and summarized in section 2. the method defines a procedure for building datasets that capture specific linguistic phenomena in a structured problem, inspired by the visual patterns detection tasks in raven's progressive matrices (rpm) @xcite , as in figure 1 . the visual rpms manipulate elements with attributes such as position, shape, colour and size. the language matrices manipulate phrases, dependencies in the syntactic tree, and lexical, grammatical and semantic attributes between connected elements of a sentence. to successfully tackle such a complex problem, a system must detect structure and patterns in the data. to complement the structure of the problem, the candidate answers set is also specifically designed -the negative answers are built following specific criteria -to help determine which facets of the problem the system is able to learn, and which it is not. by enabling different levels of analysis, from the solution of the task in different controlled setting to the analysis of the errors, this dataset intends to support the development of neural models with stronger abilities of abstraction and generalization, and more complex and compositional skills, that could learn robust models from few examples, and ultimately be deployed on lowresource languages. the code and the data are available here: https://github.com/clcl-geneva/ blm-snfdisentangling . ## blm-agrf: blackbird's language matrices for agreement the data format we present has been called blackbird's language matrices (blms) @xcite , because it requires the presentation of the linguistic phenomenon of interest in the form of a complex set of sentences that have both syntagmatic and paradigmatic relations, thereby, like in the rpm visual version, forming a matrix structure. a blm has a structure defined by a combination of rules. the starting point is defining the linguistic problem that needs to be learned (e.g. subject-verb agreement) and the grammatical rules that define it. the combination of rules can be complex and each rule can act as an interfering factor obfuscating the other rules. the next step is to devise the rules governing the abstract automatic generation process that embody the properties of the linguistic phenomenon and its underlying rules. combining these examples of grammatical rules will produce templates that can be used to automatically create large samples of data with lexical/structural variety. to allow for probing the learned model, apart from the correct answer, the answer sets contain negative examples built by corrupting some of the generating rules. this helps investigate the kind of information and structure learned, and the type of mistakes a system is prone to. ## experiments transformers and other neural architectures have shown very high performance on a variety of nlp tasks. we describe here two baselines to investigate the difficulty in learning the underlying regularities of subject-verb agreement on the proposed dataset. figure 7 shows the general process flow. each sentence in the input is encoded separately using a pre-trained multilingual transformer model, foot_0 which were shown to capture, among others, syntactic information @xcite . main clause 1 l'ordinateur avec le programme est en panne. 2 les ordinateurs avec le programme sont en panne. 3 l'ordinateur avec les programmes est en panne. 4 les ordinateurs avec les programmes sont en panne. 5 l'ordinateur avec le programme de l'expérience est en panne. 6 les ordinateurs avec le programme de l'expérience sont en panne. 7 l'ordinateur avec les programmes de l'expérience est en panne. 8 les ordinateurs avec les programmes de l'expérience sont en panne. completive clause 1 jean suppose que l'ordinateur avec le programme est en panne. 2 jean suppose que les ordinateurs avec le programme sont en panne. 3 jean suppose que l'ordinateur avec les programmes est en panne. 4 jean suppose que les ordinateurs avec les programmes sont en panne. 5 jean suppose que l'ordinateur avec le programme de l'expérience est en panne. 6 jean suppose que les ordinateurs avec le programme de l'expérience sont en panne. 7 jean suppose que l'ordinateur avec les programmes de l'expérience est en panne. 8 jean suppose que les ordinateurs avec les programmes de l'expérience sont en panne. relative clause 1 l'ordinateur avec le programme dont jean se servait est en panne. 2 les ordinateurs avec le programme dont jean se servait sont en panne. 3 l'ordinateur avec les programmes dont jean se servait est en panne. 4 les ordinateurs avec les programmes dont jean se servait sont en panne. 5 l'ordinateur avec le programme de l'expérience dont jean se servait est en panne. 6 les ordinateurs avec le programme de l'expérience dont jean se servait sont en panne. 7 l'ordinateur avec les programmes de l'expérience dont jean se servait est en panne. 8 les ordinateurs avec les programmes de l'expérience dont jean se servait sont en panne. answer set for problem constructed from lines 1-7 of the main clause sequence 1 l'ordinateur avec le programme et l'experiénce est en panne. n2 coord n3 2 les ordinateurs avec les programmes de l'experiénce sont en panne. correct 3 l'ordinateur avec le programme est en panne. figure 5 : blm-agrf instances for verb-subject agreement, with two attractors (programme, experiénce), and three clause structures. and candidate answer set for a problem constructed from lines 1-7 of the main clause sequence. les programme le [mask] avec expérience l' de panne en sont . input output serveurs, appareils, ordinateurs, connexions, logiciels top 5 les programme le serveurs avec l' de panne en sont . camembert masked language model expérience figure 6: creation of lexical variants by generating variations of a masked input using camembert (martin et al., 2020) ## related work the current paper does not have any direct comparison, as this is the first proposal of a dataset for language using a blm scheme. but it is inspired by work on generating rpm problems, and on solving such problems in computer vision, and it contributes to the investigation on learning of agreement by neural networks. ## conclusions in this paper we have introduced blm-agrf, an instance of blackbird's language matrices (blm) @xcite . this novel linguistic dataset is generatively constructed to support investigations in representation learning of grammatical rules. each instance, consisting of a sequence of sentences and a candidate answer set, was built using a combination of rules, to provide a layered and structured dataset for learning more complex models. the various layers of the dataset allow for a variety of explorations, from disentangled sentence representations to capture structure and regularities within a sentence, to modular architectures that could capture structure and regularities in the sentence sequences. the purposefully built candidate answers supports more in-depth analyses of the behaviour of a system, and provide insights into the source of prediction errors. experiments using baseline set-ups -feedforward networks and cnns -show that the task is difficult for previously successful sentence representations and neural architectures, despite the fact that the agreement rule they are supposed to discover is rather simple. this supports our hypothesis that the task the data embodies could provide a new benchmark for modeling generalization and abstraction.