Re-solve it: simulating the acquisition of core semantic competences from small data (2020)

Abstract: Many tasks are considered to be ‘solved’ in the computational linguistics literature, but the corresponding algorithms operate in ways which are radically different from human cognition. I illustrate this by coming back to the notion of semantic competence, which includes basic linguistic skills encompassing both referential phenomena and generic knowledge, in particular a) the ability to denote, b) the mastery of the lexicon, or c) the ability to model one’s language use on others. Even though each of those faculties has been extensively tested individually, there is still no computational model that would account for their joint acquisition under the conditions experienced by a human. In this paper, I focus on one particular aspect of this problem: the amount of linguistic data available to the child or machine. I show that given the first competence mentioned above (a denotation function), the other two can in fact be learned from very limited data (2.8M token), reaching state-of-the-art performance. I argue that both the nature of the data and the way it is presented to the system matter to acquisition.

## introduction many tasks and datasets are considered solved problems in the computational linguistics literature. however, the data, training regimes and system architectures required to obtain top performance are unrealistic from the point of view of human cognition. thus, state-of-the-art data-driven frameworks can be considered excellent engineering solutions to particular linguistic tasks, but they are not usable as 'models' of language acquisition, and thus of limited applicability to test hypotheses about human language. this paper argues that core problems in computational linguistics should be 're-solved' -solved again -not as tasks, but as phenomena to simulate. this would involve a more careful attention to i) the type of data fed to the system; ii) the knowledge already in-built in its architecture; iii) the mode of learning implied by the training regime; iv) the specific features exploited by the learning process; and of course, v) the theoretical proposals explaining the phenomenon. some of these desiderata have started being explored in the literature: the blackbox nlp events, for instance, are currently fostering important discussions on the interpretability of artificial neural systems @xcite . still, the field remains far from satisfying all of them. the work described in the following pages is a step towards the simulation of a particular phenomenon: the acquisition of core semantic competences. its specific focus is on data: more particularly, the type and size of the corpus a system is exposed to. as we will see, talking about input data (desideratum i. above) naturally brings in questions about learning mechanisms (ii. and iii.), and about representation (iv.) let us first note that an nlp system is typically exposed to at least hundreds of millions of words, if not billions. in contrast, a 3-year-old us child has only observed 25m words; a mayan child of the same age will hear as little as 5m words @xcite . in spite of the limited data they are exposed to, a child will reliably learn their language -this is referred to as the 'poverty of stimulus' in chomsky's work. if the stimulus is poor, we have to posit the existence of extra cognitive mechanisms to compensate for the lack of explicit linguistic evidence. for human syntax, chomsky famously advocated the existence of an innate universal grammar. i argue that there is an equivalent question to be asked in machine learning: indeed, the architecture (ii.) and hyperparameters (iii.) of a system, as well as the specific representation of the input data (iv.), are 'innate' features which are important to make explicit when describing a 'data-driven' system. in what follows, i investigate a particular configuration of a semantic acquisition model. specifically, i ask whether a particular type of input, based on individual grounded entities, can make up for data sparsity. following this hypothesis, i propose a model nicknamed eva (entity vector aggregator) 1 and compare it to the behaviour of a characterbased language model with no access to referential information. i perform a battery of tests including similarity, compatibility and acceptability judgements, as well as lexical relation categorisation, and demonstrate that when fed with the right data and the right representation, the model learns core semantic competences from as little as 2.8m words. ## semantic competence in the linguistic literature the notion of linguistic competence was introduced by chomsky aspects of the theory of syntax @xcite : competence is 'knowing one's language', and it must be distinguished from performance, 'using one's language'. according to chomsky, the study of linguistics is the study of competence. the linguist should try and elucidate the underlying structure of the mental phenomenon that leads to observable performance. in syntax, competence is usually defined in terms of grammaticality. the semantic equivalent is more difficult to pinpoint, and various proposals have been made. we will focus on three major positions in this paper: semantic competence as mastery of a) the lexicon; b) reference; c) language use. 1. mastery of the lexicon: following chomskian grammar, @xcite propose that the goals of semantics can be obtained by "subtracting grammar from the goals of a description of a language" (p172). according to them, this subtraction results in elements of lexical semantics, including relations such as hyponymy or antonymy, as well as word senses. semantic competence is then the ability to say that the paint is silent is not felicitous, that the bill is large is ambiguous, or again that there are two chairs in the room entails there are at least two things in the room. this paper starts from the assumption that all three definitions should be satisfied to speak of semantic competence. that is, i will posit that we need meaning representations that allow us to denote (to satisfy 2), for which we have descriptions or referring expressions by actual language users (to satisfy 3), and over which we can learn lexical relations (to satisfy 1). to achieve this, i will hypothesise a semantics based on instances (which can be aggregated into sets in a formal semantics fashion), but represented in terms of the statistical properties of language use. i will propose a representation which satisfies both requirements in §4. ## (small) data the input data we will work with is a set of grounded 'utterances' extracted from annotations in the visual genome (vg) dataset @xcite . this annotated set displays several important properties. first, it is small (around 2.8m tokens), so compatible in scale with the limited data a learner is exposed to. second, while it does not quite correspond to the type of sentences a child might be exposed to, it has some similarities with a realistic 'early' linguistic diet: the simple image annotations can be regarded as utterances of the type look! the dog is playing with the ball. third, it encodes the particular representational aspects we want to investigate: it is anchored in a clear notion of grounded instances (the individual objects in an image) and corresponding language use (the human-generated captions/annotations associated with each bounding box). the vg itself consists of a set of 108,077 images annotated with 5.4m region descriptions as well as 3.8m object referents, foot_1 2.8m attributes and 2.3m relationships. all objects are associated with a unique identifier, meaning that we can use such identifiers as a set of object variables for the particular universe defined by the vg. i follow the methodology introduced by kuzmenko and @xcite , who extract information about vg instances and use it to create a 'settheoretic' vector space. the example below shows a subset of the annotation for image id 1, after some initial pre-processing of the data. i assume that each image corresponds to some 'situation', in the spirit of @xcite . so situation 1 contains a tall brick building, identified by variable 1058508, on which we find a black sign, identified by variable 1058507. object types are recognisable through their suffix (e.g. building.n, sign.n), attributes consist of all other one-place predicates (e.g. tall, made|of |bricks); and relationships consist of all two-place predicates (e.g. on). we can straightforwardly obtain shallow logical forms associated with each situation, e.g.: building.n (1058508), tall (1058508), brick (1058508), sign.n (1058507), black (1058507), on(1058507, 1058508) for simplicity (and because each entity only occurs once in vg), i transform twoplace predicates into two one-place predicates: e.g. on(1058507, 1058508) becomes on(1058507, building.n ), on(sign.n , 1058508), respectively denoting the set of things that are on buildings, and the set of things that signs are on. the provided annotations together with the associated objects, attributes and relations can be taken to be a partial description of some subset of the real world (i.e. the subset encapsulated by the images). this can be illustrated by considering the following two instances of bear (objects referents 158539 and 1617277), together will all their annotated relations: we see that two instances can be annotated with different degrees of granularity in the vg. the first instance above includes many more details about the physical appearance of the bear, although the second includes the relation 'has fur', which is missed by the first one. that is, we have two different 'experiences' of bears, associated with utterances which, in a realistic situation, could have come from the learner's carer ('look at the bear next to the grass, look at its claws!') this is a typical example of the 'poverty of the stimulus' effect: the performance data associated with those instances of bear is both incomplete (the linguistic data only describes part of the bears) and inconsistent (the two descriptions are very dissimilar). in order to fully exploit the information in the vg, the annotated attributes and relationships are supplemented with a third type of linguistic information: simple extensional co-occurrences are computed, thus modelling an implicit logical and (the comma in the shallow logical form). i.e., if a bear occurs in an image under a cloudy sky, the model registers the co-occurrence of a bear entity with a sky entity. in what follows, i refer to such implicit relations under the general term of situational co-occurrences, to express the fact that the extensional co-occurrence takes place within a single situation. ## models in the field of computational linguistics, we often take models to be 'algorithms', independently of the data they are trained on, and often, independently from the assumptions that the algorithm is built upon. but as pointed out in the introduction to this paper, what is in the data, what is inbuilt in the algorithm, and how the data is presented to the learning process determines the extent to which one can speak of a scientific model of such or such phenomenon. therefore, i will talk of a 'model' as a combination of a particular system / algorithm (with its specific assumptions) and a particular type of data. in what follows, in the spirit of fixing the learning mechanism as much as possible, i present three models based on very similar algorithms (variants of skip-gram language models). i however vary the data input into the system, both in size and representation. pretrained fasttext (ft): the first model under consideration is a pre-trained, state-of-the-art set of vectors, generated with fasttext @xcite . the system is a character-based language model and thus unsuitable for encoding extensions (that is, it will not satisfy the ability to refer in our set of semantic competences). however, it provides a helpful upper-bound for the tasks that language models excel at. the fasttext vectors foot_2 were obtained from training over 16b tokens from a wikipedia snapshot, the umbc webbase corpus @xcite and statmt.org news dataset @xcite . fasttext trained on vg (ftvg): being based on simple character ngrams, fasttext is well suited to learning from smaller data @xcite . a fasttext model is trained with default settings on a portion of the visual genome's 5.4m region descriptions. such descriptions are short phrases or sentences of the type man wearing red and black surf apparel or red bus has advertisements that says 123 current account santander. from those descriptions, 2.8m tokens are used to match the size of the next system's background data (see 'eva' below). ftvg differs from ft not only with respect to the size but also the presentation of its data: while ftvg is exposed to raw utterances like ft, those utterances are broken down by instance (the data contains one description per line, so a target word is only ever found in contexts that pertain to the same instance). eva: finally, a third model is proposed. nicknamed eva (entity vector aggregator), it is generated straight from the extensional information contained in the vg annotations (the attributes, relationships and situational co-occurrences described in §3). before being fed to the skip-gram, the data is converted into a form akin to a set-theoretic vector space, using the procedure below. first, let p l be the predicates in some logic and u the entities in some universe. let us define a vector space model by using some interpretation function ||.|| to return the denotations of p l : an example of such a vector space is shown on the left of fig 1. i will refer to it as an entity matrix: each predicate is associated with a point expressed in terms of a vector basis u (so each dimension corresponds to an entity). the point is a straightforward representation of the extension of the predicate, and shows the entities that the predicate is true of. for instance, following the first row of the matrix, we find that the set of bears in our toy space is {x1, x2}. we can then define an aggregation function a d which groups context elements by predicate (e.g. all objects that are bears are aggregated into a single bear vector by pointwise addition): this operation results in a vector space such as the one shown on the right of fig 1. i will refer to it as a predicate matrix, since the basis is now made of the predicates in p l . an entity and predicate matrix are built for the vg, using the following restrictions. we ignore objects which are not annotated with any attribute or relation and would result in 0 vectors, thus obtaining around 2m entities. further, the entity matrix is constructed for predicates with frequency over 100. the result of this pre-processing is a 2m ×8284 matrix, where the predicates include 1188 object types, 798 attributes and 6283 relationships. figure 1 : left: an entity matrix, showing the entities that a predicate is true of. right: the corresponding predicate matrix, after aggregation with function a d . the first row is simply the pointwise addition of the first two columns in the entity matrix (the two bear entities). we can compare the figures above to the size of the large ft pretrained model by counting the number of unique tokens in the visual genome data, where 'unique' means that the token -whether object type, attribute or relationship -appears with a specific entity. since two-place predicates are transformed into two one-place predicates, the token is incremented for each argument separately (e.g. tree(3787077) is one token but parked-on(1058515,1058539) gives two tokens). this comes to 1,590,861 tokens for one-place predicates (object type and attributes) and 1,224,582 tokens for two-place predicates (relationships), thus around 2.8m tokens in total. 4 so eva is exposed to around 5700 less data than ft. it however has the advantage of being grounded in a clear notion of entity, thus matching the type of situated speech that forms most of a child's diet @xcite . further, the corpus size is in line with the number of tokens that a child might get directed at them in around a year of early life. the challenge for both ftvg and eva is to deal with the poverty of the stimulus. it is worth recalling that landauer and dumais (1997) suggested a solution to the problem which involved the use of principal component analysis (pca) as a dimensionality reduction method over a distributional matrix. the use of pca was meant to capture the main axes of variance over the limited data given to the model, allowing for fast generalisation. later models of distributional semantics, in particular neural architectures (e.g. the skip-gram of @xcite , do not explicitly mention dimensionality reduction as a way to successfully generalise over insufficient data, but the intuition remains implicit in the choice of dimensionality of the embedding layers. ftvg can rely on this mechanism, as well as its character-ngram model. eva needs its own way to surmount the issue, and because it encodes extensional information, it has to deal with the poverty of the stimulus not only at the level of the linguistic input, but also at the denotation level. (as we have seen before, the vg is in no way an exhaustive and accurate representation of the world.) in other words, we want eva to learn word embeddings at reduced dimensionality like its competitors, but from co-occurring extensions. the implementation of eva's embedding function is extremely simple and does away with some of the hyperparameters used by the original skipgram model. it takes a predicate matrix of size m × m, as would be produced by the aggregation function a d , subsamples the counts in that matrix (lowering very frequent counts), and performs a prediction task. that task consists in predicting whether a 'target' predicate (from the rows of the matrix) and a 'context' predicate (from the columns of the matrix) have been seen together in the description of a unique, grounded entity. a 'positive' example for the target bear might be the predicate brown (some bear entity has been seen to be brown). a 'negative' sample for the same target bear might be transparent . positive samples are taken in shuffled order from the subsampled matrix, while negative samples are randomly chosen amongst the zero values of the matrix. as in the original skip-gram with negative sampling, embeddings for each predicate are first randomly initialised and fine-tuned in the process of doing the prediction task. a dimensionality of 300 gives optimal results in preliminary experiments over our development set. therefore, results shown in the next sections are for this dimensionality only. we use two main approaches for testing representations: a) for the similarity task, as is standard in the literature, we directly compute the relative position of embeddings in the space using the cosine metric; b) for other tasks, we use the vectors as input to a very simple feedforward neural net architectures, learning dedicated weights for each task over frozen background representations. the architecture used across tasks is shown in fig. 2 . it consists of an input taking two word vectors, mapped to a hidden layer with reduced dimensionality. all hidden layer representations are then concatenated and passed through a second hidden layer which is then fed to the output layer. the output layer may consist of a single node or several, depending on whether the task requires regression or classification. a relu non-linearity is applied to the input layer and first (concatenated) hidden layer. a softmax is used on the last layer for classification tasks. the training regime for all three models is as follows. a single grid search is performed over the hyperparameter space, using 200 iterations of bayesian optimisation 5 with early stopping. for eva, i follow results by @xcite showing that linguistic phenomena are not all modelled by the same feature types in the vg. the validation data is used to select the best combination of feature types for a task (attributes, relations, situational co-occurrences), running the hyperparameter optimisation over all possible combinations. for all models, the five best hyperparameter sets are then selected according to validation results, and their stability is checked by performing 10 extra validation runs on each set, yielding 10 models per combination. the 10 models corre-5 this step uses the package available at https:// github.com/fmfn/bayesianoptimization . hyperparameters are optimised in the following ranges: learning rate and regularisation, [0.001 -0.01]; epochs, [100 -500]; minibatch size, [16 -1024]; size of hidden layer, from 100 to initial vector size for fasttext, and eva; and in the range 50 -100 for ftvg. ## evaluation procedure the entity matrix is evaluated in terms of the three aspects of competence we discussed in §2: knowledge of core lexical relations @xcite , knowledge of 'acceptable' use of a term (again, @xcite @xcite , and of course, ability to retrieve the extension of a term @xcite . lexical relations: the models are evaluated on three different datasets encoding different aspects of lexical knowledge, namely the relation of similarity, the ability of the model to classify specific relations such as hyponymy or meronymy, and finally the relation of incompatibility. first, similarity is evaluated against simlex-999 @xcite , a set of 999 pairs meant to capture similar rather than merely related items. the second test is to evaluate the ability of the model to distinguish between particular relations, as encoded in the bless dataset @xcite . bless contains 26554 pairs annotated for hyponymy, meronymy, co-hyponymy, attribute and event relations (an additional class is included for the absence of relation and is marked as 'random'). finally, the models are fed the incompatibility dataset of @xcite . this dataset contains 17973 word pairs associated with a compatibility judgement elicited from human annotators, on a scale from 1 to 7. so for instance, the pair airplane-baby has a mean score of 1 (fully incompatible), dessert-vegetable a score of 3 (somewhat compatible) and airplane-jet a score of 6.6 (close to full compatibility). all datasets are pre-processed to only keep the instances containing words present in the vg corpus, thus reducing the size of each available resource. the three models are evaluated on the same data. the overall number of tested instances is shown for each dataset in extensions: reference is deterministically encoded in eva. to make this clear, the next section provides illustrative examples of composition over vg categories. it also shows how referents are retrieved by the model and how dimensions can be aggregated to quantify over instances of subkinds. ## results this section contains results obtained on the validation and test portions of our datasets (see other datasets are less affected by feature selection but still show a preference for certain inputs. notably, bless performs at its best when using situation information. this is perhaps due to the distinctions that the model has to perform between classes such as taxonomic siblings, meronyms and 'other' relations. meronymy, in particular, requires to distinguish between items that simply co-occur in a situation (cat and garden) and those that cooccur but are also part of a relation (cat has-a paw). finally, relations seem crucial to get best performance on the incompatibility dataset. moving to the test set, we only retain the models with highest performance on the validation data (att+rel for simlex-999, sit for bless, att for acceptability and att+sit for incompatibility). overall results are provided in lexical relations: on the similarity task (simlex-999), eva outperforms ftvg by 10 points and lags behind the huge pre-trained ft by only one point. the classification of lexical relations (bless) is achieved by all systems with high accuracy, without significant differences. finally, performance on incompatibility is slightly over state-of-the-art level for both systems trained on the vg. eva gives the best overall score, outperforming pretrained ft by two points. in other words, the system built on denotations is the overall winner when considering lexical semantic acceptability: this time, we see that ftvg slightly outperforms eva (ρ = 0.58 vs ρ = 0.56), possibly by virtue of being a language model and thus more suited to encoding word usage, in the sense of 'distributional consistency' (see §2). it is nevertheless striking that minimal training over data which encodes no surface information achieves very reasonable performance, in the range of pretrained ft (ρ = 0.59). this can be taken as confirmation that acceptability can be learned successfully from an extensional representation. extensions: to complete the above results, let us recall that eva encodes reference by default, since the raw entity matrix (before aggregation and dimensionality reduction) captures how predicates are associated with entities. denotations are therefore returned fully deterministically. to illustrate this, i give here an example of basic intersective composition in the vg model. given the entity matrix, set intersection is simply expressed as pointwise multiplication. for instance, the denotation of the phrase brown bear can be obtained by multiplication of the bear and brown entity vectors. the operation returns brown bear entities in the visual genome with their other properties, as exemplified below: to illustrate this, let us consider the three subkinds 'playing white bears', 'playing brown bears' and 'cute teddy bears'. for each subkind, having applied intersective composition to the vectors in the entity matrix by pointwise multiplication, we obtain a denotation vector which can be aggregated using a d . the result of such operation is shown in fig 3 , with some relevant property vectors. following normalisation, the weight of a vector along a dimension can be read as the probability of an instance of the set represented by the dimension to have the property of the vector. so for instance, there is a 0.6 probability that a playing white bear is in water, versus a 0.36 probability for playing brown bears in the vg data. while being preliminary, such observations about the behaviour of the eva representations indicate that it could encode a number of important set-theoretic properties, making it properly compatible with formal semantics approaches. ## conclusion this paper made the case for solving existing tasks with models more in line with cognitive reality, ar-playing white bears playing brown bears cute teddy bears guing in particular for the use of smaller corpora. coming back to the points made in our introduction, i summarise the contributions of the model with respect to desirable aspects of a language acquisition 'simulation'. first, we have seen that systems trained on 2.8m tokens of vg data are competitive with a large language model pretrained on 5700 times more data when tested on core lexical tasks. on the similarity task, the reference-based system eva considerably outperforms a language model trained on the same amount of data, emerging as the best 'small data' model. notably, eva exploits a very specific presentation of the data, capitalising on its access to individual instances and its ability to choose the semantic information relevant for solving a given task. recent work has argued that language modeling is not enough for natural language understanding, and in particular that the relation between language and world(s) matters to comprehension (bender and koller, 2020). the results presented here support this view: not only are instances the basic building blocks of reference, but they might also be crucial to support the acquisition of lexical competences. with respect to the 'innate' mechanisms of the new model presented here, several shortcomings must be pointed out. first and foremost, eva assumes the availability of a denotation functionsome oracle able to map words to entities. this is of course something that children actually have to learn in the process of acquisition, and which probably proceeds in parallel with the training of other semantic competences. ideally, this assumption should be relaxed in future versions of the model to understand how much the system learns when its reference module is imperfect (for instance, by linking eva to an object recognition system from the language and vision literature). further, the learning mechanisms involved in eva may be too generic. we have used simple cooccurrence prediction for the acquisition of word semantics and non-linear regression/classification for task-specific competences, which goes well with claims that language can be acquired via generic cognitive functions. but the actual training regime used by those mechanisms may not be as plausible as it could be. in particular, it is unclear how much supervision is involved in the human acquisition of skills such as lexical relation recognition or acceptability judgements (see e.g. saxton, 2000 on the amount of negative input received by children from their carer). it is for instance dubious to argue that meronymies should be learned in a supervised fashion, with the learner being explicitly told that an ear or a whisker is 'part of' a cat. this aspect will have to be investigated further before claiming plausibility of the model. finally, the data used for our experiments is currently anchored in a visual dataset, and is therefore focused on concrete entities. linguistic competence involves mastery of abstract vocabulary, as well as reference to 'possible worlds', which can be different from the universe we perceive. it remains to be seen how the competences acquired over purely perceptual data can be usefully brought into skills that involve abstraction and modality.