Exploring sentence informativeness (2019)

Abstract: This study is a preliminary exploration of the concept of informativeness –how much information a sentence gives about a word it contains– and its potential benefits to building quality word representations from scarce data. We propose several sentence-level classifiers to predict informativeness, and we perform a manual annotation on a set of sentences. We conclude that these two measures correspond to different notions of informativeness. However, our experiments show that using the classifiers’ predictions to train word embeddings has an impact on embedding quality.

## introduction building robust and high-quality word representations is a key step for most nlp tasks. the quality mostly relies on the training corpus (its size and relevance), along with the training criterion and strategy. some noisy sentences might for instance damage the quality of embeddings, while many sentences do not contribute significantly in improving the representation of a word's meaning (see consequently, a criterion for sentence selection could be of great help in the context of low-quality or low-volume corpora. the concept at stake is the informativeness of a sentence towards a word. this criterion can be defined as follows : a sentence is informative with respect to one of its words if a person ignoring this word can correctly infer its meaning from this sentence to some extent. the sentences from the training corpora can be selected depending on their informativeness during embeddings' training. to this end, the informativeness of a sentence needs to be defined and modelled. the contribution of this paper is threefold. first, we propose a method to automatically get an artificial informativeness measure of any sentence towards a target word, using sentence classification. then, we design a labelling process in terms of informativeness scores to evaluate our models' results. finally, we give preliminary results on the use of the artificial informativeness measure for the training of word embeddings. a : i went to the *** c : it was good *** in those terms. b : i went to the *** to withdraw money d : the aquarium did a blood *** to determine their gender. ## related work while dealing with low-resource languages and specific domains has received much attention in the nlp community, to the best of our knowledge, this is the first attempt to investigate the potential impact of the notion of informativeness on building meaning representations from limited data. several attempts at improving or adapting word embeddings to restricted tasks and languages involve making use of morphological information @xcite ; fine-tuning pre-trained, global-purpose embeddings on a restricted domain @xcite ; refining them with the help of semantic resources @xcite , known as retrofitting ; or using attention mechanisms on contexts to better represent rare words @xcite . @xcite show that, when training word embeddings, using an in-domain corpus specialized in a task is better than having a large, mixed-domain corpus, which can lead to a decrease in performance. the closest study to the present one is that of @xcite . the authors argue that just as humans do not need many examples to learn the meaning of a word, the word2vec architecture can be adapted to learn a competitive representation of a word from 2-6 occurrences only. this can be critical to build an embedding for rare words or in situations of scarce data. their experiments involve both wikipedia definitions -in principle, maximally informative from a human perspectiveand naturally-occurring sentences. given the better performance of the former, they observe that accounting for the informativeness of a sentence can be useful to learn good representations. a line of work related to informativeness concerns the automatic extraction of sentence examples for lexicographic resources or knowledge bases. @xcite restrict their search with different linguistic criteria designed to help a reader grasp the meaning of a word more easily, such as sentence length and the presence or abscence of rare words, pronouns, or typical collocations. another approach for finding knowledge-rich contexts, as coined by @xcite , consists in finding sentences with knowledge patterns, that is, linguistic expressions that describe the semantic relations of a word, such as "is a kind of" to denote hypernymy @xcite . according to our definition, if a sentence is very informative with respect to one of its words, a reader who does not know that word can infer its meaning from the sentential context. intuitively, in such a sentence, a person who knows the word should be able to predict it with high accuracy. we create a task based on this intuition : for a given target word, a model gets as input sentences with a blank at the target position. the model must learn to distinguish sentences that originally contained the target word from those which did not. for instance in ## labelling data given a sentence, the classification algorithm outputs a probability of belonging to the class of the target word. according to our hypothesis, this method gives us a measure of informativeness for each pair [sentence, target word]. in order to evaluate this measure, two annotators label a set of sentences. then, we perform an inter-annotator agreement study to select the best methodology and scoring scale. for a first agreement, the two annotators both label 150 sentences randomly extracted from a portion of the annotated english gigaword corpus 4 @xcite . in each sentence, a target word is randomly masked. it has to be a noun, verb, adjective or adverb, excluding proper nouns and auxiliary verbs. the annotators give two scores to each sentence : inf o1 before seeing the masked word (from 1 to 10, indicating how much they can guess about the target word given the sentence) and inf o2 after seeing this word (from 1 to 10, expressing to what extent they expected seeing the true word). the spearman correlations between both annotators' scores are relatively low (mean @xmath0.29 for inf o1 and 0.37 for inf o2), especially for adverbs (@xmath1.12 for inf o2). overall, the annotators' remarks show that these measures are very subjective. consequently a second labelling agreement is designed, relying on a more explicit measure : inf o3. the range of scores is reduced to be from 1 to 5, and precise scoring guidelines are designed to ensure a common interpretation of scores across annotators. the inf o3 measure answers the question : how much information does this sentence give about the meaning of the target word ? medium scores (between 2 and 4). however, the correlation is very low for adjectives (0.080). to sum up, the second labelling agreement gives a more objective scale, making the annotators usually have close scores, and agreeing on extreme scores. thus, we keep this process for the rest of the labeling. 20 words are selected out of the 50, excluding adjectives : call, go, range, carry, charge, coach, hold, return, check, investigator, shot, education, paper, side, figure, post, tell, fire, put, test. the annotators label together five more sentences for each of the target words, among which we include definitions from wordnet and the online cambridge dictionary 5 to ensure the presence of highly informative sentences in the manual annotations. the final evaluation corpus consists of 7 sentences for each of these 20 target words. ## experiments on test and annotated data we extract 20,000 sentences for each of the selected target words : 10,000 containing the target word and 1,000 for each of its 10 distractors. 80% are used for training, 10% for development and 10% for testing. we use a different portion of gigaword than the one used for the manual annotation. we compare our methods to a simple c2v-based baseline. given a sentence with a target word, we calculate the similarity between the vectors of all distractors, as well as of the target word, with the c2v context vector of the sentence. the potential fillers are sorted by similarity and the rank of the target word is used as an indicator of informativeness (the more similar, the more informative). the classifiers described in section 3.2 are trained on the extracted sentences for each selected target word. results on the test set are found in examples of sentences for the target word tell can be found in classifier accuracy on test data spearman's r on manual data linguistic-based 0.890 -0.195 fasttext 0.868 0.101 c2v logistic regression 0.927 -0.179 c2v feed-forward nn 0.946 -0.162 c2v baseline 0.759 -0.070 ## experiments on word embeddings training we concluded in the previous section that the classifiers output a different kind of informativeness than the human annotations. in this section, we test the effect of the classifiers' informativeness on word embeddings training. we use the probabilities outputted by the fasttext classifier for this task. independently of the correlations with the human annotation, this classifier's distribution of outputted probability is the least skewed. the others assign very high probabilities for a large portion of sentences, preventing a clear discrimination between informative and not informative sentences. following @xcite , for each target word, we sort the sentences of the test set by probability of belonging to the class of the target word according to the classifier. we select the 250 sentences with lowest and highest probability, and make also a random selection. the target word is replaced by a new token "target_word_new" in all sentences. we initialize all weights of a word2vec model @xcite using pre-trained word embeddings. 6 unknown words are initialized randomly. we fine-tune the pre-trained embeddings on each set of sentences. then, we compute the similarities between the vector of "target_word_new" and the pre-trained embedding of the target word (gold standard). the second part of we investigate the reason behind the large disparities among words for this task. we consider two indicators for each word : its frequency in google unigrams @xcite and its polysemy in wordnet @xcite . the spearman correlation between the word frequency and the difference between sim-inf and sim-random is high and negative (-0.6) with a p-@xmath2.05. thus, for frequent words, the difference in similarity is low between highly informative sentences and random sentences. on the contrary, in the case of infrequent words, the informative sentences provide better embeddings than the random ones. the correlations with the other columns of in this study, we have introduced the notion of informativeness and proposed an automatic method to predict it for a given set of words. we have performed a manual annotation of informativeness and used it to evaluate our models. despite their efficiency in classifying sentences, classifiers do not perform well on the manual dataset, suggesting that what the models are learning is different from our definition of informativeness. however, when using the informative and uninformative sentences predicted by the classifier on the task of word embeddings training, we observe that training on uninformative sentences leads in general to lower quality embeddings. moreover, the average informativeness score of sentences varies a lot depending on the target word : for infrequent target words, informative sentences usually provide a higher gain compared to random sentences. while not allowing for strong claims about the impact of informativeness on word representations, we believe the results of the present study put forward several interesting questions worth of further research. first of all, it remains to be seen whether the human concept of informativeness can be of help to nlp applications. several modifications can be introduced to our algorithms, such as the number of distractors or a variety of source corpora ; and annotating a bigger dataset could possibly allow for supervised learning of informativeness. another open question is whether embeddings might benefit more from an alternative concept of informativeness. for instance, although definitions are -or should be-informative for humans, they are not very common in most corpora. for this reason, language representations trained on common corpora, like the ones our classifiers rely on, may find them atypical. with a better understanding of informativeness and automatic predictors, we believe that a study of the features that make a sentence informative for a word would be of great theoretical as well as practical interest, possibly allowing to build target-word-independent predictors. such features could involve context words sharing a topic with the target, or the degree of polysemy of context words.