Data Collection vs. Knowledge Graph Completion: What is Needed to Improve Coverage? (2021)

Abstract: This survey/position paper discusses ways to improve coverage of resources such as WordNet. Rapp estimated correlations, rho, between corpus statistics and pyscholinguistic norms. rho improves with quantity (corpus size) and quality (balance). 1M words is enough for simple estimates (unigram frequencies), but at least 100x more is required for good estimates of word associations and embeddings. Given such estimates, WordNet’s coverage is remarkable. WordNet was developed on SemCor, a small sample (200k words) from the Brown Corpus. Knowledge Graph Completion (KGC) attempts to learn missing links from subsets. But Rapp’s estimates of sizes suggest it would be more profitable to collect more data than to infer missing information that is not there.

## wordnet coverage and semcor wordnet foot_8 @xcite @xcite @xcite is widely cited because of accessibility foot_9 as well as coverage. why is the coverage as good as it is, and how can it be improved? unlike other methods for constructing lexical resources @xcite @xcite , wordnet was developed in tandem with semcor, a small subset of the brown corpus, tagged with pointers into wordnet. the team constantly tracked coverage as indicated by the reference to 96% below: while semcor has much to recommend it, there are also some obvious concerns. semcor is only 200k words, probably not enough given rapp's estimates above. coverage of wordnet could be improved by building something like semcor, but based on a larger corpus of more modern material. alternatively, it might be possible to combine small annotated corpora with larger unannotated corpora. ## knowledge graph completion (kgc) an alternative suggestion for improving word-net coverage is: knowledge graph completion (kgc) 11 @xcite @xcite . a standard kgc benchmark is wn18. 12 wn18 is a graph @xmath0, e). there are 41k vertices, v . each vertex is a wordnet synset, a pointer to a set of synonymous lemmas in wordnet. there are 118k such synsets in wordnet. the edges, e, connect two vertices with one of 18 relations. the relations also come from word-net. some relations are more frequent than others. many of the relations come in pairs, as shown in the kgc task is to infer subsets of these graphs from other subsets of these graphs. that is, kgc splits e randomly into three sets: train, validation and test. wn18 consists of 141k edges in train, 5k in validation and 5k in test. for each set, we have a set of input features, x, and a set of output labels, y . the standard procedure uses x train and y train to fit a model. this model is used to predict ŷtest from x test . the predicted values, ŷtest , are compared with the gold labels, y test , to compute a score. there is a considerable literature on kgc methods, e.g., trans[dehrm], kg2e, conve, complex, distmult @xcite @xcite @xcite @xcite the next two subsections address two concerns with the kgc literature and the wn18 benchmark: in particular, of the 1074 derivationally related edges in the wn18rr test set, all of them are also in one of the other sets, but in the reverse direction. the 1074 reversed edges are split across test (24), train (1011) and valid (39). because of this leakage, a system can do very well on this benchmark without learning anything useful about wordnet. simply reverse edges in the training set and predict that those reversed edges will appear in the test set (unless they have already been seen in the training or validation sets). such a system will correctly predict 1 -24/1074 = 98% of the derviationally related edges (1074/3134 = 34% of the test set). one could correct for this leakage by removing the redundant edges, just as we removed redundant edges to reduce wn18 to wn18rr. ## conclusions what is needed to improve wordnet coverage? we started with rapp's estimates of ρ, correlations of corpus statistics and psycholinguistic norms. ρ improves with quantity (corpus size) and quality (balance). unbalanced corpora need to be larger (100x) than balanced. estimates of second order quantities (word associations and edges in wordnet) require at least 100x more data than first order quantities (frequency/familiarity). rapp's estimates suggest there is more information in larger samples than in smaller samples. wordnet is based on semcor. it is remarkable that wordnet works as well as it does, given rapp's estimates. one approach to improving coverage is knowledge graph completion (kgc). kgc attempts to learn missing links from subsets. the kgc benchmarks, wn18 and wn18rr, are deeply flawed. information is leaking between training and test sets. some of this leakage has been previously reported, and some has not. but more seriously, if semcor is already too small and dated, data collection is more likely to succeed than attempts to infer information that is not there.