Fine-grained General Entity Typing in G erman using G erma N et (2021)

Abstract: Fine-grained entity typing is important to tasks like relation extraction and knowledge base construction. We find however, that fine-grained entity typing systems perform poorly on general entities (e.g. “ex-president”) as compared to named entities (e.g. “Barack Obama”). This is due to a lack of general entities in existing training data sets. We show that this problem can be mitigated by automatically generating training data from WordNets. We use a German WordNet equivalent, GermaNet, to automatically generate training data for German general entity typing. We use this data to supplement named entity data to train a neural fine-grained entity typing system. This leads to a 10% improvement in accuracy of the prediction of level 1 FIGER types for German general entities, while decreasing named entity type prediction accuracy by only 1%.

## introduction the task of fine-grained entity typing is to assign a semantic label (e.g. '/person/politician' or '/location/city') to an entity in a natural language sentence. in contrast to coarse grained entity typing it uses a larger set of types (e.g. 112 types in the figer ontology @xcite ), and a multilevel type hierarchy. an example of fine grained entity typing can be seen in figure 1 . finegrained entity typing is an important initial step in context sensitive tasks such as relation extraction @xcite , question answering @xcite and knowledge base construction @xcite . entities can appear in text in many forms. in the sentences 'barack obama visited hawaii. the expresident enjoyed the fine weather.' both 'barack obama' and 'ex-president' should be assigned the type '/person/politician' by a fine-grained entity typing system. while the typing of the named entity (ne) 'barack obama' can be performed by state of the art entity typing systems, it is unclear how well these systems perform on general entities (ges) like 'ex-president'. we find that accuracy and f1 score of a state-of-the-art german fine-grained entity typing system are 17% lower on general entities than on named entities (see because manual annotation of training data is costly and time intensive we propose an approach that uses existing resources to create silver annotated ge typing data. for this we use german text taken from wikipedia, germanet (a german word-net equivalent, @xcite ) and the figer type ontology @xcite . the resulting data can be added to existing ne typing data for the training of a neural entity typing system. in our approach we use the hierarchical typing model of @xcite , which builds upon contextualized word embeddings. it has shown good performance on public benchmarks and is freely available. we compare our approach against using only ne data for training and a rule-based approach and achieve 10% improvement in accuracy of the prediction of level 1 figer types for german general entities, while decreasing named entity prediction accuracy by only 1%. our approach can be seen as a proof of concept and a blueprint for the use of existing wordnet resources to improve entity typing quality in other languages and domains. ## related work the problem of ge typing performance has not been examined specifically before, nor has it been addressed for the case of german. @xcite create a fine-grained entity typing system that is capable of typing both ge and ne in english by integrating ges into their training data. their approach relies on large amounts of manually annotated data, and is therefore not feasible for our case. moreover they propose a new type hierarchy, while we stick to the widely used figer type hierarchy, to make the output of our system consistent with that of other systems for tasks like multilingual knowledge graph construction. recent advances in typing ne in english have harnessed the power of contextualized word embeddings @xcite to encode entities and their context. these approaches use the aida, bnn, ontonotes and figer ontologies, which come with their own human annotated data sets @xcite @xcite . by choosing to use the model of @xcite , we build upon their strengths to enable ge typing in german. german ne typing suffers from a lack of manually annotated resources. two recent approaches by by @xcite and @xcite use manually annotated data from biographic interviews and court proceedings. owing to the specific domains, the authors modify existing type onthologies (ontonotes in the case of biographic interviews) or come up with their own type ontology (in the case of court proceedings). this limits the way their models can be applied to other domains or used for multilingual tasks. weber and steedman (2021, under submission) use annotation projection to create a training data set of wikipedia text annotated with figer types. we build upon their data set to create a german model that types both nes and ges. we manually map the 112 figer types to nouns in germanet. starting from a german translation of the type name (e.g. the type 'person' translates to 'mensch') we add terms that best describe the figer type. this mapping enables us to look up a word in germanet and check if any of its hypernyms are mapped to a figer type. if this is the case, we can assign the corresponding figer type to the word in question. figure 2 illustrates this method. we use this method to generate german ge training data and as our rule-based baseline. we use this ge training data in addition to german ne typing data to train the hierarchical typing model of @xcite . in this model the entity and its context are encoded using xlm-roberta @xcite . for each type in the figer ontology the model learns a type embedding. we pass the concatenated entity and context vector trough a 2-layer feed-forward network that maps into the same space as the type embedding. the score is an inner product between the transformed entity and context vector and the type embedding. for further model details refer to @xcite . to create the ge training data, we use the german portion of the wikimatrix corpus. by using the same genre we make sure that no additional noise is added by domain differences. moreover, the original english figer data set was created from wikipedia text, so we can assume that all figer types are well represented in the wikimatrix data. ## results the test set performance on ne is best when no ge data is added, although ne performance is worsened by adding ge training data, the decrease in level 1 performance in both accuracy and f1 is only 1% for 20k and 40k ge sentences, with a maximum decrease of 3% when 100k ge sentences are added. adding ge training data has a smaller effect on level 2 performance than on level 1 performance, with level 2 accuracy and f1 on the ge test set increasing by 5% when 60k sentences of ge data are added. adding ge training data initially decreases performance on ne level 2 types, but at 60k sentences of ge data is just as good as without them. adding more than 60k sentences of ge data does not improve ge test set performance, but decreases both ne and ge test set performance in accuracy and f1 score. we can also see that the germanet baseline is outperformed by all systems, although its performance on level 2 ge types is close to our best models. we will discuss possible explanations in the next section. ## discussion the results show that the models' performance on ge typing can be improved using a simple data augmentation method using wordnet, while only lightly impacting the performance on ne typing. all neural models outperform the germanet baseline. this raises the question why the neural systems were able to perform better than germanet on ge, although the training data was generated from germanet. we speculate that the hierarchical typing model is very context sensitive because of its usage of contextualized word embeddings (xlm-roberta) to encode entities and their context during training. while our ge data provides it with high confidence non-polysemous examples, it is able to learn which context goes with which type. at test time this awareness of context enables the neural systems to disambiguate polysemous cases, even though it has not observed these cases at training time. this intuition is supported by our test results: for the best performing model (240k) 40% of the general entities that occur in our test set are never seen in the training data. a second reason why the neural models outperform germanet is that germanet does not represent every german noun. a certain word might not be part of germanet and therefor no type can be assigned. this is the case for 23% of words seen during training data generation. the neural models do not have this problem because our vocabulary is larger than the 16.000 words contained in ger-manet and because the neural models assign type labels to out of vocabulary words on the basis of the language model xml-roberta. despite these factors the neural models' performance is closely matched by the germanet baseline on level 2 labels. level 2 types are underrepresented in the data, because their prevalence follows their occurrence in the wikipedia data. this leads to some low-level types being very rare: a signal that is too weak to be learned sufficiently by a neural model. on the other hand, a lookup of words in a preexisting data base like germanet is not affected by this issue. while the neural models offer high recall at low precision, germanet has higher precision at low recall. the results also show that 20k sentences of ge data produce the highest increase of ge performance while impacting ne performance least. adding ge data beyond 60k sentences does not only worsen ne performance by also ge performance. this is due to noise in the ge typing data. a manual error analysis of 100 ge training data sentences shows that 35% have incorrect type assignments. with more ge training data the model starts to overfit to this noise, which leads to decreasing test set performance, affecting ne performance slightly more than ge performance. ## conclusion and future work in this paper we have shown that it is possible to improve the performance of a german fine-grained entity typing system using germanet. we create silver annotated general entity typing data for training a fine-grained entity typing model that builds upon contextualised word embeddings (in our case, xlm-roberta). our results can be taken as a blueprint for improving fine-grained entity typing performance in other languages and domains, as there are wordnets for over 40 different languages. moreover, the manual mapping we introduced could be replaced by machine-translating english type labels into the language of the word-net, which would require less resources for human annotation than a manual mapping. avenues for future work could be a combination between high-precison but low recall wordnets and neural models, e.g. through incorporating the models' prediction confidence to make a decision whether a wordnet look-up should be trusted over the models' own prediction. the problem of general entity typing could also be viewed through the lens of coreference resolution: the type of a general entity could be inferred from a named entity that the general entity refers to. however, there might be cases in which no named entity referent exists, or domains and languages where coreference resolution systems are unavailable. in all of these cases combining our method with existing approaches opens new possibilities.