SHINRA2020-ML: Categorizing 30-language Wikipedia into ﬁne-grained NE based on “Resource by Collaborative Contribution” scheme
Abstract
This paper describes a Knowledge Base construction project, SHINRA and particularly the SHINRA2020-ML task.
The SHINRA2020-ML task is to categorize 30-language Wikipedia pages into ﬁne-grained named entity categories, called “Extended Named Entity (ENE)”.
It is one of the three tasks conducted on SHINRA since 2018.
SHINRA is a collaborative contribution scheme that utilize Automatic Knowledge Base Construction (AKBC) systems.
This project aims to create a huge and well-structured knowledge base essential for many NLP applications, such as QA, dialogue systems and explainable NLP systems.
In our “Resource by Collaborative Contribution (RbCC)” scheme, we conducted a shared task of structuring Wikipedia to attract participants but simultaneously submitted results are used to construct a knowledge base.
One trick is that the participants are not notiﬁed of the test data, so they have to run their systems on all entities in Wikipedia, although the evaluation results are reported for only a small portion of the test data among the entire data.
Using this method, the organizers receive multiple outputs of the entire data from the participants.
The submitted outputs are publicly accessible and are applied to building better structured knowledge using ensemble learning, for example.
In other words, this project uses AKBC systems to construct a huge and well-structured Knowledge Base collaboratively. The
“SHINRA2020-ML” task is also based on the RbCC scheme.
The task categorizes 30-language Wikipedia pages into ENE.
We previously categorized the entire Japanese
Wikipedia entities (920 thousand entities) into ENE by ML and then checked by hands.
For SHINRA2020-ML task participants, we provided the training data using categories of the Japanese Wikipedia and language links from the page.
For example, out of 920K Japanese Wikipedia pages, 275K have language links to German pages.
These data are used to create the training data for German and the task is to categorize the remaining 1,946K pages.
We conducted a shared task for 30 languages with the largest active users, and 10 groups participated.
We showed that the results by simple ensemble learning, i.e., majority voting, exceed top results in 17 languages, thereby proving the usefulness of the “RbCC” scheme.
We are conducting two tasks in 2021, SHINRA2021-LinkJP and SHINRA2021-ML tasks.
We will introduce these tasks in a later section of the paper.
1. Introduction
Wikipedia consists of large volumes of entities (articles), which are great resources for knowledge used in many NLP tasks.
To maximize the use of such knowledge, resources created from Wikipedia are structured for inference, reasoning, or any other purposes in many NLP applications.
Several machine readable knowledge bases exist, including Cyc [Lenat, 1995], DBpedia [Lehmann et al., 2015], Yet another greater ontology (YAGO) [Mahdisoltani et al., 2015], Freebase [Bollacker et al., 2008], Wikidata [Vrandeˇci´c and Kr¨otzsch, 2014] and so on, each of them has problems to be solved.
Recent KBs are created in a bottom-up manner by crowdsourcing, causing a signiﬁcant amount of undesirable noises in the knowledge base.
We believe that the structure of the knowledge should be deﬁned in a top-down manner rather than a bottom-up manner to create cleaner and more valuable knowledge bases.
Instead of existing cumbersome Wikipedia categories, we rely on a well-deﬁned and ﬁne-grained category.
Extended Named Entity (ENE) [ENE-HP] is one of the well-deﬁned name ontologies with 219 hierarchical categories and a set of attributes is deﬁned for each category.
We employed ENE for as the structure of the knowledge base in this study.
The automatic knowledge base construction (AKBC) and document categorization shared task have been popular for decades.
These popular shared-tasks are common in the ﬁeld of Information Retrieval, Information Extraction, Knowledge Base population, and attribute extraction, such as TREC, MUC, ACE, KBP [U.S. National Institute of Standards and Technology (NIST) , 2018] and CoNLL.
However, most of these tasks are designed only to compare the system performance and ﬁnd the best ranked systems on limited test data.
The outputs of the participating systems are not shared and the systems may be abandoned once the task is over.
Our contribution is geared to improve the situation by the following changes:
Designing the shared-task to construct a knowledge base rather than evaluating only on limited test data 2.
Making the outputs of all systems publicly accessible so that anyone can run ensemble learning or other algorithms to create results better than the best single system 3.
repeating the task with the larger and better training data from the output of the previous tasks
Project “SHINRA” started in 2017 with the aforementioned scheme, which we call “Resource by Collaborative Contribution (RbCC)”
[shi, a]. The SHINRA tasks in 2018, 2019, and 2020 include the attribute extraction task in Japanese Wikipedia [Sekine et al., 2019].
Here, we report the ﬁrst multi-lingual task and present future directions of the project.
The ﬁnal goal of the SHINRA project is to create the structured knowledge base of Wikipedia, including the attribute.
However, as a ﬁrst step, we classify each Wikipedia entry into one or more ENE categories before extracting attribute values.
The SHINRA2020-ML task resolves around classifying 30-language Wikipedia pages into ENE [ENE-HP] (ver.8.0) categories [shi, b].
We classiﬁed most of the Japanese Wikipedia pages (920K pages) into the ENE categories already.
We used language links to create the training data in 30 languages (for example, there are 275K language links from the Japanese to German Wikipedia).
So, the task is to categorize the remaining pages in 30 languages using the training data and evaluate the output based on a small portion of the pages that have no link from
Japanese Wikipedia. This project is not only to compare the participating systems to see which system perform best, but also to create the knowledge base using outputs of the participated systems.
We used the “ensemble learning technologies” to gather the outputs of the systems and create KB as accurately as possible.
2. Previous Work
Structured knowledge bases have been considered as one of the most important knowledge resources in the ﬁelds of Natural Language Processing.
Previously, several major projects to construct structured knowledge bases have been conducted.
One of the earliest projects is Cyc, and more recently are Wikipedia-based projects, such as DBpedia, YAGO, Freebase, and Wikidata.
Moreover, some shared tasks aimed at building techniques for knowledge base structuring, such as KBP and CoNLL.
Here, we introduce these resources and projects and describe the points we consider as issues to be solved in those projects.
Cyc ontology is a large knowledge base constructed as common sense knowledge[Lenat, 1995].
It was one of the large projects in AI between 1980 and 1990, which mainly used human labor to construct a knowledge base.
The cost of construction and maintenance of handmade knowledge bases for the general domain was very high. The knowledge base suﬀered the “knowledge acquisition bottleneck” which includes problems in coverage and consistency.
DBpedia is a more recent project that constructs structured information from semistructured data in Wikipedia, such as infoboxes and categories [Lehmann et al., 2015].
However, DBpedia is challenged with inaccuracy, low coverage, and lack of coherence.
Like Cyc, infobox and categories in Wikipedia are also created by humans, however, they are nonexperts of ontology.
For example, we can easily notice that the categories are very noisy.
In
Japanese Wikipedia, “Shinjuku Station,” which is a railway station, has a category “Odakyu
Electric Railway”, which is a major railway company using the station.
However, a station is not an instance of a railway company, so this is not an appropriate category.
There are so many examples like this in DBpedia.
Also several inconsistencies are observed in the category structure, it is not even a hierarchy and there is a loop in the category structure.
The attributes deﬁned in DBpedia are not well organized in many categories.
YAGO is an ontology constructed by mapping Wikipedia articles to the WordNet synsets [Mahdisoltani et al., 2015].
Similar to DBpedia, YAGO adopts attribute information extracted from infoboxes because no attribute is deﬁned in WordNet synsets.
Freebase is a project to construct a structured knowledge base using crowd-sourcing, similar to Wikipedia [Bollacker et al., 2008].
However, using the crowd-sourcing approach, Freebase does not have a well-organized ontology.
It has many noises and lacks coherence since it was created using unorganized crowds.
Currently, the Freebase project has been paused and integrated into Wikidata.
Wikidata aims to be a structured knowledge base, based on the crowd-sourcing scheme [Vrandeˇci´c and Kr¨otzsch, 2014].
It has noises and lacks coherence since it was constructed by a bottom-up approach, similar to Wikipedia and Freebase.
For example, by comparing the deﬁnition of a “city”, “town” and “human settlements”, we can easily observe inconsistencies in the property (the numbers of properties are 30, 0 and 6, respectively), there are very biased properties such as “Danish urban area code” in “human settlements”, there are many related ambiguous entities, such as “like a city”, “city/town” and so on.
Also, the category inconsistency is observed easily, for example, “city museum”, “mayor” are subcategory of “city”, although a mayor is not an instance of a “city.”
Wikipedia allows topics to be included in a category, however, this policy prevents making the category hierarchy a welldesigned ontology.
KBP is a shared task organized by NIST for establishing a technology to construct a structured knowledge base from non-structured documents[U.S. National Institute of Standards and Technology (NIST) , 2018].
It consists mainly of two tasks, one is Entity Discovery and Linking (EDL) which involves ﬁnding and identifying an entity deﬁned in DB from documents.
The other is Slot Filling, which involves extracting attribute information of the entity.
It is a competition-based project and not for resource creation purposes.
FIne Grained Entity Recognition (FIGER) is a project to identify 112 named entity classes that are ﬁnely deﬁned, such as ENE, from documents[Ling and Weld, 2012].
The category of FIGER is biased, and it does not have attribute deﬁnitions for each category.
3. Data
This section introduces two datasets used for the SHINRA2020-ML task.
One is the ENE deﬁnition which is an ontology for the named entity, and hence, Wikipedia entities are categorized.
The other one is the categorized Japanese Wikipedia data, which will be used to create the training data for 30 languages using Wikipedia’s language links.
3.1 Extended Named Entity
To construct a structured knowledge base useful for NLP applications, well-structured ontology is essential and must be designed in a top-down manner.
The structures of the knowledge base in DBpedia, Freebase, and Wikidata were created by crowds in a bottomup manner, and all have similar characteristics including inconsistent categories, imbalanced ontologies, and ad hoc attributes.
This is majorly attributed to the bottom-up manner of their knowledge base design.
A top-down strategy is essential to design the ontology and attributes consistently.
For a top-down designed ontology for named entities, we employed the ENE [ENE-HP].
ENE is a named entity classiﬁcation hierarchy that includes the attribute deﬁnition for each category ([Sekine et al., 2002], [Sekine and Nobata, 2004], [Sekine, 2008]).
It includes 219 ﬁne-grained categories of named entities in a hierarchy of up to four layers.
It contains not only the ﬁne-grained categories of the typical NE categories, such as “city” and “lake” for “location”, and “company” and “political party” for “organization”, but also contains new named entity types such as “products”, “event”, and “natural object”.
These categories are designed to cover a wide range of entities in the general world, which are often mentioned in an encyclopedia and many other resources.
Figure 1 shows the ENE deﬁnition, version 8.0.
Attributes are designed so important attributes in the Wikipedia entities of each category are covered based on the investigation of sample entities.
For example, the attributes for “airport” categories include the following: “Reading”, “IATA code”, “ICAO code”, “nickname”, “name origin”, “number of users per year”, “the year of the statistic”, “the number of airplane landings per year”, “old name”, “elevation”, “big city nearby”, “number of runaways”.
Please refer to the ENE homepage [ENE-HP] for the complete deﬁnitions.
3.2 Categorized Japanese Wikipedia
We categorized 920K Japanese Wikipedia pages into one or more of the 219 ENE categories before this project.
For the categorization process, we excluded the less popular entities, i.e. having less than ﬁve incoming links (151K entities), and non-entity pages (about 53K pages) such as common nouns and simple numbers.
This categorization was done using the machine learning method followed by hand-checking all data [Masatoshi et al., 2018].
The accuracy of the categorization was conﬁrmed as 98.5%.
The remaining 1.5% cover those that are ambiguous and are diﬃcult even for human annotators, or for those having multiple categories.
Table 1 shows the most frequent categories in the data of 920K pages.
4. Task Description
SHINRA2020-ML is the ﬁrst shared-task of text categorization in the SHINRA project.
It tackled the problem of classifying 30 language Wikipedia entities in ﬁne-grained categories with 219 categories deﬁned in ENE (ver. 8.0).
Participants selected one or more target languages, and for each language, the system classiﬁed all Wikipedia pages in the target language(s).
We provided the training data for 30 languages, created by the categorized [Table] Japanese Wikipedia of 920K pages and Wikipedia language links for 31 languages (including
Japanese). Out of 2,263K German Wikipedia pages, 275K pages had a language links from Japanese Wikipedia, which served as silver (i.e. bit noisy) training data for German.
Thus, the task became “to classify the remaining 1,988K pages into 219 categories, based on the training data” (actually, the participants are requested to categorize the training data with their system, as well).
The same holds for the other 29 languages, as shown in the data statistics shown in Table 2.
4.1 Schedule
SHINRA2020-ML has been conducted according to the schedule listed in Table:3.
4.2 Participants
Ten groups from Seven countries participated in this study.
The list of participant groups and the task they participated in are shown in Table 4.
The description of most of systems are reported in the NTCIR-15 conference [Bui and Le-Hong, 2020]
[Cardoso et al., 2020] [Abhishek et al., 2020]
[Nakayama and Sekine, 2020]
[Yoshioka and Koitabashi, 2020] [Nishikawa and Yamada, 2020] [Yoshikawa et al., 2020].
4.3 Evaluation Results
For each target language, a group can submit up to three runs based on diﬀerent methods.
A system classiﬁes each page into one or more of the ENE (ver.8.0) categories correctly.
The evaluation pages are selected from those not linked from Japanese pages.
The selection method for the evaluation data is not disclosed since we can compare future results using the same data.
Suppose the estimated category is not an exact match, the system get no score for that output.
We evaluated the performance of systems on a multi-label classiﬁcation using the micro average F1 measure, i.e. the harmonic mean of precision and micro-averaged recall.
Note, the distribution of the category in the test data may diﬀer from that of target or training data.
Table 5 shows the F-measure of participating systems.
“Late submission” indicates those who submit the results after the deadline.
4.4 Results on RbCC
We tried a simple ensemble learning, called the majority voting.
We used systems from all but one participants, which claims that they are not aiming at building a good performance [Table] [Table] Bulgarian
[Table]
system, but aiming to show the degree of usefulness of Wikipedia category for this task. The results are shown in Figure 6. In this table, languages that achieved better results by majority voting than the single best system are marked green.
Seventeen languages which got the better results, indicating a great performance for the RbCC scheme. In the table, ”oracle F1” denotes the F1 score for selecting the correct answer from the pool of outputs by all systems. Date
[Table]
January, Data release April, Homepage and CFP open
August
[Table]
Registration and Result submission deadline
September
[Table]
Evaluation results due back to participants
Dec 8-
[Table]
NTCIR-15 Conference
[Table]
5. SHINRA tasks
Project SHINRA has been ongoing since 2017.
The tasks are not limited to only categorizing multi-lingual Wikipedia, but also entail the attribute extraction and linking tasks for
Japanese Wikipedia, as shown in Table 7.
We are conducting two tasks in 2021 [shi, a].
6. Conclusion
This paper report progress from the ongoing SHINRA project, particularly, the SHINRA2020ML task.
We proposed a knowledge base creation scheme, “Resource by Collaborative Contribution (RbCC)”.
The SHINRA2020-ML task categorizes 30-language Wikipedias into
Extended Named Entity (ENE), the top-down deﬁnition of NE categories.
Ten groups participated in the task, and the ensemble learning results show that the RbCC scheme is promising.
Furthermore, we plan to conduct the SHINRA2021-ML and SHINRA2021LinkJP tasks based on the RbCC scheme.
We express our deep appreciation to all participants and collaborators who helped this project.
With many participation, we can try ensemble learning and achieve the goal.
We hope to expand the idea of the RbCC scheme to other ML tasks in general, not limited to sHINRA or similar task and resource.
Acknowledgments
This work was supported by JSPS KAKENHI Grant Number JP20269633.