pwc-archive/papers-with-abstracts huggingface dataset:
Revisiting Evaluation of Knowledge Base Completion Models
Authors: Pouya Pezeshkpour, Yifan Tian, Sameer Singh
Date: 2020-02-14 00:00:00
Proceeding: AKBC 2020 6
Tasks: Knowledge Base Completion, Knowledge Graph Completion, Knowledge Graphs, Triple Classification

Representing knowledge graphs (KGs) by learning embeddings for entities and relations has led to accurate models for existing KG completion benchmarks. However, due to the open-world assumption of existing KGs, evaluation of KG completion uses ranking metrics and triple classification with negative samples, and is thus unable to directly assess models on the goals of the task: completion. In this paper, we first study the shortcomings of these evaluation metrics. Specifically, we demonstrate that these metrics (1) are unreliable for estimating how calibrated the models are, (2) make strong assumptions that are often violated, and 3) do not sufficiently, and consistently, differentiate embedding methods from each other, or from simpler approaches. To address these issues, we gather a semi-complete KG referred as YAGO3-TC, using a random subgraph from the test and validation data of YAGO3-10, which enables us to compute accurate triple classification accuracy on this data.  Conducting thorough experiments on existing models, we provide new insights and directions for the KG completion research. Along with the dataset and the open source implementation of the models, we also provide a leaderboard for knowledge graph completion that consists of a hidden, and growing, test set, available at https://pouyapez.github.io/yago3-tc/.
Representing Joint Hierarchies with Box Embeddings
Authors: Dhruvesh Patel, Shib Sankar Dasgupta, Michael Boratko, Xiang Li, Luke Vilnis, Andrew McCallum
Date: 2020-02-14 00:00:00
Proceeding: AKBC 2020 6

Learning representations for hierarchical and multi-relational knowledge has emerged as an active area of research. Box Embeddings  [Vilnis et al., 2018, Li et al., 2019] represent concepts with hyperrectangles in $n$-dimensional space and are shown to be capable of modeling tree-like structures efficiently by training on a large subset of the transitive closure of the WordNet hypernym graph. In this work, we evaluate the capability of box embeddings to learn the transitive closure of a tree-like hierarchical relation graph with far fewer edges from the transitive closure. Box embeddings are not restricted to tree-like structures, however, and we demonstrate this by modeling the WordNet meronym graph, where nodes may have multiple parents. We further propose a method for modeling multiple relations jointly in a single embedding space using box embeddings. In all cases, our proposed method outperforms or is at par with all other embedding methods.
Sampo: Unsupervised Knowledge Base Construction for Opinions and Implications
Authors: Nikita Bhutani, Aaron Traylor, Chen Chen, Xiaolan Wang, Behzad Golshan, Wang-Chiew Tan
Date: 2020-02-14 00:00:00
Proceeding: AKBC 2020 6
Tasks: Knowledge Base Construction

Knowledge bases (KBs) have long been the backbone of many real-world applications and services. There are many KB construction (KBC) methods that can extract factual information, where relationships between entities are explicitly stated in text. However, they cannot model implications between opinions which are abundant in user-generated text such as reviews and often have to be mined. Our goal is to develop a technique to build KBs that can capture both opinions and their implications. Since it can be expensive to obtain training data to learn to extract implications for each new domain of reviews, we propose an unsupervised KBC system, Sampo, Specifically, Sampo is tailored to build KBs for domains where many reviews on the same domain are available. We generate KBs for 20 different domains using Sampo and manually evaluate KBs for 6 domains. Our experiments show that KBs generated using Sampo capture information otherwise missed by other KBC methods. Specifically, we show that our KBs can provide additional training data to fine-tune language models that are used for downstream tasks such as review comprehension.
Exploiting Semantic Relations for Fine-grained Entity Typing
Authors: Hongliang Dai, Yangqiu Song, Xin Li
Date: 2020-02-14 00:00:00
Proceeding: AKBC 2020 6
Tasks: Entity Typing, Relation, Sentence, TAG

Fine-grained entity typing results can serve as important information for entities while constructing knowledge bases. It is a challenging task due to the use of large tag sets and the requirement of understanding the context.
We find that, in some cases, existing neural fine-grained entity typing models may ignore the semantic information in the context that is important for typing. 
To address this problem, we propose to exploit semantic relations extracted from the sentence to improve the use of context. The used semantic relations are mainly those that are between the mention and the other words or phrases in the sentence. 
We investigate the use of two types of semantic relations: hypernym relation, and verb-argument relation. Our approach combine the predictions made based on different semantic relations and the predictions of a base neural model to produce the final results. We conduct experiments on two commonly used datasets: FIGER (GOLD) and BBN. Our approach achieves at least 2\% absolute strict accuracy improvement on both datasets compared with a strong BERT based model.
Learning Relation Entailment with Structured and Textual Information
Authors: Zhengbao Jiang, Jun Araki, Donghan Yu, Ruohong Zhang, Wei Xu, Yiming Yang, Graham Neubig
Date: 2020-02-14 00:00:00
Proceeding: AKBC 2020 6
Tasks: Question Answering, Relation, Relation Extraction, Representation Learning

Relations among words and entities are important for semantic understanding of text, but previous work has largely not considered relations between relations, or meta-relations. In this paper, we specifically examine relation entailment, where the existence of one relation can entail the existence of another relation. Relation entailment allows us to construct relation hierarchies, enabling applications in representation learning, question answering, relation extraction, and summarization. To this end, we formally define the new task of predicting relation entailment and construct a dataset by expanding the existing Wikidata relation hierarchy without expensive human intervention. We propose several methods that incorporate both structured and textual information to represent relations for this task. Experiments and analysis demonstrate that this task is challenging, and we provide insights into task characteristics that may form a basis for future work. The dataset and code have been released at https://github.com/jzbjyb/RelEnt.
Enriching Knowledge Bases with Interesting Negative Statements
Authors: Hiba Arnaout, Simon Razniewski, Gerhard Weikum
Date: 2020-02-14 00:00:00
Proceeding: AKBC 2020 6
Tasks: Question Answering

Knowledge bases (KBs), pragmatic collections of knowledge about notable entities, are an important asset in applications such as search, question answering and dialogue. Rooted in a long tradition in knowledge representation, all popular KBs only store positive information, but abstain from taking any stance towards statements not contained in them.

In this paper, we make the case for explicitly stating interesting statements which are not true. Negative statements would be important to overcome current limitations of question answering, yet due to their potential abundance, any effort towards compiling them needs a tight coupling with ranking. We introduce two approaches towards automatically compiling negative statements. (i) In peer-based statistical inferences, we compare entities with highly related entities in order to derive potential negative statements, which we then rank using supervised and unsupervised features. (ii) In pattern-based query log extraction, we use a pattern-based approach for harvesting search engine query logs. Experimental results show that both approaches hold promising and complementary potential. Along with this paper, we publish the first datasets on interesting negative information, containing over 1.4M statements for 130K popular Wikidata entities.
Manifold Alignment across Geometric Spaces for Knowledge Base Representation Learning
Authors: Huiru Xiao, Yangqiu Song
Date: 2021-06-22 00:00:00
Proceeding: AKBC 2021 10
Tasks: Entity Typing, Representation Learning

Knowledge bases have multi-relations with distinctive properties. Most properties such as symmetry, inversion, and composition can be handled by the Euclidean embedding models. Nevertheless, transitivity is a special property that cannot be modeled efficiently in the Euclidean space. Instead, the hyperbolic space characterizes the transitivity naturally because of its tree-like properties. However, the hyperbolic space reveals its weakness for other relations. Therefore, building a representation learning framework for all relation properties is highly difficult. In this paper, we propose to learn the knowledge base embeddings in different geometric spaces and apply manifold alignment to align the shared entities. The aligned embeddings are evaluated on the out-of-taxonomy entity typing task, where we aim to predict the types of the entities from the knowledge graph. Experimental results on two datasets based on YAGO3 demonstrate that our approach has significantly good performances, especially in low dimensions and on small training rates.
Abg-CoQA: Clarifying Ambiguity in Conversational Question Answering
Authors: Meiqi Guo, Mingda Zhang, Siva Reddy, Malihe Alikhani
Date: 2021-06-22 00:00:00
Proceeding: AKBC 2021 10
Tasks: 4k, Conversational Question Answering, Question Answering, Text Generation

Effective  communication  is  about  the  dissemination  of  properly  worded  meaningful ideas/messages that are comprehensible to both sender and receiver and which ultimately can attract the desired response or feedback.  For machines to engage in a conversation, it is therefore essential to enable them to clarify ambiguity and achieve a common ground.  We introduce Abg-CoQA, a novel dataset for clarifying ambiguity in Conversational Question Answering systems.  Our dataset contains 9k questions with answers where 1k questions are ambiguous, obtained from 4k text passages from five diverse domains.  For ambiguous questions,  a  clarification  conversational  turn  is  collected.   We  evaluate  strong  language generation models and conversational question answering models on Abg-CoQA. The best-performing system achieves a BLEU-1 score of 12.9% on generating clarification question, which  is  27.9  points  behind  human  performance  (40.8%);  and  a  F1  score  of  40.1%  on question  answering  after  clarification,  which  is  35.1  points  behind  human  performance (75.2%), indicating there is ample room for improvement.
Neural Concept Formation in Knowledge Graphs
Authors: Agnieszka Dobrowolska, Antonio Vergari, Pasquale Minervini
Date: 2021-06-22 00:00:00
Proceeding: AKBC 2021 10
Tasks: Knowledge Graphs, Link Prediction, Novel Concepts

In this work, we investigate how to learn novel concepts in Knowledge Graphs (KGs) in a principled way, and how to effectively exploit them to produce more accurate neural link prediction models. Specifically, we show how concept membership relationships learned via unsupervised clustering of entities can be reified and used to augment a KG. In a thorough set of experiments, we confirm that neural link predictors trained on these augmented KGs, or in a joint Expectation-Maximization iterative scheme, can generalize better and produce more accurate predictions for infrequent relationships. For instance, our method yields relative improvements of up to 8.6% MRR on WN18RR for rare predicates, and up to 82% in small-data regimes, where the model has access to just a small subset of the training triples. Furthermore, our proposed models are able to learn meaningful concepts.
A Joint Training Framework for Open-World Knowledge Graph Embeddings
Authors: Karthik V, Beethika Tripathi, Mitesh M Khapra, Balaraman Ravindran
Date: 2021-06-22 00:00:00
Proceeding: AKBC 2021 10
Tasks: Dialogue Generation, Entity Embeddings, Knowledge Graph Completion, Knowledge Graph Embeddings, Knowledge Graphs, Natural Language Inference, Question Answering, World Knowledge

Knowledge Graphs(KGs) represent factual information as graphs of entities connected by relations. Knowledge graph embeddings have emerged as a popular approach to encode this information for various downstream tasks like natural language inference, question answering and dialogue generation. As knowledge bases expand, we are presented with newer (open-world) entities, often with textual descriptions. We require techniques to embed new entities as they arrive using the textual information at hand. This task of open-world KG completion has received some attention in recent years. However, we find that existing approaches suffer from one or more of four drawbacks – 1) They are not modular with respect to the choice of the KG embedding model 2) They ignore best practices for aligning two embedding spaces 3) They do not account for differences in training strategy needed when presented with datasets with different description sizes and 4) They do not produce entity embeddings for use by downstream tasks. To address these problems, we propose FOlK (Framework for Open-World KG embeddings) - a technique that jointly learns embeddings for KG entities from descriptions and KG structure for open-world knowledge graph completion. Additionally, we modify existing data sources and make available YAGO3-10- Open and WN18RR-Open two datasets that are well suited for demonstrating the efficacy of open-world KG completion approaches. Finally, we empirically demonstrate the effectiveness of our model in improving upon state-of-the-art baselines on several tasks resulting in performance increases of up to 72% on mean reciprocal rank.
A Case Study in Bootstrapping Ontology Graphs from Textbooks
Authors: Vinay K. Chaudhri, Matthew Boggess, Han Lin Aung, Debshila Basu Mallick, Andrew C Waters, Richard Baraniuk
Date: 2021-06-22 00:00:00
Proceeding: AKBC 2021 10
Tasks: Knowledge Graphs, Language Modeling, Language Modelling, Term Extraction

Ontology graphs are graphs in which the nodes are generic classes and edges have labels that specify the relationships between the classes.  In this paper, we address the question:to what extent can automated extraction and crowdsourcing techniques be combined to boostrap  the  creation  of  comprehensive  and  accurate  ontology  knowledge  graphs?   By adapting  the  state-of-the-art  language  model  BERT  to  the  task,  and  leveraging  a  novel relationship selection task, we show that even though it is difficult to achieve a high precision and recall, automated term extraction and crowd sourcing provide a way to bootstrap the ontology graph creation for further refinement and improvement through human effort.
Commonsense Reasoning with Implicit Knowledge in Natural Language
Authors: Pratyay Banerjee, Swaroop Mishra, Kuntal Kumar Pal, Arindam Mitra, Chitta Baral
Date: 2021-06-22 00:00:00
Proceeding: AKBC 2021 10
Tasks: Knowledge Graphs

Commonsense  Reasoning  is a research  challenge studied from the early days of AI. In  recent  years, several natural language QA task have been proposed where commonsense reasoning is important. Two common approaches to this are (i) Use  of well-structured commonsense present in knowledge graphs, and (ii) Use of progressively larger transformer language models. While acquiring  and  representing  commonsense  in  a  formal representation is challenging in approach (i), approach (ii) gets more and more resource-intensive.  In this work, we take a middle ground where we use smaller language models together with a relatively smaller but targeted natural language text corpora. The advantages of such an approach is that it is less resource intensive and yet at the same time it can use unstructured text corpora. We define different unstructured commonsense  knowledge  sources,  explore  three strategies  for  knowledge  incorporation,  and propose  four methods competitive to state-of-the-art methods to reason with implicit commonsense.
Prompt Tuning or Fine-Tuning - Investigating Relational Knowledge in Pre-Trained Language Models
Authors: Leandra Fichtel, Jan-Christoph Kalo, Wolf-Tilo Balke
Date: 2021-06-22 00:00:00
Proceeding: AKBC 2021 10
Tasks: fill-mask, Fill Mask, Knowledge Graphs, Language Modeling, Language Modelling, Sentence, Transfer Learning

Extracting relational knowledge from large pre-trained language models by a cloze-style sentence serving as a query has shown promising results. In particular, language models can be queried similar to knowledge graphs. The performance of the relational fact extraction task depends significantly on the query sentence, also known under the term prompt. Tuning these prompts has shown to increase the precision on standard language models by a maximum of around 12% points. However, usually large amounts of data in the form of existing knowledge graph facts and large text corpora are needed to train the required additional model.
In this work, we propose using a completely different approach: Instead of spending resources on training an additional model, we simply perform an adaptive fine-tuning of the pre-trained language model on the standard fill-mask task using a small training dataset of existing facts from a knowledge graph. We investigate the differences between complex prompting techniques and adaptive fine-tuning in an extensive evaluation. Remarkably, adaptive fine-tuning outperforms all baselines, even by using significantly fewer training facts. Additionally, we analyze the transfer learning capabilities of this adapted language model by training on a restricted set of relations to show that even fewer training relations are needed to achieve high knowledge extraction quality.
Do Boat and Ocean Suggest Beach? Dialogue Summarization with External Knowledge
Authors: Tianqing Fang, Haojie Pan, Hongming Zhang, Yangqiu Song, Kun Xu, Dong Yu
Date: 2021-06-22 00:00:00
Proceeding: AKBC 2021 10
Tasks: Text Generation

In human dialogues, utterances do not necessarily carry all the details. As pragmatics studies suggest,  humans can infer meaning from the situational context even though the meaning is not literally expressed.  It is crucial for natural language processing models to understand such an inference process. In this paper, we address the problem of inferencing Concepts Out of the Dialogue Context (CODC) in the dialogue summarization task. We propose a novel framework comprised of a CODC inference module leveraging external knowledge from WordNet and a knowledge attention module aggregating the inferred knowledge into a neural summarization model. To evaluate the inference capability of different methods, we also propose a new evaluation metric based on CODC. Experiments suggest that current automatic evaluation metrics of natural language generation may not be enough to understand the quality of out-of-context inference in generation results, and our proposed summarization model can provide statistically significant improvements on both CODC inference and traditional automatic evaluation metrics, e.g., CIDEr. Human evaluation of the model's inference ability also demonstrates the superiority of the proposed model.
Behavioral Testing of Knowledge Graph Embedding Models for Link Prediction
Authors: Wiem Ben Rim, Carolin Lawrence, Kiril Gashteovski, Mathias Niepert, Naoaki Okazaki
Date: 2021-06-22 00:00:00
Proceeding: AKBC 2021 10
Tasks: Graph Embedding, Knowledge Graph Embedding, Knowledge Graphs, Link Prediction

Knowledge graph embedding (KGE) models are often used to encode knowledge graphs in order to predict new links inside the graph. The accuracy of these methods is typically evaluated by computing an averaged accuracy metric on a held-out test set. This approach, however, does not allow the identification of \emph{where} the models might systematically fail or succeed. To address this challenge, we propose a new evaluation framework that builds on the idea of (black-box) behavioral testing, a software engineering principle that enables users to detect system failures before deployment. 
With behavioral tests, we can specifically target and evaluate the behavior of KGE models on specific capabilities deemed important in the context of a particular use case. To this end, we leverage existing knowledge graph schemas to design behavioral tests for the link prediction task. With an extensive set of experiments, we perform and analyze these tests for several KGE models. Crucially, we for example find that a model ranked second to last on the original test set actually performs best when tested for a specific capability. Such insights allow users to better choose which KGE model might be most suitable for a particular task. The framework is extendable to additional behavioral tests and we hope to inspire fellow researchers to join us in collaboratively growing this framework.
Combining Analogy with Language Models for Knowledge Extraction
Authors: Danilo Neves Ribeiro, Kenneth Forbus
Date: 2021-06-22 00:00:00
Proceeding: AKBC 2021 10
Tasks: Articles, Language Modeling, Language Modelling, World Knowledge

Learning structured knowledge from natural language text has been a long-standing challenge. Previous work has focused on specific domains, mostly extracting knowledge about named entities (e.g. countries, companies, or persons) instead of general-purpose world knowledge (e.g. information about science or everyday objects). In this paper we combine the Companion Cognitive Architecture with the BERT Language Model to extract structured knowledge from text, with the goal of automatically inferring missing commonsense facts from an existing knowledge base. Using the principles of distant supervision, the system learns functions called query cases that map statements expressed in natural language into knowledge base relations. Afterwards, the system uses such query cases to extract structured knowledge using analogical reasoning. We run experiments on 2,679 Simple English Wikipedia articles, where the system is able to learn high precision facts about a variety of subjects from a few training examples, outperforming strong baselines.
Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning
Authors: Chadi Helwe, Chloé Clavel, Fabian M. Suchanek
Date: 2021-06-22 00:00:00
Proceeding: AKBC 2021 10
Tasks: Deep Learning, Logical Reasoning, Mathematical Reasoning, Survey

Recent years have seen impressive performance of transformer-based models on different natural language processing tasks. However, it is not clear to what degree the transformers can reason on natural language. To shed light on this question, this survey paper discusses the performance of transformers on different reasoning tasks, including mathematical reasoning, commonsense reasoning, and logical reasoning. We point out successes and limitations, of both empirical and theoretical nature.
KnowFi: Knowledge Extraction from Long Fictional Texts
Authors: Cuong Xuan Chu, Simon Razniewski, Gerhard Weikum
Date: 2021-06-22 00:00:00
Proceeding: AKBC 2021 10
Tasks: Cultural Vocal Bursts Intensity Prediction, Knowledge Base Construction, Relation Extraction

Knowledge base construction has recently been extended to fictional domains like multi-volume novels and TV/movie series, aiming to support explorative queries for fans and sub-culture studies by humanities researchers. This task involves the extraction of relations between entities. State-of-the-art methods are geared for short input texts and basic relations, but fictional domains require tapping very long texts and need to cope with non-standard relations where distant supervision becomes sparse. 
This work addresses these challenges by a novel method, called KnowFi, that combines BERT-enhanced neural learning with judicious selection and aggregation of text passages. Experiments with several fictional domains demonstrate the gains that KnowFi achieves over the best prior methods for neural relation extraction.