Pre-trained Language Models (PLMs) have shown strong performance in various downstream Natural Language Processing (NLP) tasks. However, PLMs still cannot well capture the factual knowledge in the text, which is crucial for understanding the whole text, especially for document-level language understanding tasks . To address this issue, we propose a novel contrastive learning framework named ERICA in pre-training phase to obtain a deeper understanding of the entities and their relations in text. Specifically, (1) to better understand entities , we propose an entity discrimination task that distinguishes which tail entity can be inferred by the given head entity and relation . (2) Besides, to better understand relations, we employ a relation discrimination task which distinguishes whether two entity pairs are close or not in relational semantics . Experimental results demonstrate that our proposed ERICA framework achieves consistent improvements on several document-level language understanding tasks, including relation extraction and reading comprehension , especially under low resource setting. Meanwhile, ERICA achieves comparable or better performance on sentence-level tasks. We will release the datasets, source codes and pre-trained language models for further research explorations .
Pre-trained Language Models (PLMs) have shown superior performance on various downstream Natural Language Processing (NLP) tasks. However, conventional pre-training objectives do not explicitly model relational facts in text, which are crucial for textual understanding . To address this issue, we propose a novel contrastive learning framework ERICA to obtain a deep understanding of the entities and their relations in text. Specifically, we define two novel pre-training tasks to better understand entities and relations: (1) the entity discrimination task to distinguish which tail entity can be inferred by the given head entity and relation ; (2) the relation discrimination task to distinguish whether two relations are close or not semantically, which involves complex relational reasoning . Experimental results demonstrate that ERICA can improve typical PLMs (BERT and RoBERTa) on several language understanding tasks, including relation extraction , entity typing and question answering , especially under low-resource settings .
R
strong performance in
superior performance on
R
PLMs still cannot well capture the factual knowledge in the
conventional pre-training objectives do not explicitly model relational facts in
R
is crucial for understanding the whole text, especially for document-level language understanding tasks
are crucial for textual understanding
R
named ERICA in pre-training phase
ERICA
R
deeper
deep
R
(1)
we define two novel pre-training tasks
R
, we propose an
and relations: (1) the
R
that distinguishes
to distinguish
R
.
;
R
Besides, to better understand relations, we employ a
the
R
which distinguishes whether two entity pairs
to distinguish whether two relations
R
in relational semantics
semantically, which involves complex relational reasoning
R
our proposed ERICA framework achieves consistent improvements on several document-level
ERICA can improve typical PLMs (BERT and RoBERTa) on several
R
and reading comprehension
, entity typing and question answering
R
low resource setting. Meanwhile, ERICA achieves comparable or better performance on sentence-level tasks. We will release the datasets, source codes and pre-trained language models for further research explorations
low-resource settings