Pre-training Language Models with Deterministic Factual Knowledge (2022)

Abstract: Previous works show that Pre-trained Language Models (PLMs) can capture factual knowledge. However, some analyses reveal that PLMs fail to perform it robustly, e.g., being sensitive to the changes of prompts when extracting factual knowledge. To mitigate this issue, we propose to let PLMs learn the deterministic relationship between the remaining context and the masked content. The deterministic relationship ensures that the masked factual content can be deterministically inferable based on the existing clues in the context. That would provide more stable patterns for PLMs to capture factual knowledge than randomly masking. Two pre-training tasks are further introduced to motivate PLMs to rely on the deterministic relationship when filling masks. Specifically, we use an external Knowledge Base (KB) to identify deterministic relationships and continuously pre-train PLMs with the proposed methods. The factual knowledge probing experiments indicate that the continuously pre-trained PLMs achieve better robustness in factual knowledge capturing. Further experiments on question-answering datasets show that trying to learn a deterministic relationship with the proposed methods can also help other knowledge-intensive tasks.

## methods section 2.1 expatiates the deterministic masking, which includes how we align texts with triplets and identify deterministic clues and spans in texts. the clue contrastive learning and clue classification are described in sections 2.2 and 2.3, respectively. ## related work pre-training on large-scale unlabeled text can help plms capture meaningful knowledge and benefits the downstream tasks accordingly @xcite . bert @xcite proposes a mask language model (mlm) in which the model needs to recover some masked tokens based on the remaining context. the effectiveness of the mlm makes bert become the starting point for fitting many downstream tasks @xcite . afterward, several different masking methods @xcite @xcite have explored how masking methods affect performance and have obtained further performance improvement. these works push the limit of mlm and show the importance of designing better masking strategies. on the other hand, some pre-training tasks other than mlm have been proposed. @xcite trains the model to distinguish the replaced words from the original words in the context. @xcite ) let factual spans be the replacement candidates. @xcite contrasts the representations between different entities and relations. this paper views another perspective of the masking methods: whether the remaining context can uniquely determine the masked span. accordingly, we propose a deterministic masking strategy that masks deterministic spans in mlm samples. moreover, we design clue contrastive learning and clue classification as pre-training tasks to help plms identify the deterministic clues for the masked span and contrast them with the nondeterministic ones. moreover, we evaluate the performance of the proposed model with various downstream tasks. ## conclusion this paper proposes to train plms to learn a deterministic input-output relationship in mlm to improve plms on capturing factual knowledge. the deterministic relationship ensures the masked content in mlm samples is deterministically predictable based on the remaining context. to further enhance the deterministic relationship, we design a pre-training task clue contrastive learning that encourages plms to give more confident predictions when the input keeps deterministic clues, and the clue classification to train plms to predict whether the deterministic clues exist. extensive experiments show that the proposed methods can improve the accuracy and consistency of factual knowledge capturing and boost the performance of the other two knowledge-intensive tasks.