Extracting Multi-valued Relations from Language Models (2023)

Abstract: The widespread usage of latent language representations via pre-trained language models (LMs) suggests that they are a promising source of structured knowledge. However, existing methods focus only on a single object per subject-relation pair, even though often multiple objects are correct. To overcome this limitation, we analyze these representations for their potential to yield materialized multi-object relational knowledge. We formulate the problem as a rank-then-select task. For ranking candidate objects, we evaluate existing prompting techniques and propose new ones incorporating domain knowledge. Among the selection methods, we find that choosing objects with a likelihood above a learned relation-specific threshold gives a 49.5% F1 score. Our results highlight the difficulty of employing LMs for the multi-valued slot-filling task, and pave the way for further research on extracting relational knowledge from latent language representations.

## background the idea of knowledge extraction using lms representations was put forward by @xcite ; @xcite , and has since received much attention. several approaches @xcite @xcite use relational metadata and textual corpora to tune the lm using knowledge-enriched representations and efficiently extract knowledge. jiang et al. (2020a); kassner et al. (2021) focus on multilingual knowledge extraction, while dhingra et al. (2022) looks at extracting temporal knowledge from lms. current methods usually sample from correct objects for a given subject-relation pair by employing the hits@k metric and do not enforce deliberate accept/reject decisions on the outputs. ## methodology candidate list generation when probing an lm using a cloze-style prompt, the lm generates a probability distribution over the vocabulary tokens to fill in the masked position. for our task, we use prompts mentioning a subject-relation ⟨s, r⟩ pair and consider the resulting ranked list of tokens (w/ their corresponding probability scores) as candidates. in a zero-shot setting, the lm is probed using two types of prompts: (i) discrete prompts such as lpaqa @xcite and @xcite , and (ii) continuous prompts such as optiprompt @xcite and @xcite . in addition, we propose a collection of carefully designed manual prompts to evaluate and compare the generated objects for multi-valued relations. we create 50 diverse relation-specific prompts by incorporating domain knowledge, relation type, and variations in sentence structure and grammar. our prompts differ in verb form, tense, placement of the masked token (whether it is predicted in a prefix, suffix, or cloze-style), and whether or not there is a period and object type in the context. ## discussion although bert was probed for 500 objects when generating object lists, only 119.7 objects were retained after post-processing. objects with invalid types occur due to the zero-shot setting. also, other eminent masked lms, including bertbase, roberta-base, and roberta-large, achieve 60.61%, 54.82%, and 58.90% max-f1 scores. the max-f1 scores in in ## conclusion in this work, we evaluate using lm's internal representation for materializing factual knowledge on multi-valued relations. we utilize existing prompt engineering techniques and propose new prompts tailored for multi-valued relations. the suggested selection mechanism approaches help to filter out valid triples. our detailed analysis of the model's performance highlights the limitations of using zero-shot probing for the multi-valued slot-filling task. future work could aim to improve overall precision and recall and measure the impact of using lms to fill gaps in kbs. bhuwan dhingra, jeremy r. cole, julian martin eisenschlos, daniel gillick, jacob eisenstein, and william w. cohen. 2022. time-aware language models as temporal knowledge bases. transactions of the association for computational linguistics, 10:257-273. ankur goswami, akshata bhat, hadar ohana, and theodoros rekatsinas. 2020. unsupervised relation extraction from language models using constrained cloze completion. in findings of the association for computational linguistics: emnlp 2020, pages 1263-1276, online. association for computational linguistics. ari holtzman, peter west, vered shwartz, yejin choi, and luke zettlemoyer. 2021. surface form competition: why the highest probability answer isn't always right. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 7038-7051, online and punta cana, dominican republic. association for computational linguistics. zhengbao jiang, antonios anastasopoulos, jun araki, haibo ding, and graham neubig. 2020a. x-factr: multilingual factual knowledge retrieval from pretrained language models. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 5943-5959, online. association for computational linguistics. zhengbao jiang, jun araki, haibo ding, and graham neubig. 2021. how can we know when language models know? on the calibration of language models for question answering. transactions of the association for computational linguistics, 9:962-977. zhengbao jiang, frank f. xu, jun araki, and graham neubig. 2020b. how can we know what language models know? transactions of the association for computational linguistics, 8:423-438. nora kassner, philipp dufter, and hinrich schütze. 2021. multilingual lama: investigating knowledge in multilingual pretrained language models. in proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pages 3250-3258, online. association for computational linguistics. mike lewis, yinhan liu, naman goyal, marjan ghazvininejad, abdelrahman mohamed, omer levy, veselin stoyanov, and luke zettlemoyer. 2020. bart: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 7871-7880, online. association for computational linguistics. fabio petroni, tim rocktäschel, sebastian riedel, patrick lewis, anton bakhtin, yuxiang wu, and alexander miller. 2019. language models as knowledge bases? in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 2463-2473, hong kong, china. association for computational linguistics. guanghui qin and jason eisner. 2021. learning how to ask: querying lms with mixtures of soft prompts. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 5203-5212, online. association for computational linguistics. alec radford, jeff wu, rewon child, david luan, dario amodei, and ilya sutskever. 2019. language models are unsupervised multitask learners. colin raffel, noam shazeer, adam roberts, katherine lee, sharan narang, michael matena, yanqi zhou, wei li, and peter j. liu. 2020. exploring the limits of transfer learning with a unified text-to-text transformer. journal of machine learning research, 21(140):1-67. simon razniewski, andrew yates, nora kassner, and gerhard weikum. 2021. language models as or for knowledge bases. in proceedings of the workshop on deep learning for knowledge graphs (dl4kg 2021). ceur workshop proceedings. timo schick and hinrich schütze. 2021. exploiting cloze-questions for few-shot text classification and natural language inference. in proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pages 255-269, online. association for computational linguistics. taylor shin, yasaman razeghi, robert l. logan iv, eric wallace, and sameer singh. 2020. autoprompt: eliciting knowledge from language models with automatically generated prompts. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 4222-4235, online. association for computational linguistics. denny vrandečić and markus krötzsch. 2014. wikidata: a free collaborative knowledgebase. communications of acm, 57:78-85. zexuan zhong, dan friedman, and danqi chen. 2021. factual probing is [mask]: learning vs. learning to recall. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 5017-5033, online. association for computational linguistics.