Position-based Prompting for Health Outcome Generation (2022)

Abstract: Probing factual knowledge in Pre-trained Language Models (PLMs) using prompts has indirectly implied that language models (LMs) can be treated as knowledge bases. To this end, this phenomenon has been effective, especially when these LMs are fine-tuned towards not just data, but also to the style or linguistic pattern of the prompts themselves. We observe that satisfying a particular linguistic pattern in prompts is an unsustainable, time-consuming constraint in the probing task, especially because they are often manually designed and the range of possible prompt template patterns can vary depending on the prompting task. To alleviate this constraint, we propose using a position-attention mechanism to capture positional information of each word in a prompt relative to the mask to be filled, hence avoiding the need to re-construct prompts when the prompts’ linguistic pattern changes. Using our approach, we demonstrate the ability of eliciting answers (in a case study on health outcome generation) to not only common prompt templates like Cloze and Prefix but also rare ones too, such as Postfix and Mixed patterns whose masks are respectively at the start and in multiple random places of the prompt. More so, using various biomedical PLMs, our approach consistently outperforms a baseline in which the default PLMs representation is used to predict masked tokens.

## introduction language models (lms) as knowledge bases (kbs) (lm-as-kb) is a rapidly growing phenomenon attracting a lot of attention in the natural language processing (nlp) community @xcite @xcite . lm-as-kb implies the usage of lms as an alternative or at least a proxy for explicit kbs. to achieve lm-as-kb, researchers adopt prompt-based learning (pbl) in which lms learn to probabilistically predict missing information once given fill-in-the-blank prompt inputs @xcite such as "eiffel tower is located in ". pbl has generally been a success, for example, in a systematic survey of prompting methods, @xcite indicate that "pre-train, prompt and predict" is a new paradigm replacing "pre-train and finetune" paradigm in nlp. because of this success, the rationale that lms contain factual retrievable knowledge (lm-as-kb) is ostensibly justified and therefore continually explored. the prompt sequences often used in pbl have a masked token or span (denoted by @xcite in the remainder of the paper) that positionally appears either in the middle (cloze-style) @xcite @xcite or at the very end of the sequence (prefix style) @xcite . moreover, we learn that the majority of the pbl tasks probe relational knowledge possessed by pre-trained language models (plms) @xcite @xcite , which implies that the prompt inputs used in querying the plms have to contain relational information (such as "subject-relation-object" triples). furthermore, we observe that, a fair amount of time in several pbl tasks is spent reconstructing prompt inputs through manually designing templates @xcite or corrupting prompt inputs through deletion @xcite , replacement @xcite or permutation @xcite . as discussed above, we notice that, the syntactic and semantic structure of prompt inputs is a constraint encountered in pbl, notwithstanding the multitude of constraints that could arise given that pbl is inherently a text generation task @xcite . this constraint will usually require researchers to laboriously prepare supervised data with prompts whose linguistic patterns suit the objective of the prompting task, for instance, @xcite @xcite , use templates that reformulate prompts to contain relational information connecting a particular text span to the to-be filled information. however, template-based prompt reformulation has two main challenges. first, it presents a risk of corrupting the grammar of the prompts unwittingly @xcite . second, the search space of the candidate prompts is too large @xcite and is practically impossible to create templates that can enumerate all possible linguistic patterns that prompt queries can be tailored to. for example, prompt template patterns with missing information at the beginning and or with multiple missing information in a sequence are yet to be explored in prior works. to address the above-mentioned challenges, we propose a strategy we denote position-based prompting (pbp), which is less concerned about the linguistic pattern or shape the prompt takes on, but rather focuses on the words (that the prompts are composed of) and their positions relative to the @xcite . pbp is focused on shifting the emphasis on subject-relation-object triples to the masked positions as well as the interaction of all the other words with the [mask]s position. pbp is built to automatically adjust from one prompt template to another, which essentially eliminates the need to prepare hand crafted prompts in the event that an lm is to be probed for rare knowledge. in its architecture, pbp enhances contextualised word representations with position-aware representations to solve fill-in-the-blank tasks. in our approach, we fine-tune plm parameters along with position-oriented parameters to generate position-based contextualised word representations. to test our approach, we investigate how well biomedical lms store and recall information relevant to biomedical entities, with a specific interest in health outcomes, which are defined as measurements or observations used to capture and assess the effect of treatments @xcite . in addition to the prefix and cloze styles, we incorporate two rare prompt style patterns that we denote postfix and mixed, where the former contains the [mask] token/s at the beginning of the prompt sequence and the latter has multiple [mask] token/s in various positions (figure 1 ). our approach obtains mean scores (across several biomedical lms) in exact match (em) and partial match (pm) metrics that are an improvement (2.4% across both metrics) over those obtained using the vanilla plm representations, reporting a significant improvement of 6.49% in f1 on the ebm-nlp @xcite dataset. as later defined in section 4.1, em measures the percentage of predictions of all [mask] tokens (or spans) that match the ground truth, whereas pm measures the percentage of correctly predicted [mask] tokens. ## entity memorisation and recalling large-scale lms with billions of parameters have already shown to recall facts that were observed in the training data @xcite . however, the ground truth for these lms to achieve this is already laid with systematically handcrafting rules to follow in creating the prompt input sequences they receive at the training stage. for instance, the majority of the prompts created in pbl tasks embed knowledge in form of triples {subject,relation,object} such that lms could correctly predict object entities when prompted with a sequence containing a subject and relation or otherwise predict subject entities when prompted with a sequence containing an object and a relation @xcite @xcite . whichever the case, models often predict answers as shown in (1). in this work, we however do not assume any prior knowledge contained in a prompt, but rather simply locate outcome entities in the sentences extracted from randomised clinical trial (rct) abstracts and mask them, an approach we refer to as custom masking. ## method in addition to formally defining the task we undertake, this section discusses the data used as well as the different stages of our proposed pbp strategy. ## experiments in our experiments, we use several plms that are pre-trained on clinical texts such as pubmed abstracts, which often report outcomes such as biobert @xcite , scibert @xcite and biomed_roberta @xcite . additionally, we include umls-bert because it augments bert's pre-training input with semantic type embeddings aligned to clinical knowledge (semantic types) in the unified medical language system (umls) metathesaurus @xcite . we also use bert @xcite as a vanilla plm that has not been pre-trained specifically on clinical texts. ## results in this section, we evaluate how well the model generates health outcomes when queried to answer a given prompt. for example, "after patients were given sorafenib, they reported [mask]", the model should correctly generate the outcome fatigue for the [mask]. ## related work interrogating plms with fill-in-the-blank prompts to determine their knowledge and awareness of factual information is a trending paradigm in nlp. despite the emergence of subtle techniques such as automating prompt structuring @xcite , selectively updating parameters of lms and prompts (also known as continuous prompting) @xcite , or even not tuning at all @xcite , several works including these still heavily rely on handcrafted prompts to use in probing lms. our efforts are motivated by the fact that we need not worry about the nature of the prompt, but rather can leverage on information local to the prompt such as word positions to probe the lms. we attempt to enhance a word's contextualised representation with position based representations to capture the word's position relative to the mask to be filled. previously some works have used similar positionaware attention over lstms for relation extraction, sequence labelling and slot filling tasks in different datasets @xcite . to the best of our knowledge, we are the first to use an extra position-attention layer above transformer models such as bert to solve the fill-in-the-blank prompting task. ## conclusion this paper assesses the possibility of ignoring the constraint of aligning prompts to specific linguistic patterns in prompting tasks that aim to store knowledge in lms that could later be retrieved or transferred for fact generation tasks. in experiments using clinical domain datasets (supporting ebm tasks), we show that the position-based attention implemented over contextualised lms can improve the ability of plms to recall facts such as outcomes (biomedical entities) encountered during training. we further observe our proposed model is able to generalise across unseen prompts, performing considerably well for cloze and mixed (extremely rare in pbl tasks) prompts. with the obtained experimental results, despite not aligning our prompts to commonly followed linguistic patterns, we can positively answer the question posed in §3.1 by claiming that plms are knowledgeable of stored facts.