From Speculation Detection to Trustworthy Relational Tuples in Information Extraction (2023)

Abstract: Speculation detection is an important NLP task to identify text factuality. However, the extracted speculative information (e.g., speculative polarity, cue, and scope) lacks structure and poses challenges for direct utilization in downstream tasks. Open Information Extraction (OIE), on the other hand, extracts structured tuples as facts, without examining the certainty of these tuples. Bridging this gap between speculation detection and information extraction becomes imperative to generate structured speculative information and trustworthy relational tuples. Existing studies on speculation detection are defined at sentence level; but even if a sentence is determined to be speculative, not all factual tuples extracted from it are speculative. In this paper, we propose to study speculations in OIE tuples and determine whether a tuple is speculative. We formally define the research problem of tuple-level speculation detection. We then conduct detailed analysis on the LSOIE dataset which provides labels for speculative tuples. Lastly, we propose a baseline model SpecTup for this new research task.

## introduction the concept of speculation is closely connected to modality, which has been extensively explored in both linguistics and philosophy @xcite . modality is defined as the expression of the speaker's level of commitment to the events mentioned in a text @xcite . other related terms, such as "hedging", "evidentiality", "uncertainty", and "factuality", are also used when discussing different aspects of speculation. it is argued that information falling under speculation cannot be presented as factual information @xcite . in nlp applications, factual information plays a critical role in comprehending sentences or documents. consequently, the identification of speculative information, as potentially opposed to facts, has garnered considerable interest from nlp community @xcite @xcite . the conventional task of speculation detection involves two subtasks: cue identification and scope detection, which are typically performed at the sentence level. speculation cue refers to a minimal linguistic unit that indicates speculation (e.g., "might", "plans to", "subject to"). speculation scope is the text fragment governed by the corresponding cue in a sentence. consider the following example sentence: "the un plans to @xcite .", the speculative cue "plans to" affects the meaning of the scope "release a report". note that the speculation, along with its cue and scope, is often unstructured. question i arises: can unstructured speculative information be directly applied to downstream tasks? to extract structured information, we can leverage on open information extraction (oie). oie aims to generate relational factual tuples from unstructured open-domain text @xcite . the extracted tuples are in form of (arg 0 , relation, arg 1 , . . . , arg n ), also called facts. by definition, oie system is domain-independent and highly scalable, thus allowing users to obtain facts with low cost. the extracted factual tuples are beneficial to many downstream tasks such as question answering @xcite , knowledge base population @xcite , and text summarization @xcite . here, question ii arises: shall we trust all relational tuples extracted by oie as facts? apparently, the answer is no, as not all sentences state facts, partly due to speculation. current studies on oie do not consider speculation, which can lead to the use of unreliable tuples as facts, affecting the accuracy of downstream applications. to the best of our knowledge, there is no dataset specifically designed for tuple-level speculation detection. nevertheless, the recently released lsoie (solawetz and larson, 2021) provides a timely preview of this interesting research task. lsoie is a large-scale oie dataset converted from the qa-srl 2.0 dataset @xcite . we observe that lsoie provides additional annotation to some oie tuples with modal auxiliary verbs. tuple-level speculation detection is challenging, because it is common for only a certain portion of a sentence to carry speculative semantics. certain words (e.g., "may", "if ", "plan to"), also known as speculation cues, are responsible for semantic uncertainty, making part of a sentence (or the corresponding extracted tuples) vague, ambiguous, or misleading. in this paper, we develop a simple yet effective baseline model to detect speculation at tuple level (called spectup). spectup detects speculation from two perspectives: semantic and syntactic. to model relation-aware semantics, spectup adds additional relation embedding into bert transformers, and uses bert's hidden state of the tuple relation word as semantic representation. for syntactic modeling, spectup explicitly models the sub-graph of dependency structure of input sentence, which includes immediate neighbours of the tuple relation word. it adaptively aggregates nodes in the sub-graph using a novel relation-based gcn, and uses the aggregated representation as syntactic representation. the concatenated semantic and syntactic representations are then used for speculation detection. our contributions in this paper are threefold. first, we propose a new research task to detect tuple-level speculation. this task links speculation detection and information extraction. it examines the reliability of relational tuples, which aligns well with the goal of oie to extract only factual information. second, we conduct a detailed analysis on the tuple-level speculation labels from two aspects: (i) their presence in language, and (ii) the level of detection difficulty. third, we propose spectup, a baseline model to detect tuplelevel speculation. spectup leverages both semantic (bert) and syntactic (sub-dependency-graph) representations. we perform extensive experiments to analyze the research task of tuple-level speculation, and our results show that spectup is effective. ## tuple-level speculation analysis we first review the annotation processes of the qa-srl bank 2.0 and lsoie datasets, with a key focus on tuple-level speculation labels. we then study the distribution of speculation labels, by the perceived level of difficulty in detection. ## task formulation as discussed, there are no existing oie systems consider speculation when extracting tuples. to fully leverage the capabilities of existing oie models, it is more meaningful and practical to formulate speculation detection as a post-processing task, where the goal is to determine whether a tuple extracted from a sentence is speculative. formally, the inputs are the source sentence that is split into words w i (or tokens) @xmath0, . . . , w n ], and a set of relational tuples @xmath1, . . . , t m } extracted from this sentence. 5 sentence-level speculation detection makes one-pass prediction on sentence s. in comparison, the task of tuple-level speculation detection is to predict whether a tuple t i is speculative (i.e., a binary classification task), based on t i and its source sentence s. we focus on tuplelevel speculation detection in this paper. 5 each tuple ti is represented by its components @xmath2, . . . , x l ] where one x is the relation and the rest x's are arguments. each x corresponds to a contiguous span of words [wj, . . . , w j+k ]. as the tuples set t typically contains multiple tuples, sentence s will be analyzed multiple times with different t i . we assume that the relation (or relation word) of a tuple is different from the relations of all the other tuples from the same sentence and thus that a tuple can be represented with its relation. note that the task can be extended to predict if a tuple is speculative and, if speculative, which of the six modal auxiliary verbs indicates the speculation type of the tuple. ## method: spectup the proposed baseline consists of three modules: pre-trained bert to encode semantic representation of input sequence (in section 4.1), gcn encoder to capture the syntactic representation (in section 4.2), and linear layer for binary/multiclass speculation detection (in section 4.3). given an example tuple (the un, release, a report) in figure 1 , the words "plans to" in its source sentence, which modifies the relation word "release", indicate the speculation of the tuple. note that such a cue of the speculation of a tuple is typically syntactically related to the relation word of the tuple in its source sentence. we thus encode the dependency structure of a source sentence, as exemplified in figure 1 , as well as the word sequence of the sentence for the speculation detection task. ## multi-class speculation classification as mentioned in section 3, we extend the task to predict both the existence of speculation and its specific type, as defined by the 6 auxiliary modal verbs: 'might', 'can', 'will', 'would', 'should', and 'had'. in this way, we perform multi-class classification among 7 classes: non-speculative and 6 types of speculative classes. ## conclusion speculation detection, which is essential in many applications in ir and ie, has not been explored in oie. so we formally define a new research task to perform tuple-level speculation detection on oie extracted tuples. we notice that the lsoie dataset, although not dedicated for speculation detection, provides us a timely preview of this interesting research task. we conduct a detailed analysis on the speculative tuples of the lsoie dataset. to provide a glimpse of this research task, we develop a simple but effective model named spectup to detect tuplelevel speculation. as an emerging research area, we believe there is a big room for further exploration, from dataset construction to more effective models.