Multi-token Mask-filling and Implicit Discourse Relations (2025)

Abstract: Previous work has shown that simple mask-filling can provide useful information about the discourse informativeness of syntactic structures. Dong et al. (2024) first adopted this approach to investigating preposing constructions. The problem with single token mask fillers was that they were, by and large, ambiguous. We address the issue by adapting the approach of Kalinsky et al. (2023) to support the prediction of multi-token connectives in masked positions. Our first experiment demonstrates that this multi-token mask-filling approach substantially outperforms the previously considered single-token approach in recognizing implicit discourse relations. Our second experiment corroborates previous findings, providing additional empirical support for the role of preposed syntactic constituents in signaling discourse coherence. Overall, our study extends existing mask-filling methods to a new discourse-level task and reinforces the linguistic hypothesis concerning the discourse informativeness of preposed structures.

## introduction previous work has shown that simple mask-filling can provide useful information about discourse relations-in particular, about whether a preposed syntactic construction (e.g., a prepositional phrase moved from its canonical post-verbal position to the sentence-initial position) in one sentence can help identify its sense relation to its immediately preceding sentence @xcite . the discourse relations considered earlier were those that hold between adjacent english sentences that lack an explicit discourse connective providing information about how they are related (we call them implicit inter-sentential relations hereafter). since the previous work used bert @xcite , which is limited to predicting one token per @xcite , the mask fillers considered were restricted to single-token explicit dis-course connectives such as but, so, however, etc. as shown in ex. (1), the connective but is a bertpredicted mask filler inserted before the preposed prepositional phrase (pp; shown in bold) and was not present in the original sentence. in this case, but can signal a comparison relation between the two discourse arguments. to address this limitation, we adapt the extended-matrix decoder approach of @xcite to enable masked language models (mlms) to predict multi-token connectives as mask fillers. our contributions are twofold: (1) we present a system that directly generates multi-token discourse connectives in masked positions to help identify the discourse relation between text segments, and (2) we provide new empirical evidence supporting the role of preposed syntactic structures in signaling discourse relations. these results not only extend prior work but also demonstrate how language models can be used to evaluate and inform linguistic theories. specifically, we conduct two experiments. the first evaluates our multi-token completion approach against single-token mask filling on all implicit inter-sentential relations in the penn discourse treebank @xcite . results show that multi-token completion shows better performance in discourse relation recognition. in the second experiment, we implement our multi-token completion model on a subset of implicit inter-sentential relations where the second argument starts with a preposed syntactic structure. with less ambiguous fillers, our results provide further evidence for the discourse informativeness of preposed syntactic constituents. ## ambiguity in single-vs. multi-token connectives before presenting our main experiments, we begin by providing empirical support for the observation that multi-token connectives tend to be less ambiguous than single-token ones. to this end, we extracted all inter-sentential single-token and multitoken connectives from the pdtb-3, and for each of them, we also extracted its sense relations and the count of each relation. there are 104 types of single-token connectives and 61 types of multitoken connectives in the pdtb-3. as shown in figure 1 , around 60% of multi-token connectives are unambiguous (i.e., are mapped to only one sense relation), compared to just 35% of single-token connectives. among those associated with multiple senses, over 20% of single-token connectives are mapped to more than 5 senses, versus less than 10% for multi-token connectives. while the number of senses associated with a connective reflects its potential ambiguity, it does not capture how these senses are distributed. some connectives may have multiple possible senses but are strongly associated with only one or two in practice. to capture this variability, we compute the entropy of the sense distribution for each connective based on annotated frequencies in pdtb-3, where lower values indicate less ambiguity. to summarize connective-level ambiguity across the dataset, we report the average entropy. as shown in ## multi-token mask-filling model sections 4 and 5 provide the experimental setup for the experiments discussed in sections 6 and 7. ## experiment 1: multi-token vs. single-token mask filling to evaluate whether our multi-token mask-filling model can achieve comparable performance on the implicit discourse relation recognition task while reducing ambiguity, we compare it with the singletoken baseline model based on off-the-shelf bert @xcite . ## experiment 2: preposed vs. canonical syntax we then use our model to investigate whether multitoken connective mask-filling also provides evidence that preposed constituents help signal discourse coherence. this expands the empirical base of @xcite from single-token connectives to multi-token connectives, which tend to be less ambiguous (i.e., associated with fewer senses). ## conclusions and discussion as noted in section 3, single-token mask filling @xcite , despite performing fairly well, is limited by the high sense ambiguity of singletoken connectives. the current study addresses this in part, by using the efficient approach to multitoken mask-filling developed by @xcite (see section 2.3). we adapted their approach to discourse relation recognition by considering multi-token discourse connectives, which we show to be significantly less ambiguous than their single-token counterparts. the results of the first experiment provide evidence for the effectiveness of simple mask-filling with multi-token connectives for discourse relation recognition. we then assessed the effectiveness of our adaptation on the task of sense recognition of implicit discourse relations whose arg2 starts with a preposed syntactic structure. experimental results for the preposing task show that our multi-token mask-filling model achieves higher and more confident performance on recognizing discourse relations on the preposed set than the canonical set, thereby confirming the previous results from @xcite and validating that preposing indeed provides evidence for some (but not all) discourse relational senses. specifically, preposing significantly helps to indicate two sense types: expansion.instantiation.arg2-as-instance and contingency.cause.reason. our results are largely consistent with @xcite . regarding why preposing helps to signal certain discourse relations rather than others, we have an initial hypothesis. according to @xcite , preposing in english is felicitous when the information conveyed by the preposed constituent form an anaphoric link to the prior discourse-that is, either discourse-old (i.e., explicitly evoked in the prior discourse) or inferable from the prior discourse through partial ordering (e.g., type/subtype, entity/attribute, part/whole, etc.). we hypothesize that preposing helps to signal discourse relations when the linking relation between discourse entities also supports the coherence between the preposed sentence and the preceding discourse. in such cases, preposing serves as evidence for a discourse relation. otherwise, it doesn't. this hypothesis, however, requires further studies to testify. our results extend the usefulness of mask-filling, to assessing the possible discourse relevance of other non-canonical syntactic structures, such as post-posing and right extraposition (e.g., "there was a man outside, wearing a plastic raincoat.")