P rofiling News Discourse Structure Using Explicit Subtopic Structures Guided Critics (2021)

Abstract: We present an actor-critic framework to induce subtopical structures in a news article for news discourse profiling. The model uses multiple critics that act according to known subtopic structures while the actor aims to outperform them. The content structures constitute sentences that represent latent subtopic boundaries. Then, we introduce a hierarchical neural network that uses the identified subtopic boundary sentences to model multi-level interaction between sentences, subtopics, and the document. Experimental results and analyses on the NewsDiscourse corpus show that the actor model learns to effectively segment a document into subtopics and improves the performance of the hierarchical model on the news discourse profiling task.

## introduction news discourse profiling is a discourse processing task that aims to classify sentences in news articles into different content types, where each content type characterizes the specific discourse role of a sentence in describing a news story @xcite . it is vital to effectively contextualize the occurrence of a news event, which has been shown useful for extracting event structures from a document @xcite . furthermore, this task is likely to benefit a range of other nlp applications that require deep story-level text understanding, such as text summarization and complex question answering. as the discourse roles are interpreted with respect to the main event, the current approach for discourse profiling uses a hierarchical neural network model @xcite ) that relies on a sentence-level encoder to obtain sentence embeddings, that heed to the local context, and a document embedding to obtain the underlying main topic. the hierarchical model, intuitively, provides a mechanism for capturing both global and local dependencies among sentences and the main topic. however, the model is completely unaware of the underlying content organization structures that are used while producing news reports. besides, squeezing document-level features into a single vector provides limited space to learn effective document representation and model its interaction with the sentences. to extend the modeling capabilities and incorporate document-level content organization structures in the hierarchical model, we propose to decompose a document into latent subtopics by identifying subtopic boundary sentences, and model two levels of interactions-between sentences and a subtopic, and between subtopics and the document. we hypothesize that learning subtopic representations allows the model to focus on the locally relevant sentences, independent of main content, and identify its fine-grained discourse function within local subtopical context. further interactions between subtopics and the document vector helps to determine the broader role of a subtopic with respect to the main content. for instance, in news document in figure 1 , we can identify discourse role of sentence s7 by combining the two levels of information. first, sentences s6 and s8 describe events that happened years before the main event which can be modelled through interaction between document and subtopic embedding corresponding to @xcite . then, events in s7 has temporal proximity with the events in sentences s6 and s8 which can be modelled through interaction between sentence s7 embedding and subtopic embedding. several past works have independently studied the subtopic structures, and a document can exhibit multiple subtopic structures depending on the used segmentation criteria. in this paper, we consider two subtopic structures: 1) broad-genre topic segments generated by the texttiling algorithm; and 2) news genre-specific inverted pyramid structure identified through a plausible rules-based surrogate (section 5). depending on the document, one of these subtopic structures may be more suitable for identifying discourse role labels. also depending on the document, the most suitable subtopic structure may not be strictly the same as one of the known subtopic structures. for instance, in figure 1 , we show three subtopic structures, namely, texttiling, inverted pyramid structure, and a new subtopic structure that is based on temporal frames. here, three subtopic structures only partially overlap with each other. notably, sentences in each of the segments obtained by considering temporal frames exhibit homogeneous discourse role labels and could be the most suitable structure to consider. in this work, we limit ourselves to indirectly using known explicit subtopic structures as critics in a new variant of actor-critic model that selects between the standard reinforce (williams, 1992) algorithm or imitation learning for training actor. specifically, when subtopic structure identified by actor performs better than all known explicit subtopic structures, we baseline the standard reinforce (williams, 1992) algorithm with the average of reward obtained by all explicit subtopic structures. on the other hand, if one of the explicit subtopic structures perform better than the actor, we force the actor to imitate that subtopic structure. intuitively, this allows the actor model to learn to identify subtopic boundary sentences most suitable for a given document that perform better than or at least comparable to a known explicit subtopic structure on discourse profiling task. experimental results on the newsdiscourse corpus show that modeling latent subtopic structures in a hierarchical discourse model improves its performance by 2.6 and 1.3 points on average macro and micro f1 scores respectively. the improvement is consistent to different model initialization and shows that modeling the underlying sequential content organization structure enables the system to better predict content types for individual sentences. ## related work theory on news content organization the theoretical studies on the organization of different news elements @xcite @xcite @xcite @xcite have been extensively explored through case-studies in journalism and discourse. for instance, @xcite and filak (2019) studied the widely used inverted pyramid structure that follows the standard relevance ordering. specific to the content structures of hard-news reports, @xcite observed that the generic hard-news exhibit non-linear structure where sub-components such as consequences, causes, contextualization or other supportive information possess orbital relationship with the main event. further, these sub-components are organized around the main event with repetitions of the most newsworthy event, i.e. main content. this aligns with our inverted pyramid structure based on rule of relevance ordering, where a main sentence following non-main sentences indicates the segment boundaries. besides, feature news also follow welldefined content organization structures such as having an introductory anecdotes or back-grounding which is taken care by our rules to separate historical and anecdotal contents from the relevance ordering. neural models for discourse modeling deep neural networks have been successfully explored for modeling discourse @xcite , including hierarchical models @xcite @xcite to induce hierarchical structure. @xcite , however, found that some of the improvements from neural models on rst parsing are attributed to differences in evaluation procedures. nonetheless, @xcite concluded that neural models are more effective in modeling discourse, though the relative error reduction rates are lower than reported. the better discourse modeling capabilities of neural models are also evident from their widespread adoption in follow-up works such as @xcite al. (2019); zhang et al. (2020) and koto et al. (2021). for automatic text segmentation, multitude of approaches such as lexical overlap, bayesian learning or dynamic programming @xcite @xcite @xcite have been proposed. the recent works rely on neural network models to learn different aspects of text segmentation such as coherence and cohesion @xcite @xcite . ## task description news discourse profiling categorizes sentences in news articles into eight schematic categories that are defined following the news content schemata proposed by van dijk @xcite @xcite . the eight content types describe the common discourse roles of sentences in telling a news story. specifically, main event (m1) sentences introduce the main event relating to the major subjects of a news article. consequence (m2) sentences describe consequence events immediately triggered by the main event. previous events (c1) sentences describe the recent events that act as possible causes or preconditions for the main event while current context (c2) sentences describe remaining context informing contents. news articles may also describe past events that precede the main events in months and years (historical event (d1)) or unverifiable situations that are often fictional or personal accounts of incidents of an unknown person (anecdotal event (d2)). lastly, opinionated contents including reactions from immediate participants, experts, known personalities as well as journalists or news sources are covered in the evaluation (d3) category, except speculations and projected consequences that are labeled as expectation (d4). ## model we model the discourse profiling task as a two step process. given a news document x : {h, x 1 , x 2 , .., x n } comprising of headline h and n sentences with their content-type labels y : {y 1 , y 2 , .., y n }, our main goal is to learn a model f : @xmath0. in the first step, a latent function f t : @xmath1, 2, .., n} k , a classifier, is used to identify k subtopic boundary sentences in the document. these boundary sentences are used to partition documents into multiple subtopics. in the second step, a classification function f c : [x, t ] → y combines the output of latent function f t with the sentences x in document to perform final content-type classification (figure 2 ). overall, the model consists of a sentence encoder, a bilstm-based @xcite ) hierarchical encoder to obtain contextualized sentence representations used by both f t and f c , and a pointer decoder network to select subtopic boundary sentences that is exclusively used by f t . ## document-level content organization structures we experiment with two subtopic structures: 1) broad-genre topic segments and 2) news genrespecific inverted pyramid structure. note that both topic segments and inverted pyramid structures are automatically identified through statistical model or a plausible rules-based surrogate, which may not perfectly represent subtopic boundaries. ## results and analysis tables 1 and 2 compare all models on the validation and test datasets respectively. first, on the validation dataset, the best performing self-critic, text-tiling, and joint-ip models perform similar to the hierarchical model. this is also evident from the improved performance of the three actor-critic models, rl-tt, rl-ip and rl-ip/tt. actor-critic based learning helps models to learn to identify subtopic boundary sentences that are useful for news discourse profiling but not strictly same as the used subtopic structures. overall, our best performing rl-ip/tt model yields 2.72 and 0.97 points higher macro f1 and micro f1 score over the hierarchical model on the validation set, with comparable margin of improvement on the average f1 scores on test dataset. rm-ip rm-tt ip-tt rm-tt-ip overlap 324 236 139 83 the consistent improvement from different models provides evidence that learning latent content organization structure to further segment documents and modeling both local subtopic representations and global document representations leads to better main topic induction and achieve improved contenttype classification performance. between texttiling and inverted pyramid subtopic structures, we observe that the latter performs better in both joint learning and actor-critic learning frameworks. this is expected since tex-tiling is a broad-genre subtopic structure while inverted pyramid structure is specific to the news articles. further, our rules ( §5.2) used to build local inverted pyramid structures directly correlate with the content types of sentences. it is also worth noting that jointly using texttiling and inverted pyramid structures as critics performs better than each structure when used individually. ## conclusion we have presented a document-level contentorganization structures aware neural network model for discourse profiling. we explored actorcritic learning based frameworks to induce subtopic structures in a news document. then, we model two levels of interactions -between sentences and the local subtopic representation, and between subtopic representations and the document -that consistently outperformed the previous best hierarchical model. for future work, we intend to experi-ment with new modeling techniques to incorporate explicit subtopic structures. further, we plan to extend topical structures to other discourse structures such as rhetorical relations and model their interdependencies with the discourse profiling task.