S cope I t: Scoping Task Relevant Sentences in Documents (2020)

Abstract: A prominent problem faced by conversational agents working with large documents (Eg: email-based assistants) is the frequent presence of information in the document that is irrelevant to the assistant. This in turn makes it harder for the agent to accurately detect intents, extract entities relevant to those intents and perform the desired action. To address this issue we present a neural model for scoping relevant information for the agent from a large document. We show that when used as the first step in a popularly used email-based assistant for helping users schedule meetings, our proposed model helps improve the performance of the intent detection and entity extraction tasks required by the agent for correctly scheduling meetings: across a suite of 6 downstream tasks, by using our proposed method, we observe an average gain of 35% in precision without any drop in recall. Additionally, we demonstrate that the same approach can be used for component level analysis in large documents, such as signature block identification.

## proposed method in this section, we outline our approach to the problem of scoping out relevant sentences for a nlp-based scheduling assistant. our approach consists of 2 parts: a preprocessing module and a neural model. an incoming email is first passed through the preprocessing module. the preprocessed email is then tokenized, indexed and passed through the neural model to generate a confidence score for each sentence. the model is trained end-to-end with human-labeled gold scores denoting the relevant sentences of the email. we also adopt some data augmentation methods to improve model generalization. ## main results ## clustering in the embedding space we next investigate if sentence embeddings generated by scopeit exhibit any clusters that make semantic sense. on preliminary analysis, some clusters that we observed in the data were salutations, signature blocks and sentences containing entities associated with scheduling meetings. we hypothesize that similar clusters should be observed in the sentence embedding space. to test this hypothesis, we propose the following experiment: given the embedding of a sentence belonging to a certain cluster (the query sentence), retrieve the top k nearest neighbor sentence embeddings from a set of sentence embeddings generated by scopeit. if similar clusters exist in the embedding space, then sentences associated with the retrieved embeddings should belong in the same cluster as the query sentence. due to space constraints, we describe the methodology of the embedding experiment in appendix c, and report our main findings here. we observe that salutations and signatures are clustered together. we also observe sub-clusters wherein sentences containing similar entities or intents are clustered with sentences containing similar information. moreover, we find that these sentence embeddings also capture the context in which the sentences occur: syntactically similar query sentences get mapped to different clusters based on the context in which they occur. example input: "let us get together on a team's call." expected output: conference call intent this task is modeled as a multi-label classification task, and we use a context aware deep network to tackle it. we use a model similar to the one proposed in @xcite . specifically, we generate a contextual embedding using bert for each token in the email. then, an attention method @xcite , one for each label, is used to aggregate the embeddings into a document embedding, which is then passed through a sigmoid layer to generate the probability for each label. the entire model is trained end to end by minimizing the negative log likelihood of the gold labels. while using scopeit, we only select sentences that occur above a particular threshold (0.01), and feed the concatenation of those sentences as inputs to the model. ## signature block detection as described in §5, we observed that sentences with similar semantics were clustered close to each other in the sentence embedding space. we use this observation to apply our model to component detection in email, specifically for signature block identification. we show the model's performance on a publicly available dataset, and show that it outperforms the baseline model. we also hypothesize that the publicly available systems for extracting signatures are not suitable for real-world use-cases, as they are often trained on well structured emails using hand crafted features, and hence are not robust to the variety of writing styles that people employ in the real world. in order to validate this hypothesis, we test the effectiveness of the baseline on our use-case. ## related work our problem of relevance scoping in documents is similar to extractive summarization. extractive summarization deals with selecting subsets (usually sentences) of a document that succinctly summarizes it. for the case of this scheduling assistant, scoping out the relevant part in an email document is in essence selecting the subset of sentences from the email that accurately summarizes the scheduling intent and specifies the parameters necessary to schedule a meeting correctly. both traditional feature based methods using word probability, tf-idf weights, sentence position and sentence length features @xcite @xcite and recent neural methods @xcite @xcite @xcite have been used for the task of extractive summarization. @xcite show the benefit of using pretrained language models @xcite @xcite @xcite for the same task. their proposed model bertsum leverages interval segment embeddings to distinguish multiple sentences within a document. bertsum further also finetunes the bert embeddings to learn the segment embeddings during training, which potentially requires more data; and can also only encode upto 512 wordpiece long documents. in contrast, we used hierarchical rnns (similar to @xcite ), with the pretrained embeddings forming the embedding layer (fig. 1 ); thereby allowing us to encode emails much larger than 512 tokens long. intent classification and entity extraction tasks in the context of conversational understanding have been studied both in academia and corporate research laboratories @xcite . there exists a rich body of research in user intent identification from targeted queries @xcite . however, these methods don't work as well when applied to large documents. we showed that scoping out the relevant parts in a document improves performance of classification and extraction tasks on large queries. to the best of our knowledge, this is the first work to explore the utility of extractive summarization as a preprocessing step for tackling problems involving large text corpora. there has been extensive research on the topic of identifying signature blocks and reply lines from an email @xcite @xcite . @xcite present heuristic driven methods for unsupervised identification of the signature body, while @xcite present a crf based approach for identifying and extracting signature and reply lines from email. we showed in §7 that our proposed method also works well for removing signatures and also generalizes better. ## conclusion in this paper, we proposed a simple method for scoping relevant information within emails and the impact the model had on a suite of tasks that are vital for the scheduling assistant. we also showed our models applicability on the task of signature detection. we show that it performs better than existing publicly available baselines and generalizes better on real world use-cases. in this work we showed that our model works well with emails. for future work, we plan on investi-gating the impact of our proposed method for other tasks that process large textual inputs (eg: document classification, sentiment analysis on large reviews). furthermore, using bert for inference poses latency challenges in a production system. a promising direction of future work that we plan on investigating is leveraging distilled versions of bert @xcite for the task.