CAWES umm: A Contextual and Anonymous Walk Embedding Based Extractive Summarization of Legal Bills (2021)

Abstract: Extractive summarization of lengthy legal documents requires an appropriate sentence scoring mechanism. This mechanism should capture both the local semantics of a sentence as well as the global document-level context of a sentence. The search for an appropriate sentence embedding that can enable an effective scoring mechanism has been the focus of several research works in this domain. In this work, we propose an improved sentence embedding approach that combines a Legal Bert-based local embedding of the sentence with an anonymous random walk-based entire document embedding. Such combined features help effectively capture the local and global information present in a sentence. The experimental results suggest that the proposed sentence embedding approach can be very beneficial for the appropriate representation of sentences in legal documents, improving the sentence scoring mechanism required for extractive summarization of these documents.

## introduction automatic summarization of lengthy legal documents has several benefits regarding the quick understanding of these documents for various types of users like lawyers, judges, lawmakers, and the general public @xcite . one of the most popular ways of performing such automatic summarization is via extractive summarization approaches, where the main idea is to extract summary-worthy sentences directly from the original documents. this process involves the appropriate representation of the individual sentences of the document, followed by the summary worthiness scoring of these sentences. the quality of sentence representation and the subsequent scoring mechanism greatly impact the overall extractive summarization performance. this motivates the need for effective sentence embedding approaches that can capture both the meaning of the individual sentences and their global context with respect to the entire document. this work proposes an improved sentence embedding approach that combines domain-specific sentence embedding with feature-based anonymous walk embeddings (awe) of a document @xcite , which can help represent a sentence more effectively for extractive summarization. several research works have explored the problem of extractive summarization in the legal domain @xcite . casesummarizer @xcite is a tool which is specifically developed for summarizing legal judgment documents. in this approach, extractive summary is produced based on the frequency of words. in addition to the frequency, it also uses domain specific knowledge. in the recent years, neural networks based approaches have shown to be very effective for extractive summarization. most of these approaches have formulated the summarization task either as binary classification problem @xcite or classification followed by ranking @xcite of the sentences inside the documents. in order to perform such classification based summarization, researchers have extensively explored the problem of finding appropriate embeddings or representations of the individual sentences @xcite . in this work also, our main focus is on finding the appropriate sentence representations for the summarization of legal bills. representing a document in terms of a sentence connectivity graph is an idea which has been very popularly applied in the general text summarization domain @xcite @xcite , because it captures the document-level global context of each sentence effectively. this suggests that if local semantics of a sentence can be combined with a graph-based global context, appropriate summarization-centric sentence embeddings can be obtained. the key contributions of this work are given below:- following the introduction, the organization of the rest of the paper is done as follows: a brief description of the related work is given in section 2. a detailed description of our proposed method is given in section 3. the evaluation strategies are presented in section 4. the experimental results are given in section 5 along with a detailed discussion. finally section 6 concludes our work, by summarizing the key findings and the potential future research directions. ## related work there are popular classical unsupervised extractive approaches in general text summarization which either utilizes the frequency-based methods @xcite or graph-based methods @xcite for scoring sentences. finally, the top scoring sentences are picked up to form an extractive summary. several research works also find important sentences in a document, based upon the idea of singular value decomposition (svd), such as lsa @xcite . there is yet another popular classical approach in which sentences are added in a greedy manner into the summary as long as the kullback-lieber (kl) divergence keeps on decreasing between the document set and the summary set @xcite . recently a bayesian optimization (bo) based approach bo-textrank has been proposed by @xcite , in which the authors improve the textrank algorithm for extractive summarization. a neural network based supervised approach is proposed by @xcite , in which scores are assigned to each of the sentences of the document and the best among them are selected. the authors have formulated the sentence scoring task as a sentence classification problem for which the random ensemble and bert models are used as classifiers to predict the important sentences for summary formation. in @xcite , authors have proposed a novel approach called summarunner, which is a recurrent neural network (rnn) based approach in which the summarization task is formulated as the sequence classification task for extractive summarization of documents. another unsupervised neural network approach is proposed by @xcite where summary creation is done by firstly extracting the restricted boltzmann machine (rbm) based features followed by a feature enhancement step. several random walk based approaches have been proposed in the literature for the summarization of documents represented in terms of a graph. @xcite have proposed an affinitypreserving random walk for the multi-document summarization problem. the summary sentences are extracted once the random walk reaches a stationary state for the purpose of summary generation. in another work, @xcite have proposed a random walk model in which utterances are the nodes and the relationship between two utterances is determined with the help of topic relevance, opinion relevance and structure relevance features. finally, pagerank algorithm-based global ranking is done to select the relevant utterances to form an opinion summary. @xcite have proposed a topic-sensitive version of lexrank method @xcite where the sentence score is calculated based on the concept of random walks. the sentence score is determined by considering sentence's relevance to the query as well as it's similarity to other high scoring sentences. apart from these works, several efficient attention mechanisms have also been proposed in the literature for handling long documents and thus achieving better performance in downstream tasks such as summarization @xcite . from the literature review, it has been observed that most of the works ignore either the importance of domain specific knowledge or the capability to handle long documents. to deal with these shortcomings, a novel sentence representation approach is proposed in this work, which utilizes domain specific legal-bert embeddings of sentences along with awe based document graph embeddings. such combined sentence representation scheme can capture both the local sentence level information as well as the global document level information, thereby achieving better summarization of lengthy legal documents. the reason for utilizing awe is to find the accurate vectorized representation of the entire document graph. this vectorized representation can be efficiently found using anonymous walk distribution, as proven by @xcite . ## proposed approach the basic steps of our proposed approach cawe-summ (contextual anonymous walk embedding summarizer) to automatically generate the extractive summary of legal documents is presented in this section. our training dataset (d t r ) consists of {(d 1 , s 1 ), (d 2 , s 2 ), ....., (d m , s m )}, where (d i , s i ) corresponds to the i th document-summary pair in the dataset. the overall methodology has been depicted in fig. 1 . in the training phase, firstly, the legal-bert @xcite based sentence embeddings and anonymous walk embeddings of the entire document graph are combined via concatenation operation. after this, the combined representation is used in an mlp model for learning the binary classification of summary worthiness for each of the sentences. once the training phase is complete, we obtain a trained mlp model, which can be used at test time for predicting whether a sentence is summary-relevant or not. each of the individual steps in fig. 1 , which depicts the proposed summarization approach, is discussed in a detailed manner in the following subsections. ## summarization results and discussion it is important to note here that, in case of the us testing data, even though the best performances have been observed when we consider anonymous walk of length 6, still the difference between the other walk lengths are not that significant. in case of the ca testing dataset, the proposed approach has been able to outperform all the baselines and state-of-the-art methods; however here also we can see that the specific walk lengths do not have very significant impact on improving the summarization performance. based on the number of sentences in a document, the summarization performance of the document changes. this change in performance is depicted with the help of line charts in fig. 4 . from the line charts it can be clearly observed that when we consider small-sized documents(# of @xmath0, we can achieve much better summarization performances, across different anonymous walk lengths. on the other hand, as we get medium-sized(51 ≤ # of @xmath1, we see that the summarization performance decrease significantly, for both the us test and ca test datasets. this is due to the fact that, always the top 15% of the high scoring sentences are picked for summary formation, and when a larger document comes as an input, the top 15% will include some low confidence predictions as well. moreover, this causes large-sized summary formation which might be detrimental by itself. studying the inference time of the proposed approach can give appropriate insights into its realtime applicability. this analysis is presented in fig. 5 , with the help of a line chart diagram. from fig. 5 , it can be observed that for both the test sets, the average inference time for generating summaries is in the range of (1 -7) seconds. another important observation is that, as the sentence embedding dimension increases, the inference time also increases sharply. this is to be expected, since larger embedding size are due to the presence of longer anonymous random walks with more number of samplings. repeated simulation of such long walks is bound to increase the total inference time, as during inference also the awe vectors for each sentence is needed to be calculated. one of the key findings of this work is that the inclusion of anonymous random walk based document graph embeddings as part of the sentence embedding itself can significantly improve the overall quality of sentence representation. such improved representation of sentences can help in the subsequent summary worthiness prediction process, as these sentences are aware of their global context in the document. the intuition behind such performance improvement is that through the learning of different anonymous walks, the embeddings are much more informative than only contextual embeddings since the anonymous walk embeddings can efficiently model the entire document. this effectiveness of awe based graph embeddings is supported by @xcite , where the authors prove that under sufficient samplings of anonymous random walks, entire subgraphs of the embedding dimension size summary generation time (in seconds) 0 2 4 6 8 768+5 768+15 768+52 768+203 us test ca test figure 5: average running time to generate summaries for billsum testing dataset where 5,15,52 & 203 are all possible anonymous walks for @xmath2, 4, 5 & 6 respectively. ## conclusion due to the lengthy nature of legal bills, it becomes very difficult to capture the important information of the documents. to overcome this difficulty, extraction of summary worthy sentences for automatic summarization has been explored in the literature. however, in order to efficiently identify summary worthy sentences, they need to be appropriately represented in the form of numerical vectors. in this work, we propose to capture a sentence's local as well as global context information in the form of embeddings via contextual and anonymous walk embeddings. from the experimental results, it has been found that, when we incorporate the global document level information with the sentence's local information, a significant improvement can be obtained in terms of rouge scores. the experimental results suggests that the anonymous walk embeddings are very effective in capturing the entire document graph information, and can enhance the representation of a sentence for its summary worthiness prediction. such improved sentence representation are able to significantly improve the extractive summarization of the document. to further improve the representation learning, leveraging embeddings in the form of hierarchical structure of the entire legal documents will be part of our future work. moreover, a more endto-end graph based approach can also be studied, by considering the emerging area of graph neural networks.