Learning Bill Similarity with Annotated and Augmented Corpora of Bills (2021)

Abstract: Bill writing is a critical element of representative democracy. However, it is often overlooked that most legislative bills are derived, or even directly copied, from other bills. Despite the significance of bill-to-bill linkages for understanding the legislative process, existing approaches fail to address semantic similarities across bills, let alone reordering or paraphrasing which are prevalent in legal document writing. In this paper, we overcome these limitations by proposing a 5-class classification task that closely reflects the nature of the bill generation process. In doing so, we construct a human-labeled dataset of 4,721 bill-to-bill relationships at the subsection-level and release this annotated dataset to the research community. To augment the dataset, we generate synthetic data with varying degrees of similarity, mimicking the complex bill writing process. We use BERT variants and apply multi-stage training, sequentially fine-tuning our models with synthetic and human-labeled datasets. We find that the predictive performance significantly improves when training with both human-labeled and synthetic data. Finally, we apply our trained model to infer section- and bill-level similarities. Our analysis shows that the proposed methodology successfully captures the similarities across legal documents at various levels of aggregation.

## introduction in the u.s. and other democracies, a bill becomes a law through a complex political process, which includes a series of political debates and compromises within their legislative branch. a consistent pattern that arises from this political interaction is the emergence of several lengthy omnibus bills which cover a range of topics/special interest and combine parts of numerous smaller bills. in fact, most bills tend to be a mixture of a number of other bills, whether enacted or "dead." furthermore, in the u.s., many bills are modified and re-introduced between the two chambers of congress. uncovering these complex relationships among bills is important because it helps scholars better understand the legislative process, one of the most important parts of representative democracy. in this paper, we develop a methodology to characterize this relationship across bills. specifically, we leverage semantic similarities between subsections to infer bill linkages using annotated and augmented synthetic corpora. previous studies have investigated this issue in both political science @xcite and computer science @xcite . to find similar bills, they use either a simple bag-of-words model or the smith-waterman local alignment algorithm (swalign) @xcite . although this latter approach is useful in identifying exact text matches, it cannot capture semantic similarities across bills that arise from sentence reordering or paraphrasing. figure 1 shows two closely related bills that contain nearly identical policy ideas, but this similarity would be difficult to detect with swalign. we note the differences between these bills arise primarily from their distinct bill structures and insertion and deletion of words and sentences. we propose a 5-class classification task that closely reflects the nature of the bill generation process. we begin by constructing a novel humanlabeled dataset of 4,721 bill-to-bill relationships across the 111 th u.s. congress bills at the subsection level. using this data, we apply multi-stage training and perform two-stage fine-tuning on models for classification. specifically, we train several bert variants including bert @xcite , roberta @xcite , and legal-bert @xcite . our analysis shows that the two-stage training significantly improves figure 1 : an example of a subsection pair sharing similar policy ideas. we highlight the different parts. this example shows the possible text edit operations: insertion, deletion, swap, and synonym replacement. the classification performance across all models. finally, we extend the trained model to section-and bill-level pairs, which confirms that our methodology successfully determines the similarity at higher levels of aggregation between legal documents. in sum, our contributions are as follows: ## related work bill similarity several studies have examined bill linkages. @xcite measure sentence pair similarity in state legislation using a unigram bag-of-words model. others use the swalign algorithm to detect aligned sequences of text between bills @xcite @xcite . in contrast, we use transformer-based language models (roberta, legal-bert, and bert) trained with humanannotated and synthetically generated bill pairs. citation recommendation citation recommendation involves finding relationships between document-sized texts such as papers or articles @xcite . a bill is also in the form of a document and bill similarity can be posed as a task to retrieve related legislation. recently, this is done with ensemble models of traditional information retrieval (ir) models and bert variants for pre-fetching and re-ranking in @xcite . one weakness of their study is that they rely only on the obvious relationship between an eu directive and the corresponding uk regulation that complies with the directive, ignoring many other legal text pairs that may be related in other ways. to overcome this weakness, we find the linkage of bills in a bottom-up manner by splitting a bill into subsection pieces. subsections are smaller units and are easier to both annotate and measure similarity. from subsection results, we aggregate similarity scores up to the bill-level. we engage in large-scale human-annotation of legal text similarities. we are, in fact, working with magnitudes larger set of bills, as most bills are introduced but do not become law, and many are re-introduced with modifications. plagiarism detection plagiarism detection is also a task of finding a connection between two texts. work done by @xcite in plagiarism detection involves detecting similar plagiarized parts between two documents. while this method could be useful to determining similarities between bills, it is difficult to indicate the degree of similarity between two texts using this method. additionally, the dataset is collected by crowdsourcing with multiple topics, which means the data does not reflect the characteristics of bills. in this paper, we propose a new dataset that represents the similarity of bill pairs by reflecting the unique characteristics of legislation. multi-stage training multi-stage training improves downstream tasks, especially for a small dataset or a domain-specific task @xcite . there are many different bert models pretrained for various domains and tasks @xcite @xcite . these models are further trained with unlabeled large domain-specific corpora or synthetic training data @xcite . the subsection pair dataset we build is relatively small. to improve the classification performance, therefore, we apply two-stage training method with bills as text corpus. in doing so, we build a large synthetic pairs dataset and perform two-stage fine-tuning on models. ## data generation this section explains how we annotate subsection pairs and generate the synthetic dataset. ## experiment setting we conduct subsection-level classification tasks with various settings, four different models and multiple sizes of combination for both synthetic and human-annotated datasets. ## results we compare the performance of the models with various experiment settings described above. ## application we evaluate the performance of our trained model at two distinct levels of aggregation: (1) sectionlevel and (2) bill-level. our analysis is motivated by the legislative process through which political interests at the subsection-level could be aggregated into larger legislation as politicians engage in negotiations and logrolling. specifically, we develop a measure of text similarity at higher levels of aggregation. we evaluate the performance of our measurements against the section-level similarity scores developed by @xcite and the proxy bill-level similarity measure based on the co-occurrence of lobbying activities. we compare the results with roberta and swalign that were trained with the best performance setting: 10,000 synthetic pairs and 3,305 human-labeled pairs for roberta, 1,000 synthetic pairs and all 3,305 the human-labeled pairs for swalign. @xcite developed 6 alignment categories at the section-pair level according to the similarity in policy ideas: 1: clear policy idea match, 2: likely policy idea match, 3: same topic, different idea, 4: different topic, 5. boilerplate, and 6: junk. they focus on the similarity between the sections from bills that are related to the patient protection and affortable care act (ppaca), known as obamacare (h.r.3590) in the 111 th u.s. congress. ## conclusion in this paper, we developed a methodology to identify the links between legislative bills. we proposed 5-class classification task and constructed a humanlabeled dataset of 4,721 legislative text pairs. additionally, we generated synthetic datasets varying the degree of similarities to boost the classification performance. furthermore, we applied the models trained on the subsection-level to section-and bill-level pairs. we have shown that our approach successfully captures similarities across legislative bills at different levels of aggregation. our model provides an empirical framework as well as novel measurements that allow scholars to investigate bill-to-bill networks in legislative politics. this is important as policy preferences of various political actors are reflected in different parts of bills at different stages in the legislative process. bill networks can be a medium that helps scholars both to understand the legislative process and to analyze the hidden connections present between various actors such as politicians, lobbyists, and interest groups. for instance, using our model, re-searchers can conduct an in-depth analysis of how several seemingly different bills, in effect, originate from an identical policy idea and political interest @xcite . moreover, our model has broader applications in studying legal texts. in particular, it can be used to study policy diffusion across states @xcite or across countries @xcite , find lobbying patterns between politicians and lobbyists @xcite , and trace constantly changing relevant laws or regulations @xcite . much research is needed to understand the political process that underlies the bill linkages. while we have focused on the measurements, we hope that the outputs that we produced in this paper could be useful not only for improving methodological research but also for future computational social science research examining political connections in the legislative process that are often hidden to public eyes.