Tracing Influence at Scale: A Contrastive Learning Approach to Linking Public Comments and Regulator Responses (2023)

Abstract: U.S. Federal Regulators receive over one million comment letters each year from businesses, interest groups, and members of the public, all advocating for changes to proposed regulations. These comments are believed to have wide-ranging impacts on public policy. However, measuring the impact of specific comments is challenging because regulators are required to respond to comments but they do not have to specify which comments they are addressing. In this paper, we propose a simple yet effective solution to this problem by using an iterative contrastive method to train a neural model aiming for matching text from public comments to responses written by regulators. We demonstrate that our proposal substantially outperforms a set of selected text-matching baselines on a human-annotated test set. Furthermore, it delivers performance comparable to the most advanced gigantic language model (i.e., GPT-4), and is more cost-effective when handling comments and regulator responses matching in larger scale.

## introduction policymakers rely on information provided by external stakeholders to help design new regulations. for u.s. federal regulators, this process is formalized by the administrative procedures act which requires that whenever an agency is going to make a policy change (known as a "rule"), they must first publish a proposed rule and accept public comment. then, in the final rule, the agency must respond to comments they received. the number of comments received by regulators has been growing over time, and the federal government now regularly received more than a million comments per year. existing research suggest that public comments can have substantial impacts on public policy (yackee, 2019). however, measuring the influence of in-dividual organizations or tracking patterns of influence over time has been limited by the challenging nature of the data. both comments and regulator responses are in gigantic scale and take the form of complex natural language text. prior attempts at large-scale analysis have borrowed insights from the research field of nlp by measuring the lexical overlap between comments and rule text, with researchers assuming that a high degree of overlap is suggestive of influence @xcite @xcite . however, this approach provides at best a noisy measure of influence, which is difficult to verify. therefore, we aim for pursuing a more precise and efficient measure based on analyzing the regulator's responses to comments and then matching comments to specific responses. given that some responses are positive, with agencies accepting commenter's suggestions while others are negative, with the agency rejecting the comment, it is very important to link the right comments to the right responses. in this paper, we propose a simple yet effective iterative contrastive learning paradigm to train a neural-based comment-response matcher in an unsupervised manner. specifically, we first construct a pseudo training dataset comprising of hard positive and negative samples generated by the inital setup of our proposed comment-response matcher (sbert @xcite as the backbone). this matcher is then optimized on the obtained training pseudo data and subsequently utilized to generate the hard positive and negative examples for the next iteration. through empirical evaluation on a human-annotated test set, our proposed comment-response matcher not only surpasses selected unsupervised text-matching benchmarks utilized in previous literature but also achieves comparable performance with the state-ofthe-art gigantic language model -gpt-4 (openai, 2023), while remaining more cost-effective to deploy on the full-scale comment-response matching. ## comment-response matcher in this paper, we aim to design a text matching model (section 2.1) that can effectively and efficiently assess the semantic relevance between the public comment text and responses produced by regulators. in essence, given a comment chunk from public @xmath0, • • • , c m } and a regulator's response @xmath1, • • • , r n }, where each c k is a token in the comment and each r k is a token in the response, our goal is to learn a function f : (c, r) → s that predicts the score s indicating the likelihood that comment c and regulator's response r pertain to the same topic, and that the concern in c is addressed in r. as illustrated in figure 1 , we employ an iterative contrastive learning paradigm with the training procedure (section 2.2) consisting of two steps performed alternatingly, namely hard pos./neg. mining and model updating. ## conclusion in this paper, we propose a simple yet effective contrastive learning approach following the iterative data construction -model updating training scheme, aiming for automatically matching the responses in policy regulations and relevant comments they respond to. our empirical study on a real-world test set demonstrates that our proposal outperforms a set of selected benchmarks for text matching in terms of correlation with human annotations, achieves comparable performance but is more costeffective than the most advanced gigantic large language model (i.e., gpt-4) for comments and regulator responses in larger scale. our proposed approach can be easily adapted to other text matching applications dealing with text in rather different complexity, such as name matching @xcite , or extended to other more-resourced scenarios like semi-supervised settings, which we will leave as our future work.