Regulatory Compliance through D oc2 D oc Information Retrieval: A case study in EU / UK legislation where text similarity has limitations (2021)

Abstract: Major scandals in corporate history have urged the need for regulatory compliance, where organizations need to ensure that their controls (processes) comply with relevant laws, regulations, and policies. However, keeping track of the constantly changing legislation is difficult, thus organizations are increasingly adopting Regulatory Technology (RegTech) to facilitate the process. To this end, we introduce regulatory information retrieval (REG-IR), an application of document-to-document information retrieval (DOC2DOC IR), where the query is an entire document making the task more challenging than traditional IR where the queries are short. Furthermore, we compile and release two datasets based on the relationships between EU directives and UK legislation. We experiment on these datasets using a typical two-step pipeline approach comprising a pre-fetcher and a neural re-ranker. Experimenting with various pre-fetchers from BM25 to k nearest neighbors over representations from several BERT models, we show that fine-tuning a BERT model on an in-domain classification task produces the best representations for IR. We also show that neural re-rankers under-perform due to contradicting supervision, i.e., similar query-document pairs with opposite labels. Thus, they are biased towards the pre-fetcher’s score. Interestingly, applying a date filter further improves the performance, showcasing the importance of the time dimension.

## introduction major scandals in corporate history, from enron to tyco international, olympus, and tesco, foot_0 have led to the emergence of stricter regulatory mandates and highlighted the need for regulatory compliance where organizations need to ensure that they comply with relevant laws, regulations, and policies @xcite . however, keeping track of the constantly changing legislation (figure 1 ) is hard, thus organizations are increasingly adopting regulatory technology (regtech) to facilitate the process. typically, a compliance regimen includes three distinct but related types of measures, corrective, detective, and preventive @xcite . corrective measures are usually undertaken when new regulations are introduced to update existing controls. detective measures, ensure "afterthe-fact" compliance, i.e., following a procedure, a manual or automated check is carried out, to ensure that every step of the procedure complied with the corresponding regulations. finally, preventive measures ensure compliance "by design", i.e., during the creation of new controls. all types of measures include an underlying information retrieval (ir) task, where laws need to be retrieved given a control or vice versa. we identify two use cases: regulatory information retrieval (reg-ir), similarly to other applications of document-todocument (doc2doc) ir, is much more challenging than traditional ir where the query typically contains a few informative words and the documents are relatively small ( dataset domain q d ir datasets in the literature trec robust @xcite news 3 / 14 254 bioasq @xcite biomedical 9 197 ir datasets with verbose queries gov2 @xcite web 11 / 57 682 wt10g @xcite since reg-ir is a new task, our starting point is the two-step pipeline approach followed by most modern neural information retrieval systems @xcite @xcite . first, a conventional ir system (pre-fetcher) retrieves the k most prominent documents. then a neural model attempts to rank relevant documents higher than irrelevant ones. in most approaches, the pre-fetcher is based on okapi bm 25 @xcite , a bag-of-words scoring function that does not consider possible synonyms or contextual information. to overcome the first limitation, we follow @xcite who employed k nearest neighbors over tf-idf weighted centroids of word embeddings, without however improving the results, probably because the centroids are noisy considering many uninformative words. furthermore, we employ bert @xcite to extract contextualized representations for queries and documents but again the results are worse than bm 25 . we also experiment with s-bert (reimers transpositions: we have retrieved all transposition relations (approx. 3.7k) between eu directives and uk laws from the cellar database. cellar only provides the mapping between the cellar ids of eu directives and the title of each uk law. therefore we aligned the cellar ids with the official uk ids based on the law title. 7 one or more uk laws may transpose one or more eu directives. ## ir pipelines modern neural ir systems usually follow a twostep pipeline approach. first, a conventional ir system (pre-fetcher) retrieves the top-k most prominent documents aiming to maximize its recall. then a neural model attempts to re-rank the documents by scoring relevant higher than irrelevant ones. while this configuration is widely adopted in literature, the re-ranking step could be omitted provided an effective pre-fetching mechanism, i.e., the pre-fetcher will act as an end-to-end ir system. ## experimental results pre-fetching: ## related work ir in the legal domain is widely connected with the competition on legal information extraction/entailment (coliee). from 2015 to 2017 @xcite @xcite , the task was to retrieve japanese civil code articles given a question, while in coliee 2018 and 2019 @xcite , the task was to retrieve supporting cases given a short description of an unseen case. however, the texts of these competitions are small compared to our datasets. also, most submitted systems do not consider recent advances in ir, i.e, neural ranking models @xcite @xcite , which have recently managed to improve rankings of conventional ir, or end-to-end neural models which have recently been proposed @xcite . again, these end-to-end methods were applied on small texts. on the other hand, there has been some work trying to cope with larger queries, i.e., verbose or expanded queries, @xcite @xcite . nonetheless, the considered queries are at most 60 tokens long, contrary to our datasets where, depending on the setting, the average query length is 1.8k or 2.6k tokens ( ## conclusions and future work we proposed doc2doc ir, a new family of ir tasks, where the query is an entire document, thus being more challenging than traditional ir. this family of tasks is particularly useful in regulatory compliance, where organizations need to ensure that their controls comply with the existing legislation. in the absence of publicly available doc2doc datasets, we compile and release two datasets, containing eu directives and uk laws transposing these directives. experimenting with conventional (bm 25 ) and neural pre-fetchers we showed that a bert model fine-tuned on an in-domain classification task, i.e., predict eurovoc concepts, is by far the best pre-fetcher in our datasets. we also showed that neural re-rankers fail to improve the performance, as their term matching mechanisms degenerates, and over-utilize the pre-fetcher score. in the future, we would like to investigate alternatives in exploiting additional information that may be critical in the newly introduced tasks (eu2uk, uk2eu). in this direction naively utilizing chronological information leads to vast performance improvement in eu2uk dataset. one possible direction is to model the cross-document relations (e.g., amendments) using graph convolutional networks @xcite , while better modeling the dimension of time (i.e., chronological difference between a query and a document) is also crucial. further on, to better deal with long documents, we plan to investigate text summarization by employing a state-of-the-art neural summarizer, e.g., bart of @xcite , or sentence selection techniques, e.g., rationale extraction @xcite , to find the most important sections or sentences and create shorter and more informative versions of queries/documents. that implement them. we initially collect approximately 10k eu2uk pairs. in these pairs the transposed eu law is referred to by its unique portal id but the transposing uk law is referred to by its title. this is the primary challenge in modelling the transposition relations, since mapping legislation titles to unique entries in our uk corpus is not trivial. we hypothesize that these relations are manually inserted in the database and therefore human errors make performing exact matches often impossible. apart from the matching difficulties, some of the pairs in the pool are inserted mistakenly and hence need to be filtered.