Challenges in Technical Regulatory Text Variation Detection (2025)

Abstract: We present a preliminary study on the feasibility of using current natural language processing techniques to detect variations between the construction codes of different jurisdictions. We formulate the task as a sentence alignment problem and evaluate various sentence representation models for their performance in this task. Our results show that task-specific trained embeddings perform marginally better than other models, but the overall accuracy remains a challenge. We also show that domain-specific fine-tuning hurts the task performance. The results highlight the challenges of developing NLP applications for technical regulatory texts.

## introduction in canada, the regulation of building construction is the jurisdiction of the provinces and territories (p/t). national research council of canada is responsible for publishing the national model construction codes foot_0 (nmcc), a collection of model construction codes, to help promote consistency among the p/t construction codes. the model construction codes set out the technical requirements for the design and construction of new buildings, as well as the change of use and demolition of existing buildings. the nmcc are adopted individually by each p/t legislatures while modifications are made to suit local needs. as a result, there are various kinds of variations between the nmcc and each of the p/t construction codes. variations in construction codes create barriers to the free movement of construction goods, services, and investments within canada. to support inter-p/t trades and the mobility of talents, harmonizing the construction codes across canada is a key priority in the development and maintenance of the codes and tracking code variations is the first step towards code harmonization. due to the high volume and high technicality of the construction codes, tracking variations in them is a difficult and expensive task that requires labor-intensive studies done by codes advisors. some of the challenges in technical domain knowledge include identifying similarity/dissimilarity in definitions of technical terms or material specifications when they are applied in context, e.g. "sawdust" could be equivalent to "combustible pallets" in the fire codes. this study explores the potential of leveraging recent development of natural language processing (nlp) techniques to detect variations between the nmcc and the adopted codes in each p/t. the nmcc consists of the national building code (nbc), the national fire code (nfc), the national plumbing code (npc), and the national energy code for buildings(necb). each p/t has the autonomy in deciding how the nmcc are adapted, or referenced. for example, for the 2015 edition of nbc, nfc, and npc, new brunswick, nova scotia, manitoba and saskatchewan adopted them with some modifications and additions; while ontario developed their codes based on the national models, but with significant variations in content and scope and made major reorganization of the order of the provisions. the technical provisions of the construction codes (both the nmcc and the p/t construction codes) are structured, segmented and numbered and the detailed provisions are found at the sentence level. hence, the majority of variations tracked at the sentence level. the variations in the construction codes are classified by national and p/t codes advisors into the following categories: the sentences in the second example of editorial changes in in this paper, we study the potential and challenges of detecting variations between the nmcc and the construction codes in each p/t. we evaluate the performance of different sentence representation models on the sentence alignment task. our results show that developing nlp applications for technical regulatory texts requires more investments and efforts. ## sentence alignment methods sentence alignment methods have two major components: 1) the sentence representation model for reflecting how similar sentences are to each other; and 2) alignment extraction approach for deciding the sentence alignment given the sentence similarity scores. ## experiments the nmcc are updated and published once every five years and each of the p/t adopts the updated version in subsequent years. national and p/t codes advisors then work on analyzing the variations. the latest edition of nmcc with all variations tracked and classified was published in 2015 @xcite . in our experiments, we use the english version of the 2015 edition of the nmcc and the corresponding p/t construction codes. we create the training and testing sets for our experiments by doing a random 80/20 split. ## results the tf-idf weighted bag-of-words model performs better than both of the pretrained sentence embedding models (laser and labse) and only marginally worse than the task specific trained embedding models. the domain specific fine-tuning model performs significantly worse than the base model before finetuning. this is perhaps due to limited amount of in-domain data used in fine-tuning the model. overall, the aer for all our experiments are high. with an aer higher than 13% by all models before attempting to classify editorial and technical variation, we demonstrated that nlp research and development on technical regulatory texts remains an open question and great challenge. ## conclusion in this study, we explored the potential of using current natural language processing techniques to detect variations between the national model construction codes and the construction codes of different canadian p/t. we evaluated various sentence representation models on this task. the overall bad performance across all models suggests that current nlp technologies are not yet fully equipped to handle the complexity of technical regulatory text. this highlights the need for further research in developing nlp models that could acquire the necessary technical knowledge from technical regulatory text and improving the accuracy and reliability of nlp tools in technical regulatory application.