The Global Banking Standards QA Dataset ( GBS - QA ) (2021)

Abstract: A domain specific question answering (QA) dataset dramatically improves the machine comprehension performance. This paper presents a new Global Banking Standards QA dataset (GBS-QA) in the banking regulation domain. The GBS-QA has three values. First, it contains actual questions from market players and answers from global rule setter, the Basel Committee on Banking Supervision (BCBS) in the middle of creating and revising banking regulations. Second, financial regulation experts analyze and verify pairs of questions and answers in the annotation process. Lastly, the GBS-QA is a totally different dataset with existing datasets in finance and is applicable to stimulate transfer learning research in the banking regulation domain.

## introduction global banking standards have been strictly organized and gradually evolved over time to reflect the changes of financial environment and new risks emerged. the basel committee on banking supervision (bcbs) which is one of international rule setters governs to create and revise the international banking standards in cooperation with member countries and international financial supervisory agencies. creating new rules and revising current standards require clear communications with related parties such as financial supervisory authorities and market players over the globe. in the process of this work, frequently asked questions (faqs) are formally constructed and disclosed in order to help understand better how to implement provisions in practice. the faqs contains questions from market players and answers from the bcbs. however, original pairs of questions and answers in the faqs are not standardized so that they need to be re-organized and revised in order to become qualified practices on the corresponding provisions. considering that financial regulations are directly influential to the market and they require expertise to interpret and understand the context in the faqs, financial regulation experts should be involved. we propose a new global banking standards qa dataset (gbs-qa). the gbs-qa has three values. first, given that gbs-qa is composed of actual questions and answers in practice, both market players and regulators find it quite useful at work as additional guidelines. second, financial regulation experts participate in this study to construct revised qa pairs which can be properly applied into the nlp model. third, the gbs-qa is a totally different dataset with existing datasets in finance and is applicable to stimulate transfer learning research in the banking regulation domain. this paper is constructed as follows. related work is reviewed in section 2. the process of collecting and annotating the gbs-qa is described in section 3. the experimental results of the pretrained language model and its domain-adaptive model on gbs-qa are revealed in section 4. ## related work there have been many researches on constructing new datasets in various domains and utilizing them on their own purpose. in finance, financial news sentiment dataset @xcite and financial opinion mining and questions answering dataset @xcite ) are widely utilized. however, since these are basically rooted from financial news and general narratives in financial reports, they are not equipped with terminology in financial regulation and of course are not qualified for doing tasks in financial regulation domain. aside from construction of new datasets, how to make ai-enabled nlp models be more understandable and explainable is one of top priorities. recently centered on medical domain, text entailment(abacha and demner-fushman, 2016; fei et al., 2021) and deep neural reasoning @xcite are applied to do question answering tasks and combination of these approaches and input medical text contributes to achieve the goal of enhancing interpretability and explainability. from a perspective of few-shot setting, few-shot textual entailment framework combined with graphical structure in data also shows its effectiveness in the downstream tasks @xcite . in addition, domain-adaptive pretraining is proposed as one of the guaranteed way to lead to performance gains @xcite in domainspecific tasks. finbert @xcite and biobert @xcite , for example, are types of domain-adaptive models. ## gbs-qa gbs-qa is collected from the bcbs website 1 . the bcbs framework comprises the 14 standards and each standard has its provisions and the associated faqs which are matched with the corresponding provisions. as shown in figure 1 , risk-based capital requirements (rbc) which is one of the standards contains 4 sub-categories and each subcategory has a varying number of provisions with different length. the provision of 30.12, for example, raises two questions from market players so that the bcbs answers these questions with detailed explanations. overview of a gbs-qa construction process follows figure 2 . starting from bcbs framework, all standards and the associated faqs are automatically filtered out of the bcbs website. these data are organized and preprocessed by human annotation process. human annotation includes organizing provisions, matching the provisions with the 1 https://www.bis.org/basel_framework/ corresponding faqs and reviewing questions and answers in a reconciled manner. this review is conducted by independent annotators which consists of five financial regulation experts. from the human annotation process, pairs of questions and answers are classified into four types which include 1) binary answerable type, 2) wh type, 3) how type and 4) conditional type. upon question classification, questions are revised into "binary answerable type" and answers are labelled into "yes" or "no" to corresponding questions according to the gbs-qa classification guideline in appendix a. after completing the whole process in figure 2 , gbs-qa is constructed as a set of questions and answers in banking regulation domain. ## model architecture the experiments aim to show that gbs-qa can be applicable to stimulate transfer learning on the classification task. considering uniqueness and expertise of gbs-qa, domain-adaptive language models which are trained with all provisions in current standards and all answers are used to do this task. as shown in figure 4 , it starts from pre-trained models such as roberta @xcite and electra @xcite . by training domain knowledge of banking standards, pre-trained models are evolved into post-trained model which is qualified for dealing with gbs-qa. next, these post-trained models are fine-tuned for classification task with pairs of questions and answers("yes" or "no) and the associated provisions. the final model is called bankreg qa model. ## experiments due to lack of data points in the gbs-qa, there is no significant difference in experimental results between pre-trained and post-trained model in the gbs-qa. moreover, some questions have longer length than 512 which is maximum length in the model and the other questions require interpretation of equations to calculate risk factors or risk exposure so that the models can not handle them properly. however, the post-trained model shows slightly lower performance in both fiqa and phrasebank dataset. ## conclusion we propose a new qa dataset which is called gbs-qa in the banking regulation domain. the gbs-qa contains pairs of questions and answers re-organized and verified by financial regulation experts. extra information extracted from the gbs-qa can become qualified practices on the associated provisions in the regulation. from the analysis and experiments, the gbs-qa shows that it has a different words distribution compared to other existing datasets in finance and it can be applicable in the nlp models.