CBLUE : A C hinese Biomedical Language Understanding Evaluation Benchmark (2022)

Abstract: Artificial Intelligence (AI), along with the recent progress in biomedical language understanding, is gradually offering great promise for medical practice. With the development of biomedical language understanding benchmarks, AI applications are widely used in the medical field. However, most benchmarks are limited to English, which makes it challenging to replicate many of the successes in English for other languages. To facilitate research in this direction, we collect real-world biomedical data and present the first Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark: a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, single-sentence/sentence-pair classification, and an associated online platform for model evaluation, comparison, and analysis. To establish evaluation on these tasks, we report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state-of-the-art neural models perform by far worse than the human ceiling.

## introduction artificial intelligence is gradually changing the landscape of healthcare, and biomedical research @xcite . with the fast advancement of biomedical datasets, biomedical natural language processing (bionlp) has facilitated a broad range of applications such as biomedical text mining, which leverages textual data in electronic health records (ehrs). a key driving force behind such improvements and rapid iterations of models is the use of general evaluation datasets and benchmarks @xcite . pioneer benchmarks, such as blurb @xcite , pubmedqa @xcite , and others, have provided us with the opportunity to conduct research on biomedical language understanding and developing real-world applications. unfortunately, most of these benchmarks are developed in english, which makes the development of the associated machine intelligence anglo-centric. meanwhile, other languages, such as chinese, have unique linguistic characteristics and categories that need to be considered. even though chinese speakers account for a quarter of the world population, there have been no existing chinese biomedical language understanding evaluation benchmarks. to address this issue and facilitate natural language processing studies in chinese, we take the first step in introducing a comprehensive chinese biomedical language understanding evaluation (cblue) benchmark with eight biomedical language understanding tasks. these tasks include named entity recognition, information extraction, clinical diagnosis normalization, short text classification, question answering (in transfer learning setting), intent classification, semantic similarity, and so on. we evaluate several pre-trained chinese language models on cblue and report their performance. the current models still perform by far worse than the standard of single-human perfor-mance, leaving room for future improvements. we also conduct a comprehensive analysis using case studies to indicate the challenges and linguistic differences in chinese biomedical language understanding. we intend to develop a universal gluelike open platform for the chinese bionlp community, and this work helps accelerate research in that direction. overall, the main contributions of this study are as follows: ## related work several benchmarks have been developed to evaluate general language understanding over the past few years. glue @xcite ) is one of the first frameworks developed as a formal challenge affording straightforward comparison between taskagnostic transfer learning techniques. superglue @xcite , styled after glue, introduce a new set of more difficult language understanding datasets. other similarly motivated benchmarks include decanlp @xcite , which recast a set of target tasks into a general questionanswering format and prohibit task-specific parameters, and senteval @xcite , which evaluate explicitly fixed-size sentence embeddings. non-english benchmarks include rus-siansuperglue @xcite and clue @xcite , which is a community-driven benchmark with nine chinese natural language understanding tasks. these benchmarks in the general domain provide a north star goal for researchers and are part of the reason we can confidently say we have made great strides in our field. for bionlp, many datasets and benchmarks have been proposed @xcite @xcite which promote the biomedical language understanding @xcite @xcite . @xcite propose biomedical language understanding datasets as well as a competition on largescale biomedical semantic indexing and question answering. @xcite propose pubmedqa, a novel biomedical question answering dataset collected from pubmed abstracts. @xcite propose bioread, which is a publicly available cloze-style biomedical machine reading comprehension (mrc) dataset. @xcite create a leaderboard featuring the biomedical language understanding & reasoning benchmark (blurb). unlike a general domain corpus, the annotation of a biomedical corpus needs expert intervention and is labor-intensive and time-consuming. moreover, most of the benchmarks are based on english; ignoring other languages means that potentially valuable information may be lost, which can be helpful for generalization. in this study, we focus on chinese to fill the gap and aim to develop the first chinese biomedical language understanding benchmark. note that chinese biomedical text is linguistically different from english and has its domain characteristics, necessitating an evaluation bionlp benchmark designed explicitly for chinese. ## experiments baselines we conduct experiments with baselines based on different chinese pre-trained language models. we add an additional output layer (e.g., mlp) for each cblue task and fine-tune the pre-trained models. ## conclusion in this paper, we present a chinese biomedical language understanding evaluation (cblue) benchmark. we evaluate 11 current language representation models on cblue and analyzed their results. the results illustrate the limited ability of state-ofthe-art models to handle some of the more challenging tasks. in contrast to english benchmarks such as glue/superglue and blurb, whose model performance already matches human performance, we observe that this is far from the truth for chinese biomedical language understanding. dations of china (61876052 and u1813215), national natural science foundation of guangdong, china (2019a1515011158), strategic emerging industry development special fund of @xcite lgg22f030011), ningbo natural science foundation (2021j190), and yongjiang talent introduction programme (2021a-156-g).