Extending the Scope of Out-of-Domain: Examining QA models in multiple subdomains (2022)

Abstract: Past work that investigates out-of-domain performance of QA systems has mainly focused on general domains (e.g. news domain, wikipedia domain), underestimating the importance of subdomains defined by the internal characteristics of QA datasets. In this paper, we extend the scope of “out-of-domain” by splitting QA examples into different subdomains according to their internal characteristics including question type, text length, answer position. We then examine the performance of QA systems trained on the data from different subdomains. Experimental results show that the performance of QA systems can be significantly reduced when the train data and test data come from different subdomains. These results question the generalizability of current QA systems in multiple subdomains, suggesting the need to combat the bias introduced by the internal characteristics of QA datasets.

## introduction examining the out-of-domain performance of qa systems is an important focus of the research community due to its direct connection to the generalizability and robustness of qa systems especially in production environments @xcite @xcite @xcite . even though previous studies mostly focus on coarse-grained general domains @xcite , the importance of finer-grained subdomains defined by the internal characteristics of qa datasets cannot be neglected. for example, several studies exploring specific internal characteristics of qa datasets have been carried out, including @xcite , who reveal that the sentence-level answer position is a source of bias for qa models, and @xcite who investigate the effect of word-level question-context overlap. building on this prior work as well as the definition and discussion of subdomain in @xcite ; @xcite ; subdomain-1 subdomain-2 subdomain-n train data subdomain-n qa-system-1 qa-system-2 qa-system-n train train train evaluate subdomain-1 subdomain-2 test data . . . . . . . . . . . . . . . . . . varis and bojar (2021), we extend the scope of outof-domain with a view to assessing the generalizability and robustness of qa systems by investigating their out-of-subdomain performance. as shown in figure 1 , we split the qa dataset into different subdomains based on its internal characteristics. then we use the qa examples in each subdomain to train corresponding qa systems and evaluate their performance on all subdomains. we focus on extractive qa as it is not only an important task in itself @xcite but also the crucial reader component in the retriever-reader model for open-domain qa @xcite . in experiments with squad 1.1 @xcite and newsqa @xcite , we split the data into subdomains based on question type, text length (context, question and answer) and answer position. we then train qa systems on each subdomain and examine their performance on each subdomain. results show that qa systems tend to perform worse when train and test data come from different subdomains, particularly those defined by question type, answer length and answer position. ## experiments we employ the qa datasets, squad1.1 @xcite and newsqa @xcite . for squad1.1 we use the official dataset released by @xcite and for newsqa we use the data from mrqa @xcite . for question classification, we use the dataset from @xcite . we use the bert-base-uncased model from huggingface @xcite for both question classification and qa. 1 we adopt the following setup for training and evaluation: we split the original training set d into several disjoint subdomains d a , d b , d c , . . .; then we sample subsets from each subdomain using sample sizes n 1 , n 2 , n 3 , . . . in ascending order. the resulting subsets are denoted we evaluate each qa system on the test data t which is also split into disjoint subdomains t a , t b , t c , . . . similar to the training data d. ## conclusion we presented a series of experiments investigating the out-of-subdomain performance of qa systems on two popular english extractive qa datasets: squad1.1 and newsqa. the experimental results show that the subdomains defined by question type, answer length and answer position inject strong bias into qa systems, with the result that the performance of qa systems is negatively impacted when train and test data come from different subdomains. the experiments provide useful information on how to control question diversity, answer length distribution as well as answer positions when constructing qa datasets and employing qa systems. in future work, we aim to apply our analysis to multilingual data to explore how qa models behave across different languages and we plan to investigate other types of qa beyond extractive. genes with a most recent common ancestor, and thus a shared evolutionary ancestry, are known as homologs. these genes appear either from gene duplication within an organism's genome, where they are known as paralogous genes, or are the result of divergence of the genes after a speciation event, where they are known as orthologous genes,:7.6 and often perform the same or similar functions in related organisms. it is often assumed that the functions of orthologous genes are more similar than those of paralogous genes, although the difference is minimal. long how does the water vapor that rises in warm air turn into clouds? solar radiation is absorbed by the earth's land surface, oceans -which cover about 71% of the globe -and atmosphere. warm air containing evaporated water from the oceans rises, causing atmospheric circulation or convection. when the air reaches a high altitude, where the temperature is low, water vapor condenses into clouds, which rain onto the earth's surface, completing the water cycle. the latent heat of water condensation amplifies convection, producing atmospheric phenomena such as wind, cyclones and anti-cyclones. sunlight absorbed by the oceans and land masses keeps the surface at an average temperature of 14 °c. by photosynthesis green plants convert solar energy into chemically stored energy, which produces food, wood and the biomass from which fossil fuels are derived. in conformation shows, also referred to as breed shows, a judge familiar with the specific dog breed evaluates individual purebred dogs for conformity with their established breed type as described in the breed standard. as the breed standard only deals with the externally observable qualities of the dog (such as appearance, movement, and temperament), separately tested qualities (such as ability or health) are not part of the judging in conformation shows. according to a story that has often been repeated in the media, hurley and chen developed the idea for youtube during the early months of 2005, after they had experienced difficulty sharing videos that had been shot at a dinner party at chen's apartment in san francisco. karim did not attend the party and denied that it had occurred, but chen commented that the idea that youtube was founded after a dinner party ẅas probably very strengthened by marketing ideas around creating a story that was very digestible ¨. front who became chairman of the council of ministers in 1985? in the fall of 1985, gorbachev continued to bring younger and more energetic men into government. on september 27, nikolai ryzhkov replaced 79-yearold nikolai tikhonov as chairman of the council of ministers, effectively the soviet prime minister, and on october 14, nikolai talyzin replaced nikolai baibakov as chairman of the state planning committee (gosplan). at the next central committee meeting on october 15, tikhonov retired from the politburo and talyzin became a candidate. finally, on december 23, 1985, gorbachev appointed yeltsin first secretary of the moscow communist party replacing viktor grishin. front during what seasons is fog common in boston? fog is fairly common, particularly in spring and early summer, and the occasional tropical storm or hurricane can threaten the region, especially in late summer and early autumn. due to its situation along the north atlantic, the city often receives sea breezes, especially in the late spring, when water temperatures are still quite cold and temperatures at the coast can be more than 20 °f (11 °c) colder than a few miles inland, sometimes dropping by that amount near midday. thunderstorms occur from may to september, that are occasionally severe with large hail, damaging winds and heavy downpours. although downtown boston has never been struck by a violent tornado, the city itself has experienced many tornado warnings. damaging storms are more common to areas north, west, and northwest of the city. boston has a relatively sunny climate for a coastal city at its latitude, averaging over 2,600 hours of sunshine per annum.