qinchuanhui/UDA-QA huggingface dataset:
Question: How big is their created dataset?

Evidence:
raw_evidence: The dataset comprises a total of 353 conversations from 40 speakers (11 nurses, 16 patients, and 13 caregivers) with consent to the use of anonymized data for research. The speakers are 38 to 88 years old, equally distributed across gender, and comprise a range of ethnic groups (55% Chinese, 17% Malay, 14% Indian, 3% Eurasian, and 11% unspecified). The conversations cover 11 topics (e.g., medication compliance, symptom checking, education, greeting) and 9 symptoms (e.g., chest pain, cough) and amount to 41 hours.
We divide the construction of data simulation into two stages. In Section SECREF16 , we build templates and expression pools using linguistic analysis followed by manual verification. In Section SECREF20 , we present our proposed framework for generating simulated training data. The templates and framework are verified for logical correctness and clinical soundness.

Answers:
353 conversations from 40 speakers (11 nurses, 16 patients, and 13 caregivers), we build templates and expression pools using linguistic analysis
Question: Which data do they use as a starting point for the dialogue dataset?

Evidence:
raw_evidence: We used recordings of nurse-initiated telephone conversations for congestive heart failure patients undergoing telemonitoring, post-discharge from the hospital. The clinical data was acquired by the Health Management Unit at Changi General Hospital. This research study was approved by the SingHealth Centralised Institutional Review Board (Protocol 1556561515). The patients were recruited during 2014-2016 as part of their routine care delivery, and enrolled into the telemonitoring health management program with consent for use of anonymized versions of their data for research.
To analyze the linguistic structure of the inquiry-response pairs in the entire 41-hour dataset, we randomly sampled a seed dataset consisting of 1,200 turns and manually categorized them to different types, which are summarized in Table TABREF14 along with the corresponding occurrence frequency statistics. Note that each given utterance could be categorized to more than one type. We elaborate on each utterance type below.
raw_evidence: We used recordings of nurse-initiated telephone conversations for congestive heart failure patients undergoing telemonitoring, post-discharge from the hospital. The clinical data was acquired by the Health Management Unit at Changi General Hospital. This research study was approved by the SingHealth Centralised Institutional Review Board (Protocol 1556561515). The patients were recruited during 2014-2016 as part of their routine care delivery, and enrolled into the telemonitoring health management program with consent for use of anonymized versions of their data for research.

Answers:
A sample from nurse-initiated telephone conversations for congestive heart failure patients undergoing telepmonitoring, post-discharge from the Health Management Unit at Changi General Hospital
recordings of nurse-initiated telephone conversations for congestive heart failure patients
Question: What labels do they create on their dataset?

Evidence:
raw_evidence: Figure FIGREF5 (b) illustrates the proposed dialogue comprehension task using a question answering (QA) model. The input are a multi-turn symptom checking dialogue INLINEFORM0 and a query INLINEFORM1 specifying a symptom with one of its attributes; the output is the extracted answer INLINEFORM2 from the given dialogue. A training or test sample is defined as INLINEFORM3 . Five attributes, specifying certain details of clinical significance, are defined to characterize the answer types of INLINEFORM4 : (1) the time the patient has been experiencing the symptom, (2) activities that trigger the symptom (to occur or worsen), (3) the extent of seriousness, (4) the frequency occurrence of the symptom, and (5) the location of symptom. For each symptom/attribute, it can take on different linguistic expressions, defined as entities. Note that if the queried symptom or attribute is not mentioned in the dialogue, the groundtruth output is “No Answer”, as in BIBREF6 .
raw_evidence: The dataset comprises a total of 353 conversations from 40 speakers (11 nurses, 16 patients, and 13 caregivers) with consent to the use of anonymized data for research. The speakers are 38 to 88 years old, equally distributed across gender, and comprise a range of ethnic groups (55% Chinese, 17% Malay, 14% Indian, 3% Eurasian, and 11% unspecified). The conversations cover 11 topics (e.g., medication compliance, symptom checking, education, greeting) and 9 symptoms (e.g., chest pain, cough) and amount to 41 hours.
Figure FIGREF5 (b) illustrates the proposed dialogue comprehension task using a question answering (QA) model. The input are a multi-turn symptom checking dialogue INLINEFORM0 and a query INLINEFORM1 specifying a symptom with one of its attributes; the output is the extracted answer INLINEFORM2 from the given dialogue. A training or test sample is defined as INLINEFORM3 . Five attributes, specifying certain details of clinical significance, are defined to characterize the answer types of INLINEFORM4 : (1) the time the patient has been experiencing the symptom, (2) activities that trigger the symptom (to occur or worsen), (3) the extent of seriousness, (4) the frequency occurrence of the symptom, and (5) the location of symptom. For each symptom/attribute, it can take on different linguistic expressions, defined as entities. Note that if the queried symptom or attribute is not mentioned in the dialogue, the groundtruth output is “No Answer”, as in BIBREF6 .

Answers:
(1) the time the patient has been experiencing the symptom, (2) activities that trigger the symptom (to occur or worsen), (3) the extent of seriousness, (4) the frequency occurrence of the symptom, and (5) the location of symptom, No Answer
the time the patient has been experiencing the symptom, activities that trigger the symptom, the extent of seriousness, the frequency occurrence of the symptom, the location of symptom, 9 symptoms
Question: How do they select instances to their hold-out test set?

Evidence:
raw_evidence: To evaluate the effectiveness of our linguistically-inspired simulation approach, the model is trained on the simulated data (see Section SECREF20 ). We designed 3 evaluation sets: (1) Base Set (1,264 samples) held out from the simulated data. (2) Augmented Set (1,280 samples) built by adding two out-of-distribution symptoms, with corresponding dialogue contents and queries, to the Base Set (“bleeding” and “cold”, which never appeared in training data). (3) Real-World Set (944 samples) manually delineated from the the symptom checking portions (approximately 4 hours) of real-world dialogues, and annotated as evaluation samples.
raw_evidence: To evaluate the effectiveness of our linguistically-inspired simulation approach, the model is trained on the simulated data (see Section SECREF20 ). We designed 3 evaluation sets: (1) Base Set (1,264 samples) held out from the simulated data. (2) Augmented Set (1,280 samples) built by adding two out-of-distribution symptoms, with corresponding dialogue contents and queries, to the Base Set (“bleeding” and “cold”, which never appeared in training data). (3) Real-World Set (944 samples) manually delineated from the the symptom checking portions (approximately 4 hours) of real-world dialogues, and annotated as evaluation samples.

Answers:
1264 instances from simulated data, 1280 instances by adding two out-of-distribution symptoms and 944 instances manually delineated from the symptom checking portions of real-word dialogues
held out from the simulated data
Question: Which models/frameworks do they compare to?

Evidence:
raw_evidence: Table TABREF14 show the results obtained for proposed s2sL approach in comparison to that of MLP for the tasks of Speech/Music and Neutral/Sad classification, by considering different proportions of training data. The values in Table TABREF14 are mean accuracies (in %) obtained by 5-fold cross validation. It can be observed from Table TABREF14 that for both tasks, s2sL method outperforms MLP, especially at low resource conditions. s2sL shows an absolute improvement in accuracy of INLINEFORM0 % and INLINEFORM1 % over MLP for Speech/Music and Neutral/Sad classification tasks, respectively, when INLINEFORM2 of the original training data is used in experiments.
raw_evidence: In this work, we propose a novel approach to address the task of classification in low data resource scenarios. Our approach involves simultaneously considering more than one sample (in this work, two samples are considered) to train the classifier. We call this approach as simultaneous two sample learning (s2sL). The proposed approach is also applicable to low resource data suffering with data imbalance. The contributions of this paper are:
MLP, the most commonly used feed forward neural network, is considered as the base classifier to validate our proposed s2s framework. Generally, MLPs are trained using the data format given by eq. INLINEFORM0 . But to train the MLP on our s2s based data representation (as in eq. INLINEFORM1 ), the following modifications are made to the MLP architecture (refer to Figure FIGREF4 ).
Table TABREF14 show the results (in terms of INLINEFORM0 values) obtained for proposed s2sL approach in comparison to that of MLP for Anger/Happy classification (data imbalance problem). Here, state-of-the-art methods i.e., Eusboost [22] and MWMOTE [23] are also considered for comparison. It can be observed from Table TABREF14 that the s2sL method outperforms MLP, and also performs better than Eusboost and MWMOTE techniques on imbalanced data (around 3 % absolute improvement in INLINEFORM1 value for s2sL compared to MWMOTE, when INLINEFORM2 of the training data is considered). In particular, at lower amounts of training data, s2sL outperforms all the other methods, illustrating its effectiveness even for low resourced data imbalance problems. s2sL method shows an absolute improvement of 6% ( INLINEFORM3 ) in INLINEFORM4 value over the second best ( INLINEFORM5 for MWMOTE), when only INLINEFORM6 of the training data is used.

Answers:
MLP
Eusboost, MWMOTE
Question: Which classification algorithm do they use for s2sL?

Evidence:
raw_evidence: The s2sL approach proposed to address low data resource problem is explained in this Section. In this work, we use MLP (modified to handle our data representation) as the base classifier. Here, we explain the s2sL approach by considering two-class classification task.
raw_evidence: In this work, we propose a novel approach to address the task of classification in low data resource scenarios. Our approach involves simultaneously considering more than one sample (in this work, two samples are considered) to train the classifier. We call this approach as simultaneous two sample learning (s2sL). The proposed approach is also applicable to low resource data suffering with data imbalance. The contributions of this paper are:
MLP, the most commonly used feed forward neural network, is considered as the base classifier to validate our proposed s2s framework. Generally, MLPs are trained using the data format given by eq. INLINEFORM0 . But to train the MLP on our s2s based data representation (as in eq. INLINEFORM1 ), the following modifications are made to the MLP architecture (refer to Figure FIGREF4 ).

Answers:
MLP
MLP
Question: Up to how many samples do they experiment with?

Evidence:
raw_evidence: We validate the performance of the proposed s2sL by providing the preliminary results obtained on two different tasks namely, Speech/Music discrimination and emotion classification. We considered the GTZAN Music-Speech dataset [17], consisting of 120 audio files (60 speech and 60 music), for task of classifying speech and music. Each audio file (of 2 seconds duration) is represented using a 13-dimensional mel-frequency cepstral coefficient (MFCC) vector, where each MFCC vector is the average of all the frame level (frame size of 30 msec and an overlap of 10 msec) MFCC vectors. It is to be noted that our main intention for this task is not better feature selection, but to demonstrate the effectiveness of our approach, in particular for low data scenarios.
The standard Berlin speech emotion database (EMO-DB) [18] consisting of 535 utterances corresponding to 7 different emotions is considered for the task of emotion classification. Each utterance is represented by a 19-dimensional feature vector obtained by using the feature selection algorithm from WEKA toolkit [19] on the 384-dimensional utterance level feature vector obtained using openSMILE toolkit [20]. For two class classification, we considered the two most confusing emotion pairs i.e., (Neutral,Sad) and (Anger, Happy). Data corresponding to Speech/Music classification (60 speech and 60 music samples) and Neutral/Sad classification (79 neutral and 62 sad utterances) is balanced whereas Anger/Happy classification task has data imbalance, with anger forming the majority class (127 samples) and happy forming the minority class (71 samples). Therefore, we show the performance of s2sL on both, balanced and imbalanced datasets.
raw_evidence: All experimental results are validated using 5-fold cross validation (80% of data for training and 20% for testing in each fold). Further, to analyze the effectiveness of s2sL in low resource scenarios, different proportions of training data, within each fold, are considered to train the system. For this analysis, we considered 4 different proportions i.e., INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 of the training data to train the classifier. For instance, INLINEFORM4 means considering only half of the original training data to train the classifier, and INLINEFORM5 means considering the complete training data. 5-fold cross validation is considered for all data proportions. Accuracy (in %) is used as a performance measure for balanced data classification tasks (i.e., Speech/Music classification and Neutral/Sad emotion classification), whereas the more preferred INLINEFORM6 measure [21] is used as a measure for imbalanced data classification task (i.e., Anger/Happy emotion classification).

Answers:
535
we considered 4 different proportions i.e., INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 of the training data to train the classifier
Question: Do they use pretrained models?

Evidence:

Answers:
No
Question: Do they report results only on English datasets?

Evidence:
raw_evidence: A promising strategy to bridge the gap mentioned above is to integrate the neural networks of MRC models with the general knowledge of human beings. To this end, it is necessary to solve two problems: extracting general knowledge from passage-question pairs and utilizing the extracted general knowledge in the prediction of answer spans. The first problem can be solved with knowledge bases, which store general knowledge in structured forms. A broad variety of knowledge bases are available, such as WordNet BIBREF7 storing semantic knowledge, ConceptNet BIBREF8 storing commonsense knowledge, and Freebase BIBREF9 storing factoid knowledge. In this paper, we limit the scope of general knowledge to inter-word semantic connections, and thus use WordNet as our knowledge base. The existing way to solve the second problem is to encode general knowledge in vector space so that the encoding results can be used to enhance the lexical or contextual representations of words BIBREF10 , BIBREF11 . However, this is an implicit way to utilize general knowledge, since in this way we can neither understand nor control the functioning of general knowledge. In this paper, we discard the existing implicit way and instead explore an explicit (i.e. understandable and controllable) way to utilize general knowledge.
WordNet is a lexical database of English, where words are organized into synsets according to their senses. A synset is a set of words expressing the same sense so that a word having multiple senses belongs to multiple synsets, with each synset corresponding to a sense. Synsets are further related to each other through semantic relations. According to the WordNet interface provided by NLTK BIBREF12 , there are totally sixteen types of semantic relations (e.g. hypernyms, hyponyms, holonyms, meronyms, attributes, etc.). Based on synset and semantic relation, we define a new concept: semantic relation chain. A semantic relation chain is a concatenated sequence of semantic relations, which links a synset to another synset. For example, the synset “keratin.n.01” is related to the synset “feather.n.01” through the semantic relation “substance holonym”, the synset “feather.n.01” is related to the synset “bird.n.01” through the semantic relation “part holonym”, and the synset “bird.n.01” is related to the synset “parrot.n.01” through the semantic relation “hyponym”, thus “substance holonym INLINEFORM0 part holonym INLINEFORM1 hyponym” is a semantic relation chain, which links the synset “keratin.n.01” to the synset “parrot.n.01”. We name each semantic relation in a semantic relation chain as a hop, therefore the above semantic relation chain is a 3-hop chain. By the way, each single semantic relation is equivalent to a 1-hop chain.

Answers:
Yes
Question: How do the authors examine whether a model is robust to noise or not?

Evidence:
raw_evidence: MRC Dataset. The MRC dataset used in this paper is SQuAD 1.1, which contains over INLINEFORM0 passage-question pairs and has been randomly partitioned into three parts: a training set ( INLINEFORM1 ), a development set ( INLINEFORM2 ), and a test set ( INLINEFORM3 ). Besides, we also use two of its adversarial sets, namely AddSent and AddOneSent BIBREF6 , to evaluate the robustness to noise of MRC models. The passages in the adversarial sets contain misleading sentences, which are aimed at distracting MRC models. Specifically, each passage in AddSent contains several sentences that are similar to the question but not contradictory to the answer, while each passage in AddOneSent contains a human-approved random sentence that may be unrelated to the passage.
We compare KAR with other MRC models in both performance and the robustness to noise. Specifically, we not only evaluate the performance of KAR on the development set and the test set, but also do this on the adversarial sets. As for the comparative objects, we only consider the single MRC models that rank in the top 20 on the SQuAD 1.1 leader board and have reported their performance on the adversarial sets. There are totally five such comparative objects, which can be considered as representatives of the state-of-the-art MRC models. As shown in Table TABREF12 , on the development set and the test set, the performance of KAR is on par with that of the state-of-the-art MRC models; on the adversarial sets, KAR outperforms the state-of-the-art MRC models by a large margin. That is to say, KAR is comparable in performance with the state-of-the-art MRC models, and significantly more robust to noise than them.
raw_evidence: MRC Dataset. The MRC dataset used in this paper is SQuAD 1.1, which contains over INLINEFORM0 passage-question pairs and has been randomly partitioned into three parts: a training set ( INLINEFORM1 ), a development set ( INLINEFORM2 ), and a test set ( INLINEFORM3 ). Besides, we also use two of its adversarial sets, namely AddSent and AddOneSent BIBREF6 , to evaluate the robustness to noise of MRC models. The passages in the adversarial sets contain misleading sentences, which are aimed at distracting MRC models. Specifically, each passage in AddSent contains several sentences that are similar to the question but not contradictory to the answer, while each passage in AddOneSent contains a human-approved random sentence that may be unrelated to the passage.

Answers:
By evaluating their model on adversarial sets containing misleading sentences
we also use two of its adversarial sets, namely AddSent and AddOneSent BIBREF6 , to evaluate the robustness to noise
Question: What type of model is KAR?

Evidence:
raw_evidence: As shown in Figure FIGREF7 , KAR is an end-to-end MRC model consisting of five layers:
Lexicon Embedding Layer. This layer maps the words to the lexicon embeddings. The lexicon embedding of each word is composed of its word embedding and character embedding. For each word, we use the pre-trained GloVe BIBREF14 word vector as its word embedding, and obtain its character embedding with a Convolutional Neural Network (CNN) BIBREF15 . For both the passage and the question, we pass the concatenation of the word embeddings and the character embeddings through a shared dense layer with ReLU activation, whose output dimensionality is INLINEFORM0 . Therefore we obtain the passage lexicon embeddings INLINEFORM1 and the question lexicon embeddings INLINEFORM2 .
Context Embedding Layer. This layer maps the lexicon embeddings to the context embeddings. For both the passage and the question, we process the lexicon embeddings (i.e. INLINEFORM0 for the passage and INLINEFORM1 for the question) with a shared bidirectional LSTM (BiLSTM) BIBREF16 , whose hidden state dimensionality is INLINEFORM2 . By concatenating the forward LSTM outputs and the backward LSTM outputs, we obtain the passage context embeddings INLINEFORM3 and the question context embeddings INLINEFORM4 .
Coarse Memory Layer. This layer maps the context embeddings to the coarse memories. First we use knowledge aided mutual attention (introduced later) to fuse INLINEFORM0 into INLINEFORM1 , the outputs of which are represented as INLINEFORM2 . Then we process INLINEFORM3 with a BiLSTM, whose hidden state dimensionality is INLINEFORM4 . By concatenating the forward LSTM outputs and the backward LSTM outputs, we obtain the coarse memories INLINEFORM5 , which are the question-aware passage representations.
Refined Memory Layer. This layer maps the coarse memories to the refined memories. First we use knowledge aided self attention (introduced later) to fuse INLINEFORM0 into themselves, the outputs of which are represented as INLINEFORM1 . Then we process INLINEFORM2 with a BiLSTM, whose hidden state dimensionality is INLINEFORM3 . By concatenating the forward LSTM outputs and the backward LSTM outputs, we obtain the refined memories INLINEFORM4 , which are the final passage representations.
Answer Span Prediction Layer. This layer predicts the answer start position and the answer end position based on the above layers. First we obtain the answer start position distribution INLINEFORM0 : INLINEFORM1 INLINEFORM2

Answers:
Lexicon Embedding Layer, Context Embedding Layer, Coarse Memory Layer, Refined Memory Layer, Answer Span Prediction Layer