qinchuanhui/UDA-QA huggingface dataset:
Question: How does their perturbation algorihm work?

Evidence:
raw_evidence: In this section, we analyze the Hamming distance between the projections of the sentences from the enwik9 dataset and the corresponding projections of the same sentences after applying character level perturbations. We experiment with three types of character level perturbation BIBREF11 and two types of word level perturbation operations.
Perturbation Study ::: Character Level Perturbation Operations
insert(word, n) : We randomly choose n characters from the character vocabulary and insert them at random locations into the input word. We however retain the first and last characters of the word as is. Ex. transformation: $sample \rightarrow samnple$.
swap(word, n): We randomly swap the location of two characters in the word n times. As with the insert operation, we retain the first and last characters of the word as is and only apply the swap operation to the remaining characters. Ex. transformation: $sample \rightarrow sapmle$.
duplicate(word, n): We randomly duplicate a character in the word by n times. Ex. transformation: $sample \rightarrow saample$.
Perturbation Study ::: Character Level Perturbation Operations ::: Word Level Perturbation Operations
drop(sentence, n): We randomly drop n words from the sentence. Ex. transformation: This is a big cat. $\rightarrow $ This is a cat.
duplicate(sentence, n): Similar to duplicate(word, n) above, we randomly duplicate a word in the sentence n times. Ex. transformation: This is a big cat. $\rightarrow $ This is a big big cat.
swap(sentence, n): Similar to swap(word, n), we randomly swap the location of two words in the sentence n times. Ex. transformation: This is a big cat. $\rightarrow $ This cat is big.

Answers:
same sentences after applying character level perturbations
Question: Which language is divided into six dialects in the task mentioned in the paper?

Evidence:
raw_evidence: The focus of the aforementioned language and dialect identification competitions was diatopic variation and thus the data made available in these competitions was synchronic contemporary corpora. In the 2019 edition of the workshop, for the first time, a task including historical languages was organized. The CLI shared task provided participants with a dataset containing languages and dialects written in cuneiform script: Sumerian and Akkadian. Akkadian is divided into six dialects in the dataset: Old Babylonian, Middle Babylonian peripheral, Standard Babylonian, Neo Babylonian, Late Babylonian, and Neo Assyrian BIBREF14 .

Answers:
Akkadian.
Question: What is one of the first writing systems in the world?

Evidence:
raw_evidence: As evidenced in Section "Related Work" , the focus of most of these studies is the identification of languages and dialects using contemporary data. A few exceptions include the work by trieschnigg2012exploration who applied language identification methods to historical varieties of Dutch and the work by CLIarxiv on languages written in cuneiform script: Sumerian and Akkadian. Cuneiform is an ancient writing system invented by the Sumerians for more than three millennia.

Answers:
Cuneiform
Question: How do they obtain distant supervision rules for predicting relations?

Evidence:
raw_evidence: Phase 2: In order to predict the relationship between an event and the creation time of its parent document, we assign a DocRelTime random variable to every Timex3 and Event mention. For Events, these values are provided by the training data, for Timex3s we have to compute class labels. Around 42% of Timex3 mentions are simple dates (“12/29/08", “October 16", etc.) and can be naively canonicalized to a universal timestamp. This is done using regular expressions to identify common date patterns and heuristics to deal with incomplete dates. The missing year in “October 16", for example, can be filled in using the nearest preceding date mention; if that isn't available we use the document creation year. These mentions are then assigned a class using the parent document's DocTime value and any revision timestamps. Other Timex3 mentions are more ambiguous so we use a distant supervision approach. Phrases like “currently" and “today's" tend to occur near Events that overlap the current document creation time, while “ago" or “ INLINEFORM0 -years" indicate past events. These dominant temporal associations can be learned from training data and then used to label Timex3s. Finally, we define a logistic regression rule to predict entity DocRelTime values as well as specify a linear skip-chain factor over Event mentions and their nearest Timex3 neighbor, encoding the baseline system heuristic directly as an inference rule.

Answers:
dominant temporal associations can be learned from training data
Question: Which structured prediction approach do they adopt for temporal entity extraction?

Evidence:
raw_evidence: We examine a deep-learning approach to sequence labeling using a vanilla recurrent neural network (RNN) with word embeddings, as well as a joint inference, structured prediction approach using Stanford's knowledge base construction framework DeepDive BIBREF1 . Our DeepDive application outperformed the RNN and scored similarly to 2015's best-in-class extraction systems, even though it only used a small set of context window and dictionary features. Extraction performance, however lagged this year's best system submission. For document creation time relations, we again use DeepDive. Our system examined a simple temporal distant supervision rule for labeling time expressions and linking them to nearby event mentions via inference rules. Overall system performance was better than this year's median submission, but again fell short of the best system.

Answers:
DeepDive BIBREF1
Question: Which evaluation metric has been measured?

Evidence:
raw_evidence: In order to evaluate the precision of the retrieved documents in each experiment, we used "TREC_Eval" tool [3]. TREC_Eval is a standard tool for evaluation of IR tasks and its name is a short form of Text REtrieval Conference (TREC) Evaluation tool. The Mean Average Precision (MAP) reported by TREC_Eval was 27.99% without query expansion and 37.10% with query expansion which shows more than 9 percent improvement.

Answers:
Mean Average Precision
Question: What is the WordNet counterpart for Persian?

Evidence:
raw_evidence: FarsNet [20] [21] is the first WordNet for Persian, developed by the NLP Lab at Shahid Beheshti University and it follows the same structure as the original WordNet. The first version of FarsNet contained more than 10,000 synsets while version 2.0 and 2.5 contained 20,000 synsets. Currently, FarsNet version 3 is under release and contains more than 40,000 synsets [7].

Answers:
FarsNet
Question: Do they consider relations other than binary relations?

Evidence:

Answers:
Yes
Question: Are the grammar clauses manually created?

Evidence:

Answers:
Yes
Question: Do they use an NER system in their pipeline?

Evidence:

Answers:
No
Question: What large corpus is used for experiments?

Evidence:
raw_evidence: The goal of this part is to highlight the differences present in the excitation when a given speaker produces different voice qualities. The De7 database used for this study was designed by Marc Schroeder as one of the first attempts of creating diphone databases for expressive speech synthesis BIBREF2. The database contains three voice qualities (modal, soft and loud) uttered by a German female speaker, with about 50 minutes of speech available for each voice quality. In Section SECREF1, the glottal flow estimation method and glottal flow parametrization used in this work are briefly presented. The harmonicity of speech is studied via the maximum voiced frequency in Section SECREF3. As an important perceptual charactersitic, spectral tilt is analyzed in Section SECREF4. Section SECREF6 compares the so-called eigenresiduals BIBREF3 of the different voice qualities. Finally Section SECREF8 quantifies the separability between the three voice qualities for the extracted excitation features.

Answers:
The De7 database
Question: Are any of the utterances ungrammatical?

Evidence:
raw_evidence: It is worth mentioning that the collected texts contain a large quantity of errors of several types: orthographic, syntactic, code-switched words (i.e. words not in the required language), jokes, etc. Hence, the original written sentences have been processed in order to produce “cleaner” versions, in order to make the data usable for some research purposes (e.g. to train language models, to extract features for proficiency assessment, ...).

Answers:
Yes
Question: What is the state-of-the art?

Evidence:
raw_evidence: Very recently, the success of deep neural networks in many natural language processing tasks ( BIBREF20 ) has inspired new work in abstractive summarization . BIBREF2 propose a neural attention model with a convolutional encoder to solve this task. BIBREF3 build a large dataset for Chinese text summarization and propose to feed all hidden states from the encoder into the decoder. More recently, BIBREF4 extended BIBREF2 's work with an RNN decoder, and BIBREF8 proposed an RNN encoder-decoder architecture for summarization. Both techniques are currently the state-of-the-art on the DUC competition. However, the encoders exploited in these methods lack the ability to encode each word condition on the whole text, as an RNN encodes a word into a hidden vector by taking into account only the words up to that time step. In contrast, in this work we propose a `Read-Again' encoder-decoder architecture, which enables the encoder to understand each input word after reading the whole sentence. Our encoder first reads the text, and the results from the first read help represent the text in the second pass over the source text. Our second contribution is a simple copy mechanism that allows us to significantly reduce the decoder vocabulary size resulting in much faster inference times. Furthermore our copy mechanism allows us to handle out-of-vocabulary words in a principled manner. Finally our experiments show state-of-the-art performance on the DUC competition.

Answers:
neural attention model with a convolutional encoder with an RNN decoder and RNN encoder-decoder
Question: How do they identify abbreviations?

Evidence:
raw_evidence: The overview of our approach is shown in Figure FIGREF6 . Within ICU notes (e.g., text example in top-left box in Figure 2), we first identify all abbreviations using regular expressions and then try to find all possible expansions of these abbreviations from domain-specific knowledge base as candidates. We train word embeddings using the clinical notes data with task-oriented resources such as Wikipedia articles of candidates and medical scientific papers and compute the semantic similarity between an abbreviation and its candidate expansions based on their embeddings (vector representations of words).

Answers:
identify all abbreviations using regular expressions
Question: What kind of model do they build to expand abbreviations?

Evidence:
raw_evidence: We use word2vec BIBREF0 to train the word embeddings. The dimension of embeddings is empirically set to 100.

Answers:
word2vec BIBREF0
Question: Do they use any knowledge base to expand abbreviations?

Evidence:
raw_evidence: The overview of our approach is shown in Figure FIGREF6 . Within ICU notes (e.g., text example in top-left box in Figure 2), we first identify all abbreviations using regular expressions and then try to find all possible expansions of these abbreviations from domain-specific knowledge base as candidates. We train word embeddings using the clinical notes data with task-oriented resources such as Wikipedia articles of candidates and medical scientific papers and compute the semantic similarity between an abbreviation and its candidate expansions based on their embeddings (vector representations of words).

Answers:
Yes
Question: In their used dataset, do they study how many abbreviations are ambiguous?

Evidence:

Answers:
No
Question: Which dataset do they use to build their model?

Evidence:
raw_evidence: The clinical notes we used for the experiment are provided by domain experts, consisting of 1,160 physician logs of Medical ICU admission requests at a tertiary care center affiliated to Mount Sanai. Prospectively collected over one year, these semi-structured logs contain free-text descriptions of patients' clinical presentations, medical history, and required critical care-level interventions. We identify 818 abbreviations and find 42,506 candidates using domain-specific knowledge (i.e., www.allacronym.com/_medical). The enriched corpus contains 42,506 Wikipedia articles, each of which corresponds to one candidate, 6 research papers and 2 critical care medicine textbooks, besides our raw ICU data.

Answers:
1,160 physician logs of Medical ICU admission requests, 42,506 Wikipedia articles, 6 research papers and 2 critical care medicine textbooks
Question: What is the domain of their collected corpus?

Evidence:
raw_evidence: We collect utterances from the $\mathbf {C}$hinese $\mathbf {A}$rtificial $\mathbf {I}$ntelligence $\mathbf {S}$peakers (CAIS), and annotate them with slot tags and intent labels. The training, validation and test sets are split by the distribution of intents, where detailed statistics are provided in the supplementary material. Since the utterances are collected from speaker systems in the real world, intent labels are partial to the PlayMusic option. We adopt the BIOES tagging scheme for slots instead of the BIO2 used in the ATIS, since previous studies have highlighted meaningful improvements with this scheme BIBREF30 in the sequence labeling field.

Answers:
speaker systems in the real world
Question: What is the source of the CAIS dataset?

Evidence:
raw_evidence: We collect utterances from the $\mathbf {C}$hinese $\mathbf {A}$rtificial $\mathbf {I}$ntelligence $\mathbf {S}$peakers (CAIS), and annotate them with slot tags and intent labels. The training, validation and test sets are split by the distribution of intents, where detailed statistics are provided in the supplementary material. Since the utterances are collected from speaker systems in the real world, intent labels are partial to the PlayMusic option. We adopt the BIOES tagging scheme for slots instead of the BIO2 used in the ATIS, since previous studies have highlighted meaningful improvements with this scheme BIBREF30 in the sequence labeling field.

Answers:
the $\mathbf {C}$hinese $\mathbf {A}$rtificial $\mathbf {I}$ntelligence $\mathbf {S}$peakers (CAIS)
Question: What were the baselines models?

Evidence:
raw_evidence: We conduct experiments on our self-collected CAIS to evaluate the generalizability in different language. We apply two baseline models for comparison, one is the popular BiLSTMs + CRF architecture BIBREF36 for sequence labeling task, and the other one is the more powerful sententce-state LSTM BIBREF21. The results listed in Table TABREF50 demonstrate the generalizability and effectiveness of our CM-Net when handling various domains and different languages.

Answers:
BiLSTMs + CRF architecture BIBREF36, sententce-state LSTM BIBREF21
Question: Are the document vectors that the authors introduce evaluated in any way other than the new way the authors propose?

Evidence:
raw_evidence: In this section we describe how to evaluate and compare the outcomes of algorithms which assign relevance scores to words (such as LRP or SA) through intrinsic validation. Furthermore, we propose a measure of model explanatory power based on an extrinsic validation procedure. The latter will be used to analyze and compare the relevance decompositions or explanations obtained with the neural network and the BoW/SVM classifier. Both types of evaluations will be carried out in section "Results" .

Answers:
Yes
Question: According to the authors, why does the CNN model exhibit a higher level of explainability?

Evidence:
raw_evidence: Note that the BoW/SVM model being a linear predictor relying directly on word frequency statistics, it lacks expressive power in comparison to the CNN model which additionally learns intermediate hidden layer representations and convolutional filters. Moreover the CNN model can take advantage of the semantic similarity encoded in the distributed word2vec representations, while for the BoW/SVM model all words are “equidistant” in the bag-of-words semantic space. As our experiments will show, these limitations lead the BoW/SVM model to sometimes identify spurious words as relevant for the classification task. In analogy to the semantic extraction proposed in section "Word Relevance and Vector-Based Document Representation" for the CNN model, we can build vectors $d$ representing documents by leveraging the word relevances obtained with the BoW/SVM model. To this end, we introduce a binary vector $\tilde{x} \in \mathbb {R}^{V} $ whose entries are equal to one when the corresponding word from the vocabulary is present in the document and zero otherwise (i.e. $\tilde{x}$ is a binary bag-of-words representation of the document). Thereafter, we build the document summary vector $d$ component-wise, so that $d$ is just a vector of word relevances:

Answers:
CNN model which additionally learns intermediate hidden layer representations and convolutional filters. Moreover the CNN model can take advantage of the semantic similarity encoded in the distributed word2vec representations
Question: Does the LRP method work in settings that contextualize the words with respect to one another?

Evidence:
raw_evidence: Layer-wise relevance propagation (LRP) BIBREF12 , BIBREF32 is a recently introduced technique for estimating which elements of a classifier input are important to achieve a certain classification decision. It can be applied to bag-of-words SVM classifiers as well as to layer-wise structured neural networks. For every input data point and possible target class, LRP delivers one scalar relevance value per input variable, hereby indicating whether the corresponding part of the input is contributing for or against a specific classifier decision, or if this input variable is rather uninvolved and irrelevant to the classification task at all.

Answers:
Yes
Question: How do they incorporate lexicon into the neural network?

Evidence:
raw_evidence: The words are represented as fixed length vectors INLINEFORM0 resulting from the concatenation of GloVe pre-trained embeddings and DepecheMood BIBREF19 lexicon representation. Since we cannot directly concatenate token-based embeddings (provided in GloVe) with the lemma#PoS-based representation available in DepecheMood, we proceeded to re-build the latter in token-based form, applying the exact same methodology albeit with two differences: we started from a larger dataset (51.9K news articles instead of 25.3K) and used a frequency cut-off, i.e. keeping only those tokens that appear at least 5 times in the corpus.

Answers:
concatenation of GloVe pre-trained embeddings and DepecheMood BIBREF19 lexicon representation, cannot directly concatenate,  re-build the latter in token-based form
Question: What is the source of their lexicon?

Evidence:
raw_evidence: Our contribution leverages pre-trained word embeddings (GloVe, trained on wikipedia+gigaword corpus), the DepecheMood affective lexicon, and convolutional neural networks.

Answers:
DepecheMood
Question: What embeddings do they use?

Evidence:
raw_evidence: Our contribution leverages pre-trained word embeddings (GloVe, trained on wikipedia+gigaword corpus), the DepecheMood affective lexicon, and convolutional neural networks.

Answers:
GloVe
Question: did they use other pretrained language models besides bert?

Evidence:
raw_evidence: Our Recurrent +ELMo model uses the language model from BIBREF9 to provide contextualized embeddings to the baseline model outlined above, as recommended by the authors.
Our OpenAI GPT model fine-tunes the 12 layer 768 dimensional uni-directional transformer from BIBREF27 , which has been pre-trained as a language model on the Books corpus BIBREF36 .

Answers:
Yes
Question: how was the dataset built?

Evidence:
raw_evidence: Questions are gathered from anonymized, aggregated queries to the Google search engine. Queries that are likely to be yes/no questions are heuristically identified: we found selecting queries where the first word is in a manually constructed set of indicator words and are of sufficient length, to be effective.
Questions are only kept if a Wikipedia page is returned as one of the first five results, in which case the question and Wikipedia page are given to a human annotator for further processing.
Annotators label question/article pairs in a three-step process. First, they decide if the question is good, meaning it is comprehensible, unambiguous, and requesting factual information. This judgment is made before the annotator sees the Wikipedia page. Next, for good questions, annotators find a passage within the document that contains enough information to answer the question. Annotators can mark questions as “not answerable" if the Wikipedia article does not contain the requested information. Finally, annotators mark whether the question's answer is “yes" or “no". Annotating data in this manner is quite expensive since annotators need to search entire Wikipedia documents for relevant evidence and read the text carefully.

Answers:
Questions are gathered from anonymized, aggregated queries to the Google search engine. Queries that are likely to be yes/no questions are heuristically identified: we found selecting queries where the first word is in a manually constructed set of indicator words and are of sufficient length, to be effective. Questions are only kept if a Wikipedia page is returned as one of the first five results, in which case the question and Wikipedia page are given to a human annotator for further processing. Annotators label question/article pairs in a three-step process. First, they decide if the question is good, meaning it is comprehensible, unambiguous, and requesting factual information. This judgment is made before the annotator sees the Wikipedia page. Next, for good questions, annotators find a passage within the document that contains enough information to answer the question. Annotators can mark questions as “not answerable" if the Wikipedia article does not contain the requested information. Finally, annotators mark whether the question's answer is “yes" or “no"
Question: what is the size of BoolQ dataset?

Evidence:
raw_evidence: We combine 13k questions gathered from this pipeline with an additional 3k questions with yes/no answers from the NQ training set to reach a total of 16k questions. We split these questions into a 3.2k dev set, 3.2k test set, and 9.4k train set, ensuring questions from NQ are always in the train set. “Yes” answers are slightly more common (62.31% in the train set). The queries are typically short (average length 8.9 tokens) with longer passages (average length 108 tokens).

Answers:
 16k questions
Question: what processing was done on the speeches before being parsed?

Evidence:
raw_evidence: To estimate speakers' position we use Wordscore BIBREF1 – a version of the Naive Bayes classifier that is deployed for text categorization problems BIBREF22 . In a similar application, BIBREF1 have already demonstrated that Wordscore can be effectively used to derive estimates of TDs policy positions. As in the example above, we pre-process documents by removing all numbers and interjections.

Answers:
Remove numbers and interjections