qinchuanhui/UDA-QA huggingface dataset:
Question: Which subtasks do they evaluate on?

Evidence:
raw_evidence: Semantic parsing aims to solve the problem of canonicalizing language and representing its meaning: given an input sentence, it aims to extract a semantic representation of that sentence. Abstract meaning representation BIBREF0 , or AMR for short, allows us to do that with the inclusion of most of the shallow-semantic natural language processing (NLP) tasks that are usually addressed separately, such as named entity recognition, semantic role labeling and co-reference resolution. AMR is partially motivated by the need to provide the NLP community with a single dataset that includes basic disambiguation information, instead of having to rely on different datasets for each disambiguation problem. The annotation process is straightforward, enabling the development of large datasets. Alternative semantic representations have been developed and studied, such as CCG BIBREF1 , BIBREF2 and UCCA BIBREF3 .

Answers:
 entity recognition, semantic role labeling and co-reference resolution
Question: Do they test their approach on large-resource tasks?

Evidence:
raw_evidence: We first describe our corpus collection. Table. TABREF3 lists all corpora we used in the experiments. There are 16 corpora from 10 languages. To increase the variety of corpus, we selected 4 English corpora and 4 Mandarin corpora in addition to the low resource language corpora. As the target of this experiment is low resource speech recognition, we only randomly select 100,000 utterances even if there are more in each corpus. All corpora are available in LDC, voxforge, openSLR or other public websites. Each corpus is manually assigned one domain based on its speech style. Specifically, the domain candidates are telephone, read and broadcast.

Answers:
Yes
Question: By how much do they, on average, outperform the baseline multilingual model on 16 low-resource tasks?

Evidence:
raw_evidence: To evaluate our sampling strategy, we compare it with the pretrained model and fine-tuned model on 16 different corpora. The results show that our approach outperforms those baselines on all corpora: it achieves 1.6% lower phone error rate on average. Additionally, we demonstrate that our corpus-level embeddings are able to capture the characteristics of each corpus, especially the language and domain information. The main contributions of this paper are as follows:

Answers:
1.6% lower phone error rate on average
Question: How do they compute corpus-level embeddings?

Evidence:
raw_evidence: Corpus Embedding
Suppose that INLINEFORM0 is the target low-resource corpus, we are interested in optimizing the acoustic model with a much larger training corpora set INLINEFORM1 where INLINEFORM2 is the number of corpora and INLINEFORM3 . Each corpus INLINEFORM4 is a collection of INLINEFORM5 pairs where INLINEFORM6 is the input features and INLINEFORM7 is its target.
Our purpose here is to compute the embedding INLINEFORM0 for each corpus INLINEFORM1 where INLINEFORM2 is expected to encode information about its corpus INLINEFORM3 . Those embeddings can be jointly trained with the standard multilingual model BIBREF4 . First, the embedding matrix INLINEFORM4 for all corpora is initialized, the INLINEFORM5 -th row of INLINEFORM6 is corresponding to the embedding INLINEFORM7 of the corpus INLINEFORM8 . Next, during the training phase, INLINEFORM9 can be used to bias the input feature INLINEFORM10 as follows. DISPLAYFORM0
where INLINEFORM0 is an utterance sampled randomly from INLINEFORM1 , INLINEFORM2 is its hidden features, INLINEFORM3 is the parameter of the acoustic model and Encoder is the stacked bidirectional LSTM as shown in Figure. FIGREF5 . Next, we apply the language specific softmax to compute logits INLINEFORM4 and optimize them with the CTC objective BIBREF29 . The embedding matrix INLINEFORM5 can be optimized together with the model during the training process.

Answers:
First, the embedding matrix INLINEFORM4 for all corpora is initialized, during the training phase, INLINEFORM9 can be used to bias the input feature, Next, we apply the language specific softmax to compute logits INLINEFORM4 and optimize them with the CTC objective
Question: Which dataset do they use?

Evidence:
raw_evidence: In this paper, we study verifiable robustness, i.e., providing a certificate that for a given network and test input, no attack or perturbation under the specification can change predictions, using the example of text classification tasks, Stanford Sentiment Treebank (SST) BIBREF15 and AG News BIBREF16. The specification against which we verify is that a text classification model should preserve its prediction under character (or synonym) substitutions in a character (or word) based model. We propose modeling these input perturbations as a simplex and then using Interval Bound Propagation (IBP) BIBREF17, BIBREF18, BIBREF19 to compute worst case bounds on specification satisfaction, as illustrated in Figure FIGREF1. Since these bounds can be computed efficiently, we can furthermore derive an auxiliary objective for models to become verifiable. The resulting classifiers are efficiently verifiable and improve robustness on adversarial examples, while maintaining comparable performance in terms of nominal test accuracy.

Answers:
Stanford Sentiment Treebank (SST) BIBREF15 and AG News BIBREF16
Question: Which competitive relational classification models do they test?

Evidence:
raw_evidence: In this section, we consider two kinds of relational classification tasks: (1) relation prediction and (2) relation extraction. Relation prediction aims at predicting the relationship between entities with a given set of triples as training data; while relation extraction aims at extracting the relationship between two entities in a sentence.
We hope to design a simple and clear experiment setup to conduct error analysis for relational prediction. Therefore, we consider a typical method TransE BIBREF3 as the subject as well as FB15K BIBREF3 as the dataset. TransE embeds entities and relations as vectors, and train these embeddings by minimizing DISPLAYFORM0
For relation extraction, we consider the supervised relation extraction setting and TACRED dataset BIBREF10 . As for the subject model, we use the best model on TACRED dataset — position-aware neural sequence model. This method first passes the sentence into an LSTM and then calculate an attention sum of the hidden states in the LSTM by taking positional features into account. This simple and effective method achieves the best in TACRED dataset.

Answers:
For relation prediction they test TransE and for relation extraction they test position aware neural sequence model
Question: Which knowledge bases do they use?

Evidence:
raw_evidence: We show the statistics of the dataset we use in tab:statistics, and the construction procedures will be introduced in this section.
In Wikidata BIBREF8 , facts can be described as (Head item/property, Property, Tail item/property). To construct a dataset suitable for our task, we only consider the facts whose head entity and tail entity are both items. We first choose the most common 202 relations and 120000 entities from Wikidata as our initial data. Considering that the facts containing the two most frequently appearing relations (P2860: cites, and P31: instance of) occupy half of the initial data, we drop the two relations to downsize the dataset and make the dataset more balanced. Finally, we keep the triples whose head and tail both come from the selected 120000 entities as well as its relation comes from the remaining 200 relations.
ReVerb BIBREF9 is a program that automatically identifies and extracts binary relationships from English sentences. We use the extractions from running ReVerb on Wikipedia. We only keep the relations appear more than 10 times and their corresponding triples to construct our dataset.
FB15K BIBREF3 is a subset of freebase. TACRED BIBREF10 is a large supervised relation extraction dataset obtained via crowdsourcing. We directly use these two dataset, no extra processing steps were applied.

Answers:
Wikidata, ReVerb, FB15K, TACRED
Question: How do they gather human judgements for similarity between relations?

Evidence:
raw_evidence: Human Judgments
Following BIBREF11 , BIBREF12 and the vast amount of previous work on semantic similarity, we ask nine undergraduate subjects to assess the similarity of 360 pairs of relations from a subset of Wikidata BIBREF8 that are chosen to cover from high to low levels of similarity. In our experiment, subjects were asked to rate an integer similarity score from 0 (no similarity) to 4 (perfectly the same) for each pair. The inter-subject correlation, estimated by leaving-one-out method BIBREF13 , is r = INLINEFORM0 , standard deviation = INLINEFORM1 . This important reference value (marked in fig:correlation) could be seen as the highest expected performance for machines BIBREF12 .

Answers:
By assessing similarity of 360 pairs of relations from a subset of Wikidata using an integer similarity score from 0 to 4
Question: Which sampling method do they use to approximate similarity between the conditional probability distributions over entity pairs?

Evidence:
raw_evidence: Just as introduced in sec:introduction, it is intractable to compute similarity exactly, as involving INLINEFORM0 computation. Hence, we consider the monte-carlo approximation: DISPLAYFORM0
where INLINEFORM0 is a list of entity pairs sampled from INLINEFORM1 . We use sequential sampling to gain INLINEFORM6 , which means we first sample INLINEFORM7 given INLINEFORM8 from INLINEFORM9 , and then sample INLINEFORM10 given INLINEFORM11 and INLINEFORM12 from INLINEFORM13 .

Answers:
monte-carlo, sequential sampling
Question: What text classification task is considered?

Evidence:
raw_evidence: We trained word embeddings and a uni-directional GRU connected to a dense layer end-to-end for text classification on a set of scraped tweets using cross-entropy as the loss function. End-to-end training was selected to impose as few heuristic constraints on the system as possible. Each tweet was tokenized using NLTK TweetTokenizer and classified as one of 10 potential accounts from which it may have originated. The accounts were chosen based on the distinct topics each is known to typically tweet about. Tokens that occurred fewer than 5 times were disregarded in the model. The model was trained on 22106 tweets over 10 epochs, while 5526 were reserved for validation and testing sets (2763 each). The network demonstrated an insensitivity to the initialization of the hidden state, so, for algebraic considerations, INLINEFORM0 was chosen for hidden dimension of INLINEFORM1 . A graph of the network is shown in Fig.( FIGREF13 ).

Answers:
To classify a text as belonging to one of the ten possible classes.
Question: What novel class of recurrent-like networks is proposed?

Evidence:
raw_evidence: First, we propose a class of recurrent-like neural networks for NLP tasks that satisfy the differential equation DISPLAYFORM0
where DISPLAYFORM0
and where INLINEFORM0 and INLINEFORM1 are learned functions. INLINEFORM2 corresponds to traditional RNNs, with INLINEFORM3 . For INLINEFORM4 , this takes the form of RNN cells with either nested internal memories or dependencies that extend temporally beyond the immediately previous hidden state. In particular, using INLINEFORM5 for sentence generation is the topic of a manuscript presently in preparation.

Answers:
A network, whose learned functions satisfy a certain equation. The  network contains RNN cells with either nested internal memories or dependencies that extend temporally beyond the immediately previous hidden state.
Question: Is there a formal proof that the RNNs form a representation of the group?

Evidence:
raw_evidence: Second, we propose embedding schemes that explicitly embed words as elements of a Lie group. In practice, these embedding schemes would involve representing words as constrained matrices, and optimizing the elements, subject to the constraints, according to a loss function constructed from invariants of the matrices, and then applying the matrix log to obtain Lie vectors. A prototypical implementation, in which the words are assumed to be in the fundamental representation of the special orthogonal group, INLINEFORM0 , and are conditioned on losses sensitive to the relative actions of words, is the subject of another manuscript presently in preparation.

Answers:
No
Question: How much bigger is Switchboard-2000 than Switchboard-300 database?

Evidence:
raw_evidence: This study focuses on Switchboard-300, a standard 300-hour English conversational speech recognition task. Our acoustic and text data preparation follows the Kaldi BIBREF29 s5c recipe. Our attention based seq2seq model is similar to BIBREF30, BIBREF31 and follows the structure of BIBREF32.
As a contrast to our best results on Switchboard-300, we also train a seq2seq model on the 2000-hour Switchboard+Fisher data. This model consists of 10 encoder layers, and is trained for only 50 epochs. Our overall results on the Hub5'00 and other evaluation sets are summarized in Table TABREF14. The results in Fig. FIGREF12 and Table TABREF14 show that adding more training data greatly improves the system, by around 30% relative in some cases. For comparison with others, the 2000-hour system reaches 8.7% and 7.4% WER on rt02 and rt04. We observe that the regularization techniques, which are extremely important on the 300h setup, are still beneficial but have a significantly smaller effect.

Answers:
Switchboard-2000 contains 1700 more hours of speech data.
Question: How big is Switchboard-300 database?

Evidence:
raw_evidence: This study focuses on Switchboard-300, a standard 300-hour English conversational speech recognition task. Our acoustic and text data preparation follows the Kaldi BIBREF29 s5c recipe. Our attention based seq2seq model is similar to BIBREF30, BIBREF31 and follows the structure of BIBREF32.

Answers:
300-hour English conversational speech
Question: What crowdsourcing platform is used for data collection and data validation?

Evidence:
raw_evidence: The data presented in this paper was collected and validated via Mozilla's Common Voice initiative. Using either the Common Voice website or iPhone app, contributors record their voice by reading sentences displayed on the screen (see Figure (FIGREF5)). The recordings are later verified by other contributors using a simple voting system. Shown in Figure (FIGREF6), this validation interface has contributors mark $<$audio,transcript$>$ pairs as being either correct (up-vote) or incorrect (down-vote).

Answers:
the Common Voice website,  iPhone app
Question: How is validation of the data performed?

Evidence:
raw_evidence: A maximum of three contributors will listen to any audio clip. If an $<$audio,transcript$>$ pair first receives two up-votes, then the clip is marked as valid. If instead the clip first receives two down-votes, then it is marked as invalid. A contributor may switch between recording and validation as they wish.
Only clips marked as valid are included in the official training, development, and testing sets for each language. Clips which did not recieve enough votes to be validated or invalidated by the time of release are released as “other”. The train, test, and development sets are bucketed such that any given speaker may appear in only one. This ensures that contributors seen at train time are not seen at test time, which would skew results. Additionally, repetitions of text sentences are removed from the train, test, and development sets of the corpus.

Answers:
A maximum of three contributors will listen to any audio clip. If an $<$audio,transcript$>$ pair first receives two up-votes, then the clip is marked as valid. If instead the clip first receives two down-votes, then it is marked as invalid.
Question: What domains are detected in this paper?

Evidence:
raw_evidence: Experimental Setup

Answers:
Answer with content missing: (Experimental setup not properly rendered) In our experiments we used seven target domains: “Business and Commerce” (BUS), “Government and Politics” (GOV), “Physical and Mental Health” (HEA), “Law and Order” (LAW),
“Lifestyle” (LIF), “Military” (MIL), and “General Purpose” (GEN). Exceptionally, GEN does
not have a natural root category.
Question: Why do they think this task is hard?  What is the baseline performance?

Evidence:
raw_evidence: Although not done in this work, an alternative (but also natural) way to address the task is as a special case of language modelling, where the output vocabulary is restricted to the size of the `action' vocabulary. Also, note that the performance for this task is not expected to achieve a perfect accuracy, as there may be situations where more than one action is reasonable, and also because writers tell a story playing with elements such as surprise or uncertainty.

Answers:
1. there may be situations where more than one action is reasonable, and also because writers tell a story playing with elements such as surprise or uncertainty.
2. Macro F1 = 14.6 (MLR, length 96 snippet)
Weighted F1 = 31.1 (LSTM, length 128 snippet)
Question: Do they literally just treat this as "predict the next spell that appears in the text"?

Evidence:
raw_evidence: This paper explores instead a new task: action prediction from natural language descriptions of scenes. The challenge is addressed as follows: given a natural language input sequence describing the scene, such as a piece of a story coming from a transcript, the goal is to infer which action is most likely to happen next.

Answers:
Yes
Question: Do they study frequent user responses to help automate modelling of those?

Evidence:
raw_evidence: In order to identify frequent user intents, one can use existing clustering algorithms to group together all the utterances from the users. Here each cluster would correspond to a new intent and each utterance in the cluster would correspond to an example for the intent. Similarly the agents utterances can be clustered to identify system responses. However, we argue that rather than treating user utterances and agents responses in an isolated manner, there is merit in jointly clustering them. There is adjacency information of these utterances that can be utilized to identify better user intents and system responses. As an example, consider agent utterances A.2 in box A and A.2 in box C in Figure FIGREF5 (a). The utterances “Which operating system do you use?" and “What OS is installed in your machine" have no syntactic similarity and therefore may not be grouped together. However the fact that these utterances are adjacent to the similar user utterances “I am unable to start notes email client" and “Unable to start my email client" provides some evidence that the agent utterances might be similar. Similarly the user utterances “My system keeps getting rebooted" and “Machine is booting time and again" ( box B and D in Figure FIGREF5 (a))- that are syntactically not similar - could be grouped together since the adjacent agent utterances, “Is your machine heating up?" and “Is the machine heating?" are similar.

Answers:
Yes
Question: Do they use the same distance metric for both the SimCluster and K-means algorithm?

Evidence:
raw_evidence: We used the gaussian index from which an utterance pair was generated as the ground truth label, which served to provide ground truth clusters for computation of the above evaluation metrics. Table TABREF15 shows a comparison of the results on SimCluster versus K-means algorithm. Here our SimCluster algorithm improves the F1-scores from 0.412 and 0.417 in the two domains to 0.442 and 0.441. The ARI scores also improve from 0.176 and 0.180 to 0.203 and 0.204.

Answers:
Yes
Question: How do they generate the synthetic dataset?

Evidence:
raw_evidence: We performed experiments on synthetically generated dataset since it gives us a better control over the distribution of the data. Specifically we compared the gains obtained using our approach versus the variance of the distribution. We created dataset from the following generative process. [H] Generative Process [1] Generate data
Pick k points INLINEFORM0 as domain -1 means and a corresponding set of k points INLINEFORM1 as domain-2 means, and covariance matrices INLINEFORM2
iter INLINEFORM0 upto num INLINEFORM1 samples Sample class INLINEFORM2 Sample INLINEFORM3 Sample INLINEFORM4 Add q and a so sampled to the list of q,a pairs We generated the dataset from the above sampling process with means selected on a 2 dimensional grid of size INLINEFORM5 with variance set as INLINEFORM6 in each dimension.10000 sample points were generated. The parameter INLINEFORM7 of the above algorithm was set to 0.5 and k was set to 9 (since the points could be generated from one of the 9 gaussians with centroids on a INLINEFORM8 grid).

Answers:
using generative process
Question: how are multiple answers from multiple reformulated questions aggregated?

Evidence:
raw_evidence: During training, we have access to the reward for the answer returned for each reformulation $q_i$ . However, at test time we must predict the best answer $a^*$ . The selection model selects the best answer from the set $\lbrace a_i\rbrace _{i=1}^N$ observed during the interaction by predicting the difference of the F1 score to the average F1 of all variants. We use pre-trained embeddings for the tokens of query, rewrite, and answer. For each, we add a 1-dimensional CNN followed by max-pooling. The three resulting vectors are then concatenated and passed through a feed-forward network which produces the output.
Unlike the reformulation policy, we train the answer with either beam search or sampling. We can produce many rewrites of a single question from our reformulation system. We issue each rewrite to the QA environment, yielding a set of (query, rewrite, answer) tuples from which we need to pick the best instance. We train another neural network to pick the best answer from the candidates. We frame the task as binary classification, distinguishing between above and below average performance. In training, we compute the F1 score of the answer for every instance. If the rewrite produces an answer with an F1 score greater than the average score of the other rewrites the instance is assigned a positive label. We ignore questions where all rewrites yield equally good/bad answers. We evaluated FFNNs, LSTMs, and CNNs and found that the performance of all systems was comparable. We choose a CNN which offers good computational efficiency and accuracy (cf. "Training" ).

Answers:
The selection model selects the best answer from the set $\lbrace a_i\rbrace _{i=1}^N$ observed during the interaction by predicting the difference of the F1 score to the average F1 of all variants.
Question: What debate websites did they look at?

Evidence:
raw_evidence: We start by crawling the content of a few notable debating websites: idebate.com, debatewise.org, procon.org. This yields INLINEFORM0 claims, INLINEFORM1 perspectives and INLINEFORM2 evidence paragraphs (for complete statistics, see Table TABREF46 in the Appendix). This data is significantly noisy and lacks the structure we would like. In the following steps we explain how we denoise it and augment it with additional data.

Answers:
idebate.com, debatewise.org, procon.org
Question: What crowdsourcing platform did they use?

Evidence:
raw_evidence: We use crowdsourcing to annotate different aspects of the dataset. We used Amazon Mechanical Turk (AMT) for our annotations, restricting the task to workers in five English-speaking countries (USA, UK, Canada, New Zealand, and Australia), more than 1000 finished HITs and at least a 95% acceptance rate. To ensure the diversity of responses, we do not require additional qualifications or demographic information from our annotators.

Answers:
Amazon Mechanical Turk (AMT)