qinchuanhui/UDA-QA huggingface dataset:
Question: What accuracy is achieved by the speech recognition system?

Evidence:
raw_evidence: Table , extracted from BIBREF0, reports WERs obtained on evaluation data sets with a strongly adapted ASR, demonstrating the difficulty of the related speech recognition task for both languages. Refer to BIBREF10 for comparisons with a different non-native children speech data set and to scientific literature BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16, BIBREF17, BIBREF18, BIBREF19 for detailed descriptions of children speech recognition and related issues. Important, although not exhaustive of the topic, references on non-native speech recognition can be found in BIBREF20, BIBREF21, BIBREF22, BIBREF23, BIBREF24, BIBREF25, BIBREF26, BIBREF27, BIBREF28, BIBREF29.
FLOAT SELECTED: Table 8: WER results on 2017 spoken test sets.

Answers:
Accuracy not available: WER results are reported 42.6 German, 35.9 English
Question: How is the speech recognition system evaluated?

Evidence:
raw_evidence: Table , extracted from BIBREF0, reports WERs obtained on evaluation data sets with a strongly adapted ASR, demonstrating the difficulty of the related speech recognition task for both languages. Refer to BIBREF10 for comparisons with a different non-native children speech data set and to scientific literature BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16, BIBREF17, BIBREF18, BIBREF19 for detailed descriptions of children speech recognition and related issues. Important, although not exhaustive of the topic, references on non-native speech recognition can be found in BIBREF20, BIBREF21, BIBREF22, BIBREF23, BIBREF24, BIBREF25, BIBREF26, BIBREF27, BIBREF28, BIBREF29.
FLOAT SELECTED: Table 8: WER results on 2017 spoken test sets.

Answers:
Speech recognition system is evaluated using WER metric.
Question: How many of the utterances are transcribed?

Evidence:
raw_evidence: Speakers were assigned either to training or evaluation sets, with proportions of $\frac{2}{3}$ and $\frac{1}{3}$, respectively; then training and evaluation lists were built, accordingly. Table reports statistics from the spoken data set. The id All identifies the whole data set, while Clean defines the subset in which sentences containing background voices, incomprehensible speech and word fragments were excluded.
FLOAT SELECTED: Table 7: Statistics from the spoken data sets (2017) used for ASR.

Answers:
Total number of transcribed utterances including Train and Test for both Eng and Ger language is 5562 (2188 cleaned)
Question: How many utterances are in the corpus?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Spoken data collected during different evaluation campaigns. Column “#Q” indicates the total number of different (written) questions presented to the pupils.
Table reports some statistics extracted from the acquired spoken data. Speech was recorded in classrooms, whose equipment depended on each school. In general, around 20 students took the test together, at the same time and in the same classrooms, so it is quite common that speech of mates or teachers often overlaps with the speech of the student speaking in her/his microphone. Also, the type of microphone depends on the equipment of the school. On average, the audio signal quality is nearly good, while the main problem is caused by a high percentage of extraneous speech. This is due to the fact that organisers decided to use a fixed duration - which depends on the question - for recording spoken utterances, so that all the recordings for a given question have the same length. However, while it is rare that a speaker has not enough time to answer, it is quite common that, especially after the end of the utterance, some other speech (e.g. comments, jokes with mates, indications from the teachers, etc.) is captured. In addition, background noise is often present due to several sources (doors, steps, keyboard typing, background voices, street noises if the windows are open, etc). Finally, it has to be pointed out that many answers are whispered and difficult to understand.

Answers:
Total number of utterances available is: 70607 (37344 ENG + 33263 GER)
Question: By how much does their model outperform both the state-of-the-art systems?

Evidence:
raw_evidence: Evaluation on DUC2004: DUC 2004 ( BIBREF15 ) is a commonly used benchmark on summarization task consisting of 500 news articles. Each article is paired with 4 different human-generated reference summaries, capped at 75 characters. This dataset is evaluation-only. Similar to BIBREF2 , we train our neural model on the Gigaword training set, and show the models' performances on DUC2004. Following the convention, we also use ROUGE limited-length recall as our evaluation metric, and set the capping length to 75 characters. We generate summaries with 15 words using beam-size of 10. As shown in Table TABREF35 , our method outperforms all previous methods on Rouge-1 and Rouge-L, and is comparable on Rouge-2. Furthermore, our model only uses 15k decoder vocabulary, while previous methods use 69k or 200k.
FLOAT SELECTED: Table 2: Rouge-N limited-length recall on DUC2004. Size denotes the size of decoder vocabulary in a model.

Answers:
w.r.t Rouge-1 their model outperforms by 0.98% and w.r.t Rouge-L their model outperforms by 0.45%
Question: What was the performance on the self-collected corpus?

Evidence:
raw_evidence: FLOAT SELECTED: Table 6: Results on our CAIS dataset, where “†” indicates our implementation of the S-LSTM.

Answers:
F1 scores of 86.16 on slot filling and 94.56 on intent detection
Question: What is the size of their dataset?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Dataset statistics.

Answers:
10,001 utterances
Question: What was their performance?

Evidence:
raw_evidence: In this section, we report the results obtained by our model according to challenge official evaluation metric, which is based cosine-similarity and described in BIBREF27 . Results are reported for three diverse configurations: (i) the full system; (ii) the system without using word embeddings (i.e. Glove and DepecheMood); and (iii) the system without using pre-processing. In Table TABREF17 we show model's performances on the challenge training data, in a 5-fold cross-validation setting.
Further, the final performances obtained with our approach on the challenge test set are reported in Table TABREF18 . Consistently with the cross-validation performances shown earlier, we observe the beneficial impact of word-representations and basic pre-processing.
FLOAT SELECTED: Table 1: Cross-validation results
FLOAT SELECTED: Table 2: Final results

Answers:
beneficial impact of word-representations and basic pre-processing
Question: What were their results?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Test Results on the NALCS (English) and LMS (Traditional Chinese) datasets.

Answers:
Best model achieved F-score 74.7 on NALCS and F-score of 70.0 on LMS on test set
Question: What is the prediction accuracy of the model?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Predicted Mean MPA results.
FLOAT SELECTED: Table 2: S&P 500 predicted results.

Answers:
mean prediction accuracy 0.99582651
S&P 500 Accuracy 0.99582651
Question: How much does this system outperform prior work?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Accuracy on the ListOps dataset. All models have 128 dimensions. Results for models with * are taken from Nangia and Bowman (2018).

Answers:
The system outperforms by 27.7% the LSTM model, 38.5% the RL-SPINN model and 41.6% the Gumbel Tree-LSTM
Question: What are the baseline systems that are compared against?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Accuracy on the ListOps dataset. All models have 128 dimensions. Results for models with * are taken from Nangia and Bowman (2018).

Answers:
The system is compared to baseline models: LSTM, RL-SPINN and Gumbel Tree-LSTM
Question: What systems are tested?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2. % EER performance of VoxCeleb-based systems on BULATS and Linguaskill test sets.
FLOAT SELECTED: Table 1. % EER performance of BULATS-trained baseline systems on BULATS and Linguaskill test sets.
Performance of the two baseline systems is shown in Table TABREF9 in terms of equal error rate (EER). The x-vector system yielded lower EERs on both BULATS and Linguaskill test sets.
In addition to the models trained on the BULATS data, it is also interesting to investigate the application of “out-of-the-box" models for standard speaker verification tasks to this non-native speaker verification task as there is limited amounts of non-native learner English data that is publicly available. In this paper, the Kaldi-released BIBREF19 VoxCeleb x-vector/PLDA system was used as imported models, which was trained on augmented VoxCeleb 1 BIBREF17 and VoxCeleb 2 BIBREF18. There are more than 7,000 speakers in the VoxCeleb dataset with more than 2,000 hours of audio data, making it the largest publicly available speaker recognition dataset. 30 dimensional mel-frequency cepstral coefficients (MFCCs) were used as input features and system configurations were the same as the BULATS x-vector/PLDA one. It can be seen from Table TABREF10 that these out-of-domain models gave worse performance than baseline systems trained on a far smaller amount of BULATS data due to domain mismatch. Thus, two kinds of in-domain adaptation strategies were explored to make use of the BULATS training set: PLDA adaptation and x-vector extractor fine-tuning. For PLDA adaptation, x-vectors of the BULATS training set were first extracted using the VoxCeleb-trained x-vector extractor, and then employed to adapt the VoxCeleb-trained PLDA model with their mean and variance. For x-vector extractor fine-tuning, with all other layers of the VoxCeleb-trained model kept still, the output layer was re-initialised using the BULATS training set with the number of targets adjusted accordingly, and then all layers were fine-tuned on the BULATS training set. Here the PLDA adaptation system is referred to as X1 and the extractor fine-tuning system is referred to as X2. Both adaptation approaches can yield good performance gains as can be seen from Table TABREF10. PLDA adaptation is a straightforward yet effective way, while the system with x-vector extractor fine-tuning gave slightly lower EERs on both BULATS and Linguaskill test sets by virtue of a relatively “in-domain" extractor prior to the PLDA back-end.

Answers:
BULATS i-vector/PLDA
BULATS x-vector/PLDA
VoxCeleb x-vector/PLDA
PLDA adaptation (X1)
 Extractor fine-tuning (X2)
Question: What are the 12 categories devised?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Statistics of the corpus - In the Course Level column, (i) represents Institute2 .

Answers:
Economics, Genocide, Geography, History, Human Rights, Kurdish, Kurdology, Philosophy, Physics, Theology, Sociology, Social Study
Question: How big are the datasets used?

Evidence:
raw_evidence: We evaluate our approaches on two public Chinese span-extraction machine reading comprehension datasets: CMRC 2018 (simplified Chinese) BIBREF8 and DRCD (traditional Chinese) BIBREF9. The statistics of the two datasets are listed in Table TABREF29.
Note that, since the test and challenge sets are preserved by CMRC 2018 official to ensure the integrity of the evaluation process, we submitted our best-performing systems to the organizers to get these scores. The resource in source language was chosen as SQuAD BIBREF4 training data. The settings of the proposed approaches are listed below in detail.
Translation: We use Google Neural Machine Translation (GNMT) system for translation. We evaluated GNMT system on NIST MT02/03/04/05/06/08 Chinese-English set and achieved an average BLEU score of 43.24, compared to previous best work (43.20) BIBREF17, yielding state-of-the-art performance.
FLOAT SELECTED: Table 1: Statistics of CMRC 2018 and DRCD.

Answers:
Evaluation datasets used:
CMRC 2018 - 18939 questions, 10 answers
DRCD - 33953 questions, 5 answers
NIST MT02/03/04/05/06/08 Chinese-English - Not specified

Source language train data:
SQuAD - Not specified
Question: How better is gCAS approach compared to other approaches?

Evidence:
raw_evidence: FLOAT SELECTED: Table 5: Entity F1 and Success F1 at dialogue level.

Answers:
For entity  F1 in the movie, taxi and restaurant domain it results in scores of 50.86, 64, and 60.35. For success, it results it outperforms in the movie and restaurant domain with scores of 77.95 and 71.52
Question: What were the sizes of the test sets?

Evidence:
raw_evidence: Each drug label is a collection of sections (e.g., DOSAGE & ADMINISTRATION, CONTRAINDICATIONS, and WARNINGS) where each section contains one or more sentences. Each sentence is annotated with a list of zero or more mentions and interactions. The training data released for this task contains 22 drug labels, referred to as Training-22, with gold standard annotations. Two test sets of 57 and 66 drug labels, referred to as Test Set 1 and 2 respectively, with gold standard annotations are used to evaluate participating systems. As Training-22 is a relatively small dataset, we additionally utilize an external dataset with 180 annotated drug labels dubbed NLM-180 BIBREF5 (more later). We provide summary statistics about these datasets in Table TABREF3 . Test Set 1 closely resembles Training-22 with respect to the sections that are annotated. However, Test Set 1 is more sparse in the sense that there are more sentences per drug label (144 vs. 27), with a smaller proportion of those sentences having gold annotations (23% vs. 51%). Test Set 2 is unique in that it contains annotations from only two sections, namely DRUG INTERACTIONS and CLINICAL PHARMACOLOGY, the latter of which is not represented in Training-22 (nor Test Set 1). Lastly, Training-22, Test Set 1, and Test Set 2 all vary with respect to the distribution of interaction types, with Training-22, Test Set 1, and Test Set 2 containing a higher proportion of PD, UN, and PK interactions respectively.
FLOAT SELECTED: Table 1: Characteristics of datasets

Answers:
Test set 1 contained 57 drug labels and 8208 sentences and test set 2 contained 66 drug labels and 4224 sentences
Question: Which datasets are used?

Evidence:
raw_evidence: Table TABREF7 shows the ABSA datasets from the restaurants domain for English, Spanish, French, Dutch, Russian and Turkish. From left to right each row displays the number of tokens, number of targets and the number of multiword targets for each training and test set. For English, it should be noted that the size of the 2015 set is less than half with respect to the 2014 dataset in terms of tokens, and only one third in number of targets. The French, Spanish and Dutch datasets are quite similar in terms of tokens although the number of targets in the Dutch dataset is comparatively smaller, possibly due to the tendency to construct compound terms in that language. The Russian dataset is the largest whereas the Turkish set is by far the smallest one.
FLOAT SELECTED: Table 1: ABSA SemEval 2014-2016 datasets for the restaurant domain. B-target indicates the number of opinion targets in each set; I-target refers to the number of multiword targets.
Apart from the manually annotated data, we also leveraged large, publicly available, unlabelled data to train the clusters: (i) Brown 1000 clusters and (ii) Clark and Word2vec clusters in the 100-800 range.
In order to induce clusters from the restaurant domain we used the Yelp Academic Dataset, from which three versions were created. First, the full dataset, containing 225M tokens. Second, a subset consisting of filtering out those categories that do not correspond directly to food related reviews BIBREF29 . Thus, out of the 720 categories contained in the Yelp Academic Dataset, we kept the reviews from 173 of them. This Yelp food dataset contained 117M tokens in 997,721 reviews. Finally, we removed two more categories (Hotels and Hotels & Travel) from the Yelp food dataset to create the Yelp food-hotels subset containing around 102M tokens. For the rest of the languages we used their corresponding Wikipedia dumps. The pre-processing and tokenization is performed with the IXA pipes tools BIBREF30 .

Answers:
ABSA SemEval 2014-2016 datasets
Yelp Academic Dataset
Wikipedia dumps
Question: How much does it minimally cost to fine-tune some model according to benchmarking framework?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Pretraining costs of baseline models. Hardware and pretraining time are collected from original papers, with which costs are estimated with current TPU price at $8 per hour with 4 core TPU v3 chips and V100 GPU at $3.06 per hour. DistilBERT model is trained upon a pretrained BERT model. Parameter numbers are estimated using the pretrained models implemented in the Transformers (https://github.com/huggingface/ transformers) library (Wolf et al., 2019), shown in million.

Answers:
$1,728
Question: What models are included in baseline benchmarking results?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Pretraining costs of baseline models. Hardware and pretraining time are collected from original papers, with which costs are estimated with current TPU price at $8 per hour with 4 core TPU v3 chips and V100 GPU at $3.06 per hour. DistilBERT model is trained upon a pretrained BERT model. Parameter numbers are estimated using the pretrained models implemented in the Transformers (https://github.com/huggingface/ transformers) library (Wolf et al., 2019), shown in million.

Answers:
BERT, XLNET RoBERTa, ALBERT, DistilBERT
Question: How much more accurate is the model than the baseline?

Evidence:
raw_evidence: NAGMWA is much better than the other methods except NAGM, since it generates answers whose conclusions and supplements as well as their combinations closely match the questions. Thus, conclusions and supplements in the answers are consistent with each other and avoid confusion made by several different conclusion-supplement answers assigned to a single non-factoid questions. Finally, NAGM is consistently superior to the conventional attentive encoder-decoders regardless of the metric. Its ROUGE-L and BLEU-4 scores are much higher than those of CLSTM. Thus, NAGM generates more fluent sentences by assessing the context from conclusion to supplement sentences in addition to the closeness of the question and sentences as well as that of the question and sentence combinations.
The experts asked questions, which were not included in our training datasets, to the AI system and rated the answers; one answer per question. The experts rated the answers as follows: (1) the content of the answer matched the question, and the grammar was okay; (2) the content was suitable, but the grammar was poor; (3) the content was not suitable, but the grammar was okay; (4) both the content and grammar were poor. Note that our evaluation followed the DUC-style strategy. Here, we mean “grammar” to cover grammaticality, non-redundancy, and referential clarity in the DUC strategy, whereas we mean the “content matched the questions” to refer to “focus” and “structure and coherence” in the DUC strategy. The evaluators were given more than a week to carefully evaluate the generated answers, so we consider that their judgments are reliable. Each expert evaluated 50 questions. We combined the scores of the experts by summing them. They did not know the identity of the system in the evaluation and reached their decisions independently.
These results indicate that the experts were much more satisfied with the outputs of NAGM than those of CLSTM. This is because, as can be seen in Table 7, NAGM generated longer and better question-related sentences than CLSTM did. NAGM generated grammatically good answers whose conclusion and supplement statements are well matched with the question and the supplement statement naturally follows the conclusion statement.
FLOAT SELECTED: Table 4: ROUGE-L/BLEU-4 for nfL6.
FLOAT SELECTED: Table 6: Human evaluation (nfL6).

Answers:
For the Oshiete-goo dataset, the NAGM model's ROUGE-L score is higher than the highest performing conventional model, Trans, by 0.021, and its BLEU-4 score is higher than the highest performing model CLSTM by 0.037.  For the nfL6 dataset, the NAGM model's ROUGE-L score is higher than the highest performing conventional model, CLSTM, by 0.028, and its BLEU-4 score is higher than the highest performing model CLSTM by 0.040. Human evaluation of the NAGM's generated outputs for the Oshiete-goo dataset had 47% ratings of (1), the highest rating, while CLSTM only received 21% ratings of (1). For the nfL6 dataset, the comparison of (1)'s was NAGM's 50% to CLSTM's 30%.
Question: What is new state-of-the-art performance on CoNLL-2009 dataset?

Evidence:
raw_evidence: Table TABREF46 shows that our Open model achieves more than 3 points of f1-score than the state-of-the-art result, and RelAwe with DepPath&RelPath achieves the best in both Closed and Open settings. Notice that our best Closed model can almost perform as well as the state-of-the-art model while the latter utilizes pre-trained word embeddings. Besides, performance gap between three models under Open setting is very small. It indicates that the representation ability of BERT is so powerful and may contains rich syntactic information. At last, the Gold result is much higher than the other models, indicating that there is still large space for improvement for this task.
FLOAT SELECTED: Table 7: SRL results on the Chinese test set. We choose the best settings for each configuration of our model.

Answers:
In closed setting 84.22 F1 and in open 87.35 F1.
Question: What are two strong baseline methods authors refer to?

Evidence:
raw_evidence: FLOAT SELECTED: Table 7: SRL results on the Chinese test set. We choose the best settings for each configuration of our model.

Answers:
Marcheggiani and Titov (2017) and Cai et al. (2018)
Question: What languages do they use?

Evidence:
raw_evidence: In this work, the experiments are conducted using the BABEL speech corpus collected from the IARPA babel program. The corpus is mainly composed of conversational telephone speech (CTS) but some scripted recordings and far field recordings are presented as well. Table TABREF14 presents the details of the languages used in this work for training and evaluation.
FLOAT SELECTED: Table 1: Details of the BABEL data used for performing the multilingual experiments

Answers:
Train languages are: Cantonese, Bengali, Pashto, Turkish, Vietnamese, Haitian, Tamil, Kurdish, Tokpisin and Georgian, while Assamese, Tagalog, Swahili, Lao are used as target languages.
Question: How large is the corpus?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Statistical summary of multilingual corpora across English, Italian, Polish, Portuguese and Spanish. We present number of users (Users), documents (Docs), and average tokens per document (Tokens) in the corpus, plus the label distribution (HS Ratio, percent of documents labeled positive for hate speech).

Answers:
It contains 106,350 documents
Question: How large is the dataset?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Statistical summary of multilingual corpora across English, Italian, Polish, Portuguese and Spanish. We present number of users (Users), documents (Docs), and average tokens per document (Tokens) in the corpus, plus the label distribution (HS Ratio, percent of documents labeled positive for hate speech).

Answers:
over 104k documents
Question: What was their perplexity score?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3. Language Modeling Results (in perplexity).
UTF8gbsn The pointer-generator significantly outperforms the Seq2Seq with attention model by 3.58 BLEU points on the test set as shown in Table TABREF8 . Our language modeling result is given in Table TABREF9 . Based on the empirical result, adding generated samples consistently improve the performance of all models with a moderate margin around 10% in perplexity. After all, our proposed method still slightly outperforms the heuristic from linguistic constraint. In addition, we get a crucial gain on performance by adding syntax representation of the sequences.

Answers:
Perplexity score 142.84 on dev and 138.91 on test