qinchuanhui/UDA-QA huggingface dataset:
Question: What is the performance proposed model achieved on AlgoList benchmark?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Results of AlgoLisp dataset
Generating Lisp programs requires sensitivity to structural information because Lisp code can be regarded as tree-structured. Given a natural-language query, we need to generate code containing function calls with parameters. Each function call is a relational tuple, which has a function as the relation and parameters as arguments. We evaluate our model on the AlgoLisp dataset for this task and achieve state-of-the-art performance. The AlgoLisp dataset BIBREF17 is a program synthesis dataset. Each sample contains a problem description, a corresponding Lisp program tree, and 10 input-output testing pairs. We parse the program tree into a straight-line sequence of tuples (same style as in MathQA). AlgoLisp provides an execution script to run the generated program and has three evaluation metrics: the accuracy of passing all test cases (Acc), the accuracy of passing 50% of test cases (50p-Acc), and the accuracy of generating an exactly matching program (M-Acc). AlgoLisp has about 10% noisy data (details in the Appendix), so we report results both on the full test set and the cleaned test set (in which all noisy testing samples are removed). TP-N2F is compared with an LSTM seq2seq with attention model, the Seq2Tree model in BIBREF17, and a seq2seq model with a pre-trained tree decoder from the Tree2Tree autoencoder (SAPS) reported in BIBREF18. As shown in Table TABREF18, TP-N2F outperforms all existing models on both the full test set and the cleaned test set. Ablation experiments with TP2LSTM and LSTM2TP show that, for this task, the TP-N2F Decoder is more helpful than TP-N2F Encoder. This may be because lisp codes rely more heavily on structure representations.

Answers:
Full Testing Set Accuracy: 84.02
Cleaned Testing Set Accuracy: 93.48
Question: What is the performance proposed model achieved on MathQA?

Evidence:
raw_evidence: Given a natural-language math problem, we need to generate a sequence of operations (operators and corresponding arguments) from a set of operators and arguments to solve the given problem. Each operation is regarded as a relational tuple by viewing the operator as relation, e.g., $(add, n1, n2)$. We test TP-N2F for this task on the MathQA dataset BIBREF16. The MathQA dataset consists of about 37k math word problems, each with a corresponding list of multi-choice options and the corresponding operation sequence. In this task, TP-N2F is deployed to generate the operation sequence given the question. The generated operations are executed with the execution script from BIBREF16 to select a multi-choice answer. As there are about 30% noisy data (where the execution script returns the wrong answer when given the ground-truth program; see Sec. SECREF20 of the Appendix), we report both execution accuracy (of the final multi-choice answer after running the execution engine) and operation sequence accuracy (where the generated operation sequence must match the ground truth sequence exactly). TP-N2F is compared to a baseline provided by the seq2prog model in BIBREF16, an LSTM-based seq2seq model with attention. Our model outperforms both the original seq2prog, designated SEQ2PROG-orig, and the best reimplemented seq2prog after an extensive hyperparameter search, designated SEQ2PROG-best. Table TABREF16 presents the results. To verify the importance of the TP-N2F encoder and decoder, we conducted experiments to replace either the encoder with a standard LSTM (denoted LSTM2TP) or the decoder with a standard attentional LSTM (denoted TP2LSTM). We observe that both the TPR components of TP-N2F are important for achieving the observed performance gain relative to the baseline.
FLOAT SELECTED: Table 1: Results on MathQA dataset testing set

Answers:
Operation accuracy: 71.89
Execution accuracy: 55.95
Question: What previous methods is the proposed method compared against?

Evidence:
raw_evidence: FLOAT SELECTED: Table 4: Comparison results with the previous approaches and our approaches on SwDA dataset.
FLOAT SELECTED: Table 5: Experiment results about the hyperparameter W and P on SwDA dataset and online prediction result. W,P indicate the size of sliding window and context padding length during training and testing.

Answers:
BLSTM+Attention+BLSTM
Hierarchical BLSTM-CRF
CRF-ASN
Hierarchical CNN (window 4)
mLSTM-RNN
DRLM-Conditional
LSTM-Softmax
RCNN
CNN
CRF
LSTM
BERT
Question: What domains are present in the data?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: The total number of intents (services in parentheses) and dialogues for each domain across train1, dev2 and test3 sets. Superscript indicates the datasets in which dialogues from the domain are present. Multi-domain dialogues contribute to counts of each domain. The domain Services includes salons, dentists, doctors, etc.

Answers:
Alarm, Banks, Buses, Calendar, Events, Flights, Homes, Hotels, Media, Messaging, Movies, Music, Payment, Rental Cars, Restaurants, Ride Sharing, Services, Train, Travel, Weather
Question: How many texts/datapoints are in the SemEval, TASS and SENTIPOLC datasets?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Datasets details from each competition tested in this work

Answers:
Total number of annotated data:
Semeval'15: 10712
Semeval'16: 28632
Tass'15: 69000
Sentipol'14: 6428
Question: In which languages did the approach outperform the reported results?

Evidence:
raw_evidence: In BIBREF3 , BIBREF2 , the authors study the effect of translation in sentiment classifiers; they found better to use native Arabic speakers as annotators than fine-tuned translators plus fine-tuned English sentiment classifiers. In BIBREF21 , the idea is to measure the effect of the agreement among annotators on the production of a sentiment-analysis corpus. Both, on the technical side, both papers use fine tuned classifiers plus a variety of pre-processing techniques to prove their claims. Table TABREF24 supports the idea of choosing B4MSA as a bootstrapping sentiment classifier because, in the overall, B4MSA reaches superior performances regardless of the language. Our approach achieves those performance's levels since it optimizes a set of parameters carefully selected to work on a variety of languages and being robust to informal writing. The latter problem is not properly tackled in many cases.
FLOAT SELECTED: Table 5: Performance on multilingual sentiment analysis (not challenges). B4MSA was restricted to use only the multilingual set of parameters.

Answers:
Arabic, German, Portuguese, Russian, Swedish
Question: How close do clusters match to ground truth tone categories?

Evidence:
raw_evidence: To test this hypothesis, we evaluate the model on only the first syllable of every word, which eliminates carry-over and declination effects (Table TABREF14). In both Mandarin and Cantonese, the clustering is more accurate when using only the first syllables, compared to using all of the syllables.
FLOAT SELECTED: Table 3. Normalized mutual information (NMI) between cluster assignments and ground truth tones, considering only the first syllable of each word, or all syllables.

Answers:
NMI between cluster assignments and ground truth tones for all sylables is:
Mandarin: 0.641
Cantonese: 0.464
Question: what are the evaluation metrics?

Evidence:
raw_evidence: FLOAT SELECTED: Table 5: CoNLL 2003 English results.

Answers:
Precision, Recall, F1
Question: which datasets were used in evaluation?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Datasets used for training, development and evaluation. MUC7: only three classes (LOC, ORG, PER) of the formal run are used for out-of-domain evaluation. As there are not standard partitions of SONAR-1 and Ancora 2.0, the full corpus was used for training and later evaluated in-out-of-domain settings.

Answers:
CoNLL 2003, GermEval 2014, CoNLL 2002, Egunkaria, MUC7, Wikigold, MEANTIME, SONAR-1, Ancora 2.0
Question: How well does their system perform on the development set of SRE?

Evidence:
raw_evidence: In this section we present the results obtained on the protocol provided by NIST on the development set which is supposed to mirror that of evaluation set. The results are shown in Table TABREF26 . The first part of the table indicates the result obtained by the primary system. As can be seen, the fusion of MFCC and PLP (a simple sum of both MFCC and PLP scores) resulted in a relative improvement of almost 10%, as compared to MFCC alone, in terms of both INLINEFORM0 and INLINEFORM1 . In order to quantify the contribution of the different system components we have defined different scenarios. In scenario A, we have analysed the effect of using LDA instead of NDA. As can be seen from the results, LDA outperforms NDA in the case of PLP, however, in fusion we can see that NDA resulted in better performance in terms of the primary metric. In scenario B, we analysed the effect of using the short-duration compensation technique proposed in Section SECREF7 . Results indicate superior performance using this technique. In scenario C, we investigated the effects of language normalization on the performance of the system. If we replace LN-LDA with simple LDA, we can see performance degradation in MFCC as well as fusion, however, PLP seems not to be adversely affected. The effect of using QMF is also investigated in scenario D. Finally in scenario E, we can see the major improvement obtained through the use of the domain adaptation technique explained in Section SECREF16 . For our secondary submission, we incorporated a disjoint portion of the labelled development set (10 out of 20 speakers) in either LN-LDA and in-domain PLDA training. We evaluated the system on almost 6k out of 24k trials from the other portion to avoid any over-fitting, particularly important for the domain adaptation technique. This resulted in a relative improvement of 11% compared to the primary system in terms of the primary metric. However, the results can be misleading, since the recording condition may be the same for all speakers in the development set.
FLOAT SELECTED: Table 2. Performance comparison of the Intelligent Voice speaker recognition system with various analysis on the development protocol of NIST SRE 2016.

Answers:
EER 16.04, Cmindet 0.6012, Cdet 0.6107
Question: What external sources are used?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Statistics of external data.
Neural network models for NLP benefit from pretraining of word/character embeddings, learning distributed sementic information from large raw texts for reducing sparsity. The three basic elements in our neural segmentor, namely characters, character bigrams and words, can all be pretrained over large unsegmented data. We pretrain the five-character window network in Figure FIGREF13 as an unit, learning the MLP parameter together with character and bigram embeddings. We consider four types of commonly explored external data to this end, all of which have been studied for statistical word segmentation, but not for neural network segmentors.
Raw Text. Although raw texts do not contain explicit word boundary information, statistics such as mutual information between consecutive characters can be useful features for guiding segmentation BIBREF11 . For neural segmentation, these distributional statistics can be implicitly learned by pretraining character embeddings. We therefore consider a more explicit clue for pretraining our character window network, namely punctuations BIBREF10 .
Automatically Segmented Text. Large texts automatically segmented by a baseline segmentor can be used for self-training BIBREF13 or deriving statistical features BIBREF12 . We adopt a simple strategy, taking automatically segmented text as silver data to pretrain the five-character window network. Given INLINEFORM0 , INLINEFORM1 is derived using the MLP in Figure FIGREF13 , and then used to classify the segmentation of INLINEFORM2 into B(begining)/M(middle)/E(end)/S(single character word) labels. DISPLAYFORM0
Heterogenous Training Data. Multiple segmentation corpora exist for Chinese, with different segmentation granularities. There has been investigation on leveraging two corpora under different annotation standards to improve statistical segmentation BIBREF16 . We try to utilize heterogenous treebanks by taking an external treebank as labeled data, training a B/M/E/S classifier for the character windows network. DISPLAYFORM0
POS Data. Previous research has shown that POS information is closely related to segmentation BIBREF14 , BIBREF15 . We verify the utility of POS information for our segmentor by pretraining a classifier that predicts the POS on each character, according to the character window representation INLINEFORM0 . In particular, given INLINEFORM1 , the POS of the word that INLINEFORM2 belongs to is used as the output. DISPLAYFORM0

Answers:
Raw data from Gigaword, Automatically segmented text from Gigaword, Heterogenous training data from People's Daily, POS data from People's Daily
Question: How much better peformance is achieved in human evaluation when model is trained considering proposed metric?

Evidence:
raw_evidence: The experimental results are summarized in Table 1. We can see that the proposed comparative evaluator correlates far better with human judgment than BLEU and perplexity. When compared with recently proposed parameterized metrics including adversarial evaluator and ADEM, our model consistently outperforms them by a large margin, which demonstrates that our comparison-based evaluation metric is able to evaluate sample quality more accurately. In addition, we find that evaluating generated samples by comparing it with a set of randomly selected samples or using sample-level skill rating performs almost equally well. This is not surprising as the employed skill rating is able to handle the inherent variance of players (i.e. NLG models). As this variance does not exist when we regard a sample as a model which always generates the same sample.
Results are shown in Table 2. We can see that the proposed comparative evaluator with skill rating significantly outperforms all compared baselines, including comparative evaluator with averaged sample-level scores. This demonstrates the effectiveness of the skill rating system for performing model-level comparison with pairwise sample-level evaluation. In addition, the poor correlation between conventional evaluation metrics including BLEU and perplexity demonstrates the necessity of better automated evaluation metrics in open domain NLG evaluation.
FLOAT SELECTED: Table 1: Sample-level correlation between metrics and human judgments, with p-values shown in brackets.
FLOAT SELECTED: Table 2: Model-level correlation between metrics and human judgments, with p-values shown in brackets.

Answers:
Pearson correlation to human judgement - proposed vs next best metric
Sample level comparison:
- Story generation: 0.387 vs 0.148
- Dialogue: 0.472 vs 0.341
Model level comparison:
- Story generation:  0.631 vs 0.302
- Dialogue: 0.783 vs 0.553
Question: How much transcribed data is available for for Ainu language?

Evidence:
raw_evidence: The corpus we have prepared for ASR in this study is composed of text and speech. Table 1 shows the number of episodes and the total speech duration for each speaker. Among the total of eight speakers, the data of the speakers KM and UT is from the Ainu Museum, and the rest is from Nibutani Ainu Culture Museum. All speakers are female. The length of the recording for a speaker varies depending on the circumstances at the recording times. A sample text and its English translation are shown in Table 2.
FLOAT SELECTED: Table 1: Speaker-wise details of the corpus

Answers:
Transcribed data is available for duration of 38h 54m 38s for 8 speakers.
Question: What baseline approaches do they compare against?

Evidence:
raw_evidence: We chose the best system based on the dev set, and used that for submitting private test predictions on both FEVER and HotpotQA .
As can be seen in Table TABREF8, with the proposed hierarchical system design, the whole pipeline system achieves new start-of-the-art on HotpotQA with large-margin improvements on all the metrics. More specifically, the biggest improvement comes from the EM for the supporting fact which in turn leads to doubling of the joint EM on previous best results. The scores for answer predictions are also higher than all previous best results with $\sim $8 absolute points increase on EM and $\sim $9 absolute points on F1. All the improvements are consistent between test and dev set evaluation.
Similarly for FEVER, we showed F1 for evidence, the Label Accuracy, and the FEVER Score (same as benchmark evaluation) for models in Table TABREF9. Our system obtained substantially higher scores than all previously published results with a $\sim $4 and $\sim $3 points absolute improvement on Label Accuracy and FEVER Score. In particular, the system gains 74.62 on the evidence F1, 22 points greater that of the second system, demonstrating its ability on semantic retrieval.
FLOAT SELECTED: Table 1: Results of systems on HOTPOTQA.
FLOAT SELECTED: Table 2: Performance of systems on FEVER. “F1” indicates the sentence-level evidence F1 score. “LA” indicates Label Acc. without considering the evidence prediction. “FS”=FEVER Score (Thorne et al., 2018)

Answers:
HotspotQA: Yang, Ding, Muppet
Fever: Hanselowski, Yoneda, Nie
Question: Retrieval at what level performs better, sentence level or paragraph level?

Evidence:
raw_evidence: FLOAT SELECTED: Table 4: Ablation over the paragraph-level and sentence-level neural retrieval sub-modules on FEVER. “LA”=Label Accuracy; “FS”=FEVER Score; “Orcl.” is the oracle upperbound of FEVER Score assuming all downstream modules are perfect. “L-F1 (S/R/N)” means the classification f1 scores on the three verification labels: SUPPORT, REFUTE, and NOT ENOUGH INFO.
FLOAT SELECTED: Table 3: Ablation over the paragraph-level and sentence-level neural retrieval sub-modules on HOTPOTQA.
Table TABREF13 and TABREF14 shows the ablation results for the two neural retrieval modules at both paragraph and sentence level on HotpotQA and FEVER. To begin with, we can see that removing paragraph-level retrieval module significantly reduces the precision for sentence-level retrieval and the corresponding F1 on both tasks. More importantly, this loss of retrieval precision also led to substantial decreases for all the downstream scores on both QA and verification task in spite of their higher upper-bound and recall scores. This indicates that the negative effects on downstream module induced by the omission of paragraph-level retrieval can not be amended by the sentence-level retrieval module, and focusing semantic retrieval merely on improving the recall or the upper-bound of final score will risk jeopardizing the performance of the overall system.
Next, the removal of sentence-level retrieval module induces a $\sim $2 point drop on EM and F1 score in the QA task, and a $\sim $15 point drop on FEVER Score in the verification task. This suggests that rather than just enhance explainability for QA, the sentence-level retrieval module can also help pinpoint relevant information and reduce the noise in the evidence that might otherwise distract the downstream comprehension module. Another interesting finding is that without sentence-level retrieval module, the QA module suffered much less than the verification module; conversely, the removal of paragraph-level retrieval neural induces a 11 point drop on answer EM comparing to a $\sim $9 point drop on Label Accuracy in the verification task. This seems to indicate that the downstream QA module relies more on the upstream paragraph-level retrieval whereas the verification module relies more on the upstream sentence-level retrieval. Finally, we also evaluate the F1 score on FEVER for each classification label and we observe a significant drop of F1 on Not Enough Info category without retrieval module, meaning that semantic retrieval is vital for the downstream verification module's discriminative ability on Not Enough Info label.

Answers:
This seems to indicate that the downstream QA module relies more on the upstream paragraph-level retrieval whereas the verification module relies more on the upstream sentence-level retrieval.
Question: What was their F1 score on the Bengali NER corpus?

Evidence:
raw_evidence: FLOAT SELECTED: Table 5: Bengali manual annotation results. Our methods improve on state of the art scores by over 5 points F1 given a relatively small amount of noisy and incomplete annotations from non-speakers.

Answers:
52.0%
Question: Where did they get training data?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Precision (P), Recall (R), and Relative Coverage (RC) results on ConciergeQA.
FLOAT SELECTED: Table 4: Precision (P), Recall (R), and Relative Coverage (RC) results on AmazonQA dataset.
FLOAT SELECTED: Table 1: Various types of training instances.

Answers:
AmazonQA and ConciergeQA datasets
Question: Which datasets did they experiment on?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Various types of training instances.

Answers:
ConciergeQA and AmazonQA
Question: Do sluice networks outperform non-transfer learning approaches?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Accuracy scores on in-domain and out-of-domain test sets for chunking (main task) with POS tagging as auxiliary task for different target domains for baselines and Sluice networks. Out-of-domain results for each target domain are averages across the 6 remaining source domains. Average error reduction over single-task performance is 12.8% for in-domain; 8.9% for out-of-domain. In-domain error reduction over hard parameter sharing is 14.8%.

Answers:
Yes
Question: what state of the accuracy did they obtain?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Accuracies on the Stanford Sentiment Treebank 5-class classification task; except for the MVN, all results are drawn from (Lei et al., 2015).

Answers:
51.5
Question: How is the proficiency score calculated?

Evidence:
raw_evidence: Tables and report some statistics extracted from both the written and spoken data collected so far in all the campaigns. Each written or spoken item received a total score by human experts, computed by summing up the scores related to 6 indicators in 2017/2018 (from 3 to 6 in the 2016 campaign, according to the proficiency levels and the type of test). Each indicator can assume a value 0, 1, 2, corresponding to bad, medium, good, respectively.
The list of the indicators used by the experts to score written sentences and spoken utterances in the evaluations, grouped by similarity, is reported in Table . Since every utterance was scored by only one expert, it was not possible to evaluate any kind of agreement among experts. For future evaluations, more experts are expected to provide independent scoring on same data sets, so this kind of evaluation will be possible.
FLOAT SELECTED: Table 4: List of the indicators used by human experts to evaluate specific linguistic competences.

Answers:
They used 6 indicators for proficiency (same for written and spoken) each marked by bad, medium or good by one expert.
Question: What proficiency indicators are used to the score the utterances?

Evidence:
raw_evidence: FLOAT SELECTED: Table 4: List of the indicators used by human experts to evaluate specific linguistic competences.
Tables and report some statistics extracted from both the written and spoken data collected so far in all the campaigns. Each written or spoken item received a total score by human experts, computed by summing up the scores related to 6 indicators in 2017/2018 (from 3 to 6 in the 2016 campaign, according to the proficiency levels and the type of test). Each indicator can assume a value 0, 1, 2, corresponding to bad, medium, good, respectively.

Answers:
6 indicators:
- lexical richness
- pronunciation and fluency
- syntactical correctness
- fulfillment of delivery
- coherence and cohesion
- communicative, descriptive, narrative skills