qinchuanhui/UDA-QA huggingface dataset:
Question: In which setting they achieve the state of the art?

Evidence:
raw_evidence: FLOAT SELECTED: Table 4. Evaluation results on VQA dataset [1]. ”-” indicates the results are not available, and the Ours+VGG(1) and Ours+VGG(2) are the results by using different thresholds. Note that our VGGNet is same as CoAtt+VGG.

Answers:
in open-ended task esp. for counting-type questions
Question: On what dataset is Aristo system trained?

Evidence:
raw_evidence: Several methods make use of the Aristo Corpus, comprising a large Web-crawled corpus ($5 \times 10^{10}$ tokens (280GB)) originally from the University of Waterloo, combined with targeted science content from Wikipedia, SimpleWikipedia, and several smaller online science texts (BID25).
The Regents exam questions are taken verbatim from the New York Regents Examination board, using the 4th Grade Science, 8th Grade Science, and 12th Grade Living Environment examinations. The questions are partitioned into train/dev/test by exam, i.e., each exam is either in train, dev, or test but not split up between them. The ARC dataset is a larger corpus of science questions drawn from public resources across the country, spanning grades 3 to 9, and also includes the Regents 4th and 8th questions (using the same train/dev/test split). Further details of the datasets are described in (BID13). The datasets are publicly available. Dataset sizes are shown in Table TABREF34. All but 39 of the 9366 questions are 4-way multiple choice, the remaining 39 ($<$0.5%) being 3- or 5-way. A random score over the entire dataset is 25.02%.
FLOAT SELECTED: Table 3: Dataset partition sizes (number of questions).

Answers:
Aristo Corpus
Regents 4th
Regents 8th
Regents `12th
ARC-Easy
ARC-challenge
Question: How many roles are proposed?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Most common syntactic patterns for each semantic role.

Answers:
12
Question: What language technologies have been introduced in the past?

Evidence:
raw_evidence: Often, many state-of-the-art tools cannot be applied to low-resource languages due to the lack of data. Table TABREF6 describes the various technologies and their presence concerning languages with different levels of resource availability and the ease of data collection. We can observe that for low resource languages, there is considerable difficulty in adopting these tools. Machine Translation can potentially be used as a fix to bridge the gap. Translation engines can help in translating documents from minority languages to majority languages. This allows the pool of data to be used in a number of NLP tasks like sentiment analysis and summarization. Doing so allows us to leverage the existing body of work in NLP done on resource-rich languages and subsequently apply it to the resource-poor languages, thereby foregoing any attempt to reinvent the wheel for these languages. This ensures a quicker and wider impact.BIBREF16 performs sentiment analysis on Chinese customer reviews by translating them to English. They observe that the quality of machine translation systems are sufficient for sentiment analysis to be performed on the automatically translated texts without a substantial trade-off in accuracy.
FLOAT SELECTED: Table 1: Enabling language technologies, their availability and quality ( ? ? ? - excellent quality technology, ?? - moderately good but usable, ? - rudimentary and not practically useful) for differently resourced languages, and their data/knowledge requirements (? ? ? - very high data/expertise, ?? - moderate, ? - nominal and easily procurable). This information is based on authors’ analysis and personal experience.

Answers:
- Font & Keyboard
- Speech-to-Text
- Text-to-Speech
- Text Prediction
- Spell Checker
- Grammar Checker
- Text Search
- Machine Translation
- Voice to Text Search
- Voice to Speech Search
Question: what were the baselines?

Evidence:
raw_evidence: Generally, the above work is summarized in Table TABREF2 . Considering motivation, our work is most closely related to the work of BIBREF14 Fitzgerald2015, which also tackles span and dependency SRL in a uniform fashion. The essential difference is that their model employs the syntactic features and takes pre-identified predicates as inputs, while our model puts syntax aside and jointly learns and predicts predicates and arguments.
FLOAT SELECTED: Table 1: A chronicle of related work for span and dependency SRL. SA represents syntax-aware system (no + indicates syntaxagnostic system) and ST indicates sequence tagging model. F1 is the result of single model on official test set.

Answers:
2008 Punyakanok et al. 
2009 Zhao et al. + ME 
2008 Toutanova et al. 
2010 Bjorkelund et al.  
2015 FitzGerald et al. 
2015 Zhou and Xu 
2016 Roth and Lapata 
2017 He et al. 
2017 Marcheggiani et al.
2017 Marcheggiani and Titov 
2018 Tan et al. 
2018 He et al. 
2018 Strubell et al. 
2018 Cai et al. 
2018 He et al. 
2018 Li et al.
Question: Which soft-selection approaches are evaluated?

Evidence:
raw_evidence: Previous attention-based methods can be categorized as soft-selection approaches since the attention weights scatter across the whole sentence and every word is taken into consideration with different weights. This usually results in attention distraction BIBREF7, i.e., attending on noisy or misleading words, or opinion words from other aspects. Take Figure FIGREF1 as an example, for the aspect place in the sentence “the food is usually good but it certainly is not a relaxing place to go”, we visualize the attention weights from the model ATAE-LSTM BIBREF2. As we can see, the words “good” and “but” are dominant in attention weights. However, “good” is used to describe the aspect food rather than place, “but” is not so related to place either. The true opinion snippet “certainly is not a relaxing place” receives low attention weights, leading to the wrong prediction towards the aspect place.
FLOAT SELECTED: Table 2: Experimental results (accuracy %) on all the datasets. Models in the first part are baseline methods. The results in the first part (except BERT-Original) are obtained from the prior work (Tay et al., 2018). Avg column presents macro-averaged results across all the datasets.

Answers:
LSTM and BERT
Question: How big is slot filing dataset?

Evidence:
raw_evidence: In our experiments, we use Onsei Intent Slot dataset. Table TABREF21 shows the statics of this dataset. We use the following hyper parameters in our model: We set the word embedding and POS embedding to 768 and 30 respectively; The pre-trained BERT BIBREF17 embedding are used to initialize word embeddings; The hidden dimension of the Bi-LSTM, GCN and feed forward networks are 200; the hyper parameters $\alpha $, $\beta $ and $\gamma $ are all set to 0.1; We use Adam optimizer with learning rate 0.003 to train the model. We use micro-averaged F1 score on all labels as the evaluation metric.
FLOAT SELECTED: Table 1: Label Statistics

Answers:
Dataset has 1737 train, 497 dev and 559 test sentences.
Question: How large is the dataset they generate?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Number of run-on (RO) and non-run-on (Non-RO) sentences in our datasets.

Answers:
4.756 million sentences
Question: Which existing models does this approach outperform?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3. Comparisons with the Existing Models in Terms of ROUGE Metrics

Answers:
RNN-context, SRB, CopyNet, RNN-distract, DRGD
Question: What languages are represented in the dataset?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2. Twitter corpus distribution by language label.
We begin by filtering the corpus to keep only those tweets where the user's self-declared language and the tweet's detected language correspond; that language becomes the tweet's correct language label. This operation cuts out roughly half the tweets, and leaves us with a corpus of about 900 million tweets in 54 different languages. Table TABREF6 shows the distribution of languages in that corpus. Unsurprisingly, it is a very imbalanced distribution of languages, with English and Japanese together accounting for 60% of all tweets. This is consistent with other studies and statistics of language use on Twitter, going as far back as 2013. It does however make it very difficult to use this corpus to train a LID system for other languages, especially for one of the dozens of seldom-used languages. This was our motivation for creating a balanced Twitter dataset.

Answers:
EN, JA, ES, AR, PT, KO, TH, FR, TR, RU, IT, DE, PL, NL, EL, SV, FA, VI, FI, CS, UK, HI, DA, HU, NO, RO, SR, LV, BG, UR, TA, MR, BN, IN, KN, ET, SL, GU, CY, ZH, CKB, IS, LT, ML, SI, IW, NE, KM, MY, TL, KA, BO
Question: How faster is training and decoding compared to former models?

Evidence:
raw_evidence: In order to verify the time complexity analysis of our model, we measured the running time and speed of BIAF, STACKPTR and our model on PTB training and development set using the projective algorithm. The comparison in Table TABREF24 shows that in terms of convergence time, our model is basically the same speed as BIAF, while STACKPTR is much slower. For decoding, our model is the fastest, followed by BIAF. STACKPTR is unexpectedly the slowest. This is because the time cost of attention scoring in decoding is not negligible when compared with the processing speed and actually even accounts for a significant portion of the runtime.
FLOAT SELECTED: Table 4: Training time and decoding speed. The experimental environment is on the same machine with Intel i9 9900k CPU and NVIDIA 1080Ti GPU.

Answers:
Proposed vs best baseline:
Decoding: 8541 vs 8532 tokens/sec
Training: 8h vs 8h
Question: What datasets are used to evaluate the model?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: The knowledge base completion (link prediction) results on WN18 and FB15k.

Answers:
WN18 and FB15k
Question: What metadata is included?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: An example of a claim instance. Entities are obtained via entity linking. Article and outlink texts, evidence search snippets and pages are not shown.

Answers:
besides claim, label and claim url, it also includes a claim ID, reason, category, speaker, checker, tags, claim entities, article title, publish data and claim date
Question: How much important is the visual grounding in the learning of the multilingual representations?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Crosslingual semantic similarity scores (Spearman’s ρ) across six subtasks for ImageVec (our method) and previous work. Coverage is in brackets. The last column indicates the combined score across all subtasks. Best scores on each subtask are bolded.

Answers:
performance is significantly degraded without pixel data
Question: By how much does their method outperform the multi-head attention model?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Experimental results.

Answers:
Their average improvement in Character Error Rate over the best MHA model was 0.33 percent points.
Question: How large is the corpus they use?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Experimental conditions.

Answers:
449050
Question: How many shared layers are in the system?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Optimal hyperparameters used for final training on the ADE and CoNLL04 datasets.

Answers:
1
Question: How many additional task-specific layers are introduced?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Optimal hyperparameters used for final training on the ADE and CoNLL04 datasets.

Answers:
2 for the ADE dataset and 3 for the CoNLL04 dataset
Question: How many layers of self-attention does the model have?

Evidence:
raw_evidence: FLOAT SELECTED: Table 6: Evaluation of effect of self-attention mechanism using DSTC2 dataset (Att: Attetnion mechanism; UT: Universal Transformers; ACT: Adaptive Computation Time; NH: Number of attention heads)

Answers:
1, 4, 8, 16, 32, 64
Question: what are the state of the art methods?

Evidence:
raw_evidence: We also evaluate our Joint-BiLSTM structure by comparing with several other state-of-the-art baseline approaches, which exploit either local or global temporal structure. As shown in Table TABREF20 , our Joint-BiLSTM reinforced model outperforms all of the baseline methods. The result of “LSTM” in first row refer from BIBREF15 and the last row but one denotes the best model combining local temporal structure using C3D with global temporal structure utilizing temporal attention in BIBREF17 . From the first two rows, our unidirectional joint LSTM shows rapid improvement, and comparing with S2VT-VGG model in line 3, it also demonstrates some superiority. Even LSTM-E jointly models video and descriptions representation by minimizing the distance between video and corresponding sentence, our Joint-BiLSTM reinforced obtains better performance from bidirectional encoding and separated visual and language models.
FLOAT SELECTED: Table 2: Comparing with several state-of-the-art models (reported in percentage, higher is better).

Answers:
S2VT, RGB (VGG), RGB (VGG)+Flow (AlexNet), LSTM-E (VGG), LSTM-E (C3D) and Yao et al.
Question: Which four languages do they experiment with?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Datasets used for various SER experiments.

Answers:
German, English, Italian, Chinese
Question: Does DCA or GMM-based attention perform better in experiments?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3. MOS naturalness results along with 95% confidence intervals for the Lessac and LJ datasets.

Answers:
About the same performance
Question: What evaluation metric is used?

Evidence:
raw_evidence: FLOAT SELECTED: Table 10: Results of experiments on the multi-turn adversarial task. We denote the average and one standard deviation from the results of five runs. Models that use the context as input (“with context”) perform better. Encoding this in the architecture as well (via BERT dialogue segment features) gives us the best results.

Answers:
F1 and Weighted-F1
Question: Is any data-to-text generation model trained on this new corpus, what are the results?

Evidence:
raw_evidence: The NLG model we use to establish a baseline for this dataset is a standard Transformer-based BIBREF19 sequence-to-sequence model. For decoding we employ beam search of width 10 ($\alpha = 1.0$). The generated candidates are then reranked according to the heuristically determined slot coverage score. Before training the model on the ViGGO dataset, we confirmed on the E2E dataset that it performed on par with, or even slightly better than, the strong baseline models from the E2E NLG Challenge, namely, TGen BIBREF20 and Slug2Slug BIBREF21.
We evaluate our model's performance on the ViGGO dataset using the following standard NLG metrics: BLEU BIBREF22, METEOR BIBREF23, ROUGE-L BIBREF24, and CIDEr BIBREF25. Additionally, with our heuristic slot error rate (SER) metric we approximate the percentage of failed slot realizations (i.e., missed, incorrect, or hallucinated) across the test set. The results are shown in Table TABREF16.
FLOAT SELECTED: Table 4: Baseline system performance on the ViGGO test set. Despite individual models (Bo3 – best of 3 experiments) often having better overall scores, we consider the Ao3 (average of 3) results the most objective.

Answers:
Yes, Transformer based seq2seq is evaluated with average BLEU 0.519, METEOR 0.388, ROUGE 0.631 CIDEr 2.531 and SER 2.55%.
Question: What dataset did they use?

Evidence:
raw_evidence: Table TABREF14 shows the experiment measuring improvements from the MWA attention on test sets of four datasets. Generally, our method consistently outperforms all baselines on all of four tasks, which clearly indicates the advantage of introducing word segmentation information into the encoding of character sequences. Moreover, the Wilcoxon’s test shows that significant difference ($p< 0.01$) exits between our model with baseline models.
FLOAT SELECTED: Table 2: Results of word-aligned attention models on multi NLP task. All of results are f1-score evaluated on test set and each experiment are enacted five times, the average is taken as result. Part of results are similar to results from BERT-wwm technical report (Cui et al., 2019).

Answers:
weibo-100k, Ontonotes, LCQMC and XNLI
Question: What are the 12 AV approaches which are examined?

Evidence:
raw_evidence: As a basis for our experiments, we reimplemented 12 existing AV approaches, which have shown their potentials in the previous PAN-AV competitions BIBREF11 , BIBREF12 as well as in a number of AV studies. The methods are listed in Table TABREF33 together with their classifications regarding the AV characteristics, which we proposed in Section SECREF3 .
FLOAT SELECTED: Table 2: All 12 AVmethods, classified according to their properties.

Answers:
MOCC, OCCAV, COAV, AVeer, GLAD, DistAV, Unmasking, Caravel, GenIM, ImpGI, SPATIUM and NNCD
Question: What are the results achieved from the introduced method?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Comparison of different models.

Answers:
Their model resulted in values of 0.476, 0.672 and 0.893 for recall at position 1,2 and 5 respectively in 10 candidates.
Question: How big is performance improvement proposed methods are used?

Evidence:
raw_evidence: The performance of the base model described in the previous section is shown in the first row of Table TABREF8 for the Nematus cs-en ($\bar{cs}$), FB MT system cs-en (cs) and es-en (es), sequence autoencoder (seq2seq), and the average of the adversarial sets (avg). We also included the results for the ensemble model, which combines the decisions of five separate baseline models that differ in batch order, initialization, and dropout masking. We can see that, similar to the case in computer vision BIBREF4, the adversarial examples seem to stem from fundamental properties of the neural networks and ensembling helps only a little.
FLOAT SELECTED: Table 3: Accuracy over clean and adversarial test sets. Note that data augmentation and logit pairing loss decrease accuracy on clean test sets and increase accuracy on the adversarial test sets.

Answers:
Data augmentation (es)  improved Adv es by 20% comparing to baseline 
Data augmentation (cs) improved Adv cs by 16.5% comparing to baseline
Data augmentation (cs+es) improved both Adv cs and Adv es by at least 10% comparing to baseline 
All models show improvements over adversarial sets
Question: By how much does transfer learning improve performance on this task?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Transfer learning performance (Task 1)

Answers:
In task 1 best transfer learning strategy improves F1 score by 4.4% and accuracy score by 3.3%, in task 2 best transfer learning strategy improves F1 score by 2.9% and accuracy score by 1.7%
Question: How much do they outperform previous state-of-the-art?

Evidence:
raw_evidence: Results on SemEval-2014 are presented in Table TABREF35 and Table TABREF36 . We find that BERT-single has achieved better results on these two subtasks, and BERT-pair has achieved further improvements over BERT-single. The BERT-pair-NLI-B model achieves the best performance for aspect category detection. For aspect category polarity, BERT-pair-QA-B performs best on all 4-way, 3-way, and binary settings.
FLOAT SELECTED: Table 4: Test set results for Semeval-2014 task 4 Subtask 3: Aspect Category Detection. We use the results reported in XRCE (Brun et al., 2014) and NRC-Canada (Kiritchenko et al., 2014).
FLOAT SELECTED: Table 5: Test set accuracy (%) for Semeval-2014 task 4 Subtask 4: Aspect Category Polarity. We use the results reported in XRCE (Brun et al., 2014), NRCCanada (Kiritchenko et al., 2014) and ATAE-LSTM (Wang et al., 2016). “-” means not reported.

Answers:
On subtask 3 best proposed model has F1 score of 92.18 compared to best previous F1 score of 88.58.
On subtask 4 best proposed model has 85.9, 89.9 and 95.6 compared to best previous results of 82.9, 84.0 and 89.9 on 4-way, 3-way and binary aspect polarity.
Question: What are the citation intent labels in the datasets?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Characteristics of SciCite compared with ACL-ARC dataset by Jurgens et al. (2018)

Answers:
Background, extends, uses, motivation, compare/contrast, and future work for the ACL-ARC dataset. Background, method, result comparison for the SciCite dataset.
Question: What accuracy score do they obtain?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3. Performance evaluation of variations of the proposed model and baseline. Showing highest scores in boldface.

Answers:
the best performing model obtained an accuracy of 0.86
Question: What is the 12 class bilingual text?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1. Description of class label along with distribution of each class (in %) in the acquired dataset

Answers:
Appreciation, Satisfied, Peripheral complaint, Demanded inquiry, Corruption, Lagged response, Unresponsive, Medicine payment, Adverse behavior, Grievance ascribed and Obnoxious/irrelevant
Question: Was evaluation metrics and criteria were used to evaluate the output of the cascaded multimodal speech translation?

Evidence:
raw_evidence: We train all the models on an Nvidia RTX 2080Ti with a batch size of 1024, a base learning rate of 0.02 with 8,000 warm-up steps for the Adam BIBREF29 optimiser, and a patience of 10 epochs for early stopping based on approx-BLEU () for the transformers and 3 epochs for the deliberation models. After the training finishes, we evaluate all the checkpoints on the validation set and compute the real BIBREF30 scores, based on which we select the best model for inference on the test set. The transformer and the deliberation models are based upon the library BIBREF31 (v1.3.0 RC1) as well as the vanilla transformer-based deliberation BIBREF20 and their multimodal variants BIBREF7.
FLOAT SELECTED: Table 1: BLEU scores for the test set: bold highlights our best results. † indicates a system is significantly different from its text-only counterpart (p-value ≤ 0.05).

Answers:
BLEU scores
Question: What are the domains covered in the dataset?

Evidence:
raw_evidence: The 17 domains (`Alarm' domain not included in training) present in our dataset are listed in Table TABREF5. We create synthetic implementations of a total of 34 services or APIs over these domains. Our simulator framework interacts with these services to generate dialogue outlines, which are a structured representation of dialogue semantics. We then used a crowd-sourcing procedure to paraphrase these outlines to natural language utterances. Our novel crowd-sourcing procedure preserves all annotations obtained from the simulator and does not require any extra annotations after dialogue collection. In this section, we describe these steps in detail and then present analyses of the collected dataset.
FLOAT SELECTED: Table 2: The number of intents (services in parentheses) and dialogues for each domain in the train and dev sets. Multidomain dialogues contribute to counts of each domain. The domain Service includes salons, dentists, doctors etc.

Answers:
Alarm
Bank
Bus
Calendar
Event
Flight
Home
Hotel
Media
Movie
Music
RentalCar
Restaurant
RideShare
Service
Travel
Weather