qinchuanhui/UDA-QA huggingface dataset:
Question: By how much does the new parser outperform the current state-of-the-art?

Evidence:
raw_evidence: Table TABREF12 compares our novel system with other state-of-the-art transition-based dependency parsers on the PT-SD. Greedy parsers are in the first block, beam-search and dynamic programming parsers in the second block. The third block shows the best result on this benchmark, obtained with constituent parsing with generative re-ranking and conversion to dependencies. Despite being the only non-projective parser tested on a practically projective dataset, our parser achieves the highest score among greedy transition-based models (even above those trained with a dynamic oracle).
We even slightly outperform the arc-swift system of Qi2017, with the same model architecture, implementation and training setup, but based on the projective arc-eager transition-based parser instead. This may be because our system takes into consideration any permissible attachment between the focus word INLINEFORM0 and any word in INLINEFORM1 at each configuration, while their approach is limited by the arc-eager logic: it allows all possible rightward arcs (possibly fewer than our approach as the arc-eager stack usually contains a small number of words), but only one leftward arc is permitted per parser state. It is also worth noting that the arc-swift and NL-Covington parsers have the same worst-case time complexity, ( INLINEFORM2 ), as adding non-local arc transitions to the arc-eager parser increases its complexity from linear to quadratic, but it does not affect the complexity of the Covington algorithm. Thus, it can be argued that this technique is better suited to Covington than to arc-eager parsing.
FLOAT SELECTED: Table 2: Accuracy comparison of state-of-theart transition-based dependency parsers on PT-SD. The “Type” column shows the type of parser: gs is a greedy parser trained with a static oracle, gd a greedy parser trained with a dynamic oracle, b(n) a beam search parser with beam size n, dp a parser that employs global training with dynamic programming, and c a constituent parser with conversion to dependencies.

Answers:
Proposed method achieves 94.5 UAS and 92.4 LAS  compared to 94.3 and 92.2 of best state-of-the -art greedy based parser. Best state-of-the art parser overall achieves 95.8 UAS and 94.6 LAS.
Question: Could you tell me more about the metrics used for performance evaluation?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: BLUE tasks
The aim of the inference task is to predict whether the premise sentence entails or contradicts the hypothesis sentence. We use the standard overall accuracy to evaluate the performance.
The aim of the relation extraction task is to predict relations and their types between the two entities mentioned in the sentences. The relations with types were compared to annotated data. We use the standard micro-average precision, recall, and F1-score metrics.
The aim of the named entity recognition task is to predict mention spans given in the text BIBREF20 . The results are evaluated through a comparison of the set of mention spans annotated within the document with the set of mention spans predicted by the model. We evaluate the results by using the strict version of precision, recall, and F1-score. For disjoint mentions, all spans also must be strictly correct. To construct the dataset, we used spaCy to split the text into a sequence of tokens when the original datasets do not provide such information.
The sentence similarity task is to predict similarity scores based on sentence pairs. Following common practice, we evaluate similarity by using Pearson correlation coefficients.
HoC (the Hallmarks of Cancers corpus) consists of 1,580 PubMed abstracts annotated with ten currently known hallmarks of cancer BIBREF27 . Annotation was performed at sentence level by an expert with 15+ years of experience in cancer research. We use 315 ( $\sim $ 20%) abstracts for testing and the remaining abstracts for training. For the HoC task, we followed the common practice and reported the example-based F1-score on the abstract level BIBREF28 , BIBREF29 .

Answers:
BLUE utilizes different metrics for each of the tasks: Pearson correlation coefficient, F-1 scores, micro-averaging, and accuracy
Question: which tasks are used in BLUE benchmark?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: BLUE tasks

Answers:
Inference task
The aim of the inference task is to predict whether the premise sentence entails or contradicts the hypothesis sentence, Document multilabel classification
The multilabel classification task predicts multiple labels from the texts., Relation extraction
The aim of the relation extraction task is to predict relations and their types between the two entities mentioned in the sentences., Named entity recognition
The aim of the named entity recognition task is to predict mention spans given in the text , Sentence similarity
The sentence similarity task is to predict similarity scores based on sentence pairs
Question: What's the precision of the system?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3. Overall typing performance of our method and the baselines on S-Lite and R-Lite.
FLOAT SELECTED: Table 4. Overall performance of entity matching on R-Lite with and without type constraint.

Answers:
0.8320 on semantic typing, 0.7194 on entity matching
Question: What are the performances obtained for PTB and CTB?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Dependency parsing on PTB and CTB.
FLOAT SELECTED: Table 4: Comparison of constituent parsing on PTB.
FLOAT SELECTED: Table 5: Comparison of constituent parsing on CTB.
Tables TABREF17, TABREF18 and TABREF19 compare our model to existing state-of-the-art, in which indicator Separate with our model shows the results of our model learning constituent or dependency parsing separately, (Sum) and (Concat) respectively represent the results with the indicated input token representation setting. On PTB, our model achieves 93.90 F1 score of constituent parsing and 95.91 UAS and 93.86 LAS of dependency parsing. On CTB, our model achieves a new state-of-the-art result on both constituent and dependency parsing. The comparison again suggests that learning jointly in our model is superior to learning separately. In addition, we also augment our model with ELMo BIBREF48 or a larger version of BERT BIBREF49 as the sole token representation to compare with other pre-training models. Since BERT is based on sub-word, we only take the last sub-word vector of the word in the last layer of BERT as our sole token representation $x_i$. Moreover, our single model of BERT achieves competitive performance with other ensemble models.

Answers:
. On PTB, our model achieves 93.90 F1 score of constituent parsing and 95.91 UAS and 93.86 LAS of dependency parsing., On CTB, our model achieves a new state-of-the-art result on both constituent and dependency parsing.
Question: Is the proposed layer smaller in parameters than a Transformer?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Tokenized case-sensitive BLEU (in parentheses: sacreBLEU) on WMT14 En-De translation task. #Param: number of model parameters. 4Dec: decoding time (seconds)/speedup on newstest2014 dataset with a batch size of 32. 4Train: training time (seconds)/speedup per training step evaluated on 0.5K steps with a batch size of 1K target tokens. Time is averaged over 3 runs using Tensorflow on a single TITAN X (Pascal). “-”: optimization failed and no result. “?”: the same as model 1©. † and ‡: comparison against 11© and 14© respectively rather than 1©. Base: the baseline Transformer with base setting. Bold indicates best BLEU score. dpa and dpr: dropout rate on attention weights and residual connection. bs: batch size in tokens.
FLOAT SELECTED: Table 5: Translation results on different tasks. Settings for BLEU score is given in Section 7.1. Numbers in bracket denote chrF score. Our model outperforms the vanilla base Transformer on all tasks. “Ours”: DS-Init+MAtt.

Answers:
No
Question: What are the four forums the data comes from?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Test set results at the NP level in within-forum and cross-forum settings for a variety of different systems. Using either Brown clusters or gazetteers gives mixed results on cross-forum performance: only one of the improvements (†) is statistically significant with p < 0.05 according to a bootstrap resampling test. Gazetteers are unavailable for Blackhat and Nulled since we have no training data for those forums.

Answers:
Darkode,  Hack Forums, Blackhat and Nulled.
Question: What are the state-of-the-art models for the task?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Performance of various methods on the FNC-I benchmark. The first and second groups are methods introduced during and after the challenge period, respectively. Best results are in bold.
Results of our proposed method, the top three methods in the original Fake News Challenge, and the best-performing methods since the challenge's conclusion on the FNC-I test set are displayed in Table TABREF12. A confusion matrix for our method is presented in the Appendix. To the best of our knowledge, our method achieves state-of-the-art results in weighted-accuracy and standard accuracy on the dataset. Notably, since the conclusion of the Fake News Challenge in 2017, the weighted-accuracy error-rate has decreased by 8%, signifying improved performance of NLP models and innovations in the domain of stance detection, as well as a continued interest in combating the spread of disinformation.

Answers:
To the best of our knowledge, our method achieves state-of-the-art results in weighted-accuracy and standard accuracy on the dataset
Question: How much improvement do they get?

Evidence:
raw_evidence: FLOAT SELECTED: Table 7. Experimental results

Answers:
Their GTRS approach got an improvement of 3.89% compared to SVM and 27.91% compared to Pawlak.
Question: Is this approach compared to some baseline?

Evidence:
raw_evidence: Table TABREF24 gives the UAR for each feature subset individually, for all features combined, and for a combination in which one feature subset in turn is left out. The one-feature-set-at-time results suggest that prosody, speech activity and words are of increasing importance in that order. The leave-one-out analysis agrees that the words are the most important (largest drop in accuracy when removed), but on that criterion the prosodic features are more important than speech-activity. The combination of all features is 0.4% absolute better than any other subset, showing that all feature subsets are partly complementary.
FLOAT SELECTED: Table 2. Hot spot classification results with individual feature subsets, all features, and with individual feature sets left out.

Answers:
No
Question: What are the baselines?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Results of baselines and scores of top 3 participants on valid and test datasets.

Answers:
CNN, LSTM, BERT
Question: Which models do they try out?

Evidence:
raw_evidence: FLOAT SELECTED: Table 4: Performance of various methods and human.

Answers:
DocQA, SAN, QANet, ASReader, LM, Random Guess
Question: Do they compare executionttime of their model against other models?

Evidence:
raw_evidence: The base model composed of DSConv layers without grouping achieves the state-of-the-art accuracy of 96.6% on the Speech Commands test set. The low-parameter model with GDSConv achieves almost the same accuracy of 96.4% with only about half the parameters. This validates the effectiveness of GDSConv for model size reduction. Table TABREF15 lists these results in comparison with related work. Compared to the DSConv network in BIBREF1, our network is more efficient in terms of accuracy for a given parameter count. Their biggest model has a 1.2% lower accuracy than our base model while having about 4 times the parameters. Choi et al. BIBREF3 has the most competitive results while we are still able to improve upon their accuracy for a given number of parameters. They are using 1D convolution along the time dimension as well which may be evidence that this yields better performance for audio processing or at least KWS.
FLOAT SELECTED: Table 1. Comparison of results on the Speech Commands dataset [19].
FLOAT SELECTED: Table 2. Results on Speech Commands version 2 [19].

Answers:
No
Question: Which English domains do they evaluate on?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2.3: Labelled attachment scores achieved by the MST, Malt, and Mate parsers trained on the Conll training set and tested on different domains.
We further evaluate our approach on our main evaluation corpus. The method is tested on both in-domain and out-of-domain parsing. Our DLM-based approach achieved large improvement on all five domains evaluated (Conll, Weblogs, Newsgroups, Reviews, Answers). We achieved the labelled and unlabelled improvements of up to 0.91% and 0.82% on Newsgroups domain. On average we achieved 0.6% gains for both labelled and unlabelled scores on four out-of-domain test sets. We also improved the in-domain accuracy by 0.36% (LAS) and 0.4% (UAS).

Answers:
Conll, Weblogs, Newsgroups, Reviews, Answers
Question: What are the competing models?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Results on EMNLP2017 WMT News dataset. The 95 % confidence intervals from multiple trials are reported.

Answers:
TEACHER FORCING (TF), SCHEDULED SAMPLING (SS),  SEQGAN, RANKGAN, LEAKGAN.
Question: What writing styles are present in the corpus?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Stylistic domains and examples (bold marks annotated entities)

Answers:
current news, historical news, free time, sports, juridical news pieces, personal adverts, editorials.
Question: What experiment result led to conclussion that reducing the number of layers of the decoder does not matter much?

Evidence:
raw_evidence: Last, we analyze the importance of our second encoder ($enc_{src \rightarrow mt}$), compared to the source encoder ($enc_{src}$) and the decoder ($dec_{pe}$), by reducing and expanding the amount of layers in the encoders and the decoder. Our standard setup, which we use for fine-tuning, ensembling etc., is fixed to 6-6-6 for $N_{src}$-$N_{mt}$-$N_{pe}$ (cf. Figure FIGREF1), where 6 is the value that was proposed by Vaswani:NIPS2017 for the base model. We investigate what happens in terms of APE performance if we change this setting to 6-6-4 and 6-4-6. To handle out-of-vocabulary words and reduce the vocabulary size, instead of considering words, we consider subword units BIBREF19 by using byte-pair encoding (BPE). In the preprocessing step, instead of learning an explicit mapping between BPEs in the $src$, $mt$ and $pe$, we define BPE tokens by jointly processing all triplets. Thus, $src$, $mt$ and $pe$ derive a single BPE vocabulary. Since $mt$ and $pe$ belong to the same language (German) and $src$ is a close language (English), they naturally share a good fraction of BPE tokens, which reduces the vocabulary size to 28k.
The number of layers ($N_{src}$-$N_{mt}$-$N_{pe}$) in all encoders and the decoder for these results is fixed to 6-6-6. In Exp. 5.1, and 5.2 in Table TABREF5, we see the results of changing this setting to 6-6-4 and 6-4-6. This can be compared to the results of Exp. 2.3, since no fine-tuning or ensembling was performed for these three experiments. Exp. 5.1 shows that decreasing the number of layers on the decoder side does not hurt the performance. In fact, in the case of test2016, we got some improvement, while for test2017, the scores got slightly worse. In contrast, reducing the $enc_{src \rightarrow mt}$ encoder block's depth (Exp. 5.2) does indeed reduce the performance for all four scores, showing the importance of this second encoder.
FLOAT SELECTED: Table 1: Evaluation results on the WMT APE test set 2016, and test set 2017 for the PBSMT task; (±X) value is the improvement over wmt18smtbest (x4). The last section of the table shows the impact of increasing and decreasing the depth of the encoders and the decoder.

Answers:
Exp. 5.1
Question: How much is performance hurt when using too small amount of layers in encoder?

Evidence:
raw_evidence: The number of layers ($N_{src}$-$N_{mt}$-$N_{pe}$) in all encoders and the decoder for these results is fixed to 6-6-6. In Exp. 5.1, and 5.2 in Table TABREF5, we see the results of changing this setting to 6-6-4 and 6-4-6. This can be compared to the results of Exp. 2.3, since no fine-tuning or ensembling was performed for these three experiments. Exp. 5.1 shows that decreasing the number of layers on the decoder side does not hurt the performance. In fact, in the case of test2016, we got some improvement, while for test2017, the scores got slightly worse. In contrast, reducing the $enc_{src \rightarrow mt}$ encoder block's depth (Exp. 5.2) does indeed reduce the performance for all four scores, showing the importance of this second encoder.
FLOAT SELECTED: Table 1: Evaluation results on the WMT APE test set 2016, and test set 2017 for the PBSMT task; (±X) value is the improvement over wmt18smtbest (x4). The last section of the table shows the impact of increasing and decreasing the depth of the encoders and the decoder.

Answers:
comparing to the results from reducing the number of layers in the decoder, the BLEU score was 69.93 which is less than 1% in case of test2016 and in case of test2017 it was less by 0.2 %. In terms of TER it had higher score by 0.7 in case of test2016 and 0.1 in case of test2017.
Question: How much does HAS-QA improve over baselines?

Evidence:
raw_evidence: 1) HAS-QA outperforms traditional RC baselines with a large gap, such as GA, BiDAF, AQA listed in the first part. For example, in QuasarT, it improves 16.8% in EM score and 20.4% in F1 score. As RC task is just a special case of OpenQA task. Some experiments on standard SQuAD dataset(dev-set) BIBREF9 show that HAS-QA yields EM/F1:0.719/0.798, which is comparable with the best released single model Reinforced Mnemonic Reader BIBREF25 in the leaderboard (dev-set) EM/F1:0.721/0.816. Our performance is slightly worse because Reinforced Mnemonic Reader directly use the accurate answer span, while we use multiple distantly supervised answer spans. That may introduce noises in the setting of SQuAD, since only one span is accurate.
2) HAS-QA outperforms recent OpenQA baselines, such as DrQA, R ${}^3$ and Shared-Norm listed in the second part. For example, in QuasarT, it improves 4.6% in EM score and 3.5% in F1 score.
FLOAT SELECTED: Table 2: Experimental results on OpenQA datasets QuasarT, TriviaQA and SearchQA. EM: Exact Match.

Answers:
For example, in QuasarT, it improves 16.8% in EM score and 20.4% in F1 score. , For example, in QuasarT, it improves 4.6% in EM score and 3.5% in F1 score.
Question: Which dataset do they evaluate grammatical error correction on?

Evidence:
raw_evidence: We conduct experiments by incorporating NNGLM and NNJM both independently and jointly into our baseline system. The results of our experiments are described in Section SECREF23 . The evaluation is performed similar to the CoNLL 2014 shared task setting using the the official test data of the CoNLL 2014 shared task with annotations from two annotators (without considering alternative annotations suggested by the participating teams). The test dataset consists of 1,312 error-annotated sentences with 30,144 tokens on the source side. We make use of the official scorer for the shared task, M INLINEFORM0 Scorer v3.2 BIBREF19 , for evaluation. We perform statistical significance test using one-tailed sign test with bootstrap resampling on 100 samples.
FLOAT SELECTED: Table 2: Results of our experiments with NNGLM and NNJM on the CoNLL 2014 test set (* indicates statistical significance with p < 0.01)

Answers:
CoNLL 2014
Question: How large is the Dialog State Tracking Dataset?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Data used in this paper. Tasks 1-5 were generated using our simulator and share the same KB. Task 6 was converted from the 2nd Dialog State Tracking Challenge (Henderson et al., 2014a). Concierge is made of chats extracted from a real online concierge service. (∗) Tasks 1-5 have two test sets, one using the vocabulary of the training set and the other using out-of-vocabulary words.

Answers:
1,618 training dialogs, 500 validation dialogs, and 1,117 test dialogs
Question: How much is the gap between using the proposed objective and using only cross-entropy objective?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Ablation study on the development set of SQuAD.
The contributions of each part of our model are shown in Table 2 . We note that the deep residual coattention yielded the highest contribution to model performance, followed by the mixed objective. The sparse mixture of experts layer in the decoder added minor improvements to the model performance.

Answers:
The mixed objective improves EM by 2.5% and F1 by 2.2%
Question: what are the three methods presented in the paper?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: MAP scoring of new methods. The timings are in seconds for the whole dev-set, and the BERT Re-ranking figure includes the initial Iterated TF-IDF step.

Answers:
Optimized TF-IDF, iterated TF-IDF, BERT re-ranking.
Question: what datasets did the authors use?

Evidence:
raw_evidence: We trained and tested our neural network with and without sentiment information, with and without subversion, and with each corpus three times to mitigate the randomness in training. In every experiment, we used a random 70% of messages in the corpus as training data, another 20% as validation data, and the final 10% as testing data. The average results of the three tests are given in Table TABREF40 . It can be seen that sentiment information helps improve toxicity detection in all cases. The improvement is smaller when the text is clean. However, the introduction of subversion leads to an important drop in the accuracy of toxicity detection in the network that uses the text alone, and the inclusion of sentiment information gives an important improvement in that case. Comparing the different corpora, it can be seen that the improvement is smallest in the Reddit dataset experiment, which is expected since it is also the dataset in which toxicity and sentiment had the weakest correlation in Table TABREF37 .
FLOAT SELECTED: Table 7: Accuracy of toxicity detection with and without sentiment

Answers:
Kaggle
Subversive Kaggle
Wikipedia
Subversive Wikipedia
Reddit
Subversive Reddit
Question: How much performance improvements they achieve on SQuAD?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Test performance on SQuAD. Results are sorted by Test F1.
Finally, we compare our results with other top models in Table 2 . Note that all the results in Table 2 are taken from the published papers. We see that SAN is very competitive in both single and ensemble settings (ranked in second) despite its simplicity. Note that the best-performing model BIBREF14 used a large-scale language model as an extra contextual embedding, which gave a significant improvement (+4.3% dev F1). We expect significant improvements if we add this to SAN in future work.
The main experimental question we would like to answer is whether the stochastic dropout and averaging in the answer module is an effective technique for multi-step reasoning. To do so, we fixed all lower layers and compared different architectures for the answer module:
FLOAT SELECTED: Table 1: Main results—Comparison of different answer module architectures. Note that SAN performs best in both Exact Match and F1 metrics.

Answers:
Compared to baselines SAN (Table 1) shows  improvement of 1.096% on EM and 0.689% F1. Compared to other published SQuAD results (Table 2) SAN is ranked second.