qinchuanhui/UDA-QA huggingface dataset:
Question: what was their character error rate?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3. The CER and RTF of 9-layers, 2-layers regular-trained and 2-laryers distilled LSTM.
FLOAT SELECTED: Table 2. The CER of 6 to 9-layers models trained by regular Xavier Initialization, layer-wise training with CE criterion and CE + sMBR criteria. The teacher of 9-layer model is 8-layers sMBR model, while the others’ teacher is CE model.
FLOAT SELECTED: Table 4. The CER of different 2-layers models, which are Shenma distilled model, Amap model further trained with Amap dataset, and Shenma model trained with sMBR on Amap dataset.
raw_evidence: FLOAT SELECTED: Table 3. The CER and RTF of 9-layers, 2-layers regular-trained and 2-laryers distilled LSTM.
FLOAT SELECTED: Table 2. The CER of 6 to 9-layers models trained by regular Xavier Initialization, layer-wise training with CE criterion and CE + sMBR criteria. The teacher of 9-layer model is 8-layers sMBR model, while the others’ teacher is CE model.

Answers:
2.49% for  layer-wise training, 2.63% for distillation, 6.26% for transfer learning.
Their best model achieved a 2.49% Character Error Rate.
Question: which lstm models did they compare with?

Evidence:
raw_evidence: There is a high real time requirement in real world application, especially in online voice search system. Shenma voice search is one of the most popular mobile search engines in China, and it is a streaming service that intermediate recognition results displayed while users are still speaking. Unidirectional LSTM network is applied, rather than bidirectional one, because it is well suited to real-time streaming speech recognition.
FLOAT SELECTED: Table 3. The CER and RTF of 9-layers, 2-layers regular-trained and 2-laryers distilled LSTM.
FLOAT SELECTED: Table 2. The CER of 6 to 9-layers models trained by regular Xavier Initialization, layer-wise training with CE criterion and CE + sMBR criteria. The teacher of 9-layer model is 8-layers sMBR model, while the others’ teacher is CE model.

Answers:
Unidirectional LSTM networks with 2, 6, 7, 8, and  9 layers.
Question: By how much did they improve?

Evidence:
raw_evidence: Concerning the neural network architecture, we focus on Recurrent Neural Networks (RNNs) that are capable of modeling short-range and long-range dependencies like those exhibited in sequence data of arbitrary length like text. While in the traditional information retrieval paradigm such dependencies are captured using INLINEFORM0 -grams and skip-grams, RNNs learn to capture them automatically BIBREF11 . To circumvent the problems with capturing long-range dependencies and preventing gradients from vanishing, the long short-term memory network (LSTM) was proposed BIBREF12 . In this work, we use an extended version of LSTM called bidirectional LSTM (biLSTM). While standard LSTMs access information only from the past (previous words), biLSTMs capture both past and future information effectively BIBREF13 , BIBREF11 . They consist of two LSTM networks, for propagating text forward and backwards with the goal being to capture the dependencies better. Indeed, previous work on multitask learning showed the effectiveness of biLSTMs in a variety of problems: BIBREF14 tackled sequence prediction, while BIBREF6 and BIBREF15 used biLSTMs for Named Entity Recognition and dependency parsing respectively.
The models To evaluate the multitask learning approach, we compared it with several other models. Support Vector Machines (SVMs) are maximum margin classification algorithms that have been shown to achieve competitive performance in several text classification problems BIBREF16 . SVM INLINEFORM0 stands for an SVM with linear kernel and an one-vs-rest approach for the multi-class problem. Also, SVM INLINEFORM1 is an SVM with linear kernel that employs the crammer-singer strategy BIBREF18 for the multi-class problem. Logistic regression (LR) is another type of linear classification method, with probabilistic motivation. Again, we use two types of Logistic Regression depending on the multi-class strategy: LR INLINEFORM2 that uses an one-vs-rest approach and multinomial Logistic Regression also known as the MaxEnt classifier that uses a multinomial criterion.
Experimental results Table TABREF9 illustrates the performance of the models for the different data representations. The upper part of the Table summarizes the performance of the baselines. The entry “Balikas et al.” stands for the winning system of the 2016 edition of the challenge BIBREF2 , which to the best of our knowledge holds the state-of-the-art. Due to the stochasticity of training the biLSTM models, we repeat the experiment 10 times and report the average and the standard deviation of the performance achieved.
FLOAT SELECTED: Table 3 The scores on MAEM for the systems. The best (lowest) score is shown in bold and is achieved in the multitask setting with the biLSTM architecture of Figure 1.
Evaluation measure To reproduce the setting of the SemEval challenges BIBREF16 , we optimize our systems using as primary measure the macro-averaged Mean Absolute Error ( INLINEFORM0 ) given by: INLINEFORM1

Answers:
They decrease MAE in 0.34
Question: What is quantitative improvement of proposed method (the best variant) w.r.t. baseline (the best variant)?

Evidence:
raw_evidence: To evaluate the impact of our model components, we first compare scenarios Flat, Hierarchical-k, and Hierarchical-kv. As shown in Table TABREF25, we can see the lower results obtained by the Flat scenario compared to the other scenarios (e.g. BLEU $16.7$ vs. $17.5$ for resp. Flat and Hierarchical-k), suggesting the effectiveness of encoding the data-structure using a hierarchy. This is expected, as losing explicit delimitation between entities makes it harder a) for the encoder to encode semantics of the objects contained in the table and b) for the attention mechanism to extract salient entities/records.
FLOAT SELECTED: Table 1: Evaluation on the RotoWire testset using relation generation (RG) count (#) and precision (P%), content selection (CS) precision (P%) and recall (R%), content ordering (CO), and BLEU. -: number of parameters unavailable.

Answers:
Hierarchical-k
Question: What metrics are used in evaluation?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: F1 scores on an unseen (not used for training) part of the training set and the development set on BERT using different augmentation techniques.
FLOAT SELECTED: Table 3: Class-wise precision and recall with and without oversampling (OS) achieved on unseen part of the training set.
FLOAT SELECTED: Table 4: Our results on the SLC task (2nd, in bold) alongside comparable results from the competition leaderboard.
FLOAT SELECTED: Table 5: Our results on the FLC task (7th, in bold) alongside those of better performing teams from the competition leaderboard.
So as to better understand the aspects of oversampling that contribute to these gains, we perform a class-wise performance analysis of BERT with/without oversampling. The results of these experiments (Table TABREF18) show that oversampling increases the overall recall while maintaining precision. This is achieved by significantly improving the recall of the minority class (propaganda) at the cost of the recall of the majority class.
We explore the validity of this by performing several experiments with different weights assigned to the minority class. We note that in our experiments use significantly higher weights than the weights proportional to class frequencies in the training data, that are common in literature BIBREF17. Rather than directly using the class proportions of the training set, we show that tuning weights based on performance on the development set is more beneficial. Figure FIGREF22 shows the results of these experiments wherein we are able to maintain the precision on the subset of the training set used for testing while reducing its recall and thus generalising the model. The fact that the model is generalising on a dissimilar dataset is confirmed by the increase in the development set F1 score. We note that the gains are not infinite and that a balance must be struck based on the amount of generalisation and the corresponding loss in accuracy. The exact weight to use for the best transfer of classification accuracy is related to the dissimilarity of that other dataset and hence is to be obtained experimentally through hyperparameter search. Our experiments showed that a value of 4 is best suited for this task.

Answers:
precision, recall , F1 score
Question: By how much does their best model outperform the state-of-the-art?

Evidence:
raw_evidence: Table TABREF16 shows the results of our models ER-CNN (extended ranking CNN) and R-RNN (ranking RNN) in the context of other state-of-the-art models. Our proposed models obtain state-of-the-art results on the SemEval 2010 task 8 data set without making use of any linguistic features.
FLOAT SELECTED: Table 3: State-of-the-art results for relation classification
raw_evidence: Table TABREF16 shows the results of our models ER-CNN (extended ranking CNN) and R-RNN (ranking RNN) in the context of other state-of-the-art models. Our proposed models obtain state-of-the-art results on the SemEval 2010 task 8 data set without making use of any linguistic features.
FLOAT SELECTED: Table 3: State-of-the-art results for relation classification

Answers:
0.8% F1 better than the best state-of-the-art
Best proposed model achieves F1 score of 84.9 compared to best previous result of 84.1.
Question: Does the paper report the performance of the model for each individual language?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Baseline scores in the cross-lingual setting. No Italian samples were seen during training, making this a case of zero-shot cross-lingual transfer. The scores are reported as the macro-average of the F1scores for ‘favor’ and for ‘against’.
raw_evidence: FLOAT SELECTED: Table 4: Baseline scores in the cross-target setting. For each test set we separately report a German and a French score, as well as their harmonic mean.

Answers:
Yes
Yes
Question: What is the performance of the baseline?

Evidence:
raw_evidence: FLOAT SELECTED: Table 6: Performance of BERT-like models on different supervised stance detection benchmarks.
raw_evidence: FLOAT SELECTED: Table 6: Performance of BERT-like models on different supervised stance detection benchmarks.

Answers:
M-Bert had 76.6 F1 macro score.
75.1% and 75.6% accuracy
Question: What was the performance of multilingual BERT?

Evidence:
raw_evidence: To put the supervised score into context we list scores that variants of Bert have achieved on other stance detection datasets in Table TABREF46. It seems that the supervised part of x-stance has a similar difficulty as the SemEval-2016 BIBREF0 or MPCHI BIBREF22 datasets on which Bert has previously been evaluated.
FLOAT SELECTED: Table 6: Performance of BERT-like models on different supervised stance detection benchmarks.

Answers:
BERT had 76.6 F1 macro score on x-stance dataset.
Question: What is the state of the art system mentioned?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: F1-score (%) for fine-grained English all-words WSD on the test sets in the framework of Raganato et al. (2017b) (including the development set SE07). Bold font indicates best systems. The five blocks list the MFS baseline, two knowledge-based systems, two traditional word expert supervised systems, six recent neural-based systems and our systems, respectively. Results in first three blocks come from Raganato et al. (2017b), and others from the corresponding papers.

Answers:
Two knowledge-based systems,
two traditional word expert supervised systems, six recent neural-based systems, and one BERT feature-based system.
Question: Do the methods that work best on academic papers also work best on Wikipedia?

Evidence:
raw_evidence: We proposed to use visual renderings of documents to capture implicit document quality indicators, such as font choices, images, and visual layout, which are not captured in textual content. We applied neural network models to capture visual features given visual renderings of documents. Experimental results show that we achieve a 2.9% higher accuracy than state-of-the-art approaches based on textual features over Wikipedia, and performance competitive with or surpassing state-of-the-art approaches over arXiv. We further proposed a joint model, combining textual and visual representations, to predict the quality of a document. Experimental results show that our joint model outperforms the visual-only model in all cases, and the text-only model on Wikipedia and two subsets of arXiv. These results underline the feasibility of assessing document quality via visual features, and the complementarity of visual and textual document representations for quality assessment.
raw_evidence: We proposed to use visual renderings of documents to capture implicit document quality indicators, such as font choices, images, and visual layout, which are not captured in textual content. We applied neural network models to capture visual features given visual renderings of documents. Experimental results show that we achieve a 2.9% higher accuracy than state-of-the-art approaches based on textual features over Wikipedia, and performance competitive with or surpassing state-of-the-art approaches over arXiv. We further proposed a joint model, combining textual and visual representations, to predict the quality of a document. Experimental results show that our joint model outperforms the visual-only model in all cases, and the text-only model on Wikipedia and two subsets of arXiv. These results underline the feasibility of assessing document quality via visual features, and the complementarity of visual and textual document representations for quality assessment.
FLOAT SELECTED: Table 1: Experimental results. The best result for each dataset is indicated in bold, and marked with “†” if it is significantly higher than the second best result (based on a one-tailed Wilcoxon signed-rank test; p < 0.05). The results of Benchmark on Peer Review are from the original paper, where the standard deviation values were not reported.

Answers:
Yes
No
Question: What is their system's absolute accuracy?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Experimental results. The best result for each dataset is indicated in bold, and marked with “†” if it is significantly higher than the second best result (based on a one-tailed Wilcoxon signed-rank test; p < 0.05). The results of Benchmark on Peer Review are from the original paper, where the standard deviation values were not reported.

Answers:
59.4% on wikipedia dataset, 93.4% on peer-reviewed archive AI papers, 77.1%  on peer-reviewed archive Computation and Language papers, and 79.9% on peer-reviewed archive Machine Learning papers
Question: What were their best results on the benchmark datasets?

Evidence:
raw_evidence: Table TABREF32 and TABREF33 contain results of the models on SNLI and MultiNLI datasets. In SNLI, our best model achieves the new state-of-the-art accuracy of 87.0% with relatively fewer parameters. Similarly in MultiNLI, our models match the accuracy of state-of-the-art models in both in-domain (matched) and cross-domain (mismatched) test sets. Note that only the GloVe word vectors are used as word representations, as opposed to some models that introduce character-level features. It is also notable that our proposed architecture does not restrict the selection of pooling method; the performance could further be improved by replacing max-pooling with other advanced algorithms e.g. intra-sentence attention BIBREF39 and generalized pooling BIBREF19 .
Similar to the NLI experiments, GloVe pretrained vectors, 300D encoders, and 1024D MLP are used. The number of CAS-LSTM layers is fixed to 2 in PI experiments. Two sentence vectors are aggregated using Eq. EQREF29 and fed as input to the MLP. The results on the Quora Question Pairs dataset are summarized in Table TABREF34 . Again we can see that our models outperform other models by large margin, achieving the new state of the art.
FLOAT SELECTED: Table 3: Results of the models on the Quora Question Pairs dataset.
FLOAT SELECTED: Table 4: Results of the models on the SST dataset. ∗: models pretrained on large external corpora are used.
raw_evidence: Table TABREF32 and TABREF33 contain results of the models on SNLI and MultiNLI datasets. In SNLI, our best model achieves the new state-of-the-art accuracy of 87.0% with relatively fewer parameters. Similarly in MultiNLI, our models match the accuracy of state-of-the-art models in both in-domain (matched) and cross-domain (mismatched) test sets. Note that only the GloVe word vectors are used as word representations, as opposed to some models that introduce character-level features. It is also notable that our proposed architecture does not restrict the selection of pooling method; the performance could further be improved by replacing max-pooling with other advanced algorithms e.g. intra-sentence attention BIBREF39 and generalized pooling BIBREF19 .

Answers:
In SNLI, our best model achieves the new state-of-the-art accuracy of 87.0%,  we can see that our models outperform other models by large margin, achieving the new state of the art., Our models achieve the new state-of-the-art accuracy on SST-2 and competitive accuracy on SST-5
accuracy of 87.0%
Question: what linguistics features are used?

Evidence:
raw_evidence: Table TABREF17 lists the features that we used for CE recovery. We used Farasa to perform segmentation and POS tagging and to determine stem-templates BIBREF31. Farasa has a reported POS accuracy of 96% on the WikiNews dataset BIBREF31. Though the Farasa diacritizer utilizes a combination of some the features presented herein, namely segmentation, POS tagging, and stem templates, Farasa's SVM-ranking approach requires explicit specification of feature combinations (ex. $Prob(CE\Vert current\_word, prev\_word, prev\_CE)$). Manual exploration of the feature space is undesirable, and ideally we would want our learning algorithm to do so automatically. The flexibility of the DNN model allowed us to include many more surface level features such as affixes, leading and trailing characters in words and stems, and the presence of words in large gazetteers of named entities. As we show later, these additional features significantly lowered CEER.
FLOAT SELECTED: Table 1. Features with examples and motivation.

Answers:
POS, gender/number and stem POS
Question: What is best performing model among author's submissions, what performance it had?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Comparison of our system (MIC-CIS) with top-5 participants: Scores on Test set for SLC and FLC

Answers:
For SLC task, the "ltuorp" team  has the best performing  model (0.6323/0.6028/0.6649 for F1/P/R  respectively) and for FLC task the "newspeak" team  has the best performing  model (0.2488/0.2863/0.2201 for F1/P/R respectively).
Question: What extracted features were most influencial on performance?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: SLC: Scores on Dev (internal) of Fold1 and Dev (external) using different classifiers and features.
raw_evidence: Shared Task: This work addresses the two tasks in propaganda detection BIBREF3 of different granularities: (1) Sentence-level Classification (SLC), a binary classification that predicts whether a sentence contains at least one propaganda technique, and (2) Fragment-level Classification (FLC), a token-level (multi-label) classification that identifies both the spans and the type of propaganda technique(s).
Contributions: (1) To address SLC, we design an ensemble of different classifiers based on Logistic Regression, CNN and BERT, and leverage transfer learning benefits using the pre-trained embeddings/models from FastText and BERT. We also employed different features such as linguistic (sentiment, readability, emotion, part-of-speech and named entity tags, etc.), layout, topics, etc. (2) To address FLC, we design a multi-task neural sequence tagger based on LSTM-CRF and linguistic features to jointly detect propagandistic fragments and its type. Moreover, we investigate performing FLC and SLC jointly in a multi-granularity network based on LSTM-CRF and BERT. (3) Our system (MIC-CIS) is ranked 3rd (out of 12 participants) and 4th (out of 25 participants) in FLC and SLC tasks, respectively.
Table TABREF10 shows the scores on dev (internal and external) for SLC task. Observe that the pre-trained embeddings (FastText or BERT) outperform TF-IDF vector representation. In row r2, we apply logistic regression classifier with BERTSentEmb that leads to improved scores over FastTextSentEmb. Subsequently, we augment the sentence vector with additional features that improves F1 on dev (external), however not dev (internal). Next, we initialize CNN by FastTextWordEmb or BERTWordEmb and augment the last hidden layer (before classification) with BERTSentEmb and feature vectors, leading to gains in F1 for both the dev sets. Further, we fine-tune BERT and apply different thresholds in relaxing the decision boundary, where $\tau \ge 0.35$ is found optimal.

Answers:
Linguistic
BERT
Question: Which basic neural architecture perform best by itself?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: SLC: Scores on Dev (internal) of Fold1 and Dev (external) using different classifiers and features.

Answers:
BERT
Question: What participating systems had better results than ones authors submitted?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Comparison of our system (MIC-CIS) with top-5 participants: Scores on Test set for SLC and FLC

Answers:
For SLC task : Ituorp, ProperGander and YMJA  teams had better results.
For FLC task: newspeak and Antiganda teams had better results.
Question: What is the size of the StackExchange dataset?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Statistics of datasets we use in this work. Avg# and Var# indicate the mean and variance of numbers of target phrases per data point, respectively.

Answers:
around 332k questions
Question: What were the baselines?

Evidence:
raw_evidence: We report our model's performance on the present-keyphrase portion of the KP20k dataset in Table TABREF35 . To compare with previous works, we provide compute INLINEFORM0 and INLINEFORM1 scores. The new proposed F INLINEFORM2 @ INLINEFORM3 metric indicates consistent ranking with INLINEFORM4 for most cases. Due to its target number sensitivity, we find that its value is closer to INLINEFORM5 for KP20k and Krapivin where average target keyphrases is less and closer to INLINEFORM6 for the other three datasets.
FLOAT SELECTED: Table 2: Present keyphrase predicting performance on KP20K test set. Compared with CopyRNN (Meng et al., 2017), Multi-Task (Ye and Wang, 2018), and TG-Net (Chen et al., 2018b).
raw_evidence: We include four non-neural extractive models and CopyRNN BIBREF0 as baselines. We use CopyRNN to denote the model reported by BIBREF0 , CopyRNN* to denote our implementation of CopyRNN based on their open sourced code. To draw fair comparison with existing study, we use the same model hyperparameter setting as used in BIBREF0 and use exhaustive decoding strategy for most experiments. KEA BIBREF4 and Maui BIBREF8 are trained on a subset of 50,000 documents from either KP20k (Table TABREF35 ) or StackEx (Table TABREF37 ) instead of all documents due to implementation limits (without fine-tuning on target dataset).

Answers:
CopyRNN (Meng et al., 2017), Multi-Task (Ye and Wang, 2018), and TG-Net (Chen et al., 2018b)
CopyRNN BIBREF0, KEA BIBREF4 and Maui BIBREF8, CopyRNN*