qinchuanhui/UDA-QA huggingface dataset:
Question: Which state-of-the-art model is surpassed by 9.68% attraction score?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Human evaluation on three aspects: relevance, attraction, and fluency. “None” represents the original headlines in the dataset.
In terms of attraction scores in Table TABREF51, we have three findings: (1) The human-written headlines are more attractive than those from NHG, which agrees with our observation in Section SECREF1. (2) Our TitleStylist can generate more attractive headlines over the NHG and Multitask baselines for all three styles, demonstrating that adapting the model to these styles could improve the attraction and specialization of some parameters in the model for different styles can further enhance the attraction. (3) Adapting the model to the “Clickbait” style could create the most attractive headlines, even out-weighting the original ones, which agrees with the fact that click-baity headlines are better at drawing readers' attention. To be noted, although we learned the “Clickbait” style into our summarization system, we still made sure that we are generating relevant headlines instead of too exaggerated ones, which can be verified by our relevance scores.

Answers:
pure summarization model NHG
Question: What is increase in percentage of humor contained in headlines generated with TitleStylist method (w.r.t. baselines)?

Evidence:
raw_evidence: The human evaluation is to have a comprehensive measurement of the performances. We conduct experiments on four criteria, relevance, attraction, fluency, and style strength. We summarize the human evaluation results on the first three criteria in Table TABREF51, and the last criteria in Table TABREF57. Note that through automatic evaluation, the baselines NST, Fine-tuned, and Gigaword-MASS perform poorer than other methods (in Section SECREF58), thereby we removed them in human evaluation to save unnecessary work for human raters.
FLOAT SELECTED: Table 2: Human evaluation on three aspects: relevance, attraction, and fluency. “None” represents the original headlines in the dataset.
FLOAT SELECTED: Table 2: Human evaluation on three aspects: relevance, attraction, and fluency. “None” represents the original headlines in the dataset.

Answers:
Humor in headlines (TitleStylist vs Multitask baseline):
Relevance: +6.53% (5.87 vs 5.51)
Attraction: +3.72% (8.93 vs 8.61)
Fluency: 1,98% (9.29 vs 9.11)
Question: What evaluation metrics are used?

Evidence:
raw_evidence: Table TABREF34 shows the performances of the different methods. From the table, we can see that the performances of most tasks can be improved with the help of multi-task learning. FS-MTL shows the minimum performance gain from multi-task learning since it puts all private and shared information into a unified space. SSP-MTL and PSP-MTL achieve similar performance and are outperformed by ASP-MTL which can better separate the task-specific and task-invariant features by using adversarial training. Our proposed models (SA-MTL and DA-MTL) outperform ASP-MTL because we model a richer representation from these 16 tasks. Compared to SA-MTL, DA-MTL achieves a further improvement of INLINEFORM0 accuracy with the help of the dynamic and flexible query vector. It is noteworthy that our models are also space efficient since the task-specific information is extracted by using only a query vector, instead of a BiLSTM layer in the shared-private models.
FLOAT SELECTED: Table 2: Performances on 16 tasks. The column of “Single Task” includes bidirectional LSTM (BiLSTM), bidirectional LSTM with attention (att-BiLSTM) and the average accuracy of the two models. The column of “Multiple Tasks” shows several multi-task models. * is from [Liu et al., 2017] .

Answers:
Accuracy on each dataset and the average accuracy on all datasets.
Question: How much better are results of proposed model compared to pivoting method?

Evidence:
raw_evidence: First we compare our models with the conventional pivoting method, i.e., round-trip translation. As shown in Figure FIGREF15 (a)(b), either the bilingual or the multilingual model is better than the baseline in terms of relevance and diversity in most cases. In other words, with the same generation diversity (measured by both Distinct-2 and Self-BLEU), our models can generate paraphrase with more semantically similarity to the input sentence.
As shown in Table TABREF28, our method outperforms the baseline in both relevance and fluency significantly. We further calculate agreement (Cohen's kappa) between two annotators.
Both round-trip translation and our method performs well as to fluency. But the huge gap of relevance between the two systems draw much attention of us. We investigate the test set in details and find that round-trip approach indeed generate more noise as shown in case studies.
FLOAT SELECTED: Table 3: Human evaluation results.

Answers:
our method outperforms the baseline in both relevance and fluency significantly.
Question: Who was the top-scoring team?

Evidence:
raw_evidence: FLOAT SELECTED: Table 6: F-scores for Friends (%)
FLOAT SELECTED: Table 7: F-scores for EmotionPush (%)
The submissions and the final results are summarized in Tables and . Two of the submissions did not follow up with technical papers and thus they do not appear in this summary. We note that the top-performing models used BERT, reflecting the recent state-of-the-art performance of this model in many NLP tasks. For Friends and EmotionPush the top micro-F1 scores were 81.5% and 88.5% respectively.

Answers:
IDEA
Question: How strong is the correlation between the prevalence of the #MeToo movement and official reports [of sexual harassment]?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Linear regression results.
We examine other features regarding the characteristics of the studied colleges, which might be significant factors of sexual harassment. Four factual attributes pertaining to the 200 colleges are extracted from the U.S. News Statistics, which consists of Undergraduate Enrollment, Male/Female Ratio, Private/Public, and Region (Northeast, South, West, and Midwest). We also use the normalized rape-related cases count (number of cases reported per student enrolled) from the stated government resource as another attribute to examine the proximity of our dataset to the official one. This feature vector is then fitted in a linear regression to predict the normalized #metoo users count (number of unique users who posted #MeToo tweets per student enrolled) for each individual college.

Answers:
0.9098 correlation
Question: What were their accuracy results on the task?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Lemmatization accuracy using WikiNews testset

Answers:
97.32%
Question: How much F1 was improved after adding skip connections?

Evidence:
raw_evidence: Table TABREF20 reports the F1 and EM scores obtained for the experiments on the base model. The first column reports the base BERT baseline scores, while the second reports the results for the C2Q/Q2C attention addition. The two skip columns report scores for the skip connection connecting the BERT embedding layer to the coattention output (Simple Skip) and the scores for the same skip connection containing a Transformer block (Transformer Skip). The final column presents the result of the localized feature extraction added inside the C2Q/Q2C architecture (Inside Conv - Figure FIGREF8).
FLOAT SELECTED: Table 2: Performance results for experiments relative to BERT base

Answers:
Simple Skip improves F1 from 74.34 to 74.81
Transformer Skip improes F1 from 74.34 to 74.95
Question: How much gain does the model achieve with pretraining MVCNN?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Test set results of our CNN model against other methods. RAE: Recursive Autoencoders with pretrained word embeddings from Wikipedia (Socher et al., 2011b). MV-RNN: Matrix-Vector Recursive Neural Network with parse trees (Socher et al., 2012). RNTN: Recursive Neural Tensor Network with tensor-based feature function and parse trees (Socher et al., 2013). DCNN, MAX-TDNN, NBOW: Dynamic Convolution Neural Network with k-max pooling, Time-Delay Neural Networks with Max-pooling (Collobert and Weston, 2008), Neural Bag-of-Words Models (Kalchbrenner et al., 2014). Paragraph-Vec: Logistic regression on top of paragraph vectors (Le and Mikolov, 2014). SVM, BINB, MAXENT: Support Vector Machines, Naive Bayes with unigram features and bigram features, Maximum Entropy (Go et al., 2009). NBSVM, MNB: Naive Bayes SVM and Multinomial Naive Bayes with uni-bigrams from Wang and Manning (2012). CNN-rand/static/multichannel/nonstatic: CNN with word embeddings randomly initialized / initialized by pretrained vectors and kept static during training / initialized with two copies (each is a “channel”) of pretrained embeddings / initialized with pretrained embeddings while fine-tuned during training (Kim, 2014). G-Dropout, F-Dropout: Gaussian Dropout and Fast Dropout from Wang and Manning (2013). Minus sign “-” in MVCNN (-Huang) etc. means “Huang” is not used. “versions / filters / tricks / layers” denote the MVCNN variants with different setups: discard certain embedding version / discard certain filter size / discard mutual-learning or pretraining / different numbers of convolution layer.

Answers:
0.8 points on Binary; 0.7 points on Fine-Grained; 0.6 points on Senti140; 0.7 points on Subj
Question: What are the effects of extracting features of multigranular phrases?

Evidence:
raw_evidence: The block “filters” indicates the contribution of each filter size. The system benefits from filters of each size. Sizes 5 and 7 are most important for high performance, especially 7 (rows 25 and 26).
This work presented MVCNN, a novel CNN architecture for sentence classification. It combines multichannel initialization – diverse versions of pretrained word embeddings are used – and variable-size filters – features of multigranular phrases are extracted with variable-size convolution filters. We demonstrated that multichannel initialization and variable-size filters enhance system performance on sentiment classification and subjectivity classification tasks.
FLOAT SELECTED: Table 3: Test set results of our CNN model against other methods. RAE: Recursive Autoencoders with pretrained word embeddings from Wikipedia (Socher et al., 2011b). MV-RNN: Matrix-Vector Recursive Neural Network with parse trees (Socher et al., 2012). RNTN: Recursive Neural Tensor Network with tensor-based feature function and parse trees (Socher et al., 2013). DCNN, MAX-TDNN, NBOW: Dynamic Convolution Neural Network with k-max pooling, Time-Delay Neural Networks with Max-pooling (Collobert and Weston, 2008), Neural Bag-of-Words Models (Kalchbrenner et al., 2014). Paragraph-Vec: Logistic regression on top of paragraph vectors (Le and Mikolov, 2014). SVM, BINB, MAXENT: Support Vector Machines, Naive Bayes with unigram features and bigram features, Maximum Entropy (Go et al., 2009). NBSVM, MNB: Naive Bayes SVM and Multinomial Naive Bayes with uni-bigrams from Wang and Manning (2012). CNN-rand/static/multichannel/nonstatic: CNN with word embeddings randomly initialized / initialized by pretrained vectors and kept static during training / initialized with two copies (each is a “channel”) of pretrained embeddings / initialized with pretrained embeddings while fine-tuned during training (Kim, 2014). G-Dropout, F-Dropout: Gaussian Dropout and Fast Dropout from Wang and Manning (2013). Minus sign “-” in MVCNN (-Huang) etc. means “Huang” is not used. “versions / filters / tricks / layers” denote the MVCNN variants with different setups: discard certain embedding version / discard certain filter size / discard mutual-learning or pretraining / different numbers of convolution layer.

Answers:
The system benefits from filters of each size., features of multigranular phrases are extracted with variable-size convolution filters.
Question: What are the effects of diverse versions of pertained word embeddings? 

Evidence:
raw_evidence: In the block “versions”, we see that each embedding version is crucial for good performance: performance drops in every single case. Though it is not easy to compare fairly different embedding versions in NLP tasks, especially when those embeddings were trained on different corpora of different sizes using different algorithms, our results are potentially instructive for researchers making decision on which embeddings to use for their own tasks.
FLOAT SELECTED: Table 3: Test set results of our CNN model against other methods. RAE: Recursive Autoencoders with pretrained word embeddings from Wikipedia (Socher et al., 2011b). MV-RNN: Matrix-Vector Recursive Neural Network with parse trees (Socher et al., 2012). RNTN: Recursive Neural Tensor Network with tensor-based feature function and parse trees (Socher et al., 2013). DCNN, MAX-TDNN, NBOW: Dynamic Convolution Neural Network with k-max pooling, Time-Delay Neural Networks with Max-pooling (Collobert and Weston, 2008), Neural Bag-of-Words Models (Kalchbrenner et al., 2014). Paragraph-Vec: Logistic regression on top of paragraph vectors (Le and Mikolov, 2014). SVM, BINB, MAXENT: Support Vector Machines, Naive Bayes with unigram features and bigram features, Maximum Entropy (Go et al., 2009). NBSVM, MNB: Naive Bayes SVM and Multinomial Naive Bayes with uni-bigrams from Wang and Manning (2012). CNN-rand/static/multichannel/nonstatic: CNN with word embeddings randomly initialized / initialized by pretrained vectors and kept static during training / initialized with two copies (each is a “channel”) of pretrained embeddings / initialized with pretrained embeddings while fine-tuned during training (Kim, 2014). G-Dropout, F-Dropout: Gaussian Dropout and Fast Dropout from Wang and Manning (2013). Minus sign “-” in MVCNN (-Huang) etc. means “Huang” is not used. “versions / filters / tricks / layers” denote the MVCNN variants with different setups: discard certain embedding version / discard certain filter size / discard mutual-learning or pretraining / different numbers of convolution layer.

Answers:
each embedding version is crucial for good performance
Question: What is the highest accuracy score achieved?

Evidence:
raw_evidence: FLOAT SELECTED: Table 4: The performance of classifiers trained on the original and generated datasets. The classifiers were tested on original test set. The generated datasets were generated by the models from Table 3. The generated datasets were filtered with threshold 0.6.

Answers:
82.0%
Question: What is improvement in accuracy for short Jokes in relation other types of jokes?

Evidence:
raw_evidence: Our experiment with the Short Jokes dataset found the Transformer model's accuracy and F1 score to be 0.986. This was a jump of 8 percent from the most recent work done with CNNs (Table 4).
In Table 2, we see the results of our experiment with the Reddit dataset. We ran our models on the body of the joke exclusively, the punchline exclusively, and both parts together (labeled full in our table). On the full dataset we found that the Transformer achieved an accuracy of 72.4 percent on the hold out test set, while the CNN was in the high 60's. We also note that the general human classification found 66.3% of the jokes to be humorous.
The results on the Pun of the Day dataset are shown in Table 3 above. It shows an accuracy of 93 percent, close to 4 percent greater accuracy than the best CNN model proposed. Although the CNN model used a variety of techniques to extract the best features from the dataset, we see that the self-attention layers found even greater success in pulling out the crucial features.
FLOAT SELECTED: Table 2: Results of Accuracy on Reddit Jokes dataset
FLOAT SELECTED: Table 3: Comparison of Methods on Pun of the Day Dataset. HCF represents Human Centric Features, F for increasing the number of filters, and HN for the use of highway layers in the model. See (Chen and Soo, 2018; Yang et al., 2015) for more details regarding these acronyms.
FLOAT SELECTED: Table 4: Results on Short Jokes Identification

Answers:
It had the highest accuracy comparing to all datasets 0.986% and It had the highest improvement comparing to previous methods on the same dataset by 8%
Question: What is the metric used with WIKIHOP?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Accuracy of different models on WIKIHOP closed test set and public validation set. Our Entity-GCN outperforms recent prior work without learning any language model to process the input but relying on a pretrained one (ELMo – without fine-tunning it) and applying R-GCN to reason among entities in the text. * with coreference for unmasked dataset and without coreference for the masked one.

Answers:
Accuracy
Question: What performance does the Entity-GCN get on WIKIHOP?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Accuracy of different models on WIKIHOP closed test set and public validation set. Our Entity-GCN outperforms recent prior work without learning any language model to process the input but relying on a pretrained one (ELMo – without fine-tunning it) and applying R-GCN to reason among entities in the text. * with coreference for unmasked dataset and without coreference for the masked one.

Answers:
During testing: 67.6 for single model without coreference, 66.4 for single model with coreference, 71.2 for ensemble of 5 models
Question: How do they damage different neural modules?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Results for artificial Wernicke’s and Broca’s aphasia induced in the LLA-LSTM model. Damage to neural modules is done by randomly initializing their weights, causing the loss of all learned information. The inputs that we present are arbitrarily chosen, subject to the constraints listed in the text. Mean precision (Prec.) results on the test sets are also provided to demonstrate corpus-level results. An ellipses represents the repetition of the preceding word at least 1000 times.

Answers:
Damage to neural modules is done by randomly initializing their weights, causing the loss of all learned information.
Question: what metrics are used in evaluation?

Evidence:
raw_evidence: We evaluate our approach on two well benchmarked sequence tagging tasks, the CoNLL 2003 NER task BIBREF13 and the CoNLL 2000 Chunking task BIBREF14 . We report the official evaluation metric (micro-averaged INLINEFORM0 ). In both cases, we use the BIOES labeling scheme for the output tags, following previous work which showed it outperforms other options BIBREF15 . Following BIBREF8 , we use the Senna word embeddings BIBREF2 and pre-processed the text by lowercasing all tokens and replacing all digits with 0.
FLOAT SELECTED: Table 1: Test set F1 comparison on CoNLL 2003 NER task, using only CoNLL 2003 data and unlabeled text.

Answers:
micro-averaged F1
Question: what previous systems were compared to?

Evidence:
raw_evidence: Tables TABREF15 and TABREF16 compare results from TagLM with previously published state of the art results without additional labeled data or task specific gazetteers. Tables TABREF17 and TABREF18 compare results of TagLM to other systems that include additional labeled data or gazetteers. In both tasks, TagLM establishes a new state of the art using bidirectional LMs (the forward CNN-BIG-LSTM and the backward LSTM-2048-512).
FLOAT SELECTED: Table 1: Test set F1 comparison on CoNLL 2003 NER task, using only CoNLL 2003 data and unlabeled text.
FLOAT SELECTED: Table 2: Test set F1 comparison on CoNLL 2000 Chunking task using only CoNLL 2000 data and unlabeled text.
FLOAT SELECTED: Table 3: Improvements in test set F1 in CoNLL 2003 NER when including additional labeled data or task specific gazetteers (except the case of TagLM where we do not use additional labeled resources).
FLOAT SELECTED: Table 4: Improvements in test set F1 in CoNLL 2000 Chunking when including additional labeled data (except the case of TagLM where we do not use additional labeled data).

Answers:
Chiu and Nichols (2016), Lample et al. (2016), Ma and Hovy (2016), Yang et al. (2017), Hashimoto et al. (2016), Søgaard and Goldberg (2016)
Question: When they say "comparable performance", how much of a performance drop do these new embeddings result in?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Results on test data. The proposed method significantly improves interpretability while maintaining comparable performance on KG tasks (Section 4.3).

Answers:
Performance was comparable, with the proposed method quite close and sometimes exceeding performance of baseline method.
Question: How does TP-N2F compare to LSTM-based Seq2Seq in terms of training and inference speed?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Results of AlgoLisp dataset
Generating Lisp programs requires sensitivity to structural information because Lisp code can be regarded as tree-structured. Given a natural-language query, we need to generate code containing function calls with parameters. Each function call is a relational tuple, which has a function as the relation and parameters as arguments. We evaluate our model on the AlgoLisp dataset for this task and achieve state-of-the-art performance. The AlgoLisp dataset BIBREF17 is a program synthesis dataset. Each sample contains a problem description, a corresponding Lisp program tree, and 10 input-output testing pairs. We parse the program tree into a straight-line sequence of tuples (same style as in MathQA). AlgoLisp provides an execution script to run the generated program and has three evaluation metrics: the accuracy of passing all test cases (Acc), the accuracy of passing 50% of test cases (50p-Acc), and the accuracy of generating an exactly matching program (M-Acc). AlgoLisp has about 10% noisy data (details in the Appendix), so we report results both on the full test set and the cleaned test set (in which all noisy testing samples are removed). TP-N2F is compared with an LSTM seq2seq with attention model, the Seq2Tree model in BIBREF17, and a seq2seq model with a pre-trained tree decoder from the Tree2Tree autoencoder (SAPS) reported in BIBREF18. As shown in Table TABREF18, TP-N2F outperforms all existing models on both the full test set and the cleaned test set. Ablation experiments with TP2LSTM and LSTM2TP show that, for this task, the TP-N2F Decoder is more helpful than TP-N2F Encoder. This may be because lisp codes rely more heavily on structure representations.

Answers:
Full Testing Set accuracy: 84.02
Cleaned Testing Set accuracy: 93.48