qinchuanhui/UDA-QA huggingface dataset:
Question: How much is proposed model better in perplexity and BLEU score than typical UMT models?

Evidence:
raw_evidence: As illustrated in Table TABREF12 (ID 1). Given the vernacular translation of each gold poem in test set, we generate five poems using our models. Intuitively, the more the generated poem resembles the gold poem, the better the model is. We report mean perplexity and BLEU scores in Table TABREF19 (Where +Anti OT refers to adding the reinforcement loss to mitigate over-fitting and +Anti UT refers to adding phrase segmentation-based padding to mitigate under-translation), human evaluation results in Table TABREF20.
FLOAT SELECTED: Table 3: Perplexity and BLEU scores of generating poems from vernacular translations. Since perplexity and BLEU scores on the test set fluctuates from epoch to epoch, we report the mean perplexity and BLEU scores over 5 consecutive epochs after convergence.

Answers:
Perplexity of the best model is 65.58 compared to best baseline 105.79.
Bleu of the best model is 6.57 compared to best baseline 5.50.
Question: What web and user-generated NER datasets are used for the analysis?

Evidence:
raw_evidence: Since the goal of this study is to compare NER performance on corpora from diverse domains and genres, seven benchmark NER corpora are included, spanning newswire, broadcast conversation, Web content, and social media (see Table 1 for details). These datasets were chosen such that they have been annotated with the same or very similar entity classes, in particular, names of people, locations, and organisations. Thus corpora including only domain-specific entities (e.g. biomedical corpora) were excluded. The choice of corpora was also motivated by their chronological age; we wanted to ensure a good temporal spread, in order to study possible effects of entity drift over time.
FLOAT SELECTED: Table 1 Corpora genres and number of NEs of different classes.

Answers:
MUC, CoNLL, ACE, OntoNotes, MSM, Ritter, UMBC
Question: How big are the datasets?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Number of documents in the training/dev/test sets of the in-house and ACE05 datasets.
Our in-house dataset includes manually annotated RE data for 6 languages: English, German, Spanish, Italian, Japanese and Portuguese. It defines 56 entity types (e.g., Person, Organization, Geo-Political Entity, Location, Facility, Time, Event_Violence, etc.) and 53 relation types between the entities (e.g., AgentOf, LocatedAt, PartOf, TimeOf, AffectedBy, etc.).
The ACE05 dataset includes manually annotated RE data for 3 languages: English, Arabic and Chinese. It defines 7 entity types (Person, Organization, Geo-Political Entity, Location, Facility, Weapon, Vehicle) and 6 relation types between the entities (Agent-Artifact, General-Affiliation, ORG-Affiliation, Part-Whole, Personal-Social, Physical).

Answers:
In-house dataset consists of  3716 documents 
ACE05 dataset consists of  1635 documents
Question: What was the performance of their model?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Performance of different models on the test datasets. EM and GM report percentages, and ED corresponds to average edit distance. The symbol ↑ indicates that higher results are better in the corresponding column; ↓ indicates that lower is better.

Answers:
For test-repeated set, EM score of 61.17, F1 of 93.54, ED of 0.75 and GM of 61.36. For test-new set, EM score of 41.71, F1 of 91.02, ED of 1.22 and GM of 41.81
Question: What are their initial results on this task?

Evidence:
raw_evidence: FLOAT SELECTED: Table 5: Overall and domain-specific Precision, Recall, and F1 scores for our models. The first row computes similarity directly from the TF-IDF matrix, as in (McCoy et al., 2015). All other rows are classifier outputs.

Answers:
Achieved the highest per-domain scores on Substance (F1 ≈ 0.8) and the lowest scores on Interpersonal and Mood (F1 ≈ 0.5), and show consistency in per-domain performance rankings between MLP and RBF models.
Question: What are the different bilingual models employed?

Evidence:
raw_evidence: We use the bilingual neural-based Unsupervised Word Segmentation (UWS) approach from BIBREF6 to discover words in Mboshi. In this approach, Neural Machine Translation (NMT) models are trained between language pairs, using as source language the translation (word-level) and as target, the language to document (unsegmented phonemic sequence). Due to the attention mechanism present in these networks BIBREF7, posterior to training, it is possible to retrieve soft-alignment probability matrices between source and target sequences. These matrices give us sentence-level source-to-target alignment information, and by using it for clustering neighbor phonemes aligned to the same translation word, we are able to create segmentation in the target side. The product of this approach is a set of (discovered-units, translation words) pairs.
In this work we apply two simple methods for including multilingual information into the bilingual models from BIBREF6. The first one, Multilingual Voting, consists of merging the information learned by models trained with different language pairs by performing a voting over the final discovered boundaries. The voting is performed by applying an agreement threshold $T$ over the output boundaries. This threshold balances between accepting all boundaries from all the bilingual models (zero agreement) and accepting only input boundaries discovered by all these models (total agreement). The second method is ANE Selection. For every language pair and aligned sentence in the dataset, a soft-alignment probability matrix is generated. We use Average Normalized Entropy (ANE) BIBREF8 computed over these matrices for selecting the most confident one for segmenting each phoneme sequence. This exploits the idea that models trained on different language pairs will have language-related behavior, thus differing on the resulting alignment and segmentation over the same phoneme sequence.
Lastly, following the methodology from BIBREF8, we extract the most confident alignments (in terms of ANE) discovered by the bilingual models. Table presents the top 10 most confident (discovered type, translation) pairs. Looking at the pairs the bilingual models are most confident about, we observe there are some types discovered by all the bilingual models (e.g. Mboshi word itua, and the concatenation oboá+ngá). However, the models still differ for most of their alignments in the table. This hints that while a portion of the lexicon might be captured independently of the language used, other structures might be more dependent of the chosen language. On this note, BIBREF11 suggests the notion of word cannot always be meaningfully defined cross-linguistically.
FLOAT SELECTED: Table 3: Top 10 confident (discovered type, translation) pairs for the five bilingual models. The “+” mark means the discovered type is a concatenation of two existing true types.

Answers:
 Neural Machine Translation (NMT) models are trained between language pairs, using as source language the translation (word-level) and as target
Question: Does this approach perform better in the multi-domain or single-domain setting?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: The joint goal accuracy of the DST models on the WoZ2.0 test set and the MultiWoZ test set. We also include the Inference Time Complexity (ITC) for each model as a metric for scalability. The baseline accuracy for the WoZ2.0 dataset is the Delexicalisation-Based (DB) Model (Mrksic et al., 2017), while the baseline for the MultiWoZ dataset is taken from the official website of MultiWoZ (Budzianowski et al., 2018).

Answers:
single-domain setting
Question: What is the difference in recall score between the systems?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2. Comparison with Traditional NERs as reference
FLOAT SELECTED: Table 3. Comparison with Wikipedia titles as reference

Answers:
Between the model and Stanford, Spacy and Flair the differences are 42.91, 25.03, 69.8 with Traditional NERs as reference and  49.88, 43.36, 62.43 with Wikipedia titles as reference.
Question: What is their f1 score and recall?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2. Comparison with Traditional NERs as reference
FLOAT SELECTED: Table 3. Comparison with Wikipedia titles as reference

Answers:
F1 score and Recall are 68.66, 80.08 with Traditional NERs as reference and 59.56, 69.76 with Wikipedia titles as reference.
Question: How big is improvement in performances of proposed model over state of the art?

Evidence:
raw_evidence: We consider three models as our baselines. SyntaxSQL-con and CD-Seq2Seq are two strong baselines introduced in the SParC dataset paper BIBREF2. SyntaxSQL-con employs a BiLSTM model to encode dialogue history upon the SyntaxSQLNet model (analogous to our Turn) BIBREF23, while CD-Seq2Seq is adapted from BIBREF4 for cross-domain settings (analogous to our Turn+Tree Copy). EditSQL BIBREF5 is a STOA baseline which mainly makes use of SQL attention and token-level copy (analogous to our Turn+SQL Attn+Action Copy).
Taking Concat as a representative, we compare the performance of our model with other models, as shown in Table TABREF34. As illustrated, our model outperforms baselines by a large margin with or without BERT, achieving new SOTA performances on both datasets. Compared with the previous SOTA without BERT on SParC, our model improves Ques.Match and Int.Match by $10.6$ and $5.4$ points, respectively.
FLOAT SELECTED: Table 1: We report the best performance observed in 5 runs on the development sets of both SPARC and COSQL, since their test sets are not public. We also conduct Wilcoxon signed-rank tests between our method and the baselines, and the results show the improvements of our model are significant with p < 0.005.

Answers:
Compared with the previous SOTA without BERT on SParC, our model improves Ques.Match and Int.Match by $10.6$ and $5.4$ points, respectively.
Question: Did they compare against other systems?

Evidence:
raw_evidence: The slot extraction and intent keywords extraction results are given in Table TABREF1 and Table TABREF2 , respectively. Table TABREF3 summarizes the results of various approaches we investigated for utterance-level intent understanding. Table TABREF4 shows the intent-wise detection results for our AMIE scenarios with the best performing utterance-level intent recognizer.
FLOAT SELECTED: Table 3: Utterance-level Intent Recognition Results (10-fold CV)

Answers:
Yes
Question: How large is the gap in performance between the HMMs and the LSTMs?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Predictive loglikelihood (LL) comparison, sorted by validation set performance.

Answers:
With similar number of parameters, the log likelihood is about 0.1 lower for LSTMs across datasets. When the number of parameters in LSTMs is increased, their log likelihood is up to 0.7 lower.
Question: what was their system's f1 performance?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: F1 Results3
The approach we have developed establishes a new state of the art for classifying hate speech, outperforming previous results by as much as 12 F1 points. Table TABREF10 illustrates the robustness of our method, which often outperform previous results, measured by weighted F1.

Answers:
Proposed model achieves 0.86, 0.924, 0.71 F1 score on SR, HATE, HAR datasets respectively.
Question: Is the supervised morphological learner tested on Japanese?

Evidence:
raw_evidence: We conduct two out of the three proposed approaches for Japanese-Vietnamese translation systems and the results are given in the Table TABREF15.
FLOAT SELECTED: Table 1: Results of Japanese-Vietnamese NMT systems

Answers:
No
Question: How better is proposed method than baselines perpexity wise?

Evidence:
raw_evidence: Table TABREF34 gives the perplexity scores obtained by the three models on the two validation sets and the test set. As shown in the table, MEED achieves the lowest perplexity score on all three sets. We also conducted t-test on the perplexity obtained, and results show significant improvements (with $p$-value $<0.05$).
FLOAT SELECTED: Table 2: Perplexity scores achieved by the models. Validation set 1 comes from the Cornell dataset, while validation set 2 comes from the DailyDialog dataset.

Answers:
Perplexity of proposed MEED model is 19.795 vs 19.913 of next best result on test set.
Question: Which training dataset allowed for the best generalization to benchmark sets?

Evidence:
raw_evidence: FLOAT SELECTED: Table 4: Test accuracies (%). For the baseline results (highlighted in bold) the training data and test data have been drawn from the same benchmark corpus. ∆ is the difference between the test accuracy and the baseline accuracy for the same training set. Results marked with * are for the development set, as no annotated test set is openly available. Best scores with respect to accuracy and difference in accuracy are underlined.

Answers:
MultiNLI
Question: How many natural language explanations are human-written?

Evidence:
raw_evidence: e-SNLI-VE-2.0 is the combination of SNLI-VE-2.0 with explanations from either e-SNLI or our crowdsourced annotations where applicable. The statistics of e-SNLI-VE-2.0 are shown in Table TABREF40.
FLOAT SELECTED: Table 3. Summary of e-SNLI-VE-2.0 (= SNLI-VE-2.0 + explanations). Image-sentence pairs labelled as neutral in the training set have not been corrected.

Answers:
Totally 6980 validation and test image-sentence pairs have been corrected.
Question: What are the baseline state of the art models?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Evaluation results of different approaches compared to ours

Answers:
Stanford NER, BiLSTM+CRF, LSTM+CNN+CRF, T-NER and BiLSTM+CNN+Co-Attention
Question: What was the result of the highest performing system?

Evidence:
raw_evidence: There are 74 participants who are signing up the evaluation. The final number of participants is 28 and the number of submitted systems is 43. Table TABREF14 and TABREF15 show the evaluation results of the closed test and open test of the task 1 respectively. Due to the space limitation, we only present the top 5 results of task 1. We will add the complete lists of the evaluation results in the version of full paper.
Note that for task 2, there are 7 submitted systems. However, only 4 systems can provide correct results or be connected in a right way at the test phase. Therefore, Table TABREF16 shows the complete results of the task 2.
FLOAT SELECTED: Table 4: Top 5 results of the closed test of the task 1.
FLOAT SELECTED: Table 5: Top 5 results of the open test of the task 1.
FLOAT SELECTED: Table 6: The results of the task 2. Ratio, Satisfaction, Fluency, Turns and Guide indicate the task completion ratio, user satisfaction degree, response fluency, number of dialogue turns and guidance ability for out of scope input respectively.

Answers:
For task 1 best F1 score was 0.9391 on closed and 0.9414 on open test.
For task2 best result had: Ratio 0.3175 , Satisfaction 64.53, Fluency 0, Turns -1 and Guide 2
Question: What are the baselines that Masque is compared against?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Performance of our and competing models on the MS MARCO V2 leaderboard (4 March 2019). aSeo et al. (2017); bYan et al. (2019); cShao (unpublished), a variant of Tan et al. (2018); dLi (unpublished), a model using Devlin et al. (2018) and See et al. (2017); eQian (unpublished); fWu et al. (2018). Whether the competing models are ensemble models or not is unreported.
FLOAT SELECTED: Table 5: Performance of our and competing models on the NarrativeQA test set. aSeo et al. (2017); bTay et al. (2018); cBauer et al. (2018); dIndurthi et al. (2018); eHu et al. (2018). fResults on the NarrativeQA validation set.

Answers:
BiDAF, Deep Cascade QA, S-Net+CES2S, BERT+Multi-PGNet, Selector+CCG, VNET, DECAPROP, MHPGM+NOIC, ConZNet, RMR+A2D
Question: What is the performance achieved on NarrativeQA?

Evidence:
raw_evidence: FLOAT SELECTED: Table 5: Performance of our and competing models on the NarrativeQA test set. aSeo et al. (2017); bTay et al. (2018); cBauer et al. (2018); dIndurthi et al. (2018); eHu et al. (2018). fResults on the NarrativeQA validation set.

Answers:
Bleu-1: 54.11, Bleu-4: 30.43, METEOR: 26.13, ROUGE-L: 59.87
Question: How many uniue words are in the dataset?

Evidence:
raw_evidence: The large corpus acquired from multiple resources is rich in vocabulary. We present the complete statistics of collected corpus (see Table TABREF52) with number of sentences, words and unique tokens.
FLOAT SELECTED: Table 2: Complete statistics of collected corpus from multiple resources.

Answers:
908456 unique words are available in collected corpus.
Question: How much is the BLEU score?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: BLEU scores and computation times with varyingK and sequence length compared to baseline models with and without attention.

Answers:
Ranges from 44.22 to 100.00 depending on K and the sequence length.
Question: What are new best results on standard benchmark?

Evidence:
raw_evidence: Table TABREF15 shows the final results on Vecmap. We first compare our model with the state-of-the-art unsupervised methods. Our model based on procrustes (Ours-Procrustes) outperforms Sinkhorn-BT on all test language pairs, and shows better performance than Adv-C-Procrustes on most language pairs. Adv-C-Procrustes gives very low precision on DE-EN, FI-EN and ES-EN, while Ours-Procrustes obtains reasonable results consistently. A possible explanation is that dual learning is helpful for providing good initiations, so that the procrustes solution is not likely to fall in poor local optima. The reason why Unsup-SL gives strong results on all language pairs is that it uses a robust self-learning framework, which contains several techniques to avoid poor local optima.
FLOAT SELECTED: Table 4: Accuracy (P@1) on Vecmap. The best results are bolded. †Results as reported in the original paper. For unsupervised methods, we report the average accuracy across 10 runs.

Answers:
New best results of accuracy (P@1) on Vecmap:
Ours-GeoMMsemi: EN-IT 50.00 IT-EN 42.67 EN-DE 51.60 DE-EN 47.22 FI-EN 39.62 EN-ES 39.47 ES-EN 36.43
Question: How better is performance compared to competitive baselines?

Evidence:
raw_evidence: FLOAT SELECTED: Table 4: Accuracy (P@1) on Vecmap. The best results are bolded. †Results as reported in the original paper. For unsupervised methods, we report the average accuracy across 10 runs.
Table TABREF15 shows the final results on Vecmap. We first compare our model with the state-of-the-art unsupervised methods. Our model based on procrustes (Ours-Procrustes) outperforms Sinkhorn-BT on all test language pairs, and shows better performance than Adv-C-Procrustes on most language pairs. Adv-C-Procrustes gives very low precision on DE-EN, FI-EN and ES-EN, while Ours-Procrustes obtains reasonable results consistently. A possible explanation is that dual learning is helpful for providing good initiations, so that the procrustes solution is not likely to fall in poor local optima. The reason why Unsup-SL gives strong results on all language pairs is that it uses a robust self-learning framework, which contains several techniques to avoid poor local optima.

Answers:
Proposed method vs best baseline result on Vecmap (Accuracy P@1):
EN-IT: 50 vs 50
IT-EN: 42.67 vs 42.67
EN-DE: 51.6 vs 51.47
DE-EN: 47.22 vs 46.96
EN-FI: 35.88 vs 36.24
FI-EN: 39.62 vs 39.57
EN-ES: 39.47 vs 39.30
ES-EN: 36.43 vs 36.06
Question: What 6 language pairs is experimented on?

Evidence:
raw_evidence: Table TABREF13 shows the inconsistency rates of back translation between Adv-C and our method on MUSE. Compared with Adv-C, our model significantly reduces the inconsistency rates on all language pairs, which explains the overall improvement in Table TABREF12. Table TABREF14 gives several word translation examples. In the first three cases, our regularizer successfully fixes back translation errors. In the fourth case, ensuring cycle consistency does not lead to the correct translation, which explains some errors by our system. In the fifth case, our model finds a related word but not the same word in the back translation, due to the use of cosine similarity for regularization.
FLOAT SELECTED: Table 1: Accuracy on MUSE and Vecmap.

Answers:
EN<->ES
EN<->DE
EN<->IT
EN<->EO
EN<->MS
EN<->FI
Question: Do they compare DeepER against other approaches?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3. Question answering accuracy of RAFAEL with different entity recognition strategies: quantities only (Quant), traditional NER (Nerf, Liner2 ), deep entity recognition (DeepER) and their combination (Hybrid).

Answers:
Yes
Question: What are lyrical topics present in the metal genre?

Evidence:
raw_evidence: Table TABREF10 displays the twenty resulting topics found within the text corpus using LDA. The topics are numbered in descending order according to their prevalence (weight) in the text corpus. For each topic, a qualitative interpretation is given along with the 10 most salient terms.
FLOAT SELECTED: Table 1: Overview of the resulting topics found within the corpus of metal lyrics (n = 124,288) and their correlation to the dimensions hardness and darkness obtained from the audio signal (see section 3.2)

Answers:
Table TABREF10 displays the twenty resulting topics
Question: By how much does SPNet outperforms state-of-the-art abstractive summarization methods on evaluation metrics?

Evidence:
raw_evidence: We show all the models' results in Table TABREF24. We observe that SPNet reaches the highest score in both ROUGE and CIC. Both Pointer-Generator and Transformer achieve high ROUGE scores, but a relative low CIC scores. It suggests that the baselines have more room for improvement on preserving critical slot information. All the scaffolds we propose can be applied to different neural network models. In this work we select Pointer-Generator as our base model in SPNet because we observe that Transformer only has a small improvement over Pointer-Generator but is having a higher cost on training time and computing resources. We observe that SPNet outperforms other methods in all the automatic evaluation metrics with a big margin, as it incorporates all the three semantic scaffolds. Semantic slot contributes the most to SPNet's increased performance, bringing the largest increase on all automatic evaluation metrics.
FLOAT SELECTED: Table 1: Automatic evaluation results on MultiWOZ. We use Pointer-Generator as the base model and gradually add different semantic scaffolds.

Answers:
SPNet vs best baseline:
ROUGE-1: 90.97 vs 90.68
CIC: 70.45 vs 70.25
Question: What are state of the art methods MMM is compared to?

Evidence:
raw_evidence: FLOAT SELECTED: Table 3: Accuracy on the DREAM dataset. Performance marked by ? is reported by (Sun et al. 2019). Numbers in parentheses indicate the accuracy increased by MMM compared to the baselines.

Answers:
FTLM++, BERT-large, XLNet
Question: By how much do they improve the accuracy of inferences over state-of-the-art methods?

Evidence:
raw_evidence: We first compare the perplexity of CWVAE with baseline methods. Perplexity measures the probability of model to regenerate the exact targets, which is particular suitable for evaluating the model performance on one-to-many problem BIBREF20. Further, we employ BLEU score to evaluate the accuracy of generations BIBREF21, and the number of distinct n-gram to evaluate the diversity of generations BIBREF6. The distinct is normalized to $[0, 1]$ by dividing the total number of generated tokens.
FLOAT SELECTED: Table 4: Average perplexity and BLEU score (reported in percentages) for the top 10 generations under each inference dimension of Event2Mind. The the best result for each dimension is emboldened.
FLOAT SELECTED: Table 6: Average perplexity and BLEU scores (reported in percentages) for the top 10 generations under each inference dimension of Atomic. The the best result for each dimension is emboldened.

Answers:
ON Event2Mind, the accuracy of proposed method is improved by  absolute BLUE  2.9,  10.87, 1.79 for xIntent, xReact and oReact respectively.
On Atomic dataset, the accuracy of proposed method is improved by  absolute BLUE 3.95.   4.11, 4.49 for xIntent, xReact and oReact.respectively.