qinchuanhui/UDA-QA huggingface dataset:
Question: How much better is performance of proposed method than state-of-the-art methods in experiments?

Evidence:
raw_evidence: Experimental results of entity classification on the test sets of all the datasets is shown in Table TABREF25. The results is clearly demonstrate that our proposed method significantly outperforms state-of-art results on accuracy for three datasets. For more in-depth performance analysis, we note: (1) Among all baselines, Path-based methods and Attribute-incorporated methods outperform three typical methods. This indicates that incorporating extra information can improve the knowledge graph embedding performance; (2) Four variants of KANE always outperform baseline methods. The main reasons why KANE works well are two fold: 1) KANE can capture high-order structural information of KGs in an efficient, explicit manner and passe these information to their neighboring; 2) KANE leverages rich information encoded in attribute triples. These rich semantic information can further improve the performance of knowledge graph; (3) The variant of KANE that use LSTM Encoder and Concatenation aggregator outperform other variants. The main reasons is that LSTM encoder can distinguish the word order and concatenation aggregator combine all embedding of multi-head attention in a higher leaver feature space, which can obtain sufficient expressive power.
FLOAT SELECTED: Table 2: Entity classification results in accuracy. We run all models 10 times and report mean ± standard deviation. KANE significantly outperforms baselines on FB24K, DBP24K and Game30K.

Answers:
Accuracy of best proposed method KANE (LSTM+Concatenation) are 0.8011, 0.8592, 0.8605 compared to best state-of-the art method R-GCN + LR 0.7721, 0.8193, 0.8229 on three datasets respectively.
Question: What baseline model is used?

Evidence:
raw_evidence: FLOAT SELECTED: Table 5: Performance of human evaluators and our classifiers (trained on all features), for Dataset-H as the test set

Answers:
Human evaluators
Question: What stylistic features are used to detect drunk texts?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Our Feature Set for Drunk-texting Prediction
raw_evidence: FLOAT SELECTED: Table 1: Our Feature Set for Drunk-texting Prediction

Answers:
LDA unigrams (Presence/Count), POS Ratio, #Named Entity Mentions, #Discourse Connectors,  Spelling errors, Repeated characters, Capitalisation, Length,  Emoticon (Presence/Count ) 
 and Sentiment Ratio
LDA unigrams (Presence/Count), POS Ratio, #Named Entity Mentions, #Discourse Connectors, Spelling errors, Repeated characters, Capitalization, Length, Emoticon (Presence/Count), Sentiment Ratio.
Question: What is the accuracy of the proposed technique?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: TUPLEINF is significantly better at structured reasoning than TABLEILP.9

Answers:
51.7 and 51.6 on 4th and 8th grade question sets with no curated knowledge. 47.5 and 48.0 on 4th and 8th grade question sets when both solvers are given the same knowledge
Question: How much better was the BLSTM-CNN-CRF than the BLSTM-CRF?

Evidence:
raw_evidence: Table 2 shows our experiments on two models with and without different pre-trained word embedding – KP means the Kyubyong Park’s pre-trained word embeddings and EG means Edouard Grave’s pre-trained word embeddings.
FLOAT SELECTED: Table 2. F1 score of two models with different pre-trained word embeddings

Answers:
Best BLSTM-CNN-CRF had F1 score 86.87 vs 86.69 of best BLSTM-CRF
Question: How much performance gap between their approach and the strong handcrafted method?

Evidence:
raw_evidence: FLOAT SELECTED: Table 4: Compared with other systems (bold is best).

Answers:
0.007 MAP on Task A, 0.032 MAP on Task B, 0.055 MAP on Task C
Question: How big is their model?

Evidence:
raw_evidence: To figure out whether the proposed AEN-GloVe is a lightweight alternative of recurrent models, we study the model size of each model on the Restaurant dataset. Statistical results are reported in Table TABREF37 . We implement all the compared models base on the same source code infrastructure, use the same hyperparameters, and run them on the same GPU .
FLOAT SELECTED: Table 3: Model sizes. Memory footprints are evaluated on the Restaurant dataset. Lowest 2 are in bold.

Answers:
Proposed model has 1.16 million parameters and 11.04 MB.
Question: Which tested technique was the worst performer?

Evidence:
raw_evidence: FLOAT SELECTED: Table 4: Benchmark results (averaged 5-fold cross validation)
We computed bag-of-words-based benchmarks using the following methods:
Classification with TF-IDF + Linear SVM (TF-IDF + SVM)
Classification with Depeche++ Emotion lexicons BIBREF12 + Linear SVM (Depeche + SVM)
Classification with NRC Emotion lexicons BIBREF13, BIBREF14 + Linear SVM (NRC + SVM)
Combination of TF-IDF and NRC Emotion lexicons (TF-NRC + SVM)

Answers:
Depeche + SVM
Question: What is difference in peformance between proposed model and state-of-the art on other question types?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2. Performance of different models on DROP’s development set in terms of Exact Match (EM) and F1.

Answers:
For single-span questions, the proposed LARGE-SQUAD improve performance of the MTMSNlarge baseline for 2.1 EM and 1.55 F1.
For number type question,  MTMSNlarge baseline  have improvement over LARGE-SQUAD for 3,11  EM and  2,98 F1. 
For date question,  LARGE-SQUAD have improvements in 2,02 EM but MTMSNlarge have improvement of 4,39 F1.
Question: What is the performance of proposed model on entire DROP dataset?

Evidence:
raw_evidence: Table TABREF25 shows the results on DROP's test set, with our model being the best overall as of the time of writing, and not just on multi-span questions.
FLOAT SELECTED: Table 3. Comparing test and development set results of models from the official DROP leaderboard

Answers:
The proposed model achieves  EM 77,63 and F1 80,73  on the test and EM  76,95 and  F1 80,25 on the dev
Question: Does the system trained only using XR loss outperform the fully supervised neural system?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Average accuracies and Macro-F1 scores over five runs with random initialization along with their standard deviations. Bold: best results or within std of them. ∗ indicates that the method’s result is significantly better than all baseline methods, † indicates that the method’s result is significantly better than all baselines methods that use the aspect-based data only, with p < 0.05 according to a one-tailed unpaired t-test. The data annotations S, N and A indicate training with Sentence-level, Noisy sentence-level and Aspect-level data respectively. Numbers for TDLSTM+Att,ATAE-LSTM,MM,RAM and LSTM+SynATT+TarRep are from (He et al., 2018a). Numbers for Semisupervised are from (He et al., 2018b).

Answers:
Yes
Question: How accurate is the aspect based sentiment classifier trained only using the XR loss?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Average accuracies and Macro-F1 scores over five runs with random initialization along with their standard deviations. Bold: best results or within std of them. ∗ indicates that the method’s result is significantly better than all baseline methods, † indicates that the method’s result is significantly better than all baselines methods that use the aspect-based data only, with p < 0.05 according to a one-tailed unpaired t-test. The data annotations S, N and A indicate training with Sentence-level, Noisy sentence-level and Aspect-level data respectively. Numbers for TDLSTM+Att,ATAE-LSTM,MM,RAM and LSTM+SynATT+TarRep are from (He et al., 2018a). Numbers for Semisupervised are from (He et al., 2018b).

Answers:
BiLSTM-XR-Dev Estimation accuracy is 83.31 for SemEval-15 and 87.68 for SemEval-16.
BiLSTM-XR accuracy is 83.31 for SemEval-15 and 88.12 for SemEval-16.
Question: What metrics other than entity tagging are compared?

Evidence:
raw_evidence: Following BIBREF7, we then evaluated a metric that combines intent and entities, computed by simply summing up the two confusion matrices (Table TABREF23). Results highlight the contribution of the entity tagging task, where HERMIT outperforms the other approaches. Paired-samples t-tests were conducted to compare the HERMIT combined F1 against the other systems. The statistical analysis shows a significant improvement over Rasa $[Z=-2.803, p = .005]$, Dialogflow $[Z=-2.803, p = .005]$, LUIS $[Z=-2.803, p = .005]$ and Watson $[Z=-2.803, p = .005]$.
FLOAT SELECTED: Table 4: Comparison of HERMIT with the results in (Liu et al., 2019) by combining Intent and Entity.
In this section we report the experiments performed on the ROMULUS dataset (Table TABREF27). Together with the evaluation metrics used in BIBREF7, we report the span F1, computed using the CoNLL-2000 shared task evaluation script, and the Exact Match (EM) accuracy of the entire sequence of labels. It is worth noticing that the EM Combined score is computed as the conjunction of the three individual predictions – e.g., a match is when all the three sequences are correct.
Results in terms of EM reflect the complexity of the different tasks, motivating their position within the hierarchy. Specifically, dialogue act identification is the easiest task ($89.31\%$) with respect to frame ($82.60\%$) and frame element ($79.73\%$), due to the shallow semantics it aims to catch. However, when looking at the span F1, its score ($89.42\%$) is lower than the frame element identification task ($92.26\%$). What happens is that even though the label set is smaller, dialogue act spans are supposed to be longer than frame element ones, sometimes covering the whole sentence. Frame elements, instead, are often one or two tokens long, that contribute in increasing span based metrics. Frame identification is the most complex task for several reasons. First, lots of frame spans are interlaced or even nested; this contributes to increasing the network entropy. Second, while the dialogue act label is highly related to syntactic structures, frame identification is often subject to the inherent ambiguity of language (e.g., get can evoke both Commerce_buy and Arriving). We also report the metrics in BIBREF7 for consistency. For dialogue act and frame tasks, scores provide just the extent to which the network is able to detect those labels. In fact, the metrics do not consider any span information, essential to solve and evaluate our tasks. However, the frame element scores are comparable to the benchmark, since the task is very similar.

Answers:
We also report the metrics in BIBREF7 for consistency, we report the span F1,  Exact Match (EM) accuracy of the entire sequence of labels, metric that combines intent and entities
Question: What is the results of multimodal compared to unimodal models?

Evidence:
raw_evidence: Table TABREF31 shows the F-score, the Area Under the ROC Curve (AUC) and the mean accuracy (ACC) of the proposed models when different inputs are available. $TT$ refers to the tweet text, $IT$ to the image text and $I$ to the image. It also shows results for the LSTM, for the Davison method proposed in BIBREF7 trained with MMHS150K, and for random scores. Fig. FIGREF32 shows the Precision vs Recall plot and the ROC curve (which plots the True Positive Rate vs the False Positive Rate) of the different models.
FLOAT SELECTED: Table 1. Performance of the proposed models, the LSTM and random scores. The Inputs column indicate which inputs are available at training and testing time.

Answers:
Unimodal LSTM vs Best Multimodal (FCM)
- F score: 0.703 vs 0.704
- AUC: 0.732 vs 0.734 
- Mean Accuracy: 68.3 vs 68.4
Question: What were their performance results?

Evidence:
raw_evidence: FLOAT SELECTED: Table 6: Comparison of ACC of our proposed methods and some other non-biased models on three datasets. For LPI, we project the text under the best dimension as described in Section 4.3. For both bi-LSTM and bi-GRU based clustering methods, the binary codes generated from LPI are used to guide the learning of bi-LSTM/bi-GRU models.
FLOAT SELECTED: Table 7: Comparison of NMI of our proposed methods and some other non-biased models on three datasets. For LPI, we project the text under the best dimension as described in Section 4.3. For both bi-LSTM and bi-GRU based clustering methods, the binary codes generated from LPI are used to guide the learning of bi-LSTM/bi-GRU models.

Answers:
On SearchSnippets dataset ACC 77.01%, NMI 62.94%, on StackOverflow dataset ACC 51.14%, NMI 49.08%, on Biomedical dataset ACC 43.00%, NMI 38.18%
Question: By how much did they outperform the other methods?

Evidence:
raw_evidence: FLOAT SELECTED: Table 6: Comparison of ACC of our proposed methods and some other non-biased models on three datasets. For LPI, we project the text under the best dimension as described in Section 4.3. For both bi-LSTM and bi-GRU based clustering methods, the binary codes generated from LPI are used to guide the learning of bi-LSTM/bi-GRU models.
FLOAT SELECTED: Table 7: Comparison of NMI of our proposed methods and some other non-biased models on three datasets. For LPI, we project the text under the best dimension as described in Section 4.3. For both bi-LSTM and bi-GRU based clustering methods, the binary codes generated from LPI are used to guide the learning of bi-LSTM/bi-GRU models.

Answers:
on SearchSnippets dataset by 6.72% in ACC, by 6.94% in NMI; on Biomedical dataset by 5.77% in ACC, 3.91% in NMI
Question: What are all 15 types of modifications ilustrated in the dataset?

Evidence:
raw_evidence: We selected 15 modifications types to collect COSTRA 1.0. They are presented in annotationinstructions.
FLOAT SELECTED: Table 2: Sentences transformations requested in the second round of annotation with the instructions to the annotators. The annotators were given no examples (with the exception of nonsense) not to be influenced as much as in the first round.

Answers:
- paraphrase 1
- paraphrase 2
- different meaning
- opposite meaning
- nonsense
- minimal change
- generalization
- gossip
- formal sentence
- non-standard sentence
- simple sentence
- possibility
- ban
- future
- past
Question: What were their results on the three datasets?

Evidence:
raw_evidence: FLOAT SELECTED: Table IV ACCURACY OF DIFFERENT MODELS FOR BINARY CLASSIFICATION

Answers:
accuracy of 86.63 on STS, 85.14 on Sanders and 80.9 on HCR
Question: What semantic rules are proposed?

Evidence:
raw_evidence: In Twitter social networking, people express their opinions containing sub-sentences. These sub-sentences using specific PoS particles (Conjunction and Conjunctive adverbs), like "but, while, however, despite, however" have different polarities. However, the overall sentiment of tweets often focus on certain sub-sentences. For example:
@lonedog bwahahah...you are amazing! However, it was quite the letdown.
@kirstiealley my dentist is great but she's expensive...=(
In two tweets above, the overall sentiment is negative. However, the main sentiment is only in the sub-sentences following but and however. This inspires a processing step to remove unessential parts in a tweet. Rule-based approach can assists these problems in handling negation and dealing with specific PoS particles led to effectively affect the final output of classification BIBREF11 BIBREF16 . BIBREF11 summarized a full presentation of their semantic rules approach and devised ten semantic rules in their hybrid approach based on the presentation of BIBREF16 . We use five rules in the semantic rules set because other five rules are only used to compute polarity of words after POS tagging or Parsing steps. We follow the same naming convention for rules utilized by BIBREF11 to represent the rules utilized in our proposed method. The rules utilized in the proposed method are displayed in Table TABREF15 in which is included examples from STS Corpus and output after using the rules. Table TABREF16 illustrates the number of processed sentences on each dataset.
FLOAT SELECTED: Table I SEMANTIC RULES [12]

Answers:
rules that compute polarity of words after POS tagging or parsing steps
Question: What was the performance of both approaches on their dataset?

Evidence:
raw_evidence: FLOAT SELECTED: Table 4. EER(%) results of the i-vector and x-vector systems trained on VoxCeleb and evaluated on three evaluation sets.

Answers:
ERR of 19.05 with i-vectors and 15.52 with x-vectors
Question: What genres are covered?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1. The distribution over genres.

Answers:
genre, entertainment, interview, singing, play, movie, vlog, live broadcast, speech, drama, recitation and advertisement
Question: Which of the two speech recognition models works better overall on CN-Celeb?

Evidence:
raw_evidence: FLOAT SELECTED: Table 4. EER(%) results of the i-vector and x-vector systems trained on VoxCeleb and evaluated on three evaluation sets.

Answers:
x-vector
Question: By how much is performance on CN-Celeb inferior to performance on VoxCeleb?

Evidence:
raw_evidence: FLOAT SELECTED: Table 4. EER(%) results of the i-vector and x-vector systems trained on VoxCeleb and evaluated on three evaluation sets.

Answers:
For i-vector system, performances are 11.75% inferior to voxceleb. For x-vector system, performances are 10.74% inferior to voxceleb
Question: On what datasets is the new model evaluated on?

Evidence:
raw_evidence: SST BIBREF25 SST (Stanford Sentiment Treebank) is a dataset for sentiment classification on movie reviews, which are annotated with five labels (SST5: very positive, positive, neutral, negative, or very negative) or two labels (SST2: positive or negative).
Subj BIBREF26 Subj (Subjectivity dataset) is annotated with whether a sentence is subjective or objective.
MPQA BIBREF27 MPQA Opinion Corpus is an opinion polarity detection dataset of short phrases rather than sentences, which contains news articles from a wide variety of news sources manually annotated for opinions and other private states (i.e., beliefs, emotions, sentiments, speculations, etc.).
RT BIBREF28 RT is another movie review sentiment dataset contains a collection of short review excerpts from Rotten Tomatoes collected by Bo Pang and Lillian Lee.
TREC BIBREF29 TREC is a dataset for classification of the six question types (whether the question is about person, location, numeric information, etc.).
FLOAT SELECTED: Table 2: Accuracies of different methods for various benchmarks on two classifier architectures. CBERT, which represents conditional BERT, performs best on two classifier structures over six datasets. “w/” represents “with”, lines marked with “*” are experiments results from Kobayashi(Kobayashi, 2018).

Answers:
SST (Stanford Sentiment Treebank), Subj (Subjectivity dataset), MPQA Opinion Corpus, RT is another movie review sentiment dataset, TREC is a dataset for classification of the six question types
Question: How do the authors measure performance?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Accuracies of different methods for various benchmarks on two classifier architectures. CBERT, which represents conditional BERT, performs best on two classifier structures over six datasets. “w/” represents “with”, lines marked with “*” are experiments results from Kobayashi(Kobayashi, 2018).

Answers:
Accuracy across six datasets
Question: Are other pretrained language models also evaluated for contextual augmentation? 

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Accuracies of different methods for various benchmarks on two classifier architectures. CBERT, which represents conditional BERT, performs best on two classifier structures over six datasets. “w/” represents “with”, lines marked with “*” are experiments results from Kobayashi(Kobayashi, 2018).

Answers:
No
Question: What is the latest paper covered by this survey?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: Existing NQG models with their best-reported performance on SQuAD. Legend: QW: question word generation, PC: paragraph-level context, CP: copying mechanism, LF: linguistic features, PG: policy gradient.

Answers:
Kim et al. (2019)
Question: What is the state of the art?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Results of RobBERT fine-tuned on several downstream tasks compared to the state of the art on the tasks. For accuracy, we also report the 95% confidence intervals. (Results annotated with * from van der Burgh and Verberne (2019), ** = from de Vries et al. (2019), *** from Allein et al. (2020))
We replicated the high-level sentiment analysis task used to evaluate BERTje BIBREF8 to be able to compare our methods. This task uses a dataset called Dutch Book Reviews Dataset (DBRD), in which book reviews scraped from hebban.nl are labeled as positive or negative BIBREF19. Although the dataset contains 118,516 reviews, only 22,252 of these reviews are actually labeled as positive or negative. The DBRD dataset is already split in a balanced 10% test and 90% train split, allowing us to easily compare to other models trained for solving this task. This dataset was released in a paper analysing the performance of an ULMFiT model (Universal Language Model Fine-tuning for Text Classification model) BIBREF19.

Answers:
BERTje BIBREF8, an ULMFiT model (Universal Language Model Fine-tuning for Text Classification model) BIBREF19., mBERT
Question: How does the model compare with the MMR baseline?

Evidence:
raw_evidence: Various classes of NP-hard problems involving a submodular and non-decreasing function can be solved approximately by polynomial time algorithms with provable approximation factors. Algorithms "Detection of hypergraph transversals for text summarization" and "Detection of hypergraph transversals for text summarization" are our core methods for the detection of approximations of maximal budgeted hypergraph transversals and minimal soft hypergraph transversals, respectively. In each case, a transversal is found and the summary is formed by extracting and aggregating the associated sentences. Algorithm "Detection of hypergraph transversals for text summarization" is based on an adaptation of an algorithm presented in BIBREF30 for the maximization of submodular functions under a Knaspack constraint. It is our primary transversal-based summarization model, and we refer to it as the method of Transversal Summarization with Target Length (TL-TranSum algorithm). Algorithm "Detection of hypergraph transversals for text summarization" is an application of the algorithm presented in BIBREF20 for solving the submodular set covering problem. We refer to it as Transversal Summarization with Target Coverage (TC-TranSum algorithm). Both algorithms produce transversals by iteratively appending the node inducing the largest increase in the total weight of the covered hyperedges relative to the node weight. While long sentences are expected to cover more themes and induce a larger increase in the total weight of covered hyperedges, the division by the node weights (i.e. the sentence lengths) balances this tendency and allows the inclusion of short sentences as well. In contrast, the methods of sentence selection based on a maximal relevance and a minimal redundancy such as, for instance, the maximal marginal relevance approach of BIBREF31 , tend to favor the selection of long sentences only. The main difference between algorithms "Detection of hypergraph transversals for text summarization" and "Detection of hypergraph transversals for text summarization" is the stopping criterion: in algorithm "Detection of hypergraph transversals for text summarization" , the approximate minimal soft transversal is obtained whenever the targeted hyperedge coverage is reached while algorithm "Detection of hypergraph transversals for text summarization" appends a given sentence to the approximate maximal budgeted transversal only if its addition does not make the summary length exceed the target length $L$ .
FLOAT SELECTED: Table 2: Comparison with related graph- and hypergraph-based summarization systems.

Answers:
 Moreover, our TL-TranSum method also outperforms other approaches such as MaxCover ( $5\%$ ) and MRMR ( $7\%$ )