qinchuanhui/UDA-QA huggingface dataset:
Question: What is the performance difference of using a generated summary vs. a user-written one?

Evidence:
raw_evidence: Table TABREF34 and Table TABREF35 show the final results. Our model outperforms all the baseline models and the top-performing models with both generated summary and golden summary, for all the three datasets. In the scenario where golden summaries are used, BiLSTM+self-attention performs the best among all the baselines, which shows that attention is a useful way to integrate summary and review information. Hard-attention receives more supervision information compared with soft-attention, by supervision signals from extractive summaries. However, it underperforms the soft attention model, which indicates that the most salient words for making sentiment classification may not strictly overlap with extractive summaries. This justifies the importance of user written or automatic-generated summary.
FLOAT SELECTED: Table 4: Experimental results. Predicted indicates the use of system-predicted summaries. Star (*) indicates that hard attention model is trained with golden summaries but does not require golden summaries during inference.
FLOAT SELECTED: Table 5: Experimental results. Golden indicates the use of user-written (golden) summaries. Noted that joint modeling methods, such as HSSC (Ma et al., 2018) and SAHSSC (Wang and Ren, 2018), cannot make use of golden summaries during inference time, so their results are excluded in this table.
Experiments ::: Datasets
We empirically compare different methods using Amazon SNAP Review Dataset BIBREF20, which is a part of Stanford Network Analysis Project. The raw dataset consists of around 34 millions Amazon reviews in different domains, such as books, games, sports and movies. Each review mainly contains a product ID, a piece of user information, a plain text review, a review summary and an overall sentiment rating which ranges from 1 to 5. The statistics of our adopted dataset is shown in Table TABREF20. For fair comparison with previous work, we adopt the same partitions used by previous work BIBREF6, BIBREF7, which is, for each domain, the first 1000 samples are taken as the development set, the following 1000 samples as the test set, and the rest as the training set.

Answers:
2.7 accuracy points
Question: Is audio data per language balanced in dataset?

Evidence:
raw_evidence: The data presented in Table (TABREF12) shows the currently available data. Each of the released languages is available for individual download as a compressed directory from the Mozilla Common Voice website. The directory contains six files with Tab-Separated Values (i.e. TSV files), and a single clips subdirectory which contains all of the audio data. Each of the six TSV files represents a different segment of the voice data, with all six having the following column headers: [client_id, path, sentence, up_votes, down_votes, age, gender, accent]. The first three columns refer to an anonymized ID for the speaker, the location of the audio file, and the text that was read. The next two columns contain information on how listeners judged the $<$audio,transcript$>$ pair. The last three columns represent demographic data which was optionally self-reported by the speaker of the audio.
FLOAT SELECTED: Table 1: Current data statistics for Common Voice. Data in italics is as of yet unreleased. Other numbers refer to the data published in the June 12, 2019 release.
We made dataset splits (c.f. Table (TABREF19)) such that one speaker's recordings are only present in one data split. This allows us to make a fair evaluation of speaker generalization, but as a result some training sets have very few speakers, making this an even more challenging scenario. The splits per language were made as close as possible to 80% train, 10% development, and 10% test.
FLOAT SELECTED: Table 2: Data used in the experiments, from an earlier multilingual version of Common Voice. Number of audio clips and unique speakers.

Answers:
No
Question: What is the average length of the claims?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: A summary of PERSPECTRUM statistics
We now provide a brief summary of [wave]390P[wave]415e[wave]440r[wave]465s[wave]485p[wave]525e[wave]535c[wave]595t[wave]610r[wave]635u[wave]660m. The dataset contains about INLINEFORM0 claims with a significant length diversity (Table TABREF19 ). Additionally, the dataset comes with INLINEFORM1 perspectives, most of which were generated through paraphrasing (step 2b). The perspectives which convey the same point with respect to a claim are grouped into clusters. On average, each cluster has a size of INLINEFORM2 which shows that, on average, many perspectives have equivalents. More granular details are available in Table TABREF19 .

Answers:
Average claim length is 8.9 tokens.
Question: By how much, the proposed method improves BiDAF and DCN on SQuAD dataset?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Effect of Character Embedding

Answers:
In terms of F1 score, the Hybrid approach improved by 23.47% and 1.39% on BiDAF and DCN respectively. The DCA approach improved by 23.2% and 1.12% on BiDAF and DCN respectively.
Question: what genres do they songs fall under?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: The number of songs and artists by genre
From the Vagalume's music web page, we collect the song title and lyrics, and the artist name. The genre was collected from the page of styles, which lists all the musical genres and, for each one, all the artists. We selected only 14 genres that we consider as representative Brazilian music, shown in Table TABREF8. Figure FIGREF6 presents an example of the Vagalume's music Web page with the song “Como é grande o meu amor por você”, of the Brazilian singer Roberto Carlos. Green boxes indicate information about music that can be extracted directly from the web page. From this information, the language in which the lyrics are available can be obtained by looking at the icon indicating the flag of Brazil preceded by the “Original” word.

Answers:
Gospel, Sertanejo, MPB, Forró, Pagode, Rock, Samba, Pop, Axé, Funk-carioca, Infantil, Velha-guarda, Bossa-nova and Jovem-guarda
Question: To what other competitive baselines is this approach compared?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Automatic Evaluation Activity/Entity F1 results for baselines and our 3 models (attn means “with attention”). LSTM, HRED and VHRED are reported in Serban et al. (2017a), VHRED (attn) and Reranking-RL in Niu and Bansal (2018a), and the rest are produced by our work. All our four models have statistically significantly higher F1 values (p < 0.001) against VHRED (attn) and MMI.

Answers:
LSTMs with and without attention, HRED, VHRED with and without attention, MMI and Reranking-RL
Question: How much better were results of the proposed models than base LSTM-RNN model?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Automatic Evaluation Activity/Entity F1 results for baselines and our 3 models (attn means “with attention”). LSTM, HRED and VHRED are reported in Serban et al. (2017a), VHRED (attn) and Reranking-RL in Niu and Bansal (2018a), and the rest are produced by our work. All our four models have statistically significantly higher F1 values (p < 0.001) against VHRED (attn) and MMI.

Answers:
on diversity 6.87 and on relevance 4.6 points higher
Question: How much is proposed model better than baselines in performed experiments?

Evidence:
raw_evidence: FLOAT SELECTED: Table 2: The performance of baselines and proposed model on DSTC2 and Maluuba dataset. T imefull is the time spent on training the whole model, T imeDP is the time spent on training the dialogue policy maker.
BPRA Results: As shown in Table TABREF35, most of the models have similar performance on BPRA on these two datasets, which can guarantee a consistent impact on the dialogue policy maker. All the models perform very well in BPRA on DSTC2 dataset. On Maluuba dataset, the BPRA decreases because of the complex domains. We can notice that BPRA of CDM is slightly worse than other models on Maluuba dataset, the reason is that the CDM's dialogue policy maker contains lots of classifications and has the bigger loss than other models because of complex domains, which affects the training of the dialogue belief tracker.
APRA Results: Compared with baselines, GDP achieves the best performance in APRA on two datasets. It can be noted that we do not compare with the E2ECM baseline in APRA. E2ECM only uses a simple classifier to recognize the label of the acts and ignores the parameters information. In our experiment, APRA of E2ECM is slightly better than our method. Considering the lack of parameters of the acts, it's unfair for our GDP method. Furthermore, the CDM baseline considers the parameters of the act. But GDP is far better than CDM in supervised learning and reinforcement learning.
BLEU Results: GDP significantly outperforms the baselines on BLEU. As mentioned above, E2ECM is actually slightly better than GDP in APRA. But in fact, we can find that the language quality of the response generated by GDP is still better than E2ECM, which proves that lack of enough parameters information makes it difficult to find the appropriate sentence template in NLG. It can be found that the BLEU of all models is very poor on Maluuba dataset. The reason is that Maluuba is a human-human task-oriented dialogue dataset, the utterances are very flexible, the natural language generator for all methods is difficult to generate an accurate utterance based on the context. And DSTC2 is a human-machine dialog dataset. The response is very regular so the effectiveness of NLG will be better than that of Maluuba. But from the results, the GDP is still better than the baselines on Maluuba dataset, which also verifies that our proposed method is more accurate in modeling dialogue policy on complex domains than the classification-based methods.

Answers:
most of the models have similar performance on BPRA: DSTC2 (+0.0015), Maluuba (+0.0729)
GDP achieves the best performance in APRA: DSTC2 (+0.2893), Maluuba (+0.2896)
GDP significantly outperforms the baselines on BLEU: DSTC2 (+0.0791), Maluuba (+0.0492)
Question: Do they evaluate only on English datasets?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Tweet examples with emojis. The sentiment ground truth is given in the second column. The examples show that inconsistent sentiments exist between emojis and texts.
We construct our own Twitter sentiment dataset by crawling tweets through the REST API which consists of 350,000 users and is magnitude larger comparing to previous work. We collect up to 3,200 tweets from each user and follow the standard tweet preprocessing procedures to remove the tweets without emojis and tweets containing less than ten words, and contents including the urls, mentions, and emails.
For acquiring the sentiment annotations, we first use Vader which is a rule-based sentiment analysis algorithm BIBREF17 for text tweets only to generate weak sentiment labels. The algorithm outputs sentiment scores ranging from -1 (negative) to 1 (positive) with neutral in the middle. We consider the sentiment analysis as a binary classification problem (positive sentiment and negative sentiment), we filter out samples with weak prediction scores within INLINEFORM0 and keep the tweets with strong sentiment signals. Emoji occurrences are calculated separately for positive tweets and negative tweets, and threshold is set to 2,000 to further filter out emojis which are less frequently used in at least one type of sentimental text. In the end, we have constructed a dataset with 1,492,065 tweets and 55 frequently used emojis in total. For the tweets with an absolute sentiment score over 0.70, we keep the auto-generated sentiment label as ground truth because the automatic annotation is reliable with high sentiment scores. On the other hand, we select a subset of the tweets with absolute sentiment scores between INLINEFORM1 for manual labeling by randomly sampling, following the distribution of emoji occurrences where each tweet is labeled by two graduate students. Tweets are discarded if the two annotations disagree with each other or they are labeled as neutral. In the end, we have obtained 4,183 manually labeled tweets among which 60% are used for fine-tuning and 40% are used for testing purposes. The remainder of the tweets with automatic annotations are divided into three sets: 60% are used for pre-training the bi-sense and conventional emoji embedding, 10% for validation and 30% are for testing. We do not include a “neutral” class because it is difficult to obtain valid neutral samples. For auto-generated labels, the neutrals are the samples with low absolute confidence scores and their sentiments are more likely to be model failures other than “true neutrals”. Moreover, based on the human annotations, most of the tweets with emojis convey non-neutral sentiment and only few neutral samples are observed during the manual labeling which are excluded from the manually labeled subset.

Answers:
Yes
Question: Do they evaluate only on English datasets?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Example Twitter Customer Service Conversation

Answers:
Yes
Question: Which data sources do they use?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: An approximate number of sentence pairs for each task.

Answers:
- En-Fr (WMT14)
- En-De (WMT15)
- Skipthought (BookCorpus)
- AllNLI (SNLI + MultiNLI)
- Parsing (PTB + 1-billion word)
Question: What approach performs better in experiments global latent or sequence of fine-grained latent variables?

Evidence:
raw_evidence: Compare to baseline models, the GVT achieves relatively lower reconstruction PPL, which suggests that the global latent variable contains rich latent information (e.g., topic) for response generation. Meanwhile, the sequential latent variables of the SVT encode fine-grained latent information and further improve the reconstruction PPL.
FLOAT SELECTED: Table 1: Results of Variational Transformer compared to baselines on automatic and human evaluations.

Answers:
PPL: SVT
Diversity: GVT
Embeddings Similarity: SVT
Human Evaluation: SVT
Question: What previous approaches did this method outperform?

Evidence:
raw_evidence: The POS tagging and lemmatization results are presented in Table TABREF44. The word2vec word embeddings (WE) considerably increase performance compared to the baseline, especially in POS tagging. When only Flair embeddings are added to the baseline, we also observe an improvement, but not as high. We hypothesise that the lower performance (in contrast with the results reported in BIBREF2) is caused by the size of the training data, because we train the word2vec WE on considerably larger dataset than the Czech Flair model. However, when WE and Flair embeddings are combined, performance moderately increases, demonstrating that the two embedding methods produce at least partially complementary representations.
Table TABREF44 compares our best model with state-of-the-art results on PDT 2.0 (note that some of the related work used only a subset of PDT 2.0 and/or utilized gold morphological annotation). To our best knowledge, research on PDT parsing was performed mostly in the first decade of this century, therefore even our baseline model substantially surpasses previous works. Our best model with contextualized embeddings achieves nearly 50% error reduction both in UAS and LAS.
Table TABREF47 shows the performance of analyzed embedding methods in a joint model performing POS tagging, lemmatization, and dependency parsing on Czech PDT UD 2.3 treebank. This treebank is derived from PDT 3.5 a-layer, with original POS tags kept in XPOS, and the dependency trees and lemmas modified according to UD guidelines.
Table TABREF47 shows NER results (F1 score) on CNEC 1.1 and CNEC 2.0. Our sequence-to-sequence (seq2seq) model which captures the nested entities, clearly surpasses the current Czech NER state of the art. Furthermore, significant improvement is gained when adding the contextualized word embeddings (BERT and Flair) as optional input to the LSTM encoder. The strongest model is a combination of the sequence-to-sequence architecture with both BERT and Flair contextual word embeddings.
FLOAT SELECTED: Table 2. POS tagging and lemmatization results (accuracy) on PDT 3.5. Bold indicates the best result, italics related work. †Reported on PDT 2.0, which has the same underlying corpus, with minor changes in morphological annotation (our model results differ at 0.1% on PDT 2.0).
FLOAT SELECTED: Table 4. Dependency tree parsing results on PDT 2.0 a-layer. Bold indicates the best result, italics related work. †Possibly using gold POS tags. ‡Results as of 23 Mar 2019.
FLOAT SELECTED: Table 5. Czech PDT UD 2.3 results for POS tagging (UPOS: universal POS, XPOS: languagespecific POS, UFeats: universal morphological features), lemmatization and dependency parsing (UAS, LAS, MLAS, and BLEX scores). Bold indicates the best result, italics related work.
FLOAT SELECTED: Table 6. Named entity recognition results (F1) on the Czech Named Entity Corpus. Bold indicates the best result, italics related work.

Answers:
Table TABREF44, Table TABREF44, Table TABREF47, Table TABREF47
Question: Is the new model evaluated on the tasks that BERT and ELMo are evaluated on?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: GLUE results with and without STILTs, fine-tuning on full training data of each target task. Bold marks the best within each section. Strikethrough indicates cases where the intermediate task is the same as the target task—we substitute the baseline result for that cell. A.Ex is the average excluding MNLI and QQP because of the overlap with intermediate tasks. See text for discussion of WNLI results. Test results on STILTs uses the supplementary training regime for each task based on the performance on the development set, corresponding to the numbers shown in Best of Each. The aggregated GLUE scores differ from the public leaderboard because we report performance on QNLIv1.

Answers:
Yes
Question: Which language has the lowest error rate reduction?

Evidence:
raw_evidence: FLOAT SELECTED: Table 9 Character error rates on the validation data using successively more of the system components described above for English (en), Spanish (es), German (de), Arabic (ar), Korean (ko), Thai (th), Hindi (hi), and Chinese (zh) along with the respective number of items and characters in the test sets. Average latencies for all languages and models were computed on an Intel Xeon E5-2690 CPU running at 2.6GHz.

Answers:
thai
Question: How big is dataset used?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Statistics of the datasets.

Answers:
553,451 documents
Question: What are the baseline models?

Evidence:
raw_evidence: We compared our proposed approaches to a standard CNN that exploits a single set of word embeddings BIBREF3 . We also compared to a baseline of simply concatenating embeddings for each word to form long vector inputs. We refer to this as Concatenation-CNN C-CNN. For all multiple embedding approaches (C-CNN, MG-CNN and MGNC-CNN), we explored two combined sets of embedding: word2vec+Glove, and word2vec+syntactic, and one three sets of embedding: word2vec+Glove+syntactic. For all models, we tuned the l2 norm constraint INLINEFORM0 over the range INLINEFORM1 on a validation set. For instantiations of MGNC-CNN in which we exploited two embeddings, we tuned both INLINEFORM2 , and INLINEFORM3 ; where we used three embedding sets, we tuned INLINEFORM4 and INLINEFORM5 .
FLOAT SELECTED: Table 1: Results mean (min, max) achieved with each method. w2v:word2vec. Glv:GloVe. Syn: Syntactic embedding. Note that we experiment with using two and three sets of embeddings jointly, e.g., w2v+Syn+Glv indicates that we use all three of these.

Answers:
MC-CNN
MVCNN
CNN
Question: By how much of MGNC-CNN out perform the baselines?

Evidence:
raw_evidence: FLOAT SELECTED: Table 1: Results mean (min, max) achieved with each method. w2v:word2vec. Glv:GloVe. Syn: Syntactic embedding. Note that we experiment with using two and three sets of embeddings jointly, e.g., w2v+Syn+Glv indicates that we use all three of these.
We repeated each experiment 10 times and report the mean and ranges across these. This replication is important because training is stochastic and thus introduces variance in performance BIBREF4 . Results are shown in Table TABREF2 , and the corresponding best norm constraint value is shown in Table TABREF2 . We also show results on Subj, SST-1 and SST-2 achieved by the more complex model of BIBREF11 for comparison; this represents the state-of-the-art on the three datasets other than TREC.

Answers:
In terms of Subj the Average MGNC-CNN is better than the average score of baselines by 0.5.  Similarly, Scores of SST-1, SST-2, and TREC where MGNC-CNN has similar improvements. 
In case of Irony the difference is about 2.0.
Question: What are the comparable alternative architectures?

Evidence:
raw_evidence: We compared our proposed approaches to a standard CNN that exploits a single set of word embeddings BIBREF3 . We also compared to a baseline of simply concatenating embeddings for each word to form long vector inputs. We refer to this as Concatenation-CNN C-CNN. For all multiple embedding approaches (C-CNN, MG-CNN and MGNC-CNN), we explored two combined sets of embedding: word2vec+Glove, and word2vec+syntactic, and one three sets of embedding: word2vec+Glove+syntactic. For all models, we tuned the l2 norm constraint INLINEFORM0 over the range INLINEFORM1 on a validation set. For instantiations of MGNC-CNN in which we exploited two embeddings, we tuned both INLINEFORM2 , and INLINEFORM3 ; where we used three embedding sets, we tuned INLINEFORM4 and INLINEFORM5 .
More similar to our work, Yin and Schütze yin-schutze:2015:CoNLL proposed MVCNN for sentence classification. This CNN-based architecture accepts multiple word embeddings as inputs. These are then treated as separate `channels', analogous to RGB channels in images. Filters consider all channels simultaneously. MVCNN achieved state-of-the-art performance on multiple sentence classification tasks. However, this model has practical drawbacks. (i) MVCNN requires that input word embeddings have the same dimensionality. Thus to incorporate a second set of word vectors trained on a corpus (or using a model) of interest, one needs to either find embeddings that happen to have a set number of dimensions or to estimate embeddings from scratch. (ii) The model is complex, both in terms of implementation and run-time. Indeed, this model requires pre-training and mutual-learning and requires days of training time, whereas the simple architecture we propose requires on the order of an hour (and is easy to implement).
FLOAT SELECTED: Table 1: Results mean (min, max) achieved with each method. w2v:word2vec. Glv:GloVe. Syn: Syntactic embedding. Note that we experiment with using two and three sets of embeddings jointly, e.g., w2v+Syn+Glv indicates that we use all three of these.

Answers:
standard CNN, C-CNN, MVCNN