qinchuanhui/UDA-QA huggingface dataset:
Question: What are the research questions posed in the paper regarding EAC studies?

Evidence:
raw_evidence: In this work, a systematic review of emotionally-aware chatbots is proposed. We focus on three main issues, including, how to incorporate affective information into chatbots, what are resources that available and can be used to build EAC, and how to evaluate EAC performance. The rise of EAC was started by Parry, which uses a simple rule-based approach. Now, most of EAC are built by using a neural-based approach, by exploiting emotion classifier to detect emotion contained in the text. In the modern era, the development of EAC gains more attention since Emotion Generation Challenge shared task on NLPCC 2017. In this era, most EAC is developed by adopting encoder-decoder architecture with sequence-to-sequence learning. Some variant of the recurrent neural network is used in the learning process, including long-short-term memory (LSTM) and gated recurrent unit (GRU). There are also some datasets available for developing EAC now. However, the datasets are only available in English and Chinese. These datasets are gathered from various sources, including social media, online website and manual construction by crowdsourcing. Overall, the difference between these datasets and the common datasets for building chatbot is the presence of an emotion label. In addition, we also investigate the available affective resources which usually use in the emotion classification task. In this part, we only focus on English resources and found several resources from the old one such as LIWC and Emolex to the new one, including DepecheMood and EmoWordNet. In the final part, we gather information about how to evaluate the performance of EAC, and we can classify the approach into two techniques, including qualitative and quantitative assessment. For qualitative assessment, most studies used ISO 9241, which covers several aspects such as efficiency, effectiveness, and satisfaction. While in quantitative analysis, two techniques can be used, including automatic evaluation (by using perplexity) and manual evaluation (involving human judgement). Overall, we can see that effort to humanize chatbots by incorporation affective aspect is becoming the hot topic now. We also predict that this development will continue by going into multilingual perspective since up to now every chatbot only focusing on one language. Also, we think that in the future the studies of humanizing chatbot are not only utilized emotion information but will also focus on a contextual-aware chatbot.

Answers:
how to incorporate affective information into chatbots, what are resources that available and can be used to build EAC, and how to evaluate EAC performance
Question: What evaluation metrics did they use?

Evidence:
raw_evidence: Given the mapped embedding spaces, the translations are retrieved using a distance metric, with Cross-Lingual Similarity Scaling BIBREF12 as the most common and best performing in the literature. Intuitively, CSLS decreases the scores of pairs that lie in dense areas, increasing the scores of rarer words (which are harder to align). The retrieved pairs are compared to the gold standard and evaluated using precision at $k$ (P@$k$, evaluating how often the correct translation is within the $k$ retrieved nearest neighbours of the query). Throughout this work we report P@1, which is equivalent to accuracy, but we also provide results with P@5 and P@10 in the Appendix.

Answers:
we report P@1, which is equivalent to accuracy, we also provide results with P@5 and P@10 in the Appendix
Question: What is triangulation?

Evidence:
raw_evidence: Conclusion

Answers:
Answer with content missing: (Chapter 3) The concept can be easily explained with an example, visualized in Figure 1. Consider the Portuguese (Pt) word trabalho which, according to the MUSE Pt–En dictionary, has the words job and work as possible En translations. In turn, these two En words can be translated to 4 and 5 Czech (Cs) words respectively. By utilizing the transitive property (which translation should exhibit) we can identify the set of 7 possible Cs translations for the Pt word trabalho.
Question: Did they experiment with the dataset on some tasks?

Evidence:
raw_evidence: In this work, we publish TWNERTC dataset in which named entities and categories of sentences have been automatically annotated. We use Turkish Wikipedia dumps as the text source and Freebase to construct a large-scale gazetteers to map fine-grained types to entities. To overcome noisy and ambiguous data, we leverage domain information which is given by Freebase and develop domain-independent and domain-dependent methodologies. All versions of datasets can be downloaded from our project web-page. Our main contributions are (1) the publication of Turkish corpus for coarse-grained and fine-grained NER, and TC research, (2) six different versions of corpus according to noise reduction methodology and entity types, (3) an analysis of the corpus and (4) benchmark comparisons for NER and TC tasks against human annotators. To the best of our knowledge, these datasets are the largest datasets available for Turkish NER ad TC tasks.

Answers:
Yes
Question: Do they use pretrained word embeddings?

Evidence:
raw_evidence: In this study, we tackle a task of describing (defining) a phrase when given its local context as BIBREF2 , while allowing access to other usage examples via word embeddings trained from massive text (global contexts) BIBREF0 , BIBREF1 . We present LOG-Cad, a neural network-based description generator (Figure FIGREF1 ) to directly solve this task. Given a word with its context, our generator takes advantage of the target word's embedding, pre-trained from massive text (global contexts), while also encoding the given local context, combining both to generate a natural language description. The local and global contexts complement one another and are both essential; global contexts are crucial when local contexts are short and vague, while the local context is crucial when the target phrase is polysemous, rare, or unseen.

Answers:
Yes
Question: Do they use skipgram version of word2vec?

Evidence:
raw_evidence: In this paper we focus on the skipgram approach with random negative examples proposed in BIBREF0 . This has been found to yield the best results among the proposed variants on a variety of semantic tests of the resulting vectors BIBREF7 , BIBREF0 . Given a corpus consisting of a sequence of sentences INLINEFORM0 each comprising a sequence of words INLINEFORM1 , the objective is to maximize the log likelihood: DISPLAYFORM0

Answers:
Yes
Question: What domains are considered that have such large vocabularies?

Evidence:
raw_evidence: More recently, novel applications of word2vec involving unconventional generalized “words” and training corpuses have been proposed. These powerful ideas from the NLP community have been adapted by researchers from other domains to tasks beyond representation of words, including relational entities BIBREF1 , BIBREF2 , general text-based attributes BIBREF3 , descriptive text of images BIBREF4 , nodes in graph structure of networks BIBREF5 , and queries BIBREF6 , to name a few.

Answers:
relational entities, general text-based attributes, descriptive text of images, nodes in graph structure of networks, queries
Question: Do they perform any morphological tokenization?

Evidence:
raw_evidence: This step entails counting occurrences of all words in the training corpus and sorting them in order of decreasing occurrence. As mentioned, the vocabulary is taken to be the INLINEFORM0 most frequently occurring words, that occur at least some number INLINEFORM1 times. It is implemented in Spark as a straight-forward map-reduce job.

Answers:
No
Question: What data do they train the language models on?

Evidence:
raw_evidence: In this work, the experiments are conducted using the BABEL speech corpus collected from the IARPA babel program. The corpus is mainly composed of conversational telephone speech (CTS) but some scripted recordings and far field recordings are presented as well. Table TABREF14 presents the details of the languages used in this work for training and evaluation.

Answers:
 BABEL speech corpus
Question: Do they report BLEU scores?

Evidence:

Answers:
No
Question: What architectures are explored to improve the seq2seq model?

Evidence:
raw_evidence: Table TABREF16 shows the recognition performance of naive multilingual approach using BLSTMP and VGG model against a monolingual model trained with BLSTMP. The results clearly indicate that having a better architecture such as VGG-BLSTM helps in improving multilingual performance. Except Pashto, Georgian and Tokpisin, the multilingual VGG-BLSTM model gave 8.8 % absolute gain in average over monolingual model. In case of multilingual BLSTMP, except Pashto and Georgian an absolute gain of 5.0 % in average is observed over monolingual model. Even though the VGG-BLSTM gave improvements, we were not able to perform stage-1 and stage-2 retraining with it due to time constraints. Thus, we proceed further with multilingual BLSTMP model for retraining experiments tabulated below.
We used a character-level RNNLM, which was trained with 2-layer LSTM on character sequences. We use all available paired text in the corresponding target language to train the LM for the language. No external text data were used. All language models are trained separately from the seq2seq models. When building dictionary, we combined all the characters over all 15 languages mentioned in table TABREF14 to make them work with transferred models. Regardless of the amount of data used for transfer learning, the RNNLM provides consistent gains across all languages over different data sizes.

Answers:
VGG-BLSTM, character-level RNNLM
Question: Why is this work different from text-only UNMT?

Evidence:
raw_evidence: Our idea is originally inspired by the text-only unsupervised MT (UMT) BIBREF8 , BIBREF9 , BIBREF0 , investigating whether it is possible to train a general MT system without any form of supervision. As BIBREF0 discussed, the text-only UMT is fundamentally an ill-posed problem, since there are potentially many ways to associate target with source sentences. Intuitively, since the visual content and language are closely related, the image can play the role of a pivot “language" to bridge the two languages without paralleled corpus, making the problem “more well-defined" by reducing the problem to supervised learning. However, unlike the text translation involving word generation (usually a discrete distribution), the task to generate a dense image from a sentence description itself is a challenging problem BIBREF10 . High quality image generation usually depends on a complicated or large scale neural network architecture BIBREF11 , BIBREF12 , BIBREF13 . Thus, it is not recommended to utilize the image dataset as a pivot “language" BIBREF14 . Motivated by the cycle-consistency BIBREF15 , we tackle the unsupervised translation with a multi-modal framework which includes two sequence-to-sequence encoder-decoder models and one shared image feature extractor. We don't introduce the adversarial learning via a discriminator because of the non-differentiable $\arg \max $ operation during word generation. With five modules in our framework, there are multiple data streaming paths in the computation graph, inducing the auto-encoding loss and cycle-consistency loss, in order to achieve the unsupervised translation.

Answers:
the image can play the role of a pivot “language" to bridge the two languages without paralleled corpus
Question: What is baseline used?

Evidence:
raw_evidence: We follow the BiLSTM architecture from BIBREF3. Only the optimizer was changed for all models to NADAM BIBREF22 as this helped with convergence problems for increasing cluster numbers. The Base is trained only on clean data while Base+Noise is trained on both the clean and the noisy data without noise handling. Global-CM uses a global confusion matrix for all noisy instances to model the noise as proposed by BIBREF3 and presented in Section SECREF3. The same architecture is used for Global-ID-CM, but the confusion matrix is initialized with the identity matrix (instead of Formula DISPLAY_FORM5) and only adapted during training.
The cluster-based models we propose in Section SECREF4 are Brown-CM and K-Means-CM. We experimented with numbers of clusters of 5, 10, 25 and 50. The models that select only the largest groups $G$ are marked as *-Freq and select either 30% or 50% of the clusters. The interpolation models have the postfix *-IP with $\lambda \in \lbrace 0.3, 0.5, 0.7\rbrace $ . The combination of both is named *-Freq-IP. As for all other hyperparameters, the choice was taken on the development set.
We implemented the Cleaning BIBREF15 and Dynamic-CM BIBREF14 models. Both were not developed for sequence labeling tasks and therefore needed to be adapted. For the Cleaning model, we followed the instructions by BIBREF3. The embedding and prediction components of the Dynamic-CM model were replaced according to our base model. The output of the dense layer was used as input to the dynamic matrix generation. We experimented with and without their proposed trace loss.

Answers:
Base , Base+Noise, Cleaning , Dynamic-CM ,  Global-CM,  Global-ID-CM, Brown-CM ,  K-Means-CM
Question: Did they evaluate against baseline?

Evidence:
raw_evidence: Our contributions are as follows: We propose to cluster the input words with the help of additional, unlabeled data. Based on this partition of the feature space, we obtain different confusion matrices that describe the relationship between clean and noisy labels. We evaluate our newly proposed models and related baselines in several low-resource settings across different languages with real, distantly supervised data with non-synthetic noise. The advanced modeling of the noisy labels substantially improves the performance up to 36% over methods without noise-handling and up to 9% over all other noise-handling baselines.

Answers:
Yes
Question: How they evaluate their approach?

Evidence:
raw_evidence: Our contributions are as follows: We propose to cluster the input words with the help of additional, unlabeled data. Based on this partition of the feature space, we obtain different confusion matrices that describe the relationship between clean and noisy labels. We evaluate our newly proposed models and related baselines in several low-resource settings across different languages with real, distantly supervised data with non-synthetic noise. The advanced modeling of the noisy labels substantially improves the performance up to 36% over methods without noise-handling and up to 9% over all other noise-handling baselines.

Answers:
They  evaluate newly proposed models in several low-resource settings across different languages with real, distantly supervised data with non-synthetic noise
Question: Which document classifiers do they experiment with?

Evidence:
raw_evidence: Demographic variations root in documents, especially in social media data BIBREF26, BIBREF25, BIBREF10. Such variations could further impact the performance and fairness of document classifiers. In this study, we experiment four different classification models including logistic regression (LR), recurrent neural network (RNN) BIBREF35, convolutional neural network (CNN) BIBREF36 and Google BERT BIBREF37. We present the baseline results of both performance and fairness evaluations across the multilingual corpus.

Answers:
logistic regression (LR), recurrent neural network (RNN) BIBREF35, convolutional neural network (CNN) BIBREF36 and Google BERT BIBREF37
Question: How is some information lost in the RNN-based generation models?

Evidence:
raw_evidence: Natural language generation (NLG) is an essential component of an SDS. Given a semantic representation (SR) consisting of a dialogue act and a set of slot-value pairs, the generator should produce natural language containing the desired information. Traditionally NLG was based on templates BIBREF3 , which produce grammatically-correct sentences that contain all desired information. However, the lack of variation of these sentences made these systems seem tedious and monotonic. Trainable generators BIBREF4 , BIBREF5 can generate several sentences for the same SR, but the dependence on pre-defined operations limits their potential. Corpus-based approaches BIBREF6 , BIBREF7 learn to generate natural language directly from data without pre-defined rules. However, they usually require alignment between the sentence and the SR. Recently, Wen et al. wensclstm15 proposed an RNN-based approach, which outperformed previous methods on several metrics. However, the generated sentences often did not include all desired attributes.

Answers:
the generated sentences often did not include all desired attributes.
Question: What is the model accuracy?

Evidence:
raw_evidence: We adopt the leave-one-out evaluation protocol to evaluate the performance of our model and baselines. The leave-one-out evaluation protocol has been widely used in top-K recommendation tasks. In particular, we held the latest interaction of each user as the test set and used the remaining interactions for training. Each testing instance was paired with 99 randomly sampled negative instances. Each recommendation model ranks the 100 instances according to its predicted results. The ranked list is judged by Hit Ratio (HR) BIBREF49 and Normalized Discount Cumulative Gain (NDCG) BIBREF50 at the position 10. HR@10 is a recall-based metric, measuring the percentage of the testing item being correctly recommended in the top-10 position. NDCG@10 is a ranked evaluation metric which considers the position of the correct hit in the ranked result. Since both modules in our framework introduce randomness, we repeat each experiment 5 times with different weight initialization and randomly selecting neighbors. We report the average score of the best performance in each training process for both metrics to ensure the robustness of our framework.
Overall, our AMRAN outperforms all baselines, achieving 0.657 HR@10 and 0.410 NDCG@10. It improves HR@10 by 5.3% and NDCG@10 by 3% over the best baseline (i.e., $NAIS_{concat}$).

Answers:
Overall, our AMRAN outperforms all baselines, achieving 0.657 HR@10 and 0.410 NDCG@10.
Question: What dataset is used?

Evidence:
raw_evidence: We evaluate our proposed model on a Twitter dataset obtained from the authors of BIBREF12. The interaction behavior collected in the dataset is consistent with our definition in SECREF3. As they did for their study, we only kept users who have at least three interactions (i.e., posting at least three fact-checking messages containing fact-checking URLs). We conducted additional preprocessing step by removing users whose posts are non-English, or their tweets were inaccessible, because some of our baselines require a fact-checker's tweets. Our final dataset consists of 11,576 users (i.e, fact-checkers), 4,732 fact-checking URLs and 63,429 interactions. The dataset also contains each user's social network information. Note that each user's social relationship is restricted within available users in the dataset. And we further take available feature values of both user and URL into consideration. For instance, a category of referred fact-checking article and the name of corresponding fact-checking website reveals linguistic characteristics such as writing style and topical interest of each URL; while the number of followers and number of followees of each user indicates the credibility and influence of the fact-checker. Statistics of the final dataset is presented in Table TABREF65.

Answers:
Twitter dataset obtained from the authors of BIBREF12
Question: Did they use other evaluation metrics?

Evidence:
raw_evidence: In this section, we present the experimental settings for pointer-generator network and language model. Our experiment, our pointer-generator model has 500-dimensional hidden states and word embeddings. We use 50k words as our vocabulary for source and target. We evaluate our pointer-generator performance using BLEU score. We take the best model as our generator and during the decoding stage, we generate 1-best and 3-best using beam search with a beam size of 5. For the input, we build a parallel monolingual corpus by translating the mixed language sequence using Google NMT to English ( INLINEFORM0 ) and Mandarin ( INLINEFORM1 ) sequences. Then, we concatenate the translated English and Mandarin sequences and assign code-switching sequences as the labels ( INLINEFORM2 ).
The baseline language model is trained using RNNLM BIBREF23 . Then, we train our 2-layer LSTM models with a hidden size of 500 and unrolled for 35 steps. The embedding size is equal to the LSTM hidden size for weight tying. We optimize our model using SGD with initial learning rates of INLINEFORM1 . If there is no improvement during the evaluation, we reduce the learning rate by a factor of 0.75. In each time step, we apply dropout to both embedding layer and recurrent network. The gradient is clipped to a maximum of 0.25. Perplexity measure is used in the evaluation.

Answers:
Yes
Question: What languages are explored in this paper?

Evidence:
raw_evidence: In our experiment, we use a conversational Mandarin-English code-switching speech corpus called SEAME Phase II (South East Asia Mandarin-English). The data are collected from spontaneously spoken interviews and conversations in Singapore and Malaysia by bilinguals BIBREF21 . As the data preprocessing, words are tokenized using Stanford NLP toolkit BIBREF22 and all hesitations and punctuations were removed except apostrophe. The split of the dataset is identical to BIBREF9 and it is showed in Table TABREF6 .

Answers:
Mandarin, English
Question: What parallel corpus did they use?

Evidence:
raw_evidence: In this section, we present the experimental settings for pointer-generator network and language model. Our experiment, our pointer-generator model has 500-dimensional hidden states and word embeddings. We use 50k words as our vocabulary for source and target. We evaluate our pointer-generator performance using BLEU score. We take the best model as our generator and during the decoding stage, we generate 1-best and 3-best using beam search with a beam size of 5. For the input, we build a parallel monolingual corpus by translating the mixed language sequence using Google NMT to English ( INLINEFORM0 ) and Mandarin ( INLINEFORM1 ) sequences. Then, we concatenate the translated English and Mandarin sequences and assign code-switching sequences as the labels ( INLINEFORM2 ).

Answers:
Parallel monolingual corpus in English and Mandarin
Question: What datasets are used for experiments on three other tasks?

Evidence:
raw_evidence: By combining both NSA and GSA, we obtain an enhanced SA module. We then construct our Normalized and Geometry-aware Self-Attention Network, namely NG-SAN, by replacing the vanilla SA modules in the encoder of the self-attention network with the proposed one. Extensive experiments on MS-COCO validates the effectiveness of our proposals. In particular, our NG-SAN establishes a new state-of-the-art on the MS-COCO evaluation sever, improving the best single-model result in terms of CIDEr from 125.5 to 128.6. To demonstrate the generality of NSA, we further present video captioning, machine translation, and visual question answering experiments on the VATEX, WMT 2014 English-to-German, and VQA-v2 datasets, respectively. On top of the strong Transformer-based baselines, our methods can consistently increase accuracies on all tasks at a negligible extra computational cost.

Answers:
VATEX, WMT 2014 English-to-German, and VQA-v2 datasets