qinchuanhui/UDA-QA huggingface dataset:
Question: Which machine baselines are used?

Evidence:
raw_evidence: (Information Retrieval). This baseline has been successfully used for related tasks like Question Answering BIBREF39 . We create two versions of this baseline: one with the pool of perspectives INLINEFORM0 and one with the pool of evidences INLINEFORM1 . We use this system to retrieve a ranked list of best matching perspective/evidence from the corresponding index.

Answers:
Information Retrieval
Question: What challenges are highlighted?

Evidence:
raw_evidence: There are two aspects that we defer to future work. First, the systems designed here assumed that the input are valid claim sentences. To make use of such systems, one needs to develop mechanisms to recognize valid argumentative structures. In addition, we ignore trustworthiness and credibility issues, important research issues that are addressed in other works.

Answers:
one needs to develop mechanisms to recognize valid argumentative structures, we ignore trustworthiness and credibility issues
Question: What are the linguistic differences between each class?

Evidence:
raw_evidence: Generic Sarcasm. We first examine the different patterns learned on the Gen dataset. Table TABREF29 show examples of extracted patterns for each class. We observe that the not-sarcastic patterns appear to capture technical and scientific language, while the sarcastic patterns tend to capture subjective language that is not topic-specific. We observe an abundance of adjective and adverb patterns for the sarcastic class, although we do not use adjective and adverb patterns in our regex retrieval method. Instead, such cues co-occur with the cues we search for, expanding our pattern inventory as we show in Table TABREF31 .
Rhetorical Questions. We notice that while the not-sarcastic patterns generated for RQs are similar to the topic-specific not-sarcastic patterns we find in the general dataset, there are some interesting features of the sarcastic patterns that are more unique to the RQs.
Many of our sarcastic questions focus specifically on attacks on the mental abilities of the addressee. This generalization is made clear when we extract and analyze the verb, subject, and object arguments using the Stanford dependency parser BIBREF32 for the questions in the RQ dataset. Table TABREF32 shows a few examples of the relations we extract.
Hyperbole. One common pattern for hyperbole involves adverbs and adjectives, as noted above. We did not use this pattern to retrieve hyperbole, but because each hyperbolic sarcastic utterance contains multiple cues, we learn an expanded class of patterns for hyperbole. Table TABREF33 illustrates some of the new adverb adjective patterns that are frequent, high-precision indicators of sarcasm.
We learn a number of verbal patterns that we had not previously associated with hyperbole, as shown in Table TABREF34 . Interestingly, many of these instantiate the observations of CanoMora2009 on hyperbole and its related semantic fields: creating contrast by exclusion, e.g. no limit and no way, or by expanding a predicated class, e.g. everyone knows. Many of them are also contrastive. Table TABREF33 shows just a few examples, such as though it in no way and so much knowledge.

Answers:
Each class has different patterns in adjectives, adverbs and verbs for sarcastic and non-sarcastic classes
Question: What simple features are used?

Evidence:
raw_evidence: Supervised Learning. We restrict our supervised experiments to a default linear SVM learner with Stochastic Gradient Descent (SGD) training and L2 regularization, available in the SciKit-Learn toolkit BIBREF25 . We use 10-fold cross-validation, and only two types of features: n-grams and Word2Vec word embeddings. We expect Word2Vec to be able to capture semantic generalizations that n-grams do not BIBREF26 , BIBREF27 . The n-gram features include unigrams, bigrams, and trigrams, including sequences of punctuation (for example, ellipses or "!!!"), and emoticons. We use GoogleNews Word2Vec features BIBREF28 .

Answers:
unigrams, bigrams, and trigrams, including sequences of punctuation, Word2Vec word embeddings
Question: What lexico-syntactic cues are used to retrieve sarcastic utterances?

Evidence:
raw_evidence: Generic Sarcasm. We first examine the different patterns learned on the Gen dataset. Table TABREF29 show examples of extracted patterns for each class. We observe that the not-sarcastic patterns appear to capture technical and scientific language, while the sarcastic patterns tend to capture subjective language that is not topic-specific. We observe an abundance of adjective and adverb patterns for the sarcastic class, although we do not use adjective and adverb patterns in our regex retrieval method. Instead, such cues co-occur with the cues we search for, expanding our pattern inventory as we show in Table TABREF31 .
Many of our sarcastic questions focus specifically on attacks on the mental abilities of the addressee. This generalization is made clear when we extract and analyze the verb, subject, and object arguments using the Stanford dependency parser BIBREF32 for the questions in the RQ dataset. Table TABREF32 shows a few examples of the relations we extract.
Hyperbole. One common pattern for hyperbole involves adverbs and adjectives, as noted above. We did not use this pattern to retrieve hyperbole, but because each hyperbolic sarcastic utterance contains multiple cues, we learn an expanded class of patterns for hyperbole. Table TABREF33 illustrates some of the new adverb adjective patterns that are frequent, high-precision indicators of sarcasm.
We learn a number of verbal patterns that we had not previously associated with hyperbole, as shown in Table TABREF34 . Interestingly, many of these instantiate the observations of CanoMora2009 on hyperbole and its related semantic fields: creating contrast by exclusion, e.g. no limit and no way, or by expanding a predicated class, e.g. everyone knows. Many of them are also contrastive. Table TABREF33 shows just a few examples, such as though it in no way and so much knowledge.

Answers:
adjective and adverb patterns, verb, subject, and object arguments, verbal patterns
Question: what is the source of the song lyrics?

Evidence:
raw_evidence: In order to obtain a large number of Brazilian music lyrics, we created a crawler to navigate into the Vagalume website, extracting, for each musical genre, all the songs by all the listed authors. The implementation of a crawler was necessary because, although the Vagalume site provides an API, it is only for consultation and does not allow obtaining large amounts of data. The crawler was implemented using Scrapy, an open-source and collaborative Python library to extract data from websites.

Answers:
Vagalume website
Question: what genre was the most difficult to classify?

Evidence:
raw_evidence: The bossa-nova and jovem-guarda genres, which have few instances in the dataset, are among the most difficult ones to classify using the model. The pop genre, by contrast, has a small distribution between the number of songs and the number of artists, and could not be well classified by our model. This may indicate that our model was unable to identify a pattern due to the low number of songs per artist, or that the song lyrics of this genre cover several subjects that are confused with other genres.

Answers:
 bossa-nova and jovem-guarda genres
Question: what word embedding techniques did they experiment with?

Evidence:
raw_evidence: In this work, we use the Python Word2Vec implementation provided by the Gensim library. The Portuguese pre-trained word embeddings created by BIBREF10 and available for download was used to represent words as vectors. We only used models of dimension 300 and, for Word2Vec, Wang2Vec, and FastText, skip-gram architectured models.

Answers:
Word2Vec, Wang2Vec, and FastText
Question: Is the filter based feature selection (FSE) a form of regularization?

Evidence:
raw_evidence: Feature selection is considered an indispensable task in text classification as it removes redundant and irrelevant features of the corpus BIBREF18. Broadly, feature selection approaches can be divided into three classes namely wrapper, embedded, and filter BIBREF7, BIBREF8. In recent years, researchers have proposed various filter based feature selection methods to raise the performance of document text classification BIBREF19.
To develop the vocabulary of most discriminative features, we remove all punctuation symbols and non-significant words (stop words) as a part of the preprocessing step. Furthermore, in order to rank the terms based on their discriminative power among the classes, we use filter based feature selection method named as Normalized Difference Measure (NDM)BIBREF5. Considering the features contour plot, Rehman et al. BIBREF5 suggested that all those features which exist in top left, and bottom right corners of the contour are extremely significant as compared to those features which exist around diagonals. State-of-the-art filter based feature selection algorithms such as ACC2 treat all those features in the same fashion which exist around the diagonals BIBREF5. For instance, ACC2 assigns same rank to those features which has equal difference ($|t_{pr} -f_{pr}|$) value but different $t_{pr}$ and $f_{pr}$ values. Whereas NDM normalizes the difference ($|t_{pr} -f_{pr}|$) with the minimum of $t_{pr}$ and $f_{pr}$ (min($t_{pr}$, $f_{pr}$)) and assign different rank to those terms which have same difference value. Normalized Difference Measure (NDM) considers those features highly significant which have the following properties:

Answers:
No
Question: How is human evaluation performed, what was the criteria?

Evidence:
raw_evidence: We thus also conducted human studies on Amazon MTurk to evaluate the generated responses with pairwise comparison for dialogue quality. We compare our models with an advanced decoding algorithm MMI BIBREF2 and two models, namely LSTM BIBREF0 and VHRED BIBREF7, both with additive attention. To our best knowledge, LSTM and VHRED were the primary models with which F1's were reported on the Ubuntu dataset. Following BIBREF5 (BIBREF5), we employ two criteria: Plausibility and Content Richness. The first criterion measures whether the response is plausible given the context, while the second gauges whether the response is diverse and informative. The utterances were randomly shuffled to anonymize model identity. We only allowed annotators located in the US-located with at least an approval rate of $98\%$ and $10,000$ approved HITs. We collected 100 annotations in total after rejecting those completed by people who assign exactly the same score to all model responses. Since we evaluated 7 models, we collected 700 annotations in total, which came from a diverse pool of annotators.

Answers:
Through Amazon MTurk annotators to determine plausibility and content richness of the response
Question: Which one of the four proposed models performed best?

Evidence:
raw_evidence: We employ several complementary metrics to capture different aspects of the model. The F1 results are shown in Table TABREF24. Among all single models, LFT performs the best, followed by MinAvgOut. RL is also comparable with previous state-of-the-art models VHRED (attn) and Reranking-RL. We think that this is because LFT exerts no force in pulling the model predictions away from the ground-truth tokens, but rather just makes itself aware of how dull each response is. Consequently, its responses appear more relevant than the other two approaches. Moreover, the hybrid model (last row) outperforms all other models by a large margin. One might expect that minimizing AVGOUT causes the models to move further away from the ground-truth tokens, so that it will hurt relevance. However, our F1 results show that as the responses become more diverse, they are more likely to include information more related and specific to the input contexts, which actually makes the model gain on both diversity and relevance. This will be further confirmed by the output examples in Table TABREF29.

Answers:
the hybrid model MinAvgOut + RL
Question: What metrics are used to measure performance of models?

Evidence:
raw_evidence: BPRA: Belief Per-Response Accuracy (BPRA) tests the ability to generate the correct user intents during the dialogue. This metric is used to evaluate the accuracy of dialogue belief tracker BIBREF1.
APRA: Action Per-Response Accuracy (APRA) evaluates the per-turn accuracy of the dialogue actions generated by dialogue policy maker. For baselines, APRA evaluates the classification accuracy of the dialogue policy maker. But our model actually generates each individual token of actions, and we consider a prediction to be correct only if every token of the model output matches the corresponding token in the ground truth.
BLEU BIBREF19: The metric evaluates the quality of the final response generated by natural language generator. The metric is usually used to measure the performance of the task-oriented dialogue system.

Answers:
BPRA, APRA, BLEU
Question: What are state-of-the-art baselines?

Evidence:
raw_evidence: E2ECM BIBREF11: In dialogue policy maker, it adopts a classic classification for skeletal sentence template. In our implement, we construct multiple binary classifications for each act to search the sentence template according to the work proposed by BIBREF11.
CDM BIBREF10: This approach designs a group of classifications (two multi-class classifications and some binary classifications) to model the dialogue policy.

Answers:
E2ECM, CDM
Question: What two benchmark datasets are used?

Evidence:
raw_evidence: We adopt the DSTC2 BIBREF20 dataset and Maluuba BIBREF21 dataset to evaluate our proposed model. Both of them are the benchmark datasets for building the task-oriented dialog systems. Specifically, the DSTC2 is a human-machine dataset in the single domain of restaurant searching. The Maluuba is a very complex human-human dataset in travel booking domain which contains more slots and values than DSTC2. Detailed slot information in each dataset is shown in Table TABREF34.

Answers:
DSTC2, Maluuba
Question: Do they compare to other models appart from HAN?

Evidence:
raw_evidence: We evaluate the quality of the document embeddings learned by the different variants of CAHAN and the HAN baseline on three of the large-scale document classification datasets introduced by BIBREF14 and used in the original HAN paper BIBREF5. They fall into two categories: topic classification (Yahoo) and fine-grained sentiment analysis (Amazon, Yelp). Dataset statistics are shown in Table TABREF29. Classes are perfectly balanced, for all datasets.

Answers:
No
Question: What are the datasets used

Evidence:
raw_evidence: We evaluate the quality of the document embeddings learned by the different variants of CAHAN and the HAN baseline on three of the large-scale document classification datasets introduced by BIBREF14 and used in the original HAN paper BIBREF5. They fall into two categories: topic classification (Yahoo) and fine-grained sentiment analysis (Amazon, Yelp). Dataset statistics are shown in Table TABREF29. Classes are perfectly balanced, for all datasets.

Answers:
large-scale document classification datasets introduced by BIBREF14
Question: What evidence does visualizing the attention give to show that it helps to obtain a more robust understanding of semantics and sentiments?

Evidence:
raw_evidence: In order to obtain insights about why the more fine-grained bi-sense emoji embedding helps in understanding the complexed sentiments behind tweets, we visualize the attention weights for ATT-E-LSTM and MATT-BiE-LSTM for comparison. The example tweets with corresponding attention weights calculated by word-emoji embedding and senti-emoji embedding are shown in Figure FIGREF27 , where the contexts are presented in the captions. The emojis used are , , and , respectively.
Therefore, we construct the new input INLINEFORM0 to each LSTM unit by concatenating the original word embedding and the attention vector in Equation EQREF21 to distribute the senti-emoji information to each step. This model is called Multi-level Attention-based LSTM with Bi-sense Emoji Embedding (MATT-BiE-LSTM). We choose the same binary cross-entropy as the loss function with the same network configuration with WATT-BiE-LSTM. DISPLAYFORM0
Attention-based LSTM with emojis:We also use the word-emoji embedding to calculate the emoji-word attention following Equation EQREF20 and EQREF21 , and the only difference is that we replace the attention-derived senti-emoji embedding with the pre-trained word-emoji embedding by fasttext, denoted as ATT-E-LSTM.
In Figure FIGREF27 (a), the ATT-E-LSTM model (baseline) assigns relatively more weights on the word “no” and “pressure”, while MATT-BiE-LSTM attends mostly on the word “happy” and “lovely”. The different attention distributions suggest that the proposed senti-emoji embedding is capable of recognizing words with strong sentiments that are closely related to the true sentiment even with the presence of words with conflicting sentiments, such as “pressure” and “happy”. while ATT-E-LSTM tends to pick up all sentimental words which could raise confusions. The senti-emoji embedding is capable of extracting representations of complexed semantics and sentiments which help guide the attentions even in cases when the word sentiment and emoji sentiment are somewhat contradictory to each other. From Figure FIGREF27 (b) and (c) we can observe that the ATT-E-LSTM assigns more weights on the sentiment-irrelevant words than the MATT-BiE-LSTM such as “hoodies”, “wait” and “after”, indicating that the proposed model is more robust to irrelevant words and concentrates better on important words. Because of the senti-emoji embedding obtained through bi-sense emoji embedding and the sentence-level LSTM encoding on the text input (described in Section SECREF13 ), we are able to construct a more robust embedding based on the semantic and sentiment information from the whole context compared to the word-emoji embedding used in ATT-E-LSTM which takes only word-level information into account.

Answers:
The different attention distributions suggest that the proposed senti-emoji embedding is capable of recognizing words with strong sentiments that are closely related to the true sentiment even with the presence of words with conflicting sentiments
Question: Which SOTA models are outperformed?

Evidence:
raw_evidence: Attention mechanism aligns and performs well with bi-sense embedding. MATT-BiE-LSTM and WATT-BiE-LSTM obtain similar performances when tested on both Vader and human annotated samples, though their ways of computing the attention (weights and vectors) are different that WATT computes attention weights and the senti-emoji embeddings guided by each word, and MATT obtains the senti-emoji embedding based on the LSTM encoder on the whole contexts and computes the attention weights of the senti-emoji embedding across all words. Both models outperforms the state-of-the-art baseline models including ATT-E-LSTM. The proposed attention-based LSTM can be further extended to handle tasks involving multi-sense embedding as inputs, such as the word-sense embedding in NLP, by using context-guide attention to self-select how much to attend on each sense of the embeddings each of which correspond to a distinct sense of semantics or sentiments. In this way we are able to take advantage of the more robust and fine-grained embeddings.
Attention-based LSTM with emojis:We also use the word-emoji embedding to calculate the emoji-word attention following Equation EQREF20 and EQREF21 , and the only difference is that we replace the attention-derived senti-emoji embedding with the pre-trained word-emoji embedding by fasttext, denoted as ATT-E-LSTM.
LSTM with bi-sense emoji embedding (proposed): As we have introduced in Section SECREF13 , we propose two attention-based LSTM networks based on bi-sense emoji embedding, denoted as MATT-BiE-LSTM and WATT-BiE-LSTM.

Answers:
Attention-based LSTM with emojis
Question: What is the baseline for experiments?

Evidence:
raw_evidence: We set up the baselines and proposed models as follows:
LSTM with text embedding: CNNs and LSTMs are widely used to encode textual contents for sentiment analysis in BIBREF45 , BIBREF46 and many online tutorials. Here we select the standard LSTM with pre-trained word embedding as input, and add one fully-connected layer with sigmoid activation top of the LSTM encoder (same as all other models), denoted as T-LSTM.
LSTM with emoji embedding: We consider the emoji as one special word and input both pre-trained text and emoji embeddings into the same LSTM network, namely E-LSTM. Similarly, we concatenate the pre-trained bi-sense emoji embedding as one special word to feed into the LSTM network. This model is called BiE-LSTM.
Attention-based LSTM with emojis:We also use the word-emoji embedding to calculate the emoji-word attention following Equation EQREF20 and EQREF21 , and the only difference is that we replace the attention-derived senti-emoji embedding with the pre-trained word-emoji embedding by fasttext, denoted as ATT-E-LSTM.

Answers:
LSTM with text embedding, LSTM with emoji embedding, Attention-based LSTM with emojis
Question: What is the motivation for training bi-sense embeddings?

Evidence:
raw_evidence: The extensive use of emojis has drawn a growing attention from researchers BIBREF4 , BIBREF5 because the emojis convey fruitful semantical and sentimental information to visually complement the textual information which is significantly useful in understanding the embedded emotional signals in texts BIBREF6 . For example, emoji embeddings have been proposed to understand the semantics behind the emojis BIBREF7 , BIBREF8 , and the embedding vectors can be used to visualize and predict emoji usages given their corresponding contexts. Previous work also shows that, it is useful to pre-train a deep neural network on an emoji prediction task with pre-trained emoji embeddings to learn the emotional signals of emojis for other tasks including sentiment, emotion and sarcasm prediction BIBREF9 . However, the previous literatures lack in considerations of the linguistic complexities and diversity of emoji. Therefore, previous emoji embedding methods fail to handle the situation when the semantics or sentiments of the learned emoji embeddings contradict the information from the corresponding contexts BIBREF5 , or when the emojis convey multiple senses of semantics and sentiments such as ( and ). In practice, emojis can either summarize and emphasis the original tune of their contexts, or express more complex semantics such as irony and sarcasm by being combined with contexts of contradictory semantics or sentiments. For the examples shown in Table TABREF3 , the emoji () is of consistent sentiment with text to emphasis the sentiment, but is of the opposite sentiment (positive) to the text sentiment (negative) example 3 and 4 to deliver a sense of sarcasm. Conventional emoji analysis can only extract single embedding of each emoji, and such embeddings will confuse the following sentiment analysis model by inconsistent sentiment signals from the input texts and emojis. Moreover, we consider the emoji effect modeling different from the conventional multimodal sentiment analysis which usually includes images and texts in that, image sentiment and text sentiment are usually assumed to be consistent BIBREF10 while it carries no such assumption for texts and emojis.

Answers:
 previous emoji embedding methods fail to handle the situation when the semantics or sentiments of the learned emoji embeddings contradict the information from the corresponding contexts BIBREF5 , or when the emojis convey multiple senses of semantics and sentiments