qinchuanhui/UDA-QA huggingface dataset:
Question: How many parameters does the model have?

Evidence:
raw_evidence: Instead of reading in the tokenized input text, our model reads raw utf-8 bytes. For English text in the ASCII range, this is equivalent to processing characters as individual tokens. Non-ASCII characters (e.g. accented characters, or non-Latin scripts) are typically two or three utf-8 bytes. We use a standard “transformer decoder” (a stack of transformer layers with a causal attention mask) to process the sequence $x_{0:i-1}$ and predict the following byte $x_i$. The model's prediction is an estimate of the probability distribution over all possible 256 byte values. Our input byte embedding matrix has dimensionality 256. Our byte-level transformer model has 40 standard transformer layers with hidden size 1024, filter size 8192, and 16 heads. The model has around 836M parameters, of which only 66K are byte embeddings.

Answers:
model has around 836M parameters
Question: How many characters are accepted as input of the language model?

Evidence:
raw_evidence: Instead of reading in the tokenized input text, our model reads raw utf-8 bytes. For English text in the ASCII range, this is equivalent to processing characters as individual tokens. Non-ASCII characters (e.g. accented characters, or non-Latin scripts) are typically two or three utf-8 bytes. We use a standard “transformer decoder” (a stack of transformer layers with a causal attention mask) to process the sequence $x_{0:i-1}$ and predict the following byte $x_i$. The model's prediction is an estimate of the probability distribution over all possible 256 byte values. Our input byte embedding matrix has dimensionality 256. Our byte-level transformer model has 40 standard transformer layers with hidden size 1024, filter size 8192, and 16 heads. The model has around 836M parameters, of which only 66K are byte embeddings.

Answers:
input byte embedding matrix has dimensionality 256
Question: What dataset is used for this task?

Evidence:
raw_evidence: We used the Pasokh dataset BIBREF42 that contains 100 Persian news documents each of which is associated with 5 summaries. Each summary consists of several sentences of the original text, selected by a human expert. Some sentences are slightly modified and are not, therefore, an exact copy of any original sentences. Documents are categorized into six categories such as political, economic and so on. The length of documents ranges from 4 to 156 sentences. Overall, it has about 2,500 sentences.

Answers:
the Pasokh dataset BIBREF42
Question: Is new approach tested against state of the art?

Evidence:
raw_evidence: Another insight gained from these charts is that a random summarizer resulted in scores more than 50% in all measures, and without using document-aware features, the model achieves a small improvement over a random summarizer.

Answers:
No
Question: Is the dataset balanced across categories?

Evidence:
raw_evidence: We submit 4,000 segments, sampled equally from each domain and uniformly within domains, to Mechanical Turk to be annotated by at least five Workers each and include in each batch one of 50 “check questions” which have been previously verified by two in-house annotators. After removing annotations which failed the check questions, and data points for which at least half of the annotators selected “Can't Tell”, we are left with 3,553 labeled data points from 2,929 different posts. We take the annotators' majority vote as the label for each segment and record the percentage of annotators who agreed. The resulting dataset is nearly balanced, with 52.3% of the data (1,857 instances) labeled stressful.

Answers:
Yes
Question: What supervised methods are used?

Evidence:
raw_evidence: We first experiment with a suite of non-neural models, including Support Vector Machines (SVMs), logistic regression, Naïve Bayes, Perceptron, and decision trees. We tune the parameters for these models using grid search and 10-fold cross-validation, and obtain results for different combinations of input and features.
We finally experiment with neural models, although our dataset is relatively small. We train both a two-layer bidirectional Gated Recurrent Neural Network (GRNN) BIBREF20 and Convolutional Neural Network (CNN) (as designed in BIBREF21) with parallel filters of size 2 and 3, as these have been shown to be effective in the literature on emotion detection in text (e.g., BIBREF22, BIBREF23). Because neural models require large amounts of data, we do not cull the data by annotator agreement for these experiments and use all the labeled data we have. We experiment with training embeddings with random initialization as well as initializing with our domain-specific Word2Vec embeddings, and we also concatenate the best feature set from our non-neural experiments onto the representations after the recurrent and convolutional/pooling layers respectively.

Answers:
Support Vector Machines (SVMs), logistic regression, Naïve Bayes, Perceptron, and decision trees, a two-layer bidirectional Gated Recurrent Neural Network (GRNN) BIBREF20 and Convolutional Neural Network (CNN) (as designed in BIBREF21)
Question: What categories does the dataset come from?

Evidence:
raw_evidence: We include ten subreddits in the five domains of abuse, social, anxiety, PTSD, and financial, as detailed in tab:data-spread, and our analysis focuses on the domain level. Using the PRAW API, we scrape all available posts on these subreddits between January 1, 2017 and November 19, 2018; in total, 187,444 posts. As we will describe in sec:annotation, we assign binary stress labels to 3,553 segments of these posts to form a supervised and semi-supervised training set. An example segment is shown in fig:stress-example. Highlighted phrases are indicators that the writer is stressed: the writer mentions common physical symptoms (nausea), explicitly names fear and dread, and uses language indicating helplessness and help-seeking behavior.

Answers:
abuse, social, anxiety, PTSD, and financial
Question: What are all machine learning approaches compared in this work?

Evidence:
raw_evidence: Email classifier using machine learning ::: Machine learning approach ::: Feature selection
Ngrams are a continuous sequence of n items from a given sample of text. From title, body and OCR text words are selected. Ngrams of 3 nearby words are extracted with Term Frequency-Inverse Document Frequency (TF-IDF) vectorizing, then features are filtered using chi squared the feature scoring method.
Email classifier using machine learning ::: Machine learning approach ::: Random forest
Random Forest is a bagging Algorithm, an ensemble learning method for classification that operates by constructing a multitude of decision trees at training time and outputting the class that has highest mean majority vote of the classesBIBREF14.
Email classifier using machine learning ::: Machine learning approach ::: XGBoost
XGBoost is a decision-tree-based ensemble Machine Learning algorithm that uses a gradient boosting framework. It is used commonly in the classification problems involving unstructured dataBIBREF5.
Email classifier using machine learning ::: Machine learning approach ::: Hierarchical Model
Since the number of target labels are high, achieving the higher accuracy is difficult, while keeping all the categories under same feature selection method. Some categories performs well with lower TF-IDF vectorizing range and higher n grams features even though they showed lower accuracy in the overall single model. Therefore, hierarchical machine learning models are built to classify 31 categories in the first classification model and remaining categories are named as low-accu and predicted as one category. In the next model, predicted low-accu categories are again classified into 47 categories. Comparatively this hierarchical model works well since various feature selection methods are used for various categoriesBIBREF5.

Answers:
Feature selection, Random forest, XGBoost, Hierarchical Model
Question: Which patterns and rules are derived?

Evidence:
raw_evidence: Additionally, important patterns can emerge from analysis of the fine-grained acts in a dialogue in a post-prediction setting. For example, if an agent does not follow-up with certain actions in response to a customer's question dialogue act, this could be found to be a violation of a best practice pattern. By analyzing large numbers of dialogue act sequences correlated with specific outcomes, various rules can be derived, i.e. "Continuing to request information late in a conversation often leads to customer dissatisfaction." This can then be codified into a best practice pattern rules for automated systems, such as "A request for information act should be issued early in a conversation, followed by an answer, informative statement, or apology towards the end of the conversation."
Our analysis helps zone in on how the use of certain dialogue acts may be likely to result in different outcomes. The weights we observe vary in the amount of insight provided: for example, offering extra help at the end of a conversation, or thanking the customer yields more satisfied customers, and more resolved problems (with ratios of above 6:1). However, some outcomes are much more subtle: for example, asking yes-no questions early-on in a conversation is highly associated with problem resolution (ratio 3:1), but asking them at the end of a conversation has as similarly strong association with unsatisfied customers. Giving elaborate answers that are not a simple affirmative, negative, or response acknowledgement (i.e. Answer (Other)) towards the middle of a conversation leads to satisfied customers that are not frustrated. Likewise, requesting information towards the end of a conversation (implying that more information is still necessary at the termination of the dialogue) leads to unsatisfied and unresolved customers, with ratios of at least 4:1.

Answers:
A request for information act should be issued early in a conversation, followed by an answer, informative statement, or apology towards the end of the conversation,  offering extra help at the end of a conversation, or thanking the customer yields more satisfied customers, and more resolved problems , asking yes-no questions early-on in a conversation is highly associated with problem resolution (ratio 3:1), but asking them at the end of a conversation has as similarly strong association with unsatisfied customers, Giving elaborate answers that are not a simple affirmative, negative, or response acknowledgement (i.e. Answer (Other)) towards the middle of a conversation leads to satisfied customers that are not frustrated, requesting information towards the end of a conversation (implying that more information is still necessary at the termination of the dialogue) leads to unsatisfied and unresolved customers
Question: How are customer satisfaction, customer frustration and overall problem resolution data collected?

Evidence:
raw_evidence: Using these filters as pre-processing methods, we end up with a set of 800 conversations, spanning 5,327 turns. We conduct our annotation study on Amazon Mechanical Turk, presenting Turkers with Human Intelligence Tasks (henceforth, HITs) consisting of a single conversation between a customer and an agent. In each HIT, we present Turkers with a definition of each dialogue act, as well as a sample annotated dialogue for reference. For each turn in the conversation, we allow Turkers to select as many labels from our taxonomy as required to fully characterize the intent of the turn. Additionally, annotators are asked three questions at the end of each conversation HIT, to which they could respond that they agreed, disagreed, or could not tell:

Answers:
By annotators on Amazon Mechanical Turk.
Question: Which Twitter customer service industries are investigated?

Evidence:
raw_evidence: We first expand upon previous work and generic dialogue act taxonomies, developing a fine-grained set of dialogue acts for customer service, and conducting a systematic user study to identify these acts in a dataset of 800 conversations from four Twitter customer service accounts (i.e. four different companies in the telecommunication, electronics, and insurance industries). We then aim to understand the conversation flow between customers and agents using our taxonomy, so we develop a real-time sequential SVM-HMM model to predict our fine-grained dialogue acts while a conversation is in progress, using a novel multi-label scheme to classify each turn. Finally, using our dialogue act predictions, we classify conversations based on the outcomes of customer satisfaction, frustration, and overall problem resolution, then provide actionable guidelines for the development of automated customer service systems and intelligent agents aimed at desired customer outcomes BIBREF3 , BIBREF4 .

Answers:
 four different companies in the telecommunication, electronics, and insurance industries
Question: Which dialogue acts are more suited to the twitter domain?

Evidence:
raw_evidence: In this work, we are motivated to predict the dialogue acts in conversations with the intent of identifying problem spots that can be addressed in real-time, and to allow for post-conversation analysis to derive rules about conversation outcomes indicating successful/unsuccessful interactions, namely, customer satisfaction, customer frustration, and problem resolution. We focus on analysis of the dialogue acts used in customer service conversations as a first step to fully automating the interaction. We address various different challenges: dialogue act annotated data is not available for customer service on Twitter, the task of dialogue act annotation is subjective, existing taxonomies do not capture the fine-grained information we believe is valuable to our task, and tweets, although concise in nature, often consist of overlapping dialogue acts to characterize their full intent. The novelty of our work comes from the development of our fine-grained dialogue act taxonomy and multi-label approach for act prediction, as well as our analysis of the customer service domain on Twitter. Our goal is to offer useful analytics to improve outcome-oriented conversational systems.

Answers:
overlapping dialogue acts
Question: How many improvements on the French-German translation benchmark?

Evidence:
raw_evidence: While the mixing strategy compensates for most of the gap between the Fr-De* and the Fr*-De (3.01 $\rightarrow $ 0.17) in the De $\rightarrow $ Fr case, the resulting PSEUDOmix still shows lower BLEU than the target-originated Fr-De* corpus. We thus enhance the quality of the synthetic examples of the source-originated Fr*-De data by further training its mother translation model (En $\rightarrow $ Fr). As illustrated in Figure 2 , with the target-originated Fr-De* corpus being fixed, the quality of the models trained with the source-originated Fr*-De data and PSEUDOmix increases in proportion to the quality of the mother model for the Fr*-De corpus. Eventually, PSEUDOmix shows the highest BLEU, outperforming both Fr*-De and Fr-De* data. The results indicate that the benefit of the proposed mixing approach becomes much more evident when the quality gap between the source- and target-originated synthetic data is within a certain range.
As presented in Table 6 , we observe that fine-tuning using ground truth parallel data brings substantial improvements in the translation qualities of all NMT models. Among all fine-tuned models, PSEUDOmix shows the best performance in all experiments. This is particularly encouraging for the case of De $\rightarrow $ Fr, where PSEUDOmix reported lower BLEU than the Fr-De* data before it was fine-tuned. Even in the case where PSEUDOmix shows comparable results with other synthetic corpora in the Pseudo Only scenario, it shows higher improvements in the translation quality when fine-tuned with the real parallel data. These results clearly demonstrate the strengths of the proposed PSEUDOmix, which indicate both competitive translation quality by itself and relatively higher potential improvement as a result of the refinement using ground truth parallel corpora.

Answers:
one
Question: How do they align the synthetic data?

Evidence:
raw_evidence: By choosing English (En) as the pivot language, we perform pivot alignments for identical English segments on Europarl Fr-En and En-De parallel corpora BIBREF18 , constructing a multi-parallel corpus of Fr-En-De. Then each of the Fr*-De and Fr-De* pseudo parallel corpora is established from the multi-parallel data by applying the pivot language-based translation described in the previous section. For automatic translation, we utilize a pre-trained and publicly released NMT model for En $\rightarrow $ De and train another NMT model for En $\rightarrow $ Fr using the WMT'15 En-Fr parallel corpus BIBREF19 . A beam of size 5 is used to generate synthetic sentences. Lastly, to match the size of the training data, PSEUDOmix is established by randomly sampling half of each Fr*-De and Fr-De* corpus and mixing them together.

Answers:
By choosing English (En) as the pivot language, we perform pivot alignments for identical English segments on Europarl Fr-En and En-De parallel corpora BIBREF18 , constructing a multi-parallel corpus of Fr-En-De. Then each of the Fr*-De and Fr-De* pseudo parallel corpora is established from the multi-parallel data by applying the pivot language-based translation described in the previous section.
Question: Where do they collect the synthetic data?

Evidence:
raw_evidence: By choosing English (En) as the pivot language, we perform pivot alignments for identical English segments on Europarl Fr-En and En-De parallel corpora BIBREF18 , constructing a multi-parallel corpus of Fr-En-De. Then each of the Fr*-De and Fr-De* pseudo parallel corpora is established from the multi-parallel data by applying the pivot language-based translation described in the previous section. For automatic translation, we utilize a pre-trained and publicly released NMT model for En $\rightarrow $ De and train another NMT model for En $\rightarrow $ Fr using the WMT'15 En-Fr parallel corpus BIBREF19 . A beam of size 5 is used to generate synthetic sentences. Lastly, to match the size of the training data, PSEUDOmix is established by randomly sampling half of each Fr*-De and Fr-De* corpus and mixing them together.

Answers:
Yes
Question: Do they analyze what type of content Arabic bots spread in comparison to English?

Evidence:

Answers:
No
Question: Do they propose a new model to better detect Arabic bots specifically?

Evidence:
raw_evidence: In this work, we build a novel regression model, based on linguistic, content, behavioral and topic features to detect Arabic Twitter bots to understand the impact of bots in spreading religious hatred in Arabic Twitter space. In particular, we quantitatively code and analyze a representative sample of 450 accounts disseminating hate speech from the dataset constructed in our previous work BIBREF18 , BIBREF19 for bot-like behavior. We compare our assigned bot-likelihood scores to those of Botometer BIBREF14 , a well-known machine-learning-based bot detection tool, and we show that Botometer performs a little above average in detecting Arabic bots. Based on our analysis, we build a predictive regression model and train it on various sets of features and show that our regression model outperforms Botometer's by a significant margin (31 points in Spearman's rho). Finally, we provide a large-scale analysis of predictive features that distinguish bots from humans in terms of characteristics and behaviors within the context of social media.

Answers:
Yes
Question: How do they prevent the model complexity increasing with the increased number of slots?

Evidence:
raw_evidence: To solve this problem, we need a state tracking model independent of dialogue slots. In other words, the network should depend on the semantic similarity between slots and utterance instead of slot-specific modules. To this end, we propose the Slot-Independent Model (SIM). Our model complexity does not increase when the number of slots in dialogue tasks go up. Thus, SIM has many fewer parameters than existing dialogue state tracking models. To compensate for the exclusion of slot-specific parameters, we incorporate better feature representation of user utterance and dialogue states using syntactic information and convolutional neural networks (CNN). The refined representation, in addition to cross and self-attention mechanisms, make our model achieve even better performance than slot-specific models. For instance, on Wizard-of-Oz (WOZ) 2.0 dataset BIBREF8, the SIM model obtains a joint-accuracy score of 89.5%, 1.4% higher than the previously best model GLAD, with only 22% of the number of parameters. On DSTC2 dataset, SIM achieves comparable performance with previous best models with only 19% of the model size.

Answers:
They exclude slot-specific parameters and incorporate better feature representation of user utterance and dialogue states using syntactic information and convolutional neural networks (CNN).
Question: What network architecture do they use for SIM?

Evidence:
raw_evidence: To solve this problem, we need a state tracking model independent of dialogue slots. In other words, the network should depend on the semantic similarity between slots and utterance instead of slot-specific modules. To this end, we propose the Slot-Independent Model (SIM). Our model complexity does not increase when the number of slots in dialogue tasks go up. Thus, SIM has many fewer parameters than existing dialogue state tracking models. To compensate for the exclusion of slot-specific parameters, we incorporate better feature representation of user utterance and dialogue states using syntactic information and convolutional neural networks (CNN). The refined representation, in addition to cross and self-attention mechanisms, make our model achieve even better performance than slot-specific models. For instance, on Wizard-of-Oz (WOZ) 2.0 dataset BIBREF8, the SIM model obtains a joint-accuracy score of 89.5%, 1.4% higher than the previously best model GLAD, with only 22% of the number of parameters. On DSTC2 dataset, SIM achieves comparable performance with previous best models with only 19% of the model size.

Answers:
convolutional neural networks (CNN)
Question: How do they measure model size?

Evidence:
raw_evidence: Furthermore, as SIM has no slot-specific neural network structures, its model size is much smaller than previous models. Table TABREF20 shows that, on WoZ and DSTC2 datasets, SIM model has the same number of parameters, which is only 23% and 19% of that in GLAD model.

Answers:
By the number of parameters.
Question: Does model uses pretrained Transformer encoders?

Evidence:
raw_evidence: Our Transformer Transducer model architecture has 18 audio and 2 label encoder layers. Every layer is identical for both audio and label encoders. The details of computations in a layer are shown in Figure FIGREF10 and Table TABREF11. All the models for experiments presented in this paper are trained on 8x8 TPU with a per-core batch size of 16 (effective batch size of 2048). The learning rate schedule is ramped up linearly from 0 to $2.5\mathrm {e}{-4}$ during first 4K steps, it is then held constant till 30K steps and then decays exponentially to $2.5\mathrm {e}{-6}$ till 200K steps. During training we also added a gaussian noise($\mu =0,\sigma =0.01$) to model weights BIBREF24 starting at 10K steps. We train this model to output grapheme units in all our experiments. We found that the Transformer Transducer models trained much faster ($\approx 1$ day) compared to the an LSTM-based RNN-T model ($\approx 3.5$ days), with a similar number of parameters.

Answers:
No