qinchuanhui/UDA-QA huggingface dataset:
Question: What are the best within-language data augmentation methods?

Evidence:
raw_evidence: In this section, we consider 3 categories of data augmentation techniques that are effectively applicable to video datasets.
To further increase training data size and diversity, we can create new audios via superimposing each original audio with additional noisy audios in time domain. To obtain diverse noisy audios, we use AudioSet, which consists of 632 audio event classes and a collection of over 2 million manually-annotated 10-second sound clips from YouTube videos BIBREF28.
We consider applying the frequency and time masking techniques – which are shown to greatly improve the performance of end-to-end ASR models BIBREF23 – to our hybrid systems. Similarly, they can be applied online during each epoch of LF-MMI training, without the need for realignment.
Consider each utterance (i.e. after the audio segmentation in Section SECREF5), and we compute its log mel spectrogram with $\nu $ dimension and $\tau $ time steps:
Frequency masking is applied $m_F$ times, and each time the frequency bands $[f_0$, $f_0+ f)$ are masked, where $f$ is sampled from $[0, F]$ and $f_0$ is sampled from $[0, \nu - f)$.
Time masking is optionally applied $m_T$ times, and each time the time steps $[t_0$, $t_0+ t)$ are masked, where $t$ is sampled from $[0, T]$ and $t_0$ is sampled from $[0, \tau - t)$.
Data augmentation ::: Speed and volume perturbation
Both speed and volume perturbation emulate mean shifts in spectrum BIBREF18, BIBREF19. To perform speed perturbation of the training data, we produce three versions of each audio with speed factors $0.9$, $1.0$, and $1.1$. The training data size is thus tripled. For volume perturbation, each audio is scaled with a random variable drawn from a uniform distribution $[0.125, 2]$.

Answers:
Frequency masking, Time masking, Additive noise, Speed and volume perturbation
Question: How much of the ASR grapheme set is shared between languages?

Evidence:
raw_evidence: The character sets of these 7 languages have little overlap except that (i) they all include common basic Latin alphabet, and (ii) both Hindi and Marathi use Devanagari script. We took the union of 7 character sets therein as the multilingual grapheme set (Section SECREF2), which contained 432 characters. In addition, we deliberately split 7 languages into two groups, such that the languages within each group were more closely related in terms of language family, orthography or phonology. We thus built 3 multilingual ASR models trained on:

Answers:
Little overlap except common basic Latin alphabet and that Hindi and Marathi languages use same script.
Question: What is the performance of the model for the German sub-task A?

Evidence:
raw_evidence: The performance of our models across different languages for sub-task A are shown in table TABREF19. Our model got the first position in the German sub-task with a macro F1 score of 0.62. The results of sub-task B and sub-task C is shown in table TABREF20 and TABREF21 respectively.

Answers:
macro F1 score of 0.62
Question: Is the model tested for language identification?

Evidence:

Answers:
No
Question: Is the model compared to a baseline model?

Evidence:

Answers:
No
Question: What are the languages used to test the model?

Evidence:
raw_evidence: The dataset at HASOC 2019 were given in three languages: Hindi, English, and German. Dataset in Hindi and English had three subtasks each, while German had only two subtasks. We participated in all the tasks provided by the organisers and decided to develop a single model that would be language agnostic. We used the same model architecture for all the three languages.
In the results of subtask A, models are mainly affected by imbalance of the dataset. The training dataset of Hindi dataset was more balanced than English or German dataset. Hence, the results were around 0.78. As the dataset in German language was highly imbalanced, the results drops to 0.62. In subtask B, the highest F1 score reached was by the profane class for each language in table TABREF20. The model got confused between OFFN, HATE and PRFN labels which suggests that these models are not able to capture the context in the sentence. The subtask C was again a case of imbalanced dataset as targeted(TIN) label gets the highest F1 score in table TABREF21.

Answers:
Hindi, English and German (German task won)
Question: What datasets did they use?

Evidence:
raw_evidence: We present an extensive comparison of the differences in recognition accuracy for eight languages (Sec. SECREF5 ) and compare the accuracy of models trained on publicly available datasets where available (Sec. SECREF4 ). In addition, we propose a new standard experimental protocol for the IBM-UB-1 dataset BIBREF25 (Sec. SECREF50 ) to enable easier comparison between approaches in the future.
The IAM-OnDB dataset BIBREF42 is probably the most used evaluation dataset for online handwriting recognition. It consists of 298 523 characters in 86 272 word instances from a dictionary of 11 059 words written by 221 writers. We use the standard IAM-OnDB dataset separation: one training set, two validations sets and a test set containing 5 363, 1 438, 1 518 and 3 859 written lines, respectively. We tune the decoder weights using the validation set with 1 438 items and report error rates on the test set.
We provide an evaluation of our production system trained on our in-house datasets applied to a number of publicly available benchmark datasets from the literature. Note that for all experiments presented in this section we evaluate our current live system without any tuning specifec to the tasks at hand.
The ICDAR-2013 Competition for Online Handwriting Chinese Character Recognition BIBREF45 introduced a dataset for classifying the most common Chinese characters. We report the error rates in comparison to published results from the competition and more recent work done by others in Table TABREF56 .
In the ICFHR2018 Competition on Vietnamese Online Handwritten Text Recognition using VNOnDB BIBREF50 , our production system was evaluated against other systems. The system used in the competition is the one reported and described in this paper. Due to licensing restrictions we were unable to do any experiments on the competition training data, or specific tuning for the competition, which was not the case for the other systems mentioned here.

Answers:
IBM-UB-1 dataset BIBREF25, IAM-OnDB dataset BIBREF42, The ICDAR-2013 Competition for Online Handwriting Chinese Character Recognition BIBREF45, ICFHR2018 Competition on Vietnamese Online Handwritten Text Recognition using VNOnDB BIBREF50
Question: Do they report results only on English data?

Evidence:
raw_evidence: Dos and Don'ts for the Moral Choice Machine. The verb extraction identifies the most positive and most negative associated verbs in vocabulary, to infer socially desired and neglected behaviour. BIBREF0 (BIBREF0) extracted them with the general positive and negative association sets on the Google Slim embedding. Since those sets are expected to reflect social norms, they are referred as Dos and Don'ts hereafter.
Summarized, even though the contained positive verbs are quite diverse, all of them carry a positive attitude. Some of the verbs are related to celebration or travelling, others to love matters or physical closeness. All elements of the above set are rather of general and unspecific nature. Analogously, some of the negative words just describe inappropriate behaviour, like slur or misdeal, whereas others are real crimes as murder. As BIBREF0 (BIBREF0) describe, the listed words can be accepted as commonly agreed Dos and Don'ts.

Answers:
Yes
Question: What is the Moral Choice Machine?

Evidence:
raw_evidence: BIBREF0 (BIBREF0) developed Moral Choice Machine computes the cosine similarity in a sentence embedding space of an arbitrary action embedded in question/answer pairs. This is illustrated in Fig. FIGREF16 for the moral bias of the action murder. Since murdering is a quite destructive and generally refused behaviour, the questions are expected to lie closer to the denying response and thus to yield a negative bias. To create a more meaningful and comprehensive statistic, several question/answer prompts were conflated to a question/answer template (cf. Tab. TABREF15). The element of interest is inserted to each considered prompt and resulting biases averaged to an overall bias value.

Answers:
Moral Choice Machine computes the cosine similarity in a sentence embedding space of an arbitrary action embedded in question/answer pairs
Question: How is moral bias measured?

Evidence:
raw_evidence: Analogous to word embeddings, sentence embeddings, e.g. the Universal Sentence Encoder BIBREF8 and Sentence-BERT BIBREF6, allow one to calculate the cosine similarity of various different sentences, as for instance the similarity of a question and the corresponding answer. The more appropriate a specific answer is to a given question, the stronger is its cosine similarity expected to be. When considering two opposite answers, it is therefore possible to determine a bias value:
where $\vec{q}$ is the vector representation of the question and $\vec{a}$ and $\vec{b}$ the representations of the two answers/choices. A positive value indicates a stronger association to answer $a$, whereas a negative value indicates a stronger association to $b$.

Answers:
Answer with content missing: (formula 1) bias(q, a, b) = cos(a, q) − cos(b, q)
Bias is calculated as substraction of cosine similarities of question and some answer for two opposite answers.
Question: How do the authors define deontological ethical reasoning?

Evidence:
raw_evidence: With regards to complexity and intangibility of ethics and morals, we restrict ourselves to a rather basic implementation of this construct, following the theories of deontological ethics. These ask which choices are morally required, forbidden, or permitted instead of asking which kind of a person we should be or which consequences of our actions are to be preferred. Thus, norms are understood as universal rules of what to do and what not to do. Therefore, we focus on the valuation of social acceptance in single verbs and single verbs with surrounding context information —e.g. trust my friend or trust a machine— to figure out which of them represent a Do and which tend to be a Don't. Because we specifically chose templates in the first person, i.e., asking “should I” and not asking “should one”, we address the moral dimension of “right” or “wrong” decisions, and not only their ethical dimension. This is the reason why we will often use the term “moral”, although we actually touch upon “ethics” and “moral”. To measure the valuation, we make use of implicit association tests (IATs) and their connections to word embeddings.

Answers:
These ask which choices are morally required, forbidden, or permitted, norms are understood as universal rules of what to do and what not to do
Question: How does framework automatically chooses different curricula at the evolving learning process according to the learning status of the neural dialogue generation model?

Evidence:
raw_evidence: The adaptive multi-curricula learning framework is established upon the reinforcement learning (RL) paradigm. Figure FIGREF18 illustrates the overall learning process. The multi-curricula learning scheme is scheduled according to the model's performance on the validation set, where the scheduling mechanism acts as the policy $\pi $ interacting with the dialogue model to acquire the learning status $s$. The reward of the multi-curricula learning mechanism $m_t$ indicates how well the current dialogue model performs. A positive reward is expected if a multi-curricula scheduling action $a_t$ brings improvements on the model's performance, and the current mini-batch of training samples is drawn consulting with the scheduling action $a_t$. The neural dialogue generation model learns from those mini-batches, resulting with a new learning status $s_{t+1}$. The adaptive multi-curricula learning framework is optimized to maximize the reward. Such learning process loops continuously until the performance of the neural dialogue generation model converges.

Answers:
The multi-curricula learning scheme is scheduled according to the model's performance on the validation set, where the scheduling mechanism acts as the policy $\pi $ interacting with the dialogue model to acquire the learning status $s$. The reward of the multi-curricula learning mechanism $m_t$ indicates how well the current dialogue model performs.
Question: What human judgement metrics are used?

Evidence:
raw_evidence: We conduct a human evaluation to validate the effectiveness of the proposed multi-curricula learning framework. We employ the DailyDialog as the evaluation corpus since it is closer to our daily conversations and easier for humans to make the judgment. We randomly sampled 100 cases from the test set and compared the generated responses of the models trained with the vanilla learning procedure and the multi-curricula learning framework. Three annotators, who have no knowledge about which system the response is from, are then required to evaluate among win (response$_1$ is better), loss (response$_2$ is better) and tie (they are equally good or bad) independently, considering four aspects: coherence, logical consistency, fluency and diversity. Cases with different rating results are counted as “tie”. Table TABREF25 reveals the results of the subjective evaluation. We observe that our multi-curricula learning framework outperforms the vanilla training method on all the five dialogue models and the kappa scores indicate that the annotators came to a fair agreement in the judgment. We checked the cases on which the vanilla training method loses to our multi-curricula learning method and found that the vanilla training method usually leads to irrelevant, generic and repetitive responses, while our method effectively alleviates such defects.

Answers:
coherence, logical consistency, fluency and diversity
Question: What automatic evaluation metrics are used?

Evidence:
raw_evidence: We perform experiments using the following state-of-the-art models: (1) SEQ2SEQ: a sequence-to-sequence model with attention mechanisms BIBREF21, (2) CVAE: a conditional variational auto-encoder model with KL-annealing and a BOW loss BIBREF2, (3) Transformer: an encoder-decoder architecture relying solely on attention mechanisms BIBREF22, (4) HRED: a generalized sequence-to-sequence model with the hierarchical RNN encoder BIBREF23, (5) DialogWAE: a conditional Wasserstein auto-encoder, which models the distribution of data by training a GAN within the latent variable space BIBREF6. We adopt several standard metrics widely used in existing works to measure the performance of dialogue generation models, including BLEU BIBREF24, embedding-based metrics (Average, Extrema, Greedy and Coherence) BIBREF25, BIBREF26, entropy-based metrics (Ent-{1,2}) BIBREF0 and distinct metrics (Dist-{1,2,3} and Intra-{1,2,3}) BIBREF1, BIBREF6.

Answers:
BLEU, embedding-based metrics (Average, Extrema, Greedy and Coherence), , entropy-based metrics (Ent-{1,2}), distinct metrics (Dist-{1,2,3} and Intra-{1,2,3})
Question: What state of the art models were used in experiments?

Evidence:
raw_evidence: We perform experiments using the following state-of-the-art models: (1) SEQ2SEQ: a sequence-to-sequence model with attention mechanisms BIBREF21, (2) CVAE: a conditional variational auto-encoder model with KL-annealing and a BOW loss BIBREF2, (3) Transformer: an encoder-decoder architecture relying solely on attention mechanisms BIBREF22, (4) HRED: a generalized sequence-to-sequence model with the hierarchical RNN encoder BIBREF23, (5) DialogWAE: a conditional Wasserstein auto-encoder, which models the distribution of data by training a GAN within the latent variable space BIBREF6. We adopt several standard metrics widely used in existing works to measure the performance of dialogue generation models, including BLEU BIBREF24, embedding-based metrics (Average, Extrema, Greedy and Coherence) BIBREF25, BIBREF26, entropy-based metrics (Ent-{1,2}) BIBREF0 and distinct metrics (Dist-{1,2,3} and Intra-{1,2,3}) BIBREF1, BIBREF6.

Answers:
SEQ2SEQ, CVAE, Transformer, HRED, DialogWAE
Question: What three publicly available coropora are used?

Evidence:
raw_evidence: Intuitively, a well-organized curriculum should provide the model learning with easy dialogues first, and then gradually increase the curriculum difficulty. However, currently, there is no unified approach for dialogue complexity evaluation, where the complexity involves multiple aspects of attributes. In this paper, we prepare the syllabus for dialogue learning with respect to five dialogue attributes. To ensure the universality and general applicability of the curriculum, we perform an in-depth investigation on three publicly available conversation corpora, PersonaChat BIBREF12, DailyDialog BIBREF13 and OpenSubtitles BIBREF7, consisting of 140 248, 66 594 and 358 668 real-life conversation samples, respectively.

Answers:
PersonaChat BIBREF12, DailyDialog BIBREF13, OpenSubtitles BIBREF7
Question: Which datasets do they use?

Evidence:
raw_evidence: We validate our approach on the Wikipedia toxic comments dataset BIBREF18 . Our fairness experiments show that the classifiers trained with our method achieve the same performance, if not better, on the original task, while improving AUC and fairness metrics on a synthetic, unbiased dataset. Models trained with our technique also show lower attributions to identity terms on average. Our technique produces much better word vectors as a by-product when compared to the baseline. Lastly, by setting an attribution target of 1 on toxic words, a classifier trained with our objective function achieves better performance when only a subset of the data is present.

Answers:
 Wikipedia toxic comments
Question: What metrics are used for evaluation?

Evidence:
raw_evidence: The Density Ratio method consistently outperformed Shallow Fusion for the cross-domain scenarios examined, with and without fine-tuning to audio data from the target domain. Furthermore, the gains in WER over the baseline are significantly larger for the Density Ratio method than for Shallow Fusion, with up to 28% relative reduction in WER (17.5% $\rightarrow $ 12.5%) compared to up to 17% relative reduction (17.5% $\rightarrow $ 14.5%) for Shallow Fusion, in the no fine-tuning scenario.

Answers:
word error rate
Question: How much training data is used?

Evidence:
raw_evidence: The following data sources were used to train the RNN-T and associated RNN-LMs in this study.
Source-domain baseline RNN-T: approximately 120M segmented utterances (190,000 hours of audio) from YouTube videos, with associated transcripts obtained from semi-supervised caption filtering BIBREF28.
Source-domain normalizing RNN-LM: transcripts from the same 120M utterance YouTube training set. This corresponds to about 3B tokens of the sub-word units used (see below, Section SECREF30).
Target-domain RNN-LM: 21M text-only utterance-level transcripts from anonymized, manually transcribed audio data, representative of data from a Voice Search service. This corresponds to about 275M sub-word tokens.
Target-domain RNN-T fine-tuning data: 10K, 100K, 1M and 21M utterance-level {audio, transcript} pairs taken from anonymized, transcribed Voice Search data. These fine-tuning sets roughly correspond to 10 hours, 100 hours, 1000 hours and 21,000 hours of audio, respectively.

Answers:
163,110,000 utterances
Question: How is the training data collected?

Evidence:
raw_evidence: The following data sources were used to train the RNN-T and associated RNN-LMs in this study.
Source-domain baseline RNN-T: approximately 120M segmented utterances (190,000 hours of audio) from YouTube videos, with associated transcripts obtained from semi-supervised caption filtering BIBREF28.
Source-domain normalizing RNN-LM: transcripts from the same 120M utterance YouTube training set. This corresponds to about 3B tokens of the sub-word units used (see below, Section SECREF30).
Target-domain RNN-LM: 21M text-only utterance-level transcripts from anonymized, manually transcribed audio data, representative of data from a Voice Search service. This corresponds to about 275M sub-word tokens.
Target-domain RNN-T fine-tuning data: 10K, 100K, 1M and 21M utterance-level {audio, transcript} pairs taken from anonymized, transcribed Voice Search data. These fine-tuning sets roughly correspond to 10 hours, 100 hours, 1000 hours and 21,000 hours of audio, respectively.

Answers:
from YouTube videos, with associated transcripts obtained from semi-supervised caption filtering, from a Voice Search service
Question: was bert used?

Evidence:
raw_evidence: BERT BIBREF6 is a state-of-the-art model, which has been pre-trained on a large corpus and is suitable to be fine-tuned for various downstream NLP tasks. The main innovation between this model and existing language models is in how the model is trained. For BERT, the text conditioning happens on both the left and right context of every word and is therefore bidirectional. In previous models BIBREF7, a unidirectional language model was usually used in the pre-training. With BERT, two fully unsupervised tasks are performed. The Masked Language Model and the Next Sentence Prediction (NSP).
For this study, the NSP is used as a proxy for the relevance of response. Furthermore, in order to improve performance, we fine-tune on a customized dataset which achieved an accuracy of 82.4%. For the main analysis, we used the single-turn dataset, which gave us a correlation of 0.28 between the mean of the AMT evaluation and the BERT NSP. Next, we put each score into a category. For example, if the average score is 2.3, this would be placed in category 2. We then displayed the percentage of positive and negative predictions in a histogram for each of the categories. As seen in Figure FIGREF5, a clear pattern is seen between the higher scores and the positive prediction, and the lower scores and the negative predictions. details of how they are combined to create a final classification layer.

Answers:
Yes
Question: what datasets did they use?

Evidence:
raw_evidence: Single-Turn: This dataset consists of single-turn instances of statements and responses from the MiM chatbot developed at Constellation AI BIBREF5. The responses provided were then evaluated using Amazon Mechanical Turk (AMT) workers. A total of five AMT workers evaluated each of these pairs. The mean of the five evaluations is then used as the target variable. A sample can be seen in Table TABREF3. This dataset was used during experiments with results published in the Results section.
Multi-Turn: This dataset is taken from the ConvAI2 challenge and consists of various types of dialogue that have been generated by human-computer conversations. At the end of each dialogue, an evaluation score has been given, for each dialogue, between 1–4.

Answers:
Single-Turn, Multi-Turn
Question: which existing metrics do they compare with?

Evidence:
raw_evidence: To create a final metric, we combine the individual components from Section 2 as features into a Support Vector Machine. The final results for our F1-score from this classification technique are 0.52 and 0.31 for the single and multi-turn data respectively.
We compare our results for both the single-turn and multi-turn experiments to the accuracies on the test data based off the BLEU score. We see an increase of 6% for our method with respect to the BLEU score in the single turn data, and a no change when using the multi-turn test set.

Answers:
F1-score, BLEU score