qinchuanhui/UDA-QA huggingface dataset:
Question: Is human evaluation performed?

Evidence:
raw_evidence: BLEU: We use the Bilingual Evaluation Understudy (BLEU) BIBREF34 metric which is commonly used in machine translation tasks. The BLEU metric can be used to evaluate dialogue generation models as in BIBREF5, BIBREF35. The BLEU metric is a word-overlap metric which computes the co-occurrence of N-grams in the reference and the generated response and also applies the brevity penalty which tries to penalize far too short responses which are usually not desired in task-oriented chatbots. We compute the BLEU score using all generated responses of our systems.
Per-turn Accuracy: Per-turn accuracy measures the similarity of the system generated response versus the target response. Eric and Manning eric2017copy used this metric to evaluate their systems in which they considered their response to be correct if all tokens in the system generated response matched the corresponding token in the target response. This metric is a little bit harsh, and the results may be low since all the tokens in the generated response have to be exactly in the same position as in the target response.
Per-Dialogue Accuracy: We calculate per-dialogue accuracy as used in BIBREF8, BIBREF5. For this metric, we consider all the system generated responses and compare them to the target responses. A dialogue is considered to be true if all the turns in the system generated responses match the corresponding turns in the target responses. Note that this is a very strict metric in which all the utterances in the dialogue should be the same as the target and in the right order.
F1-Entity Score: Datasets used in task-oriented chores have a set of entities which represent user preferences. For example, in the restaurant domain chatbots common entities are meal, restaurant name, date, time and the number of people (these are usually the required entities which are crucial for making reservations, but there could be optional entities such as location or rating). Each target response has a set of entities which the system asks or informs the user about. Our models have to be able to discern these specific entities and inject them into the generated response. To evaluate our models we could use named-entity recognition evaluation metrics BIBREF36. The F1 score is the most commonly used metric used for the evaluation of named-entity recognition models which is the harmonic average of precision and recall of the model. We calculate this metric by micro-averaging over all the system generated responses.

Answers:
No
Question: What are the three datasets used?

Evidence:
raw_evidence: We use three different datasets for training the models. We use the Dialogue State Tracking Competition 2 (DSTC2) dataset BIBREF27 which is the most widely used dataset for research on task-oriented chatbots. We also used two other datasets recently open-sourced by Google Research BIBREF28 which are M2M-sim-M (dataset in movie domain) and M2M-sim-R (dataset in restaurant domain). M2M stands for Machines Talking to Machines which refers to the framework with which these two datasets were created. In this framework, dialogues are created via dialogue self-play and later augmented via crowdsourcing. We trained on our models on different datasets in order to make sure the results are not corpus-biased. Table TABREF12 shows the statistics of these three datasets which we will use to train and evaluate the models.

Answers:
DSTC2, M2M-sim-M, M2M-sim-R
Question: Did they experiment with the corpus?

Evidence:
raw_evidence: In this section, we propose new simple disentanglement models that perform better than prior methods, and re-examine prior work. The models we consider are:

Answers:
Yes
Question: How were the feature representations evaluated?

Evidence:
raw_evidence: Our work extends these explorations of the geometry of internal representations. Investigating how BERT represents syntax, we describe evidence that attention matrices contain grammatical representations. We also provide mathematical arguments that may explain the particular form of the parse tree embeddings described in BIBREF8 . Turning to semantics, using visualizations of the activations created by different pieces of text, we show suggestive evidence that BERT distinguishes word senses at a very fine level. Moreover, much of this semantic information appears to be encoded in a relatively low-dimensional subspace.
To formalize what it means for attention matrices to encode linguistic features, we use an attention probe, an analog of edge probing BIBREF11 . An attention probe is a task for a pair of tokens, $(token_i, token_j)$ where the input is a model-wide attention vector formed by concatenating the entries $a_{ij}$ in every attention matrix from every attention head in every layer. The goal is to classify a given relation between the two tokens. If a linear model achieves reliable accuracy, it seems reasonable to say that the model-wide attention vector encodes that relation. We apply attention probes to the task of identifying the existence and type of dependency relation between two words.

Answers:
attention probes, using visualizations of the activations created by different pieces of text
Question: What linguistic features were probed for?

Evidence:
raw_evidence: To formalize what it means for attention matrices to encode linguistic features, we use an attention probe, an analog of edge probing BIBREF11 . An attention probe is a task for a pair of tokens, $(token_i, token_j)$ where the input is a model-wide attention vector formed by concatenating the entries $a_{ij}$ in every attention matrix from every attention head in every layer. The goal is to classify a given relation between the two tokens. If a linear model achieves reliable accuracy, it seems reasonable to say that the model-wide attention vector encodes that relation. We apply attention probes to the task of identifying the existence and type of dependency relation between two words.
Our first experiment is an exploratory visualization of how word sense affects context embeddings. For data on different word senses, we collected all sentences used in the introductions to English-language Wikipedia articles. (Text outside of introductions was frequently fragmentary.) We created an interactive application, which we plan to make public. A user enters a word, and the system retrieves 1,000 sentences containing that word. It sends these sentences to BERT-base as input, and for each one it retrieves the context embedding for the word from a layer of the user's choosing.

Answers:
dependency relation between two words, word sense
Question: Does the paper describe experiments with real humans?

Evidence:
raw_evidence: Thus, when size adjectives are noisier than color adjectives, the model produces overinformative referring expressions with color, but not with size – precisely the pattern observed in the literature BIBREF5 , BIBREF0 . Note also that no difference in adjective cost is necessary for obtaining the overinformativeness asymmetry, though assuming a greater cost for size than for color does further increase the observed asymmetry. We defer a discussion of costs to Section "Experiment 1: scene variation in modified referring expressions" , where we infer the best parameter values for both the costs and the semantic values of size and color, given data from a reference game experiment.
We recruited 58 pairs of participants (116 participants total) over Amazon's Mechanical Turk who were each paid $1.75 for their participation. Data from another 7 pairs who prematurely dropped out of the experiment and who could therefore not be compensated for their work, were also included. Here and in all other experiments reported in this paper, participants' IP address was limited to US addresses and only participants with a past work approval rate of at least 95% were accepted.

Answers:
Yes
Question: What are bottleneck features?

Evidence:
raw_evidence: One way to re-use information extracted from other multilingual corpora is to use multilingual bottleneck features (BNFs), which has shown to perform well in conventional ASR as well as intrinsic evaluations BIBREF19 , BIBREF26 , BIBREF27 , BIBREF20 , BIBREF28 , BIBREF29 . These features are typically obtained by training a deep neural network jointly on several languages for which labelled data is available. The bottom layers of the network are normally shared across all training languages. The network then splits into separate parts for each of the languages, or has a single shared output. The final output layer has phone labels or HMM states as targets. The final shared layer often has a lower dimensionality than the input layer, and is therefore referred to as a `bottleneck'. The intuition is that this layer should capture aspects that are common across all the languages. We use such features from a multilingual neural network in our CNN-DTW keyword spotting approach. The BNFs are trained on a set of well-resourced languages different from the target language.

Answers:
Bulgarian, Czech, French, German, Korean, Polish, Portuguese, Russian, Thai, Vietnamese, South African English, These features are typically obtained by training a deep neural network jointly on several languages for which labelled data is available., The final shared layer often has a lower dimensionality than the input layer, and is therefore referred to as a `bottleneck'.
Question: What languages are considered?

Evidence:
raw_evidence: A 6-layer 10-language TDNN was trained on the GlobalPhone corpus, also using 40-high resolution MFCC features as input, as described in BIBREF20 . For speaker adaptation, a 100-dimensional i-vector was appended to the the MFCC input features. The TDNN was trained with a block-softmax, with the hidden layers shared across all languages and a separate output layer for each language. Each of the six hidden layers had 625 dimensions, and was followed by a 39-dimensional bottleneck layer with ReLU activations and batch normalisation. Training was accomplished using the Kaldi Babel receipe using 198 hours of data in 10 languages (Bulgarian, Czech, French, German, Korean, Polish, Portuguese, Russian, Thai, Vietnamese) from GlobalPhone.

Answers:
Bulgarian, Czech, French, German, Korean, Polish, Portuguese, Russian, Thai, Vietnamese
Question: How do they obtain language identities?

Evidence:
raw_evidence: In this paper, we propose an improved RNN-T model with language bias to alleviate the problem. The model is trained to predict language IDs as well as the subwords. To ensure the model can learn CS information, we add language IDs in the CS point of transcription, as illustrated in Fig. 1. In the figure, we use the arrangements of different geometric icons to represent the CS distribution. Compared with normal text, the tagged data can bias the RNN-T to predict language IDs in CS points. So our method can model the CS distribution directly, no additional LID model is needed. Then we constrain the input word embedding with its corresponding language ID, which is beneficial for model to learn the language identity information from transcription. In the inference process, the predicted language IDs are used to adjust the output posteriors. The experiment results on CS corpus show that our proposed method outperforms the RNN-T baseline (without language bias) significantly. Overall, our best model achieves 16.2% and 12.9% relative error reduction on two test sets, respectively. To our best knowledge, this is the first attempt of using the RNN-T model with language bias as an end-to-end CSSR strategy.

Answers:
model is trained to predict language IDs as well as the subwords, we add language IDs in the CS point of transcriptio
Question: What other multimodal knowledge base embedding methods are there?

Evidence:
raw_evidence: A number of methods utilize an extra type of information as the observed features for entities, by either merging, concatenating, or averaging the entity and its features to compute its embeddings, such as numerical values BIBREF26 (we use KBLN from this work to compare it with our approach using only numerical as extra attributes), images BIBREF27 , BIBREF28 (we use IKRL from the first work to compare it with our approach using only images as extra attributes), text BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , and a combination of text and image BIBREF35 . Further, BIBREF7 address the multilingual relation extraction task to attain a universal schema by considering raw text with no annotation as extra feature and using matrix factorization to jointly embed KB and textual relations BIBREF36 . In addition to treating the extra information as features, graph embedding approaches BIBREF37 , BIBREF38 consider observed attributes while encoding to achieve more accurate embeddings.

Answers:
merging, concatenating, or averaging the entity and its features to compute its embeddings, graph embedding approaches, matrix factorization to jointly embed KB and textual relations
Question: What is the data selection paper in machine translation

Evidence:
raw_evidence: We observe that merely adding more tasks cannot provide much improvement on the target task. Thus, we propose two MTL training algorithms to improve the performance. The first method simply adopts a sampling scheme, which randomly selects training data from the auxiliary tasks controlled by a ratio hyperparameter; The second algorithm incorporates recent ideas of data selection in machine translation BIBREF7 . It learns the sample weights from the auxiliary tasks automatically through language models. Prior to this work, many studies have used upstream datasets to augment the performance of MRC models, including word embedding BIBREF5 , language models (ELMo) BIBREF8 and machine translation BIBREF1 . These methods aim to obtain a robust semantic encoding of both passages and questions. Our MTL method is orthogonal to these methods: rather than enriching semantic embedding with external knowledge, we leverage existing MRC datasets across different domains, which help make the whole comprehension process more robust and universal. Our experiments show that MTL can bring further performance boost when combined with contextual representations from pre-trained language models, e.g., ELMo BIBREF8 .
We develop a novel re-weighting method to resolve these problems, using ideas inspired by data selection in machine translation BIBREF26 , BIBREF7 . We use $(Q^{k},P^{k},A^{k})$ to represent a data point from the $k$ -th task for $1\le k\le K$ , with $k=1$ being the target task. Since the passage styles are hard to evaluate, we only evaluate data points based on $Q^{k}$ and $A^k$ . Note that only data from auxiliary task ( $2\le k\le K$ ) is re-weighted; target task data always have weight 1.

Answers:
BIBREF7, BIBREF26
Question: Do they compare computational time of AM-softmax versus Softmax?

Evidence:

Answers:
No
Question: Do they visualize the difference between AM-Softmax and regular softmax?

Evidence:
raw_evidence: In the epoch 96, the proposed method has already an FER more than INLINEFORM0 better than the original SincNet for almost every value of INLINEFORM1 excluding INLINEFORM2 . The difference keeps increasing over the epochs, and at epoch 352 the proposed method has an FER of INLINEFORM3 ( INLINEFORM4 ) against INLINEFORM5 from SincNet, which means that at this epoch AM-SincNet has a Frame Error Rate approximately INLINEFORM6 better than traditional SincNet. The Figure FIGREF7 plots the Frame Error Rate on the test data for both methods along the training epochs. For the AM-SincNet, we used the margin parameter INLINEFORM7 .
FLOAT SELECTED: Fig. 3. Comparison of Frame Error Rate (%) from SincNet and AM-SincNet (m=0.50) over the training epochs for TIMIT dataset.

Answers:
Yes
Question: what metrics were used for evaluation?

Evidence:
raw_evidence: BLEU BIBREF28 , METEOR BIBREF29 , ROUGE-L BIBREF30 and CIDEr BIBREF31 are common evaluation metrics in image and video description, the first three were originally proposed to evaluate machine translation at the earliest and CIDEr was proposed to evaluate image description with sufficient reference sentences. To quantitatively evaluate the performance of our bidirectional recurrent based approach, we adopt METEOR metric because of its robust performance. Contrasting to the other three metrics, METEOR could capture semantic aspect since it identifies all possible matches by extracting exact matcher, stem matcher, paraphrase matcher and synonym matcher using WordNet database, and compute sentence level similarity scores according to matcher weights. The authors of CIDEr also argued for that METEOR outperforms CIDEr when the reference set is small BIBREF31 .

Answers:
METEOR
Question: What datasets do they use for the tasks?

Evidence:
raw_evidence: To compare fairly, we train and evaluate these systems on the VLSP corpora. In particular, we conduct experiments on Viet Treebank corpus for POS tagging and chunking tasks, and on VLSP shared task 2016 corpus for NER task. All of these corpora are converted to CoNLL format. The corpus of POS tagging task consists of two columns namely word, and POS tag. For chunking task, there are three columns namely word, POS tag, and chunk in the corpus. The corpus of NER task consists of four columns. The order of these columns are word, POS tag, chunk, and named entity. While NER corpus has been separated into training and testing parts, the POS tagging and chunking data sets are not previously divided. For this reason, we use INLINEFORM0 of these data sets as a training set, and the remaining as a testing set. Because our system adopts early stopping method, we use INLINEFORM1 of these data sets from the training set as a development set when training NNVLP system. Table TABREF24 and Table TABREF25 shows the statistics of each corpus.

Answers:
 Viet Treebank corpus for POS tagging and chunking tasks, and on VLSP shared task 2016 corpus for NER task
Question: What evaluation metrics do they use?

Evidence:
raw_evidence: For de-identification tasks, the three metrics we will use to evaluate the performance of our architecture are Precision, Recall and INLINEFORM0 score as defined below. We will compute both the binary INLINEFORM1 score and the three metrics for each PHI type for both data sets. Note that binary INLINEFORM2 score calculates whether or not a token was identified as a PHI as opposed to correctly predicting the right PHI type. For de-identification, we place more importance on identifying if a token was a PHI instance with correctly predicting the right PHI type as a secondary objective. INLINEFORM3 INLINEFORM4

Answers:
Precision, Recall and INLINEFORM0 score
Question: Do they use BERT?

Evidence:

Answers:
No
Question: What is their baseline?

Evidence:
raw_evidence: As a benchmark, we will use the results of the systems by Burckhardt et al. BIBREF22 , Liu et al. BIBREF18 , Dernoncourt et al. BIBREF9 and Yang et al. BIBREF10 on the i2b2 dataset and the performance of Burckhardt et al. on the nursing corpus. Note that Burckhardt et al. used the entire data set for their results as it is an unsupervised learning system while we had to split our data set into 60% training data and 40% testing data.

Answers:
Burckhardt et al. BIBREF22, Liu et al. BIBREF18, Dernoncourt et al. BIBREF9, Yang et al. BIBREF10
Question: Which two datasets is the system tested on?

Evidence:
raw_evidence: The two main data sets that we will use to evaluate our architecture are the 2014 i2b2 de-identification challenge data set BIBREF2 and the nursing notes corpus BIBREF3 .

Answers:
2014 i2b2 de-identification challenge data set BIBREF2, nursing notes corpus BIBREF3
Question: How they compare varioius mechanisms in terms of naturalness?

Evidence:
raw_evidence: We evaluate CBA, LSA, DCA, and GMMv2b using mean opinion score (MOS) naturalness judgments produced by a crowd-sourced pool of raters. Scores range from 1 to 5, with 5 representing “completely natural speech”. The Lessac and LJ models are evaluated on their respective test sets (hence in-domain), and the results are shown in Table TABREF17. We see that for these utterances, the LSA, DCA, and GMMV2b mechanisms all produce equivalent scores around 4.3, while the content-based mechanism is a bit lower due to occasional catastrophic attention failures.

Answers:
using mean opinion score (MOS) naturalness judgments produced by a crowd-sourced pool of raters
Question: What datasets are used?

Evidence:
raw_evidence: To this end, various datasets have been created to benchmark progress in the field. In hate speech detection, recently BIBREF5 compiled and released a dataset of over 24,000 tweets labeled as containing hate speech, offensive language, or neither. The TRAC shared task on Aggression Identification, a dataset of over 15,000 Facebook comments labeled with varying levels of aggression, was released as part of a competition BIBREF14. In order to benchmark toxic comment detection, The Wikipedia Toxic Comments dataset (which we study in this work) was collected and extracted from Wikipedia Talk pages and featured in a Kaggle competition BIBREF12, BIBREF15. Each of these benchmarks examine only single-turn utterances, outside of the context in which the language appeared. In this work we recommend that future systems should move beyond classification of singular utterances and use contextual information to help identify offensive language.

Answers:
The Wikipedia Toxic Comments dataset
Question: Is the origin of the dialogues in corpus some video game and what game is that?

Evidence:

Answers:
No
Question: How the authors made sure that corpus is clean despite being crowdsourced?

Evidence:
raw_evidence: The main contribution of our work is thus a new parallel data-to-text NLG corpus that (1) is more conversational, rather than information seeking or question answering, and thus more suitable for an open-domain dialogue system, (2) represents a new, unexplored domain which, however, has excellent potential for application in conversational agents, and (3) has high-quality, manually cleaned human-produced utterances.

Answers:
manually cleaned human-produced utterances
Question: Do they build a generative probabilistic language model for sign language?

Evidence:

Answers:
No
Question: How do they obtain human generated policies?

Evidence:
raw_evidence: Specifying reward functions in reinforcement learning dialogue agents is often a difficult aspect. We propose to derive rewards from human-human dialogues by assigning positive values to contextualised responses seen in the data, and negative values to randomly chosen responses due to lacking coherence (also referred to as `non-human-like responses') – see example in Tables TABREF29 and TABREF30 . An episode or dialogue reward can thus be computed as INLINEFORM0 , where index INLINEFORM1 refers to the dialogue in focus, index INLINEFORM2 to the dialogue turn in focus, and INLINEFORM3 is given according to DISPLAYFORM0

Answers:
derive rewards from human-human dialogues by assigning positive values to contextualised responses seen in the data, and negative values to randomly chosen responses due to lacking coherence
Question: How many agents do they ensemble over?

Evidence:
raw_evidence: Ensemble, which selects a sentence using 100 agents trained on clustered dialogues as described in section SECREF4 – the agent in focus is chosen using a regressor as predictor of dialogue reward INLINEFORM0 using a similar neural net as the ChatDQN agents except for the final layer having one node and using Batch Normalisation BIBREF44 between hidden layers as in BIBREF36 ;

Answers:
100