qinchuanhui/UDA-QA huggingface dataset:
Question: How do they quantify alignment between the embeddings of a document and its translation?

Evidence:
raw_evidence: We construct the embeddings for each document using BERT models finetuned on MLDoc. We mean-pool each document embedding to create a single vector per document. We then calculate the cosine similarity between the embeddings for the English document and its translation. In Table TABREF21 , we observe that the median cosine similarity increases dramatically with adversarial training, which suggests that the embeddings became more language-independent.

Answers:
median cosine similarity
Question: Does adversarial learning have stronger performance gains for text classification, or for NER?

Evidence:
raw_evidence: In Table TABREF13 , we report the classification accuracy for all of the languages in MLDoc. Generally, adversarial training improves the accuracy across all languages, and the improvement is sometimes dramatic versus the BERT non-adversarial baseline.
In Table TABREF19 , we report the F1 scores for all of the CoNLL NER languages. When combined with adversarial learning, the BERT cross-lingual F1 scores increased for German over the non-adversarial baseline, and the scores remained largely the same for Spanish and Dutch. Regardless, the BERT zero-resource performance far exceeds the results published in previous work.

Answers:
classification
Question: Do any of the evaluations show that adversarial learning improves performance in at least two different language families?

Evidence:
raw_evidence: In Figure FIGREF15 , we plot the zero-resource German and Japanese test set accuracy as a function of the number of steps taken, with and without adversarial training. The plot shows that the variation in the test accuracy is reduced with adversarial training, which suggests that the cross-lingual performance is more consistent when adversarial training is applied. (We note that the batch size and learning rates are the same for all the languages in MLDoc, so the variation seen in Figure FIGREF15 are not affected by those factors.)

Answers:
Yes
Question: what experiments are conducted?

Evidence:
raw_evidence: We carried out a reliability study for the proposed scheme using two pairs of expert annotators, P1 and P2. The annotators were trained by coding one transcript at a time and discussing disagreements. Five text-based discussions were used for testing reliability after training: pair P1 annotated discussions of The Bluest Eye, Death of a Salesman, and Macbeth, while pair P2 annotated two separate discussions of Ain't I a Woman. 250 argument moves (discussed by over 40 students and consisting of over 8200 words) were annotated. Inter-rater reliability was assessed using Cohen's kappa: unweighted for argumentation and knowledge domain, but quadratic-weighted for specificity given its ordered labels.

Answers:
a reliability study for the proposed scheme
Question: how do they measure discussion quality?

Evidence:
raw_evidence: As a first step towards developing an automated system for detecting the features of student talk that lead to high quality discussions, we propose a new annotation scheme for student talk during ELA “text-based" discussions - that is, discussions that center on a text or piece of literature (e.g., book, play, or speech). The annotation scheme was developed to capture three aspects of classroom talk that are theorized in the literature as important to discussion quality and learning opportunities: argumentation (the process of systematically reasoning in support of an idea), specificity (the quality of belonging or relating uniquely to a particular subject), and knowledge domain (area of expertise represented in the content of the talk). We demonstrate the reliability and validity of our scheme via an annotation study of five transcripts of classroom discussion.

Answers:
Measuring three aspects: argumentation, specificity and knowledge domain.
Question: do they use a crowdsourcing platform?

Evidence:
raw_evidence: We carried out a reliability study for the proposed scheme using two pairs of expert annotators, P1 and P2. The annotators were trained by coding one transcript at a time and discussing disagreements. Five text-based discussions were used for testing reliability after training: pair P1 annotated discussions of The Bluest Eye, Death of a Salesman, and Macbeth, while pair P2 annotated two separate discussions of Ain't I a Woman. 250 argument moves (discussed by over 40 students and consisting of over 8200 words) were annotated. Inter-rater reliability was assessed using Cohen's kappa: unweighted for argumentation and knowledge domain, but quadratic-weighted for specificity given its ordered labels.

Answers:
No
Question: Is the model evaluated against the baseline also on single-aspect sentences?

Evidence:
raw_evidence: Secondly, we compare the performance of three BERT-related methods. The performance of BERT-Original and BERT-Soft are similar by comparing their average scores. The reason may be that the original BERT has already modeled the deep relationships between the sentence and the aspect. BERT-Original can be thought of as a kind of soft-selection approach as BERT-Soft. We also observe that the snippet selection by reinforcement learning improves the performance over soft-selection approaches in almost all settings. However, the improvement of BERT-Hard over BERT-Soft is marginal. The average score of BERT-Hard is better than BERT-Soft by 0.68%. The improvement percentages are between 0.36% and 1.49%, while on the Laptop dataset, the performance of BERT-Hard is slightly weaker than BERT-Soft. The main reason is that the datasets only contain a small portion of multi-aspect sentences with different polarities. The distraction of attention will not impact the sentiment prediction much in single-aspect sentences or multi-aspect sentences with the same polarities.

Answers:
No
Question: Is the accuracy of the opinion snippet detection subtask reported?

Evidence:

Answers:
No
Question: What were the baselines?

Evidence:
raw_evidence: To compare with our model, we select a number of strong summarization models as baseline systems. $\textsc {Lead-X}$ uses the top $X$ sentences as a summary BIBREF19. The value of $X$ is 3 for NYT and CNN/DailyMail and 1 for XSum to accommodate the nature of summary length. $\textsc {PTGen}$ BIBREF4 is the pointer-generator network. $\textsc {DRM}$ BIBREF10 leverages deep reinforcement learning for summarization. $\textsc {TConvS2S}$ BIBREF2 is based on convolutional neural networks. $\textsc {BottomUp}$ BIBREF11 uses a bottom-up approach to generate summarization. ABS BIBREF26 uses neural attention for summary generation. DRGD BIBREF27 is based on a deep recurrent generative decoder. To compare with our pretrain-only model, we include several unsupervised abstractive baselines: SEQ$^3$ BIBREF28 employs the reconstruction loss and topic loss for summarization. BottleSum BIBREF23 leverages unsupervised extractive and self-supervised abstractive methods. GPT-2 BIBREF7 is a large-scaled pretrained language model which can be directly used to generate summaries.

Answers:
$\textsc {Lead-X}$, $\textsc {PTGen}$, $\textsc {DRM}$, $\textsc {TConvS2S}$,  $\textsc {BottomUp}$, ABS, DRGD, SEQ$^3$, BottleSum, GPT-2
Question: What metric was used in the evaluation step?

Evidence:
raw_evidence: We employ the standard ROUGE-1, ROUGE-2 and ROUGE-L metrics BIBREF29 to evaluate all summarization models. These three metrics respectively evaluate the accuracy on unigrams, bigrams and longest common subsequence. ROUGE metrics have been shown to highly correlate with the human judgment BIBREF29. Following BIBREF22, BIBREF23, we use F-measure ROUGE on XSUM and CNN/DailyMail, and use limited-length recall-measure ROUGE on NYT and DUC. In NYT, the prediction is truncated to the length of the ground-truth summaries; in DUC, the prediction is truncated to 75 characters.

Answers:
ROUGE-1, ROUGE-2 and ROUGE-L, F-measure ROUGE on XSUM and CNN/DailyMail, and use limited-length recall-measure ROUGE on NYT and DUC
Question: What did they pretrain the model on?

Evidence:
raw_evidence: Pretraining. We collect three years of online news articles from June 2016 to June 2019. We filter out articles overlapping with the evaluation data on media domain and time range. We then conduct several data cleaning strategies.

Answers:
hree years of online news articles from June 2016 to June 2019
Question: What does the data cleaning and filtering process consist of?

Evidence:
raw_evidence: First, many news articles begin with reporter names, media agencies, dates or other contents irrelevant to the content, e.g. “New York (CNN) –”, “Jones Smith, May 10th, 2018:”. We therefore apply simple regular expressions to remove these prefixes.
Second, to ensure that the summary is concise and the article contains enough salient information, we only keep articles with 10-150 words in the top three sentences and 150-1200 words in the rest, and that contain at least 6 sentences in total. In this way, we filter out i) articles with excessively long content to reduce memory consumption; ii) very short leading sentences with little information which are unlikely to be a good summary. To encourage the model to generate abstrative summaries, we also remove articles where any of the top three sentences is exactly repeated in the rest of the article.
Third, we try to remove articles whose top three sentences may not form a relevant summary. For this purpose, we utilize a simple metric: overlapping words. We compute the portion of non-stopping words in the top three sentences that are also in the rest of an article. A higher portion implies that the summary is representative and has a higher chance of being inferred by the model using the rest of the article. To verify, we compute the overlapping ratio of non-stopping words between human-edited summary and the article in CNN/DailyMail dataset, which has a median value of 0.87. Therefore, in pretraining, we keep articles with an overlapping word ratio higher than 0.65.

Answers:
many news articles begin with reporter names, media agencies, dates or other contents irrelevant to the content, to ensure that the summary is concise and the article contains enough salient information, we only keep articles with 10-150 words in the top three sentences and 150-1200 words in the rest, and that contain at least 6 sentences in total, we try to remove articles whose top three sentences may not form a relevant summary
Question: What unlabeled corpus did they use?

Evidence:
raw_evidence: Pretraining. We collect three years of online news articles from June 2016 to June 2019. We filter out articles overlapping with the evaluation data on media domain and time range. We then conduct several data cleaning strategies.

Answers:
three years of online news articles from June 2016 to June 2019
Question: What is the TREC-CAR dataset?

Evidence:
raw_evidence: Introduced by BIBREF16 , in this dataset, the input query is the concatenation of a Wikipedia article title with the title of one of its section. The relevant passages are the paragraphs within that section. The corpus consists of all of the English Wikipedia paragraphs, except the abstracts. The released dataset has five predefined folds, and we use the first four as a training set (approximately 3M queries), and the remaining as a validation set (approximately 700k queries). The test set is the same one used to evaluate the submissions to TREC-CAR 2017 (approx. 1,800 queries).

Answers:
in this dataset, the input query is the concatenation of a Wikipedia article title with the title of one of its section. The relevant passages are the paragraphs within that section
Question: How does their model utilize contextual information for each work in the given sentence in a multi-task setting? setting?

Evidence:
raw_evidence: Prior work have shown that contextual information could be useful for SF. They utilize contextual information either in word level representation (i.e., via contextualize embedding e.g., BERT BIBREF0) or in the model computation graph (e.g., concatenating the context feature to the word feature BIBREF1). However, such methods fail to capture the explicit dependence between the context of the word and its label. Moreover, such limited use of contextual information (i.e., concatenation of the feature vector and context vector) in the model cannot model the interaction between the word representation and its context. In order to alleviate these issues, in this work, we propose a novel model to explicitly increase the predictability of the word label using its context and increasing the interactivity between word representations and its context. More specifically, in our model we use the context of the word to predict its label and by doing so our model learns label-aware context for each word in the sentence. In order to improve the interactivity between the word representation and its context, we increase the mutual information between the word representations and its context. In addition to these contributions, we also propose an auxiliary task to predict which labels are expressed in a given sentence. Our model is trained in a mutli-tasking framework. Our experiments on a SF dataset for identifying semantic concepts from natural language request to edit an image show the superiority of our model compared to previous baselines. Our model achieves the state-of-the-art results on the benchmark dataset by improving the F1 score by almost 2%, which corresponds to a 12.3% error rate reduction.

Answers:
we use the context of the word to predict its label and by doing so our model learns label-aware context for each word in the sentence
Question: What metris are used for evaluation?

Evidence:
raw_evidence: In our experiments, we use Onsei Intent Slot dataset. Table TABREF21 shows the statics of this dataset. We use the following hyper parameters in our model: We set the word embedding and POS embedding to 768 and 30 respectively; The pre-trained BERT BIBREF17 embedding are used to initialize word embeddings; The hidden dimension of the Bi-LSTM, GCN and feed forward networks are 200; the hyper parameters $\alpha $, $\beta $ and $\gamma $ are all set to 0.1; We use Adam optimizer with learning rate 0.003 to train the model. We use micro-averaged F1 score on all labels as the evaluation metric.

Answers:
micro-averaged F1 score
Question: How better is proposed model compared to baselines?

Evidence:
raw_evidence: We compare our method with the models trained using Adobe internal NLU tool, Pytext BIBREF18 and Rasa BIBREF19 NLU tools. Table TABREF22 shows the results on Test set. Our model improves the F1 score by almost 2%, which corresponds to a 12.3% error rate reduction. This improvements proves the effectiveness of using contextual information for the task of slot filling.

Answers:
 improves the F1 score by almost 2%, which corresponds to a 12.3% error rate reduction
Question: What are the baselines?

Evidence:
raw_evidence: We compare our method with the models trained using Adobe internal NLU tool, Pytext BIBREF18 and Rasa BIBREF19 NLU tools. Table TABREF22 shows the results on Test set. Our model improves the F1 score by almost 2%, which corresponds to a 12.3% error rate reduction. This improvements proves the effectiveness of using contextual information for the task of slot filling.

Answers:
Adobe internal NLU tool, Pytext, Rasa
Question: Which machine learning models do they use to correct run-on sentences?

Evidence:
raw_evidence: In this paper, we analyze the task of automatically correcting run-on sentences. We develop two methods: a conditional random field model (roCRF) and a Seq2Seq attention model (roS2S) and show that they outperform models from the sister tasks of punctuation restoration and whole-sentence grammatical error correction. We also experiment with artificially generating training examples in clean, otherwise grammatical text, and show that models trained on this data do nearly as well predicting artificial and naturally occurring run-on sentences.

Answers:
conditional random field model, Seq2Seq attention model
Question: What are least important components identified in the the training of VQA models?

Evidence:
raw_evidence: In this study we observed the actual impact of several components within VQA models. We have shown that transformer-based encoders together with GRU models provide the best performance for question representation. Notably, we demonstrated that using pre-trained text representations provide consistent performance improvements across several hyper-parameter configurations. We have also shown that using an object detector fine-tuned with external data provides large improvements in accuracy. Our experiments have demonstrated that even simple fusion strategies can achieve performance on par with the state-of-the-art. Moreover, we have shown that attention mechanisms are paramount for learning top performing networks, once they allow producing question-aware image representations that are capable of encoding spatial relations. It became clear that Top-Down is the preferred attention method, given its results with ReLU activation. It is is now clear that some configurations used in some architectures (e.g., additional RNN layers) are actually irrelevant and can be removed altogether without harming accuracy. For future work, we expect to expand this study in two main ways: (i) cover additional datasets, such as Visual Genome BIBREF37; and (ii) study in an exhaustive fashion how distinct components interact with each other, instead of observing their impact alone on the classification performance.

Answers:
 some configurations used in some architectures (e.g., additional RNN layers) are actually irrelevant
Question: What type of experiments are performed?

Evidence:
raw_evidence: More specifically, we observe the impact of: (i) pre-trained word embeddings BIBREF11, BIBREF12, recurrent BIBREF13 and transformer-based sentence encoders BIBREF14 as question representation strategies; (ii) distinct convolutional neural networks used for visual feature extraction BIBREF15, BIBREF16, BIBREF17; and (iii) standard fusion strategies, as well as the importance of two main attention mechanisms BIBREF18, BIBREF19. We notice that even using a relatively simple baseline architecture, our best models are competitive to the (maybe overly-complex) state-of-the-art models BIBREF20, BIBREF21. Given the experimental nature of this work, we have trained over 130 neural network models, accounting for more than 600 GPU processing hours. We expect our findings to be useful as guidelines for training novel VQA models, and that they serve as a basis for the development of future architectures that seek to maximize predictive performance.

Answers:
pre-trained word embeddings BIBREF11, BIBREF12, recurrent BIBREF13, transformer-based sentence encoders BIBREF14, distinct convolutional neural networks, standard fusion strategies,  two main attention mechanisms BIBREF18, BIBREF19
Question: What components are identified as core components for training VQA models?

Evidence:
raw_evidence: In this study we observed the actual impact of several components within VQA models. We have shown that transformer-based encoders together with GRU models provide the best performance for question representation. Notably, we demonstrated that using pre-trained text representations provide consistent performance improvements across several hyper-parameter configurations. We have also shown that using an object detector fine-tuned with external data provides large improvements in accuracy. Our experiments have demonstrated that even simple fusion strategies can achieve performance on par with the state-of-the-art. Moreover, we have shown that attention mechanisms are paramount for learning top performing networks, once they allow producing question-aware image representations that are capable of encoding spatial relations. It became clear that Top-Down is the preferred attention method, given its results with ReLU activation. It is is now clear that some configurations used in some architectures (e.g., additional RNN layers) are actually irrelevant and can be removed altogether without harming accuracy. For future work, we expect to expand this study in two main ways: (i) cover additional datasets, such as Visual Genome BIBREF37; and (ii) study in an exhaustive fashion how distinct components interact with each other, instead of observing their impact alone on the classification performance.

Answers:
pre-trained text representations, transformer-based encoders together with GRU models, attention mechanisms are paramount for learning top performing networks, Top-Down is the preferred attention method
Question: what approaches are compared?

Evidence:
raw_evidence: In our experiments, we compared about 60 elementary metrics, which can be organised as follows:
This creates a challenge for the use of reference-based MT metrics for TS evaluation. However, TS has the advantage of being a monolingual translation-like task, the source being in the same language as the output. This allows for new, non-conventional ways to use MT evaluation metrics, namely by using them to compare the output of a TS system with the source sentence, thus avoiding the need for reference data. However, such an evaluation method can only capture at most two of the three above-mentioned dimensions, namely meaning preservation and, to a lesser extent, grammaticality.
BLEU, ROUGE, METEOR, TERp
Variants of BLEU: BLEU_1gram, BLEU_2gram, BLEU_3gram, BLEU_4gram and seven smoothing methods from NLTK BIBREF32 .
Intermediate components of TERp inspired by BIBREF18 : e.g. number of insertions, deletions, shifts...
Readability metrics and other sentence-level features: FKGL and FRE, numbers of words, characters, syllables...
Metrics based on the baseline QuEst features (17 features) BIBREF28 , such as statistics on the number of words, word lengths, language model probability and INLINEFORM0 -gram frequency.
Metrics based on other features: frequency table position, concreteness as extracted from BIBREF33 's BIBREF33 list, language model probability of words using a convolutional sequence to sequence model from BIBREF34 , comparison methods using pre-trained fastText word embeddings BIBREF35 or Skip-thought sentence embeddings BIBREF36 .

Answers:
MT metrics, Readability metrics and other sentence-level features, Metrics based on the baseline QuEst features, Metrics based on other features
Question: What model do they use a baseline to estimate satisfaction?

Evidence:
raw_evidence: To evaluate the proposed BiLSTM model with attention (BiLSTM+att), it is compared with three of its own variants: a BiLSTM without attention (BiLSTM) as well as a single forward-LSTM layer with attention (LSTM+att) and without attention (LSTM). Additional baselines are defined by BIBREF32 who already proposed an LSTM-based architecture that only uses non-temporal features, and the SVM-based estimation model as originally used for reward estimation by BIBREF24.

Answers:
a BiLSTM without attention (BiLSTM) as well as a single forward-LSTM layer with attention (LSTM+att) and without attention (LSTM), baselines are defined by BIBREF32 who already proposed an LSTM-based architecture that only uses non-temporal features, and the SVM-based estimation model as originally used for reward estimation by BIBREF24