qinchuanhui/UDA-QA huggingface dataset:
Question: What are the languages they consider in this paper?

Evidence:
raw_evidence: To test this hypothesis we first trained UG-WGAN in English, Chinese and German following the procedure described in Section "UG-WGAN" . The embedding size of the table was 300 and the internal LSTM hidden size was 512. A dropout rate of $0.1$ was used and trained with the ADAM optimization method BIBREF23 . Since we are interested in the zero-shot capabilities of our representation, we trained our sentiment analysis model only on the english IMDB Large Movie Review dataset and tested it on the chinese ChnSentiCorp dataset and german SB-10K BIBREF24 , BIBREF25 . We binarize the label's for all the datasets.
For this experiment we trained UG-WGAN on the English and Russian language following the procedure described in Section "UG-WGAN" . We kept the hyper-parameters equivalent to the Sentiment Analysis experiment. All of the NLI model tested were run over the fixed UG embeddings. We trained two different models from literature, Densely-Connected Recurrent and Co-Attentive Network by BIBREF30 and Multiway Attention Network by BIBREF31 . Please refer to this papers for further implementation details.
One way to measure universality is by studying perplexity of our multi-lingual language model as we increase the number of languages. To do so we trained 6 UG-WGAN models on the following languages: English, Russian, Arabic, Chinese, German, Spanish, French. We maintain the same procedure as described above. The hidden size of the language model was increased to 1024 with 16K BPE tokens being used. The first model was trained on English Russian, second was trained on English Russian Arabic and so on. For arabic we still trained from left to right even though naturally the language is read from right to left. We report the results in Figure 5 . As the number of languages increases the gap between a UG-WGAN without any distribution matching and one with diminishes. This implies that the efficiency and representative power of UG-WGAN grows as we increase the number of languages it has to model.

Answers:
The languages considered were English, Chinese, German, Russian, Arabic, Spanish, French
Question: Did they experiment with tasks other than word problems in math?

Evidence:
raw_evidence: To test this hypothesis we first trained UG-WGAN in English, Chinese and German following the procedure described in Section "UG-WGAN" . The embedding size of the table was 300 and the internal LSTM hidden size was 512. A dropout rate of $0.1$ was used and trained with the ADAM optimization method BIBREF23 . Since we are interested in the zero-shot capabilities of our representation, we trained our sentiment analysis model only on the english IMDB Large Movie Review dataset and tested it on the chinese ChnSentiCorp dataset and german SB-10K BIBREF24 , BIBREF25 . We binarize the label's for all the datasets.
A natural language inference task consists of two sentences; a premise and a hypothesis which are either contradictions, entailments or neutral. Learning a NLI task takes a certain nuanced understanding of language. Therefore it is of interest whether or not UG-WGAN captures the necessary linguistic features. For this task we use the Stanford NLI (sNLI) dataset as our training data in english BIBREF29 . To test the zero-shot learning capabilities we created a russian sNLI test set by random sampling 400 sNLI test samples and having a native russian speaker translate both premise and hypothesis to russian. The label was kept the same.

Answers:
They experimented with sentiment analysis and natural language inference task
Question: Do the authors offer any hypothesis as to why the transformations sometimes disimproved performance?

Evidence:

Answers:
No
Question: What preprocessing techniques are used in the experiments?

Evidence:
raw_evidence: To explore how helpful these transformations are, we incorporated 20 simple transformations and 15 additional sequences of transformations in our experiment to see their effect on different type of metrics on four different ML models (See Figure FIGREF3 ).
FLOAT SELECTED: Fig. 1: List of transformations.

Answers:
See Figure FIGREF3
Question: What state of the art models are used in the experiments?

Evidence:
raw_evidence: We used four classification algorithms: 1) Logistic regression, which is conventionally used in sentiment classification. Other three algorithms which are relatively new and has shown great results on sentiment classification types of problems are: 2) Naïve Bayes with SVM (NBSVM), 3) Extreme Gradient Boosting (XGBoost) and 4) FastText algorithm with Bidirectional LSTM (FastText-BiLSTM).

Answers:
2) Naïve Bayes with SVM (NBSVM), 3) Extreme Gradient Boosting (XGBoost), 4) FastText algorithm with Bidirectional LSTM (FastText-BiLSTM)
Question: What dataset did they use?

Evidence:
raw_evidence: We use 16 different datasets from several popular review corpora used in BIBREF20 . These datasets consist of 14 product review datasets and two movie review datasets.
We use CoNLL 2000 BIBREF22 sequence labeling dataset for both POS Tagging and Chunking tasks. There are 8774 sentences in training data, 500 sentences in development data and 1512 sentences in test data. The average sentence length is 24 and has a total vocabulary size as 17k.

Answers:
16 different datasets from several popular review corpora used in BIBREF20, CoNLL 2000 BIBREF22
Question: What tasks did they experiment with?

Evidence:
raw_evidence: Exp I: Sentiment Classification
We first conduct a multi-task experiment on sentiment classification.
We use 16 different datasets from several popular review corpora used in BIBREF20 . These datasets consist of 14 product review datasets and two movie review datasets.
All the datasets in each task are partitioned randomly into training set, development set and testing set with the proportion of 70%, 10% and 20% respectively. The detailed statistics about all the datasets are listed in Table TABREF27 .
Exp II: Transferability of Shared Sentence Representation 
With attention mechanism, the shared sentence encoder in our proposed models can generate more generic task-invariant representations, which can be considered as off-the-shelf knowledge and then be used for unseen new tasks.
Exp III: Introducing Sequence Labeling as Auxiliary Task
A good sentence representation should include its linguistic information. Therefore, we incorporate sequence labeling task (such as POS Tagging and Chunking) as an auxiliary task into the multi-task learning framework, which is trained jointly with the primary tasks (the above 16 tasks of sentiment classification). The auxiliary task shares the sentence encoding layer with the primary tasks and connected to a private fully connected layer followed by a softmax non-linear layer to process every hidden state INLINEFORM0 and predicts the labels.

Answers:
Sentiment Classification, Transferability of Shared Sentence Representation, Introducing Sequence Labeling as Auxiliary Task
Question: What multilingual parallel data is used for training proposed model?

Evidence:
raw_evidence: We adopt the mixture of two multilingual translation corpus as our training data: MultiUN BIBREF20 and OpenSubtitles BIBREF21. MultiUN consists of 463,406 official documents in six languages, containing around 300M words for each language. OpenSubtitles is a corpus consisting of movie and TV subtitles, which contains 2.6B sentences over 60 languages. We select four shared languages of the two corpora: English, Spanish, Russian and Chinese. Statistics of the training corpus are shown in Table TABREF14. Sentences are tokenized by Wordpiece as in BERT. A multilingual vocabulary of 50K tokens is used. For validation and testing, we randomly sample 10000 sentences respectively from each language pair. The rest data are used for training. For monolingual pre-training, we use English Wikipedia corpus, which contains 2,500M words.

Answers:
MultiUN BIBREF20, OpenSubtitles BIBREF21
Question: What kind of Youtube video transcripts did they use?

Evidence:
raw_evidence: We focused evaluation over a small but diversified dataset composed by 10 YouTube videos in the English language in the news context. The selected videos cover different topics like technology, human rights, terrorism and politics with a length variation between 2 and 10 minutes. To encourage the diversity of content format we included newscasts, interviews, reports and round tables.

Answers:
youtube video transcripts on news covering different topics like technology, human rights, terrorism and politics
Question: Which SBD systems did they compare?

Evidence:
raw_evidence: To exemplify the INLINEFORM0 score we evaluated and compared the performance of two different SBD systems over a set of YouTube videos in a multi-reference enviroment. The first system (S1) employs a Convolutional Neural Network to determine if the middle word of a sliding window corresponds to a SU boundary or not BIBREF30 . The second approach (S2) by contrast, introduces a bidirectional Recurrent Neural Network model with attention mechanism for boundary detection BIBREF31 .

Answers:
Convolutional Neural Network , bidirectional Recurrent Neural Network model with attention mechanism
Question: What makes it a more reliable metric?

Evidence:
raw_evidence: Results form Table TABREF31 may give an idea that INLINEFORM0 is just an scaled INLINEFORM1 . While it is true that they show a linear correlation, INLINEFORM2 may produce a different system ranking than INLINEFORM3 given the integral multi-reference principle it follows. However, what we consider the most profitable about INLINEFORM4 is the twofold inclusion of all available references it performs. First, the construction of INLINEFORM5 to provide a more inclusive reference against to whom be evaluated and then, the computation of INLINEFORM6 , which scales the result depending of the agreement between references.
In this paper we presented WiSeBE, a semi-automatic multi-reference sentence boundary evaluation protocol based on the necessity of having a more reliable way for evaluating the SBD task. We showed how INLINEFORM0 is an inclusive metric which not only evaluates the performance of a system against all references, but also takes into account the agreement between them. According to your point of view, this inclusivity is very important given the difficulties that are present when working with spoken language and the possible disagreements that a task like SBD could provoke.

Answers:
It takes into account the agreement between different systems
Question: How much in experiments is performance improved for models trained with generated adversarial examples?

Evidence:
raw_evidence: Adversarial training can often improve model robustness BIBREF25, BIBREF27. We also fine-tune the target models using adversarial training. At each training step, we train the model with a batch of original examples along with adversarial examples with balanced labels. The adversarial examples account for around 10% in a batch. During training, we generate adversarial examples with the current model as the target and update the model parameters with the hybrid batch iteratively. The beam size for generation is set to 1 to reduce the computation cost, since the generation success rate is minor in adversarial training. We evaluate the adversarially trained models, as shown in Table TABREF18.
After adversarial training, the performance of all the target models raises significantly, while that on the original examples remain comparable. Note that since the focus of this paper is on model robustness which can hardly be reflected in original data, we do not expect performance improvement on original data. The results demonstrate that adversarial training with our adversarial examples can significantly improve the robustness we focus on without remarkably hurting the performance on original data. Moreover, although the adversarial example generation is constrained by a BERT language model, BiMPM and DIIN which do not use the BERT language model can also significantly benefit from the adversarial examples, further demonstrating the effectiveness of our method.

Answers:
Answer with content missing: (Table 1) The performance of all the target models raises significantly, while that on the original
examples remain comparable (e.g. the overall accuracy of BERT on modified examples raises from 24.1% to 66.0% on Quora)
Question: How much dramatically results drop for models on generated adversarial examples?

Evidence:
raw_evidence: After adversarial modifications, the performance of the original target models (those without the “-adv” suffix) drops dramatically (e.g. the overall accuracy of BERT on Quora drops from 94.6% to 24.1%), revealing that the target models are vulnerable to our adversarial examples. Particularly, even though our generation is constrained by a BERT language model, BERT is still vulnerable to our adversarial examples. These results demonstrate the effectiveness of our algorithm for generating adversarial examples and also revealing the corresponding robustness issues. Moreover, we present some generated adversarial examples in the appendix.

Answers:
BERT on Quora drops from 94.6% to 24.1%
Question: What is discriminator in this generative adversarial setup?

Evidence:
raw_evidence: Adversarial training can often improve model robustness BIBREF25, BIBREF27. We also fine-tune the target models using adversarial training. At each training step, we train the model with a batch of original examples along with adversarial examples with balanced labels. The adversarial examples account for around 10% in a batch. During training, we generate adversarial examples with the current model as the target and update the model parameters with the hybrid batch iteratively. The beam size for generation is set to 1 to reduce the computation cost, since the generation success rate is minor in adversarial training. We evaluate the adversarially trained models, as shown in Table TABREF18.

Answers:
 current model
Question: What are benhmark datasets for paraphrase identification?

Evidence:
raw_evidence: We adopt the following two datasets:
Quora BIBREF1: The Quora Question Pairs dataset contains question pairs annotated with labels indicating whether the two questions are paraphrases. We use the same dataset partition as BIBREF5, with 384,348/10,000/10,000 pairs in the training/development/test set respectively.
MRPC BIBREF34: The Microsoft Research Paraphrase Corpus consists of sentence pairs collected from online news. Each pair is annotated with a label indicating whether the two sentences are semantically equivalent. There are 4,076/1,725 pairs in the training/test set respectively.

Answers:
Quora, MRPC
Question: What representations are presented by this paper?

Evidence:
raw_evidence: Following work by doukhan2018open, we wanted to explore the corpora looking at the number of speakers of each gender category as well as their speech duration, considering both variables as good features to account for gender representation. After the download, we manually extracted information about gender representation in each corpus.

Answers:
the number of speakers of each gender category, their speech duration
Question: How is the delta-softmax calculated?

Evidence:
raw_evidence: We reason that, if a token is important for predicting the correct label, masking it will degrade the model's classification accuracy, or at least reduce its reported classification certainty. In SECREF36, it seems reasonable to assume that masking the word `To' has a greater impact on predicting the label purpose than masking the word `provide', and even less so, the following noun `information'. We therefore use reduction in softmax probability of the correct relation as our signaling strength metric for the model. We call this metric ${\Delta }_s$ (for delta-softmax), which can be written as:
where $rel$ is the true relation of the EDU pair, $t_i$ represents the token at index $i$ of $N$ tokens, and $X_{mask=i}$ represents the input sequence with the masked position $i$ (for $i \in 1 \ldots N$ ignoring separators, or $\phi $, the empty set).

Answers:
Answer with content missing: (Formula) Formula is the answer.
Question: Are some models evaluated using this metric, what are the findings?

Evidence:
raw_evidence: As a framework, we use the sentence classifier configuration of FLAIR BIBREF46 with a biLSTM encoder/classifier architecture fed by character and word level representations composed of a concatenation of fixed 300 dimensional GloVe embeddings BIBREF47, pre-trained contextualized FLAIR word embeddings, and pre-trained contextualized character embeddings from AllenNLP BIBREF48 with FLAIR's default hyperparameters. The model's architecture is shown in Figure FIGREF30.

Answers:
Yes
Question: Where does proposed metric differ from juman judgement?

Evidence:
raw_evidence: In other cases, the model points out plausible signals which were passed over by an annotator, and may be considered errors in the gold standard. For example, the model easily notices that question marks indicate the solutionhood relation, even where these were skipped by annotators in favor of marking WH words instead:
. [RGB]230, 230, 230Which [RGB]230, 230, 230previous [RGB]230, 230, 230Virginia [RGB]230, 230, 230Governor(s) [RGB]230, 230, 230do [RGB]230, 230, 230you [RGB]230, 230, 230most [RGB]230, 230, 230admire [RGB]230, 230, 230and [RGB]230, 230, 230why [RGB]12, 12, 12? $\xrightarrow[\text{pred:solutionhood}]{\text{gold:solutionhood}}$ [RGB]230, 230, 230Thomas [RGB]230, 230, 230Jefferson [RGB]183, 183, 183.
Unsurprisingly, the model sometimes make sporadic errors in signal detection for which good explanations are hard to find, especially when its predicted relation is incorrect, as in SECREF43. Here the evaluative adjective remarkable is missed in favor of neighboring words such as agreed and a subject pronoun, which are not indicative of the evaluation relation in this context but are part of several cohorts of high scoring words. However, the most interesting and interpretable errors arise when ${\Delta }_s$ scores are high compared to an entire document, and not just among words in one EDU pair, in which most or even all words may be relatively weak signals. As an example of such a false positive with high confidence, we can consider SECREF43. In this example, the model correctly assigns the highest score to the DM so marking a purpose relation. However, it also picks up on a recurring tendency in how-to guides in which the second person pronoun referring to the reader is often the benefactee of some action, which contributes to the purpose reading and helps to disambiguate so, despite not being considered a signal by annotators.

Answers:
model points out plausible signals which were passed over by an annotator, it also picks up on a recurring tendency in how-to guides in which the second person pronoun referring to the reader is often the benefactee of some action
Question: Where does proposed metric overlap with juman judgement?

Evidence:
raw_evidence: Looking at the model's performance qualitatively, it is clear that it can detect not only DMs, but also morphological cues (e.g. gerunds as markers of elaboration, as in SECREF43), semantic classes and sentiment, such as positive and negative evaluatory terms in SECREF43, as well as multiple signals within the same EDU, as in SECREF43. In fact, only about 8.3% of the tokens correctly identified by the model in Table TABREF45 below are of the DM type, whereas about 7.2% of all tokens flagged by human annotators were DMs, meaning that the model frequently matches non-DM items to discourse relation signals (see Performance on Signal Types below). It should also be noted that signals can be recognized even when the model misclassifies relations, since ${\Delta }_s$ does not rely on correct classification: it merely quantifies the contribution of a word in context toward the correct label's score. If we examine the influence of each word on the score of the correct relation, that impact should and does still correlate with human judgments based on what the system may tag as the second or third best class to choose.

Answers:
influence of each word on the score of the correct relation, that impact should and does still correlate with human judgments
Question: Which baseline performs best?

Evidence:
raw_evidence: Statistical significance is assessed for select results using bootstrapping with 1000 samples in each of 100 iterations. This test shows that conditioning on the introduction of the source document improves performance compared to conditioning on the abstract when using the SciGPT2 model. However, we see that IR methods perform better than the best neural models. We do not find enough evidence to reject the null hypothesis regarding what context from the cited document should be used.

Answers:
IR methods perform better than the best neural models
Question: Which baselines are explored?

Evidence:
raw_evidence: To fine-tune GPT2 for text generation, it is typical to concatenate the conditioning context $X = x_1 \ldots x_n$ and citing sentence $Y = y_1 \ldots y_m$ with a special separator token $\mho $. The model learns to approximate next token probabilities for each index after $\mho $:
We measure the closeness of two pairs of documents by measuring cosine distances between vector representations of their content. The abstract of each document is embedded into a single dense vector by averaging the contextualized embeddings provided by the SciBERT model of BIBREF11 and normalizing. The distance between $(S,C)$ and candidate $(N_S,N_C)$ is computed as:

Answers:
GPT2, SciBERT model of BIBREF11
Question: What is the size of the corpus?

Evidence:
raw_evidence: For source and cited documents, we use English-language computer science articles and annotation from the S2-GORC dataset BIBREF7. S2-GORC is a large citation graph dataset which includes full texts of 8.1 million scientific documents. We select a subset of 154K computer science articles as our corpus. From these, we extract 622K citing sentences that link back to other documents in our corpus. We hold 2500 examples for each of the validation and test sets. Detailed statistics can be found in Table TABREF4.

Answers:
8.1 million scientific documents, 154K computer science articles, 622K citing sentences
Question: Are any machine translation sysems tried with these embeddings, what is the performance?

Evidence:

Answers:
No
Question: Are any experiments performed to try this approach to word embeddings?

Evidence:
raw_evidence: In the literature, two main types of datasets are used for machine translation: Word-aligned data and Sentence-aligned data. The first one is basically a dictionary between the two languages, where there is a direct relation between same words in different languages. The second one has the relation between corresponding sentences in the two languages. We decided to start with the sentence aligned corpus, since it was more interesting to infer dependency from contexts among words. For our experiment we decided to use the Europarl dataset, using the data from the WMT11 .The Europarl parallel corpus is extracted from the proceedings of the European Parliament. It includes versions in 21 European languages: Romanic (French, Italian, Spanish, Portuguese, Romanian), Germanic (English, Dutch, German, Danish, Swedish), Slavik (Bulgarian, Czech, Polish, Slovak, Slovene), Finni-Ugric (Finnish, Hungarian, Estonian), Baltic (Latvian, Lithuanian), and Greek. For this experience, we used the English-French parallel corpus, which contains 2,007,723 sentences and the English-Italian corpus, that contains 1,909,115 sentences.

Answers:
Yes