qinchuanhui/UDA-QA huggingface dataset:
Question: What aspects of discussion are relevant to instructor intervention, according to the attention mechanism?

Evidence:
raw_evidence: In order to understand if context inference is useful to intervention prediction, we ablate the attention components and experiment with the vanilla hierarchical LSTM model. Row 3 of Table TABREF17 shows the macro averaged result from this experiment. The UPA and PPA attention models better the vanilla hLSTM by 5% and 2% on average in INLINEFORM0 respectively. Recall that the vanilla hLSTM already has access to a context consisting of all posts (from INLINEFORM1 through INLINEFORM2 ). In contrast, the UPA and PPA models selectively infers a context for INLINEFORM3 and INLINEFORM4 posts, respectively, and use it to predict intervention. The improved performance of our attention models that actively select their optimal context, over a model with the complete thread as context, hLSTM, shows that the context inference improves intervention prediction over using the default full context.

Answers:
context inference
Question: What was the previous state of the art for this task?

Evidence:
raw_evidence: Baselines. We compare our models against a neural baseline models, hierarchical LSTM (hLSTM), with the attention ablated but with access to the complete context, and a strong, open-sourced feature-rich baseline BIBREF7 . We choose BIBREF7 over other prior works such as BIBREF0 since we do not have access to the dataset or the system used in their papers for replication. BIBREF7 is a logistic regression classifier with features inclusive of bag-of-words representation of the unigrams and thread length, normalised counts of agreements to previous posts, counts of non-lexical reference items such as URLs, and the Coursera forum type in which a thread appeared. We also report aggregated results from a hLSTM model with access only to the last post as context for comparison. Table TABREF17 compares the performance of these baselines against our proposed methods.
raw_evidence: Baselines. We compare our models against a neural baseline models, hierarchical LSTM (hLSTM), with the attention ablated but with access to the complete context, and a strong, open-sourced feature-rich baseline BIBREF7 . We choose BIBREF7 over other prior works such as BIBREF0 since we do not have access to the dataset or the system used in their papers for replication. BIBREF7 is a logistic regression classifier with features inclusive of bag-of-words representation of the unigrams and thread length, normalised counts of agreements to previous posts, counts of non-lexical reference items such as URLs, and the Coursera forum type in which a thread appeared. We also report aggregated results from a hLSTM model with access only to the last post as context for comparison. Table TABREF17 compares the performance of these baselines against our proposed methods.

Answers:
hLSTM
hLSTM
Question: What type of latent context is used to predict instructor intervention?

Evidence:
raw_evidence: In this paper, we improve the state-of-the-art for instructor intervention in MOOC forums. We propose the first neural models for this prediction problem. We show that modelling the thread structure and the sequence of posts explicitly improves performance. Instructors in different MOOCs from different subject areas intervene differently. For example, on a Science, Technology, Engineering and Mathematics (STEM) MOOC, instructors may often intervene early as possible to resolve misunderstanding of the subject material and prevent confusion. However, in a Humanities MOOC, instructors allow for the students to explore open-ended discussions and debate among themselves. Such instructors may prefer to intervene later in the discussion to encourage further discussion or resolve conflicts among students. We therefore propose attention models to infer the latent context, i.e., the series of posts that trigger an intervention. Earlier studies on MOOC forum intervention either model the entire context or require the context size to be specified explicitly.

Answers:
the series of posts that trigger an intervention
Question: Do they report results only on English dataset?

Evidence:
raw_evidence: We build and test our MMT models on the Multi30K dataset BIBREF21 . Each image in Multi30K contains one English (EN) description taken from Flickr30K BIBREF22 and human translations into German (DE), French (FR) and Czech BIBREF23 , BIBREF24 , BIBREF25 . The dataset contains 29,000 instances for training, 1,014 for development, and 1,000 for test. We only experiment with German and French, which are languages for which we have in-house expertise for the type of analysis we present. In addition to the official Multi30K test set (test 2016), we also use the test set from the latest WMT evaluation competition, test 2018 BIBREF25 .
raw_evidence: We build and test our MMT models on the Multi30K dataset BIBREF21 . Each image in Multi30K contains one English (EN) description taken from Flickr30K BIBREF22 and human translations into German (DE), French (FR) and Czech BIBREF23 , BIBREF24 , BIBREF25 . The dataset contains 29,000 instances for training, 1,014 for development, and 1,000 for test. We only experiment with German and French, which are languages for which we have in-house expertise for the type of analysis we present. In addition to the official Multi30K test set (test 2016), we also use the test set from the latest WMT evaluation competition, test 2018 BIBREF25 .

Answers:
No
No
Question: What dataset does this approach achieve state of the art results on?

Evidence:
raw_evidence: Table TABREF14 shows the results of our main experiments on the 2016 and 2018 test sets for French and German. We use Meteor BIBREF31 as the main metric, as in the WMT tasks BIBREF25 . We compare our transformer baseline to transformer models enriched with image information, as well as to the deliberation models, with or without image information.
We first note that our multimodal models achieve the state of the art performance for transformer networks (constrained models) on the English-German dataset, as compared to BIBREF30 . Second, our deliberation models lead to significant improvements over this baseline across test sets (average INLINEFORM0 , INLINEFORM1 ).

Answers:
the English-German dataset
Question: How much training data from the non-English language is used by the system?

Evidence:
raw_evidence: Do multilingual models always need to be trained from scratch? Can we transfer linguistic knowledge learned by English pre-trained models to other languages? In this work, we develop a technique to rapidly transfer an existing pre-trained model from English to other languages in an energy efficient way BIBREF8. As the first step, we focus on building a bilingual language model (LM) of English and a target language. Starting from a pre-trained English LM, we learn the target language specific parameters (i.e., word embeddings), while keeping the encoder layers of the pre-trained English LM fixed. We then fine-tune both English and target model to obtain the bilingual LM. We apply our approach to autoencoding language models with masked language model objective and show the advantage of the proposed approach in zero-shot transfer. Our main contributions in this work are:

Answers:
No data. Pretrained model is used.
Question: Is the system tested on low-resource languages?

Evidence:
raw_evidence: We evaluate our approach for six target languages: French (fr), Russian (ru), Arabic (ar), Chinese (zh), Hindi (hi), and Vietnamese (vi). These languages belong to four different language families. French, Russian, and Hindi are Indo-European languages, similar to English. Arabic, Chinese, and Vietnamese belong to Afro-Asiatic, Sino-Tibetan, and Austro-Asiatic family respectively. The choice of the six languages also reflects different training conditions depending on the amount of monolingual data. French and Russian, and Arabic can be regarded as high resource languages whereas Hindi has far less data and can be considered as low resource.
raw_evidence: We evaluate our approach for six target languages: French (fr), Russian (ru), Arabic (ar), Chinese (zh), Hindi (hi), and Vietnamese (vi). These languages belong to four different language families. French, Russian, and Hindi are Indo-European languages, similar to English. Arabic, Chinese, and Vietnamese belong to Afro-Asiatic, Sino-Tibetan, and Austro-Asiatic family respectively. The choice of the six languages also reflects different training conditions depending on the amount of monolingual data. French and Russian, and Arabic can be regarded as high resource languages whereas Hindi has far less data and can be considered as low resource.

Answers:
Yes
Yes
Question: What languages are the model transferred to?

Evidence:
raw_evidence: We evaluate our approach for six target languages: French (fr), Russian (ru), Arabic (ar), Chinese (zh), Hindi (hi), and Vietnamese (vi). These languages belong to four different language families. French, Russian, and Hindi are Indo-European languages, similar to English. Arabic, Chinese, and Vietnamese belong to Afro-Asiatic, Sino-Tibetan, and Austro-Asiatic family respectively. The choice of the six languages also reflects different training conditions depending on the amount of monolingual data. French and Russian, and Arabic can be regarded as high resource languages whereas Hindi has far less data and can be considered as low resource.
raw_evidence: We evaluate our approach for six target languages: French (fr), Russian (ru), Arabic (ar), Chinese (zh), Hindi (hi), and Vietnamese (vi). These languages belong to four different language families. French, Russian, and Hindi are Indo-European languages, similar to English. Arabic, Chinese, and Vietnamese belong to Afro-Asiatic, Sino-Tibetan, and Austro-Asiatic family respectively. The choice of the six languages also reflects different training conditions depending on the amount of monolingual data. French and Russian, and Arabic can be regarded as high resource languages whereas Hindi has far less data and can be considered as low resource.

Answers:
French (fr), Russian (ru), Arabic (ar), Chinese (zh), Hindi (hi), and Vietnamese (vi)
French (fr), Russian (ru), Arabic (ar), Chinese (zh), Hindi (hi), and Vietnamese (vi)
Question: How is the model transferred to other languages?

Evidence:
raw_evidence: Do multilingual models always need to be trained from scratch? Can we transfer linguistic knowledge learned by English pre-trained models to other languages? In this work, we develop a technique to rapidly transfer an existing pre-trained model from English to other languages in an energy efficient way BIBREF8. As the first step, we focus on building a bilingual language model (LM) of English and a target language. Starting from a pre-trained English LM, we learn the target language specific parameters (i.e., word embeddings), while keeping the encoder layers of the pre-trained English LM fixed. We then fine-tune both English and target model to obtain the bilingual LM. We apply our approach to autoencoding language models with masked language model objective and show the advantage of the proposed approach in zero-shot transfer. Our main contributions in this work are:

Answers:
Build a bilingual language model,   learn the target language specific parameters starting from a pretrained English LM , fine-tune both English and target model to obtain the bilingual LM.
Question: What metrics are used for evaluation?

Evidence:
raw_evidence: Since pre-trained models operate on subword level, we need to estimate subword translation probabilities. Therefore, we subsample 2M sentence pairs from each parallel corpus and tokenize the data into subwords before running fast-align BIBREF13.
We build on top of RAMEN a graph-based dependency parser BIBREF27. For the purpose of evaluating the contextual representations learned by our model, we do not use part-of-speech tags. Contextualized representations are directly fed into Deep-Biaffine layers to predict arc and label scores. Table TABREF34 presents the Labeled Attachment Scores (LAS) for zero-shot dependency parsing.
raw_evidence: Table TABREF32 shows the XNLI test accuracy. For reference, we also include the scores from the previous work, notably the state-of-the-art system XLM BIBREF6. Before discussing the results, we spell out that the fairest comparison in this experiment is the comparison between mBERT and RAMEN$_{\textsc {base}}$+BERT trained with monolingual only.
We build on top of RAMEN a graph-based dependency parser BIBREF27. For the purpose of evaluating the contextual representations learned by our model, we do not use part-of-speech tags. Contextualized representations are directly fed into Deep-Biaffine layers to predict arc and label scores. Table TABREF34 presents the Labeled Attachment Scores (LAS) for zero-shot dependency parsing.

Answers:
translation probabilities, Labeled Attachment Scores (LAS)
accuracy, Labeled Attachment Scores (LAS)
Question: What datasets are used for evaluation?

Evidence:
raw_evidence: For experiments that use parallel data to initialize foreign specific parameters, we use the same datasets in the work of BIBREF6. Specifically, we use United Nations Parallel Corpus BIBREF18 for en-ru, en-ar, en-zh, and en-fr. We collect en-hi parallel data from IIT Bombay corpus BIBREF19 and en-vi data from OpenSubtitles 2018. For experiments that use only monolingual data to initialize foreign parameters, instead of training word-vectors from the scratch, we use the pre-trained word vectors from fastText BIBREF14 to estimate word translation probabilities (Eq. DISPLAY_FORM13). We align these vectors into a common space using orthogonal Procrustes BIBREF20, BIBREF15, BIBREF16. We only use identical words between the two languages as the supervised signal. We use WikiExtractor to extract extract raw sentences from Wikipedias as monolingual data for fine-tuning target embeddings and bilingual LMs (§SECREF15). We do not lowercase or remove accents in our data preprocessing pipeline.

Answers:
United Nations Parallel Corpus, IIT Bombay corpus, OpenSubtitles 2018
Question: what are the existing approaches?

Evidence:
raw_evidence: We will refer to our model as EGEL (Embedding GEographic Locations), and will consider the following variants. EGEL-Tags only uses the information from the Flickr tags (i.e. component INLINEFORM0 ), without using any negative examples and without feature selection. EGEL-Tags+NS is similar to EGEL-Tags but with the addition of negative examples. EGEL-KL(Tags+NS) additionally considers term selection. EGEL-All is our full method, i.e. it additionally uses the structured information. We also consider the following baselines. BOW-Tags represents locations using a bag-of-words representation, using the same tag weighting as the embedding model. BOW-KL(Tags) uses the same representation but after term selection, using the same KL-based method as the embedding model. BOW-All combines the bag-of-words representation with the structured information, encoded as proposed in BIBREF7 . GloVe uses the objective from the original GloVe model for learning location vectors, i.e. this variant differs from EGEL-Tags in that instead of INLINEFORM1 we use the number of co-occurrences of tag INLINEFORM2 near location INLINEFORM3 , measured as INLINEFORM4 .

Answers:
BOW-Tags, BOW-KL(Tags), BOW-All, GloVe
Question: what dataset is used in this paper?

Evidence:
raw_evidence: There is a wide variety of structured data that can be used to describe locations. In this work, we have restricted ourselves to the same datasets as BIBREF7 . These include nine (real-valued) numerical features, which are latitude, longitude, elevation, population, and five climate related features (avg. temperature, avg. precipitation, avg. solar radiation, avg. wind speed, and avg. water vapor pressure). In addition, 180 categorical features were used, which are CORINE land cover classes at level 1 (5 classes), level 2 (15 classes) and level 3 (44 classes) and 116 soil types (SoilGrids). Note that each location should belong to exactly 4 categories: one CORINE class at each of the three levels and a soil type.
raw_evidence: Predicting the distribution of 100 species across Europe, using the European network of nature protected sites Natura 2000 dataset as ground truth. For each of these species, a binary classification problem is considered. The set of locations INLINEFORM0 is defined as the 26,425 distinct sites occurring in the dataset.
There is a wide variety of structured data that can be used to describe locations. In this work, we have restricted ourselves to the same datasets as BIBREF7 . These include nine (real-valued) numerical features, which are latitude, longitude, elevation, population, and five climate related features (avg. temperature, avg. precipitation, avg. solar radiation, avg. wind speed, and avg. water vapor pressure). In addition, 180 categorical features were used, which are CORINE land cover classes at level 1 (5 classes), level 2 (15 classes) and level 3 (44 classes) and 116 soil types (SoilGrids). Note that each location should belong to exactly 4 categories: one CORINE class at each of the three levels and a soil type.
Predicting people's subjective opinions of landscape beauty in Britain, using the crowdsourced dataset from the ScenicOrNot website as ground truth. The set INLINEFORM0 is chosen as the set of locations of 191 605 rated locations from the ScenicOrNot dataset for which at least one georeferenced Flickr photo exists within a 1 km radius.

Answers:
 the same datasets as BIBREF7
same datasets as BIBREF7
Question: How is keyphrase diversity measured?

Evidence:
raw_evidence: To verify our assumption that target encoding and orthogonal regularization help to boost the diversity of generated sequences, we use two metrics, one quantitative and one qualitative, to measure diversity of generation.
First, we simply calculate the average unique predictions produced by both INLINEFORM0 and INLINEFORM1 in experiments shown in Section SECREF36 . The resulting numbers are 20.38 and 89.70 for INLINEFORM2 and INLINEFORM3 respectively. Second, from the model running on the KP20k validation set, we randomly sample 2000 decoder hidden states at INLINEFORM4 steps following a delimiter ( INLINEFORM5 ) and apply an unsupervised clustering method (t-SNE BIBREF35 ) on them. From the Figure FIGREF46 we can see that hidden states sampled from INLINEFORM6 are easier to cluster while hidden states sampled from INLINEFORM7 yield one mass of vectors with no obvious distinct clusters. Results on both metrics suggest target encoding and orthogonal regularization indeed help diversifying generation of our model.
To illustrate the difference of predictions between our proposed models, we show an example chosen from the KP20k validation set in Appendix SECREF10 . In this example there are 29 ground truth phrases. Neither of the models is able to generate all of the keyphrases, but it is obvious that the predictions from INLINEFORM0 all start with “test”, while predictions from INLINEFORM1 are diverse. This to some extent verifies our assumption that without the target encoder and orthogonal regularization, decoder states following delimiters are less diverse.

Answers:
average unique predictions, illustrate the difference of predictions between our proposed models, we show an example chosen from the KP20k validation set
Question: How was the StackExchange dataset collected?

Evidence:
raw_evidence: Inspired by the StackLite tag recommendation task on Kaggle, we build a new benchmark based on the public StackExchange data. We use questions with titles as source, and user-assigned tags as target keyphrases.
Since oftentimes the questions on StackExchange contain less information than in scientific publications, there are fewer keyphrases per data point in StackEx. Furthermore, StackExchange uses a tag recommendation system that suggests topic-relevant tags to users while submitting questions; therefore, we are more likely to see general terminology such as Linux and Java. This characteristic challenges models with respect to their ability to distill major topics of a question rather than selecting specific snippets from the text.

Answers:
they obtained computer science related topics by looking at titles and user-assigned tags
Question: What two metrics are proposed?

Evidence:
raw_evidence: First, we simply calculate the average unique predictions produced by both INLINEFORM0 and INLINEFORM1 in experiments shown in Section SECREF36 . The resulting numbers are 20.38 and 89.70 for INLINEFORM2 and INLINEFORM3 respectively. Second, from the model running on the KP20k validation set, we randomly sample 2000 decoder hidden states at INLINEFORM4 steps following a delimiter ( INLINEFORM5 ) and apply an unsupervised clustering method (t-SNE BIBREF35 ) on them. From the Figure FIGREF46 we can see that hidden states sampled from INLINEFORM6 are easier to cluster while hidden states sampled from INLINEFORM7 yield one mass of vectors with no obvious distinct clusters. Results on both metrics suggest target encoding and orthogonal regularization indeed help diversifying generation of our model.

Answers:
average unique predictions, randomly sample 2000 decoder hidden states at INLINEFORM4 steps following a delimiter ( INLINEFORM5 ) and apply an unsupervised clustering method (t-SNE BIBREF35 )
Question: Can the findings of this paper be generalized to a general-purpose task?

Evidence:
raw_evidence: The variability in generalization on uncharacteristic inputs (and thus, the extent of linguistic generalization) given different random seeds is alarming, particularly given the fact that the standard test set performance remains mostly the same regardless. The task presented here was easy and simple to analyze, however, future work may be done on natural language tasks. If these properties hold it might indicate that a new evaluation paradigm for NLP should be pushed; one that emphasizes performance on uncharacteristic (but structurally sound) inputs in addition to the data typically seen in training.
raw_evidence: Real world NLP tasks are complex, and as such, it can be difficult to precisely define what a model should and should not learn during training. As done in previous work BIBREF8 , BIBREF9 , we ease analysis by looking at a simple formal task. The task is set up to mimic (albeit, in an oversimplified manner) the input-output symbol alignments and local syntactic properties that models must learn in many natural language tasks, such as translation, tagging and summarization.

Answers:
Yes
Yes
Question: Why does the proposed task a good proxy for the general-purpose sequence to sequence tasks?

Evidence:
raw_evidence: Real world NLP tasks are complex, and as such, it can be difficult to precisely define what a model should and should not learn during training. As done in previous work BIBREF8 , BIBREF9 , we ease analysis by looking at a simple formal task. The task is set up to mimic (albeit, in an oversimplified manner) the input-output symbol alignments and local syntactic properties that models must learn in many natural language tasks, such as translation, tagging and summarization.

Answers:
The task is set up to mimic (albeit, in an oversimplified manner) the input-output symbol alignments and local syntactic properties that models must learn in many natural language tasks, such as translation, tagging and summarization.