qinchuanhui/UDA-QA huggingface dataset:
Question: Why does the model improve in monolingual spaces as well? 

Evidence:
raw_evidence: As a complement of this analysis we show some qualitative results which give us further insights on the transformations of the vector space after our average approximation. In particular, we analyze the reasons behind the higher quality displayed by our bilingual embeddings in monolingual settings. While VecMap and MUSE do not transform the initial monolingual spaces, our model transforms both spaces simultaneously. In this analysis we focus on the source language of our experiments (i.e., English). We found interesting patterns which are learned by our model and help understand these monolingual gains. For example, a recurring pattern is that words in English which are translated to the same word, or to semantically close words, in the target language end up closer together after our transformation. For example, in the case of English-Spanish the following pairs were among the pairs whose similarity increased the most by applying our transformation: cellphone-telephone, movie-film, book-manuscript or rhythm-cadence, which are either translated to the same word in Spanish (i.e., teléfono and película in the first two cases) or are already very close in the Spanish space. More generally, we found that word pairs which move together the most tend to be semantically very similar and belong to the same domain, e.g., car-bicycle, opera-cinema, or snow-ice.

Answers:
because word pair similarity increases if the two words translate to similar parts of the cross-lingual embedding space
Question: Do the authors test their annotation projection techniques on tasks other than AMR?

Evidence:
raw_evidence: Cross-lingual techniques can cope with the lack of labeled data on languages when this data is available in at least one language, usually English. The annotation projection method, which we follow in this work, is one way to address this problem. It was introduced for POS tagging, base noun phrase bracketing, NER tagging, and inflectional morphological analysis BIBREF29 but it has also been used for dependency parsing BIBREF30 , role labeling BIBREF31 , BIBREF32 and semantic parsing BIBREF26 . Another common thread of cross-lingual work is model transfer, where parameters are shared across languages BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36 , BIBREF37 .

Answers:
No
Question: How is annotation projection done when languages have different word order?

Evidence:
raw_evidence: AMR is not grounded in the input sentence, therefore there is no need to change the AMR annotation when projecting to another language. We think of English labels for the graph nodes as ones from an independent language, which incidentally looks similar to English. However, in order to train state-of-the-art AMR parsers, we also need to project the alignments between AMR nodes and words in the sentence (henceforth called AMR alignments). We use word alignments, similarly to other annotation projection work, to project the AMR alignments to the target languages.
Our approach depends on an underlying assumption that we make: if a source word is word-aligned to a target word and it is AMR aligned with an AMR node, then the target word is also aligned to that AMR node. More formally, let $S = s_1 \dots s_{\vert s \vert }$ be the source language sentence and $T = t_1 \dots t_{\vert t \vert }$ be the target language sentence; $A_s(\cdot )$ be the AMR alignment mapping word tokens in $S$ to the set of AMR nodes that are triggered by it; $A_t(\cdot )$ be the same function for $T$ ; $v$ be a node in the AMR graph; and finally, $W(\cdot )$ be an alignment that maps a word in $S$ to a subset of words in $T$ . Then, the AMR projection assumption is: $T = t_1 \dots t_{\vert t \vert }$0
Word alignments were generated using fast_align BIBREF10 , while AMR alignments were generated with JAMR BIBREF11 . AMREager BIBREF12 was chosen as the pre-existing English AMR parser. AMREager is an open-source AMR parser that needs only minor modifications for re-use with other languages. Our multilingual adaptation of AMREager is available at http://www.github.com/mdtux89/amr-eager-multilingual. It requires tokenization, POS tagging, NER tagging and dependency parsing, which for English, German and Chinese are provided by CoreNLP BIBREF13 . We use Freeling BIBREF14 for Spanish, as CoreNLP does not provide dependency parsing for this language. Italian is not supported in CoreNLP: we use Tint BIBREF15 , a CoreNLP-compatible NLP pipeline for Italian.

Answers:
Word alignments are generated for parallel text, and aligned words are assumed to also share AMR node alignments.
Question: What is the reasoning method that is used?

Evidence:
raw_evidence: The DBpedia lookup service, which is based on the Spotlight index BIBREF18 , is used for entity lookup (retrieval). The DBpedia SPARQL endpoint is used for query answering and reasoning. The reported results are based on the following settings: the Adam optimizer together with cross-entropy loss are used for network training; $d_r$ and $d_a$ are set to 200 and 50 respectively; $N_0$ is set to 1200; word2vec trained with the latest Wikipedia article dump is adopted for word embedding; and ( $T_s$ , $T_p$ , $T_l$ ) are set to (12, 4, 12) for S-Lite and (12, 4, 15) for R-Lite. The experiments are run on a workstation with Intel(R) Xeon(R) CPU E5-2670 @2.60GHz, with programs implemented by Tensorflow.

Answers:
SPARQL
Question: What KB is used in this work?

Evidence:
raw_evidence: In this study, we investigate KB literal canonicalization using a combination of RNN-based learning and semantic technologies. We first predict the semantic types of a literal by: (i) identifying candidate classes via lexical entity matching and KB queries; (ii) automatically generating positive and negative examples via KB sampling, with external semantics (e.g., from other KBs) injected for improved quality; (iii) training classifiers using relevant subject-predicate-literal triples embedded in an attentive bidirectional RNN (AttBiRNN); and (iv) using the trained classifiers and KB class hierarchy to predict candidate types. The novelty of our framework lies in its knowledge-based learning; this includes automatic candidate class extraction and sampling from the KB, triple embedding with different importance degrees suggesting different semantics, and using the predicted types to identify a potential canonical entity from the KB. We have evaluated our framework using a synthetic literal set (S-Lite) and a real literal set (R-Lite) from DBpedia BIBREF0 . The results are very promising, with significant improvements over several baselines, including the existing state-of-the-art.

Answers:
DBpedia
Question: How did they measure effectiveness?

Evidence:
raw_evidence: We trained the DQN agent using an $\epsilon $-greedy policy with $\epsilon $ decreasing linearly from 1 to $0.1$ over $100,000$ steps. Additionally, we tuned a window size to include previous dialog turns as input and set $\gamma $ to $0.9$. We ran the method 30 times for $150,000$ steps, and in each run, after every 10,000 steps, we sampled $3,000$ dialog episodes with no exploration to evaluate the performance. The optimal parameters were found using Hyperopt BIBREF23 (see Appendix B). Figure FIGREF9 shows the simulation results during training. The Y-axis in the figure is the success rate of the agent (measured in terms of number of dialogs that resulted in launching a skill divided by total number of dialogs), and the X-axis is the number of learning steps. Given our choice of reward function, the increase in success rate is indicative of the agent learning to improve its policy over time. Furthermore, the RL agent outperformed the rule-based agent with average success rate of $68.00\% (\pm 2\%$) in simulation.

Answers:
number of dialogs that resulted in launching a skill divided by total number of dialogs
Question: Which of the two ensembles yields the best performance?

Evidence:
raw_evidence: Figure FIGREF5 shows that E-BERT performs comparable to BERT and ERNIE on unfiltered LAMA. However, E-BERT is less affected by filtering on LAMA-UHN, suggesting that its performance is more strongly due to factual knowledge. Recall that we lack entity embeddings for 46% of Google-RE subjects, i.e., E-BERT cannot improve over BERT on almost half of the Google-RE tuples.

Answers:
Answer with content missing: (Table 2) CONCAT ensemble
Question: What are the two ways of ensembling BERT and E-BERT?

Evidence:
raw_evidence: We ensemble BERT and E-BERT by (a) mean-pooling their outputs (AVG) or (b) concatenating the entity and its name with a slash symbol (CONCAT), e.g.: Jean_Marais / Jean Mara ##is.

Answers:
mean-pooling their outputs (AVG), concatenating the entity and its name with a slash symbol (CONCAT)
Question: How is it determined that a fact is easy-to-guess?

Evidence:
raw_evidence: Filter 1: The string match filter deletes all KB triples where the correct answer (e.g., Apple) is a case-insensitive substring of the subject entity name (e.g., Apple Watch). This simple heuristic deletes up to 81% of triples from individual relations (see Appendix for statistics and examples).
Filter 2: Of course, entity names can be revealing in ways that are more subtle. As illustrated by our French actor example, a person's name can be a useful prior for guessing their native language and by extension, their nationality, place of birth, etc. Our person name filter uses cloze-style questions to elicit name associations inherent in BERT, and deletes KB triples that correlate with them. Consider our previous example (Jean_Marais, native-language, French). We whitespace-tokenize the subject name into Jean and Marais. If BERT considers either name to be a common French name, then a correct answer is insufficient evidence for factual knowledge about the entity Jean_Marais. On the other hand, if neither Jean nor Marais are considered French, but a correct answer is given nonetheless, then we consider this sufficient evidence for factual knowledge.

Answers:
 filter deletes all KB triples where the correct answer (e.g., Apple) is a case-insensitive substring of the subject entity name (e.g., Apple Watch), person name filter uses cloze-style questions to elicit name associations inherent in BERT, and deletes KB triples that correlate with them
Question: How is dependency parsing empirically verified?

Evidence:
raw_evidence: We evaluate our model on two benchmark treebanks, English Penn Treebank (PTB) and Chinese Penn Treebank (CTB5.1) following standard data splitting BIBREF30, BIBREF31. POS tags are predicted by the Stanford Tagger BIBREF32. For constituent parsing, we use the standard evalb tool to evaluate the F1 score. For dependency parsing, we apply Stanford basic dependencies (SD) representation BIBREF4 converted by the Stanford parser. Following previous work BIBREF27, BIBREF33, we report the results without punctuations for both treebanks.
Tables TABREF17, TABREF18 and TABREF19 compare our model to existing state-of-the-art, in which indicator Separate with our model shows the results of our model learning constituent or dependency parsing separately, (Sum) and (Concat) respectively represent the results with the indicated input token representation setting. On PTB, our model achieves 93.90 F1 score of constituent parsing and 95.91 UAS and 93.86 LAS of dependency parsing. On CTB, our model achieves a new state-of-the-art result on both constituent and dependency parsing. The comparison again suggests that learning jointly in our model is superior to learning separately. In addition, we also augment our model with ELMo BIBREF48 or a larger version of BERT BIBREF49 as the sole token representation to compare with other pre-training models. Since BERT is based on sub-word, we only take the last sub-word vector of the word in the last layer of BERT as our sole token representation $x_i$. Moreover, our single model of BERT achieves competitive performance with other ensemble models.
Multitask learning (MTL) is a natural solution in neural models for multiple inputs and multiple outputs, which is adopted in this work to decode constituent and dependency in a single model. BIBREF15 indicates that when tasks are sufficiently similar, especially with syntactic nature, MTL would be useful. In contrast to previous work on deep MTL BIBREF16, BIBREF17, our model focuses on more related tasks and benefits from the strong inherent relation. At last, our model is evaluated on two benchmark treebanks for both constituent and dependency parsing. The empirical results show that our parser reaches new state-of-the-art for all parsing tasks.

Answers:
 At last, our model is evaluated on two benchmark treebanks for both constituent and dependency parsing. The empirical results show that our parser reaches new state-of-the-art for all parsing tasks.
Question: How are different network components evaluated?

Evidence:
raw_evidence: Shared Self-attention Layers As our model providing two outputs from one input, there is a bifurcation setting for how much shared part should be determined. Both constituent and dependency parsers share token representation and 8 self-attention layers at most. Assuming that either parser always takes input information flow through 8 self-attention layers as shown in Figure FIGREF4, then the number of shared self-attention layers varying from 0 to 8 may reflect the shared degree in the model. When the number is set to 0, it indicates only token representation is shared for both parsers trained for the joint loss through each own 8 self-attention layers. When the number is set to less than 8, for example, 6, then it means that both parsers first shared 6 layers from token representation then have individual 2 self-attention layers.
For different numbers of shared layers, the results are in Table TABREF14. We respectively disable the constituent and the dependency parser to obtain a separate learning setting for both parsers in our model. The comparison in Table TABREF14 indicates that even though without any shared self-attention layers, joint training of our model may significantly outperform separate learning mode. At last, the best performance is still obtained from sharing full 8 self-attention layers.

Answers:
For different numbers of shared layers, the results are in Table TABREF14. We respectively disable the constituent and the dependency parser to obtain a separate learning setting for both parsers in our model.
Question: What are the models used to perform constituency and dependency parsing?

Evidence:
raw_evidence: Using an encoder-decoder backbone, our model may be regarded as an extension of the constituent parsing model of BIBREF18 as shown in Figure FIGREF4. The difference is that in our model both constituent and dependency parsing share the same token representation and shared self-attention layers and each has its own individual Self-Attention Layers and subsequent processing layers. Our model includes four modules: token representation, self-attention encoder, constituent and dependency parsing decoder.
Our Model ::: Constituent Parsing Decoder
Our Model ::: Dependency Parsing Decoder

Answers:
token representation, self-attention encoder,, Constituent Parsing Decoder,  Dependency Parsing Decoder
Question: What is the new initialization method proposed in this paper?

Evidence:
raw_evidence: Training our model implicitly embeds the vocabulary of words and collection of entities in a common space. However, we found that explicitly initializing these embeddings with vectors pre-trained over a large collection of unlabeled data significantly improved performance (see Section "Effects of initialized embeddings and corrupt-sampling schemes" ). To this end, we implemented an approach based on the Skip-Gram with Negative-Sampling (SGNS) algorithm by mikolov2013distributed that simultaneously trains both word and entity vectors.

Answers:
They initialize their word and entity embeddings with vectors pre-trained over a large corpus of unlabeled data.
Question: How was a quality control performed so that the text is noisy but the annotations are accurate?

Evidence:
raw_evidence: Wikilinks can be seen as a large-scale, naturally-occurring, crowd-sourced dataset where thousands of human annotators provide ground truths for mentions of interest. This means that the dataset contains various kinds of noise, especially due to incoherent contexts. The contextual noise presents an interesting test-case that supplements existing datasets that are sourced from mostly coherent and well-formed text.
We prepare our dataset from the local-context version of Wikilinks, and resolve ground-truth links using a Wikipedia dump from April 2016. We use the page and redirect tables for resolution, and keep the database pageid column as a unique identifier for Wikipedia entities. We discard mentions where the ground-truth could not be resolved (only 3% of mentions).

Answers:
The authors believe that the Wikilinks corpus  contains ground truth annotations while being noisy. They discard mentions that cannot have ground-truth verified by comparison with Wikipedia.
Question: Is it a neural model? How is it trained?

Evidence:
raw_evidence: Here we describe the components of our probabilistic model of question generation. Section "Compositionality and computability" describes two key elements of our approach, compositionality and computability, as reflected in the choice to model questions as programs. Section "A grammar for producing questions" describes a grammar that defines the space of allowable questions/programs. Section "Probabilistic generative model" specifies a probabilistic generative model for sampling context-sensitive, relevant programs from this space. The remaining sections cover optimization, the program features, and alternative models (Sections "Optimization" - "Alternative models" ).
The details of optimization are as follows. First, a large set of 150,000 questions is sampled in order to approximate the gradient at each step via importance sampling. Second, to run the procedure for a given model and training set, we ran 100,000 iterations of gradient ascent at a learning rate of 0.1. Last, for the purpose of evaluating the model (computing log-likelihood), the importance sampler is also used to approximate the normalizing constant in Eq. 14 via the estimator $Z \approx \mathbb {E}_{x\sim q}[\frac{p(x;\mathbf {\theta })}{q(x)}]$ .

Answers:
No, it is a probabilistic model trained by finding feature weights through gradient ascent
Question: How are the clusters related to security, violence and crime identified?

Evidence:

Answers:
Yes
Question: What are the features of used to customize target user interaction? 

Evidence:
raw_evidence: We represent each instance of image, question, and user choice as a triplet consisting of image feature, question feature, and the label vector for the user's answer. In addition, collecting multiple choices from identical users enables us to represent any two instances by the same user as a pair of triplets, assuming source-target relation. With these pairs of triplets, we can train the system to predict a user's choice on a new image and a new question, given the same user's choice on the previous image and its associated question. User's choice $x_{ans_i}$ is represented as one-hot vector where the size of the vector is equal to the number of possible choices. We refer to the fused feature representation of this triplet consisting of image, question, and the user's choice as choice vector.
As discussed earlier, we attempt to reflect user's interest by asking questions that provide visual context. The foremost prerequisite for the interactive questions to perform that function is the possibility of various answers or interpretations. In other words, a question whose answer is so obvious that it can be answered in an identical way would not be valid as an interactive question. In order to make sure that each generated question allows for multiple possible answers, we internally utilize the VQA module. The question generated by the VQG module is passed on to VQA module, where the probability distribution $p_{ans}$ for all candidate answers $C$ is determined. If the most likely candidate $c_i=\max p_{ans}$ , where $c_i \in C$ , has a probability of being answer over a certain threshold $\alpha $ , then the question is considered to have a single obvious answer, and is thus considered ineligible. The next question generated by VQG is passed on to VQA to repeat the same process until the the following requirement is met:
In our experiments, we set $\alpha $ as 0.33. We also excluded the yes/no type of questions. Figure 4 illustrates an example of a question where the most likely answer had a probability distribution over the threshold (and is thus ineligible), and another question whose probability distribution over the candidate answers was more evenly distributed (and thus proceeds to narrative generation stage).

Answers:
image feature, question feature, label vector for the user's answer
Question: How they evaluate quality of generated output?

Evidence:
raw_evidence: Automatic evaluation of Natural Language Generation (NLG) systems is a challenging task BIBREF22. For QG, $n$-gram based similarity metrics are commonly used. These measures evaluate how similar the generated text is to the corresponding reference(s). While they are known to suffer from several shortcomings BIBREF29, BIBREF30, they allow to evaluate specific properties of the developed models. In this work, the metrics detailed below are proposed and we evaluate their quality through a human evaluation in subsection SECREF32.
In addition to the automatic metrics, we proceeded to a human evaluation. We chose to use the data from our SQuAD-based experiments in order to also to measure the effectiveness of the proposed approach to derive Curiosity-driven QG data from a standard, non-conversational, QA dataset. We randomly sampled 50 samples from the test set. Three professional English speakers were asked to evaluate the questions generated by: humans (i.e. the reference questions), and models trained using pre-training (PT) or (RL), and all combinations of those methods.
Before submitting the samples for human evaluation, the questions were shuffled. Ratings were collected on a 1-to-5 likert scale, to measure to what extent the generated questions were: answerable by looking at their context; grammatically correct; how much external knowledge is required to answer; relevant to their context; and, semantically sound. The results of the human evaluation are reported in Table TABREF33.

Answers:
Through human evaluation where they are asked to evaluate the generated output on a likert scale.