qinchuanhui/UDA-QA huggingface dataset:
Question: What is the size of the labelled dataset?

Evidence:
raw_evidence: We collected a total of 96,566 tweets from TwiNL, which we filtered in a number of ways. First, retweets were removed, as we wanted to focus on unique messages. This led to a removal of 31% of the messages. Second, we filtered out messages that contain a URL. Such messages often share a news headline and include a URL to refer to the complete news message. As a news headline does not reflect the stance of the person who posted the tweet, we decided to apply this filtering step. It is likely that part of the messages with a URL do include a message composed by the sender itself, but this step helps to clean many unwanted messages. Third, we removed messages that include a word related to animals and traveling (`dier’, animal; `landbouw’, agriculture; and `teek’, tick), as we strictly focus on messages that refer to vaccination that is part of the governmental vaccination program. 27,534 messages were left after filtering. This is the data set that is used for experimentation.

Answers:
27,534 messages
Question: Which features do they use to model Twitter messages?

Evidence:
raw_evidence: To properly distinguish word tokens and punctuation we tokenized the tweets by means of Ucto, a rule-based tokenizer with good performance on the Dutch language, and with a configuration specific for Twitter. Tokens were lowercased in order to focus on the content. Punctuation was maintained, as well as emoji and emoticons. Such markers could be predictive in the context of a discussion such as vaccination. To account for sequences of words and characters that might carry useful information, we extracted word unigrams, bigrams, and trigrams as features. Features were coded binary, i.e. set to 1 if a feature is seen in a message and set to 0 otherwise. During training, all features apart from the top 15,000 most frequent ones were removed.

Answers:
word unigrams, bigrams, and trigrams
Question: Do they allow for messages with vaccination-related key terms to be of neutral stance?

Evidence:
raw_evidence: The stance towards vaccination was categorized into `Negative’, `Neutral’, `Positive’ and `Not clear’. The latter category was essential, as some posts do not convey enough information about the stance of the writer. In addition to the four-valued stance classes we included separate classes grouped under relevance, subject and sentiment as annotation categories. With these additional categorizations we aimed to obtain a precise grasp of all possibly relevant tweet characteristics in relation to vaccination, which could help in a machine learning setting.

Answers:
Yes
Question: Is this a span-based (extractive) QA task?

Evidence:
raw_evidence: We evaluate our approaches on two public Chinese span-extraction machine reading comprehension datasets: CMRC 2018 (simplified Chinese) BIBREF8 and DRCD (traditional Chinese) BIBREF9. The statistics of the two datasets are listed in Table TABREF29.

Answers:
Yes
Question: What datasets are used for training/testing models? 

Evidence:
raw_evidence: The experiment dataset comes from Microsoft Research (MSR) . It contains three domains: movie, taxi, and restaurant. The total count of dialogues per domain and train/valid/test split is reported in Table TABREF11. At every turn both user and agent acts are annotated, we use only the agent side as targets in our experiment. The acts are ordered in the dataset (each output sentence aligns with one act). The size of the sets of acts, slots, and act-slot pairs are also listed in Table TABREF11. Table TABREF12 shows the count of turns with multiple act annotations, which amounts to 23% of the dataset. We use MSR's dialogue management code and knowledge base to obtain the state at each turn and use it as input to every model.

Answers:
Microsoft Research dataset containing movie, taxi and restaurant domains.
Question: What is specific to gCAS cell?

Evidence:
raw_evidence: In this paper, we introduce a novel policy model to output multiple actions per turn (called multi-act), generating a sequence of tuples and expanding agents' expressive power. Each tuple is defined as $(\textit {continue}, \textit {act}, \textit {slots})$, where continue indicates whether to continue or stop producing new acts, act is an act type (e.g., inform or request), and slots is a set of slots (names) associated with the current act type. Correspondingly, a novel decoder (Figure FIGREF5) is proposed to produce such sequences. Each tuple is generated by a cell called gated Continue Act Slots (gCAS, as in Figure FIGREF7), which is composed of three sequentially connected gated units handling the three components of the tuple. This decoder can generate multi-acts in a double recurrent manner BIBREF18. We compare this model with baseline classifiers and sequence generation models and show that it consistently outperforms them.

Answers:
It has three sequentially connected units to output continue, act and slots generating multi-acts in a doble recurrent manner.
Question: What dataset do they evaluate their model on?

Evidence:
raw_evidence: The CoNLL-2012 shared task BIBREF21 corpus is used as the evaluation dataset, which is selected from the Ontonotes 5.0. Following conventional approaches BIBREF9 , BIBREF11 , for each pronoun in the document, we consider candidate $n$ from the previous two sentences and the current sentence. For pronouns, we consider two types of them following BIBREF9 , i.e., third personal pronoun (she, her, he, him, them, they, it) and possessive pronoun (his, hers, its, their, theirs). Table 1 reports the number of the two type pronouns and the overall statistics for the experimental dataset. According to our selection range of candidate $n$ , on average, each pronoun has 4.6 candidates and 1.3 correct references.

Answers:
CoNLL-2012 shared task BIBREF21 corpus
Question: What is the source of external knowledge?

Evidence:
raw_evidence: The second type is the selectional preference (SP) knowledge. For this knowledge, we create a knowledge base by counting how many times a predicate-argument tuple appears in a corpus and use the resulted number to represent the preference strength. Specifically, we use the English Wikipedia as the base corpus for such counting. Then we parse the entire corpus through the Stanford parser and record all dependency edges in the format of (predicate, argument, relation, number), where predicate is the governor and argument the dependent in the original parsed dependency edge. Later for sentences in the training and test data, we firstly parse each sentence and find out the dependency edge linking $p$ and its corresponding predicate. Then for each candidate $n$ in a sentence, we check the previously created SP knowledge base and find out how many times it appears as the argument of different predicates with the same dependency relation (i.e., nsubj and dobj). The resulted frequency is grouped into the following buckets [1, 2, 3, 4, 5-7, 8-15, 16-31, 32-63, 64+] and we use the bucket id as the final SP knowledge. Thus in the previous example:

Answers:
counts of predicate-argument tuples from English Wikipedia
Question: Which of their proposed attention methods works better overall?

Evidence:
raw_evidence: Tables and summarize the results from the link-prediction experiments on all three datasets, where a different ratio of edges are used for training. Results from models other than GANE are collected from BIBREF9 , BIBREF10 and BIBREF34 . We have also repeated these experiments on our own, and the results are consistent with the ones reported. Note that BIBREF34 did not report results on DMTE. Both GANE variants consistently outperform competing solutions. In the low-training-sample regime our solutions lead by a large margin, and the performance gap closes as the number of training samples increases. This indicates that our OT-based mutual attention framework can yield more informative textual representations than other methods. Note that GANE-AP delivers better results compared with GANE-OT, suggesting the attention parsing mechanism can further improve the low-level mutual attention matrix. More results on Cora and Hepth are provided in the SM.

Answers:
attention parsing
Question: Which dataset of texts do they use?

Evidence:
raw_evidence: We consider three benchmark datasets: ( INLINEFORM0 ) Cora, a paper citation network with text information, built by BIBREF44 . We prune the dataset so that it only has papers on the topic of machine learning. ( INLINEFORM1 ) Hepth, a paper citation network from Arxiv on high energy physics theory, with paper abstracts as text information. ( INLINEFORM2 ) Zhihu, a Q&A network dataset constructed by BIBREF9 , which has 10,000 active users with text descriptions and their collaboration links. Summary statistics of these three datasets are summarized in Table . Pre-processing protocols from prior studies are used for data preparation BIBREF10 , BIBREF34 , BIBREF9 .

Answers:
Cora, Hepth, Zhihu
Question: Do they measure how well they perform on longer sequences specifically?

Evidence:
raw_evidence: We further explore the effect of INLINEFORM0 -gram length in our model (i.e., the filter size for the covolutional layers used by the attention parsing module). In Figure FIGREF39 we plot the AUC scores for link prediction on the Cora dataset against varying INLINEFORM1 -gram length. The performance peaked around length 20, then starts to drop, indicating a moderate attention span is more preferable. Similar results are observed on other datasets (results not shown). Experimental details on the ablation study can be found in the SM.

Answers:
Yes
Question: Which other embeddings do they compare against?

Evidence:
raw_evidence: To demonstrate the effectiveness of the proposed solutions, we evaluated our model along with the following strong baselines. ( INLINEFORM0 ) Topology only embeddings: MMB BIBREF45 , DeepWalk BIBREF1 , LINE BIBREF33 , Node2vec BIBREF46 . ( INLINEFORM1 ) Joint embedding of topology & text: Naive combination, TADW BIBREF5 , CENE BIBREF6 , CANE BIBREF9 , WANE BIBREF10 , DMTE BIBREF34 . A brief summary of these competing models is provided in the Supplementary Material (SM).

Answers:
MMB, DeepWalk, LINE,  Node2vec, TADW, CENE, CANE, WANE, DMTE
Question: What training data did they use?

Evidence:
raw_evidence: Each drug label is a collection of sections (e.g., DOSAGE & ADMINISTRATION, CONTRAINDICATIONS, and WARNINGS) where each section contains one or more sentences. Each sentence is annotated with a list of zero or more mentions and interactions. The training data released for this task contains 22 drug labels, referred to as Training-22, with gold standard annotations. Two test sets of 57 and 66 drug labels, referred to as Test Set 1 and 2 respectively, with gold standard annotations are used to evaluate participating systems. As Training-22 is a relatively small dataset, we additionally utilize an external dataset with 180 annotated drug labels dubbed NLM-180 BIBREF5 (more later). We provide summary statistics about these datasets in Table TABREF3 . Test Set 1 closely resembles Training-22 with respect to the sections that are annotated. However, Test Set 1 is more sparse in the sense that there are more sentences per drug label (144 vs. 27), with a smaller proportion of those sentences having gold annotations (23% vs. 51%). Test Set 2 is unique in that it contains annotations from only two sections, namely DRUG INTERACTIONS and CLINICAL PHARMACOLOGY, the latter of which is not represented in Training-22 (nor Test Set 1). Lastly, Training-22, Test Set 1, and Test Set 2 all vary with respect to the distribution of interaction types, with Training-22, Test Set 1, and Test Set 2 containing a higher proportion of PD, UN, and PK interactions respectively.

Answers:
Training-22, NLM-180
Question: What domains do they experiment with?

Evidence:
raw_evidence: With unsupervised domain adaptation, one has access to labeled sentence specificity in one source domain, and unlabeled sentences in all target domains. The goal is to predict the specificity of target domain data. Our source domain is news, the only domain with publicly available labeled data for training BIBREF1 . We crowdsource sentence specificity for evaluation for three target domains: Twitter, Yelp reviews and movie reviews. The data is described in Section SECREF4 .

Answers:
Twitter, Yelp reviews and movie reviews
Question: What games are used to test author's methods?

Evidence:
raw_evidence: TextWorld uses a grammar to generate similar games. Following BIBREF7, we use TextWorld's “home” theme to generate the games for the question-answering system. TextWorld is a framework that uses a grammar to randomly generate game worlds and quests. This framework also gives us information such as instructions on how to finish the quest, and a list of actions that can be performed at each step based on the current world state. We do not let our agent access this additional solution information or admissible actions list. Given the relatively small quest length for TextWorld games—games can be completed in as little as 5 steps—we generate 50 such games and partition them into train and test sets in a 4:1 ratio. The traces are generated on the training set, and the question-answering system is evaluated on the test set.
We choose the game, 9:05 as our target task game due to similarities in structure in addition to the vocabulary overlap. Note that there are multiple possible endings to this game and we pick the simplest one for the purpose of training our agent.
For the horror domain, we choose Lurking Horror to train the question-answering system on. The source and target task games are chosen as Afflicted and Anchorhead respectively. However, due to the size and complexity of these two games some modifications to the games are required for the agent to be able to effectively solve them. We partition each of these games and make them smaller by reducing the final goal of the game to an intermediate checkpoint leading to it. This checkpoints were identified manually using walkthroughs of the game; each game has a natural intermediate goal. For example, Anchorhead is segmented into 3 chapters in the form of objectives spread across 3 days, of which we use only the first chapter. The exact details of the games after partitioning is described in Table TABREF7. For Lurking Horror, we report numbers relevant for the oracle walkthrough. We then pre-prune the action space and use only the actions that are relevant for the sections of the game that we have partitioned out. The majority of the environment is still available for the agent to explore but the game ends upon completion of the chosen intermediate checkpoint.

Answers:
Lurking Horror, Afflicted, Anchorhead, 9:05, TextWorld games
Question: How is the domain knowledge transfer represented as knowledge graph?

Evidence:
raw_evidence: The agent also has access to all actions accepted by the game's parser, following BIBREF2. For general interactive fiction environments, we develop our own method to extract this information. This is done by extracting a set of templates accepted by the parser, with the objects or noun phrases in the actions replaces with a OBJ tag. An example of such a template is "place OBJ in OBJ". These OBJ tags are then filled in by looking at all possible objects in the given vocabulary for the game. This action space is of the order of $A=\mathcal {O}(|V| \times |O|^2)$ where $V$ is the number of action verbs, and $O$ is the number of distinct objects in the world that the agent can interact with. As this is too large a space for a RL agent to effectively explore, the knowledge graph is used to prune this space by ranking actions based on their presence in the current knowledge graph and the relations between the objects in the graph as in BIBREF7

Answers:
the knowledge graph is used to prune this space by ranking actions based on their presence in the current knowledge graph and the relations between the objects in the graph as in BIBREF7
Question: What was the baseline?

Evidence:
raw_evidence: Early approaches to Opinion Target Extraction (OTE) were unsupervised, although later on the vast majority of works have been based on supervised and deep learning models. To the best of our knowledge, the first work on OTE was published by BIBREF8 . They created a new task which consisted of generating overviews of the main product features from a collection of customer reviews on consumer electronics. They addressed such task using an unsupervised algorithm based on association mining. Other early unsupervised approaches include BIBREF9 which used a dependency parser to obtain more opinion targets, and BIBREF10 which aimed at extracting opinion targets in newswire via Semantic Role Labelling. From a supervised perspective, BIBREF11 presented an approach which learned the opinion target candidates and a combination of dependency and part-of-speech (POS) paths connecting such pairs. Their results improved the baseline provided by BIBREF8 . Another influential work was BIBREF12 , an unsupervised algorithm called Double Propagation which roughly consists of incrementally augmenting a set of seeds via dependency parsing.
In spite of this, Table TABREF20 shows that our system outperforms the best previous approaches across the five languages. In some cases, such as Turkish and Russian, the best previous scores were the baselines provided by the ABSA organizers, but for Dutch, French and Spanish our system is significantly better than current state-of-the-art. In particular, and despite using the same system for every language, we improve over GTI's submission, which implemented a CRF system with linguistic features specific to Spanish BIBREF28 .

Answers:
the baseline provided by BIBREF8, the baselines provided by the ABSA organizers
Question: Which six languages are experimented with?

Evidence:
raw_evidence: In this section we report on the experiments performed using the system and data described above. First we will present the English results for the three ABSA editions as well as a comparison with previous work. After that we will do the same for 5 additional languages included in the ABSA 2016 edition: Dutch, French, Russian, Spanish and Turkish. The local and clustering features, as described in Section SECREF11 , are the same for every language and evaluation setting. The only change is the clustering lexicons used for the different languages. As stated in section SECREF11 , the best cluster combination is chosen via 5-fold cross validation (CV) on the training data. We first try every permutation with the Clark and Word2vec clusters. Once the best combination is obtained, we then try with the Brown clusters obtaining thus the final model for each language and dataset.

Answers:
Dutch, French, Russian, Spanish , Turkish, English
Question: What shallow local features are extracted?

Evidence:
raw_evidence: The system consists of: (i) Local, shallow features based mostly on orthographic, word shape and n-gram features plus their context; and (ii) three types of simple clustering features, based on unigram matching: (i) Brown BIBREF32 clusters, taking the 4th, 8th, 12th and 20th node in the path; (ii) Clark BIBREF33 clusters and, (iii) Word2vec BIBREF34 clusters, based on K-means applied over the extracted word vectors using the skip-gram algorithm.

Answers:
 Local, shallow features based mostly on orthographic, word shape and n-gram features plus their context
Question: Do they compare results against state-of-the-art language models?

Evidence:
raw_evidence: All models outperform previously reported results for mlstm BIBREF8 despite lower parameter counts. This is likely due to our relatively small batch size. However, they perform fairly similarly. Encouraged by these results, we built an mgru with both hidden and intermediate state sizes set to that of the original mlstm (700). This version highly surpasses the previous state of the art while still having fewer parameters than previous work.

Answers:
Yes
Question: Which dataset do they train their models on?

Evidence:
raw_evidence: Character-level language modeling (or character prediction) consists in predicting the next character while reading a document one character at a time. It is a common benchmark for rnn because of the heightened need for shared parametrization when compared to word-level models. We test mgru on two well-known datasets, the Penn Treebank and Text8.

Answers:
Penn Treebank, Text8
Question: It looks like learning to paraphrase questions, a neural scoring model and a answer selection model cannot be trained end-to-end. How are they trained?

Evidence:
raw_evidence: BIBREF11 revisit bilingual pivoting in the context of neural machine translation (NMT, BIBREF12 , BIBREF13 ) and present a paraphrasing model based on neural networks. At its core, NMT is trained end-to-end to maximize the conditional probability of a correct translation given a source sentence, using a bilingual corpus. Paraphrases can be obtained by translating an English string into a foreign language and then back-translating it into English. NMT-based pivoting models offer advantages over conventional methods such as the ability to learn continuous representations and to consider wider context while paraphrasing.

Answers:
using multiple pivot sentences
Question: What multimodal representations are used in the experiments?

Evidence:
raw_evidence: Evaluation of Representation Models ::: Experiments ::: Multimodal representation.
We combined textual and image representations in two simple ways. The first method is concatenation of the text and image representation (concat). Before concatenation we applied the L2 normalization to each of the modalities. The second method it to learn a common space for the two modalities before concatenation (project).
The projection of each modality learns a space of $d$-dimensions, so that $h_{1}, h_{2} \in \mathbb {R}^{d}$. Once the multimodal representation is produced ($h_{m}$) for the left and right pairs, vectors are directly plugged into the regression layers. Projections are learned end-to-end with the regression layers and the MSE as loss function.

Answers:
The second method it to learn a common space for the two modalities before concatenation (project), The first method is concatenation of the text and image representation (concat)
Question: How much better is inference that has addition of image representation compared to text-only representations? 

Evidence:
raw_evidence: Table TABREF31 summarizes the contribution of the images on text representations in test partition. The contribution is consistent through all text-based representations. We measure the absolute difference (Diff) and the error reduction (E.R) of each textual representation with the multimodal counterpart. For the comparison we chose the best text model for each representation. As expected we obtain the largest improvement ($22-26\%$ E.R) when text-based unsupervised models are combined with image representations. Note that unsupervised models are not learning anything about the specific task, so the more information in the representation, the better. In the case of use and vse++ the improvement is significant but not as large as the purely unsupervised models. The best text-only representation is the one fine-tuned on a multimodal task, VSE++, which is noteworthy, as it is better than a textual representation fine-tuned in a text-only inference task like USE.

Answers:
 largest improvement ($22-26\%$ E.R) when text-based unsupervised models are combined with image representations
Question: How they compute similarity between the representations?

Evidence:
raw_evidence: In the unsupervised scenario similarity is computed as the cosine of the produced $h_{L}$ and $h_{R}$ sentence/image representations.

Answers:
similarity is computed as the cosine of the produced $h_{L}$ and $h_{R}$ sentence/image representations