qinchuanhui/UDA-QA huggingface dataset:
Question: What challenges remain unresolved?

Evidence:
raw_evidence: blackIn the paper, we first proposed a taxonomy to organize GAN based text-to-image synthesis frameworks into four major groups: semantic enhancement GANs, resolution enhancement GANs, diversity enhancement GANs, and motion enhancement GANs. The taxonomy provides a clear roadmap to show the motivations, architectures, and difference of different methods, and also outlines their evolution timeline and relationships. Following the proposed taxonomy, we reviewed important features of each method and their architectures. We indicated the model definition and key contributions from some advanced GAN framworks, including StackGAN, StackGAN++, AttnGAN, DC-GAN, AC-GAN, TAC-GAN, HDGAN, Text-SeGAn, StoryGAN etc. Many of the solutions surveyed in this paper tackled the highly complex challenge of generating photo-realistic images beyond swatch size samples. In other words, beyond the work of BIBREF8 in which images were generated from text in 64$\times $64 tiny swatches. Lastly, all methods were evaluated on datasets that included birds, flowers, humans, and other miscellaneous elements. We were also able to allocate some important papers that were as impressive as the papers we finally surveyed. Though, these notable papers have yet to contribute directly or indirectly to the expansion of the vast computer vision AI field. Looking into the future, an excellent extension from the works surveyed in this paper would be to give more independence to the several learning methods (e.g. less human intervention) involved in the studies as well as increasing the size of the output images.

Answers:
give more independence to the several learning methods (e.g. less human intervention) involved in the studies, increasing the size of the output images
Question: Where do they employ feature-wise sigmoid gating?

Evidence:
raw_evidence: The vector gate is inspired by BIBREF11 and BIBREF12 , but is different to the former in that the gating mechanism acts upon each dimension of the word and character-level vectors, and different to the latter in that it does not rely on external sources of information for calculating the gating mechanism.

Answers:
gating mechanism acts upon each dimension of the word and character-level vectors
Question: Which model architecture do they use to obtain representations?

Evidence:
raw_evidence: To enable sentence-level classification we need to obtain a sentence representation from the word vectors INLINEFORM0 . We achieved this by using a BiLSTM with max pooling, which was shown to be a good universal sentence encoding mechanism BIBREF13 .

Answers:
BiLSTM with max pooling
Question: Are there datasets with relation tuples annotated, how big are datasets available?

Evidence:
raw_evidence: We focus on the task of extracting multiple tuples with overlapping entities from sentences. We choose the New York Times (NYT) corpus for our experiments. This corpus has multiple versions, and we choose the following two versions as their test dataset has significantly larger number of instances of multiple relation tuples with overlapping entities. (i) The first version is used by BIBREF6 (BIBREF6) (mentioned as NYT in their paper) and has 24 relations. We name this version as NYT24. (ii) The second version is used by BIBREF11 (BIBREF11) (mentioned as NYT10 in their paper) and has 29 relations. We name this version as NYT29. We select 10% of the original training data and use it as the validation dataset. The remaining 90% is used for training. We include statistics of the training and test datasets in Table TABREF11.

Answers:
Yes
Question: Which one of two proposed approaches performed better in experiments?

Evidence:
raw_evidence: Among the baselines, HRL achieves significantly higher F1 scores on the two datasets. We run their model and our models five times and report the median results in Table TABREF15. Scores of other baselines in Table TABREF15 are taken from previous published papers BIBREF6, BIBREF11, BIBREF14. Our WordDecoding (WDec) model achieves F1 scores that are $3.9\%$ and $4.1\%$ higher than HRL on the NYT29 and NYT24 datasets respectively. Similarly, our PtrNetDecoding (PNDec) model achieves F1 scores that are $3.0\%$ and $1.3\%$ higher than HRL on the NYT29 and NYT24 datasets respectively. We perform a statistical significance test (t-test) under a bootstrap pairing between HRL and our models and see that the higher F1 scores achieved by our models are statistically significant ($p < 0.001$). Next, we combine the outputs of five runs of our models and five runs of HRL to build ensemble models. For a test instance, we include those tuples which are extracted in the majority ($\ge 3$) of the five runs. This ensemble mechanism increases the precision significantly on both datasets with a small improvement in recall as well. In the ensemble scenario, compared to HRL, WDec achieves $4.2\%$ and $3.5\%$ higher F1 scores and PNDec achieves $4.2\%$ and $2.9\%$ higher F1 scores on the NYT29 and NYT24 datasets respectively.

Answers:
WordDecoding (WDec) model
Question: What is previous work authors reffer to?

Evidence:
raw_evidence: We compare our model with the following state-of-the-art joint entity and relation extraction models:
(1) SPTree BIBREF4: This is an end-to-end neural entity and relation extraction model using sequence LSTM and Tree LSTM. Sequence LSTM is used to identify all the entities first and then Tree LSTM is used to find the relation between all pairs of entities.
(2) Tagging BIBREF5: This is a neural sequence tagging model which jointly extracts the entities and relations using an LSTM encoder and an LSTM decoder. They used a Cartesian product of entity tags and relation tags to encode the entity and relation information together. This model does not work when tuples have overlapping entities.
(3) CopyR BIBREF6: This model uses an encoder-decoder approach for joint extraction of entities and relations. It copies only the last token of an entity from the source sentence. Their best performing multi-decoder model is trained with a fixed number of decoders where each decoder extracts one tuple.
(4) HRL BIBREF11: This model uses a reinforcement learning (RL) algorithm with two levels of hierarchy for tuple extraction. A high-level RL finds the relation and a low-level RL identifies the two entities using a sequence tagging approach. This sequence tagging approach cannot always ensure extraction of exactly two entities.
(5) GraphR BIBREF14: This model considers each token in a sentence as a node in a graph, and edges connecting the nodes as relations between them. They use graph convolution network (GCN) to predict the relations of every edge and then filter out some of the relations.
(6) N-gram Attention BIBREF9: This model uses an encoder-decoder approach with N-gram attention mechanism for knowledge-base completion using distantly supervised data. The encoder uses the source tokens as its vocabulary and the decoder uses the entire Wikidata BIBREF15 entity IDs and relation IDs as its vocabulary. The encoder takes the source sentence as input and the decoder outputs the two entity IDs and relation ID for every tuple. During training, it uses the mapping of entity names and their Wikidata IDs of the entire Wikidata for proper alignment. Our task of extracting relation tuples with the raw entity names from a sentence is more challenging since entity names are not of fixed length. Our more generic approach is also helpful for extracting new entities which are not present in the existing knowledge bases such as Wikidata. We use their N-gram attention mechanism in our model to compare its performance with other attention models (Table TABREF17).

Answers:
SPTree, Tagging, CopyR, HRL, GraphR, N-gram Attention
Question: what were the baselines?

Evidence:
raw_evidence: We compare our system to a variety of baselines. (1) Rank by the number of times a citation is mentioned in the document. (2) Rank by the number of times the citation is cited in the literature (citation impact). (3) Rank using Google Scholar Related Articles. (4) Rank by the TF*IDF weighted cosine similarity. (5) Rank using a learning-to-rank model trained on text similarity rankings. The first two baseline systems are models where the values are ordered from highest to lowest to generate the ranking. The idea behind them is that the number of times a citation is mentioned in an article, or the citation impact may already be good indicators of their closeness. The text similarity model is trained using the same features and methods used by the annotation model, but trained using text similarity rankings instead of the author's judgments.
raw_evidence: We compare our system to a variety of baselines. (1) Rank by the number of times a citation is mentioned in the document. (2) Rank by the number of times the citation is cited in the literature (citation impact). (3) Rank using Google Scholar Related Articles. (4) Rank by the TF*IDF weighted cosine similarity. (5) Rank using a learning-to-rank model trained on text similarity rankings. The first two baseline systems are models where the values are ordered from highest to lowest to generate the ranking. The idea behind them is that the number of times a citation is mentioned in an article, or the citation impact may already be good indicators of their closeness. The text similarity model is trained using the same features and methods used by the annotation model, but trained using text similarity rankings instead of the author's judgments.

Answers:
Rank by the number of times a citation is mentioned in the document,  Rank by the number of times the citation is cited in the literature (citation impact). , Rank using Google Scholar Related Articles., Rank by the TF*IDF weighted cosine similarity. , ank using a learning-to-rank model trained on text similarity rankings
(1) Rank by the number of times a citation is mentioned in the document., (2) Rank by the number of times the citation is cited in the literature (citation impact)., (3) Rank using Google Scholar Related Articles., (4) Rank by the TF*IDF weighted cosine similarity., (5) Rank using a learning-to-rank model trained on text similarity rankings.
Question: what is the supervised model they developed?

Evidence:
raw_evidence: Support Vector Machine (SVM) ( BIBREF25 ) is a commonly used supervised classification algorithm that has shown good performance over a range of tasks. SVM can be thought of as a binary linear classifier where the goal is to maximize the size of the gap between the class-separating line and the points on either side of the line. This helps avoid over-fitting on the training data. SVMRank is a modification to SVM that assigns scores to each data point and allows the results to be ranked ( BIBREF26 ). We use SVMRank in the experiments below. SVMRank has previously been used in the task of document retrieval in ( BIBREF27 ) for a more traditional short query task and has been shown to be a top-performing system for ranking.

Answers:
SVMRank
Question: what is the size of this built corpus?

Evidence:
raw_evidence: We asked authors to rank documents by how “close to your work” they were. The definition of closeness was left to the discretion of the author. The dataset is composed of 90 annotated documents with 5 citations each ranked 1 to 5, where 1 is least relevant and 5 is most relevant for a total of 450 annotated citations.

Answers:
90 annotated documents with 5 citations each ranked 1 to 5, where 1 is least relevant and 5 is most relevant for a total of 450 annotated citations
Question: what crowdsourcing platform is used?

Evidence:
raw_evidence: Given the full text of a scientific publication, we want to rank its citations according to the author's judgments. We collected recent publications from the open-access PLoS journals and asked the authors to rank by closeness five citations we selected from their paper. PLoS articles were selected because its journals cover a wide array of topics and the full text articles are available in XML format. We selected the most recent publications as previous work in crowd-sourcing annotation shows that authors' willingness to participate in an unpaid annotation task declines with the age of publication ( BIBREF23 ). We then extracted the abstract, citations, full text, authors, and corresponding author email address from each document. The titles and abstracts of the citations were retrieved from PubMed, and the cosine similarity between the PLoS abstract and the citation's abstract was calculated. We selected the top five most similar abstracts using TF*IDF weighted cosine similarity, shuffled their order, and emailed them to the corresponding author for annotation. We believe that ranking five articles (rather than the entire collection of the references) is a more manageable task for an author compared to asking them to rank all references. Because the documents to be annotated were selected based on text similarity, they also represent a challenging baseline for models based on text-similarity features. In total 416 authors were contacted, and 92 responded (22% response rate). Two responses were removed from the dataset for incomplete annotation.

Answers:
asked the authors to rank by closeness five citations we selected from their paper
Question: Which deep learning model performed better?

Evidence:
raw_evidence: To evaluate the performance of the proposed approach, precision (1), recall (2), f-Measure (3), and prediction accuracy (4) have been used as a performance matrices. The experimental results are shown in Table 1, where it can be seen that autoencoders outperformed MLP and CNN outperformed autoencoders with the highest achieved accuracy of 82.6%. DISPLAYFORM0 DISPLAYFORM1
raw_evidence: In the literature, deep learning based automated feature extraction has been shown to outperform state-of-the-art manual feature engineering based classifiers such as Support Vector Machine (SVM), Naive Bayes (NB) or Multilayer Perceptron (MLP) etc. One of the important techniques in deep learning is the autoencoder that generally involves reducing the number of feature dimensions under consideration. The aim of dimensionality reduction is to obtain a set of principal variables to improve the performance of the approach. Similarly, CNNs have been proven to be very effective in sentiment analysis. However, little work has been carried out to exploit deep learning based feature representation for Persian sentiment analysis BIBREF8 BIBREF9 . In this paper, we present two deep learning models (deep autoencoders and CNNs) for Persian sentiment analysis. The obtained deep learning results are compared with MLP.
To evaluate the performance of the proposed approach, precision (1), recall (2), f-Measure (3), and prediction accuracy (4) have been used as a performance matrices. The experimental results are shown in Table 1, where it can be seen that autoencoders outperformed MLP and CNN outperformed autoencoders with the highest achieved accuracy of 82.6%. DISPLAYFORM0 DISPLAYFORM1

Answers:
autoencoders
CNN
Question: What was their performance on the dataset?

Evidence:
raw_evidence: To evaluate the performance of the proposed approach, precision (1), recall (2), f-Measure (3), and prediction accuracy (4) have been used as a performance matrices. The experimental results are shown in Table 1, where it can be seen that autoencoders outperformed MLP and CNN outperformed autoencoders with the highest achieved accuracy of 82.6%. DISPLAYFORM0 DISPLAYFORM1

Answers:
accuracy of 82.6%
Question: Did the authors use crowdsourcing platforms?

Evidence:
raw_evidence: We crowd-sourced the collection of the dataset on Amazon Mechanical Turk (MTurk). We use the MTurk interface of ParlAI BIBREF6 to render 360 images via WebGL and dynamically display neighborhood maps with an HTML5 canvas. Detailed task instructions, which were also given to our workers before they started their task, are shown in Appendix SECREF15 . We paired Turkers at random and let them alternate between the tourist and guide role across different HITs.
raw_evidence: We introduce the Talk the Walk dataset, where the aim is for two agents, a “guide” and a “tourist”, to interact with each other via natural language in order to achieve a common goal: having the tourist navigate towards the correct location. The guide has access to a map and knows the target location, but does not know where the tourist is; the tourist has a 360-degree view of the world, but knows neither the target location on the map nor the way to it. The agents need to work together through communication in order to successfully solve the task. An example of the task is given in Figure FIGREF3 .
We crowd-sourced the collection of the dataset on Amazon Mechanical Turk (MTurk). We use the MTurk interface of ParlAI BIBREF6 to render 360 images via WebGL and dynamically display neighborhood maps with an HTML5 canvas. Detailed task instructions, which were also given to our workers before they started their task, are shown in Appendix SECREF15 . We paired Turkers at random and let them alternate between the tourist and guide role across different HITs.

Answers:
Yes
Yes
Question: How was the dataset collected?

Evidence:
raw_evidence: We crowd-sourced the collection of the dataset on Amazon Mechanical Turk (MTurk). We use the MTurk interface of ParlAI BIBREF6 to render 360 images via WebGL and dynamically display neighborhood maps with an HTML5 canvas. Detailed task instructions, which were also given to our workers before they started their task, are shown in Appendix SECREF15 . We paired Turkers at random and let them alternate between the tourist and guide role across different HITs.

Answers:
crowd-sourced the collection of the dataset on Amazon Mechanical Turk (MTurk)
Question: What language do the agents talk in?

Evidence:
raw_evidence: Tourist: I can't go straight any further.
Guide: ok. turn so that the theater is on your right.
Guide: then go straight
Tourist: That would be going back the way I came
Guide: yeah. I was looking at the wrong bank
Tourist: I'll notify when I am back at the brooks brothers, and the bank.
Tourist: ACTION:TURNRIGHT ACTION:TURNRIGHT
Guide: make a right when the bank is on your left
Tourist: ACTION:FORWARD ACTION:FORWARD ACTION:TURNRIGHT
Tourist: Making the right at the bank.
Tourist: ACTION:FORWARD ACTION:FORWARD ACTION:FORWARD ACTION:TURNLEFT ACTION:TURNLEFT
Tourist: I can't go that way.
Tourist: ACTION:TURNLEFT ACTION:TURNLEFT ACTION:TURNLEFT
Tourist: Bank is ahead of me on the right
Tourist: ACTION:FORWARD ACTION:FORWARD ACTION:TURNLEFT
Guide: turn around on that intersection
Tourist: I can only go to the left or back the way I just came.
Guide: you're in the right place. do you see shops on the corners?
Guide: If you're on the corner with the bank, cross the street
Tourist: I'm back where I started by the shop and the bank.

Answers:
English
Question: What evaluation metrics did the authors look at?

Evidence:
raw_evidence: In this section, we describe the findings of various experiments. First, we analyze how much information needs to be communicated for accurate localization in the Talk The Walk environment, and find that a short random path (including actions) is necessary. Next, for emergent language, we show that the MASC architecture can achieve very high localization accuracy, significantly outperforming the baseline that does not include this mechanism. We then turn our attention to the natural language experiments, and find that localization from human utterances is much harder, reaching an accuracy level that is below communicating a single landmark observation. We show that generated utterances from a conditional language model leads to significantly better localization performance, by successfully grounding the utterance on a single landmark observation (but not yet on multiple observations and actions). Finally, we show performance of the localization baseline on the full task, which can be used for future comparisons to this work.

Answers:
localization accuracy
Question: What data did they use?

Evidence:
raw_evidence: Talk The Walk is the first task to bring all three aspects together: perception for the tourist observing the world, action for the tourist to navigate through the environment, and interactive dialogue for the tourist and guide to work towards their common goal. To collect grounded dialogues, we constructed a virtual 2D grid environment by manually capturing 360-views of several neighborhoods in New York City (NYC). As the main focus of our task is on interactive dialogue, we limit the difficulty of the control problem by having the tourist navigating a 2D grid via discrete actions (turning left, turning right and moving forward). Our street view environment was integrated into ParlAI BIBREF6 and used to collect a large-scale dataset on Mechanical Turk involving human perception, action and communication.

Answers:
 dataset on Mechanical Turk involving human perception, action and communication
Question: How is the accuracy of the system measured?

Evidence:
raw_evidence: The graphs in figure 1 show the distances between duplicate and non-duplicate questions using different embedding systems. The X axis shows the euclidean distance between vectors and the Y axis frequency. A perfect result would be a blue peak to the left and an entirely disconnected orange spike to the right, showing that all non-duplicate questions have a greater euclidean distance than the least similar duplicate pair of questions. As can be clearly seen in the figure above, Elmo BIBREF23 and Infersent BIBREF13 show almost no separation and therefore cannot be considered good models for this problem. A much greater disparity is shown by the Google USE models BIBREF14 , and even more for the Google USE Large model. In fact the Google USE Large achieved a F1 score of 0.71 for this task without any specific training, simply by choosing a threshold below which all sentence pairs are considered duplicates.
In order to test whether these results generalised to our domain, we devised a test that would make use of what little data we had to evaluate. We had no original data on whether sentences were semantically similar, but we did have a corpus of articles clustered into stories. Working on the assumption that similar claims would be more likely to be in the same story, we developed an equation to judge how well our corpus of sentences was clustered, rewarding clustering which matches the article clustering and the total number of claims clustered. The precise formula is given below, where INLINEFORM0 is the proportion of claims in clusters from one story cluster, INLINEFORM1 is the proportion of claims in the correct claim cluster, where they are from the most common story cluster, and INLINEFORM2 is the number of claims placed in clusters. A,B and C are parameters to tune. INLINEFORM3

Answers:
F1 score of 0.71 for this task without any specific training, simply by choosing a threshold below which all sentence pairs are considered duplicates, distances between duplicate and non-duplicate questions using different embedding systems