z-uo/qasper-squad huggingface dataset:
Paper: A Survey and Taxonomy of Adversarial Neural Networks for Text-to-Image Synthesis
In the early stages of research, text-to-image synthesis was mainly carried out through a search and supervised learning combined process BIBREF4, as shown in Figure FIGREF4. In order to connect text descriptions to images, one could use correlation between keywords (or keyphrase) & images that identifies informative and “picturable” text units; then, these units would search for the most likely image parts conditioned on the text, eventually optimizing the picture layout conditioned on both the text and the image parts. Such methods often integrated multiple artificial intelligence key components, including natural language processing, computer vision, computer graphics, and machine learning.
Question: What challenges remain unresolved?
Paper: A Survey and Taxonomy of Adversarial Neural Networks for Text-to-Image Synthesis
Although generative model based text-to-image synthesis provides much more realistic image synthesis results, the image generation is still conditioned by the limited attributes. In recent years, several papers have been published on the subject of text-to-image synthesis. Most of the contributions from these papers rely on multimodal learning approaches that include generative adversarial networks and deep convolutional decoder networks as their main drivers to generate entrancing images from text BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11.
Question: What is the conclusion of comparison of proposed solution?
Paper: A Survey and Taxonomy of Adversarial Neural Networks for Text-to-Image Synthesis
With the growth and success of GANs, deep convolutional decoder networks, and multimodal learning methods, these techniques were some of the first procedures which aimed to solve the challenge of image synthesis. Many engineers and scientists in computer vision and AI have contributed through extensive studies and experiments, with numerous proposals and publications detailing their contributions. Because GANs, introduced by BIBREF9, are emerging research topics, their practical applications to image synthesis are still in their infancy. Recently, many new GAN architectures and designs have been proposed to use GANs for different applications, e.g. using GANs to generate sentimental texts BIBREF18, or using GANs to transform natural images into cartoons BIBREF19.
Question: What is typical GAN architecture for each text-to-image synhesis group?
Answer: Semantic Enhancement GANs: DC-GANs, MC-GAN
Resolution Enhancement GANs: StackGANs, AttnGAN, HDGAN
Diversity Enhancement GANs: AC-GAN, TAC-GAN etc.
Motion Enhancement GAGs: T2S, T2V, StoryGAN
Paper: Gating Mechanisms for Combining Character and Word-level Word Representations: An Empirical Study
Incorporating sub-word structures like substrings, morphemes and characters to the creation of word representations significantly increases their quality as reflected both by intrinsic metrics and performance in a wide range of downstream tasks BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 .
Question: Where do they employ feature-wise sigmoid gating?
Paper: Gating Mechanisms for Combining Character and Word-level Word Representations: An Empirical Study
We are interested in studying different ways of combining word representations, obtained from different hierarchies, into a single word representation. Specifically, we want to study how combining word representations (1) taken directly from a word embedding lookup table, and (2) obtained from a function over the characters composing them, affects the quality of the final word representations.
Question: Which model architecture do they use to obtain representations?
Paper: Gating Mechanisms for Combining Character and Word-level Word Representations: An Empirical Study
The function INLINEFORM0 is composed of an embedding layer, an optional context function, and an aggregation function.
Question: Which downstream sentence-level tasks do they evaluate on?
Paper: Gating Mechanisms for Combining Character and Word-level Word Representations: An Empirical Study
We tested three different methods for combining INLINEFORM0 with INLINEFORM1 : simple concatenation, a learned scalar gate BIBREF11 , and a learned vector gate (also referred to as feature-wise sigmoidal gate). Additionally, we compared these methods to two baselines: using pre-trained word vectors only, and using character-only features for representing words. See fig:methods for a visual description of the proposed methods.
Question: Which similarity datasets do they use?
Paper: Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction
Distantly-supervised information extraction systems extract relation tuples with a set of pre-defined relations from text. Traditionally, researchers BIBREF0, BIBREF1, BIBREF2 use pipeline approaches where a named entity recognition (NER) system is used to identify the entities in a sentence and then a classifier is used to find the relation (or no relation) between them. However, due to the complete separation of entity detection and relation classification, these models miss the interaction between multiple relation tuples present in a sentence.
Question: Are there datasets with relation tuples annotated, how big are datasets available?
Paper: Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction
A relation tuple consists of two entities and a relation. Such tuples can be found in sentences where an entity is a text span in a sentence and a relation comes from a pre-defined set $R$. These tuples may share one or both entities among them. Based on this, we divide the sentences into three classes: (i) No Entity Overlap (NEO): A sentence in this class has one or more tuples, but they do not share any entities. (ii) Entity Pair Overlap (EPO): A sentence in this class has more than one tuple, and at least two tuples share both the entities in the same or reverse order. (iii) Single Entity Overlap (SEO): A sentence in this class has more than one tuple and at least two tuples share exactly one entity. It should be noted that a sentence can belong to both EPO and SEO classes. Our task is to extract all relation tuples present in a sentence.
Question: Which one of two proposed approaches performed better in experiments?
Paper: Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction
In this task, input to the system is a sequence of words, and output is a set of relation tuples. In our first approach, we represent each tuple as entity1 ; entity2 ; relation. We use `;' as a separator token to separate the tuple components. Multiple tuples are separated using the `$\vert $' token. We have included one example of such representation in Table TABREF1. Multiple relation tuples with overlapping entities and different lengths of entities can be represented in a simple way using these special tokens (; and $\vert $). During inference, after the end of sequence generation, relation tuples can be extracted easily using these special tokens. Due to this uniform representation scheme, where entity tokens, relation tokens, and special tokens are treated similarly, we use a shared vocabulary between the encoder and decoder which includes all of these tokens. The input sentence contains clue words for every relation which can help generate the relation tokens. We use two special tokens so that the model can distinguish between the beginning of a relation tuple and the beginning of a tuple component. To extract the relation tuples from a sentence using the encoder-decoder model, the model has to generate the entity tokens, find relation clue words and map them to the relation tokens, and generate the special tokens at appropriate time. Our experiments show that the encoder-decoder models can achieve this quite effectively.
Question: What is previous work authors reffer to?
Paper: Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction
We create a single vocabulary $V$ consisting of the source sentence tokens, relation names from relation set $R$, special separator tokens (`;', `$\vert $'), start-of-target-sequence token (SOS), end-of-target-sequence token (EOS), and unknown word token (UNK). Word-level embeddings are formed by two components: (1) pre-trained word vectors (2) character embedding-based feature vectors. We use a word embedding layer $\mathbf {E}_w \in \mathbb {R}^{\vert V \vert \times d_w}$ and a character embedding layer $\mathbf {E}_c \in \mathbb {R}^{\vert A \vert \times d_c}$, where $d_w$ is the dimension of word vectors, $A$ is the character alphabet of input sentence tokens, and $d_c$ is the dimension of character embedding vectors. Following BIBREF7 (BIBREF7), we use a convolutional neural network with max-pooling to extract a feature vector of size $d_f$ for every word. Word embeddings and character embedding-based feature vectors are concatenated ($\Vert $) to obtain the representation of the input tokens.
Question: How higher are F1 scores compared to previous work?
Paper: Learning to Rank Scientific Documents from the Crowd
[block]I.1em
Question: what were the baselines?
Paper: Learning to Rank Scientific Documents from the Crowd
The number of biomedical research papers published has increased dramatically in recent years. As of October, 2016, PubMed houses over 26 million citations, with almost 1 million from the first 3 quarters of 2016 alone . It has become impossible for any one person to actually read all of the work being published. We require tools to help us determine which research articles would be most informative and related to a particular question or document. For example, a common task when reading articles is to find articles that are most related to another. Major research search engines offer such a “related articles” feature. However, we propose that instead of measuring relatedness by text-similarity measures, we build a model that is able to infer relatedness from the authors' judgments.
Question: what is the supervised model they developed?
Paper: Learning to Rank Scientific Documents from the Crowd
In order to develop and evaluate ranking algorithms we need a benchmark dataset. However, to the best of our knowledge, we know of no openly available benchmark dataset for bibliographic query-by-document systems. We therefore created such a benchmark dataset.
Question: what is the size of this built corpus?
Paper: Learning to Rank Scientific Documents from the Crowd
Learning-to-rank is a technique for reordering the results returned from a search engine query. Generally, the initial query to a search engine is concerned more with recall than precision: the goal is to obtain a subset of potentially related documents from the corpus. Then, given this set of potentially related documents, learning-to-rank algorithms reorder the documents such that the most relevant documents appear at the top of the list. This process is illustrated in Figure FIGREF6 .
Question: what crowdsourcing platform is used?
Paper: Exploiting Deep Learning for Persian Sentiment Analysis
In recent years, social media, forums, blogs and other forms of online communication tools have radically affected everyday life, especially how people express their opinions and comments. The extraction of useful information (such as people's opinion about companies brand) from the huge amount of unstructured data is vital for most companies and organizations BIBREF0 . The product reviews are important for business owners as they can take business decision accordingly to automatically classify user’s opinions towards products and services. The application of sentiment analysis is not limited to product or movie reviews but can be applied to different fields such as news, politics, sport etc. For example, in online political debates, the sentiment analysis can be used to identify people's opinions on a certain election candidate or political parties BIBREF1 BIBREF2 BIBREF3 . In this context, sentiment analysis has been widely used in different languages by using traditional and advanced machine learning techniques. However, limited research has been conducted to develop models for the Persian language.
Question: Which deep learning model performed better?
Paper: Exploiting Deep Learning for Persian Sentiment Analysis
In the literature, extensive research has been carried out to model novel sentiment analysis models using both shallow and deep learning algorithms. For example, the authors in BIBREF10 proposed a novel deep learning approach for polarity detection in product reviews. The authors addressed two major limitations of stacked denoising of autoencoders, high computational cost and the lack of scalability of high dimensional features. Their experimental results showed the effectiveness of proposed autoencoders in achieving accuracy upto 87%. Zhai et al., BIBREF11 proposed a five layers autoencoder for learning the specific representation of textual data. The autoencoders are generalised using loss function and derived discriminative loss function from label information. The experimental results showed that the model outperformed bag of words, denoising autoencoders and other traditional methods, achieving accuracy rate up to 85% . Sun et al., BIBREF12 proposed a novel method to extract contextual information from text using a convolutional autoencoder architecture. The experimental results showed that the proposed model outperformed traditional SVM and Nave Bayes models, reporting accuracy of 83.1 %, 63.9% and 67.8% respectively.
Question: By how much did the results improve?
Paper: Exploiting Deep Learning for Persian Sentiment Analysis
The novel dataset used in this work was collected manually and includes Persian movie reviews from 2014 to 2016. A subset of dataset was used to train the neural network (60% training dataset) and rest of the data (40%) was used to test and validate the performance of the trained neural network (testing set (30%), validation set (10%)). There are two types of labels in the dataset: positive or negative. The reviews were manually annotated by three native Persian speakers aged between 30 and 50 years old.
Question: What was their performance on the dataset?
Paper: Exploiting Deep Learning for Persian Sentiment Analysis
Sentiment analysis has been used extensively for a wide of range of real-world applications, ranging from product reviews, surveys feedback, to business intelligence, and operational improvements. However, the majority of research efforts are devoted to English-language only, where information of great importance is also available in other languages. In this work, we focus on developing sentiment analysis models for Persian language, specifically for Persian movie reviews. Two deep learning models (deep autoencoders and deep CNNs) are developed and compared with the the state-of-the-art shallow MLP based machine learning model. Simulations results revealed the outperformance of our proposed CNN model over autoencoders and MLP. In future, we intend to exploit more advanced deep learning models such as Long Short-Term Memory (LSTM) and LSTM-CNNs to further evaluate the performance of our developed novel Persian dataset.
Question: How large is the dataset?
Paper: Talk the Walk: Navigating New York City through Grounded Dialogue
0pt0.03.03 *
Question: Did the authors use crowdsourcing platforms?
Paper: Talk the Walk: Navigating New York City through Grounded Dialogue
As artificial intelligence plays an ever more prominent role in everyday human lives, it becomes increasingly important to enable machines to communicate via natural language—not only with humans, but also with each other. Learning algorithms for natural language understanding, such as in machine translation and reading comprehension, have progressed at an unprecedented rate in recent years, but still rely on static, large-scale, text-only datasets that lack crucial aspects of how humans understand and produce natural language. Namely, humans develop language capabilities by being embodied in an environment which they can perceive, manipulate and move around in; and by interacting with other humans. Hence, we argue that we should incorporate all three fundamental aspects of human language acquisition—perception, action and interactive communication—and develop a task and dataset to that effect.
Question: How was the dataset collected?
Paper: Talk the Walk: Navigating New York City through Grounded Dialogue
We create a perceptual environment by manually capturing several neighborhoods of New York City (NYC) with a 360 camera. Most parts of the city are grid-like and uniform, which makes it well-suited for obtaining a 2D grid. For Talk The Walk, we capture parts of Hell's Kitchen, East Village, the Financial District, Williamsburg and the Upper East Side—see Figure FIGREF66 in Appendix SECREF14 for their respective locations within NYC. For each neighborhood, we choose an approximately 5x5 grid and capture a 360 view on all four corners of each intersection, leading to a grid-size of roughly 10x10 per neighborhood.
Question: What language do the agents talk in?
Answer: English
Paper: Talk the Walk: Navigating New York City through Grounded Dialogue
For the Talk The Walk task, we randomly choose one of the five neighborhoods, and subsample a 4x4 grid (one block with four complete intersections) from the entire grid. We specify the boundaries of the grid by the top-left and bottom-right corners INLINEFORM0 . Next, we construct the overhead map of the environment, i.e. INLINEFORM1 with INLINEFORM2 and INLINEFORM3 . We subsequently sample a start location and orientation INLINEFORM4 and a target location INLINEFORM5 at random.
Question: What evaluation metrics did the authors look at?
Paper: Talk the Walk: Navigating New York City through Grounded Dialogue
We crowd-sourced the collection of the dataset on Amazon Mechanical Turk (MTurk). We use the MTurk interface of ParlAI BIBREF6 to render 360 images via WebGL and dynamically display neighborhood maps with an HTML5 canvas. Detailed task instructions, which were also given to our workers before they started their task, are shown in Appendix SECREF15 . We paired Turkers at random and let them alternate between the tourist and guide role across different HITs.
Question: What data did they use?
Paper: Real-time Claim Detection from News Articles and Retrieval of Semantically-Similar Factchecks
In recent years, the spread of misinformation has become a growing concern for researchers and the public at large BIBREF1 . Researchers at MIT found that social media users are more likely to share false information than true information BIBREF2 . Due to renewed focus on finding ways to foster healthy political conversation, the profile of factcheckers has been raised.
Question: Do the authors report results only on English data?
Paper: Real-time Claim Detection from News Articles and Retrieval of Semantically-Similar Factchecks
It is important to decide what sentences are claims before attempting to cluster them. The first such claim detection system to have been created is ClaimBuster BIBREF6 , which scores sentences with an SVM to determine how likely they are to be politically pertinent statements. Similarly, ClaimRank BIBREF7 uses real claims checked by factchecking institutions as training data in order to surface sentences that are worthy of factchecking.
Question: How is the accuracy of the system measured?
Paper: Real-time Claim Detection from News Articles and Retrieval of Semantically-Similar Factchecks
It is much easier to build a dataset and reliably evaluate a model if the starting definitions are clear and objective. Questions around what is an interesting or pertinent claim are inherently subjective. For example, it is obvious that a politician will judge their opponents' claims to be more important to factcheck than their own.
Question: How is an incoming claim used to retrieve similar factchecked claims?
Paper: Real-time Claim Detection from News Articles and Retrieval of Semantically-Similar Factchecks
In order to choose an embedding, we sought a dataset to represent our problem. Although no perfect matches exist, we decided upon the Quora duplicate question dataset BIBREF22 as the best match. To study the embeddings, we computed the euclidean distance between the two questions using various embeddings, to study the distance between semantically similar and dissimilar questions.
Question: What existing corpus is used for comparison in these experiments?
Paper: Real-time Claim Detection from News Articles and Retrieval of Semantically-Similar Factchecks
We decided to follow a methodology upon the DBScan method of clustering BIBREF24 . DBScan considers all distances between pairs of points. If they are under INLINEFORM0 then those two are linked. Once the number of connected points exceeds a minimum size threshold, they are considered a cluster and all other points are considered to be unclustered. This method is advantageous for our purposes because unlike other methods, such as K-Means, it does not require the number of clusters to be specified. To create a system that can build clusters dynamically, adding one point at a time, we set the minimum cluster size to one, meaning that every point is a member of a cluster.
Question: What are the components in the factchecking algorithm?
Paper: RC-QED: Evaluating Natural Language Derivations in Multi-Hop Reading Comprehension
Reading comprehension (RC) has become a key benchmark for natural language understanding (NLU) systems and a large number of datasets are now available BIBREF0, BIBREF1, BIBREF2. However, these datasets suffer from annotation artifacts and other biases, which allow systems to “cheat”: Instead of learning to read texts, systems learn to exploit these biases and find answers via simple heuristics, such as looking for an entity with a matching semantic type BIBREF3, BIBREF4. To give another example, many RC datasets contain a large number of “easy” problems that can be solved by looking at the first few words of the question Sugawara2018. In order to provide a reliable measure of progress, an RC dataset thus needs to be robust to such simple heuristics.
Question: What is the baseline?
Paper: RC-QED: Evaluating Natural Language Derivations in Multi-Hop Reading Comprehension
We formally define RC-QED as follows:
Question: What dataset was used in the experiment?
Paper: RC-QED: Evaluating Natural Language Derivations in Multi-Hop Reading Comprehension
This paper instantiates RC-QED by employing multiple choice, entity-based multi-hop QA BIBREF0 as a testbed (henceforth, RC-QED$^{\rm E}$). In entity-based multi-hop QA, machines need to combine relational facts between entities to derive an answer. For example, in Figure FIGREF1, understanding the facts about Barracuda, Little Queen, and Portrait Records stated in each article is required. This design choice restricts a problem domain, but it provides interesting challenges as discussed in Section SECREF46. In addition, such entity-based chaining is known to account for the majority of reasoning types required for multi-hop reasoning BIBREF2.
Question: Did they use any crowdsourcing platform?
Paper: RC-QED: Evaluating Natural Language Derivations in Multi-Hop Reading Comprehension
To acquire a large-scale corpus of NLDs, we use crowdsourcing (CS). Although CS is a powerful tool for large-scale dataset creation BIBREF2, BIBREF8, quality control for complex tasks is still challenging. We thus carefully design an incentive structure for crowdworkers, following Yang2018HotpotQA:Answering.
Question: How was the dataset annotated?
Paper: RC-QED: Evaluating Natural Language Derivations in Multi-Hop Reading Comprehension
Given a statement and articles, workers are asked to judge whether the statement can be derived from the articles at three grades: True, Likely (i.e. Answerable), or Unsure (i.e. Unanswerable). If a worker selects Unsure, we ask workers to tell us why they are unsure from two choices (“Not stated in the article” or “Other”).
Question: What is the source of the proposed dataset?