z-uo/qasper-squad huggingface dataset:
Paper: Doc2Vec on the PubMed corpus: study of a new approach to generate related articles
To do so, each document is tokenized into many topics $S_{i}$. Then, the probability $P(C|D)$ that the user will find relevant the document C when reading the document D will be calculated. For this purpose, the authors brought the concept of eliteness. Briefly, a topic $S_{i}$ is presented as elite topic for a given document if a word $W_{i}$ representing $S_{i}$ is used with a high frequency in this document. This work allows to bring closer documents sharing a maximum of elite topics. In the article presenting the pmra model, authors claim that “the deployed algorithm in PubMed also takes advantage of MeSH terms, which we do not discuss here”. We can thus assume that a similar score is computed thanks to the associated MeSH terms with both documents D and C. Such an indexing is highly time-consuming and has to be manually performed.
Question: What Doc2Vec architectures other than PV-DBOW have been tried?
Paper: Doc2Vec on the PubMed corpus: study of a new approach to generate related articles
Nowadays, embedding models allow to represent a text into a vector of fixed dimensions. The primary purpose of this mathematical representation of documents was to be able to use texts as input of deep neural networks. However, these models have been used by the IR community as well: once all fitted in the same multidimensional space, the cosine distance between two documents vectors can estimate the proximity between these two texts. In 2013, Mikolov et al. released a word embedding method called Word2Vec (W2V) BIBREF2. Briefly, this algorithm uses unsupervised learning to train a model which embeds a word as a vector while preserving its semantic meaning. Following this work, Mikolov and Le released in 2014 a method to vectorize complete texts BIBREF3. This algorithm, called Doc2Vec (D2V), is highly similar to W2V and comes with two architectures. The Distributed Memory Model of Paragraph Vectors (PV-DM) first trains a W2V model. This word embedding will be common for all texts from a given corpus C on which it was trained. Then, each document $D_{x}$ from C will be assigned to a randomly initialised vector of fixed length, which will be concatenated with vectors of words composing $D_{x}$ during the training time (words and documents vectors are sharing the same number of dimensions). This concatenation will be used by a final classifier to predict the next token of a randomly selected window of words. The accuracy of this task can be calculated and used to compute a loss function, used to back-propagate errors to the model, which leads to a modification of the document’s representation. The Distributed Bag of Words version of Paragraph Vector (PV-DBOW) is highly similar to the PV-DM, the main difference being the goal of the final classifier. Instead of concatenating vector from the document with word vectors, the goal here is to output words from this window just by using the mathematical representation of the document.
Question: What four evaluation tasks are defined to determine what influences proximity?
Paper: Doc2Vec on the PubMed corpus: study of a new approach to generate related articles
Doc2Vec has been used for many cases of similar document retrieval. In 2016, Lee et al. used D2V to clusterize positive and negative sentiments with an accuracy of 76.4% BIBREF4. The same year, Lau and Baldwin showed that D2V provides a robust representation of documents, estimated with two tasks: document similarity to retrieve 12 different classes and sentences similarity scoring BIBREF5. Recently, studies started to use documents embedding on the PubMed corpus. In 2017, Gargiulo et al. used a combination of words vectors coming from the abstract to bring closer similar documents from Pubmed BIBREF6. Same year, Wang and Koopman used the PubMed database to compare D2V and their own document embedding method BIBREF7. Their designed accuracy measurement task was consisting in retrieving documents having a small cosine distance with the embedding of a query. Recently, Chen et al. released BioSentVec, a set of sentence vectors created from PubMed with the algorithm sent2vec BIBREF8, BIBREF9. However, their evaluation task was based on public sentences similarity datasets, when the goal here is to embed entire abstracts as vectors and to use them to search for similar articles versus the pmra model. In 2008, the related articles feature of PubMed has been compared (using a manual evaluation) with one that uses both a TF-IDF BIBREF10 representation of the documents and Lin’s distance BIBREF11 to compare their MeSH terms BIBREF12. Thus, no study was designed so far to compare documents embedding and the pmra algorithm. The objectives of this study were to measure the ability of these two models to infer the similarity between documents from PubMed and to search what impacts the most this proximity. To do so, different evaluation tasks were defined to cover a wide range of aspects of document analogy, from their context to their morphological similarities.
Question: What six parameters were optimized with grid search?
Paper: Multi-Perspective Fusion Network for Commonsense Reading Comprehension
Content: Task Definition
Question: What baseline models do they compare against?
Answer: SLQA, Rusalka, HMA Model (single), TriAN (single), jiangnan (ensemble), MITRE (ensemble), TriAN (ensemble), HMA Model (ensemble)
Paper: Identifying Clickbait: A Multi-Strategy Approach Using Neural Networks
The Internet provides instant access to a wide variety of online content, news included. Formerly, users had static preferences, gravitating towards their trusted sources, incurring an unwavering sense of loyalty. The same cannot be said for current trends since users are likely to go with any source readily available to them.
Question: What are the differences with previous applications of neural networks for this task?
Answer: This approach considers related images
Paper: Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
Neural sequence models trained with maximum likelihood estimation (MLE) have become a standard approach to modeling sequences in a variety of natural language applications such as machine translation BIBREF0, dialogue modeling BIBREF1, and language modeling BIBREF2. Despite this success, MLE-trained neural sequence models have been shown to exhibit issues such as length bias BIBREF3, BIBREF4 and degenerate repetition BIBREF5. These issues are suspected to be related to the maximum likelihood objective's local normalization, which results in a discrepancy between the learned model's distribution and the distribution induced by the decoding algorithm used to generate sequences BIBREF6, BIBREF7. This has prompted the development of alternative decoding methods BIBREF8, BIBREF5 and training objectives BIBREF9, BIBREF10. In this paper, we formalize and study this discrepancy between the model and the decoding algorithm.
Question: How much improvement is gained from the proposed approaches?
Answer: It eliminates non-termination in some models fixing for some models up to 6% of non-termination ratio.
Paper: Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
We begin our discussion by establishing background definitions. First, we define a sequence which is the main object of our investigation.
Question: Is the problem of determining whether a given model would generate an infinite sequence is a decidable problem?
Paper: Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
A recurrent language model is an autoregressive model of a sequence distribution, where each conditional probability is parameterized with a neural network. Importantly, we assume that all tokens in a sequence are dependent on each other under a recurrent language model. This allows us to avoid cases in which the model degenerates to a Markovian language model, such as an $n$-gram model with a finite $n$.
Question: Is infinite-length sequence generation a result of training with maximum likelihood?
Answer: There are is a strong conjecture that it might be the reason but it is not proven.
Paper: Modality-Balanced Models for Visual Dialogue
When we pursue conversations, context is important to keep the topic consistent or to answer questions which are asked by others, since most new utterances are made conditioned on related mentions or topic clues in the previous utterances in the conversation history. However, conversation history is not necessarily needed for all interactions, for instance, someone can change topics during a conversation and can ask a sudden new question which is not related to the context. This is similar to the setup in the Visual Dialog task BIBREF0, in which one agent (say the `asker') keeps asking questions and the other one (say the `answerer') keeps answering the questions based on an image for multiple rounds. The asker can ask a question from the conversation context. Then the answerer should answer the question by considering the conversation history as well as the image information, e.g., if the asker asks a question, “Are they in pots?” (Q4 in Fig. FIGREF1), the answerer should find a clue in the past question-answer pairs “Is there a lot of plants?” - “I only see 2.” (Q3-A3 in Fig. FIGREF1) and figure out what `they' means first to answer the question correctly. On the other hand, some questions in this task are independent of the past conversation history, e.g., “Can you see a building?” (Q8 in Fig. FIGREF1), where the answerer does not need to look at conversation context and can answer the question only based on the image information.
Question: What metrics are used in challenge?
Paper: Modality-Balanced Models for Visual Dialogue
Visual question answering is a task in which a machine is asked to answer a question about an image. The recent success of deep neural networks and massive data collection BIBREF2 has made the field more active. One of the most challenging parts of the task is to ground the meaning of text on visual evidence. Co-attention BIBREF3 is proposed to integrate information from different modalities (i.e., image and language) and more advanced approaches have shown good performance BIBREF4, BIBREF5, BIBREF6. A bilinear approach has also been proposed to replace simple addition or concatenation approaches for fusing the two modalities BIBREF7, BIBREF8, BIBREF9, BIBREF10. In our work, we employ multi-modal factorized bilinear pooling (MFB) BIBREF11 to fuse a question and image-history features.
Question: What model was winner of the Visual Dialog challenge 2019?
Paper: Modality-Balanced Models for Visual Dialogue
The Visual Dialog task BIBREF0 can be seen as an extended version of the VQA task, with multiple rounds of sequential question-answer pairs as dialog history, including an image caption, which should be referred to before answering a given question. This conversation history can help a model better predict correct answers by giving direct or indirect clues for the answers, or proper context for co-reference resolution. However, having conversation history also means that a model should extract relevant information from the history and introduces another challenge to the task. Many approaches have been proposed to handle this challenge. BIBREF12 tries to extract the clues from history recursively while BIBREF13 and BIBREF14 employ co-attention to fuse visual, history, and question features. In our work, we employ BIBREF15's approach to fuse visual and history features before they are attended by a question. Our joint model with fused features has much information from history and we find that it is in complementary relation with our image-only model. Thus, we combine the two models to take the most appropriate information from each model to answer questions.
Question: What model was winner of the Visual Dialog challenge 2018?
Paper: Modality-Balanced Models for Visual Dialogue
In the Visual Dialog task BIBREF0, two agents interact via natural language with respect to an image. The asker keeps asking about the image given an image caption without seeing the image. The other agent (i.e., answerer) keeps answering the questions by viewing the image. They conduct multiple rounds of conversation accumulating question-answer pairs which are called `history' (Figure FIGREF1). The full history $\textrm {HISTORY}$ consists of question-answer pairs as well as an image caption which describes the given image, such that at a current time point $t$, the previous history is $\textrm {HISTORY}_t = \lbrace C, (Q_{1},A_{1}), (Q_{2},A_{2}), ..., (Q_{t-1},A_{t-1}) \rbrace $, where $C$ is the image caption and $Q_{t-1}$ and $A_{t-1}$ are the question and answer at round $t-1$, respectively. Then, given a new current time-stamp question $Q_t$, the history $\textrm {HISTORY}_t$, and the image, the model has to rank 100 candidate answers from the answerer's perspective.
Question: Which method for integration peforms better ensemble or consensus dropout fusion with shared parameters?
Paper: Modality-Balanced Models for Visual Dialogue
Visual Features: For visual features, we use object features which are extracted from an image by using Faster R-CNN BIBREF16. The visual feature, $V_{rcnn} \in \mathbb {R}^{k \times d_{v}}$, is a matrix whose rows correspond to objects, where $k$ is the number of objects (k=36 in our experiment), $d_{v}$ is dimension size of visual feature ($d_{v}$ = 2048 for ResNet backbone).
Question: How big is dataset for this challenge?
Answer: 133,287 images
Paper: Logician: A Unified End-to-End Neural Approach for Open-Domain Information Extraction
Semantic applications typically work on the basis of intermediate structures derived from sentences. Traditional word-level intermediate structures, such as POS-tags, dependency trees and semantic role labels, have been widely applied. Recently, entity and relation level intermediate structures attract increasingly more attentions.
Question: What open relation extraction tasks did they experiment on?
Paper: Logician: A Unified End-to-End Neural Approach for Open-Domain Information Extraction
When reading a sentence in natural language, humans are able to recognize the facts involved in the sentence and accurately express them. In this paper, Symbolic Aided Open Knowledge Expression (SAOKE) is proposed as the form for honestly recording these facts. SAOKE expresses the primary information of sentences in n-ary tuples $(subject,predicate,object_{1},\cdots ,object_{N})$ , and (in this paper) neglects some auxiliary information. In the design of SAOKE, we take four requirements into consideration: completeness, accurateness, atomicity and compactness.
Question: How is Logician different from traditional seq2seq models?
Paper: Logician: A Unified End-to-End Neural Approach for Open-Domain Information Extraction
After having analyzed a large number of sentences, we observe that the majority of facts can be classified into the following classes:
Question: What's the size of the previous largest OpenIE dataset?
Paper: RTFM: Generalising to Novel Environment Dynamics via Reading
Reinforcement learning (RL) has been successful in a variety of areas such as continuous control BIBREF0, dialogue systems BIBREF1, and game-playing BIBREF2. However, RL adoption in real-world problems is limited due to poor sample efficiency and failure to generalise to environments even slightly different from those seen during training. We explore language-conditioned policy learning, where agents use machine reading to discover strategies required to solve a task, thereby leveraging language as a means to generalise to new environments.
Question: How is data for RTFM collected?
Paper: RTFM: Generalising to Novel Environment Dynamics via Reading
A growing body of research is learning policies that follow imperative instructions. The granularity of instructions vary from high-level instructions for application control BIBREF11 and games BIBREF5, BIBREF6 to step-by-step navigation BIBREF7. In contrast to learning policies for imperative instructions, BIBREF4, BIBREF9 infer a policy for a fixed goal using features extracted from high level strategy descriptions and general information about domain dynamics. Unlike prior work, we study the combination of imperative instructions and descriptions of dynamics. Furthermore, we require that the agent learn to filter out irrelevant information to focus on dynamics relevant to accomplishing the goal.
Question: How better is performance of proposed model compared to baselines?
Answer: Proposed model achive 66+-22 win rate, baseline CNN 13+-1  and baseline FiLM 32+-3 .
Paper: RTFM: Generalising to Novel Environment Dynamics via Reading
Language grounding refers to interpreting language in a non-linguistic context. Examples of such context include images BIBREF12, games BIBREF13, BIBREF14, robot control BIBREF15, BIBREF16, and navigation BIBREF17. We study language grounding in interactive games similar to BIBREF11, BIBREF5 or BIBREF8, where executable semantics are not provided and the agent must learn through experience. Unlike prior work, we require grounding between an underspecified goal, a document of environment dynamics, and world observations. In addition, we focus on generalisation to not only new goal descriptions but new environments dynamics.
Question: How does propose model model that capture three-way interactions?
Paper: ISS-MULT: Intelligent Sample Selection for Multi-Task Learning in Question Answering
Transferring knowledge between related domains could help to improve a learner on a special domain. Moreover, gathering data from different related resources proves very time-consuming and expensive. Therefore, this necessitates the development of machine learning methods to extract knowledge from these resources with different properties and characteristics. Transfer learning is a field in which machine learning methods encounter this challenge.
Question: Do transferring hurt the performance is the corpora are not related?
Paper: ISS-MULT: Intelligent Sample Selection for Multi-Task Learning in Question Answering
Transfer learning has a long history, and many researchers try to utilize the knowledge from relevant datasets to improve the performance on the target dataset [9, 7]. Moreover, transfer learning has been well-studied for deep learning methods in computer vision. Similarity between datasets is a key factor for performing transfer learning. Based on the similarity between two domains, different methods could be used. Bengio et. al [11] examine the ability of the neural networks in transferring knowledge from different domain.
Question: Is accuracy the only metric they used to compare systems?
Paper: ISS-MULT: Intelligent Sample Selection for Multi-Task Learning in Question Answering
Five different datasets (SQuAD, SelQA , WikiQA, WikiQA and InforbaxQA) are used for evaluation of the INIT, MULT and ISS-MULT methods. These datasets were proposed in recent years for the question answering problems. These datasets are produced differently. Therefore, they are may not be semanticaly related datasets, and this feature plays an important role in transfer learning in NLP tasks.
Question: How do they transfer the model?
Paper: Improving Span-based Question Answering Systems with Coarsely Labeled Data
Question answering (QA) systems can provide most value for users by showing them a fine-grained short answer (answer span) in a context that supports the answer (paragraph in a document). However, fine-grained short answer annotations for question answering are costly to obtain, whereas non-expert annotators can annotate coarse-grained passages or documents faster and with higher accuracy. In addition, coarse-grained annotations are often freely available from community forums such as Quora. Therefore, methods that can learn to select short answers based on more abundant coarsely annotated paragraph-level data can potentially bring significant improvements. As an example of the two types of annotation, Figure 1 shows on the left a question with corresponding short answer annotation (underlined short answer) in a document, and on the right a question with a document annotated at the coarse-grained paragraph relevance level. In this work we study methods for learning short answer models from small amounts of data annotated at the short answer level and larger amounts of data annotated at the paragraph level. min-seo-hajishirzi:2017:Short recently studied a related problem of transferring knowledge from a fine-grained QA model to a coarse-grained model via multi-task learning and showed that finely annotated data can help improve performance on the coarse-grained task. We investigate the opposite and arguably much more challenging direction: improving fine-grained models using coarse-grained data.
Question: Will these findings be robust through different datasets and different question answering algorithms?
Paper: Improving Span-based Question Answering Systems with Coarsely Labeled Data
The fine-grained short question answering task asks to select an answer span in a document containing multiple paragraphs. In the left example in Figure 1, the short answer to the question What was Nikola Tesla's ethnicity? is the phrase Serbian in the first paragraph in the document.
Question: What is the underlying question answering algorithm?
Paper: Improving Span-based Question Answering Systems with Coarsely Labeled Data
We define the fine-grained task of interest $T_y$ as predicting outputs $y$ from a set of possible outputs ${\cal {Y}}(x)$ given inputs $x$ . We say that a task $T_z$ to predict outputs $z$ given inputs $x$ is a coarse-grained counterpart of $T_y$ , iff each coarse label $z$ determines a sub-set of possible labels ${\cal {Y}}(z,x) \subset {\cal {Y}}(x)$ , and each fine label $y$0 has a deterministically corresponding single coarse label $y$1 . We refer to the fine-grained and coarse-grained training data as $y$2 and $y$3 respectively.
Question: What datasets have this method been evaluated on?
Paper: AandP: Utilizing Prolog for converting between active sentence and passive sentence with three-steps conversion
Language plays a vital role in the human life. A language is a structured system of communication BIBREF2. There are various language systems in the world with the estimated number being between 5,000 and 7,000 BIBREF3. Natural Language Processing (NLP) which we commonly hear is a subfield of linguistics. NLP aims to provide interactions between computers and human languages. The performance of NLP is evaluated by how computers can process and analyze large amounts of natural language data BIBREF4. In terms of language processing, we cannot but mention Computational Linguistics BIBREF5. Computational Linguistics is the scientific study of language from a computational perspective, and thus an interdisciplinary field, involving linguistics, computer science, mathematics, logic, cognitive science, and cognitive psychology.
Question: Is there a machine learning approach that tries to solve same problem?
Paper: AandP: Utilizing Prolog for converting between active sentence and passive sentence with three-steps conversion
The main challenge of this work is how much it can handle cases. There are a variety of cases in terms of active sentence and passive sentence. The cases that I solved in this work are shown as follows.
Question: What DCGs are used?
Answer: Author's own DCG rules are defined from scratch.
Paper: AandP: Utilizing Prolog for converting between active sentence and passive sentence with three-steps conversion
The objective of this work is sentences: active sentence and passive sentence, so I need to determine the representation of both active sentence and passive sentence.
Question: What else is tried to be solved other than 12 tenses, model verbs and negative form?
Paper: AandP: Utilizing Prolog for converting between active sentence and passive sentence with three-steps conversion
User interacts with the program by posing the query with the form (Figure FIGREF56):
Question: What is used for evaluation of this approach?