z-uo/qasper-squad huggingface dataset:
Paper: Irony Detection in a Multilingual Context
This paper proposes the first multilingual ID in tweets. We show that simple monolingual architectures (either neural or feature-based) trained separately on each language can be successfully used in a multilingual setting providing a cross-lingual word representation or basic surface features. Our monolingual results are comparable to state of the art for the three languages. The CNN architecture trained on cross-lingual word representation shows that irony has a certain similarity between the languages we targeted despite the cultural differences which confirm that irony is a universal phenomena, as already shown in previous linguistic studies BIBREF39, BIBREF25, BIBREF40. The manual analysis of the common misclassified tweets across the languages in the multilingual setup, shows that classification errors are due to three main factors. (1) First, the absence of context where writers did not provide sufficient information to capture the ironic sense even in the monolingual setting, as in نبدا تاني يسقط يسقط حسني مبارك !! > (Let's start again, get off get off Mubarak!!) where the writer mocks the Egyptian revolution, as the actual president "Sisi" is viewed as Mubarak's fellows. (2) Second, the presence of out of vocabulary (OOV) terms because of the weak coverage of the mutlilingual embeddings which make the system fails to generalize when the OOV set of unseen words is large during the training process. We found tweets in all the three languages written in a very informal way, where some characters of the words were deleted, duplicated or written phonetically (e.g phat instead of fat). (3) Another important issue is the difficulty to deal with the Arabic language. Arabic tweets are often characterized by non-diacritised texts, a large variations of unstandardized dialectal Arabic (recall that our dataset has 4 main varieties, namely Egypt, Gulf, Levantine, and Maghrebi), presence of transliterated words (e.g. the word table becomes طابلة> (tabla)), and finally linguistic code switching between Modern Standard Arabic and several dialects, and between Arabic and other languages like English and French. We found some tweets contain only words from one of the varieties and most of these words do not exist in the Arabic embeddings model. For example in مبارك بقاله كام يوم مامتش .. هو عيان ولاه ايه #مصر > (Since many days Mubarak didn't die .. is he sick or what? #Egypt), only the words يوم> (day), مبارك> (Mubarak), and هو> (he) exist in the embeddings. Clearly, considering only these three available words, we are not able to understand the context or the ironic meaning of the tweet. To conclude, our multilingual experiments confirmed that the door is open towards multilingual approaches for ID. Furthermore, our results showed that ID can be applied to languages that lack of annotated data. Our next step is to experiment with other languages such as Hindi and Italian.
Question: What monolingual word representations are used?
Answer: AraVec for Arabic, FastText for French, and Word2vec Google News for English.
Paper: Deep Health Care Text Classification
With the expansion of micro blogging platforms such as Twitter, the Internet is progressively being utilized to spread health information instead of similarly as a wellspring of data BIBREF0 , BIBREF1 . Twitter allows users to share their status messages typically called as tweets, restricted to 140 characters. Most of the time, these tweets expresses the opinions about the topics. Thus analysis of tweets has been considered as a significant task in many of the applications, here for health related applications.
Question: Does the proposed method outperform a baseline?
Paper: Deep Health Care Text Classification
This section discusses the concepts of tweet representation and deep learning algorithms particularly recurrent neural network (RNN) and long short-term memory (LSTM) in a mathematical way.
Question: What type of RNN is used?
Paper: A Novel ILP Framework for Summarizing Content with High Lexical Variety
Summarization is a promising technique for reducing information overload. It aims at converting long text documents to short, concise summaries conveying the essential content of the source documents BIBREF0 . Extractive methods focus on selecting important sentences from the source and concatenating them to form a summary, whereas abstractive methods can involve a number of high-level text operations such as word reordering, paraphrasing, and generalization BIBREF1 . To date, summarization has been successfully exploited for a number of text domains, including news articles BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , product reviews BIBREF6 , online forum threads BIBREF7 , meeting transcripts BIBREF8 , scientific articles BIBREF9 , BIBREF10 , student course responses BIBREF11 , BIBREF12 , and many others.
Question: What do they constrain using integer linear programming?
Paper: A Novel ILP Framework for Summarizing Content with High Lexical Variety
Extractive summarization has undergone great development over the past decades. It focuses on extracting relevant sentences from a single document or a cluster of documents related to a particular topic. Various techniques have been explored, including maximal marginal relevance BIBREF22 , submodularity BIBREF23 , integer linear programming BIBREF24 , BIBREF25 , BIBREF26 , BIBREF27 , BIBREF4 , minimizing reconstruction error BIBREF28 , graph-based models BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , determinantal point processes BIBREF33 , neural networks and reinforcement learning BIBREF34 , BIBREF35 among others. Nonetheless, most studies are bound to a single dataset and few approaches have been evaluated in a cross-domain setting. In this paper, we propose an enhanced ILP framework and evaluate it on a broad range of datasets. We present an in-depth analysis of the dataset characteristics derived from both source documents and reference summaries to understand how domain-specific factors may affect the applicability of the proposed approach.
Question: Do they build one model per topic or on all topics?
Answer: One model per topic.
Paper: A Novel ILP Framework for Summarizing Content with High Lexical Variety
Let INLINEFORM0 be a set of documents that consist of INLINEFORM1 sentences in total. Let INLINEFORM2 , INLINEFORM3 indicate if a sentence INLINEFORM4 is selected ( INLINEFORM5 ) or not ( INLINEFORM6 ) in the summary. Similarly, let INLINEFORM7 be the number of unique concepts in INLINEFORM8 . INLINEFORM9 , INLINEFORM10 indicate the appearance of concepts in the summary. Each concept INLINEFORM11 is assigned a weight of INLINEFORM12 , often measured by the number of sentences or documents that contain the concept. The ILP-based summarization approach BIBREF20 searches for an optimal assignment to the sentence and concept variables so that the selected summary sentences maximize coverage of important concepts. The relationship between concepts and sentences is captured by a co-occurrence matrix INLINEFORM13 , where INLINEFORM14 indicates the INLINEFORM15 -th concept appears in the INLINEFORM16 -th sentence, and INLINEFORM17 otherwise. In the literature, bigrams are frequently used as a surrogate for concepts BIBREF24 , BIBREF21 . We follow the convention and use `concept' and `bigram' interchangeably in this paper.
Question: Do they quantitavely or qualitatively evalute the output of their low-rank approximation to verify the grouping of lexical items?
Answer: They evaluate quantitatively.
Paper: A Novel ILP Framework for Summarizing Content with High Lexical Variety
Because of the lexical diversity problem, we suspect the co-occurrence matrix INLINEFORM0 may not establish a faithful correspondence between sentences and concepts. A concept may be conveyed using multiple bigram expressions; however, the current co-occurrence matrix only captures a binary relationship between sentences and bigrams. For example, we ought to give partial credit to “bicycle parts” given that a similar expression “bike elements” appears in the sentence. Domain-specific synonyms may be captured as well. For example, the sentence “I tried to follow along but I couldn't grasp the concepts” is expected to partially contain the concept “understand the”, although the latter did not appear in the sentence.
Question: Do they evaluate their framework on content of low lexical variety?
Paper: Word Embeddings to Enhance Twitter Gang Member Profile Identification
Street gangs are defined as “a coalition of peers, united by mutual interests, with identifiable leadership and internal organization, who act collectively to conduct illegal activity and to control a territory, facility, or enterprise” BIBREF0 . They promote criminal activities such as drug trafficking, assault, robbery, and threatening or intimidating a neighborhood BIBREF1 . Today, over 1.4 million people, belonging to more than 33,000 gangs, are active in the United States BIBREF2 , of which 88% identify themselves as being members of a street gang. They are also active users of social media BIBREF2 ; according to 2007 National Assessment Center's survey of gang members, 25% of individuals in gangs use the Internet for at least 4 hours a week BIBREF3 . More recent studies report approximately 45% of gang members participate in online offending activities such as threatening, harassing individuals, posting violent videos or attacking someone on the street for something they said online BIBREF4 , BIBREF5 . They confirm that gang members use social media to express themselves in ways similar to their offline behavior on the streets BIBREF6 .
Question: Do the authors report on English datasets only?
Paper: Word Embeddings to Enhance Twitter Gang Member Profile Identification
Researchers have begun investigating the gang members' use of social media and have noticed the importance of identifying gang members' Twitter profiles a priori BIBREF6 , BIBREF7 . Before analyzing any textual context retrieved from their social media posts, knowing that a post has originated from a gang member could help systems to better understand the message conveyed by that post. Wijeratne et al. developed a framework to analyze what gang members post on social media BIBREF7 . Their framework could only extract social media posts from self identified gang members by searching for pre-identified gang names in a user's Twitter profile description. Patton et al. developed a method to collect tweets from a group of gang members operating in Detroit, MI BIBREF11 . However, their approach required the gang members' Twitter profile names to be known beforehand, and data collection was localized to a single city in the country. These studies investigated a small set of manually curated gang member profiles, often from a small geographic area that may bias their findings.
Question: Which supervised learning algorithms are used in the experiments?
Paper: Word Embeddings to Enhance Twitter Gang Member Profile Identification
A word embedding model is a neural network that learns rich representations of words in a text corpus. It takes data from a large, INLINEFORM0 -dimensional `word space' (where INLINEFORM1 is the number of unique words in a corpus) and learns a transformation of the data into a lower INLINEFORM2 -dimensional space of real numbers. This transformation is developed in a way that similarities between the INLINEFORM3 -dimensional vector representation of two words reflects semantic relationships among the words themselves. These semantics are not captured by typical bag-of-words or INLINEFORM4 -gram models for classification tasks on text data BIBREF14 , BIBREF10 .
Question: How in YouTube content translated into a vector format?
Paper: Word Embeddings to Enhance Twitter Gang Member Profile Identification
Gang member tweets and profile descriptions tend to have few textual indicators that demonstrate their gang affiliations or their tweets/profile text may carry acronyms which can only be deciphered by others involved in gang culture BIBREF9 . These gang-related terms are often local to gangs operating in neighborhoods and change rapidly when they form new gangs. Consequently, building a database of keywords, phrases, and other identifiers to find gang members nationally is not feasible. Instead, we use heterogeneous sets of features derived not only from profile and tweet text but also from the emoji usage, profile images, and links to YouTube videos reflecting their music preferences and affinity. In this section, we briefly discuss the feature types and their broad differences in gang and non-gang member profiles. An in-depth explanation of these feature selection can be found in BIBREF9 .
Question: How is the ground truth of gang membership established in this dataset?
Paper: A Byte-sized Approach to Named Entity Recognition
While NER tasks across domains share similar problems of ambiguous abbreviations, homonyms, and other entity variations, the domain of biomedical text poses some unique challenges. While, in principle, there is a known set of biomedical entities (e.g., all known proteins), there is a surprising amount of variation for any given entity. For example, PPCA, C4 PEPC, C4 PEPCase, and Photosynthetic PEPCase all refer to the same entity. Additionally, certain entities such as proteins and genes can naturally span less than a “word" (e.g., HA and APG12 are separate proteins in pHA-APG12). Most state-of-the-art NER methods tag entities at the “word" level, and rely on pre- or post-processing rules to extract subword entities. Our goal is to develop a subword approach that does not rely on ad hoc processing steps.
Question: Do they evaluate ablated versions of their CNN+RNN model?
Paper: The Intelligent Voice 2016 Speaker Recognition System
Compared to previous years, the 2016 NIST speaker recognition evaluation (SRE) marked a major shift from English towards Austronesian and Chinese languages. The task like previous years is to perform speaker detection with the focus on telephone speech data recorded over a variety of handset types. The main challenges introduced in this evaluation are duration and language variability. The potential variation of languages addressed in this evaluation, recording environment, and variability of test segments duration influenced the design of our system. Our goal was to utilize recent advances in language normalization, domain adaptation, speech activity detection and session compensation techniques to mitigate the adverse bias introduced in this year's evaluation.
Question: Do they single out a validation set from the fixed SRE training set?
Paper: The Intelligent Voice 2016 Speaker Recognition System
The fixed training condition is used to build our speaker recognition system. Only conversational telephone speech data from datasets released through the linguistic data consortium (LDC) have been used, including NIST SRE 2004-2010 and the Switchboard corpora (Switchboard Cellular Parts I and II, Switchboard2 Phase I,II and III) for different steps of system training. A more detailed description of the data used in the system training is presented in Table TABREF1 . We have also included the unlabelled set of 2472 telephone calls from both minor (Cebuano and Mandarin) and major (Tagalog and Cantonese) languages provided by NIST in the system training. We will indicate when and how we used this set in the training in the following sections.
Question: How well does their system perform on the development set of SRE?
Answer: EER 16.04, Cmindet 0.6012, Cdet 0.6107
Paper: The Intelligent Voice 2016 Speaker Recognition System
In this section we will provide a description of the main steps in front-end processing of our speaker recognition system including speech activity detection, acoustic and i-vector feature extraction.
Question: Which are the novel languages on which SRE placed emphasis on?
Paper: Synonym Discovery with Etymology-based Word Embeddings
Word embedding is a very active area of research. It consists of using a text corpus to characterize and embed words into rich high-dimensional vector spaces. By mining a text corpus, it is possible to embed words in a continuous space where semantically similar words are embedded close together. By encoding words into vectors, it is possible to represent semantic properties of these words in a way that is more expressive and useful for tasks of natural language processing. Word embeddings have been effectively used for sentiment analysis, machine translation, and other and other language-related tasks BIBREF0 , BIBREF1 .
Question: Does this approach perform better than context-based word embeddings?
Paper: Synonym Discovery with Etymology-based Word Embeddings
There exists limited research on etymological networks in the English language. Particularly BIBREF7 , and BIBREF8 use an etymological network-based approach to study movie scripts and reviews in English.
Question: Have the authors tried this approach on other languages?
Paper: Luminoso at SemEval-2018 Task 10: Distinguishing Attributes Using Text Corpora and Relational Knowledge
Word embeddings are most effective when they learn from both unstructured text and a graph of general knowledge BIBREF0 . ConceptNet 5 BIBREF1 is an open-data knowledge graph that is well suited for this purpose. It is accompanied by a pre-built word embedding model known as ConceptNet Numberbatch, which combines skip-gram embeddings learned from unstructured text with the relational knowledge in ConceptNet.
Question: What features did they train on?
Paper: Coarse-grain Fine-grain Coattention Network for Multi-evidence Question Answering
A requirement of scalable and practical question answering (QA) systems is the ability to reason over multiple documents and combine their information to answer questions. Although existing datasets enabled the development of effective end-to-end neural question answering systems, they tend to focus on reasoning over localized sections of a single document BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . For example, BIBREF4 find that 90% of the questions in the Stanford Question Answering Dataset are answerable given 1 sentence in a document. In this work, we instead focus on multi-evidence QA, in which answering the question requires aggregating evidence from multiple documents BIBREF5 , BIBREF6 .
Question: How big is the test set?
Paper: Coarse-grain Fine-grain Coattention Network for Multi-evidence Question Answering
The coarse-grain module and fine-grain module of the correspond to and strategies. The coarse-grain module summarizes support documents without knowing the candidates: it builds codependent representations of support documents and the query using coattention, then produces a coarse-grain summary using self-attention. In contrast, the fine-grain module retrieves specific contexts in which each candidate occurs: it identifies coreferent mentions of the candidate, then uses coattention to build codependent representations between these mentions and the query. While low-level encodings of the inputs are shared between modules, we show that this division of labour allows the attention hierarchies in each module to focus on different parts of the input. This enables the model to more effectively represent a large number of potentially long support documents.
Question: What is coattention?
Paper: Unsupervised Question Decomposition for Question Answering
Question answering (QA) systems have become remarkably good at answering simple, single-hop questions but still struggle with compositional, multi-hop questions BIBREF0, BIBREF1. In this work, we examine if we can answer hard questions by leveraging our ability to answer simple questions. Specifically, we approach QA by breaking a hard question into a series of sub-questions that can be answered by a simple, single-hop QA system. The system's answers can then be given as input to a downstream QA system to answer the hard question, as shown in Fig. FIGREF1. Our approach thus answers the hard question in multiple, smaller steps, which can be easier than answering the hard question all at once. For example, it may be easier to answer “What profession do H. L. Mencken and Albert Camus have in common?” when given the answers to the sub-questions “What profession does H. L. Mencken have?” and “Who was Albert Camus?”
Question: What off-the-shelf QA model was used to answer sub-questions?
Paper: Unsupervised Question Decomposition for Question Answering
We now formulate the problem and overview our high-level approach, with details in the following section. We aim to leverage a QA model that is accurate on simple questions to answer hard questions, without using supervised question decompositions. Here, we consider simple questions to be “single-hop” questions that require reasoning over one paragraph or piece of evidence, and we consider hard questions to be “multi-hop.” Our aim is then to train a multi-hop QA model $M$ to provide the correct answer $a$ to a multi-hop question $q$ about a given a context $c$ (e.g., several paragraphs). Normally, we would train $M$ to maximize $\log p_M(a | c, q)$. To help $M$, we leverage a single-hop QA model that may be queried with sub-questions $s_1, \dots , s_N$, whose “sub-answers” to each sub-question $a_1, \dots , a_N$ may be provided to the multi-hop QA model. $M$ may then instead maximize the (potentially easier) objective $\log p_M(a | c, q, [s_1, a_1], \dots , [a_N, s_N])$.
Question: How large is the improvement over the baseline?
Paper: Unsupervised Question Decomposition for Question Answering
To train a decomposition model, we need appropriate training data. We assume access to a hard question corpus $Q$ and a simple question corpus $S$. Instead of using supervised $(q, d)$ training examples, we design an algorithm that constructs pseudo-decompositions $d^{\prime }$ to form $(q, d^{\prime })$ pairs from $Q$ and $S$ using an unsupervised approach (§SECREF4). We then train a model to map $q$ to a decomposition. We explore learning to decompose with standard and unsupervised sequence-to-sequence learning (§SECREF6).
Question: What is the strong baseline that this work outperforms?
Paper: Simultaneous Neural Machine Translation using Connectionist Temporal Classification
Simultaneous translation is a translation task where the translation process starts before the end of an input. It helps real-time spoken language communications such as human conversations and public talks. A usual machine translation system works in the sentence level and starts its translation process after it reads the end of a sentence. It would not be appropriate for spoken languages due to roughly two issues: (1) sentence boundaries are not clear and (2) a large latency occurs for a long input.
Question: Which dataset do they use?
Paper: Simultaneous Neural Machine Translation using Connectionist Temporal Classification
First, we review a general NMT model following the formulation by BIBREF1 and the “Wait-k" model BIBREF2 that is the baseline model for simultaneous NMT.
Question: Do they trim the search space of possible output sequences?
Paper: Simultaneous Neural Machine Translation using Connectionist Temporal Classification
The encoder takes a sequence of a source sentence $X$ as inputs and returns forward hidden vectors $\overrightarrow{\textbf {h}_i}(1 \le i \le I)$ of the forward RNNs:
Question: Which model architecture do they use to build a model?
Paper: Simultaneous Neural Machine Translation using Connectionist Temporal Classification
The decoder takes source hidden vectors as inputs and returns target language words one-by-one with the attention mechanism. The decoder RNNs recurrently generates target words using its hidden state and an output context. The conditional generation probability of the target word $\textbf {y}_i$ defined as follows:
Question: Do they compare simultaneous translation performance to regular machine translation?
Paper: Simultaneous Neural Machine Translation using Connectionist Temporal Classification
In this work, we proposed the method to decide the output timing adaptively. The proposed method introduces a special token <wait> which is output instead of delaying translation to target-side vocabulary.
Question: Which metrics do they use to evaluate simultaneous translation?