z-uo/qasper-squad huggingface dataset:
Paper: $\rho$-hot Lexicon Embedding-based Two-level LSTM for Sentiment Analysis
Deep learning (including word embedding BIBREF12 ) has been applied to almost all text-related applications, such as translation BIBREF13 , quality assurance BIBREF14 , recommendation BIBREF15 , and categorization BIBREF16 . Popular deep neural networks are divided into convolutional neural networks (CNNs) BIBREF17 and recurrent neural network (RNNs) BIBREF18 BIBREF19 . Both are utilized in sentiment classification BIBREF20 . Kim investigated the use of CNN in sentence sentiment classification and achieved promising results BIBREF2 . LSTM BIBREF21 , a classical type of RNN, is the most popular network used for sentiment classification. A binary-directional LSTM BIBREF22 with an attention mechanism is demonstrated to be effective in sentiment analysis.
Question: How long are the datasets?
Answer: Travel dataset contains 4100 raw samples, 11291 clauses, Hotel dataset contains 3825 raw samples, 11264 clauses, and the Mobile dataset contains 3483 raw samples and 8118 clauses
Paper: $\rho$-hot Lexicon Embedding-based Two-level LSTM for Sentiment Analysis
This section first introduces our two-stage labeling procedure. A two-level LSTM is then proposed. Lexicon embedding is finally leveraged to incorporate lexical cues.
Question: What are the sources of the data?
Answer: User reviews written in Chinese collected online for hotel, mobile phone, and travel domains
Paper: $\rho$-hot Lexicon Embedding-based Two-level LSTM for Sentiment Analysis
As stated earlier, sentiment is subjective, and texts usually contain mixed sentiment orientations. Therefore, texts¡¯ sentiment orientations are difficult to label. In our study, three sentiment labels, namely, positive, neutral, and negative, are used. The following sentences are taken as examples.
Question: What is the new labeling strategy?
Answer: They use a two-stage labeling strategy where in the first stage single annotators label a large number of short texts with relatively pure sentiment orientations and in the second stage multiple annotators label few text samples with mixed sentiment orientations
Paper: Unsupervised Pre-training for Natural Language Generation: A Literature Review
Unsupervised pre-training has sparked a sensational research interest in the natural language processing (NLP) community. This technology provides a promising way to exploit linguistic information from large-scale unlabelled textual data, which can serve as an auxiliary prior knowledge to benefit a wide range of NLP applications. In the literature, language modeling (LM) is a prevalent task for pre-training, where the target words are predicted conditioned on a given context. Therefore, it is intuitive to employ the pre-trained LMs for natural language generation, as the pre-training objective naturally accords with the goal of NLG. However, revolutionary improvements are only observed in the field of NLU.
Question: Which future direction in NLG are discussed?
Paper: Unsupervised Pre-training for Natural Language Generation: A Literature Review
Learning fine-grained language representations is a perennial topic in natural language understanding. In restrospect, compelling evidences suggest that good representations can be learned through unsupervised pre-training.
Question: What experimental phenomena are presented?
Paper: Unsupervised Pre-training for Natural Language Generation: A Literature Review
NLG systems are usually built with an encoder-decoder framework, where the encoder reads the context information and the decoder generates the target text from the encoded vectorial representations. A direct way to utilize the pre-trained models is to initialize part of the encoder (when dealing with textual context) and/or the decoder with pre-trained parameters. For the encoder, pre-training is expected to provide better sentence representations, as we discussed in Section SECREF2. For the decoder, the intuition is to endow the model with some rudimentary ability for text generation.
Question: How strategy-based methods handle obstacles in NLG?
Paper: Unsupervised Pre-training for Natural Language Generation: A Literature Review
Separately initializing the encoder and the decoder with LMs neglects the interaction between the two modules at the pre-training stage, which is sub-optimal. For NLG tasks that can be modeled as Seq2Seq learning, it is feasible to jointly pre-train the encoder and the decoder. Existing approaches for this sake can be categorized into three variants: Denoising autoencoders (DAEs), conditional masked language models (CMLMs) and sequence to sequence language models (Seq2Seq LMs).
Question: How architecture-based method handle obstacles in NLG?
Paper: Incrementalizing RASA's Open-Source Natural Language Understanding Pipeline
There is no shortage of services that are marketed as natural language understanding (nlu) solutions for use in chatbots, digital personal assistants, or spoken dialogue systems (sds). Recently, Braun2017 systematically evaluated several such services, including Microsoft LUIS, IBM Watson Conversation, API.ai, wit.ai, Amazon Lex, and RASA BIBREF0 . More recently, Liu2019b evaluated LUIS, Watson, RASA, and DialogFlow using some established benchmarks. Some nlu services work better than others in certain tasks and domains with a perhaps surprising pattern: RASA, the only fully open-source nlu service among those evaluated, consistently performs on par with the commercial services.
Question: How are their changes evaluated?
Answer: The changes are evaluated based on accuracy of intent and entity recognition on SNIPS dataset
Paper: Cross-Lingual Induction and Transfer of Verb Classes Based on Word Vector Space Specialisation
Playing a key role in conveying the meaning of a sentence, verbs are famously complex. They display a wide range of syntactic-semantic behaviour, expressing the semantics of an event as well as relational information among its participants BIBREF0 , BIBREF1 , BIBREF2 .
Question: What baseline is used for the verb classification experiments?
Paper: Cross-Lingual Induction and Transfer of Verb Classes Based on Word Vector Space Specialisation
Our departure point is a state-of-the-art specialisation model for fine-tuning vector spaces termed Paragram BIBREF49 . The Paragram procedure injects similarity constraints between word pairs in order to make their vector space representations more similar; we term these the Attract constraints. Let INLINEFORM0 be the vocabulary consisting of the source language and target language vocabularies INLINEFORM1 and INLINEFORM2 , respectively. Let INLINEFORM3 be the set of word pairs standing in desirable lexical relations; these include: 1) verb pairs from the same VerbNet class (e.g. (en_transport, en_transfer) from verb class send-11.1); and 2) the cross-lingual synonymy pairs (e.g. (en_peace, fi_rauha)). Given the initial distributional space and collections of such Attract pairs INLINEFORM4 , the model gradually modifies the space to bring the designated word vectors closer together, working in mini-batches of size INLINEFORM5 . The method's cost function can be expressed as:
Question: What clustering algorithm is used on top of the VerbNet-specialized representations?
Paper: Cross-Lingual Induction and Transfer of Verb Classes Based on Word Vector Space Specialisation
Given the initial distributional or specialised collection of target language vectors INLINEFORM0 , we apply an off-the-shelf clustering algorithm on top of these vectors in order to group verbs into classes. Following prior work BIBREF56 , BIBREF57 , BIBREF17 , we employ the MNCut spectral clustering algorithm BIBREF58 , which has wide applicability in similar NLP tasks which involve high-dimensional feature spaces BIBREF59 , BIBREF60 , BIBREF18 . Again, following prior work BIBREF17 , BIBREF61 , we estimate the number of clusters INLINEFORM1 using the self-tuning method of Zelnik:2004nips. This algorithm finds the optimal number by minimising a cost function based on the eigenvector structure of the word similarity matrix. We refer the reader to the relevant literature for further details.
Question: How many words are translated between the cross-lingual translation pairs?
Paper: Cross-Lingual Induction and Transfer of Verb Classes Based on Word Vector Space Specialisation
Cross-Lingual Transfer Model F-1 verb classification scores for the six target languages with different sets of constraints are summarised in Fig. FIGREF29 . We can draw several interesting conclusions. First, the strongest results on average are obtained with the model which transfers the VerbNet knowledge from English (as a resource-rich language) to the resource-lean target language (providing an answer to question Q3, Sect. SECREF1 ). These improvements are visible across all target languages, empirically demonstrating the cross-lingual nature of VerbNet-style classifications. Second, using cross-lingual constraints alone (XLing) yields strong gains over initial distributional spaces (answering Q1 and Q2). Fig. FIGREF29 also shows that cross-lingual similarity constraints are more beneficial than the monolingual ones, despite a larger total number of the monolingual constraints in each language (see Tab. TABREF18 ). This suggests that such cross-lingual similarity links are strong implicit indicators of class membership. Namely, target language words which map to the same source language word are likely to be synonyms and consequently end up in the same verb class in the target language. However, the cross-lingual links are even more useful as means for transferring the VerbNet knowledge, as evidenced by additional gains with XLing+VerbNet-EN.
Question: What are the six target languages?
Answer: Answer with content missing: (3 Experimental Setup) We experiment with six target languages: French (FR), Brazilian Portuguese (PT), Italian (IT), Polish (PL), Croatian (HR), and Finnish (FI).
Paper: We Built a Fake News&Click-bait Filter: What Happened Next Will Blow Your Mind!
Fake news are written and published with the intent to mislead in order to gain financially or politically, often targeting specific user groups. Another type of harmful content on the Internet are the so-called click-baits, which are distinguished by their sensational, exaggerated, or deliberately false headlines that grab attention and deceive the user into clicking an article with questionable content.
Question: what classifiers were used in this paper?
Paper: We Built a Fake News&Click-bait Filter: What Happened Next Will Blow Your Mind!
Trustworthiness and veracity analytics of on-line statements is an emerging research direction BIBREF0 . This includes predicting credibility of information shared in social media BIBREF1 , stance classification BIBREF2 and contradiction detection in rumours BIBREF3 . For example, Castillo:2011:ICT:1963405.1963500 studied the problem of finding false information about a newsworthy event. They compiled their own dataset, focusing on tweets using a variety of features including user reputation, author writing style, and various time-based features. Canini:2011 analysed the interaction of content and social network structure, and Morris:2012:TBU:2145204.2145274 studied how Twitter users judge truthfulness. They found that this is hard to do based on content alone, and instead users are influenced by heuristics such as user name.
Question: what are their evaluation metrics?
Paper: We Built a Fake News&Click-bait Filter: What Happened Next Will Blow Your Mind!
We use a corpus of Bulgarian news over a fixed period of time, whose factuality had been questioned. The news come from 377 different sources from various domains, including politics, interesting facts and tips&tricks. The dataset was prepared for the Hack the Fake News hackathon. It was provided by the Bulgarian Association of PR Agencies and is available in Gitlab. The corpus was automatically collected, and then annotated by students of journalism. Each entry in the dataset consists of the following elements: URL of the original article, date of publication, article heading, article content, a label indicating whether the article is fake or not, and another label indicating whether it is a click-bait.
Question: what types of features were used?
Paper: We Built a Fake News&Click-bait Filter: What Happened Next Will Blow Your Mind!
We propose a general framework for finding fake news focusing on the text only. We first create some resources, e.g., dictionaries of words strongly correlated with fake news, which are needed for feature extraction. Then, we design features that model a number of interesting aspects about an article, e.g., style, intent, etc. Moreover, we use a deep neural network to learn task-specific representations of the articles, which includes an attention mechanism that can focus on the most discriminative sentences and words.
Question: what lexical features did they experiment with?
Paper: We Built a Fake News&Click-bait Filter: What Happened Next Will Blow Your Mind!
As our work is the first attempt at predicting click-baits in Bulgarian, it is organized around building new language-specific resources and analyzing the task.
Question: what is the size of the dataset?
Paper: We Built a Fake News&Click-bait Filter: What Happened Next Will Blow Your Mind!
Fake news are written with the intent to deceive, and their authors often use a different style of writing compared to authors that create genuine content. This could be either deliberately, e.g., if the author wants to adapt the text to a specific target group or wants to provoke some particular emotional reaction in the reader, or unintentionally, e.g., because the authors of fake news have different writing style and personality compared to journalists in mainstream media. Disregarding the actual reason, we use features from author profiling and style detection BIBREF28 .
Question: what datasets were used?
Paper: We Built a Fake News&Click-bait Filter: What Happened Next Will Blow Your Mind!
As we mentioned above, our method is purely text-based. Thus, we ignored the publishing date of the article. In future work, it could be explored as a useful piece of information about the credibility of the article, as there is interesting research in this direction BIBREF24 . We also disregarded the article source (the URL) because websites that specialize in producing and distributing fake content are often banned and then later reappear under another name. We recognize that the credibility of a specific website could be a very informative feature, but, for the sake of creating a robust method for fake news detection, our system relies only on the text when predicting whether the target article is likely to be fake. We describe our features in more detail below.
Question: what are the three reasons everybody hates them?
Paper: Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?
The wide use and success of monolingual word embeddings in NLP tasks BIBREF0 , BIBREF1 has inspired further research focus on the induction of cross-lingual word embeddings (CLWEs). CLWE methods learn a shared cross-lingual word vector space where words with similar meanings obtain similar vectors regardless of their actual language. CLWEs benefit cross-lingual NLP, enabling multilingual modeling of meaning and supporting cross-lingual transfer for downstream tasks and resource-lean languages. CLWEs provide invaluable cross-lingual knowledge for, inter alia, bilingual lexicon induction BIBREF2 , BIBREF3 , information retrieval BIBREF4 , BIBREF5 , machine translation BIBREF6 , BIBREF7 , document classification BIBREF8 , cross-lingual plagiarism detection BIBREF9 , domain adaptation BIBREF10 , cross-lingual POS tagging BIBREF11 , BIBREF12 , and cross-lingual dependency parsing BIBREF13 , BIBREF14 .
Question: How are seed dictionaries obtained by fully unsupervised methods?
Paper: Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?
We now dissect a general framework for unsupervised CLWE learning, and show that the “bag of tricks of the trade” used to increase their robustness (which often slips under the radar) can be equally applied to (weakly) supervised projection-based approaches, leading to their fair(er) comparison.
Question: How does BLI measure alignment quality?
Paper: Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?
In short, projection-based CLWE methods learn to (linearly) align independently trained monolingual spaces $\mathbf {X}$ and $\mathbf {Z}$ , using a word translation dictionary $D_0$ to guide the alignment process. Let $\mathbf {X}_D \subset \mathbf {X}$ and $\mathbf {Z}_D \subset \mathbf {Z}$ be the row-aligned subsets of monolingual spaces containing vectors of aligned words from $D_0$ . Alignment matrices $\mathbf {X}_D$ and $\mathbf {Z}_D$ are then used to learn orthogonal transformations $\mathbf {W}_x$ and $\mathbf {W}_z$ that define the joint bilingual space $\mathbf {Z}$0 . While supervised projection-based CLWE models learn the mapping using a provided external (clean) dictionary $\mathbf {Z}$1 , their unsupervised counterparts automatically induce the seed dictionary in an unsupervised way (C1) and then refine it in an iterative fashion (C2).
Question: What methods were used for unsupervised CLWE?
Paper: Open Information Extraction on Scientific Text: An Evaluation
 This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/
Question: What is the size of the released dataset?
Answer: 440 sentences, 2247 triples extracted from those sentences, and 11262 judgements on those triples.
Paper: Open Information Extraction on Scientific Text: An Evaluation
OIE systems analyze sentences and emit relations between one predicate and two or more arguments (e.g. Washington :: was :: president). The arguments and predicates are not fixed to a given domain. (Note, that throughout this paper we use the word `triple” to refer interchangeably to binary relations.) Existing evaluation approaches for OIE systems have primarily taken a ground truth-based approach. Human annotators analyze sentences and determine correct relations to be extracted. Systems are then evaluated with respect to the overlap or similarity of their extractions to the ground truth annotations, allowing the standard metrics of precision and recall to be reported.
Question: Were the OpenIE systems more accurate on some scientific disciplines than others?
Paper: Open Information Extraction on Scientific Text: An Evaluation
We evaluate two OIE systems (i.e. extractors). The first, OpenIE 4 BIBREF5 , descends from two popular OIE systems OLLIE BIBREF10 and Reverb BIBREF10 . We view this as a baseline system. The second was MinIE BIBREF7 , which is reported as performing better than OLLIE, ClauseIE BIBREF9 and Stanford OIE BIBREF9 . MinIE focuses on the notion of minimization - producing compact extractions from sentences. In our experience using OIE on scientific text, we have found that these systems often produce overly specific extractions that do not provide the redundancy useful for downstream tasks. Hence, we thought this was a useful package to explore.
Question: What is the most common error type?
Paper: Open Information Extraction on Scientific Text: An Evaluation
We used two different data sources in our evaluation. The first dataset (WIKI) was the same set of 200 sentences from Wikipedia used in BIBREF7 . These sentences were randomly selected by the creators of the dataset. This choice allows for a rough comparison between our results and theirs.
Question: Which OpenIE systems were used?
Answer: OpenIE4 and MiniIE
Paper: Open Information Extraction on Scientific Text: An Evaluation
We employed the following annotation process. Each OIE extractor was applied to both datasets with the settings described above. This resulted in the generation of triples for 199 of the 200 WIKI sentences and 206 of the 220 SCI sentences. That is there were some sentences in which no triples were extracted. We discuss later the sentences in which no triples were extracted. In total 2247 triples were extracted.
Question: What is the role of crowd-sourcing?
Paper: RaKUn: Rank-based Keyword extraction via Unsupervised learning and Meta vertex aggregation
Keywords are terms (i.e. expressions) that best describe the subject of a document BIBREF0 . A good keyword effectively summarizes the content of the document and allows it to be efficiently retrieved when needed. Traditionally, keyword assignment was a manual task, but with the emergence of large amounts of textual data, automatic keyword extraction methods have become indispensable. Despite a considerable effort from the research community, state-of-the-art keyword extraction algorithms leave much to be desired and their performance is still lower than on many other core NLP tasks BIBREF1 . The first keyword extraction methods mostly followed a supervised approach BIBREF2 , BIBREF3 , BIBREF4 : they first extract keyword features and then train a classifier on a gold standard dataset. For example, KEA BIBREF4 , a state of the art supervised keyword extraction algorithm is based on the Naive Bayes machine learning algorithm. While these methods offer quite good performance, they rely on an annotated gold standard dataset and require a (relatively) long training process. In contrast, unsupervised approaches need no training and can be applied directly without relying on a gold standard document collection. They can be further divided into statistical and graph-based methods. The former, such as YAKE BIBREF5 , BIBREF6 , KP-MINER BIBREF7 and RAKE BIBREF8 , use statistical characteristics of the texts to capture keywords, while the latter, such as Topic Rank BIBREF9 , TextRank BIBREF10 , Topical PageRank BIBREF11 and Single Rank BIBREF12 , build graphs to rank words based on their position in the graph. Among statistical approaches, the state-of-the-art keyword extraction algorithm is YAKE BIBREF5 , BIBREF6 , which is also one of the best performing keyword extraction algorithms overall; it defines a set of five features capturing keyword characteristics which are heuristically combined to assign a single score to every keyword. On the other hand, among graph-based approaches, Topic Rank BIBREF9 can be considered state-of-the-art; candidate keywords are clustered into topics and used as vertices in the final graph, used for keyword extraction. Next, a graph-based ranking model is applied to assign a significance score to each topic and keywords are generated by selecting a candidate from each of the top-ranked topics. Network-based methodology has also been successfully applied to the task of topic extraction BIBREF13 .
Question: How are meta vertices computed?
Paper: RaKUn: Rank-based Keyword extraction via Unsupervised learning and Meta vertex aggregation
We first discuss how the texts are transformed to graphs, on which RaKUn operates. Next, we formally state the problem of keyword extraction and discuss its relation to graph centrality metrics.
Question: How are graphs derived from a given text?