z-uo/qasper-squad huggingface dataset:
Paper: Solving Arithmetic Word Problems Automatically Using Transformer and Unambiguous Representations
We view math word problem solving as a sequence-to-sequence translation problem. RNNs have excelled in sequence-to-sequence problems such as translation and question answering. The recent introduction of attention mechanisms has improved the performance of RNN models. Vaswani et al. BIBREF0 introduced the Transformer network, which uses stacks of attention layers instead of recurrence. Applications of Transformers have achieved state-of-the-art performance in many NLP tasks. We use this architecture to produce character sequences that are arithmetic expressions. The models we experiment with are easy and efficient to train, allowing us to test several configurations for a comprehensive comparison. We use several configurations of Transformer networks to learn the prefix, postfix, and infix notations of MWP equations independently.
Question: Are the Transformers masked?
Paper: Solving Arithmetic Word Problems Automatically Using Transformer and Unambiguous Representations
We work with four individual datasets. The datasets contain addition, subtraction, multiplication, and division word problems.
Question: How is this problem evaluated?
Paper: Solving Arithmetic Word Problems Automatically Using Transformer and Unambiguous Representations
We take a simple approach to convert infix expressions found in the MWPs to the other two representations. Two stacks are filled by iterating through string characters, one with operators found in the equation and the other with the operands. From these stacks, we form a binary tree structure. Traversing an expression tree in pre-order results in a prefix conversion. Post-order traversal gives us a postfix expression. Three versions of our training and testing data are created to correspond to each type of expression. By training on different representations, we expect our test results to change.
Question: What datasets do they use?
Paper: What Do You Mean I'm Funny? Personalizing the Joke Skill of a Voice-Controlled Virtual Assistant
Voice-controlled virtual assistants (VVA) such as Siri and Alexa have experienced an exponential growth in terms of number of users and provided capabilities. They are used by millions for a variety of tasks including shopping, playing music, and even telling jokes. Arguably, their success is due in part to the emotional and personalized experience they provide. One important aspect of this emotional interaction is humor, a fundamental element of communication. Not only can it create in the user a sense of personality, but also be used as fallback technique for out-of-domain queries BIBREF0. Usually, a VVA's humorous responses are invoked by users with the phrase "Tell me a joke". In order to improve the joke experience and overall user satisfaction with a VVA, we propose to personalize the response to each request. To achieve this, a method should be able to recognize and evaluate humor, a challenging task that has been the focus of extensive work. Some authors have applied traditional NLP techniques BIBREF1, while others deep learning models BIBREF2. Moreover, BIBREF3 follows a semantic-based approach, while BIBREF4 and BIBREF5 tackle the challenge from a cognitive and linguistic perspective respectively.
Question: What evaluation metrics were used?
Paper: What Do You Mean I'm Funny? Personalizing the Joke Skill of a Voice-Controlled Virtual Assistant
Generating labels for this VVA skill is challenging. Label generation through explicit user feedback is unavailable since asking users for feedback creates friction and degrade the user experience. In addition, available humor datasets such as BIBREF3, BIBREF11 only contain jokes and corresponding labels, but not the additional features we need to personalize the jokes.
Question: Where did the real production data come from?
Paper: What Do You Mean I'm Funny? Personalizing the Joke Skill of a Voice-Controlled Virtual Assistant
All models have access to the same raw features, which we conceptually separate into user, item and contextual features. Examples of features in each of these categories are shown in Table TABREF4. Some of these are used directly by the models, while others need to be pre-processed. The manner in which each model consumes them is explained next.
Question: What feedback labels are used?
Paper: A Measure of Similarity in Textual Data Using Spearman's Rank Correlation Coefficient
Over the past few years, the term big data has become an important key point for research into data mining and information retrieval. Through the years, the quantity of data managed across enterprises has evolved from a simple and imperceptible task to an extent to which it has become the central performance improvement problem. In other words, it evolved to be the next frontier for innovation, competition and productivity BIBREF0. Extracting knowledge from data is now a very competitive environment. Many companies process vast amounts of customer/user data in order to improve the quality of experience (QoE) of their customers. For instance, a typical use-case scenario would be a book seller that performs an automatic extraction of the content of the books a customer has bought, and subsequently extracts knowledge of what customers prefer to read. The knowledge extracted could then be used to recommend other books. Book recommending systems are typical examples where data mining techniques should be considered as the primary tool for making future decisions BIBREF1.
Question: What representations for textual documents do they use?
Paper: A Measure of Similarity in Textual Data Using Spearman's Rank Correlation Coefficient
In this section we provide a brief background of vector space representation of TDs and existing similarity measures that have been widely used in statistical text analysis. To begin with, we consider the representation of documents.
Question: Which dataset(s) do they use?
Paper: A Measure of Similarity in Textual Data Using Spearman's Rank Correlation Coefficient
A document $d$ can be defined as a finite sequence of terms (independent textual entities within a document, for example, words), namely $d=(t_1,t_2,\dots ,t_n)$. A general idea is to associate weight to each term $t_i$ within $d$, such that
Question: How do they evaluate knowledge extraction performance?
Paper: CamemBERT: a Tasty French Language Model
Pretrained word representations have a long history in Natural Language Processing (NLP), from non-neural methods BIBREF0, BIBREF1, BIBREF2 to neural word embeddings BIBREF3, BIBREF4 and to contextualised representations BIBREF5, BIBREF6. Approaches shifted more recently from using these representations as an input to task-specific architectures to replacing these architectures with large pretrained language models. These models are then fine-tuned to the task at hand with large improvements in performance over a wide range of tasks BIBREF7, BIBREF8, BIBREF9, BIBREF10.
Question: What is CamemBERT trained on?
Paper: CamemBERT: a Tasty French Language Model
The first neural word vector representations were non-contextualised word embeddings, most notably word2vec BIBREF3, GloVe BIBREF4 and fastText BIBREF14, which were designed to be used as input to task-specific neural architectures. Contextualised word representations such as ELMo BIBREF5 and flair BIBREF6, improved the expressivity of word embeddings by taking context into account. They improved the performance of downstream tasks when they replaced traditional word representations. This paved the way towards larger contextualised models that replaced downstream architectures in most tasks. These approaches, trained with language modeling objectives, range from LSTM-based architectures such as ULMFiT BIBREF15 to the successful transformer-based architectures such as GPT2 BIBREF8, BERT BIBREF7, RoBERTa BIBREF9 and more recently ALBERT BIBREF16 and T5 BIBREF10.
Question: Which tasks does CamemBERT not improve on?
Paper: CamemBERT: a Tasty French Language Model
Since the introduction of word2vec BIBREF3, many attempts have been made to create monolingual models for a wide range of languages. For non-contextual word embeddings, the first two attempts were by BIBREF17 and BIBREF18 who created word embeddings for a large number of languages using Wikipedia. Later BIBREF19 trained fastText word embeddings for 157 languages using Common Crawl and showed that using crawled data significantly increased the performance of the embeddings relatively to those trained only on Wikipedia.
Question: What is the state of the art?
Answer: POS and DP task: CONLL 2018
NER task: (no extensive work) Strong baselines CRF and BiLSTM-CRF
NLI task: mBERT or XLM (not clear from text)
Paper: CamemBERT: a Tasty French Language Model
Following the success of large pretrained language models, they were extended to the multilingual setting with multilingual BERT , a single multilingual model for 104 different languages trained on Wikipedia data, and later XLM BIBREF12, which greatly improved unsupervised machine translation. A few monolingual models have been released: ELMo models for Japanese, Portuguese, German and Basque and BERT for Simplified and Traditional Chinese and German.
Question: How much better was results of CamemBERT than previous results on these tasks?
Paper: CamemBERT: a Tasty French Language Model
Our approach is based on RoBERTa BIBREF9, which replicates and improves the initial BERT by identifying key hyper-parameters for more robust performance.
Question: Was CamemBERT compared against multilingual BERT on these tasks?
Paper: CamemBERT: a Tasty French Language Model
Similar to RoBERTa and BERT, CamemBERT is a multi-layer bidirectional Transformer BIBREF21. Given the widespread usage of Transformers, we do not describe them in detail here and refer the reader to BIBREF21. CamemBERT uses the original BERT $_{\small \textsc {BASE}}$ configuration: 12 layers, 768 hidden dimensions, 12 attention heads, which amounts to 110M parameters.
Question: How long was CamemBERT trained?
Paper: CamemBERT: a Tasty French Language Model
We train our model on the Masked Language Modeling (MLM) task. Given an input text sequence composed of $N$ tokens $x_1, ..., x_N$, we select $15\%$ of tokens for possible replacement. Among those selected tokens, 80% are replaced with the special $<$mask$>$ token, 10% are left unchanged and 10% are replaced by a random token. The model is then trained to predict the initial masked tokens using cross-entropy loss.
Question: What data is used for training CamemBERT?
Paper: Vocabulary-based Method for Quantifying Controversy in Social Media
Controversy is a phenomenom with a high impact at various levels. It has been broadly studied from the perspective of different disciplines, ranging from the seminal analysis of the conflicts within the members of a karate club BIBREF0 to political issues in modern times BIBREF1, BIBREF2. The irruption of digital social networks BIBREF3 gave raise to new ways of intentionally intervening on them for taking some advantage BIBREF4, BIBREF5. Moreover highly contrasting points of view in some groups tend to provoke conflicts that lead to attacks from one community to the other by harassing, “brigading”, or “trolling” it BIBREF6. The existing literature shows different issues that controversy brings up such as splitting of communities, biased information, hateful discussions and attacks between groups, generally proposing ways to solve them. For example, Kumar, Srijan, et al. BIBREF6 analyze many techniques to defend us from attacks in Reddit while Stewart, et al. BIBREF4 insinuate that there was external interference in Twitter during the 2016 US presidential elections to benefit one candidate. Also, as shown in BIBREF7, detecting controversy could provide the basis to improve the “news diet" of readers, offering the possibility to connect users with different points of views by recommending them new content to read BIBREF8.
Question: What are the state of the art measures?
Paper: Vocabulary-based Method for Quantifying Controversy in Social Media
Many previous works are dedicated to quantifying the polarization observed in online social networks and social media BIBREF1, BIBREF19, BIBREF20, BIBREF21, BIBREF22, BIBREF23. The main characteristic of those works is that the measures proposed are based on the structural characteristics of the underlying graph. Among them, we highlight the work of Garimella et al.BIBREF23 that presents an extensive comparison of controversy measures, different graph-building approaches, and data sources, achieving the best performance of all. In their research they propose different metrics to measure polarization on Twitter. Their techniques based on the structure of the endorsement graph can successfully detect whether a discussion (represented by a set of tweets), is controversial or not regardless of the context and most importantly, without the need of any domain expertise. They also consider two different methods to measure controversy based on the analysis of the posts contents, but both fail when used to create a measure of controversy.
Question: What controversial topics are experimented with?
Paper: Vocabulary-based Method for Quantifying Controversy in Social Media
Our approach to measuring controversy is based on a systematic way of characterizing social media activity through its content. We employ a pipeline with five stages, namely graph building, community identification, model training, predicting and controversy measure. The final output of the pipeline is a value that measures how controversial a topic is, with higher values corresponding to higher degrees of controversy. The method is based on analysing posts content through Fasttext BIBREF34, a library for efficient learning of word representations and sentence classification developed by Facebook Research team. In short, our method works as follows: through Fasttext we train a language-agnostic model which can predict the community of many users by their jargon. Then we take there predictions and compute a score based on the physic notion Dipole Moment using a language approach to identify core or characteristic users and set the polarity trough them. We provide a detailed description of each stage in the following.
Question: What datasets did they use?
Paper: Vocabulary-based Method for Quantifying Controversy in Social Media
In this section we report the results obtained by running the above proposed method over different discussions.
Question: What social media platform is observed?
Paper: Vocabulary-based Method for Quantifying Controversy in Social Media
In the literature, a topic is often defined by a single hashtag. However, this might be too restrictive in many cases. In our approach, a topic is operationalized as an specific hashtags or key words. Sometimes a discussion in a particular moment could not have a defined hashtag but it could be around a certain keyword, i.e. a word or expression that is not specifically a hashtag but it is widely used in the topic. For example during the Brazilian presidential elections in 2018 we captured the discussion by the mentions to the word Bolsonaro, that is the principal candidate's surname.
Question: How many languages do they experiment with?
Paper: Semantic Sentiment Analysis of Twitter Data
Microblog sentiment analysis; Twitter opinion mining
Question: What is the current SOTA for sentiment analysis on Twitter at the time of writing?
Paper: Semantic Sentiment Analysis of Twitter Data
Sentiment Analysis: This is text analysis aiming to determine the attitude of a speaker or a writer with respect to some topic or the overall contextual polarity of a piece of text.
Question: What difficulties does sentiment analysis on Twitter have, compared to sentiment analysis in other domains?
Answer: Tweets noisy nature, use of creative spelling and punctuation, misspellings, slang, new words, URLs, and genre-specific terminology and abbreviations, short (length limited) text
Paper: Semantic Sentiment Analysis of Twitter Data
Sentiment analysis on Twitter is the use of natural language processing techniques to identify and categorize opinions expressed in a tweet, in order to determine the author's attitude toward a particular topic or in general. Typically, discrete labels such as positive, negative, neutral, and objective are used for this purpose, but it is also possible to use labels on an ordinal scale, or even continuous numerical values.
Question: What are the metrics to evaluate sentiment analysis on Twitter?
Paper: COSTRA 1.0: A Dataset of Complex Sentence Transformations
Vector representations are becoming truly essential in majority of natural language processing tasks. Word embeddings became widely popular with the introduction of word2vec BIBREF0 and GloVe BIBREF1 and their properties have been analyzed in length from various aspects.
Question: How many sentence transformations on average are available per unique sentence in dataset?
Answer: 27.41 transformation on average of single seed sentence is available in dataset.
Paper: COSTRA 1.0: A Dataset of Complex Sentence Transformations
As hinted above, there are many methods of converting a sequence of words into a vector in a highly dimensional space. To name a few: BiLSTM with the max-pooling trained for natural language inference BIBREF13, masked language modeling and next sentence prediction using bidirectional Transformer BIBREF14, max-pooling last states of neural machine translation among many languages BIBREF15 or the encoder final state in attentionless neural machine translation BIBREF16.
Question: What annotations are available in the dataset?
Answer: For each source sentence, transformation sentences that are transformed according to some criteria (paraphrase, minimal change etc.)
Paper: COSTRA 1.0: A Dataset of Complex Sentence Transformations
We acquired the data in two rounds of annotation. In the first one, we were looking for original and uncommon sentence change suggestions. In the second one, we collected sentence alternations using ideas from the first round. The first and second rounds of annotation could be broadly called as collecting ideas and collecting data, respectively.
Question: How are possible sentence transformations represented in dataset, as new sentences?
Answer: Yes, as new sentences.
Paper: COSTRA 1.0: A Dataset of Complex Sentence Transformations
We manually selected 15 newspaper headlines. Eleven annotators were asked to modify each headline up to 20 times and describe the modification with a short name. They were given an example sentence and several of its possible alternations, see tab:firstroundexamples.
Question: What are all 15 types of modifications ilustrated in the dataset?
Answer: - paraphrase 1
- paraphrase 2
- different meaning
- opposite meaning
- nonsense
- minimal change
- generalization
- gossip
- formal sentence
- non-standard sentence
- simple sentence
- possibility
- ban
- future
- past
Paper: COSTRA 1.0: A Dataset of Complex Sentence Transformations
We selected 15 modifications types to collect COSTRA 1.0. They are presented in annotationinstructions.
Question: Is this dataset publicly available?
Paper: COSTRA 1.0: A Dataset of Complex Sentence Transformations
The source sentences for annotations were selected from Czech data of Global Voices BIBREF24 and OpenSubtitles BIBREF25. We used two sources in order to have different styles of seed sentences, both journalistic and common spoken language. We considered only sentences with more than 5 and less than 15 words and we manually selected 150 of them for further annotation. This step was necessary to remove sentences that are:
Question: Are some baseline models trained on this dataset?
Paper: COSTRA 1.0: A Dataset of Complex Sentence Transformations
The annotation is a challenging task and the annotators naturally make mistakes. Unfortunately, a single typo can significantly influence the resulting embedding BIBREF26. After collecting all the sentence variations, we applied the statistical spellchecker and grammar checker Korektor BIBREF27 in order to minimize influence of typos to performance of embedding methods. We manually inspected 519 errors identified by Korektor and fixed 129, which were identified correctly.
Question: Do they do any analysis of of how the modifications changed the starting set of sentences?
Paper: COSTRA 1.0: A Dataset of Complex Sentence Transformations
In the second round, we collected 293 annotations from 12 annotators. After Korektor, there are 4262 unique sentences (including 150 seed sentences) that form the COSTRA 1.0 dataset. Statistics of individual annotators are available in tab:statistics.
Question: How do they introduce language variation?
Paper: COSTRA 1.0: A Dataset of Complex Sentence Transformations
We embedded COSTRA sentences with LASER BIBREF15, the method that performed very well in revealing linear relations in BaBo2019. Having browsed a number of 2D visualizations (PCA and t-SNE) of the space, we have to conclude that visually, LASER space does not seem to exhibit any of the desired topological properties discussed above, see fig:pca for one example.
Question: Do they use external resources to make modifications to sentences?
Paper: Learning to Create Sentence Semantic Relation Graphs for Multi-Document Summarization
Today's increasing flood of information on the web creates a need for automated multi-document summarization systems that produce high quality summaries. However, producing summaries in a multi-document setting is difficult, as the language used to display the same information in a sentence can vary significantly, making it difficult for summarization models to capture. Given the complexity of the task and the lack of datasets, most researchers use extractive summarization, where the final summary is composed of existing sentences in the input documents. More specifically, extractive summarization systems output summaries in two steps : via sentence ranking, where an importance score is assigned to each sentence, and via the subsequent sentence selection, where the most appropriate sentence is chosen, by considering 1) their importance and 2) their frequency among all documents. Due to data sparcity, models heavily rely on well-designed features at the word level BIBREF0, BIBREF1, BIBREF2, BIBREF3 or take advantage of other large, manually annotated datasets and then apply transfer learning BIBREF4. Additionally, most of the time, all sentences in the same collection of documents are processed independently and therefore, their relationships are lost.
Question: How big is dataset domain-specific embedding are trained on?
Paper: Learning to Create Sentence Semantic Relation Graphs for Multi-Document Summarization
Let $C$ denote a collection of related documents composed of a set of documents $\lbrace D_i|i \in [1,N]\rbrace $ where $N$ is the number of documents. Moreover, each document $D_i$ consists of a set of sentences $\lbrace S_{i,j}|j \in [1,M]\rbrace $, $M$ being the number of sentences in $D_i$. Given a collection of related documents $C$, our goal is to produce a summary $Sum$ using a subset of these in the input documents ordered in some way, such that $Sum = (S_{i_1,j_1},S_{i_2,j_2},...,S_{i_n,j_m})$.
Question: How big is unrelated corpus universal embedding is traned on?