z-uo/qasper-squad huggingface dataset:
Paper: Generating Personalized Recipes from Historical User Preferences
In the kitchen, we increasingly rely on instructions from cooking websites: recipes. A cook with a predilection for Asian cuisine may wish to prepare chicken curry, but may not know all necessary ingredients apart from a few basics. These users with limited knowledge cannot rely on existing recipe generation approaches that focus on creating coherent recipes given all ingredients and a recipe name BIBREF0. Such models do not address issues of personal preference (e.g. culinary tastes, garnish choices) and incomplete recipe details. We propose to approach both problems via personalized generation of plausible, user-specific recipes using user preferences extracted from previously consumed recipes.
Question: What metrics are used for evaluation?
Answer: Byte-Pair Encoding perplexity  (BPE PPL),
BLEU-1,
BLEU-4,
ROUGE-L,
percentage of distinct unigram (D-1),
percentage of distinct bigrams(D-2),
user matching accuracy(UMA),
Mean Reciprocal Rank(MRR)
Pairwise preference over baseline(PP)
 Distinct-1/2, UMA = User Matching Accuracy, MRR
= Mean Reciprocal Rank, PP = Pairwise preference over baseline (evaluated for 310 recipe pairs per model)
Paper: Generating Personalized Recipes from Historical User Preferences
Large-scale transformer-based language models have shown surprising expressivity and fluency in creative and conditional long-text generation BIBREF6, BIBREF7. Recent works have proposed hierarchical methods that condition on narrative frameworks to generate internally consistent long texts BIBREF8, BIBREF9, BIBREF10. Here, we generate procedurally structured recipes instead of free-form narratives.
Question: What natural language(s) are the recipes written in?
Answer: English
English
Paper: Generating Personalized Recipes from Historical User Preferences
Our model's input specification consists of: the recipe name as a sequence of tokens, a partial list of ingredients, and a caloric level (high, medium, low). It outputs the recipe instructions as a token sequence: $\mathcal {W}_r=\lbrace w_{r,0}, \dots , w_{r,T}\rbrace $ for a recipe $r$ of length $T$. To personalize output, we use historical recipe interactions of a user $u \in \mathcal {U}$.
Question: What were their results on the new dataset?
Paper: Generating Personalized Recipes from Historical User Preferences
We collect a novel dataset of 230K+ recipe texts and 1M+ user interactions (reviews) over 18 years (2000-2018) from Food.com. Here, we restrict to recipes with at least 3 steps, and at least 4 and no more than 20 ingredients. We discard users with fewer than 4 reviews, giving 180K+ recipes and 700K+ reviews, with splits as in tab:recipeixnstats.
Question: What are the baseline models?
Paper: Generating Personalized Recipes from Historical User Preferences
For training and evaluation, we provide our model with the first 3-5 ingredients listed in each recipe. We decode recipe text via top-$k$ sampling BIBREF7, finding $k=3$ to produce satisfactory results. We use a hidden size $d_h=256$ for both the encoder and decoder. Embedding dimensions for vocabulary, ingredient, recipe, techniques, and caloric level are 300, 10, 50, 50, and 5 (respectively). For prior recipe attention, we set $k=20$, the 80th %-ile for the number of user interactions. We use the Adam optimizer BIBREF28 with a learning rate of $10^{-3}$, annealed with a decay rate of 0.9 BIBREF29. We also use teacher-forcing BIBREF30 in all training epochs.
Question: How did they obtain the interactions?
Paper: Generating Personalized Recipes from Historical User Preferences
In this paper, we propose a novel task: to generate personalized recipes from incomplete input specifications and user histories. On a large novel dataset of 180K recipes and 700K reviews, we show that our personalized generative models can generate plausible, personalized, and coherent recipes preferred by human evaluators for consumption. We also introduce a set of automatic coherence measures for instructional texts as well as personalization metrics to support our claims. Our future work includes generating structured representations of recipes to handle ingredient properties, as well as accounting for references to collections of ingredients (e.g. “dry mix").
Question: Where do they get the recipes from?
Paper: Resolving the Scope of Speculation and Negation using Transformer-Based Architectures
The task of speculation detection and scope resolution is critical in distinguishing factual information from speculative information. This has multiple use-cases, like systems that determine the veracity of information, and those that involve requirement analysis. This task is particularly important to the biomedical domain, where patient reports and medical articles often use this feature of natural language. This task is commonly broken down into two subtasks: the first subtask, speculation cue detection, is to identify the uncertainty cue in a sentence, while the second subtask: scope resolution, is to identify the scope of that cue. For instance, consider the example:
Question: What were the baselines?
Paper: Resolving the Scope of Speculation and Negation using Transformer-Based Architectures
We use the methodology by Khandelwal and Sawant (BIBREF12), and modify it to support experimentation with multiple models.
Question: Does RoBERTa outperform BERT?
Paper: Resolving the Scope of Speculation and Negation using Transformer-Based Architectures
We use a default train-validation-test split of 70-15-15 for each dataset. For the speculation detection and scope resolution subtasks using single-dataset training, we report the results as an average of 5 runs of the model. For training the model on multiple datasets, we perform a 70-15-15 split of each training dataset, after which the train and validation part of the individual datasets are merged while the scores are reported for the test part of the individual datasets, which is not used for training or validation. We report the results as an average of 3 runs of the model. Figure FIGREF8 contains results for speculation cue detection and scope resolution when trained on a single dataset. All models perform the best when trained on the same dataset as they are evaluated on, except for BF, which gets the best results when trained on BA. This is because of the transfer learning capabilities of the models and the fact that BF is a smaller dataset than BA (BF: 2670 sentences, BA: 11871 sentences). For speculation cue detection, there is lesser generalizability for models trained on BF or BA, while there is more generalizability for models trained on SFU. This could be because of the different nature of the biomedical domain.
Question: Which multiple datasets did they train on during joint training?
Answer: BioScope Abstracts, SFU, and BioScope Full Papers
Paper: Resolving the Scope of Speculation and Negation using Transformer-Based Architectures
We use a default train-validation-test split of 70-15-15 for each dataset, and use all 4 datasets (BF, BA, SFU and Sherlock). The results for BERT are taken from BIBREF12. The results for XLNet and RoBERTa are averaged across 5 runs for statistical significance. Figure FIGREF14 contains results for negation cue detection and scope resolution. We report state-of-the-art results on negation scope resolution on BF, BA and SFU datasets. Contrary to popular opinion, we observe that XLNet is better than RoBERTa for the cue detection and scope resolution tasks. A few possible reasons for this trend are:
Question: What were the previously reported results?
Paper: Resolving the Scope of Speculation and Negation using Transformer-Based Architectures
In this paper, we expanded on the work of Khandelwal and Sawant (BIBREF12) by looking at alternative transfer-learning models and experimented with training on multiple datasets. On the speculation detection task, we obtained a gain of 0.42 F1 points on BF, 1.98 F1 points on BA and 0.29 F1 points on SFU, while on the scope resolution task, we obtained a gain of 8.06 F1 points on BF, 4.27 F1 points on BA and 11.87 F1 points on SFU, when trained on a single dataset. While training on multiple datasets, we observed a gain of 10.6 F1 points on BF and 1.94 F1 points on BA on the speculation detection task and 2.16 F1 points on BF and 0.25 F1 points on SFU on the scope resolution task over the single dataset training approach. We thus significantly advance the state-of-the-art for speculation detection and scope resolution. On the negation scope resolution task, we applied the XLNet and RoBERTa and obtained a gain of 3.16 F1 points on BF, 0.06 F1 points on BA and 0.3 F1 points on SFU. Thus, we demonstrated the usefulness of transformer-based architectures in the field of negation and speculation detection and scope resolution. We believe that a larger and more general dataset would go a long way in bolstering future research and would help create better systems that are not domain-specific.
Question: What is the size of SFU Review corpus?
Paper: Paraphrase-Supervised Models of Compositionality
Numerous lexical semantic properties are captured by representations encoding distributional properties of words, as has been demonstrated in a variety of tasks BIBREF0 , BIBREF1 , BIBREF2 . However, this distributional account of meaning does not scale to larger units like phrases and sentences BIBREF3 , BIBREF4 , motivating research into compositional models that combine word representations to produce representations of the semantics of longer units BIBREF5 , BIBREF6 , BIBREF7 . Previous work has learned these models using autoencoder formulations BIBREF8 or limited human supervision BIBREF5 . In this work, we explore the hypothesis that the equivalent knowledge about how words compose can be obtained through monolingual paraphrases that have been extracted using word alignments and an intermediate language BIBREF9 . Confirming this hypothesis would allow the rapid development of compositional models in a large number of languages.
Question: Do they study numerical properties of their obtained vectors (such as orthogonality)?
Paper: Paraphrase-Supervised Models of Compositionality
We formalize composition as a function INLINEFORM0 that maps INLINEFORM1 -dimensional vector representations of phrase constituents INLINEFORM2 to an INLINEFORM3 -dimensional vector representation of the phrase, i.e., the composed representation. A phrase is defined as any contiguous sequence of words of length 2 or greater, and does not have to adhere to constituents in a phrase structure grammar. This definition is in line with our MT application and ignores “gappy” noncontiguous phrases, but this pragmatic choice does exclude many verb-object relations BIBREF13 . We assume the existence of word-level vector representations for every word in our vocabulary of size INLINEFORM4 . Compositionality is modeled as a bilinear map, and two classes of linear models with different levels of parametrization are proposed. Unlike previous work BIBREF6 , BIBREF7 , BIBREF14 where the functions are word-specific, our compositional functions operate on part-of-speech (POS) tag pairs, which facilitates learning by drastically reducing the number of parameters, and only requires a shallow syntactic parse of the input.
Question: How do they score phrasal compositionality?
Paper: Paraphrase-Supervised Models of Compositionality
Our first class of models is a generalization of the additive models introduced in Mitchell2008: DISPLAYFORM0 
Question: Which translation systems do they compare against?
Paper: Automatic Judgment Prediction via Legal Reading Comprehension
Automatic judgment prediction is to train a machine judge to determine whether a certain plea in a given civil case would be supported or rejected. In countries with civil law system, e.g. mainland China, such process should be done with reference to related law articles and the fact description, as is performed by a human judge. The intuition comes from the fact that under civil law system, law articles act as principles for juridical judgments. Such techniques would have a wide range of promising applications. On the one hand, legal consulting systems could provide better access to high-quality legal resources in a low-cost way to legal outsiders, who suffer from the complicated terminologies. On the other hand, machine judge assistants for professionals would help improve the efficiency of the judicial system. Besides, automated judgment system can help in improving juridical equality and transparency. From another perspective, there are currently 7 times much more civil cases than criminal cases in mainland China, with annual rates of increase of INLINEFORM0 and INLINEFORM1 respectively, making judgment prediction in civil cases a promising application BIBREF0 .
Question: what are their results on the constructed dataset?
Paper: Automatic Judgment Prediction via Legal Reading Comprehension
Automatic judgment prediction has been studied for decades. At the very first stage of judgment prediction studies, researchers focus on mathematical and statistical analysis of existing cases, without any conclusions or methodologies on how to predict them BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 . Recent attempts consider judgment prediction under the text classification framework. Most of these works extract efficient features from text (e.g., N-grams) BIBREF15 , BIBREF4 , BIBREF1 , BIBREF16 , BIBREF17 or case profiles (e.g., dates, terms, locations and types) BIBREF2 . All these methods require a large amount of human effort to design features or annotate cases. Besides, they also suffer from generalization issue when applied to other scenarios.
Question: what evaluation metrics are reported?
Paper: Automatic Judgment Prediction via Legal Reading Comprehension
As the basis of previous judgment prediction works, typical text classification task takes a single text content as input and predicts the category it belongs to. Recent works usually employ neural networks to model the internal structure of a single input BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 .
Question: what civil field is the dataset about?
Paper: Automatic Judgment Prediction via Legal Reading Comprehension
Reading comprehension is a relevant task to model heterogeneous and complementary inputs, where an answer is predicted given two channels of inputs, i.e. a textual passage and a query. Considerable progress has been made BIBREF6 , BIBREF24 , BIBREF5 . These models employ various attention mechanism to model the interaction between passage and query. Inspired by the advantage of reading comprehension models on modeling multiple inputs, we apply this idea into the legal area and propose legal reading comprehension for judgment prediction.
Question: what are the state-of-the-art models?
Paper: Automatic Judgment Prediction via Legal Reading Comprehension
Conventional reading comprehension BIBREF25 , BIBREF26 , BIBREF7 , BIBREF8 usually considers reading comprehension as predicting the answer given a passage and a query, where the answer could be a single word, a text span of the original passage, chosen from answer candidates, or generated by human annotators.
Question: what is the size of the real-world civil case dataset?
Answer: 100 000 documents
Paper: Automatic Judgment Prediction via Legal Reading Comprehension
Existing works usually formalize judgment prediction as a text classification task and focus on extracting well-designed features of specific cases. Such simplification ignores that the judgment of a case is determined by its fact description and multiple pleas. Moreover, the final judgment should act up to the legal provisions, especially in civil law systems. Therefore, how to integrate the information (i.e., fact descriptions, pleas, and law articles) in a reasonable way is critical for judgment prediction.
Question: what datasets are used in the experiment?
Paper: A Framework for the Computational Linguistic Analysis of Dehumanization
computational sociolinguistics, dehumanization, lexical variation, language change, media, New York Times, LGBTQ
Question: Do they model semantics
Paper: A Framework for the Computational Linguistic Analysis of Dehumanization
Despite the American public's increasing acceptance of LGBTQ people and recent legal successes, LGBTQ individuals frequently remain the targets of hate and violence BIBREF0, BIBREF1, BIBREF2. At the core of this issue is dehumanization, “the act of perceiving or treating people as less than human” BIBREF3, a process that heavily contributes to extreme intergroup bias BIBREF4. Language is central to studying this phenomenon; like other forms of bias BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, dehumanizing attitudes are expressed through subtle linguistic manipulations, even in carefully-edited texts. It is crucial to understand the use of such linguistic signals in mainstream media, as the media's representation of marginalized social groups has far-reaching implications for social acceptance, policy, and safety.
Question: How do they identify discussions of LGBTQ people in the New York Times?
Paper: A Framework for the Computational Linguistic Analysis of Dehumanization
Our lexical semantic analysis involves quantifying linguistic correlates of component psychological processes that contribute to dehumanization. Our approaches are informed by social psychology research on dehumanization, which is briefly summarized here. Prior work has identified numerous related processes that comprise dehumanization BIBREF4. One such component is likening members of the target group to non-human entities, such as machines or animals BIBREF4, BIBREF11, BIBREF12. By perceiving members of a target group to be non-human, they are “outside the boundary in which moral values, rules, and considerations of fairness apply" BIBREF13, which thus leads to violence and other forms of abuse. Metaphors and imagery relating target groups to vermin are particularly insidious and played a prominent role in the genocide of Jews in Nazi Germany and Tutsis in Rwanda BIBREF14. More recently, the vermin metaphor has been invoked by the media to discuss terrorists and political leaders of majority-Muslim countries after September 11 BIBREF15. According to BIBREF16, the vermin metaphor is particularly powerful because it conceptualizes the target group as “engaged in threatening behavior, but devoid of thought or emotional desire".
Question: Do they analyze specific derogatory words?
Paper: Text Summarization with Pretrained Encoders
Language model pretraining has advanced the state of the art in many NLP tasks ranging from sentiment analysis, to question answering, natural language inference, named entity recognition, and textual similarity. State-of-the-art pretrained models include ELMo BIBREF1, GPT BIBREF2, and more recently Bidirectional Encoder Representations from Transformers (Bert; BIBREF0). Bert combines both word and sentence representations in a single very large Transformer BIBREF3; it is pretrained on vast amounts of text, with an unsupervised objective of masked language modeling and next-sentence prediction and can be fine-tuned with various task-specific objectives.
Question: What is novel about their document-level encoder?
Paper: Text Summarization with Pretrained Encoders
Pretrained language models BIBREF1, BIBREF2, BIBREF0, BIBREF12, BIBREF13 have recently emerged as a key technology for achieving impressive gains in a wide variety of natural language tasks. These models extend the idea of word embeddings by learning contextual representations from large-scale corpora using a language modeling objective. Bidirectional Encoder Representations from Transformers (Bert; BIBREF0) is a new language representation model which is trained with a masked language modeling and a “next sentence prediction” task on a corpus of 3,300M words.
Question: What rouge score do they achieve?
Answer: Best results on unigram:
CNN/Daily Mail: Rogue F1 43.85
NYT: Rogue Recall 49.02
XSum: Rogue F1 38.81
Highest scores for ROUGE-1, ROUGE-2 and ROUGE-L on CNN/DailyMail test set are 43.85, 20.34 and 39.90 respectively; on the XSum test set 38.81, 16.50 and 31.27 and on the NYT test set 49.02, 31.02 and 45.55
Paper: Text Summarization with Pretrained Encoders
Extractive summarization systems create a summary by identifying (and subsequently concatenating) the most important sentences in a document. Neural models consider extractive summarization as a sentence classification problem: a neural encoder creates sentence representations and a classifier predicts which sentences should be selected as summaries. SummaRuNNer BIBREF7 is one of the earliest neural approaches adopting an encoder based on Recurrent Neural Networks. Refresh BIBREF8 is a reinforcement learning-based system trained by globally optimizing the ROUGE metric. More recent work achieves higher performance with more sophisticated model structures. Latent BIBREF17 frames extractive summarization as a latent variable inference problem; instead of maximizing the likelihood of “gold” standard labels, their latent model directly maximizes the likelihood of human summaries given selected sentences. Sumo BIBREF18 capitalizes on the notion of structured attention to induce a multi-root dependency tree representation of the document while predicting the output summary. NeuSum BIBREF19 scores and selects sentences jointly and represents the state of the art in extractive summarization.
Question: What are the datasets used for evaluation?
Paper: Distant supervision for emotion detection using Facebook reactions
This work is licenced under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/
Question: What was their performance on emotion detection?
Answer: Answer with content missing: (Table 3) Best author's model B-M average micro f-score is 0.409, 0.459, 0.411 on Affective, Fairy Tales and ISEAR datasets respectively.
Paper: Distant supervision for emotion detection using Facebook reactions
For years, on Facebook people could leave comments to posts, and also “like” them, by using a thumbs-up feature to explicitly express a generic, rather underspecified, approval. A “like” could thus mean “I like what you said", but also “I like that you bring up such topic (though I find the content of the article you linked annoying)".
Question: Which existing benchmarks did they compare to?
Paper: Distant supervision for emotion detection using Facebook reactions
Three datasets annotated with emotions are commonly used for the development and evaluation of emotion detection systems, namely the Affective Text dataset, the Fairy Tales dataset, and the ISEAR dataset. In order to compare our performance to state-of-the-art results, we have used them as well. In this Section, in addition to a description of each dataset, we provide an overview of the emotions used, their distribution, and how we mapped them to those we obtained from Facebook posts in Section SECREF7 . A summary is provided in Table TABREF8 , which also shows, in the bottom row, what role each dataset has in our experiments: apart from the development portion of the Affective Text, which we used to develop our models (Section SECREF4 ), all three have been used as benchmarks for our evaluation.
Question: Which Facebook pages did they look at?