z-uo/qasper-squad huggingface dataset:
Paper: Generating Diverse Numbers of Diverse Keyphrases
Keyphrase generation is the task of automatically predicting keyphrases given a source text. Desired keyphrases are often multi-word units that summarize the high-level meaning and highlight certain important topics or information of the source text. Consequently, models that can successfully perform this task should be capable of not only distilling high-level information from a document, but also locating specific, important snippets therein.
Question: How is keyphrase diversity measured?
Paper: Generating Diverse Numbers of Diverse Keyphrases
Traditional keyphrase extraction has been studied extensively in past decades. In most existing literature, keyphrase extraction has been formulated as a two-step process. First, lexical features such as part-of-speech tags are used to determine a list of phrase candidates by heuristic methods BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 . Second, a ranking algorithm is adopted to rank the candidate list and the top ranked candidates are selected as keyphrases. A wide variety of methods were applied for ranking, such as bagged decision trees BIBREF8 , BIBREF9 , Multi-Layer Perceptron, Support Vector Machine BIBREF9 and PageRank BIBREF10 , BIBREF11 , BIBREF12 . Recently, BIBREF13 , BIBREF14 , BIBREF15 used sequence labeling models to extract keyphrases from text. Similarly, BIBREF16 used Pointer Networks to point to the start and end positions of keyphrases in a source text.
Question: How was the StackExchange dataset collected?
Answer: they obtained computer science related topics by looking at titles and user-assigned tags
Paper: Generating Diverse Numbers of Diverse Keyphrases
Sequence to Sequence (Seq2Seq) learning was first introduced by BIBREF17 ; together with the soft attention mechanism of BIBREF18 , it has been widely used in natural language generation tasks. BIBREF19 , BIBREF20 used a mixture of generation and pointing to overcome the problem of large vocabulary size. BIBREF21 , BIBREF22 applied Seq2Seq models on summary generation tasks, while BIBREF23 , BIBREF24 generated questions conditioned on documents and answers from machine comprehension datasets. Seq2Seq was also applied on neural sentence simplification BIBREF25 and paraphrase generation tasks BIBREF26 .
Question: What does the TextWorld ACG dataset contain?
Paper: Generating Diverse Numbers of Diverse Keyphrases
Given a piece of source text, our objective is to generate a variable number of multi-word phrases. To this end, we opt for the sequence-to-sequence framework (Seq2Seq) as the basis of our model, combined with attention and pointer softmax mechanisms in the decoder.
Question: What is the size of the StackExchange dataset?
Answer: around 332k questions
Paper: Generating Diverse Numbers of Diverse Keyphrases
In the following subsections, we use INLINEFORM0 to denote input text tokens, INLINEFORM1 to denote token embeddings, INLINEFORM2 to denote hidden states, and INLINEFORM3 to denote output text tokens. Superscripts denote time-steps in a sequence, and subscripts INLINEFORM4 and INLINEFORM5 indicate whether a variable resides in the encoder or the decoder of the model, respectively. The absence of a superscript indicates multiplicity in the time dimension. INLINEFORM6 refers to a linear transformation and INLINEFORM7 refers to it followed by a non-linear activation function INLINEFORM8 . Angled brackets, INLINEFORM9 , denote concatenation.
Question: What were the baselines?
Answer: CopyRNN (Meng et al., 2017), Multi-Task (Ye and Wang, 2018), and TG-Net (Chen et al., 2018b)
Paper: Generating Diverse Numbers of Diverse Keyphrases
There are usually multiple keyphrases for a given source text because each keyphrase represents certain aspects of the text. Therefore keyphrase diversity is desired for the keyphrase generation. Most previous keyphrase generation models generate multiple phrases by over-generation, which is highly prone to generate similar phrases due to the nature of beam search. Given our objective to generate variable numbers of keyphrases, we need to adopt new strategies for achieving better diversity in the output.
Question: What two metrics are proposed?
Paper: The Fine Line between Linguistic Generalization and Failure in Seq2Seq-Attention Models
It is well known that language has certain structural properties which allows natural language speakers to make “infinite use of finite means" BIBREF3 . This structure allows us to generalize beyond the typical machine learning definition of generalization BIBREF4 (which considers performance on the distribution that generated the training set), permitting the understanding of any utterance sharing the same structure, regardless of probability. For example, sentences of length 100 typically do not appear in natural text or speech (our personal 'training set'), but can be understood regardless due to their structure. We refer to this notion as linguistic generalization .
Question: Can the findings of this paper be generalized to a general-purpose task?
Paper: The Fine Line between Linguistic Generalization and Failure in Seq2Seq-Attention Models
Real world NLP tasks are complex, and as such, it can be difficult to precisely define what a model should and should not learn during training. As done in previous work BIBREF8 , BIBREF9 , we ease analysis by looking at a simple formal task. The task is set up to mimic (albeit, in an oversimplified manner) the input-output symbol alignments and local syntactic properties that models must learn in many natural language tasks, such as translation, tagging and summarization.
Question: Why does the proposed task a good proxy for the general-purpose sequence to sequence tasks?
Paper: Measuring Offensive Speech in Online Political Discourse
The apparent rise in political incivility has attracted substantial attention from scholars in recent years. These studies have largely focused on the extent to which politicians and elected officials are increasingly employing rhetoric that appears to violate norms of civility BIBREF0 , BIBREF1 . For the purposes of our work, we use the incidence of offensive rhetoric as a stand in for incivility. The 2016 US presidential election was an especially noteworthy case in this regard, particularly in terms of Donald Trump's campaign which frequently violated norms of civility both in how he spoke about broad groups in the public (such as Muslims, Mexicans, and African Americans) and the attacks he leveled at his opponents BIBREF2 . The consequences of incivility are thought to be crucial to the functioning of democracy since “public civility and interpersonal politeness sustain social harmony and allow people who disagree with one another to maintain ongoing relationships" BIBREF3 .
Question: What was the baseline?
Paper: Measuring Offensive Speech in Online Political Discourse
Our study makes use of multiple datasets in order to identify and characterize trends in offensive speech.
Question: What was their system's performance?
Paper: Measuring Offensive Speech in Online Political Discourse
In order to identify offensive speech, we propose a fully automated technique that classifies comments into two classes: Offensive and Not Offensive.
Question: What other political events are included in the database?
Paper: Measuring Offensive Speech in Online Political Discourse
At a high-level, our approach works as follows:
Question: What classifier did they use?
Paper: Trouble on the Horizon: Forecasting the Derailment of Online Conversations as they Develop
“Ché saetta previsa vien più lenta.”
Question: What labels for antisocial events are available in datasets?
Answer: The Conversations Gone Awry dataset is labelled as either containing a personal attack from withint (i.e. hostile behavior by one user in the conversation directed towards another) or remaining civil throughout. The Reddit Change My View dataset is labelled with whether or not a coversation eventually had a comment removed by a moderator for violation of Rule 2: "Don't be rude or hostile to others users."
Paper: Trouble on the Horizon: Forecasting the Derailment of Online Conversations as they Develop
Question: What are two datasets model is applied to?
Answer: An expanded version of the existing 'Conversations Gone Awry' dataset and the ChangeMyView dataset, a subreddit whose only annotation is whether the conversation required action by the Reddit moderators.
Paper: Identifying Radiological Findings Related to COVID-19 from Medical Literature
Coronavirus disease 2019 (COVID-19) is an infectious disease that has affected more than one million individuals all over the world and caused more than 55,000 deaths, as of April 3 in 2020. The science community has been working very actively to understand this new disease and make diagnosis and treatment guidelines based on the findings. One major stream of efforts are focused on discovering the correlation between radiological findings in the lung areas and COVID-19. There have been several works BIBREF0, BIBREF1 publishing such results. However, existing studies are mostly conducted separately by different hospitals and medical institutes. Due to geographic affinity, the populations served by different hospitals have different genetic, social, and ethnic characteristics. As a result, the radiological findings from COVID-19 patient cases in different populations are different. This population bias incurs inconsistent or even conflicting conclusions regarding the correlation between radiological findings and COVID-19. As a result, medical professionals cannot make informed decisions on how to use radiological findings to guide diagnosis and treatment of COVID-19.
Question: What is the CORD-19 dataset?
Paper: Identifying Radiological Findings Related to COVID-19 from Medical Literature
We used the COVID-19 Open Research Dataset (CORD-19) BIBREF2 for our study. In response to the COVID-19 pandemic, the White House and a coalition of research groups prepared the CORD-19 dataset. It contains over 45,000 scholarly articles, including over 33,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. These articles are contributed by hospitals and medical institutes all over the world. Since the outbreak of COVID-19 is after November 2019, we select articles published after November 2019 to study, which include a total of 2081 articles and about 360000 sentences. Many articles report the radiological findings related to COVID-19. Table TABREF4 shows some examples.
Question: How large is the collection of COVID-19 literature?
Paper: Semi-supervised Thai Sentence Segmentation Using Local and Distant Word Representations
Automatic summarization, machine translation, question answering, and semantic parsing operations are useful for processing, analyzing, and extracting meaningful information from text. However, when applied to long texts, these tasks usually require some minimal syntactic structure to be identified, such as sentences BIBREF0 , BIBREF1 , BIBREF2 , which always end with a period (“.”) in English BIBREF3 .
Question: Which deep learning architecture do they use for sentence segmentation?
Paper: Semi-supervised Thai Sentence Segmentation Using Local and Distant Word Representations
This section includes three subsections. The first subsection concerns Thai sentence segmentation, which is the main focus of this work. The task of English punctuation restoration, which is similar to our main task, is described in the second subsection. The last subsection describes the original Cross-View Training initially proposed in BIBREF20 .
Question: How do they utilize unlabeled data to improve model representations?
Paper: A Surprising Density of Illusionable Natural Speech
A growing body of work on adversarial examples has identified that for machine-learning (ML) systems that operate on high-dimensional data, for nearly every natural input there exists a small perturbation of the point that will be misclassified by the system, posing a threat to its deployment in certain critical settings BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 . More broadly, the susceptibility of ML systems to adversarial examples has prompted a re-examination of whether current ML systems are truly learning or if they are assemblages of tricks that are effective yet brittle and easily fooled BIBREF9 . Implicit in this line of reasoning is the assumption that instances of ”real" learning, such as human cognition, yield extremely robust systems. Indeed, at least in computer vision, human perception is regarded as the gold-standard for robustness to adversarial examples.
Question: What is the McGurk effect?
Answer: a perceptual illusion, where listening to a speech sound while watching a mouth pronounce a different sound changes how the audio is heard
When the perception of what we hear is influenced by what we see.
Paper: A Surprising Density of Illusionable Natural Speech
Illusionable instances for humans are similar to adversarial examples for ML systems. Strictly speaking, however, our investigation of the density of natural language for which McGurk illusions can be created, is not the human analog of adversarial examples. The adversarial examples for ML systems are datapoints that are misclassified, despite being extremely similar to a typical datapoint (that is correctly classified). Our illusions of misdubbed audio are not extremely close to any typically encountered input, since our McGurk samples have auditory signals corresponding to one phoneme/word and visual signals corresponding to another. Also, there is a compelling argument for why the McGurk confusion occurs, namely that human speech perception is bimodal (audio-visual) in nature when lip reading is available BIBREF20 , BIBREF21 . To the best of our knowledge, prior to our work, there has been little systematic investigation of the extent to which the McGurk effect, or other types of illusions, can be made dense in the set of instances encountered in everyday life. The closest work is BIBREF22 , where the authors demonstrate that some adversarial examples for computer vision systems also fool humans when humans were given less than a tenth of second to view the image. However, some of these examples seem less satisfying as the perturbation acts as a pixel-space interpolation between the original image and the “incorrect” class. This results in images that are visually borderline between two classes, and as such, do not provide a sense of illusion to the viewer. In general, researchers have not probed the robustness of human perception with the same tools, intent, or perspective, with which the security community is currently interrogating the robustness of ML systems.
Question: Are humans and machine learning systems fooled by the same kinds of illusions?
Paper: Context-Aware Monolingual Repair for Neural Machine Translation
Machine translation has made remarkable progress, and studies claiming it to reach a human parity are starting to appear BIBREF0. However, when evaluating translations of the whole documents rather than isolated sentences, human raters show a stronger preference for human over machine translation BIBREF1. These findings emphasize the need to shift towards context-aware machine translation both from modeling and evaluation perspective.
Question: how many humans evaluated the results?
Paper: Context-Aware Monolingual Repair for Neural Machine Translation
We propose a monolingual DocRepair model to correct inconsistencies between sentence-level translations of a context-agnostic MT system. It does not use any states of a trained MT model whose outputs it corrects and therefore can in principle be trained to correct translations from any black-box MT system.
Question: what was the baseline?
Paper: Context-Aware Monolingual Repair for Neural Machine Translation
We use contrastive test sets for evaluation of discourse phenomena for English-Russian by BIBREF11. These test sets allow for testing different kinds of phenomena which, as we show, can be captured from monolingual data with varying success. In this section, we provide test sets statistics and briefly describe the tested phenomena. For more details, the reader is referred to BIBREF11.
Question: what phenomena do they mention is hard to capture?
Answer: Four discourse phenomena - deixis, lexical cohesion, VP ellipsis, and ellipsis which affects NP inflection.
Paper: Context-Aware Monolingual Repair for Neural Machine Translation
There are four test sets in the suite. Each test set contains contrastive examples. It is specifically designed to test the ability of a system to adapt to contextual information and handle the phenomenon under consideration. Each test instance consists of a true example (a sequence of sentences and their reference translation from the data) and several contrastive translations which differ from the true one only in one specific aspect. All contrastive translations are correct and plausible translations at the sentence level, and only context reveals the inconsistencies between them. The system is asked to score each candidate translation, and we compute the system accuracy as the proportion of times the true translation is preferred to the contrastive ones. Test set statistics are shown in Table TABREF15. The suites for deixis and lexical cohesion are split into development and test sets, with 500 examples from each used for validation purposes and the rest for testing. Convergence of both consistency scores on these development sets and BLEU score on a general development set are used as early stopping criteria in models training. For ellipsis, there is no dedicated development set, so we evaluate on all the ellipsis data and do not use it for development.
Question: by how much did the BLEU score improve?
Answer: On average 0.64
Paper: A Biomedical Information Extraction Primer for NLP Researchers
The explosion of available scientific articles in the Biomedical domain has led to the rise of Biomedical Information Extraction (BioIE). BioIE systems aim to extract information from a wide spectrum of articles including medical literature, biological literature, electronic health records, etc. that can be used by clinicians and researchers in the field. Often the outputs of BioIE systems are used to assist in the creation of databases, or to suggest new paths for research. For example, a ranked list of interacting proteins that are extracted from biomedical literature, but are not present in existing databases, can allow researchers to make informed decisions about which protein/gene to study further. Interactions between drugs are necessary for clinicians who simultaneously administer multiple drugs to their patients. A database of diseases, treatments and tests is beneficial for doctors consulting in complicated medical cases.
Question: What is NER?
Answer: Named Entity Recognition, including entities such as proteins, genes, diseases, treatments, drugs, etc. in the biomedical domain
Paper: A Biomedical Information Extraction Primer for NLP Researchers
Named Entity Recognition (NER) in the Biomedical domain usually includes recognition of entities such as proteins, genes, diseases, treatments, drugs, etc. Fact extraction involves extraction of Named Entities from a corpus, usually given a certain ontology. When compared to NER in the domain of general text, the biomedical domain has some characteristic challenges:
Question: Does the paper explore extraction from electronic health records?
Paper: jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models
This paper introduces jiant, an open source toolkit that allows researchers to quickly experiment on a wide array of NLU tasks, using state-of-the-art NLP models, and conduct experiments on probing, transfer learning, and multitask training. jiant supports many state-of-the-art Transformer-based models implemented by Huggingface's Transformers package, as well as non-Transformer models such as BiLSTMs.
Question: Does jiant involve datasets for the 50 NLU tasks?
Paper: jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models
Transfer learning is an area of research that uses knowledge from pretrained models to transfer to new tasks. In recent years, Transformer-based models like BERT BIBREF17 and T5 BIBREF18 have yielded state-of-the-art results on the lion's share of benchmark tasks for language understanding through pretraining and transfer, often paired with some form of multitask learning.
Question: Is jiant compatible with models in any programming language?
Paper: Prose for a Painting
Neural networks have been successfully used to describe images with text using sequence-to-sequence models BIBREF0. However, the results are simple and dry captions which are one or two phrases long. Humans looking at a painting see more than just objects. Paintings stimulate sentiments, metaphors and stories as well. Therefore, our goal is to have a neural network describe the painting artistically in a style of choice. As a proof of concept, we present a model which generates Shakespearean prose for a given painting as shown in Figure FIGREF1. Accomplishing this task is difficult with traditional sequence to sequence models since there does not exist a large collection of Shakespearean prose which describes paintings: Shakespeare's works describes a single painting shown in Figure FIGREF3. Fortunately we have a dataset of modern English poems which describe images BIBREF1 and line-by-line modern paraphrases of Shakespeare's plays BIBREF2. Our solution is therefore to combine two separately trained models to synthesize Shakespearean prose for a given painting.
Question: What models are used for painting embedding and what for language style transfer?
Paper: Prose for a Painting
A general end-to-end approach to sequence learning BIBREF3 makes minimal assumptions on the sequence structure. This model is widely used in tasks such as machine translation, text summarization, conversational modeling, and image captioning. A generative model using a deep recurrent architecture BIBREF0 has also beeen used for generating phrases describing an image. The task of synthesizing multiple lines of poetry for a given image BIBREF1 is accomplished by extracting poetic clues from images. Given the context image, the network associates image attributes with poetic descriptions using a convolutional neural net. The poem is generated using a recurrent neural net which is trained using multi-adversarial training via policy gradient.
Question: What applicability of their approach is demonstrated by the authors?
Paper: Prose for a Painting
We use a total three datasets: two datasets for generating an English poem from an image, and Shakespeare plays and their English translations for text style transfer.
Question: What limitations do the authors demnostrate of their model?
Paper: Prose for a Painting
For generating a poem from images we use an existing actor-critic architecture BIBREF1. This involves 3 parallel CNNs: an object CNN, sentiment CNN, and scene CNN, for feature extraction. These features are combined with a skip-thought model which provides poetic clues, which are then fed into a sequence-to-sequence model trained by policy gradient with 2 discriminator networks for rewards. This as a whole forms a pipeline that takes in an image and outputs a poem as shown on the top left of Figure FIGREF4. A CNN-RNN generative model acts as an agent. The parameters of this agent define a policy whose execution determines which word is selected as an action. When the agent selects all words in a poem, it receives a reward. Two discriminative networks, shown on the top right of Figure FIGREF4, are defined to serve as rewards concerning whether the generated poem properly describes the input image and whether the generated poem is poetic. The goal of the poem generation model is to generate a sequence of words as a poem for an image to maximize the expected return.
Question: How does final model rate on Likert scale?