z-uo/qasper-squad huggingface dataset:
Paper: Satirical News Detection with Semantic Feature Extraction and Game-theoretic Rough Sets
Satirical news, which uses parody characterized in a conventional news style, has now become an entertainment on social media. While news satire is claimed to be pure comedic and of amusement, it makes statements on real events often with the aim of attaining social criticism and influencing change BIBREF0. Satirical news can also be misleading to readers, even though it is not designed for falsifications. Given such sophistication, satirical news detection is a necessary yet challenging natural language processing (NLP) task. Many feature based fake or satirical news detection systems BIBREF1, BIBREF2, BIBREF3 extract features from word relations given by statistics or lexical database, and other linguistic features. In addition, with the great success of deep learning in NLP in recent years, many end-to-end neural nets based detection systems BIBREF4, BIBREF5, BIBREF6 have been proposed and delivered promising results on satirical news article detection.
Question: How much improvement do they get?
Answer: Their GTRS approach got an improvement of 3.89% compared to SVM and 27.91% compared to Pawlak.
Paper: Satirical News Detection with Semantic Feature Extraction and Game-theoretic Rough Sets
Satirical news detection is an important yet challenging NLP task. Many feature based models have been proposed. Burfoot et al. extracted features of headline, profanity, and slang using word relations given by statistical metrics and lexical database BIBREF1. Rubin et al. proposed a SVM based model with five features (absurdity, humor, grammar, negative affect, and punctuation) for fake news document detection BIBREF2. Yang et al. presented linguistic features such as psycholinguistic feature based on dictionary and writing stylistic feature from part-of-speech tags distribution frequency BIBREF17. Shu et al. gave a survey in which a set of feature extraction methods is introduced for fake news on social media BIBREF3. Conroy et al. also uses social network behavior to detect fake news BIBREF18. For satirical sentence classification, Davidov et al. extract patterns using word frequency and punctuation features for tweet sentences and amazon comments BIBREF19. The detection of a certain type of sarcasm which contracts positive sentiment with a negative situation by analyzing the sentence pattern with a bootstrapped learning was also discussed BIBREF20. Although word level statistical features are widely used, with advanced word representations and state-of-the-art part-of-speech tagging and named entity recognition model, we observe that semantic features are more important than word level statistical features to model performance. Thus, we decompose the syntactic tree and use word vectors to more precisely capture the semantic inconsistencies in different structural parts of a satirical news tweet.
Question: How large is the dataset?
Paper: Satirical News Detection with Semantic Feature Extraction and Game-theoretic Rough Sets
In this section, we will describe the composition and preprocessing of our dataset and introduce our model in detail. We create our dataset by collecting legitimate and satirical news tweets from different news source accounts. Our model aims to detect whether the content of a news tweet is satirical or legitimate. We first extract the semantic features based on inconsistencies in different structural parts of the tweet sentences, and then use these features to train game-theoretic rough set decision model.
Question: What features do they extract?
Paper: Combining Acoustics, Content and Interaction Features to Find Hot Spots in Meetings
A definition of the meeting “hot spots” was first introduced in BIBREF2, where it was investigated whether human annotators could reliably identify regions in which participants are “highly involved in the discussion”. The motivation was that meetings generally have low information density and are tedious to review verbatim after the fact. An automatic system that could detect regions of high interest (as indicated by the involvement of the participants during the meeting) would thus be useful. Relatedly, automatic meeting summarization could also benefit from such information to give extra weight to hot spot regions in selecting or abstracting material for inclusion in the summary. Later work on the relationship between involvement and summarization BIBREF3 defined a different approach: hot spots are those regions chosen for inclusion in a summary by human annotators (“summarization hot spots”). In the present work we stick with the original “involvement hot spot” notion, and refer to such regions simply as “hot spots”, regardless of their possible role in summarization. We note that high involvement may be triggered both by a meeting's content (“what is being talked about”, and “what may be included in a textual summary”), as well as behavioral and social factors, such as a desire to participate, to stake out a position, or to oppose another participant. As related notion in dialog system research is “level of interest” BIBREF4.
Question: What they use as a metric of finding hot spots in meeting?
Paper: Combining Acoustics, Content and Interaction Features to Find Hot Spots in Meetings
The ICSI Meeting Corpus BIBREF11 is a collection of meeting recordings that has been thoroughly annotated, including annotations for involvement hot spots BIBREF12, linguistic utterance units, and word time boundaries based on forced alignment. The dataset is comprised of 75 meetings and about 70 hours of real-time audio duration, with 6 speakers per meeting on average. Most of the participants are well-acquainted and friendly with each other. Hot spots were originally annotated with 8 levels and degrees, ranging from `not hot' to `luke warm' to `hot +'. Every utterance was labeled with one of these discrete labels by a single annotator. Hightened involvement is rare, being marked on only 1% of utterances.
Question: Is this approach compared to some baseline?
Paper: Combining Acoustics, Content and Interaction Features to Find Hot Spots in Meetings
As stated above, the corpus was originally labeled for hot spots at the utterance level, where involvement was marked by either a `b' or a `b+' label. Training and test samples for our experiments correspond to 60 s-long sliding windows, with a 15 s step size. If a certain window, e.g., a segment spanning the times 15 s ...75 s, overlaps with any involved speech utterance, then we label that whole window as `hot'. Fig. FIGREF6 gives a visual representation.
Question: How big is ICSI meeting corpus?
Paper: Combining Acoustics, Content and Interaction Features to Find Hot Spots in Meetings
In spite of the windowing approach, the class distribution is still skewed, and an accuracy metric would reflect the particular class distribution in our data set. Therefore, we adopt the unweighted average recall (UAR) metric commonly used in emotion classification research. UAR is a reweighted accuracy where the samples of both classes are weighted equally in aggregate. UAR thus simulates a uniform class distribution. To match the objective, our classifiers are trained on appropriately weighted training data. Note that chance performance for UAR is by definition 50%, making results more comparable across different data sets.
Question: What annotations are available in ICSI meeting corpus?
Paper: Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference
Natural Language Inference (NLI) is often used to gauge a model's ability to understand a relationship between two texts BIBREF0 , BIBREF1 . In NLI, a model is tasked with determining whether a hypothesis (a woman is sleeping) would likely be inferred from a premise (a woman is talking on the phone). The development of new large-scale datasets has led to a flurry of various neural network architectures for solving NLI. However, recent work has found that many NLI datasets contain biases, or annotation artifacts, i.e., features present in hypotheses that enable models to perform surprisingly well using only the hypothesis, without learning the relationship between two texts BIBREF2 , BIBREF3 , BIBREF4 . For instance, in some datasets, negation words like “not” and “nobody” are often associated with a relationship of contradiction. As a ramification of such biases, models may not generalize well to other datasets that contain different or no such biases.
Question: Is such bias caused by bad annotation?
Paper: Good-Enough Compositional Data Augmentation
This paper proposes a data augmentation protocol for sequence modeling problems. Our approach aims to supply a simple and model-agnostic bias toward compositional reuse of previously observed sequence fragments in novel environments. Consider a language modeling task in which we wish to estimate a probability distribution over a family of sentences with the following finite sample as training data:
Question: How do they determine similar environments for fragments in their data augmentation scheme?
Paper: Good-Enough Compositional Data Augmentation
Recent years have seen tremendous success at natural language transduction and generation tasks using black-box function approximators, especially recurrent BIBREF9 and attentional BIBREF10 neural models. With enough training data, these models are often more accurate than than approaches built on traditional tools from the computational linguistics literature—formal models like regular transducers or context-free grammars BIBREF11 can be brittle and challenging to efficiently infer from large datasets.
Question: Do they experiment with language modeling on large datasets?
Paper: Good-Enough Compositional Data Augmentation
Consider again the example in fig:teaser. Our data augmentation protocol aims to discover substitutable sentence fragments (highlighted), with the fact a pair of fragments appear in some common sub-sentential environment (underlined) taken as evidence that the fragments belong to a common category. To generate a new examples for the model, an occurrence of one fragment is removed from a sentence to produce a sentence template, which is then populated with the other fragment.
Question: Which languages do they test on?
Answer: Answer with content missing: (Applications section) We use Wikipedia articles
in five languages
(Kinyarwanda, Lao, Pashto, Tok Pisin, and a subset of English) as well as the Na dataset of Adams
et al. (2017).
Select:
Kinyarwanda, Lao, Pashto, Tok Pisin, and a subset of English
Paper: DpgMedia2019: A Dutch News Dataset for Partisanship Detection
In a survey across 38 countries, the Pew Research Center reported that the global public opposed partisanship in news media BIBREF0 . It is, however, challenging to assess the partisanship of news articles on a large scale. We thus made an effort to create a dataset of articles annotated with political partisanship so that content analysis systems can benefit from it.
Question: What limitations are mentioned?
Answer: deciding publisher partisanship, risk annotator bias because of short description text provided to annotators
Paper: DpgMedia2019: A Dutch News Dataset for Partisanship Detection
DpgMedia2019 is a Dutch dataset that was collected from the publications within DPG Media. We took 11 publishers in the Netherlands for the dataset. These publishers include 4 national publishers, Algemeen Dagblad (AD), de Volkskrant (VK), Trouw, and Het Parool, and 7 regional publishers, de Gelderlander, Tubantia, Brabants Dagblad, Eindhovens Dagblad, BN/De Stem PZC, and de Stentor. The regional publishers are collectively called Algemeen Dagblad Regionaal (ADR). A summary of the dataset is shown in Table TABREF3 .
Question: What examples of applications are mentioned?
Paper: DpgMedia2019: A Dutch News Dataset for Partisanship Detection
We used an internal database that stores all articles written by journalists and ready to be published to collect the articles. From the database, we queried all articles that were published between 2017 and 2019. We filtered articles to be non-advertisement. We also filtered on the main sections so that the articles were not published under the sports and entertainment sections, which we assumed to be less political. After collecting, we found that a lot of the articles were published by several publishers, especially a large overlap existed between AD and ADR. To deal with the problem without losing many articles, we decided that articles that appeared in both AD and its regional publications belonged to AD. Therefore, articles were processed in the following steps:
Question: Did they crowdsource the annotations?
Paper: Task-Oriented Language Grounding for Language Input with Multiple Sub-Goals of Non-Linear Order
Task-oriented language grounding refers to the process of extracting semantically meaningful representations of language by mapping it to visual elements and actions in the environment in order to perform the task specified by the instruction BIBREF0.
Question: Why they conclude that the usage of Gated-Attention provides no competitive advantage against concatenation in this setting?
Paper: CAIL2019-SCM: A Dataset of Similar Case Matching in Legal Domain
Similar Case Matching (SCM) plays a major role in legal system, especially in common law legal system. The most similar cases in the past determine the judgment results of cases in common law systems. As a result, legal professionals often spend much time finding and judging similar cases to prove fairness in judgment. As automatically finding similar cases can benefit to the legal system, we select SCM as one of the tasks of CAIL2019.
Question: What was the best team's system?
Paper: CAIL2019-SCM: A Dataset of Similar Case Matching in Legal Domain
We first define the task of CAIL2019-SCM here. The input of CAIL2019-SCM is a triplet $(A,B,C)$, where $A,B,C$ are fact descriptions of three cases. Here we define a function $sim$ which is used for measuring the similarity between two cases. Then the task of CAIL2019-SCM is to predict whether $sim(A,B)>sim(A,C)$ or $sim(A,C)>sim(A,B)$.
Question: What are the baselines?
Answer: CNN, LSTM, BERT
Paper: Limits of Detecting Text Generated by Large-Scale Language Models
Building on a long history of language generation models that are based on statistical knowledge that people have BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, large-scale, neural network-based language models (LMs) that write paragraph-length text with the coherence of human writing have emerged BIBREF6, BIBREF7, BIBREF8. Such models have raised concerns about misuse in generating fake news, misleading reviews, and hate speech BIBREF9, BIBREF10, BIBREF8, BIBREF11, BIBREF12. The alarming consequences of such machine-generated misinformation present an urgent need to discern fake content from genuine, as it is becoming more and more difficult for people to do so without cognitive support tools BIBREF13. Several recent studies have used supervised learning to develop classifiers for this task BIBREF8, BIBREF14, BIBREF9, BIBREF15, BIBREF16 and interpreted their properties. Here we take inspiration from our recent work on information-theoretic limits for detecting audiovisual deepfakes generated by GANs BIBREF17 to develop information-theoretic limits for detecting the outputs of language models. In particular, we build on the information-theoretic study of authentication BIBREF18 to use a formal hypothesis testing framework for detecting the outputs of language models.
Question: What semantic features help in detecting whether a piece of text is genuine or generated? of 
Answer: No feature is given, only discussion that semantic features are use in practice and yet to be discovered how to embed that knowledge into statistical decision theory framework.
Paper: Limits of Detecting Text Generated by Large-Scale Language Models
Consider a language $L$ like English, which has tokens drawn from a finite alphabet $\mathcal {A}$; tokens can be letters, words, or other such symbols. A language model assigns probabilities to sequences of tokens $(a_1,a_2,\ldots ,a_m)$ so the more likely a sequence is in $L$, the greater its probability. Language models discussed in Sec. SECREF1 estimate this probability $Q$ as a product of each token's probability $q$ given its preceding tokens:
Question: Which language models generate text that can be easier to classify as genuine or generated?
Paper: Limits of Detecting Text Generated by Large-Scale Language Models
Recall that the distribution of authentic text is denoted $P$ and the distribution of text generated by the language model is $Q$. Suppose we have access to $n$ tokens of generated text from the language model, which we call $Y_1, Y_2, Y_3, \ldots , Y_n$. We can then formalize a hypothesis test as:
Question: Is the assumption that natural language is stationary and ergodic valid?
Answer: It is not completely valid for natural languages because of diversity of language - this is called smoothing requirement.
Paper: ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
[color=red!20,size=,fancyline,caption=,disable]ben:It is a little weird that RECORD is not spelled out in the abstract, but especially odd that it isn't spelled out in the Introduction. I would remove the footnote, put that content in the Introduction
Question: Which models do they try out?
Answer: DocQA, SAN, QANet, ASReader, LM, Random Guess
Paper: Small-Footprint Keyword Spotting on Raw Audio Data with Sinc-Convolutions
Speech processing enables natural communication with smart phones or smart home assistants, e.g., Amazon Echo, Google Home. However, continuously performing speech recognition is not energy-efficient and would drain batteries of smart devices. Instead, most speech recognition systems passively listen for utterances of certain wake words such as “Ok Google", “Hey Siri", “Alexa", etc. to trigger the continuous speech recognition system on demand. This task is referred to as keyword spotting (KWS). There are also uses of KWS where a view simple speech commands (e.g. “on", “off") are enough to interact with a device such as a voice-controlled light bulb.
Question: Do they compare executionttime of their model against other models?
Paper: Small-Footprint Keyword Spotting on Raw Audio Data with Sinc-Convolutions
Recently, CNNs have been successfully applied to KWS BIBREF1, BIBREF2, BIBREF3. Zhang et al. evaluated different neural network architectures (such as CNNs, LSTMs, GRUs) in terms of accuracy, computational operations and memory footprint as well as their deployment on embedded hardware BIBREF1. They achieved their best results using a CNN with DSConvs. Tang et al. explored the use of Deep Residual Networks with dilated convolutions to achieve a high accuracy of $95.8\%$ BIBREF2, while keeping the number of parameters comparable to BIBREF1. Choi et al. build on this work as they also use a ResNet-inspired architecture. Instead of using 2D convolution over a time-frequency representation of the data they convolve along the time dimension and treat the frequency dimension as channels BIBREF3.
Question: What is the memory footprint decrease of their model in comparison to other models?
Paper: Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks
Any finite training set is consistent with multiple generalizations. Therefore, the way that a learner generalizes to unseen examples depends not only on the training data but also on properties of the learner. Suppose a learner is told that a blue triangle is an example of a blick. A learner preferring shape-based generalizations would conclude that blick means “triangle,” while a learner preferring color-based generalizations would conclude that blick means “blue object” BIBREF0. Factors that guide a learner to choose one generalization over another are called inductive biases.
Question: What architectural factors were investigated?
Paper: Racial Bias in Hate Speech and Abusive Language Detection Datasets
Recent work has shown evidence of substantial bias in machine learning systems, which is typically a result of bias in the training data. This includes both supervised BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 and unsupervised natural language processing systems BIBREF4 , BIBREF5 , BIBREF6 . Machine learning models are currently being deployed in the field to detect hate speech and abusive language on social media platforms including Facebook, Instagram, and Youtube. The aim of these models is to identify abusive language that directly targets certain individuals or groups, particularly people belonging to protected categories BIBREF7 . Bias may reduce the accuracy of these models, and at worst, will mean that the models actively discriminate against the same groups they are designed to protect.
Question: Any other bias may be detected?
Paper: Think Globally, Embed Locally --- Locally Linear Meta-embedding of Words
Representing the meanings of words is a fundamental task in Natural Language Processing (NLP). One popular approach to represent the meaning of a word is to embed it in some fixed-dimensional vector space (). In contrast to sparse and high-dimensional counting-based distributional word representation methods that use co-occurring contexts of a word as its representation (), dense and low-dimensional prediction-based distributed word representations have obtained impressive performances in numerous NLP tasks such as sentiment classification (), and machine translation (). Several distributed word embedding learning methods based on different learning strategies have been proposed (;;;;).
Question: What is the introduced meta-embedding method introduced in this paper?
Paper: Simultaneous Speech Recognition and Speaker Diarization for Monaural Dialogue Recordings with Target-Speaker Acoustic Models
Our main goal is to develop a monaural conversation transcription system that can not only perform automatic speech recognition (ASR) of multiple talkers but also determine who spoke the utterance when, known as speaker diarization BIBREF0, BIBREF1. For both ASR and speaker diarization, the main difficulty comes from speaker overlaps. For example, a speaker-overlap ratio of about 15% was reported in real meeting recordings BIBREF2. For such overlapped speech, neither conventional ASR nor speaker diarization provides a result with sufficient accuracy. It is known that mixing two speech significantly degrades ASR accuracy BIBREF3, BIBREF4, BIBREF5. In addition, no speaker overlaps are assumed with most conventional speaker diarization techniques, such as clustering of speech partitions (e.g. BIBREF0, BIBREF6, BIBREF7, BIBREF8, BIBREF9), which works only if there are no speaker overlaps. Due to these difficulties, it is still very challenging to perform ASR and speaker diarization for monaural recordings of conversation.
Question: How long are dialogue recordings used for evaluation?
Paper: Modeling Global Syntactic Variation in English Using Dialect Classification
This paper combines grammar induction (Dunn, 2018a, 2018b, 2019) and text classification (Joachims, 1998) to model syntactic variation across national varieties of English. This classification-based approach is situated within the task of dialect identification (Section 2) and evaluated against other baselines for the task (Sections 7 and 8). But the focus is modelling syntactic variation on a global-scale using corpus data. On the one hand, the problem is to use a model of syntactic preferences to predict an author's dialect membership (Dunn, 2018c). On the other hand, the problem is to take a spatially-generic grammar of English that is itself learned from raw text (c.f., Zeman, et al., 2017; Zeman, et al., 2018) and adapt that grammar using dialect identification as an optimization task: which constructions are more likely to occur in a specific regional variety?
Question: What do the models that they compare predict?