z-uo/qasper-squad huggingface dataset:
Paper: Frozen Binomials on the Web: Word Ordering and Language Conventions in Online Text
Previous work has one main measure of binomials — their `frozen-ness'. A binomial is `frozen' if it always appears with a particular order. For example, if the pair {`arrow', `bow'} always occurs as [`bow', `arrow'] and never as [`arrow', `bow'], then it is frozen. This leaves open the question of how describe the large number of binomials that are not frozen. To address this point, we instead consider the ordinality of a list, or how often the list is `in order' according to some arbitrary underlying reference order. Unless otherwise specified, the underlying order is assumed to be alphabetical. If the list [`cat', `dog'] appears 40 times and the list [`dog', `cat'] 10 times, then the list {`cat', `dog'} would have an ordinality of 0.8.
Question: How was performance of previously proposed rules at very large scale?
Paper: Frozen Binomials on the Web: Word Ordering and Language Conventions in Online Text
Let the point $(A,M,G)_{x,y}$ be a vector of the asymmetry, movement, and agreement for some unordered list $\lbrace x,y\rbrace $. These vectors then define a 3-dimensional space in which each list occupies a point. Since our measures for asymmetry, agreement, and movement are all defined from 0 to 1, their domains form a unit cube (Fig. FIGREF8). The corners of this cube correspond to points with coordinates are entirely made up of 0s or 1s. By examining points near the corners of this cube, we can get a better understanding of the range of binomials. Some corners are natural — it is easy to imagine a high asymmetry, low movement, high agreement binomial — such as {`arrow', `bow'} from earlier. On the other hand, we have found no good examples of a high asymmetry, low movement, low agreement binomial. There are a few unusual examples, such as {10, 20}, which has 0.4 asymmetry, 0.2 movement, and 0.1 agreement and is clearly visible as an isolated point in Fig. FIGREF8.
Question: What previously proposed rules for predicting binoial ordering are used?
Paper: Frozen Binomials on the Web: Word Ordering and Language Conventions in Online Text
In this section, we establish a null model under which different communities or time slices have the same probability of ordering a binomial in a particular way. With this, we would expect to see variation in binomial asymmetry. We find that our data shows smaller variation than this null model predicts, suggesting that binomial orderings are extremely stable across communities and time. From this, we might also expect that orderings are predictable; but we find that standard predictors in fact have limited success.
Question: What online text resources are used to test binomial lists?
Paper: Casting Light on Invisible Cities: Computationally Engaging with Literary Criticism
Literary critics form interpretations of meaning in works of literature. Building computational models that can help form and test these interpretations is a fundamental goal of digital humanities research BIBREF0 . Within natural language processing, most previous work that engages with literature relies on “distant reading” BIBREF1 , which involves discovering high-level patterns from large collections of stories BIBREF2 , BIBREF3 . We depart from this trend by showing that computational techniques can also engage with literary criticism at a closer distance: concretely, we use recent advances in text representation learning to test a single literary theory about the novel Invisible Cities by Italo Calvino.
Question: How do they model a city description using embeddings?
Paper: Casting Light on Invisible Cities: Computationally Engaging with Literary Criticism
Before describing our method and results, we first review critical opinions on both sides of whether Calvino's thematic groups meaningfully characterize his city descriptions.
Question: How do they obtain human judgements?
Answer: Using crowdsourcing
Paper: Casting Light on Invisible Cities: Computationally Engaging with Literary Criticism
We focus on measuring to what extent computers can recover Calvino's thematic groupings when given just raw text of the city descriptions. At a high level, our approach (Figure FIGREF4 ) involves (1) computing a vector representation for every city and (2) performing unsupervised clustering of these representations. The rest of this section describes both of these steps in more detail.
Question: Which clustering method do they use to cluster city description embeddings?
Paper: Scalable and Accurate Dialogue State Tracking via Hierarchical Sequence Generation
A Dialogue State Tracker (DST) is a core component of a modular task-oriented dialogue system BIBREF7 . For each dialogue turn, a DST module takes a user utterance and the dialogue history as input, and outputs a belief estimate of the dialogue state. Then a machine action is decided based on the dialogue state according to a dialogue policy module, after which a machine response is generated.
Question: Does this approach perform better in the multi-domain or single-domain setting?
Answer: single-domain setting
Paper: Scalable and Accurate Dialogue State Tracking via Hierarchical Sequence Generation
f1 shows a multi-domain dialogue in which the user wants the system to first help book a train and then reserve a hotel. For each turn, the DST will need to track the slot-value pairs (e.g. (arrive by, 20:45)) representing the user goals as well as the domain that the slot-value pairs belongs to (e.g. train, hotel). Instead of representing the belief state via a hierarchical structure, one can also combine the domain and slot together to form a combined slot-value pair (e.g. (train; arrive by, 20:45) where the combined slot is “train; arrive by"), which ignores the subordination relationship between the domain and the slots.
Question: What are the performance metrics used?
Paper: Scalable and Accurate Dialogue State Tracking via Hierarchical Sequence Generation
Given a dialogue $D$ which consists of $T$ turns of user utterances and system actions, our target is to predict the state at each turn. Different from previous methods which formulate multi-label state prediction as a collection of binary prediction problems, COMER adapts the task into a sequence generation problem via a Seq2Seq framework.
Question: Which datasets are used to evaluate performance?
Paper: Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization
State-of-the-art models for almost all popular natural language processing tasks are based on deep neural networks, trained on massive amounts of data. A key question that has been raised in many different forms is to what extent these models have learned the compositional generalizations that characterize language, and to what extent they rely on storing massive amounts of exemplars and only make `local' generalizations BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 . This question has led to (sometimes heated) debates between deep learning enthusiasts that are convinced neural networks can do almost anything, and skeptics that are convinced some types of generalization are fundamentally beyond reach for deep learning systems, pointing out that crucial tests distinguishing between generalization and memorization have not been applied.
Question: How does the automatic theorem prover infer the relation?
Paper: Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization
The data generation process is inspired by BIBREF13 : an artificial language is defined, sentences are generated according to its grammar and the entailment relation between pairs of such sentences is established according to a fixed background logic. However, our language is significantly more complex, and instead of natural logic we use FOL.
Question: If these model can learn the first-order logic on artificial language, why can't it lear for natural language?
Paper: Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization
Our main model is a recurrent network, sketched in Figure 4 . It is a so-called `Siamese' network because it uses the same parameters to process the left and the right sentence. The upper part of the model is identical to BIBREF13 's recursive networks. It consists of a comparison layer and a classification layer, after which a softmax function is applied to determine the most probable target class. The comparison layer takes the concatenation of two sentence vectors as input. The number of cells equals the number of words, so it differs per sentence.
Question: How many samples did they generate for the artificial language?
Answer: 70,000
Paper: A Simple Method for Commonsense Reasoning
Although deep neural networks have achieved remarkable successes (e.g., BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 ), their dependence on supervised learning has been challenged as a significant weakness. This dependence prevents deep neural networks from being applied to problems where labeled data is scarce. An example of such problems is common sense reasoning, such as the Winograd Schema Challenge BIBREF0 , where the labeled set is typically very small, on the order of hundreds of examples. Below is an example question from this dataset:
Question: Which of their training domains improves performance the most?
Paper: A Simple Method for Commonsense Reasoning
Unsupervised learning has been used to discover simple commonsense relationships. For example, Mikolov et al. BIBREF15 , BIBREF16 show that by learning to predict adjacent words in a sentence, word vectors can be used to answer analogy questions such as: Man:King::Woman:?. Our work uses a similar intuition that language modeling can naturally capture common sense knowledge. The difference is that Winograd Schema questions require more contextual information, hence our use of LMs instead of just word vectors.
Question: Do they fine-tune their model on the end task?
Paper: Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology
One of the biggest challenges faced by modern natural language processing (NLP) systems is the inadvertent replication or amplification of societal biases. This is because NLP systems depend on language corpora, which are inherently “not objective; they are creations of human design” BIBREF0 . One type of societal bias that has received considerable attention from the NLP community is gender stereotyping BIBREF1 , BIBREF2 , BIBREF3 . Gender stereotypes can manifest in language in overt ways. For example, the sentence he is an engineer is more likely to appear in a corpus than she is an engineer due to the current gender disparity in engineering. Consequently, any NLP system that is trained such a corpus will likely learn to associate engineer with men, but not with women BIBREF4 .
Question: Why does not the approach from English work on other languages?
Answer: Because, unlike other languages, English does not mark grammatical genders
Paper: Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology
Men and women are mentioned at different rates in text BIBREF11 . This problem is exacerbated in certain contexts. For example, the sentence he is an engineer is more likely to appear in a corpus than she is an engineer due to the current gender disparity in engineering. This imbalance in representation can have a dramatic downstream effect on NLP systems trained on such a corpus, such as giving preference to male engineers over female engineers in an automated resumé filtering system. Gender stereotypes of this sort have been observed in word embeddings BIBREF5 , BIBREF3 , contextual word embeddings BIBREF12 , and co-reference resolution systems BIBREF13 , BIBREF9 inter alia.
Question: How do they measure grammaticality?
Answer: by calculating log ratio of grammatical phrase over ungrammatical phrase
Paper: Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology
In this section, we present a Markov random field BIBREF17 for morpho-syntactic agreement. This model defines a joint distribution over sequences of morpho-syntactic tags, conditioned on a labeled dependency tree with associated part-of-speech tags. Given an intervention on a gendered word, we can use this model to infer the manner in which the remaining tags must be updated to preserve morpho-syntactic agreement.
Question: Which model do they use to convert between masculine-inflected and feminine-inflected sentences?
Paper: Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study
Humans deploy structure-sensitive expectations to guide processing during natural language comprehension BIBREF0. While it has been shown that neural language models show similar structure-sensitivity in their predictions about upcoming material BIBREF1, BIBREF2, previous work has focused on dependencies that are conditioned by features attached to a single word, such as subject number BIBREF3, BIBREF4 or wh-question words BIBREF5. There has been no systematic investigation into models' ability to compute phrase-level features—features that are attached to a set of words—and whether models can deploy these more abstract properties to drive downstream expectations.
Question: What is the performance achieved by the model described in the paper?
Paper: Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study
To determine whether state-of-the-art neural architectures are capable of learning humanlike CoordNP/verb agreement properties, we adopt the psycholinguistics paradigm for model assessment. In this paradigm the models are tested using hand-crafted sentences designed to test underlying network knowledge. The assumption here is that if a model implicitly learns humanlike linguistic knowledge during training, its expectations for upcoming words should qualitatively match human expectations in novel contexts. For example, BIBREF1 and BIBREF6 assessed how well neural models had learned the subject/verb number agreement by feeding them with the prefix The keys to the cabinet .... If the models predicted the grammatical continuation are over the ungrammatical continuation is, they can be said to have learned the number agreement insofar as the number of the head noun and not the number of the distractor noun, cabinet, drives expectations about the number of the matrix verb.
Question: What is the best performance achieved by supervised models?
Paper: Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study
are trained to output the probability distribution of the upcoming word given a context, without explicitly representing the structure of the context BIBREF9, BIBREF10. We trained two two-layer recurrent neural language models with long short-term memory architecture BIBREF11 on a relatively small corpus. The first model, referred as `LSTM (PTB)' in the following sections, was trained on the sentences from Penn Treebank BIBREF12. The second model, referred as `LSTM (FTB)', was trained on the sentences from French Treebank BIBREF13. We set the size of input word embedding and LSTM hidden layer of both models as 256.
Question: What is the size of the datasets employed?
Paper: Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study
models the linearized bracketed tree structure of a sentence by learning to predict the next action required to construct a phrase-structure parse BIBREF16. The action space consists of three possibilities: open a new non-terminal node and opening bracket; generate a terminal node; and close a bracket. To compute surprisal values for a given token, we approximate $P(w_i|w_{1\cdots i-1)}$ by marginalizing over the most-likely partial parses found by word-synchronous beam search BIBREF17.
Question: What are the baseline models?
Paper: Investigating Linguistic Pattern Ordering in Hierarchical Natural Language Generation
Spoken dialogue systems that can help users to solve complex tasks have become an emerging research topic in artificial intelligence and natural language processing areas BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . With a well-designed dialogue system as an intelligent personal assistant, people can accomplish certain tasks more easily via natural language interactions. Today, there are several virtual intelligent assistants, such as Apple's Siri, Google's Home, Microsoft's Cortana, and Amazon's Alexa, in the market. A typical dialogue system pipeline can be divided into several parts: a recognized result of a user's speech input is fed into a natural language understanding module (NLU) to classify the domain along with domain-specific intents and fill in a set of slots to form a semantic frame BIBREF4 , BIBREF5 , BIBREF6 . A dialogue state tracking (DST) module predicts the current state of the dialogue by means of the semantic frames extracted from multi-turn conversations. Then the dialogue policy determines the system action for the next step given the current dialogue state. Finally the semantic frame of the system action is then fed into a natural language generation (NLG) module to construct a response utterance to the user BIBREF7 , BIBREF8 .
Question: What evaluation metrics are used?
Paper: Investigating Linguistic Pattern Ordering in Hierarchical Natural Language Generation
The framework of the proposed hierarchical NLG model is illustrated in Figure FIGREF2 , where the model architecture is based on an encoder-decoder (seq2seq) structure with attentional hierarchical decoders BIBREF14 , BIBREF15 . In the encoder-decoder architecture, a typical generation process includes encoding and decoding phases: First, a given semantic representation sequence INLINEFORM0 is fed into a RNN-based encoder to capture the temporal dependency and project the input to a latent feature space; the semantic representation sequence is also encoded into an one-hot representation as the initial state of the encoder in order to maintain the temporal-independent condition as shown in the left part of Figure FIGREF2 . The recurrent unit of the encoder is bidirectional gated recurrent unit (GRU) BIBREF14 , DISPLAYFORM0 
Question: What datasets did they use?
Paper: Deep Enhanced Representation for Implicit Discourse Relation Recognition
 This work is licenced under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/
Question: Why does their model do better than prior models?
Paper: Detecting Potential Topics In News Using BERT, CRF and Wikipedia
Named-Entity-Recognition(NER) approaches can be categorised broadly in three types. Detecting NER with predefined dictionaries and rulesBIBREF2, with some statistical approachesBIBREF3 and with deep learning approachesBIBREF4.
Question: What is the difference in recall score between the systems?
Answer: Between the model and Stanford, Spacy and Flair the differences are 42.91, 25.03, 69.8 with Traditional NERs as reference and  49.88, 43.36, 62.43 with Wikipedia titles as reference.
Paper: Detecting Potential Topics In News Using BERT, CRF and Wikipedia
We need good amount of data to try deep learning state-of-the-art algorithms. There are lot of open datasets available for names, locations, organisations, but not for topics as defined in Abstract above. Also defining and inferring topics is an individual preference and there are no fix set of rules for its definition. But according to our definition, we can use wikipedia titles as our target topics. English wikipedia dataset has more than 18 million titles if we consider all versions of them till now. We had to clean up the titles to remove junk titles as wikipedia title almost contains all the words we use daily. To remove such titles, we deployed simple rules as follows -
Question: What is their f1 score and recall?
Answer: F1 score and Recall are 68.66, 80.08 with Traditional NERs as reference and 59.56, 69.76 with Wikipedia titles as reference.
Paper: Detecting Potential Topics In News Using BERT, CRF and Wikipedia
We tried multiple variations of LSTM and GRU layes, with/without CRF layer. There is a marginal gain in using GRU layers over LSTM. Also, we saw gain in using just one layers of GRU instead of more. Finally, we settled on the architecture, shown in Figure 1 for the final training, based on validation set scores with sample training set.
Question: How many layers does their system have?
Answer: 4 layers
Paper: Detecting Potential Topics In News Using BERT, CRF and Wikipedia
Trained the topic model on single 32gb NVidia-V100 and it took around 50 hours to train the model with sequence length 512. We had to take 256gb ram machine to accommodate all data in memory for faster read/write. Also, trained model with 64 sequence length in around 17 hours.
Question: Which news corpus is used?
Paper: Detecting Potential Topics In News Using BERT, CRF and Wikipedia
Comparison with existing open-source NER libraries is not exactly fair as they are NOT trained for detecting topics and important n-grams, also NOT trained for case-less text. But they are useful in testing and benchmarking if our model is detecting traditional NERs or not, which it should capture, as Wikipedia titles contains almost all Names, Places and Organisation names. You can check the sample output here
Question: How large is the dataset they used?
Paper: Gender Bias in Coreference Resolution
There is a classic riddle: A man and his son get into a terrible car crash. The father dies, and the boy is badly injured. In the hospital, the surgeon looks at the patient and exclaims, “I can't operate on this boy, he's my son!” How can this be?
Question: Which coreference resolution systems are tested?
Paper: How Far are We from Effective Context Modeling ? An Exploratory Study on Semantic Parsing in Context
Semantic parsing, which translates a natural language sentence into its corresponding executable logic form (e.g. Structured Query Language, SQL), relieves users from the burden of learning techniques behind the logic form. The majority of previous studies on semantic parsing assume that queries are context-independent and analyze them in isolation. However, in reality, users prefer to interact with systems in a dialogue, where users are allowed to ask context-dependent incomplete questions BIBREF0. That arises the task of Semantic Parsing in Context (SPC), which is quite challenging as there are complex contextual phenomena. In general, there are two sorts of contextual phenomena in dialogues: Coreference and Ellipsis BIBREF1. Figure FIGREF1 shows a dialogue from the dataset SParC BIBREF2. After the question “What is id of the car with the max horsepower?”, the user poses an elliptical question “How about with the max mpg?”, and a question containing pronouns “Show its Make!”. Only when completely understanding the context, could a parser successfully parse the incomplete questions into their corresponding SQL queries.
Question: How big is improvement in performances of proposed model over state of the art?
Paper: How Far are We from Effective Context Modeling ? An Exploratory Study on Semantic Parsing in Context
In the task of semantic parsing in context, we are given a dataset composed of dialogues. Denoting $\langle \mathbf {x}_1,...,\mathbf {x}_n\rangle $ a sequence of natural language questions in a dialogue, $\langle \mathbf {y}_1,...,\mathbf {y}_n\rangle $ are their corresponding SQL queries. Each SQL query is conditioned on a multi-table database schema, and the databases used in test do not appear in training. In this section, we first present a base model without considering context. Then we introduce 6 typical context modeling methods and describe how we equip the base model with these methods. Finally, we present how to augment the model with BERT BIBREF10.
Question: What two large datasets are used for evaluation?