z-uo/qasper-squad huggingface dataset:
Paper: Many Languages, One Parser
Developing tools for processing many languages has long been an important goal in NLP BIBREF0 , BIBREF1 , but it was only when statistical methods became standard that massively multilingual NLP became economical. The mainstream approach for multilingual NLP is to design language-specific models. For each language of interest, the resources necessary for training the model are obtained (or created), and separate parameters are fit for each language separately. This approach is simple and grants the flexibility of customizing the model and features to the needs of each language, but it is suboptimal for theoretical and practical reasons. Theoretically, the study of linguistic typology tells us that many languages share morphological, phonological, and syntactic phenomena BIBREF3 ; therefore, the mainstream approach misses an opportunity to exploit relevant supervision from typologically related languages. Practically, it is inconvenient to deploy or distribute NLP tools that are customized for many different languages because, for each language of interest, we need to configure, train, tune, monitor, and occasionally update the model. Furthermore, code-switching or code-mixing (mixing more than one language in the same discourse), which is pervasive in some genres, in particular social media, presents a challenge for monolingually-trained NLP models BIBREF4 .
Question: How does the model work if no treebank is available?
Paper: Many Languages, One Parser
Our goal is to train a dependency parser for a set of target languages ${L}^t$ , given universal dependency annotations in a set of source languages ${L}^s$ . Ideally, we would like to have training data in all target languages (i.e., $L^t \subseteq L^s$ ), but we are also interested in the case where the sets of source and target languages are disjoint (i.e., $L^t \cap L^s = \emptyset $ ). When all languages in $L^t$ have a large treebank, the mainstream approach has been to train one monolingual parser per target language and route sentences of a given language to the corresponding parser at test time. In contrast, our approach is to train one parsing model with the union of treebanks in $L^s$ , then use this single trained model to parse text in any language in $L^t$ , hence the name “Many Languages, One Parser” (MaLOPa). MaLOPa strikes a balance between: (1) enabling cross-lingual model transfer via language-invariant input representations; i.e., coarse POS tags, multilingual word embeddings and multilingual word clusters, and (2) tweaking the behavior of the parser depending on the current input language via language-specific representations; i.e., fine-grained POS tags and language embeddings.
Question: How many languages have this parser been tried on?
Paper: Semi-Supervised Neural Text Generation by Joint Learning of Natural Language Generation and Natural Language Understanding Models
Natural Language Generation (NLG) is an NLP task that consists in generating a sequence of natural language sentences from non-linguistic data. Traditional approaches of NLG consist in creating specific algorithms in the consensual NLG pipeline BIBREF0, but there has been recently a strong interest in End-to-End (E2E) NLG systems which are able to jointly learn sentence planning and surface realization BIBREF1, BIBREF2, BIBREF3, BIBREF4. Probably the most well known effort of this trend is the E2E NLG challenge BIBREF5 whose task was to perform sentence planing and realization from dialogue act-based Meaning Representation (MR) on unaligned data. For instance, Figure FIGREF1 presents, on the upper part, a meaning representation and on the lower part, one possible textual realization to convey this meaning. Although the challenge was a great success, the data used in the challenge contained a lot of redundancy of structure and a limited amount of concepts and several reference texts per MR input (8.1 in average). This is an ideal case for machine learning but is it the one that is encountered in all E2E NLG real-world applications?
Question: Do they use attention?
Paper: Semi-Supervised Neural Text Generation by Joint Learning of Natural Language Generation and Natural Language Understanding Models
E2E Natural Language Generation systems are typically based on the Recurrent Neural Network (RNN) architecture consisting of an encoder and a decoder also known as seq2seq BIBREF8. The encoder takes a sequence of source words $\mathbf {x}~=~\lbrace {x_1},{x_2}, ..., {x_{T_x}}\rbrace $ and encodes it to a fixed length vector. The decoder then decodes this vector into a sequence of target words $\mathbf {y}~=~\lbrace {y_1},{y_2}, ..., {y_{T_y}}\rbrace $. Seq2seq models are able to treat variable sized source and target sequences making them a great choice for NLG and NLU tasks.
Question: What non-annotated datasets are considered?
Paper: Story Ending Generation with Incremental Encoding and Commonsense Knowledge
Story generation is an important but challenging task because it requires to deal with logic and implicit knowledge BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . Story ending generation aims at concluding a story and completing the plot given a story context. We argue that solving this task involves addressing the following issues: 1) Representing the context clues which contain key information for planning a reasonable ending; and 2) Using implicit knowledge (e.g., commonsense knowledge) to facilitate understanding of the story and better predict what will happen next.
Question: Did they compare to Transformer based large language models?
Paper: Story Ending Generation with Incremental Encoding and Commonsense Knowledge
The corpus we used in this paper was first designed for Story Cloze Test (SCT) BIBREF10 , which requires to select a correct ending from two candidates given a story context. Feature-based BIBREF11 , BIBREF12 or neural BIBREF8 , BIBREF13 classification models are proposed to measure the coherence between a candidate ending and a story context from various aspects such as event, sentiment, and topic. However, story ending generation BIBREF14 , BIBREF15 , BIBREF16 is more challenging in that the task requires to modeling context clues and implicit knowledge to produce reasonable endings.
Question: Which baselines are they using?
Paper: Dataset for the First Evaluation on Chinese Machine Reading Comprehension
Machine Reading Comprehension (MRC) has become enormously popular in recent research, which aims to teach the machine to comprehend human languages and answer the questions based on the reading materials. Among various reading comprehension tasks, the cloze-style reaing comprehension is relatively easy to follow due to its simplicity in definition, which requires the model to fill an exact word into the query to form a coherent sentence according to the document material. Several cloze-style reading comprehension datasets are publicly available, such as CNN/Daily Mail BIBREF0 , Children's Book Test BIBREF1 , People Daily and Children's Fairy Tale BIBREF2 .
Question: What two types the Chinese reading comprehension dataset consists of?
Answer: cloze-style reading comprehension and user query reading comprehension questions
Paper: Dataset for the First Evaluation on Chinese Machine Reading Comprehension
In this section, we will introduce several public cloze-style reading comprehension dataset.
Question: For which languages most of the existing MRC datasets are created?
Answer: English
Paper: Joint learning of ontology and semantic parser from text
One of the ultimate goals of Natural Language Processing (NLP) is machine reading BIBREF0 , the automatic, unsupervised understanding of text. One way of pursuing machine reading is by semantic parsing, which transforms text into its meaning representation. However, capturing the meaning is not the final goal, the meaning representation needs to be predefined and structured in a way that supports reasoning. Ontologies provide a common vocabulary for meaning representations and support reasoning, which is vital for understanding the text. To enable flexibility when encountering new concepts and relations in text, in machine reading we want to be able to learn and extend the ontology while reading. Traditional methods for ontology learning BIBREF1 , BIBREF2 are only concerned with discovering the salient concepts from text. Thus, they work in a macro-reading fashion BIBREF3 , where the goal is to extract facts from a large collection of texts, but not necessarily all of them, as opposed to a micro-reading fashion, where the goal is to extract every fact from the input text. Semantic parsers operate in a micro-reading fashion. Consequently, the ontologies with only the salient concepts are not enough for semantic parsing. Furthermore, the traditional methods learn an ontology for a particular domain, where the text is used just as a tool. On the other hand, ontologies are used just as tool to represent meaning in the semantic parsing setting. When developing a semantic parser it is not trivial to get the best meaning representation for the observed text, especially if the content is not known yet. Semantic parsing datasets have been created by either selecting texts that can be expressed with a given meaning representation, like Free917 dataset BIBREF4 , or by manually deriving the meaning representation given the text, like Atis dataset BIBREF5 . In both datasets, each unit of text has its corresponding meaning representation. While Free917 uses Freebase BIBREF6 , which is a very big multi-domain ontology, it is not possible to represent an arbitrary sentence with Freebase or any other existing ontology.
Question: How did they induce the CFG?
Paper: Joint learning of ontology and semantic parser from text
In this section, we propose a semi-automatic bootstrapping procedure for grammar induction, which searches for the most frequent patterns and constructs new production rules from them. One of the main challenges is to make the induction in a way that minimizes human involvement and maximizes the quality of semantic trees.
Question: How big is their dataset?
Paper: A Continuous Relaxation of Beam Search for End-to-end Training of Neural Sequence Models
[t] Standard Beam Search [1] INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 t = 0 to T i = 1 to k INLINEFORM4 INLINEFORM5 INLINEFORM6 is the local output scoring function INLINEFORM7 top-k-max INLINEFORM8 Top k values of the input matrix INLINEFORM9 top-k-argmax INLINEFORM10 Top INLINEFORM11 argmax index pairs of the input matrix i = 1 to k INLINEFORM12 embedding( INLINEFORM13 ) INLINEFORM14 INLINEFORM15 is a nonlinear recurrent function that returns state at next step INLINEFORM16 INLINEFORM17 follow-backpointer( INLINEFORM18 ) INLINEFORM19 Sequence-to-sequence (seq2seq) models have been successfully used for many sequential decision tasks such as machine translation BIBREF0 , BIBREF1 , parsing BIBREF2 , BIBREF3 , summarization BIBREF4 , dialog generation BIBREF5 , and image captioning BIBREF6 . Beam search is a desirable choice of test-time decoding algorithm for such models because it potentially avoids search errors made by simpler greedy methods. However, the typical approach to training neural sequence models is to use a locally normalized maximum likelihood objective (cross-entropy training) BIBREF0 . This objective does not directly reason about the behaviour of the final decoding method. As a result, for cross-entropy trained models, beam decoding can sometimes yield reduced test performance when compared with greedy decoding BIBREF7 , BIBREF8 , BIBREF9 . These negative results are not unexpected. The training procedure was not search-aware: it was not able to consider the effect that changing the model's scores might have on the ease of search while using a beam decoding, greedy decoding, or otherwise.
Question: By how much do they outperform basic greedy and cross-entropy beam decoding?
Paper: A Continuous Relaxation of Beam Search for End-to-end Training of Neural Sequence Models
We denote the seq2seq model parameterized by INLINEFORM0 as INLINEFORM1 . We denote the input sequence as INLINEFORM2 , the gold output sequence as INLINEFORM3 and the result of beam search over INLINEFORM4 as INLINEFORM5 . Ideally, we would like to directly minimize a final evaluation loss, INLINEFORM6 , evaluated on the result of running beam search with input INLINEFORM7 and model INLINEFORM8 . Throughout this paper we assume that the evaluation loss decomposes over time steps INLINEFORM9 as: INLINEFORM10 . We refer to this idealized training objective that directly evaluates prediction loss as the “direct loss” objective and define it as: DISPLAYFORM0 
Question: Do they provide a framework for building a sub-differentiable for any final loss metric?
Paper: A Continuous Relaxation of Beam Search for End-to-end Training of Neural Sequence Models
[t] continuous-top-k-argmax [1] INLINEFORM0 INLINEFORM1 , s.t. INLINEFORM2 INLINEFORM3 INLINEFORM4 = 1 to k peaked-softmax will be dominated by scores closer to INLINEFORM5 INLINEFORM6 The square operation is element-wise Formally, beam search is a procedure with hyperparameter INLINEFORM7 that maintains a beam of INLINEFORM8 elements at each time step and expands each of the INLINEFORM9 elements to find the INLINEFORM10 -best candidates for the next time step. The procedure finds an approximate argmax of a scoring function defined on output sequences.
Question: Do they compare partially complete sequences (created during steps of beam search) to gold/target sequences?
Paper: A Continuous Relaxation of Beam Search for End-to-end Training of Neural Sequence Models
[t] Continuous relaxation to beam search [1] INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 t = 0 to T INLINEFORM5 i=1 to k INLINEFORM6 INLINEFORM7 is a local output scoring function INLINEFORM8 INLINEFORM9 is used to compute INLINEFORM10 INLINEFORM11 Call Algorithm 2 i = 1 to k INLINEFORM12 Soft back pointer computation INLINEFORM13 Contribution from vocabulary items INLINEFORM14 Peaked distribution over the candidates to compute INLINEFORM15 INLINEFORM16 INLINEFORM17 INLINEFORM18 j = 1 to k Get contributions from soft backpointers for each beam element INLINEFORM19 INLINEFORM20 INLINEFORM21 INLINEFORM22 is a nonlinear recurrent function that returns state at next step INLINEFORM23 Pick the loss for the sequence with highest model score on the beam in a soft manner.
Question: Which loss metrics do they try in their new training procedure evaluated on the output of beam search?
Paper: Weighed Domain-Invariant Representation Learning for Cross-domain Sentiment Analysis
Sentiment analysis aims to predict sentiment polarity of user-generated data with emotional orientation like movie reviews. The exponentially increase of online reviews makes it an interesting topic in research and industrial areas. However, reviews can span so many different domains and the collection and preprocessing of large amounts of data for new domains is often time-consuming and expensive. Therefore, cross-domain sentiment analysis is currently a hot topic, which aims to transfer knowledge from a label-rich source domain (S) to the label-few target domain (T).
Question: How are different domains weighted in WDIRL?
Paper: Weighed Domain-Invariant Representation Learning for Cross-domain Sentiment Analysis
For expression consistency, in this work, we consider domain adaptation in the unsupervised setting (however, we argue that our analysis and solution also applies to the supervised and semi-supervised domain adaptation settings). In the unsupervised domain adaptation setting, there are two different distributions over $\rm {X} \times \rm {Y}$: the source domain $\rm {P}_S(\rm {X},\rm {Y})$ and the target domain $\rm {P}_T(\rm {X},\rm {Y})$. And there is a labeled data set $\mathcal {D}_S$ drawn $i.i.d$ from $\rm {P}_S(\rm {X},\rm {Y})$ and an unlabeled data set $\mathcal {D}_T$ drawn $i.i.d.$ from the marginal distribution $\rm {P}_T(\rm {X})$:
Question: How is DIRL evaluated?
Paper: Weighed Domain-Invariant Representation Learning for Cross-domain Sentiment Analysis
Domain invariant representation learning (DIRL) is a very popular framework for performing domain adaptation in the cross-domain sentiment analysis field BIBREF23, BIBREF4, BIBREF24, BIBREF7. It is heavily motivated by the following theorem BIBREF25.
Question: Which sentiment analysis tasks are addressed?
Answer: 12 binary-class classification and multi-class classification of reviews based on rating
Paper: The State of NLP Literature: A Diachronic Analysis of the ACL Anthology
The ACL Anthology (AA) is a digital repository of tens of thousands of articles on Natural Language Processing (NLP) / Computational Linguistics (CL). It includes papers published in the family of ACL conferences as well as in other NLP conferences such as LREC and RANLP. AA is the largest single source of scientific literature on NLP.
Question: Which NLP area have the highest average citation for woman author?
Paper: The State of NLP Literature: A Diachronic Analysis of the ACL Anthology
Q. How big is the ACL Anthology (AA)? How is it changing with time?
Question: Which 3 NLP areas are cited the most?
Answer: machine translation, statistical machine, sentiment analysis
Paper: The State of NLP Literature: A Diachronic Analysis of the ACL Anthology
NLP, like most other areas of research, suffers from poor demographic diversity. There is very little to low representation from certain nationalities, race, gender, language, income, age, physical abilities, etc. This impacts the breadth of technologies we create, how useful they are, and whether they reach those that need it most. In this section, we analyze three specific attributes among many that deserve attention: gender (specifically, the number of women researchers in NLP), age (more precisely, the number of years of NLP paper publishing experience), and the amount of research in various languages (which loosely correlates with geographic diversity).
Question: Which journal and conference are cited the most in recent years?
Answer: CL Journal and EMNLP conference
Paper: The State of NLP Literature: A Diachronic Analysis of the ACL Anthology
The ACL Anthology does not record demographic information about the paper authors. (Until recently, ACL and other NLP conferences did not record demographic information of the authors.) However, many first names have strong associations with a male or female gender. We will use these names to estimate the percentage of female first authors in NLP.
Question: Which 5 languages appear most frequently in AA paper titles?
Answer: English, Chinese, French, Japanese and Arabic
Paper: The State of NLP Literature: A Diachronic Analysis of the ACL Anthology
While the actual age of NLP researchers might be an interesting aspect to explore, we do not have that information. Thus, instead, we can explore a slightly different (and perhaps more useful) attribute: NLP academic age. We can define NLP academic age as the number of years one has been publishing in AA. So if this is the first year one has published in AA, then their NLP academic age is 1. If one published their first AA paper in 2001 and their latest AA paper in 2018, then their academic age is 18.
Question: What aspect of NLP research is examined?
Paper: The State of NLP Literature: A Diachronic Analysis of the ACL Anthology
Automatic systems with natural language abilities are growing to be increasingly pervasive in our lives. Not only are they sources of mere convenience, but are crucial in making sure large sections of society and the world are not left behind by the information divide. Thus, the limits of what automatic systems can do in a language, limit the world for the speakers of that language.
Question: Are the academically younger authors cited less than older?
Paper: The State of NLP Literature: A Diachronic Analysis of the ACL Anthology
Natural Language Processing addresses a wide range of research questions and tasks pertaining to language and computing. It encompasses many areas of research that have seen an ebb and flow of interest over the years. In this section, we examine the terms that have been used in the titles of ACL Anthology (AA) papers. The terms in a title are particularly informative because they are used to clearly and precisely convey what the paper is about. Some journals ask authors to separately include keywords in the paper or in the meta-information, but AA papers are largely devoid of this information. Thus titles are an especially useful source of keywords for papers—keywords that are often indicative of the area of research.
Question: How many papers are used in experiment?
Paper: BERTQA -- Attention on Steroids
Through this CS224N Pre-trained Contextual Embeddings (PCE) project, we tackle the question answering problem which is one of the most popular in NLP and has been brought to the forefront by datasets such as SQUAD 2.0. This problem's success stems from both the challenge it presents and the recent successes in approaching human level function. As most, if not all, of the problems humans solve every day can be posed as a question, creating an deep learning based solution that has access to the entire internet is a critical milestone for NLP. Through our project, our group had tested the limits of applying attention in BERT BIBREF0 to improving the network's performance on the SQUAD2.0 dataset BIBREF1. BERT applies attention to the concatenation of the query and context vectors and thus attends these vectors in a global fashion. We propose BERTQA BIBREF2 which adds Context-to-Query (C2Q) and Query-to-Context (Q2C) attention in addition to localized feature extraction via 1D convolutions. We implemented the additions ourselves, while the Pytorch baseline BERT code was obtained from BIBREF3. The SQUAD2.0 answers span from a length of zero to multiple words and this additional attention provides hierarchical information that will allow the network to better learn to detect answer spans of varying sizes. We applied the empirical findings from this part of our project to the large BERT model, which has twice as many layers as the base BERT model. We also augmented the SQUAD2.0 dataset with additional backtranslated examples. This augmented dataset will be publicly available on our github BIBREF4 on the completion of this course. After performing hyperparameter tuning, we ensembled our two best networks to get F1 and EM scores of 82.317 and 79.442 respectively. The experiments took around 300 GPU hours to train.
Question: What ensemble methods are used for best model?
Paper: BERTQA -- Attention on Steroids
The SQUAD2.0 creators proposed this dataset as a means for networks to actually understand the text they were being interrogated about rather than simply being extractive parsers. Many networks stepped up to the challenge including BERT, BIDAF, and QANET. BERT is a fully feed forward network that is based on the transformer architecture BIBREF5. The base BERT model has 12 transformer encoder layers that terminate in an interchangeable final layer which can be finetuned to the specific task. We chose this network as our baseline because of its use of contextual embeddings and global attention and because of the speed advantage derived from an RNN free architecture. We derived inspiration for our modifications from the BIDAF and QANET models. BIDAF is an LSTM based network that uses character, word, and contextual embeddings which are fed through Context-to-Query (C2Q) and Query-to-Context (Q2C) layers. The final logits are derived from separate Start and End output layers, as opposed to BERT which produces these logits together. Our C2Q/Q2C addition to BERT and the Dense Layer/LSTM based separate final Start and End logit prediction layer were inspired by this paper. We also refered to the QANET model, which is also a fully feed forward network that emphasizes the use of convolutions to capture the local structure of text. Based on this paper, we created a convolutional layer within the C2Q/Q2C architecture to add localized information to BERT's global attention and the C2Q/Q2C coattention.
Question: What hyperparameters have been tuned?