z-uo/qasper-squad huggingface dataset:
Paper: Table-to-Text Generation with Effective Hierarchical Encoder on Three Dimensions (Row, Column and Time)
The input to the model are tables $S=\lbrace s^{1}, s^{2}, s^{3}\rbrace $. $s^{1}$, $s^{2}$, and $s^{3}$ contain records about players' performance in home team, players' performance in visiting team and team's overall performance respectively. We regard each cell in the table as record. Each record $r$ consists of four types of information including value $r.v$ (e.g. 18), entity $r.e$ (e.g. Al Jefferson), type $r.c$ (e.g. POINTS) and a feature $r.f$ (e.g. visiting) which indicate whether a player or a team compete in home court or not. Each player or team takes one row in the table and each column contains a type of record such as points, assists, etc. Also, tables contain the date when the match happened and we let $k$ denote the date of the record. We also create timelines for records. The details of timeline construction is described in Section SECREF4. For simplicity, we omit table id $l$ and record date $k$ in the following sections and let $r_{i,j}$ denotes a record of $i^{th}$ row and $j^{th}$ column in the table. We assume the records come from the same table and $k$ is the date of the mentioned record. Given those information, the model is expected to generate text $y=(y_{1}, ..., y_{t}, ..., y_{T})$ describing these tables. $T$ denotes the length of the text.
Question: What is the strong baseline?
Paper: Conversational flow in Oxford-style debates
Public debates are a common platform for presenting and juxtaposing diverging viewpoints As opposed to monologues where speakers are limited to expressing their own beliefs, debates allow for participants to interactively attack their opponents' points while defending their own. The resulting flow of ideas is a key feature of this conversation genre.
Question: what aspects of conversation flow do they look at?
Answer: The time devoted to self-coverage, opponent-coverage, and the number of adopted discussion points.
Paper: Conversational flow in Oxford-style debates
In this study we use transcripts and results of Oxford-style debates from the public debate series “Intelligence Squared Debates” (IQ2 for short). These debates are recorded live, and contain motions covering a diversity of topics ranging from foreign policy issues to the benefits of organic food. Each debate consists of two opposing teams—one for the motion and one against—of two or three experts in the topic of the particular motion, along with a moderator. Each debate follows the Oxford-style format and consists of three rounds. In the introduction, each debater is given 7 minutes to lay out their main points. During the discussion, debaters take questions from the moderator and audience, and respond to attacks from the other team. This round lasts around 30 minutes and is highly interactive; teams frequently engage in direct conversation with each other. Finally, in the conclusion, each debater is given 2 minutes to make final remarks.
Question: what debates dataset was used?
Paper: Robust Named Entity Recognition in Idiosyncratic Domains
Information extraction tasks have become very important not only in the Web, but also for in-house enterprise settings. One of the crucial steps towards understanding natural language is named entity recognition (NER), which aims to extract mentions of entity names in text. NER is necessary for many higher-level tasks such as entity linking, relation extraction, building knowledge graphs, question answering and intent based search. In these scenarios, NER recall is critical, as candidates that are never generated can not be recovered later BIBREF0 .
Question: what is the state of the art?
Answer: Babelfy, DBpedia Spotlight, Entityclassifier.eu, FOX, LingPipe MUC-7, NERD-ML, Stanford NER, TagMe 2
Paper: Robust Named Entity Recognition in Idiosyncratic Domains
We abstract the task of NER as sequential word labeling problem. Figure FIGREF15 illustrates an example for sequential transformation of a sentence into word labels. We express each sentence in a document as a sequence of words: INLINEFORM0 , e.g. INLINEFORM1 Aspirin. We define a mention as the longest possible span of adjacent tokens that refer to a an entity or relevant concept of a real-world object, such as Aspirin (ASA). We further assume that mentions are non-recursive and non-overlapping. To encode boundaries of the mention span, we adapt the idea of ramshaw1995text, which has been adapted as BIO2 standard in the CoNLL2003 shared task BIBREF15 . We assign labels INLINEFORM2 to each token to mark begin, inside and outside of a mention from left to right. We use the input sequence INLINEFORM3 together with a target sequence INLINEFORM4 of the same length that contains a BIO2 label for each word: INLINEFORM5 , e.g. INLINEFORM6 B. To predict the most likely label INLINEFORM7 of a token regarding its context, we utilize recurrent neural networks.
Question: what standard dataset were used?
Paper: Offensive Language Detection: A Comparative Analysis
In this digital era, online discussions and interactions has become a vital part of daily life of which a huge part is covered by social media platforms like twitter, facebook, instagram etc. Similar to real life there exist anti-social elements in the cyberspace, who take advantage of the anonymous nature in cyber world and indulge in vulgar and offensive communications. This includes bullying, trolling, harassment BIBREF0, BIBREF1 and has become a growing concern for governments. Youth experiencing such victimization was recorded to have psychological symptoms of anxiety, depression, loneliness BIBREF1. Thus it is important to identify and remove such behaviours at the earliest. One solution to this is the automatic detection using machine learning algorithms.
Question: Do they perform error analysis?
Paper: Offensive Language Detection: A Comparative Analysis
Data pre-processing is a very crucial step which needs to be done before applying any machine learning tasks, because the real time data could be very noisy and unstructured. For the two models used in this work, pre-processing of tweets is done separately:
Question: How do their results compare to state-of-the-art?
Paper: Offensive Language Detection: A Comparative Analysis
Word embeddings are ubiquitous for any NLP problem, as algorithms cannot process the plain text or strings in its raw form. Word emeddings are vectors that captures the semantic and contextual information of words. The word embedding used for this work are:
Question: What is the Random Kitchen Sink approach?
Answer: Random Kitchen Sink method uses a kernel function to map data vectors to a space where linear separation is possible.
Paper: Edinburgh Neural Machine Translation Systems for WMT 16
We participated in the WMT 2016 shared news translation task by building neural translation systems for four language pairs: English INLINEFORM0 Czech, English INLINEFORM1 German, English INLINEFORM2 Romanian and English INLINEFORM3 Russian. Our systems are based on an attentional encoder-decoder BIBREF0 , using BPE subword segmentation for open-vocabulary translation with a fixed vocabulary BIBREF1 . We experimented with using automatic back-translations of the monolingual News corpus as additional training data BIBREF2 , pervasive dropout BIBREF3 , and target-bidirectional models.
Question: what are the baseline systems?
Paper: Equation Embeddings
Equations are an important part of scientific articles, but many existing machine learning methods do not easily handle them. They are challenging to work with because each is unique or nearly unique; most equations occur only once. An automatic understanding of equations, however, would significantly benefit methods for analyzing scientific literature. Useful representations of equations can help draw connections between articles, improve retrieval of scientific texts, and help create tools for exploring and navigating scientific literature.
Question: What word embeddings do they test?
Paper: Equation Embeddings
Word embeddings were first introduced in BIBREF2 , BIBREF3 and there have been many variants BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 . Common for all of them is the idea that words can be represented by latent feature vectors. These feature vectors are optimized to maximize the conditional probability of the dataset. Recently BIBREF1 extended the idea of word embeddings to other types of data. EqEmb expand the idea of word embeddings to a new type of data points – equations.
Question: How do they define similar equations?
Answer: By using Euclidean distance computed between the context vector representations of the equations
Paper: Template-free Data-to-Text Generation of Finnish Sports News
Automated, or robotic, journalism aims at news generation from structured data sources, either as the final product or as a draft for subsequent post-editing. At present, automated journalism typically focuses on domains such as sports, finance and similar statistics-based reporting, where there is a commercial product potential due to the high volume of news, combined with the expectation of a relatively straightforward task.
Question: What evaluation criteria and metrics were used to evaluate the generated text?
Paper: Feature Studies to Inform the Classification of Depressive Symptoms from Twitter Data for Population Health
In recent years, there has been a movement to leverage social medial data to detect, estimate, and track the change in prevalence of disease. For example, eating disorders in Spanish language Twitter tweets BIBREF0 and influenza surveillance BIBREF1 . More recently, social media has been leveraged to monitor social risks such as prescription drug and smoking behaviors BIBREF2 , BIBREF3 , BIBREF4 as well as a variety of mental health disorders including suicidal ideation BIBREF5 , attention deficient hyperactivity disorder BIBREF6 and major depressive disorder BIBREF7 . In the case of major depressive disorder, recent efforts range from characterizing linguistic phenomena associated with depression BIBREF8 and its subtypes e.g., postpartum depression BIBREF5 , to identifying specific depressive symptoms BIBREF9 , BIBREF10 e.g., depressed mood. However, more research is needed to better understand the predictive power of supervised machine learning classifiers and the influence of feature groups and feature sets for efficiently classifying depression-related tweets to support mental health monitoring at the population-level BIBREF11 .
Question: Do they evaluate only on English datasets?
Paper: Feature Studies to Inform the Classification of Depressive Symptoms from Twitter Data for Population Health
Specifically, we conducted a feature ablation study to assess the informativeness of each feature group and a feature elimination study to determine the optimal feature sets for classifying Twitter tweets. We leveraged an existing, annotated Twitter dataset that was constructed based on a hierarchical model of depression-related symptoms BIBREF12 , BIBREF13 . The dataset contains 9,473 annotations for 9,300 tweets. Each tweet is annotated as no evidence of depression (e.g., “Citizens fear an economic depression") or evidence of depression (e.g., “depressed over disappointment"). If a tweet is annotated evidence of depression, then it is further annotated with one or more depressive symptoms, for example, depressed mood (e.g., “feeling down in the dumps"), disturbed sleep (e.g., “another restless night"), or fatigue or loss of energy (e.g., “the fatigue is unbearable") BIBREF10 . For each class, every annotation (9,473 tweets) is binarized as the positive class e.g., depressed mood=1 or negative class e.g., not depressed mood=0.
Question: What are the three steps to feature elimination?
Answer: reduced the dataset by eliminating features, apply feature selection to select highest ranked features to train and test the model and rank the performance of incrementally adding features.
Paper: Feature Studies to Inform the Classification of Depressive Symptoms from Twitter Data for Population Health
Furthermore, this dataset was encoded with 7 feature groups with associated feature values binarized (i.e., present=1 or absent=0) to represent potentially informative features for classifying depression-related classes. We describe the feature groups by type, subtype, and provide one or more examples of words representing the feature subtype from a tweet:
Question: How is the dataset annotated?
Answer: The annotations are based on evidence of depression and further annotated by the depressive symptom if there is evidence of depression
Paper: Feature Studies to Inform the Classification of Depressive Symptoms from Twitter Data for Population Health
Feature ablation studies are conducted to assess the informativeness of a feature group by quantifying the change in predictive power when comparing the performance of a classifier trained with the all feature groups versus the performance without a particular feature group. We conducted a feature ablation study by holding out (sans) each feature group and training and testing the support vector model using a linear kernel and 5-fold, stratified cross-validation. We report the average F1-score from our baseline approach (all feature groups) and report the point difference (+ or -) in F1-score performance observed by ablating each feature set.
Question: What dataset is used for this study?
Paper: Extracting clinical concepts from user queries
Medical search engines are an essential component for many online medical applications, such as online diagnosis systems and medical document databases. A typical online diagnosis system, for instance, relies on a medical search engine. The search engine takes as input a user query that describes some symptoms and then outputs clinical concept entries that provide relevant information to assist in diagnosing the problem. One challenge medical search engines face is the segmentation of individual clinical entities. When a user query consists of multiple clinical entities, a search engine would often fail to recognize them as separate entities. For example, the user query “fever joint pain weight loss headache” contains four separate clinical entities: “fever”, “joint pain”, “weight loss”, and “headache”. But when the search engine does not recognize them as separate entities and proceeds to retrieve results for each word in the query, it may find "pain" in body locations other than "joint pain", or it may miss "headache" altogether, for example. Some search engines allow the users to enter a single clinical concept by selecting from an auto-completion pick list. But this could also result in retrieving inaccurate or partial results and lead to poor user experience.
Question: what were their performance results?
Paper: Extracting clinical concepts from user queries
An effective model that has been commonly used for NER problem is a Bi-directional LSTM with a Conditional Random Field (CRF) on the top layer (BiLSTM-CRF), which is described in the next section. Combining LSTM’s power of representing relations between words and CRF’s capability of accounting for tag sequence constraints, Huang et al. BIBREF2 proposed the BiLSTM-CRF model and used handcrafted word features as the input to the model. Lample et al. BIBREF3 used a combination of character-level and word-level word embeddings as the input to BiLSTM-CRF. Since then, similar models with variation in types of word embeddings have been used extensively for clinical CE tasks and produced state-of-the-art results BIBREF4, BIBREF5, BIBREF6, BIBREF7. Word embeddings have become the cornerstone of the neural models in NLP since the famous Word2vec BIBREF8 model demonstrated its power in word analogy tasks. One well-known example is that after training Word2vec on a large amount of news data, we can get word relations such as $vector(^{\prime }king^{\prime }) - vector(^{\prime }queen^{\prime }) + vector(^{\prime }woman^{\prime }) \approx vector(^{\prime }man^{\prime })$. More sophisticated word embedding technique emerged since Word2vec. It has been shown empirically that better quality in word embeddings leads to better performance in many downstream NLP including entity tagging BIBREF9, BIBREF10. Recently, contextualized word embeddings generated by deep learning models, such as ELMo BIBREF11, BERT BIBREF12, and Flair BIBREF13, have been shown to be more effective in various NLP tasks. In our project, we make use of a fine-tuned ELMo model and a fine-tuned Flair model in the medical domain. We experiment with the word embeddings from the two fine-tuned models as the input to the BiLSTM-CRF model separately and compare the results.
Question: where did they obtain the annotated clinical notes from?
Paper: Attention-based Wav2Text with Feature Transfer Learning
Conventional large-vocabulary continuous speech recognition (LVCSR) systems typically perform multi-level pattern recognition tasks that map the acoustic speech waveform into a hierarchy of speech units such as sub-words (phonemes), words, and strings of words (sentences). Such systems basically consist of several sub-components (feature extractor, acoustic model, pronunciation lexicon, language model) that are trained and tuned separately BIBREF0 . First, the speech signal is processed into a set of observation features based on a carefully hand-crafted feature extractor, such as Mel frequency cepstral coefficients (MFCC) or Mel-scale spectrogram. Then the acoustic model classifies the observation features into sub-unit or phoneme classes. Finally, the search algorithm finds the most probable word sequence based on the evidence of the acoustic model, the lexicon, and the language model. But, it is widely known that information loss in the earlier stage can propagate through the later stages.
Question: Which architecture do they use for the encoder and decoder?
Answer: In encoder they use convolutional, NIN and bidirectional LSTM layers and in decoder they use unidirectional LSTM
Paper: Attention-based Wav2Text with Feature Transfer Learning
The encoder-decoder model is a neural network that directly models conditional probability INLINEFORM0 where INLINEFORM1 is the source sequence with length INLINEFORM2 and INLINEFORM3 is the target sequence with length INLINEFORM4 . It consists of encoder, decoder and attention modules. The encoder task processes an input sequence INLINEFORM5 and outputs representative information INLINEFORM6 for the decoder. The attention module is an extension scheme that assists the decoder to find relevant information on the encoder side based on the current decoder hidden states BIBREF12 , BIBREF13 . Usually, the attention module produces context information INLINEFORM7 at time INLINEFORM8 based on the encoder and decoder hidden states: DISPLAYFORM0 
Question: How does their decoder generate text?
Answer: Decoder predicts the sequence of phoneme or grapheme at each time based on the previous output and context information with a beam search strategy
Paper: Attention-based Wav2Text with Feature Transfer Learning
Deep learning is well known for its ability to learn directly from low-level feature representation such as raw speech BIBREF1 , BIBREF3 . However, in most cases such models are already conditioned on a fixed input size and a single target output (i.e., predicting one phoneme class for each input frame). In the attention-based encoder-decoder model, the training process is not as easy as in a standard neural network model BIBREF10 because the attention-based model needs to jointly optimize three different modules simultaneously: (1) an encoder module for producing representative information from a source sequence; (2) an attention module for calculating the correct alignment; and (3) a decoder module for generating correct transcriptions. If one of these modules has difficulty fulfilling its own tasks, then the model will fail to produce good results.
Question: Which dataset do they use?
Paper: Contextualize, Show and Tell: A Neural Visual Storyteller
Over the past few years, generating text from images and videos has gained a lot of attention in the Computer Vision and Natural Language Processing communities and several related tasks have been proposed, such as image labeling, image and video description and visual question answering. In particular, prominent results have been achieved in image description with various deep neural network architectures, e.g. BIBREF1 , BIBREF2 , BIBREF3 , BIBREF0 . However, the need of generating more narrative texts from images which may reflect experiences, rather than just listing objects and their attributes, has given rise to tasks such as visual storytelling BIBREF4 . This task is about generating a story from a sequence of images. Figure 1 shows the difference between descriptions of images in isolation and stories for images in sequence.
Question: What model is used to encode the images?
Paper: Contextualize, Show and Tell: A Neural Visual Storyteller
The work by BIBREF6 presented probably the first system for generating stories from an album of images. This early approach involved the use of the NYC and Disney datasets mined from blog posts by the authors.
Question: How is the sequential nature of the story captured?
Paper: Contextualize, Show and Tell: A Neural Visual Storyteller
Our model extends the image description model by BIBREF0 , which consists of an encoder-decoder architecture. The encoder is a Convolutional Neural Network (CNN) and the decoder is a Long Short-Term Memory (LSTM) network, as presented in Figure 2 . The image is passed through the encoder generating the image representation that is used by the decoder to know the content of the image and generate the description word by word. In the following, we describe how we extended this model for the visual storytelling task.
Question: Is the position in the sequence part of the input?
Paper: Contextualize, Show and Tell: A Neural Visual Storyteller
The model's first component is a Recurrent Neural Network (RNN), more precisely an LSTM that summarizes the sequence of images. At every timestep $t$ the network takes as input an image $I_i$ where $i\in \lbrace 1,2,3,4,5\rbrace $ from the sequence. At time $t=5$ , the LSTM has encoded the 5 images and provides the sequence's context through its last hidden state denoted by $h_e^{(t)}$ . The representation of the images was obtained through Inception V3.
Question: Do the decoder LSTMs all have the same weights?
Paper: DOLORES: Deep Contextualized Knowledge Graph Embeddings
Knowledge graphs BIBREF0 enable structured access to world knowledge and form a key component of several applications like search engines, question answering systems and conversational assistants. Knowledge graphs are typically interpreted as comprising of discrete triples of the form (entityA, relationX, entityB) thus representing a relation (relationX) between entityA and entityB. However, one limitation of only a discrete representation of triples is that it does not easily enable one to infer similarities and potential relations among entities which may be missing in the knowledge graph. Consequently, one popular alternative is to learn dense continuous representations of entities and relations by embedding them in latent continuous vector spaces, while seeking to model the inherent structure of the knowledge graph. Most knowledge graph embedding methods can be classified into two major classes: one class which operates purely on triples like RESCAL BIBREF1 , TransE BIBREF2 , DistMult BIBREF3 , TransD BIBREF4 , ComplEx BIBREF5 , ConvE BIBREF6 and the second class which seeks to incorporate additional information (like multi-hops) BIBREF7 . Learning high-quality knowledge graph embeddings can be quite challenging given that (a) they need to effectively model the contextual usages of entities and relations (b) they would need to be useful for a variety of predictive tasks on knowledge graphs.
Question: Is fine-tuning required to incorporate these embeddings into existing models?