z-uo/qasper-squad huggingface dataset:
Paper: On the Relationship between Self-Attention and Convolutional Layers
In this work, we put forth theoretical and empirical evidence that self-attention layers can (and do) learn to behave similar to convolutional layers:
Question: Is there a way of converting existing convolution layers into self-attention to perform very same convolution?
Paper: On the Relationship between Self-Attention and Convolutional Layers
We here recall the mathematical formulation of self-attention layers and emphasize the role of positional encodings.
Question: What authors mean by sufficient number of heads?
Paper: On the Relationship between Self-Attention and Convolutional Layers
Let $\in ^{T\times D_{\textit {in}}}$ be an input matrix consisting of $T$ tokens in of ${D_{\textit {in}}}$ dimensions each. While in NLP each token corresponds to a word in a sentence, the same formalism can be applied to any sequence of $T$ discrete objects, e.g. pixels. A self-attention layer maps any query token $t \in [T]$ from $D_{\textit {in}}$ to $D_{\textit {out}}$ dimensions as follows: Self-Attention()t,: := ( t,: ) val, where we refer to the elements of the $T \times T$ matrix := qrykey as attention scores and the softmax output as attention probabilities. The layer is parametrized by a query matrix $_{\!\textit {qry}}\in ^{D_{\textit {in}} \times D_{k}}$, a key matrix $_{\!\textit {key}}\in ^{D_{\textit {in}} \times D_{k}}$ and a value matrix $_{\!\textit {val}}\in ^{D_{\textit {in}} \times D_{\textit {out}}}$.For simplicity, we exclude any residual connections, batch normalization and constant factors. A key property of the self-attention model described above is that it is equivariant to reordering, that is, it gives the same output independently of how the $T$ input tokens are shuffled. This is problematic for cases we expect the order of things to matter. To alleviate the limitation, a positional encoding is learned for each token in the sequence (or pixel in an image), and added to the representation of the token itself before applying self-attention := (+ ) qrykey(+ ), where $\in ^{T \times D_{\textit {in}}}$ contains the embedding vectors for each position. More generally, $$ may be substituted by any function that returns a vector representation of the position.
Question: Is there any nonnumerical experiment that also support author's claim, like analysis of attention layers in publicly available networks?
Paper: On the Relationship between Self-Attention and Convolutional Layers
Convolutional layers are the de facto choice for building neural networks that operate on images. We recall that, given an image tensor $~\in ~^{W\times H \times D_{\textit {in}}}$ of width $W$, height $H$ and $D_{\textit {in}}$ channels, the output of a convolutional layer for pixel $(i,j)$ is given by Conv()i,j,: := (1, 2) K 1,2,:,: i+1, j+2, : + , where $$ is the $K \times K \times D_{\textit {out}} \times D_{\textit {in}}$ weight tensor , $\in ^{D_{\textit {out}}}$ is the bias vector and the set
Question: What numerical experiments they perform?
Paper: Learning to Discover, Ground and Use Words with Segmental Neural Language Models
How infants discover the words of their native languages is a long-standing question in developmental psychology BIBREF0 . Machine learning has contributed much to this discussion by showing that predictive models of language are capable of inferring the existence of word boundaries solely based on statistical properties of the input BIBREF1 , BIBREF2 , BIBREF3 . Unfortunately, the best language models, measured in terms of their ability to model language, segment quite poorly BIBREF4 , BIBREF5 , while the strongest models in terms of word segmentation are far too weak to adequately predict language BIBREF3 , BIBREF6 . Moreover, since language acquisition is ultimately a multimodal process, neural models which simplify working with multimodal data offer opportunities for future research. However, as BIBREF7 have argued, current neural models' inability to discover meaningful words is too far behind the current (non-neural) state-of-the-art to be a useful foundation.
Question: What dataset is used?
Paper: Learning to Discover, Ground and Use Words with Segmental Neural Language Models
We now describe the segmental neural language model (SNLM). Refer to Figure FIGREF1 for an illustration. The SNLM generates a character sequence INLINEFORM0 , where each INLINEFORM1 is a character in a finite character set INLINEFORM2 . Each sequence INLINEFORM3 is the concatenation of a sequence of segments INLINEFORM4 where INLINEFORM5 measures the length of the sequence in segments and each segment INLINEFORM6 is a sequence of characters, INLINEFORM7 . Intuitively, each INLINEFORM8 corresponds to one word. Let INLINEFORM9 represent the concatenation of the characters of the segments INLINEFORM10 to INLINEFORM11 , discarding segmentation information; thus INLINEFORM12 . For example if INLINEFORM13 , the underlying segmentation might be INLINEFORM14 (with INLINEFORM15 and INLINEFORM16 ), or INLINEFORM17 , or any of the INLINEFORM18 segmentation possibilities for INLINEFORM19 .
Question: What language do they look at?
Paper: Text Length Adaptation in Sentiment Classification
Text classification can be categorized according to the text length of the data, from sentence-level classification BIBREF0 to document-level classification BIBREF1, BIBREF2. One kind of such task is sentiment classification BIBREF3, a subtask of sentiment analysis BIBREF4, BIBREF5 where we are to predict the sentiment/rating given a review written by a user. In some domains, the length of these reviews varies widely. For example, well-known review websites in East Asia such as Naver Movies and Douban Movies provide two channels for users to write reviews, depending on their preferred length. Figure FIGREF4 shows the review channels provided in Naver Movies.
Question: What dierse domains and languages are present in new datasets?
Paper: Vietnamese Semantic Role Labelling
In this paper, we study semantic role labelling (SRL), a subtask of semantic parsing of natural language sentences. SRL is the task of identifying semantic roles of arguments of each predicate in a sentence. In particular, it answers a question Who did what to whom, when, where, why?. For each predicate in a sentence, the goal is to identify all constituents that fill a semantic role, and to determine their roles, such as agent, patient, or instrument, and their adjuncts, such as locative, temporal or manner.
Question: Are their corpus and software public?
Paper: Emotionally-Aware Chatbots: A Survey
Conversational agents or dialogue systems development are gaining more attention from both industry and academia BIBREF0 , BIBREF1 in the latest years. Some works tried to model them into domain-specific tasks such as customer service BIBREF2 , BIBREF3 , and shopping assistance BIBREF4 . Other works design a multi-purpose agents such as SIRI, Amazon Alexa, and Google Assistance. This domain is a well-researched area in Human-Computer Interaction research community but still become a hot topic now. The main development focus right now is to have an intelligent and humanizing machine to have a better engagement when communicating with human BIBREF5 . Having a better engagement will lead to higher user satisfaction, which becomes the main objective from the industry perspective.
Question: How are EAC evaluated?
Answer: Qualitatively through efficiency, effectiveness and satisfaction aspects and quantitatively through metrics such as precision, recall, accuracy, BLEU score and even human judgement.
Paper: Emotionally-Aware Chatbots: A Survey
The early development of chatbot was inspired by Turing test in 1950 BIBREF20 . Eliza was the first publicly known chatbot, built by using simple hand-crafted script BIBREF21 . Parry BIBREF22 was another chatbot which successfully passed the Turing test. Similar to Eliza, Parry still uses a rule-based approach but with a better understanding, including the mental model that can stimulate emotion. Therefore, Parry is the first chatbot which involving emotion in its development. Also, worth to be mentioned is ALICE (Artificial Linguistic Internet Computer Entity), a customizable chatbot by using Artificial Intelligence Markup Language (AIML). Therefore, ALICE still also use a rule-based approach by executing a pattern-matcher recursively to obtain the response. Then in May 2014, Microsoft introduced XiaoIce BIBREF23 , an empathetic social chatbot which is able to recognize users' emotional needs. XiaoIce can provide an engaging interpersonal communication by giving encouragement or other affective messages, so that can hold human attention during communication.
Question: What are the currently available datasets for EAC?
Paper: Emotionally-Aware Chatbots: A Survey
As we mentioned before that emotion is an essential aspect of building humanize chatbot. The rise of the emotionally-aware chatbot is started by Parry BIBREF22 in early 1975. Now, most of EAC development exploits neural-based model. In this section, we will try to review previous works which focus on EAC development. Table TABREF10 summarizes this information includes the objective and exploited approach of each work. In early development, EAC is designed by using a rule-based approach. However, in recent years mostly EAC exploit neural-based approach. Studies in EAC development become a hot topic start from 2017, noted by the first shared task in Emotion Generation Challenge on NLPCC 2017 BIBREF31 . Based on Table TABREF10 this research line continues to gain massive attention from scholars in the latest years.
Question: What are the research questions posed in the paper regarding EAC studies?
Paper: Should All Cross-Lingual Embeddings Speak English?
Continuous distributional vectors for representing words (embeddings) BIBREF0 have become ubiquitous in modern, neural NLP. Cross-lingual representations BIBREF1 additionally represent words from various languages in a shared continuous space, which in turn can be used for Bilingual Lexicon Induction (BLI). BLI is often the first step towards several downstream tasks such as Part-Of-Speech (POS) tagging BIBREF2, parsing BIBREF3, document classification BIBREF4, and machine translation BIBREF5, BIBREF6, BIBREF7.
Question: What evaluation metrics did they use?
Paper: Should All Cross-Lingual Embeddings Speak English?
In the supervised bilingual setting, as formulated by BIBREF1, given two languages $\mathcal {L} = \lbrace l_1,l_2\rbrace $ and their pre-trained row-aligned embeddings $\mathcal {X}_1, \mathcal {X}_2,$ respectively, a transformation matrix $$ is learned such that:
Question: What is triangulation?
Answer: Answer with content missing: (Chapter 3) The concept can be easily explained with an example, visualized in Figure 1. Consider the Portuguese (Pt) word trabalho which, according to the MUSE Pt–En dictionary, has the words job and work as possible En translations. In turn, these two En words can be translated to 4 and 5 Czech (Cs) words respectively. By utilizing the transitive property (which translation should exhibit) we can identify the set of 7 possible Cs translations for the Pt word trabalho.
Paper: Should All Cross-Lingual Embeddings Speak English?
One of the most common downstream evaluation tasks for the learned cross-lingual word mappings is Lexicon Induction (LI), the task of retrieving the most appropriate word-level translation for a query word from the mapped embedding spaces. Specialized evaluation (and training) dictionaries have been created for multiple language pairs, with the MUSE dictionaries BIBREF12 most often used, providing word translations between English (En) and 48 other high- to mid-resource languages, as well as on all 30 pairs among 6 very similar Romance and Germanic languages (English, French, German, Spanish, Italian, Portuguese).
Question: What languages are explored in this paper?
Paper: Automatically Annotated Turkish Corpus for Named Entity Recognition and Text Categorization using Large-Scale Gazetteers
Named-entity recognition (NER) is an information extraction (IE) task that aims to detect and categorize entities to pre-defined types in a text. On the other hand, the goal of text categorization (TC) is to assign correct categories to texts based on their content. Most NER and TC studies focus on English, hence accessing available English datasets is not a issue. However, the annotated datasets for Turkish NER and TC are scarce. It is hard to manually construct datasets for these tasks due to excessive human effort, time and budget. In this paper, our motivation is to construct an automatically annotated dataset that would be very useful for NER and TC researches in Turkish.
Question: Did they experiment with the dataset on some tasks?
Paper: Hybrid Tiled Convolutional Neural Networks for Text Sentiment Classification
Sentiment analysis or opinion mining is a sort of text classification that assigns a sentiment orientation to documents based on the detected contextual polarity BIBREF0. In the past, research work has focused only on the overall sentiment of a document, trying to determine if the entire text is positive, neutral or negative BIBREF1. However, besides predicting the general sentiment, a better understanding of the reviews could be undertaken using the more in-depth aspect based sentiment analysis (ABSA) BIBREF2, BIBREF1. Specifically, ABSA’s goal is to predict the sentiment polarity of the corpus’ target entities and aspects (e.g. the possible aspects of the entity “movie” could be “the plot”, “the actors’ acting” or “the special effects”). While it is easy to establish the text entities based on the a priori information about text corpus, the aspect terms are difficult to infer and usually the training process requires to use a predefined set of term categories BIBREF3, BIBREF4, BIBREF5.
Question: How better does the hybrid tiled CNN model perform than its counterparts?
Paper: Learning to Describe Phrases with Local and Global Contexts
When we read news text with emerging entities, text in unfamiliar domains, or text in foreign languages, we often encounter expressions (words or phrases) whose senses we are unsure of. In such cases, we may first try to examine other usages of the same expression in the text, in order to infer its meaning from this context. Failing to do so, we may consult a dictionary, and in the case of polysemous words, choose an appropriate meaning based on the context. Acquiring novel word senses via dictionary definitions is known to be more effective than contextual guessing BIBREF3 , BIBREF4 . However, very often, hand-crafted dictionaries do not contain definitions for rare or novel phrases/words, and we eventually give up on understanding them completely, leaving us with only a shallow reading of the text.
Question: Do they use pretrained word embeddings?
Paper: Network-Efficient Distributed Word2vec Training System for Large Vocabularies
Embedding words in a common vector space can enable machine learning algorithms to achieve better performance in natural language processing (NLP) tasks. Word2vec BIBREF0 is a recently proposed family of algorithms for training such vector representations from unstructured text data via shallow neural networks. The geometry of the resulting vectors was shown in BIBREF0 to capture word semantic similarity through the cosine similarity of the corresponding vectors as well as more complex semantic relationships through vector differences, such as vec(“Madrid”) - vec(“Spain”) + vec(“France”) INLINEFORM0 vec(“Paris”).
Question: Do they use skipgram version of word2vec?
Paper: Network-Efficient Distributed Word2vec Training System for Large Vocabularies
Sponsored search is a popular advertising model BIBREF14 used by web search engines, such as Google, Microsoft, and Yahoo, in which advertisers sponsor the top web search results in order to redirect user's attention from organic search results to ads that are highly relevant to the entered query.
Question: What domains are considered that have such large vocabularies?
Paper: Network-Efficient Distributed Word2vec Training System for Large Vocabularies
In this paper we focus on the skipgram approach with random negative examples proposed in BIBREF0 . This has been found to yield the best results among the proposed variants on a variety of semantic tests of the resulting vectors BIBREF7 , BIBREF0 . Given a corpus consisting of a sequence of sentences INLINEFORM0 each comprising a sequence of words INLINEFORM1 , the objective is to maximize the log likelihood: DISPLAYFORM0 
Question: Do they perform any morphological tokenization?
Paper: Network-Efficient Distributed Word2vec Training System for Large Vocabularies
Several existing word2vec training systems are limited to running on a single machine, though with multiple parallel threads of execution operating on different segments of training data. These include the original open source implementation of word2vec BIBREF0 , as well as those of Medallia BIBREF16 , and Rehurek BIBREF17 . As mentioned in the introduction, these systems would require far larger memory configurations than available on typical commodity-scale servers.
Question: How many nodes does the cluster have?
Paper: Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling
The sequence-to-sequence (seq2seq) model proposed in BIBREF0 , BIBREF1 , BIBREF2 is a neural architecture for performing sequence classification and later adopted to perform speech recognition in BIBREF3 , BIBREF4 , BIBREF5 . The model allows to integrate the main blocks of ASR such as acoustic model, alignment model and language model into a single framework. The recent ASR advancements in connectionist temporal classification (CTC) BIBREF5 , BIBREF4 and attention BIBREF3 , BIBREF6 based approaches has created larger interest in speech community to use seq2seq models. To leverage performance gains from this model as similar or better to conventional hybrid RNN/DNN-HMM models requires a huge amount of data BIBREF7 . Intuitively, this is due to the wide-range role of the model in performing alignment and language modeling along with acoustic to character label mapping at each iteration.
Question: What data do they train the language models on?
Paper: Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling
In this work, we use the attention based approach BIBREF1 as it provides an effective methodology to perform sequence-to-sequence (seq2seq) training. Considering the limitations of attention in performing monotonic alignment BIBREF18 , BIBREF19 , we choose to use CTC loss function to aid the attention mechanism in both training and decoding. The basic network architecture is shown in Fig. FIGREF7 .
Question: Do they report BLEU scores?
Paper: Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling
In this work, the experiments are conducted using the BABEL speech corpus collected from the IARPA babel program. The corpus is mainly composed of conversational telephone speech (CTS) but some scripted recordings and far field recordings are presented as well. Table TABREF14 presents the details of the languages used in this work for training and evaluation.
Question: What languages do they use?
Answer: Train languages are: Cantonese, Bengali, Pashto, Turkish, Vietnamese, Haitian, Tamil, Kurdish, Tokpisin and Georgian, while Assamese, Tagalog, Swahili, Lao are used as target languages.
Paper: Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling
Multilingual approaches used in hybrid RNN/DNN-HMM systems BIBREF10 have been used for for tackling the problem of low-resource data condition. Some of these approaches include language adaptive training and shared layer retraining BIBREF29 . Among them, the most benefited method is the parameter sharing technique BIBREF10 . To incorporate the former approach into encoder, CTC and attention decoder model, we performed the following experiments:
Question: What architectures are explored to improve the seq2seq model?
Paper: Unsupervised Multi-modal Neural Machine Translation
Our long-term goal is to build intelligent systems that can perceive their visual environment and understand the linguistic information, and further make an accurate translation inference to another language. Since image has become an important source for humans to learn and acquire knowledge (e.g. video lectures, BIBREF1 , BIBREF2 , BIBREF3 ), the visual signal might be able to disambiguate certain semantics. One way to make image content easier and faster to be understood by humans is to combine it with narrative description that can be self-explainable. This is particularly important for many natural language processing (NLP) tasks as well, such as image caption BIBREF4 and some task-specific translation–sign language translation BIBREF5 . However, BIBREF6 demonstrates that most multi-modal translation algorithms are not significantly better than an off-the-shelf text-only machine translation (MT) model for the Multi30K dataset BIBREF7 . There remains an open question about how translation models should take advantage of visual context, because from the perspective of information theory, the mutual information of two random variables $I(X,Y)$ will always be no greater than $I(X;Y,Z)$ , due to the following fact. 
Question: Why is this work different from text-only UNMT?
Paper: Feature-Dependent Confusion Matrices for Low-Resource NER Labeling with Noisy Labels
Most languages, even with millions of speakers, have not been the center for natural language processing and are counted as low-resource for tasks like named entity recognition (NER). Similarly, even for high-resource languages, there exists only few labeled data for most entity types beyond person, location and organization. Distantly- or weakly-supervised approaches have been proposed to solve this issue, e.g., by using lists of entities for labeling raw text BIBREF0, BIBREF1. This allows obtaining large amounts of training data quickly and cheaply. Unfortunately, these labels often contain errors and learning with this noisily-labeled data is difficult and can even reduce overall performance (see, e.g. BIBREF2).
Question: What is baseline used?
Paper: Feature-Dependent Confusion Matrices for Low-Resource NER Labeling with Noisy Labels
A popular approach is modeling the relationship between noisy and clean labels, i.e., estimating $p(\hat{y}|y)$ where $y$ is the clean and $\hat{y}$ the noisy label. For example, this can be represented as a noise or confusion matrix between the clean and the noisy labels, as explained in Section SECREF3. Having its roots in statistics BIBREF4, this or similar ideas have been recently studied in NLP BIBREF2, BIBREF3, BIBREF5, image classification BIBREF6, BIBREF7, BIBREF8 and general machine learning settings BIBREF9, BIBREF10, BIBREF11. All of these methods, however, do not take the features into account that are used to represent the instances during classification. In BIBREF12 only the noise type depends on $x$ but not the actual noise model. BIBREF13 and BIBREF14 use the learned feature representation $h$ to model $p(\hat{y}|y,h(x))$ for image classification and relation extraction respectively. In the work of BIBREF15, $p(y|\hat{y},h(x))$ is estimated to clean the labels for an image classification task. The survey by BIBREF16 gives a detailed overview about other techniques for learning in the presence of noisy labels.
Question: Did they evaluate against baseline?