z-uo/qasper-squad huggingface dataset:
Paper: A Multi-cascaded Deep Model for Bilingual SMS Classification
Social media such as Facebook, Twitter, and Short Text Messaging Service (SMS) are popular channels for getting feedback from consumers on products and services. In Pakistan, with the emergence of e-government practices, SMS is being used for getting feedback from the citizens on different public services with the aim to reduce petty corruption and deficient delivery in services. Automatic classification of these SMS into predefined categories can greatly decrease the response time on complaints and consequently improve the public services rendered to the citizens. While Urdu is the national language of Pakistan, English is treated as the official language of the country. This leads to the development of a distinct dialect of communication known as Roman Urdu, which utilizes English alphabets to write Urdu. Hence, the SMS texts contain multilingual text written in the non-native script and informal diction. The utilization of two or more languages simultaneously is known as multilingualism BIBREF0. Consequently, alternation of two languages in a single conversation, a phenomenon known as code-switching, is inevitable for a multilingual speaker BIBREF1. Factors like informal verbiage, improper grammar, variation in spellings, code-switching, and short text length make the problem of automatic bilingual SMS classification highly challenging.
Question: What accuracy score do they obtain?
Answer: the best performing model obtained an accuracy of 0.86
Paper: A Multi-cascaded Deep Model for Bilingual SMS Classification
The dataset consists of SMS feedbacks of the citizens of Pakistan on different public services availed by them. The objective of collecting these responses is to measure the performance of government departments rendering different public services. Preprocessing of the data is kept minimal. All records having only single word in SMS were removed as cleaning step. To construct the “gold standard", $313,813$ samples are manually annotated into 12 predefined categories by two annotators in supervision of a domain-expert. Involvement of the domain-expert was to ensure the practicality and quality of the “gold standard". Finally, stratified sampling method was opted for splitting the data into train and test partitions with $80-20$ ratio (i.e., $80\%$ records for training and $20\%$ records for testing). This way, training split has $251,050$ records while testing split has $62,763$ records. The rationale behind stratified sampling was to maintain the ratio of every class in both splits. The preprocessed and annotated data along with train and test split is made available . Note that the department names and service availed by the citizens is mapped to an integer identifier for anonymity.
Question: What is their baseline model?
Paper: A Multi-cascaded Deep Model for Bilingual SMS Classification
The proposed model, named McM, is mainly inspired by the findings by Reimers, N., & Gurevych (2017) , who concluded that deeper model have minimal effect on the predictive performance of the model BIBREF7. McM manifests a wider model, which employ three feature learners (cascades) that are trained for classification independently (in parallel).
Question: What is the size of the dataset?
Paper: A Multi-cascaded Deep Model for Bilingual SMS Classification
CNN learner is employed to learn $n$-gram features for identification of relationships between words. A 1-d convolution filter is used with a sliding window (kernel) of size $k$ (number of $n$-grams) in order to extract the features. A filter $W$ is defined as $W \in \mathbb {R}^{k \times d}$ for the convolution function. The word vectors starting from the position $j$ to the position $j + k -1$ are processed by the filter $W$ at a time. The window $h_j$ is expressed as:
Question: What is the 12 class bilingual text?
Answer: Appreciation, Satisfied, Peripheral complaint, Demanded inquiry, Corruption, Lagged response, Unresponsive, Medicine payment, Adverse behavior, Grievance ascribed and Obnoxious/irrelevant
Paper: Masakhane -- Machine Translation For Africa
2144 of all 7111 (30.15%) living languages today are African languages BIBREF1. But only a small portion of linguistic resources for NLP research are built for African languages. As a result, there are only few NLP publications: In all ACL conferences in 2019, only 5 out of 2695 (0.19%) author affiliations were based in Africa BIBREF2. This stark contrast of linguistic richness versus poor representation of African languages in NLP is caused by multiple factors.
Question: Which languages do they focus on?
Paper: Simplify the Usage of Lexicon in Chinese NER
Named Entity Recognition (NER) is concerned with identifying named entities, such as person, location, product, and organization names, in unstructured text. In languages where words are naturally separated (e.g., English), NER was conventionally formulated as a sequence labeling problem, and the state-of-the-art results have been achieved by those neural-network-based models BIBREF1, BIBREF2, BIBREF3, BIBREF4.
Question: Which are the sequence model architectures this method can be transferred across?
Answer: The sequence model architectures which this method is transferred to are: LSTM and Transformer-based models
Paper: Simplify the Usage of Lexicon in Chinese NER
In this section, we provide a concise description of the generic character-based neural NER model, which conceptually contains three stacked layers. The first layer is the character representation layer, which maps each character of a sentence into a dense vector. The second layer is the sequence modeling layer. It plays the role of modeling the dependence between characters, obtaining a hidden representation for each character. The final layer is the label inference layer. It takes the hidden representation sequence as input and outputs the predicted label (with probability) for each character. We detail these three layers below.
Question:  What percentage of improvement in inference speed is obtained by the proposed method over the newest state-of-the-art methods?
Answer: Across 4 datasets, the best performing proposed model (CNN) achieved an average of 363% improvement over the state of the art method (LR-CNN)
Paper: Sampling strategies in Siamese Networks for unsupervised speech representation learning
Current speech and language technologies based on Deep Neural Networks (DNNs) BIBREF0 require large quantities of transcribed data and additional linguistic resources (phonetic dictionary, transcribed data). Yet, for many languages in the world, such resources are not available and gathering them would be very difficult due to a lack of stable and widespread orthography BIBREF1 .
Question: What is the metric that is measures in this paper?
Answer: error rate in a minimal pair ABX discrimination task
Paper: Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU
Attentional sequence-to-sequence models have become the new standard for machine translation over the last two years, and with the unprecedented improvements in translation accuracy comes a new set of technical challenges. One of the biggest challenges is the high training and decoding costs of these neural machine translation (NMT) system, which is often at least an order of magnitude higher than a phrase-based system trained on the same data. For instance, phrasal MT systems were able achieve single-threaded decoding speeds of 100-500 words/sec on decade-old CPUs BIBREF0 , while BIBREF1 reported single-threaded decoding speeds of 8-10 words/sec on a shallow NMT system. BIBREF2 was able to reach CPU decoding speeds of 100 words/sec for a deep model, but used 44 CPU cores to do so. There has been recent work in speeding up decoding by reducing the search space BIBREF3 , but little in computational improvements.
Question: Do they only test on one dataset?
Paper: Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU
The data set we evaluate on in this work is WMT English-French NewsTest2014, which has 380M words of parallel training data and a 3003 sentence test set. The NewsTest2013 set is used for validation. In order to compare our architecture to past work, we train a word-based system without any data augmentation techniques. The network architecture is very similar to BIBREF4 , and specific details of layer size/depth are provided in subsequent sections. We use an 80k source/target vocab and perform standard unk-replacement BIBREF1 on out-of-vocabulary words. Training is performed using an in-house toolkit.
Question: What baseline decoder do they use?
Paper: Transformer-based Cascaded Multimodal Speech Translation
The recently introduced How2 dataset BIBREF2 has stimulated research around multimodal language understanding through the availability of 300h instructional videos, English subtitles and their Portuguese translations. For example, BIBREF3 successfully demonstrates that semantically rich action-based visual features are helpful in the context of machine translation (MT), especially in the presence of input noise that manifests itself as missing source words. Therefore, we hypothesize that a speech-to-text translation (STT) system may also benefit from the visual context, especially in the traditional cascaded framework BIBREF4, BIBREF5 where noisy automatic transcripts are obtained from an automatic speech recognition system (ASR) and further translated into the target language using a machine translation (MT) component. The dataset enables the design of such multimodal STT systems, since we have access to a bilingual corpora as well as the corresponding audio-visual stream. Hence, in this paper, we propose a cascaded multimodal STT with two components: (i) an English ASR system trained on the How2 dataset and (ii) a transformer-based BIBREF0 visually grounded MMT system.
Question: Was evaluation metrics and criteria were used to evaluate the output of the cascaded multimodal speech translation?
Answer: BLEU scores
Paper: Transformer-based Cascaded Multimodal Speech Translation
In this section, we briefly describe the proposed multimodal speech translation system and its components.
Question: What dataset was used in this work?
Paper: Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning
Learning sentence representations from unlabelled data is becoming increasingly prevalent in both the machine learning and natural language processing research communities, as it efficiently and cheaply allows knowledge extraction that can successfully transfer to downstream tasks. Methods built upon the distributional hypothesis BIBREF0 and distributional similarity BIBREF1 can be roughly categorised into two types:
Question: How do they evaluate the sentence representations?
Paper: Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning
Learning vector representations for words with a word embedding matrix as the encoder and a context word embedding matrix as the decoder BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 can be considered as a word-level example of our approach, as the models learn to predict the surrounding words in the context given the current word, and the context word embeddings can also be utilised to augment the word embeddings BIBREF14 , BIBREF16 . We are thus motivated to explore the use of sentence decoders after learning instead of ignoring them as most sentence encoder-decoder models do.
Question: What are the two decoding functions?
Answer: a linear projection and a bijective function with continuous transformation though  ‘affine coupling layer’ of (Dinh et al.,2016).
Paper: Automatic Creation of Text Corpora for Low-Resource Languages from the Internet: The Case of Swiss German
Swiss German (“Schwyzerdütsch” or “Schwiizertüütsch”, abbreviated “GSW”) is the name of a large continuum of dialects attached to the Germanic language tree spoken by more than 60% of the Swiss population BIBREF0. Used every day from colloquial conversations to business meetings, Swiss German in its written form has become more and more popular in recent years with the rise of blogs, messaging applications and social media. However, the variability of the written form is rather large as orthography is more based on local pronunciations and emerging conventions than on a unique grammar.
Question: How is language modelling evaluated?
Paper: A Semi-automatic Method for Efficient Detection of Stories on Social Media
In recent years, Twitter, a social media platform with hundreds of millions of users, has become a major source of news for people BIBREF0 . This is especially true for breaking-news about real-world events BIBREF1 . The 2011 Japanese earthquake, the 2013 Boston marathon bombings, and the 2015 Paris shootings are just three examples of an event where Twitter played major role in the dissemination of information. However, given the great volume of tweets generated during these events, it becomes extremely difficult to make sense of all the information that is being shared. In this paper, we present a semi-automatic tool that combines state-of-the-art natural language processing and clustering algorithms in a novel way, enabling users to efficiently and accurately identify and track stories that spread on Twitter about particular events. The output of our system can also be used by rumor verification systems to substantiate the veracity of rumors on Twitter BIBREF2 .
Question: Why there is only user study to evaluate the model?
Paper: MDE: Multi Distance Embeddings for Link Prediction in Knowledge Graphs
While machine learning methods conventionally model functions given sample inputs and outputs, a subset of statistical relational learning(SRL) BIBREF0 , BIBREF1 approaches specifically aim to model “things” (entities) and relations between them. These methods usually model human knowledge which is structured in the form of multi-relational Knowledge Graphs(KG). KGs allow semantically rich queries in search engines, natural language processing (NLP) and question answering. However, they usually miss a substantial portion of true relations, i.e. they are incomplete. Therefore, the prediction of missing links/relations in KGs is a crucial challenge for SRL approaches.
Question: What datasets are used to evaluate the model?
Paper: Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Virtual assistants help users accomplish tasks including but not limited to finding flights, booking restaurants and, more recently, navigating user interfaces, by providing a natural language interface to services and APIs on the web. The recent popularity of conversational interfaces and the advent of frameworks like Actions on Google and Alexa Skills, which allow developers to easily add support for new services, has resulted in a major increase in the number of application domains and individual services that assistants need to support, following the pattern of smartphone applications.
Question: How did they gather the data?
Paper: Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Task-oriented dialogue systems have constituted an active area of research for decades. The growth of this field has been consistently fueled by the development of new datasets. Initial datasets were limited to one domain, such as ATIS BIBREF6 for spoken language understanding for flights. The Dialogue State Tracking Challenges BIBREF7, BIBREF8, BIBREF9, BIBREF10 contributed to the creation of dialogue datasets with increasing complexity. Other notable related datasets include WOZ2.0 BIBREF11, FRAMES BIBREF2, M2M BIBREF1 and MultiWOZ BIBREF0. These datasets have utilized a variety of data collection techniques, falling within two broad categories:
Question: What are the domains covered in the dataset?
Answer: Alarm
Bank
Bus
Calendar
Event
Flight
Home
Hotel
Media
Movie
Music
RentalCar
Restaurant
RideShare
Service
Travel
Weather
Paper: Neural Machine Translation System of Indic Languages -- An Attention based Approach
India is a highly diverse multilingual country in the world. In India, people of different regions use their own regional speaking languages, which makes India a country having world's second highest number of languages. Human spoken languages in India belongs to several language families. Two main of those families are typically known as Indo-Aryan languages having 78.05 percentage Indian speakers BIBREF0 and Dravidian languages having 19.64 BIBREF0 percentage Indian speakers. Hindi and Gujarati are among constitutional languages of India having nearly 601,688,479 BIBREF0 Indian speakers almost 59 BIBREF0 percentage of total country population. Constitute of India under Article 343 offers English as second additional official language having only 226,449 BIBREF0 Indian speakers and nearly 0.02 percentages of total country population BIBREF0. Communication and information exchange among people is necessary for sharing knowledge, feelings, opinions, facts, and thoughts. Variation of English is used globally for human communication. The content available on the Internet is exceptionally dominated by English. Only 20 percent of the world population speaks in English, while in India it is only 0.02 BIBREF0. It is not possible to have a human translator in the country having this much language diversity. In order to bridge this vast language gap we need effective and accurate computational approaches, which require minimum human intervention. This task can be effectively done using machine translation.
Question: What is their baseline?
Paper: Harvey Mudd College at SemEval-2019 Task 4: The Clint Buchanan Hyperpartisan News Detector
SemEval Task 4 BIBREF1 tasked participating teams with identifying news articles that are misleading to their readers, a phenomenon often associated with “fake news” distributed by partisan sources BIBREF2 .
Question: Do they use the cased or uncased BERT model?
Paper: Harvey Mudd College at SemEval-2019 Task 4: The Clint Buchanan Hyperpartisan News Detector
We build upon the Bidirectional Encoder Representations from Transformers (BERT) model. BERT is a deep bidirectional transformer that has been successfully tuned to a variety of tasks BIBREF0 . BERT functions as a language model over character sequences, with tokenization as described by BIBREF3 . The transformer architecture BIBREF4 is based upon relying on self-attention layers to encode a sequence. To allow the language model to be trained in a bidirectional manner instead of predicting tokens autoregressively, BERT was pre-trained to fill in the blanks for a piece of text, also known as the Cloze task BIBREF5 .
Question: How are the two different models trained?
Answer: They pre-train the models using 600000 articles as an unsupervised dataset and then fine-tune the models on small training set.
Paper: Harvey Mudd College at SemEval-2019 Task 4: The Clint Buchanan Hyperpartisan News Detector
Next, we describe the variations of the BERT model used in our experiments, the data we used, and details of the setup of each of our experiments.
Question: How long is the dataset?
Answer: 645, 600000
Paper: Adaptive Scheduling for Multi-Task Learning
Multiple tasks may often benefit from others by leveraging more available data. For natural language tasks, a simple approach is to pre-train embeddings BIBREF0, BIBREF1 or a language model BIBREF2, BIBREF3 over a large corpus. The learnt representations may then be used for upstream tasks such as part-of-speech tagging or parsing, for which there is less annotated data. Alternatively, multiple tasks may be trained simultaneously with either a single model or by sharing some model components. In addition to potentially benefit from multiple data sources, this approach also reduces the memory use. However, multi-task models of similar size as single-task baselines often under-perform because of their limited capacity. The underlying multi-task model learns to improve on harder tasks, but may hit a plateau, while simpler (or data poor) tasks can be over-trained (over-fitted). Regardless of data complexity, some tasks may be forgotten if the schedule is improper, also known as catastrophic forgetting BIBREF4.
Question: How big are negative effects of proposed techniques on high-resource tasks?
Answer: The negative effects were insignificant.
Paper: Adaptive Scheduling for Multi-Task Learning
A common approach for multi-task learning is to train on each task uniformly BIBREF5. Alternatively, each task may be sampled following a fixed non-uniform schedule, often favoring either a specific task of interest or tasks with larger amounts of data BIBREF6, BIBREF8. Kipperwasser and Ballesteros BIBREF8 also propose variable schedules that increasingly favor some tasks over time. As all these schedules are pre-defined (as a function of the training step or amount of available training data), they offer limited control over the performance of all tasks. As such, we consider adaptive schedules that vary based on the validation performance of each task during training.
Question: What datasets are used for experiments?
Paper: Adaptive Scheduling for Multi-Task Learning
Explicit schedules may possibly be too restrictive in some circumstances, such as models trained on a very high number of tasks, or when one task is sampled much more often than others. Instead of explicitly varying task schedules, a similar impact may be achieved through learning rate or gradient manipulation. For example, the GradNorm BIBREF9 algorithm scales task gradients based on the magnitude of the gradients as well as on the training losses.
Question: Are this techniques used in training multilingual models, on what languages?
Answer: English to French and English to German
Paper: Adaptive Scheduling for Multi-Task Learning
Scaling either the gradients $g_t$ or the per-task learning rates $\alpha $ is equivalent with standard stochastic gradient descent, but not with adaptive optimizers such as Adam BIBREF7, whose update rule is given in Eq. DISPLAY_FORM5.
Question: What baselines non-adaptive baselines are used?
Paper: Improved Semantic-Aware Network Embedding with Fine-Grained Word Alignment
Networks are ubiquitous, with prominent examples including social networks (e.g., Facebook, Twitter) or citation networks of research papers (e.g., arXiv). When analyzing data from these real-world networks, traditional methods often represent vertices (nodes) as one-hot representations (containing the connectivity information of each vertex with respect to all other vertices), usually suffering from issues related to the inherent sparsity of large-scale networks. This results in models that are not able to fully capture the relationships between vertices of the network BIBREF0 , BIBREF1 . Alternatively, network embedding (i.e., network representation learning) has been considered, representing each vertex of a network with a low-dimensional vector that preserves information on its similarity relative to other vertices. This approach has attracted considerable attention in recent years BIBREF2 , BIBREF0 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 .
Question: What text sequences are associated with each vertex?
Paper: Improved Semantic-Aware Network Embedding with Fine-Grained Word Alignment
A network (graph) is defined as $G = \lbrace V,E\rbrace $ , where $V$ and $E$ denote the set of $N$ vertices (nodes) and edges, respectively, where elements of $E$ are two-element subsets of $V$ . Here we only consider undirected networks, however, our approach (introduced below) can be readily extended to the directed case. We also define $W$ , the symmetric $\mathbb {R}^{N \times N}$ matrix whose elements, $w_{ij}$ , denote the weights associated with edges in $V$ , and $V$0 , the set of text sequences assigned to each vertex. Edges and weights contain the structural information of the network, while the text can be used to characterize the semantic properties of each vertex. Given network $V$1 , with the network embedding we seek to encode each vertex into a low-dimensional vector $V$2 (with dimension much smaller than $V$3 ), while preserving structural and semantic features of $V$4 .
Question: How long does it take for the model to run?