z-uo/qasper-squad huggingface dataset:
Paper: An Analysis of Word2Vec for the Italian Language
The W2V structure consists of a simple two-level neural network (Figure FIGREF1) with one-hot vectors representing words at the input. It can be trained in two different modes, algorithmically similar, but different in concept: Continuous Bag-of-Words (CBOW) model and Skip-Gram model. While CBOW tries to predict the target words from the context, Skip-Gram instead aims to determine the context for a given target word. The two different approaches therefore modify only the way in which the inputs and outputs are to be managed, but in any case, the network does not change, and the training always takes place between single pairs of words (placed as one-hot in input and output).
Question: Are the word embeddings tested on a NLP task?
Paper: An Analysis of Word2Vec for the Italian Language
The common words (such as “the", “of", etc.) carry very little information on the target word with which they are coupled, and through backpropagation they tend to have extremely small representative vectors in the embedding space. To solve both these problems the W2V algorithm implements a particular “subsampling" BIBREF11, which acts by eliminating some words from certain sentences. Note that the elimination of a word directly from the text means that it no longer appears in the context of any of the words of the sentence and, at the same time, a number of pairs equal to (at most) twice the size of the window relating to the deleted word will also disappear from the training set.
Question: Are the word embeddings evaluated?
Paper: An Analysis of Word2Vec for the Italian Language
Working with one-hot pairs of words means that the size of the network must be the same at input and output, and must be equal to the size of the vocabulary. So, although very simple, the network has a considerable number of parameters to train, which lead to an excessive computational cost if we are supposed to backpropagate all the elements of the one-hot vector in output.
Question: How big is dataset used to train Word2Vec for the Italian Language?
Paper: An Analysis of Word2Vec for the Italian Language
The dataset needed to train the W2V was obtained using the information extracted from a dump of the Italian Wikipedia (dated 2019.04.01), from the main categories of Italian Google News (WORLD, NATION, BUSINESS, TECHNOLOGY, ENTERTAINMENT, SPORTS, SCIENCE, HEALTH) and from some anonymized chats between users and a customer care chatbot (Laila). The dataset (composed of 2.6 GB of raw text) includes $421\,829\,960$ words divided into $17\,305\,401$ sentences.
Question: How does different parameter settings impact the performance and semantic capacity of resulting model?
Paper: An Analysis of Word2Vec for the Italian Language
To analyse the results we chose to use the test provided by BIBREF10, which consists of $19\,791$ analogies divided into 19 different categories: 6 related to the “semantic" macro-area (8915 analogies) and 13 to the “syntactic" one (10876 analogies). All the analogies are composed by two pairs of words that share a relation, schematized with the equation: $a:a^{*}=b:b^{*}$ (e.g. “man : woman = king : queen"); where $b^{*}$ is the word to be guessed (“queen"), $b$ is the word coupled to it (“king"), $a$ is the word for the components to be eliminated (“man"), and $a^{*}$ is the word for the components to be added (“woman").
Question: Are the semantic analysis findings for Italian language similar to English language version?
Paper: An Analysis of Word2Vec for the Italian Language
We first analysed 6 different implementations of the Skip-gram model each one trained for 20 epochs. Table TABREF10 shows the accuracy values (only on possible analogies) at the 20th epoch for the six models both using 3COSADD and 3COSMUL. It is interesting to note that the 3COSADD total metric, respect to 3COSMUL, seems to have slightly better results in the two extreme cases of limited learning (W5N5 and W10N20) and under the semantic profile. However, we should keep in mind that the semantic profile is the one best captured by the network in both cases, which is probably due to the nature of the database (mainly composed of articles and news that principally use an impersonal language). In any case, the improvements that are obtained under the syntactic profile lead to the 3COSMUL metric obtaining better overall results.
Question: What dataset is used for training Word2Vec in Italian language?
Paper: Improving Character-based Decoding Using Target-Side Morphological Information for Neural Machine Translation
Morphologically complex words (MCWs) are multi-layer structures which consist of different subunits, each of which carries semantic information and has a specific syntactic role. Table 1 gives a Turkish example to show this type of complexity. This example is a clear indication that word-based models are not suitable to process such complex languages. Accordingly, when translating MRLs, it might not be a good idea to treat words as atomic units as it demands a large vocabulary that imposes extra overhead. Since MCWs can appear in various forms we require a very large vocabulary to $i$ ) cover as many morphological forms and words as we can, and $ii$ ) reduce the number of OOVs. Neural models by their nature are complex, and we do not want to make them more complicated by working with large vocabularies. Furthermore, even if we have quite a large vocabulary set, clearly some words would remain uncovered by that. This means that a large vocabulary not only complicates the entire process, but also does not necessarily mitigate the OOV problem. For these reasons we propose an NMT engine which works at the character level.
Question: How are the auxiliary signals from the morphology table incorporated in the decoder?
Paper: Improving Character-based Decoding Using Target-Side Morphological Information for Neural Machine Translation
There are several models for NMT of MRLs which are designed to deal with morphological complexities. garcia2016factored and sennrich-haddow:2016:WMT adapted the factored machine translation approach to neural models. Morphological annotations can be treated as extra factors in such models. jean-EtAl:2015:ACL-IJCNLP proposed a model to handle very large vocabularies. luong-EtAl:2015:ACL-IJCNLP addressed the problem of rare words and OOVs with the help of a post-translation phase to exchange unknown tokens with their potential translations. sennrich2015neural used subword units for NMT. The model relies on frequent subword units instead of words. costajussa-fonollosa:2016:P16-2 designed a model for translating from MRLs. The model encodes source words with a convolutional module proposed by kim2015character. Each word is represented by a convolutional combination of its characters.
Question: What type of morphological information is contained in the "morphology table"?
Paper: Learning Twitter User Sentiments on Climate Change with Limited Labeled Data
Much prior work has been done at the intersection of climate change and Twitter, such as tracking climate change sentiment over time BIBREF2 , finding correlations between Twitter climate change sentiment and seasonal effects BIBREF3 , and clustering Twitter users based on climate mentalities using network analysis BIBREF4 . Throughout, Twitter has been accepted as a powerful tool given the magnitude and reach of samples unattainable from standard surveys. However, the aforementioned studies are not scalable with regards to training data, do not use more recent sentiment analysis tools (such as neural nets), and do not consider unbiased comparisons pre- and post- various climate events (which would allow for a more concrete evaluation of shocks to climate change sentiment). This paper aims to address these three concerns as follows.
Question: Do they report results only on English data?
Paper: Learning Twitter User Sentiments on Climate Change with Limited Labeled Data
We henceforth refer to a tweet affirming climate change as a “positive" sample (labeled as 1 in the data), and a tweet denying climate change as a “negative" sample (labeled as -1 in the data). All data were downloaded from Twitter in two separate batches using the “twint" scraping tool BIBREF5 to sample historical tweets for several different search terms; queries always included either “climate change" or “global warming", and further included disaster-specific search terms (e.g., “bomb cyclone," “blizzard," “snowstorm," etc.). We refer to the first data batch as “influential" tweets, and the second data batch as “event-related" tweets.
Question: Do the authors mention any confounds to their study?
Paper: Learning Twitter User Sentiments on Climate Change with Limited Labeled Data
Our first goal is to train a sentiment analysis model (on training and validation datasets) in order to perform classification inference on event-based tweets. We experimented with different feature extraction methods and classification models. Feature extractions examined include Tokenizer, Unigram, Bigram, 5-char-gram, and td-idf methods. Models include both neural nets (e.g. RNNs, CNNs) and standard machine learning tools (e.g. Naive Bayes with Laplace Smoothing, k-clustering, SVM with linear kernel). Model accuracies are reported in Table FIGREF3 .
Question: Which machine learning models are used?
Paper: Learning Twitter User Sentiments on Climate Change with Limited Labeled Data
Our second goal is to compare the mean values of users' binary sentiments both pre- and post- each natural disaster event. Applying our highest-performing RNN to event-related tweets yields the following breakdown of positive tweets: Bomb Cyclone (34.7%), Mendocino Wildfire (80.4%), Hurricane Florence (57.2%), Hurricane Michael (57.6%), and Camp Fire (70.1%). As sanity checks, we examine the predicted sentiments on a subset with geographic user information and compare results to the prior literature.
Question: What methodology is used to compensate for limited labelled data?
Answer: Influential tweeters ( who they define as individuals certain to have a classifiable sentiment regarding the topic at hand) is used to label tweets in bulk in the absence of manually-labeled tweets.
Paper: Learning Twitter User Sentiments on Climate Change with Limited Labeled Data
In Figure FIGREF8 , we see that overall sentiment averages rarely show movement post-event: that is, only Hurricane Florence shows a significant difference in average tweet sentiment pre- and post-event at the 1% level, corresponding to a 0.12 point decrease in positive climate change sentiment. However, controlling for the same group of users tells a different story: both Hurricane Florence and Hurricane Michael have significant tweet sentiment average differences pre- and post-event at the 1% level. Within-cohort, Hurricane Florence sees an increase in positive climate change sentiment by 0.21 points, which is contrary to the overall average change (the latter being likely biased since an influx of climate change deniers are likely to tweet about hurricanes only after the event). Hurricane Michael sees an increase in average tweet sentiment of 0.11 points, which reverses the direction of tweets from mostly negative pre-event to mostly positive post-event. Likely due to similar bias reasons, the Mendocino wildfires in California see a 0.06 point decrease in overall sentiment post-event, but a 0.09 point increase in within-cohort sentiment. Methodologically, we assert that overall averages are not robust results to use in sentiment analyses.
Question: Which five natural disasters were examined?
Paper: A multimodal deep learning approach for named entity recognition from social media
A common social media delivery system such as Twitter supports various media types like video, image and text. This media allows users to share their short posts called Tweets. Users are able to share their tweets with other users that are usually following the source user. Hovewer there are rules to protect the privacy of users from unauthorized access to their timeline BIBREF0. The very nature of user interactions in Twitter micro-blogging social media is oriented towards their daily life, first witness news-reporting and engaging in various events (sports, political stands etc.). According to studies, news in twitter is propagated and reported faster than conventional news media BIBREF1. Thus, extracting first hand news and entities occurring in this fast and versatile online media gives valuable information. However, abridged and noisy content of Tweets makes it even more difficult and challenging for tasks such as named entity recognition and information retrieval BIBREF2.
Question: Which social media platform is explored?
Paper: A multimodal deep learning approach for named entity recognition from social media
Many algorithms and methods have been proposed to detect, classify or extract information from single type of data such as audio, text, image etc. However, in the case of social media, data comes in a variety of types such as text, image, video or audio in a bounded style. Most of the time, it is very common to caption a video or image with textual information. This information about the video or image can refer to a person, location etc. From a multimodal learning perspective, jointly computing such data is considered to be more valuable in terms of representation and evaluation. Named entity recognition task, on the other hand, is the task of recognizing named entities from a sentence or group of sentences in a document format.
Question: What datasets did they use?
Paper: A multimodal deep learning approach for named entity recognition from social media
The recognition of named entities from only textual data (unimodal learning approach) is a well studied and explored research criteria. For a prominent example of this category, the Stanford NER is a widely used baseline for many applications BIBREF18. The incorporation of non-local information in information extraction is proposed by the authors using of Gibbs sampling. The conditional random field (CRF) approach used in this article, creates a chain of cliques, where each clique represents the probabilistic relationship between two adjacent states. Also, Viterbi algorithm has been used to infer the most likely state in the CRF output sequence. Equation DISPLAY_FORM5 shows the proposed CRF method.
Question: What are the baseline state of the art models?
Answer: Stanford NER, BiLSTM+CRF, LSTM+CNN+CRF, T-NER and BiLSTM+CNN+Co-Attention
Paper: Uncover Sexual Harassment Patterns from Personal Stories by Joint Key Element Extraction and Categorization
Sexual violence, including harassment, is a pervasive, worldwide problem with a long history. This global problem has finally become a mainstream issue thanks to the efforts of survivors and advocates. Statistics show that girls and women are put at high risk of experiencing harassment. Women have about a 3 in 5 chance of experiencing sexual harassment, whereas men have slightly less than 1 in 5 chance BIBREF0, BIBREF1, BIBREF2. While women in developing countries are facing distinct challenges with sexual violence BIBREF3, however sexual violence is ubiquitous. In the United States, for example, there are on average >300,000 people who are sexually assaulted every year BIBREF4. Additionally, these numbers could be underestimated, due to reasons like guilt, blame, doubt and fear, which stopped many survivors from reporting BIBREF5. Social media can be a more open and accessible channel for those who have experienced harassment to be empowered to freely share their traumatic experiences and to raise awareness of the vast scale of sexual harassment, which then allows us to understand and actively address abusive behavior as part of larger efforts to prevent future sexual harassment. The deadly gang rape of a medical student on a Delhi bus in 2012 was a catalyst for protest and action, including the development of Safecity, which uses online and mobile technology to work towards ending sexual harassment and assault. More recently, the #MeToo and #TimesUp movements, further demonstrate how reporting personal stories on social media can raise awareness and empower women. Millions of people around the world have come forward and shared their stories. Instead of being bystanders, more and more people become up-standers, who take action to protest against sexual harassment online. The stories of people who experienced harassment can be studied to identify different patterns of sexual harassment, which can enable solutions to be developed to make streets safer and to keep women and girls more secure when navigating city spaces BIBREF6. In this paper, we demonstrated the application of natural language processing (NLP) technologies to uncover harassment patterns from social media data. We made three key contributions:
Question: What is the size of the dataset?
Paper: Uncover Sexual Harassment Patterns from Personal Stories by Joint Key Element Extraction and Categorization
Conventional surveys and reports are often used to study sexual harassment, but harassment on these is usually under-reported BIBREF2, BIBREF5. The high volume of social media data available online can provide us a much larger collection of firsthand stories of sexual harassment. Social media data has already been used to analyze and predict distinct societal and health issues, in order to improve the understanding of wide-reaching societal concerns, including mental health, detecting domestic abuse, and cyberbullying BIBREF11, BIBREF12, BIBREF13, BIBREF14.
Question: What model did they use?
Paper: Uncover Sexual Harassment Patterns from Personal Stories by Joint Key Element Extraction and Categorization
We obtained 9,892 stories of sexual harassment incidents that was reported on Safecity. Those stories include a text description, along with tags of the forms of harassment, e.g. commenting, ogling and groping. A dataset of these stories was published by Karlekar and Bansal karlekar2018safecity. In addition to the forms of harassment, we manually annotated each story with the key elements (i.e. “harasser", “time", “location", “trigger"), because they are essential to uncover the harassment patterns. An example is shown in Figure FIGREF3. Furthermore, we also assigned each story classification labels in five dimensions (Table TABREF4). The detailed definitions of classifications in all dimensions are explained below.
Question: What patterns were discovered from the stories?
Paper: Uncover Sexual Harassment Patterns from Personal Stories by Joint Key Element Extraction and Categorization
The key elements can be very informative when categorizing the incidents. For instance, in Figure 1, with identified key elements, one can easily categorize the incident in dimensions of “age of harasser” (adult), “single/multiple harasser(s)” (single), “type of harasser” (unspecified), “type of location” (park) , “time of day” (day time). Therefore, we proposed two joint learning schemes to extract the key elements and categorize the incidents together. In the models' names, “J”, “A”, “SA” stand for joint learning, attention, and supervised attention, respectively.
Question: Did they use a crowdsourcing platform?
Paper: Domain Adaptation of Recurrent Neural Networks for Natural Language Understanding
Slot filling models are a useful method for simple natural language understanding tasks, where information can be extracted from a sentence and used to perform some structured action. For example, dates, departure cities and destinations represent slots to fill in a flight booking task. This information is extracted from natural language queries leveraging typical context associated with each slot type. Researchers have been exploring data-driven approaches to learning models for automatic identification of slot information since the 90's, and significant advances have been made BIBREF0 . Our paper builds on recent work on slot-filling using recurrent neural networks (RNNs) with a focus on the problem of training from minimal annotated data, taking an approach of sharing data from multiple tasks to reduce the amount of data for developing a new task.
Question: Does the performance increase using their method?
Paper: Domain Adaptation of Recurrent Neural Networks for Natural Language Understanding
Our model has a word embedding layer, followed by a bi-directional LSTM (bi-LSTM), and a softmax output layer. The bi-LSTM allows the model to use information from both the right and left contexts of each word when making predictions. We choose this architecture because similar models have been used in prior work on slot filling and have achieved good results BIBREF16 , BIBREF11 . The LSTM gates are used as defined by Sak et al. including the use of the linear projection layer on the output of the LSTM BIBREF22 . The purpose of the projection layer is to produce a model with fewer parameters without reducing the number of LSTM memory cells. For the multi-task model, the word embeddings and the bi-LSTM parameters are shared across tasks but each task has its own softmax layer. This means that if the multi-task model has half a million parameters, only a couple thousand of them are unique to each task and the other 99.5% are shared between all of the tasks.
Question: What tasks are they experimenting with in this paper?
Paper: Domain Adaptation of Recurrent Neural Networks for Natural Language Understanding
Crowd-sourced data was collected simulating common use cases for four different apps: United Airlines, Airbnb, Greyhound bus service and OpenTable. The corresponding actions are booking a flight, renting a home, buying bus tickets, and making a reservation at a restaurant. In order to elicit natural language, crowd workers were instructed to simulate a conversation with a friend planning an activity as opposed to giving a command to the computer. Workers were prompted with a slot type/value pair and asked to form a reply to their friend using that information. The instructions were to not include any other potential slots in the sentence but this instruction was not always followed by the workers.
Question: What is the size of the open vocabulary?
Paper: Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
Pre-trained language representation models, including feature-based methods BIBREF0 , BIBREF1 and fine-tuning methods BIBREF2 , BIBREF3 , BIBREF4 , can capture rich language information from text and then benefit many NLP tasks. Bidirectional Encoder Representations from Transformers (BERT) BIBREF4 , as one of the most recently developed models, has produced the state-of-the-art results by simple fine-tuning on various NLP tasks, including named entity recognition (NER) BIBREF5 , text classification BIBREF6 , natural language inference (NLI) BIBREF7 , question answering (QA) BIBREF8 , BIBREF9 , and has achieved human-level performances on several datasets BIBREF8 , BIBREF9 .
Question: How do they select answer candidates for their QA task?
Paper: What we write about when we write about causality: Features of causal statements across large-scale social discourse
Social media and online social networks now provide vast amounts of data on human online discourse and other activities BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 . With so much communication taking place online and with social media being capable of hosting powerful misinformation campaigns BIBREF7 such as those claiming vaccines cause autism BIBREF8 , BIBREF9 , it is more important than ever to better understand the discourse of causality and the interplay between online communication and the statement of cause and effect.
Question: How do they extract causality from text?
Answer: They identify documents that contain the unigrams 'caused', 'causing', or 'causes'