z-uo/qasper-squad huggingface dataset:
Paper: Deeper Task-Specificity Improves Joint Entity and Relation Extraction
Multi-task learning (MTL) refers to machine learning approaches in which information and representations are shared to solve multiple, related tasks. Relative to single-task learning approaches, MTL often shows improved performance on some or all sub-tasks and can be more computationally efficient BIBREF0, BIBREF1, BIBREF2, BIBREF3. We focus here on a form of MTL known as hard parameter sharing. Hard parameter sharing refers to the use of deep learning models in which inputs to models first pass through a number of shared layers. The hidden representations produced by these shared layers are then fed as inputs to a number of task-specific layers.
Question: Do they repot results only on English data?
Paper: Deeper Task-Specificity Improves Joint Entity and Relation Extraction
We focus in this section on previous deep learning approaches to solving the tasks of NER and RE, as this work is most directly comparable to our proposal. Most work on joint NER and RE has adopted a BIO or BILOU scheme for the NER task, where each token is labeled to indicate whether it is the (B)eginning of an entity, (I)nside an entity, or (O)utside an entity. The BILOU scheme extends these labels to indicate if a token is the (L)ast token of an entity or is a (U)nit, i.e. the only token within an entity span.
Question: What were the variables in the ablation study?
Paper: Deeper Task-Specificity Improves Joint Entity and Relation Extraction
The architecture proposed here is inspired by several previous proposals BIBREF10, BIBREF11, BIBREF12. We treat the NER task as a sequence labeling problem using BIO labels. Token representations are first passed through a series of shared, BiRNN layers. Stacked on top of these shared BiRNN layers is a sequence of task-specific BiRNN layers for both the NER and RE tasks. We take the number of shared and task-specific layers to be a hyperparameter of the model. Both sets of task-specific BiRNN layers are followed by task-specific scoring and output layers. Figure FIGREF4 illustrates this architecture. Below, we use superscript $e$ for NER-specific variables and layers and superscript $r$ for RE-specific variables and layers.
Question: How many shared layers are in the system?
Answer: 1
Paper: Deeper Task-Specificity Improves Joint Entity and Relation Extraction
We obtain contextual token embeddings using the pre-trained ELMo 5.5B model BIBREF13. For each token in the input text $t_i$, this model returns three vectors, which we combine via a weighted averaging layer. Each token $t_i$'s weighted ELMo embedding $\mathbf {t}^{elmo}_{i}$ is concatenated to a pre-trained GloVe embedding BIBREF14 $\mathbf {t}^{glove}_{i}$, a character-level word embedding $\mathbf {t}^{char}_i$ learned via a single BiRNN layer BIBREF15 and a one-hot encoded casing vector $\mathbf {t}^{casing}_i$. The full representation of $t_i$ is given by $\mathbf {v}_i$ (where $\circ $ denotes concatenation):
Question: How many additional task-specific layers are introduced?
Answer: 2 for the ADE dataset and 3 for the CoNLL04 dataset
Paper: A neural network system for transformation of regional cuisine style
With growing diversity in personal food preference and regional cuisine style, personalized information systems that can transform a recipe into any selected regional cuisine style that a user might prefer would help food companies and professional chefs create new recipes.
Question: What is barycentric Newton diagram?
Paper: Measuring Social Bias in Knowledge Graph Embeddings
Recent work in the word embeddings literature has shown that embeddings encode gender and racial biases, BIBREF0, BIBREF1, BIBREF2. These biases can have harmful effects in downstream tasks including coreference resolution, BIBREF3 and machine translation, BIBREF4, leading to the development of a range of methods to try to mitigate such biases, BIBREF0, BIBREF5. In an adjacent literature, learning embeddings of knowledge graph (KG) entities and relations is becoming an increasingly common first step in utilizing KGs for a range of tasks, from missing link prediction, BIBREF6, BIBREF7, to more recent methods integrating learned embeddings into language models, BIBREF8, BIBREF9, BIBREF10.
Question: Do they propose any solution to debias the embeddings?
Paper: Measuring Social Bias in Knowledge Graph Embeddings
Graph embeddings are a vector representation of dimension $d$ of all entities and relations in a KG. To learn these representations, we define a score function $g(.)$ which takes as input the embeddings of a fact in triple form and outputs a score, denoting how likely this triple is to be correct.
Question: How are these biases found?
Paper: Self-Attentional Models Application in Task-Oriented Dialogue Generation Systems
Task-oriented chatbots are a type of dialogue generation system which tries to help the users accomplish specific tasks, such as booking a restaurant table or buying movie tickets, in a continuous and uninterrupted conversational interface and usually in as few steps as possible. The development of such systems falls into the Conversational AI domain which is the science of developing agents which are able to communicate with humans in a natural way BIBREF0. Digital assistants such as Apple's Siri, Google Assistant, Amazon Alexa, and Alibaba's AliMe are examples of successful chatbots developed by giant companies to engage with their customers.
Question: How many layers of self-attention does the model have?
Answer: 1, 4, 8, 16, 32, 64
Paper: Self-Attentional Models Application in Task-Oriented Dialogue Generation Systems
End-to-end architectures are among the most used architectures for research in the field of conversational AI. The advantage of using an end-to-end architecture is that one does not need to explicitly train different components for language understanding and dialogue management and then concatenate them together. Network-based end-to-end task-oriented chatbots as in BIBREF4, BIBREF8 try to model the learning task as a policy learning method in which the model learns to output a proper response given the current state of the dialogue. As discussed before, all encoder-decoder sequence modelling methods can be used for training end-to-end chatbots. Eric and Manning eric2017copy use the copy mechanism augmentation on simple recurrent neural sequence modelling and achieve good results in training end-to-end task-oriented chatbots BIBREF9.
Question: Is human evaluation performed?
Paper: Self-Attentional Models Application in Task-Oriented Dialogue Generation Systems
Sequence modelling methods usually fall into recurrence-based, convolution-based, and self-attentional-based methods. In recurrence-based sequence modeling, the words are fed into the model in a sequential way, and the model learns the dependencies between the tokens given the context from the past (and the future in case of bidirectional Recurrent Neural Networks (RNNs)) BIBREF14. RNNs and their variations such as Long Short-term Memory (LSTM) BIBREF15, and Gated Recurrent Units (GRU) BIBREF16 are the most widely used recurrence-based models used in sequence modelling tasks. Convolution-based sequence modelling methods rely on Convolutional Neural Networks (CNN) BIBREF17 which are mostly used for vision tasks but can also be used for handling sequential data. In CNN-based sequence modelling, multiple CNN layers are stacked on top of each other to give the model the ability to learn long-range dependencies. The stacking of layers in CNNs for sequence modeling allows the model to grow its receptive field, or in other words context size, and thus can model complex dependencies between different sections of the input sequence BIBREF18, BIBREF19. WaveNet van2016wavenet, used in audio synthesis, and ByteNet kalchbrenner2016neural, used in machine translation tasks, are examples of models trained using convolution-based sequence modelling.
Question: What are the three datasets used?
Paper: A Large-Scale Corpus for Conversation Disentanglement
When a group of people communicate in a common channel there are often multiple conversations occurring concurrently. Often there is no explicit structure identifying conversations or their structure, such as in Internet Relay Chat (IRC), Google Hangout, and comment sections on websites. Even when structure is provided it often has limited depth, such as threads in Slack, which provide one layer of branching. In all of these cases, conversations are entangled: all messages appear together, with no indication of separate conversations. Automatic disentanglement could be used to provide more interpretable results when searching over chat logs, and to help users understand what is happening when they join a channel. Over a decade of research has considered conversation disentanglement BIBREF0 , but using datasets that are either small BIBREF1 or not released BIBREF2 .
Question: Did they experiment with the corpus?
Paper: Visualizing and Measuring the Geometry of BERT
Neural networks for language processing have advanced rapidly in recent years. A key breakthrough was the introduction of transformer architectures BIBREF0 . One recent system based on this idea, BERT BIBREF1 , has proven to be extremely flexible: a single pretrained model can be fine-tuned to achieve state-of-the-art performance on a wide variety of NLP applications. This suggests the model is extracting a set of generally useful features from raw text. It is natural to ask, which features are extracted? And how is this information represented internally?
Question: How were the feature representations evaluated?
Paper: Visualizing and Measuring the Geometry of BERT
Our object of study is the BERT model introduced in BIBREF1 . To set context and terminology, we briefly describe the model's architecture. The input to BERT is based on a sequence of tokens (words or pieces of words). The output is a sequence of vectors, one for each input token. We will often refer to these vectors as context embeddings because they include information about a token's context.
Question: What linguistic features were probed for?
Paper: When redundancy is rational: A Bayesian approach to 'overinformative' referring expressions
Reference to objects is one of the most basic and prevalent uses of language. In order to refer, speakers must choose from among a wealth of referring expressions they have at their disposal. How does a speaker choose whether to refer to an object as the animal, the dog, the dalmatian, or the big mostly white dalmatian? The context within which the object occurs (other non-dogs, other dogs, other dalmatians) plays a large part in determining which features the speaker chooses to include in their utterance – speakers aim to be sufficiently informative to establish unique reference to the intended object. However, speakers' utterances often exhibit what has been claimed to be overinformativeness: referring expressions are often more specific than necessary for establishing unique reference, and they are more specific in systematic ways. For instance, speakers are likely to produce referring expressions like the small blue pin instead of the small pin in contexts like Figure 1 , even though the color modifier provides no additional information BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . Similar use of redundant size modifiers, in contrast, is rare. Providing a unified theory for speakers' systematic patterns of overinformativeness has so far proven elusive.
Question: Does the paper describe experiments with real humans?
Paper: ASR-free CNN-DTW keyword spotting using multilingual bottleneck features for almost zero-resource languages
Social media has become a popular medium for individuals to express opinions and concerns on issues impacting their lives BIBREF0 , BIBREF1 , BIBREF2 . In countries without adequate internet infrastructure, like Uganda, communities often use phone-in talk shows on local radio stations for the same purpose. In an ongoing project by the United Nations (UN), radio-browsing systems have been developed to monitor such radio shows BIBREF3 , BIBREF4 . These systems are actively and successfully supporting UN relief and developmental programmes. The development of such systems, however, remains dependent on the availability of transcribed speech in the target languages. This dependence has proved to be a key impediment to the rapid deployment of radio-browsing systems in new languages, since skilled annotators proficient in the target languages are hard to find, especially in crisis conditions.
Question: What are bottleneck features?
Paper: ASR-free CNN-DTW keyword spotting using multilingual bottleneck features for almost zero-resource languages
The first radio browsing systems implemented as part of the UN's humanitarian monitoring programmes rely on ASR systems BIBREF3 . Human analysts filter speech segments identified by the system and add these to a searchable database to support decision making. To develop the ASR system, at least a small amount of annotated speech in the target language is required BIBREF4 . However, the collection of even a small fully transcribed corpus has proven difficult or impossible in some settings. In recent work, we have therefore proposed an ASR-free keyword spotting system based on CNNs BIBREF18 . CNN classifiers typically require a large number of training examples, which are not available in our setting. Instead, we therefore use a small set of recorded isolated keywords, which are then matched against a large collection of untranscribed speech drawn from the target domain using a DTW-based approach. The resulting DTW scores are then used as targets for a CNN. The key is that it is not necessary to know whether or not the keywords do in fact occur in this untranscribed corpus; the CNN is trained simply to emulate the behaviour of the DTW. Since the CNN does not perform any alignment, it is computationally much more efficient than DTW. The resulting CNN-DTW model can therefore be used to efficiently detect the presence of keywords in new input speech. Figure FIGREF2 show the structure of this CNN-DTW radio browsing system.
Question: What languages are considered?
Paper: Rnn-transducer with language bias for end-to-end Mandarin-English code-switching speech recognition
Code-switching (CS) speech is defined as the alternation of languages in an utterance, it is a pervasive communicative phenomenon in multilingual communities. Therefore, developing a CS speech recognition (CSSR) system is of great interest.
Question: Do they compare speed performance of their model compared to the ones using the LID model?
Paper: Rnn-transducer with language bias for end-to-end Mandarin-English code-switching speech recognition
Although CTC has been applied successfully in the context of speech recognition, it assumes that outputs at each step are independent of the previous predictions BIBREF6. RNN-T is an improved model based on CTC, it augments with a prediction network, which is explicitly conditioned on the previous outputs BIBREF10, as illustrated in Fig. 2(a).
Question: How do they obtain language identities?
Paper: Embedding Multimodal Relational Data for Knowledge Base Completion
Knowledge bases (KB) are an essential part of many computational systems with applications in search, structured data management, recommendations, question answering, and information retrieval. However, KBs often suffer from incompleteness, noise in their entries, and inefficient inference under uncertainty. To address these issues, learning relational knowledge representations has been a focus of active research BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . These approaches represent relational triples, that consist of a subject entity, relation, and an object entity, by learning fixed, low-dimensional representations for each entity and relation from observations, encoding the uncertainty and inferring missing facts accurately and efficiently. The subject and the object entities come from a fixed, enumerable set of entities that appear in the knowledge base. Knowledge bases in the real world, however, contain a wide variety of data types beyond these direct links. Apart from relations to a fixed set of entities, KBs often not only include numerical attributes (such as ages, dates, financial, and geoinformation), but also textual attributes (such as names, descriptions, and titles/designations) and images (profile photos, flags, posters, etc.). These different types of data can play a crucial role as extra pieces of evidence for knowledge base completion. For example the textual descriptions and images might provide evidence for a person's age, profession, and designation. In the multimodal KB shown in Figure 1 for example, the image can be helpful in predicting of Carles Puyol's occupation, while the description contains his nationality. Incorporating this information into existing approaches as entities, unfortunately, is challenging as they assign each entity a distinct vector and predict missing links (or attributes) by enumerating over the possible values, both of which are only possible if the entities come from a small, enumerable set. There is thus a crucial need for relational modeling that goes beyond just the link-based view of KB completion, by not only utilizing multimodal information for better link prediction between existing entities, but also being able to generate missing multimodal values.
Question: What other multimodal knowledge base embedding methods are there?
Paper: Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
Machine Reading Comprehension (MRC) has gained growing interest in the research community BIBREF0 , BIBREF1 . In an MRC task, the machine reads a text passage and a question, and generates (or selects) an answer based on the passage. This requires the machine to possess strong comprehension, inference and reasoning capabilities. Over the past few years, there has been much progress in building end-to-end neural network models BIBREF2 for MRC. However, most public MRC datasets (e.g., SQuAD, MS MARCO, TriviaQA) are typically small (less than 100K) compared to the model size (such as SAN BIBREF3 , BIBREF4 with around 10M parameters). To prevent over-fitting, recently there have been some studies on using pre-trained word embeddings BIBREF5 and contextual embeddings in the MRC model training, as well as back-translation approaches BIBREF1 for data augmentation.
Question: What is the data selection paper in machine translation
Paper: Additive Margin SincNet for Speaker Recognition
Speaker Recognition is an essential task with applications in biometric authentication, identification, and security among others BIBREF0 . The field is divided into two main subtasks: Speaker Identification and Speaker Verification. In Speaker Identification, given an audio sample, the model tries to identify to which one in a list of predetermined speakers the locution belongs. In the Speaker Verification, the model verifies if a sampled audio belongs to a given speaker or not. Most of the literature techniques to tackle this problem are based on INLINEFORM0 -vectors methods BIBREF1 , which extract features from the audio samples and classify the features using methods such as PLDA BIBREF2 , heavy-tailed PLDA BIBREF3 , and Gaussian PLDA BIBREF4 .
Question: Do they compare computational time of AM-softmax versus Softmax?
Paper: Additive Margin SincNet for Speaker Recognition
For some time, INLINEFORM0 -vectors BIBREF1 have been used as the state-of-the-art feature extraction method for speaker recognition tasks. Usually, the extracted features are classified using PLDA BIBREF2 or other similar techniques, such as heavy-tailed PLDA BIBREF3 and Gauss-PLDA BIBREF4 . The intuition behind these traditional methods and how they work can be better seem in BIBREF18 . Although they have been giving us some reasonable results, it is clear that there is still room for improvements BIBREF18 .
Question: Do they visualize the difference between AM-Softmax and regular softmax?
Paper: Bidirectional Long-Short Term Memory for Video Description
With the development of digital media technology and popularity of Mobile Internet, online visual content has increased rapidly in recent couple of years. Subsequently, visual content analysis for retrieving BIBREF0 , BIBREF1 and understanding becomes a fundamental problem in the area of multimedia research, which has motivated world-wide researchers to develop advanced techniques. Most previous works, however, have focused on classification task, such as annotating an image BIBREF2 , BIBREF3 or video BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 with given fixed label sets. With some pioneering methods BIBREF8 , BIBREF9 tackling the challenge of describing images with natural language proposed, visual content understanding has attracted more and more attention. State-of-the-art techniques for image captioning have been surpassed by new advanced approaches in succession BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 . Recent researches BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 have been focusing on describing videos with more comprehensive sentences instead of simple keywords. Different from image, video is sequential data with temporal structure, which may pose significant challenge to video caption. Most of the existing works in video description employed max or mean pooling across video frames to obtain video-level representation, which failed to capture temporal knowledge. To address this problem, Yao et al. proposed to use 3-D Convolutional Neural Networks to explore local temporal information in video clips, where the most relevant temporal fragments were automatically chosen for generating natural language description with attention mechanism BIBREF17 . In BIBREF19 , Venugopanlan et al. implemented a Long-Short Term Memory (LSTM) network, a variant of Recurrent Neural Networks (RNNs), to model the global temporal structure in whole video snippet. However, these methods failed to exploit bidirectional global temporal structure, which could benefit from not only previous video frames, but also information in future frames. Also, existing video captioning schemes cannot adaptively learn dense video representation and generate sparse semantic sentences.
Question: what metrics were used for evaluation?
Paper: Bidirectional Long-Short Term Memory for Video Description
In this section, we elaborate the proposed video captioning framework, including an introduction of the overall flowchart (as illustrated in Figure FIGREF1 ), a brief review of LSTM-based Sequential Model, the joint visual modelling with bidirectional LSTM and CNNs, as well as the sentence generation process.
Question: what are the state of the art methods?
Answer: S2VT, RGB (VGG), RGB (VGG)+Flow (AlexNet), LSTM-E (VGG), LSTM-E (C3D) and Yao et al.
Paper: NNVLP: A Neural Network-Based Vietnamese Language Processing Toolkit
We implement the deep neural network model described in BIBREF5 . This model is a combination of Bi-directional Long Short-Term Memory (Bi-LSTM), Convolutional Neural Network (CNN), and Conditional Random Field (CRF). In particular, this model takes as input a sequence of the concatenation of word embedding pre-trained by word2vec tool and character-level word feature trained by CNN. That sequence is then passed to a Bi-LSTM, and then a CRF layer takes as input the output of the Bi-LSTM to predict the best named entity output sequence. Figure FIGREF9 and Figure FIGREF10 describe the architectures of BI-LSTM-CRF layers, and CNN layer respectively.
Question: What datasets do they use for the tasks?
Paper: A Deep Learning Architecture for De-identification of Patient Notes: Implementation and Evaluation
[block] 1 5mm *
Question: What evaluation metrics do they use?