z-uo/qasper-squad huggingface dataset:
Paper: How Language-Neutral is Multilingual BERT?
Multilingual BERT (mBERT; BIBREF0) is gaining popularity as a contextual representation for various multilingual tasks, such as dependency parsing BIBREF1, BIBREF2, cross-lingual natural language inference (XNLI) or named-entity recognition (NER) BIBREF3, BIBREF4, BIBREF5.
Question: How they demonstrate that language-neutral component is sufficiently general in terms of modeling semantics to allow high-accuracy word-alignment?
Paper: How Language-Neutral is Multilingual BERT?
Since the publication of mBERT BIBREF0, many positive experimental results were published.
Question: Are language-specific and language-neutral components disjunctive?
Paper: How Language-Neutral is Multilingual BERT?
Following BIBREF3, we hypothesize that a sentence representation in mBERT is composed of a language-specific component, which identifies the language of the sentence, and a language-neutral component, which captures the meaning of the sentence in a language-independent way. We assume that the language-specific component is similar across all sentences in the language.
Question: How they show that mBERT representations can be split into a language-specific component and a language-neutral component?
Paper: How Language-Neutral is Multilingual BERT?
We employ five probing tasks to evaluate the language neutrality of the representations.
Question: What challenges this work presents that must be solved to build better language-neutral representations?
Paper: CAiRE: An End-to-End Empathetic Chatbot
Empathetic chatbots are conversational agents that can understand user emotions and respond appropriately. Incorporating empathy into the dialogue system is essential to achieve better human-robot interaction because naturally, humans express and perceive emotion in natural language to increase their sense of social bonding. In the early development stage of such conversational systems, most of the efforts were put into developing hand-crafted rules of engagement. Recently, a modularized empathetic dialogue system, XiaoIce BIBREF0 achieved an impressive number of conversational turns per session, which was even higher than average conversations between humans. Despite the promising results of XiaoIce, this system is designed using a complex architecture with hundreds of independent components, such as Natural Language Understanding and Response Generation modules, using a tremendous amount of labeled data for training each of them.
Question: What is the performance of their system?
Paper: CAiRE: An End-to-End Empathetic Chatbot
As shown in Figure FIGREF4 , our user interface is based solely on text inputs. Users can type anything in the input box and get a response immediately from the server. A report button is added at the bottom to allow users to report unethical dialogues, which will then be marked and saved in our back-end server separately. To facilitate the need for teaching our chatbot how to respond properly, we add an edit button next to the response. When the user clicks it, a new input box will appear, and the user can type in the appropriate response they think the chatbot should have replied with.
Question: What evaluation metrics are used?
Paper: CAiRE: An End-to-End Empathetic Chatbot
Due to the high demand for GPU computations during response generation, the computation cost needs to be well distributed across different GPUs to support multiple users. We adopt several approaches to maximize the utility of GPUs without crashing the system. Firstly, we set up two independent processes in each GTX 1080Ti, where we found the highest GPU utilities to be around 90%, with both processes working stably. Secondly, we employ a load-balancing module to distribute the requests to idle processes based on their working loads. During a stress testing, we simulated users sending requests every 2 seconds, and using 8 GPUs, we were able to support more than 50 concurrent requests.
Question: What is the source of the dialogues?
Paper: CAiRE: An End-to-End Empathetic Chatbot
We apply the Generative Pre-trained Transformer (GPT) BIBREF2 as our pre-trained language model. GPT is a multi-layer Transformer decoder with a causal self-attention which is pre-trained, unsupervised, on the BooksCorpus dataset. BooksCorpus dataset contains over 7,000 unique unpublished books from a variety of genres. Pre-training on such large contiguous text corpus enables the model to capture long-range dialogue context information. Furthermore, as existing EmpatheticDialogue dataset BIBREF4 is relatively small, fine-tuning only on such dataset will limit the chitchat topic of the model. Hence, we first integrate persona into CAiRE, and pre-train the model on PersonaChat BIBREF3 , following a previous transfer-learning strategy BIBREF1 . This pre-training procedure allows CAiRE to have a more consistent persona, thus improving the engagement and consistency of the model. We refer interested readers to the code repository recently released by HuggingFace. Finally, in order to optimize empathy in CAiRE, we fine-tune this pre-trained model using EmpatheticDialogue dataset to help CAiRE understand users' feeling.
Question: What pretrained LM is used?
Paper: Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?
Fueled by recent advances in deep-learning and language processing, NLP systems are increasingly being used for prediction and decision-making in many fields BIBREF0, including sensitive ones such as health, commerce and law BIBREF1. Unfortunately, these highly flexible and highly effective neural models are also opaque. There is therefore a critical need for explaining learning-based models' decisions.
Question: What approaches they propose?
Paper: Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?
There is considerable research effort in attempting to define and categorize the desiderata of a learned system's interpretation, most of which revolves around specific use-cases BIBREF17, BIBREF15.
Question: What faithfulness criteria does they propose?
Paper: Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?
A distinction is often made between two methods of achieving interpretability: (1) interpreting existing models via post-hoc techniques; and (2) designing inherently interpretable models. BIBREF29 argues in favor of inherently interpretable models, which by design claim to provide more faithful interpretations than post-hoc interpretation of black-box models.
Question: Which are three assumptions in current approaches for defining faithfulness?
Paper: Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?
While explanations have many different use-cases, such as model debugging, lawful guarantees or health-critical guarantees, one other possible use-case with particularly prominent evaluation literature is Intelligent User Interfaces (IUI), via Human-Computer Interaction (HCI), of automatic models assisting human decision-makers. In this case, the goal of the explanation is to increase the degree of trust between the user and the system, giving the user more nuance towards whether the system's decision is likely correct, or not. In the general case, the final evaluation metric is the performance of the user at their task BIBREF34. For example, BIBREF35 evaluate various explanations of a model in a setting of trivia question answering.
Question: Which are key points in guidelines for faithfulness evaluation?
Paper: Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference
Deep learning has achieved tremendous success for many NLP tasks. However, unlike traditional methods that provide optimized weights for human understandable features, the behavior of deep learning models is much harder to interpret. Due to the high dimensionality of word embeddings, and the complex, typically recurrent architectures used for textual data, it is often unclear how and why a deep learning model reaches its decisions.
Question: Did they use the state-of-the-art model to analyze the attention?
Paper: Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference
In NLI BIBREF4 , we are given two sentences, a premise and a hypothesis, the goal is to decide the logical relationship (Entailment, Neutral, or Contradiction) between them.
Question: What is the performance of their model?
Paper: Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference
In this work, we are primarily interested in the internal workings of the NLI model. In particular, we focus on the attention and the gating signals of LSTM readers, and how they contribute to the decisions of the model.
Question: How many layers are there in their model?
Paper: Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference
Attention has been widely used in many NLP tasks BIBREF12 , BIBREF13 , BIBREF14 and is probably one of the most critical parts that affects the inference decisions. Several pieces of prior work in NLI have attempted to visualize the attention layer to provide some understanding of their models BIBREF5 , BIBREF15 . Such visualizations generate a heatmap representing the similarity between the hidden states of the premise and the hypothesis (Eq. 19 of Appendix). Unfortunately the similarities are often the same regardless of the decision.
Question: Did they compare with gradient-based methods?
Paper: Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering
Enabling computers to understand given documents and answer questions about their content has recently attracted intensive interest, including but not limited to the efforts as in BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . Many specific problems such as machine comprehension and question answering often involve modeling such question-document pairs.
Question: What MC abbreviate for?
Answer: machine comprehension
Paper: Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering
Recent advance on reading comprehension and question answering has been closely associated with the availability of various datasets. BIBREF0 released the MCTest data consisting of 500 short, fictional open-domain stories and 2000 questions. The CNN/Daily Mail dataset BIBREF1 contains news articles for close style machine comprehension, in which only entities are removed and tested for comprehension. Children's Book Test (CBT) BIBREF2 leverages named entities, common nouns, verbs, and prepositions to test reading comprehension. The Stanford Question Answering Dataset (SQuAD) BIBREF3 is more recently released dataset, which consists of more than 100,000 questions for documents taken from Wikipedia across a wide range of topics. The question-answer pairs are annotated through crowdsourcing. Answers are spans of text marked in the original documents. In this paper, we use SQuAD to evaluate our models.
Question: how much of improvement the adaptation model can get?
Paper: Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering
Our baseline model is composed of the following typical components: word embedding, input encoder, alignment, aggregation, and prediction. Below we discuss these components in more details.
Question: what is the architecture of the baseline model?
Paper: Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering
The interplay of syntax and semantics of natural language questions is of interest for question representation. We attempt to incorporate syntactic information in questions representation with TreeLSTM BIBREF13 , BIBREF14 . In general a TreeLSTM could perform semantic composition over given syntactic structures.
Question: What is the exact performance on SQUAD?
Paper: SUM-QE: a BERT-based Summary Quality Estimation Model
Quality Estimation (QE) is a term used in machine translation (MT) to refer to methods that measure the quality of automatically translated text without relying on human references BIBREF0, BIBREF1. In this study, we address QE for summarization. Our proposed model, Sum-QE, successfully predicts linguistic qualities of summaries that traditional evaluation metrics fail to capture BIBREF2, BIBREF3, BIBREF4, BIBREF5. Sum-QE predictions can be used for system development, to inform users of the quality of automatically produced summaries and other types of generated text, and to select the best among summaries output by multiple systems.
Question: What are their correlation results?
Answer: High correlation results range from 0.472 to 0.936
Paper: SUM-QE: a BERT-based Summary Quality Estimation Model
Summarization evaluation metrics like Pyramid BIBREF5 and ROUGE BIBREF3, BIBREF2 are recall-oriented; they basically measure the content from a model (reference) summary that is preserved in peer (system generated) summaries. Pyramid requires substantial human effort, even in its more recent versions that involve the use of word embeddings BIBREF8 and a lightweight crowdsourcing scheme BIBREF9. ROUGE is the most commonly used evaluation metric BIBREF10, BIBREF11, BIBREF12. Inspired by BLEU BIBREF4, it relies on common $n$-grams or subsequences between peer and model summaries. Many ROUGE versions are available, but it remains hard to decide which one to use BIBREF13. Being recall-based, ROUGE correlates well with Pyramid but poorly with linguistic qualities of summaries. BIBREF14 proposed a regression model for measuring summary quality without references. The scores of their model correlate well with Pyramid and Responsiveness, but text quality is only addressed indirectly.
Question: What dataset do they use?
Paper: SUM-QE: a BERT-based Summary Quality Estimation Model
We use datasets from the NIST DUC-05, DUC-06 and DUC-07 shared tasks BIBREF7, BIBREF19, BIBREF20. Given a question and a cluster of newswire documents, the contestants were asked to generate a 250-word summary answering the question. DUC-05 contains 1,600 summaries (50 questions x 32 systems); in DUC-06, 1,750 summaries are included (50 questions x 35 systems); and DUC-07 has 1,440 summaries (45 questions x 32 systems).
Question: What simpler models do they look at?
Answer: BiGRUs with attention, ROUGE, Language model, and next sentence prediction
Paper: SUM-QE: a BERT-based Summary Quality Estimation Model
In Sum-QE, each peer summary is converted into a sequence of token embeddings, consumed by an encoder $\mathcal {E}$ to produce a (dense vector) summary representation $h$. Then, a regressor $\mathcal {R}$ predicts a quality score $S_{\mathcal {Q}}$ as an affine transformation of $h$:
Question: What linguistic quality aspects are addressed?
Answer: Grammaticality, non-redundancy, referential clarity, focus, structure & coherence
Paper: Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction
Knowledge graphs are usually collections of factual triples—(head entity, relation, tail entity), which represent human knowledge in a structured way. In the past few years, we have witnessed the great achievement of knowledge graphs in many areas, such as natural language processing BIBREF0, question answering BIBREF1, and recommendation systems BIBREF2.
Question: What benchmark datasets are used for the link prediction task?
Paper: Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction
In this section, we will describe the related work and the key differences between them and our work in two aspects—the model category and the way to model hierarchy structures in knowledge graphs.
Question: What are state-of-the art models for this task?
Paper: Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction
Roughly speaking, we can divide knowledge graph embedding models into three categories—translational distance models, bilinear models, and neural network based models. Table TABREF2 exhibits several popular models.
Question: How better does HAKE model peform than state-of-the-art methods?
Paper: Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction
Another related problem is how to model hierarchy structures in knowledge graphs. Some recent work considers the problem in different ways. BIBREF25 embed entities and categories jointly into a semantic space and designs models for the concept categorization and dataless hierarchical classification tasks. BIBREF13 use clustering algorithms to model the hierarchical relation structures. BIBREF12 proposed TKRL, which embeds the type information into knowledge graph embeddings. That is, TKRL requires additional hierarchical type information for entities.
Question: How are entities mapped onto polar coordinate system?
Paper: Machine Translation from Natural Language to Code using Long-Short Term Memory
Removing computer-human language barrier is an inevitable advancement researchers are thriving to achieve for decades. One of the stages of this advancement will be coding through natural human language instead of traditional programming language. On naturalness of computer programming D. Knuth said, “Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.”BIBREF0. Unfortunately, learning programming language is still necessary to instruct it. Researchers and developers are working to overcome this human-machine language barrier. Multiple branches exists to solve this challenge (i.e. inter-conversion of different programming language to have universally connected programming languages). Automatic code generation through natural language is not a new concept in computer science studies. However, it is difficult to create such tool due to these following three reasons–
Question: What additional techniques are incorporated?
Paper: Machine Translation from Natural Language to Code using Long-Short Term Memory
Code repositories (i.e. Git, SVN) flourished in the last decade producing big data of code allowing data scientists to perform machine learning on these data. In 2017, Allamanis M et al. published a survey in which they presented the state-of-the-art of the research areas where machine learning is changing the way programmers code during software engineering and development process BIBREF1. This paper discusses what are the restricting factors of developing such text-to-code conversion method and what problems need to be solved–
Question: What dataset do they use?
Answer: A parallel corpus where the source is an English expression of code and the target is Python code.
Paper: Machine Translation from Natural Language to Code using Long-Short Term Memory
According to the sources, there are more than a thousand actively maintained programming languages, which signifies the diversity of these language . These languages were created to achieve different purpose and use different syntaxes. Low-level languages such as assembly languages are easier to express in human language because of the low or no abstraction at all whereas high-level, or Object-Oriented Programing (OOP) languages are more diversified in syntax and expression, which is challenging to bring into a unified human language structure. Nonetheless, portability and transparency between different programming languages also remains a challenge and an open research area. George D. et al. tried to overcome this problem through XML mapping BIBREF2. They tried to convert codes from C++ to Java using XML mapping as an intermediate language. However, the authors encountered challenges to support different features of both languages.
Question: Do they compare to other models?
Paper: Machine Translation from Natural Language to Code using Long-Short Term Memory
One of the motivations behind this paper is - as long as it is about programming, there is a finite and small set of expression which is used in human vocabulary. For instance, programmers express a for-loop in a very few specific ways BIBREF3. Variable declaration and value assignment expressions are also limited in nature. Although all codes are executable, human representation through text may not due to the semantic brittleness of code. Since high-level languages have a wide range of syntax, programmers use different linguistic expressions to explain those. For instance, small changes like swapping function arguments can significantly change the meaning of the code. Hence the challenge remains in processing human language to understand it properly which brings us to the next problem-
Question: What is the architecture of the system?
Paper: Machine Translation from Natural Language to Code using Long-Short Term Memory
Although there is a finite set of expressions for each programming statements, it is a challenge to extract information from the statements of the code accurately. Semantic analysis of linguistic expression plays an important role in this information extraction. For instance, in case of a loop, what is the initial value? What is the step value? When will the loop terminate?
Question: How long are expressions in layman's language?
Paper: Machine Translation from Natural Language to Code using Long-Short Term Memory
The use of machine learning techniques such as SMT proved to be at most 75% successful in converting human text to executable code. BIBREF9. A programming language is just like a language with less vocabulary compared to a typical human language. For instance, the code vocabulary of the training dataset was 8814 (including variable, function, class names), whereas the English vocabulary to express the same code was 13659 in total. Here, programming language is considered just like another human language and widely used SMT techniques have been applied.
Question: What additional techniques could be incorporated to further improve accuracy?
Paper: Machine Translation from Natural Language to Code using Long-Short Term Memory
SMT techniques are widely used in Natural Language Processing (NLP). SMT plays a significant role in translation from one language to another, especially in lexical and grammatical rule extraction. In SMT, bilingual grammatical structures are automatically formed by statistical approaches instead of explicitly providing a grammatical model. This reduces months and years of work which requires significant collaboration between bi-lingual linguistics. Here, a neural network based machine translation model is used to translate regular text into programming code.
Question: What programming language is target language?
Paper: Machine Translation from Natural Language to Code using Long-Short Term Memory
SMT techniques require a parallel corpus in thr source and thr target language. A text-code parallel corpus similar to Fig. FIGREF12 is used in training. This parallel corpus has 18805 aligned data in it . In source data, the expression of each line code is written in the English language. In target data, the code is written in Python programming language.
Question: What dataset is used to measure accuracy?
Paper: A Survey and Taxonomy of Adversarial Neural Networks for Text-to-Image Synthesis
“ (GANs), and the variations that are now being proposed is the most interesting idea in the last 10 years in ML, in my opinion.” (2016)
Question: Is text-to-image synthesis trained is suppervized or unsuppervized manner?