wanyu/IteraTeR_full_sent huggingface dataset:
Original:  Extensive experiments on our generated probe sets  show that PTLMs perform no better than random guessing (even with fine-tuning) , are heavily impacted by statistical biases, and are not robust to perturbation attacks.
Revised:  Extensive experiments on our generated probe sets show that PTLMs perform no better than random guessing on the zero-shot setting , are heavily impacted by statistical biases, and are not robust to perturbation attacks.
Edit intent: clarity
Original:  Our framework and probe sets can help future work improve PTLMs ' inference abilities and robustness to linguistic variations--bringing us closer to more fluid communication .
Revised:  We also find that fine-tuning on similar statements offer limited gains, as PTLMs still fail to generalize to unseen inferences. Our new large-scale benchmark exposes a significant gap between PTLMs and human-level language understanding and offers a new challenge for PTLMs to demonstrate commonsense .
Edit intent: meaning-changed
Original:  yet we lack a realistic setup to study neural networks' capability in human-like language acquisition . In this paper, we propose a realistic setup by simulating children's language acquisition process. We formulate language acquisition as a masked language modeling task where the model visits a stream of data with continuously shifting distribution. Our training and evaluation encode two important challenges in human's language learning, namely the continual learning and the compositionality. We show the performance of existing continual learning algorithms is far from satisfactory. We also study the interactions between memory based continual learning algorithms and compositional generalization and conclude that overcoming overfitting and compositional overfitting may be crucial for a good performance in our problem setup.Our code and data can be found at URL
Revised:  yet we lack a realistic setup to study neural networks' capability in human-like language acquisition ability, we present VisCOLL, a visually grounded language learning task, which simulates the continual acquisition of compositional phrases from streaming visual scenes. In the task, models are trained on a paired image-caption stream which has shifting object distribution; while being constantly evaluated by a visually-grounded masked language prediction task on held-out test sets. VisCOLL compounds the challenges of continual learning (i.e., learning from continuously shifting data distribution) and compositional generalization (i.e., generalizing to novel compositions). To facilitate research on VisCOLL, we construct two datasets, COCO-shift and Flickr-shift, and benchmark them using different continual learning methods. Results reveal that SoTA continual learning approaches provide little to no improvements on VisCOLL, since storing examples of all possible compositions is infeasible. We conduct further ablations and analysis to guide future work.
Edit intent: meaning-changed
Original:  First, we formalize this form of disconnected reasoning and propose contrastive support sufficiency as a better test of multifact reasoning.
Revised:  First, we formalize such undesirable behavior as disconnected reasoning across subsets of supporting facts. This allows developing a model-agnostic probe for measuring how much any model can cheat via disconnected reasoning. Second, using a notion of contrastive support sufficiency as a better test of multifact reasoning.
Edit intent: meaning-changed
Original:  First, we formalize this form of disconnected reasoning and propose contrastive support sufficiency as a better test of multifact reasoning. To this end , we introduce an automated sufficiency-based dataset transformation that considers all possible partitions of supporting facts, capturing disconnected reasoning.
Revised:  First, we formalize this form of disconnected reasoning and propose contrastive support sufficiency , we introduce an automated sufficiency-based dataset transformation that considers all possible partitions of supporting facts, capturing disconnected reasoning.
Edit intent: clarity
Original:  Second, we develop a probe to measure how much can a model cheat (via non-multifact reasoning) on existing tests and our sufficiency test. Third, we conduct experiments using a transformer based model (XLNet), demonstrating that the sufficiency transform not only reduces the amount of non-multifact reasoning in this model by 6.5\% but is also harder to cheat -- a non-multifact model sees a 20.8\% (absolute) reduction in score compared to previous metrics .
Revised:  Third, our experiments demonstrate that there hasn't been much progress in multifact reasoning. For a recent large-scale model (XLNet), demonstrating that the sufficiency transform not only reduces the amount of non-multifact reasoning in this model by 6.5\% but is also harder to cheat -- a non-multifact model sees a 20.8\% (absolute) reduction in score compared to previous metrics .
Edit intent: clarity
Original:  Third, we conduct experiments using a transformer based model (XLNet), demonstrating that the sufficiency transform not only reduces the amount of non-multifact reasoning in this model by 6.5\% but is also harder to cheat -- a non-multifact model sees a 20.8\% (absolute) reduction in score compared to previous metrics .
Revised:  Third, we conduct experiments using a transformer based model (XLNet), we show that only 18\% of its answer score is obtained through multifact reasoning, roughly the same as that of a simpler RNN baseline. Our transformation shows a substantial reduction in disconnected reasoning (nearly 19 points in answer F1). It is complementary to adversarial approaches, yielding further reductions in conjunction .
Edit intent: meaning-changed
Original:  For a recent large-scale model (XLNet), we show that only 18 \% of its answer score is obtained through multifact reasoning, roughly the same as that of a simpler RNN baseline.
Revised:  For a recent large-scale model (XLNet), we show that only 18 points out of its answer score is obtained through multifact reasoning, roughly the same as that of a simpler RNN baseline.
Edit intent: clarity
Original:  For a recent large-scale model (XLNet), we show that only 18 \% of its answer score is obtained through multifact reasoning, roughly the same as that of a simpler RNN baseline.
Revised:  For a recent large-scale model (XLNet), we show that only 18 \% of its answer F1 score of 72 on HotpotQA are obtained through multifact reasoning, roughly the same as that of a simpler RNN baseline.
Edit intent: meaning-changed
Original:   SimpleTOD also improves all three metrics used to evaluate action  and response generation in the most complete setting for task-oriented dialog systems: inform rate by 8.1 points, success rate by 9.7 points, and BLEU by 23.5 points.
Revised:  More impressively, SimpleTOD also improves all three metrics used to evaluate action and response generation in the most complete setting for task-oriented dialog systems: inform rate by 8.1 points, success rate by 9.7 points, and BLEU by 23.5 points.
Edit intent: coherence
Original:   SimpleTOD also improves all three metrics used to evaluate action  and response generation in the most complete setting for task-oriented dialog systems: inform rate by 8.1 points, success rate by 9.7 points, and BLEU by 23.5 points.
Revised:  SimpleTOD also improves the main metrics used to evaluate action and response generation in the most complete setting for task-oriented dialog systems: inform rate by 8.1 points, success rate by 9.7 points, and BLEU by 23.5 points.
Edit intent: clarity
Original:   SimpleTOD also improves all three metrics used to evaluate action  and response generation in the most complete setting for task-oriented dialog systems: inform rate by 8.1 points, success rate by 9.7 points, and BLEU by 23.5 points.
Revised:  SimpleTOD also improves all three metrics used to evaluate action decisions and response generation in the most complete setting for task-oriented dialog systems: inform rate by 8.1 points, success rate by 9.7 points, and BLEU by 23.5 points.
Edit intent: clarity
Original:   SimpleTOD also improves all three metrics used to evaluate action  and response generation in the most complete setting for task-oriented dialog systems: inform rate by 8.1 points, success rate by 9.7 points, and BLEU by 23.5 points.
Revised:  SimpleTOD also improves all three metrics used to evaluate action and response generation in an end-to-end setting for task-oriented dialog systems: inform rate by 8.1 points, success rate by 9.7 points, and BLEU by 23.5 points.
Edit intent: clarity
Original:   SimpleTOD also improves all three metrics used to evaluate action  and response generation in the most complete setting for task-oriented dialog systems: inform rate by 8.1 points, success rate by 9.7 points, and BLEU by 23.5 points.
Revised:  SimpleTOD also improves all three metrics used to evaluate action and response generation in the most complete setting for task-oriented dialog systems: inform rate by 8.1 points, success rate by 9.7 points, and combined score by 7.2 points.
Edit intent: meaning-changed
Original: Advances in extractive machine reading comprehension (MRC) rely heavily on the collection of large scale human-annotated training data ( in the form of "question-paragraph-answer span") . A single question-answer example provides limited supervision, while an explanation in natural language describing human's deduction process may generalize to many other questions that share similar solution patterns.
Revised: Advances in machine reading comprehension (MRC) rely heavily on the collection of large scale human-annotated training data ( in the form of "question-paragraph-answer span") . A single question-answer example provides limited supervision, while an explanation in natural language describing human's deduction process may generalize to many other questions that share similar solution patterns.
Edit intent: clarity
Original: Advances in extractive machine reading comprehension (MRC) rely heavily on the collection of large scale human-annotated training data ( in the form of "question-paragraph-answer span") . A single question-answer example provides limited supervision, while an explanation in natural language describing human's deduction process may generalize to many other questions that share similar solution patterns.
Revised: Advances in extractive machine reading comprehension (MRC) rely heavily on the collection of large scale human-annotated examples in the form of "question-paragraph-answer span") . A single question-answer example provides limited supervision, while an explanation in natural language describing human's deduction process may generalize to many other questions that share similar solution patterns.
Edit intent: clarity
Original: Advances in extractive machine reading comprehension (MRC) rely heavily on the collection of large scale human-annotated training data ( in the form of "question-paragraph-answer span") . A single question-answer example provides limited supervision, while an explanation in natural language describing human's deduction process may generalize to many other questions that share similar solution patterns.
Revised: Advances in extractive machine reading comprehension (MRC) rely heavily on the collection of large scale human-annotated training data ( in the form of (question, paragraph, answer) triples. In contrast, humans are typically able to generalize with only a few examples, relying on deeper underlying world knowledge, linguistic sophistication, and/or simply superior deductive powers . A single question-answer example provides limited supervision, while an explanation in natural language describing human's deduction process may generalize to many other questions that share similar solution patterns.
Edit intent: meaning-changed
Original: Advances in extractive machine reading comprehension (MRC) rely heavily on the collection of large scale human-annotated training data ( in the form of "question-paragraph-answer span") . A single question-answer example provides limited supervision, while an explanation in natural language describing human's deduction process may generalize to many other questions that share similar solution patterns.
Revised: Advances in extractive machine reading comprehension (MRC) rely heavily on the collection of large scale human-annotated training data ( in the form of "question-paragraph-answer span") . 
Edit intent: coherence
Original:  In this paper, we focus on "teaching" machines on reading comprehensionwith ( a small number of ) natural language explanations . We propose a data augmentation framework that exploits the compositional nature of explanations to rapidly create pseudo-labeled data for training downstream MRC models.
Revised:  In this paper, we focus on "teaching" machines on reading comprehensionwith ( a small number of semi-structured explanations that explicitly inform machines why answer spans are correct. We extract structured variables and rules from explanations and compose neural module teachers that annotate instances for training downstream MRC models.
Edit intent: meaning-changed
Original:  Structured variables and rules are extracted from each explanation and formulated into neural module teacher, which employs softened neural modules and combinatorial search to handle linguistic variations and overcome sparse coverage . The proposed work is particularly effective when limited annotation effort is available, and achieved a practicable F1 score of 59.80\% with supervision from 52 explanationson the SQuAD dataset .
Revised:  Structured variables and rules are extracted from each explanation and formulated into neural module teacher, which employs softened neural modules and combinatorial search to handle linguistic variations and overcome sparse coverage ; the modules are jointly optimized with the MRC model to improve final performance. On the SQuAD dataset, our proposed method achieves 70.14\% F1 score of 59.80\% with supervision from 52 explanationson the SQuAD dataset .
Edit intent: meaning-changed
Original:  The proposed work is particularly effective when limited annotation effort is available, and achieved a practicable F1 score of 59.80\% with supervision from 52 explanationson the SQuAD dataset .
Revised:  The proposed work is particularly effective when limited annotation effort is available, and achieved a practicable F1 score of 59.80\% with supervision from 26 explanations, comparable to plain supervised learning using 1,100 labeled instances, yielding a 12x speed up .
Edit intent: meaning-changed
Original:  We discuss in more detail Lehre's (PPSN 2010) negative drift in populations method, one of the most general tools to prove lower bounds on the runtime of non-elitist evolutionary algorithms .
Revised:  We discuss in more detail Lehre's (PPSN 2010) negative drift in populations method, one of the most general tools to prove lower bounds on the runtime of non-elitist mutation-based evolutionary algorithms for discrete search spaces .
Edit intent: meaning-changed
Original:  As one particular result, we apply this method and a novel domination argument to show an exponential lower bound for the runtime of the mutation-only simple GA on \onemax for arbitrary population size.
Revised:  For the special case of algorithms using standard bit mutation with a random mutation rate (called uniform mixing in the language of hyper-heuristics), we prove the result stated by Dang and Lehre (PPSN 2016) and extend it to mutation rates other than \Theta(1/n), which includes the heavy-tailed mutation operator proposed by Doerr, Le, Makhmara, and Nguyen (GECCO 2017). We finally apply our method and a novel domination argument to show an exponential lower bound for the runtime of the mutation-only simple GA on \onemax for arbitrary population size.
Edit intent: meaning-changed
Original:  The directed transmission graph G^{\rightarrow}_D for D is the directed graph with vertex set D in which there is an edge from a disk D_1 \in D to a disk D_2 \in D if and only if D_1 contains the center of D_2. Given D and two non-intersecting disks s, t \in D, we show that a minimum s-t vertex cut in G_D or in G^{\rightarrow}_\mathcal{D} can be found in O(n^{3/2} %DIFDELCMD < \operatorname{polylog} %%%  n) expected time.
Revised:  The directed transmission graph G^{\rightarrow}_D for D is the directed graph with vertex set D in which there is an edge from a disk D_1 \in D to a disk D_2 \in D if and only if D_1 contains the center of D_2. Given D and two non-intersecting disks s, t \in D, we show that a minimum s-t vertex cut in G_D or in G^{\rightarrow}_\mathcal{D} can be found in O(n^{3/2} %DIFDELCMD < \operatorname{polylog} %%% n) expected time.
Edit intent: fluency
Original:  The directed transmission graph G^{\rightarrow}_D for D is the directed graph with vertex set D in which there is an edge from a disk D_1 \in D to a disk D_2 \in D if and only if D_1 contains the center of D_2. Given D and two non-intersecting disks s, t \in D, we show that a minimum s-t vertex cut in G_D or in G^{\rightarrow}_\mathcal{D} can be found in O(n^{3/2} %DIFDELCMD < \operatorname{polylog} %%%  n) expected time.
Revised:  The directed transmission graph G^{\rightarrow}_D for D is the directed graph with vertex set D in which there is an edge from a disk D_1 \in D to a disk D_2 \in D if and only if D_1 contains the center of D_2. Given D and two non-intersecting disks s, t \in D, we show that a minimum s-t vertex cut in G_D or in G^{\rightarrow}_\mathcal{D} can be found in O(n^{3/2} %DIFDELCMD < \operatorname{polylog} %%% \text{polylog n) expected time.
Edit intent: meaning-changed
Original:  Using our improved algorithm for minimum cuts in disk graphs, we can solve the barrier resilience problem in O(n^{3/2} %DIFDELCMD < \operatorname{polylog} %%%  n) expected time.
Revised:  Using our improved algorithm for minimum cuts in disk graphs, we can solve the barrier resilience problem in O(n^{3/2} %DIFDELCMD < \operatorname{polylog} %%% \text{polylog n) expected time.
Edit intent: meaning-changed
Original:  This paper presents a novel numerical procedure to efficiently compute the moments of a given quadratic form. Based on that, a gamma distribution with matched skewness-kurtosis ratio is proposed to approximate its distribution.
Revised:  This paper presents a novel numerical procedure to efficiently compute the moments of the quadratic forms. Second, we establish a general moment-matching framework for distribution approximation, which covers existing approximation methods for the distributions of the quadratic forms of Gaussian variables. Under this framework, a novel moment-ratio method (MR) is proposed to approximate its distribution.
Edit intent: meaning-changed
Original:  Based on that, a gamma distribution with matched skewness-kurtosis ratio is proposed to approximate its distribution. Comparing with existing methods, the new method is significantly more accurate in getting the right-tail probability. The new method may facilitate the hypothesis testing in the analysis of big data , where efficient and accurate calculation of  small p-values is desired.
Revised:  Based on that, a gamma distribution with matched skewness-kurtosis ratio is proposed to match the ratio of skewness and kurtosis based on the gamma distribution. Our extensive simulations show that 1) MR is almost as accurate as the exact distribution calculation and is much more efficient; 2) comparing with existing approximation methods, MR significantly improves the accuracy of approximating far right tail probabilities. The proposed method has wide applications. For example, it is a better choice than existing methods for facilitating hypothesis testing in the analysis of big data , where efficient and accurate calculation of small p-values is desired.
Edit intent: meaning-changed
Original:   In this work, we examine female first author percentages and the citations to their papers in Natural Language Processing . We find that only about 29\% of first authors are female and only about 25\% of last authors are female.
Revised:  In this work, we examine female first author percentages and the citations to their papers in Natural Language Processing (1965 to 2019). We determine aggregate-level statistics using an existing manually curated author--gender list as well as first names strongly associated with a gender. We find that only about 29\% of first authors are female and only about 25\% of last authors are female.
Edit intent: meaning-changed
Original: Question Generation (QG) is an important task in Natural Language Processing (NLP) that involves generating questions automatically when given a context paragraph. While many techniques exist for the task of QG , they employ complex model architectures, extensive features, and additional mechanisms to boost model performance. In this work, we show that transformer-based finetuning techniques can be used to create robust question generation systems using only a single pretrained language model , without the use of additional mechanisms, answer metadata, and extensive features. Our best modeloutperforms previous more complex RNN-based Seq2Seq models, with an 8.62 and a 14.27 increase in METEOR and ROUGE_L scores, respectively. We show that it also performs on par with Seq2Seq models that employ answer-awareness and other special mechanisms, despite being only a single-model system. We analyze how various factors affect the model's performance, such as input data formatting, the length of the context paragraphs, and the use of answer-awareness. In addition, we also look into the modes of failure that the model experiences and identify the reasons why it fails .
Revised: Question Generation (QG) is a natural language generation task where a model is trained to ask questions corresponding to some input text. Most recent approaches frame QG as a sequence-to-sequence problem and rely on additional features and mechanisms to increase performance; however, these often increase model complexity, and can rely on auxiliary data unavailable in practical use. A single Transformer-based unidirectional language model leveraging transfer learning can be used to produce high quality questions while disposing of additional task-specific complexity. Our QG model, finetuned from GPT-2 Small, outperforms several paragraph-level QG baselines on the SQuAD dataset by 0.95 METEOR points. Human evaluators rated questions as easy to answer, relevant to their context paragraph, and corresponding well to natural human speech. Also introduced is a new set of baseline scores on the RACE dataset, which has not previously been used for QG tasks. Further experimentation with varying model capacities and datasets with non-identification type questions is recommended in order to further verify the robustness of pretrained Transformer-based LMs as question generators .
Edit intent: meaning-changed
Original: We consider variations in the rate of spread of COVID-19 , firstly comparing between European countries and secondly comparing between US states. We show that the population density has a small but significant effect on the rate of spread of the virus. However we show that measures of ` lived population density', which capture density as perceived by a randomly chosen person, do a better job of explaining variations in the rate of spread , achieving R^2 = 0.45 in Europe.
Revised: We consider variations in the rate of spread of COVID-19 pandemic has progressed with varying degrees of intensity in individual countries, suggesting it is important to analyse factors that vary between them. We study measures of ` lived population density', which capture density as perceived by a randomly chosen person, do a better job of explaining variations in the rate of spread , achieving R^2 = 0.45 in Europe.
Edit intent: meaning-changed