GAPX: Generalized Autoregressive Paraphrase-Identification X
Anonymous Author(s)
AffiliationAddress email
Abstract
Paraphrase Identification is a fundamental task in Natural Language Processing.
While much progress has been made in the field, the performance of many state-ofthe-art models often suffer from distribution shift during inference time.
We verifythat a major source of this performance drop comes from biases introduced bynegative examples.
To overcome these biases, we propose in this paper to train twoseparate models, one that only utilizes the positive pairs and the other the negativepairs.
This enables us the option of deciding how much to utilize the negativemodel, for which we introduce a perplexity based out-of-distribution metric thatwe show can effectively and automatically determine how much weight it shouldbe given during inference.
We support our findings with strong empirical results
Introduction
Paraphrases are sentences or phrases that convey the same meaning using different wording, andis fundamental to the understanding of languages [7].
Paraphrase Identification is a well-studiedtask of identifying if a given pair of sentences has the same meaning [50, 46, 56, 57, 31], andhas many important downstream applications such as machine translation [61, 43, 39, 27], andquestion-answering [11, 35].
What causes stool color to change to yellow?
What can cause stool to come out as little balls?
QQP similar topic
Emma Townshend is represented by David Godwin at
DGA Associates .
David David Godwin is represented at DGA Associates by Emma Townshend .
PAWS same bag-of-word
But it changed the name and IDN and there was nothing to deal with.
WMT poor machine translation
That amber alert was getting annoying
Why do I get amber alerts tho
PIT similar topics
The company then changed its company ID number, and that was it.
Recently, researchers have observed that neural network architectures trained on different datasetscould achieve state-of-the-art performances for the task of paraphrase identification [51, 16, 49].
While these advances are encouraging for the research community, it has however been observedthat these models can be especially fragile in the face of distribution shift [44].
In other words, whena model trained on a source dataset D s is tested on another dataset, D t , collected and annotatedindependently, and with a distribution shift, the classification accuracy drops significantly [62].
This paper presents our findings and observations that negative pairs (i.e., non-paraphrase pairs) inthe training set, as opposed to the positive pairs, do not generalize well to out-of-distribution testpairs.
Intuitively, negative pairs only represent a limited perspective of how the meanings of sentencescan be different (and indeed it is practically infeasible to represent every possible perspective).
Weconjecture that negative pairs are so specific to the dataset that they adversely encourage the modelto learn biased representations.
We show this observation in Figure 1.
Quora Question Pair (QQP) 28extracts its negative pairs from similar topics.
Paraphrase Adversarials from Word Scrambling(PAWS)
Zhang et al.
[62] generate negative pairs primarily from word swapping.
World Machine
Translation Metrics Task 2017 (WMT) [8] considers negative examples as poor machine translations.
We therefore hypothesize that biases introduced by the different ways negative pairs are mined aremajor causes of the poor generalizability of paraphrase identification models.
Based on this observation, we would like to be able to control the reliance on negative pairs forout-of-distribution prediction.
In order to achieve this, we propose to explicitly train two separatemodels for the positive and negative pairs (we will refer to them as the positive and negative modelsrespectively), which will give us the option to choose when to use the negative model.
It is wellknown that just training on positive pairs alone can lead to a degenerate solution [21, 59, 41, 13], e.g.,a constant function network would still produce a perfect training loss.
To prevent this, we propose anovel generative framework where we use an autoregressive transformer [38, 48], specifically BART[30].
Given two sentences, we condition the prediction of the next token in the second sentence onthe first sentence and the previous tokens.
In a Bayesian sense, this would mean that the next tokenpredicted has a higher probability of being a positive/negative pair to the first sentence for the positiveand negative model respectively.
This learning strategy has no degenerate solutions even when weare training the positive and negative models separately.
We call our proposed approach GAP tostand for Generalized Autoregressive Paraphrase-Identification.
One potential pitfall of GAP is that itignores the “interplay” between positive and negative pairs that would otherwise be learned if theyare utilized in training together.
This is especially important when the test pairs are in-distribution.
To overcome, we utilize an extra discriminative model, trained with both positive and negative pairs,to capture the interplay.
We call this extension GAPX (pronounced as “Gaps”) to capture the e X tradiscriminative model used.
For all practical purposes, the weights we placed on the positive, negative and/or discriminativemodel in GAP and GAPX need to be determined automatically during inference.
For in-distributionpairs, we desire to use them all, while for out-of-distribution pairs, we hope to rely on the positivemodel much more heavily.
This obviously leads to a question of how to determine whether a givenpair is in or out of distribution [9, 14, 42, 17, 20].
During testing, our method ensembles the positivemodel, the negative model, and the discriminative model based on the degree of the similarity of thetest pair to the training pairs, and found that this works well for our purpose.
We measure this degreeof similarity with probability cumulative density function (cdf) in terms of perplexity [25], and showthat it is superior to other measures.
In summary, our contributions are as follow:
We report new research insights, supported by empirical results, that the negative pairs of a
dataset could potentially introduce biases that will prevent a paraphrase identification model
from generalizing to out-of-distribution pairs.
To overcome, we propose a novel autoregressive modeling approach to train both a positive
and a negative model
and ensemble them automatically during inference.
Further, we
observe that the interplay between positive and negative pairs are important for in-distribution
inference, for which we add a discriminative model.
We then introduce a new perplexity
based approach to determine whether a given pair is out-of-distribution to achieve auto
ensembling.
We support our proposal with strong empirical results
Compared to state-of-the-art transformers in out-of-distribution performance, our model achieves an average of 11.5% and
8.4% improvement in terms of macro F1 and accuracy respectively over 7 different outof-distribution scenarios.
Our method is especially robust to paraphrase adversarials like
PAWS, while keeping comparable performance for in-distribution prediction.
Related Works2.1 Distribution Shift and Debiasing Models in NLP
The issue of dataset bias and model debiasing has been widely studied in a lot of field in NLPsuch as Natural Language Inference [23, 3] and Question Answering [36, 2, 4].
Notable work by[22, 5, 10, 15, 47] utilize ensembling to reduce models’ reliance on dataset bias.
These models sharethe same paradigm where they break down a given sample x into signal feature x s and biased featurex b , in the hope of preventing their model from relying on x b , which has been shown to be the limitingfactor preventing the model from generalizing to out-of-distribution samples [5].
Here, a separatemodel is first either trained on x b or on datasets with known biases [10, 15, 22], or acquired frommodels known to have limited generalization capability.
Then they train their main model with aregularization term that encourages the main model to produce predictions that deviate from that ofthe “biased model”.
However, this type of approach has shown limited success [44] in debiasingparaphrase identification models.
In contrast, our method is based on our observation that negativepairs limit the generalization of paraphrase identification models.
2.2 Out-of-distribution Detection
Another line of work relevant to this paper is the task of detecting out-of-distribution samples[9, 14, 42, 17, 20, 55].
Researchers have proposed methods to detect anomaly samples by examiningthe softmax scores [32] or energy scores [33, 60] produced by discriminative models, while others takea more probabilistic approach to estimate the probability density [14, 25, 66, 1] or reconstruction error[37].
In this paper, we introduce a novel perplexity based out-of-distribution detection method that weshow empirically to work well for our purpose.
Specifically, during inference, an out-of-distributionscore is utilized to weigh the contributions from the positive and negative models: the higher theconfidence that the sample is out-of-distribution, the lesser the negative model’s contribution.
2.3 Text Generation Metrics
Finally, we would like to note the difference between our work and autoregressive methods that havebeen explored [58, 45] for evaluating text generation.
Our work differs as follow: 1) Paraphraseidentification seeks to assign an label of paraphrase or not while text generation metrics seeks toassign a score to measure the similarity of sentences; 2) Current text generation metrics either cannotbe trained to fit to a specifc distribution [61, 58, 63] or are limited to the i.i.d.
setting [43, 40] ofthe training distribution.
In contrast, our method not only significantly improves out-of-distributionperformances but is also competitive with state-of-the-art paraphrase identification methods forin-distribution predictions.
Methodology
We observe that negative pairs in paraphrase identification constitute the main source of bias.
Toovercome this, we propose the following training paradigms to learn a significantly less biasedparaphrase identification model.
We employ an autoregressive conditional sentence generators withtransformer architecture as the backbone of our model.
Specifically, we train a positive and negativemodel to estimate the distribution of positive and negative pairs in a dataset respectively.
Duringtesting, the two models are ensembled based on how likely the input pair is out of distribution.
Thissection provides details on our method.
[Figure]
3.1 Separation of Dependence on Positive and Negative Pairs
[Figure]
Let S be the space of all sentences, X =
( s 1 , s 2 ) be the random variable representing a sample pairfrom S , and Y the random variable representing the labels, with Y =
1 indicating that s 1 and s 2 areparaphrases
and otherwise when Y = 0 .
We seek to separate the dependence between the distributionof positive and negative pairs, motivated by the observation of the presence of bias in the negativepairs.
To begin, we model the distribution of sentences by splitting the sentence s 2 of length n intothe autoregressive product of individual words, where w ( i ) 2 denotes the i th word in s 2 .
By applying
Bayesian Inference Law, we have:
[Equation]
Subtracting the logarithm for Y = 1 and Y = 0 , we get:
[Equation]
s
[Equation]
+ ( n (cid:88)
[Equation]
+ ( n (cid:88) i = log
[Equation]
In this way, we break the probability inference into 3 terms resulting in Eqn.
2: (1) (log P ( Y =1) − log P
( Y = 0)) , which should just be a constant; (2) ( (cid:80) ni =1 log P ( w ( i )
| s 1 , Y = 1 , w (1: i − 1)) ,which depends only on the distribution of positive pairs; (3) ( (cid:80) ni =1 log P ( w ( i ) 2 | s 1 , Y = 0 , w (1: i − 1)) ,which depends only on the distribution of negative pairs.
We define the score of confidence as follows:
[Equation]
In the above, we are now left with two terms, the first representing the positive model and the secondthe negative model.
If we were to train the two terms together, the effects of the negative pairs inthe resulting model can never be removed during inference, which we have observed to be a majorsource of bias.
To avoid this, we propose to train the first term and the second term separately, andthen subsequently ensemble them based on the degree that a given pair is out of distribution.
We traineach model on top of the pretrained autoregressive
Given s 1 and s 2 , we feed s 1 into the encoder as the condition, shift s 2 to the right by one-token, andfeed shifted s 2 to the decoder.
While the decoder proceeds autoregressively, we record the next-wordprobability distribution.
We calculate the cross entropy between the next-word probability distributionand the target token in s 2 to update the model parameters.
3.2 Ensembling
To combine the positive and negative model, if we know a priori D t is in the same distribution as D s ,we can directly substitute the prediction of the positive and negative model into Eqn. 3.
We will referto this as the In-distribution Predictor (IDP).
If we have reason to believe that there is a significantdistribution shift between D s and D t (e.g., different sources of corpus and different dataset collectionprocedure), we observe empirically that we should only utilize the positive model and disregard thenegative model due to the bias it introduces.
We will refer to this as the Out-of-distribution Predictor(OODP).
3.2.1 Automatic Ensembling
However, in most cases, we have little or no knowledge of the testing distribution, in which case weneed to automatically decide how important the negative model is by detecting how much a test pairis in the same distribution as the training set.
We adopt a weighted interpolation between a constantand the negative model in addition to the positive model as follow:
[Equation]
[Equation]
where λ ( s 1 , s 2 ) is a weight parameter depending on s 1 and s 2 , and C is a constant that achievesa regularization effect.
See Appendix for ablations on how C can be set.
To automatically assignλ ( s 1 , s 2 ) for different sentence pairs, we measure an out-of-distribution score for ( s 1 , s 2 ) with regardto the training distribution.
Specifically, we hold back a set of validation data, comprising bothpositive and negative pairs, from D s , on which we train another autoregressive model, which we willrefer to as the distribution model.
The distribution model is trained by feeding an empty string intothe encoder and the concatenated s 1 and s 2 into the decoder, with the training goal of predicting thenext token.
We measure the perplexity of each sentence pair ( s 1 , s 2 ) using the distribution modelbased on the following formula, w i being the i th token of the concatenated ( s 1 , s 2 ) of length n :
[Equation]
We then fit a Weibull distribution to the perplexity of the validation data, so that it can better modelthe right-skewed property of the distribution.
We derive the exponential parameter a , the shapeparameter c , the location parameter loc , and the scale parameter scale .
During testing, λ ( s 1 , s 2 ) cannow be determined as:
[Equation]
For final prediction, we predict the sentence pair to be paraphrase if S ( s 1 , s 2 ) ≥ 0 and non-paraphraseotherwise.
This forms what we referred to earlier as GAP (Generalized Autoregressive ParaphraseIdentification).
3.2.2 Capturing Interplay Between Positive and Negative Pairs
In practice, training a positive and negative model separately disregards the interplay between thepositive and negative pairs, which could be important when the test pairs are in-distribution.
Tocapture such interplay, we utilize both positive and negative pairs to train a discriminative model forsequence classification.
Specifically, we first define a thresholding function based on the value of λ :
[Equation]
We then ensemble the discriminative model and GAP using the value of τ ( λ ) :
[Equation]
where P ( Y = 1 | s 1 , s 2 ) can be estimated by any discriminative model, and M is a sufficiently largeconstant.
Note that this definition is essentially the same as trusting the discriminative model whenwe do not have statistical evidence that the pair is out-of-distribution (p-test < 10%) while trustingthe GAP model otherwise.
For final prediction, we predict the sentence pair to be paraphrase if
S ∗ ( s 1 , s 2 ) ≥ 0 and non-paraphrase otherwise.
This defines GAPX (Generalized Autoregressive
Paraphrase Identfication X), for which we set M to be sufficiently large ( > 1000 ), so that whencomparing the model confidence for different pairs, the score of the discriminative model will beprioritized.
Experiments
Our experiments are designed to (1) verify that the task of paraphrase identification suffers frombiases in the datasets that is the main obstacle to generalization in this field of study, (2) test theaccuracy of our perplexity based out-of-distribution detection method, and (3) test that balancing theutilization of the negative model can help outperform the state-of-the-art in the face of distributionshift.
4.1 Datasets
We compare our method against the other state-of-the-art methods on different combinations of thefollowing datasets: Quora Question Pair (QQP) (10k subset), World Machine Translation Metrics
Task 2017 (WMT)[8], Paraphrase and Semantic Similarity in Twitter (PIT)[53, 54], and Paraphrase
Adversarials from Word Scrambling (PAWS)[62].
Details on the datasets are given in Appendix.
A.
With these datasets, we perform experiments where different models (Sec. 4.2) are trained on onedataset and evaluated on another in order to observe whether their performance hold in the face ofdistribution shift.
4.2 Benchmarks
We benchmark the paraphrase identification task with these methods: (1) BOW (Bag-Of-Words) [62],which represents two input sentences with bag of words; (2) BiLSTM [28], which encodes each ofthe two input sentences with a bidirectional LSTM network; (3) BERT+EP , which applies ExpertProduct [15] to the standard BERT approach to avoid known bias.
Similar to He et al.
[22], weuse a discriminative model trained on the task of making predictions relying on only one of thesentences as the bias model; (4) IDP is a model trained based on Eqn. 3; (5) OODP as described in Sec. 3.2; (6) GAP as described by Eqn. 3.2; (7) GAPX as given in Eqn.
8 - we use RoBERTa forthe discriminative model as we found it to be the strongest baseline for in-distribution prediction.
In addition, we also trained 3 state-of-the-art transformers for conventional sequence classificationwith [CLS] token embeddings.
They are BERT [16], BART [30], and RoBERTa [34].
We use allbase forms of transformers because of GPU RAM limitations, relying on the implementation of [52].
Detailed explanations can be found in Appendix.
B.
Following the previous literature in sentence matching [51, 16], we mainly use macro F1 score (F1)and accuracy score (ACC) to evaluate the models.
Results based on Area-under-curve of the Receiver
Operating Characteristic Curve (AUROC), a common metrics used to evaluate out-of-distributionmetrics, are also provided in Appendix. C.
4.3 Measuring Distribution Shift
To situate our experiments properly, we note that different datasets does not equate to differentdistributions.
It is thus important for us to be able to measure the distribution gap between datasets,and shed light on the models that perform the best when transferring between datasets with highdistribution gap.
Metrics that is per sample based such as our perplexity measure are not suitablefor measuring at dataset level.
To this end, we take a look at the Reverse Classification Accuracy*(RCA*) metric that has been proposed to predict the drop of model performance [18] and modelselection [19, 64].
Here, given D s and D t , we first train a model M 1 from the training set of D s .
Wethen take a certain amount of samples from the target distribution D t (1000 in our experiments), anduse M 1 to relabel them.
The relabeled pairs are then utilized to train a new model M 2 .
We measurethe performance (in terms of AUC or ACC) of M 2 on a held-out test set from D s .
As a control, wetrain another model M 3 following the same procedure except the relabeled data comes from D s .
Wedenote the performance drop from M 3 to M 2 as the RCA* score indicative of the distribution gap.
To calibrate RCA*, for each dataset, we randomly selected 1000 held-out pairs and measure itsshift from the dataset itself (which we expect to characterize an in-distribution RCA*).
After 100repetitions of measurements, we get a probability distribution of RCA* scores for each distribution initself.
Calibration results are reported in Appendix. 9.
All the distributions share a mean of around 0and a standard deviation around 2.
By a p-test of 10%, it is unlikely that datasets with a RCA* greaterthan 4.0 would belong to the same distribution.
Based on this observation, most of the transfersbetween different datasets are likely out-of-distribution, except for transfers from QQP to PIT andvice versa, which have a RCA* score of 2.8 and 3.4 respectively.
This is probably because QQP and
PIT are curated in a similar fashion, where they extract sentences with similar topics from socialplatforms, while all other datasets adopt different strategies to collect their data.
Finally, it is important to note that although RCA* provides a good estimate of the distributionshift at the dataset level, it’s utility does not easily extend to Eqn. 4 and 8 as opposed to per sampleout-of-distribution metrics.
RCA* assumes availability of the entire test set, while in real world, weare much more likely to get a single or small batches of test pairs at a time, all of which could evenbe from different distributions.
4.4 Implementation details
In practice, for testing, it helps to average the cross entropy by the length of the sentence, and averagethe cross entropy of generating s 2 from s 1 and generating s 1 from s 2 .
To optimize conditionalsentence generators, we use Adam optimizer with learning rate 2e-5.
We adopt cross entropy loss foreach word logit.
All experiments are run on Nvidia
Ti with 11 GB memory.
[Figure]
4.5 Main Results
Bias in Negative Pairs To understand whether negative pairs are major sources of bias, we plotthe Macro F1 and ACC score against RCA* in Fig. 3(a) and Fig. 4(a), comparing the performancesof OODP,
[Figure]
Model
QQP
PIT PIT
PAWS PAWS
PAWS
QQP average -> WMT -> WMT -> PAWS -> QQP -> PIT -> WMT -> PAWS (5.5) (11.4) (32.4) (33.2) (34.5) (35.6) (43.7) BOW 34.6/51.
34.4/52.0 BiLSTM 34.4/51.
40.8/51.2 BERT 67.4/67.
54.4/56.8 BERT+EP 66.5/66.
[Table]
[Table]
->PAWS ->PIT ->PIT ->QQP (0) (0) (0) (2.8) (3.4)
BOW 51.3/57.
40.7/52.9 BiLSTM 61.6/63.
49.7/53.9 BERT 82.5/82.
77.8/78.1 BERT+EP 81.6/81.
74.5/74.5 BART 82.6/82.
79.7/80.0 RoBERTa 84.4/84.
OODP 65.3/73.
60.4/70.4 IDP 79.0/79.
75.6/76.2 GAP 68.8/71.
68.8/73.9 GAPX 84.4/84.
[Table]
in ascending order of RCA* between pairs of datasets given in Table 1 and 2.
There are three pairsthat have RCA* score of 0 (Table 2), for which we average the performance in the plots.
Both the
F1 and ACC plots share a similar pattern.
In the in-distribution region, where the RCA* score isless than 4% (Fig. 9), BART and IDP achieves similarly high performances of 79.7% and 75.6% F1on average respectively.
The 4.1% gap in F1 is potentially due to the fact that IDP trains a negativemodel and positive model separately, neglecting the interaction between positive and negative pairs.
Comparatively, the performance of OODP is significantly inferior to the other two, with only 60.4%average F1.
This changes in the out-of-distribution region, where the RCA* score is now greaterthan 4%.
OODP turns out to be the leading model over BART and IDP.
With increasing degree ofdistribution shift, we observe that both BART and IDP are especially fragile and their performancedrop significantly.
When the RCA* is greater than 20, both models’ F1 drop to as low as around60%, which can be hardly useful in practice.
In contrast, OODP maintains an advantage in F1 of asmuch as 10-20% throughout the out-of-distribution region.
Since the only difference between OODPand IDP is that OODP transfers only the positive model, it confirms our hypothesis that the negativemodel does not generalize as well as the positive model.
Importance of the Interplay between Positive and Negative Pairs To understand the necessityof capturing the interplay for test pairs that are in distribution, we compare the performances of
GAP, GAPX, and BART in Fig. 3(a) and Fig. 4(a).
GAP only ensembles the positive and negativemodel trained separately, so does not contain any interplay information.
On the other hand, both
BART, trained with positive and negative pairs together, and GAPX (Sec. 3.2.2) capture interplayinformation.
As shown in the plot, the performance of GAP in the in-distribution region is not directlycomparable to BART with a gap of 10.9% in macro F1.
In contrast, GAPX’s in-distribution result hasa much smaller margin of 1.6% macro F1 compared to BART, yet, by automatically weighing thecontribution of the discriminative model, GAPX also closely matches the performance of GAP in theout-of-distribution region with only a 2.3% loss in macro F1.
Effectiveness of Perplexity-based Ensembling We also substitute PP ( s 1 , s 2 ) with other state-ofthe-art out-of-distribution metrics used for estimating the probability in Eqn. 6 for GAPX.
Specifically,we compare our perplexity metric with Maximum Softmax Probability (SOFTMAX) [24, 26, 6],
Energy Score (ENERGY)
[33], Mahalanobis Distance (MAHA) [29], and COSINE [65].
See
Appendix for more details.
The results are given in Fig. 4(b) and Fig. 3(b).
F1 and ACC are allsimilar in the in-distribution region.
However, in the out-of-distribution region, SOFTMAX, MAHA,and COSINE start to perform poorly.
ENERGY turned out to be the most robust but is still obviouslynot matching our perplexity metric, often by a large margin.
Generalization We implemented one of the most popular methods for domain generalization inparaphrase identification, Expert Product [15], as a potential strong baseline for handling datasetbias.
As described in Sec.
4.2, we train a BERT classifier with only the first sentence s 1 as thebiased model for BERT+EP.
We report the results in Fig. 4(c) and Fig. 5(c), together with BERT,
BART and RoBERTa.
In addition, we also provide performance of traditional methods like BoW [62]and ESIM [12] in Table 1 and Table 2.
BERT, BART, and RoBERTa all produce similar resultson all combinations of datasets.
We observe that their performances are consistently better thantraditional methods like BoW and ESIM, showing that pretraining and finetuning can indeed improvethe generalizability of classifiers for paraphrase identification.
However, their performances in out-ofdistribution setting are still far from their in-distribution performances, with accuracy below 65%in most of the cases (around 20% drop).
GAPX maintains an absolute margin of around 10% interms of ACC and an absolute margin of 7-20% in terms of F1 in the out-of-distribution region.
Thebest transformer-based models in the out-of-distribution region is BART, with an average of 56.2%in F1 and 59.8% in ACC, while GAPX maintains an average of 67.7% in F1 and 68.2% in ACC,with an absolute gain of 11.5% in F1 and 8.4% in ACC.
What is also encouraging is that GAPX’sperformance is close to that of OODP, which is promising that we do not need a priori information onthe domain gap between the source and target.
Lastly, BERT+EP fails to provide much gain, whichwe conjecture is due to the difficulty of “finding” the right bias model or features.
Conclusion
We have shown that negative samples introduce bias that prevent the generalization of paraphraseidentification models.
To overcome, we present a novel paradigm to train separate models for thedistribution of positive and negative samples independently, and utilize a perplexity based out-ofdistribution detection to ensemble them automatically.
Experiments show that our method achieves anaverage of 11.5% gain of F1 and 8.4% gain of ACC in various different out-of-distribution scenariosover other state-of-the-art methods.
5.1 Limitations
Our methodology is specifically designed for only “verification” problems, where the samples comein pairs.
Scenarios that involve other types of bias will require non-trivial turn-key formulations toexplicitly model the source of bias (much like the negative model).
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] .
See our listed contributions in the introduction.(b) Did you describe the limitations of your work?
[Yes] .
See the end of our conclusion,Sec. 5.1.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] .
This is a
traditional Natural Language Processing research topic.
We do not see any conceivable
negative social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] .
See Sec. 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] .
See Section.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] .(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] .
See Sec.4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
The baseline models in NLP that we are comparing to
do not report error bars in their paper.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] .
See Sec. 4.4.
If you are using existing assets (e
(a) If your work uses existing assets, did you cite the creators?
[Yes] .
See Appendix A
(b) Did you mention the license of the assets?
[N/A] .
Those are all public datasets.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A] .
Those are all public datasets, allowing researching purpose.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifi-
[Table]
able information or offensive content?
[N/A] .
Those data does not contain personal
[Table]
information
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]