On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning
Anonymous Author(s)
AffiliationAddress
Abstract
Rehearsal approaches enjoy immense popularity with Continual Learning (CL)practitioners.
These methods collect samples from previously encountered datadistributions in a small memory buffer; subsequently, they repeatedly optimizeon the latter to prevent catastrophic forgetting.
This work draws attention to ahidden pitfall of this widespread practice: repeated optimization on a small pool ofdata inevitably leads to tight and unstable decision boundaries, which are a majorhindrance to generalization.
To address this issue, we propose Lipschitz-DrivEn
Rehearsal (LiDER), a surrogate objective that induces smoothness in the backbonenetwork by constraining its layer-wise Lipschitz constants w.r t. replay examples.
By means of extensive experiments, we show that applying LiDER delivers a stableperformance gain to several state-of-the-art rehearsal CL methods across multipledatasets, both in the presence and absence of pre-training.
Through additionalablative experiments, we highlight peculiar aspects of buffer overfitting in CL andbetter characterize the effect produced by LiDER.
Introduction
The last few years have seen a renewed interest in aiding Deep Neural Networks (DNNs) to acquirenew knowledge and, at the same time, retain high performance on previously encountered data.
Inthis regard, the mitigation of catastrophic forgetting [44] has driven the recent research towards novelincremental methods [35, 56], often framed under the field of Continual Learning (CL).
Among other valid strategies, rehearsal approaches caught the attention of a large body of literature [52, 18, 12] thanks to their advantages.
Simply, they maintain a small fixed-size buffer containinga fraction of examples from previous tasks; afterward, these examples are mixed together with theones of the current task, hence provided continuously as training data.
In this respect, differentapproaches establish different regularization strategies on top of the retained examples [53, 42], aswell as which kind of information to store ( e.g. model responses [12, 10], explanations [22], etc.).
In spite of their widespread application, these approaches fall into a common pitfall: as the memorybuffer holds only a small fraction of past examples, there is a high risk of overfitting on thatmemory [62], thus harming generalization.
Several approaches mitigate such an issue through dataaugmentation techniques, either by generating different versions of the same buffer datapoint [6, 12]or by combining different examples into a single one [11, 9].
Other works [3, 4, 13, 70], instead,select carefully the valuable samples that should be inserted into the buffer: they argue that randomselection may pick non-informative and noisy instances, affecting the model generalization.
This work tackles the issue described above from a different perspective, viewing catastrophicforgetting in the light of the progressive deterioration of decision boundaries between classes.
Indeed,while for the examples of the current task we expect the decision boundaries to be already smooth and
Random x Perturbation Magnitude −
A d v .
x
P e r t u r ba t i on
M agn i t ude −−   τττ
Rehearsed Examples ττ− 40 −−   τ−  τ−  τ
Non-Rehearsed Examples −  τ−  τ- - - - -
D e c
i s i on M agn i t ude
S ( x ) Such an intuition motivates our research of novel mechanisms for guaranteeing the robustness ofthe decision boundaries.
To this aim, we resort to enforcing the Lipschitz continuity of the modelw.r.t.
its input: indeed, a long-standing research trend [69, 59, 38, 39, 67, 25] has pointed out thatsuch a property favors generalization capabilities and robustness to adversarial attacks.
In particular,constraining the Lipschitz constant of a model – intuitively, a bound on how much the model’sresponse can change in proportion to a change in its input [5] – has proven to strengthen the decisionsurface around a point [19, 38, 39, 67, 76], preventing attacks of a given magnitude from changingthe output of the classifier.1k
2.5k 4k .
A d v .
A cc .
|M|
L .
While these works assessed Lipschitz regularization in the classical scenario ( i.e. , single joint i.i.d.
task), we advocate that it iseven more beneficial in continual learning, particularly for thoseapproaches based on replay memories.
As shown in the insetfigure, without explicit regularization, the Lipschitz constant amodel increases for smaller memory buffers: in other terms, itscorresponding function space becomes increasingly sensitivew.r.t.
local input perturbations (as also highlighted by the lower accuracy attained in presence ofadversarial attacks). In light of the above considerations, we ascribe such a tendency to the higheruncertainty, which derives from subjecting the model to a low-data training regime.
To the best of our knowledge, our work is the first attempt to assess the effectiveness of Lipschitzconstrained DNNs in continual learning, with several experiments and ablative studies supportingour intuition.
In particular, we have equipped several widely-known and state-of-the-art rehearsalapproaches with our Lipschitz-guided optimization objective named Lipschitz-DrivEn Rehearsal(LiDER) , showing that it systematically leads to better results in several benchmarks.
Related works
Continual Learning CL examines the capability of a deep model to learn from a sequence ofnon-i.i.d.
classification tasks [20, 49] while preventing the onset of catastrophic forgetting [44].
Toachieve this goal, models are trained according to specifically designed strategies, meant to influencetheir evolution and maximize the retention of previously acquired knowledge.
Among them, regularization methods work by introducing additional constraints in the form of lossterms; they are designed to limit the amount of total change either in parameter space [35, 74, 16, 2]or functional space [40, 8].
Differently, structural methods purposefully organize the allocationof model capacity to prevent interference and facilitate parameter sharing [1, 43, 54, 31].
Lastly,rehearsal methods store and reuse a subset of previously seen data-points to prevent overfittingon current data and avert forgetting [13, 42, 4, 3].
While rehearsal strategies are by far the mostfrequently adopted thanks to their effectiveness and flexibility [23, 4], it is not uncommon to adoptsolutions combining multiple approaches [28, 1, 12, 50, 15].
In this paper, we similarly propose astrategy that leverages an existing replay memory buffer to compute an additional regularization term,aimed at conditioning the learning dynamics and avoiding overfitting.
CL evaluations are often carried out in the so-called Task-Incremental setting (TIL) [35, 74, 42, 20, 16]– that is, the model is provided a task identifier at test time to restrict its predictions and avoidinterference across logits of distinct tasks.
However, recent works put an increasing focus on theharder Class-Incremental setting (CIL) [4, 28, 12, 68], which entails the production of a unifiedprediction encompassing all seen classes.
W.r.t.
the latter, TIL has been criticized as a less challengingand realistic benchmark [23, 61, 4]; we therefore conduct our main experiments on state-of-the-art
CL models in the CIL setting 1 .
Lipschitz-based Regularization Naturally trained DNNs typically suffer from their overparametrization [75, 48], leading to the tendency to overfitt the training data by producing jagged decision boundaries that closely fit the seen examples.
On the contrary, a model’s reliability depends on its capabilityfor generalization, which is linked to the appearance of smooth decision boundaries [69, 7, 71, 25].
Starting from the first studies focusing on this simple dichotomy, the Lipschitz constant L of a DNNhas been established as a commonplace measure of both smoothness and generalization [69, 59, 33]and still constitutes a key ingredient for current evaluations of model capacity [7, 26].
Most notably, Szegedy et al.
[59] verify that constraining L reduces the model’s vulnerability toadversarial perturbations.
Many current approaches to Adversarial Learning similarly operate eitherby minimizing L at the global or local level [39, 60, 38] or by devising models characterized by asmall L by design [19, 29].
In other areas, the smoothing effect of L -based regularization has beenfavorably applied to both GAN training [46] and neural fields [41].
Method
A CL problem usually involves learning a function f from a stream of data, which we formalize as asuccession of separate datasets T
= { τ 0 , τ 1 , . . .
, τ | T | } , where τ t = { ( x i , y i ) } N t i =1 and τ i ∩
τ j = ∅ ; the 106 label set Y t for each τ t are non-overlapping.
In this setting, the ideal objective consists in minimizingthe overall loss over all tasks experienced, formally:
[Equation]
(1) where L is an appropriate loss for solving the task at hand.
In a continual scenario, only data fromthe current task τ t is freely available; therefore, CL methods need to maintain knowledge from thepast t − 1 tasks in order to solve the overall problem.
For the sake of simplicity, we consider a feed forward neural network f ( · ) =
( H K ◦ σ K ◦ H K − 1 ◦σ K − 1 ◦ . . .
H 1 )( · ) , i.e. , a sequence of σ -activated linear functions H k
( h ) =
W Tk h (biases are omit- ted).
A final projection head g ( · ) = softmax( · ) is applied to produce per-class output probabilities.
max-pool) can also be seen in terms of matrix multiplications, thus making our approach applicableto more complex networks.
Lipschitz continuity .
A function f is said to be Lipschitz continuous if there exists a value L ∈ R +such that the following inequality holds:
[Equation]
If such a value exists, the smallest L that satisfies the condition is usually referred to as the Lipschitznorm ||
f || L .
Therefore, for a single point x ∈ R n , we obtain:
[Equation]
Unfortunately, computing the Lipschitz constant of even the most simple multi-layer perceptron is a
NP-hard problem [64].
Therefore, several works relied on its estimation by computing reliable upperbounds.
As stated in [71, 55], an effective way to bound the Lipschitz constant of f ( · ) is to computethe constants of each linear projections H k
and then aggregate them.
In more details:
[Equation]
where σ max ( W k ) is the largest singular value of the weight matrix W k (also know as its spectralnorm || W k || SN ).
To account for non-linear composite functions ( e.g. , the residual building blocks of 127 most convolutional architecture), we leverage the following inequality:
[Equation]
, where g ( · ) and z ( · ) are two Lipschitz continuous functions characterized by the constants
||
g ||
L and|| z || L .
In the case of ReLU-activated networks (but the following result can be extended to othercommon non-linear functions), the forward pass through σ l l = 1 , 2 , . . .
, L can be re-arranged asa matrix multiplication by a diagonal matrix D l ∈ R d l × d l +1 whose diagonal elements equal eitherzero or one.
Therefore, their corresponding Lipschitz constant || σ l || L ≤ 1 .
On top of that, we can 133 compute an upper bound on the Lipschitz constant of the entire network:
[Equation]
Computing the spectral norm of weights matrices .
The computation of || W k || SN can be done [46, 135 25 ] naively through the Singular Value Decomposition (SVD), yielding, among the others, the largestsingular value.
Such approach has been applied in recent works [46, 25]; however, for complexstructures ( e.g. , convolutions or entire residual blocks) the SVD decomposition is inaccessible.
Hence,we rely on the approximation introduced in [55] and compute the largest eigenvalue λ k 1 of the
Transmitting Matrix TM k (which represents a good proxy of || W k || SN 2 ):
[Equation]
where F k ∈ R B × d k is the L2-normalized feature map produced by the l -th layer from a batch of
B samples.
Finally, our approach exploits the power iteration method [47] to compute the largesteigenvalue of TM k , which is backpropagation-friendly.
3.1 Lipschitz-Driven Rehearsal
In a continual setting, a model is asked i) to be adaptable to incoming samples from the stream(plasticity), and ii) to be accurate on past tasks (stability).
We seek to ensure a balance between theseclashing objectives through the two following loss terms.
Controlling Lipschitz-continuity .
To mitigate overfitting on buffer datapoints, we firstly impose thateach layer behaves as a c -Lipschitz continuous function, for a given real positive target constant c k :
[Equation]
During the computation of each λ k 1 , we discard the activation maps incoming from the examples ofthe current task.
Indeed, as we have access to the entire training set (and not a subset as holds forold tasks), additional regularization is not needed: the decision boundaries tied to the current taskare less prone to the risk of over-adapting to certain points.
Regarding the target constants c k , wecould fix them as hyperparameters of our learning objective (as done in [41]) and exploit them as asort of budget assigned to each layer; however, we empirically observed that it is beneficial, instead,learning these targets by means of gradient descent (see Sec. 5.4), especially in a CL scenario wherethere is no access to the full data distribution.
Indeed, these can be interpreted as additional learnableparameters, which represent the appropriate level of strictness each layer should be subjected to.
Therefore, to avoid trivial solutions, we add another regularization term, which asks these constant tobe as much as possible close to zero:
[Equation]
Intuitively, when λ k 1 → 0 , the outputs of the corresponding k -th layer has low sensitivity to changesin its input.
In our intentions, this could relieve continuous rehearsal from eroding the decisionsurface in a way that fits well only certain examples ( i.e. , those retained in the memory buffer).
Overall objective .
The overall objective of LiDER combines the two introduced loss terms; formally:
[Equation]
This objective can be plugged in almost any rehearsal approach.
For such a reason, we keep it generaland avoid reporting the common loss terms asking for accurate predictions, as their form dependon the specific choices made by each approach.
Finally, we further remark that the introduced lossterms require minimal additional computation.
Moreover, they do not need additional samples to beretained, besides those that are already present in the memory buffer.
Experiments
To assess the effectiveness of our proposal, we introduce a suite of experiments encompassing theprevalent applications of rehearsal in modern literature.
In particular, we show that our method canbe easily applied to state-of-the-art replay methods and enhance their performance in a wide varietyof challenging settings and backbone architectures.
Moreover, we show that our proposal remainsrewarding and can improve the generalization capabilities of CL models even when a pre-trainedmodel is employed.
Such scenario is important for a twofold reason: i) as shown in [45], pre-trainingimplicitly mitigates forgetting by widening the local minima found in function space, thus making themodel more robust to input perturbations; additionally, ii) we accommodate for real-world scenarioswhere pre-training is usually involved as an initial step.
Due to space constraints, for additionalexperimental details we kindly refer the reader to the supplementary material.
Benchmarked models
We conduct experiments on multiple state-of-the-art rehearsal-based models
If not specified, all methods use reservoir sampling [65] to update the memory buffer.
• Experience Replay with Asymmetric Cross-Entropy (ER-ACE)
[14]: the authors devise an improvement over standard ER with a separate loss for data coming from the stream and from the
buffer.
Consequently, they observe a significant performance gain, which shows the importance of
reducing the drift in incrementally learned representations.
• Dark Experience Replay (DER)
[12]: the authors propose an improvement over ER to takeadvantage high-level knowledge previously learned by means of self-distillation [24].
In additionto storing the one-hot labels, they also include the pre-softmax activations (logits) of the modelsampled along the optimization trajectory; subsequently, they enforce consistent responses through
time by minimizing the L 2 norm between current and past logits.
In this work, we use DER ++ , a stronger baseline that combines standard replay and self-distillation.
• Incremental Classifier and Representational Learning (iCaRL) [52]: instead of training a
classifier to solve the task directly, the authors propose to learn a representation suitable for a nearest-neighbor classification w.r.t.
class prototypes stored in the buffer.
Additionally, forgetting
is prevented by distilling the responses of the model’s snapshot at the previous task to both current and replay examples
Since samples in the buffer have a major role as anchors for classifications, the buffer is managed through the herding strategy
• Greedy Sampler and Dumb Learner (GDumb) [51]: the authors challenge the improvements made on CL literature, in which the authors propose to simply train the model at the task boundarysolely on data from the buffer
4.1 Datasets & measures
We focus our evaluation on the CIL setting and rely on commonly used and challenging imageclassification tasks.
In each experiment, classes from the main dataset are split into separate anddisjoint sets, which are then used sequentially to train the evaluated models.
Split CIFAR-100 .
An initial evaluation is carried by splitting the 32 × 32 images from the 100classes of CIFAR-100 [36] into 10 tasks.
In detail, we run two tests: i) using a randomly initialized
ResNet18 [27] – which constitutes a common scenario in literature [74, 52, 18, 13]; ii) adopting thesame backbone but with its parameters pre-trained on Tiny ImageNet [58].
Split mini ImageNet .
This setting is designed to evaluate the robustness of models on longer sequencesof tasks, by splitting the 100 classes of mini ImageNet [63] – a subset of the ImageNet dataset, whereeach image is resized to 84 × 84 – into 20 consecutive tasks.
For this test, we opt for an EfficientGCNB2 [57] backbone with no pre-train.
Split CUB-200 .
A final, more challenging benchmark involves classifying large-scale 224 × 224images from the Caltech-UCSD Birds-200-2011 [66] dataset, organized in a stream of 10 20 -foldclassification tasks.
This evaluation is designed to highlight the importance of protecting pre-trainedweights from forgetting.
Indeed, due to the limited size of the training set, competitive performancecan only be achieved if each task can benefit from the initialization [17, 73].
As backbone network,we opt for the commonly available ResNet50 architecture pre-trained on the ImageNet dataset [21].
For each benchmark we measure performance in terms of the classification accuracy averaged acrossall seen tasks (Final Average Accuracy - FAA) and of Final Forgetting (FF) [16], formally defined as:
[Equation]
where a ti indicates the accuracy on task
τ i after training on the t th task.
4.2 Results
Results of our evaluation on Split CIFAR-100 can be found in Tab. 1, while results on Split CIFAR100, and Split mini ImageNet are in Tab. 2.
For further reference, each table includes the results ofa model jointly trained on all classes – which represents an ideal non-CL upper bound – and of amodel sequentially trained on each task, with no countermeasure to forgetting.
Across the board, we find LiDER to be capable of improving the performance of all base methods inall evaluated scenarios, both in terms of FAA and FF metrics.
Most notably, we find it especiallybeneficial for methods that feature the most compelling results – usually DER++ and iCaRL –, suggesting that their higher generalization capability can still benefit from increased smoothness in latentspace.
Differently, scenarios where a method fail to prevent forgetting i.e. GDumb with a reducedbuffer size usually feature a lower degree of benefit from LiDER but still a notable improvement.
However, as it takes advantage both from increased buffers and pre-trained initialization, we observea considerable performance gap with our proposal.
Notably, on Split CIFAR-100, GDumb with abuffer size of 500 moves from a performance gain of +0 .
98% to +6 .
46% when increasing the bufferto 2000 samples and +2 .
99% with pre-train.
Finally, we observe an average performance gain of 2 .
08% , and 4 .
36% on Split CIFAR-100,
Split mini ImageNet, and Split CUB-200 respectively.
Ablation studies
In the following sections we provide further insights on the effectiveness of our proposal in controllingand improving the generalization capabilities of the
[Table] [Table]
[Table]
5.1 Buffer Poisoning
To evaluate the performance of a ML model, practitioners usually resort to synthetic benchmarks thatsimulate the progressive arrival of novel knowledge from an unknown source.
While this methodologyis useful to compare the effectiveness of new proposals a key element is usually neglected: the labelsgiven to each sample may – and in the real-world usually does – be incorrect.
Notably, this aspectis even more relevant in a CL scenario and especially when a rehearsal strategy is employed.
Aspreviously shown, samples included in the buffer are the ones that are most likely to be overfitted,which would induce a severe loss of performance if the label is incorrect.
In the following, we propose to measure the negative effect of overfitting the wrong label for samplesincluded in the memory buffer.
Specifically, we simulate noise in the labels by randomly assigningwith probability p a new label to the samples as these are added in the buffer, a technique which wename label poisoning .
To better suit a CL scenario, poisoned labels are chosen from those of thecurrent task.
Results reported in Tab.
3 show the evaluation performed on top of DER ++ for the Split CIFAR-100benchmark.
As one could expect, performance degrades as p increases; however, we also observe that
LiDER retains an higher level of accuracy against poisoning.
This effect suggests that our proposalleads to a looser decision boundary around the items store in the buffer.
[Figure]
[Table]
[Figure]
Guessing
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure]
[Equation]
[Figure]
[Equation]
[Table]
5.2 Buffer Guessing Game
In this section, we set up a novel experiment aimed at further illustrating the peculiar overfittingbehavior of rehearsal-based CL models.
As mentioned in Sec. 1, we posit that a substantially differentregime affects replay and stream examples: the former plays a much larger role in shaping thedecision boundary w.r.t.
the latter.
To validate this intuition, we propose a simple buffer guessinggame : given a rehearsal-based model f fully trained on a CL benchmark and the dataset τ 0 used inits first encountered task, we aim to find M ∩ τ 0 ( i.e. the subset of data-points that are included inthe model’s buffer).
We approach the game by associating each x ∈ τ 0 with a score s x computed as the mean height ofthe decision surface of f in a neighborhood of x 3 ; finally, we evaluate the ROC curve obtained fromthese scores.
In Fig.
2, we report the results obtained for ER-ACE and ER-ACE+LiDER on Split
CIFAR-100 across different buffer sizes.
We see that: i) ER-ACE makes it is easier to reconstruct thecontent of the buffer, as indicated by larger ROC-AUC scores w.r.t.
ER-ACE+LiDER; ii) in line withour expectations, this effect is increased when employing smaller memory buffers, as this leads to therepeated optimization of a smaller pool of data.
5.3 Generalization Measures
Previous CL works proposed investigating the generalization capabilities of a CL model by evaluatingthe flatness of its attained minima [12, 10].
We likewise evaluate the effect of LiDER both interms of its resilience to weight perturbations [48, 33] and the eigenvalues of the Hessian of theloss function [33, 30] in Fig.
We see that DER ++ and ER-ACE combined with LiDER seean improvement of both metrics.
This is expected, as the Lipschitz constant of a DNN has beencommonly interpreted as a generalization measure [69, 59].
[Figure]
5.4 Optimization with a fixed target
Instead of constraining the optimization of the Lipschitz targets of Eq.
7 by an additional contribution,one might argue that a similar effect could be obtained by fixing an initial value and force the modelto reduce its capacity until it meets the desiderata [41].
In Tab. 4 we empirically show that suchapproach does not lead to satisfactory results in a CL setting, with our proposal consistently exceedingits performance on various settings and base methods.
5.5 Decision Landscape of LiDER
In Fig.
1, we presented a pictorial demonstration of a model’s tolerance to input perturbations inthe form of a decision surface plot [72].
Such a visualization is constructed by focusing on a set ofperturbations x p ≜ x + i · α + j · β computed around a data-point x
( α is a random divergence directionand β corresponds to the direction induced by the first step of a non-targeted FGSM attack [37]).
Theplot shows the respective values of the decision function S ( x p ) , where S ( x ) ≜
f ( x ) t − max i ̸
= t f ( x ) i(with f ( · ) indicating pre-softmax responses) and highlights decision boundary of the model ( i.e. thelocus of { x p | S ( x p ) = 0 } ), in correspondence of which the model accuracy fails.
In Fig.
4, we adopt the same approach to compare the decision boundaries around rehearsed τ 0examples for ER-ACE with and without LiDER.
It can be observed that, while both models startwith a similar robust decision landscape in τ 0 , later tasks reveal a clear shrinking behavior in ERACE.
On the contrary, introducing L -based regularization hardens the backbone against adversarialperturbations, leading to minimal decision boundary deterioration in later tasks.
Conclusions
We present LiDER, a novel regularization strategy to compensate for the phenomenon of bufferoverfitting in rehearsal-based Continual Learning; with its carefully-designed loss term it enforcesthe decision boundary of replayed samples to be smoother by bounding the complexity of the model.
We show that such approach can be readily applied to diverse state-of-the-art models and remainscompetitive across various settings and backbone architecture.
Finally, we provide an extensiveinvestigation to assess the validity of theoretical assumptions.
Limitations & Societal Impact
Our approach heavily relies on the estimation of the Lipschitz constant and, in particular, on its upperbound (as reported in the previous section, the exact computation is a NP-hard problem).
We highlightthat tighter bounds have been recently proposed, at the expense of higher complexity and hence slowercomputation speed.
Moreover, the approximation exploited in this work cannot be naively applied tothose architectures comprising of not-Lipschitz continuous layers ( e.g. , cross-attention) [34].
Concerning societal impact, we do not feel that this work will have detrimental applications that 314 might affect any public.
We only remark that, as our approach is based on rehearsal techniques thatstore in-plain data, it cannot be used in those scenario where privacy constraints are crucial.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code for reproducing our experiments can be found in the supplemental material and will be made publicly available at publication.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multipletimes)?
[Yes] Due to space constraints, variance values for our experiments are included in the supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] Experiments were conducted on an on-premise desktopmachine, equipped with a RTX 2080 consumer GPU.
However, we did not keep a detailed recordof total computation time.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] In our work we use original code, with the only exceptions for the evaluation datasets whose original papers are cited.
(b) Did you mention the license of the assets?
[N/A]
Our work uses original code and freely available datasets.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
[N/A] We use datasets commonly established and used in literature, for which, to the best of our knowledge, the issues mentioned above have never been found to
affect them.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on
[Table]