CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks
Anonymous Author(s)
AffiliationAddress email
Abstract
Previous works have validated that text generation APIs can be stolen throughimitation attacks, causing IP violations.
In order to protect the IP of text generation
APIs, a recent work has introduced a watermarking algorithm and utilized thenull-hypothesis test as a post-hoc ownership verification on the imitation models.
However, we find that it is possible to detect those watermarks via sufficient statistics of the frequencies of candidate watermarking words.
To address this drawback,in this paper, we propose a novel Conditional wATERmarking framework (CATER)for protecting the IP of text generation APIs.
An optimization method is proposedto decide the watermarking rules that can minimize the distortion of overall worddistributions while maximizing the change of conditional word selections.
Theoretically, we prove that it is infeasible for even the savviest attacker (they know how
CATER works) to reveal the used watermarks from a large pool of potential wordpairs based on statistical inspection.
Empirically, we observe that high-order conditions lead to an exponential growth of suspicious (unused) watermarks, making ourcrafted watermarks more stealthy.
In addition, CATER can effectively identify the
IP infringement under architectural mismatch and cross-domain imitation attacks,with negligible impairments on the generation quality of victim APIs.
We envisionour work as a milestone for stealthily protecting the IP of text generation APIs.
Introduction
Nowadays, many technology corporations, such as Google, Amazon, Microsoft, have invested aplethora of workforce and computation to data collection and model training, in order to deploywell-trained commercial models as pay-as-you-use services on their cloud platforms.
Therefore, thesecorporations own the intellectual property (IP) of their trained models.
Unfortunately, previous workshave validated that the functionality of a victim API can be stolen through imitation attacks, whichinquire the victim with carefully designed queries and train an imitation model based on the outputsof the target API.
Such attacks cause severe IP violations of the target API and stifle the creativityand motivation of our research community [36, 40, 15, 9].
In fact, imitation attacks work not only on laboratory models, but also on commercial APIs [40, 41],since the enormous commercial benefit allures competing companies or individual users to extract orsteal these successful APIs.
For instance, some leading companies in NLP business have been caughtimitating their competitors’ models [33].
Beyond imitation attacks, the attacker could potentially surpass victims by conducting unsupervised domain adaptation and multi-victim ensemble [ d i ff e r en t ne w g r ea t h
i gh good w
ea t
he r w i nd s peed . . . .
U n i on p r opo s a l E U
no v e l M
r
e m i nen
t d i ss i m il a r e s t i m ab l e 10e10e-  10e+
10e+
# w o r d
( ben i gn ) /
# w o r d
( w a t e r m a r k ed )
In order to protect the IP of victim models, He et al.
[10] first introduced a watermarking algorithmto text generation and utilized the null-hypothesis test as a post-hoc ownership verification on theimitation models.
However, traditional watermarking methods generally distort the word distribution,which could be utilized by attackers to infer the watermarked words via sufficient statistics ofthe frequency change of candidate watermarking words.
As an example shown in Figure 1, thereplaced words and their substitutions are those with most frequency decrease ratios and frequencyincrease ratios, respectively.
To address this drawback, we are motivated to develop a more stealthywatermarking method to protect the IP of text generation APIs.
The stealthiness of the new watermarksis achieved by incorporating high-order linguistic features as conditions that trigger correspondingwatermarking rules.
Overall, our main contributions are as follows:
• We propose a novel Conditional wATERmarking framework (CATER) for protecting text generation APIs.
An optimization method is proposed to decide the watermarking rules that can i)minimize the distortion of overall word distributions, while ii) maximize the change of conditionalword selections.•
Theoretically, we prove that a small number of the used watermarks could be blended in andcamouflaged by a large number of suspicious watermarks when attackers attempt to inverse the
backend watermarking rules.
• Empirically, we observe that high-order conditions lead to the exponential growth of suspicious(unused) watermarks, which encourages better stealthiness of the proposed defense with little hurtto the generation of victim models.
Preliminary and Background2.1 Imitation Attack
An imitation attack (a.k.a model extraction) aims to emulate the behavior of the victim model V , suchthat the adversary can either sidestep the service charges or launch a competitive service [36, 15, 40,9, 41].
Malicious users can achieve this goal through interaction with the victim model V withoutknowing its internals, such as the model architecture, hyperparameters, training data, etc.
Adversariesfirst craft a set of queries Q based on the documentation of a target model.
Then Q will be sent to Vto obtain the corresponding predictions Y .
Finally, an imitation model S can be attained by learninga function to map Q to Y .
Most prior imitation attacks are limited to classification tasks [36, 24, 9].
The imitation for textgeneration, a crucial task in natural language processing, has been under-developed until recently.
Inspired by the efficacy of sequence-level knowledge distillation [13], Wallace et al.
[40] and Xu et al.
[41] propose mimicking the functionally of commercial text generation APIs.
Similar to thestandard imitation attack, adversaries can query V with Q .
For generation tasks, Y is a sequence oftokens ( y 1 , ..., y L ) , where L is the length of the sequence.
According to their empirical studies, onecan rival the performance of these APIs, which poses a severe threat to cloud platforms.
2.2 Identification of IP Infringement
Prior works have utilized watermarking avenues to achieve a post-hoc verification of the ownership [37, 19, 20].
However, this line of work assumes the model owners can watermark victim model
V by altering its neurons before releasing V to end-users.
This operation is not feasible for theimitation attack, as V cannot access the parameters of S .
The only thing under the control of V isthe responses to adversaries.
Hence, some recent works propose creating a backdoor to S duringthe interaction with attackers [15, 34].
Specifically, V can select a small fraction of queries andanswer them with incorrect predictions, in a similar way to the popular choice of watermarks inthe computer vision domain (adopting some arbitrary features as the trigger to evaluate) [7, 43].
Erroneous predictions are so abrupt that S will memorize these outliers [4, 16].
As such, V can utilizethese watermarks as evidence of ownership.
Albeit the efficacy, the drawbacks of backdoor approaches are tangible as well.
First, since V does notimpose regulations on users’ usage, one cannot distinguish a malicious user from a regular user basedon their querying behaviors 1 .
Thus, V has to fairly serve all users and store all mislabeled queries,which leads to a massive storage consumption and a negative impact on the users’ experiences.
Moreover, as the identity of imitation models is unknown to V , V has to iterate over all the mislabeledqueries, which is computationally prohibitive.
Finally, as S tends to adopt the pay-as-you-use policyfor the sake of profits, the brute-force interaction with S can cause drastic financial costs.
As a remedy, He et al.
[10] utilize a lexical watermark to identify IP infringement brought by imitationattacks.
They point out that a neat watermarking algorithm must follow two principles: i ) it cannotsignificantly impair customer experience, and ii ) it should not be reverse-engineered by malicioususers.
In order to fulfill these requirements, they first select a set of words W from the training dataof the victim model V .
Then for each w ∈ W , they find R − 1 semantically equivalent substitutionsfor it.
Next, they employ W and their substitutions T to compose watermarking words M .
Finally,they replace W with M .
The rationale behind this avenue is to alter the distribution of words suchthat the imitation model can learn this biased pattern.
To verify such a biased pattern of the wordchoice, He et al.
[10] employ a null hypothesis test [29] for evaluation.
More concretely, He et al.
[10] utilize an evaluation set O to conduct the null hypothesis test.
They formulate the null hypothesis as: the tested model generates outputs without preference forwatermarks .
A null hypothesis can be either rejected or accepted via the calculation of a p-value [29].
They assume that all words { w i | w i ∈ W ∪ T } follow a binomial distribution Pr ( k ; n, p ) , where k isthe number of words in M appearing in O , n is the number of words in W ∪ T found in O , and p isthe probability of watermarks observed in the natural language.
According to their algorithm, p isapproximated by 1 /R .
Now, one can compute the p-value from as follows:
[Equation]
The p-value indicates how one can confidently reject the hypothesis.
Lower p-value suggests that thetested model should be more likely subject to an imitator.
CATER
This section introduces our proposed CATER , leveraging conditional watermarks to watermark the imitation model, which can be served as a post-hoc identification of an IP infringement.
Figure 2provides an overview of CATER, consisting of two stages: i)
Watermarking Stage and ii) Identification
Stage .
i)
Watermarking Stage The victimmodel V employs CATER to build alist of watermarking rules (with conditions) from training data.
When received a query Q from an end-user,
V initially produces a tentative answer Y .
Next, V utilizes CATER to conduct over ii) Identification Stage If a model
S is under suspicion, V can query itwith a verification set O .
After obtaining the responses Y from S , V canleverage CATER to testify whether S violates the IP right of V .
3.1 Watermarking Rule Optimization
Watermarking some words to a deterministic substitutions could distort the overall word distribution.
Therefore, some watermarks could be reversely inferred and eliminated by analyzing the worddistribution, as demonstrated in Figure 1.
We propose to inject the watermarks in conditional worddistribution, while maintaining the original word distribution.
The substitutions can be conditionedon linguistic features as illustrated in Figure 3.
Remarkably, given a condition c ∈ C and a group ofsemantically equivalent words W , one can replace any words w ∈ W with each other.
We formulatethe objective of conditional watermarking rules as:
[Equation]
[Table]
The two factors reflect two essential desiderata:
For each W , with w ∈ W , the overall word distributions before optimization P ( w ) =
(cid:80) P ( w | c ) P ( c ) and after optimization ˆ P ( w ) =
(cid:80) ˆ P ( w | c ) P ( c ) should be close to each
other, as the indistinguishable objective in Equation 2;
For a particular condition c ∈ C , the conditional word distributions should still be distinct to their original distributions, reflected by the dissimilarity between P ( w ( i ) | c ) and ˆ P ( w ( i ) | c ) ,
as the distinct objective in Equation 2.
This guarantees the conditional watermarks are identifiable in verification
In practice, we utilize multiple synonym word sets as a group G = {W ( i ) } |G| i =1 .
For each W ( i ) ,we can formulate Equation 2 as a mixed integer quadratic programming using ℓ 2 -norm as distancemeasurement function:
[Equation]
We define matrix X =
[ ˆ P ( w ( i ) | c )] |W ( i ) |×|C| as the variables for optimization.
Matrix W =[ P ( w ( i ) | c )] |W ( i ) |×|C| and vector c =
[ P ( c )] |C|× 1 are constant variables, decided by calculatingcorresponding distributions in a large training corpus.
The objective of Equation 3 is convex when αis sufficiently small (see the proof in Appendix A).
We optimize the watermark assignments ˆ P ( w ( i )
| c )using Gurobi [8]
3.2 Constructing Watermarking Conditions using Linguistic Features
This part will concentrate on practical ways to construct the watermarking conditions C .
We considertwo fundamental linguistic features F , i) part-of-speech and ii) dependency tree, and their highorder variations as conditions.
Such linguistic features were widely and successfully used for textclassification [3, 42], sequence labeling [17, 30], etc.
[Figure]
Part-of-Speech Part-of-Speech (POS) tagging is a grammatical grouping algorithm, which cancluster words according to their grammatical properties, such as syntactic and morphological behaviors [12].
The POS tag for each token is demonstrated in a colored box, in Figure 3.
Given a word w in a sentence, we denote its POS as l 0 , and use l − k or l + k to represent the POS of thek -th word to the left or right of w .
We consider a single label l − 1 as our first-order condition.
In orderto reduce the identifiability of our conditional watermark, we can construct high-order conditionsfrom the same feature set, e.g. , ( l − 1 , l +1 ) as second-order condition and ( l − 2 , l − 1 , l +1 ) as third- condition.
Note that if l − k or l + k does not exist, we use a pseudo tag “[none]" by default.
describes grammatical roles of words and its classes are limited combination of POS of ananchor and its neighbors should also be bounded the bond
as the condition
Dependency Tree Dependency Tree (DEP) is a syntactic structure, which describes directedbinary grammatical relations between words [12], as shown in Figure 3.
A dependency tree can berepresented by an acyclic directed graph G =
( V, E ) , where V is a set of vertices corresponding toall words in a given sentence, E is a set of ordered pairs of vertices, denoted as arcs .
An arc e ∈ Edescribes a grammatical relation between two vertices in V , i.e., source vertex named as head andtarget vertex coined as dependent .
Except the root vertex, each vertex is connected to by exactly onehead.
Consequently, there exists a unique path from each vertex to the root node in a dependency tree.
Analagously for POS features, we can design first-order and high-order DEP features as watermarkingconditions.
Given a word w , and its incoming DEP arc, we use the DEP label of the arc as thefirst-order features ( d 1 ) .
We construct high-order condition recursively using the labels of incoming
DEP arcs ( d 1 , d 2 , · · · ) .
A pseudo arc label “[none]" is used when there is no parent node in recursion.
3.3 Identifiability of Conditional Watermark
In this section, we discuss the identifiability of our watermark if the attackers attempt to infer theused watermarks.
We assume the worst case that the attackers have access to i) the watermarkingalgorithm, ii) all possible word sets for substitution G , and iii) combination of feature sets F aswartermarking conditions C .
However, the exact watermarking rules are unobservable to attackers.
The attackers may identify the watermark rules by suspecting those observed P ( w ( i ) | c ) with extremedistributions, i.e. , only a single word in a synonym set is selected given a specific condition.
Given a limited budget, we assume that an imitator has queried our watermarked API and has acquired
N tokens in (cid:83)
i W ( i ) .
The system has incorporated watermarks with K -order features c ∈ C , where
C =
( F 1 , F 2 , · · · F K ) , the total number of possible conditions is |C| =
(cid:81)
Ki =
1 |F i | .
We simplify ourdiscussion by using the same feature set F (POS or DEP), then |C| = |F| K .
Theorem 3.1.
If |F| K > N and there exist t conditions that have less or equal to m ∈ Z + supportsamples, then t ≥ |F| K − N/ ( m + 1) .
The attacker would suspect conditional word distributions that are extremely imbalanced , namelyonly a single dominant choice of word is observed within the responses to the attacker.
Theorem 3.2.
Having m support samples for a specific condition c , the possibility of observingextremely imbalanced word choice is I ( W , c, m ) =
(cid:80) w i ∈W P ( w i
| c ) m .
If m ′ ≤ m and m, m ′
∈
Z + ,
I ( W , c, m ′ ) ≥ I ( W , c, m ) .
The proofs of Thm 3.1 and Thm 3.2 can be found in Appendix B and C. Thm 3.1 guarantees alower bound for the number of conditions that attackers will have less or equal to m observedsamples.
Moreover, the lower bound grows exponentially with regard to the feature orders used aswatermarking conditions.
Thm 3.2 guarantees the high probability of the conditions with extremelyimbalanced word selection when m is small.
Combining these two theorems, the total number ofthe suspicious watermark rules could be huge compared with the limited used watermark rules ifwe are utilizing high-order linguistic features as conditions.
We further empirically demonstrate thesignificant confusion between suspected and used watermark rules in Section 4.2.
Experiments
Text Generation Tasks.
We examine two widespread text generation tasks: machine translationand document summarization, which have been successfully
deployed as commercial APIs.
• Machine Translation: We consider WMT14 German (De) →
English (En) translation [1] asthe testbed.
We follow the official split: train (4.5M) / dev (3,000) / test (3,003).
Moses [14] isapplied to pre-process all corpora, with a cased tokenizer.
We use BLEU [26] and BERTScore [44]to evaluate the translation quality.
BLEU concentrates on lexical similarity via n-grams match,whereas BERTScore targets at semantic equivalence through contextualized embeddings.• Document summarization: CNN/DM [11] utilizes informative headlines as summaries of newsarticles.
We reuse the dataset preprocessed by See et al.
[31] with a partition of train/dev/test as287K / 13K / 11K. Rouge [21] and BERTScore [44] are employed for the evaluation metric of thesummary quality.
We use 32K and 16K BPE vocabulary [32] for experiments on WMT14 and CNN/DM, respectively.
Models.
For the primary experiments, we consider Transformer-base [38] as the backbone of bothvictim models and the imitation models.
Following He et al.
[10], we use a 3-layer Transformerfor the summarization task.
Because of their superior performance, pre-trained language models(PLMs) have been deployed on cloud platforms.
Hence, we also consider using two popular PLMs:i) BART (summarization ) [18] and ii) mBART (translation) [22] as the victim model.
Regardingthe imitation model, since the architecture of the victim model is unknown to the adversary, wesimulate this black-box setting by using three different architectures as the imitator, namely (m)BART,
Transformer-base, and ConvS2S [6].
The training details are summarized in Appendix D.
Basic Settings.
As a proof-of-concept, we start our evaluation with a most straightforward case.
We assume the victim model V and the imitation model S use the same training data, but S usesthe response y ′ with CATER instead of the ground-truth y .
We set the size of synonyms to 2 and [Table] vary this value in Appendix E.1.
The detailed construction of watermarks and approximation of p in
Equation 1 for CATER is provided in Appendix D.
Baselines.
We compare our approach with [39] and [10].
Venugopal et al.[39] proposed watermarking the generated output with a sequence of bits under the representation of either n-grams orthe complete sentence.
He et al.
[10] devises two effective watermarking approaches.
The first onereplaces all the watermarked words with their synonyms.
The second one watermarks the victim APIoutputs by mixing American and British spelling systems.
4.1 Performance of CATER
Table 1 presents the watermark identifiability and generation quality of studied text generation tasks.
Both [10] and CATER obtain a sizeable gap in the p-value, and demonstrate a negligible degradationin BLEU, ROUGE, and BERTScore, compared to the non-watermarking baseline.
However, [39] fallsshort of injecting detectable watermarks.
Although CATER is slightly inferior to [10] in p-value, weargue that watermarks in [10] can be easily erased, as their replacement techniques are not invisible.
As shown in Figure 1, the synonyms used by [10] can be identified due to the tangible distributionshift on the watermarks.
In addition, one can eliminate the spelling watermarks by consistently usingone spelling system.
Unless otherwise stated, we use the first-order POS as the default setting for CATER, due to itsefficacy in terms of watermark identifiability and generation quality. [Table]
IP Identification under Architectural Mismatch
The architectures of remote APIs are usuallyunknown to the adversary.
However, recent works have shown that the imitation attack is effectiveeven if there is an architectural mismatch between the victim model and the imitator [40, 9].
Todemonstrate that our approach is model-agnostic, we use BART-family models as victim models andvary architectures of imitation models.
Table 2 summarizes p-value and generation quality of CATER on WMT14 and CNN/DM datasets.
Similar to Table 1, CATER can confidently identify the IP infringement when the architecture of the imitation model is the same as that of the victim model, with a gap of p-value between watermarkedmodel and benign model being 3 orders of magnitude.
In addition, this gap applies to the case, wherewe use distinct architectures for the victim model and the imitator.
The generation quality exhibitsnegligible drops, within a range of 0.3.
Note that the generation quality of Transformer and ConvS2Simitators degrades due to the capacity gap, compared to powerful BART-family models.
[Table]
IP Identification on Cross-domain Imitation Similarly, the training data of the victim model is confidential and remains unknown to the public.
Thus, therecould be a domain mismatch between the training dataof the victim model and queries from the adversary.
In order to exhibit that our approach is exempt fromthe domain shift, we use two out-of-domain datasetsto conduct the imitation attack for the machine translation task.
The first is IWSLT14 data [2] with 250K German sentences, and the second is OPUS(Law) data [35] consisting of 2.1M German sentences.
Table 3 suggests that despite the domainmismatch, CATER can still watermark the imitation model, and one can identify watermarks withhigh confidence.
High-order Conditions We have shown that the first-order CATER effectively performs varioustasks and settings.
We argue that CATER is not limited to the first-order condition.
Instead, onecan use high-order CATER as mentioned in Section 3.2, which can consolidate the invisibility asdiscussed in Section 3.3.
Therefore, we investigate the efficacy of the high-order CATER to thetranslation task and provide the study on the summarization task in Appendix E.2.
Figure 4 shows that with the increase of conditional POS orders, there is no side effect on thegeneration quality.
Although the gap between the benign and watermarked models is mitigated usinghigh-order conditions, it is still significantly distinguishable.
[Figure]
4.2 Analysis on Adaptive Attacks
The previous sections illustrates the efficacy of CATER for watermarking and detecting potentialimitation attacks.
Given the case that a savvy attacker might be aware of the existence of watermarks,they might launch countermeasures to remove the effects of the watermark.
This section exploresand analyzes possible adaptive attacks based on varying degrees of prior knowledge of our defensivestrategy.
Specifically, we examine two types of adaptive attacks that try to erase the effects of thewatermark: i) vanilla watermark removal , and ii) watermarking algorithm leakage .
Vanilla Watermark Removal.
Under this setting, we assume the attackers are aware of theexistence of watermarks, but they are not aware of the details of the watermarking algorithm.
[Table]
Following such a setting of attacker knowledge, we assume the attacker would adopt an existingwatermark removal technique in their vanilla form.
To evaluate, we employ ONION, a populardefensive avenue for data poisoning in the natural language processing field, which adopts GPT2 [28] to expel outlier words.
The defense results are shown in Table 4.
We find that ONION cannoterase the injected watermarks.
Meanwhile, it drastically diminishes the generation quality of theimitation model.
[Figure]
Watermarking Algorithm Leakage.
Underthis case study, we assume attackers have access to the full details of our watermarking algorithm, i.e., the same watermarking dictionaryand the features for constructing watermarking conditions.
We note that this is the mostsubstantial attacker knowledge assumption wecan imagine, aside from the infeasible case thatthey know the complete pairs of watermarkswe used.
After collecting responses from thevictim model, the attackers can leverage theleaked knowledge to analyze the responses tofind the used watermarks.
As shown in Section 3.3, we theoretically prove that such areverse-engineering is infeasible.
In addition,
Figure 5 shows that even with such a strong attacker knowledge, the amount of potential candidate watermarks (orange curve) is still astronomical times larger than the used number of watermarks (blue curve).
Thus, malicious users wouldhave difficulty in removing watermarks from theresponses; unless they lean toward modifyingall potential watermarks.
Such a brute-force approach can drastically debilitate the performance ofthe imitation attack, causing a feeble imitation.
Finally, we demonstrate the upper bound (greencurve) to show that without the curated knowledge about the watermarking conditions, the difficultyof identifying the watermarks from top 200 words can be combinatorially exacerbated.
Conclusion
In this work, we are keen on protecting text generation APIs.
We first discover that it is possibleto detect previously proposed watermarks via sufficient statistics of the frequencies of candidatewatermarking words.
We then propose a novel Conditional wATERmarking framework (CATER), forwhich, an optimization method is proposed to decide the watermarking rules that can minimize thedistortion of overall word distributions while maximizing the change of conditional word selections.
Theoretically, we prove that it is infeasible for even the savviest attackers, who know how CATERalgorithms, to reveal the used watermarks from a large pool of potential watermarking rules basedon statistical inspection.
Empirically, we observe that high-order conditions lead to an exponentialgrowth of suspicious (unused) watermarks, rendering our crafted watermarks more stealthy.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section X.
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] See Section 4.1(b)
Did you describe the limitations of your work?
[Yes] See Appendix F(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix F(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sections 3.1and 3.3(b)
Did you include complete proofs of all theoretical results?
[Yes] See Appendices A, Band C3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] {+Seehttps://anonymous.4open .science/r/cater_neurips-
+}
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.1 and
Appendix D
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Appendix D
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]