Prompt Learning with Optimal Transport for Vision-Language Models
Anonymous Author(s)
AffiliationAddress email
Abstract
With the increasing attention to large vision-language models such as CLIP, therehas been a significant amount of effort dedicated to building efficient prompts.
Unlike conventional methods of only learning one single prompt, we proposeto learn multiple comprehensive prompts to describe diverse characteristics ofcategories such as intrinsic attributes or extrinsic contexts.
However, directlymatching each prompt to the same visual feature is problematic, as it pushes theprompts to converge to one point.
To solve this problem, we propose to applyoptimal transport to match the vision and text modalities.
Specifically, we firstmodel images and the categories with visual and textual feature sets.
Then, weapply a two-stage optimization strategy to learn the prompts.
In the inner loop, weoptimize the optimal transport distance to align visual features and prompts by the
Sinkhorn algorithm, while in the outer loop, we learn the prompts by this distancefrom the supervised data.
Extensive experiments are conducted on the few-shotrecognition task and the significant improvement demonstrates the superiority ofour method.
Introduction
In the past few years, large-scale vision-language pre-trained (VLP) models, such as CLIP [39],
ALIGN [17], and BLIP [23] have achieved remarkable success in open-world visual concept learning.
These methods have brought new light but also pose a new question: how to efficiently adapt theknowledge from pretraining to the downstream tasks since these models are typically of massive sizeswhich are not feasible for normal users to re-train.
Brambling
A bird that lives in winter wood
A bird with dark fan-tail
A bird with orange and black texture
A bird with black crown and eye
One of the conventionalparadigms of utilizing pretrainedknowledge is “pre-training,fine-tuning”, which fixes thearchitecture of the pre-trainedneural network and tunes itsparameters using task-specificobjective functions.
Beyondfine-tuning the parameters,many recent methods [62, 63]introduce the concept of promptlearning from the field of NLP to the vision domain and achieve striking performance gain for thefew-shot visual classification.
They fix the model parameters and instead learn suitable promptsby turning a template sentence into a set of learnable vectors.
Then, these prompts are learned byminimizing the distance between the visual features and prompt-based language features.
Despite significant improvements over manual prompts, learning only a sentence is intuitivelyinsufficient to represent a class.
One class can be described by many intrinsic characteristics andeven extrinsic context relations.
Thus, for one object, we may have multiple prompt candidateswhich focus on different attributes.
As shown in Figure 1, we can describe the class “Brambling” indifferent views: such as the color of the wing, the color of the crown and eyes, the shape and color ofthe tail, and even the living environment information.
It motivates us to learn multiple prompts tocomprehensively represent the class and thus facilitate classification.
The most natural solution is to directly learn multiple prompts by respectively matching each promptwith the visual features.
However, it is the same as matching the mean of prompt features and thevisual features.
This solution is problematic since all prompts are encouraged to be closer to one singlepoint and thus tend to learn the same characteristics.
It contradicts our purpose to learn comprehensiveprompts.
To solve this problem, we tested adding some constraints to push away the prompt fromeach other, but found that this solution still fails to learn representative and comprehensive prompts.
This solution treats the visual representation as one single point, and such a unified view of visualfeatures ignores the fact that different prompts may only focus on one or a subset of characteristics.
To address this problem, in this paper, we propose Prompt Learning with Optimal Transport (PLOT),which applies optimal transport (OT) to align the local visual features and multiple textual prompts.
Optimal transport can calculate the distance between two distributions under the form of multiplesampling.
In our prompts learning framework, we formulate local visual features and multiple promptsas the samplings of two discrete distributions and use OT to encourage fine-grained cross-modalmatching.
Specifically, to obtain the local visual features with different semantic clues, we extract allfeature maps as the visual representation instead of the single global representation.
Fortunately, wecan easily obtain the visual feature maps from the visual encoder of CLIP by using all outputs of themulti-head self-attention layer [42].
Then the problem comes down to how to calculate the distancebetween two feature sets.
We solve this problem by introducing the optimal transport theory [50] and formulate the feature setsas a discrete probability distribution where each feature has an equal probability value.
Furthermore,to reduce the computational cost and avoid the extra model parameters, we learn the prompts witha two-stage optimization strategy.
At the first stage in the inner loop, we fix both visual and textfeatures and optimize the optimal transport problem by a fast Sinkhorn distances algorithm
Then,in the outer loop, we fix all parameters of optimal transport and back-propagate the gradient to learnthe prompts with different characteristics.
We conduct comprehensive experiments on 11 datasetsfollowing the standard setting of CLIP [39] and CoOp [62] to evaluate our method.
These experimentsspan the visual classification on generic objects, scenes, actions, fine-grained categories and so on.
The significant result improvement demonstrates that PLOT can effectively learn representative andcomprehensive prompts.
Related Work
Optimal Transport The Optimal Transport [30] is initially introduced to solve the problem of howto reduce the cost when moving simultaneously several items.
Recently, OT theory has drawn wideattention in the machine learning and computer vision community by comparing distributions readilyavailable to them under the form of feature sets [37].
Due to the brilliant property of distributionmatching, OT has been applied in many theoretic and application tasks including generative models [1,44, 59], structural matching [4, 56, 60, 55] (e.g. sequence matching [4] and graph matching [55]),and other distribution-based tasks (such as clustering [22], distribution estimation [2], and causaldiscovery [49]).
In this paper, we use OT distance to align the features of vision and languagemodalities and propose a two-stage learning strategy to guide the learning of multiple prompts.
Vision-Language Pre-trained Models Vision-Language Pre-trained (VLP) models aim to explorethe semantic correspondence between the vision and language modalities through large-scale pretraining.
Recently, VLP models have achieved an exciting performance improvement in the zero-shotand few-shot visual recognition [39, 10, 62, 63, 58], which shows the great potential to promoteopen-world visual understanding with the help of language.
One key part of learning VLP models isthe self-supervised learning objective on two modalities.
The popular VLP objectives can be dividedinto reconstruction [25, 15, 8, 20], contrastive matching [39, 17, 16], or the combination of bothtwo [24, 53, 19].
Besides, recent progresses in the filed of VLP also benefit a lot from large scale pair-wised datasets.
For example, CLIP [39] apply 400 million image-text pairs for the contrastivelearning, while ALIGN even exploits 1.8 billion data pairs.
Beyond recognition, these VLP modelsalso show great potential for other downstream applications, such as dense prediction [42, 61], imagegeneration [31, 41, 35], and action understanding [52, 47].
Prompt Learning Prompt learning is introduced from the field of NLP to efficiently adapt the largelanguage model to downstream tasks.
Different from the conventional “pre-training, fine-tuning”paradigm which initializes the pre-trained model and tunes the parameters of the network usingdownstream task-specific objective functions, prompt learning applies textual prompt to reformulatethe downstream tasks as the original pretrained task [27, 36].
By the prompt, the domain shift betweenpretrained task and downstream application is reduced and thus the pretrained knowledge can beeasier adapted to downstream tasks.
The concept of prompt learning [36, 40, 38] begins from thesuccess of GPT [40] series.
Early prompt learning methods (such as Petroni et al.
[36] and Pörner etal.
[38]) always manually create templates based on human prior knowledge.
Furthermore, somemining-based methods [18] and gradient-based methods [45] are proposed to automatically search forappropriate templates.
Beyond search in the discrete space, some methods [26, 48, 28] remove theconstraint that the prompts are “words” and instead learn prompts in the continuous embedding space.
Recently, CoOp [62] and its extended version [63] introduce prompt learning into open-world visualunderstanding to adapt the knowledge from the large-scale visual-language pretrained models andachieve great performance improvement on the few-shot visual recognition.
Compared with CoOp,our PLOT method further improve prompt learning by introducing the optimal transport distance tolearn multiple local prompts and achieve fine-grained vision-language matching.
Approach
In this section we will first revisit the baseline method CoOp 3.1, review the preliminaries ofoptimal transport 3.2, and then introduce our proposed PLOT 3.3 to show how we can learn multiplecomprehensive prompts.
3.1 A Revisit of CoOp
CoOp [62] is one of the pioneering methods to learn the prompts for using vision language pretrainedknowledge (such as CLIP [39]) for downstream open-world visual recognition.
Different from CLIPthat manually designs the prompt templates, CoOp sets a part of context words in the template ascontinuous learnable parameters which can be learned from the few-shot data.
Then the classificationweights can be represented by the distance between the learned prompt and visual feature.
Specifically, given an image x , a visual feature f = f ( x ) is obtained by the visual encoder f of
CLIP.
Then, the textual prompt can be formulated as t k = { v 1 , v 2 , . .
. , v L , c k } , where c k is theword embedding of the class name, { v l | Ll =1 } are learnable vectors with the same dimension as theoriginal word embedding and L is the length of context words.
With prompt t k as the input, the textencoder g outputs the textual feature as g k = g ( t k ) .
The final prediction probability is computed bythe matching score as follows:
[Equation]
where sim ( · , · ) denotes a metric function such as cosine similarity, and τ stands for the temperatureof Softmax.
Then we can optimize the parameters of { v l | Ll =1 } with the cross-entropy loss betweenthe prediction and the labeled target.
3.2 Optimal Transport
Optimal transport (OT) distance is a widely used metric for the comparison of distributions.
Here, weonly focus on the discrete situation which is more related to our framework.
Assuming we have twosets of points (features), the discrete distributions are formulated as:
[Equation]
[X X
[Equation]
CLASS
[Equation]
Text
Encoder
ImageEncoder
Brambling, House finch, …
[Figure]
OptimalTransportDistance where u and v are the discrete probability vectorss that sum to 1, and δ f is a Dirac delta functionplaced at support point f in the embedding space. Then, the total distance of these two distributionsare written as:
[Equation]
We call C the cost matrix in which each point denotes the cost between f m and g n , such as
C m,n = 1 − sim
( f m , g n ) .
While the T is called the transport plan, which is learned to minimize thetotal distance.
The optimization problem of optimal transport is formulated as:
[Equation]
As directly optimizing the above objective is always time-consuming, we apply the Sinkhorn distance [6] to use an entropic constraint for fast optimization.
The optimization problem with a
Lagrange multiplier of the entropy constraint is:
[Equation]
subject to
[Equation]
[Equation]
where h ( · ) is entropy and λ ≥ 0 is a hyper-parameter.
Then we can have a fast optimization solutionwith few iterations as:
[Equation]
6) where t denotes iteration and in each iteration u t = u / (( exp ( − C /λ ) v t − 1 ) and v t =v / (( exp ( − C /λ )
T u t ) , with the initiation v 0 = 1 .
3.3 Prompt Learning with Optimal Transport
In this subsection, we introduce the details of our PLOT, which learns multiple prompts to describedifferent characteristics of the category by minimizing the OT distance.
Specifically, as shown in Figure 2, given an image x , we first feed it to the visual encoder branch of
CLIP.
Apart from the global visual feature f , we can also obtain a set of local features { f m |
Mm
=1 } .
The visual encoder has a multi-head attention pooling layer in which the input is the combination ofthe global feature and a set of local features (feature map) and the output is a tensor with the shape 154 R ( H × W +1)
× C , where H and W is the height and width of feature map and C is the feature dimension.
Therefore, we can obtain M = H × W local features and a global feature.
At the same time, forclass k , we can initialize N local prompts as { t k,n | Nn =1 } with learnable vectors { v l,n | L,Nl =1 ,n =1 } , whereeach is same as the prompt in CoOp.
With both visual and textual encoders, we can obtain localvisual features F = { f m | Mm =1 } ∈ R M × C and prompt features G k = { g n | Nn =1 } ∈ R N × C .
In the inner loop, we learn the transport plan T with these fixed support sets F , G k , by minimizingthe following OT distance to push G k to F :
[Equation]
where C = F T G k denotes that we use the cosine similarity between F and G k as the cost matrix.
Then we can obtain the solution of transport plan T ∗ as Eq (6) and the final OT distance d O T ( k ) .
Given the OT distance between G k and F , we reformulate the prediction probability as:
[Equation]
In the outer loop, we fix the transport plan T ∗ and apply the cross entropy loss to optimize the{ v l,n | L,Nl =1 ,n =1 } as:
[Equation]
where y x is a one-hot label vector.
The detail algorithm can be found in the supplementary materials.
Experiments
Extensive experiments are conducted to evaluate our method, including comparison with CoOp,ablation studies, parameter analysis extensibility analysis, computing cost analysis and visualization.
4.1 Datasets
We followed the experimental settings in the CoOp [62] for the few-shot learning evaluation.
Theexperiments are conducted on the 11 visual recognition datasets, including Caltech101 [9], DTD [5],
EuroSAT [12], FGVCAircraft [29], Flowers102 [32], Food101 [3], ImageNet [7], OxfordPets [33],
StandfordCars [21], SUN397 [54], and UCF101 [46].
These datasets span visual classification ongeneric objects, scenes, actions, fine-grained categories and so on, which constitutes a comprehensiveevaluation of our method.
All experiments adopted the few-shot evaluation protocol used in CLIP [39]and CoOp [62], where we respectively choose 1, 2, 4, 8 and 16 shots for model training and useoriginal test set for evaluation.
Besides, we also evaluated the robustness of our method with domainshift.
Following CoOp, we used the ImageNet as the source domain and evaluate our method with
ImageNet-based robustness evaluation datasets including ImageNetV2 [43], ImageNet-Sketch [51],
ImageNet-A [14], and ImageNet-R [13].
The detailed introduction of each dataset can be found inthe supplementary materials.
4.2 Implementation details
We chose CoOp [62] as our main competitor to evaluate our method.
Compared with CoOp whichonly learns a global prompt for one class, our PLOT method learns multiple local prompts and appliesthe OT distance for better fine-grained alignment.
Besides, we also reported the performance oftraining a linear classifier with the CLIP [39] features.
It is also a widely-used strategy to adapt thepretrained knowledge for the downstream task [45].
We reproduced the performance of CoOp and CLIP linear probe with the released official code.
The original CoOp method has different versions with different class token positions and parameterinitialization strategies.
We applied the default model that fixes the class token positions in the end dueto the limited performance gap between two different ways of positioning the class token.
Besides, weused the random parameter initialization strategy but not the class-specific context version.
Followingthe widely used setting in [62, 63, 10, 57], we also chose RN50 [11] as the backbone network of thevisual branch and set the length of learnable context tokens as 16.
All the code of our method is basedon CoOp, which adopted the SGD optimizer with 0.002 initial learning rate, CosineAnnealingLRschedule, and a warmup trick with 1e-5 learning rate.
Besides, we also followed the epoch strategy totrain more epochs for more shots.
For the optimal transport distance, we apply N = 4 prompts for each category and use M = 7 × 7due to the feature map size.
We set the hyper-parameters in the Sinkhorn distances algorithm
[6] asλ = 0 . 1 for all the datasets.
We set the maximum iteration number of the inner loop as 100 and willearly stop the iteration when the average absolute update value Λ < 0 .
We initialize all values inthe vector v and µ as 1 /N and 1 /M respectively.
All models are conducted on the Pytorch [34] 1.7.1and trained on 4 NVIDIA A100 GPUs.
We repeated the experiments three times with different seedsand reported the average.
4.3 Comparison With CoOp
In this subsection, we compare our PLOT with the baseline CoOp on the few-shot recognition anddomain generalization tasks.
We summarized the experimental results in Figure 3 where the red line denotes our PLOT method,the blue one denotes CoOp, and the green one is CLIP linear probe.
The detailed accuracy can befound in the supplementary materials.
We observed that both prompt learning methods (PLOT and CoOp) outperform the linear probe method by a large margin.
Besides, PLOT can further improvethe performance of CoOp on most of the datasets.
Taking the average accuracy (at the left top) asthe example, Plot respectively gained 3 .
61% performance boost over CoOp at 1 , 2 , 4 , 8 , 16 shots.
We found the performance gap will reduce when shots increase.
It is notsurprising since both CoOp and PLOT focus on utilizing the pre-trained knowledge, and the effectof pre-training diminishes given more training data.
Among all datasets, PLOT achieves a largerimprovement over CoOp on the FOOD101 and DTD datasets and achieves comparable performanceonly on the StanfordCars datasets.
For the FGVCAircraft dataset in which the CoOp only obtains7 .
77% accuracy, our PLOT can achieve an accuracy of 17 .
79% , twice as high as that of CoOp.
Notethat we don’t use the class-specific context, thus the performance on the fine-grained classificationdatasets is lower, e.g. the performance of both CoOp and PLOT without class-specific context islower than the linear probing on FGVCAircraft.
All these performance comparisons can serve asexperimental evidence to demonstrate that multiple local prompts and optimal transport distancefacilitate the prompt learning of vision-language models.
4.3.2 Domain generalization
The robustness also plays a critical role in model applications since the real-world environment mayhave large domain shifts with the training data.
Therefore, we conducted a robustness evaluation toinvestigate the transferability of models learned by PLOT.
[Table] [Table] [Table] Table 1 summarize the results of our PLOT method and CoOp on four ImageNet-based robustnessevaluation datasets.
For both methods, we trained the models on ImageNet with 16 shots per class.
For PLOT, we set the number of prompts as N = 4 .
We can observe that PLOT outperforms CoOpconsistently on both source and target domains.
These experimental results demonstrate that theperformance improvement of our learning multiple prompts doesn’t rely on single-domain overfitting.
4.4 Ablation Studies and More Analysis
In this subsection, we conducted the ablation studies to investigate the effectiveness of differentcomponents, in order to answer the following questions.
Q: Can we directly learn multiple prompts by respectively matching each prompt with theglobal visual feature?
A: No .
As shown in Table 2, we report the performance of directly matchingthe global visual feature (notated as “G”) and compare it with the baseline CoOp and our PLOT onthree datasets including Caltech101, DTD, and FOOD101.
We observe that there is no improvement [Table] over the baseline on some datasets (such as Caltech101 and FOOD101) if we only directly matchprompts and global features.
Though “G” obtained the improvement on the DTD dataset, thisimprovement is still less than that of PLOT.
It is because this “G” method is incentivized to learn theindistinguishable prompts, which contradicts our purpose to learn multiple comprehensive prompts.
We further add some constraints to push away the prompt from each other.
For example, we add anobjective function to add the distance between every two prompts as a regularization term, whichis notated as “V”.
However, comparing “G” and “G+V”, we do not find significant and consistentimprovement when using variance loss.
Q: Does the improvement mainly come from using all feature maps?
A: No .
In PLOT, we applyall feature maps of the visual encoder branch, where each feature is a local embedding at one spatialposition.
Compared with the global feature, these local features are more informative and containfine-grained clues.
However, we demonstrate that the improvement of PLOT does not only rely onusing all feature maps.
On the contrary, directly using the feature map to replace the global featurecauses a large performance drop.
For example, on all three datasets, directly using the feature map(“M” or “M+V”) has an around 20% 1 shot accuracy drop over using the global visual feature.
Itis not surprising since the original CLIP model is trained by matching the global visual feature andlanguage feature.
Without using the OT method, the distance between the feature map and multipletextual prompts degenerates to the mean distance of each feature-prompt pair.
Besides, when usingthe feature map, adding the variance loss works well, especially for more shots.
For example, theaccuracy on 16 shots DTD is improved by a large margin (from 26.99 to 56.85).
Q: How many prompts are needed?
A: 4 prompts are enough One important hyper-parameterin PLOT is the number of prompts.
To analyze the effect of the number of prompts, we conductedthe experiments on three datasets with 1 , 2 , 4 , 8 prompts.
The results are summarized in the whitepart of Table 2.
We can observe that the performance obviously increases when adding the numberof prompts from 1 to 4.
For example, PLOT (N=4) respectively obtains 1 .
64% , and 1 .
68%1-shot accuracy improvement over PLOT (N=1) on three datasets.
Besides, when we further increasethe number of prompts, the improvement is not consistent.
To balance the improvement and cost,we set N = 4 as the default configuration of our PLOT model.
In the experiments, we tuned thishyper-parameter on the Caltech101 dataset and applied it to other datasets.
Q: Can PLOT benefit zero shot learning?
A: No . CLIP [39] shows that manually designing theprompts can still achieve good performance.
we obtain 7 prompts by prompt engineering on the
ImageNet dataset and can further ensemble them to obtain 60 .
38 % top 1 accuracy.
In this section,we replace the cosine distance between the global visual feature and prompt ensemble with the OTdistance between the feature map and all 7 prompts.
However, without any learning, the OT distanceonly obtains 58 .
78 % accuracy.
We argue there are two reasons why the OT distance does not workwithout learning: 1) prompt engineering selects prompts based on the global feature and cosinedistance, instead of OT distance with feature map; 2) all these selected prompts are closed to theglobal feature and lack the complementarity.
Q: Can PLOT benefit Adapter-based methods?
A:
Yes .
Adapter-based methods [10, 57] is anotherresearch direction of the efficient adaptation of pre-trained vision-language models.
Different fromthe prompt learning that fixes the model parameters and tunes the language prompt, adapter-basedmethods [10, 57] allow for fine-tuning a part of the network or adding an extra model for training.
Recently, adapter-based methods also achieve good performance on few-shot visual recognition.
Therefore, we want to explore whether our PLOT method can benefit them, and how.
We apply the Tip-adapter-F [57] as our baseline method, which learns a Linear ( d, N cls × K shots )model to describe one image by the similarity with all training samples, where d is the dimension ofvisual feature, N cls is the number of categories (e.g. 1000 in ImageNet), and K shots is the numberof shots.
Then, the final similarity consists of the original distance between the visual feature andprompt ensembling and the new distance calculated by the learned feature and one-hot vector of [Figure] labels (whose dimension is ( N cls × K shots , N cls ) ).
Please find details in Tip-adapter-F [57].
Tointroduce PLOT to this framework, we first used the feature map to replace the global feature andthen learned multiple linear models.
As a result, with different local features and different linearmodels, we can obtain a M × N distance matrix and applied the Sinkhorn algorithm [6] to calculatethe OT distance.
Furthermore, we can apply the learned prompts as co-partner of the ensemblingprompt to refine the final similarity.
Table 3 summarizes the few-shot recognition results of the original Tip-Adapter-F method and ouradapter-based PLOT methods on ImageNet.
From this table, We observe that using the OT distancecan improve the performance of the adapter-based method.
Using the learned prompts, we can furtherpromote the accuracy of all settings.
Q: What is the extra computation time cost of PLOT over CoOp baseline?
A: Around 10 %inference speed and 5 % training time .
Please see the detailed comparisons and analysis in thesupplementary materials.
4.5 Visualization
In this subsection, we provide some visualization examples of the transport plans T related to differentprompts (N=4).
We translate each transport plan into colorful heatmaps and resize them into theiroriginal size and combine them with the raw image.
As shown in Figure 4, we provide the heatmapsof 4 categories in ImageNet.
We observe that different transport plans highlight different regions ofthe image, which demonstrates that the learned multiple prompts are complementary.
For the class“Brambling”, the prompts respectively focus on the head, tail, wing, and environment.
For “Dog
Sled”, the prompts are related to dogs, the sled, some ties, and the snow environment.
Conclusion
In this paper, we present a method, named PLOT, to learn multiple comprehensive prompts todescribe diverse characteristics of one category.
To avoid convergence to one point, we propose toapply the optimal transport to achieve the fine-grained alignment between both vision and languagedomains.
We apply a two-stage optimization strategy where the inner loop fixes the prompts andlearns the transport plan to calculate the cross-modality distance, and the outer loop uses this distanceto optimize the prompt learner.
We build our method on the base of CoOp and achieve significantimprovement on the few-shot recognition task in various datasets, which demonstrates the advantageto learn multiple prompts instead of a single one.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu-
tions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See our analysis in Section 4.4. 1)
Our method is still need few-shot data for optimization, which cannot be applied in zero-shot setting.
The method needs more computing cost than CoOp.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
We propose
a general framework for using the vision-language pre-trained model.
It is not for specific
applications, which does not directly involve societal issues
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experiments
multiple times)?
[Yes]
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
[Yes] See Section 4.2.
[Table]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
[Table]
If your work uses existing assets, you cite the creators
[Yes]
[Table]
you the license of the assets
N you
any
either in the supplemental material or as a URL
[No]
[Table]
you discuss and consent was obtained from people whose data you’re us-
[Table]
ing/curating
N
[Table]
the you are /curating contains personally identifiable informa-
[Table]
tion or offensive content
[Table]
crowdsourcing or conducted research with subjects
[Table]
screenshots
applicable
[Table]
[Table]
[Table]
[Table]
[Table]