A Differentiable Semantic Metric Approximation in Probabilistic Embedding for Cross-Modal Retrieval
Abstract
Cross-modal retrieval aims to build correspondence between multiple modalities bylearning a common representation space.
Typically, an image can match multipletexts semantically, and vice versa, which greatly increases the difficulty of this task.
To tackle this problem, probabilistic embeddings are proposed to quantify thesemany-to-many relationships.
However, existing datasets ( e . g ., MS-COCO) and metrics ( e .
g ., Recall@K) are hard to fully represent these diversity correspondences dueto non-exhaustive annotations.
Based on this observation, we utilize semantic correlation computed by CIDEr to find the potential correspondences.
Then we presentan effective metric, named Average Semantic Precision (ASP), which can measurethe ranking precision of semantic correlation for retrieval sets.
Additionally, weintroduce a novel and concise objective, coined Differentiable ASP Approximation(DAA).
Concretely, DAA can optimize ASP directly by making the ranking function of ASP differentiable through a sigmoid function.
To verify the effectivenessof our approach, extensive experiments are conducted on MS-COCO and CUB
Captions, which are commonly used in probabilistic embedding for cross-modalretrieval.
The results show that our approach obtains superior performance over thestate-of-the-art approaches on all metrics.
The code and trained models are releasedat https://anonymous.4open.science/r/2022-NeurIPS-DAA-4F1F .
Introduction
As one of the most fundamental tasks in cross-modal retrieval image-text retrieval is in great demanddue to the explosive increase of multimedia data in social activities.
Given an image or a text asa
query, it aims to retrieve the most relevant items in the database of other modalities.
Althoughsignificant progress has been made [20, 11, 39, 34, 23, 22], this task remains challenging since theheterogeneity across different modalities.
To tackle this problem, most existing methods map thequery and retrieval items from distinct modalities into a common embedding space, which allowscomputing the similarity between different modalities.
A line of methods [20, 36, 11] extracts representations of images and texts by constructing a twostream deep neural network, which can learn the correspondence of inter-modalities well.
Anotherthread of work [39, 34, 47] focuses on the relationship within the intra-modality.
Recent studies [23,22, 26] have shown that vision-language pre-training is effective to learn generic representations frommassive image-text pairs.
These methods have achieved impressive results on existing benchmarks.
However, most of the prior works [20, 36, 11, 39, 34, 47] use projection functions to map a sampleas a deterministic embedding, which can only quantify one-to-one relationships.
It has no abilityto represent the semantic diversity in images or texts, e . g ., given a text query, the first four imagesin the second row of Figure 1 (c) can match it semantically, but these works using deterministicembedding only regard the ground-truth image (green) as positive.
To tackle this problem, Chun et
[Figure]
number of image-text pairs greater than the CIDEr threshold that there are a large number of image-text pairs with high semantic correspondence besides the ground truth.
(c) Both retrieved sets obtain the same score by AP or Recall@K. Nonetheless, compared with the first retrieved set, the second one predicted by the model using DAA is more semantically related to the text query.
Below each image is the semantic similarity computed by CIDEr.
It shows that the similarities computed by
CIDEr are generally consistent with the real semantic similarities between the text queries and the images.
al .
[7] introduce probabilistic embedding for cross-modal retrieval, which is powerful to representmany-to-many relationships, by converting the samples into diversified representations according to adistribution.
The multiplicity of probabilistic embedding heavily depends on the well-matched datasets, which areexpensive to annotate manually.
However, commonly used datasets for the cross-modal retrieval tasksuffer from non-exhaustive annotations, which miss several positive pairs.
These missing positivepairs (MPPs) enormously limit the diversity expressed in training and evaluation processes.
To findout all potential positives, we utilize CIDEr [41], a metric to evaluate the similarity of two captions inimage captioning, i .
e ., we regard all image-text pairs with CIDEr scores greater than the threshold aspositives.
Then, we observe that a caption with a CIDEr score greater than 0.3 can typically describeimages correctly (see Figure 1 (a)).
We measure the data scale of potential positives with different
CIDEr thresholds and find that there are 128,050 image-text pairs with the CIDEr scores
greaterthan 0.3 (see Figure 1 (b)), which means that each image equally matches 25.61 captions (only 5ground-truth captions) on the semantic.
The results strongly demonstrate that a large number ofpotential positives are wrongly regarded as negatives, which severely impede diversity in training andevaluation processes.
Nonetheless, most optimization methods [11, 14, 12, 43] and metrics revolve around the accurateprediction of positives in the ground truth, ignoring the correlations between other retrieved itemsand the query.
As the situation in Figure 1 (c), the retrieved set at the bottom is more in line with ourexpectations than the one above, even though they obtain the same score by the metrics only focusingon the ground truth, such as AP or Recall@K.
It means that ground-truth-based metrics are hard tofully represent the multiplicity for cross-modal retrieval.
To address these problems, we carry out thefollowing work.
In this work, we give a solution to alleviate the hindrance of MPPs to multiplicity for probabilisticrepresentations in the training process.
Firstly, to evaluate with non-ground-truth relevant items,we propose a new evaluation metric, named Average Semantic Precision (ASP).
ASP computesthe average deviation between prediction ranking and semantic ranking.
It indicates the semanticranking precision of each retrieved item.
We utilize CIDEr as the criterion for judging the semanticcorrelations of captions.
Secondly, we propose a concise and novel Differentiable ASP Approximation(DAA) method, which optimizes ASP directly by relaxing the discrete ranking function in the nondifferentiable ASP with a sigmoid function.
Eventually, our approach achieves state-of-the-artperformance on several multiplicity metrics.
The main contributions and novelties of this paper could be summarized below.
i)
We count thenumber of image-text pairs with strong semantic correlations on the most widely used dataset
MS-COCO.
The result shows that incompletely annotating MPPs is a crucial factor restricting manyto-many mapping for probabilistic embedding.
Thus, we propose an effective evaluation metricnamed ASP, instead of the existing ones, which are powerless to reflect the correlations between eachretrieved item and the query.
ii)
To eliminate the adverse effect of MPPs in the training process, wepropose a novel and concise metric learning method DAA, which approximates and optimizes ASPdirectly. iii) To verify the effectiveness of our method, we conduct experiments on two challengingbenchmarks and multiple sensible evaluation metrics.
Related Work
In this section, we briefly introduce some recent developments in probabilistic embedding for crossmodal retrieval and metric learning.
2.1 Probabilistic Embedding for Cross-modal Retrieval
Cross-modal retrieval requires a coordinated representation [3] that allows computing the similaritybetween the query and the retrieved item from distinct modalities.
Early works [11, 14] focused onone-to-one mapping, which utilizes multiple projection functions to map the sample as a deterministicembedding.
For example, canonical correlation analysis (CCA) is proposed to learn a linear projectionfunction and maximize the correspondences between positive images and texts.
Faghri et al .
[11]fully utilize the hardest negative to boost the matching performance.
Some prior works attempt toimprove the multiplicity of representations.
For example, Song and Soleymani [38] use a multi-headself-attention module to generate multiple candidate representations for a given input.
VSRN [21]adopts a graph neural network for semantic reasoning on the region proposals generated by a FasterRCNN [33] detector.
It gains significant performance improvements with the large increase incomputing expenses.
Recently, Chun et al .
[7] use probabilistic embedding in cross-modal retrievalto fix the defect of deterministic embedding.
Probabilistic embedding has shown strong performancein representing many-to-many correlations.
Although there have been feasible approaches [38, 21]to represent the multiplicity of samples, and promising results have been achieved in recent years,the existing not well-aligned datasets (NWD) are hard to support diversity learning.
To addressthis problem, Chun et al .
[7] propose a many-to-many metric Plausible Match R-Precision (PMRP),which computes similarities between different samples by the multiple class labels in MS-COCO.
Itregards two samples in MS-COCO whose class label difference is less than 2 as positive, which willmake a lot of false matches.
Meanwhile, while labeling well-aligned datasets which is as large as
MS-COCO, the cost is enormous.
Moreover, limited by the incomplete annotations of datasets, thereis no appropriate metric to represent the diversity performance of the model trained in NWD.
ASPis a metric considering the semantic precision of retrieved items.
The diversity performance of themodel can be accurately evaluated by ASP even if tested in NWD.
2.2 Metric Learning
Metric Learning maps input data into an embedding space, such that distance metrics can be appliedeasily.
A great body of work has focused on metric learning [2, 1, 44, 40, 37, 6, 45].
For instance,contrastive [6] loss and triplet [45] loss are two of the most widely used methods.
The former forcesall positive instances closer, while negative pairs are only separated by a fixed margin.
In the latter,positive pairs are pushed closer than negative pairs.
Despite impressive success has been achieved,the motivation to only minimize distance is limited.
The aforementioned methods ignore the ranking orders, which are essential for rank-based metrics such as Average Precision (AP).
Brown et al .
[5]propose a differentiable AP approximation loss that can optimize AP directly in image retrieval.
Compared with triplet loss, it shows better performance in AP, a ranking-based metric.
Impressiveprogress in optimizing AP has been achieved [40, 37, 10, 17, 16, 30, 35], but AP is a limited metric,which only focuses on the ground truth.
In our work, a ranking-based metric ASP is proposed toexploit potential semantic information.
Besides, we propose a Differentiable ASP Approximation(DAA) to optimize ASP directly.
To our knowledge, DAA is the first work that uses direct metricoptimization for cross-modal retrieval.
Preliminaries
[Figure]
Probabilistic embedding.
Chun et al .
[7] have a previous attempt at Probabilistic Cross-Modal
Embedding (PCME) to address the multiplicity of cross-modal correlation.
PCME models eachsample as a normal distribution (See Figure 2), which could be described by mean and variancetogether.
Therefore, the key to generating probabilistic embedding is determining the mean embeddingand the variance matrix.
Given an image v ∈ V and a caption t ∈ T as inputs, probability-basedmodel will learn their normal distributions as:
[Equation]
where f v is the feature map extracted from the visual encoder and f t is the feature sequence extractedfrom the textual encoder.
E ( · ) is a function mapping input to probabilistic embedding.
h µ and h σare two head modules used to compute the mean embedding and the variance matrix in PCME [7],respectively.
To learn the joint probabilistic embedding, the soft cross-modal contrastive loss isused [13], which can be defined as:
[Equation]
where θ =
( θ V , θ T ) are the parameters for the visual and the textual encoders, respectively.
Given animage-text pair ( v, t ) as input, p θ ( m | v, t ) is the match probability from Euclidean distances can besimilarly denoted as:
[Equation]
where E ( v ) j and E ( t )
j ′ are samples from the embedding distribution p ( E ( v ) | v ) and p ( E ( t ) | t ) via a
Monte-Carlo estimator [28], respectively.
( a, b ) are learnable scalars and ϕ ( · ) is a sigmoid function.
Approach
In this section, we elaborate on the proposed method.
In Section 4.1, we first delineate our novel evaluation metric, i .
e ., Average Semantic Precision (ASP), which can calculate the semantic correlationscores of the retrieved set easily and precisely even in noisy datasets.
In Section 4.2, we detail howto optimize ASP directly in probabilistic embedding by using Differentiable ASP Approximation(DAA).
4.1 Average Semantic Precision (ASP)
Existing diversity metrics [7, 29] for not well-aligned datasets (NWD) are hard to fully representthe semantic ranking accuracy of retrieved sets limited by incomplete correlation labels, such as
PMRP [7] or CxC [29].
The former treats samples in MS-COCO [24] whose class label differencenumbers are less than 2 as positives, but it will make a lot of false matches.
The latter manuallyannotates the testing set of MS-COCO, but still misses many potential positive pairs.
Meanwhile, bothof them are merely applicable to MS-COCO, i .
e ., they are not universal.
To solve these problems, wepropose Average Semantic Precision (ASP).
Given a query set Ψ =
{ Q i , i = 0 , ..., n } , a query q ∈ Ψcan simply calculate the semantic correlation scores of the retrieved set Ω
= { G i , i = 0 , ..., m } invarious image-text retrieval datasets.
Mathematically,
[Equation]
) where R ( i, C ) and R ( i, S ) refer to the rankings of the instance i in semantic list C (defined as Eq.
6)computed by CIDEr [41], and similarity list S (defined as Eq. 5) predicted by the model.
In Eq. 4,
ASP utilizes the overall gap between the semantic ranking and the similarity ranking to represent theprecision of a retrieved set.
[Equation] [Equation]
The SEM ( i, q ) is the semantic correlation score between query q and retrieved item i .
When q and irefer to an image and a text, we will select all positive texts (5 texts in MS-COCO) corresponding tothe image, and calculate their semantic similarities with i through the CIDEr metric, respectively.
Eventually, the mean of these semantic similarities will represent the semantic similarity betweenq and i .
Conversely, the positive texts corresponding to image i will be selected to calculate thesemantic correlation scores with text q .
The specific equation is as follows:
[Equation]
where P os ( q ) is the positive texts of q , P is the number of texts in P os ( q ) , ψ stands for the CIDErmetric.
4.2 Differentiable ASP Approximation (DAA)
Contrastive [6] and triplet [45] losses are two widely used metric learning methods, which are drivenby minimizing distance.
They both ignore the ranking order, which is important for ranking-basedmetrics.
Thus, the ranking-based metrics will be indirectly optimized in these methods.
ASP is aranking-based metric that is neither differentiable nor decomposable.
Consequently, we propose a
Differentiable ASP Approximation (DAA) to optimize ASP directly.
Given the ranking-based metric ASP, the reason why it is non-differentiable is that the derivativeof a ranking function R for instance i defined Eq gradient discontinuous
[Equation]
where 1 {·} serves as an Indicator function, which select 1 while ( u i − u j ) < 0 , otherwise select 0 [31].
U is the CIDEr score set C or the similarity score set S .
The difference matrix of D U ∈ R m × m canbe computed as:
[Equation]
The ASP for the query q can be converted from Eq.
4 to:
[Equation]
To make ASP differentiable, which could achieve direct optimization of ASP, we utilize a sigmoid function ϕ ( · ) to replace the Indicator function 1 {·} .
After substituting ϕ ( · ) into Eq. 8, theapproximation of ASP is denoted as:
[Equation]
(11) which is differentiable and could be optimized with gradient-based methods.
Therefore, the final lossfunction is defined as:
[Equation]
Experiment
In this section, we conduct experiments to verify the effectiveness of DAA, which could optimize
ASP
directly.
We describe the evaluation protocol including datasets and metrics, as well as the implementation details.
Then we report experimental results on MS-COCO [24], and CUB Captions [7]for the cross-modal retrieval task.
After that, we carry out a series of ablation studies to verify theeffectiveness of DAA in improving multiplicity.
Meanwhile, we present an experimental analysis ofthe effectiveness of DAA.
5.1 Evaluation Protocol
5.1.1 Datasets
We conduct the experiments on MS-COCO [24] and CUB Captions [7] to show the performance ofprobabilistic embedding using DAA for cross-modal retrieval.
MS-COCO is a widely used dataset for the cross-modal retrieval task.
It contains 123,287 imageswith five captions per image.
We follow the data partition in [19], which consists of 113,287 imagesfor training, 5,000 images for validation, and 5,000 images for testing.
We report our results on both5K and 1K testing sets (the average over 5-folds of 1K testing images).
CUB Captions [7, 32, 42] is a benchmark proposed to reduce the impact of false matching forprobabilistic embedding in MS-COCO.
It contains 11,788 images of 200 fine-grained bird categorieswith 10 captions per image.
All the image-text pairs which belong to the same class are considered aspositive pairs, hence false negatives rarely exist.
Meanwhile, the homogeneity of image-text pairsin the same class could largely suppress false positives.
We follow the class splits in [46], whichconsists of 150 classes for training and validation, and the remaining 50 classes for testing.
5.1.2 Metrics
We take Recall@K (R@K) as a measurement in both benchmarks.
R@K is a widely used metric incross-modal retrieval.
We report R@1, R@5, and R@10 for a comprehensive evaluation.
However,
R@K only focuses on the position of the first relevant item according to the ground truth, reflectinglimited information.
R-Precision (R-P) is an alternative proposed by Musgrave et al .
For each query, R-P computesthe ratio of matched items in the top- r retrieved items, where r is the number of ground-truth matches.
In short, for all matched items, the fewer negatives before, the higher the R-P score is.
Comparedwith R@K, R-P could better evaluate the one-to-many matching for cross-modal retrieval.
We use
R-P to evaluate the performance of DAA on CUB Captions.
As MS-COCO, we take Plausible Match R-Precision (PMRP) [7] metric as a measurement, whichutilizes the class labels for MS-COCO to find out all hidden positive image-text pairs.
Concretely,
PMRP regards the pair ( v, t ) , whose binary label vectors y v , y t ∈ { 0 , 1 } d label , differ at most at ζpositions ( ζ ∈ { 0 , 1 , 2 } ), as positive matches.
The above metrics can only reflect the precision of ground-truth items, ignoring the correlationsbetween other non-matched retrieved items and queries.
Therefore, we refer to ASP as an evaluationmeasurement for both benchmarks.
In most cases, we pay more attention to the relevance of header retrieved items.
However, the top-kretrieved items predicted by current high-performance models are typically highly correlated, onlywith a small CIDEr difference.
Concretely, as the second row in Figure 1 (c), the first four imagescan match with the query semantically, they should be equally treated as good matches even if theyare not ranked according to CIDEr score.
Thus, we apply a variant of ASP to evaluate the semanticcorrelations in the top-K retrieved items, named ASP@K (A@K).
A@K computes the ratio betweenthe total CIDEr scores of top-K retrieved items predicted by the model, and the sum of the largest
K in CIDEr list C (defined as Eq. 6).
We compute A@5, A@10, A@20 in the ablation studies on
MS-COCO (5K).
5.2 Implementation Details
We use ResNet [15] and the basic version of the pre-trained BERT [8] as visual and textual encoders.
There are three phases in training proceeds: The first is a pre-training phase, where only the headmodules in the visual encoder and all parameters in the textual encoder are trained.
A warm-upphase lasting three epochs is used in the pre-training phase.
Finally, all parameters will be trainedin an end-to-end fine-tuning phase.
Concretely, for MS-COCO, we use ResNet-152 backbone withembedding dimension D = 1024 .
For CUB, we use ResNet-50 backbone with embedding dimension
D = 512 .
Meanwhile, We utilize AdamP Optimizer [18] for model training with the batch size of128 and 150 for MS-COCO and CUB, respectively.
For both datasets, we set the probabilistic samplenumber as 5 and the initial learning rate as 0.00015 with the cosine learning rate scheduler [25].
Besides, models are always trained on a single RTX A6000 GPU with Cutout [9] and random captiondropping [4] augmentation strategies with 0.2 and 0.1 erasing ratios, respectively.
[Table]
Ours 45.
5.3 Comparisions with State of The Arts
In this section, we compare our method with previous works using ResNet as the visual encoderon the two datasets.
The comparison methods include VSE0 [11], VSE++ [11], PVSE [38], and
PCME [7].
Our model uses the basic version of BERT as the textual encoder.
To be fair, PCME isalso retrained with BERT as the textual encoder and compared with our approach.
Results on MS-COCO.
Table 1 shows the quantitative results of MS-COCO 1K and 5K testingsets for image-to-text (i2t) and text-to-image (t2i) retrieval.
We observe that our approach obtains aconsistent improvement in performance on all metrics, compared with PCME, which is the state-of- the-art model using probabilistic embedding for cross-modal retrieval.
Exactly, PCME-bert usingimplementation strategy in Section 5.2 emerges better performance than PCME [7] on R@1. Then, DAA is employed in the training process of PCME-bert, resulting in significant boosts on R@1, PMRP, and ASP.
As for MS-COCO (1K), compared with PCME, we achieve absolute boosts on PMRP, R@1, and ASP in both i2t and t2i, which are 0.9%, 2.4%, 3.7% and 0.7%, 3.2%, 1.9%,respectively.
As for MS-COCO (5K), compared with PCME, the boosts on PMRP, R@1 and ASP are0.5%, 4.0%, 3.8% and 0.5%, 3.2%, 2.0%.
The above results show that DAA could effectively enhancethe scores of multiple metrics, which indicates that semantics is useful to boost the performance ofthe model in cross-modal retrieval.
[Table]
Results on CUB Captions.
To further demonstrate the robustness of the proposed model, weevaluate our model on CUB Captions.
Compared with MS-COCO, R-P is more suitable for
CUB due to rare false pairs, which is acceptablefor evaluating the multiplicity of probabilisticembedding.
The results on CUB are presentedin Table 2.
It can be seen that DAA outperformsall the state-of-the-art methods.
Especially, forimage-to-text retrieval, DAA surpasses the previous best method on R-P, R@1, and ASP withan increase of 1.9%, 6.3%, and 0.3% respectively.
Besides, for text-to-image retrieval, DAAalso outperforms the best counterpart by a significant margin (1.7% for R-P, 2.5% for R@1, and 0.8% for ASP).
The above results demonstrate theeffectiveness and importance of exploiting semantic information in cross-modal retrieval.
5.4 Ablation Studies
In this section, we perform several ablation studies on MS-COCO 5K to explore the superiority of
DAA over the optimization method only focusing on ground truth ( e .
g ., Smooth-AP [5]), on theprobability-based model.
To explore the effect of DAA and Smooth-AP, we test the impact of using these two methods onmultiple metrics.
The results are shown in Table 3.
To start with, we first evaluate the model using
BERT as the baseline, where the sample number is 5.
Subsequently, we train our model with SmoothAP and find that there are only a few improvements on PMRP, R@1, and A@K (Section 5.1.2).
Finally, we add the DAA to the baseline, resulting in considerable boosts on all metrics, especially R@1 (2.6% for i2t, 1.9% for t2i) and ASP (4.1% for i2t, 2.8% for t2i).
The significant improvement of ASP shows that DAA shifts retrieved items with high relevance forward overall.
Besides, DAA obtainimpressive improvements on A@K (K=5, 10, 20), which reflects that DAA can improve the semanticranking precision even for the retrieval set with high semantic correlation.
These results implicate thatexcavating semantic correlation can not only improve the multiplicity of probability-based models,but also improve the traditional ground-truth-based metrics, such as R@K.
The visualization ofretrieval results is shown in the second row of Figure 1 (c).
[Table]
The results in Table 3 prove that DAA can increase the semantic similarity in the whole retrieval setand top-K ranked retrieval subset, which can be regarded as two sets with large and small CIDErscore variance.
To validate the effect of DAA for subsets with different CIDEr score variance, werank the retrieved set according to the CIDEr score in descending order and sample the top N tocompute ASP.
We draw the contrastive curves of ASP in Baseline, DAA, and Smooth-AP [5] with Nfrom 0 to the length of the whole retrieval set in Figure 3. We can observe that DAA always performs
[Figure]
Given a retrieval set in descending order according to CIDEr score, we compute the
[Figure]
better than others and the gap becomes larger along with the increase of N.
The results indicate that
DAA could enhance diversity performance in any retrieval set.
Meanwhile, the curves of Smooth-APand baseline are almost consistent, which shows that only focusing on ground truth is limited toimproving multiplicity.
To make clear the impacts of different methods on multiplicity in the training process, we test ASP ineach epoch for baseline, Smooth-AP, and DAA.
The results are shown in Figure 4, which providesa quantitative analysis of ASP against epochs.
We can observe that with the increase of trainingrounds, the curve of the training strategy using DAA grows constantly.
It substantially verifies [Figure] that DAA effectively captures complex semantic correspondences and enhances multiplicity in thetraining process.
The ASP curves of baseline and Smooth-AP strategies are almost invariant, while
Recall@K improves constantly.
This means that boosting the prediction accuracy of the items inground truth may not lead to the improvement of multiplicity.
All of these ablation studies stronglydemonstrate that DAA is superior to the optimization method focusing only on ground truth inimproving multiplicity.
Conclusion
This paper attempts to address a critical problem in probabilistic embedding for cross-modal retrieval,i .
e ., the not well-aligned datasets and existing metrics can hardly train and evaluate the multiplicityperformance of the model.
To solve this problem, we introduce Average Semantic Precision (ASP) toevaluate the performance of multiplicity in these not well-aligned datasets.
Moreover, we propose anovel Differentiable ASP Approximation (DAA), which could make the model diversified even iftraining in these noisy datasets.
We have demonstrated the effectiveness of our DAA for boosting themultiplicity of probabilistic embedding for cross-modal retrieval by outperforming the state-of-the-artmodels.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section
(b) Did you describe the limitations of your work?
[No] It need to be explored.(c)
Did you discuss any potential negative societal impacts of your work?
[No] No explicitnegative impacts.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section (b) Did you include complete proofs of all theoretical results?
[Yes] See Section
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See abstract, asa URL
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section
(b) Did you mention the license of the assets?
[Yes] See Section
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See abstract, as a URL
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](
[Table]
paid [Table] and the [Table]
[Table] on [Table] compensation
A [Table]