Towards Effective Multi-Modal Interchanges in Zero-Resource Sounding Object Localization
Anonymous Author(s)
AffiliationAddress email
Abstract
Aiming to locate the object that emits a specified sound in complex scenes, thetask of sounding object localization bridges two perception-oriented modalities ofvision and acoustics, and brings enormous research value to the comprehensiveperceptual understanding of machine intelligence.
Although there are massivetraining data collected in this field, few of them contain accurate bounding boxannotations, hindering the learning process and further application of proposedmodels.
In order to address this problem, we try to explore an effective multi-modalknowledge transfer strategy to obtain precise knowledge from other similar tasksand transfer it through well-aligned multi-modal data to deal with this task in a zeroresource manner.
Concretely, we design and propose a novel Two-stream Universal
Referring localization Network (TURN), which is composed of a localizationstream and an alignment stream to carry out different functions.
The former isutilized to extract the knowledge related to referring object localization from theimage grounding task, while the latter is devised to learn a universal semanticspace shared between texts and audios.
Moreover, we further develop an adaptivesampling strategy to automatically identify the overlap between different datadomains, thus boosting the performance and stability of our model.
The extensiveexperiments on various publicly-available benchmarks demonstrate that TURN canachieve competitive performance compared with the state-of-the-art approacheswithout using any data in this field, which verifies the feasibility of our proposedmechanisms and strategies.
Introduction
When interacting with the external environment, humans can gather and integrate much information through different types of perception (such as vision, hearing, smell, and taste), thus makingcorresponding judgments and decisions based on this.
To effectively mimic this ability to bridgeknowledge from various modalities, researchers have been trying to explore the potential of machineintelligence from different aspects.
Among them, as a crucial part of audio-visual learning, the taskof sounding object localization has gained an increasing amount of attention.
Given a specific sound, the goal of sounding object localization is to ascertain the object that emitsit in a complex scene.
To tackle this problem, a lot of efforts are made to model the co-occurrencepatterns in the visual and acoustic data and accordingly identify the possible target region.
And forthe research and development in this field, there have been numerous aligned data pairs collected intoseveral publicly-available benchmarks.
Nevertheless, few of them contain available box annotations,which forces the models to acquire knowledge in a weakly supervised manner and increases thedifficulty in the training process.
Apart from this, the annotations in the test split are usually givenas bounding boxes, which keeps consistent with other spatial localization tasks.
But restricted by the architecture and learning diagram, most of the proposed models can only generate results in theformat of heatmaps but not bounding boxes, bringing much inconvenience to the further utilization oflocated regions.
In spite of this, when we look into another subproblem of referring object localization, i.e., imagegrounding, we may find that the data in this field are usually equipped with accurate bounding boxannotations, which brings higher information density and better model performances.
So is there anysuitable solution to extract and migrate the knowledge from this field to our target scenario?
Theanswer is Yes .
As with the pivot-based alignment strategies adopted in the zero-resource machinetranslation [60, 40, 23, 51], we can turn to the datasets of audio retrieval and use a series of alignedtext-audio pairs as intermediaries to establish an efficient knowledge transfer pipeline to tacklethe target problem in a zero-resource manner.
Although it sounds straightforward and feasibletheoretically, such a learning scheme is still faced with two kinds of practical challenges: 1) Domaingap , which consists of the essential discrepancies between all these three data domains.
As depictedin the green part of Figure 1, the queries in image grounding put more emphasis on the appearance,quantity and spatial relationship of the objects, while those in audio retrieval tend to describe action,state and temporal change about the targets.
Meanwhile, similar gaps also exist in the visual inputs(illustrated in the red part) and the acoustic data in the other two modality pairs.
2) Modality gap ,which stems from the intrinsic properties within different modalities and the complicated relationshipsbetween them.
The noise and redundant information in these three modalities will hamper the transferprocess from different aspects, and the non-monotonic alignment between modalities also makes ithard to learn cross-modal representation in fine granularity.
To this end, we propose a novel Two-stream Universal Referring localization Network (TURN) tohandle the problem of sounding object localization in a zero-resource way and reduce the potentialperformance degradation from the challenges described above.
Specifically, we utilize a localizationstream to model the relationship between visual elements and referring queries so as to unearthvaluable location knowledge from image grounding data.
Meanwhile, we develop an alignmentstream to construct a universal semantic space for the linguistic and acoustic data, thus enablingefficient knowledge transfer to our target task.
Moreover, considering that different data items willaffect the transferability of knowledge to a varying extent, we further devise an adaptive samplingstrategy to automatically identify the overlap between various data domains and accordingly modulatethe sampling weights of data, which enhances the stability of the learning process and improves theall-around performance of our model.
In conclusion, our contributions in this paper can be summarized in the following aspects:
• We explore the task of sounding object localization from a zero-resource perspective for the
first time and discuss the feasibility of transfer-based solutions in this setting.
• We devise a novel adaptive sampling mechanism in the area of multi-modal knowledgetransfer, which can automatically discover the overlap between different data domains andadjust the sampling weights of data, thus enhancing the comprehensive performance of thelearning model.•
We propose an effective two-stream architecture for the target of zero-resource soundingobject localization.
Without using any training data in this field, TURN achieves competitiveperformance on publicly-available benchmarks, including MUSIC and VGGSS, whichdemonstrates the feasibility of our proposed solution.
Related Work
Audio-Text Representation Learning Cross-modal representation learning aims to capture therelationship between different modalities and project embeddings to the same latent space.
Thoughmany works have studied image-text [22, 15, 44] and audio-visual representation learning [5, 38, 52],audio-text representation learning is still under-studied.
Some works [38, 8, 35] directly computedthe similarity between embeddings but cannot well encode the semantic and acoustic informationinto the embeddings at the same time.
Other works such as Chen et al.
[12] attempted to learninteractive cross-modal representation with both acoustic and textual information followed the spiritof contrastive language–image pre-training, but might be deeply affected by the insufficiency oftraining data.
Considering that VQ-VAE [55, 6, 67] has shown its outstanding performance in discreterepresentation learning, we try to learn the audio-text representation via cross-modal reconstructionand codebook sharing based on the structure of VQ-VAE.
Transfer
Learning Typical transfer learning is usually applied to solve the problem of datadistribution inconsistency between source and target domains.
The mainstream approaches in thisarea usually tackle this problem from one of these three perspectives: data inputs [16, 3, 65, 11],intermediate representations [9, 41, 20, 31, 57, 66] and learning paradigms [36, 30, 32, 18, 46].
However, these works are based on the assumption that the source and target domains are conceptuallyclose and modally consistent.
In order to handle the situation where data from the source domain andtarget domain are not closely related, Tan et al.
[53] formulated the paradigm of Transitive Transfer
Learning (TTL) and exploited an auxiliary annotated intermediate domain to transfer knowledge.
And considering the cost of constructing the intermediate domain, Tan et al.
[54] further proposed anovel learning schema Distant Domain Transfer Learning (DDTL), in which the intermediate domainwill be automatically selected and constructed from multiple datasets without labels, thus achievingthe knowledge transfer between completely unrelated domains.
Recently, some researchers have alsodiscussed the usage of transfer learning strategies in the multi-modal scenarios, which covers the fieldof image caption [39], audio caption [70], machine translation [40] and so forth.
In this paper, wefurther explore the application of transfer learning paradigms in the task of zero-resource soundingobject localization as a complement to this field.
Referring Object Localization Given a complicated scene with multiple objects and a specificreferential query, the task of referring object localization is to ascertain the location of the object thatbest matches the given reference.
Depending on the modality of queries, this field can be subdividedinto Image Grounding and Sounding Object Localization .
In the scenario of image grounding, thequeries are given in natural language descriptions, and fully-annotated data are denoted to facilitatethis area.
Consequently, a lot of supervised approaches and architectures are proposed to tackle thisproblem, which can be further categorized into proposal-based [24, 28, 34, 58, 59, 61, 68, 71] andproposal-free ones [17, 33, 47, 63, 62].
The former estimate the confidences of all the pre-definedproposals and choose the best one in a ranking-based way, while the latter directly regress thecoordinates of bounding boxes based on the fully aggregated and fused representations.
When itcomes to acoustic queries, clear box annotations are usually unavailable in the training partition ofpreviously proposed datasets [14, 48, 69].
Therefore, researchers in this field turn to explore thesolutions [26, 42, 45, 50, 1, 25] based on contrastive learning paradigms and produce pixel-levelheatmap results in a weakly-supervised way.
Although sounding object localization shares lots ofcommonalities in essence with image grounding , there is still a lack of discussions and analyses oftheir relationships and characteristics.
In this paper, we make a pioneering attempt to link these twosubproblems and explore modality-independent universal referring object localization.
Method
3.1 Problem Formulation
As depicted in Figure 2, our goal is to extract adequate knowledge from the data of image groundingand modulate it with the assistance of well-aligned audio-text corpora in the field of audio retrieval,thus achieving an acceptable generalization performance on the task of sounding object localization.
For the sake of clarity, we regard the data of image grounding, audio retrieval and sounding objectlocalization are from the source , intermediate and target domains with the notations of D s , D m and
D t , respectively.
Under this definition, this problem setting can be also considered as a knowledgetransfer and generalization from D s to D t via D m .
To keep consistent with the other prior works andthe characteristics of different domains, only a limited amount of data from source and intermediatedomains are accessible to the model during training, and the annotated testing data of target domainwill be finally used to measure the overall generalization ability in the evaluation.
3.2 Model Architecture and Data Specification
Overall Network Structure
The overall workflow of our proposed model is depicted in Figure 2.
Specifically, our model conducts the calculations in two parallel pathways referred to as localizationand alignment streams in the following descriptions.
The localization stream focuses on modelingdependencies within visual elements and capturing relationships between inputs and queries, thusaccordingly producing the target bounding boxes.
In this part, a series of pre-trained backbones arefirst utilized to embed the input images into a group of object-aware features.
And these featureswill be further refined and aggregated with the guidance of query conditions and then used to predictthe coordinates of target regions.
Meanwhile, the alignment stream is developed to align the datadistribution of different domains and modalities into a shared space and generate debiased queryconditions to serve as references for the localization stream.
In order to construct a semanticallycomplete latent space of query embeddings, we exploit a group of shared vector quantizers and devisea cross-reconstruction mechanism to alleviate the domain and modality gaps existing in the inputaudio and text data.
Moreover, we additionally employ an auxiliary classifier to adjust the samplingweight during the training phase to help our model adaptively absorb knowledge from differentdomains.
Inputs and Annotations Before the detailed descriptions of our proposed architecture, we firstformulate the concrete symbolic definitions of the inputs, annotations and representations.
Tobegin with, the original data from source , intermediate and target domains are denoted as D s =
{ ( V s , T s , b s ) ∈ ( V s , T s , B ) } , D m = { ( T m ,
A m ) ∈
( V m , T m ) } and D t = { ( V t , A t , b t ) ∈
( V t , T t , B ) } .
For simplification, we omit the superscript in the following explanations when there isno need to distinguish between different domains.
In these notations, V ∈ [0 , 1] 3 × H × W , T ∈ N L tand A ∈ R L m × c represent the pixel matrix, word index sequence, and log-mel spectrogram of theimage, text and audio inputs, respectively.
As for the annotation format, b =
[ x, y, w, h ] ∈ [0 , 1] 4represents the target bounding box of queried object.
3.3 Localization Stream
As the basic component of referring object localization, the localization stream is applied to obtainlocation-related knowledge from the source domain and produce target bounding box predictions.
Given an image containing multiple objects V ∈ [0 , 1] 3 × H × W and a semantically condensedquery condition q ∈ R d , we first project the visual input into a series of object-aware embeddings
ˆV = { ˆv 1 , . .
, ˆv n v } via a pre-trained DETR backbone [10] and a group of multi-layer perceptrons,where ˆv ∈ R d and n v is the number of patch features.
After the initial extraction procedure, we combine the guidance information from query conditionsinto the visual representations via a simple element-wise multiplication.
And then, we employ anothertransformer encoder component to model the dependencies and relations among these query-awarevisual features and generate a bunch of fully fused representations, which can be given by¯v = 1 n v (cid:80) (Θ v ( ˆv 1 , ˆv 2 , . .
, ˆv n v )) , where ˆv i = v i ·
q / || v i · q || 2 .
In this formula, the notation Θ( · · · ) stands for the transformer encoder module and we conduct al 2 -norm to prevent numerical instability caused by the dot product calculation.
Finally, we calculate the aggregated representation by averaging all the patch features and utilize amulti-layer perceptron to generate the prediction of bounding boxes, given by
[Equation]
where W v1 ∈ R d × d , W v2 ∈ R 4 × d and c v1 , c v2 ∈ R
d are all learnable parameters, σ ( · ) is the sigmoidfunction and ˆb ∈ [0 , 1] 4 is the predicted bounding box.
3.4 Alignment Stream
In the overall knowledge transfer procedure, the alignment stream plays a crucial role in extractingthe domain- and modality-invariant representations from the original inputs and providing querycondition features as the reference of the localization stream.
In the training phase, we first project the text data into a series of continuous word embeddings,denoted as ˆT s = { ˆt s1 , . . .
ˆt s L t } and ˆT m = { ˆt m1 , . .
, ˆt m L t } , respectively.
And the log-mel spectrogramof audio data will be compressed via a stack of convolutional layers, given as ˆA m = { ˆa m1 , . . .
, ˆa m L a } .
Afterward, we utilize two transformer encoders to generate context-aware linguistic and acoustic features and aggregate them into into a single semantically-rich representation ¯t , ¯a ∈ R d , as formulatedin Equation 3.
It’s worth noting that the word embedding table and text encoder are shared betweensource and intermediate domains so as to filter out the domain-specific bias within the original data.
[Equation]
Next, we expect to learn a universal semantic representation space where the data with the samesemantics but from different domains or modalities will be mapped to adjacent positions so that thelocalization of target objects can be independent of the query source.
Regarding the remarkableperformance of variational autoencoder in fitting distributions of variables, we employ the straightthrough VQ-VAE [55] module to achieve this goal.
Following the strategy adopted in [7], we splitthe aggregated features into a total of q chunks and utilize a group of codebooks { Ψ 1 ( · ) , . . .
, Ψ q ( · ) }to project them into multiple shared semantic spaces and reorganize them into a series of debiasedquery representations ˇ t , ˇa ∈ R d , which will serve as the query condition term q in Equation 1 andcan be given by
[Equation]
where [; ] is the concatenation operator and ¯t i ,
¯a i ∈ R d/q represents the i -th part of ¯t and ¯a
(i.e.,¯t =
[ ¯t 1 ; . .
; ¯t q ] and ¯a =
[ ¯a 1 ; . .
; ¯a q ] )
Afterward, we further devise a self-cross-reconstruction mechanism to improve the robustness ofquery representations and enhance the mutual information between aligned texts and audios.
[Equation]
where T mask and A mask are the modified inputs in which a certain proportion of the original datainto the <mask> embedding and Ω( X , Y ) is a mask-predict transformer decoder to reconstruct themasked elements of Y with the reference of X .
As formulated in Equation 5, the texts from the sourcedomain are reconstructed with the guidance of itself and the reconstruction in the intermediate domainwill be performed in a cross-modality way.
Thereafter, the aggregated features of one modality willbe able to get modulated in a data-driven manner to serve as reasonable substitutes of the other one.
3.5 Training and Inference
As mentioned above, the entire architecture can be trained in an end-to-end mechanism.
Given theannotation b from the source domain, we follow the learning strategy adopted in [17] and apply acombination of regression-based and IoU-based loss functions to optimize the localization stream,formulated as
[Equation]
) where the L smooth − l 1 ( · , · ) is the smooth l 1 loss to optimize the regressions of each coordinateseparately, and L giou ( · , · ) is the GIoU loss to conduct an overall constraint on predicted boxes.
As for the alignment stream, the optimization target is composed of the following three aspects, whichwill be presented in the logical order of calculations.
Initially, in order to maximize the possibility ofmapping the aligned linguistic and acoustic data to the same discrete code, we apply a contrastivelearning paradigm with the InfoNCE loss [49, 56] in every mini-batch to constrain the distribution ofaggregation vectors, given by
[Equation]
where b is the size of each mini-batch and τ c is the temperature hyper-parameter.
D ( x , y ) measuresthe distance between two vectors x and y , and we use the the normalized inner-product to calculatethe cosine-similarity distance in this part.
And next, all the codebooks will conduct calculations in parallel and get optimized in a K -meansbased manner individually, which can be abstracted as
[Equation]
where f denotes the input of codebooks and sg ( · ) represents the stop-gradient operation.
Moreover, two reconstruction losses will be utilized to indirectly modulate condition representations,given by
[Equation]
where L ce ( · , · ) and L l 1 ( · , · ) are the cross-entropy and l 1 loss functions, respectively.
In conclusion, the optimization of alignment stream is summarized as
[Equation]
Finally, the overall loss function is combined as L =
L loc + L aln to learn our proposed architecture,where λ 1 , . . .
, λ 5 are the balancing factors to control the magnitude of corresponding terms.
And inthe process of inference, we can simply take the projected embeddings of audio data as the conditionrepresentations to the localization stream and generate the calculation result.
3.6 Adaptive Sampling Strategy
Although we have already obtained a set of domain- and modality-invariant representations andperformed sufficient alignments, the huge inherent gaps between the source and intermediate domainsstill heavily hamper the transferability and generalization ability.
As illustrated previously, there tendto be some domain-specific data samples in the corpora.
For example, “The half of the sandwich onthe right” might be a good item for image grounding, but there merely exists any acoustic informationin this scenario.
And it will also be too abstract to imagine the concrete object corresponding to “Aloud high-pitched sound takes place.”
Generally, a sample like “A car was going on the road.”
mayserve as an excellent point to bridge knowledge.
To this end, we try to estimate the domain-exclusivityof data items and take this as a reference for data sampling.
Specifically, given all the queries from
T s and T m , we first employ an extra classifier Φ( · ) to discriminate between them and use the domainlabels to optimize this module, formulated as
[Equation]
After this training procedure, the classifier will respond with higher confidence to the data from thesource domain and lower confidence to those from the intermediate domain.
On this basis, we can safely assume that t s with lower confidence Φ( t s ) and t m with higher Φ( t m ) ought to be sharedbetween these two domains and located within the intersection part.
Consequently, the data sampling weights will be assigned adaptively via the following mechanism.
[Equation]
12) and we have
[Equation]
where w ∗ i is the probability for the i -th item to be selected into mini-batches in the training phase.
In this formula, the term τ ∗ is applied to smooth the distribution of sampling weights and k is theexpected ratio of the maximum and minimum in all weights, constraining the range of possible resultsand reducing the impacts of outliers to some extent.
Experiments and Results
4.1 Datasets and Metrics
Image Grounding Datasets We choose RefCOCO [64]/RefCOCO+ [64]/RefCOCOg [37] as theimage grounding datasets and conduct in-depth studies on the RefCOCOg dataset.
The statisticinformation of these three datasets can be found on the webpage of Tensorflow.
1 We follow the datapartitions from Yu et al.
[64] 2 and only use the training set in the overall learning process.
Audio Retrieval Datasets
As for the audio-text datasets, we use Clotho [19] and AudioCaps [29].
Clotho is sourced from the online platform Freesound [21] and contains 4,981 audio samples and24,905 captions (5 captions for each audio clip).
AudioCaps contains video clips and correspondingnatural language captions, and we only use the audio clips extracted from the videos, resulting intotally 50,960 audio-text pairs for training.
Sounding Object Localization Datasets For the dev set, we use the annotated subset of FlickrSoundNet [48], which is built on Flickr-SoundNet dataset [4] and contains 2,786 annotated imageaudio pairs.
For the test set, we choose VGGSS [14] and MUSIC [69].
VGGSS is derived from
VGGSound [13] and contains 5,158 audio-image pairs.
MUSIC, as a musical instrument video dataset,consists of 685 untrimmed videos of musical solos and duets spanning 11 instrument categories.
Wefollow the data process pipeline of Hu et al.
[27], and finally we use 489 solo videos for test.
Metrics Following [43, 48, 14], we adopt Consensus Intersection over Union (cIoU) and Area Under
Curve (AUC) to evaluate the performance of our model.
It’s noteworthy that cIoU and vanilla IoUwill be equivalent under the circumstance of n = c = 1 as shown in Equation 14.
[Equation]
where { m j } nj =
and ˆm are the binary masks converted from ground-truth and predicted boxes.
4.2 Comparisons with State-of-the-art Methods
Though TURN is designed for the zero-resource scenario, it still reaches a counterpart to otherfull-resource state-of-the-art sounding object localization methods.
Here we compare our TURN withprevious leading methods on VGGSS and MUSIC-Solo as Table 1 and Table 2 respectively.
And itis noteworthy that we retain all the samples with multiple box annotations of VGGSS in the entireevaluation process.
We can find that TURN outperforms all of the previous methods in VGGSS inboth cIoU and AUC metrics.
In MUSIC-Solo, the results are similar – TURN obtains the best
AUCamong all the methods with a reasonable IoU score.
It is also worth noting that the cIoU and AUCof both datasets suffer a sharp drop when a random vector takes the place of the actual query in thelocalization stream (denoted as TURN (w/o Query)).
This observation shows that it is unreliable toretrieve the salient region as the predictions by only the localization stream and further indicates thevalidity of our proposed knowledge transfer strategy.
[Table]
[Table]
4.3 Ablation Studies
In this part, we conduct ablation studies to verify the effectiveness of each design detail in TURNaccording to the results on VGGSS and MUSIC-Solo.
Choice of Source and Intermediate Datasets TURN needs to generalize knowledge from the sourcedomain to the target domain with the assistance of the intermediate domain.
Therefore, we makea comparison between different selections of source and intermediate datasets, which is illustratedin Table 3.
Consistent with our expectations, the best choice for the source domain is RefCOCOg.
Compared with RefCOCO and RefCOCO+, the grounded textual descriptions in RefCOCOg
containmore details about the object itself, which is vital when aligned to the texts from the intermediatedomain.
As for intermediate domain datasets, it is straightforward to find that AudioCaps is moresuitable for the MUSIC-Solo dataset, and Clotho helps VGGSS more.
The reason can be attributed tothe fact that AudioCaps contains more instrument-related entries and indicates the names of differentinstruments in the descriptions, while the audio quality in Clotho is much higher, and there aremultiple descriptions assigned to the same audio, which reduces noise disturbances in general.
[Table]
Effect of Different Training Settings In order to verify the effectiveness of regularization termsin our model, we conduct an individual ablation study on each term.
As illustrated in Table 4, thecontrastive loss takes the most significant effect, implying it plays a leading role in the alignmentof data from different modalities.
And the codebook loss optimizes the learning process of sharedsemantic space, determining the effect of knowledge transfer.
Although the reconstruction loss hasno direct influence on representations, it can still improve the performance by enhancing mutualinformation between different modalities.
In addition, we also evaluate the adaptive sampling strategy,and the performance margin indicates the benefit it brings to the learning process.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] ✓
[Table] Choice of Hyper-parameters in Codebooks As described in Section 3.4, we use a group ofcodebooks to project aggregated features in the alignment stream.
Here we investigate how the groupnumber of codebooks and total codebook size influence the localization performance.
From Figure 3,we can observe that the total codebook size cannot be either too large (depicted by the green line, theembeddings from different modalities might be scattered into different partitions of the codebookspace) or too small (shown by the blue line, the codebook capacity is not enough).
And given a fixedtotal size, the codebook group number should also be tuned in a proper range.
According to theexperiment results, when the codebook number is set to be 2 and the total codebook size is set to be512, TURN gets the highest cIoU and AUC.
[Figure]
4.4 Qualitative Analyses
[Figure]
In order to comprehensively evaluate the performance of our TURN, we will analyze some successcases on both the VGGSS and MUSIC-Solo datasets, which are shown in Figure 4.
(The failure oneswill be discussed in the supplementary material 3 .)
From these cases, we can easily find that ourmodel obtains the basic ability to discriminate the corresponding target according to the specifiedsound from a complex scene.
Limited by the structural design, when faced with the multi-boxscenarios of the VGGSS dataset, our model can only try to randomly choose one (c) or take theirunion (d) as the prediction among multiple equivalent candidates.
Although this will result in a lowermetric score, these cases can still qualitatively indicate the satisfactory performance of our proposedarchitecture.
[Figure]
Conclusion
In this paper, we develop TURN, an effective two-stream architecture for zero-resource soundingobject localization, which consists of a localization stream and an alignment stream.
We explorethe task of sounding object localization from a zero-resource perspective for the first time anddiscuss the feasibility of transfer-based solutions in this setting.
Without using any training data fromthe field of sounding object localization, our proposed TURN reaches a counterpart to other fullresource SOTA methods on publicly-available benchmarks (including MUSIC and VGGSS), whichdemonstrates the feasibility of our TURN.
More detailed analyses further verify the effectiveness ofthe overall mechanism and architecture, and the qualitative analyses also provide a more intuitive andcomprehensive perception of the results.
Checklist
For all authors
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions
and scope? [Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
[Table]
) Did you report error bars (e
[Table]
g., with respect
[Table]
the
[Table]
seed
[Table]
experi
[Table]
[No [Table] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
[Table]
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]