Obj2Seq: Formatting Objects as Sequences with Class Prompt for Visual Tasks
Anonymous Author(s)
AffiliationAddress email
Abstract
Visual tasks vary a lot in their output formats and concerned contents, thereforeit is hard to process them with an identical structure.
In this paper, we proposea unified vision framework, Obj2Seq.
Obj2Seq takes objects as basic units, andregards most visual tasks as sequence generation problems of objects.
Therefore,these visual tasks can be decoupled into two steps.
First recognize objects of givencategories, and then generate a sequence to describe each of these objects.
Thedefinition of the output sequence varies for different tasks.
Obj2Seq is able toflexibly determine input categories to satisfy customized requirements, and be easilyextended to different visual tasks.
When experimenting on MS COCO, Obj2Seqachieves 45.7% AP on object detection, 89.0% AP on multi-label classification and65.0% AP on human pose estimation.
These results demonstrate its potential to begenerally applied to different visual tasks.
[Figure]
Introduction
Deep learning has made great progress in the field of computer vision, and recently researchers areseeking for a unified framework for various visual tasks.
This idea has been widely discussed in
Natural Language Processing (NLP), which results in the emergence of general language models.
In NLP, all tasks are formatted as sequence-to-sequence problems in order to be processed by asingle transformer [40].
Meanwhile, prompt learning [16] is utilized so that the general model iscompelled to perform a specific task.
With the help of large-scale text data, numerous pre-trainedlanguage models are developed [13, 1, 46], and they achieve excellent performance on a wide rangeof language tasks.
With these encouraging achievements in NLP, it is natural to ask whether we can implement a similarframework in computer vision as well.
One of the two obstacles to realize this is that visual tasksare defined in various formats.
They can hardly be unified like sequence-to-sequence problems in
NLP.
The other is that the model may encounter diverse categories even for an identical task.
If themodel is not aware of what it will process in advance, we cannot expect it to satisfy customizedrequirements flexibly.
There are some works taking a step forward to solve these problems.
M-DETR[22] takes sentences as inputs in addition to the images, so that it is able to flexibly detect specifictargets that mentioned and realize target-aware inference.
However, M-DETR is designed for objectdetection only, and hard to extend for general usage.
Pix2Seq [9] is a forward looking work to unifythe formats of different tasks.
It turns object detection into a sequence generation task, and directlyapplies next token generation to yield bounding boxes and categories.
Even so, Pix2Seq is not awareof the desired targets before inference, and the output sequence might become extremely long undersophisticated scenes.
In this paper, we propose a unified visual framework, Obj2Seq. It regards various tasks as sequencegeneration problems, and formats them into a two-step process.
First, with images and class promptsas inputs, Obj2Seq determines whether each given category exists or not in an image, and detectsobjects of these categories.
Then a sequence is generated as the output for each object in responseto the desired task.
Many visual tasks can be filled in this template by changing class promptsand the definition of output sequences.
For each specific task, class prompts provide additionalinstructions to impel the framework to attend to objects of specified categories, and the outputsequences are defined accordingly to include information required by the task.
This gives the creditto two new modules, Prompted Visual Indicator that receives class prompts from the inputs, and General Sequence Predictor that outputs sequences in a unified format for each object.
Obj2Seq hastwo main advantages.
On one hand, the behavior of Obj2Seq can be controlled and manipulated.
Bytaking different class prompts as inputs, Obj2Seq is able to focus on different objects and flexiblysatisfy specific requirements under practical scenes.
On the other hand, the sequences generated by Obj2Seq are of a unified format to describe objects, which makes it ready to be extended for a widerange of visual tasks.
In order to validate the versatility of our framework, we conduct experiments on MS COCO [25]for some common visual tasks.
Obj2Seq achieves 45.7% AP on object detection and 89.0% AP onmulti-label classification when trained for object detection, while it also achieves 65.0% AP on humanpose estimation with keypoint annotations.
These results indicate that our framework is effective fordifferent visual tasks, and has potential for general usage.
Related Works2.1 General Frameworks for Visual Tasks
The idea of pretrain a general model is firstly proposed in NLP [13, 1].
Currently, a similar trendappears in computer vision as well.
Researchers first borrow transformer as the base model [14, 39,27, 42, 12], and then design novel self-supervised algorithms to pretrain them with the help of hugeamounts of unlabeled data and modern hardware [10, 5, 24, 18, 8].
The pretrained models are capableof extracting robust feature for downstream tasks.
However, since only the pretrained backbone isprovided, downstream tasks still have to design their specific head structures and need further trainingto get desired results.
Recently, the academic raises a higher requirement to train a general framework that can be directlyapplied in different tasks.
It should be trained in an end-to-end way, and does not need finetuningfor a specific task.
Multi Task Learning (MTL) is one related field [6, 35, 17].
It constructs a sharedbackbone and multiple heads to process a fixed combination of tasks.
However, since MTL needsindividually designed heads and sufficient labels for every sample, it is expansive and unable to beextended to other tasks.
Recently, works like ML-Decoder [34] and X-DETR [2] attempt to constructa general framework for a set of some similar tasks.
However, they can hardly be generalized fortasks with totally different definitions.
Pix2Seq [9] provides a straight-forward solution to convert animage into a sequence of words.
However, it lacks interface to flexibly adopt for different scenes.
Inaddition, the whole image is sometimes too complex to be represented in a single sequence.
Different from previous work, we take objects as the basic unit in visual tasks, and unify them assequence generation for each single object.
Additional class prompts are used to specify the objectsin need.
2.2 Prompt Learning
Prompt learning [16] is firstly develop in NLP in order to directly apply pre-trained language modelsin different tasks.
It does not need finetuning model parameters, but rather modifies the inputs anduses a template to indicate the proposed task.
Therefore, the model handles all tasks in a unifiedmanner.
In order to train vision models with easily available image-text pair data, the concept of prompt isfirstly introduced to construct training targets with label text in computer vision [31, 45].
Thesealgorithms help supervision with large-scale data only for a certain task.
The prompts are used aslabels, and they cannot control the inference.
VPT [21] and FP-DETR [41] add prompt parametersin their framework.
These parameters are trained in the finetune stage for each task.
However,they do not make use of language embeddings, and the effect of these modules is hard to explain.
ML-Decoder [34] and MDETR [22] use text to impel the model to focus on specific categories orobjects.
It allows us to interfere the model behavior, however, these frameworks are limited to agroup of similar tasks.
In this paper, class prompts are used to give instructions about what categoriesto detect, and Obj2Seq is able to perform a wide range of vision tasks.
2.3 Sequence Prediction
Sequence prediction is a basic formulation for various NLP tasks, since the words in sentences aresequential naturally.
It has the characteristic that the forward process at time t depends on featuresor outputs from previous time steps.
Currently transformer [40] is widely utilized to solve theseproblems.
In order to generate a sequence, the output token at time t is taken as the input token of thenext step.
Some up-to-date pre-train algorithm use the input sentence as the target output, in order toperform next-token prediction for unsupervised training [1, 46].
As to computer vision, there is no preset order in the input image.
Sequence prediction is onlyavailable in image generation since they can generate an output image pixel by pixel [29].
Whentransformer becomes popular, it is firstly applied to process a group of image patches [14] or anunordered set of objects [4].
Recently, the idea of next-token prediction is borrowed in order todevelop novel self-supervised training algorithms [7, 20].
However, these algorithms are still closelyrelated to image generation, as they predict the image patches in sequence.
Pix2Seq [9] is the firstattempt to apply sequential prediction as a general method in a traditional visual task, and achievesa decent performance.
It regards the whole detection results as a single sequence, which abandonssemantic strutures in the original image.
On the contrary, our framework considers a more basicunit, objects, in the field of computer vision.
This makes Obj2Seq more suitable for visual tasks, andachieves better results.
Methods
3.1 Overview
Obj2Seq is a unified visual framework.
It takes an image I ∈ R 3 × H 0 × W 0 and a group of K classprompts C =
[ c (1) ...
c ( K ) ]
∈ R K × d as inputs, and outputs a set of objects, each with a sequence todescribe it.
It contains four main parts listed as below:
• Image Feature Encoder E to extract image features.
• Prompted Visual Indicator I to receive class prompts from inputs and initialize objectqueries.
• Object Transformer Decoder D to encode object-related features into object queries.
• General Sequence Predictor P to generate a sequence of desired attributes for each object.
When the inputs I and C are provided, Obj2Seq first extracts image features F I with Image Feature
Encoder E .
Then, Prompted Visual Indicator I receives class prompts.
It judges the existence of
ImageFeatureEncoder
[Figure]
Object
TransformerDecoder
Sequence Predictor
No Dogs 𝑠𝑠 :0.9, 𝑥𝑥 :0, 𝑦𝑦 :17, 𝑤𝑤 :148, ℎ :
Fig 2 v 𝐹𝐹 𝐼𝐼 𝐹𝐹 𝑂𝑂 𝑠𝑠 :0.7, 𝑥𝑥 :197, 𝑦𝑦 :84, 𝑤𝑤 :185, ℎ :468, 𝑥𝑥 1 :308, 𝑦𝑦 1 :204, 𝑥𝑥 2 :315, 𝑦𝑦 2 :193,…
[Figure]
[Figure]
𝑥𝑥 𝑦𝑦 𝑎𝑎𝑛𝑛𝑎𝑎𝑎𝑎𝑛𝑛 𝑥𝑥 𝑦𝑦 𝑤𝑤 ℎ 𝑦𝑦 𝑤𝑤 ℎ 𝑥𝑥 𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 𝑦𝑦 𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 𝑥𝑥 𝑛𝑛𝑒𝑒𝑛𝑛 𝑥𝑥
[Figure]
𝑦𝑦 𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 𝑥𝑥
[Figure]
Forward
Forward and Stop Gradient
Detection Results
KeypointResults
[Figure]
Cross Attn
[Figure]
𝑥𝑥 𝑦𝑦 𝑎𝑎𝑛𝑛𝑎𝑎𝑎𝑎𝑛𝑛 𝑥𝑥
[Figure]
𝑤𝑤 ℎ 𝑦𝑦 𝑤𝑤 ℎ 𝑥𝑥 𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 𝑦𝑦 𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 𝑥𝑥 𝑛𝑛𝑒𝑒𝑛𝑛 𝑥𝑥 𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 𝑦𝑦 𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 𝑥𝑥 𝑎𝑎𝑛𝑛𝑎𝑎𝑎𝑎𝑛𝑛
[Figure]
Pos Embs
Forward
Forward and Stop Gradient
Detection Results
KeypointResults
[Figure]
each category for classification, and initializes class-aware object queries ˆ F O .
Each object queryˆ f o ∈ R 1 × d is responsible for detecting and describing one object of a specific class.
Afterwards,these object queries are sent into Object Transformer Decoder D .
The decoder extracts object-relatedfeatures from F I according to ˆ F O , and transforms ˆ F O into well-encoded object queries F O .
Thesenew queries contain sufficient information to describe their corresponding objects as the task requires.
Finally, General Sequence Predictor P generates a sequence for each object query, which containsthe desired attributes and constitutes the final result.
The whole pipeline is shown in Figure 2a.
In the following sections, we mainly elaborate on Prompted Visual Indicator I and General Sequence
Predictor P , and then supply additional details in other parts of our implemention.
3.2 Prompted Visual Indicator
Different from processing with a complete category set, human beings usually know what categoriesto focus on in advance of each task.
This prior information makes their behavior more compact andaccurate.
Also, the categories can be flexibly changed according to specific requirements.
Therefore,is it possible that we provide not only the images but also additional instructions as inputs, so that themodel is forced to focus on objects of specified categories?
Prompted Visual Indicator borrows the idea of prompt learning from NLP [16].
It receives additionalclass prompts as inputs, therefore it can classify the input image with specific categories, and generateinput-dependent object queries to focus on desired objects.
As shown in Figure 2b, each class promptis a vector representing a certain category c ( k ) ∈ R d .
These input vectors are sent into promptblocks with cross-attention modules that can extract class-related features from F I .
After processingby these blocks, the output vectors f ( k ) c represent features about specified categories in the inputimage.
These vectors can be classified by a class-related binary classifier as in Eq.
The classifier compares them with original class vectors, and outputs a score s ( k ) C to indicate the existence of eachcategory.
Moreover, these features are utilized to generate class-aware object queries for furtherinference.
We retain K ′ categories that are more likely to exist with a certain retention policy, andperform Cartesian product with a set of N position embeddings to generate N object queries foreach retained class.
Each object query ˆ f o is the sum of a class vector f ( k ) c and a position embedding.
These object queries are sent into Object Transformer Decoder for further process.
The retentionpolicy is detailed in Section 3.4.
[Equation]
With the help of Prompted Visual Indicator, Obj2Seq can take not only images but also desiredcategories as inputs.
The model can flexibly classify and detect desired objects with given categories,and therefore is able to achieve better performance.
Moreover, each category is processed in anisolated way.
Obj2Seq is able to take any category subsets as the input, without any influence inresults.
This makes our framework flexible and easy to implement in different scenes.
3.3 General Sequence Predictor
In the field of NLP, a task-agnostic language model usually converts all language tasks into a sequenceto-sequence format.
This is intuitive, because we always answer questions by speaking a sentence out.
As to visual tasks, we can also describe the input image with a sentence.
Since the image containsmore sophisticated information, the sentence should be better organized with a more basic visualunit, objects.
In order to satisfy different requirements, we may need the bounding box, keypointoffsets or some other attributes in this sentence to describe an object.
Therefore, we construct General
Sequence Predictor, that generates a sequence of attributes to describe an object.
This paradigm canbe generalized among a wide range of visual tasks.
General Sequence Predictor P takes object queries processed by Object Transformer Decoder asinputs, and outputs a sequence of attributes for each object.
We illustrate how it works in Figure 2c,taking object detection and human pose estimation as examples.
For simplicity, we only elaborateon the prediction process for one object query, and omit the superscript ( i ) .
The predictor P firsttakes the object query f
o as the input z in, 0 of the first time step.
The input is then sent into self- andcross-attention layers to get the output feature z out, 0 .
For any other time step t , P takes the previousoutput feature z out,t − 1 as the input z in,t , and the self-attention layers always take all previous inputsz in, 0: t to calculate keys and values.
After the output feature z out,t is obtained, a linear layer isutilized to convert it into a logit z t .
Finally, logits from all time steps are further interpreted intospecific attributes in a non-parametric way, according to different tasks.
We formulate this process asin Eq.
[Equation]
General Sequence Predictor provides a unified way to generate output attributes for each object.
It can handle different visual tasks with an identical structure, which makes Obj2Seq ready to beimplemented for general usage.
3.4 Other Details
Image Feature Encoder Image Feature Encoder E consists of a resnet backbone and 6 multi-scaledeformable encoder layers, which is exactly the same as [49].
The backbone extracts multi-scalefeature maps { F ( l ) B } 4 l =1 from C 3 to C 5 and an additional downsampled scale C 6 .
These feature mapsare transformed to have d = 256 channels, and then processed by deformable encoder layers to obtain
F I .
Object Transformer Decoder Object Transformer Decoder D consists of 4 multi-scale deformabledecoder layers.
A decoder layer contains self-attention and multi-scale deformable attention modules,which is elaborated in [49].
Here we only implement 4 decoder layers for fair comparison incomputation with other methods.
Retention policy Here we discuss about how Prompted Visual Indicator retains categories andgenerates class-aware object queries.
During training, categories with ground truth objects are firstguaranteed to be retained.
Then we select categories with the highest scores to fullfill a fixed numberof K ′ classes.
While in inference, we can retain either top- K ′ categories or all categories over a fixedthreshold.
The influence of these policies is discussed in Section 4.4.3.
Objectness branch Besides describing an object with a sequence, we implement a separateobjectness branch to recognize if the object exists.
This branch provides a score s ( i ) for each objectas in Eq.
We first compute the probability of each object with given image and categories, andthen multiply it with its corresponding class existence s ( k ) C .
In Section 4.4.2, we will elaborate on theefficacy of this objectness branch .
[Equation]
Training Criterion
The outputs of Obj2Seq is a set of objects with specified categories.
Theseobjects are equivalent, and do not appear in a certain order.
Therefore, we use bipartite matchingloss for supervision [4].
It matches object queries with ground truth so that the loss of matched pairsis minimized, and then optimizes this optimal match.
This loss makes sure that each object is onlypredicted with one object query, eliminating the need of non-maximum suppression.
The definition ofthe detailed loss function varies for tasks, which will be elaborated in Appendix.
As to the classifierin Prompted Visual Indicator, we regard it as an auxiliary multi-label classification problem, andsupervise it with asymmetric loss [33].
Experiments
In this section, we first conduct experiments on three different visual tasks separately, includingobject detection, multi-label classification and human pose estimation.
These results demonstrate how
Obj2Seq solves visual tasks in a unified way.
After that, some ablation studies are provided to bettercomprehend the detailed options in Obj2Seq.
Here we only provide overall results.
Implementationdetails please refer to Appendix.
4.1 Experiments on Object Detection
In Table 1, we compare Obj2Seq with a group of popular detection frameworks on MS COCO [25].
We mainly compare it with different variations of DETR [4] here, because they are also end-to-endobject detection frameworks without NMS, and Obj2Seq shares some similarity with Deformable
DETR [49].
All variations use multi-scale features or dilated convolution in stage 5 (DC5) for similarcomputation cost.
Meanwhile, we also present some representative convolution-based frameworks[32, 37] too, and they are trained with longer schedulers as reported in former papers.
Obj2Seq achieves 45.7% mAP with R50 as the backbone, without bells and whistles.
It is +1.2%higher than Deformable DETR using the same multi-scale deformable attention, and outperformsother detection frameworks as well, no matter DETR-like or convolution-based frameworks witha longer scheduler.
With R101 as the backbone, Obj2Seq also achieves 46.1%.
We notice that
DAB-DETR achieves significantly higher AP L than Obj2Seq.
This possibly attributes to its enhancedposition embeddings, since reference points with learnable wh may better fit objects with extremescales.
In general, these results indicates that Obj2Seq is an effective framework for object detection.
It achieves excellent performance.
4.2 Experiments on Multi-Label Classfication
Prompted Visual Indicator does not only generate object queries for Object Transformer Decoder, butalso classify if the input categories exist in the image, and output scores for them.
Therefore, we areable to perform multi-label classification as well with the outputs from Prompted Visual Indicator.
We evaluate the model trained in Section 4.1 on MS COCO multi-label benchmark.
The classifier in
Prompted Visual Indicator achieves 89.0 mAP, which is comparable to most multi-label classificationsolutions.
We will elaborate on its detailed behavior in Appendix.
[Table] [Table]
[Table] [Table]
4.3 Experiments on Human Pose Estimation
Current human pose estimation algorithms are generally divided into three categories, bottom-upmethods, top-down methods with image crop, and top-down methods in an end-to-end way.
Allof them involves sophisticate structures, such as image crop [44, 47, 11] or RoI Align [23, 19] fortop-down methods and group post-processing for bottom-up methods [30, 3].
There are few worksdirectly obtaining keypoints from the input image [36].
In Obj2Seq, General Sequence Predictorcan directly read the coordinates out from object queries without any task-specific model structures.
We also provide a baseline for better comparision.
The only difference between the baseline and
Obj2Seq is that the baseline model use a simple MLP instead of General Sequence Predictor.
Thedetails in hyperparameters and training configurations are elaborated in Appendix.
Our experiments are conducted on the COCO keypoint challange [25], and we mainly list otherend-to-end top-down methods in Table 2 for comparison.
Obj2Seq achieves 60.1% mAP with a50-epoch training scheduler.
It reaches a reasonable performance as an end-to-end framework with nodelicate structures.
The result with General Sequence Predictor is +2.9% higher than the MLP-headbaseline, which indicates that it is a more effective predictor.
This improvement may attribute to theinterdependence between different keypoints.
Knowing the exact position of some keypoints benefitsthe prediction of others.
We also attempt to train Obj2Seq with a longer training schedule.
Trainingfor 150 epochs can further improves mAP to 65.0%.
Obj2Seq currently falls behind of start-of-the-
[Table] [Table]
art methods (listed in Appendix)
[Table]
but has potential for further improvements.
Nevertheless, theseexperiments validate that Obj2Seq can be implemented in human pose estimation
[Table]
and we believe itis also available for other visual tasks.
4.4 Ablation Studies
In this section, we examine different parts and ablate on some important options in Obj2Seq.
Weconduct these experiments on object detection for demonstration.
4.4.1 Overall Structure
We mainly elaborate the existence of two modules in Obj2Seq, Prompted Visual Indicator and General
Sequence Predictor.
Removing them leading to a framework similar to Deformable DETR.
Therefore,we directly takes Deformable DETR as the baseline.
Table 3 shows that both modules provide steadyimprovements for Obj2Seq.
With the help of Prompted Visual Indicator, Obj2Seq is able to processobjects of different categories separately, and distinguish important class-aware features with givenprompts.
This makes Obj2Seq work more efficiently, leading to an improvement of +0.6%.
General
Sequence Predictor is initially designed for better generality.
However, the precision is also increasedby +0.6%, which actually surprises us.
We suppose that attributes of most object-level tasks are notindependent.
Some known attributes may help with the prediction of the others.
For example, itwould be more accuracy to estimate the scale of a box if its center is determined.
4.4.2 Objectness Branch
In Section 3.4, we have introduced the additional objectness branch.
It is constructed in order torecognize if the object exists, rather than to describe it.
Meanwhile, it is also a valid way if wemerge objectness into the output sequence of General Sequence Predictor, which means insertingan additional word representing the objectness score before the original sequence.
We validate thenecessity of the objectness branch in Table 3.
Predictor without a separate objectness branch achievesthe same performance as the original MLP.
However, a separate objectness branch further improvesmAP by +0.6%.
Both options are extensible for different tasks.
We explain this phenomenon in twoaspects.
On one hand, a single linear layer is not that good at judging the existence of objects ofdifferent categories.
It needs help from the class vectors.
On the other hand, checking the existenceof an object is different from describing it.
Their functions may conflict in the same head.
Therefore,it is better to build a separate objectness branch.
4.4.3 Ablation of Prompted Visual Indicator
Input prompt vector Obj2Seq receives class prompts as inputs to detect objects of certain categories, while how to generate class prompts remains a question.
The simplest way is learning a promptvector for each class from scratch, just like how object queries are learned in DETR [4].
As shownin Table 4, this straight-forward solution achieves 45.4%.
CLIP [31] is a robust feature extractorpretrained with a huge amount of image-text data pairs.
It can provide better representations for label
     
  1XPEHURIFDWRJRULHVUHWDLQHG       P $    
  
  * ) O RS V P$3ZVFRUHRIFDWHJRULHVP$3ZRVFRUHRIFDWHJRULHV&RPSXWDWLRQ&RVW (a)     7KUHVKROGWRUHWDLQFDWHJRULHV   
   P $    
  
  * ) O RS V P$3ZVFRUHRIFDWHJRULHVP$3ZRVFRUHRIFDWHJRULHV&RPSXWDWLRQ&RVW (
[Figure]
text of each category.
Initializing prompt vectors with CLIP features achieves +0.3% improvement.
However, fixing these prompt vectors leads to a slight decrease in precision.
Vectors generated by
CLIP may not be well aligned with image features.
It needs to be tuned in the training process.
Retention Policies As mentioned in Section 3.2, Prompted Visual Indicator is not only used forclassification, but also to initialize object queries for retained categories.
If one category is filteredout, it means that Obj2Seq will not detect any objects of this category.
It may be doubtful whether
Obj2Seq is sensitive to the retention policy.
Here we demonstrate how mAP changes when we usedifferent policies in evaluation.
As shown in Figure 3, we mainly test with two policies, retainingthe top- K ′ categories or any categories over a threshold.
Obj2Seq achieves steady results with bothpolicies (see the blue curves).
The evaluation metric fluctuates within a range of 0.2% when thenumber of remaining classes changes from 10 to 80, and mAP is also stable when the threshold islower than 0.3.
These results indicate that Obj2Seq is insensitive to the retention policy.
In Eq.
(3), the scores for class existence s ( k ) C is further multiplied in the objectness branch to obtainfinal scores for objects.
It does not improves the accuracy much if we use the same retention policyin training and evaluation, but makes Obj2Seq a more robust framework with different policiesin inference.
Figure 3 provides both curves with and without the participance of class existences ( k ) C .
Without s ( k ) C , the precision drops more significantly when we modify the retention policy inevaluation.
This phenomenon emerges because Object Transformer Decoder and General Sequence
Predictor receives inputs in different distributions when the policy changes.
Inputs of some categoriesare filtered out by Prompted Visual Indicator during training, therefore Obj2Seq cannot make accuratepredictions if they are seen in inference.
By multiplying with the class existence s ( k ) C , we are able tomake it up to some extent.
Conclusion
In this paper, we propose a controllable and versatile framework for visual tasks, Obj2Seq.
Thisframework takes class prompts to determine the objects it should attend to, and generates a sequenceof attributes to describe each object in the image.
Obj2Seq can be applied in different visual tasks,such as object detection, human pose estimation and image classification, and achieves comparableresults with other task-specific methods.
There are a few problems unsolved in Obj2Seq, which need further discussions.
Firstly, Obj2Seqmainly focuses on a unified definition of different visual tasks, but barely pays attention to the detailedstructure of the feature extractor.
Designing a new feature extractor for general usage may achievebetter performance.
Secondly, Obj2Seq can be implemented in a wide range of other tasks.
Finally,
Obj2Seq only takes class prompts to guide the model.
In order to satisfy finer requirements, a moredetailed indicator is worth developing.
All in all, Obj2Seq points a promising direction for generalvision models, and it may inspire further studies.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See conclusion.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our work
proposes a framework for visual tasks.
It does not involve social impacts.(d)
Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] The code is
included in the supplemental material.
We will also release it later.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix and the code.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Our code is included in the supplemental material
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]