OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regression
Abstract
This paper presents a language-powered paradigm for ordinal regression.
Existingmethods usually treat each rank as a category and employ a set of weights to learnthese concepts.
These methods are easy to overfit and usually attain unsatisfactoryperformance as the learned concepts are mainly derived from the training set.
Recent large pre-trained vision-language models like CLIP have shown impressiveperformance on various visual tasks.
In this paper, we propose to learn the rankconcepts from the rich semantic CLIP latent space.
Specifically, we reformulate thistask as an image-language matching problem with a contrastive objective, whichregards labels as text and obtains a language prototype from a text encoder for eachrank.
While prompt engineering for CLIP is extremely time-consuming, we propose
OrdinalCLIP, a differentiable prompting method for adapting CLIP for ordinalregression.
OrdinalCLIP consists of learnable context tokens and learnable rankembeddings; The learnable rank embeddings are constructed by explicitly modelingnumerical continuity, resulting in well-ordered, compact language prototypes in the
CLIP space.
Once learned, we can only save the language prototypes and discardthe huge language model, resulting in zero additional computational overheadcompared with the linear head counterpart.
Experimental results show that ourparadigm achieves competitive performance in general ordinal regression tasks, andgains improvements in few-shot and distribution shift settings for age estimation.
Introduction
Ordinal regression aims to predict a rank number [6] or continue value [21] y for a given sample x .
The core difference between ordinal regression and classification is that its targets are continuousand ordered numbers. As a fundamental machine learning problem, ordinal regression has attractedincreasing attention in the field of computer vision due to its broad applications [18, 27, 24, 15].
A simple solution for ordinal regression is to directly regress a scale value [35].
It is implementedwith a single neuron and optimized with L 1 or L 2 loss.
However, it suffers from limited performance.
Popular approaches [36, 40, 35] usually discretize the continuous target space into different binsor treat the rank labels as different numbers.
Then they regard these discrete values as differentcategories and perform the classification using a linear layer.
These methods are essentially learninga set of weights to model the concept of order .
As the learned concepts are mainly derived fromthe training set, these approaches are prone to overfit and usually attain unsatisfactory performance.
The human language contains rich semantic information and prior knowledge.
CLIP [33] learnsan image encoder and a text encoder with large-scale image-text pairs, which shows impressiveperformance on zero/few-shot knowledge transfer with robust and expressive latents.
In this paper,we propose a language-powered paradigm for ordinal regression, which learns a language prototypefor each rank from a well-defined language latent space .
We aim to associate each rank category language concept to alleviate the overfitting issue and obtain latents that obey the ordinality.
Specifically, we convert each rank label into text input (prompt) and employ the pre-trained giant textencoder in CLIP to extract language prototypes for all ranks.
Therefore, each rank label is mappedto a language prototype with a fixed language model.
Then we reformulate ordinal regression asan image-text matching framework, where image embeddings are obtained from a learnable imageencoder and a set of language prototypes are served as the text features to be matched.
We to the into the well via a contrastive the is a with can exploit additional
Recent research [48] has found that the text input (prompt) plays a key role in the performance andchoosing the right prompt is extremely time-consuming.
Instead of using hard prompts, we proposeto learn rank prompts for ordinal regression.
The proposed OrdinalCLIP consists of learnable contexttokens and learnable rank embeddings.
Learnable context tokens are shared across all ranks andused to model the context words.
As large language models usually do not fully learn the continuousproperty of the numbers [42], we also propose to learn rank embeddings by interpolating betweenseveral base features, which explicitly model the continuous nature of rank embeddings.
The key ideaof this paper is shown in Figure 1.
Although our paradigm leverages the language priors with giantlanguage models, we only utilize the learned language prototypes during inference, which does notbring any additional computational and storage overhead compared with the linear head alternative.
In other words, our method obtains language knowledge for free from the perspective of deployment.
We summarize the main contributions of this paper as follows: (1) We present a new paradigmthat learns language prototypes from a fixed language model for ordinal regression.
To the best ofour knowledge, our method is the first language-powered paradigm for ordinal regression.
(2) Weintroduce learnable rank embeddings, which explicitly model the continuous nature of ranks to betterexploit language priors from the text encoder in CLIP.
(3) Comprehensive experiments illustrate theeffectiveness of the paradigm.
OrdinalCLIP achieves competitive performance on three real-worldtasks, along with improvements in few-shot and distribution shift settings for age estimation.
Related Work
Ordinal Regression:
Given an input sample, ordinal regression aims to map the input to a rank or acontinuous value.
Although we can directly regress this value, it suffers from limited performance [35].
Rothe et al.
[35] posed this task as a deep classification problem.
Then they computed expectedvalues over the softmax-normalized probabilities for inference.
They also contributed a large-scaleage estimation dataset named IMDB-WIKI.
Geng et al.
[10] proposed label distribution, whichregards a sample as an instance associated with a specific label distribution.
The mean-variance losswas introduced in [28] to leverage the label ambiguity.
Chen et al.
[3] proposed ranking-CNN with aseries of basic CNNs to learn the rank.
There are some papers [11, 20, 12, 43, 14] modeling ordinalregression from the perspective of probability.
Gustafsson et al.
[11] proposed an energy-basedmodel within a probabilistic regression formulation.
Li et al.
[20] introduced probabilistic ordinalembeddings to model the data uncertainty for regression.
DeepGUM was proposed in [17], whichused a Gaussian-uniform mixture model to achieve robust performance for outliers.
Pan et al.
[29]proposed self-paced deep regression forests to achieve more robust and less biased models.
Althoughthese methods achieve good results on the general regression tasks, they concept from the data, which makes them easy to overfit and leads to sub-optimal performance.
Ourparadigm aims to learn the concept from the language to alleviate the above issues.
Language Model: Recent years have witnessed the great success of pre-training general languagerepresentations [4, 2, 34, 32, 5].
ELMo was proposed in [32] to extract context-sensitive features.
BERT [5] employed masked language models to learn deep bidirectional representations.
CLIP [33]proposed to learn representations from image-text pairs.
The experimental results show that the CLIPis much more efficient at zero/few-shot transfer, which demonstrates the strong power of language.
Encouraged by the excellent performance of CLIP, many CLIP-powered methods have been proposed.
Patashnik et al.
[30] proposed StyleCLIP, which leveraged the power of CLIP models to develop a textbased interface for StyleGAN.
Wang et al.
[44] introduced ActionCLIP for video action recognition.
ActionCLIP follows a paradigm “pre-train, prompt and finetune”, which achieves strong performanceon action recognition tasks.
Gao et al.
[9] proposed the CLIP-Adapter to use the CLIP model ondownstream tasks.
The CLIP-Adapter uses feature adapters for fine-tuning.
Zhang et al.
[47] furtherproposed Tip-Adapter to enhance the CLIP’s few-shot capability without training.
Wav2CLIP [46]was proposed to learn robust audio representations by distilling from CLIP models.
In this paper, weleverage the language priors from the CLIP model.
To better exploit the language priors from CLIP,we further propose differentiable rank prompts to model the continuous property of the numbers.
Themost relevant works are [48] and [49].
Zhou et al.
[48] presented a new approach named contextoptimization.
This method learns the context of prompts, which attains significant performanceimprovements.
Zhou [49] also proposed an improved version of [48] by incorporating the visualfeatures into the text features, which improves the generalizability of the model.
Note that [48] onlyoptimizes the context embeddings.
Thus directly applying it to ordinal regression leads to degradedperformance due to the lack of ordinal property.
We do not compare with [49], as we want to solelyinvestigate how language priors introduced as language prototypes could help ordinal regression tasks.
Proposed Approach
We first present the problem statement.
Then we elaborate on how to leverage prior knowledge froma language model with language prototypes.
Lastly, we detail the learnable rank prompts.
3.1 Problem Statement
The goal of ordinal regression is to estimate the rank number or continue value for a given sample.
Although we can directly regress this value with a single neuron, a popular baseline is to regard itas a classification task and employ additional constraints trying to make use of the ordinal property.
Mathematically, we denote the rank space as R =
{ r j | 0 ≤ j < C } , where r j denotes a rank categoryand C is the number of ranks categories.
During training, we randomly sample a batch of images
X = { x i | 0 ≤ i < B } and the associated labels Y =
{ y i | 0 ≤ i < B } , where B represents the batchsize and y i ∈ { 0 , 1 } C .
The classification baseline treats each rank as a class number.
Assuming therank label for a sample x i is r j , we construct the one-hot label by setting y i,l = 1 if l = j , otherwisey i,l = 0 , where l is an index and 0 ≤ l < C .
The images are first sent to an image encoder to extract image features f = { f 0 , f 1 , . . .
, f B − 1 } ,where f i ∈ R
d is the extracted features of image x i .
We use d to denote the feature dimension.
Thenthe image features are sent to a linear layer with a set of parameters W =
[ w 0 , w 1 , . .
. , w C − 1 ] ⊤ ∈ 118 R C × d and b = [ b 0 , b 1 , . . .
, b C − 1 ] ∈ R C :
[Equation]
(1) where l i,j is the predicted logit for rank r j of image x i .
Then the model is optimized with across-entropy loss:
[Equation]
Many existing methods develop additional technologies based on the classification formulation, suchas label distribution [10, 8, 7], mean-variance loss [28], and so on.
However, these methods simplytreat each rank as a number and learn the rank concept mainly from the training data leading to theoverfitting problem along with unsatisfactory performance.
In this paper, we propose a languagepowered paradigm, which learns the rank concept from language to address the above issues.
Prompt Context
Text Encoder
Rank Prompt 𝑹
C ...
[𝑬] 𝟐 [𝑬][𝑬] 𝒎
[𝑬]𝐁𝐚𝐬𝐞𝐑𝐚𝐧𝐤𝐄𝐦𝐛𝐞𝐝𝑹′ 𝒓 0 𝒓 1 𝒓𝒓 𝒋
Mapping
Ordinal Property
Language Latent Space
Image Feature Language Prototype
ImageEncoder
Image
Ordinal Property 𝒓′ C’- 𝒓′
𝒓′ 𝒌 𝒓 C-
3.2 Language Prototypes
We believe that natural language contains rich prior knowledge.
To preserve ordinality, we propose tolink the learned rank category to its language concept.
To leverage the language power, we employthe text encoder of CLIP [33] as the language model.
CLIP follows a vision-language pre-trainingframework and learns representations from image-text pairs, resulting in a joint vision-languagelatent space.
As the CLIP model shows strong performance in downstream tasks, we implement ourmethod with the CLIP-powered text encoder.
We reformulate the ordinal regression task as an image-language matching problem, which consistsof a text encoder and an image encoder.
The text encoder is used to extract language features with thefixed CLIP language model.
As the parameters of the text encoder are fixed, the text is mapped to awell-learned language latent space, which encodes rich language priors.
To leverage the language priors with the text encoder, we treat the rank categories as words.
Thenwe obtain a set of words for the rank categories { [ r 0 ] , [ r 1 ] , . . .
, [ r C − 1 ] } .
To obtain the languageembedding for each rank, we construct a sentence for these ranks and send the entire sentence to thetext encoder.
For example, we can construct a sentence of “a person at the age of [ r j ] ” for the rank[ r j ] .
Then each word of this sentence is mapped to a 512-D word embedding vector.
All the wordembeddings are sent to the text encoder to obtain a text embedding.
In this way, each rank category ismapped to a text embedding in the language latent space.
We treat the mapped embedding as theprototype of its corresponding rank category.
As these prototypes lie on the well-learned languagemanifold and encode the language concept, we name them as language prototypes .
Formally, we use{ p j ∈ R d | 0 ≤ j < C − 1 } to represent the ℓ 2 normalized language prototypes, where p i correspondsto the language prototype of the rank r j and d is the feature dimension.
The input images are processed by an image encoder to extract image features.
We can employ anypopular vision backbones as the image encoder.
While the parameters of the text encoder are fixed,we train the entire image encoder to align image features to the language latent space.
For a batch ofimages X , we denote the ℓ 2 normalized image features as I =
{ I i ∈ R d | 0 ≤ i < B − 1 } , where d isthe feature dimension.
As an image-language matching framework, we calculate the similarity scoresbetween the two modalities through the inner product and obtain a similarity matrix A =
[ a i,j ] :
[Equation]
We use image-text contrastive loss to optimize the network, which consists of an image-to-textloss and a text-to-image loss following the practice of CLIP [33].
For image-to-text loss, we firstnormalize the rows of the matrix A using a softmax layer to obtain the matrix A ′
=
[ a ′ i,j ] :
[Equation]
where T is the temperature parameter.
We minimize the discrepancy between the matrix A ′ and label
Y using KL divergence.
For text-to-image loss, there may be zero or multiple hits.
Therefore weconstruct the label matrix Y ′′ by normalizing the non-zero columns of Y [44].
Besides, we obtain acolumn-normalized matrix A ′′ =
[ a ′′ i,j ] :
[Equation]
We also employ a KL divergence to minimize the discrepancy between the matrix A ′′ and the labelmatrix Y ′′ .
In the end, we have the image-text contrastive loss:
[Equation]
3.3 Learning Rank Prompts
As illustrated in [33, 48], different text input, which is known as prompt, may give very differentperformances.
Instead of designing the prompt context, we directly learn m word embeddings
[ E ] 0 , [ E ] 1 , . . .
, [ E ] m − 1 to serve as the prompt context, which is concatenated with the word embeddings of ranks and is sent to the text encoder.
To further strengthen the ordinality of our model and improve its performance, we propose to learnthe rank embeddings to preserve the order of the language prototypes in the language latent space.
The reason why we cannot directly apply CLIP to regression is that the human-built prompts fromthe rank labels lack ordinality (see Figure 3), which leads to degraded performance.
Inspired by thearithmetic property in the word embeddings space [25, 31, 1], whose typical example is the followingembedding arithmetic equation: “King − Man + Woman = Queen”, we choose to maintain the orderof rank embeddings to preserve the order of the language prototypes.
By explicitly constructingcontinuous rank embeddings, we can also improve the ordinality of the language prototypes (also see
Figure 3).
We regard this ordinality as the natural outcome of continuity.
We derive the ordinal rank embeddings from a set of base rank embeddings via interpolation.
Supposewe have a set of base rank embeddings B ′ =
[ r ′ 0 , r ′ 1 , . . . , r ′ C ′ − 1 ] , where C ′ denotes the number ofthe embeddings, and R ′ ∈ R 512 × C ′ .
Note that in practice, C ′ should be a relatively small numbercompared with C ( i.e. , C ′ << C ).
Then the j-th rank embedding could be obtained by:
[Equation]
where 0 ≤ k < C ′ , and W ′ =
[ w ′ j,k ] ∈ R C × C ′ is the interpolation weights:
[Equation]
where I ( · ) is the interpolation function, and | · | outputs absolute value.
We simply propose twointerpolation approaches, which are linear interpolation : I ( w j,k ) = 1 − w j,k C − 1 and inverse-proportioninterpolation : I ( w j,k ) =
1 w j,k + ϵ .
We set ϵ to 1e-5 for numerical stability.
Figure 2 presents thedetails of the rank prompt of OrdinalCLIP.
Although our method uses a huge language model to exploit the language concepts for training.
Wecan only employ the learned language prototypes during testing.
In this way, our method does notbring any additional computational and storage overhead compared with the linear head alternative.
Experiments
4.1 Age
Estimation
Datasets: The task of facial age estimation is to predict the age of a given facial image.
We test ourmethod on the widely-used MORPH II [13] dataset and Adience [19] dataset.
Morph II contains
[Table] [Table]
[Table]
55,134 portraits from 13,618 individuals.
Each portrait image is labeled with an age value from 16 to77.
Popular evaluation protocol [35, 39, 21, 20] is adopted in our experiments, which only contains5,492 images of Caucasian descent to remove the cross-race interference.
For general regression,we select 80% of the images for training, and others for testing.
Note the long-tailed distribution inthe dataset.
Besides, the distribution of the test set constructed by random sampling is consistentwith that of the training data.
Adience contains 26,580 Flickr photos from 2,284 subjects.
Thelabel is annotated with eight groups, which are 0-2, 4-6, 8-13, 15-20, 25-32, 38-43, 48-53, and over60-year-old.
We adopt the same five-fold cross-validation protocol as used in [22].
We adoptedmean average error (MAE) to measure the absolute differences between the ground truth labels andthe predicted ones.
In addition, classification accuracy is adopted for Adience.
For the detailedexperimental settings please refer to Appendix.
Comparison with State-of-the-art methods: We report the results in Table 1.
The state-of-the-artmethods are usually designed especially for age estimation [45, 41].
DRFs [39] learned a Gaussianmixture model by extending the deep neural decision forests [16] for regression.
AVDL [45] leverageda meta-learning framework to learn an adaptive variance for label distribution.
POE [20] proposeda general regression framework with uncertainty modeling and achieved superior performance.
Compared with the advanced approaches, zero-shot CLIP only obtains an MAE of 14.45, implyingthe pool modeling of numerical and ordinal concepts.
CoOp scores an MAE of 2.39, illustrating theeffectiveness of the language priors.
OrdinalCLIP achieves a superior MAE of 2.32 without bells andwhistles, resulting from the explicit ordinal constraint.
We found similar results in the Adience [22]dataset.
Specifically, CoOp attains 60.5% for regression accuracy, which is better than the previousbest.
Our OrdinalCLIP outperforms previous state-of-the-arts with an accuracy of 61.2% and is onpar with the previous best method under the MAE metric of 0.67.
Parameter Analysis: We first analyze the influence of different numbers of base rank embeddingsand different interpolation types.
With all parameter combinations, the models can all achievecomparable MAEs on MORPH II(Table 3).
When varying the number of base rank embeddings, theresults improve slightly until a certain sweet point.
The best MAE of 2.32 is reported with 3 baseranks.
For inverse-proportion one, the best MAE of 2.35 is attained with 7 base ranks.
However,its overall performance is sub-optimal compared with the linear interpolation one.
In the followingexperiments, we chose linear interpolation and 3 base ranks as the default hyperparameters.
Ordinality of Learned Language Prototypes: We validate the ordinality of the rank embeddings bymeasuring the similarity of their corresponding language prototypes.
OrdinalCLIP makes predictionsby measuring the similarity between the normalized image embedding and each normalized languageprototype in the CLIP latent space.
Ideally, the structure of the normalized language prototypesshould preserve the ordinality - “lie on a line” in the CLIP latent space.
This is aligned with humanperception and judgment while facing ordinal regression tasks.
We compute the similarity matricesfollowed by Equation 4 with both inputs being language prototypes.
The derived matrices are then
[Figure]
[Table]
max-normalized by their maximum similarity.
The ordinality score is the percentage of languageprototypes that obey the ordinal property {+
+}
Figure 3 visualizes the ordinality under various situations.
The CNN baseline (Figure 3(a)) has theleast ordinality score of 49.13%; Half of the language prototypes violate the ordinal property andall the prototypes are distributed quite sparsely across the latent space.
Figure 3(b) and Figure 3(c)are results computed directly from the CLIP model.
Though the initialized context embeddings helpto improve ordinality (from 49.71% to 54.84%), almost half of the language prototypes violate theordinal properties.
[33] found that CLIP models give degraded predictions facing abstract conceptslike numbers.
We attribute this to the loss of ordinality in the CLIP latent.
Figure 3(d) shows thattrained context embeddings from CoOp could further improve the ordinal property with an ordinalityscore of 59.92%.
However, the learned language prototypes are still “twisted” in the CLIP space,leading to degraded predictions.
Our OrdinalCLIP (Figure 3(e)) learns smoother language prototypeswith a boosted ordinality score of 65.94%, which validates the effectiveness of the learned baserank and their interpolation.
Note that the similarity matrices reflect the long-tail property of thedataset.
Due to the insufficient samples for the older people, there is a “red blob” in the right-downcorner.
CLIP and CoOp cannot push the “older” labels away and leave them closer to the well-learned“younger” labels, which may lead to corrupted predictions.
Our OrdinalCLIP reinforces the ordinalityand as a result, outperforms the above two methods.
Ablation Study: To validate the effectiveness of OrdinalCLIP, we compare OrdinalCLIP withpreceded CoOp [48] using different optimization and prompt context initialization strategies.
Thereare four major observations in Table 4.
First, for both CoOP and OrdinalCLIP, solely optimizing either rank embeddings or context embeddings leads to sub-optimal performance, yet training them together results in optimal results.
Notethat CoOp only optimizes the context embeddings while leaving the rank embeddings untouched asthey are initialized with the rank labels.
However, after jointly training with the rank embeddings, theperformance has been boosted from 2.44 to 2.39.
For OrdinalCLIP, that is from 2.37 to 2.36 evenwith random base rank embeddings, resulting from the explicit ordinality from our method.
Second, the initialized rank embeddings are prone to be degraded, which is another observationvalidating that the CLIP latent space struggles to represent abstract concepts like numbers [33].
Whenonly optimizing rank embeddings, CoOp obtains an MAE of 2.41 with initialized context prompts.
Incontrast, our OrdinalCLIP could achieve a better MAE of 2.38 even with all the context and base rankembeddings initialized randomly.
We attribute this to the explicit ordinal modeling – the interpolationbetween the base ranks.
[Table] [Table]
Third, initializing prompts with task-specific context could boost performance, no matter whether itis CoOp or OrdinalCLIP, or which kind of embeddings are optimized.
For example, our OrdinalCLIPcould achieve the best MAE of 2.32 when training both embeddings and initializing the context.
Last, OrdinalCLIP outperforms CoOp under all settings in the ablation, which further validates theeffectiveness of the proposed learned base rank embeddings and their interpolation.
Few-Shot Learning: We validate the generalization ability of OrdinalCLIP for the few-shot regression learning.
The full dataset is split into 80% for training and 20% for testing.
We use the entiretest set for validation and only choose 1/2/4/8/16/32/64 samples in the training set from each class ofthe labels for training.
We report the results in Table 5.
We observe the language models consistentlyoutperform the baseline by a large margin.
By incorporating the language prior, the MAE is improvedby 2.95 compared with baseline under 1 shot setting for CoOp; Our OrdinalCLIP further improvesthe MAE to 4.94.
For other shot settings, we also obverse the same performance gains, which showsthe effectiveness of the proposed learned rank embeddings.
Distribution Shift: Another generalization experiment about data distribution shifts is conducted onthe MORPH II database.
We also use the entire test set of the general regression setting.
For the training set, we randomly choose several rank labels, and then randomly discard a large portion of samplesin those classes.
Table 6 shows the comparison results.
We observe that by incorporating languageprior, CoOp improves performance by a large margin compared with the baseline.
While OrdinalCLIPboosts performance furthermore, illustrating our OrdinalCLIP can better resist the distribution shiftof dataset by the explicit ordinal constraint.
Specifically, with 40 rank labels selected and 90% oftheir samples discarded (most severe distribution shift setting), CoOp obtains a 0.35 improvementof MAE compared with baseline, while OrdinalCLIP further attains a performance gain of 0.18.
4.2 Image Aesthetics Assessment
Dataset.
The image aesthetic dataset [37] contains 13,929 available Flickr photos of four categories,i.e, nature, animal, urban, and people.
Five absolute rating scales are employed to evaluate theaesthetics quality: “unacceptable”, “flawed”, “ordinary”, “professional”, and “exceptional”.
Eachimage is judged by at least five different examiners and uses the median rank as ground truth.
Thewhole dataset is randomly split into three non-overlapped subsets following [37].
Five-fold crossvalidation was used for fair comparisons.
We reported mean and standard deviation for both MAEand classification accuracy.
We detail the experimental settings in Appendix.
Results: Table 7 shows the comparison with the previous SOTA methods.
Language priors assist themodel to achieve better performance, while zero-shot CLIP still struggles to distinguish ordinal concepts.
Our OrdinalCLIP attains an overall MAE of 0.280 and overall accuracy of 72.58%, outperforming the previous best method.
The results across all nominal categories either outperform or are on parwith previous state-of-the-art.
Consistent improvements are observed across all categories comparedwith baseline and CoOp, showing the effectiveness of learned rank prompts with the ordinal property.
[Table]
4.3 Historical Image Dating
[Table]
[Table]
The historical image datingdataset [26] is a benchmark for automaticallypredicting the decade of the historical coloredimage.
There are five-decade categories in thedatabase from the 1930s to the 1970s, whereeach category contains 265 images.
For the general regression setting, the data of each categoryis divided into three parts: 210 for training, 5for validation, and the rest for testing.
Ten-foldcross-validation is adopted following [22, 26].
We report the mean and standard deviationfor both MAE and accuracy metrics.
Refer to
Appendix for the detailed experimental settings.
Results: We compare our OrdinalCLIP withother state-of-the-art models using the full dataset.
Table 8 demonstrates that CoOp improves theaverage accuracy by 9.34% compared with the baseline method, which again shows the importanceof exploiting language priors.
Zero-shot CLIP is still suffering from degraded results due to poolordinal transferability.
Our OrdinalCLIP further improves the accuracy by 4.45%.
Compared withthe previous best, OrdinalCLIP attains a competitive MAE of 0.67 and achieves a new state-of-the-artaccuracy of 56.44% with only half of the standard deviation, which validates the effectiveness of theproposed learned rank embeddings.
Discussions and Conclusions
OrdinalCLIP is a general approach for ordinal regression.
It can be deployed on many vision taskssuch as age estimation.
These tasks may be used for surveillance without user permission, whichcould lead to privacy issues.
Therefore, strict technical controls may be needed.
Since our methoddirectly employs existing language models, the bias in the language models will be also inherited byour method.
How to alleviate the bias in the language model still needs further research.
In this paper, we have presented the language-powered paradigm for ordinal regression.
Existingordinal regression methods usually suffer from the overfitting problem and lack of ordinality withinthe latents, as they mainly learn the rank concepts from the training data.
We propose OrdinalCLIP,which associates each rank category with its language concept derived from the CLIP text encoder.
Toleverage the language priors, each rank is mapped to a corresponding language prototype.
We furtherpropose learnable rank prompts to explicit learning ordinal rank embeddings to preserve the order of
the language prototypes in the language latent space.
Extensive experimental results from three tasks including age estimation, historical image dating and image aesthetics assessment show that
our
OrdinalCLIP attains very competitive performance in general ordinal regression tasks.
OrdinalCLIP also gains improvements in few-shot and distribution shift settings for age estimation
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]