Original: More Than Eleven Thousand Words: Towards Using Language Models for Robotic Sorting of Unseen Objects into Arbitrary CategoriesAnonymous Author(s)
Revised: More Than Eleven Thousand Words: Towards Using Language Models for Robotic Sorting of Unseen Objects into Arbitrary Categories

Original: AffiliationAddress email

Original: Abstract:
Revised: Abstract:

Original: We consider the task of automatically sorting previously unseen objects into arbitrary categories.
Revised: We consider the task of automatically sorting previously unseen objects into arbitrary categories.

Original: We aim to sort into general, high-level categories in contrastto traditional methods that sort on visually discernible features or by other sensormeasurements.
Revised: We aim to sort into general, high-level categories in contrast to traditional methods that sort on visually discernible features or by other sensor measurements.

Original: This paper explores a method where we divide the categorizationinto two sub-tasks: object detection and categorization.
Revised: This paper explores a method where we divide the categorization into two sub-tasks: object detection and categorization.

Original: In a set of experiments,it is shown that splitting the categorization task into a two-stage process removeshighly important information for robust categorization and performs far worsethan using an open vocabulary object detector.
Revised: In a set of experiments, it is shown that splitting the categorization task into a two-stage process removes highly important information for robust categorization and performs less robustly than an open vocabulary object detector.

Original: We hope these results are helpfulfor exploring the limits of Language Models for robotic tasks.
Revised: We hope these results are helpful for exploring the limits of Language Models for robotic tasks.

Original:  Introduction
Revised:  Introduction

Original: We recognize the ubiquity of sorting tasks.
Revised: We recognize the ubiquity of sorting tasks.

Original: From industrial settings, second-hand stores, and household services,the act of sorting objects provides value.
Revised: From industrial settings, second-hand stores, and household services, sorting objects provides value.

Original: So far, however, many sorting tasks havebeen restricted to human execution.
Revised: So far, however, many sorting tasks have been restricted to human execution.

Original: Considering this, we consider the question: Can we devise arobotic system that can sort objects efficiently, in a highly general manner?
Revised: Considering this, we consider the question: Can we devise a robotic system that can sort objects efficiently and in a highly general manner?

Original: A usual approach to sorting systems is training on a fixed set of categories, such as color, material,or other visually discernible or measurable features  with sensors or cameras.
Revised: A usual approach to sorting systems is training on a fixed set of categories, such as color, material, or other features measurable with sensors or cameras.

Original: Compared to humans,these methods suffer from two significant distinctions that hinder their flexibility: (i) they need to beretrained when presented with new categories, and (ii) the categories must be discernible from sensorinput.
Revised: Compared to humans, these methods suffer from two significant distinctions that hinder their flexibility: (i) they need to be retrained when presented with new categories, and (ii) the categories must be discernible from sensor input.

Original: These characteristics limit the use of robotics in cases where categories might change oftenand are provided as high-level expressions.
Revised: These characteristics limit the use of robotics in cases where categories might change often and are provided as high-level expressions.

Original: Consider a service robot in a specific household wherethe pots and pans are stored in one drawer and children’s toys in another.
Revised: Consider a service robot in a specific household where the pots and pans are stored in one drawer and children’s toys in another.

Original: Reprogramming the sorting system for each house is limiting.
Revised: Reprogramming the sorting system to recognize these two specific categories is limiting.

Original: Another application is second-hand stores, such as Goodwill,which might have predefined categories they sort after, determined by factors such as target groups,seasons, or campaigns.
Revised: Another application is second-hand stores, such as Goodwill, which might have specific categories they sort after, determined by factors such as target groups, seasons, or campaigns.

Original: These categories might change rapidly, and to compete with humans, thesorting systems should allow rapid switching between categories.
Revised: These categories might change rapidly, and to compete with humans, the sorting systems should allow rapid switching between categories.

Original: From an environmental perspec- tive, improving the throughput of the reuse industry will positively impact the circular economy and lower the demand for production
Revised: From an environmental perspective, improving the throughput of the reuse industry will positively impact the circular economy and lower the demand for production.

Revised: An apparent challenge is that the categories are previously unseen, separating this from traditional classification in computer vision.

Revised: We must capture the relation between an arbitrary object and an arbitrary category.

Revised: A challenging aspect of this task is the wide range of objects and categories we can encounter.

Revised: For many categories, modeling the relationship between the sensor observation and category membership relies on a complex semantic understanding.

Revised: Consider the class of children’s toys, a category containing objects of all shapes, weights, colors, and sizes.

Original: Several works have recently shown that combining Language Models (LMs) with robotics systemsallows for understanding natural language and complex reasoning for long-horizon planning.
Revised: Several works have recently shown that combining Language Models (LMs) with robotics systems allows for understanding natural language and complex reasoning for long-horizon planning.

Original: An emerging challenge is ensuring that the proposed actions are executable in practice.
Revised: An emerging challenge is ensuring that the proposed actions are executable in practice.

Original: Given theproblem of flexible and high-level sorting and the advancements of LMs for robotics, we want toanswer the pertinent question: Can we incorporate the semantic understanding of LMs into a roboticsystem to achieve efficient yet flexible sorting behavior?
Revised: Given the problem of flexible and high-level sorting and the advancements of LMs for robotics, we want to answer the pertinent question: Can we incorporate the semantic understanding of LMs into a robotic system to achieve efficient yet flexible sorting behavior?

Original: The sorting task can be separated into the following tasks: (i) Detect a single object in the sceneand determine its category and subsequently (ii) determine pick-and-place positions, and lastly (iii)execute the pick-and-place action with the robot.
Revised: The sorting task can be separated into the following tasks: (i) Detect a single object in the scene and determine its category and subsequently (ii) determine pick-and-place positions, and lastly (iii) execute the pick-and-place action with the robot.

Original: We recognize that detection and categorizationis the primary limiting factor for allowing a larger set of classes and objects.
Revised: We recognize that detection and categorization is the primary limiting factor for allowing a more extensive set of classes and objects.

Original: Responding to thischallenge, we explore a novel approach for categorization by separating the task into two steps.
Revised: Responding to this challenge, we explore an approach for categorization by separating the task into two steps.

Original: First,perform classification, namely assigning the membership to a specific class, and categorization,assigning this class to a higher-level category.
Revised: First, perform classification, namely assigning the membership to a specific class, and categorization, assigning this class to a higher-level category.

Original: Figure 2 shows a schematic view of this approach.
Revised: Figure 2 shows a schematic view of this approach.

Original: We conduct experiments on images to measure the  performance of the methods.
Revised: We conduct experiments on images to measure the method’s performance compared to directly using open-vocabulary classifiers. We present and discuss failure modes and promising tactics to improve the method.

Original: The results showthat it is beneficial to predict the category directly from the image, pointing to the fact that visualappearance provides essential cues for determining the category of the object.
Revised: In general, the results show that it is beneficial to predict the category directly from the image, pointing to the fact that visual appearance provides essential cues for determining the category of the object.

Original: Section 2 will provide an overview of the related work, Section 3 will describe the approach of ourmethod, Section 4 will present experimental results, and Section 5 will conclude and point to furtherresearch.
Revised: Section 2 will provide an overview of the related work, Section 3 will describe the approach of our method, Section 4 will present experimental results, and Section 5 will conclude and point to further research.

Original:  Related work
Revised:  Related work

Original: Robotic sorting Several systems perform sorting of objects into predefined categories; however, tothe best of our knowledge, no previous work has considered the problem of open vocabulary roboticobject sorting.
Revised: Robotic sorting Several systems perform sorting of objects into predefined categories; however, to the best of our knowledge, no previous work has considered the problem of open vocabulary robotic object sorting.

Original: Several works have investigated the sorting of objects based on specific propertiessuch as color, shape, and material.
Revised: Several works have investigated the sorting of objects based on specific properties such as color, shape, and material.

Original: [3, 4, 5] An application for these methods is waste sorting, where
Revised: [3, 4, 5] An application for these methods is waste sorting, where

Original: Lukka et al.
Revised: Lukka et al.

Original: [6] performs sorting by material properties.
Revised: [6] performs sorting by material properties.

Original: Similarly, Kujala et al.
Revised: Similarly, Kujala et al.

Original: [7] sorts objects bycolor off a conveyor belt.
Revised: [7] sorts objects by color off a conveyor belt.

Original: Automatic sorting Gu ´ erin et al.
Revised: Automatic sorting Gu ´ erin et al.

Original: [8] explore the problem of sorting objects without specified categories.
Revised: [8] explore the problem of sorting objects without specified categories.

Original: They use a Convolutional Neural Network in combination with a clustering algorithm togroup objects into a given set of bins.
Revised: They use a convolutional neural network in combination with a clustering algorithm to group objects into a given set of bins.

Original: Their method is, however, restricted to sorting objects ofsimilar appearance.
Revised: Their method is, however, restricted to sorting objects of similar appearance.

Original: The sorting system is also unaware of the semantic meaning of the differentclasses, and the system does not allow us to provide categories.
Revised: The sorting system is also unaware of the semantic meaning of the different classes, and the system does not allow the operator to specify categories.

Original: Language models for robotics Zeng et al.
Revised: Language models for robotics Zeng et al.

Original: [2] show that combining LMs with Visual Language
Revised: [2] show that combining LMs with Visual Language

Original: Models (VLMs), allowing them to communicate through language, results in an overall system witha high understanding of the scene.
Revised: Models (VLMs), allowing them to communicate through language, results in an overall system with a high understanding of the scene.

Original: Specifically, they use VLMs to inform the system what objects arein the scene, and a prompt format guides the LM to output code-like responses, e.g., pick-and-placeactions.
Revised: Specifically, they use VLMs to inform the system what objects are in the scene, and a prompt format guides the LM to output code-like responses, e.g., pick-and-place actions.

Original: Ahn et al.
Revised: Ahn et al.

Original: [1] compares the predictions from the LM to affordance functions, predicting themost relevant action given the robot’s surroundings and the given task description.
Revised: [1] compares the predictions from the LM to affordance functions, predicting the most relevant action given the robot’s surroundings and the given task description.

Original: Object retrieval Nguyen et al.
Revised: Object retrieval Nguyen et al.

Original: [9] focus on the problem of retrieving the most relevant object givena command containing a verb, imbuing the robot with semantic understanding.
Revised: [9] focus on the problem of retrieving the most relevant object given a command containing a verb, imbuing the robot with semantic understanding.

Original: Their approach islimited to retrieval given an action that can be done with the object, whereas our approach considersa free space of categories.Proposed method
Revised: Their approach is limited to retrieval given an action that can be done with the object, whereas our approach considers a free space of categories.

Original: We propose SortingBot, which consists of several parts.

Original: We divide the challenge of sorting theobjects into (i) object detection and classification and (ii) grasp synthesis and pick-and-place operations execution.

Original: This paper focuses on the former task and investigates how LMs could be used forthis purpose.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We will discuss two approaches to solving object categorization:
Revised: We will discuss two approaches to performing object categorization:

Original: Categorizing directly using an open vocabulary object detector.
Revised: Categorizing directly using an open vocabulary object detector.

Original: First, detect objects using a large vocabulary and then categorize them using a languagemodel.
Revised: First, detect objects using a large vocabulary and then categorize them using a language model.

Original: These will be described in detail in Section 3.2 and Section 3.1.
Revised: These will be described in detail in Section 3.1 and Section 3.2.

Original: 3.1 Categorization directly
Revised: 3.1 Categorization directly

Original: Here, we aim to categorize the objects directly, meaning we find the associated category withoutclassifying the objects first.
Revised: As shown in Figure 1, we aim to categorize the objects directly, meaning we find the associated category without classifying the objects first.

Original: This aligns with the task of an open-language object detector, which candetect objects using an arbitrary vocabulary.
Revised: This aligns with the task of an open-language object detector, which can detect objects using an arbitrary vocabulary.

Original: We hypothesize that this method will work well for categories within its training distribution.
Revised: We hypothesize that this method will work well for categories within its training distribution.

Original: Thisis not restricted to purely classes that the detector is trained on, but also includes  the  level of abstraction.
Revised: This includes categories on the same level of abstraction.

Original: Perhaps classes such as “cat,” “poodle,” and “bicycle” are more straightforward to recognizethan “travel-related,” “furniture,” and “kids toys”?
Revised: Perhaps classes such as “cat,” “poodle,” and “bicycle” are more straightforward to recognize than “travel-related,” “furniture,” and “kids toys”?

Original: 3.2 Separating detection and categorization
Revised: 3.2 Separating detection and categorization

Original: An overview of this approach is shown in Figure 2.
Revised: An overview of this approach is shown in Figure 2.

Original: In the spirit of Ahn et al.
Revised: In the spirit of Ahn et al.

Original: [1], we incorporate an
Revised: [1], we incorporate an

Original: LM in the task of assigning the object to a category.
Revised: LM in the task of assigning the object to a category.

Original: We achieve this by splitting the task into classification and categorization.
Revised: We achieve this by splitting the task into classification and categorization.

Original: In short, we use an object classification model to obtain P ( class | image ) ,where “class” is an class in the vocabulary of the classification model (e.g. “apple”).
Revised: In short, we use an object classification model to obtain P ( class | image ) , where “class” is an class in the vocabulary of the classification model (e.g. “apple”).

Original: Subsequently,we query an LM to obtain P ( category | class ) , where “category” is one of the given categories (e.g.“food”).
Revised: Subsequently, we query an LM to obtain P ( category | class ) , where “category” is one of the given categories (e.g. “food”).

Original: We can express the original task of estimating the category C from an image I as finding:
Revised: We can express the original task of estimating the category C from an image I as finding:

Original: [Equation]
Revised: [Equation]

Original: Incorporating an LM, we reformulate the categorization problem to depend  on the object’s class.
Revised: Incorporating an LM, we reformulate the categorization problem to depend purely on the object’s class.

Original: Then, we want to assign two attributes to the object in the image I , namely the class of the object O and its corresponding category C .
Revised: This implies assigning two attributes to the object in the image I , namely the class of the object O and its corresponding category C .

Original: We are then interested in maximizing:
Revised: We are then interested in maximizing the following:

Original: [Equation]
Revised: [Equation]

Original: We write the above expression as the two conditional probabilities:
Revised: We write the above expression as the two conditional probabilities:

Original: [Equation]
Revised: [Equation]

Original: We now make two assumptions to allow the separation of concerns.
Revised: We now make two assumptions to allow the separation of concerns.

Original: First, we assume that
Revised: First, we assume that

Original: P ( O ∗ | I ) = 1 , meaning that we have a perfect object classifier predicting only one relevant class.
Revised: P ( O ∗ | I ) = 1 , meaning that we have a perfect object classifier predicting only one relevant class.

Original: The maximization is then only over the categories.
Revised: The maximization is then only over the categories.

Original: Secondly, we assume that information aboutthe object’s class is sufficient to estimate the category, meaning that the category C is conditionallyindependent of the image I .
Revised: Secondly, we assume that information about the object’s class is sufficient to estimate the category, meaning that the category C is conditionally independent of the image I .

Original: We simplify the expression to:
Revised: We simplify the expression to:

Original: [Equation]
Revised: [Equation]

Original: With classification, we mean detecting the objects in the scene and classifying them into a set ofclasses.
Revised: Here, classification means detecting the objects in the scene and classifying them into a set of classes.

Original: The classes can be fixed, and the classifier can be trained specifically for them.
Revised: The classes can be fixed, and the classifier can be trained specifically for them.

Original: Therefore,this task can be performed by an object detector providing instance masks.
Revised: Therefore, this task can be performed by an object detector for separate objects or an image classifier for the whole image. The instance masks from the former help predict grasps downstream.

Original: For categorization, wecan use an LM to get the probability that an object class is assigned to a given class by carefulprompt engineering.
Revised: For categorization, we can use an LM to get the probability that an object class is assigned to a given class by careful prompt engineering.

Original: The details are provided in Section 4.
Revised: The details are provided in Section 4.

Original: We hypothesize that this two-stage approach is beneficial when it is challenging to determine thecategory given the detected object class.
Revised: We hypothesize that this two-stage approach is beneficial when it is challenging to determine the category given the detected object class.

Original: This might be the case when reasoning is required todetermine the category.
Revised: This might be the case when reasoning is required to determine the category.

Original: For example, it might be easier for an LM to capture that and “apple”should be in the “food” category than it is for an object detector to detect something as “food” in theimage directly.
Revised: For example, it might be easier for an LM to capture that and “apple” should be in the “food” category than it is for an object detector to detect something as “food” in the image directly.

Original:  Experiments
Revised:  Experiments

Original: Here, we present experimental results from each of the presented methods in Section 3 for categorizing objects.
Revised: Here, we present experimental results from each of the presented methods in Section 3 for categorizing objects.

Original: We benchmark the methods on a task of classifying images of singular objects into aset of given categories.
Revised: We benchmark the methods for classifying images of singular objects into a set of given categories.

Original: For the direct categorization approach, we use an open-language object detector.
Revised: We use an open-vocabulary object detector and an open-vocabulary image classifier for the direct categorization approach.

Original: Specifically, wechoose ViLD [10], representing state of the art on several datasets.
Revised: Specifically, we choose ViLD [10] and CLIP [11], both representing state of the art on several datasets.

Original: Pre-trained models are publiclyavailable.
Revised: Pre-trained models are publicly available for both algorithms.

Original: We choose ViLD as the object detector with  a fixed vocabulary and GPT-3  as the LM [11] forseparating the classification and categorization.
Revised: For the two-step approach, we conduct experiments with ViLD and CLIP using a fixed vocabulary for object detection. GPT-3 [12] is used as the LM predicting the category given the detected class.

Original: As we want the object detector to be as specificas possible, we use the class labels from the Tencent ML Image dataset [12] as this vocabulary.
Revised: As we want the object detector to be as specific as possible, we use the class labels from the TencentML Images Database [13] as this vocabulary.

Original: It combines the categories from both ImageNet [13] and Open Images [14], resulting in 11 166categories.
Revised: It combines the categories from both ImageNet11k [14] and Open Images [15], resulting in 11 166 categories.

Original: We do not  do any fine-tuning of the object detector on the Tencent ML dataset and notethat this might help improve this approach.
Revised: We do not evaluate a fixed vocabulary object detector, nor do any fine-tuning of the object detector on the Tencent-ML dataset. This might help improve this approach.

Original: We use GPT-3 with the following prompt:prompt
Revised: We use GPT-3 with the following prompt: prompt

Original: [Equation]
Revised: [Equation]

Original: Classify each of the following objects as either " 
Revised: Classify each of the following objects as either " prompt

Original: [Equation]
Revised: [Equation]

Revised: .join

Original: [Equation]
Revised: [Equation]

Revised: category_list

Revised: [Equation]

Original:  where detected_object is the detected object class, and category_list is the each of the provided categories.
Revised: [Equation] [Equation] where detected_object is the detected object class, and category_list is the each of the provided categories.

Original: We make a prompt for each category, where the relevant category is appended tothe prompt.
Revised: We make a prompt for each category, where the relevant category is appended to the prompt.

Original: Using the language model, we choose the category corresponding to the most probableprompt.
Revised: The resulting category is associated with the prompt with the highest probability from the language model.

Original: 4.1 Open source images in the wild
Revised: 4.1 Images in the wild

Original: To indicate the robustness of the two methods, we samplea small set of 23 images from the web andmeasure the classification accuracy for each method. 
Revised: To indicate the robustness of the two methods, we sample 23 images from the web and measure the categorization accuracy for each method. [Table]

Original: [Table]
Revised: [Table]

Original: cutleryvasesornamentscandlestickstableclothsdecorativepillowswoolblanketspicturesandpaintings
Revised: Motivated by the second-hand industry

Original: [Table]
Revised: [Table]

Original: Motivated by the second-hand industry, we sort the images intoa set of categories that might beused when receiving second-hand goods.
Revised: we sort the images into categories that might be used when receiving second-hand goods.

Original: These are listed in the first column of Table 1.
Revised: These are listed in the first column of Table 1.

Revised: We label each image by the most suitable category.

Revised: 4.2 Results

Original: The accuracy for  each method is listed in the first column of Table 2.
Revised: The accuracy for categorization into the narrow categories for each method is listed in the first column of Table 2.

Original: Here, the direct approach isan order of magnitude better than the two-stage approach, at 45% in comparison to 4.5%.
Revised: For both CLIP and ViLD, the unified one-step approach achieves a higher result, an order of magnitude in the case of ViLD, at 45% compared to 4.5%.

Original: Figure 3and Figure 4 show the result on a sample of images using the open vocabulary object detector andthe two-stage approach.
Revised: Figure 3 and Figure 4 show the result on a sample of images using the one-step and the two-step approach, respectively.

Original: Here too, we see that the direct approach yields higher accuracy.
Revised: Here too, we see that the direct approach yields higher accuracy.

Original: 1 Categorization into broader categories
Revised: 4.3 Categorization into broader categories.

Original: Motivated by the hypothesis in Section 3.2, we test the methods on fewer and broader categories.
Revised: Motivated by the hypothesis in Section 3.2 that the two-step approach might be more robust with more abstract categories, we test the methods on fewer and broader categories.

Original: These are listed in the second column of Table 1.
Revised: These are listed in the second column of Table 1.

Original: The second column of Table 2 lists the respective accuracies on this task.
Revised: The second column of Table 2 lists the performance of all the approaches.

Original: As previously, the directmethod outperforms the two-stage approach.
Revised: In the case of ViLD, the direct method outperforms the two-stage approach.

Original: However, both are close to the performance of a random classifier, which would yield an accuracy of 33% .
Revised: Note that, due to only having three categories, a random categorization method would yield an accuracy of 33% .

Original: This is also visible in the visualization on thesubset of images shown in Figure 5 for the direct approach and Figure 6 for the two-stage method.
Revised: For CLIP, however, we see that the two-step approach outperforms the one-step approach.

Original: 4.2 Categorization of YCB-objects
Revised: 4.4 Categorization of YCB-objects

Original: We apply both approaches to images of objects in the YCB-dataset [15], which amounts to 48images.
Revised: We apply both approaches to images of objects in the YCB-dataset [16], which amounts toimages.

Original: We use the categories  in the third column of Table 1.
Revised: The categories are listed in the third column of Table 1.

Revised: The images of YCB objects typically include a single object against a neutral background, whereas the In-the-wild images have varying backgrounds and distracting objects.

Original: The results are summarized by their accuracy in Table 2.
Revised: The results are summarized by their accuracy in Table 2.

Original: The direct approach shows superior performance in this task too.
Revised: The one-step approach shows superior performance for both CLIP and ViLD.

Original: Thedirect approach results  for a set of images are shown in Figure 7, andcomparing these to Figure 8 showing the results of VILD+GPT-3, we can see that the direct methodperforms more robustly.
Revised: The results of the two-step approach for a set of images are shown in Figure 5.

Original: 4.3 Discussion
Revised: 4.5 Classification and categorization evaluation

Original:  For both methods, a significant failure mode is inappropriate bounding boxes.
Revised: We evaluate the two parts of the two-step system in the following way. For classification, we note how frequently the main object in the image belongs to the predicted class. The results are summarized in Table 3.

Original: In the case wherethe bounding box is too small, the categorization is challenging due to the lack of knowledge of 
Revised: CLIP classifies substantially more robustly compared to ViLD, indicating the additional challenge of predicting instance masks and relevant bounding boxes. For the classification by the LM, we count how many of the categories predicted by the LM include the given class. On average, over all experiments, GPT-3 achieves a categorization accuracy of 87.4%.

Original: [Table]
Revised: [Table]

Original:  the  context.
Revised: In the case of ViLD, a significant failure mode is inappropriate bounding boxes. It can capture an irrelevant part of the image or just part of the object of interest. When the bounding box is too small, the categorization is challenging due to the lack of context.

Original: For example, consider the children’s toy in Figure 3.
Revised: For example, consider the children’s toy in Figure 3, where only a small part of the object is included in the bounding box.

Original: In this case, both methods fail tocategorize the whole object correctly.
Revised: This provides limited context for object categorization downstream.

Original: Looking at Table 2, we see that ViLD directly performs better than ViLD+GPT-3 overall.
Revised: Among the tactics to improve this is fine-tuning hyperparameters for the bounding box prediction in ViLD. While the one-step method either fails or succeeds in categorizing correctly, the two-stage can fail or succeed in several ways. This is illustrated in Figure 6, showing the flow of images through the two-step system. Here, there are two ways in which an image can end up miscategorized.

Original: Thispoints to  the fact that important information is lost in the classification process.
Revised: The first is an initial misclassification, leading to a miscategorization by the LM. The other is when a correctly classified image is still incorrectly categorized. Table 3 reveals that there is room for improvement in the classification step. Aside from a standard classification error where the detector chooses the wrong class, another cause for misclassification can be a lacking vocabulary.

Original: Take the drawingof the hand in Figure 4 as an example.
Revised: An example is the Christmas ball present in the rightmost image in Figure 4.

Original: Here, ViLD detects a bounding box around the finger of thedrawn hand.
Revised: The Tencent-ML Image Database does not include the class “Christmas ball”, and the detector predicts a visually similar, but wrong, class. Improving the classifier or extending the vocabulary are tactics to reduce the number of classification errors.

Original: How this is handled further is different for the two methods.
Revised: The miscategorization of an already correctly classified image can be split into two cases. The first is an incorrect category prediction from the LM. We see empirically that this is relatively rare, as the LM correctly categorizes the class 87.4% of the time in our experiments. The other case is due to the predicted class lacking specificity, such as the class prediction for the mustard bottle in Figure 5.

Original: For ViLD+GPT-3, this isclassified as a “knife blade”, which arguably has a similar appearance as  the depicted finger. Thisis then categorized as “cutlery” by GPT-3.
Revised: Here, the predicted class “bottle” is too vague for the LM to determine that this is a food-related bottle. A more specific classification, such as “mustard bottle” would improve the categorization accuracy.

Original: Applying ViLD directly to this bounding box, however,results in  the correct prediction of “decorative”.
Revised: Again, simply improving the vocabulary is a promising approach to alleviate such errors. On the other hand, the two-step approach can also succeed in two ways. As shown in Figure 6, some of the previously misclassified images are still correctly categorized by the LM. An example picture is shown in the rightmost subfigure in Figure 5, where a peach is incorrectly classified as an apple, which is also a member of the overall category “food”.

Original: We theorize that the texture and more minor detailsare highly relevant to  the correct classification, and these crucial cues are lost when first conductingthe classification task.
Revised: Empirically, we see that the assumptions made to satisfy Equation 1 are not valid. Firstly, the classifier is imperfect, and propagating its uncertainty to the classification stage could be an exciting extension.

Original: Referring to Equation 1, we see that assuming the conditional independence P ( category | class ∪image ) ≈ P ( category | class ) is a faulty assumption, as in many cases information such as texture ishighly relevant to predicting the category.
Revised: Secondly, the conditional independence P ( category | class ∪ image ) ≈ P ( category | class ) is a faulty assumption, as a textual description of the object might lack sufficient information for downstream decisions.

Original: In addition, the assumption of a perfect classifier is alsonot valid, as the categorization fails frequently due to  the wrong class being detected.
Revised: Specifically, the textual description might be too vague to provide the relevant information to categorize the object correctly.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we compare two approaches for sorting objects in RGB images motivated by improving robotic sorting.
Revised: In this paper, we compare two approaches for sorting objects in RGB images motivated by improving robotic sorting.

Original: We implement an approach where the task of determining the class of an object(e.g. , “apple”) is separated from categorizing the object into categories (e.g., “food”).
Revised: We implement an approach where the task of determining the class of an object (e.g., “apple”) is separated from categorizing the object into categories (e.g., “food”).

Original: In a set ofexperiments, we measure the accuracy of previously unseen images and categories.
Revised: In a set of experiments, we measure the accuracy of previously unseen images and categories.

Original: After that, wepresent insights into where the different methods fail and succeed.
Revised: Additionally, we present insights into where the different methods fail and succeed.

Original: The results indicate that  informationabout appearance, such as detailed texture, ishighly relevant to determining the category.
Revised: The results indicate that access to visual information is relevant to determining the category and that describing the object sufficiently by its class is restrictive.

Original: Therefore, a direct approach behaves more robustly in our experiments.
Revised: Therefore, a direct approach behaves more robustly in our experiments.

Original: As directions for further work, improving direct, open-vocabulary categorization methods emergesas a promising direction.
Revised: As directions for further work, improving direct, open-vocabulary categorization methods emerges as a promising direction.

Original: A study of the effect of prompt engineering could provide some interestinghints on how to improve VLMs in general.
Revised: A study of the effect of prompt engineering could provide some interesting hints on how to improve VLMs in general.

Original: Additionally, in the case of second-hand stores, it is notonly crucial to group objects into categories, but also evaluate whether the object is in good conditionand applicable for reselling.
Revised: Additionally, in the case of second-hand stores, it is not only crucial to group objects into categories but also evaluate whether the object is in good condition and applicable for reselling.

Original: This new dimension is helpful to consider in further work to progresstowards more effective second-hand sorting systems.
Revised: This new dimension is helpful to consider in further work to progress towards more effective second-hand sorting systems.

Original: Another point of further work is investigating the gain of using RGB-D images for classification, as such observations are typically available onmodern robotic platforms.
Revised: Another point of further work is investigating the gain of using RGB-D images for classification, as such observations are typically available on modern robotic platforms.

Original: This might help increase the detection, classification, and categorizationrobustness.
Revised: This might help increase the detection, classification, and categorization robustness. Acknowledgments

Revised: The authors would like to thank the reviewers for their helpful feedback and interesting perspectives.

Revised: This work was funded by the Norwegian Research Council under Project Number 237896, SFI

Revised: Offshore Mechatronics.