Original: U NIFIED -IO: A UNIFIED MODEL FOR VISION , LANGUAGE , AND MULTI - MODAL TASKS
Revised: U NIFIED -IO: A UNIFIED MODEL FOR VISION , LANGUAGE , AND MULTI - MODAL TASKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose U NIFIED -IO, a model that performs a large variety of AI tasks spanning classical computer vision tasks, including pose estimation, object detection, depth estimation and image generation, vision-and-language tasks such as region captioning and referring expression, to natural language processing tasks such as question answering and paraphrasing.
Revised: We propose U NIFIED -IO, a model that performs a large variety of AI tasks spanning classical computer vision tasks, including pose estimation, object detection, depth estimation and image generation, vision-and-language tasks such as region captioning and referring expression, to natural language processing tasks such as question answering and paraphrasing.

Original: Developing a single unified model for such a large variety of tasks poses unique challenges due to the heterogeneous inputs and outputs pertaining to each task, including RGB images, per-pixel maps, binary masks, bounding boxes, and language.
Revised: Developing a single unified model for such a large variety of tasks poses unique challenges due to the heterogeneous inputs and outputs pertaining to each task, including RGB images, per-pixel maps, binary masks, bounding boxes, and language.

Original: We achieve this unification by homogenizing every supported input and output into a sequence of discrete vocabulary tokens.
Revised: We achieve this unification by homogenizing every supported input and output into a sequence of discrete vocabulary tokens.

Original: This common representation across all tasks allows us to train a single transformer-based architecture, jointly on over 90 diverse datasets in the vision and language fields.
Revised: This common representation across all tasks allows us to train a single transformer-based architecture, jointly on over 90 diverse datasets in the vision and language fields.

Original: U NIFIED -IO is the first model capable of performing alltasks on the GRIT benchmark and produces strong results across 16 diverse benchmarks like NYUv2-Depth, ImageNet, VQA2.0, OK-VQA, Swig, VizWizGround, BoolQ, and SciTail, with no task-specific fine-tuning.
Revised: U NIFIED -IO is the first model capable of performing all 7 tasks on the GRIT benchmark and produces strong results across 16 diverse benchmarks like NYUv2-Depth, ImageNet, VQA2.0, OK-VQA, Swig, VizWizGround, BoolQ, and SciTail, with no task-specific fine-tuning.

Original: Code and pre-trained models will be made publicly available.
Revised: Code and demos for U NIFIED -IO are available at: unified-io.allenai.org

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We present U NIFIED -IO, the first neural model to jointly perform a large and diverse set of AI tasks spanning classical computer vision (such as object detection, segmentation, and depth estimation), image synthesis (such as image generation and image in-painting), vision-and-language (like visual question answering, image captioning, and referring expression) and NLP (such as question answering and paraphrasing).
Revised: We present U NIFIED -IO, the first neural model to jointly perform a large and diverse set of AI tasks spanning classical computer vision (such as object detection, segmentation and depth estimation), image synthesis (such as image generation and image in-painting), vision-and-language (like visual question answering, image captioning and referring expression) and NLP (such as question answering and paraphrasing).

Original: Unified general-purpose models avoid the need for task-specific design, learn and perform a wide range of tasks with a single architecture, can utilize large, diverse data corpora, can effectively transfer concept knowledge across tasks, and even perform tasks unknown and unobserved at design and training time.
Revised: Unified general-purpose models avoid the need for task-specific design, learn and perform a wide range of tasks with a single architecture, can utilize large, diverse data corpora, can effectively transfer concept knowledge across tasks, and even perform tasks unknown and unobserved at design and training time.

Original: Building unified models for computer vision has proven to be quite challenging since vision tasks have incredibly diverse input and output representations.
Revised: Building unified models for computer vision has proven to be quite challenging since vision tasks have incredibly diverse input and output representations.

Original: For instance, object detection produces bounding boxes around objects in an image, segmentation produces binary masks outlining regions in an image, visual question answering produces an answer as text, and depth estimation produces a map detailing the distance of each pixel from the camera.
Revised: For instance, object detection produces bounding boxes around objects in an image, segmentation produces binary masks outlining regions in an image, visual question answering produces an answer as text, and depth estimation produces a map detailing the distance of each pixel from the camera.

Original: This heterogeneity makes it very challenging to architect a single model for all these tasks.
Revised: This heterogeneity makes it very challenging to architect a single model for all these tasks.

Original: In contrast, while the landscape of natural language processing (NLP) tasks, datasets, and benchmarks is large and diverse, their inputs and desired outputs can often be uniformly represented as sequences of tokens.
Revised: In contrast, while the landscape of natural language processing (NLP) tasks, datasets, and benchmarks is large and diverse, their inputs and desired outputs can often be uniformly represented as sequences of tokens.

Original: Sequence to sequence (Seq2Seq) architectures (Raffel et al., 2020; Brown et al., 2020), specifically designed to accept and produce such sequences of tokens, are thus widely applicable to many tasks.
Revised: Sequence to sequence (Seq2Seq) architectures (Raffel et al., 2020; Brown et al., 2020), specifically designed to accept and produce such sequences of tokens, are thus widely applicable to many tasks.

Original: Unified models employing such architectures have been central to much recent progress in NLP.
Revised: Unified models employing such architectures have been central to much recent progress in NLP.

Original: Unified models for computer vision typically use a shared visual backbone to produce visual embeddings but then employ individual branches for each of the desired tasks.
Revised: Unified models for computer vision typically use a shared visual backbone to produce visual embeddings but then employ individual branches for each of the desired tasks.

Original: These include models  like Mask R-CNN (He et al., 2017) for classical visual tasks that use an ImageNet pre-trained encoder followed by branches for detection and segmentation, trained in a fully supervised manner.
Revised: These include models [Figure] like Mask R-CNN (He et al., 2017) for classical visual tasks that use an ImageNet pre-trained encoder followed by branches for detection and segmentation, trained in a fully supervised manner.

Original: In the vision and language (V&L) domain, CNN backbones feed visual features to transformer architectures that also combine language, followed by task-specific heads for visual question answering,[Figure] referring expression, visual commonsense reasoning, etc.
Revised: In the vision and language (V&L) domain, CNN backbones feed visual features to transformer architectures that also combine language, followed by task-specific heads for visual question answering, referring expression, visual commonsense reasoning, etc.

Original: (Lu et al., 2019;
Revised: (Lu et al., 2019;

Original: Li et al., 2019; Tan &
Revised: Li et al., 2019; Tan &

Original: Bansal, 2019).
Revised: Bansal, 2019).

Original: A more recent trend has seen the emergence of unified architectures that do away with task-specific heads and instead introduce modality-specific heads (Hu & Singh, 2021; Cho et al., 2021; Gupta et al., 2022a; Wang et al., 2022b) – for instance, a single language decoder that serves multiple tasks requiring language output like captioning and classification.
Revised: A more recent trend has seen the emergence of unified architectures that do away with task-specific heads and instead introduce modality-specific heads (Hu & Singh, 2021; Cho et al., 2021; Gupta et al., 2022a; Wang et al., 2022b) – for instance, a single language decoder that serves multiple tasks requiring language output like captioning and classification.

Original: However, most progress in unified models continues to be centered around V&L tasks, owing to the simplicity of building shared language decoders, and is often limited to supporting just a handful of tasks.
Revised: However, most progress in unified models continues to be centered around V&L tasks, owing to the simplicity of building shared language decoders and is often limited to supporting just a handful of tasks.

Original: U NIFIED -IO is a Seq2Seq model capable of performing a variety of tasks using a unified architecture without a need for either task or even modality-specific branches.
Revised: U NIFIED -IO is a Seq2Seq model capable of performing a variety of tasks using a unified architecture without a need for either task or even modality-specific branches.

Original: This broad unification is achieved by homogenizing every task’s output into a sequence of discrete tokens.
Revised: This broad unification is achieved by homogenizing every task’s output into a sequence of discrete tokens.

Original: Dense structured outputs such as images, segmentation masks and depth maps are converted to sequences using a vector quantization variational auto-encoder (VQ-GAN) (Esser et al., 2021), sparse structured outputs such as bounding boxes, and human joint locations are transcribed into sequences of coordinate tokens, and language outputs are converted to sequences using byte-pair encoding.
Revised: Dense structured outputs such as images, segmentation masks and depth maps are converted to sequences using a vector quantization variational auto-encoder (VQ-GAN) (Esser et al., 2021), sparse structured outputs such as bounding boxes, and human joint locations are transcribed into sequences of coordinate tokens, and language outputs are converted to sequences using byte-pair encoding.

Original: This unification enables
Revised: This unification enables

Original: Unified-IO to jointly train on over 90 datasets spanning computer vision, V&L, and NLP tasks with a single streamlined transformer encoder-decoder architecture (Raffel et al., 2020).
Revised: Unified-IO to jointly train on over 90 datasets spanning computer vision, V&L, and NLP tasks with a single streamlined transformer encoder-decoder architecture (Raffel et al., 2020).

Original: Our jointly trained U NIFIED -IO is the first model to support all 7 tasks in the General Robust Image Task (GRIT) Benchmark (Gupta et al., 2022b) and obtains the top overall score of 64.3 when averaging across all tasks, handily beating the second best model by 32.0.
Revised: Our jointly trained U NIFIED -IO is the first model to support all 7 tasks in the General Robust Image Task (GRIT) Benchmark (Gupta et al., 2022b) and obtains the top overall score of 64.3 when averaging across all tasks, handily beating the second best model by 32.0.

Original: We further evaluate
Revised: We further evaluate

Original: U NIFIED -IO on 16 diverse benchmarks across computer vision and NLP, without any fine-tuning towards any individual benchmark, and find that it performs remarkably well compared to specialized (or fine-tuned) state-of-the-art models.
Revised: U NIFIED -IO on 16 diverse benchmarks across computer vision and NLP, without any fine-tuning towards any individual benchmark, and find that it performs remarkably well compared to specialized (or fine-tuned) state-of-the-art models.

Original:  V ISION , L ANGUAGE AND M ULTI -M ODAL T ASKS
Revised:  V ISION , L ANGUAGE AND M ULTI -M ODAL T ASKS

Original: U NIFIED -IO is designed to handle a wide range of language, vision and language, and classic vision tasks in a unified way.
Revised: U NIFIED -IO is designed to handle a wide range of language, vision and language, and classic vision tasks in a unified way.

Original: To fully test this capability, we gather 95 vision, language, and multi-modal datasets from 62 publicly available data sources as targets for our model to learn during multi-task training.
Revised: To fully test this capability, we gather 95 vision, language, and multi-modal [Table] ✓ - - - ✓ - - [Table]

Original: These datasets cover a wide range of tasks, skills, and modalities.
Revised: [Table] ✓ - - - ✓ - - [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: ✓ ✓ ✓ ✓ ✓ ✓ datasets from 62 publicly available data sources as targets for our model to learn during multi-task training.

Revised: These datasets cover a wide range of tasks, skills, and modalities.

Original: We categorize the input and output modalities of each task into 4 different types:
Revised: We categorize the input and output modalities of each task into 4 different types:

Original: Text – natural language tokens; Image – RGB images; Sparse – a small number of location coordinates within the image; Dense – per-pixel labels such as depth maps, surface normal maps, etc .
Revised: Text – natural language tokens; Image – RGB images; Sparse – a small number of location coordinates within the image; Dense – per-pixel labels such as depth maps, surface normal maps, etc .

Original: We group related datasets into 8 groups and 22 tasks to facilitate our training and analysis:
Revised: We group related datasets into 8 groups and 22 tasks to facilitate our training and analysis:

Original: Image Synthesis.
Revised: Image Synthesis.

Original: Given a text description, partially occluded image and inpainting target, or segmentation map containing a semantic class for some pixels, generate a matching image.
Revised: Given a text description, partially occluded image and inpainting target, or segmentation map containing a semantic class for some pixels, generate a matching image.

Original: Data sources with image and text pairs (Desai et al., 2021), bounding boxes (Krishna et al., 2017) or semantic segmentation (Gupta et al., 2019) can be used to build these tasks.
Revised: Data sources with image and text pairs (Desai et al., 2021), bounding boxes (Krishna et al., 2017) or semantic segmentation (Gupta et al., 2019) can be used to build these tasks.

Original: Sparse Labelling.
Revised: Sparse Labelling.

Original: Given an image and a natural language query, identify the target regions or keypoint locations that are being referred to.
Revised: Given an image and a natural language query, identify the target regions or keypoint locations that are being referred to.

Original: Tasks include object detection (Kuznetsova et al., 2020), object localization (Rhodes et al., 2017), human pose estimation (Lin et al., 2014) and referring expression (Kazemzadeh et al., 2014).
Revised: Tasks include object detection (Kuznetsova et al., 2020), object localization (Rhodes et al., 2017), human pose estimation (Lin et al., 2014), and referring expression (Kazemzadeh et al., 2014).

Original: Dense Labelling.
Revised: Dense Labelling.

Original: Given an image, produce per-pixel labels for that image.
Revised: Given an image, produce per-pixel labels for that image.

Original: Labels include the distance of that pixel to the camera (Nathan Silberman & Fergus, 2012), surface orientation (Bae et al., 2021) or semantic class (Lin et al., 2014).
Revised: Labels include the distance of that pixel to the camera (Nathan Silberman & Fergus, 2012), surface orientation (Bae et al., 2021) or semantic class (Lin et al., 2014).

Original: Image Classification.
Revised: Image Classification.

Original: Given an image and optionally a target bounding box, generate a class name or tag of that image or target region.
Revised: Given an image and optionally a target bounding box, generate a class name or tag of that image or target region.

Original: This group includes image classification (Deng et al., 2009) and object categorization (Pinz et al., 2006) datasets.
Revised: This group includes image classification (Deng et al., 2009) and object categorization (Pinz et al., 2006) datasets.

Original: Image Captioning.
Revised: Image Captioning.

Original: Given an image and optionally a bounding box, generate a natural language description of that image or target region.
Revised: Given an image and optionally a bounding box, generate a natural language description of that image or target region.

Original: We include both crowd-sourced (Chen et al., 2015) and webly supervised (Changpinyo et al., 2021) captions.
Revised: We include both crowd-sourced (Chen et al., 2015) and webly supervised (Changpinyo et al., 2021) captions.

Revised: Transformer Decoder

Original: [Figure]
Revised: [Figure]

Original: What
Revised: embedding

Original: [Figure] color [Figure] the cat?
Revised: What is the color of the cat?

Original: SentPieceencoder
Revised: SentPieceencoder

Original: Localize the cat
Revised: Localize the cat

Original: SentPieceencoder
Revised: SentPieceencoder

Original: Segment the cat
Revised: Segment the cat

Original: SentPieceencoder
Revised: SentPieceencoder

Original: What is the depth map of the image?
Revised: What is the depth map of the image?

Original: SentPieceencoder
Revised: SentPieceencoder

Original: VQGANdecoder
Revised: VQGANdecoder

Original: SentPiecedecoder brown
Revised: SentPiecedecoder brown

Original: VQGANdecoder
Revised: VQGANdecoder

Original: SentPiecedecoder  loc
Revised: SentPiecedecoder loc750 loc940 loc480 loc

Original: [Figure]
Revised: [Figure]

Original: loc
Revised: Transformer Encoder quantization arg

Original: [Figure]
Revised: [Figure]

Original: quantization
Revised: VQGANEncoder (CNN) VQGANDecoder( [Figure] [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: Discrete

Original: [Figure]
Revised: [Figure]

Revised: serves

Original: [Figure]
Revised: [Figure]

Original: Vision & Language.
Revised: Vision & Language.

Original: A broad category for other tasks that require jointly reason over image content and a natural language query.
Revised: A broad category for other tasks that require joint reason over image content and a natural language query.

Original: There are many popular vision and language datasets, and we categories these datasets into 3 tasks – visual question answering (Antol et al., 2015); relationship detection (Lu et al., 2016) and grounded VQA (Chen et al., 2022a).
Revised: There are many popular vision and language datasets, and we categories these datasets into 3 tasks – visual question answering (Antol et al., 2015); relationship detection (Lu et al., 2016) and grounded VQA (Chen et al., 2022a).

Original: NLP.
Revised: NLP.

Original: Tasks with text as the only input and output modalities, including text classification (Williams et al., 2018), question answering (Rajpurkar et al., 2016) and text summarization (Graff et al., 2003).
Revised: Tasks with text as the only input and output modalities, including text classification (Williams et al., 2018), question answering (Rajpurkar et al., 2016) and text summarization (Graff et al., 2003).

Original: Language Modeling.
Revised: Language Modeling.

Original: The masking language modeling pre-training task (See Section 3.3) using text from C4 (Raffel et al., 2020) and Wikipedia (Foundation), which we include to ensure the knowledge gained from language pre-training is not lost during multi-task training.
Revised: The masking language modeling pre-training task (See Section 3.3) using text from C4 (Raffel et al., 2020) and Wikipedia (Foundation), which we include to ensure the knowledge gained from language pre-training is not lost during multi-task training.

Original: Other pre-training tasks are not included because the relevant datasets are already used in other supervised tasks ( e.g ., for captioning or classification).
Revised: Other pre-training tasks are not included because the relevant datasets are already used in other supervised tasks ( e.g ., for captioning or classification).

Original: Table 1 shows the details of tasks 
Revised: Table 1 shows the details of tasks and

Original:  U NIFIED -IO
Revised:  U NIFIED -IO

Original: Our goal is to build a single unified model that can support a diverse set of tasks across computer vision and language with little to no need for task-specific customizations and parameters.
Revised: Our goal is to build a single unified model that can support a diverse set of tasks across computer vision and language with little to no need for task-specific customizations and parameters.

Original: Such unified architectures can be applied to new tasks with little to no knowledge of the underlying machinery, enable general pre-training to benefit many diverse downstream applications, be jointly trained on a large number of tasks, and better allows knowledge to be shared between tasks.
Revised: Such unified architectures can be applied to new tasks with little to no knowledge of the underlying machinery, enable general pre-training to benefit many diverse downstream applications, be jointly trained on a large number of tasks, and better allows knowledge to be shared between tasks.

Original: 3. U NIFIED T ASK R EPRESENTATIONS
Revised: 3. U NIFIED T ASK R EPRESENTATIONS

Original: Supporting a variety of modalities such as images, language, boxes, binary masks, segmentation masks, etc . without task-specific heads requires representing these modalities in a shared and unified space.
Revised: Supporting a variety of modalities such as images, language, boxes, binary masks, segmentation masks, etc . without task-specific heads requires representing these modalities in a shared and unified space.

Original: To do this, we discretize the text, images, and other structured outputs in our tasks and represent them with tokens drawn from a unified and finite vocabulary.
Revised: To do this, we discretize the text, images, and other structured outputs in our tasks and represent them with tokens drawn from a unified and finite vocabulary.

Original: Text representation.
Revised: Text representation.

Original: Following Raffel et al.
Revised: Following Raffel et al.

Original: (2020), text inputs and outputs are tokenized using
Revised: (2020), text inputs and outputs are tokenized using

Original: SentencePiece (Kudo & Richardson, 2018).
Revised: SentencePiece (Kudo & Richardson, 2018).

Original: Following past works such as McCann et al.
Revised: Following past works such as McCann et al.

Original: Raffel et al.
Revised: Raffel et al.

Original: ; Gupta et al.
Revised: ; Gupta et al.

Original: (2022a)
Revised: (2022a)

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2022b)
Revised: (2022b)

Original: we also specify each task with a natural language prompt (excluding some tasks like VQA, which are fully specified by their text inputs) in order to indicate what task should be performed.
Revised: we also specify each task with a natural language prompt (excluding some tasks like VQA, which are fully specified by their text inputs) in order to indicate what task should be performed.

Original: For example, “ What is the depth map of the image? ”
Revised: For example, “ What is the depth map of the image? ”

Original: for depth estimation or “ What region does “cat” describe? ”
Revised: for depth estimation or “ What region does “cat” describe? ”

Original: for object localization.
Revised: for object localization.

Original: Images and dense structures representation.
Revised: Images and dense structures representation.

Original: A variety of tasks in computer vision requires the model to produce high-dimensional outputs such as images ( e.g ., image in-painting) or per-pixel labels ( e.g ., depth estimation).
Revised: A variety of tasks in computer vision requires the model to produce high-dimensional outputs such as images ( e.g ., image in-painting) or per-pixel labels ( e.g ., depth estimation).

Original: To handle these modalities, we first convert per-pixel labels into
Revised: To handle these modalities, we first convert per-pixel labels into

Original: RGB images.
Revised: RGB images.

Original: For depth, we construct a grayscale image by normalizing the depth map.
Revised: For depth, we construct a grayscale image by normalizing the depth map.

Original: For surface normal estimation, we convert the x/y/z orientations into r/g/b values.
Revised: For surface normal estimation, we convert the x/y/z orientations into r/g/b values.

Original: For segmentation, we map each instance present in the image to a unique color.
Revised: For segmentation, we map each instance present in the image to a unique color.

Original: We randomly select colors for each instance and specify the color-to-class mapping in the text instead of using universal color-to-class mapping.
Revised: We randomly select colors for each instance and specify the color-to-class mapping in the text instead of using universal color-to-class mapping.

Original: This avoids requiring a fixed list of classes and avoids having colors that may only be marginally different due to the presence of a large number of classes.
Revised: This avoids requiring a fixed list of classes and avoids having colors that may only be marginally different due to the presence of a large number of classes.

Original: Then we encode these images as discrete tokens using a VQ-GAN.
Revised: Then we encode these images as discrete tokens using a VQ-GAN.

Original: In particular, we use the imagenet-pretrained VQ-GAN from Esser et al.
Revised: In particular, we use the imagenet-pretrained VQ-GAN from Esser et al.

Original: (2021) with 256 × 256 resolution,  compression ratio of 16 , and 16384 codebook size.
Revised: (2021) with 256 × 256 resolution, a compression ratio of 16 , and 16384 codebook size.

Original: The VQ-GAN codebook is added to the vocabulary as additional tokens that can be generated by the decoder.
Revised: The VQ-GAN codebook is added to the vocabulary as additional tokens that can be generated by the decoder.

Original: During training, the tokens for the target image are used as targets.
Revised: During training, the tokens for the target image are used as targets.

Original: During inference, the VQ-GAN decoder is used to convert the generated image tokens into an output image.
Revised: During inference, the VQ-GAN decoder is used to convert the generated image tokens into an output image.

Original: Sparse structures representation.
Revised: Sparse structures representation.

Original: We encode sparse structures such as bounding boxes or human joints by adding 1000 special tokens to the vocabulary to represent discretized image coordinates (Chen et al., 2022b).
Revised: We encode sparse structures such as bounding boxes or human joints by adding 1000 special tokens to the vocabulary to represent discretized image coordinates (Chen et al., 2022b).

Original: Points are then encoded with a sequence of two such tokens, one for the x and one for the y coordinates, and boxes are encoded using a sequence of four tokens, two for the upper right corner and two for the lower left corner.
Revised: Points are then encoded with a sequence of two such tokens, one for the x and one for the y coordinates, and boxes are encoded using a sequence of four tokens, two for the upper right corner and two for the lower left corner.

Original: Labeled boxes are encoded as a box followed by a text class label, and joints are encoded as a sequence of points followed by a text visibility label.
Revised: Labeled boxes are encoded as a box followed by a text class label, and joints are encoded as a sequence of points followed by a text visibility label.

Original: This allows us to handle a wide variety of tasks that use these elements in their inputs or output (see
Revised: This allows us to handle a wide variety of tasks that use these elements in their inputs or output (see

Original: Appendix A.1 for examples).
Revised: Appendix A.1 for examples).

Original: 3. U NIFIED A RCHITECTURE
Revised: 3. U NIFIED A RCHITECTURE

Original: Universally representing a wide variety of tasks as input and output sequences of discrete tokens enables us to employ architectures that have been proven successful in natural language processing.
Revised: Universally representing a wide variety of tasks as input and output sequences of discrete tokens enables us to employ architectures that have been proven successful in natural language processing.

Original: In
Revised: In

Original: U NIFIED -IO, we propose a pure transformer model largely following the design of T5 (Raffel et al., 2020).
Revised: U NIFIED -IO, we propose a pure transformer model largely following the design of T5 (Raffel et al., 2020).

Original: In particular, U NIFIED -IO is an encoder-decoder architecture where both the encoder and decoder are composed of stacked transformer layers, which in turn are composed of self-attention transformers, cross-attention transformers (in the decoder), and feed-forward neural networks.
Revised: In particular, U NIFIED -IO is an encoder-decoder architecture where both the encoder and decoder are composed of stacked transformer layers, which in turn are composed of self-attention transformers, cross-attention transformers (in the decoder), and feed-forward neural networks.

Original: The layers are applied residually, and layer norms are applied before each transformer and feed-forward network.
Revised: The layers are applied residually, and layer norms are applied before each transformer and feed-forward network.

Original: See Raffel et al.
Revised: See Raffel et al.

Original: (2020) for details.
Revised: (2020) for details.

Original: We make a few architectural changes to adapt the T5 architecture to our setting.
Revised: We make a few architectural changes to adapt the T5 architecture to our setting.

Original: First, to handle input images, we reshape the image into a sequence of patches that are embedded with linear projection similar to Dosovitskiy et al.
Revised: First, to handle input images, we reshape the image into a sequence of patches that are embedded with linear projection similar to Dosovitskiy et al.

Original: Second, we expand the vocabulary to include the location tokens and the image tokens used in the VQ-GAN.
Revised: Second, we expand the vocabulary to include the location tokens and the image tokens used in the VQ-GAN.

Original: Third, we extend the 1-d relative embedding (Dosovitskiy et al., 2021) to 2-d with a fixed number of learned embeddings.
Revised: Third, we extend the 1-d relative embedding (Dosovitskiy et al., 2021) to 2-d with a fixed number of learned embeddings.

Original: We also add absolute position embedding to the token embedding following Devlin et al.
Revised: We also add absolute position embedding to the token embedding following Devlin et al.

Original: (2019), since the absolute position information is essential to image tasks.
Revised: (2019), since the absolute position information is essential to image tasks.

Original: We use a maximum of 256 and 128 text tokens for inputs and outputs respectively, and a maximum length of 576 ( i.e . 24 × 24 patch encoding from a 384 × 384 image) for image inputs and 256 ( i.e .
Revised: We use a maximum of 256 and 128 text tokens for inputs and outputs respectively, and a maximum length of 576 ( i.e . 24 × 24 patch encoding from a 384 × 384 image) for image inputs and 256 ( i.e .

Original: 16 × 16 latent codes from a 256 × 256 image) for image outputs.
Revised: 16 × 16 latent codes from a 256 × 256 image) for image outputs.

Original: In this work, we present four versions of U NIFIED -IO ranging from 71 million to 2.9 billion parameters, as detailed in Table 2.
Revised: In this work, we present four versions of U NIFIED -IO ranging from 71 million to 2.9 billion parameters, as detailed in Table 2.

Original: 3. T RAINING
Revised: 3. T RAINING

Original: U NIFIED -IO is trained in two stages – A pre-training stage that uses unsupervised losses from text, image, and paired image-text data, and a massive multi-task stage where the model is jointly trained
Revised: U NIFIED -IO is trained in two stages – A pre-training stage that uses unsupervised losses from text, image, and paired image-text data, and a massive multi-task stage where the model is jointly trained

Original: [Table]
Revised: [Table]

Original: on a large variety of tasks.
Revised: on a large variety of tasks.

Original: Since our goal is to examine whether a single unified model can solve a variety of tasks simultaneously, we do not perform task-specific fine-tuning although prior work (Lu et al., 2020; Wang et al., 2022b) shows it can further improve task performance.
Revised: Since our goal is to examine whether a single unified model can solve a variety of tasks simultaneously, we do not perform task-specific fine-tuning although prior work (Lu et al., 2020; Wang et al., 2022b) shows it can further improve task performance.

Original: Pre-training.
Revised: Pre-training.

Original: To learn good representations from large-scale webly supervised image and text data, we consider two pre-training tasks: text span denoising and masked image denoising .
Revised: To learn good representations from large-scale webly supervised image and text data, we consider two pre-training tasks: text span denoising and masked image denoising .

Original: The text span denoising task follows Raffel et al.
Revised: The text span denoising task follows Raffel et al.

Original: (2020) – randomly corrupt 15% of the tokens and replace the consecutive corrupted tokens with a unique mask token.
Revised: (2020) – randomly corrupt 15% of the tokens and replace the consecutive corrupted tokens with a unique mask token.

Original: The masked image denoising task follows
Revised: The masked image denoising task follows

Original: Bao et al.
Revised: Bao et al.

Original: and
Revised: and

Original: He et al.
Revised: He et al.

Original: (2022) – randomly masked 75% of the image patches, and the goal is to recover the whole image.
Revised: (2022) – randomly masked 75% of the image patches, and the goal is to recover the whole image.

Original: When another modality is present, i.e . image or text, the model can use information from that modality to complete the tasks.
Revised: When another modality is present, i.e . image or text, the model can use information from that modality to complete the tasks.

Original: We construct the pre-training dataset by incorporating publicly available language data (i.e., plain texts from Common Crawl), vision data (i.e., raw images from different datasets), and V&L data (i.e., image caption and image label pairs).
Revised: We construct the pre-training dataset by incorporating publicly available language data (i.e., plain texts from Common Crawl), vision data (i.e., raw images from different datasets), and V&L data (i.e., image caption and image label pairs).

Original: For V&L data, we add a simple prompt “ An image of ” at the beginning of  caption or categories to indicate it is multi-modal data (Wang et al., 2022d).
Revised: For V&L data, we add a simple prompt “ An image of ” at the beginning of the caption or categories to indicate it is multi-modal data (Wang et al., 2022d).

Original: We pre-train U NIFIED -IO on this combination of datasets with an in-batch mixing strategy.
Revised: We pre-train U NIFIED -IO on this combination of datasets with an in-batch mixing strategy.

Original: We equally sample data with the text and image denoising objective.
Revised: We equally sample data with the text and image denoising objective.

Original: For text denoising, half of the samples are from pure text data, i.e . C4 and Wikipedia.
Revised: For text denoising, half of the samples are from pure text data, i.e . C4 and Wikipedia.

Original: The other half is constructed from image and class data, such as Imagenet21k (Ridnik et al., 2021) or image and caption data, such as YFCC15M
Revised: The other half is constructed from image and class data, such as Imagenet21k (Ridnik et al., 2021) or image and caption data, such as YFCC15M

Original: (Radford et al., 2021).
Revised: (Radford et al., 2021).

Original: For image denoising, we also use the same caption and class data and some image-only data from datasets for our vision tasks.
Revised: For image denoising, we also use the same caption and class data and some image-only data from datasets for our vision tasks.

Original: We sample from datasets in proportion to dataset size.
Revised: We sample from datasets in proportion to dataset size.

Original: See Appendix A.2 for details.
Revised: See Appendix A.2 for details.

Original: Multi-tasking.
Revised: Multi-tasking.

Original: To build a single unified model for diverse vision, language, and V&L tasks, we construct a massive multi-tasking dataset by ensembling 95 datasets from 62 publicly available data sources.
Revised: To build a single unified model for diverse vision, language, and V&L tasks, we construct a massive multi-tasking dataset by ensembling 95 datasets from 62 publicly available data sources.

Original: See Section 2 for task details and Appendix A.1 for dataset visualizations.
Revised: See Section 2 for task details and Appendix A.1 for dataset visualizations.

Original: We jointly train U NIFIED -IO on this large set of datasets by mixing examples from these datasets within each batch.
Revised: We jointly train U NIFIED -IO on this large set of datasets by mixing examples from these datasets within each batch.

Original: We equally sample each group ( 1 / 8 ) except for image synthesis ( 3 / 16 ) and dense labeling ( 1 / 16 ) since dense labeling has significantly fewer data and image synthesis has significantly more data than other groups.
Revised: We equally sample each group ( 1 / 8 ) except for image synthesis ( 3 / 16 ) and dense labeling ( 1 / 16 ) since dense labeling has significantly fewer data and image synthesis has significantly more data than other groups.

Original: Within each group, we sample datasets proportional to the square root of their size to better expose the model to underrepresented tasks.
Revised: Within each group, we sample datasets proportional to the square root of their size to better expose the model to underrepresented tasks.

Original: Due to the large variance in dataset size, some tasks are still rarely sampled ( e.g . depth estimation only has a 0 .
Revised: Due to the large variance in dataset size, some tasks are still rarely sampled ( e.g . depth estimation only has a 0 .

Original: 43% chance of being sampled).
Revised: 43% chance of being sampled).

Original: See Appendix A.3 for details and visualizations.
Revised: See Appendix A.3 for details and visualizations.

Original: Implementation Details.
Revised: Implementation Details.

Original: Due to space limitation, see Appendix A.4 for implementation details.
Revised: Due to space limitation, see Appendix A.4 for implementation details.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We now present results for U NIFIED -IO on the GRIT benchmark (Sec 4.1), ablate training data via the GRIT ablation benchmark (Sec 4.2) and evaluate U NIFIED -IO on 16 other benchmarks in computer vision and NLP (Sec 4.3).
Revised: We now present results for U NIFIED -IO on the GRIT benchmark (Sec 4.1), ablate training data via the GRIT ablation benchmark (Sec 4.2) and evaluate U NIFIED -IO on 16 other benchmarks in computer vision and NLP (Sec 4.3).

Original: Appendix A.5 shows  evaluation on same concept and new concept on GRIT and A.6 shows the prompt generalization.
Revised: Appendix A.5 shows the evaluation of the same concept and new concept on GRIT and A.6 shows the prompt generalization.

Original: Qualitative examples are in A.8.
Revised: Qualitative examples are in A.9.

Original: 4. R ESULTS ON GRIT
Revised: 4. R ESULTS ON GRIT

Original: The General Robust Image Task (GRIT) Benchmark (Gupta et al., 2022b) is an evaluation-only benchmark designed to measure the performance of models across multiple tasks, concepts, and data sources.
Revised: The General Robust Image Task (GRIT) Benchmark (Gupta et al., 2022b) is an evaluation-only benchmark designed to measure the performance of models across multiple tasks, concepts, and data sources.

Original: GRIT aims to encourage the building of unified and general purpose vision models and is thus well suited to evaluate U NIFIED -IO.
Revised: GRIT aims to encourage the building of unified and general purpose vision models and is thus well suited to evaluate U NIFIED -IO.

Original: GRIT has seven tasks that cover a range of visual skills with varying input and output modalities and formats: categorization, localization, VQA, refer expression, segmentation, keypoint, and surface normal estimation.
Revised: GRIT has seven tasks that cover a range of visual skills with varying input and output modalities and formats: categorization, localization, VQA, refer expression, segmentation, keypoint, and surface normal estimation.

Original: [Table]
Revised: [Table]

Original: U NIFIED -IO is the first model to support all seven tasks in GRIT.
Revised: U NIFIED -IO is the first model to support all seven tasks in GRIT.

Original: As seen in Table 3, U NIFIED -IO XL outperforms all prior submissions to GRIT obtaining average accuracy of 64.3 on test.
Revised: As seen in Table 3, U NIFIED -IO XL outperforms all prior submissions to GRIT obtaining average accuracy of 64.3 on test.

Original: The next best submission is GPV-2 (Kamath et al., 2022) which obtains 32.0 and can only support 4 out of 7 tasks.
Revised: The next best submission is GPV-2 (Kamath et al., 2022) which obtains 32.0 and can only support 4 out of 7 tasks.

Original: U NIFIED -IO XL also outperforms the multi-task checkpoint of OFA LARGE (Wang et al., 2022b) on
Revised: U NIFIED -IO XL also outperforms the multi-task checkpoint of OFA LARGE (Wang et al., 2022b) on

Original: VQA, refer expression and categorization.
Revised: VQA, refer expression and categorization.

Original: Mask R-CNN (He et al., 2017) is a strong baseline for core vision tasks.
Revised: Mask R-CNN (He et al., 2017) is a strong baseline for core vision tasks.

Original: U NIFIED -IO XL outperforms Mask R-CNN on localization and segmentation.
Revised: U NIFIED -IO XL outperforms Mask R-CNN on localization and segmentation.

Original: The reason is U NIFIED -IO XL shows little degradation in performance between  same concept and  new concept as discussed in Appendix A.5.
Revised: The reason is U NIFIED -IO XL shows little degradation in performance between the same concept and the new concept as discussed in Appendix A.5.

Original: On keypoint, our model is worse compared to Mask R-CNN (68.1 vs . 70.8).
Revised: On keypoint, our model is worse compared to Mask R-CNN (68.1 vs . 70.8).

Original: The reason is we have 2-stage inference for keypoint – first locate the person using the object localization prompt, then find keypoints for each detected region.
Revised: The reason is we have 2-stage inference for keypoint – first locate the person using the object localization prompt, then find keypoints for each detected region.

Original: NLL-AngMF (Bae et al., 2021) is a SOTA model for surface normal estimation.
Revised: NLL-AngMF (Bae et al., 2021) is a SOTA model for surface normal estimation.

Original: Our model gets strong results compared to NLL-AngMF (44.3 vs . 49.6).
Revised: Our model gets strong results compared to NLL-AngMF (44.3 vs . 49.6).

Original: Since our image tokenizer is only pretrained on ImageNet without any surface normal data, the upper bound of our method through reconstruction is 59.8 on FrameNet (Kazemzadeh et al., 2014).
Revised: Since our image tokenizer is only pretrained on ImageNet without any surface normal data, the upper bound of our method through reconstruction is 59.8 on FrameNet (Kazemzadeh et al., 2014).

Original: This suggests our score could be considerably improved by training a stronger image tokenizer.
Revised: This suggests our score could be considerably improved by training a stronger image tokenizer.

Original: 4. A BLATIONS
Revised: 4. A BLATIONS

Original: To better understand how multi-tasking affects learning, we perform ablations by leaving out individual task groups from multi-task training.
Revised: To better understand how multi-tasking affects learning, we perform ablations by leaving out individual task groups from multi-task training.

Original: Due to computational constraints, we ablate
Revised: Due to computational constraints, we ablate

Original: U NIFIED -IO LARGE and train for 250k steps.
Revised: U NIFIED -IO LARGE and train for 250k steps.

Original: If ablating a task group, we reduce the number of training steps so that all models are trained on approximately the same number of examples for each of the remaining task groups.
Revised: If ablating a task group, we reduce the number of training steps so that all models are trained on approximately the same number of examples for each of the remaining task groups.

Original: Results are shown in Table 4 on GRIT and MNLI (Williams et al., 2018).
Revised: Results are shown in Table 4 on GRIT and MNLI (Williams et al., 2018).

Original: In spite of supporting a large number of heterogeneous tasks, Unified-IO is able to perform well across all tasks.
Revised: In spite of supporting a large number of heterogeneous tasks, Unified-IO is able to perform well across all tasks.

Original: Reducing this heterogeneity by removing task groups does not impact the performance of individual tasks significantly.
Revised: Reducing this heterogeneity by removing task groups does not impact the performance of individual tasks significantly.

Original: This is notable since removing a task group significantly reduces the scope of what a model needs to learn while keeping the model capacity fixed.
Revised: This is notable since removing a task group significantly reduces the scope of what a model needs to learn while keeping the model capacity fixed.

Original: This empirically demonstrates the effectiveness of the proposed unified architecture for massive heterogeneous task support.
Revised: This empirically demonstrates the effectiveness of the proposed unified architecture for massive heterogeneous task support.

Original: An exception is that removing the NLP group significantly boosts categorization, which might indicate that the sentence classification task interferes with image classification.
Revised: An exception is that removing the NLP group significantly boosts categorization, which might indicate that the sentence classification task interferes with image classification.

Original: Removing captioning also boosts performances on VQA and a few other tasks, which might be caused by captioning requiring a relatively large amount of model capacity to learn free-form text generation, in contrast to VQA that requires short answer phrases from a limited vocabulary.
Revised: Removing captioning also boosts performances on VQA and a few other tasks, which might be caused by captioning requiring a relatively large amount of model capacity to learn free-form text generation, in contrast to VQA that requires short answer phrases from a limited vocabulary.

Original: Removing image synthesis causes a major regression in keypoint.
Revised: Removing image synthesis causes a major regression in keypoint.

Original: Manual inspection shows that the model predicts standinghuman shaped keypoints even for people in very different postures, suggesting the model learned to rely on priors instead of the image content.
Revised: Manual inspection shows that the model predicts standinghuman shaped keypoints even for people in very different postures, suggesting the model learned to rely on priors instead of the image content.

Original: We also see minor regressions in localization and referring expression, suggesting that image synthesis tasks, possibly image in-painting in particular, had a surprising positive transfer to understanding sparse structured outputs.
Revised: We also see minor regressions in localization and referring expression, suggesting that image synthesis tasks, possibly image in-painting in particular, had a surprising positive transfer to understanding sparse structured outputs.

Original: It is possible that an ablation analysis on the XL model may yield different outcomes, but we are unable to perform an
Revised: It is possible that an ablation analysis on the XL model may yield different outcomes, but we are unable to perform an

Original: XL -based analysis due to limited compute.
Revised: XL -based analysis due to limited compute.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. R ESULTS ON A DDITIONAL T ASKS
Revised: 4. R ESULTS ON A DDITIONAL T ASKS

Original: We report results on 16 additional tasks used in our training setup.
Revised: We report results on 16 additional tasks used in our training setup.

Original: For these tasks, we do not expect to get state-of-the-art results since specialized models are usually designed and hyper-parameter tuned for a single task while we are evaluating a single jointly trained model.
Revised: For these tasks, we do not expect to get state-of-the-art results since specialized models are usually designed and hyper-parameter tuned for a single task, while we are evaluating a single jointly trained model.

Original: We also avoid extensive task-specific tricks like color jittering, horizontal flipping, CIDEr optimization and label smoothing, which are often responsible for considerable gains in individual task performance.
Revised: We also avoid extensive task-specific tricks like color jittering, horizontal flipping, CIDEr optimization and label smoothing, which are often responsible for considerable gains in individual task performance.

Original: We leave such task-specific tuning for future work.
Revised: We leave such task-specific tuning for future work.

Original: See Table 5 for the results.
Revised: See Table 5 for the results.

Original: When possible, we additionally report the best prior result on these tasks from a unified model, meaning a model that is trained in a multi-task setting and a unified architecture (no task-specific head or customizations) with at least three other tasks.
Revised: When possible, we additionally report the best prior result on these tasks from a unified model, meaning a model that is trained in a multi-task setting and a unified architecture (no task-specific head or customizations) with at least three other tasks.

Original: U NIFIED -IO provides strong performance on all these tasks despite being massively multi-tasked.
Revised: U NIFIED -IO provides strong performance on all these tasks despite being massively multi-tasked.

Original: We review more fine-grained results below.
Revised: We review more fine-grained results below.

Original: Depth Estimation.
Revised: Depth Estimation.

Original: On depth estimation, U NIFIED -IO achieves 0.385 rmse, which is behind
Revised: On depth estimation, U NIFIED -IO achieves 0.475 rmse, which is behind

Original: SOTA (Li et al., 2022e) but ahead of the recently proposed unified model, UViM (Kolesnikov et al., 2022), despite being trained to do far more tasks.
Revised: SOTA (Li et al., 2022e) but similar the recently proposed unified model, UViM (Kolesnikov et al., 2022), despite being trained to do far more tasks.

Revised: More discussion can be found in A.8.

Original: Image Classification.
Revised: Image Classification.

Original: U NIFIED -IO achieves 79.1 on ImageNet and 53.2 on Places365, showing the model was able to retain the knowledge of many fine-grained classes despite being massively multi-tasked.
Revised: U NIFIED -IO achieves 79.1 on ImageNet and 53.2 on Places365, showing the model was able to retain the knowledge of many fine-grained classes despite being massively multi-tasked.

Original: Notably, we achieve this without the extensive data augmentations methods typically used by SOTA models (Yu et al., 2022a;
Revised: Notably, we achieve this without the extensive data augmentations methods typically used by SOTA models (Yu et al., 2022a;

Original: He et al., 2022).
Revised: He et al., 2022).

Original: Visual Question Answering.
Revised: Visual Question Answering.

Original: U NIFIED -IO is competitive with fine-tuned models on VQA (Alayrac et al., 2022; Kamath et al., 2022; Gui et al., 2021), and achieves SOTA results on A-OKVQA.
Revised: U NIFIED -IO is competitive with fine-tuned models on VQA (Alayrac et al., 2022; Kamath et al., 2022; Gui et al., 2021), and achieves SOTA results on A-OKVQA.

Original: Relative to Flamingo, U NIFIED -IO performs better on VizWiz-QA but worse on OK-VQA.
Revised: Relative to Flamingo, U NIFIED -IO performs better on VizWiz-QA but worse on OK-VQA.

Original: Image Captioning.
Revised: Image Captioning.

Original: Despite the lack of CIDEr optimization, U NIFIED -IO is a strong captioning model and generalizes well to nocaps.
Revised: Despite the lack of CIDEr optimization, U NIFIED -IO is a strong captioning model and generalizes well to nocaps.

Original: Since U NIFIED -IO is trained on many captioning datasets, it is likely the use of style tags following Cornia et al.
Revised: Since U NIFIED -IO is trained on many captioning datasets, it is likely the use of style tags following Cornia et al.

Original: (2021) would offer additional improvement by signaling U NIFIED -IO to specifically generate COCO-style captions during inference.
Revised: (2021) would offer additional improvement by signaling U NIFIED -IO to specifically generate COCO-style captions during inference.

Original: NLP tasks.
Revised: NLP tasks.

Original: : U NIFIED -IO achieves respectable results on three NLP tasks but lags behind SOTA models (Smith et al., 2022; Zoph et al., 2022; He et al., 2021).
Revised: : U NIFIED -IO achieves respectable results on three NLP tasks but lags behind SOTA models (Smith et al., 2022; Zoph et al., 2022; He et al., 2021).

Original: This can partly be attributed to scale.
Revised: This can partly be attributed to scale.

Original: Modern NLP models contain 100 billion+ parameters and with more extensive NLP pre-training.
Revised: Modern NLP models contain 100 billion+ parameters and with more extensive NLP pre-training.

Original: 4. L IMITATIONS
Revised: L IMITATIONS

Original: For object detection, while U NIFIED -IO generally produces accurate outputs (see Appendix A.8), we find the recall is often poor in cluttered images.
Revised: For object detection, while U NIFIED -IO generally produces accurate outputs (see Appendix A.9), we find the recall is often poor in cluttered images.

Original: Prior work (Chen et al., 2022b) has shown this can be overcome with extensive data augmentation techniques, but these methods are not currently integrated into U NIFIED
Revised: Prior work (Chen et al., 2022b) has shown this can be overcome with extensive data augmentation techniques, but these methods are not currently integrated into U NIFIED

Original: -IO.
Revised: -IO.

Original: Our use of a pre-trained VQ-GAN greatly simplifies our training and is surprisingly effective for dense prediction tasks.
Revised: Our use of a pre-trained VQ-GAN greatly simplifies our training and is surprisingly effective for dense prediction tasks.

Original: However, it does mean U NIFIED -IO has limited image generation capabilities (recent works (Yu et al., 2022b) have shown this method can be greatly improved but was not available at the time of development).
Revised: However, it does mean U NIFIED -IO has limited image generation capabilities (recent works (Yu et al., 2022b) have shown this method can be greatly improved but was not available at the time of development).

Original: We also found in a small-scale study that our model does not always understand prompts not in the training data (see Appendix A.6).
Revised: We also found in a small-scale study that our model does not always understand prompts not in the training data (see Appendix A.6).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Constructing models that can learn to solve many different tasks has been of long-standing interest to researchers.
Revised: Constructing models that can learn to solve many different tasks has been of long-standing interest to researchers.

Original: A traditional approach to this problem is to build models with task-specialized heads on top of shared backbones
Revised: A traditional approach to this problem is to build models with task-specialized heads on top of shared backbones

Original: (He et al., 2017; Liu et al., 2019; Lu et al., 2020).
Revised: (He et al., 2017; Liu et al., 2019; Lu et al., 2020).

Original: However, this requires manually designing a specialized head for each task and potentially limits transfer across tasks.
Revised: However, this requires manually designing a specialized head for each task and potentially limits transfer across tasks.

Original: An alternative is to build unified models – models that can complete many different tasks without taskspecialized components.
Revised: An alternative is to build unified models – models that can complete many different tasks without taskspecialized components.

Original: In NLP, this approach has achieved a great deal of success through the use of pre-trained generative models (Raffel et al., 2020; Brown et al., 2020; Chowdhery et al., 2022).
Revised: In NLP, this approach has achieved a great deal of success through the use of pre-trained generative models (Raffel et al., 2020; Brown et al., 2020; Chowdhery et al., 2022).

Original: Inspired by this success, there has been a recent trend to build unified models that can be additionally applied to tasks with visual or structured inputs and outputs.
Revised: Inspired by this success, there has been a recent trend to build unified models that can be additionally applied to tasks with visual or structured inputs and outputs.

Original: Many models have been proposed for tasks with text and/or image input and text output (Cho et al., 2021;
Revised: Many models have been proposed for tasks with text and/or image input and text output (Cho et al., 2021;

Original: Wang et al., 2022d; Li et al., 2022b;
Revised: Wang et al., 2022d; Li et al., 2022b;

Original: Wang et al., 2021; Kaiser et al., 2017; Sun et al., 2022; Chen et al., 2022d; Wang et al., 2022c).
Revised: Wang et al., 2021; Kaiser et al., 2017; Sun et al., 2022; Chen et al., 2022d; Wang et al., 2022c).

Original: However, these models can not produce any structured or visual output.
Revised: However, these models can not produce any structured or visual output.

Original: More recent unified models can additionally support image locations, which allows tasks like object detection or region captioning.
Revised: More recent unified models can additionally support image locations, which allows tasks like object detection or region captioning.

Original: This can be done by using bounding boxes proposed by an object detector (Cho et al., 2021; Kamath et al., 2022) or including a bounding box output head (Gupta et al., 2022a; Dou et al., 2022; Chen et al., 2022c; Kamath et al., 2021; Li et al., 2022d).
Revised: This can be done by using bounding boxes proposed by an object detector (Cho et al., 2021; Kamath et al., 2022) or including a bounding box output head (Gupta et al., 2022a; Dou et al., 2022; Chen et al., 2022c; Kamath et al., 2021; Li et al., 2022d).

Original: Alternatively, image locations can be encoded as special tokens in the input/output text (Yang et al., 2021;
Revised: Alternatively, image locations can be encoded as special tokens in the input/output text (Yang et al., 2021;

Original: Yao et al., 2022; Zhu et al., 2022) following Chen et al.
Revised: Yao et al., 2022; Zhu et al., 2022) following Chen et al.

Original: (2022b).
Revised: (2022b).

Original: U NIFIED -IO follows this design, but applies it to a wider set of tasks than previous works, including key-point estimation, image in-painting, and region captioning.
Revised: U NIFIED -IO follows this design, but applies it to a wider set of tasks than previous works, including key-point estimation, image in-painting, and region captioning.

Original: Concurrent to our work, OFA (Wang et al., 2022b) proposes a similar approach that also supports image locations and text-to-image synthesis.
Revised: Concurrent to our work, OFA (Wang et al., 2022b) proposes a similar approach that also supports image locations and text-to-image synthesis.

Original: However, OFA does not support dense labeling tasks such as depth estimation, segmentation, and surface normal estimation.
Revised: However, OFA does not support dense labeling tasks such as depth estimation, segmentation, and surface normal estimation.

Original: Other closely related models include UViM (Kolesnikov et al., 2022) which generates a discrete guiding code for a D-VAE to build an autoregressive model for panoptic segmentation, depth prediction, and colorization; Pix2Seq v (Chen et al., 2022c) which extends Pix2Seq to segmentation, keypoint estimation, and image captioning; Visual Prompting (Bar et al., 2022) adapts the pre-trained visual model to novel downstream tasks by image inpainting.
Revised: Other closely related models include UViM (Kolesnikov et al., 2022) which generates a discrete guiding code for a D-VAE to build an autoregressive model for panoptic segmentation, depth prediction, and colorization; Pix2Seq v (Chen et al., 2022c) which extends Pix2Seq to segmentation, keypoint estimation, and image captioning; Visual Prompting (Bar et al., 2022) adapts the pre-trained visual model to novel downstream tasks by image inpainting.

Original: U NIFIED -IO covers all these tasks, and focuses on multi-tasking rather then task-specific fine-tuning.
Revised: U NIFIED -IO covers all these tasks, and focuses on multi-tasking rather then task-specific fine-tuning.

Original: Additional discussions are presented in Appendix A.9.
Revised: Additional discussions are presented in Appendix A.10.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have presented U NIFIED -IO, a unified architecture that supports a large variety of computer vision and NLP tasks with diverse inputs and outputs, including images, continuous maps, binary masks, segmentation masks, text, bounding boxes, and keypoints.
Revised: We have presented U NIFIED -IO, a unified architecture that supports a large variety of computer vision and NLP tasks with diverse inputs and outputs, including images, continuous maps, binary masks, segmentation masks, text, bounding boxes, and keypoints.

Original: This unification is made possible by homogenizing each of these modalities into a sequence of discrete tokens.
Revised: This unification is made possible by homogenizing each of these modalities into a sequence of discrete tokens.

Original: The 2.9B parameter
Revised: The 2.9B parameter

Original: U NIFIED -IO XL model is jointly trained on 90+ datasets, is the first model to perform all 7 tasks on the GRIT benchmark and obtains impressive results across 16 other vision and NLP benchmarks, with no benchmark fine-tuning or task-specific modifications.
Revised: U NIFIED -IO XL model is jointly trained on 90+ datasets, is the first model to perform all 7 tasks on the GRIT benchmark and obtains impressive results across 16 other vision and NLP benchmarks, with no benchmark fine-tuning or task-specific modifications.

Original: A A PPENDIX A. T ASKS D ETAILS
Revised: A A PPENDIX A. T ASKS D ETAILS

Original: U NIFIED -IO is jointly trained on a large and diverse set of vision, language and vision & language tasks.
Revised: U NIFIED -IO is jointly trained on a large and diverse set of vision, language and vision & language tasks.

Original: In this section, we describe these tasks in detail and show the prompts we use during training and inference (text on the left of example cards).
Revised: In this section, we describe these tasks in detail and show the prompts we use during training and inference (text on the left of example cards).

Original: We also provide qualitative examples of both the ground truth and the predictions made by U NIFIED -IO.
Revised: We also provide qualitative examples of both the ground truth and the predictions made by U NIFIED -IO.

Original: A.1. I MAGE S YNTHESIS T ASKS
Revised: A.1. I MAGE S YNTHESIS T ASKS

Original: Image Synthesis from Text.
Revised: Image Synthesis from Text.

Original: This task requires generating an image that matches a sentence.
Revised: This task requires generating an image that matches a sentence.

Original: Training data comes from 4 captioning datasets: COCO Caption (Chen et al., 2015), Conceptual Captions 3M and 12M (Changpinyo et al., 2021), and RedCaps (Desai et al., 2021) as well datasets used for image classification using the object class as the input caption.
Revised: Training data comes from 4 captioning datasets: COCO Caption (Chen et al., 2015), Conceptual Captions 3M and 12M (Changpinyo et al., 2021), and RedCaps (Desai et al., 2021) as well datasets used for image classification using the object class as the input caption.

Original: Specialized image generation models like DALL·E 2 (Ramesh et al., 2022) use an order of magnitude more data, but we limit our sources to these sets for training efficiency.
Revised: Specialized image generation models like DALL·E 2 (Ramesh et al., 2022) use an order of magnitude more data, but we limit our sources to these sets for training efficiency.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: U

Original: [Figure]
Revised: [Figure]

Original: N
Revised: H

Revised: P R

Original: [Figure]
Revised: [Figure]

Original: RA
Revised: D I C T I O N

Original: [Figure]
Revised: [Figure]

Revised: RA

Original: [Table]
Revised: [Table]

Original: O
Revised: O

Original: N
Revised: N

Original: Image Inpainting.
Revised: Image Inpainting.

Original: This task requires filling in a region of an image with a target object.
Revised: This task requires filling in a region of an image with a target object.

Original: Training data for this task is built from object bounding box annotations from Open Images (Kuznetsova et al., 2020), Visual Genome (Krishna et al., 2017) and COCO (Lin et al., 2014).
Revised: Training data for this task is built from object bounding box annotations from Open Images (Kuznetsova et al., 2020), Visual Genome (Krishna et al., 2017) and COCO (Lin et al., 2014).

Original: For each object, the input image becomes the source image with the object’s bounding box blanked out.
Revised: For each object, the input image becomes the source image with the object’s bounding box blanked out.

Original: The input prompt provides the bounding box’s location and the target category.
Revised: The input prompt provides the bounding box’s location and the target category.

Original: The target output is the original image.
Revised: The target output is the original image.

Original: [Figure]
Revised: [Figure]

Revised: N

Original:  [Table]
Revised: [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: Image Synthesis from Segmentation.
Revised: Image Synthesis from Segmentation.

Original: This task involves generating an image that matches an input semantic segmentation, i.e., a set of class labels for some or all of the pixels in the image.
Revised: This task involves generating an image that matches an input semantic segmentation, i.e., a set of class labels for some or all of the pixels in the image.

Original: U NIFIED -IO is trained for this task using segmentation annotations from COCO (Lin et al., 2014),
Revised: U NIFIED -IO is trained for this task using segmentation annotations from COCO (Lin et al., 2014),

Original: Open Images (Kuznetsova et al., 2020), and LVIS (Gupta et al., 2019) as input.
Revised: Open Images (Kuznetsova et al., 2020), and LVIS (Gupta et al., 2019) as input.

Original: Following the method from Section 3.1 the segmentation input is converted into a RGB image paired with a prompt listing the color-to-class mapping, and the target output is the source image.
Revised: Following the method from Section 3.1 the segmentation input is converted into a RGB image paired with a prompt listing the color-to-class mapping, and the target output is the source image.

Original: T R U T
Revised: T R U T

Original: H
Revised: H

Original: P R E D
Revised: P R E D

Original: I C T
Revised: I C T

Original: I O
Revised: I O

Original: N
Revised: N

Revised: [Figure]

Revised: SE

Original: [Figure]
Revised: [Figure]

Revised: BA SE

Original: D I M A
Revised: D I M A

Original: G E
Revised: G E

Original: G E N E
Revised: G E N E

Original: RA T
Revised: RA T

Original: I O
Revised: I O

Original: N
Revised: N

Original: A.1. S PARSE L ABELLING T ASKS
Revised: A.1. S PARSE L ABELLING T ASKS

Original: Object Detection.
Revised: Object Detection.

Original: U NIFIED -IO is trained on object detection annotations from Visual Genome,
Revised: U NIFIED -IO is trained on object detection annotations from Visual Genome,

Original: Open Images, and COCO.
Revised: Open Images, and COCO.

Original: For this task the input is a static prompt and an image, and the output text includes the bounding boxes and class names of all objects in the image.
Revised: For this task the input is a static prompt and an image, and the output text includes the bounding boxes and class names of all objects in the image.

Original: We randomize the order of the output objects during training, but for simplicity leave integrating more complex dataaugmentation techniques (Chen et al., 2022b) to future work.[Table] I [Table]
Revised: We randomize the order of the output objects during training, but for simplicity leave integrating more complex dataaugmentation techniques (Chen et al., 2022b) to future work.

Original: J E

Original: C

Original: [Table]

Original: C T

Original: I O

Original: N

Original: [Table]
Revised: [Table]

Original: Object Localization.
Revised: Object Localization.

Original: Object localization requires returning bounding boxes around all objects of a given category.
Revised: Object localization requires returning bounding boxes around all objects of a given category.

Original: Training data is derived from our object detection training data by constructing a training example from each category of objects present in an image.
Revised: Training data is derived from our object detection training data by constructing a training example from each category of objects present in an image.

Original: The input is then the image, a prompt specifying the target class, and the output is a list of all boxes that contain an instance of that class.
Revised: The input is then the image, a prompt specifying the target class, and the output is a list of all boxes that contain an instance of that class.

Original: The class for each box (which is always the class specified in the prompt) is included in the output for the sake of keeping the output format consistent with the object detection output.
Revised: The class for each box (which is always the class specified in the prompt) is included in the output for the sake of keeping the output format consistent with the object detection output.

Original: Object localization can use input categories which are not present in the image.
Revised: Object localization can use input categories which are not present in the image.

Original: To handle this, we construct negative samples by randomly selecting categories not present in the image to use as input, in which case the output is an empty sequence.[Table] U [Table]
Revised: To handle this, we construct negative samples by randomly selecting categories not present in the image to use as input, in which case the output is an empty sequence.

Original: R E

Original: [Table]

Original: I C T

Original: I O

Original: N

Original: [Table]
Revised: [Table]

Original: Referring Expression Comprehension.
Revised: Referring Expression Comprehension.

Original: The task requires the model to localize an image region described by a natural language expression.
Revised: The task requires the model to localize an image region described by a natural language expression.

Original: The annotation is similar to Object Localization, except that the target is specified with natural language expression instead of class name.
Revised: The annotation is similar to Object Localization, except that the target is specified with natural language expression instead of class name.

Original: Datasets for this task include RefCOCO (Kazemzadeh et al., 2014), RefCOCO+ (Kazemzadeh et al., 2014) and
Revised: Datasets for this task include RefCOCO (Kazemzadeh et al., 2014), RefCOCO+ (Kazemzadeh et al., 2014) and

Original: RefCOCOg
Revised: RefCOCOg

Original: (Mao et al., 2016).
Revised: (Mao et al., 2016).

Original: T R U T
Revised: T R U T

Original: H
Revised: H

Original: P R E D
Revised: P R E D

Original: I C T
Revised: I C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: R E F E
Revised: R E F E

Original: RR I N G
Revised: RR I N G

Original: EXP R
Revised: EXP R

Original: ESS
Revised: ESS

Original: I O
Revised: I O

Original: N S
Revised: N S

Original: [Figure]
Revised: [Table]

Original: Keypoint Estimation.
Revised: Keypoint Estimation.

Original: Keypoint estimation requires returning the location of 17 keypoints on a human body (e.g., eyes, nose, feet, etc.)
Revised: Keypoint estimation requires returning the location of 17 keypoints on a human body (e.g., eyes, nose, feet, etc.)

Original: for each person in an image.
Revised: for each person in an image.

Original: While it is possible to perform this task in one pass by listing the keypoints of all people in the image in a single output sequence, this can result in an extremely long output sequence, so U NIFIED -IO uses a multi-step approach instead.
Revised: While it is possible to perform this task in one pass by listing the keypoints of all people in the image in a single output sequence, this can result in an extremely long output sequence, so U NIFIED -IO uses a multi-step approach instead.

Original: To do this U NIFIED -IO is trained to complete the subtask of detecting the keypoints for single a person in a given region.
Revised: To do this U NIFIED -IO is trained to complete the subtask of detecting the keypoints for single a person in a given region.

Original: For this subtask, the input prompt specifies the target region and and the output is a list of 17 points (a pair of locations tokens for the x and y coordinates) along with a visibility labels (1 for not visible, 2 for partly visible, 3 for fully visible).
Revised: For this subtask, the input prompt specifies the target region and and the output is a list of 17 points (a pair of locations tokens for the x and y coordinates) along with a visibility labels (1 for not visible, 2 for partly visible, 3 for fully visible).

Original: Non-visible points are preceded by two copies of a new special tokens that indicate there are no valid coordinates.
Revised: Non-visible points are preceded by two copies of a new special tokens that indicate there are no valid coordinates.

Original: The keypoint metric does not award points for correctly identifying non-visible points, so during inference we mask that special token so the model makes a best-effort guess for the coordinates of every single point.
Revised: The keypoint metric does not award points for correctly identifying non-visible points, so during inference we mask that special token so the model makes a best-effort guess for the coordinates of every single point.

Original: Training data for this subtask comes from COCO human pose data (Lin et al., 2014) with the ground-truth person regions as input.
Revised: Training data for this subtask comes from COCO human pose data (Lin et al., 2014) with the ground-truth person regions as input.

Original: During inference we locate person regions using the object localization prompt, then apply U NIFIED -IO again to find keypoints for each detected region.T R U T H P R E D
Revised: During inference we locate person regions using the object localization prompt, then apply U NIFIED -IO again to find keypoints for each detected region.

Original: I C T I O N
Revised: [Table]

Original: K EYP
Revised: K EYP

Original: O I N T
Revised: O I N T

Original: ES T M
Revised: ES T M

Original: A T I
Revised: A T I

Original: O
Revised: O

Original: N
Revised: N

Original: Find

Original: [Table]
Revised: [Table]

Original: A.1. D ENSE L ABELLING T ASKS
Revised: A.1. D ENSE L ABELLING T ASKS

Original: Object Segmentation.
Revised: Object Segmentation.

Original: Object segmentation requires finding the binary segmentation mask of each instance of a particular category in an image.
Revised: Object segmentation requires finding the binary segmentation mask of each instance of a particular category in an image.

Original: The input is an image and a prompt that includes the target class, while the output is an RGB image with black background and instances of that class filled in with unique colors following the method in Section 3.1.
Revised: The input is an image and a prompt that includes the target class, while the output is an RGB image with black background and instances of that class filled in with unique colors following the method in Section 3.1.

Original: The output image is resized to match the input image if needed using a nearest-neighbor resizing method, and binary masks are built from each unique color.
Revised: The output image is resized to match the input image if needed using a nearest-neighbor resizing method, and binary masks are built from each unique color.

Original: In practice the output image from U NIFIED -IO can have slightly nonuniform colors or extraneous background pixels, likely due to limitation in what the D-VAE can decode/encode, so the output pixels are clustered by color and and connected components of less than 8 pixels are removed to build cleaned instance masks.
Revised: In practice the output image from U NIFIED -IO can have slightly nonuniform colors or extraneous background pixels, likely due to limitation in what the D-VAE can decode/encode, so the output pixels are clustered by color and and connected components of less than 8 pixels are removed to build cleaned instance masks.

Original: Segmentation annotations come from
Revised: Segmentation annotations come from

Original: Open Images LVIS, and COCO.
Revised: Open Images LVIS, and COCO.

Original: What is [Table]
Revised: What is the segmentation of “apple” ?

Original: T R U T
Revised: T R U T

Original: H
Revised: H

Original: P R E D
Revised: P R E D

Original: I C T
Revised: I C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: O B J E
Revised: O B J E

Original: C T
Revised: C T

Original: SE G M
Revised: SE G M

Original: E N [Table] A
Revised: E N T A

Original: T I [Table] N
Revised: T I O N

Original: Depth Estimation.
Revised: Depth Estimation.

Original: Depth estimation requires assigning each pixel in an image a depth value.
Revised: Depth estimation requires assigning each pixel in an image a depth value.

Original: This task uses a static prompt as input, and the output is a grayscale image representing the normalized depth at each pixel.
Revised: This task uses a static prompt as input, and the output is a grayscale image representing the normalized depth at each pixel.

Original: The generated output image is reiszed to the same size as the input image and then pixel values are rescaled to the maximum depth in the training to get an output depth map.
Revised: The generated output image is reiszed to the same size as the input image and then pixel values are rescaled to the maximum depth in the training to get an output depth map.

Original: Training data comes from the NYU Depth Dataset V2 (Nathan Silberman & Fergus, 2012).
Revised: Training data comes from the NYU Depth Dataset V2 (Nathan Silberman & Fergus, 2012).

Original: [Table]
Revised: [Table]

Original: T R U T
Revised: T R U T

Original: H
Revised: H

Original: P R E D
Revised: P R E D

Original: I C T
Revised: I C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: D EP T
Revised: D EP T

Original: H
Revised: H

Original: ES T
Revised: ES T

Original: I M
Revised: I M

Original: A T I
Revised: A T I

Original: O
Revised: O

Original: N
Revised: N

Original: Surface Normal Estimation.
Revised: Surface Normal Estimation.

Original: U NIFIED -IO is trained on FrameNet (Huang et al., 2019a) and BlendedMVS (Yao et al., 2020)
Revised: U NIFIED -IO is trained on FrameNet (Huang et al., 2019a) and BlendedMVS (Yao et al., 2020)

Original: surface normal estimation datasets.
Revised: surface normal estimation datasets.

Original: For this task the input is a static prompt and an image and the output is an RGB representation of the x/y/z orientation of the surface at each pixel.
Revised: For this task the input is a static prompt and an image and the output is an RGB representation of the x/y/z orientation of the surface at each pixel.

Original: The generated output image is resized to match the input image and converted back to x/y/z orientations to produce the final output.
Revised: The generated output image is resized to match the input image and converted back to x/y/z orientations to produce the final output.

Original: T R U T
Revised: T R U T

Original: H
Revised: H

Original: P R E D
Revised: P R E D

Original: I C T
Revised: I C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: S U R F
Revised: S U R F

Original: A C E
Revised: A C E

Original: N O R
Revised: N O R

Original: M A L
Revised: M A L

Original: ES T
Revised: ES T

Original: I M
Revised: I M

Original: A T I
Revised: A T I

Original: O
Revised: O

Original: N
Revised: N

Original: [Table]
Revised: [Table]

Original: A.1. I MAGE C LASSIFICATION T ASKS Image Classification.
Revised: A.1. I MAGE C LASSIFICATION T ASKS Image Classification.

Original: U NIFIED -IO is trained on 6 image classification datasets: ImageNet 2012 (Deng et al., 2009), ImageNet21k (Ridnik et al., 2021), Places365 (Zhou et al., 2017), Sun (Xiao et al., 2010), iNaturalist (Van Horn et al., 2018) and Caltech birds 2011 (Welinder et al., 2010).
Revised: U NIFIED -IO is trained on 6 image classification datasets: ImageNet 2012 (Deng et al., 2009), ImageNet21k (Ridnik et al., 2021), Places365 (Zhou et al., 2017), Sun (Xiao et al., 2010), iNaturalist (Van Horn et al., 2018) and Caltech birds 2011 (Welinder et al., 2010).

Original: For this task the input is an image and a static prompt, and the output is a class name.
Revised: For this task the input is an image and a static prompt, and the output is a class name.

Original: During inference we compute the log-probability of each class label in the dataset being evaluated and return the highest scoring one.
Revised: During inference we compute the log-probability of each class label in the dataset being evaluated and return the highest scoring one.

Original: This ensures U NIFIED
Revised: This ensures U NIFIED

Original: -IO does not return a category from a different categorization dataset that is a synonym or hypernym of the correct label.
Revised: -IO does not return a category from a different categorization dataset that is a synonym or hypernym of the correct label.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: T R U T
Revised: R U T

Original: H
Revised: H

Original: P R E D
Revised: P R E D

Original: I C T
Revised: I C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Object Categorization.
Revised: Object Categorization.

Original: This task identifies which label, from a given set, best corresponds to an image region defined by an input image and bounding box.
Revised: This task identifies which label, from a given set, best corresponds to an image region defined by an input image and bounding box.

Original: The input is the image, a prompt specifying the image region and the output is the target class name.
Revised: The input is the image, a prompt specifying the image region and the output is the target class name.

Original: We convert object detection annotations from Visual Genome, Open Images, and COCO for this task.
Revised: We convert object detection annotations from Visual Genome, Open Images, and COCO for this task.

Original: Inference is constrained to return a valid label for the target label set just as with image classification.
Revised: Inference is constrained to return a valid label for the target label set just as with image classification.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: A.1. I MAGE C APTIONING T ASKS
Revised: A.1. I MAGE C APTIONING T ASKS

Original: Image Captioning.
Revised: Image Captioning.

Original: Image captioning data comes from the same manually annotated and unsupervised sources used for Image Generation.
Revised: Image captioning data comes from the same manually annotated and unsupervised sources used for Image Generation.

Original: In this case the inputs and output are reversed, the input is an image and the static prompt, and the output is a caption that matches the image.
Revised: In this case the inputs and output are reversed, the input is an image and the static prompt, and the output is a caption that matches the image.

Original: [Table]
Revised: [Table]

Original:  U T
Revised: T R U T

Original: H
Revised: H

Original: P R E D
Revised: P R E D

Original: I C T
Revised: I C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: I M A G
Revised: I M A G

Original: E
Revised: E

Original: C A P T
Revised: C A P T

Original: I
Revised: I

Original: O N I N
Revised: O N I N

Original: G
Revised: G

Original: [Table]
Revised: [Table]

Original: Region Captioning.
Revised: Region Captioning.

Original: Region captioning tasks a model with generating a caption that describes a specific region in the image.
Revised: Region captioning tasks a model with generating a caption that describes a specific region in the image.

Original: Our format for this task is identical to Image Captioning except the region is included in the input prompt.
Revised: Our format for this task is identical to Image Captioning except the region is included in the input prompt.

Original: Visual Genome (Krishna et al., 2017) is used for the training data. 
Revised: Visual Genome (Krishna et al., 2017) is used for the training data. [Table] H [Table]

Revised: I

Original: [Table]
Revised: [Table]

Original: I
Revised: I

Original: [Table]
Revised: [Table]

Original: N
Revised: N

Original: R E G I
Revised: R E G I

Original: O N
Revised: O N

Original: C A P T
Revised: C A P T

Original: I
Revised: I

Original: O N I N
Revised: O N I N

Original: G
Revised: G

Original: [Table]
Revised: [Table]

Original: A.1. V ISION & L ANGUAGE T ASKS
Revised: A.1. V ISION & L ANGUAGE T ASKS

Original: Visual Question Answering.
Revised: Visual Question Answering.

Original: U NIFIED -IO is trained on a collection of VQA datasets including
Revised: U NIFIED -IO is trained on a collection of VQA datasets including

Original: VQA 2.0 (Goyal et al., 2017), Visual Genome, VizWizVQA (Gurari et al., 2018), OKVQA (Marino et al., 2019) and A-OKVQA (Schwenk et al., 2022).
Revised: VQA 2.0 (Goyal et al., 2017), Visual Genome, VizWizVQA (Gurari et al., 2018), OKVQA (Marino et al., 2019) and A-OKVQA (Schwenk et al., 2022).

Original: For VQA, the question is used as the prompt, and the output is the answer text.
Revised: For VQA, the question is used as the prompt, and the output is the answer text.

Original: For VQA, it is common to constrain the model to predict an answer from a fixed last of common VQA answers (Wang et al., 2022b;d) during inference, but we avoid doing this since we find it does not benefit U NIFIED -IO in practice.
Revised: For VQA, it is common to constrain the model to predict an answer from a fixed last of common VQA answers (Wang et al., 2022b;d) during inference, but we avoid doing this since we find it does not benefit U NIFIED -IO in practice.

Original: We additionally convert data from several other datasets in a VQA format, including imSitu (Yatskar et al., 2016), where we treat predicting the verb and then the related slots as separate VQA questions, VisualCOMMET (Park et al., 2020) where we convert the before/after/intent into questions by converting the input regions into location tokens, SNLI-VE
Revised: We additionally convert data from several other datasets in a VQA format, including imSitu (Yatskar et al., 2016), where we treat predicting the verb and then the related slots as separate VQA questions, VisualCOMMET (Park et al., 2020) where we convert the before/after/intent into questions by converting the input regions into location tokens, SNLI-VE

Original: (Xie et al., 2019) where we integrate the entailed text into an input question, and VCR (Zellers et al., 2019a) where we again integrate the input regions into the prompt by encoding them with location tokens and integrate the rationales into the target text for the answer justification task.
Revised: (Xie et al., 2019) where we integrate the entailed text into an input question, and VCR (Zellers et al., 2019a) where we again integrate the input regions into the prompt by encoding them with location tokens and integrate the rationales into the target text for the answer justification task.

Original: [Table]
Revised: [Table]

Original: T R U T
Revised: U [Table]

Original: H
Revised: H

Original: P R E D
Revised: P R E D

Original: I C T
Revised: I C T

Original: I O
Revised: I O

Original: N

Original: V I S U

Original: A L Q

Original: [Table]

Original: ES

Original: I

Original: [Table]

Original: N
Revised: N

Original: [Table]

Original: E

Original: [Table]

Original: I

Original: [Table]

Original: G

Original: [Table]
Revised: [Table]

Original: Answer-Grounded Visual Question Answering.
Revised: Answer-Grounded Visual Question Answering.

Original: This task requires both answering a question and returning a binary mask specifying the region of the image used to answer the question.
Revised: This task requires both answering a question and returning a binary mask specifying the region of the image used to answer the question.

Original: The format for this task follows the one for VQA except that a binary mask is also used as an additional output.
Revised: The format for this task follows the one for VQA except that a binary mask is also used as an additional output.

Original: Training data comes from VizWiz-VQA (Chen et al., 2022a), a dataset designed to train models that could benefit people with visual impairments. 
Revised: Training data comes from VizWiz-VQA (Chen et al., 2022a), a dataset designed to train models that could benefit people with visual impairments. [Figure] [Table] N

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: Relationship Detection.
Revised: Relationship Detection.

Original: This task requires predicating a relationship between a pair of objects which are grounded by bounding boxes.
Revised: This task requires predicating a relationship between a pair of objects which are grounded by bounding boxes.

Original: The prompt contains both the object regions, and the output is the predicted predicate.
Revised: The prompt contains both the object regions, and the output is the predicted predicate.

Original: There are 2 datasets in this tasks: Visual Genome (Krishna et al., 2017) and Open Images (Kuznetsova et al., 2020). 
Revised: There are 2 datasets in this tasks: Visual Genome (Krishna et al., 2017) and Open Images (Kuznetsova et al., 2020). [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: A.1. N ATURAL L ANGUAGE P ROCESSING T ASKS
Revised: A.1. N ATURAL L ANGUAGE P ROCESSING T ASKS

Original: Question Answering.
Revised: Question Answering.

Original: Following prior work in natural language processing (Raffel et al., 2020), QA tasks are formatted by placing both the question and any text context (e.g., an paragraph containing the answer) into the prompt and training the model to generate the text answer.
Revised: Following prior work in natural language processing (Raffel et al., 2020), QA tasks are formatted by placing both the question and any text context (e.g., an paragraph containing the answer) into the prompt and training the model to generate the text answer.

Original: U NIFIED -IO is trained on several QA datasets including SQuAD 2.0 (Rajpurkar et al., 2016), other training datasets from the MRQA (Fisch et al., 2019) shared tasks (Trischler et al., 2017; Joshi et al., 2017; Dunn et al., 2017; Yang et al., 2018; Kwiatkowski et al., 2019), QA datasets from SuperGLUE (Wang et al., 2019;
Revised: U NIFIED -IO is trained on several QA datasets including SQuAD 2.0 (Rajpurkar et al., 2016), other training datasets from the MRQA (Fisch et al., 2019) shared tasks (Trischler et al., 2017; Joshi et al., 2017; Dunn et al., 2017; Yang et al., 2018; Kwiatkowski et al., 2019), QA datasets from SuperGLUE (Wang et al., 2019;

Original: Clark et al., 2019; Khashabi et al., 2018; Roemmele et al., 2011), Cosmos QA (Huang et al., 2019b), OpenBookQA (Mihaylov et al., 2018), and HellaSwag (Zellers et al., 2019b).
Revised: Clark et al., 2019; Khashabi et al., 2018; Roemmele et al., 2011), Cosmos QA (Huang et al., 2019b), OpenBookQA (Mihaylov et al., 2018), and HellaSwag (Zellers et al., 2019b).

Original: If the text context is longer then our maximum sequence length we use a sliding-window approach following Devlin et al.
Revised: If the text context is longer then our maximum sequence length we use a sliding-window approach following Devlin et al.

Original: (2019) which exposes the model to different windows of text from the context and returns the highest-confidence answer.
Revised: (2019) which exposes the model to different windows of text from the context and returns the highest-confidence answer.

Original: [Table]
Revised: [Table]

Revised: A

Revised: [Table]

Original: Text Classification.
Revised: Text Classification.

Original: Also following past work (Raffel et al., 2020), text classification tasks are formatted by placing the input sentences and a query in the prompt and training the model to generate the target class.
Revised: Also following past work (Raffel et al., 2020), text classification tasks are formatted by placing the input sentences and a query in the prompt and training the model to generate the target class.

Original: Datasets include tasks from GLUE and SuperGLUE (Wang et al., 2018; 2019;
Revised: Datasets include tasks from GLUE and SuperGLUE (Wang et al., 2018; 2019;

Original: Warstadt et al., 2018;
Revised: Warstadt et al., 2018;

Original: Socher et al., 2013; Dolan & Brockett, 2005; Iyer et al., 2017; Cer et al., 2017;
Revised: Socher et al., 2013; Dolan & Brockett, 2005; Iyer et al., 2017; Cer et al., 2017;

Original: Williams et al., 2018; Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009; Levesque et al., 2012; Williams et al., 2018; De Marneff et al., 2019; Pilehvar & os’e
Revised: Williams et al., 2018; Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009; Levesque et al., 2012; Williams et al., 2018; De Marneff et al., 2019; Pilehvar & os’e

Original: Camacho-Collados, 2018), as well as SNLI (Bowman et al., 2015), SciTail (Khot et al., 2018),
Revised: Camacho-Collados, 2018), as well as SNLI (Bowman et al., 2015), SciTail (Khot et al., 2018),

Original: IMDB Reviews (Maas et al., 2011), and PAWS (Zhang et al., 2019). 
Revised: IMDB Reviews (Maas et al., 2011), and PAWS (Zhang et al., 2019). T R U T

Original: [Figure]
Revised: H

Original: [Table]
Revised: [Table]

Original: I
Revised: I

Original: [Table]
Revised: [Table]

Revised: N

Original:  Text  data.
Revised: [Table] Text Summerization. Text summarization is done again by providing the input paragraph and a prompt as input and generating a summary as output. We use the Gigaword dataset (Graff et al., 2003; Rush et al., 2015) for training data.

Original: T R U T H P
Revised: T R U T H P

Original: R E D
Revised: R E D

Original: I C T
Revised: I C T

Original: I O N T R U T H P
Revised: I O N T R U T H P

Original: R E D
Revised: R E D

Original: I C T
Revised: I C T

Original: I O N
Revised: I O N

Original: P
Revised: “apple” [Table]

Original: Which regions does the text “giraffe” describe ?
Revised: Which regions does the text “giraffe” describe ?

Original: T R U T H P
Revised: T R U T H P

Original: R E D
Revised: R E D

Original: I C T
Revised: I C T

Original: I O N Which region does the text “man in solid black” describe ?
Revised: I O N Which region does the text “man in solid black” describe ?

Original: T R U T H P
Revised: T R U T H P

Original: R E D
Revised: R E D

Original: I C T
Revised: I C T

Original: I O N O B J E C T
Revised: I O N O B J E C T

Original: L O C A L I Z
Revised: L O C A L I Z

Original: A T I O N R E F E RR
Revised: A T I O N R E F E RR

Original: I N G EXP R
Revised: I N G EXP R

Original: ESS
Revised: ESS

Original: I O N S S
Revised: I O N S S

Original: U R F
Revised: U R F

Original: A C E
Revised: A C E

Original: N O R M
Revised: N O R M

Original: A L ES T
Revised: A L ES T

Original: I M A T I O N K EYP
Revised: I M A T I O N K EYP

Original: O I N T ES T M A T
Revised: O I N T ES T M A T

Original: I O N
Revised: I O N

Original: What is the surface normal of the image ?
Revised: What is the surface normal of the image ?

Original: SE
Revised: SE

Original: [Table]
Revised: [Table]

Original: A.1. L ANGUAGE M ODELING T ASKS
Revised: A.1. L ANGUAGE M ODELING T ASKS

Original: Mask Language Modeling.
Revised: Mask Language Modeling.

Original: Following T5 (Raffel et al., 2020), the mask language modelling objective randomly samples and then drops out 15% of tokens in the input sequence.
Revised: Following T5 (Raffel et al., 2020), the mask language modelling objective randomly samples and then drops out 15% of tokens in the input sequence.

Original: All consecutive spans of dropped-out tokens are replaced by a single sentinel token.
Revised: All consecutive spans of dropped-out tokens are replaced by a single sentinel token.

Original: The target is to recover the dropped tokens given the sentinel token.
Revised: The target is to recover the dropped tokens given the sentinel token.

Original: We use C4 (Raffel et al., 2020) and Wikipedia (Foundation) datasets.
Revised: We use C4 (Raffel et al., 2020) and Wikipedia (Foundation) datasets.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: Lan

Original: [Table]
Revised: [Table]

Original: M o d
Revised: u

Original: [Table]
Revised: [Table]

Original:  li n g
Revised: g e M o d e li n g

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Denoising
Revised: Denoising

Original: Text Denoising Im a g e
Revised: I

Original: [Figure]
Revised: [Figure]

Original: Image & Caption

Original: I m a g e
Revised: I m a g e

Original: Image & Class
Revised: [Figure] Class

Original: Image & Caption
Revised: Image [Figure]

Original: Text
Revised: Text

Original: C O C
Revised: C O C

Original: O V C R V
Revised: O V C R V

Original: i s u a l G e n o m e L V I S
Revised: i s u a l G e n o m e L V I S

Original: O p e n I m a g e s
Revised: O p e n I m a g e s

Original: S U N 

Revised: S U N

Original: i N a t u r a li s t
Revised: i N a t u r a li s t

Original: I m a g e n e t
Revised: I m a g e n e t

Original: P l a c e s
Revised: P l a c e s

Original: Imagenet 21k
Revised: Imagenet 21k

Original: CC3M
Revised: CC3M

Original: CC12M
Revised: CC12M

Original: RedCaps
Revised: RedCaps

Original: YFCC 15M
Revised: YFCC 15M

Original: C O C O S U N 

Revised: C O C O S U N

Original: i N a t u r a li s t
Revised: i N a t u r a li s t

Original: I m a g e n e t P l a c e s
Revised: I m a g e n e t P l a c e s

Original: Imagenet 21k
Revised: Imagenet 21k

Original: CC3M
Revised: CC3M

Original: CC12M
Revised: CC12M

Original: RedCaps
Revised: [Figure]

Original: YFCC 15M
Revised: YFCC 15M

Original: Wikipedia

Original: [Figure]
Revised: [Figure]

Original: A. P RE -T RAINING D ATA D ISTRIBUTION
Revised: A. P RE -T RAINING D ATA D ISTRIBUTION

Original: Figure 3 shows a visualization of pre-training data distribution used by U NIFIED -IO.
Revised: Figure 3 shows a visualization of pre-training data distribution used by U NIFIED -IO.

Original: As discussed in Section 3.3, we equally sample data with the text denoising and image denoising objective (inner circle of Figure 3).
Revised: As discussed in Section 3.3, we equally sample data with the text denoising and image denoising objective (inner circle of Figure 3).

Original: For text denoising, half of the samples are from pure text data, i.e . C4 and
Revised: For text denoising, half of the samples are from pure text data, i.e . C4 and

Original: Wikipedia.
Revised: Wikipedia.

Original: The other half is constructed from image and class, such as Imagenet21k (Ridnik et al., 2021) or image and caption, such as YFCC15M
Revised: The other half is constructed from image and class, such as Imagenet21k (Ridnik et al., 2021) or image and caption, such as YFCC15M

Original: (Radford et al., 2021).
Revised: (Radford et al., 2021).

Original: For image denoising, we use the text information when class and caption are present in the data source and sample the dataset proportional to the dataset size.
Revised: For image denoising, we use the text information when class and caption are present in the data source and sample the dataset proportional to the dataset size.

Original: For both text and image denoising, we random drop both modalitieswith 10% of the time if both text and image as inputs. 
Revised: For both text and image denoising, we randomly drop both modalities 10% of the time if both text and image as inputs. [Table] t

Original: [Table]
Revised: [Table]

Revised: 3 M

Original: C C 2 1 M
Revised: C C 2 1 M

Original: R ed C ap s
Revised: R ed C ap s

Original: N Y U D ep t h B l ended M VS F r a m ene t
Revised: N Y U D ep t h B l ended M VS F r a m ene t

Original: L V I S S eg m en t a t i on
Revised: L V I S S eg m en t a t i on

Original: C O C O S eg m en t a t i on
Revised: C O C O S eg m en t a t i on

Original: O pen I m age s S eg m en t a t i on C O
Revised: O pen I m age s S eg m en t a t i on C O

Original: C O P
Revised: C O P

Original: o s e
Revised: o s e

Original: R e f C O C
Revised: R e f C O C

Original: O +
Revised: O +

Original: UNC
Revised: UNC

Original: R e f C O C
Revised: R e f C O C

Original: O
Revised: O

Original: UNC
Revised: UNC

Original: R e f C O C
Revised: R e f C O C

Original: O G oog l e
Revised: O G oog l e

Original: V G O b j e c t s
Revised: V G O b j e c t s

Original: C O C O O
Revised: C O C O O

Original: b j e c t s
Revised: b j e c t s

Original: O pen I m age s O
Revised: O pen I m age s O

Original: b
Revised: b

Original: j e c t s
Revised: j e c t s

Original: C O C O O
Revised: C O C O O

Original: b j e c t s V G
Revised: b j e c t s V G

Original: O b j e c t s
Revised: O b j e c t s

Original: O pen I m age s O
Revised: O pen I m age s O

Original: b
Revised: b

Original: j e c t s
Revised: j e c t s

Original: L V I S S eg m en t a t i on C O
Revised: L V I S S eg m en t a t i on C O

Original: C O S eg m en t a t i on
Revised: C O S eg m en t a t i on

Original: V G O b j e c t s C O C
Revised: V G O b j e c t s C O C

Original: O O b j e c t s O p e n I m a g e s
Revised: O O b j e c t s O p e n I m a g e s

Original: O b j e c t s
Revised: O b j e c t s

Original: C a l T e c h B i r d s C
Revised: C a l T e c h B i r d s C

Original: O C O C ap t i on s i N a t u r a li s t I 
Revised: O C O C ap t i on s i N a t u r a li s t I m

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Figure 4 shows a visualization of the multi-task training distribution used by U NIFIED -IO from
Revised: Figure 4 shows a visualization of the multi-task training distribution used by U NIFIED -IO from

Original: Table 1.
Revised: Table 1.

Original: As discussed in Section 3.3, we equally sample each group ( 1 / 8 ) except image synthesis ( 3 / 16 ) and dense labeling ( 1 / 16 ) since dense labeling has a much smaller sample size compared to image synthesis.
Revised: As discussed in Section 3.3, we equally sample each group ( 1 / 8 ) except image synthesis ( 3 / 16 ) and dense labeling ( 1 / 16 ) since dense labeling has a much smaller sample size compared to image synthesis.

Original: We sample tasks and datasets (middle and outer circle) with a temperature-scaled mixing strategy to make sure the model is sufficiently exposed to underrepresented tasks.
Revised: We sample tasks and datasets (middle and outer circle) with a temperature-scaled mixing strategy to make sure the model is sufficiently exposed to underrepresented tasks.

Original: We raise each task’s mixing rate to the power of 1 /T and then renormalize the rates so that they sum to 1.
Revised: We raise each task’s mixing rate to the power of 1 /T and then renormalize the rates so that they sum to 1.

Original: Following Raffel et al.
Revised: Following Raffel et al.

Original: , we use T = 2 in our experiments.
Revised: , we use T = 2 in our experiments.

Original: Due to the large variance in dataset size, some of the tasks are rarely sampled.
Revised: Due to the large variance in dataset size, some of the tasks are rarely sampled.

Original: For example, the depth estimation task only has the NYU Depth dataset source (Nathan Silberman & Fergus, 2012) and thus the sampling rate is only 0 .
Revised: For example, the depth estimation task only has the NYU Depth dataset source (Nathan Silberman & Fergus, 2012) and thus the sampling rate is only 0 .

Original: However, the model still works well for depth estimation tasks, even outperforming concurrent work (Kolesnikov et al., 2022) (0.385 vs .
Revised: However, the model still works well for depth estimation tasks, even outperforming concurrent work (Kolesnikov et al., 2022) (0.385 vs .

Original: 0.467 RMSE).
Revised: 0.467 RMSE).

Original: We suspect the large model capacity and masked image denoising pre-training improves the performance.
Revised: We suspect the large model capacity and masked image denoising pre-training improve the performance.

Original: Similarly,
Revised: Similarly,

Original: Grounding VQA (Chen et al., 2022a) has 0 .
Revised: Grounding VQA (Chen et al., 2022a) has 0 .

Original: 15% sample rate, but the model can still achieve stateof-the-art performance on this task partly because it is trained on many related datasets for VQA and segmentation.
Revised: 15% sample rate, but the model can still achieve stateof-the-art performance on this task partly because it is trained on many related datasets for VQA and segmentation.

Original: A. I MPLEMENTATION D ETAILS
Revised: A. I MPLEMENTATION D ETAILS

Original: The total vocabulary size is 49536, with 32152 language tokens, 1000 location tokens, andvision tokens.
Revised: The total vocabulary size is 49536, with 32152 language tokens, 1000 location tokens, andvision tokens.

Original: We use the imagenet pretrained VQ-GAN checkpoints with 16384 tokens and f = 16 1 .
Revised: We use the imagenet pretrained VQ-GAN checkpoints with 16384 tokens and f = 16 1 .

Original: Please refer (Esser et al., 2021)
Revised: Please refer (Esser et al., 2021)

Original: During training, we random sub-sample 128 image patches for pre-training state and 256 image patches (out of 576) for multi-task stage.
Revised: During training, we random sub-sample 128 image patches for pre-training state and 256 image patches (out of 576) for multi-task stage.

Original: We do not use dropout.
Revised: We do not use dropout.

Original: Adafactor (Shazeer & Stern, 2018) optimizer is used to save memory.
Revised: Adafactor (Shazeer & Stern, 2018) optimizer is used to save memory.

Original: We use a learning rate of 10 − 2 for the first 10,000 steps and then decay at a rate of 1 / √ k .
Revised: We use a learning rate of 10 − 2 for the first 10,000 steps and then decay at a rate of 1 / √ k .

Original: We train with β 1 = 0 .
Revised: We train with β 1 = 0 .

Original: 9 and β 2 = 1 . 

Revised: 9 and β 2 = 1 .

Original: − k − 0 .
Revised: − k − 0 .

Original: 8 , where k is the step number.
Revised: 8 , where k is the step number.

Original: We use global norm gradient clipping with 1.0 and find this is crucial to stabilized XL training.
Revised: We use global norm gradient clipping with 1. and find this is crucial to stabilized XL training.

Original: We train the Small , Base and Large with  batch size of 204 8 and XL with batch size of 1024 due to memory consideration.
Revised: We train the Small , Base and Large with a batch size of 204 8 and XL with batch size of 1024 due to memory consideration.

Original: 4-way in-layer parallelism and 128-way data parallelism  used to scale the 3B model training.
Revised: 4-way in-layer parallelism and 128-way data parallelism were used to scale the 3B model training.

Original: For all models, we train 1000 k steps – 500 k for pre-training and multi-task training respectively.
Revised: For all models, we train 1000 k steps – 500 k for pre-training and multi-task training respectively.

Original: A. E VALUATION ON  SAME CONCEPT AND NEW CONCEPT
Revised: A. E VALUATION ON THE SAME CONCEPT AND NEW CONCEPT

Original: [Table]
Revised: [Table]

Original: GRIT provides a breakdown of metrics into two groups: same for samples that only contain concepts seen in the primary training data (a set of common datasets like COCO, ImageNet and Visual
Revised: GRIT provides a breakdown of metrics into two groups: same for samples that only contain concepts seen in the primary training data (a set of common datasets like COCO, ImageNet, and Visual

Original: Genome), and new for samples containing at least one concept unseen in primary training data.
Revised: Genome), and new for samples containing at least one concept unseen in primary training data.

Original: Table 6 shows results for U NIFIED -IO and other leaderboard entries for the ablation set, divided into  same and new concepts.
Revised: Table 6 shows results for U NIFIED -IO and other leaderboard entries for the ablation set, divided into the same and new concepts.

Original: U NIFIED -IO XL shows little degradation in performance between same and new , compared to competing entries.
Revised: U NIFIED -IO XL shows little degradation in performance between same and new , compared to competing entries.

Original: On some tasks U NIFIED -IO is even able to outperform on the new split compared to the same .
Revised: On some tasks U NIFIED -IO is even able to outperform on the new split compared to the same .

Original: This indicates that the volume of training data used to train U NIFIED -IO has a broad coverage of concepts, and provides almost as effective a level of supervision as provided by large standard vision datasets like COCO.
Revised: This indicates that the volume of training data used to train U NIFIED -IO has a broad coverage of concepts, and provides almost as effective a level of supervision as provided by large standard vision datasets like COCO.

Original: Furthermore, since U NIFIED -IO is a uniquely unified architecture with no task-specific parameters, it is very likely able to effectively transfer knowledge across different tasks.
Revised: Furthermore, since U NIFIED -IO is a uniquely unified architecture with no task-specific parameters, it is very likely able to effectively transfer knowledge across different tasks.

Original: In comparison to Mask-RCNN (row 1), GRIT metrics show U NIFIED -IO (row 14) is better by a large margin on new concepts, i.e., non-COCO examples (74.4 vs 40.8 for localization and 64.2 vs 0.3 on segmentation), but is still superior on the COCO-like examples (65.6 vs 51.9 for localization and 53.0 vs 44.9 on segmentation).
Revised: In comparison to Mask-RCNN (row 1), GRIT metrics show U NIFIED -IO (row 14) is better by a large margin on new concepts, i.e., non-COCO examples (74.4 vs 40.8 for localization and 64.2 vs 0.3 on segmentation), but is still superior on the COCO-like examples (65.6 vs 51.9 for localization and 53.0 vs 44.9 on segmentation).

Original: U NIFIED -IO is also able to beat GPV-2 (row 5) on new concepts by large margins across all 4 tasks supported by GPV-2 even though GPV-2 is exposed to these concepts via webly supervised data and is designed to transfer concept knowledge across skills.
Revised: U NIFIED -IO is also able to beat GPV-2 (row 5) on new concepts by large margins across all 4 tasks supported by GPV-2 even though GPV-2 is exposed to these concepts via webly supervised data and is designed to transfer concept knowledge across skills.

Original: A. P ROMPT G ENERALIZATION C ASE S TUDY
Revised: A. P ROMPT G ENERALIZATION C ASE S TUDY

Original: To better understand how different prompts affect
Revised: To better understand how different prompts affect

Original: U NIFIED -IO, we do a case study on referring expressions.
Revised: U NIFIED -IO, we do a case study on referring expressions.

Original: In particular, we re-evaluate U NIFIED -IO on the GRIT referring expression ablation set while replacing the prompt used during training (first row in the table) with a paraphrase (following rows).
Revised: In particular, we re-evaluate U NIFIED -IO on the GRIT referring expression ablation set while replacing the prompt used during training (first row in the table) with a paraphrase (following rows).

Original: Results are shown in Table 7.
Revised: Results are shown in Table 7.

Original: [Table]
Revised: [Table]

Original: Overall, we find that the model has some capacity to generalize to paraphrases of the prompt ( e.g ., row 3 works reasonably well despite using completely different words), but there are paraphrases that result in  very significant performance decrease ( e.g . rows 5, 6, and 8).
Revised: Overall, we find that the model has some capacity to generalize to paraphrases of the prompt ( e.g ., row 3 works reasonably well despite using completely different words), but there are paraphrases that result in a very significant performance decrease ( e.g . rows 5, 6, and 8).

Original: We also find removing the spaces around the punctuation sometimes results in minor regressions (row 0 vs row 1) and sometimes in sharply reduced performance (row 6 vs row 7), showing U NIFIED -IO can be sensitive to formatting details.
Revised: We also find removing the spaces around the punctuation sometimes results in minor regressions (row 0 vs row 1) and sometimes in sharply reduced performance (row 6 vs row 7), showing U NIFIED -IO can be sensitive to formatting details.

Original: We hypothesize that this  caused by the SentencePiece tokenizer changing the tokenization of the referring expressing if the quotes are not separated from it by spaces.
Revised: We hypothesize that this is caused by the SentencePiece tokenizer changing the tokenization of the referring expression if the quotes are not separated from it by spaces.

Original: Building multi-task models that can generalize to different prompts, and ideally to prompts for completely new tasks, is an exciting avenue for future work.
Revised: Building multi-task models that can generalize to different prompts, and ideally to prompts for completely new tasks, is an exciting avenue for future work.

Original: A. C ROSS T ASK G ENERALIZATION C ASE S TUDY
Revised: A. C ROSS T ASK G ENERALIZATION C ASE S TUDY

Original: Qualitative examples for U NIFIED -IO when applied to two out-of-domain settings, surface normal detection on COCO images and animal pose estimation, are included in Figure 5 with the other qualitative examples.
Revised: Qualitative examples for U NIFIED -IO when applied to two out-of-domain settings, surface normal detection on COCO images and animal pose estimation, are included in Figure 5 with the other qualitative examples.

Original: For surface normal detection, we find that the model produces plausible images even for objects like cats or humans that are not in the surface normal training data.
Revised: For surface normal detection, we find that the model produces plausible images even for objects like cats or humans that are not in the surface normal training data.

Original: However, other scenes, such as outdoor scenes (Figure 5 bottom right), are less coherent.
Revised: However, other scenes, such as outdoor scenes (Figure 5 bottom right), are less coherent.

Original: Despite not being trained on animal pose estimation, U NIFIED -IO is able to sometimes find animal keypoints.
Revised: Despite not being trained on animal pose estimation, U NIFIED -IO is able to sometimes find animal keypoints.

Original: For animals standing or crouching on two legs, the keypoints are reasonably accurate (first two images), however for animals standing on four legs the model will find leg and eye points but then guess arm positions that would make sense for a person instead of attaching points to the other legs.
Revised: For animals standing or crouching on two legs, the keypoints are reasonably accurate (first two images), however for animals standing on four legs the model will find leg and eye points but then guess arm positions that would make sense for a person instead of attaching points to the other legs.

Original: While this hints that the model was able to combine skills learned from human pose estimation data with the knowledge of animals learned from other tasks, it also shows that more work is needed to fully realize this potential.
Revised: While this hints that the model was able to combine skills learned from human pose estimation data with the knowledge of animals learned from other tasks, it also shows that more work is needed to fully realize this potential.

Original: A. Q UALITATIVE E XAMPLES
Revised: A. NYU V 2 R ESULTS

Revised: The first version of U NIFIED -IO multi-tasking data distribution contains two sources of depth dataset. nyu depth v2 from Tensorflow Dataset 2 and a pre-processed version from sparse-to-dense.pytorch 3 . Since the original NYUv2 dataset has a lot of 0-distance regions (holes) which can be problematic for sequence training, we included the latter source because it replaces the 0-distance holes with approximations.

Revised: The second version of U NIFIED -IO model we trained for code release is trained on an updated multi-tasking data distribution that only contains the Tensorflow source of the NYUv2 dataset, and we find a significant drop in performance for XL model 0.475 vs .

Revised: 0.385 while other tasks maintain similar performance.

Revised: We suspect the reason is sparse-to-dense.pytorch has a different split and contaminates the training data for

Revised: NYUv2 evaluation.

Revised: Our final result is a little worse compared to UViM 0.475 vs . 0.467.

Revised: I N P U

Revised: T

Revised: P R E D

Revised: I

Revised: C T

Revised: I O

Revised: N

Revised: ANIMAL POSE ESTIMATION

Revised: Find the animal joints in this region.

Revised: I N P U

Revised: T

Revised: P R E D

Revised: I

Revised: C T

Revised: I O

Revised: N

Revised: COCO SURFACE NORMAL ESTIMATION

Revised: What is the surface normal of the image?

Original:  Here we present qualitative examples of predictions from U NIFIED -IO for all training tasks.
Revised: [Figure] Q [Figure] Here we present qualitative examples of predictions from U NIFIED -IO for all training tasks.

Original: For brevity, if prompts are identical for each example we only show the prompt once, and if the prompt follows the same template for each example we show the template with parts that would be substituted with different words or location tokens underlined, and then show just the substitution with individual examples.
Revised: For brevity, if prompts are identical for each example we only show the prompt once, and if the prompt follow the same template for each example we show the template with parts that would be substituted with different words or location tokens underlined, and then show just the substitution with individual examples.

Original: A. O THER R ELATED W ORK
Revised: A. O THER R ELATED W ORK

Original: Other Modalities.
Revised: Other Modalities.

Original: Multi-modal models for video (Li et al., 2022c;a; Wang et al., 2022a; Alayrac et al., 2022; Zellers et al., 2021;
Revised: Multi-modal models for video (Li et al., 2022c;a; Wang et al., 2022a; Alayrac et al., 2022; Zellers et al., 2021;

Original: Yu et al., 2022a), audio (Zellers et al., 2022; Jaegle et al., 2022), and other modalities including game-playing and robot controlling (Reed et al., 2022; Jaegle et al., 2022; Liang et al., 2022) have also been studied.
Revised: Yu et al., 2022a), audio (Zellers et al., 2022; Jaegle et al., 2022), and other modalities including game-playing and robot controlling (Reed et al., 2022; Jaegle et al., 2022; Liang et al., 2022) have also been studied.

Original: Integrating these modalities is an important line of research, however existing models often do not even support sparse structured output, and do I N P U T P R E D
Revised: Integrating these modalities is an important line of research, however existing models often do not even support sparse structured output and do not support dense structured outputs, so they do not meet our objective of supporting classic vision tasks.

Original: I

Original: C T

Original: I O

Original: N

Original: ANIMAL POSE ESTIMATION

Original: Find the animal joints in this region.

Original: I N P U

Original: T

Original: P R E D

Original: I

Original: C T

Original: I O

Original: N

Original: COCO SURFACE NORMAL ESTIMATION

Original: What is the surface normal of the image?

Original: not support dense structured outputs, so they do not meet our objective of supporting classic vision tasks.

Original: Vision & Language Pre-Training.
Revised: Vision & Language Pre-Training.

Original: Vision and language pre-training has become standard practice for multi-modal models, including both unified models and non-unified models that require task-specific heads to  trained from scratch during fine-tuning.
Revised: Vision and language pre-training has become standard practice for multi-modal models, including both unified models and non-unified models that require task-specific heads to be trained from scratch during fine-tuning.

Original: Many initial pre-training strategies were inspired by BERT (Devlin et al., 2019) and included masked-language-modeling, image-textmatching, or mask-region-modeling objectives, often supplemented with objectives using the predictions of a strong object detector model (e.g, VILBERT (Lu et al., 2019), LXMERT (Tan & Bansal, 2019), VisualBERT (Li et al., 2019)).
Revised: Many initial pre-training strategies were inspired by BERT (Devlin et al., 2019) and included masked-language-modeling, imagetext-matching, or mask-region-modeling objectives, often supplemented with objectives using the predictions of a strong object detector model (e.g, VILBERT (Lu et al., 2019), LXMERT (Tan & Bansal, 2019), VisualBERT (Li et al., 2019)).

Original: More recently contrastive-image-text losses (Radford et al., 2021;
Revised: More recently contrastive-image-text losses (Radford I N P U T P R E D

Original: Li et al., 2022b; 2021) or auto-regressive generation losses (Wang et al., 2022d;a; Yu et al., 2022a), have become common.
Revised: I C T I O N

Revised: IMAGE SYNTHESIS

Revised: [Figure]

Revised: many

Revised: [Figure]

Revised: the

Revised: [Figure]

Revised: on the tracks in the station

Revised: [Figure]

Revised: with black birds

Revised: [Figure]

Revised: it

Revised: I N P U

Revised: T

Revised: P R E D

Revised: I

Revised: C T

Revised: I O

Revised: N

Revised: IMAGE INPAINTING

Revised: Fill in the blank region with this

Revised: [Figure]

Revised: I N P U

Revised: T

Revised: P R E D

Revised: I

Revised: C T

Revised: I O

Revised: N

Revised: [Figure]

Revised: et al., 2021; Li et al., 2022b; 2021) or auto-regressive generation losses (Wang et al., 2022d;a; Yu et al., 2022a), have become common.

Original: Several works have also directly used object detection or segmentation datasets for pre-training Yuan et al.
Revised: Several works have also directly used object detection or segmentation datasets for pre-training Yuan et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2022b);
Revised: (2022b);

Original: Sun et al.
Revised: Sun et al.

Original: The generalized masked-data-modeling objective used in U NIFIED -IO is similar to  ones used is several recent works (Wang et al., 2022c; Peng et al., 2022; Singh et al., 2022).
Revised: The generalized masked-data-modeling objective used in U NIFIED -IO is similar to the ones used in several recent works (Wang et al., 2022c; Peng et al., 2022; Singh et al., 2022).

Original: I N P U
Revised: I N P U

Original: T
Revised: T

Original: P R E D
Revised: P R E D

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N

Original: IMAGE SYNTHESIS

Original: [Figure]

Original: I N P U

Original: T

Original: P R E D

Original: I

Original: C T

Original: I O

Original: N

Original: IMAGE

Original: [Figure]

Original: in the blank region

Original: [Figure]

Original: I N P U

Original: T

Original: P R E D

Original: I

Original: C T

Original: I O

Original: N
Revised: N

Original: IMAGE GENERATION FROM SEGMENTATION What is the complete image? Segmentation color [Figure]
Revised: OBJECT DETECTION bike chair bed umbrella car person plant person dog frisbee person glove ball

Original: I N P U
Revised: I N P U

Original: T
Revised: T

Original: P R E D
Revised: P R E D

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: OBJECT DETECTION bike chair bed umbrella car person plant person dog frisbee person glove ball

Original: I N P U

Original: T

Original: P R E D

Original: I

Original: C T

Original: I O

Original: N

Original: OBJECT LOCALIZATION
Revised: OBJECT LOCALIZATION

Original: What region does this describe?
Revised: What region does this describe?

Original: person skis parking-meter bottle
Revised: person skis parking-meter bottle

Original: I N P U
Revised: I N P U

Original: T
Revised: T

Original: P R E D
Revised: P R E D

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: REFERRING EXPRESSIONS
Revised: REFERRING EXPRESSIONS

Original: What region does this describe?
Revised: What region does this describe?

Original: chair at white table animal most out of water gray-tie man dark coat with hood
Revised: chair at white table animal most out of water gray-tie man dark coat with hood

Original: I N P U
Revised: I N P U

Original: T
Revised: T

Original: P R E D
Revised: P R E D

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: KEYPOINT ESTIMATION
Revised: KEYPOINT ESTIMATION

Original: Find the human joints in this region.
Revised: Find the human joints in this region.

Original: I N P U
Revised: I N P U

Original: T
Revised: T

Original: P R E D
Revised: P R E D

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: DEPTH ESTIMATION
Revised: DEPTH ESTIMATION

Original: What is the depth map of the image?
Revised: What is the depth map of the image?

Original: I N P U
Revised: I N P U

Original: T
Revised: T

Original: P R E D
Revised: P R E D

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: SURFACE NORMAL ESTIMATION
Revised: SURFACE NORMAL ESTIMATION

Original: What is the surface normal of the image?
Revised: What is the surface normal of the image?

Original: I N P U
Revised: I N P U

Original: T
Revised: T

Original: P R E D
Revised: P R E D

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: OBJECT SEGMENTATION
Revised: OBJECT SEGMENTATION

Original: What is the segmentation of this?
Revised: What is the segmentation of this?

Original: [Figure]

Original:  bed apple grass
Revised: pizza bed apple grass

Original: I N P U
Revised: I N P U

Original: T
Revised: T

Original: P R E 
Revised: P R E D

Original: [Figure]

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: [Figure]
Revised: [Figure]

Original: U
Revised: U

Original: T
Revised: T

Original: P R E D
Revised: P R E D

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: [Figure]
Revised: [Figure]

Original: I N P U
Revised: I N P U

Original: T
Revised: T

Original: P R E D
Revised: P R E D

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: [Figure]
Revised: IMAGE CAPTIONING

Original: What does this image describe?
Revised: What does this image describe?

Original: A large clock in a large room with a glass ceiling.
Revised: A large clock in a large room with a glass ceiling.

Original: Aflock of birds flying in a cloudy sky over a beach.
Revised: Aflock of birds flying in a cloudy sky over a beach.

Original: Aman with a thin slice of pizza in his mouth.
Revised: Aman with a thin slice of pizza in his mouth.

Original: Ablack and white cat wearing a purple knit hat
Revised: Ablack and white cat wearing a purple knit hat

Original: I N P U
Revised: I N P U

Original: T
Revised: T

Original: P R E D
Revised: P R E D

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: REGION CAPTIONING [Figure]
Revised: REGION CAPTIONING tall grass along the water a women holding a kite green shirt on mannequin a surfboard hanging on a wall

Original: What does [Figure] region describe?
Revised: What does the region describe?

Original: I N P U
Revised: I N P U

Original: T
Revised: T

Original: P R E D
Revised: P R E D

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: VISUAL QUESTION ANSWERING
Revised: VISUAL QUESTION ANSWERING

Original: [Figure]
Revised: [Figure]

Original: I N P U
Revised: I N P U

Original: T
Revised: T

Original: P R E D
Revised: P R E D

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: GROUNDED VQA
Revised: GROUNDED VQA

Original: [Figure]
Revised: [Figure]

Original: I N P U
Revised: I N P U

Original: T
Revised: T

Original: P R E D
Revised: P R E D

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: RELATIONSHIP DETECTION
Revised: RELATIONSHIP DETECTION

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: I N P U

Original: T

Original: P R E D

Original: I

Original: C T I O N SENTENCE CLASSIFICATION
Revised: SENTENCE CLASSIFICATION

Original: [Table]
Revised: [Table]

Original: Site sentence2:
Revised: sentence

Original: [Table]
Revised: [Table]

Original: Rhine Gorge is between Koblenz and what other city?
Revised: is between [Table] and [Table]

Original: question:
Revised: other

Original: [Table]
Revised: [Table]

Original: the first sentence contain the answer to the second sentence?
Revised: to [Table] second sentence?

Original: no entailment no
Revised: no entailment no

Original: yes
Revised: yes

Original: I N P U
Revised: I N P U

Original: T
Revised: T

Original: P R E D
Revised: P R E D

Original: I
Revised: I

Original: C T
Revised: C T

Original: I O
Revised: I O

Original: N
Revised: N

Original: TEXT SUMMERIZATION context: European stocks finished on a mixed note Tuesday, as continental markets recouped earlier losses after a positive start to trading in U.S. equity markets .
Revised: TEXT SUMMERIZATION context: European stocks finished on a mixed note Tuesday, as continental markets recouped earlier losses after a positive start to trading in U. [Table] equity markets .

Original: question: What is a one sentence summary of this document?
Revised: question: What is a one sentence summary of this document?

Original: context: Hungary and Bulgaria, which both share borders with Serbia and
Revised: context: Hungary and Bulgaria, which both share borders with Serbia and

Original: Croatia, want U.S. and otherinternational troops to stay longer than currently planned in Bosnia to keep the
Revised: Croatia, want [Table] [Table] and other

Original: [Table]
Revised: [Table]

Original: in
Revised: longer

Original: [Table]
Revised: [Table]

Original: former Yugoslavia.
Revised: currently planned in

Original: question:What is a one sentence summary of this document?
Revised: [Table]

Original: context: Toyota rolled out its first UNK
Revised: to

Original: [Table]
Revised: [Table]

Original: model Tuesday as
Revised: Yugoslavia. question

Original: [Table]
Revised: [Table]

Original: world
Revised: sentence

Original: [Table]
Revised: [Table]

Original: top automaker seeks to
Revised: document

Original: [Table]
Revised: [Table]

Original: itself around by pushing the increasingly popular green technology. question
Revised: technology

Original: [Table]
Revised: [Table]

Original: What is a one sentence summary of this document?

Original: European stocks end mixed
Revised: European stocks end mixed

Original: Hungary Bulgaria want longer
Revised: Hungary Bulgaria want longer

Original: [Table]
Revised: [Table]

Original: [Table]

Original: presence
Revised: presence

Original: [Table]

Original: Bosnia

Original: [Table]

Original: says

Original: [Table]

Original: Toyota

Original: [Table]

Original: out first

Original: [Table]

Original: context:Federal

Original: [Table]

Original: sought to

Original: [Table]

Original: Street and congress Tuesday

Original: [Table]

Original: the

Original: [Table]

Original: [Table]

Original: [Table]

Original: be

Original: [Table]
Revised: [Table]

Original: to reel in its extraordinary economic stimulus and

Original: [Table]
Revised: [Table]

Original: a

Original: [Table]

Original: [Table] What [Table] a one sentence summary of this document?
Revised: summary of this document?