GiT: Towards Generalist Vision Transformer through Universal Language Interface

Dear BK,

Thank you for your question. :)

Regarding the first question about whether to discard the visual encoder? First, we want to create a unified general model, a data-centric, end-to-end model that only requires non-parametric design at the input and output ends, or a light tokenizer, etc. Why do this? First, it greatly simplifies the training process. The training of the model is a one-stage process, no longer requiring the pre-training of a visual encoder. Imagine, if you need to scale an adapter-based vision-language model, you need to scale the visual encoder separately and then scale the LLM (Large Language Model) separately again, and the vision-language fine-tuning. Three-stage. This is a troublesome thing. The future data-centric model I envision has a very simple structure, continuously training with data, allowing the model to become increasingly strong. The network's capability is strong enough to handle all modalities and tasks on its own. The Extensibility of the model is very important. Here, extensibility means the model continuously becomes stronger, more tasks and more modalities, all end-to-end, rather than the current non-end-to-end, multi-stage training (requiring pre-training of visual encoder and LLM). In short, what we want to do is a supremely simple end-to-end multi-layer transformer that can learn almost all tasks and data, including but not limited to 2D vision, language, point cloud, etc. Point cloud can refer to our previous work that unified image and point cloud using a plain transformer, 'UniTR: A Unified and Efficient Multi-Modal Transformer for Bird’s-Eye-View Representation'.

Regarding the second question, for tasks involving dense prediction, there indeed exist flaws in the current paper. However, the multiple parallel decoding mentioned in the paper can greatly mitigate this issue. The appendix includes our segmentation speed, which is roughly on par with SAM's speed at the same resolution. We haven't tried more points. More points mean a higher degree of parallelism and faster speed. Efficiency is where we aim to optimize and improve in the next phase. Please stay tuned to our work. A LLM-like framework means that all acceleration techniques applicable to LLMs can be used by us, and I believe the speed can be significantly increased. Our current work is not perfect yet, and there are many areas for improvement, which we can strive to enhance together.

We want to create a framework that is simple to the extreme and data-centric, not limited to vision-language. This framework can be embedded with various other modalities and data types, such as point clouds and graphs, etc. Secondly, why combine vision-language. Vision-language must inevitably move towards unification. Language is very special; I believe it is not on the same level as vision. It is a modality that ultimately aligns with the human understanding space. In a sense, all vision tasks' ground truth is language, all are texts. Segmentation is a mask because people visualized the segmentation label in the form of a 2D image. For example, in the perception of an autonomous driving scenario, you need to segment the scene, and the result of the segmentation still needs to be converted into language. Because only language is a program that a machine or computer can execute, the visualized segmentation mask is not. Third, if the model has good extensibility, allowing different tasks to promote each other, then this model is very promising and can continuously become stronger (with better performance on individual tasks, and the ability to handle more tasks, just like humans).

You also mentioned that we humans have different brain areas for different modalities, and I agree with this view. The development of deep learning has evolved from manually designing features to models being able to learn features. I believe the structure of models should also follow this path. Previously, people manually designated which parameters were responsible for different modalities. Perhaps now, the network itself can learn which parameters are responsible for which modalities, in an implicit way. In summary, it's about being data-centric and continuously reducing the part of human hand-craft design.

Of course, I do not deny the work of the previous adapter. On the contrary, I think it works very well at this stage. We just want to offer a new idea and solution, and whether it is the right path can be observed later. Thank you very much for your question, let's make the community better together.

Best,
Haiyang