A rch BERT : Bi-Modal Understanding of Neural Architectures and Natural Languages (2023)

Abstract: Building multi-modal language models has been a trend in the recent years, where additional modalities such as image, video, speech, etc. are jointly learned along with natural languages (i.e., textual information). Despite the success of these multi-modal language models with different modalities, there is no existing solution for neural network architectures and natural languages. Providing neural architectural information as a new modality allows us to provide fast architecture-2-text and text-2-architecture retrieval/generation services on the cloud with a single inference. Such solution is valuable in terms of helping beginner and intermediate ML users to come up with better neural architectures or AutoML approaches with a simple text query. In this paper, we propose ArchBERT, a bi-modal model for joint learning and understanding of neural architectures and natural languages, which opens up new avenues for research in this area. We also introduce a pre-training strategy named Masked Architecture Modeling (MAM) for a more generalized joint learning. Moreover, we introduce and publicly release two new bi-modal datasets for training and validating our methods. The ArchBERT’s performance is verified through a set of numerical experiments on different downstream tasks such as architecture-oriented reasoning, question answering, and captioning (summarization). Datasets, codes, and demos are available as supplementary materials.

## introduction existing machine learning models are mostly based on uni-modal learning, where a single modality is learned for the desired tasks. example scenarios include image classification with image-only data; or language translation with text-only data (raffel et al., 2020; @xcite . despite the success of existing uni-modal learning methods at traditional single-modal tasks, they are usually insufficient @xcite to model the complete aspects of human's reasoning and understanding of the environment. the alternative solution for this problem is to use multi-modal learning, where a model can jointly learn from multiple modalities such as text, image, or video to yield more abstract and generalized representations. as a result, a better understanding of various senses in information can be achieved and many new challenges that concern multi-modality can be handled. such solution also enables the possibility of supplying a missing modality based on the observed ones. as an example, in textbased image generation, we aim to generate photorealistic images which are semantically consistent with some given text description @xcite . one of the most popular multi-modal solutions is multi-modal language models (lms), where an extra modality (e.g., image or video) is jointly used and learned along with the natural languages (i.e., textual information). some of the recent multimodal lms include vilbert for image+text @xcite , videobert for video+text @xcite , codebert for code+text @xcite , and also @xcite . although many multi-modal lms with different modalities have been introduced so far, there is no existing solution for joint learning of neural network architectures and natural languages. providing neural architectural information as a new modality allows us to perform many architectureoriented tasks such as architecture search (as), architecture reasoning (ar), architectural question answering (aqa), and architecture captioning (ac) (figure 1 ). the real-world applications of such solution include fast architecture-2-text and text-2-architecture retrieval/generation services on the cloud with a single inference. such solution is valuable in terms of helping users to come up with better neural architectures or automl approaches with a simple text query especially for beginner and intermediate ml users. for instance, ac can be used for automatically generating descriptions or model card information on a model hub (i.e., machine learning models repository). furthermore, ar is helpful when a model is uploaded to a repository or cloud along with some textual description provided by the user, where the relevancy of the user's description for the given model can be automatically verified. if not verified, alternative autogenerated descriptions by a architecture-2-text solution can be proposed to the user. in this paper, we propose archbert as a bimodal solution for neural architecture and natural language understanding, where the semantics of both modalities and their relations can be jointly learned (figure 1 ). to this end, we learn joint embeddings from the graph representations of architectures and their associated descriptions. moreover, a pre-training strategy called masked architecture modelling (mam) for a more generalized and robust learning of architectures is proposed. we also introduce two new bi-modal datasets called tvhf and autonet for training and evaluating archbert. to the best of our knowledge, archbert is the first solution for joint learning of architecture-language modalities. in addition, archbert can work with any natural languages and any type of neural network architectures designed for different machine learning tasks. the main contributions of this paper are as follows: multi-modal models are used in many sub-fields in machine learning. for example, @xcite and @xcite introduced the audio-visual models trained on input acoustic speech signal and video frames of the speaker for speech enhancement, speech separation, and emotion recognition. multi-modal models used in biomedical @xcite , remote-sensing @xcite , and autonomous driving @xcite applications have also proven to provide more accurate prediction and detection than the unimodal models. among different types of multi-modal lms in the literature, transformer-based ones have shown significant performance, especially for vision-andlanguage tasks like visual question answering, image captioning, and visual reasoning. in visual-bert @xcite , a stack of transformers is used to align the elements of text and image pairs. vilbert @xcite extended bert to a multi-modal double-stream model based on coattentional transformer layers. in lxmert @xcite , three encoders including language, object relation, and cross modality encoders are used. a single-stream vision-language model was introduced in vl-beit @xcite , where unpaired and paired image-text modalities were used for pre-training. video is another modality that is used with language in multi-modal models. videobert @xcite ) is a single-stream video-language model, which learns a joint visual-linguistic representation from input video-text pairs. violet @xcite is another example that employs a video transformer to model the temporal dynamics of videos, and achieves sota results on video question answering and text-to-video retrieval. programming language is also an emerging modality that has been used along with language. for example, codebert @xcite ) is a multistream model, which uses lms in each stream, where the input code is regarded as a sequence of tokens. on the other hand, @xcite proposes a structure-aware pre-training technique to consider the inherent structure of the code by mapping it to a data flow graph. there are several prior works that combine more than two modalities. in multimodal transformer (mult) @xcite , cross-modal attention modules are added to the transformers to learn representations from unaligned multi-modal streams, including the language, the facial gestures, and the acoustic behaviors. vatt @xcite also used video, audio, and text transformers along with a self-supervised learning strategy to obtain multi-modal representations from unlabeled data. it is worth mentioning that chatgpt (openai, 2022) can be used for information retrieval, question answering, and also summarization over the textual descriptions of well-known neural architectures such alexnet @xcite or faster-rcnn @xcite . however, unlike archbert, it does not have a bi-modal understanding of both neural architectures (i.e., graphs) and natural languages, especially for newly proposed architectures and models. ## proposed method: archbert the overall archbert framework is shown in figure 2 . the major components of archbert include a text encoder, an architecture encoder, a cross encoder, and a pooling module. first, the input text represented by a sequence of n words w " tw i |i p r1, nsu is tokenized to a sequence of n tokens t " tt i |i p r1, nsu. then, the text encoder e t is utilized to map them to some word/token embeddings denoted by m t p r pnˆdq with the embedding size of d: m t " e t pt q. on the other hand, the architecture encoder is responsible for encoding the input neural architecture. in this procedure, the computational graph of the input architecture is first extracted and represented with a directed acyclic graph g " tv, a, su where v " tv i |i p r1, msu denotes a sequence of m nodes representing the operations and layers (e.g., convolutions, fully-connected layers, summations, etc.) and a p t0, 1u mˆm denotes a binary adjacency matrix describing the edges and the connectivity between the nodes. in addition to the nodes and edges, we also extract the shape of the param-eters associated with each node (i.e., input/output channel dimensions and kernel sizes), denoted by s " tps i p n 4 q|i p r1, msu. the nodes and the shapes are separately encoded using the node and shape embedders e v and e s , respectively. the adjacency matrix along with the summation of the resulting nodes and shapes embeddings are then given to a graph attention network (gat) @xcite for computing the final architecture (graph) embeddings denoted by m g p r pmˆdq with the embedding size of d: in general, gat is designed to operate on graphstructured data in which a set of graph features (node+shape embeddings in our case) is transformed into higher-level features. given the adjacency matrix, the gat model also allows all nodes to attend over their neighborhoods' features based on a self-attention strategy. for joint learning of textual and architectural embeddings and share learning signals between both modalities, a cross transformer encoder, e c , is used to process both embeddings in parallel. these embeddings are then average-pooled to fixed-size 1d representations j t p r p1ˆdq and j g p r p1ˆdq : as in s-bert (reimers and gurevych, 2019), we use the cosine similarity loss as a regression objective function to learn the similarity/dissimilarity between architectures and language embeddings. first, the cosine similarity between j t and j g are computed. given a target soft score y p r0, 1s (i.e., 0: dissimilar, 1: similar), the following mean squared-error (mse) loss is then employed: ## datasets for pre-training the archbert model, a dataset of neural architectures labeled with some relevant descriptions is required. to the best of our knowledge, there is no such bi-modal dataset in the literature. in this paper, we introduce two datasets called tvhf and autonet for bi-modal learning of neural architectures and natural languages. the numerical and the statistical details of tvhf and autonet datasets are summarized in note that all the labels and descriptions in the proposed datasets have been manually checked and refined by human. there may be some minor noise in the dataset (i.e., an inevitable nature of any dataset, especially the very first versions), but in overall, the datasets are of sufficient quality for our proof-of-concept experiments. ## experimental results in this section, the performance of archbert on the following downstream tasks is evaluated and numerically analyzed. since there is no related prior works, we compare our method with some uni-modal baselines for each of the above tasks. an ablation study over different components of archbert is also presented. in this work, we employ the bert-base model (with 12 heads) as our archbert's cross encoder. we pre-trained archbert on both tvhf and au-tonet datasets with a batch size of 80, embedding size of @xmath0, and the adam optimizer with learning rate of 2e-5 for 6 hours. the training on tvhf and autonet was respectively done for 20 and 10 epochs. since there is a large scale difference between the l sim and l m am loss values in the weighted loss in equation 5, where l m am "l sim , we set α=5e-2 to balance the total loss value (obtained experimentally). a batch size of 80 is used for all the tests with the pre-trained archbert. ## conclusion in this paper, we proposed archbert, a bi-modal solution for joint learning of neural architectures and natural languages. we also introduced a new pre-training technique called masked architecture modeling (mam) for a better generalization of archbert. in addition, two new bi-modal benchmark datasets called tvhf and autonet were presented on which the proposed model was trained and evaluated for different downstream tasks. five architecture-language-related tasks and applications were introduced in this work to verify the performance of archbert. this work has opened up new avenues for research in the area of architecturelanguage joint understanding, particularly the proposed benchmarks. potential research directions to this work include text-based neural architecture generation and bi-modal learning of languages and other graph-structured modalities such as knowledge graphs and social network graphs. tv-only, and hf-only datasets, and evaluate their performance on each other. the corresponding experimental results are summarized in as observed in the table, the models trained on tv and hf subsets do not generalize to each other due to the difference in their data distributions, which results in poor performance. the distribution plots for tv and hf subsets are shown in figure 8 . as given in