M u G : A Multimodal Classification Benchmark on Game Data with Tabular, Textual, and Visual Fields (2023)

Abstract: Previous research has demonstrated the advantages of integrating data from multiple sources over traditional unimodal data, leading to the emergence of numerous novel multimodal applications. We propose a multimodal classification benchmark MuG with eight datasets that allows researchers to evaluate and improve their models. These datasets are collected from four various genres of games that cover tabular, textual, and visual modalities. We conduct multi-aspect data analysis to provide insights into the benchmark, including label balance ratios, percentages of missing features, distributions of data within each modality, and the correlations between labels and input modalities. We further present experimental results obtained by several state-of-the-art unimodal classifiers and multimodal classifiers, which demonstrate the challenging and multimodal-dependent properties of the benchmark. MuG is released at https://github.com/lujiaying/MUG-Bench with the data, tutorials, and implemented baselines.

## introduction the world surrounding us is multimodal. realworld data is often stored in well-structured databases that contain tabular fields, with textual and visual fields co-occurring. numerous automated classification systems have been deployed on these multimodal data to provide efficient and scalable services. for instance, medical decision support systems @xcite utilize patients' electronic health record data that contains tabular inputs (e.g., ages, genders, races), textual inputs (e.g., notes, prescriptions, written reports), and visual inputs (e.g., x-rays, magnetic resonance imaging, ct-scans) to help precise disease prediction. similarly, e-commerce product classification systems @xcite categorize products based on their categorical/numerical quantities, textual descriptions, and teasing pictures, thus enhancing user search experiences and recommendation outcomes. therefore, accurate classification models for table-text-image input are desired. deep neural networks have shown significant progress in multimodal learning tasks, such as clip @xcite for image-text retrieval and fuse-transformer @xcite for tabular-with-text classification. this progress has been made with large-scale datasets provided to train the data-eager models. so far, there exist many datasets @xcite @xcite @xcite @xcite that cover one or two modalities. however, the progress in tabular-text-image multimodal learning lags due to the lack of available resources. in this paper, we provide a multimodal benchmark, namely mug, that contains eight datasets for researchers to examine their al-gorithms' multimodal perception ability. mug contains data samples with tabular, textual, and visual fields that are collected from various genres of games. we have made necessary cleaning, transformations, and modifications to the original data to make mug easy to use. we further conduct comprehensive data analysis to demonstrate the diverse and multimodal-dependent properties of mug. mug can enable future studies of many multimodal tasks, and we focus on the multimodal classification task in this paper. for the primary classification evaluation, we incorporate two stateof-the-art (sota) unimodal classifiers for each of the three input modalities, resulting in a total of six, along with two sota multimodal classifiers. we also propose a novel baseline model mugnet based on the graph attention network @xcite . in addition to capturing the interactions among the three input modalities, our mugnet takes the sample-wise similarity into account, yielding a compatible performance to existing multimodal classifiers. we further conduct efficiency evaluations to reflect the practical requirements of many machine learning systems. ## mug: the benchmark we create and release mug with eight datasets for multimodal classification with tabular, text, and image fields to the community for future studies. raw data and examples of how to appropriately load the data are provided in https://github. com/lujiaying/mug-bench . mug is under the "cc by-nc-sa 4.0" license 2 , and is designated to use for research purposes. ## baseline models we employ several state-of-the-art unimodal classifiers and multimodal classifiers in the experiments. we also proposed our own graph neural networkbased multimodal classifier as one baseline model to be compared. ## conclusion this paper presents a benchmark dataset mugnet along with multiple baselines as a starting point for the machine learning community to improve upon. mugnet is a multimodal classification benchmark on game data that covers tabular, textual, and visual modalities. all eight datasets and nine evaluated baselines are open-source and easily-extended to motivate rapid iteration and reproducible experiments for researchers. a comprehensive set of analyses is included to provide insight into the characteristics of the benchmark. the experimental results reported in the paper are obtained from models trained with constrained resources, which is often required by real-life applications. however, we also welcome future works that utilize enormous resources. finally, we hope this work can facilitate research on multimodal learning, and we encourage any extensions to mugnet to support new tasks or applications such as open-domain retrieval, ai-generated content, multimodal qa, etc.