Paper: ResBit: Residual Bit Vector for Categorical Values (ICLR-2024)

Reviewer comment (Evaluation):
On 5 tabular datasets where such an encoding method would be of most use, the proposed method is clearly better only on 2 of the tasks (CC, AR), whereas on BD and AD performance is on par, and results on IS are discounted due to the small size of the dataset (1338). Similarly, when used for conditioning of GANs, visually speaking res-bit results seem to be worse (much less diverse) and have no strong edge over one-hot in classification tasks. Considering these observations, it is hard to say that residual bitwidth representation of categorical values is a good encoding in general.

Author response:
Results from the remaining datasets experimented in the TabDDPM paper, which include categorical columns, are presented in the COMMON RESPONSE. According to the findings, the results are comparable for the cardio dataset and slightly inferior for the abalone dataset. However, the facebook dataset, with 157,638 training data, demonstrates significant results. If we exclude abalone and IS due to the small dataset sizes, TRBD overall equals or surpasses TabDDPM. The above, coupled with the runtime results (refer to Section 4.1.5 and Table 4), are significant in that the same or better results can be achieved in a shorter time. Concerning the conditioning of GANs, when using CIFAR10, it can be visually said that more mode collapse can be observed in the one-hot case compared to ResBit (see Appendix. C). Furthermore, ResBit is superior to one-hot in terms of FID, so it is not a result that the superiority of ResBit is not generally recognized. As for image classification, there is certainly no strong advantage. However, as mentioned **twice** in our paper (see Sections 4.2 and 4.3), the main purpose of these two experiments is not to show the superiority in terms of accuracy. **We aim to confirm that ResBit can be used as an alternative to one-hot while reducing the number of dimensions (see Section 4.4)**. Therefore, the lack of strong significance in terms of accuracy is not a concern. In addition, even if it is somewhat inferior in terms of accuracy, in this experimental setting, the runtime has an advantage (refer to the table below). Considering the trade-off between runtime and quality, there is no compelling reason to strongly negate this advantage.


Table 1. Comparison the runtime using SNGAN and 40 selected class from the Food-101.
|  |  | | | |
| :------: | :--------------------: | :--------------: | :--------------: | :--------------: |
| Method  | Label dim | #params D | #params G | Training Time (hours)
| one-hot | 40 | 38,446,736 | 32,806,147 | 8.75 |
| ResBit | **9** | **38,424,896** | **32,298,243** | **8.25** |

Reviewer comment (Theory):
There are several false claims in the paper. First, ResBit may not fully address the “out-of-index” issue. Since \(N=50=32+16+2\), the example given in the paper is free from the issue. Proof for any natural number is missing. One can find a counterexample by finding the ResBit representation of \(N=51\)?

Author response:
Since ResBit is to be applied when the number of classes is known in advance, the reviewer seems to have a major misunderstanding. In the analog bits, "out of index" occurs when the length of a class increases, even if the number of classes is known in advance. The $N=50$ shown in our paper is a perfect example. If $N=2^k$, "out of index" does not occur. ResBit is an expression that depends on the number of classes, similar to one-hot, etc. In the example of the states of the United States, one-hot considers a 50-dimensional vector because one-hot depends on the number of classes.　In the following, we formulate the ResBit. The number of classes is set to $N\geq2$. When $N=1$, there is no problem in the actual task, so we do not consider it (e.g., if conditioning, it corresponds to no condition). Considering 0-index, ResBit can be written as follows.

$$
    N-1=\sum_{k=0}^{\infty}a_k(2^{k}-1)\qquad \mathrm{s.t.}\min\sum_{k=0}^{\infty}a_kk
$$

The $k$-th term can represent any natural number in the range $[0, a_k(2^k-1)]$. It is easy to decompose $i\in\mathbb{Z}$ with $0\leq i<N$ as the sum of such natural numbers.

Reviewer comment (Experiments):
The ResBit does not really improve or at least achieve no worse results compared to their baselines. In some cases, ResBit even performs much worse than the baselines.

Author response:
Many of our experiments show applicability. Learning with ResBit is equivalent to multi-label classification, and in some cases, devising multi-class classification alone may not be sufficient. In our experiments, the one-hot method is slightly more advantageous than the one-hot method because the one-hot classification employs top-1. However, this is not the case in most cases in the experiment. Therefore, only some bad results should not be emphasized.

Reviewer comment (Experiments):
In Section 4.1.4, the authors state that the loss exploded or disappeared during the training phase of TabDDPM for certain datasets and argue that that is probably due to the very large number of dimensions. This seems to be a strange reason because the dimensions are not too large in these problems and usually this kind of problem can be addressed by normalizing the features or using smaller learning rates.

Author response:
It is unfair if the experiment is not conducted in the same learning rate search space (fairness is what you pointed out as a weakness). In the same search space, the fact that the loss exploded or disappeared for TabDDPM but not for TRBD may be due to the number of dimensions.　This is because there is no other difference between TabDDPM and TRBD.　In the experiments with Multinomial Diffusion [2], the number of classes is 8, 27, and 256, so compared to those, 6k and 7k are very large and not something to consider in general terms.

Reviewer comment (Theory):
Can we prove that ResBit does not have the “out-of-index” issue mathematically?

Author response:
Since ResBit is to be applied when the number of classes is known in advance, the reviewer seems to have a major misunderstanding. In the analog bits, "out of index" occurs when the length of a class increases, even if the number of classes is known in advance. The $N=50$ shown in our paper is a perfect example. If $N=2^k$, "out of index" does not occur. ResBit is an expression that depends on the number of classes, similar to one-hot, etc. In the example of the states of the United States, one-hot considers a 50-dimensional vector because one-hot depends on the number of classes.　In the following, we formulate the ResBit. The number of classes is set to $N\geq2$. When $N=1$, there is no problem in the actual task, so we do not consider it (e.g., if conditioning, it corresponds to no condition). Considering 0-index, ResBit can be written as follows.

$$
    N-1=\sum_{k=0}^{\infty}a_k(2^{k}-1)\qquad \mathrm{s.t.}\min\sum_{k=0}^{\infty}a_kk
$$

The $k$-th term can represent any natural number in the range $[0, a_k(2^k-1)]$. It is easy to decompose $i\in\mathbb{Z}$ with $0\leq i<N$ as the sum of such natural numbers.

Reviewer comment (Experiments):
Given that ResBit is proposed for reducing the representation complexity of categorical data, have you tried to run ResBit for image classification on ImageNet? Does it maintain the performance compared to one-hot encoding while achieving lower complexity?

Author response:
As described in Section 4.4, this is a future work. Note that there are two possible ways to evaluate the output of ResBit in this experiment. In both cases, the output of the model is rounded.

- Check if the output of the model matches the ResBit of the correct label
- Convert the model output to an integer value and check if it matches the correct label

Since the accuracy was the same in both cases, the model is considered to correctly represent ResBit. Therefore, we expect the same degree of difficulty in experiments on datasets with numerous labels, such as ImageNet-1k, as in the one-hot case. Note that image classification in ImageNet is also possible by ensembling many classifiers like ACGAN[1].

Reviewer comment (Experiments):
In TabDDPM, authors propose experiments on 15 datasets with many baselines. Authors should follow such an experimental setting.

Author response:
> In TabDDPM, authors propose experiments on 15 datasets with many  baselines. Authors should follow such an experimental setting.

> Maybe authors should introduce fewer components but add more detailed comparisons with existing methods.

Of the 15 datasets listed in TabDDPM, the results of those not covered in the paper are shown in the COMMON RESPONSE. Note that we do not conduct the experiments for data that have only numerical datasets, since TRBD and TabDDPM are the same in such datasets. We use CatBoost in the TSTR framework. When integrated with the results in our paper, TRBD outperforms or equals TabDDPM on most datasets. This shows the superiority of the proposed method in generating tabular data. Also, since TabDDPM has been reported to significantly outperform existing deep generative models [3, 4, 5] in the TabDDPM paper, we do not believe that we need to use them as a baseline.

Reviewer comment (Experiments):
Maybe authors should introduce fewer components but add more detailed comparisons with existing methods.

Author response:
> In TabDDPM, authors propose experiments on 15 datasets with many  baselines. Authors should follow such an experimental setting.

> Maybe authors should introduce fewer components but add more detailed comparisons with existing methods.

Of the 15 datasets listed in TabDDPM, the results of those not covered in the paper are shown in the COMMON RESPONSE. Note that we do not conduct the experiments for data that have only numerical datasets, since TRBD and TabDDPM are the same in such datasets. We use CatBoost in the TSTR framework. When integrated with the results in our paper, TRBD outperforms or equals TabDDPM on most datasets. This shows the superiority of the proposed method in generating tabular data. Also, since TabDDPM has been reported to significantly outperform existing deep generative models [3, 4, 5] in the TabDDPM paper, we do not believe that we need to use them as a baseline.

Reviewer comment (Experiments):
Maybe it is possible to run the algorithms on an image dataset.

Author response:
> Maybe it is possible to run the algorithms on an image dataset.

Representing the pixels of an image in ResBit is the same as representing them in Analog Bits (Since the pixels of an image are integer values of [0, 255]). An experiment to represent image pixels using Analog Bits and image generation using Diffusion Models has been conducted in [6]. There is no need for us to perform the experiments already validated in [6].

Reviewer comment (Writing):
How the ResBit algorithm works concerning Analog Bits.

Author response:
> How the ResBit algorithm works concerning Analog Bits.

ResBit is a fusion of part of the Analog Bits concept, which represents discrete data as a sequence of bits, and the RVQ concept, which further vector quantizes the difference of vector quantization. The full application of the Analog Bits concept requires the high expressive power of Diffusion Models, and TRBD is an example of its application. On the other hand, ResBit can be considered to work the same as one-hot if it is considered as a method of expressing ResBit as  0/1. Its application to image classification and conditional image generation is a verification of this. In addition, if the number of classes is represented by $2^m$, it is truly the same as the analog bits. Representing the pixels of an image in such a way is exactly what was experimented in the analog bits paper.

Reviewer comment (Experiments):
Maybe it is possible to show some practical tasks to show that ResBit works better than Analog Bits.

Author response:
> Maybe it is possible to show some practical tasks to show that ResBit works better than Analog Bits

> The model proposes three elements: ResBit, TRBD, and conditioning GAN. Unfortunately, none of such components are well evaluated. Especially ResBit should be compared with Analog Bits.

We cannot evaluate in the tabular data generation task due to the "out of index" problem described in Section 3.1. On the other hand, we can evaluate the other two applications. Let $N$ be the number of classes and consider the smallest $m\in\mathbb{N}$ satisfying $N<2^m$ holds. In this experiment, $m$-dimensional vectors are used to represent the classes. For example, for $N=10$, $m=4$. The table below are the experimental results when using CIFAR-10. The results for one-hot and ResBit are those in the paper.

   
|  |  | | |
| :------: | :--------------------: | :--------------: | :--------------: |
| Task  | Image Classification | Image Classification | Image Generation |
| Used Model | MobileNetV3-small | MobileNetV3-large | InfoGAN | 
| Metrics | Accuracy (%) | Accuracy (%) | FID |
| Result (one-hot) | 66.52 | **74.63** | 95.52 |
| Result (ResBit) | **66.70** | 72.55 | 82.21 |
| Result (analog) | 62.70 | 70.62 | **63.54** |

In the context of conditioning, "out of index" scenario is not deemed a significant issue; rather, it leads to a straightforward reduction in the number of dimensions, which is believed to enhance accuracy. Conversely, in the realm of image classification, predicting an "out of index" situation could result in diminished accuracy. Consequently, we consider that Analog Bits are perceived as less versatile than ResBit.