To be Continuous, or to be Discrete, Those are Bits of Questions (2024)

Abstract: Recently, binary representation has been proposed as a novel representation that lies between continuous and discrete representations. It exhibits considerable information-preserving capability when being used to replace continuous input vectors. In this paper, we investigate the feasibility of further introducing it to the output side, aiming to allow models to output binary labels instead. To preserve the structural information on the output side along with label information, we extend the previous contrastive hashing method as structured contrastive hashing. More specifically, we upgrade CKY from label-level to bit-level, define a new similarity function with span marginal probabilities, and introduce a novel contrastive loss function with a carefully designed instance selection strategy. Our model achieves competitive performance on various structured prediction tasks, and demonstrates that binary representation can be considered a novel representation that further bridges the gap between the continuous nature of deep learning and the discrete intrinsic property of natural languages.

## introduction bridging the gap between the continuous nature of deep learning and the discrete intrinsic property of natural languages has been one of the most fundamental and essential questions since the very beginning. continuous representation makes the training of neural networks effective and efficient. nowadays, representing discrete natural languages in continuous format is the first and foremost step to leveraging the capabilities of deep learning. one could even argue that the exhilarating advancements in natural language processing in the past decade can largely be attributed to the word embedding technique, as it is the first successful attempt. word embedding @xcite technique replaces the vocabulary-µ < l a t e x i t s h a 1 _ b a s e 6 4 = " f w h k z j e m p w n t q 1 i j z s e j 1 u o / c 8 @xmath0 figure 1 : the model architecture. the attention hash layer produces span scores (pink circles), we only use the upper triangular part of these scores and feed them into the bit-level cky to obtain the marginal probabilities of all valid spans (purple circles). during training, we only select the spans on the target trees for structured contrastive hashing and leave the other spans unused (transparent purple circles). during inference, as shown at the bottom, our model parses sentences by returning trees with label codes (hexadecimal numbers), which are then translated back to the original labels. recently, @xcite introduced a novel binary representation that lies between continuous and discrete representations. they proposed a contrastive hashing method to compress continuous hidden states into binary codes. these codes contain all the necessary task-relevant information, and using them as the only inputs can reproduce the performance of the original models. unlike associating each token with only a single vector, their method allocates multiple bits to each token, and the token representation can be constructed by concatenating these bit vectors. in other words, their binary representation breaks tokens down into combinations of semantic subspaces. as a result, replacing the token embedding matrix in the input layer with a tiny bit embedding matrix without sacrificing performance becomes possible. in this paper, we explore the possibility of further introducing this representation to output layers. in the input layer, structural information can only be implicitly obtained by introducing the task loss as an auxiliary. however, the output layers often involve complex intra-label constraints, especially for structured prediction tasks, structural information can and should be explicitly preserved along with plain label information. therefore, we attempt to endow models with this capability by extending previous contrastive hashing to @xmath1 figure 2 : an example of the geometric center issue. orange circles are positive to the black circle instance, while the dotted orange circle is their geometric center. the difference between l sup and our l max is that we target the closest positive instead of their geometric center. we begin by upgrading the cky, which parses sentences, returns spans with discrete labels, to support binary format labels ( §3.1). subsequently, we define a new similarity function by using span marginal probabilities obtained from this bit-level cky ( §3.2) to jointly learn label and structural information. furthermore, we conduct a detailed analysis of several widely used contrastive learning losses, identifying the geometric center issue, and introduce a novel contrastive learning loss to remedy it ( §3.3) through carefully selecting instances. by doing so, we show that it is feasible to introduce binary representation to output layers and have them output binary labels on trees. moreover, since our model is based on contrastive learning, it also benefits from its remarkable representation learning capability, resulting in better performance than existing models. we conduct experiments on constituency parsing and nested named entity recognition. experimental results ( §4.2) demonstrate that our models achieve competitive performance with only around 12 and 8 bits, respectively. ## proposed methods our model attempts to learn parsing and hashing simultaneously with a single structured contrastive hashing loss. in other words, we try to introduce the binary representation to output layers and eliminate the need for the l task above. to achieve this, we first extend the cky module to support binary labels ( §3.1). then, we replace the cosine similarity with a newly defined similarity function ( §3.2) based on span marginal probabilities, because it contains not only label information but also structural information. after that, we analyze several commonly used contrastive losses, and propose a new one ( §3.3) to mitigate the geometric center issue as shown in figure 2 . after training, we build code vocabulary by mapping binary codes back to their most frequently coinciding labels. ## related work the ease of training has allowed contiguous representation @xcite to achieve great success in applications such as masked language models @xcite and causal language models @xcite @xcite . different from them, autoencoder @xcite @xcite and contrastive learning @xcite do not classify instances into specific classes, but rather reconstruct the input or determine whether a given pair belongs to the same class. the advantage of these approaches is that they do not require introducing embedding matrices, making representation learning more effective. however, because embeddings are not introduced, continuous vectors are hard to be discretized to specific classes and need to train an additional classifier for such purpose. learning discrete representation has also been widely focused on since the early era of deep learning @xcite @xcite , as it is a necessary component of many downstream applications. for example, languages are inherently discrete, and mapping discrete tokens to continuous vectors and back is the foremost step for modern natural language processing. additionally, many applications involving the control and interpretation of intermediate variables also require interleaving non-differentiable layers. however, the key challenge is that backpropagating gradients through discrete representation is generally intractable, making them hard to train. the most straightforward solution is the straightthrough estimator @xcite , which simply copies gradients across nondifferentiable layers, treating the jacobian matrix as an identity matrix. spigot @xcite further introduces a structured projection to reduce gradient errors. building upon this previous work, vq-vae (van den @xcite allows interleaving a discrete layer with continuous inputs mapped as discrete codes. these returned informative codes preserve necessary information for downstream tasks, and at the same time, provide an interface for controlling and interpreting the hidden states in intermediate layers of neural networks. another line of research turns to relaxation techniques, as they approximate discrete representation with continuous relaxation to mitigate the intractability issue. the gumbel-softmax estimator @xcite was first proposed by using the gumbel-max trick @xcite to provide a differentiable approximation of the arg max operation. the marginal approaches @xcite @xcite even further relax the discreteness restriction, not limiting themselves to selecting only a single discrete token, but instead aggregating information from all discrete tokens according to softmax distribution. different from all of them, our binary represen-tation lies between continuous and discrete representations. on the one hand, the easy-to-train property of continuous representation allows us to avoid designing complex gradient estimators while also benefiting from state-of-the-art representation learning methods, i.e., contrastive learning. on the other hand, the easy-to-discretize property allows us to instantly transform these continuous representations into discrete representations, thus making the imposition of various task-relevant structures tractable and efficient. besides, in other potential applications such as machine translation and language models, using binary representation removes the need for a large softmax, thus, reduces memory consumption and accelerates training and inference. additionally, the final step of re-interpreting binary codes as labels or tokens allows the model to preserve much more contextual information within the codes. consequently, this implicit clustering capability may potentially address issues like polysemy through discovering subclasses ( §4.4). ## conclusions in this paper, we demonstrated that introducing the binary representation to the output side is also feasible. binary representation inherits the advantages of easy training from contiguous representation and the structure modeling capability from discrete representation. therefore, binary representation can be considered as a novel representation that further bridges the gap between the continuous nature of deep learning and the discrete intrinsic property of natural languages. we achieved this by extending previous contrastive hashing into structured contrastive hashing. specifically, we designed a bit-level cky, defined a similarity function based on marginal probability, and proposed a novel contrastive hashing loss to mitigate the geometric center issue. experiments show that our methods perform remarkably on various structured prediction tasks by using only a few bits. our model also demonstrates a certain degree of implicit clustering capability in discovering subclasses. most importantly, although we primarily focus on structured prediction tasks, our method can be easily applied to other natural language processing tasks by equipping the corresponding structures.