C OMPRESSING E NCODING OF W ORDS FOR USE IN C HARACTER - LEVEL C ONVOLUTIONAL N ETWORKS FOR T EXT C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper reports the empirical exploration of a novel approach to encode words using compressing inspired techniques for use on convolutional neural networks at character-level.
This approach reduces drastically the number of parameters to be optimized using deep learning, resulting in competitive accuracies in only a fraction of the time spent by characters level convolutional neural networks, enabling training in simpler hardware.
I NTRODUCTION
Classiﬁcation of documents, that is, assigning a document or extract of a text to a category based on their content is a daily need of modern life.
Everyday we manipulate digital documents organizing them into folders, seeking to group characteristics, ideas and subjects for future references.
Such organization makes possible to relate other people’s thoughts and experiences for feed the development of our own ideas.
With the increasing availability of texts, especially through the internet, the need for artiﬁcial intelligence tools for automate this function is increasing very fast.
Spam detectors, sentiment analysis, news archiving are all technologies based on text classiﬁcators.
As a basic task in Natural Language Processing, the literature on text classiﬁers presents an extensive range of approaches to its realization (Aggarwal & Zhai, 2012;
Hotho et al., 2005; Kosala & Blockeel, 2000) .
Many are based on the discrimination of the word as an atom of the text structure and the model works through statistics of words occurrence (Zhang & LeCun, 2015).
The great variability of words and structures belonging to a language makes this task quite difﬁcult and
such models have superior quality mainly in very speciﬁc domains, where the vocabulary can be restricted to a much smaller number of words, possibly chosen by a specialist.
Moreover, such modeling becomes speciﬁc to a language, causing the replication process in another language to be carried out from scratch (Zhang & LeCun, 2015).
With the advent of Deep Learning methodologies (Goodfellow et al., 2016), some techniques have been updated but the clear majority are still based on the tokenization of words and the inference of their statistics.
LSTM (Hochreiter & Schmidhuber, 1997) and Word2Vec (Mikolov et al., 2013) are some of the most popular strategies.
The development of convolution techniques (LeCun et al., 1998) coupled with the popularization of parallel computing libraries for using Graphical Processing Units- GPUs (Bergstra et al., 2010;
Chollet et al., 2015) has been quite successful in image classiﬁcation (Krizhevsky et al., 2012).
The replication of this success in text classiﬁcation suffers, among other reasons, by the difﬁculty of representing one word as a vector.
Using the one hot encoding technique, where the vector of the term position is 1 and all other 0, would generate a representation of thousands or even millions of coordinates, making its use impractical both in terms of memory needed to store this vector, as time required to compute its optimizations.
Another problem is that typing errors or simple declensions of words in test data could not be codiﬁed if they are unseen on train data.
In an audacious paper, Zhang & LeCun (2015) suggest an approach that consider the character as the atomic level, reducing the vocabulary of symbols to only 69 characters and allowing the use of one hot encoding.
They considered each of the 1014 characters of a text represented by the line of a matrix where the column referring to the character of the vocabulary is 1 and all the others 0.
With this representation on hand, they applied a model with deep convolutional neural networks and obtained results competitive with other techniques, and in some cases the state of the art.
Building upon the work of Zhang & LeCun (2015)
[Table]
we suggest a novel approach to word representation that encode a word as a sequence of chars in a vector with size order of some hundreds, using the knowledge of character frequency imbalance, and inspired by traditional compression techniques.
This procedure made possible to represent larger portions of texts, perform the training in shorter times and applying simpler hardware, preserving the ability to codify any word, even unseen ones.
O UR APPROACH 2. E NCODING P ROCEDURE
In all of ours experiments
[Table]
we used the following procedure to encode words:
• Obtaining a character frequency rank : We read the text database and count the frequency of each character, generating a list sorted by frequency of occurrence.
Then we create a rank with only the relative positions of the characters.
For a given language, this rank is quite stable since only the order of the rank is used.
This means that if all documents are in the same idiom, this procedure can be replaced by a list with the characters rank of frequency for that language.
• Creating a mapping from character to compress code : To encode each character, we assign the value 1 to begin and end each char code.
Between these values, we insert the digitin the same amount of the rank position of the character.
Table 1 have some examples of encoded characters.
[Table]
’0’+’1’
To encode each word, we just concatenate the codes of each character.
As an example, we provide some codiﬁed words:
• science: • scien tist: • art: • artist:
• tl;dr:
[Table]
• u2:
[Table]
• e a (eˆa)
[Table]
Given a document, we consider that it is composed of words, being those any set of characters limited by the space character.
This means ”word” could be equations, web site address, latex code, computer commands, etc.
In the matrix representation of the document, each row represents a properly encoded word.
The code for each word is then embedded on a matrix of number of words × code size with its ﬁrst symbol in column 1.
Columns that were not occupied are padded with 0, larger codes are represented up to the chosen limit.
Unoccupied lines are ﬁlled with 0 and larger documents are represented only up to the chosen maximum number of words, ignoring the last remaining ones.
As an example, we represent a document in an 8x65 matrix in Table 2:
[Table]
We use 8x65 (520 elements) to encode the text with a certain amount of slack.
At the very least, we would need 7x64 (448 elements).
The approach of Zhang & LeCun (2015) would employ at least 69x32 (2208) elements to represent the same sentence.
In our experiments, 256 coordinates were enough to represent 99.5 of the words from one of the databases studied.
In all datasets studied in this paper, we choose 128 as a limit of words to represent a document, encoding each text in a 128x256 matrix.
2. T RAINING E NVIRONMENT AND N ETWORK ARCHITECTURE
For all the trainings, we used the environment and parameters settings listed on Table 3:
[Table]
We choose a very simple network architecture described in Table 4.
Our main concern was to be fast enough to execute tests in all datasets applied by Zhang & LeCun (2015) approach, enabling an unbiased comparison.
[Table] [Table]
[Table]
pooling1d
[Table] [Table]
[Table] [Table]
[Table]
conv1d 5 (5) (None, 164, 256) mergemax pooling1d 5 (3)
(None, 54, 256)
[Table]
conv1d 6 (5) (None, 50, 256)
[Table]
max pooling1d 6 (4) (None, 12, 256)
[Table]
ﬂatten(None, 3072)
[Table]
dense 1 (128)
(None, 128) ﬂattendense
[Table]
Output)
(None, #classes) (
[Table] [Table]
classes dense
To perform the training of this architecture with an output of 4 classes, it is necessary to optimize 1,837,188 parameters.
As a comparison, in the architecture suggested by Zhang and LeCun it is necessary to optimize 11,337,988 parameters (Zhang et al., 2015).
D ATASETS AND R ESULTS
The databases used were the same as those cited in an article by Zhang et al.
(2015), where there is an extensive description of them.
In this article, we will only summarize the main descriptions on
Table 5.
[Table]
One note is worth recall: the sogou news dataset is originally in Chinese.
Zhang & LeCun (2015) used software packages to produce transcriptions in Pinyin - a Chinese phonetic romanization.
We do not use any preprocessing strategy except the use of lowercase letters.
No data enhancement technique was employed.
The comparison results obtained are summarized in the Table 6. [Table] [Table] [Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
D ISCUSSION
The main objective of this research was to evaluate the possibility of using a coding approach that contemplated the construction of words using characters as basic tokens.
Our main contribution is demonstrate that such approach allow reduce the dimensionality of the encoding matrix, allowing substantial shorter optimization times and the use of devices with lower computational power.
The effort to use Neural Convolution Networks for text classiﬁcation tasks is justiﬁed by the possibility of appropriating tools from the recent developments of techniques, libraries and hardware used especially in the image classiﬁcation (LeCun et al., 1998; Bergstra et al., 2010; Chollet et al., 2015; Krizhevsky et al., 2012).
Such technologies can generate more general algorithms, facilitating the use of the same approach for several applications, independent of database language.
Establishing the character as the basic unit of word formation provides a better opportunity to be robust to typos, acceptance of neologisms, equations and chemical formulas, abbreviations and idiosyncrasies of written language on the internet, such as emoticons, slang and dialects of the online world.
Assuming the word as a base item, much of this ability is lost, especially when models assumes that text producers uses a formal language.
The article by Zhang & LeCun (2015) was a great innovation in this regard.
The results obtained by them suggest that language can be treated as a sequence of signals, like any other (Zhang &
LeCun, 2015).
However, the training times and the dimensionality of the matrices to be computed are still obstacles to the most effective use of the technique, especially by startups that could generate new products and ideas based on this technology.
Zhang & LeCun (2015), report that training of
Ag News database take 1-3 hours (60-180 min) per epoch, Dbpedia database lasted 5 hours ( min) per epoch and Amazon 2-5 days (2880-7200 min) per epoch on an Nvidia K40 GPU.
As a comparison, our trainings take 3 min, 18 min and 133 min per epoch, respectively on a NVIDIA
Geforce 1080ti and 25 min, 116 min and 744 min per epoch respectively on NVIDIA 930M 4GB, a much inferior GPU.
The improvement of performance is more impressive if we consider that our results were obtained in 5 epochs and Zhang & LeCun (2015) on 50 epochs!
Large portion of this time is still used to encode and deliver matrices to the GPUs.
To calculate the weights in each superbatch of 10000 arrays, a Geforce 1080ti GPU spends only 6 seconds.
In a more modest notebook with 16 Gb of RAM and a Nvidia Geforce 930M card, the optimization time for these same arrays is about 100 seconds, a reasonable timeframe even to motivate students and small startups who do not have access to a GPU as modern as GeForce 1080ti to use this approach.
The major bottleneck for analyzing this gigantic amount of matrixed text is the need for intensive use of RAM.
Our approach generates a 128x256 matrix, smaller than 1014x69 generated by Zhang & LeCun (2015), but a large set of them quickly occupy the available memory on the computer.
On the machine we used, there were 16 GB available, which is not uncommon in modern personal computers.
Therefore, the use of generators to control the number of matrices generated and sent to
GPU is an important detail in the implementation of this optimization algorithm.
If your computer has only 8 GB of RAM or less, you should reduce the number of superbatch to ﬁt the memory.
The encoding used in this article was inspired by the Tagged Huffman Code, developed by Silva de
Moura et al.
(2000), differing only that in the original conception, 2 digits 0 is added for each position in the rank of the characters.
As the goal was to use the smallest possible code to represent words in a vector of the document matrix, we chose to use only one 0 digit.
As in the Silva de Moura et al.
(2000) approach, the coding we employ has the following advantages (Brisaboa et al., 2003):
No character code is a preﬁx of another, that is, the match is one-to-one. 2.
It allows direct search, that is, to search for a word in a codiﬁed document, just encode the word and use traditional methods of comparing strings with the encoded document.
This encoding approach is a compression technique, so it also allows saving already encoded text documents permanently using a binary system, requiring less storage space.
These advantages become attractive especially if the goal is to extract knowledge about ﬁles in a repository to perform various classiﬁcations on different dimensions with the same ﬁles.
A possible advantage of this encoding is to better respond to unseen words on training data, once that the network at least have some preﬁx to guess the meaning of the word, the same way we humans do.
This is especially interesting in languages where there is a lot of declensions like portuguese, spanish, italian, french, russian, german, arabic and turkish, for example.
The coding procedure is not restricted to any size of vocabulary, the only problem is that less frequent characteres will generate bigger codes.
If your database have a lot of this characters, you could use a higher word code size.
The results obtained strongly indicate that the use of this coding is a possibility.
We emphasize that little effort was made to obtain a better network architecture.
The one we use is a fairly simple network, enough to demonstrate the feasibility of the approach with the time and computational resources that we have.
Possibly there are several architectures that can generate better results.
In addition, the dimensionality reduction provided enables several architectures to be veriﬁed in a reasonable time.
C ONCLUSION
The main conclusion is that using a compressing technique is a very convenient possibility to represent words for use in Convolutional Neural Networks to classify text.
Codifying words using character could be relevant specially on less curated texts datasets, being robust to typos, slangs and others usual characteristics of internet texts.
With our results, we reinforce Zhang et al.
(2015) conclusions, which states that language could be treated as a signal, no different of any other.
We demonstrated that using the approach suggested reduce the dimensionality of matrix representation of texts and allow the use of simpler hardware than character level one hot encoding, allowing a drastic reduction of training time.
We performed a comparative test with the approach of Zhang et al.
and others traditional techniques and found competitive results.
We speculate that new efforts should best employed in obtaining others architectures that may further favor this strategy.
A CKNOWLEDGMENTS
We gratefully acknowledge the support of NVIDIA Corporation with the donation of a GPU used for this research.
The ﬁrst author is grateful to the CAPES for the award of a research scholarship.