Original: P REDICTING A UCTION P RICE OF V EHICLE L ICENSE P LATE WITH D EEP R ECURRENT N EURAL N ETWORK
Revised: P REDICTING A UCTION P RICE OF V EHICLE L ICENSE P LATE WITH D EEP R ECURRENT N EURAL N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In Chinese societies, superstition is of paramount importance, and vehicle license plates with desirable numbers can fetch very high prices in auctions.
Revised: In Chinese societies, superstition is of paramount importance, and vehicle license plates with desirable numbers can fetch very high prices in auctions.

Original: Unlike other valuable items, license plates are not allocated an estimated price before auction.
Revised: Unlike other valuable items, license plates are not allocated an estimated price before auction.

Original: I propose that the task of predicting plate prices can be viewed as a natural language processing (NLP) task, as the value depends on the meaning of each individual character on the plate and its semantics.
Revised: I propose that the task of predicting plate prices can be viewed as a natural language processing (NLP) task, as the value depends on the meaning of each individual character on the plate and its semantics.

Original: I construct a deep recurrent neural network (RNN) to predict the prices of vehicle license plates in Hong Kong, based on the characters on a plate.
Revised: I construct a deep recurrent neural network (RNN) to predict the prices of vehicle license plates in Hong Kong, based on the characters on a plate.

Original: I demonstrate the importance of having a deep network and of retraining.
Revised: I demonstrate the importance of having a deep network and of retraining.

Original: Evaluated on 13 years of historical auction prices, the deep
Revised: Evaluated on 13 years of historical auction prices, the deep

Original: RNN’s predictions can explain over 80 percent of price variations, outperforming previous models by a signiﬁcant margin.
Revised: RNN’s predictions can explain over 80 percent of price variations, outperforming previous models by a signiﬁcant margin.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Chinese societies place great importance on numerological superstition.
Revised: Chinese societies place great importance on numerological superstition.

Original: Numbers such as 8 (representing prosperity) and 9 (longevity) are often used solely because of the desirable qualities they represent.
Revised: Numbers such as 8 (representing prosperity) and 9 (longevity) are often used solely because of the desirable qualities they represent.

Original: For example, the Beijing Olympic opening ceremony occurred on 2008/8/8 at 8 p.m., the
Revised: For example, the Beijing Olympic opening ceremony occurred on 2008/8/8 at 8 p.m., the

Original: Bank of China (Hong Kong) opened on 1988/8/8, and the Hong Kong dollar is linked to the U.S. dollar at a rate of around 7.8.
Revised: Bank of China (Hong Kong) opened on 1988/8/8, and the Hong Kong dollar is linked to the U.S. dollar at a rate of around 7.8.

Original: License plates represent a very public display of numbers that people can own, and can therefore unsurprisingly fetch an enormous amount of money.
Revised: License plates represent a very public display of numbers that people can own, and can therefore unsurprisingly fetch an enormous amount of money.

Original: Governments have not overlooked this, and plates of value are often auctioned off to generate public revenue.
Revised: Governments have not overlooked this, and plates of value are often auctioned off to generate public revenue.

Original: Unlike the auctioning of other valuable items, however, license plates generally do not come with a price estimate, which has been shown to be a signiﬁcant factor affecting the sale price Ashenfelter (1989); Milgrom & Weber (1982).
Revised: Unlike the auctioning of other valuable items, however, license plates generally do not come with a price estimate, which has been shown to be a signiﬁcant factor affecting the sale price (Ashenfelter, 1989; Milgrom & Weber, 1982).

Original: The large number of character combinations and of plates per auction makes it difﬁcult to provide reasonable estimates.
Revised: The large number of character combinations and of plates per auction makes it difﬁcult to provide reasonable estimates.

Original: This study proposes that the task of predicting a license plate’s price based on its characters can be viewed as a natural language processing (NLP) task.
Revised: This study proposes that the task of predicting a license plate’s price based on its characters can be viewed as a natural language processing (NLP) task.

Original: Whereas in the West numbers can be desirable (such as 7) or undesirable (such as 13) in their own right for various reasons, in Chinese societies numbers derive their superstitious value from the characters they rhyme with.
Revised: Whereas in the West numbers can be desirable (such as 7) or undesirable (such as 13) in their own right for various reasons, in Chinese societies numbers derive their superstitious value from the characters they rhyme with.

Original: As the
Revised: As the

Original: Chinese language is logosyllabic and analytic, combinations of numbers can stand for sound-alike phrases.
Revised: Chinese language is logosyllabic and analytic, combinations of numbers can stand for sound-alike phrases.

Original: Combinations of numbers that rhyme with phrases that have positive connotations are thus desirable.
Revised: Combinations of numbers that rhyme with phrases that have positive connotations are thus desirable.

Original: For example, “168,” which rhythms with “all the way to prosperity” in Chinese, is the URL of a major Chinese business portal (http://www.168.com).
Revised: For example, “168,” which rhythms with “all the way to prosperity” in Chinese, is the URL of a major Chinese business portal (http://www.168.com).

Original: Looking at the historical data analyzed in this study, license plates with the number 168 fetched an average price of US$10,094 and as much as $113,462 in one instance.
Revised: Looking at the historical data analyzed in this study, license plates with the number 168 fetched an average price of US$10,094 and as much as $113,462 in one instance.

Original: Combinations of numbers that rhyme with phrases possessing negative connotations are equally undesirable.
Revised: Combinations of numbers that rhyme with phrases possessing negative connotations are equally undesirable.

Original: Plates with the number 888 are generally highly sought after, selling for an average of $4,105 in the data, but adding a 5 (rhymes with “no”) in front drastically lowers the average to $342.
Revised: Plates with the number 888 are generally highly sought after, selling for an average of $4,105 in the data, but adding a 5 (rhymes with “no”) in front drastically lowers the average to $342.

Original: As these examples demonstrate, the value of a certain combination of characters depends on both the meaning of each individual character and the broader semantics.
Revised: As these examples demonstrate, the value of a certain combination of characters depends on both the meaning of each individual character and the broader semantics.

Original: The task at hand is thus closely related to sentiment analysis and machine translation, both of which have advanced signiﬁcantly in recent years.
Revised: The task at hand is thus closely related to sentiment analysis and machine translation, both of which have advanced signiﬁcantly in recent years.

Original: Using a deep recurrent neural network (RNN), I demonstrate that a good estimate of a license plate’s price can be obtained.
Revised: Using a deep recurrent neural network (RNN), I demonstrate that a good estimate of a license plate’s price can be obtained.

Original: The predictions from this study’s deep RNN were signiﬁcantly more accurate than previous attempts to model license plate prices, and are able to explain over 80 percent of price variations.
Revised: The predictions from this study’s deep RNN were signiﬁcantly more accurate than previous attempts to model license plate prices, and are able to explain over 80 percent of price variations.

Original: There are two immediate applications of the ﬁndings in this paper: ﬁrst, an accurate prediction model facilitates arbitrage, allowing one to detect underpriced plates that can potentially fetch for a higher price in the active second-hand market.
Revised: There are two immediate applications of the ﬁndings in this paper: ﬁrst, an accurate prediction model facilitates arbitrage, allowing one to detect underpriced plates that can potentially fetch for a higher price in the active second-hand market.

Original: Second, the feature vectors extracted from the last recurrent layer of the model can be used to construct a search engine for historical plate prices.
Revised: Second, the feature vectors extracted from the last recurrent layer of the model can be used to construct a search engine for historical plate prices.

Original: among other uses, the search engine can provide highly-informative justiﬁcation for the predicted price of any given plate.
Revised: Among other uses, the search engine can provide highly-informative justiﬁcation for the predicted price of any given plate.

Original: In a more general sense, this study demonstrates the value of deep networks and NLP in making accurate price predictions, which is of practical importance in many industries and has led to a huge volume of research.
Revised: In a more general sense, this study demonstrates the value of deep networks and NLP in making accurate price predictions, which is of practical importance in many industries and has led to a huge volume of research.

Original: As detailed in the following review, studies to date have relied on small, shallow networks.
Revised: As detailed in the following review, studies to date have relied on small, shallow networks.

Original: The use of text data is also rare, despite the large amount of business text data available.
Revised: The use of text data is also rare, despite the large amount of business text data available.

Original: By demonstrating how a deep network can be trained to predict prices from sequential data, this study provides an approach that may improve prediction accuracy in many industrial applications.
Revised: By demonstrating how a deep network can be trained to predict prices from sequential data, this study provides an approach that may improve prediction accuracy in many industrial applications.

Original: The paper is organized as follows: Section 2 describes Hong Kong license plate auctions, followed by a review of related studies in Section 3.
Revised: The paper is organized as follows: Section 2 describes Hong Kong license plate auctions, followed by a review of related studies in Section 3.

Original: Section 4 details the model, which is tested in Section 5.
Revised: Section 4 details the model, which is tested in Section 5.

Original: Section 7 explores enhancements to the model, including the effect of retraining over time.
Revised: Section 6 explores enhancements to the model, including the effect of retraining over time.

Original: Sectionconcludes the paper, and includes a discussion of further developments that have potential practical uses.
Revised: Sectionconcludes the paper, and includes a discussion of further developments that have potential practical uses.

Original:  L ICENSE P LATE A UCTIONS IN H ONG K ONG
Revised:  L ICENSE P LATE A UCTIONS IN H ONG K ONG

Original: License plates have been sold through government auctions in Hong Kong since 1973, and restrictions are placed on the reselling of plates.
Revised: License plates have been sold through government auctions in Hong Kong since 1973, and restrictions are placed on the reselling of plates.

Original: Between 1997 and 2009, 3,812 plates were auctioned per year, on average.
Revised: Between 1997 and 2009, 3,812 plates were auctioned per year, on average.

Original: Traditional plates, which were the only type available before September 2006, consist of either a two-letter preﬁx or no preﬁx, followed by up to four digits (e.g., AB 1, LZ 3360, or 168).
Revised: Traditional plates, which were the only type available before September 2006, consist of either a two-letter preﬁx or no preﬁx, followed by up to four digits (e.g., AB 1, LZ 3360, or 168).

Original: Traditional plates can be divided into the mutually exclusive categories of special plates and ordinary plates.
Revised: Traditional plates can be divided into the mutually exclusive categories of special plates and ordinary plates.

Original: Special plates are deﬁned by a set of legal rules and include the most desirable plates.
Revised: Special plates are deﬁned by a set of legal rules and include the most desirable plates.

Original: 2 Ordinary plates are issued by the government when a new vehicle is registered.
Revised: 1 Ordinary plates are issued by the government when a new vehicle is registered.

Original: If the vehicle owner does not want the assigned plate, she can return the plate and bid for another in an auction.
Revised: If the vehicle owner does not want the assigned plate, she can return the plate and bid for another in an auction.

Original: The owner can also reserve any unassigned plate for auction.
Revised: The owner can also reserve any unassigned plate for auction.

Original: Only ordinary plates can be resold.
Revised: Only ordinary plates can be resold.

Original: In addition to traditional plates, personalized plates allow vehicle owners to propose the string of characters used.
Revised: In addition to traditional plates, personalized plates allow vehicle owners to propose the string of characters used.

Original: These plates must then be purchased from auctions.
Revised: These plates must then be purchased from auctions.

Original: The data used in this study do not include this type of plate.
Revised: The data used in this study do not include this type of plate.

Original: Auctions are open to the public and held on weekends twice a month by the Transport Department.
Revised: Auctions are open to the public and held on weekends twice a month by the Transport Department.

Original: The number of plates to be auctioned ranged from 90 per day in the early years to 280 per day in later years, and the list of plates available is announced to the public well in advance.
Revised: The number of plates to be auctioned ranged from 90 per day in the early years to 280 per day in later years, and the list of plates available is announced to the public well in advance.

Original: The English oral ascending auction format is used, with payment settled on the spot, either by debit card or check.
Revised: The English oral ascending auction format is used, with payment settled on the spot, either by debit card or check.

Original:  R ELATED S TUDIES
Revised:  R ELATED S TUDIES

Original: Most relevant to the current study is the limited literature on the modeling price of license plates, which uses hedonic regressions with a larger number of handcrafted features Woo & Kwok (1994);
Revised: Most relevant to the current study is the limited literature on the modeling price of license plates, which uses hedonic regressions with a larger number of handcrafted features (Woo & Kwok, 1994;

Original: Woo et al. (2008); Ng et al.
Revised: Woo et al., 2008; Ng et al., 2010).

Original: These highly ad-hoc models rely on handcrafted features, so they adapt poorly to new data, particularly if they include combinations of characters not previously seen.
Revised: These highly ad-hoc models rely on handcrafted features, so they adapt poorly to new data, particularly if they include combinations of characters not previously seen.

Original: In contrast, the deep RNN considered in this study learns the value of each combination of characters from its auction price, without the involvement of any handcrafted features.
Revised: In contrast, the deep RNN considered in this study learns the value of each combination of characters from its auction price, without the involvement of any handcrafted features.

Original: The literature on using neural networks to make price predictions is very extensive and covers areas such as stock prices Baba & Kozaki (1992); Olson & Mossman (2003); Guresen et al. (2011); de Oliveira et al. (2013), commodity prices Kohzadi et al. ; Kristjanpoller & Minutolo (2015; 2016), real estate prices Do & Grudnitski (1992); Evans et al.
Revised: The literature on using neural networks to make price predictions is very extensive and covers areas such as stock prices (Baba & Kozaki, 1992; Olson & Mossman, 2003; Guresen et al., 2011; en public services/auction_of_vehicle_registration_marks/ how_to_obtain_your_favourite_vehicle_registration/schedule/index.html .

Original: ; Worzola et al.

Original:  electricity prices Weron (2014); Dudek (2016), movie revenues Sharda & Delen (2006); Yu et al. Zhang et al. ; Ghiassi et al. (2015), automobile prices Iseri & Karlik (2009) and food prices 
Revised: de Oliveira et al., 2013), commodity prices (Kohzadi et al., 1996; Kristjanpoller & Minutolo, 2015; 2016), real estate prices (Do & Grudnitski, 1992; Evans et al., 1992; Worzola et al., 1995), electricity prices (Weron, 2014; Dudek, 2016), movie revenues (Sharda & Delen, 2006; Yu et al., 2008; Zhang et al., 2009; Ghiassi et al., 2015), automobile prices (Iseri & Karlik, 2009) and food prices (Haofei et al., 2007).

Original: Haofei et al.

Original: Most studies focus on numeric data and use small, shallow networks, typically using a single hidden layer of fewer than 20 neurons.
Revised: Most studies focus on numeric data and use small, shallow networks, typically using a single hidden layer of fewer than 20 neurons.

Original: The focus of this study is very different: predicting prices from combinations of alphanumeric characters.
Revised: The focus of this study is very different: predicting prices from combinations of alphanumeric characters.

Original: Due to the complexity of this task, the networks used are much larger (up to 1,024 hidden units per layer) and deeper (up to 9 layers).
Revised: Due to the complexity of this task, the networks used are much larger (up to 1,024 hidden units per layer) and deeper (up to 9 layers).

Original: The approach is closely related to sentiment analysis.
Revised: The approach is closely related to sentiment analysis.

Original: A particularly relevant line of research is the use of Twitter feeds to predict stock price movements Bollen et al. ; Bing et al.
Revised: A particularly relevant line of research is the use of Twitter feeds to predict stock price movements (Bollen et al., 2011; Bing et al., 2014;

Original: ; Pagolu et al. (2016), although the current study has signiﬁcant differences.
Revised: Pagolu et al., 2016), although the current study has signiﬁcant differences.

Original: A single model is used in this study to generate predictions from character combinations, rather than treating sentiment analysis and price prediction as two distinct tasks, and the actual price level is predicted rather than just the direction of price movement.
Revised: A single model is used in this study to generate predictions from character combinations, rather than treating sentiment analysis and price prediction as two distinct tasks, and the actual price level is predicted rather than just the direction of price movement.

Original: This end-to-end approach is feasible because the causal relationship between sentiment and price is much stronger for license plates than for stocks.
Revised: This end-to-end approach is feasible because the causal relationship between sentiment and price is much stronger for license plates than for stocks.

Original: Deep RNNs have been shown to perform very well in tasks that involve sequential data, such as machine translation Cho et al.

Original: ; Sutskever et al.
Revised: Finally, Akita et al.

Original: (2014); Zaremba et al. Amodei et al. (2016) and classiﬁcation based on text description
Revised: (2016) utilizes a Long-Short-Term Memory (LSTM) network to study the collective price movements of 10 Japanese stocks.

Original: Ha et al. and are therefore used in this study.
Revised: The neural network in that study was solely used as a time-series model, taking in vectorized textual information from two simplier, non-neural-networkbased models. In contrast, this study utilizies a neural network directly on textual information. Deep RNNs have been shown to perform very well in tasks that involve sequential data, such as machine translation (Cho et al., 2014; Sutskever et al., 2014; Zaremba et al., 2014; Amodei et al., 2016) and classiﬁcation based on text description (Ha et al., 2016), and are therefore used in this study.

Original: Predicting the price of a license plate is relatively simple: the model only needs to predict a single value based on a string of up to six characters.
Revised: Predicting the price of a license plate is relatively simple: the model only needs to predict a single value based on a string of up to six characters.

Original: This simplicity makes training feasible on the relatively small volume of license plate auction data used in the study, compared with datasets more commonly used in training deep RNN.
Revised: This simplicity makes training feasible on the relatively small volume of license plate auction data used in the study, compared with datasets more commonly used in training deep RNN.

Original:  M ODELING L ICENSE P LATE P RICE WITH A D EEP R ECURRENT N EURAL N ETWORK
Revised:  M ODELING L ICENSE P LATE P RICE WITH A D EEP R ECURRENT N EURAL N ETWORK

Original: The input from each sample is an array of characters (e.g., [“X,” “Y,” “1,” “2,” “8”]), padded to the same length with a special character.
Revised: The input from each sample is an array of characters (e.g., [“X,” “Y,” “1,” “2,” “8”]), padded to the same length with a special character.

Original: Each character s t is converted by a lookup table  to a vector representation h t 0 , known as character embedding .
Revised: Each character s t is converted by a lookup table g to a vector representation (cid:126)h t 0 , known as character embedding :

Original:  The dimension of the character embedding is a hyperparameter while the values  are  learned through training.
Revised: [Equation] The dimension of the character embedding, n , is a hyperparameter. The values h t 0 , 1 , ..., h t 0 ,n are initialized with random values and learned through training.

Original: The embedding is fed into the neural network sequentially, denoted by the time step t .
Revised: The embedding is fed into the neural network sequentially, denoted by the time step t .

Original: The neural network consists of multiple bidirectional recurrent layers, followed by one or more fully connected layers Schuster & Paliwal (1997).
Revised: The neural network consists of multiple bidirectional recurrent layers, followed by one or more fully connected layers (Schuster & Paliwal, 1997).

Original: The bidirectionality allows the network to access hidden states from both the previous and next time steps, improving its ability to understand each character in context.
Revised: The bidirectionality allows the network to access hidden states from both the previous and next time steps, improving its ability to understand each character in context.

Original: The network also uses batch normalization, which has been shown to speed up convergence Laurent et al.
Revised: The network also uses batch normalization, which has been shown to speed up convergence (Laurent et al., 2016).

Original: Each recurrent layer is implemented as follows:
Revised: Each recurrent layer is implemented as follows:

Original: [Equation]
Revised: [Equation]

Original: where f is the rectiﬁed-linear unit, h tl − 1 is the vector of activations from the previous layer at the same time step t ,  h t − 1 l represents the activations from the current layer at the previous time step  t − 1 , and h t +1 l represents the activations from the current layer at the next time step t + 1 .
Revised: where f is the rectiﬁed-linear unit, (cid:126)h tl − 1 is the vector of activations from the previous layer at the same time step t , (cid:126) h t − 1 l represents the activations from the current layer at the previous time step [Figure] t − 1 , and (cid:126)h t +1 l represents the activations from the current layer at the next time step t + 1 .

Original: B is the
Revised: B is the

Original: BatchNorm transformation, and ˆ x is the within-mini-batch-standardized version of x . 3 W , U , γ and β are weights learnt by the network through training.
Revised: BatchNorm transformation, and ˆ x is the within-mini-batch-standardized version of x . 2 W , U , γ and β are weights learnt by the network through training.

Original: [Figure]

Original: The fully connected layers are implemented as
Revised: The fully connected layers are implemented as

Original: [Equation]
Revised: [Equation]

Original: except for the last layer, which is implemented as
Revised: except for the last layer, which is implemented as

Original: [Equation]
Revised: [Equation]

Original: b l is a bias vector learnt from training.
Revised: b l is a bias vector learnt from training.

Original: The outputs from all time steps in the ﬁnal recurrent layer are added together before being fed into the ﬁrst fully connected layer.
Revised: The outputs from all time steps in the ﬁnal recurrent layer are added together before being fed into the ﬁrst fully connected layer.

Original: To prevent overﬁtting, dropout is applied after every layer except the last Hinton et al.
Revised: To prevent overﬁtting, dropout is applied after every layer except the last (Hinton et al., 2012).

Original: The model’s hyperparameters include the dimension of character embeddings, number of recurrent layers, number of fully connected layers, number of hidden units in each layer, and dropout rate.
Revised: The model’s hyperparameters include the dimension of character embeddings, number of recurrent layers, number of fully connected layers, number of hidden units in each layer, and dropout rate.

Original: These parameters must be selected ahead of training.
Revised: These parameters must be selected ahead of training.

Original:  E XPERIMENT 5. D ATA
Revised:  E XPERIMENT 5. D ATA

Original: The data used are the Hong Kong license plate auction results from January 1997 to July 2010, obtained from the HKSAR government.
Revised: The data used are the Hong Kong license plate auction results from January 1997 to July 2010, obtained from the HKSAR government.

Original: The data contain 52,926 auction entries, each consisting of i. the characters on the plate, ii.
Revised: The data contain 52,926 auction entries, each consisting of i. the characters on the plate, ii.

Original: the sale price (or a speciﬁc symbol if the plate was unsold), and iii.
Revised: the sale price (or a speciﬁc symbol if the plate was unsold), and iii.

Original: the auction date.
Revised: the auction date.

Original: Figure 1 plots the distribution of prices within the data.
Revised: Figure 2 plots the distribution of prices within the data.

Original: The ﬁgure shows that the prices are highly skewed: while the median sale price is $641, the mean sale price is $2,073.
Revised: The ﬁgure shows that the prices are highly skewed: while the median sale price is $641, the mean sale price is $2,073.

Original: The most expensive plate in the data is “12,” which was sold for $910,256 in February 2005.
Revised: The most expensive plate in the data is “12,” which was sold for $910,256 in February 2005.

Original: To compensate for this skewness, log prices were used in training and inference.
Revised: To compensate for this skewness, log prices were used in training and inference.

Original: Ordinary plates start at a reserve price of HK$1,000 ($128.2), with $5,000 ($644.4) for special plates.
Revised: Ordinary plates start at a reserve price of HK$1,000 ($128.2), with $5,000 ($644.4) for special plates.

Original: The reserve prices mean that not every plate is sold, and 5.1 percent of the plates in the data were unsold.
Revised: The reserve prices mean that not every plate is sold, and 5.1 percent of the plates in the data were unsold.

Original: As these plates did not possess a price, we followed previous studies in dropping them from the dataset, leaving 50,698 entries available for the experiment.
Revised: As these plates did not possess a price, we followed previous studies in dropping them from the dataset, leaving 50,698 entries available for the experiment.

Revised: The ﬁnalized data were divided into three parts, in two different ways: the ﬁrst way divided the data randomly, while the second divided the data sequentially into non-overlapping parts.

Revised: The second way creates a more realistic scenario, as it represents what a model in practical deployment would face.

Revised: It is also a signiﬁcantly more difﬁcult scenario: because the government releases plates alphabetically through time, plates that start with later alphabets would not be available in sequentiallysplit data.

Revised: For example, plates that start with “M” were not available before 2005, and plates that start with “P” would not until 2010.

Revised: It is therefore very difﬁcult for a model trained on sequentiallysplit data to learn the values of plates starting with later alphabets.

Original: The ﬁnalized data were randomly divided into three: training was conducted with 64 percent of the data, validation was conducted with 16 percent, and the remaining 20 percent served as the test set.
Revised: In both cases, training was conducted with 64 percent of the data, validation was conducted withpercent, and the remaining 20 percent served as the test set.

Original: 5. T RAINING
Revised: 5. T RAINING

Original: I conducted a grid search to investigate the properties of different combinations of hyperparameters, varying the dimension of character embeddings ( 12 , 24 , 48 , 96 , 128 , 256 ), the number of recurrent layers ( 1 , 3 , 5 , 7 , 9 ), the number of fully connected layers ( 1 , 3 ), the number of hidden units in each layer ( 64 , 128 , 256 , 512 , 1024 , 2048 ) and the dropout rate ( 0 , .
Revised: I conducted a grid search to investigate the properties of different combinations of hyperparameters, varying the dimension of character embeddings ( 12 , 24 , 48 , 96 , 128 , 256 ), the number of recurrent layers ( 1 , 3 , 5 , 7 , 9 ), the number of fully connected layers ( 1 , 3 ), the number of hidden units in each layer ( 64 , 128 , 256 , 512 , 1024 , 2048 ) and the dropout rate ( 0 , .

Original: A total of 1080 sets of hyperparameters were investigated.
Revised: A total of 1080 sets of hyperparameters were investigated.

Original: Training was repeated under each set of hyperparameters 30 times with different initializations. During each training session, a network was trained for 40 epochs under mean-squared error.
Revised: The grid search was conducted in three passes: In the ﬁrst pass, a network was trained for 40 epochs under each set of hyperparameters, repeated 4 times. In the second pass, training was repeatedtimes for each of the 10 best sets of hyperparameters from the ﬁrst pass, based on median validation

Original: An
Revised: RMSE.

Original: [Table]
Revised: In the ﬁnal pass, training was repeated for 30 times under the best set of hyperparameters from the second pass, again based on median validation RMSE. Training duration in the second and the third passes was 120 epochs. During each training session, a network was trained under mean-squared error with different initializations.

Original: Adagrad optimizer with a learning rate of 0.001and a gradient clip of 15 was used throughout Duchi et al.
Revised: An Adam optimizer with a learning rate of 0.001 was used throughout (Kingma & Ba, 2014).

Original: After training was completed, the best state based on the validation error was reloaded for inference.
Revised: After training was completed, the best state based on the validation error was reloaded for inference.

Original: Training was conducted with a pair of NVIDIA GTX 1080s.
Revised: Training was conducted with four of NVIDIA GTX 1080s.

Original: To fully use the GPUs, a large minibatch size of 2,048 was used.
Revised: To fully use the GPUs, a large mini-batch size of 2,048 was used.

Original: The median training time on a single GPU ranged from 8 seconds for a 2-layer, 64-hidden-unit network with an embedding dimension of 12, to 1 minute 57 seconds for an 8-layer, 1,024-hidden-unit network with an embedding dimension of 24, and to 7 minutesseconds for a 12-layer 2,048-hidden-unit network with an embedding dimension of 256.
Revised: During the ﬁrst pass, the median training time on a single GPU ranged from 8 seconds for a 2-layer, 64-hidden-unit network with an embedding dimension of 12, to 1 minute 57 seconds for an 8-layer, 1,024-hidden-unit network with an embedding dimension of 24, and tominutes 50 seconds for a 12-layer 2,048-hidden-unit network with an embedding dimension of 256.

Original: Finally, I also trained recreations of models from previous studies as well as a series of fullyconnected networks and character n -gram models for comparison.
Revised: Finally, I also trained recreations of models from previous studies as well as a series of fullyconnected networks and character n -gram models for comparison.

Original: Given that the maximum length of a plate is six characters, for the n -gram models I focused on n ≤ 4 , and in each case calculated a predicted price based on the median and mean of k closest neighbors from the training data, where k = 1 , 3 , 5 , 10 , 20 .
Revised: Given that the maximum length of a plate is six characters, for the n -gram models I focused on n ≤ 4 , and in each case calculated a predicted price based on the median and mean of k closest neighbors from the training data, where k = 1 , 3 , 5 , 10 , 20 .

Original: 5. M ODEL P ERFORMANCE
Revised: 5. M ODEL P ERFORMANCE

Original: Table 1 reports the summary statistics for the best two sets of parameters out of the 1080 sets speciﬁed in section 5.2, based on the median validation RMSE.
Revised: Table 1 reports the summary statistics for the set of parameters out of the 1080 sets speciﬁed in section 5.2, based on the median validation RMSE.

Original: The performance levels of these models were quite close, with both able to explain more than 80 percent of the variation in prices.
Revised: The model was able to explain more thanpercent of the variation in prices when the data was randomly split.

Original: As a comparison, Woo et al.
Revised: As a comparison, Woo et al.

Original: (2008) and Ng et al.
Revised: (2008) and Ng et al.

Original: (2010) , which represent recreations of the regression models in Woo et al. (2008) and Ng et al. (2010), respectively, were capable of explaining onlypercent of the variation at most.
Revised: (2010) , which represent recreations of the regression models in (Woo et al., 2008) and (Ng et al., 2010), respectively, were capable of explaining only 70 percent of the variation at most.

Original: The importance of having recurrent layers can be seen from the inferior performance of the fullyconnected network (MLP) with the same embedded dimension, number of layers and neurons as the best RNN model.
Revised: The importance of having recurrent layers can be seen from the inferior performance of the fullyconnected network (MLP) with the same embedded dimension, number of layers and neurons as the best RNN model.

Original: This model was only capable of explaining less than 66 percent of the variation in prices.
Revised: This model was only capable of explaining less than 66 percent of the variation in prices.

Original: Even with double the neurons a MLP was only capable of explaining 69 percent of the variation.
Revised: [Table] [Table] [Table]

Original: In the interest of space, I include only two best-performing n -gram models based on median prices of neighbors.
Revised: In the interest of space, I include only two best-performing n -gram models based on median prices of neighbors.

Original: Both models were signiﬁcantly inferior to RNN and hedonic regressions, being able to explain only 40 percent of the variation in prices.
Revised: Both models were signiﬁcantly inferior to RNN and hedonic regressions, being able to explain only 40 percent of the variation in prices.

Original: For unigram, the best validation performance [Figure] al (2008) [Figure]
Revised: For unigram, the best validation performance was achieved when k = 10 .

Original: et al ( [Figure] was achieved when k = 10 . For n > 2 , models with unlimited features have very poor performance, as they generate a large number of features that rarely appear in the data.
Revised: For n > 2 , models with unlimited features have very poor performance, as they generate a large number of features that rarely appear in the data.

Original: Restricting the number of features based on occurances and allowing a range of n within a single model improve performance, but never surpassing the performance of the simple unigram.
Revised: Restricting the number of features based on occurances and allowing a range of n within a single model improve performance, but never surpassing the performance of the simple unigram.

Original: The performance of using median price and using mean price are very close, with a difference smaller than 0.05 in all cases.
Revised: The performance of using median price and using mean price are very close, with a difference smaller than 0.05 in all cases.

Revised: All models took a signiﬁcant performance hit when the data was split sequentially, with the RNN maintaining its performance lead over other models.

Revised: The impact was particularly severe for the test set, because it was drawn from a time period furthest away from that of the train set.

Original: Figure 2 plots the relationship between predicted price and actual price from a representative run of the best model, grouped in bins of HK$1,000 ($128.2).
Revised: Figure 3 plots the relationship between predicted price and actual price from a representative run of the best model, grouped in bins of HK$1,000 ($128.2).

Original: The model performed well for a wide range of prices, with bins tightly clustered along the 45-degree line.
Revised: The model performed well for a wide range of prices, with bins tightly clustered along the 45-degree line.

Original: It consistently underestimated the price of the most expensive plates, however, suggesting that the buyers of these plates had placed on them exceptional value that the model could not capture.
Revised: It consistently underestimated the price of the most expensive plates, however, suggesting that the buyers of these plates had placed on them exceptional value that the model could not capture.

Original: Figure 4 plots the variation in performance as the hyperparameters deviate from the best-performing model.

Original: The effectiveness of deep networks has been previously noted and is also demonstrated here.

Original: Performance improved signiﬁcantly with a hidden unit count and a recurrent layer count, leveling off around 512 hidden units and 7 recurrent layers.

Original: Under the hyperparameters of the best model, the median RMSE of a 1-layer model was 35 percent higher than that of a 7-layer model, while that of a 64-units-per-layer model was 11 times that of a 1,024-units-per-layer model.

Original: However, there appears to be no beneﬁt in stacking fully connected layers: the model with three fully connected layers had a median RSME 24 percent higher than that of the one-layer version.

Original: Performance peaked out relatively early with the dimensionality of character embedding, which is not surprising given there were only 33 possible characters.

Original: 6 Unlike the hidden unit count or the recurrent layer count, there is a clear sweet spot for the dimensionality of embedding, and performance worsens rapidly as the dimensionality increases beyond that point.

Original: A small amount of dropout was necessary to achieve good performance.

Original: Without dropout, the model was much more likely to converge to local maxima, resulting in poor ﬁt in many cases.

Original: 5. M ODEL S TABILITY
Revised: 5. M ODEL S TABILITY

Original: Unlike hedonic regressions, which give the same predictions and achieve the same performance in every run, a neural network is susceptible to ﬂuctuations due to convergence to local maxima.
Revised: Unlike hedonic regressions, which give the same predictions and achieve the same performance in every run, a neural network is susceptible to ﬂuctuations due to convergence to local maxima.

Original: These[Figure] [Table] ﬂuctuations can be smoothed out by combining the predictions of multiple runs of the same model, although the number of runs necessary to achieve good results is then a practical concern.
Revised: These ﬂuctuations can be smoothed out by combining the predictions of multiple runs of the same model, although the number of runs necessary to achieve good results is then a practical concern.

Original: Figure 3 plots the kernel density estimate of validation RMSEs for the best model’s 30 training runs.
Revised: Figure 4 plots the kernel density estimate of validation RMSEs for the best model’s 30 training runs.

Original: The errors are tightly clustered, with only one run producing particularly inaccurate predictions.
Revised: The errors are tightly clustered, with only one run producing particularly inaccurate predictions.

Original: Excluding that run, the standard deviation of validation RMSE was only 0.034, which suggests that in practice several runs should sufﬁce.
Revised: Excluding that run, the standard deviation of validation RMSE was only 0.027, which suggests that in practice several runs should sufﬁce.

Original: E XPLAINING THE  P REDICTIONS
Revised:  P ERFORMANCE E NHANCEMENTS 6. R ETRAINING O VER T IME

Original: Compared to models such as regression and n -gram it is relatively hard to understand the rationale behind a RNN model’s prediction, given the large number of parameters involved and the complexity of the their interaction. If the RNN model is to be deployed in the ﬁeld, it would need to be able to explain its prediction in order to convince human users to adopt it in practice. One way to do so is to extract a feature vector for each plate by summing up the output of the last recurrent layer over time.

Original: This feature vector is of the same size as the number of neurons in the last layer, which can be fed into astandard k -nearest-neighbor model to provide a “rationale” for the model’s prediction. To demonstrate this procedure, I use the best RNN model in Table 1 to generate feature vectors for all training samples. These samples are used to setup a k -NN model. When the user submit a query, a price prediction is made with the RNN model, while a number of examples are provided by the k -NN model as rationale.
Revised: Over time, a model could conceivably become obsolete if, for example, taste or the economic environment changed.

Original: Table 2 illustrate the outcome of this procedure with three examples. The model was asked to predict the price of three plates, ranging from low to high value.
Revised: In this section, I investigate the effect of periodically retraining the model with

Original: The predicted prices are listed in the Prediction section, while the Historical Examples section lists for each query the top four entries returned by the k -NN model. Notice how the procedure focused on the numeric part for the low-value plate and the alphabetical part for the middle-value plate, reﬂecting the value of having identical digits and identical alphabets respectively. The procedure was also able to inform the user that a plate has been sold before. Finally, the examples provided for the high-value plate show why it is hard to obtain an accurate prediction for such plates, as the historical prices for similar plates are also highly variable.
Revised: [Figure]

Original: Combining several models is known to improve prediction accuracy. Two combinations are considered in this section: a combination of the preceding neural network and Woo et al. (2008), and a combination of these two models plus features not related to the characters on plates.
Revised: Woo et al (2008)

Original: In each case, the combination was conducted through linear regression, with the prediction of each model acting as features. The two models were thus implemented as follows: [Equation] where y rnn is the prediction of the neural network, y woo the prediction of Woo et al.
Revised: Ng et al (

Original: (2008)’s regression model with only the license-plate-speciﬁc features, and x i a series of additional features, including the year and month of the auction, whether it was an afternoon session, the plate’s position within the session’s ordering, the existence of a preﬁx, the number of digits, a log of the local market stock index, and a log of the consumer price index. α , δ and ν were estimated by linear regression on the training data. Both models performed better than the neural network alone. The simple combined model shown ( Combined in Table 1) improved performance by slightly more than 1 percent as measured by R 2 , while the additional features ( Combined + Extra ) improved performance by another 1.6 percentage points.
Revised: [Figure]

Original: Overall, the ensemble models improved accuracy, but only by a small amount, suggesting that the neural network was successful in explaining most of the variation in prices. 7. R ETRAINING O VER T IME Over time, a model could conceivably become obsolete if, for example, taste or the economic environment changed. In this section, I investigate the effect of periodically retraining the model.
Revised: [Figure]

Original: To retain sufﬁcient samples for subsequent testing and retraining, the dataset was roughly divided into two. Initial training was conducted with the ﬁrst 8 years of data, which contained 25, samples.
Revised: the sequentially-split data.

Original: In the subsequent 5 years, retraining was conducted  yearly, monthly, or never.
Revised: Speciﬁcally, retraining was conducted throughout the test data yearly, monthly, or never.

Original: The best RNN-only model and the Combined + Extra model were used, with the sample size kept constant  in each retraining.
Revised: The best RNN-only model was used, with the sample size kept constant at 25,990 in each retraining, which is roughly ﬁve years of data.

Original: The process was repeated 30 times as before.
Revised: The process was repeated 30 times as before.

Original: Figure 5 plots the median RMSE and R 2 , evaluated monthly.
Revised: Figure 5 plots the median RMSE and R 2 , evaluated monthly.

Original: For the RNN model with no retraining prediction, accuracy dropped rapidly by both measures.
Revised: For the RNN model with no retraining prediction, accuracy dropped rapidly by both measures.

Original: RMSE dropped an average of 0.7 percent per month, while R 2 dropped 0.3 percent per month.
Revised: RMSE increases an average of 0.017 per month, while R 2 dropped 0.01 per month.

Original: Yearly retraining was signiﬁcantly better, with a 12.1 percent lower RMSE and a 7.2 percent higher R 2 .
Revised: Yearly retraining was signiﬁcantly better, with a 8. percent lower RMSE and a 6.9 percent higher R 2 .

Original: The additional beneﬁt of monthly retraining was, however, much smaller.
Revised: The additional beneﬁt of monthly retraining was, however, much smaller.

Original: Compared with the yearly retraining, there was only a 3. percent reduction in the RMSE and a 1.5 percent increase in the explanatory power.
Revised: Compared with the yearly retraining, there was only a 3.3 percent reduction in the RMSE and a 2.6 percent increase in the explanatory power.

Original: The differences were statistically signiﬁcant.
Revised: The differences were statistically signiﬁcant.

Revised: [Table]

Original: For the ensemble model, the performance between different retraining frequencies was very close, witha less than 4 percent difference in the RMSE and a less than 1 percent difference in R 2 when going from no retraining to monthly retraining.
Revised: Combining several models is known to improve prediction accuracy. This section considers a combination of the preceding neural network, (Woo et al., 2008) plus features not related to the characters on plates. The combination was conducted through linear regression, with the prediction of each model acting as features.

Revised: The model was thus implemented as follows:

Revised: [Equation]

Revised: where y rnn is the prediction of the neural network, y woo the prediction of (Woo et al., 2008)’s regression model with only the license-plate-speciﬁc features, and x i a series of additional features, including the year and month of the auction, whether it was an afternoon session, the plate’s position within the session’s ordering, the existence of a preﬁx, the number of digits, a log of the local market stock index, and a log of the consumer price index.

Revised: α , δ and ν were estimated by linear regression on the training data.

Revised: For this ensemble model, the performance between different retraining frequencies was very close, with a less than 1 percent difference in the RMSE and a less than 2 percent difference in R 2 when going from no retraining to monthly retraining.

Original: Nevertheless, the differences remained statistically signiﬁcant, as retraining every month did improve accuracy.
Revised: Nevertheless, the differences remained statistically signiﬁcant, as retraining every month did improve accuracy.

Original: The performance of the ensemble model was also considerably more stable than the RNN alone, with only half of the volatility at every retraining frequency.
Revised: The performance of the ensemble model was also considerably more stable than the RNN alone, with only half of the volatility at every retraining frequency.

Original: The primary reason behind this difference was the RNN’s inability to account for extreme prices, particularly at month 12, 24 and 46.
Revised: The primary reason behind this difference was the RNN’s inability to account for extreme prices, particularly at month 12, 24 and 46.

Original: Auctions in these three months had an unusually high number of valuable plates, resulting in average sold prices that were 15 to [Figure] times higher than the overall average.
Revised: Auctions in these three months had an unusually high number of valuable plates, resulting in average sold prices that were 15 totimes higher than the overall average.

Original: The ensemble model was able to predict these extreme prices because Woo et al. (2008) handcrafted features speciﬁcally for these valuable plates.
Revised: The ensemble model was able to predict these extreme prices because (Woo et al., 2008) handcrafted features speciﬁcally for these valuable plates.

Original: These results suggest that while there is a clear beneﬁt in periodical retraining, this beneﬁt diminishes rapidly beyond a certain threshold.
Revised: These results suggest that while there is a clear beneﬁt in periodical retraining, this beneﬁt diminishes rapidly beyond a certain threshold.

Original: Moreover, while deep RNN generally outperforms handcrafted features, the latter could be used to capture outliers.
Revised: Moreover, while deep RNN generally outperforms handcrafted features, the latter could be used to capture outliers.

Original:  C ONCLUDING R EMARKS
Revised:  E XPLAINING THE P REDICTIONS

Revised: Compared to models such as regression and n -gram it is relatively hard to understand the rationale behind a RNN model’s prediction, given the large number of parameters involved and the complexity of the their interaction.

Revised: If the RNN model is to be deployed in the ﬁeld, it would need to be able to explain its prediction in order to convince human users to adopt it in practice.

Revised: One way to do so is to extract a feature vector for each plate by summing up the output of the last recurrent layer over time.

Revised: This feature vector is of the same size as the number of neurons in the last layer, which can be fed into a standard k -nearest-neighbor model to provide a “rationale” for the model’s prediction.

Revised: To demonstrate this procedure, I use the best RNN model in Table 1 to generate feature vectors for all training samples.

Revised: These samples are used to setup a k -NN model.

Revised: When the user submit a query, a price prediction is made with the RNN model, while a number of examples are provided by the k -NN model as rationale.

Revised: Table 2 illustrate the outcome of this procedure with three examples.

Revised: The model was asked to predict the price of three plates, ranging from low to high value.

Revised: The predicted prices are listed in the Prediction section, while the Historical Examples section lists for each query the top four entries returned by the k -NN model.

Revised: Notice how the procedure focused on the numeric part for the low-value plate and the alphabetical part for the middle-value plate, reﬂecting the value of having identical digits and identical alphabets respectively.

Revised: The procedure was also able to inform the user that a plate has been sold before.

Revised: Finally, the examples provided for the high-value plate show why it is hard to obtain an accurate prediction for such plates, as the historical prices for similar plates are also highly variable.

Revised: [Table]

Revised: [Table]

Revised: the Prediction section are user

Revised: [Table]

Revised: and the prices are predictions

Revised: [Table]

Revised: The plates and their corresponding prices listed in the Historical Examples

Revised: [Table]

Revised: historical

Revised: [Table]

Revised: [Figure]

Revised:  E STIMATING THE D ISTRIBUTION OF P RICES

Revised: While the RNN model outputs only a single price estimate, auctions that provide estimates typically give both a high and a low estimate.

Revised: The k -NN model from the previous section can provide reasonably good estimates for common, low-value plates, but works poorly for rare, high-value plates due to the lack of other similar plates in record.

Revised: To tackle this problem, this section uses a Mixture

Revised: Density Network (MDN) to estimate the distribution of plate prices (Bishop, 1994).

Revised: A MDN with a single hidden layer of fully-connected 256 neurons was used to estimate 24 Gaussian distributions.

Revised: The network was trained for 5000 epochs.

Revised: Figure 6 demonstrates the network’s ability to ﬁt the distribution of prices.

Revised: The estimated density resembles the distribution of common, low-value plates, while producing a density that is noticeably wider than the distribution of actual prices for rare, high-value plates.

Revised:  C ONCLUDING R EMARKS

Original: This study demonstrates that a deep recurrent neural network can provide good estimates of license plate prices, with signiﬁcantly higher accuracy than other models.
Revised: This study demonstrates that a deep recurrent neural network can provide good estimates of license plate prices, with signiﬁcantly higher accuracy than other models.

Original: The deep RNN is capable of learning the prices from the raw characters on the plates, while other models must rely on handcrafted features.
Revised: The deep RNN is capable of learning the prices from the raw characters on the plates, while other models must rely on handcrafted features.

Original: With modern hardware, it takes only a few minutes to train the best-performing model described previously, so it is feasible to implement a system in which the model is constantly retrained for accuracy.
Revised: With modern hardware, it takes only a few minutes to train the best-performing model described previously, so it is feasible to implement a system in which the model is constantly retrained for accuracy.

Original: Notwithstanding this good performance, several areas can be further improved.

Original: First, the ensemble models in this study were constructed with linear regression. The use of another neural network instead of linear regression may better capture the high-order interactions between the characters on the plate and the other features, thereby further improving the performance ofthe model. Second, while the model outputs only a single price estimate, auctions that provide estimates typically give both a high and a low estimate. As there is only one realized price to train on for each sample, designing a model that can output a meaningful range of estimates is a non-trivial task; a range that is too narrow will often be violated, while one that is too broad will be of no practical use.
Revised: A natural next step along this line of research is the construction of a model for personalized plates.

Original: Finally, the performance of the model on personalized plates has yet to be studied.

Original: Personalized plates contain owner-submitted sequences of characters and so may have vastly more complex meanings.
Revised: Personalized plates contain owner-submitted sequences of characters and so may have vastly more complex meanings.

Original: Exactly how the model should be designed—for example, whether there should be separate models for different types of plates, or whether pre-training on another text corpus could help— remains to be studied.
Revised: Exactly how the model should be designed—for example, whether there should be separate models for different types of plates, or whether pre-training on another text corpus could help—remains to be studied.

Original:  A CKNOWLEDGMENTS
Revised:  A CKNOWLEDGMENTS

Original: I would like to thank Travis Ng for providing the license plate data used in this study.
Revised: I would like to thank Travis Ng for providing the license plate data used in this study