Original: Hierarchical Lattice Layer for Partially Monotone Neural Networks
Revised: Hierarchical Lattice Layer for Partially Monotone Neural Networks

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Partially monotone regression is a regression analysis in which the target valuesare monotonically increasing with respect to a subset of input features.
Revised: Partially monotone regression is a regression analysis in which the target values are monotonically increasing with respect to a subset of input features.

Original: The
Revised: The

Original: TensorFlow Lattice library is one of the standard machine learning libraries forpartially monotone regression.
Revised: TensorFlow Lattice library is one of the standard machine learning libraries for partially monotone regression.

Original: It consists of several neural network layers, and itscore component is the lattice layer.
Revised: It consists of several neural network layers, and its core component is the lattice layer.

Original: One of the problems of the lattice layer is itsrequirement for a special training algorithm  to satisfy monotonicity constraints.
Revised: One of the problems of the lattice layer is that it requires the projected gradient descent algorithm with many constraints to train it.

Original: Another problem is that it cannot receive a high-dimensional input vector due tothe resultant memory consumption.
Revised: Another problem is that it cannot receive a high-dimensional input vector due to the memory consumption.

Original: We propose a novel neural network layer,the hierarchical lattice layer (HLL), as an extension of the lattice layer so thatwe can use a standard neural network algorithm to train HLL while satisfyingmonotonicity constraints and so that it can receive a high-dimensional input vector.
Revised: We propose a novel neural network layer, the hierarchical lattice layer (HLL), as an extension of the lattice layer so that we can use a standard stochastic gradient descent algorithm to train HLL while satisfying monotonicity constraints and so that it can receive a high-dimensional input vector.

Original: Our experiments demonstrate that HLL did not sacriﬁce its prediction performanceon real datasets compared with the lattice layer
Revised: Our experiments demonstrate that HLL did not sacriﬁce its prediction performance on real datasets compared with the lattice layer.

Original:  Introduction
Revised:  Introduction

Original: In regression analysis, we often have prior knowledge that the target values are monotonicallyincreasing (or decreasing) with respect to a subset of input features.
Revised: In regression analysis, we often have prior knowledge that the target values are monotonically increasing (or decreasing) with respect to a subset of input features.

Original: For example, when we estimatehouse prices from house data (e.g., the area size, the number of rooms, and the distance to the nearestsupermarket), we know that the target value (i.e., the house price) is monotonically increasing withrespect to some of the input features (e.g., the area size) if the other features are equal [Potharst and
Revised: For example, when we estimate house prices from house data (e.g., the area size, the number of rooms and the distance to the nearest supermarket), we know that the target value (i e., the house price) is monotonically increasing with respect to some of the input features (e.g., the area size) if the other features are equal [Potharst and

Original: Feelders, 2002].
Revised: Feelders, 2002].

Original: Partially monotone regression is a regression analysis that exploits such prior knowledge.
Revised: Partially monotone regression is a regression analysis that exploits such prior knowledge.

Original: The inputof this regression is a feature vector x ∈ R d , which can be partitioned into x = ( p, q )
Revised: The input of this regression is a feature vector x ∈ R d , which can be partitioned into x = ( p, q )

Original: such thatp ∈ R d − m is non-monotone features and q ∈ R m is monotone features.
Revised: such that p ∈ R d − m is non-monotone features and q ∈ R m is monotone features.

Original: The goal of this regressionis to ﬁnd a function f such that f ( x ) =
Revised: The goal of this regression is to ﬁnd a function f such that f ( x ) =

Original: f ( p, q ) ∈ R is monotonically increasing on q .
Revised: f ( p, q ) ∈ R is monotonically increasing on q .

Original: Partiallymonotone regression models have advantages over the standard regression models.
Revised: Partially monotone regression models have advantages over the standard regression models.

Original: For example, theyhave a better regularization capability Dugas et al. [2000]; Fard et al. 
Revised: For example, they have a better regularization capability [Dugas et al. , 2000; Fard et al. , 2016;

Original: [2016]; You et al. [2017] andbetter interpretability Gupta et al. 
Revised: You et al. , 2017] and better interpretability [Gupta et al. , 2016].

Original: [2016], and they can be used for fair machine learning Wang and
Revised: They can be used for fair machine learning [Wang and

Original: Gupta [2020].
Revised: Gupta, 2020].

Original: Many machine learning models have been developed for partially monotone regression.
Revised: Many machine learning models have been developed for partially monotone regression.

Original: For example,there are decision-tree based models Potharst and Feelders [2002], SVM models Lauer and Bloch[2008], boosting models Bartley et al. [2018], and random forest models Bartley et al. 
Revised: For example, there are decision-tree based models [Potharst and Feelders, 2002], SVM models [Lauer and Bloch, 2008], boosting models [Bartley et al. , 2018], and random forest models [Bartley et al. , 2019].

Original: Manypopular open source libraries for gradient boosting can handle partial monotonicity (e.g., CatBoost,
Revised: Many popular open source libraries for gradient boosting can handle partial monotonicity (e.g., CatBoost,

Original: LightGBM, and XGBoost).
Revised: LightGBM, and XGBoost).

Original: See a survey paper Cano et al. [2019] and the references in Gupta et al.[2016] for comprehensive surveys.
Revised: See a survey paper [Cano et al. , 2019] and the references in [Gupta et al. , 2016] for comprehensive surveys.

Original: As for neural network models, the TensorFlow Lattice library Canini et al. [2016]; Gupta et al.  You et al. [2017] is a standard model for partially monotone regression, and it ispublicly available as an open source library.
Revised: As for neural network models, the TensorFlow Lattice library [Canini et al. , 2016; Gupta et al. , 2016; You et al. , 2017] is a standard model for partially monotone regression, and it is available as an open source library.

Original: It consists of several neural network layers, and its core component isthe lattice layer Gupta et al. 
Revised: It consists of several neural network layers, and its core component is the lattice layer [Gupta et al. , 2016].

Original: Hereafter, we call this layer TL Lattice.
Revised: Hereafter, we call this layer TL Lattice.

Original: TL Lattice constructsa hypercube with k d vertices as an internal structure, where k ≥ 2 is an integer hyperparameterfor specifying the granularity of the hypercube, and d is the dimension of the input vector x ∈ R d .
Revised: TL Lattice constructs a hypercube with k d vertices as an internal structure, where k ≥ 2 is an integer hyperparameter for specifying the granularity of the hypercube, and d is the dimension of the input vector x ∈ R d .

Original: One of its problems is its memory consumption in storing k d parameters, and therefore, it cannotreceive a high-dimensional input vector with a large d .
Revised: One of its problems is its memory consumption in storing k d parameters, and therefore, it cannot receive a high-dimensional input vector with a large d .

Original: This means that we need to combine multipleneural network layers to handle an input vector with a large d (e.g., by using small lattices Caniniet al. 
Revised: This means that we need to combine multiple neural network layers to handle an input vector with a large d (e.g., by using small lattices [Canini et al. , 2016;

Original: [2016]; You et al. 
Revised: You et al. , 2017]).

Original: Another problem is its requirement for a special algorithm  to trainit to handle monotonicity constraints, which means that we cannot use a standard neural networkalgorithm to train it.
Revised: Another problem is its requirement for the projected gradient descent algorithm with many constraints to train it to handle monotonicity constraints. This means that we cannot use a standard stochastic gradient descent algorithm to train it.

Original: The details of thespecial training algorithms used for the TensorFlow Latticelibrary are described in Gupta et al. 
Revised: The details of the training algorithms used for the TensorFlow Lattice library are described in [Gupta et al. , 2016;

Original: [2016]; You et al. 
Revised: You et al. , 2017].

Original: In this paper, we propose a novel neural network layer, the hierarchical lattice layer (HLL), asan extension of TL Lattice.
Revised: In this paper, we propose a novel neural network layer, the hierarchical lattice layer (HLL), as an extension of TL Lattice.

Original: The internal structure of HLL is similar to that of TL Lattice inthat both layers construct hypercube structures.
Revised: The internal structure of HLL is similar to that of TL Lattice in that both layers construct hypercube structures.

Original: However, they are different in several aspects.
Revised: However, they are different in several aspects.

Original: For example, HLL constructs an m -dimensional hypercube, whereas TL Lattice constructs a ddimensional hypercube, where d and m are the dimensions of the input vector x ∈ R d and itsmonotone feature q ∈ R m , respectively.
Revised: For example, HLL constructs an m -dimensional hypercube, whereas TL Lattice constructs a ddimensional hypercube, where d and m are the dimensions of the input vector x ∈ R d and its monotone feature q ∈ R m , respectively.

Original: Since m is usually a small integer in practice, the numberof vertices k m in HLL is signiﬁcantly smaller than that of TL Lattice (i.e., k d ).
Revised: Since m is usually a small integer in practice, the number of vertices k m in HLL is signiﬁcantly smaller than that of TL Lattice (i.e., k d ).

Original: Therefore, HLLcan receive a high-dimensional input vector as long as m is small.
Revised: Therefore, HLL can receive a high-dimensional input vector as long as m is small.

Original: For another example, theparameterization of HLL is designed to represent a class of partially monotone functions (i.e., HLLcannot represent a function f ( p, q ) that is not monotonic on q ) by using the hierarchical relationshipbetween the vertices of the hypercube.
Revised: For another example, the parameterization of HLL is designed to represent a class of partially monotone functions (i.e., HLL cannot represent a function f ( p, q ) that is not monotonic on q ) by using the hierarchical relationship between the vertices of the hypercube.

Original: Therefore, the internal structure of HLL ensures partialmonotonicity regardless of the training algorithm used.
Revised: Therefore, the internal structure of HLL ensures partial monotonicity regardless of the training algorithm used.

Original: In contrast, the internal structure of TL
Revised: In contrast, the internal structure of TL

Original: Lattice is a lookup table, and the table itself can represent any function f
Revised: Lattice is a lookup table, and the table itself can represent any function f

Original: ( p, q ) , which means that theparameterization of TL Lattice is not designed to satisfy monotonicity constraints.Therefore, TL
Revised: ( p, q ) , which means that the parameterization of TL Lattice is not designed to satisfy monotonicity constraints.

Original: Lattice requires a special algorithm to train its parameters to ensure these constraints.

Original: In our experiments, we demonstrate that HLL did not sacriﬁce its prediction performance comparedwith TL Lattice.
Revised: In our experiments, we demonstrate that HLL did not sacriﬁce its prediction performance compared with TL Lattice.

Original: Note that HLL is designed to satisfy monotonicity constraints and is not designedto improve the prediction performance compared with TL Lattice.
Revised: Note that HLL is designed to satisfy monotonicity constraints and is not designed to improve the prediction performance compared with TL Lattice.

Original: Therefore, the purpose of ourexperiments was not to show improvements in prediction performance compared with TL Lattice butto show that HLL did not sacriﬁce the prediction performance to satisfy the constraints.
Revised: Therefore, the purpose of our experiments was not to show improvements in prediction performance compared with TL Lattice but to show that HLL did not sacriﬁce the prediction performance to satisfy the constraints.

Original: In addition,we also demonstrate that we could use HLL for high-dimensional input vectors with a small m .
Revised: In addition, we demonstrate that we could use HLL for high-dimensional input vectors with a small m .

Original:  Partially Monotone Regression
Revised:  Partially Monotone Regression

Original: Regression analysis is a method for ﬁnding a function f : R d → R such that f ( x i ) approximatesy i for a given dataset D
Revised: Regression analysis is a method for ﬁnding a function f : R d → R such that f ( x i ) approximates y i for a given dataset D

Original: = { ( x i , y i ) } ni
Revised: = { ( x i , y i ) } ni

Original: =1 , where x i ∈ R
Revised: =1 , where x i ∈ R

Original: d is a feature vector, and y i ∈ R is a targetvalue.
Revised: d is a feature vector, and y i ∈ R is a target value.

Original: In partially monotone regression, we assume that we can partition a feature vector x ∈ R d intox = ( p, q ) ∈ R d − m × R m , where [-
-]
Revised: In partially monotone regression, we assume that we can partition a feature vector x ∈ R d into x = ( p, q ) ∈ R d − m × R m , where [-
-]0 < m <

Original: < m < d , such that f ( x ) ∈ R is monotonic on q .
Revised: d , such that f ( x ) ∈ R is monotonic on q .

Original: A functionf ( x ) for x = ( p, q ) is monotonic on q if this inequality holds:
Revised: A function f ( x ) for x = ( p, q ) is monotonic on q if this inequality holds:

Original: [Equation]
Revised: [Equation]

Original: where q ≤ q (cid:48) denotes the inequality for all the elements (i.e., q [ i ] ≤ q (cid:48)
Revised: where q ≤ q (cid:48) denotes the inequality for all the elements (i.e., q [ i ] ≤ q (cid:48)

Original: [ i ] for all 1 ≤ i ≤ m , whereq [ i ] denotes the i -th element of q ).
Revised: [ i ] for all 1 ≤ i ≤ m , where q [ i ] denotes the i -th element of q ).

Original: Partially monotone regression has many applications in various ﬁelds, and it has important applicationseven if m = 1 .
Revised: Partially monotone regression has many applications in various ﬁelds, and it has important applications even if m = 1 .

Original: An example of such application with m = 1 is quantile regression Koenker and
Revised: An example of such application with m = 1 is quantile regression [Koenker and

Original: Bassett [1978]; Zhou et al. [2020], which is used to estimate the conditional quantiles of a targetvariable distribution.
Revised: Bassett, 1978; Zhou et al. , 2020], which is used to estimate the conditional quantiles of a target variable distribution.

Original: We show that the output q ( x ) =
Revised: We show that the output q ( x ) =

Original: q ( x (cid:48) , τ ) of the quantile regression for an inputx = ( x (cid:48) , τ ) is monotonic on τ , where x (cid:48) ∈
Revised: q ( x (cid:48) , τ ) of the quantile regression for an input x = ( x (cid:48) , τ ) is monotonic on τ , where x (cid:48) ∈

Original: R d − 1 is a feature vector, and τ ∈ [0 , 1] is a quantile level.
Revised: R d − 1 is a feature vector, and τ ∈ [0 , 1] is a quantile level.

Original: In quantile regression, we estimate the quantile value q ( x (cid:48) , τ ) deﬁned byq ( x (cid:48) , τ ) = inf { y | F ( y | X = x (cid:48) )
Revised: In quantile regression, we estimate the quantile value q ( x (cid:48) , τ ) deﬁned by

Original: ≥ τ } , where F ( y | X
Revised: [Equation]

Original:  = x (cid:48) ) is the conditional cumulative distribution function of Y given X = x (cid:48) : F ( y | X = x (cid:48) )
Revised: where F ( y | X = x (cid:48) ) is the conditional cumulative distribution function of Y given X = x (cid:48) : F ( [Equation]

Original:  =
Revised: ) =

Original: Pr ( Y ≤ y | X = x (cid:48) ) .
Revised: Pr [Equation] Y ≤ y | X = x (cid:48) ) .

Original: Since the function q ( x (cid:48) , τ ) is monotonically increasing on τ , this problem can be modeled as partiallymonotone regression with m = 1 .
Revised: Since the function q ( x (cid:48) , τ ) is monotonically increasing on τ , this problem can be modeled as partially monotone regression with m = 1 .

Revised: Note that, if we use a standard multi-layer perceptron without any monotonicity constraints to estimate the quantile value q ( x (cid:48) , τ ) , it sometimes fails to satisfy the monotonicity constraints, which is known as the crossing problem (see e.g., [Tagasovska and

Revised: Lopez-Paz, 2019]).

Revised: In quantile regression, many algorithms have been proposed to resolve the crossing problem (e.g., [Park et al. , 2022]).

Original: Another example of partially monotone regression with m = 1 is survival analysis, which is alsoknown as time-to-event analysis Wang et al. [2019]; Rindt et al. 
Revised: Another example of partially monotone regression with m = 1 is survival analysis, which is also known as time-to-event analysis [Wang et al. , 2019; Rindt et al. , 2022].

Original: We show that the outputf ( x ) =
Revised: We show that the output f ( x ) =

Original: f
Revised: f

Original: ( x (cid:48) , t ) of the survival analysis for an input x = ( x (cid:48) , t ) is monotonic on t , where x (cid:48) ∈ R d − 1is a feature vector of a patient, and t ∈ R is the time of an event of interest of this patient.
Revised: ( x (cid:48) , t ) of the survival analysis for an input x = ( x (cid:48) , t ) is monotonic on t , where x (cid:48) ∈ R d −is a feature vector of a patient, and t ∈ R is the time of an event of interest of this patient.

Original: In survivalanalysis, the event of interest typically corresponds to the death of a patient.
Revised: In survival analysis, the event of interest typically corresponds to the death of a patient.

Original: The goal of the analysisis to estimate a survival function S ( x (cid:48) , t ) for patient x (cid:48) , which represents the probability that theevent does not occur until time t for patient x (cid:48) .
Revised: The goal of the analysis is to estimate a survival function S ( x (cid:48) , t ) for patient x (cid:48) , which represents the probability that the event does not occur until time t for patient x (cid:48) .

Original: By setting f ( x (cid:48) , t ) = 1 − S ( x (cid:48) , t ) , we can model thesurvival analysis as a partially monotone regression with m = 1 , where the value f
Revised: By setting f ( x (cid:48) , t ) = 1 − S ( x (cid:48) , t ) , we can model the survival analysis as a partially monotone regression with m = 1 , where the value f

Original: ( x (cid:48) , t ) shows themortality rate of patient x (cid:48) at time t , and f ( x (cid:48) , t ) is monotonically increasing on t .
Revised: ( x (cid:48) , t ) shows the mortality rate of patient x (cid:48) at time t , and f ( x (cid:48) , t ) is monotonically increasing on t .

Original:  Hierarchical Lattice Layer
Revised:  Hierarchical Lattice Layer

Original: In this section, we explain our new hierarchical lattice layer (HLL) for neural network models forpartially monotone regression.
Revised: In this section, we explain our new hierarchical lattice layer (HLL) for neural network models for partially monotone regression.

Original: It can be seen as an extension of TL Lattice Gupta et al. 
Revised: It can be seen as an extension of TL Lattice [Gupta et al. , 2016].

Original: Notethat the input of HLL is a d -dimensional vector x ∈ [0 , 1] d , and its output f ( x ) is in [0 , 1] , where theinput vector x can be partitioned into two subvectors p ∈ [0 , 1] d − m and q ∈ [0 , 1] m such that f ( x ) ismonotonic on q .
Revised: Note that the input of HLL is a d -dimensional vector x ∈ [0 , 1] d , and its output f ( x ) is in [0 , 1] , where the input vector x can be partitioned into two subvectors p ∈ [0 , 1] d − m and q ∈ [0 , 1] m such that f ( x ) is monotonic on q .

Original: In HLL, we create the m -dimensional unit hypercube [0 , 1] m , and we decompose it into ( k 1 − 1) ×
Revised: We create the m -dimensional unit hypercube [0 , 1] m as an internal structure of HLL, and we decompose it into ( k 1 − 1) ×

Original: ( k 2 − 1) × · · · × ( k m − 1) subhypercubes by dividing the interval [0 , 1] of the i -th dimension intok i − 1 subintervals, where each k i ≥ 2 is an integer hyperparameter that speciﬁes the granularity ofthe hypercube.
Revised: ( k 2 − 1) × · · · × ( k m − 1) subhypercubes by dividing the interval [0 , 1] of the i -th dimension into k i − 1 subintervals, where each k i ≥ 2 is an integer hyperparameter that speciﬁes the granularity of the hypercube.

Original: Unless otherwise stated, we assume that k 1 = k 2 = · · · = k m =
Revised: Unless otherwise stated, we assume that k 1 = k 2 = · · · = k m =

Original: k , and the interval[0 , 1] is divided into k − 1 equal-length subintervals just for simplicity.
Revised: k , and the interval [0 , 1] is divided into k − 1 equal-length subintervals just for simplicity.

Original: This means that the length ofan edge of a subhypercube is 1 / ( k − 1) , and we have the k m vertices in the unit hypercube.
Revised: This means that the length of an edge of a subhypercube is 1 / ( k − 1) , and we have the k m vertices in the unit hypercube.

Original: We denotethe set of the vertices in the unit hypercube by V m .
Revised: We denote the set of the vertices in the unit hypercube by V m .

Original: In this paper, we do not distinguish between avertex v ∈ V m and its coordinate q v ∈ R m .
Revised: In this paper, we do not distinguish between a vertex v ∈ V m and its coordinate q v ∈ [0 , 1] m .

Original: This means that the value f ( p, q v ) for coordinate q v iswritten as f ( p, v ) .
Revised: This means that the value f ( p, q v ) for coordinate q v is written as f ( p, v ) .

Original: To construct HLL, we assume that there is a total order σ =  ( v 0 , v 1  , . .
Revised: Before describing the details of HLL, we explain the key idea of HLL by using a simple case with m = 1 and k = 3 . In this case, HLL is supposed to learn a function f ( p, q ) such that 0 ≤ f ( p, q 0 ) ≤ f ( p, q 1 ) ≤ f ( p, q 2 ) ≤ 1 holds for any p ∈ R d − m and q 1 = 0 , q 2 = 1 / 2 , and q 3 = 1 .

Revised: Hereafter, we use v 1 , v 2 , and v 3 to represent q 1 , q 2 , and q 3 , respectively, to emphasize that they correspond to the vertices in V m .

Revised: In HLL, we associate a function g ( p, v ) for each vertex v ∈ V m so that the function f ( p, v ) for v ∈ V m can be represented by using the function g ( p, v ) .

Revised: We show how the function g ( p, v ) is deﬁned here.

Revised: First, the function g ( p, v 0 ) is deﬁned as g ( p, v 0 )

Revised: =

Revised: f ( p, v 0 ) .

Revised: Second, the function g ( p, v 1 ) is deﬁned as the function that satisﬁes this equation:

Revised: [Equation]

Revised: Note that 0 ≤ g ( p, v 1 ) ≤ 1 always holds because f ( p, v 0 )

Revised: ≤ f ( p, v 1 ) ≤ 1 .

Revised: Similarly, the function g ( p, v 2 ) is deﬁned as the function that satisﬁes this equation:

Revised: [Equation]

Revised: See Figure 1 for an illustration of these deﬁnitions.

Revised: By using these deﬁnitions, the task of learning the function f ( p, v ) with monotonicity constraints 0 ≤ f ( p, v 0 ) ≤ f ( p, v 1 ) ≤ f ( p, v 2 ) ≤ 1 is casted as the task of learning the function g ( p, v ) such that 0 ≤ g ( p, v ) ≤ 1 without any monotonicity constraints.

Revised: We generalize this idea for m > 1 in the following.

Revised: We note here that HLL for the case with m = 1 is similar to some existing approaches used in survival analysis (see e.g., [Ren et al. , 2019; Zheng et al. , 2019]).

Revised: More speciﬁcally, the survival function is decomposed into a product of hazard rates in these approaches, and the survival function and the hazard rate can be represented by using f ( p, v ) and g ( p, v ) in HLL, respectively.

Revised: g ( p, v 0 ) : 1 − g ( p, v 0 ) g

Revised: [Equation]

Original: . , v | V m |− 1 ) of vertices in
Revised: [Equation] [Equation] [Equation] ) g ( p,

Revised: [Equation]

Revised: f ( p, v 0 )

Original:  V m .
Revised: f ( p, v 1 ) f ( p, v 2 ) 3.2 Hierarchical Relationship between Vertices In HLL, we assume that there is a total order σ = ( v 0 , v 1 , . . . , v | V m |− 1 ) of vertices in V m .

Original: We can use any total order σ to construct HLL, but unless otherwise stated, we assume thatσ is determined by the increasing order of the Manhattan distance (i.e., L 1 distance) from theorigin vertex v 

Revised: We can use any total order σ to construct HLL, but unless otherwise stated, we assume that σ is determined by the increasing order of the Manhattan distance (i.e., L 1 distance) from the origin vertex v

Original: = (0 , 0 , . .
Revised: = (0 , 0 , . .

Original: . , 0) ∈
Revised: . , 0) ∈

Original: [0 , 1] m of the unit hypercube with an arbitrary tie-breaker.
Revised: [0 , 1] m of the unit hypercube with an arbitrary tie-breaker.

Original: See Figure 1 for an example of a decomposed unit hypercube with m = 2 and k = 4 and the total orderσ = ( v 0 , v 1 , . .
Revised: See Figurefor an example of a decomposed unit hypercube with m = 2 and k = 4 and the total order σ = ( v 0 , v 1 , . .

Original:  , v 15 ) determined by the Manhattan distance.
Revised: . , v 15 ) determined by the Manhattan distance.

Original: We deﬁne the domination relationship between vertices in V m , and we deﬁne the minimal dominatedand dominating sets based on σ .
Revised: We deﬁne the domination relationship between vertices in V m , and we deﬁne the minimal dominated and dominating sets based on σ .

Original: Deﬁnition 3.1 (Domination.)
Revised: Deﬁnition 3.1 (Domination.)

Original: A vertex v ∈ R m is said to dominate another vertex u ∈ R m if andonly if u [ i ] ≤ v [ i ] holds for all i ∈ { 1 , 2 , . . .
Revised: A vertex v ∈ R m is said to dominate another vertex u ∈ R m if and only if u [ i ] ≤ v [ i ] holds for all i ∈ { 1 , 2 , . . .

Original: , m } and there exists an index i (cid:48) ∈ { 1 , 2 , . .
Revised: , m } and there exists an index i (cid:48) ∈ { 1 , 2 , . .

Original: , m } suchthat u
Revised: , m } such that u

Original: [ i (cid:48) ] <
Revised: [ i (cid:48) ] <

Original: v [ i (cid:48) ] .
Revised: v [ i (cid:48) ] .

Original: Deﬁnition 3.2 (Minimal dominated set.)
Revised: Deﬁnition 3.2 (Minimal dominated set.)

Original: The set of vertices L v,σ denotes the minimal set of verticesthat satisfy three conditions (i) every vertex u ∈ L v,σ is dominated by v , (ii) vertex u precedes v in σ ,and (iii) no other vertex w ∈ L v,σ dominates u .
Revised: The set of vertices L v,σ denotes the minimal set of vertices that satisfy three conditions (i) every vertex u ∈ L v,σ is dominated by v , (ii) vertex u precedes v in σ , and (iii) no other vertex w ∈ L v,σ dominates u .

Original: Deﬁnition 3.3 (Minimal dominating set.)
Revised: Deﬁnition 3.3 (Minimal dominating set.)

Original: The set of vertices U v,σ denotes the minimal set of verticesthat satisfy three conditions (i) every vertex u ∈ U v,σ dominates v , (ii) vertex u precedes v in σ , and(iii) no other vertex w ∈ U v,σ is dominated by u .
Revised: The set of vertices U v,σ denotes the minimal set of vertices that satisfy three conditions (i) every vertex u ∈ U v,σ dominates v , (ii) vertex u precedes v in σ , and (iii) no other vertex w ∈ U v,σ is dominated by u .

Original: Note that the domination relation means that f ( p, u ) ≤ f ( p, v ) holds for any p ∈ R d − m if vertex vdominates vertex u .
Revised: Intuitively the domination relation means that f ( p, u ) ≤ f ( p, v ) holds for any p ∈ R d − m if vertex v dominates vertex u .

Original: Regarding the minimal dominated and dominating sets, there can be multipleminimal sets that satisfy the conditions.
Revised: Regarding the minimal dominated and dominating sets, there can be multiple minimal sets that satisfy the conditions.

Original: Throughout this paper, we ﬁx an (arbitrary) minimaldominated set L v,σ and an (arbitrary) minimal dominating set U v,σ for each v ∈ σ .
Revised: Throughout this paper, we ﬁx an (arbitrary) minimal dominated set L v,σ and an (arbitrary) minimal dominating set U v,σ for each v ∈ σ .

Original: Note also that wehave | L v,σ | ≤ m and | U v,σ | = 0 if σ is determined by the Manhattan distance.
Revised: Note also that we have | L v,σ | ≤ m and | U v,σ | = 0 if σ is determined by the Manhattan distance.

Original: This is why we usethe Manhattan distance to determine σ , because we have weaker upper bounds | L v,σ | ≤ | V m | and| U v,σ | ≤ | V m | if σ is an arbitrary ordering.[-vvvvvvvvvvvvvvvvv 
-]
Revised: This is why we use the Manhattan distance to determine σ , because we have weaker upper bounds | L v,σ | ≤ | V m | and | U v,σ | ≤ | V m | if σ is an arbitrary ordering.[-vvvvvvvvvvvvvvvvv 
-]

Original: = (0 , 0) v 9 = (0 , 1) v 6 = (1 , 0) v 15 = (1 , 1) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108)

Original: Example.
Revised: Example.

Original: In the hypercube shown in Figure 1, vertex v 8 dominates the vertices in{ v 0 , v 1 , v 2 , v 4 , v 5 } and is dominated by the vertices in { v 11 , v 12 , v 13 , v 14 , v 15 } .
Revised: In the hypercube shown in Figure 2, vertex v 8 dominates the vertices in { v 0 , v 1 , v 2 , v 4 , v 5 } and is dominated by the vertices in { v 11 , v 12 , v 13 , v 14 , v 15 } .

Original: By the deﬁnition ofthe minimal dominated set, we have L v 8 ,σ = { v 4 , v 5 } because vertices v 0 , v 1 , and v 2 are dominatedby v 4 or v 5 .
Revised: By the deﬁnition of the minimal dominated set, we have L v 8 ,σ = { v 4 , v 5 } because vertices v 0 , v 1 , and v 2 are dominated by v 4 or v 5 .

Original: By the deﬁnition of the minimal dominating set, we have U v 8 ,σ = ∅ because all thevertices that dominate v 8 appear after v 8 in ordering σ .
Revised: By the deﬁnition of the minimal dominating set, we have U v 8 ,σ = ∅ because all the vertices that dominate v 8 appear after v 8 in ordering σ .

Original: 3.2 Auxiliary Neural Network
Revised: 3.3 Auxiliary Neural Network

Original: For HLL, we construct an auxiliary neural network gθ ( p, v )  . To explain its construction, we deﬁnetwo functions l ( p, v ) and u ( p, v ) for p ∈ R d − m and a vertex v ∈ V m as 
Revised: We explain the function g ( p, v ) associated with each vertex v ∈ V m . To deﬁne it, we deﬁne two functions l ( p, v ) and u ( p, v ) for p ∈ R d − m and a vertex v ∈ V m as [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where L v,σ and U v,σ are the minimal dominated and dominating sets, respectively.
Revised: vvvvvvvvvvvvvvvvv 0 = (0 , 0) v 9 = (0 , 1) v 6 = (1 , 0) v 15 = (1 , 1) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) (cid:108) where L v,σ and U v,σ are the minimal dominated and dominating sets, respectively.

Original: Note that l ( p, v )and u ( p, v ) are deﬁned so as to satisfy the inequality l ( p, v ) ≤ f ( p, v ) ≤ u ( p, v ) .
Revised: Note that l ( p, v ) and u ( p, v ) are deﬁned so as to satisfy the inequality l ( p, v ) ≤ f ( p, v ) ≤ u ( p, v ) .

Original: By using thisinequality, we deﬁnea third function g ( p, v ) as a function that satisﬁes
Revised: By using this inequality, we deﬁne g ( p, v ) as a function that satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Since we have l ( p, v ) ≤ f ( p, v ) ≤ u ( p, v ) , there always exists 0 ≤ g ( p, v ) ≤ 1 that satisﬁes Eq.
Revised: Since we have l ( p, v ) ≤ f ( p, v ) ≤ u ( p, v ) , there always exists 0 ≤ g ( p, v ) ≤ 1 that satisﬁes Eq.

Original: We show an example of the deﬁnitions of the three functions for a simple case with m = 1 and k = 3in Appendix A.

Original: The auxiliary neural network g θ ( p, v ) in HLLis trained to learn the function g ( p, v ) .
Revised: We construct an auxiliary neural network g θ ( p, v ) in HLL to learn the function g ( p, v ) .

Original: To satisfy theconstraint 0 ≤ g ( p, v ) ≤ 1 , we use the sigmoid function in the last layer of g θ ( p, v ) as the activationfunction.
Revised: To satisfy the constraint 0 ≤ g ( p, v ) ≤ 1 , we use the sigmoid function in the last layer of g θ ( p, v ) as the activation function.

Original: Note that, by the deﬁnitions of the three functions l ( p, v ) , u ( p, v ) , and g ( p, v ) , we cancompute f ( p, v ) for any p and v by using g θ ( p, v ) .
Revised: Note that, by the deﬁnitions of the three functions l ( p, v ) , u ( p, v ) , and g ( p, v ) , we can compute f ( p, v ) for any p and v by using g θ ( p, v ) .

Original: 3.3 Inference and Training Algorithms
Revised: Example of Parameterization of HLL. We show an example of the functions l ( p, v ) , u ( p, v ) , and g ( p, v ) for a simple case with m = 1 and k = 3 (see Figure 1). In this case, the hypercube has three vertices V m = { v 0 , v 1 , v 2 } , and we use the total ordering σ = ( v 0 , v 1 , v 2 ) here. For the ﬁrst element v 0 of σ , we have l ( p, v 0 )

Original: {+= 
+}
Revised: {+= 
+}

Revised: and u ( p, v 0 )

Original: {+= 
+}
Revised: {+= 
+}

Revised: because L v 0 ,σ = U v 0 ,σ = ∅ and g ( p, v 0 )

Revised: =

Revised: f ( p, v 0 ) , which satisﬁes l ( p, v 0 ) ≤ f ( p, v 0 ) ≤ u ( p, v 0 ) and 0 ≤ g ( p, v 0 ) ≤ 1 .

Revised: For the second element v 1 of σ , we have l ( p, v 1 ) =

Revised: f ( p, v 0 ) and u ( p, v 1 )

Revised: = 1 because L v 1 ,σ = { v 0 } and U v 1 ,σ = ∅ .

Revised: The function g ( p, v 1 ) is deﬁned as a function that satisﬁes

Revised: [Equation]

Revised: Note that this deﬁnition satisﬁes l ( p,

Revised: [Equation]

Revised: 1 ) ≤ f

Revised: [Equation]

Revised: p, v 1 ) ≤ u ( p, v 1 ) and 0 ≤ g ( p, v 1 ) ≤ 1 , which means that the monotonicity constraint f ( p, v 0 ) ≤ f ( p, v 1 ) ≤ 1 is satisﬁed.

Original: The inference algorithm of HLL is similar to TL Lattice, and Figure 2 illustrates it.{+) = 
 because L v 2 ,σ = { v 1 } and U v 2 ,σ = ∅ .+}
Revised: Similarly, for the third element v 2 in σ , we have l ( p, v 2 ) = f ( p, v 1 ) and u ( p, v 2 {+) = 
 because L v 2 ,σ = { v 1 } and U v 2 ,σ = ∅ .+}

Original: Given an inputx =
Revised: The function g ( p, v 2 ) is deﬁned as a function that satisﬁes [Equation]

Original:  ( p, q )  , 
Revised: g ( p, v 2 )) · l ( p, v 2 ) + g ( p, v 2 ) · u ( p, v 2 ) . Note that this deﬁnition satisﬁes l ( p, v 2 ) ≤ f ( p, v 2 ) ≤ u ( p, v 2 ) and 0 ≤ g ( p, v 2 ) ≤ 1 , which means that the monotonicity constraint f ( p, v 1 ) ≤ f ( p, v 2 ) ≤ 1 is satisﬁed. 3.4 Inference and Training Algorithms The inference algorithm of HLL is similar to TL Lattice, and Figure 3 illustrates it.

Original:  we ﬁrst calculate f ( p, v ) for all v ∈ V q by using g θ ( p, v ) , where V q ⊆
Revised: Given an input x = ( p, q ) , we ﬁrst calculate f ( p, v ) for all v ∈ V q by using g θ ( p, v ) , where V q ⊆

Original: V m is the set ofthe vertices that surround q in the hypercube.
Revised: V m is the set of the vertices that surround q in the hypercube.

Original: Then, we estimate f ( p, q ) as an interpolation of thevalues f ( p, v ) for v ∈ V q .
Revised: Then, we estimate f ( p, q ) as an interpolation of the values f ( p, v ) for v ∈ V q .

Original: As for the interpolation algorithm, we can use multilinear interpolationor simplex interpolation Gupta et al. 
Revised: As for the interpolation algorithm, we can use multilinear interpolation or simplex interpolation [Gupta et al. , 2016].

Original: Note that the size of V q depends on the interpolationalgorithm (i.e., | V q | = 2 m if we use multilinear interpolation, and | V q | =
Revised: Note that the size of V q depends on the interpolation algorithm (i.e., | V q | = 2 m if we use multilinear interpolation, and | V q | =

Original: O ( m log m ) if we usesimplex interpolation).
Revised: O ( m log m ) if we use simplex interpolation).

Original: The training algorithm of HLL is the same as that of the standard neural network layer.
Revised: The training algorithm of HLL is the same as that of the standard neural network layer.

Original: For example,if we train a neural network model that consists of a single HLL, the parameters of HLL (i.e. theparameters θ of the auxiliary neural network g θ ) are trained to minimize the empirical loss 
Revised: For example, if we train a neural network model that consists of a single HLL, the parameters of HLL (i.e. the parameters θ of the auxiliary neural network g θ ) are trained to minimize the empirical loss min g θ n (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original:  x = ( p, q ) (cid:45)
Revised: (cid:96) ( y i , f ( x i )) , x = ( p, q ) (cid:45)

Original: Compute f ( p, v ) by using g θ f ( p, v 1 ) f ( p, v 5 ) f ( p, v 7 ) (cid:113) q f ( p, v 2 ) f
Revised: Compute f ( p, v ) by using g θ f ( p, v 1 ) f ( p, v 5 ) f ( p, v 7 ) (cid:113) q f ( p, v 2 ) f

Original: ( p, v 4 ) f ( p, v 8 ) (cid:0)(cid:0) (cid:0)(cid:0) (cid:0)(cid:0) (cid:0)(cid:0) (cid:45)
Revised: ( p, v 4 ) f ( p, v 8 ) (cid:0)(cid:0) (cid:0)(cid:0) (cid:0)(cid:0) (cid:0)(cid:0) (cid:45)

Original: Interpolate f ( p, q ) where { ( x i , y i ) } ni =1 is a dataset, f ( x i ) is the output of the inference algorithm of HLL for input x i ,and (cid:96) is a loss function.
Revised: Interpolate f ( p, q ) where { ( x i , y i ) } ni =1 is a dataset, f ( x i ) is the output of the inference algorithm of HLL for input x i , and (cid:96) is a loss function.

Original: 3.4 Partial Monotonicity
Revised: 3.5 Partial Monotonicity

Original: As already proven in Gupta et al. [2016], the function f ( p, q ) computed by an interpolation algorithmsatisﬁes monotonicity constraints if the values corresponding to the vertices of the hypercube satisfythe constraints.
Revised: As already proven in [Gupta et al. , 2016], the function f ( p, q ) computed by an interpolation algorithm satisﬁes monotonicity constraints if the values corresponding to the vertices of the hypercube satisfy the constraints.

Original: Therefore, we show that we have f ( p, u ) ≤ f ( p, v ) for any pair of vertices u, v ∈ V msuch that u is dominated by v .
Revised: Therefore, we show that we have f ( p, u ) ≤ f ( p, v ) for any pair of vertices u, v ∈ V m such that u is dominated by v .

Original: Note that this lemma holds for any vertex ordering σ and is notrestricted to the vertex ordering determined by the Manhattan distance.
Revised: Note that this proposition holds for any vertex ordering σ and is not restricted to the vertex ordering determined by the Manhattan distance.

Original: Lemma 3.4 In HLL, f ( p, u ) ≤ f ( p, v ) holds for any p ∈ R d − m and any pair of vertices u, v ∈ V msuch that u is dominated by v
Revised: Proposition 3.4 In HLL, f ( p, u ) ≤ f ( p, v ) holds for any p ∈ R d − m and any pair of vertices u, v ∈ V m such that u is dominated by v

Original: Proof.
Revised: Proof.

Original: Suppose that u precedes v in the total ordering σ .
Revised: Suppose that u precedes v in the total ordering σ .

Original: By the deﬁnition of the minimal dominatedset, there exists a sequence of vertices u 0 (= u ) , u 1 , u 2 , . .
Revised: By the deﬁnition of the minimal dominated set, there exists a sequence of vertices u 0 (= u ) , u 1 , u 2 , . .

Original: , u J (= v )
Revised: , u J (= v )

Original: such that u j − 1 ∈
Revised: such that u j − 1 ∈

Original: L u j ,σ holdsfor j = 1 , 2 , . . .
Revised: L u j ,σ holds for j = 1 , 2 , . . .

Original: , J .
Revised: , J .

Original: Since u j − 1 ∈
Revised: Since u j − 1 ∈

Original: L u j ,σ , we have f ( p, u j − 1 ) ≤ l ( p, u j ) ≤ f ( p, u j ) by Eq.
Revised: L u j ,σ , we have f ( p, u j − 1 ) ≤ l ( p, u j ) ≤ f ( p, u j ) by Eq.

Original: Hence, we have f ( p, u ) =
Revised: Hence, we have f ( p, u ) =

Original: f ( p, u 0 ) ≤ f ( p, u 1 ) ≤ · · · ≤ f ( p, u J ) =
Revised: f ( p, u 0 ) ≤ f ( p, u 1 ) ≤ · · · ≤ f ( p, u J ) =

Original: f ( p, v ) .
Revised: f ( p, v ) .

Original: Similarly, suppose that v precedes u in the total ordering σ .
Revised: Similarly, suppose that v precedes u in the total ordering σ .

Original: By the deﬁnition of the minimaldominating set, there exists a sequence of vertices v 0 (= v ) , v 1 , v 2 , . .
Revised: By the deﬁnition of the minimal dominating set, there exists a sequence of vertices v 0 (= v ) , v 1 , v 2 , . .

Original: . , v J (= u ) such that v j − 1 ∈
Revised: . , v J (= u ) such that v j − 1 ∈

Original: U v j ,σ holds for j = 1 , 2 , . . .
Revised: U v j ,σ holds for j = 1 , 2 , . . .

Original: , J .
Revised: , J .

Original: Since v j − 1 ∈ U v j ,σ , we have f ( p, v j − 1 ) ≥ u ( p, v j ) ≥ f ( p, v j ) by
Revised: Since v j − 1 ∈ U v j ,σ , we have f ( p, v j − 1 ) ≥ u ( p, v j ) ≥ f ( p, v j ) by

Original: Eq.
Revised: Eq.

Original: Hence, we have f ( p, v ) =
Revised: Hence, we have f ( p, v ) =

Original: f ( p, v 0 ) ≥ f
Revised: f ( p, v 0 ) ≥ f

Original: ( p, v 1 ) ≥ · · · ≥ f ( p, v J ) =
Revised: ( p, v 1 ) ≥ · · · ≥ f ( p, v J ) =

Original: f ( p, u ) .
Revised: f ( p, u ) .

Original: 3.5 Complexity Analysis
Revised: 3.6 Complexity Analysis

Original: We show that the time complexity of the inference algorithm of HLL is O (( T ( g θ ) + m ) | V m | ) , where
Revised: We show that the time complexity of the inference algorithm of HLL is O (( T ( g θ ) + m ) | V m | ) , where

Original: T ( g θ ) is the time complexity of computing g θ ( p, v ) for an input ( p, v ) ∈ R d − m × R m .
Revised: T ( g θ ) is the time complexity of computing g θ ( p, v ) for an input ( p, v ) ∈ R d − m × R m .

Original: First, thetime complexity to compute f ( p, v ) for all vertices v ∈ V m is O
Revised: First, the time complexity to compute f ( p, v ) for all vertices v ∈ V m is O

Original: (( T ( g θ ) + m ) | V m | ) since it takes
Revised: ( T ( g θ ) + m ) | V m | ) since it takes

Original: O ( T ( g θ ) + | L v,σ | + | U v,σ | ) time to compute f ( p, v ) for a single vertex v ∈ V m , and we have| L v,σ | + | U v,σ | ≤ m .
Revised: O ( T ( g θ ) + | L v,σ | + | U v,σ | ) time to compute f ( p, v ) for a single vertex v ∈ V m , and we have | L v,σ | + | U v,σ | ≤ m .

Original: Then, the time complexity for the interpolation algorithm is O (2 m ) if we usemultilinear interpolation and O ( m log m ) if we use simplex interpolation Gupta et al. 
Revised: Then, the time complexity for the interpolation algorithm is O (2 m ) if we use multilinear interpolation and O ( m log m ) if we use simplex interpolation [Gupta et al. , 2016].

Original: Sincewe have 2 m ≤ | V m | , the time complexity for the interpolation algorithm is smaller than that forthe computation of f ( p, v ) .
Revised: Since we have 2 m ≤ | V m | , the time complexity for the interpolation algorithm is smaller than that for the computation of f ( p, v ) .

Original: Hence, the overall time complexity remains O (( T ( g θ ) + m ) | V m | ) .
Revised: Hence, the overall time complexity remains O (( T ( g θ ) + m ) | V m | ) .

Original: Ifwe can assume that m is a small integer (i.e., m is a constant), the computation time for inferencecan be written simply as O ( T ( g θ )) .
Revised: If we can assume that m is a small integer (i.e., m is a constant), the computation time for inference can be written simply as O ( T ( g θ )) .

Original: Note that, if the time complexity T ( g θ ) is a linear functionon d , the time complexity of the inference algorithm for HLL is smaller than that for TL Latticebecause the inference algorithm of TL Lattice takes O (2 d ) and O ( d log d ) times if we use multilinearinterpolation and simplex interpolation, respectively Gupta et al. 
Revised: Note that, if the time complexity T ( g θ ) is a linear function on d , the time complexity of the inference algorithm for HLL is smaller than that for TL Lattice because the inference algorithm of TL Lattice takes O (2 d ) and O ( d log d ) times if we use multilinear interpolation and simplex interpolation, respectively [Gupta et al. , 2016].

Original: The space complexity of HLL is O ( | g θ | + m | V m | ) if the vertex ordering σ is determined by the
Revised: The space complexity of HLL is O ( | g θ | + m | V m | ) if the vertex ordering σ is determined by the

Original: Manhattan distance, where we denote the size of the auxiliary neural network g θ ( p, v ) by | g θ | , and the
Revised: Manhattan distance, where we denote the size of the auxiliary neural network g θ ( p, v ) by | g θ | , and the

Original: O ( m | V m | ) term is the space required to store L v,σ and U v,σ for all v ∈ V m .
Revised: O ( m | V m | ) term is the space required to store L v,σ and U v,σ for all v ∈ V m .

Original: Note that, if we use anarbitrary vertex ordering σ , we require O ( | V m | 2 ) space to store L v,σ and U v,σ for all v ∈ V m becausewe know only that | L v,σ | ≤ | V m | and | U v,σ | ≤ | V m | .
Revised: Note that, if we use an arbitrary vertex ordering σ , we require O ( | V m | 2 ) space to store L v,σ and U v,σ for all v ∈ V m because we know only that | L v,σ | ≤ | V m | and | U v,σ | ≤ | V m | .

Original: Since | V m | 2 is much larger than m | V m | , werecommend using the Manhattan distance to determine σ in HLL.
Revised: Since | V m | 2 is much larger than m | V m | , we recommend using the Manhattan distance to determine σ in HLL.

Original:  Experiments
Revised:  Experiments

Original: In our experiments, we demonstrate that HLL did not sacriﬁce its prediction performance comparedwith TL Lattice.
Revised: In our experiments, we demonstrate that HLL did not sacriﬁce its prediction performance compared with TL Lattice.

Original: Note that HLL is designed to satisfy monotonicity constraints and is not designedto improve the prediction performance compared with TL Lattice.
Revised: Note that HLL is designed to satisfy monotonicity constraints and is not designed to improve the prediction performance compared with TL Lattice.

Original: Therefore, the purpose of ourexperiments was not to show such improvements but to show that HLL did not sacriﬁce the predictionperformance to satisfy the constraints.
Revised: Therefore, the purpose of our experiments was not to show such improvements but to show that HLL did not sacriﬁce the prediction performance to satisfy the constraints.

Original: In addition, wealso demonstrate that we could use HLL forhigh-dimensional input vectors with a small m .
Revised: In addition, we demonstrate that we could use HLL for high-dimensional input vectors with a small m .

Original: All our experiments were conducted on a virtualmachine with an Intel Xeon CPU (3.30 GHz) processor without any GPU and 64 GB of memoryrunning Red Hat Enterprise Linux Server 7.6.
Revised: All our experiments were conducted on a virtual machine with an Intel Xeon CPU (3.30 GHz) processor without any GPU and 64 GB of memory running Red Hat Enterprise Linux Server 7.6.

Original: 4.1 Neural Network Models
Revised: 4.1 Neural Network Models

Original: In our experiments, we compared these four neural network models.
Revised: In our experiments, we compared these four neural network models.

Original: • MLP .
Revised: • MLP .

Original: A multi-layer perceptron model with three hidden layers.
Revised: A multi-layer perceptron model with three hidden layers.

Original: We used the ReLU acti- vation layer for the hidden layers and the sigmoid function for the output layer.
Revised: We used the ReLU activation layer for the hidden layers and the sigmoid function for the output layer.

Original: Note thatthis neural network is not designed for partially monotone regression and cannot handlemonotonicity constraints.• HLL .
Revised: Note that this neural network is not designed for partially monotone regression and cannot handle monotonicity constraints. • HLL .

Original: A neural network model consisting of a single hierarchical lattice layer.
Revised: A neural network model consisting of a single hierarchical lattice layer.

Original: We used anMLP model with three hidden layers for the auxiliary neural network g θ ( p, v ) , and we usedmultilinear interpolation in the inference algorithm.• TL Lattice .
Revised: We used an MLP model with three hidden layers for the auxiliary neural network g θ ( p, v ) , and we used multilinear interpolation in the inference algorithm. • TL Lattice .

Original: A neural network model consisting of a single TL Lattice layer Gupta et al.[2016].
Revised: A neural network model consisting of a single TL Lattice layer [Gupta et al. , 2016].

Original: This layer uses O ( k d ) space, and therefore, it can be used only for low-dimensionalinputs (i.e., d is a small integer).
Revised: This layer uses O ( k d ) space, and therefore, it can be used only for low-dimensional inputs (i.e., d is a small integer).

Original: We used multilinear interpolation in the inference algorithm.• TL RTL.
Revised: We used multilinear interpolation in the inference algorithm. • TL RTL.

Original: A neural network model consisting of a single RTL layer Canini et al. [2016] fromthe TensorFlow Lattice library, which is a random ensemble of tiny lattices (i.e., TL Lattices).This layer consists of (cid:100) d/r (cid:101) tiny lattices, and each tiny lattice receives an r -dimensionalinput vector, where r is an integer hyperparameter.
Revised: A neural network model consisting of a single RTL layer [Canini et al. , 2016] from the TensorFlow Lattice library, which is a random ensemble of tiny lattices (i e., TL Lattices). This layer consists of (cid:100) d/r (cid:101) tiny lattices, and each tiny lattice receives an r -dimensional input vector, where r is an integer hyperparameter.

Original: Hence, this layer uses O (( d/r ) k r ) space,which is signiﬁcantly smaller than that of TL Lattice when r (cid:28) d .
Revised: Hence, this layer uses O (( d/r ) k r ) space, which is signiﬁcantly smaller than that of TL Lattice when r (cid:28) d .

Original: Note that a theoreticalanalysis in Canini et al. [2016] shows that TL RTL with d/r tiny lattices is equivalent to thesingle d -dimensional TL Lattice.
Revised: Note that a theoretical analysis in [Canini et al. , 2016] shows that TL RTL with d/r tiny lattices is equivalent to the single d -dimensional TL Lattice.

Original: We used this model for datasets with high-dimensionalinputs (i.e., d is a large integer).
Revised: We used this model for datasets with high-dimensional inputs (i.e., d is a large integer).

Original: We used Python 3.8.12 and PyTorch 1.8.1 to implement MLP and HLL.
Revised: We used Python 3.8.12 and PyTorch 1.8.1 to implement MLP and HLL, and the implementation of HLL is available at https://github.com/IBM/pmlayer .

Original: As for TL Lattice and TL RTL, we used TensorFlow 2.3.0 and TensorFlow Lattice 2.0.10, which is the implementation providedby the authors Canini et al. [2016]; Gupta et al. [2016] and available as an open source library under
Revised: As for TL Lattice and TL RTL, we used TensorFlow 2.3.0 and TensorFlow Lattice 2.0.10, which is the implementation provided by the authors [Canini et al. , 2016; Gupta et al. , 2016] and available as an open source library under

Original: Apache License 2.0 at https://www.tensorflow.org/lattice .
Revised: Apache License 2.0 at https://www.tensorflow.org/lattice .

Original: The TensorFlow Lattice libraryalso provides an implementation of the Kronecker-Factored lattice layerMorioka et al.
Revised: The TensorFlow Lattice library also provides an implementation of the Kronecker-Factored lattice layer

Original: [2021], whichuses a smaller amount of memory than TL Lattice.
Revised: [Morioka et al. , 2021], which uses a smaller amount of memory than TL Lattice.

Original: However, we did not include the experimentalresults for this layer because TL RTL signiﬁcantly outperformed it for the datasets we used in termsof prediction performance.
Revised: However, we did not include the experimental results for this layer because TL RTL signiﬁcantly outperformed it for the datasets we used in terms of prediction performance.

Original: Throughout our experiments, all problems were solved as regression problems, and we used meansquared error (MSE) for the evaluation metric.
Revised: Throughout our experiments, all problems were solved as regression problems, and we used mean squared error (MSE) for the evaluation metric.

Original: For the training of the neural network models, weused the Adam optimizer Kingma and Ba [2015].
Revised: For the training of the neural network models, we used the Adam optimizer [Kingma and Ba, 2015].

Original: In HLL, TL Lattice, and TL RTL, the number ofvertices in each dimension of the hypercube was set to k by default.
Revised: In HLL, TL Lattice, and TL RTL, the number of vertices in each dimension of the hypercube was set to k by default.

Original: However, the numbers of thevertices in the dimensions that correspond to small categorical values were set to the numbers of thecategories.
Revised: However, the numbers of the vertices in the dimensions that correspond to small categorical values were set to the numbers of the categories.

Original: For example, if the i -th dimension corresponds to a binary feature, then we set k i = 2(instead of k i = k ).
Revised: For example, if the i -th dimension corresponds to a binary feature, then we set k i =instead of k i = k .

Original: 4.2 Real Datasets
Revised: 4.2 Real Datasets

Original: We used 12 real datasets taken from the UCI Machine Learning Repository Dua and Graff [2017] inour experiments.
Revised: We used 12 real datasets taken from the UCI Machine Learning Repository [Dua and Graff, 2017] in our experiments.

Original: We used six small datasets
Revised: We used six small datasets: Energy Efﬁciency [Tsanas and Xifara, 2012], QSAR Aquatic Toxicity [Cassotti et al. , 2014], Concrete Compressive Strength [Yeh, 1998], Contraceptive Method Choice [Lim et al. , 2000], Abalone [Nash et al. , 1994], and Shill Bidding [Alzahrani and Sadaoui, 2020], and six large datasets: Online Shoppers Purchasing Intention [Sakar et al. , 2019],

Original: [Table]
Revised: [Table]

Original: [2015], Bank Marketing Moro et al. 
Revised: Adult, Online News Popularity [Fernandes et al. , 2015], Facebook Comment Volume [Singh et al. , 2015], Bank Marketing

Original: [2014], and Blog Feedback Buza [2014].
Revised: [Moro et al. , 2014], and Blog Feedback [Buza, 2014].

Original: We conﬁrmed thatnone of these datasets contained any personally identiﬁable data points nor any offensive content.
Revised: We conﬁrmed that none of these datasets contained any personally identiﬁable data points nor any offensive content.

Original: Table 1 shows a summary of these 12 datasets.
Revised: Table 1 shows a summary of these 12 datasets.

Original: Some of the datasets were already split into trainingand test data, and the others were unsplit (i.e., a single dataset was given).
Revised: Some of the datasets were already split into training and test data, and the others were unsplit (i.e., a single dataset was given).

Original: Column n of this tableshows the number of data points in the training dataset for the split datasets and the number of datapoints for the unsplit datasets.
Revised: Column n of this table shows the number of data points in the training dataset for the split datasets and the number of data points for the unsplit datasets.

Original: In our experiments, we removed all the data points that had missingfeature values.
Revised: In our experiments, we removed all the data points that had missing feature values.

Original: Column d of Table 1 shows the number of features after our feature encoding.
Revised: Column d of Table 1 shows the number of features after our feature encoding.

Original: We brieﬂy explainour feature encoding here.
Revised: We brieﬂy explain our feature encoding here.

Original: We normalized all the continuous values (including the target values) to arange between 0 and 1 by using the min-max scaling algorithm (i.e. a simple linear transformation).
Revised: We normalized all the continuous values (including the target values) to a range between 0 and 1 by using the min-max scaling algorithm (i.e. a simple linear transformation).

Original: We used label encoding to encode the ordinal categorical values.
Revised: We used label encoding to encode the ordinal categorical values.

Original: For example, we replaced the ordinalcategorical values { low , med , high } with low = 0 , med = 0 .
Revised: For example, we replaced the ordinal categorical values { low , med , high } with low = 0 , med = 0 .

Original: 5 , and high = 1 so that the replacedvalues were in the range between 0 and 1.
Revised: 5 , and high = 1 so that the replaced values were in the range between 0 and 1.

Original: We used one-hot encoding for the other categorical values.
Revised: We used one-hot encoding for the other categorical values.

Original: When we used TL Lattice and TL RTL, we renormalized the input feature vectors from [0 , 1] d to[0 , k − 1] d , where k is a hyperparameter, to conform to the input speciﬁcations of these two models.
Revised: When we used TL Lattice and TL RTL, we renormalized the input feature vectors from [0 , 1] d to [0 , k − 1] d , where k is a hyperparameter, to conform to the input speciﬁcations of these two models.

Original: The last column of Table 1 shows the monotone features we used in our experiments.
Revised: The last column of Table 1 shows the monotone features we used in our experiments.

Original: The italicfeature names represent monotonically decreasing features, and the others represent monotonicallyincreasing features.
Revised: The italic feature names represent monotonically decreasing features, and the others represent monotonically increasing features.

Original: We should choose monotone features by using domain knowledge for eachdataset, but we used the SHAP values Lundberg and Lee [2017] to ﬁnd candidates of monotonefeatures for the datasets for which we had little domain knowledge.
Revised: We should choose monotone features by using domain knowledge for each dataset, but we used the SHAP values [Lundberg and Lee, 2017] to ﬁnd candidates of monotone features for the datasets for which we had little domain knowledge.

Original: We expected that there would bean almost monotonic relationship between the SHAP value and the i -th feature value (i.e., x [ i ] ) if thetarget value was monotonic on the i -th feature.
Revised: We expected that there would be an almost monotonic relationship between the SHAP value and the i -th feature value (i.e., x [ i ] ) if the target value was monotonic on the i -th feature.

Original: In our experiments, we constructed an MLP model foreach dataset and computed its SHAP values.
Revised: In our experiments, we constructed an MLP model for each dataset and computed its SHAP values.

Original: Then, we checked for each feature if the SHAP valueswere monotonically increasing (or decreasing) with respect to the feature values by using Spearman’srank correlation coefﬁcients.
Revised: Then, we checked for each feature if the SHAP values were monotonically increasing (or decreasing) with respect to the feature values by using Spearman’s rank correlation coefﬁcients.

Original: Note that such correlation between SHAP values and feature valuesdoes not mean a monotone relationship, but we found this approach helpful to ﬁnd candidates ofmonotone features for datasets with little domain knowledge.
Revised: Note that such correlation between SHAP values and feature values does not mean a monotone relationship, but we found this approach helpful to ﬁnd candidates of monotone features for datasets with little domain knowledge.

Original: 4.3 Experimental Results
Revised: 4.3 Experimental Results

Original: In our experiments, we split the data points into training, validation, and test data points.
Revised: In our experiments, we split the data points into training, validation, and test data points.

Original: For theunsplit datasets, we divided the data points into training (60%), validation (20%), and test (20%).
Revised: For the unsplit datasets, we divided the data points into training (60%), validation (20%), and test (20%).

Original: Weused a single random split to search for the best hyperparameters and ﬁve random splits to obtain theaverage prediction performance.
Revised: We used a single random split to search for the best hyperparameters and ﬁve random splits to obtain the average prediction performance.

Original: For the datasets that were already split into training and test datasets,we further divided the data points in the training dataset into training (80%) and validation (20%) andkept the test dataset unchanged.
Revised: For the datasets that were already split into training and test datasets, we further divided the data points in the training dataset into training (80%) and validation (20%) and kept the test dataset unchanged.

Original: We used a single random split to search for the best hyperparametersand ﬁve random splits to obtain the average prediction performance without changing the test dataset.
Revised: We used a single random split to search for the best hyperparameters and ﬁve random splits to obtain the average prediction performance without changing the test dataset.

Original: Regarding the hyperparameters, we chose the best hyperparameters for each combination of a neuralnetwork and a dataset; the learning rate was chosen from { 1 , 10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 } , the

Original: [Figure]
Revised: [Figure]

Revised: Regarding the hyperparameters, we chose the best hyperparameters for each combination of a neural network and a dataset; the learning rate was chosen from { 1 , 10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 } , the batch size was chosen from { 8 , 16 , 32 , ..., 4096 } , the number of neurons in the hidden layers was chosen from { 16 , 32 , 64 , ..., 512 } for MLP and HLL, and the hyperparameter r was chosen from { 4 , 5 , 6 , 7 } for TL RTL.

Revised: We used only the training and validation datasets to determine the best hyperparameters, and we used Optuna [Akiba et al. , 2019] to search for the best hyperparameters.

Original: Results for Small Datasets.
Revised: Results for Small Datasets.

Original: We compared the prediction performances of MLP, HLL, and TL
Revised: We compared the prediction performances of MLP, HLL, and TL

Original: Lattice by using the six small datasets (i.e., d was small).
Revised: Lattice by using the six small datasets (i.e., d was small).

Original: We trained these neural network models for1000 epochs with certain early stopping criteria, and Figure 3 shows the results for various k .
Revised: We trained these neural network models for 1000 epochs with certain early stopping criteria, and Figure 4 shows the results for various k .

Original: The redlines show the average of the MSEs for ﬁve runs of MLP, and we regard these values as baselines ofthe prediction performance.
Revised: The red dashed lines show the average of the MSEs for ﬁve runs of MLP, and we regard these values as baselines of the prediction performance.

Original: The blue and orange lines show the average MSEs together with theirstandard deviations for ﬁve runs of HLL and TL Lattice, respectively.
Revised: The blue and orange lines show the average MSEs together with their standard deviations for ﬁve runs of HLL and TL Lattice, respectively.

Original: These results showed that our HLL performed comparably to MLP for all six datasets.
Revised: These results showed that our HLL performed comparably to MLP for all six datasets.

Original: Somewhat surprisingly, we note that using aﬁner-grained lattice (i.e., increasing k ) did not necessarily improve the MSEs.
Revised: Somewhat surprisingly, we note that using a ﬁner-grained lattice (i.e., increasing k ) did not necessarily improve the MSEs.

Original: This phenomenon wasalready mentioned in Gupta et al. [2016] for TL Lattice, and we conﬁrmed this phenomenon both for TL Lattice and HLL in our experiments.
Revised: This phenomenon was already mentioned in [Gupta et al. , 2016] for TL Lattice, and we observed this phenomenon both for TL Lattice and HLL in our experiments.

Original: Figure 3 also shows that the prediction performance of HLLwere also comparable with TL Lattice.
Revised: Figure 4 also shows that the prediction performance of HLL were also comparable with TL Lattice.

Original: Although TL Lattice showed worseMSEs than MLP and
Revised: Although TL Lattice showed worse

Original:  HLL for some datasets, these differences in prediction performance were supposed to come from twofactors.
Revised: MSEs than MLP and HLL for some datasets, these differences in prediction performance were supposed to come from two factors.

Original: One was the difference of the parameterization; HLL used a hypercube of size k m , whereas TL Lattice used a hypercube of size k d .
Revised: One was the difference of the parameterization; HLL used a hypercube of size k m , whereas TL Lattice used a hypercube of size k d .

Original: The other one was the difference of the neural networktraining algorithm; MLP and HLL used the standard training algorithm, whereas TL Lattice used itsown special training algorithm.
Revised: The other one was the difference of the neural network training algorithm; MLP and HLL used the standard stochastic gradient descent algorithm, whereas TL Lattice used the projected gradient descent algorithm.

Revised: In addition, we compared the prediction performace of HLL and TL Lattice with the calibration layer, and the results are shown in Appendix A.4.

Original: Results for Large Datasets.
Revised: Results for Large Datasets.

Original: For the datasets with a large d , we could not use TL Lattice due to itsspace complexity O ( k d ) , so we used TL RTL instead.
Revised: For the datasets with a large d , we could not use TL Lattice due to its space complexity O ( k d ) , so we used TL RTL instead.

Original: TL Lattice requires that at least 2 52 parametersbe stored even if k = 2 since d ≥ 52 , but we cannot store such a large number of parameters inmemory.
Revised: TL Lattice requires that at least 2 52 parameters be stored even if k = 2 since d ≥ 52 , but we cannot store such a large number of parameters in memory.

Original: In contrast, we could use HLL since m is small in the datasets we used.
Revised: In contrast, we could use HLL since m is small in the datasets we used.

Original: We compared the prediction performance of MLP, HLL, and TL RTL by using the six large datasets.
Revised: We compared the prediction performance of MLP, HLL, and TL RTL by using the six large datasets.

Original: We trained these neural network models for 100 epochs with certain early stopping criteria, and
Revised: We trained these neural network models for 100 epochs with certain early stopping criteria, and

Original: Figure 4 shows the results for various k .
Revised: Figure 5 shows the results for various k .

Original: The red  lines show the average MSEs for ﬁve runs of MLP, and we regard these values as baselines of the prediction performance.
Revised: The red dashed lines show the average MSEs for ﬁve runs of MLP, and we regard these values as baselines of the prediction performance.

Original: The blue and greenlines show the average MSEs together with their standard deviations for ﬁve runs of HLL and TL 
Revised: The blue and green lines show the average MSEs together with their standard deviations for ﬁve runs of HLL and TL [Figure]

Original: RTL, respectively.
Revised: RTL, respectively.

Original: Note that the result for TL RTL withk =5 on the Blog Feedback dataset wasmissing due to a memory limitation in our computer environment.
Revised: Note that the result for TL RTL with = on the was missing due to a memory limitation in our computer environment.

Original: These results showed that our
Revised: These results showed that our

Original: HLL performed comparably to MLP and TL RTL regardless of k for the six large datasets.We think [Figure] [Figure]
Revised: HLL performed comparably to MLP and TL RTL regardless of k for the six large datasets.

Original: RTL

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: e

Original: [Figure] RTL [Figure] that HLL performed better than  for some  thanks 
Revised: We think that HLL performed better than MLP for some datasets thanks to the regularization effects coming from the monotonicity constraints.

Original:  Related Work
Revised:  Related Work

Original: There are some neural networks for partially monotone regression other than the TensorFlow Latticelibrary.
Revised: There are some neural networks for partially monotone regression other than the TensorFlow Lattice library.

Original: One of the simplest approaches is to enforce non-negativity constraints on neural networkweights that are related to monotone inputs (e.g., Daniels and Velikova [2010]; You et al. 
Revised: One of the simplest approaches is to enforce non-negativity constraints on neural network weights that are related to monotone inputs (e.g., [Daniels and Velikova, 2010; You et al. , 2017]).

Original: Another simple approach is to use a regularization term to enforce monotonicity Gupta et al. 
Revised: Another simple approach is to use a regularization term to enforce monotonicity [Gupta et al. , 2019;

Original: Monteiro et al. [2021], but a prediction model that uses this approach may not always satisfy themonotonicity constraints.
Revised: Monteiro et al. , 2021], but a prediction model that uses this approach may not always satisfy the monotonicity constraints.

Original: More complex approaches are proposed in Liu et al. [2020]; Sivaramanet al. 
Revised: More complex approaches are proposed in [Liu et al. , 2020; Sivaraman et al. , 2020].

Original: These approaches use a mixed integer linear programming solver or a satisﬁabilitymodulo theories (SMT) solver to ensure the monotonicity of the neural network.
Revised: These approaches use a mixed integer linear programming solver or a satisﬁability modulo theories (SMT) solver to ensure the monotonicity of the neural network.

Original: The disadvantagesof these approaches are their computational complexity and the costs of using the solvers.
Revised: The disadvantages of these approaches are their computational complexity and the costs of using the solvers.

Original:  Conclusion
Revised:  Conclusion

Original: We proposed the hierarchical lattice layer (HLL) for partially monotone neural networks as anextension of TL Lattice Gupta et al. 
Revised: We proposed the hierarchical lattice layer (HLL) for partially monotone neural networks as an extension of TL Lattice [Gupta et al. , 2016].

Original: We demonstrated that HLL resolved two problems with TL Lattice: the  requirement fora special training algorithm and the memory consumption.
Revised: We demonstrated that HLL resolved two problems with TL Lattice: the memory consumption and the requirement for the projected gradient descent algorithm with many constraints.

Original: In ourexperiments, we showed that HLL did not sacriﬁce prediction performances to resolve these problems.
Revised: In our experiments, we showed that HLL did not sacriﬁce prediction performances to resolve these problems.

Original: Note that our experiments were limited to a comparison between single layers (i.e., HLL and TL Lattice).
Revised: Note that our experiments were limited to a comparison between single layers (i.e., HLL and TL Lattice).

Original: The prediction performance of these models might be able to be improved by combiningmultiple layers (e.g., You et al. [2017]) or any other techniques (e.g., changing feature encoding andincreasing the number of layers in MLP) and the memory consumption might also be reduced byusing smaller lattices, but note that the requirement for a special training algorithm  remains for TL Lattice.
Revised: The prediction performance of these models might be able to be improved by combining multiple layers (e.g., [You et al. , 2017]) or any other techniques (e.g., changing feature encoding and increasing the number of layers in MLP) and the memory consumption might also be reduced by using smaller lattices, but note that the requirement for the projected gradient descent algorithm with many constraints remains for TL Lattice.

Original: A limitation of our HLL is that it cannot be used for the applications with a large m .
Revised: A limitation of our HLL is that it cannot be used for the applications with a large m .

Original: If m is large, weneed to combine small HLL layers to construct a large neural network as demonstrated in Canini etal.
Revised: If m is large, we need to combine small HLL layers to construct a large neural network as demonstrated in [Canini et al. , 2016;

Original: [2016]; You et al. [2017] for TL Lattice.
Revised: You et al. , 2017] for TL Lattice.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes]  [No] or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer  either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .•
Revised: [Yes] See Section ?? .

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data are proprietary.

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]Please do not modify the questions and only use
Revised: [N/A]

Original:  the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count 
Revised: Checklist section does not count towards the page limit.

Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] We discussed the limitation ofour work in Sec. 
Revised: [Yes] We discussed the limitation of our work in Sec. 6.

Original: 6.(c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We believe this paper does not have any negative societal impacts, because we propose a general machine learning technique applicable to any application domain.
Revised: [No] We believe this paper does not have any negative societal impacts, because we propose a general machine learning technique applicable to any application domain.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We included the codes used in our experiments in the supplemental material.
Revised: [Yes] We included the codes used in our experiments in the supplemental material.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] We described all the training details in Sec. 4.3.
Revised: [Yes] We described all the training details in Sec. 4.3.

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We described our computing environment in Sec. 4.
Revised: [Yes] We described our computing environment in Sec. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We described the details of the asset TensorFlow Lattice in Sec. 
Revised: [Yes] We described the details of the asset TensorFlow Lattice in Sec. 4.1.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] We described the details of the asset TensorFlow Lattice in Sec. 
Revised: [Yes] We described the details of the asset TensorFlow Lattice in Sec. 4.1.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] We included the codes used in our experiments in the supplemental material.
Revised: [Yes] We included the codes used in our experiments in the supplemental material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] We obtained all datasets from the UCI Machine Learning 
Revised: [Yes] We obtained all datasets from the UCI Machine Learning Repository

Original: Repository Dua and Graff [2017], and we follow the citation requests by each datasetprovider in Sec.4.2.
Revised: [Dua and Graff, 2017], and we follow the citation requests by each dataset provider in Sec.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [Yes] We conﬁrmed that the datasets we used do not contain such information in Sec. 
Revised: [Yes] We conﬁrmed that the datasets we used do not contain such information in Sec. 4.2.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: g ( p, v 1 ) : 1 − g ( p, v 1 ) g ( p, v 2 ) : 1 − g ( p, v 2 ) g

Original: ( p, v 3 ) : 1 − g ( p, v 3 ) f

Original: ( p, v 0 )

Original: f ( p, v 1 ) f ( p, v 2 )

Original: A Parameterization of HLL for Simple Case

Original: We show an example of the functions l ( p, v ) , u ( p, v ) , and g ( p, v ) for a simple case with m =

Original: 1and k = 3 .

Original: In this case, the hypercube has three vertices V m = { v 0 , v 1 , v 2 } , and we use the totalordering σ = ( v 0 , v 1 , v 2 ) here.

Original: Recall that HLL is constructed to satisfy the monotonicity constraint0 ≤ f ( p, v 0 ) ≤ f ( p, v 1 ) ≤ f ( p, v 2 ) ≤ 1 for any p ∈ [0 , 1] d − 1 .

Original: See Figure 5 for an illustration of thedeﬁnitions of g ( p, v 0 ) , g ( p, v 1 ) , and g ( p, v 2 ) .

Original: The three functions l ( p, v ) , u ( p, v ) , and g ( p, v ) are deﬁned to satisfy l ( p, v ) ≤ f ( p, v ) ≤ u ( p, v )and 0 ≤ g ( p, v ) ≤ 1 for any p ∈ [0 , 1] d − 1 and any v ∈ V m .

Original: For the ﬁrst element v 0 of σ , we havel ( p, v 0 )

Original: [-= 
-]
Revised: [-= 
-]

Original: and u ( p, v 0 )

Original: [-= 
-]
Revised: [-= 
-]

Original: because L v 0 ,σ = U v 0 ,σ = ∅ and g ( p, v 0 )

Original: =

Original: f ( p, v 0 ) , which satisﬁesl ( p, v 0 ) ≤ f ( p, v 0 ) ≤ u ( p, v 0 ) and 0 ≤ g ( p, v 0 ) ≤ 1 .

Original: For the second element v 1 of σ , we havel ( p, v 1 ) =

Original: f ( p, v 0 ) and u ( p, v 1 )

Original: = 1 because L v 1 ,σ = { v 0 } and U v 1 ,σ = ∅ .

Original: The function g ( p, v 1 ) isdeﬁned as a function that satisﬁes

Original: [Equation]

Original: Note that this deﬁnition satisﬁes l ( p, v 1 ) ≤ f ( p, v 1 ) ≤ u ( p, v 1 ) and 0 ≤ g ( p, v 1 ) ≤ 1 , which meansthat the monotonicity constraint f ( p, v 0 ) ≤ f ( p, v 1 ) ≤ 1 is satisﬁed.

Original: Similarly, for the third elementv 2 in σ , we have l ( p, v 2 ) =

Original: f ( p, v 1 ) and u ( p, v 2 )

Original: [-= 
-]
Revised: [-= 
-]

Original: because L v 2 ,σ = { v 1 } and U v 2 ,σ = ∅ .

Original: Thefunction g ( p, v 2 ) is deﬁned as a function that satisﬁes

Original: [Equation]

Original: Note that this deﬁnition satisﬁes l ( p, v 2 ) ≤ f ( p, v 2 ) ≤ u ( p, v 2 ) and 0 ≤ g ( p, v 2 ) ≤ 1 , which meansthat the monotonicity constraint f ( p, v 1 ) ≤ f ( p, v 2 ) ≤ 1 is satisﬁed.

Original: B Multilinear Interpolation

Original: For the completeness of this paper, we brieﬂy describe multilinear interpolation algorithm Gupta et al.[2016] here.

Original: For simplicity, we assume that we are given a vector q ∈ [0 , 1] m , and the set of verticesthat surround q is the set of vertices of the (undivided) unit hypercube V q

Original: = { v | v ∈ { 0 , 1 } m } .

Original: Assuming that we know the value f

Original: ( p, v ) for any v ∈ V q and p ∈ R d − m , we can compute themultilinear interpolation by

Original: [Equation]

Original: where q [ i ] and v [ i ] represent the i -th dimension of vectors q and v , respectively.

Original: Example.

Original: We can compute f ( p, q ) for q = (0 . 3 , 0 . 6) by

Original: [Equation]

Original: when m = 2 and V q

Original: = { v (0 , 0) , v (0 , 1) , v (1 , 0) , v (1 , 1) } .

Original: C Experiments on Synthetic Datasets
Revised: A Experiments A.1 1D Synthetic Datasets

Original: [Figure]

Original: 1D Synthetic Datasets.

Original: We show our experimental results on one-dimensional synthetic datasets D square

Revised: We show our experimental results on one-dimensional synthetic datasets D square

Original: = { ( x i , y i ) } ni [-=
-] such that y 
Revised: = { ( x i , y i ) } ni [-=
-]= such that y i

Original: i

Original: =
Revised: =

Original: x 2 i and D sqrt1 = { ( x i , y i ) } ni =1 such that y i = √ x i .
Revised: x 2 i and D sqrt1 = { ( x i , y i ) } ni =1 such that y i = √ x i .

Original: Figures 6 and 7 show the prediction results with various k for these two datasets.
Revised: Figures 6 and 7 show the prediction results with various k for these two datasets.

Original: The blue lines showthe predictions by HLL, and the black lines show the ground truth.
Revised: The blue lines show the predictions by HLL, and the black lines show the ground truth.

Original: Both ﬁgures show that increasingk yields better prediction results on these synthetic datasets.
Revised: Both ﬁgures show that increasing k yields better prediction results on these synthetic datasets.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  2D Synthetic Datasets.
Revised: A.2 2D Synthetic Datasets.

Original: We show our experimental results on two-dimensional synthetic datasets D square

Revised: We show our experimental results on two-dimensional synthetic datasets D square

Original: = { ( x i , y i ) } ni =1 such that y i
Revised: = { ( x i , y i ) } ni = such that y i

Original: =
Revised: =

Original: ( x i [1] 2 + x i [2] 2 ) / 2 and D sqrt

Revised: ( x i [1] 2 + x i [2] 2 ) / 2 and D sqrt

Original: = { ( x i , y i ) } ni =1 such thaty i
Revised: = { ( x i , y i ) } ni =1 such that y i

Original: =
Revised: =

Original: ( (cid:112) x
Revised: ( (cid:112) x

Original: i [1] + (cid:112) x
Revised: i [1] + (cid:112) x

Original: i [2]) / 2 (recall that the i -th element of vector x is denoted as x [ i ] ).
Revised: i [2]) /(recall that the i -th element of vector x is denoted as x [ i ] ).

Original: Figures 8 and9 show the prediction results of HLL, TL Lattice, and TL RTL with k = 4 against the ground truth.
Revised: Figures 8 and 9 show the prediction results of HLL, TL Lattice [Figure] and TL RTL with k = 4 against the ground truth.

Original: These prediction models show similar prediction results on these synthetic datasets.
Revised: These prediction models show similar prediction results on these synthetic datasets [Figure] [Figure]

Revised: HLL

Original: [Figure]
Revised: [Figure]

Revised: TL Lattice

Original: [Figure]
Revised: [Figure]

Original: D Experiments on Training Time
Revised: TL RTL 0. A.3 Training Time

Original: We compared the training times of HLL and TL Lattice on the small datasets.
Revised: We compared the training times of HLL and TL Lattice on the small datasets.

Original: In these experiments,we ﬁxed the hyperparameters; k was set to 2 , the learning rate was set to 0 .
Revised: In these experiments, we ﬁxed the hyperparameters; k was set to 2 , the learning rate was set to 0 .

Original: 001 , and the batch size wasset to 128 .
Revised: 001 , and the batch size was set to 128 .

Original: The number of epochs was set to 100, and we did not use any early stopping.
Revised: The number of epochs was set to 100, and we did not use any early stopping.

Original: Figure 10shows the results, where we varied the number of neurons in the auxiliary neural network g θ ( p, v )from 16 to 512 for HLL
Revised: Figureshows the results, where we varied the number of neurons in the auxiliary neural network g θ ( p, v ) from 16 to 512 for HLL

Original: [Figure]
Revised: [Figure]

Original: and the execution times were the averages of ﬁve runs.
Revised: and the execution times were the averages of ﬁve runs.

Original: These results showthat the training times of HLL were shorter than those of TL Lattice for these six small datasets
Revised: These results show that the training times of HLL were shorter than those of TL Lattice for these six small datasets

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: E Experiments with Calibration Layers
Revised: A.4 Combination with Calibration Layers

Original: We conducted additional experiments by using the calibration layers [You et al. , 2017].
Revised: We conducted additional experiments by using the calibration layers [You et al. , 2017].

Original: Morespeciﬁcally, we added the calibration layer for each dimension of the input x before the input of thelattice layer.
Revised: More speciﬁcally, we added the calibration layer for each dimension of the input x before the input of the lattice layer and the monotonicity constraint were imposed on the calibration layers for the monotone inputs.

Original: The number of the end points in each calibration layer is set to 100.
Revised: The number of the end points in each calibration layer was set to 100 and the end points were uniformly distributed over the input domain.

Original: We used both themultilinear interpolation and the simplex interpolation for TL Lattice.
Revised: We used both the multilinear interpolation and the simplex interpolation for TL Lattice.

Original: Figure 11 shows the results.
Revised: Figure 11 shows the results, where the red dashed lines show the average of the MSEs for ﬁve runs of MLP. The blue, orange, and brown lines show the average MSEs together with their standard deviations for ﬁve runs of HLL, TL Lattice with multilinear interpolation, and TL Lattice with simplex interpolation, respectively.

Original: These results show that the prediction performance of TL Lattice with thecalibration layers are improved for the three datasets (a) Energy Efﬁciency, (c)Concrete Compressive
Revised: These results showed that the prediction performance of TL Lattice with the calibration layers were improved for the three datasets (a) Energy Efﬁciency, (c)

Original:  Strength, and (f) Shill Bidding compared with the results shown in Figure 3.
Revised: Concrete Compressive Strength, and (f) Shill Bidding compared with the results shown in Figure 4.

Original: However, they are stillnot better than HL Lattice.
Revised: However, they were still not better than HL Lattice.

Original: We also note that there are no clear advantage of the simplex interpolationcompared to the multilinear interpolation with respect to MSEs.
Revised: We also note that there were no clear advantage of the simplex interpolation compared to the multilinear interpolation with respect to MSEs.

Original: [Figure]
Revised: [Figure]

Original: the neural network models with various k on small datasets.
Revised: the neural network models with various k on small datasets.

Revised: B Multilinear Interpolation

Revised: For the completeness of this paper, we brieﬂy describe multilinear interpolation algorithm

Revised: [Gupta et al. , 2016] here.

Revised: For simplicity, we assume that we are given a vector q ∈ [0 , 1] m , and the set of vertices that surround q is the set of vertices of the (undivided) unit hypercube V q

Revised: = { v | v ∈ { 0 , 1 } m } .

Revised: Assuming that we know the value f

Revised: ( p, v ) for any v ∈ V q and p ∈ R d − m , we can compute the multilinear interpolation by

Revised: [Equation]

Revised: where q [ i ] and v [ i ] represent the i -th dimension of vectors q and v , respectively.

Revised: Example.

Revised: We can compute f ( p, q ) for q = (0 . 3 , 0 . 6) by

Revised: [Equation]

Revised: when m = 2 and V q

Revised: = { v (0 , 0) , v (0 , 1) , v (1 , 0) , v (1 , 1) } .