Original: I N S EARCH FOR A SAT- FRIENDLY B INARIZED N EURAL N ETWORK A RCHITECTURE.
Revised: I N S EARCH FOR A SAT- FRIENDLY B INARIZED N EURAL N ETWORK A RCHITECTURE

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Analyzing the behavior of neural networks is one of the most pressing challenges in deep learning.
Revised: Analyzing the behavior of neural networks is one of the most pressing challenges in deep learning.

Original: Binarized Neural Networks are an important class of networks that allow equivalent representation in Boolean logic and can be analyzed formally with logic-based reasoning tools like SAT solvers.
Revised: Binarized Neural Networks are an important class of networks that allow equivalent representation in Boolean logic and can be analyzed formally with logic-based reasoning tools like SAT solvers.

Original: Such tools can be used to answer existential and probabilistic queries about the network, perform explanation generation, etc.
Revised: Such tools can be used to answer existential and probabilistic queries about the network, perform explanation generation, etc.

Original: However, the main bottleneck for all methods is their ability to reason about large BNNs efﬁciently.
Revised: However, the main bottleneck for all methods is their ability to reason about large BNNs efﬁciently.

Original: In this work, we analyze architectural design choices of BNNs and discuss how they affect the performance of logic-based reasoners.
Revised: In this work, we analyze architectural design choices of BNNs and discuss how they affect the performance of logic-based reasoners.

Original: We propose changes to the BNN architecture and the training procedure to get a simpler network for SAT solvers without sacriﬁcing accuracy on the primary task.
Revised: We propose changes to the BNN architecture and the training procedure to get a simpler network for SAT solvers without sacriﬁcing accuracy on the primary task.

Original: Our experimental results demonstrate that our approach scales to larger deep neural networks compared to existing work for existential and probabilistic queries, leading to signiﬁcant speed ups on all tested datasets.
Revised: Our experimental results demonstrate that our approach scales to larger deep neural networks compared to existing work for existential and probabilistic queries, leading to signiﬁcant speed ups on all tested datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks are among the most successful AI technologies making impact in a variety of practical applications ranging from vision to speech recognition and natural language (Goodfellow et al., 2016).
Revised: Deep neural networks are among the most successful AI technologies making impact in a variety of practical applications ranging from vision to speech recognition and natural language (Goodfellow et al., 2016).

Original: However, many concerns have been raised about the decision making process behind machine learning technology.
Revised: However, many concerns have been raised about the decision making process behind machine learning technology.

Original: For instance, can we trust decisions that neural networks make (EU
Revised: For instance, can we trust decisions that neural networks make (EU

Original: Data Protection Regulation, 2016; Goodman & Flaxman, 2017; NIPS IML Symposium, 2017)?
Revised: Data Protection Regulation, 2016; Goodman & Flaxman, 2017; NIPS IML Symposium, 2017)?

Original: One way to address this problem is to deﬁne properties that we expect the network to satisfy.
Revised: One way to address this problem is to deﬁne properties that we expect the network to satisfy.

Original: Verifying whether the network satisﬁes these properties sheds light on the properties of the function that it represents.
Revised: Verifying whether the network satisﬁes these properties sheds light on the properties of the function that it represents.

Original: Veriﬁcation guarantees can reassure the user that the network behaves as expected.
Revised: Veriﬁcation guarantees can reassure the user that the network behaves as expected.

Original: There are two main approaches to neural network analysis.
Revised: There are two main approaches to neural network analysis.

Original: The ﬁrst approach, the certiﬁcation of neural networks, trains a veriﬁed network that satisﬁes given properties, e.g. a network that is guaranteed to be robust to adversarial perturbations (Wong & Kolter, 2018; Dvijotham et al., 2018;
Revised: The ﬁrst approach, the certiﬁcation of neural networks, trains a veriﬁed network that satisﬁes given properties, e.g. a network that is guaranteed to be robust to adversarial perturbations (Wong & Kolter, 2018; Dvijotham et al., 2018;

Original: Raghunathan et al., 2018; Mirman et al., 2018).
Revised: Raghunathan et al., 2018; Mirman et al., 2018).

Original: However, a set of properties must be known in advance, which might not always be possible.
Revised: However, a set of properties must be known in advance, which might not always be possible.

Original: Moreover, enforcing a set of properties during the training procedure can signiﬁcantly affect the accuracy of the network on the primary task.
Revised: Moreover, enforcing a set of properties during the training procedure can signiﬁcantly affect the accuracy of the network on the primary task.

Original: Finally, certiﬁcation techniques work with relaxation of the original problem and might not be able to certify robust inputs.
Revised: Finally, certiﬁcation techniques work with relaxation of the original problem and might not be able to certify robust inputs.

Original: The second approach, the veriﬁcation of neural networks, takes a trained network as input and focuses only on the veriﬁcation task (Katz et al., 2017;
Revised: The second approach, the veriﬁcation of neural networks, takes a trained network as input and focuses only on the veriﬁcation task (Katz et al., 2017;

Original: Weng et al., 2018; Singh et al., 2019; Xiao et al., 2019).
Revised: Weng et al., 2018; Singh et al., 2019; Xiao et al., 2019).

Original: A number of veriﬁcation frameworks were proposed over the last few years that can be roughly divided into complete (Katz et al., 2017; 2019; Tjeng et al., 2019) and incomplete methods (Weng et al., 2018; Zhang et al., 2018; Singh et al., 2018).
Revised: A number of veriﬁcation frameworks were proposed over the last few years that can be roughly divided into complete (Katz et al., 2017; 2019; Tjeng et al., 2019) and incomplete methods (Weng et al., 2018; Zhang et al., 2018; Singh et al., 2018).

Original: As the training and veriﬁcation tasks are separated, a wide set of properties can be checked.
Revised: As the training and veriﬁcation tasks are separated, a wide set of properties can be checked.

Original: However, the scalability of this approach remains an issue, especially for complete methods.
Revised: However, the scalability of this approach remains an issue, especially for complete methods.

Original: In this work we tackle the scalability problem of the complete veriﬁcation approach, in the context of an important class of networks, Binarized Neural Networks(BNNs)
Revised: In this work we tackle the scalability problem of the complete veriﬁcation approach, in the context of an important class of networks, Binarized Neural Networks(BNNs)

Original: (Hubara et al., 2016).
Revised: (Hubara et al., 2016).

Original: A BNN is an extreme case of quantized neural networks where parameters are primarily binary.
Revised: A BNN is an extreme case of quantized neural networks where parameters are primarily binary.

Original: These networks have a number of important features that are useful in resource constrained environments, like embedded devices or mobile phones (McDanel et al., 2017; Kung et al., 2017).
Revised: These networks have a number of important features that are useful in resource constrained environments, like embedded devices or mobile phones (McDanel et al., 2017; Kung et al., 2017).

Original: They are memory efﬁcient as only one bit per weight must be stored and are computationally efﬁcient as all activations are binary, which enables the use of specialized algorithms for fast binary matrix multiplication.
Revised: They are memory efﬁcient as only one bit per weight must be stored and are computationally efﬁcient as all activations are binary, which enables the use of specialized algorithms for fast binary matrix multiplication.

Original: [Table] More importantly, these networks admit  exact representation in Boolean logic (Narodytska et al., 2018; Cheng et al., 2018).
Revised: More importantly, these networks admit an exact representation in Boolean logic (Narodytska et al., 2018; Cheng et al., 2018).

Original: Such a representation enables us to apply powerful Boolean reasoning tools to the analysis of BNNs.
Revised: Such a representation enables us to apply powerful Boolean reasoning tools to the analysis of BNNs.

Original: For example, we can perform a rich set of queries, ranging from existential queries (e.g., is there a faulty input to the network?), to counting queries (e.g., how many faulty inputs exist?), using logic-based reasoners (Baluta et al., 2019; Shih et al., 2019).
Revised: For example, we can perform a rich set of queries, ranging from existential queries (e.g., is there a faulty input to the network?), to counting queries (e.g., how many faulty inputs exist?), using logic-based reasoners (Baluta et al., 2019; Shih et al., 2019).

Original: This paper makes two main contributions.
Revised: This paper makes two main contributions.

Original: • First, we analyze how different architectural design choices for BNNs affect the performance of SAT solvers.
Revised: • First, we analyze how different architectural design choices for BNNs affect the performance of SAT solvers.

Original: To identify inﬂuential parts of the design, we scrutinize BNNs at three levels of granularity: individual neurons, blocks of layers, and network as a whole.
Revised: To identify inﬂuential parts of the design, we scrutinize BNNs at three levels of granularity: individual neurons, blocks of layers, and network as a whole.

Original: Our work continues work of (Narodytska et al., 2018; Cheng et al.,2018) where BNNs were analyzed on the network level.
Revised: Our work continues work of (Narodytska et al., 2018; Cheng et al.,

Revised: ) where BNNs were analyzed on the network level.

Original: For the block and network levels, we only analyse bottlenecks, propose possible research directions, and position existing work on the network level.
Revised: For the block and network levels, we only analyse bottlenecks, propose possible research directions, and position existing work on the network level.

Original: Our main contribution here is within the network level.
Revised: Our main contribution here is within the network level.

Original: Second, we exploit our ﬁndings to train SAT-friendly BNNs.
Revised: Second, we exploit our ﬁndings to train SAT-friendly BNNs.

Original: We propose a modiﬁed training procedure that makes the resulting network easier for logic-based veriﬁcation tools, like SAT solvers, to reason about.
Revised: We propose a modiﬁed training procedure that makes the resulting network easier for logic-based veriﬁcation tools, like SAT solvers, to reason about.

Original: Modiﬁcations to training are crucially performed so that the accuracy of the network is unaffected.
Revised: Modiﬁcations to training are crucially performed so that the accuracy of the network is unaffected.

Original: Overall, our approach (a) preserves the separation between training and veriﬁcation, by not committing to a certain property during training and (b) boosts the performance of logic-based veriﬁcation.
Revised: Overall, our approach (a) preserves the separation between training and veriﬁcation, by not committing to a certain property during training and (b) boosts the performance of logic-based veriﬁcation.

Original: We implemented the proposed methods and demonstrated signiﬁcant performance gains over previous work (Narodytska et al., 2018;
Revised: We implemented the proposed methods and demonstrated signiﬁcant performance gains over previous work (Narodytska et al., 2018;

Original: Khalil et al., 2019; Baluta et al., 2019).
Revised: Khalil et al., 2019; Baluta et al., 2019).

Original: We get more than 10x-20x improvements on tested benchmarks for both veriﬁcation and quantitative queries, e g., ﬁnding the probability that a perturbation yields an adversarial example.
Revised: We get more than 10x-20x improvements on tested benchmarks for both veriﬁcation and quantitative queries, e.g., ﬁnding the probability that a perturbation yields an adversarial example.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Boolean satisﬁability (SAT).
Revised: Boolean satisﬁability (SAT).

Original: We assume notation and deﬁnitions standard in Boolean Satisﬁability (SAT), i.e. the decision problem for propositional logic (Biere et al., 2009).
Revised: We assume notation and deﬁnitions standard in Boolean Satisﬁability (SAT), i.e. the decision problem for propositional logic (Biere et al., 2009).

Original: SAT formulas are deﬁned over a set of Boolean variables { x 1 , . . .
Revised: SAT formulas are deﬁned over a set of Boolean variables { x 1 , . . .

Original: , x n } .
Revised: , x n } .

Original: A literal l i is a variable x
Revised: A literal l i is a variable x

Original: i (‘positive’ polarity) or its complement ¯ x i (‘negative’ polarity).
Revised: i (‘positive’ polarity) or its complement ¯ x i (‘negative’ polarity).

Original: Cardinality constraints over Boolean variable are constraints of the form (cid:80) ni
Revised: Cardinality constraints over Boolean variable are constraints of the form (cid:80) ni

Original: =

Revised: =

Original: l i ≥ k , where l i ∈ { x i , ¯ x i } .
Revised: l i ≥ k , where l i ∈ { x i , ¯ x i } .

Original: In this work we employ reiﬁed cardinality constraints : x (cid:48) ⇔ (cid:80) ni =

Revised: In this work we employ reiﬁed cardinality constraints : x (cid:48) ⇔ (cid:80) ni =

Original: l i ≥ k .
Revised: l i ≥ k .

Original: We use the full sequential counters encoding (Sinz, 2005b) to model this constraint in SAT.
Revised: We use the full sequential counters encoding (Sinz, 2005b) to model this constraint in SAT (see Appendix A for details of the encoding).

Revised: However, other encodings can be used.

Original: We also work with (approximate) solutions counting solvers, e.g. ApproxMC3 .
Revised: We also work with (approximate) solutions counting solvers, e.g. ApproxMC3 .

Original: See details on such methods in Appendix  A.
Revised: See details on such methods in Appendix as well A.

Original: Binarized neural networks We summarise standard BNN structure (Hubara et al., 2016).
Revised: Binarized neural networks We summarise standard BNN structure (Hubara et al., 2016).

Original: A
Revised: A

Original: BNN consists of blocks of layers, each mapping binary vectors to binary vectors.
Revised: BNN consists of blocks of layers, each mapping binary vectors to binary vectors.

Original: We deﬁne a block (referred to as B LOCK ) as a function mapping an input to an output, i.e., B LOCK i : {− 1 , 1 } n i → {− 1 , 1 } n i +1 , i = 1 , . . .
Revised: We deﬁne a block (referred to as B LOCK ) as a function mapping an input to an output, i.e., B LOCK i : {− 1 , 1 } n i → {− 1 , 1 } n i +1 , i = 1 , . . .

Original: , d − 1 , where d − 1 is the number of hidden layers.
Revised: , d − 1 , where d − 1 is the number of hidden layers.

Original: The last block, denoted
Revised: The last block, denoted

Original: O UTPUT , has a slightly different structure: O UTPUT : {− 1 , 1 } n d → R p , where p is the number of outputs of the network.
Revised: O UTPUT , has a slightly different structure: O UTPUT : {− 1 , 1 } n d → R p , where p is the number of outputs of the network.

Original: Each B LOCK takes an input vector x and applies three transformations: a linear transformation (L IN ), batch normalization (B N ) and binarization (B IN ) 1 .
Revised: Each B LOCK takes an input vector x and applies three transformations: a linear transformation (L IN ), batch normalization (B N ) and binarization (B IN ) 1 .

Original: Table 1 shows transformations of internal and output blocks in detail. 
Revised: Table 1 shows transformations of internal and output blocks in detail. [Table]

Original:  L OGIC - BASED ANALYSIS OF BNN S
Revised:  L OGIC - BASED ANALYSIS OF BNN S

Original: An important property of BNNs is that they allow exact encoding into SAT: for each BNN there exists a SAT formula such that solutions of the formula are exactly the set of all valid input/output pairs of the BNN (Narodytska et al., 2018; Cheng et al., 2018).
Revised: An important property of BNNs is that they allow exact encoding into SAT: for each BNN there exists a SAT formula such that solutions of the formula are exactly the set of all valid input/output pairs of the BNN (Narodytska et al., 2018; Cheng et al., 2018).

Original: Such translation opens many possibilities for logic-based analysis of BNNs.
Revised: Such translation opens many possibilities for logic-based analysis of BNNs.

Original: In this section, we describe several such methods.
Revised: In this section, we describe several such methods.

Original: We highlight that reasoning tools available for BNNs are more diverse and powerful compared to frameworks available for the analysis of conventional networks, which focus mostly on veriﬁcation and certiﬁcation of NNs.
Revised: We highlight that reasoning tools available for BNNs are more diverse and powerful compared to frameworks available for the analysis of conventional networks, which focus mostly on veriﬁcation and certiﬁcation of NNs.

Original: Since logic-based analysis of BNNs exploits their logical representation, we start by revisiting how to obtain such a representation before discussing logic-based BNN analysis tools.
Revised: Since logic-based analysis of BNNs exploits their logical representation [Table] we start by revisiting how to obtain such a representation before discussing logic-based BNN analysis tools [Table]

Original: 3. SAT ENCODING OF BNN S
Revised: 3. SAT ENCODING OF BNN S

Revised: We consider how to generate a logical encoding of a BNN (Narodytska et al., 2018).

Revised: Consider a single block B LOCK i , which applies three transformations to the input vector.

Revised: The ﬁrst step is a linear (afﬁne) transformation (L IN ) of the input vector.

Revised: The linear transformation can be based on a fully connected layer or a convolutional layer.

Original: We give details of a propositional encoding of BNNs in Appendix B. Here, we highlight the main part of the encoding. 
Revised: The linear transformation is followed by a scaling which is performed with a batch normalization operation (B N ) (Ioffe & Szegedy, 2015a). Finally, a binarization is performed using the sign function to obtain a binary output vector (B IN ). We will use the following running example to demonstrate the encoding. Example 3.1.

Revised: Consider a block with three inputs and two outputs.

Revised: We deﬁne transformation parameters as follows:

Revised: [Equation]

Revised: We encode these transformations as a system of logical constraints.

Original: We introduce a vector of Boolean variables x i ∈ { 0 , 1 } n i to encode inputs and x i +1 ∈ { 0 , 1 } n i +1 to encode outputs of B LOCK i .
Revised: We introduce a vector of binary variables x i ∈ {− 1 , 1 } n i to encode inputs and x i +1 ∈ {− 1 , 1 } n i +1 to encode outputs of B LOCK i .

Revised: Since transformations are applied sequentially, we can encode the block as their composition:

Revised: [Equation]

Revised: where a ij,t and b ij are parameters of L IN , α ij , σ ij , µ ij and γ ij are parameters of B N .

Original: The encoding converts a network into a set of cardinality constraints.
Revised: To obtain a SAT encoding, we need to eliminate binary variables as a SAT formula is deﬁned over Boolean variables. We introduce Boolean variables x t ∈ { 0 , 1 } , t = 1 , . . . , n i and relate them to the corresponding binary variables x t : x t = 2 x t − 1 , t = 1 , . . . , n i . We get that if x t = 0 then x t = − 1 , otherwise x t = 1 . We can now rewrite the last expression as x i +1 j = sign (cid:32)

Revised: α ij σ i j ( n i (cid:88) t = a ij,t 2 x it − n i (cid:88) t = a ij,t + b ij − µ ij )

Original: Each row j of A i , [ a  ij 1 , . .
Revised: + γ ij (cid:33) , j = 1 . . n i +1 . We denote C j = ( − (cid:80) n i t =1 a ij,t + b ij − µ ij ) , as these are just constants. We get

Revised: [Equation]

Revised: Example 3.2.

Revised: Consider how the encoding works on Example 3.1.

Revised: We introduce three binary variables to encode inputs x 0 i , i = 0 , 1 , 2 , and two binary variables to encode outputs x 10 and x 11 .

Revised: Inputs and outputs are connected as follows:

Revised: [Equation]

Revised: Next we perform variable replacement as explained above using the relation x t = 2 x t − 1 .

Revised: Note that α 01 and α 02 are both positive so we get:

Revised: [Equation]

Original: , a ijn i ] , i ∈
Revised: [Equation] [Equation] [Equation] [Equation]

Original: [1 , d − 1] is encoded using  acardinality constraint over Boolean variables:
Revised: The ﬁnal step is to encode reiﬁed cardinality constraints using Boolean variables instead of binary (integer) variables in Equation 3. We recall the following tautology for a Boolean variable x t + ¯ x t = 1 , where ¯ x t is a negation of x t . As all coefﬁcients a ijt ∈ {− 1 , 1 } , Equation 3 contains linear constraints with unary coefﬁcients (see the last transformation in Example 3.2). We denote

Revised: N j = |{ a ijt < 0 , t = 1 , . .

Revised: . , n i +1 }| the number of negative coefﬁcients in the j th row and assume α ij (cid:54) = 0 .

Original: [Equation]
Revised: [Equation]

Revised: i

Revised: [Equation]

Original: where l it =
Revised: where l it =

Original: [Equation]
Revised: [Equation]

Original: x 
Revised: x it if

Original: [Equation]
Revised: [Equation]

Original: and h j is a constant computed from parameters of the network.
Revised: We recall that a constraint of type Equation 4 is a reiﬁed cardinality constraint that can be translated into Boolean formulae (Narodytska et al., 2018).

Original: We  refer to a SAT encoding of B LOCKi as B IN B LOCK i ( x i , x i +1 )and a SAT encoding of O UTPUT block as B IN O UTPUT ( x d , o ) .
Revised: We present details of such translation in Appendix A. We refer to a SAT encoding of B LOCK as B IN B LOCK i ( x i , x i +1 ) .

Original: We can now represent the entire network asa Boolean formula: [Equation]
Revised: Similarly, we can encode the O UTPUT block that we show in details below as our method needs to make modiﬁcations to the original encoding (see Appendix B.1). We refer to encoding of O UTPUT as

Original: i =1 B IN B LOCK
Revised: B IN O UTPUT .

Original:  i ( x i , x i +1 )
Revised: We can now represent the entire network as a Boolean formula: B IN B NN ( x , o ) ≡ (cid:86) d −i =1 B IN B LOCK i ( x i , x i +1 )

Original: ∧ B IN O UTPUT ( x d , o ) , where B IN B LOCK i ( x i , x i +1 )
Revised: ∧ B IN O UTPUT ( x d , o ) , where B IN B LOCK i ( x i , x i +1 )

Original: encodes the i th block B LOCK i with input x i and output x i +1 , i ∈ [0 , d − 1] , B IN O UTPUT ( x d , o ) is a Boolean encoding of the last layer.
Revised: encodes the i th block B LOCK i with input x i and output x i +1 , i ∈ [0 , d − 1] , B IN O UTPUT ( x d , o ) is a Boolean encoding of the last layer.

Original: 3. BNN’ S REASONERS
Revised: 3. BNN’ S REASONERS

Original: We consider three types of reasoners designed for BNNs.
Revised: We consider three types of reasoners designed for BNNs.

Original: The ﬁrst type includes property checkers that can handle existential queries, e.g. ‘is there an input of the network that violates a given property?’.
Revised: The ﬁrst type includes property checkers that can handle existential queries, e.g. ‘is there an input of the network that violates a given property?’.

Original: The second type deals with probabilistic queries, e.g. ‘what is an approximate probability that a valid input will violate a given property?’.
Revised: The second type deals with probabilistic queries, e.g. ‘what is an approximate probability that a valid input will violate a given property?’.

Original: Finally, the most powerful reasoner answers a large set of queries, like generating explanations, ﬁnding exact probabilities of property violations, etc.
Revised: Finally, the most powerful reasoner answers a large set of queries, like generating explanations, ﬁnding exact probabilities of property violations, etc.

Original: Property checkers.
Revised: Property checkers.

Original: Property checking of BNNs using a SAT-based approach was proposed in (Narodytska et al., 2018; Cheng et al., 2018).
Revised: Property checking of BNNs using a SAT-based approach was proposed in (Narodytska et al., 2018; Cheng et al., 2018).

Original: Given a precondition prec on the inputs x and a property prop on the outputs o , we check if the following statement is valid:prec ( x ) ∧
Revised: Given a precondition prec on the inputs x and a property prop on the outputs o , we check if the following statement is valid:

Original: B IN B NN ( x , o ) ⇒ prop ( o ) .
Revised: [Equation]

Original: To decide if there exists a counterexample to this property, we look for a model of:prec ( x ) ∧ B IN B NN ( x , o )
Revised: To decide if there exists a counterexample to this property, we look for a model of:

Original: ∧ ¬ prop ( o ) .
Revised: [Equation]

Original: An example of property checking is to check for the existence of adversarial perturbations.
Revised: An example of property checking is to check for the existence of adversarial perturbations.

Original: In this case, prec deﬁnes an (cid:15) -ball of valid perturbations and prop states that the classiﬁcation should not change under small perturbations.
Revised: In this case, prec deﬁnes an (cid:15) -ball of valid perturbations and prop states that the classiﬁcation should not change under small perturbations.

Original: An optimization version of this problem was considered in (Khalil et al., 2019) using ILP rather than SAT solvers.
Revised: An optimization version of this problem was considered in (Khalil et al., 2019) using ILP rather than SAT solvers to reason about BNNs.

Original: Quantitative reasoners.
Revised: Quantitative reasoners.

Original: In many practical applications, it is not sufﬁcient to check for the existence of a counterexample to a given property.
Revised: In many practical applications, it is not sufﬁcient to check for the existence of a counterexample to a given property.

Original: We would like to know precise or approximate probability of the undesired behavior.
Revised: We would like to know precise or approximate probability of the undesired behavior.

Original: (Baluta et al., 2019), proposes a framework to answer such probabilistic queries using approximate model counting tools.
Revised: (Baluta et al., 2019), proposes a framework to answer such probabilistic queries using approximate model counting tools.

Original: The main technical challenge is to perform efﬁcient approximate solution counting of a SAT formula that contains a B IN B NN as a subformula.
Revised: The main technical challenge is to perform efﬁcient approximate solution counting of a SAT formula that contains a B IN B NN as a subformula.

Original: Consider the property veriﬁcation formula above.
Revised: Consider the property veriﬁcation formula above.

Original: By approximately counting solutions, we obtain an estimate of the probability of a valid input leading to a property violation with a controllable and bounded error 2 .
Revised: By approximately counting solutions, we obtain an estimate of the probability of a valid input leading to a property violation with a controllable and bounded error 2 .

Original: Baluta et al.
Revised: Baluta et al.

Original: identiﬁed three applications of this framework in the security domain: robustness, trojan attacks, and fairness.
Revised: identiﬁed three applications of this framework in the security domain: robustness, trojan attacks, and fairness.

Original: Narodytska et al.
Revised: Narodytska et al.

Original: (2019) investigated how a similar type of quantitative reasoning can be used to assess the quality of ML explanations.
Revised: (2019) investigated how a similar type of quantitative reasoning can be used to assess the quality of ML explanations.

Original: Knowledge compilation engines.
Revised: Knowledge compilation engines.

Original: The compilation of BNNs is an interesting research direction that aims to compile a B IN B NN into a tractable structure, a logic graph-based representation of the formula (Shih et al., 2019; Choi et al., 2019) that supports a wide range of queries about the original
Revised: The compilation of BNNs is an interesting research direction that aims to compile a B IN B NN into a tractable structure, a logic graph-based representation of the formula (Shih et al., 2019; Choi et al., 2019) that supports a wide range of queries about the original

Original: BNN, including exact solution counting for probabilistic queries or generating logical explanations for network decisions.
Revised: BNN, including exact solution counting for probabilistic queries or generating logical explanations for network decisions.

Original: Interestingly, BNNs often admit succinct representations as the networks contain redundancies that can be eliminated by compilation.
Revised: Interestingly, BNNs often admit succinct representations as the networks contain redundancies that can be eliminated by compilation.

Original: For example, (Choi et al., 2019) reports large reduction in the representation size on some benchmarks.
Revised: For example, (Choi et al., 2019) reports large reduction in the representation size on some benchmarks.

Original: As can be seen from this overview, the success of these applications depends on the ability to reason efﬁciently about the underlying formula in the corresponding logical reasoner, e.g., a SAT solver, an approximate model counting method, or a knowledge compilation engine.
Revised: As can be seen from this overview, the success of these applications depends on the ability to reason efﬁciently about the underlying formula in the corresponding logical reasoner, e.g., a SAT solver, an approximate model counting method, or a knowledge compilation engine.

Original: On the one hand, BNNs are potentially easier to reason about compared to full-precision networks 3 .
Revised: On the one hand, BNNs are potentially easier to reason about compared to full-precision networks 3 .

Original: On the other hand, SAT is a hard combinatorial problem.
Revised: On the other hand, SAT is a hard combinatorial problem.

Original: Hence, developing efﬁcient decision procedures require exploiting the special structure of BNNs.
Revised: Hence, developing efﬁcient decision procedures require exploiting the special structure of BNNs.

Original:  A NALYSIS OF BNN’ S FROM THE SAT STANDPOINT
Revised:  A NALYSIS OF BNN’ S FROM THE SAT STANDPOINT

Original: In this section we analyze the properties of the formula B IN B NN from the perspective of a solver developer.
Revised: In this section we analyze the properties of the formula B IN B NN from the perspective of a solver developer.

Original: We discuss how these properties affect solver performance both positively and negatively.
Revised: We discuss how these properties affect solver performance both positively and negatively.

Original: We highlight parts of the BNN architecture that can be exploited to speed up logical reasoners.
Revised: We highlight parts of the BNN architecture that can be exploited to speed up logical reasoners.

Original: To structure our analysis, we consider BNN’s at three levels of granularity that naturally correspond to sub-formulas of B IN B NN .
Revised: To structure our analysis, we consider BNN’s at three levels of granularity that naturally correspond to sub-formulas of B IN B NN .

Original: (Narodytska et al., 2018; Cheng et al., 2018; Khalil et al., 2019) discussed and exploited properties of BNNs on the highest level of granularity, but we believe that there are more opportunities to take advantage of the structure if we look at all levels systematically.
Revised: (Narodytska et al., 2018; Cheng et al., 2018; Khalil et al., 2019) discussed and exploited properties of BNNs on the highest level of granularity, but we believe that there are more opportunities to take advantage of the structure if we look at all levels systematically.

Original: Neuron level We start with the level of an individual neuron.
Revised: Neuron level We start with the level of an individual neuron.

Original: Recall that the value of a neuron is determined by Equation 1.
Revised: Recall that the value of a neuron is determined by Equation 4.

Original: Let us examine it in more detail.
Revised: Let us examine it in more detail.

Original: We note that A is a dense matrix by design as all entries must be 1 or − 1 .
Revised: We note that A is a dense matrix by design as all entries must be 1 or − 1 .

Original: At the same time, the number of literals in Equation 1 is equal to the width of A for fully-connected layers or the ﬁlter size for convolutional layers.
Revised: At the same time, the number of literals in Equation 4 is equal to the width of A for fully-connected layers or the ﬁlter size for convolutional layers.

Original: Therefore, the high density of A leads to a large number of variables in the cardinality constraint of Equation 1.
Revised: Therefore, the high density of A leads to a large number of variables in the cardinality constraint of Equation 4.

Original: To make things worse, depending on the encoding, the number of auxiliary variables to encode Equation 1 depends on n i and h j , e.g., the number of auxiliary variables introduced is O ( n i h j ) for the sequential counters encoding, leading to large SAT representations.
Revised: To make things worse, depending on the encoding, the number of auxiliary variables to encode Equation 4 depends on n i and h j , e.g., the number of auxiliary variables introduced is O ( n i h j ) for the sequential counters encoding, leading to large SAT representations.

Original: Hence, dense matrices lead to large encodings.
Revised: Hence, dense matrices lead to large encodings.

Original: Next, consider the logical structure of Equation 1.
Revised: Next, consider the logical structure of Equation 4.

Original: Note that we have an equivalence relation between the value of the neuron and the cardinality constraint.
Revised: Note that we have an equivalence relation between the value of the neuron and the cardinality constraint.

Original: A key component of a SAT solver is the inference procedure, that at each point of the search infers values of unassigned variables.
Revised: A key component of a SAT solver is the inference procedure, that at each point of the search infers values of unassigned variables.

Original: Suppose Equation 1 produces the following constraint x 11 ⇔ (¯ x 00 + ¯ x 01 + x 02 ≥ 2) .
Revised: Suppose Equation 4 produces the following constraint x 11 ⇔ (¯ x 00 + ¯ x 01 + x 02 ≥ 2) .

Original: Moreover, suppose x 10 and x 02 have not been ﬁxed yet and x 00 = 1 .
Revised: Moreover, suppose x 10 and x 02 have not been ﬁxed yet and x 00 = 1 .

Original: Note that any setting of variables x 01 , x 02 can be extended to a valid assignment in this constraint.
Revised: Note that any setting of variables x 01 , x 02 can be extended to a valid assignment in this constraint.

Original: Hence, no inference is possible.
Revised: Hence, no inference is possible.

Original: In contrast, assume we only had cardinality constraints, i.e. ¯ x 00{+0 
+} + ¯ x 01 + x 02 ≥ 2 , then we can infer that x 01 = 0 and x 02 = 1 at this point.
Revised: In contrast, assume we only had cardinality constraints, i.e. ¯ x {+0 
+} + ¯ x 0 1 + x 0 2 ≥ 2 , then we can infer that x 0 1 = 0 and x 0 2 = 1 at this point.

Original: As the vast majority of constraints in the encoding contain the equivalence relation, this hinders the inference algorithm’s ability to ﬁnd implied variables, making the solver dive deep before a conﬂict occurs.
Revised: As the vast majority of constraints in the encoding contain the equivalence relation, this hinders the inference algorithm’s ability to ﬁnd implied variables, making the solver dive deep before a conﬂict occurs.

Original: Block level Next we go up one level of granularity and focus on the encoding of B LOCK .
Revised: Block level Next we go up one level of granularity and focus on the encoding of B LOCK .

Original: We note that a block is encoded by a set of constraints over the same variables.
Revised: We note that a block is encoded by a set of constraints over the same variables.

Original: We count over these variables therefore multiple times.
Revised: We count over these variables therefore multiple times.

Original: This observation hints at an opportunity to identify shared computations and exploit them to get more succinct encodings.
Revised: This observation hints at an opportunity to identify shared computations and exploit them to get more succinct encodings.

Original: For example, if we have two constraints ( x 00 − x 01 + x 02 ≥ 2) and ( − x 00 − x 01 + x 02 ≥ − 2) then we encode the partial sum − x 01 + x 02 for both constraints and this can be shared.
Revised: For example, if we have two constraints ( x 00 − x 01 + x 02 ≥ 2) and ( − x 00 − x 01 + x 02 ≥ − 2) then we encode the partial sum − x 01 + x 02 for both constraints and this can be shared.

Original: Our ability to share computations depends on the patterns in which 1 and −appear in the matrix A as these coefﬁcients control whether variables appear positively or negatively in each constraint.
Revised: Our ability to share computations depends on the patterns in which 1 and −appear in the matrix A as these coefﬁcients control whether variables appear positively or negatively in each constraint.

Original: As A is a full matrix and no patterns are enforced on occurrences of 1 and −between rows the amount of sharing is rather small for the standard BNN architecture.
Revised: As A is a full matrix and no patterns are enforced on occurrences of 1 and −between rows the amount of sharing is rather small for the standard BNN architecture.

Original: Network level Finally, we consider the entire BNN as a chain of blocks.
Revised: Network level Finally, we consider the entire BNN as a chain of blocks.

Original: As each block is encoded individually, B IN B NN is a conjunction of B IN B LOCK formulas that are loosely connected via variables x
Revised: As each block is encoded individually, B IN B NN is a conjunction of B IN B LOCK formulas that are loosely connected via variables x

Original: i , i = 1 , . . .
Revised: i , i = 1 , . . .

Original: , d − 1 .
Revised: , d − 1 .

Original: As discussed above, x i variables are a small fraction of the total number of variables.
Revised: As discussed above, x i variables are a small fraction of the total number of variables.

Original: Two factors affect solver performance.
Revised: Two factors affect solver performance.

Original: First, the block-wise structure of the B IN B NN formula suggests that formula decomposition can be exploited by the solver.
Revised: First, the block-wise structure of the B IN B NN formula suggests that formula decomposition can be exploited by the solver.

Original: Second, the B IN B NN formula effectively simulates the network function for all possible inputs.
Revised: Second, the B IN B NN formula effectively simulates the network function for all possible inputs.

Original: Additional constraints on inputs/outputs of the network, e.g., in the veriﬁcation problem in Section 3.2, are the main source of inconsistencies.
Revised: Additional constraints on inputs/outputs of the network, e.g., in the veriﬁcation problem in Section 3.2, are the main source of inconsistencies.

Original: Therefore, conﬂict-driven learning, which is the core of all SAT solvers, should perform directed search , starting from either the last or the ﬁrst blocks of the formula moving toward the ﬁrst (last) blocks as search progresses rather than jumping unguided between layers.
Revised: Therefore, conﬂict-driven learning, which is the core of all SAT solvers, should perform directed search , starting from either the last or the ﬁrst blocks of the formula moving toward the ﬁrst (last) blocks as search progresses rather than jumping unguided between layers.

Original: Such guidance might help to discover conﬂicts faster.
Revised: Such guidance might help to discover conﬂicts faster.

Original:  T OWARD SAT- FRIENDLY BNN S
Revised:  T OWARD SAT- FRIENDLY BNN S

Original: We discuss how alternative architectural design choices for BNNs can be used to take advantage of the identiﬁed properties to reason faster.
Revised: We discuss how alternative architectural design choices for BNNs can be used to take advantage of the identiﬁed properties to reason faster.

Original: We observe that, assuming a ﬁxed network architecture, our design options are limited.
Revised: We observe that, assuming a ﬁxed network architecture, our design options are limited.

Original: We therefore make an important assumption that we can change the architecture of the BNN and have access to the training procedure.
Revised: We therefore make an important assumption that we can change the architecture of the BNN and have access to the training procedure.

Original: While this is a strong assumption, we believe that BNN designers will be willing to incorporate changes into the training procedure as long as they do not cause accuracy loss.
Revised: While this is a strong assumption, we believe that BNN designers will be willing to incorporate changes into the training procedure as long as they do not cause accuracy loss.

Original: As before, we structure our solutions and observations using different levels of granularity.
Revised: As before, we structure our solutions and observations using different levels of granularity.

Original: Neuron level As discussed above, there are two properties of Equation 1 that need to be addressed.
Revised: Neuron level As discussed above, there are two properties of Equation 4 that need to be addressed.

Original: First issue is that A is a dense matrix by design of BNNs.
Revised: First issue is that A is a dense matrix by design of BNNs.

Original: To deal with this issue, we propose using a ternary quantization procedure instead of a binary quantization to learn sparse matrices for linear transformations.
Revised: To deal with this issue, we propose using a ternary quantization procedure instead of a binary quantization to learn sparse matrices for linear transformations.

Original: We recall that BNN is trained using two matrices: a full-precision matrix W i that is updated during the gradient decent and a binarized matrix
Revised: We recall that BNN is trained using two matrices: a full-precision matrix W i that is updated during the gradient decent and a binarized matrix

Original: A i , connected as follows A i = sign( W i ) .
Revised: A i , connected as follows A i = sign( W i ) .

Original: A i is used on the forward path and during inference.
Revised: A i is used on the forward path and during inference.

Original: The modiﬁcation we make is a simple twosided ternary quantization of A i during training:
Revised: The modiﬁcation we make is a simple twosided ternary quantization of A i during training:

Original: [Equation]
Revised: [Equation]

Original: where T is a parameter.
Revised: where T is a parameter.

Original: In principle, T can be a learnable parameter (Zhu et al., 2017).
Revised: In principle, T can be a learnable parameter (Zhu et al., 2017).

Original: However, in (Darabi et al., 2018), it was observed that weights of W are naturally grouped around points − 1 , 1 and 0 during training.
Revised: However, in (Darabi et al., 2018), it was observed that weights of W are naturally grouped around points − 1 , 1 and 0 during training.

Original: Therefore, we use a ﬁxed threshold T , obtained from the distribution of weights of a trained vanilla BNN.
Revised: Therefore, we use a ﬁxed threshold T , obtained from the distribution of weights of a trained vanilla BNN.

Original: Allowing zero weights in A requires some modiﬁcations in the original encoding of the last layer that we discuss in Appendix B.2.
Revised: Allowing zero weights in A requires some modiﬁcations in the original encoding of the last layer that we discuss in Appendix B.1.

Original: Consider how ternary quantization helps to reduce the encoding in our example.
Revised: Consider how ternary quantization helps to reduce the encoding in our example.

Original: Example 5.1.
Revised: Example 5.1.

Original: Suppose thatbefore quantization the ﬁrst row of A is [1 , − 1 , 1] and the corresponding  cardinality constraintto encode it is  ( x 00 − x 01 + x 02 ≥ 1 / 2)  (see Example B.1 and Example B.2 in
Revised: Consider Example 3.1. We recall that the ﬁrst row of A is [1 , − 1 , 1] and the corresponding reiﬁed cardinality constraint is x 10 ⇔ ( x 00 − x 01 + x 02 ≥ 1 / 2) , (see Example 3.2).

Original: Appendix for more details).

Original: Now we assume that after using ternary quantization the ﬁrst row of A is [1 , 0 , 0] .
Revised: Now we assume that after using ternary quantization the ﬁrst row of A is [1 , 0 , 0] .

Original: Then the last  cardinality constraint is simpliﬁed to  ( x 00 ≥ 1) .
Revised: Then the last reiﬁed cardinality constraint is simpliﬁed to x 1 0 ⇔ ( x 0 0 ≥ 1) , which is mush simpler constraint.

Original: Interestingly, our approach resembles the lottery ticket hypothesis training framework introduced by Frankle & Carbin (2019).
Revised: Interestingly, our approach resembles the lottery ticket hypothesis training framework introduced by Frankle & Carbin (2019).

Original: The main conceptual difference is that pruning and training are tightly coupled in our case as ternary quantization of weights is performed during every forward path.
Revised: The main conceptual difference is that pruning and training are tightly coupled in our case as ternary quantization of weights is performed during every forward path.

Original: In Frankle & Carbin (2019), training and pruning are two separate phases.
Revised: In Frankle & Carbin (2019), training and pruning are two separate phases 4 .

Original: The second issue at this level is that SAT solver inference is weak due to equivalence relations in
Revised: The second issue at this level is that SAT solver inference is weak due to equivalence relations in

Original: Equation 1.
Revised: Equation 4.

Original: Here we take the stabilization approach proposed in (Xiao et al., 2019) and adjust it to work for BNNs.
Revised: Here we take the stabilization approach proposed in (Xiao et al., 2019) and adjust it to work for BNNs.

Original: The idea is to perform bounds propagation of the input bounds through the network to estimate lower and upper bounds of each neuron in the network.
Revised: The idea is to perform bounds propagation of the input bounds through the network to estimate lower and upper bounds of each neuron in the network.

Original: We denote z ij an input of the sign function of a neuron x
Revised: We denote z j an input of the sign function of a neuron x

Original: i +1 j , x i +1 j = sign( z ij ) (Equation 5 deﬁnes z ij in Appendix B.1).
Revised: i +1 j , x i +1 j = sign( z j ) (Equation 1 deﬁnes z j ).

Original: If sign( ub ( z ij ))
Revised: If sign( ub ( z j ))

Original: =
Revised: =

Original: sign( lb ( z ij ))
Revised: sign( lb ( z j ))

Original: then the sign operator is stable in the sense that we know the value of x i +1 j , e.g. if both bounds are positive then x i +1 j = 1 , otherwise − 1 .
Revised: then the sign operator is stable in the sense that we know the value of x i +1 j , e.g. if both bounds are positive then x i +1 j = 1 , otherwise − 1 .

Original: For stable neurons, we can remove the equivalence relation and ﬁx x
Revised: For stable neurons, we can remove the equivalence relation and ﬁx x

Original: i +1 j simplifying the encoding (see Appendix C for more details on bounds propagation and an example).
Revised: i +1 j simplifying the encoding (see Appendix C for more details on bounds propagation and an example).

Original: Estimates of lower and upper bounds must be binarized as the bounds computation goes across B LOCK s. To mimic the computation ﬂow on the bounds computation path, we apply hard tanh before the sign operator: lb ( x i +1 j )
Revised: Estimates of lower and upper bounds must be binarized as the bounds computation goes across B LOCK s. To mimic the computation ﬂow on the bounds computation path, we apply hard tanh before the sign operator: lb ( x i +1 j )

Original: = sign( H ARD T ANH ( lb ( z ij ))) and ub ( x i +1 j )
Revised: = sign( H ARD T ANH ( lb ( z j ))) and ub ( x i +1 j )

Original: = sign( H ARD T ANH ( ub ( z ij ))) .
Revised: = sign( H ARD T ANH ( ub ( z j ))) .

Original: We must also take into account that parameters of the B N layer are learned in a different way.
Revised: We must also take into account that parameters of the B N layer are learned in a different way.

Original: Namely, µ i j and σ i j are computed from the batch of samples, while other parameters are learned via gradient descent (Ioffe & Szegedy, 2015).
Revised: Namely, µ ij and σ ij are computed from the batch of samples, while other parameters are learned via gradient descent (Ioffe & Szegedy, 2015b).

Original: When we compute bounds, we pass them through the B N layer, but we should exclude these fake inputs from the computation of µ ij and σ ij .
Revised: When we compute bounds, we pass them through the B N layer, but we should exclude these fake inputs from the computation of µ i j and σ i j .

Original: Finally, to achieve the effect of stabilization during training, for each neuron, we encourage ub ( z ij )) and lb ( z ij ) to take the same sign by adding an extra term to the loss function that approximates sign( ub ( z ij )) sign( lb ( z ij )) as − tanh(1 + ub ( z ij ) lb ( z ij ))
Revised: Finally, to achieve the effect of stabilization during training, for each neuron, we encourage ub ( z j )) and lb ( z j ) to take the same sign by adding an extra term to the loss function that approximates sign( ub ( z j )) sign( lb ( z j )) as − tanh(1 + ub ( z j ) lb ( z j ))

Original: (Xiao et al., 2019).
Revised: (Xiao et al., 2019).

Original: Block level We recall that the main issue we identiﬁed is that it is not obvious how to take advantage of shared partial computations in constraints of a block.
Revised: Block level We recall that the main issue we identiﬁed is that it is not obvious how to take advantage of shared partial computations in constraints of a block.

Original: One solution could be to enforce a pattern of zero and non-zero coefﬁcients for each matrix A .
Revised: One solution could be to enforce a pattern of zero and non-zero coefﬁcients for each matrix A .

Original: A recent and successful example of this is a butterﬂy matrix (Dao et al., 2019b;a).
Revised: A recent and successful example of this is a butterﬂy matrix (Dao et al., 2019b;a).

Original: This is a structured matrix with a predeﬁned sparsity pattern.
Revised: This is a structured matrix with a predeﬁned sparsity pattern.

Original: The original matrix is replaced with a composition of highly sparse matrices of a rigid structure that provides a very good approximation of the original matrix (Dao et al., 2019a).
Revised: The original matrix is replaced with a composition of highly sparse matrices of a rigid structure that provides a very good approximation of the original matrix (Dao et al., 2019a).

Original: This new matrix can potentially be used to create a natural decomposition of our block constraints.
Revised: This new matrix can potentially be used to create a natural decomposition of our block constraints.

Original: However, it is a matter of future research if butterﬂy matrices can be used alongside quantization.
Revised: However, it is a matter of future research if butterﬂy matrices can be used alongside quantization.

Original: Another solution that we can use is a different binary quantization, e.g. − 1 / 0 or 0 / 1 , for each row of the matrix instead of ternary quantization.
Revised: Another solution that we can use is a different binary quantization, e.g. − 1 / 0 or 0 / 1 , for each row of the matrix instead of ternary quantization.

Original: This will ensure that variables either occur in the negative/positive polarity or are absent in each constraint.
Revised: This will ensure that variables either occur in the negative/positive polarity or are absent in each constraint.

Original: Keeping the same polarity per constraint may foster shared computations.
Revised: Keeping the same polarity per constraint may foster shared computations.

Original: However, the distribution of zeroes is also important here to get a good level of sharing.
Revised: However, the distribution of zeroes is also important here to get a good level of sharing.

Original: Hence, we believe that using predeﬁned patterns might be more promising.
Revised: Hence, we believe that using predeﬁned patterns might be more promising.

Original: Network level Recall that we found two avenues for optimizing encoding at the network level: formula decomposition and directed search .
Revised: Network level Recall that we found two avenues for optimizing encoding at the network level: formula decomposition and directed search .

Original: In (Narodytska et al., 2018; Cheng et al., 2018), decomposability of B IN B NN was exploited in limited forms there the authors independently proposed using CEGAR-like search to reason about property veriﬁcation.
Revised: In (Narodytska et al., 2018; Cheng et al., 2018), decomposability of B IN B NN was exploited in limited forms there the authors independently proposed using CEGAR-like search to reason about property veriﬁcation.

Original: (Khalil et al., 2019) took advantage of both decomposition and directed search by proceeding from the last to the ﬁrst layer for a similar problem.
Revised: (Khalil et al., 2019) took advantage of both decomposition and directed search by proceeding from the last to the ﬁrst layer for a similar problem.

Original: However, while all these ideas appear to be helpful, there is no solver designed to incorporate such techniques.
Revised: However, while all these ideas appear to be helpful, there is no solver designed to incorporate such techniques.

Original: On the other hand, a number of veriﬁcation frameworks exploit similar domain properties for bounded model checking (Bradley, 2011; Gurﬁnkel et al., 2015).
Revised: On the other hand, a number of veriﬁcation frameworks exploit similar domain properties for bounded model checking (Bradley, 2011; Gurﬁnkel et al., 2015).

Original: So, it would be interesting to see if these solvers can be adapted for property veriﬁcation of ML models.
Revised: So, it would be interesting to see if these solvers can be adapted for property veriﬁcation of ML models.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We focus on the improvements we proposed for the neuron level of granularity.
Revised: We focus on the improvements we proposed for the neuron level of granularity.

Original: We use three datasets from (Narodytska et al., 2018;
Revised: We use three datasets from (Narodytska et al., 2018;

Original: Baluta et al., 2019; Khalil et al., 2019).
Revised: Baluta et al., 2019; Khalil et al., 2019).

Original: For each, we experiment with two tasks.
Revised: For each, we experiment with two tasks.

Original: First, we check if there is an untargeted adversarial attack (Goodfellow et al., 2015).
Revised: First, we check if there is an untargeted adversarial attack (Goodfellow et al., 2015).

Original: Second, we compute an approximate number of adversarial examples for a given image.
Revised: Second, we compute an approximate number of adversarial examples for a given image.

Original: We used 4 hidden layers and one output layer.
Revised: We used 4 hidden layers and one output layer.

Original: The input of the network is an image of size 28 × 28 .
Revised: The input of the network is an image of size 28 × 28 .

Original: The network has the following dimensions for all layers: [784 , 500 , 300 , 200 , 100 , 10] .
Revised: The network has the following dimensions for all layers: [784 , 500 , 300 , 200 , 100 , 10] .

Original: This gives 623 K parameters, which is 3 to 10 times bigger than the networks in (Narodytska et al., 2018; Baluta et al., 2019).
Revised: This gives 623 K parameters, which is 3 to 10 times bigger than the networks in (Narodytska et al., 2018; Baluta et al., 2019).

Original: We used a full-precision trained network to seed weights before binarized training (Alizadeh et al., 2019).
Revised: We used a full-precision trained network to seed weights before binarized training (Alizadeh et al., 2019).

Original: As the ﬁrst layer inputs are reals, we used an additional B N + sign layer after the  input layer to binarize inputs (see Appendix D.1).
Revised: As the ﬁrst layer inputs are reals, we used an additional B N + sign layer after the [Table] input layer to binarize inputs (see Appendix D.1).

Original: We used the PySAT tool (Ignatiev et al., 2018) to encode logical constraints to SAT and Glucose as a SAT solver (Audemard & Simon, 2018).
Revised: We used the PySAT tool (Ignatiev et al., 2018) to encode logical constraints to SAT and Glucose as a SAT solver (Audemard & Simon, 2018).

Original: Untargeted adversarial attacks.
Revised: Untargeted adversarial attacks.

Original: In this task, we look for a perturbation x (cid:48) of the given input x where | x − x (cid:48) | ≤ (cid:15) .
Revised: In this task, we look for a perturbation x (cid:48) of the given input x where | x − x (cid:48) | ≤ (cid:15) .

Original: We used ﬁve values of epsilon, (cid:15) ∈ { 1 , 3 , 5 , 10 , 15 , 20 } 4 .
Revised: We used ﬁve values of epsilon, (cid:15) ∈ { 1 , 3 , 5 , 10 , 15 , 20 } 5 .

Original: We train three groups of networks.
Revised: We train three groups of networks.

Original: First, we trained vanilla BNNs.
Revised: First, we trained vanilla BNNs.

Original: Then we trained two types of BNNs using our ideas from the neuron level.
Revised: Then we trained two types of BNNs using our ideas from the neuron level.

Original: We trained BNNs with the ternary quantization method (’Sparse’).
Revised: We trained BNNs with the ternary quantization method (’Sparse’).

Original: We used the value of T = 0 .
Revised: We used the value of T = 0 .

Original: 03 for M NIST BG and F ASHION and T = 0 .
Revised: 03 for M NIST BG and F ASHION and T = 0 .

Original: 04 for M NIST using 60%- 80% quantile depending on the dataset and the accuracy of the resulting network.
Revised: 04 for M NIST using 60%- 80% quantile depending on the dataset and the accuracy of the resulting network.

Original: Finally, we trained
Revised: Finally, we trained

Original: BNNs with stabilization of neurons (’Sparse+Stable’).
Revised: BNNs with stabilization of neurons (’Sparse+Stable’).

Original: We seeded initial weights of ’Sparse+Stable’ network with weights of the trained ’Sparse’ network as such a setup up gave better accuracy.
Revised: We seeded initial weights of ’Sparse+Stable’ network with weights of the trained ’Sparse’ network as such a setup up gave better accuracy.

Original: In addition, we investigate the effect of a popular L1 regularizer on ternary quantization during the training (’Sparse+L1’) and it stabilization version (’Sparse+L1+Stable’).
Revised: In addition, we investigate the effect of a popular L1 regularizer on ternary quantization during the training (’Sparse+L1’) and it stabilization version (’Sparse+L1+Stable’).

Original: Table 2 summarizes test accuracy results and the number of non-zero parameters per network for all trained networks and datasets.
Revised: Table 2 summarizes test accuracy results and the number of non-zero parameters per network for all trained networks and datasets.

Original: From Table 2, we can see that we can signiﬁcantly reduce the number of parameters using ternary quantization during the training (‘Sparse’).
Revised: From Table 2, we can see that we can signiﬁcantly reduce the number of parameters using ternary quantization during the training (‘Sparse’).

Original: Interestingly, we do[Table] not lose test accuracy compared to the vanilla BNNs in all cases.
Revised: Interestingly, we do not lose test accuracy compared to the vanilla BNNs in all cases.

Original: If we consider the effect of L regulazation then we see that the number of parameters reduces even further but the accuracy drops a bit as well: the more we reduce the number of non-zero parameters the more we lose in terms of the accuracy.
Revised: If we consider the effect of L regulazation then we see that the number of parameters reduces even further but the accuracy drops a bit as well: the more we reduce the number of non-zero parameters the more we lose in terms of the accuracy.

Original: Using additional stabilization can also hurt the accuracy a bit for both ‘Sparse’ and ‘Sparse+L1’ networks.
Revised: Using additional stabilization can also hurt the accuracy a bit for both ‘Sparse’ and ‘Sparse+L1’ networks.

Original: However, on average, we get about 40% of sign s stabilized.
Revised: However, on average, we get about 40% of sign s stabilized.

Original: Figure 1 shows the performance of a SAT solver on three datasets.
Revised: Figure 1 shows the performance of a SAT solver on three datasets.

Original: Results are averaged overbenchmarks.
Revised: Results are averaged overbenchmarks.

Original: The average size of the encoding for all datasets and (cid:15) = 5 are shown in Table 3.
Revised: The average size of the encoding for all datasets and (cid:15) = 5 are shown in Table 4.

Original: Note that the size of the encoding is 50x smaller compared to millions of variables and classes reported in Narodytska et al.
Revised: Note that the size of the encoding is 50x smaller compared to millions of variables and classes reported in Narodytska et al.

Original: (2018) for M NIST and M NIST BG for a much smaller network and the same (cid:15) .
Revised: (2018) for M NIST and M NIST BG for a much smaller network and the same (cid:15) .

Original: Another interesting observation is that stabilization helps a lot to reduce the size of the encoding.
Revised: Another interesting observation is that stabilization helps a lot to reduce the size of the encoding.

Original: If the PySAT tool takes more than 10G of memory to generate a SAT encoding then we terminate the generation process.
Revised: If the PySAT tool takes more than 10G of memory to generate a SAT encoding then we terminate the generation process.

Original: If a solver time/memory outs on more than 70% of benchmarks we do not plot these results.
Revised: If a solver time/memory outs on more than 70% of benchmarks we do not plot these results.

Original: Therefore, we do not have a plot for vanilla BNNs in these datasets as we either had a timeout or (mostly) memory out exceptions.
Revised: Therefore, we do not have a plot for vanilla BNNs in these datasets as we either had a timeout or (mostly) memory out exceptions.

Original: Each plot shows the average time to solve the untargeted adversarial attack problem for each value of (cid:15) .
Revised: Each plot shows the average time to solve the untargeted adversarial attack problem for each value of (cid:15) .

Revised: In addition, Figure 3 shows the number of solved problems for each (cid:15) value within the timeout (100 sec) by the SAT solver.

Original: In all except one case, at least 95% of benchmarks were solved within the 100 sec timeout(See Appendix D.2, Figure 4 for details on the number of solved instances.).
Revised: As can be seen from these plots, in all except one case, at least 95% of benchmarks were solved within the 100 sec timeout

Original: The plots show that ternary quantization greatly improves performance on three datasets as we were not even able to solve any benchmarks for the vanilla case.
Revised: The plots show that ternary quantization greatly improves performance on three datasets as we were not even able to solve any benchmarks for the vanilla case.

Original: Using the ‘L1’ regularizer slightly improves performance in two datasets and helps a lot on M NIST .
Revised: Using the ‘L1’ regularizer slightly improves performance in two datasets and helps a lot on M NIST .

Original: Finally, the stabilization of sign s method consistently improves performance of a SAT solver across datasets for both ‘Sparse’ and ‘Sparse+L1’ networks, e.g. we get from 3x to 6x speedup due to stabilization.
Revised: Finally, the stabilization of sign s method consistently improves performance of a SAT solver across datasets for both ‘Sparse’ and ‘Sparse+L1’ networks, e.g. we get from 3x to 6x speedup due to stabilization.

Original: Figure 2 shows the number of successful attacks as it is an interesting measure to understand properties of a network.
Revised: Figure 2 shows the number of successful attacks as it is an interesting measure to understand properties of a network.

Original: Each plot shows the ratio of successful adversarial attacks for each value of (cid:15) .
Revised: Each plot shows the ratio of successful adversarial attacks for each value of (cid:15) .

Original: Naturally, this ration plateaus at 1 as the value of perturbation (cid:15) grows.
Revised: Naturally, this ration plateaus at 1 as the value of perturbation (cid:15) grows.

Original: The results are mixed here.
Revised: The results are mixed here.

Original: In some cases, using ’L1’ does not signiﬁcantly change the vulnerability of the network compare to ‘Sparse’ network, e.g. F ASHION and M NIST BG, while it does help for M NIST .
Revised: In some cases, using ’L1’ does not signiﬁcantly change the vulnerability of the network compare to ‘Sparse’ network, e.g. F ASHION and M NIST BG, while it does help for M NIST .

Original: Interestingly, stabilization of sign s can lead to increase in vulnarabilities if we compare a network with and without stabilization.
Revised: Interestingly, stabilization of sign s can lead to increase in vulnarabilities if we compare a network with and with- out stabilization.

Original: Consider for example, M NIST with (cid:15) = 5 .
Revised: Consider for example, M NIST with (cid:15) = 5 .

Original: Only 13 out of 100 benchmarks were successfully attacked for ‘Sparse+L1’ while 43 were attacked for ‘Sparse+L1+Stable’.
Revised: Only 13 out of 100 benchmarks were successfully attacked for ‘Sparse+L1’ while 43 were attacked for ‘Sparse+L1+Stable’.

Revised: Finally, Table 3 shows original (the top row) and perturbed (the bottom row) samples of images from M NIST BG dataset for the following values of (cid:15)

Revised: = 3 , 5 , 15 , 20 (from left to right).

Revised: As can be seen from these sample, if (cid:15) = 3 then original and perturbed images are identical.

Revised: For (cid:15) = 20 (last column), perturbations are visible.

Original: Likelihood of adversarial examples.
Revised: Likelihood of adversarial examples.

Original: Next we estimate the probability of a perturbation to be an adversarial example.
Revised: Next we estimate the probability of a perturbation to be an adversarial example.

Original: We reproduce a setup from (Baluta et al., 2019) where an approximate model[Table] counting, ApproxMC3 , was used to perform such estimate.
Revised: We reproduce a setup from (Baluta et al., 2019) where an approximate model counting, ApproxMC3 , was used to perform such estimate.

Original: The main goal of this experiment is to demonstrate that our approach allows approximate model counting techniques to scale to larger networks.
Revised: The main goal of this experiment is to demonstrate that our approach allows approximate model counting techniques to scale to larger networks.

Original: We focus on robustness as above, so SAT formulas check for existence of untargeted adversarial attacks.
Revised: We focus on robustness as above, so SAT formulas check for existence of untargeted adversarial attacks.

Original: Hence, each solution corresponds to an adversarial attack.
Revised: Hence, each solution corresponds to an adversarial attack.

Original: We perform model counting on binarized inputs (see Appendix D.1) with a timeout of 600 sec.
Revised: We perform model counting on binarized inputs (see Appendix D.1) with a timeout of 600 sec.

Original: We use default parameters of ApproxMC3 .
Revised: We use default parameters of ApproxMC3 .

Original: Table 3 summarizes results.
Revised: Table 4 summarizes results.

Original: We report the mean value of the solution count estimate divided by the total number of solutions (P(adv)) and the percentage of benchmarks solved by ApproxMC3 out of all benchmarks that we know are susceptible to adversarial perturbation (#solved).
Revised: We report the mean value of the solution count estimate divided by the total number of solutions (P(adv)) and the percentage of benchmarks solved by ApproxMC3 out of all benchmarks that we know are susceptible to adversarial perturbation (#solved).

Original: On average, we can solve 80% of benchmarks within the timeout.
Revised: On average, we can solve 80% of benchmarks within the timeout.

Original: This contrasts with results reported in (Baluta et al., 2019), where it took 24 hours to perform approximate model counting for a BNN with about 50K parameters and a large fraction of benchmarks were not solved.
Revised: This contrasts with results reported in (Baluta et al., 2019), where it took 24 hours to perform approximate model counting for a BNN with about 50K parameters and a large fraction of benchmarks were not solved.

Original: Table 3 shows that the probability of adversarial attack is small for two out of three datasets.
Revised: Table 4 shows that the probability of adversarial attack is small for two out of three datasets.

Revised: Another observation is sparsiﬁcation and stabilization do not signiﬁcantly change the probability of an input to be adversarial.

Revised: [Table]

Original: A ILP solver performance.
Revised: A ILP solver performance.

Original: For completeness of the evaluation, we report results of an ILP solver on the same benchmarks for the untargeted adversarial attacks.
Revised: For completeness of the evaluation, we report results of an ILP solver on the same benchmarks for the untargeted adversarial attacks.

Original: The second experiment (to estimate the probability of a perturbation to be an adversarial example) is not currently possible for ILP formulations as we are not aware of an efﬁcient approximate model counting solver for ILPs.
Revised: The second experiment (to estimate the probability of a perturbation to be an adversarial example) is not currently possible for ILP formulations as we are not aware of an efﬁcient approximate model counting solver for ILPs.

Original: We used the CPLEX solver with default conﬁguration.
Revised: We used the CPLEX solver with default conﬁguration.

Original: Figure 3 shows results.
Revised: Figure 4 shows results.

Original: As before Figureshows the performance of a ILP solver on the untargeted adversarial example task for each value of (cid:15) .
Revised: As before Figureshows the performance of a ILP solver on the untargeted adversarial example task for each value of (cid:15) .

Original: Results are averaged over 100 benchmarks.
Revised: Results are averaged over 100 benchmarks.

Original: We observe similar patterns as for ILP solvers: sparsiﬁcation and stabilization are mostly helpful for all datasets as we cannot solve most of the benchmarks if we do not apply these techniques.
Revised: We observe similar patterns as for ILP solvers: sparsiﬁcation and stabilization are mostly helpful for all datasets as we cannot solve most of the benchmarks if we do not apply these techniques.

Original: As can be seen from the graph, the ILP solver exhibits a bell shape in two datasets, M NIST and F ASHION : if (cid:15) is small or large then the instance is easier to solve.
Revised: As can be seen from the graph, the ILP solver exhibits a bell shape in two datasets, M NIST and F ASHION : if (cid:15) is small or large then the instance is easier to solve.

Original: (cid:15) values in the middle are the hardest to solve for an ILP solver.
Revised: (cid:15) values in the middle are the hardest to solve for an ILP solver.

Original: We have not observed this bell shape for SAT solvers.
Revised: We have not observed this bell shape for SAT solvers.

Original: Overall, if we compare performance of SAT and ILP solvers, the SAT solver is more efﬁcient and solves more benchmarks (see Figure 5 for the number of solved benchmarks in Appendix D.3).
Revised: Overall, if we compare performance of SAT and ILP solvers, the SAT solver is more efﬁcient and solves more benchmarks (see Figure 5 for the number of solved benchmarks in Appendix D.2).

Original: In summary, our results show that a signiﬁcant speed up can be obtained if we design an architecture of BNNs and its training procedure taking into account potential bottlenecks of SAT solvers.
Revised: In summary, our results show that a signiﬁcant speed up can be obtained if we design an architecture of BNNs and its training procedure taking into account potential bottlenecks of SAT solvers.

Original: We achieved 10-20x speed 
Revised: We achieved 10-20x speed up in veriﬁcation and in probabilistic queries compared to state of the art results in (Narodytska et al., 2018; Baluta et al., 2019).

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work, we analyze architectural design choices of BNNs and discuss how they affect the performance of logic-based reasoners, focusing on the individual neuron and block levels.
Revised: In this work, we analyze architectural design choices of BNNs and discuss how they affect the performance of logic-based reasoners, focusing on the individual neuron and block levels.

Original: We demonstrated how to train BNNs so that the resulting network is easier to verify for logic-based engines, like SAT solver or approximate model counting solvers.
Revised: We demonstrated how to train BNNs so that the resulting network is easier to verify for logic-based engines, like SAT solver or approximate model counting solvers.

Original: A B ACKGROUND . A DDITIONAL DETAILS .
Revised: A B ACKGROUND . A DDITIONAL DETAILS .

Original: Model counting.
Revised: Model counting.

Original: The problem of model counting is to calculate the number satisfying assignments or models of a given SAT formula.
Revised: The problem of model counting is to calculate the number satisfying assignments or models of a given SAT formula.

Original: This problem is complete for the complexity class # P
Revised: This problem is complete for the complexity class # P

Original: (Valiant, 1979), which contains the entire polynomial hierarchy.
Revised: (Valiant, 1979), which contains the entire polynomial hierarchy.

Original: Despite the high complexity, a number of tools for model counting have been developed (Sang et al., 2005; Thurley, 2006; Muise et al., 2012; Lagniez & Marquis, 2017), which were shown to work well on certain benchmarks arising in practice.
Revised: Despite the high complexity, a number of tools for model counting have been developed (Sang et al., 2005; Thurley, 2006; Muise et al., 2012; Lagniez & Marquis, 2017), which were shown to work well on certain benchmarks arising in practice.

Original: However, for many applications, obtaining exact counts is not necessary and a good approximation often sufﬁces, especially when it leads to better scalability.
Revised: However, for many applications, obtaining exact counts is not necessary and a good approximation often sufﬁces, especially when it leads to better scalability.

Original: These requirements have led to the emergence of approximate counting approaches (Ermon et al., 2013; Chakraborty et al., 2013; Soos & Meel, 2019) which employ universal hash functions along with specialized SAT solvers (Soos et al., 2009) for balancing accuracy and scalability.
Revised: These requirements have led to the emergence of approximate counting approaches (Ermon et al., 2013; Chakraborty et al., 2013; Soos & Meel, 2019) which employ universal hash functions along with specialized SAT solvers (Soos et al., 2009) for balancing accuracy and scalability.

Original: The most successful approximate counters provide Probabilistically Approximately Correct (PAC)-style guarantees (Valiant, 1984) on accuracy, while scaling to formulas with thousands of variables in practice.
Revised: The most successful approximate counters provide Probabilistically Approximately Correct (PAC)-style guarantees (Valiant, 1984) on accuracy, while scaling to formulas with thousands of variables in practice.

Original: In our experiments, we use a state-of-the-art tool called ApproxMC3 (Soos & Meel, 2019) which has been shown to scale much better than other exact and approximate counters.
Revised: In our experiments, we use a state-of-the-art tool called ApproxMC3 (Soos & Meel, 2019) which has been shown to scale much better than other exact and approximate counters.

Original: A key advantage of ApproxMC3 is that it supports projected model counting, i.e. counting over a subset of the variables in the given formula, which is essential for our purposes as we count over variables that represent inputs of the network.
Revised: A key advantage of ApproxMC3 is that it supports projected model counting, i.e. counting over a subset of the variables in the given formula, which is essential for our purposes as we count over variables that represent inputs of the network.

Original: Sequential counters encoding.
Revised: Sequential counters encoding.

Original: We recall the deﬁnition of sequential counters encoding from (Sinz, 2005a;b) that are used to encode cardinality constraints.
Revised: We recall the deﬁnition of sequential counters encoding from (Sinz, 2005a;b) that are used to encode cardinality constraints.

Original: Consider a cardinality constraint: (cid:80) mi =1 l i ≥ k , where l i ∈ { 0 , 1 } is a Boolean variable, x i , or its negation, ¯ x i , and k is a constant.
Revised: Consider a cardinality constraint: (cid:80) mi =1 l i ≥ k , where l i ∈ { 0 , 1 } is a Boolean variable, x i , or its negation, ¯ x i , and k is a constant.

Original: Then the sequential counter encoding, can be written as the following Boolean formula using auxiliary Boolean variables r ( j,k ) :
Revised: Then the sequential counter encoding, can be written as the following Boolean formula using auxiliary Boolean variables r ( j,k ) :

Original: [Equation]
Revised: [Equation]

Original: where i ∈ { 2 , . .
Revised: where i ∈ { 2 , . .

Original: , m } .
Revised: , m } .

Original: Essentially, this encoding computes the cumulative sums (cid:80) ki =1 l i and the computation is performed in unary, i.e., the Boolean variable r ( j,k ) is true iff (cid:80) ji =1 l i ≥ k .
Revised: Essentially, this encoding computes the cumulative sums (cid:80) ki =1 l i and the computation is performed in unary, i.e., the Boolean variable r ( j,k ) is true iff (cid:80) ji =1 l i ≥ k .

Original: In particular, r ( m,k ) encodes whether (cid:80) mi =1 l i is greater than or equal to k .
Revised: In particular, r ( m,k ) encodes whether (cid:80) mi =1 l i is greater than or equal to k .

Original: B P ROPOSITION ENCODING OF BNN S . B. SAT ENCODING OF BNN S

Original: We consider how to generate a logical encoding of a BNN (Narodytska et al., 2018).

Original: Consider a single block B LOCK i , which applies three transformations: L IN , B N and B IN to the input vector.

Original: We will use the following running example to demonstrate the encoding.

Original: Example B.1.

Original: Consider a block with three inputs and two outputs.

Original: We deﬁne transformation parameters as follows:

Original: L IN : A = [1 , − 1 , 1; − 1 , − 1 , 1] , b

Original: =

Original: [0 , 0] and B N : α = [0 . 1 , 0 . 1] , µ = [1 , − 1] , σ = [1 , 1] , γ = [0 . 1 , 0 . 2] .

Original: We encode these transformations as a system of logic constraints.

Original: We introduce a vector of binary variables x i ∈ {− 1 , 1 } n i to encode inputs and x i +1 ∈ {− 1 , 1 } n i +1 to encode outputs of B LOCK i .

Original: Since transformations are applied sequentially, we can encode the block as their composition:

Original: [Equation]

Original: where a i j,t and b i j are parameters of L IN , α i j , σ i j , µ i j and γ i j are parameters of B N .

Original: To obtain a SAT encoding, we need to eliminate binary variables as a SAT formula is deﬁned over

Original: Boolean variables.

Original: We introduce Boolean variables x t ∈ { 0 , 1 } , t = 1 , . . . , n i and relate them to the corresponding binary variables x t : x t = 2 x t − 1 , t = 1 , . . . , n i .

Original: We get that if x t = 0 then x t = − 1 , otherwise x t = 1 .

Original: We can now rewrite the last expression as x i +1 j = sign (cid:16) α ij

Original: [Equation]

Original: ij (

Original: [Equation]

Original: n i

Original: [Equation]

Original: =1 a ij

Original: [Equation]

Original: x it

Original: [Equation]

Original: n

Original: [Equation]

Original: (cid:17)

Original: [Equation]

Original: j = 1 . . .

Original: n i +1 .

Original: We denote C j = ( − (cid:80) n i t =1 a ij,t + b ij − µ ij ) , as these are just constants.

Original: We get

Original: [Equation]

Original: i

Original: [Equation]

Original: Example B.2.

Original: Consider how the encoding works on Example B.1.

Original: We introduce three binary variables to encode inputs x 0 i , i = 0 , 1 , 2 , and two binary variables to encode outputs x 10 and x 11 .

Original: Inputs and outputs are connected as follows:

Original: [Equation]

Original: Next we perform variable replacement as explained above using the relation x t = 2 x t − 1 .

Original: Note that α 01 and α 02 are both positive so we get:

Original: [Equation]

Original: 2 ≥ 0) ⇔ ( − x 00 − x 01 + x 02 ≥ − 4 / 2) .

Original: The ﬁnal step is to encode reiﬁed cardinalities using Boolean variables instead of binary (integer) variables in Equation 6.

Original: We recall the following tautology for a Boolean variable x t +¯ x t = 1 , where ¯ x t is a negation of x t .

Original: As all coefﬁcients a ijt ∈ {− 1 , 1 } , Equation 6 contains linear constraints with unary coefﬁcients (see the last transformation in Example B.2).

Original: We denote N j = |{ a ijt < 0 , t = 1 , . . .

Original: , n i +1 }| the number of negative coefﬁcients in the j th row and assume α ij (cid:54) = 0 .

Original: [Equation]

Original: where l it =

Original: [Equation]

Original: x

Original: [Equation]

Original: We recall that a constraint of type Equation 7 is a reiﬁed cardinality constraint that can be translated into Boolean formulae (Sinz, 2005b).

Original: We refer to a SAT encoding of B LOCK as

Original: B IN B LOCK i ( x i , x i +1 ) .

Original: Similarly, we can encode the O UTPUT block that we show in details below as out method needs to make modiﬁcations to the original encoding.

Original: We can now represent the entire network as a Boolean formula: B IN B NN ( x , o ) ≡ (cid:86) d − 1 i =1 B IN B LOCK i ( x i , x i +1 )

Original: ∧

Original: B IN O UTPUT ( x d , o ) , where B IN B LOCK i ( x i , x i +1 )

Original: encodes the i th block B LOCK i with input x i and output x i +1 , i ∈ [0 , d − 1] , B IN O UTPUT ( x d , o ) is a Boolean encoding of the last layer.

Original:  B. E NCODING OF THE O UTPUT LAYER AND ITS ADJUSTMENT FOR TERNARY QUANTIZATION
Revised: B P ROPOSITION ENCODING OF BNN S . B. E NCODING OF THE O UTPUT LAYER AND ITS ADJUSTMENT FOR TERNARY QUANTIZATION

Original: We recall that the O UTPUT block is just a linear transformation: o =
Revised: We recall that the O UTPUT block is just a linear transformation: o =

Original: Cx d − 1 + q .
Revised: Cx d − 1 + q .

Original: Example B.3.
Revised: Example B.1.

Original: We continue Example B.1.
Revised: We continue Example 3.1.

Original: Suppose our single block is followed by an output block with two inputs and two outputs.
Revised: Suppose our single block is followed by an output block with two inputs and two outputs.

Original: We deﬁne transformation parameters as follows.
Revised: We deﬁne transformation parameters as follows.

Original: [Equation]
Revised: [Equation]

Original: Hence, we can write the following system of linear constraints over {− 1 , 1 } variables.
Revised: Hence, we can write the following system of linear constraints over {− 1 , 1 } variables.

Original: [Equation]
Revised: [Equation]

Original: As before, we perform variable replacement to obtain constraints over Boolean variables only.
Revised: As before, we perform variable replacement to obtain constraints over Boolean variables only.

Original: [Equation]
Revised: [Equation]

Original: After rearrangements, we get
Revised: After rearrangements, we get

Original: [Equation]
Revised: [Equation]

Original: Usually, we are not interested in absolute values of o .
Revised: Usually, we are not interested in absolute values of o .

Original: Rather, we need to know the index of the maximum element of o , which corresponds to the top prediction in the classiﬁcation task.
Revised: Rather, we need to know the index of the maximum element of o , which corresponds to the top prediction in the classiﬁcation task.

Original: Hence, we need to encode the ordering relation over variables o .
Revised: Hence, we need to encode the ordering relation over variables o .

Original: To do so, we use an additional set of Boolean variables d ij , i = 1 , . .
Revised: To do so, we use an additional set of Boolean variables d ij , i = 1 , . .

Original: . , p, j = i +1 , . .
Revised: . , p, j = i +1 , . .

Original: , p .
Revised: , p .

Original: Applying variable replacements and rearranging terms, we get:
Revised: Applying variable replacements and rearranging terms, we get:

Original: [Equation]
Revised: [Equation]

Original: we denote Rij =
Revised: we denote R

Original: ( p (cid:88) t =

Original: [Equation]

Original: p

Original: [Equation]
Revised: [Equation]

Original: c jt + q j ) , so we get
Revised: c jt + q j ) , so we get

Original: [Equation]
Revised: [Equation]

Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: Finally, we note that c
Revised: we note that c

Original: it , c jt ∈ {− 1 , 1 } are applied to the same variable x dt .
Revised: it , c jt ∈ {− 1 , 1 } are applied to the same variable x dt .

Original: Hence, depending on the sign of these coefﬁcients, the difference | c it − c
Revised: Hence, depending on the sign of these coefﬁcients, the difference | c it − c

Original: jt | is either 0 or 2 .
Revised: jt | is either 0 or 2 .

Original: So we can rewrite:
Revised: So we can rewrite:

Original: [Equation]
Revised: [Equation]

Original: Example B.4.
Revised: Example B.2.

Original: Coming back to our Example B.3, as we have two variables, we introduce one
Revised: Coming back to our Example B.1, as we have two variables, we introduce one

Original: Boolean variable d 01 to signal the top prediction:
Revised: Boolean variable d 01 to signal the top prediction:

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal observation is that if C is a sparse matrix then we lose the property that the difference c | it − c
Revised: The ﬁnal observation is that if C is a sparse matrix then we lose the property that the difference c | it − c

Original: jt | is either 0 or 2 as c it , c jt ∈ {− 1 , 0 , 1 } .
Revised: jt | is either 0 or 2 as c it , c jt ∈ {− 1 , 0 , 1 } .

Original: Hence, the last division by two (Equation 14) is not possible.
Revised: Hence, the last division by two (Equation 13) is not possible.

Original: However, we take advantage of the fact that coefﬁcients are small, | c it − c jt | ≤ 2 .
Revised: However, we take advantage of the fact that coefﬁcients are small, | c it − c jt | ≤ 2 .

Original: So, we can duplicate variables to model non-unary coefﬁcients, for example.
Revised: So, we can duplicate variables to model non-unary coefﬁcients, for example.

Original: Example B.5.
Revised: Example B.3.

Original: Suppose, we get the following matrix C due to ternary quantization:
Revised: Suppose, we get the following matrix C due to ternary quantization:

Original: [Equation]
Revised: [Equation]

Original: Then we have the following ordering constraint:
Revised: Then we have the following ordering constraint:

Original: [Equation]
Revised: [Equation]

Original: We can introduce an extra variable x 1 (cid:48) 0 and replace 2 x 10 + x 11 with x 10 + x 1 (cid:48) 

Revised: We can introduce an extra variable x 1 (cid:48) 0 and replace 2 x 10 + x 11 with x 10 + x 1 (cid:48)

Original: + x 11 and an equality constraint x 10 = x 1 (cid:48) 0 .
Revised: + x 11 and an equality constraint x 10 = x 1 (cid:48) 0 .

Original: C B OUNDS PROPAGATION
Revised: C B OUNDS PROPAGATION

Original: Let us split B LOCK computation into two parts that correspond to L IN and B N layers: y ij = (cid:80) n i t =1 a ij,t x it + b ij and z ij = α ij σ ij ( y ij − µ ij ) + γ ij to demonstrate computations of lower and upper bounds on y ij ( z ij are computed similarly).
Revised: Let us split B LOCK computation into two parts that correspond to L IN and B N layers: y j = (cid:80) n i t =1 a ij,t x it + b ij and z j = α ij σ ij ( y j − µ ij ) + γ ij to demonstrate computations of lower and upper bounds on y j ( z j are computed similarly).

Original: We get that
Revised: We get that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: t =
Revised: y j

Original: [Equation]
Revised: [Equation]

Original: a
Revised: =

Original: [Equation]
Revised: [Equation]

Original: t
Revised: n

Original: [Equation]
Revised: [Equation]

Original: n i

Original: [Equation]

Original: t =
Revised: t =

Original: [Equation]

Original: a

Original: [Equation]
Revised: [Equation]

Original: t

Original: [Equation]

Original: a

Original: [Equation]

Original: Example C.1.
Revised: Example C.1.

Original: Consider our example for B LOCK and the second neuron computation:
Revised: Consider our example for B LOCK and the second neuron computation:

Original: [Equation]
Revised: [Equation]

Original: Suppose we derive that for all possible samples, ub ( x 00 )
Revised: Suppose we derive that for all possible samples, ub ( x 00 )

Original: = ub ( x 01 )
Revised: = ub ( x 01 )

Original: =
Revised: =

Original: − 1 .
Revised: − 1 .

Original: In this case, we have lb ( x 00 )
Revised: In this case, we have lb ( x 00 )

Original: = lb ( x 01 )
Revised: = lb ( x 01 )

Original: = − 1 as all activations are in {− 1 , 1 } .
Revised: = − 1 as all activations are in {− 1 , 1 } .

Original: We can compute bounds on the all variables y and z .
Revised: We can compute bounds on the all variables y and z .

Original: [Equation]
Revised: [Equation]

Revised: ×

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: ub ( z 1 ) = 0 .

Revised: 1 × ub ( y 1 + 1) + 0 .

Revised: 2 = 0 .

Original: So, we get that sign( lb ( z 11 ))
Revised: So, we get that sign( lb ( z 1 ))

Original: = sign( ub ( z 11 )) , so we know that x 11 = [-
-] and we can use this information to reduce the encodings.
Revised: = sign( ub ( z 1 )) , so we know that x 11 = [-
-]1 and we can use this information to reduce the encodings.

Original: D E XPERIMENTAL EVALUATION . D. B INARIZATION OF INPUTS .
Revised: D E XPERIMENTAL EVALUATION . D. B INARIZATION OF INPUTS .

Original: As we discussed in Section 6, we introduce an additional block before the ﬁrst internal layer that mimics binarization of inputs.
Revised: As we discussed in Section 6, we introduce an additional block before the ﬁrst internal layer that mimics binarization of inputs.

Original: This layer constraints B N and sign operators only (hard tanh is used during the training).
Revised: This layer constraints B N and sign operators only (hard tanh is used during the training).

Original: Consider how this layer works.
Revised: Consider how this layer works.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: First, we consider two extreme values of the expression that is an input of sign w.r.t.
Revised: First, we consider two extreme values of the expression that is an input of sign w.r.t.

Original: (cid:15) .
Revised: (cid:15) .

Original: Suppose α t ≥ 0 ( α t < 0 is analogous).
Revised: Suppose α t ≥ 0 ( α t < 0 is analogous).

Original: [Equation]
Revised: [Equation]

Original: If min t ≥ 0 then we know that x 0 t = 1 .
Revised: If min t ≥ 0 then we know that x 0 t = 1 .

Original: If max j < 0 then we know that x 0 t = 0 .
Revised: If max j < 0 then we know that x 0 t = 0 .

Original: In other cases, x 0 t ∈ { 0 , 1 } . Consider the reverse transformation.
Revised: In other cases, x 0 t ∈ { 0 , 1 } . Consider the reverse transformation.

Original: If x 0 j = 0 is a solution of a problem produced by the SAT solver then we can map back to x t = min t .
Revised: If x 0 j = 0 is a solution of a problem produced by the SAT solver then we can map back to x t = min t .

Original: Similarly, if x 0 t = 1 then x t = max t .
Revised: Similarly, if x 0 t = 1 then x t = max t .

Original: As each transformation in Equation 18–Equation 19 is performed on a single input, we can build a valid input from a solution over Boolean variables.
Revised: As each transformation in Equation 17–Equation 18 is performed on a single input, we can build a valid input from a solution over Boolean variables.

Original: D. E XPERIMENTAL EVALUATION USING SAT SOLVERS . 
Revised: D. E XPERIMENTAL EVALUATION USING ILP SOLVERS . M ISSING

Original: Number of solved benchmarks.

Original: Figure 4 shows the number of solved problems for each (cid:15) value within the timeout by the SAT solver.

Original: As can be seen from these plots, in all except one case, at least 95% of benchmarks were solved with the 100 sec timeout.

Original: Examples of perturbed images. Table 4 shows original (the top row) and perturbed (the bottom row) samples of images for (cid:15) = 3 , 5 , 15 , 20 (from left to right). As can be seen from these sample, if (cid:15) = 3 then original and perturbed images are identical.
Revised: FIGURE .

Original: For (cid:15) = 20 (last column), perturbations are visible.

Original: D. E XPERIMENTAL EVALUATION USING ILP SOLVERS . M ISSING FIGURE .

Original: Figure 5 shows the number of solved problems for each (cid:15) value within the timeout.
Revised: Figure 5 shows the number of solved problems for each (cid:15) value within the timeout.