Original: Learning Symmetric Rules with SATNet
Revised: Learning Symmetric Rules with SATNet

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: SATNet is a differentiable constraint solver with a custom backpropagation algorithm, which can be used as a layer in a deep-learning system.
Revised: SATNet is a differentiable constraint solver with a custom backpropagation algorithm, which can be used as a layer in a deep-learning system.

Original: It is a promisingproposal for bridging deep learning and logical reasoning.
Revised: It is a promisingproposal for bridging deep learning and logical reasoning.

Original: In fact, SATNet hasbeen successfully applied to learn, among others, the rules of a complex logicalpuzzle, such as Sudoku, just from input and output pairs where inputs are givenas images.
Revised: In fact, SATNet hasbeen successfully applied to learn, among others, the rules of a complex logicalpuzzle, such as Sudoku, just from input and output pairs where inputs are givenas images.

Original: In this paper, we show how to improve the learning of SATNet byexploiting symmetries in the target rules of a given but unknown logical puzzleor more generally a logical formula.
Revised: In this paper, we show how to improve the learning of SATNet byexploiting symmetries in the target rules of a given but unknown logical puzzleor more generally a logical formula.

Original: We present SymSATNet, a variant of SATNet that translates the given symmetries of the target rules to a condition on theparameters of SATNet and requires that the parameters should have a particularparametric form that guarantees the condition.
Revised: We present SymSATNet, a variant of SATNet that translates the given symmetries of the target rules to a condition on theparameters of SATNet and requires that the parameters should have a particularparametric form that guarantees the condition.

Original: The requirement dramatically reduces the number of parameters to learn for the rules with enough symmetries, andmakes the parameter learning of SymSATNet much easier than that of SATNet.
Revised: The requirement dramatically re- duces the number of parameters to learn for the rules with enough symmetries, andmakes the parameter learning of SymSATNet much easier than that of SATNet.

Original: Wealso describe a technique for automatically discovering symmetries of the targetrules from examples.
Revised: We also describe a technique for automatically discovering symmetries of the targetrules from examples.

Original: Our experiments with Sudoku and Rubik’s cube show thesubstantial improvement of SymSATNet over the baseline SATNet.
Revised: Our experiments with Sudoku and Rubik’s cube show thesubstantial improvement of SymSATNet over the baseline SATNet.

Original:  Introduction
Revised:  Introduction

Original: Bringing the ability of reasoning to the deep-learning systems has been the aim of a large amount ofrecent research efforts [Yang et al., 2017, Evans and Grefenstette, 2018, Cingillioglu and Russo, 2019, Wang et al., 2019, Topan et al., 2021].
Revised: Bringing the ability of reasoning to the deep-learning systems has been the aim of a large amount ofrecent research efforts [28, 14, 5, 26, 23].

Original: One notable outcome of these endeavours is SATNet [Wanget al., 2019], a differentiable constraint solver with an efﬁcient custom backpropagation algorithm.
Revised: One notable outcome of these endeavours is SATNet [26],a differentiable constraint solver with an efficient custom backpropagation algorithm.

Original: SATNet can be used as a component of a deep-learning system and make the system capableof learning and reasoning about sophisticated logical rules.
Revised: SATNet canbe used as a component of a deep-learning system and make the system capable of learning andreasoning about sophisticated logical rules.

Original: Its potential has been demonstratedsuccessfully with the tasks of learning the rules of complex logical puzzles, such as Sudoku, justfrom input-output examples where the inputs are given as images.
Revised: Its potential has been demonstrated successfully withthe tasks of learning the rules of complex logical puzzles, such as Sudoku, just from input-outputexamples where the inputs are given as images.

Original: We show how to improve the rule (or constraint) learning of SATNet, when the target rules havesymmetries.
Revised: We show how to improve the rule (or constraint) learning of SATNet, when the target rules havepermutation symmetries.

Original: Bysymmetries, we mean transformations of candidate solutions of those rules, and byrules having symmetries, we meanthat the solution set of those rules is closed under the transformations of those symmetries.
Revised: By having symmetries, we mean the solutions of the rules are closed underthe permutations of those symmetries.

Original: For example, in Sudoku, if a completed 9 ⇥ 9 Sudoku board is a solution,permuting the numbers 1 to 9 in the board, the ﬁrst three rows, or the last three columns always givesrise to another solution.
Revised: For example, in Sudoku, if a completed 9 × 9 Sudoku board isa solution, permuting the numbers 1 to 9 in the board, the first three rows, or the last three columnsalways gives rise to another solution.

Original: Thus, these transformations are symmetries of Sudoku.
Revised: Thus, these permutations are symmetries of Sudoku.

Original: Our improvement is a variant of SATNet, called SymSATNet, which abbreviates symmetry-aware
Revised: Our improvement is a variant of SATNet, called SymSATNet, which abbreviates symmetry-aware

Original: SATNet.
Revised: SATNet.

Original: SymSATNet assumes that some symmetries of the target rules are given a priori although therules themselves are unknown.
Revised: SymSATNet assumes that some symmetries of the target rules are given a priori although therules themselves are unknown.

Original: It then translates these symmetries into a condition on the parametermatrix C 2 R n ⇥ n of SATNet (or our minor generalisation), and requires that the parameters have aparticular parametric form that guarantees the condition.
Revised: It then translates these symmetries into a condition on the parametermatrix C ∈ R n × n of SATNet (or our minor generalisation), and requires that the parameters have aparticular parametric form that guarantees the condition.

Original: Concretely, the translated condition says thatthe matrix C regarded as a linear map should be equivariant with respect to the group G determinedby the given symmetries, and the requirement is that C should be a linear combination of elements in a basis for the space of G -equivariant symmetric matrices.
Revised: Concretely, the translated condition says thatthe matrix C regarded as a linear map should be equivariant with respect to the group G determinedby the given symmetries, and the requirement is that C should be a linear combination of elements ina basis for the space of G -equivariant symmetric matrices.

Original: The coefﬁcients of this linear combinationare the parameters of our SymSATNet, and their number is often substantially smaller than that of theparameters of SATNet.
Revised: The coefficients of this linear combinationare the parameters of our SymSATNet, and their number is often substantially smaller than that of the parameters of SATNet.

Original: For Sudoku, the former is 18 , while the latter is 729 2 or k · 729 for somek 2 N at best.
Revised: For Sudoku, the former is 18 , while the latter is 729 2 or k · 729 for somek ∈ N at best.

Original: The reduced number of parameters implies that SymSATNet has to tackle an easierlearning problem than SATNet, and has a potential to learn faster and generalise better than SATNet.
Revised: The reduced number of parameters implies that SymSATNet has to tackle an easierlearning problem than SATNet, and has a potential to learn faster and generalise better than SATNet.

Original: Who provides symmetries for SymSATNet?
Revised: Who provides symmetries for SymSATNet?

Original: The default answer is domain experts, but we show thata better alternative is possible for some applications.
Revised: The default answer is domain experts, but a betteralternative is possible.

Original: We present an  algorithm that automaticallydiscovers symmetries.
Revised: We present an automatic algorithm to discover symmetries.

Original: Our algorithm is based onour empirical observation thatfor Sudoku and aproblem related to Rubik’s cube, symmetries emerge in the parameter matrix C of SATNet in theearly phase of training, as clusters of similar entries in C .
Revised: Our algorithm isbased on empirical observation that symmetries emerge in the parameter matrix C of SATNet in theearly phase of training, as clusters of similar entries.

Original: Our algorithm takes a snapshot ofthematrix C at some training epoch of SATNet, and searches for a group G such that (i) whenever twoentries of C are required to have similar values by the G -equivariance condition,they indeed havesuch values, and (ii) the number of SymSATNet parameters under G is minimised.
Revised: Our algorithm takes a snapshot of C at sometraining epoch of SATNet, and finds a group G such that (i) specific entries of C share similar valuesby G -equivariance condition, and (ii) the number of SymSATNet parameters under G is minimised.

Original: We empirically evaluate SymSATNet and our symmetry-discovering algorithm with Sudoku and aproblem related to Rubik’s cube.
Revised: We empirically evaluate SymSATNet and our symmetry-discovering algorithm with Sudoku and aproblem related to Rubik’s cube.

Original: For both problems, our algorithm discovered nontrivial symmetries,and SymSATNet with manually speciﬁed or automatically found symmetries outperformed thebaseline SATNet in learning the rules, in terms of both efﬁciency and generalisation.
Revised: For both problems, our algorithm discovered nontrivial symmetries,and SymSATNet with manually specified or automatically found symmetries outperformed thebaseline SATNet in learning the rules, in terms of both efficiency and generalisation.

Original: Related work There have been multiple studies on discovering symmetries present in conjunctivenormal form (CNF) formulas in order to reduce the search space of satisﬁability (SAT) solvers.
Revised: Related work There have been multiple studies on discovering symmetries present in conjunctivenormal form (CNF) formulas in order to reduce the search space of satisfiability (SAT) solvers.

Original: Crawford [1992] proved that the symmetry-detection problem is equivalent to the graph isomorphismproblem, and showed how to reduce the complexity of pigeonhole problems using symmetries.
Revised: Crawford [7] proved that the symmetry-detection problem is equivalent to the graph isomorphismproblem, and showed how to reduce the complexity of pigeonhole problems using symmetries.

Original: Crawford et al.
Revised: Crawford et al.

Original: [1996] proposed symmetry-breaking predicates (SBPs), and Aloul et al.
Revised: [8] proposed symmetry-breaking predicates (SBPs), and Aloul et al.

Original: [2003, 2006]developed SBPs with more efﬁcient constructions.
Revised: [1, 2] developed SBPs with more efficient constructions.

Original: For automatic symmetry detection, Darga et al.[2004] presented a method that improves the partition reﬁnement procedure introduced by McKay[1976, 1981], and Darga et al.
Revised: For automatic symmetry detection, Darga et al. [9] presenteda method that improves the partition refinement procedure introduced by McKay [19, 20], and Dargaet al.

Original: [2008] proposed an algorithm that achieve efﬁciency by exploitingthe sparsity of symmetries.
Revised: [10] proposed an algorithm that achieve efficiency by exploiting the sparsity of symmetries.

Original: In contrast to these global and static methods, Benhamou et al.
Revised: Incontrast to these global and static methods, Benhamou et al.

Original:  Devriendt et al.
Revised: [4] and Devriendt et al.

Original: [2017] handled local symmetries that dynamically appear during search.
Revised: [12] handledlocal symmetries that dynamically arise during search.

Revised: The use of symmetries also appears in

Revised: NeuroSAT [22], which learns how to solve SAT problems from examples.

Revised: NeuroSAT solves a given

Original:  In thecontext of deep learning, Basu et al.
Revised: SAT formula by message passing over a graph constructed from the formula, and in so doing, itslearnable solver can exploit symmetries in the formulas. All of these techniques use symmetriesto help solve given formulas, whereas our approach uses symmetries to help learn such formulas. Another difference is that those techniques find hard symmetries of given formulas, whereas ourapproach discovers soft or approximate symmetries in a given SATNet parameter matrix. In thecontext of deep learning, Basu et al.

Original: [2021], Dehmamy et al.
Revised: [3] and Dehmamy et al.

Original: [2021] described algorithms that ﬁndand exploit symmetries via group decompositions and Lie algebra convolutions.
Revised: [11] described algorithms that find andexploit symmetries via group decompositions and Lie algebra convolutions.

Revised: But these techniques arenot designed to find symmetries in logical formulas.

Original: Our work is alsorelated to the studies on learning logical rules from examples using gradients.
Revised: Our work is related to the studies on learning logical rules from examples using gradients.

Original: Yang et al.
Revised: Yang etal.

Original: [2017]proposed neural logic programming, an end-to-end differentiable system which learns ﬁrst-orderlogical rules, Evans and Grefenstette [2018] proposed a differentiable inductive logic programmingsystem which is robust to noise of training data, and Cingillioglu and Russo [2019] introduced an RNN-based model to learn logical reasoning tasks end-to-end.
Revised: [28] proposed neural logic programming, an end-to-end differentiable system which learns firstorder logical rules, Evans and Grefenstette [14] proposed a differentiable inductive logic programmingsystem which is robust to noise of training data, and Cingillioglu and Russo [5] introduced an RNNbased model to learn logical reasoning tasks end-to-end.

Original: Wang et al.
Revised: Want et al.

Original: [2018, 2019] presented SATNet using the mixing method, and Topan et al.
Revised: [25, 26] presented SATNet usingthe mixing method, and Topan et al.

Original: [2021] further improved SATNet by solving thesymbol grounding problem, a key challenge of SATNet.
Revised: [23] further improved SATNet by solving the symbol groundingproblem, a key challenge of SATNet.

Original: Our work extends these lines of work byproposing how to discover and exploit symmetries  when learning logical rules with SATNet.
Revised: Our work extends these lines of work by proposing how todiscover and exploit symmetries from examples when learning logical rules with SATNet.

Original:  Background
Revised:  Background

Original: We review SATNet, the formalisation of symmetries using groups, and equivariant maps.
Revised: We review SATNet, the formalisation of symmetries using groups, and equivariant maps.

Original: For anatural number n , let [ n ] = { 1 , . . . , n } , and for a matrix M , let M i,j be the ( i, j ) -th entry of M .{+.2.
+}
Revised: For anatural number n , let [ n ] = { 1 , . . . , n } , and for a matrix M , let M i,j be the ( i, j ) -th entry of M {+.2.
+}

Original: 2.1 SATNet
Revised: SATNet

Original: A good starting point for learning about SATNet is to look at its origin, the mixing method [Wanget al., 2018], which is an efﬁcient algorithm for solving semideﬁnite programming problems withdiagonal constraints.
Revised: A good starting point for learning about SATNet is to look at its origin, the mixing method [25], whichis an efficient algorithm for solving semidefinite programming problems with diagonal constraints.

Original: Let n, k 2 N and C be a real-valued symmetric matrix in R n ⇥ n .
Revised: Let n, k ∈ N and C be a real-valued symmetric matrix in R n × n .

Original: The mixingmethod aims at solving the following optimisation problem:
Revised: The mixing method aims at solvingthe following optimisation problem:

Original: [Equation]
Revised: [Equation]

Original: where v i is the i -th column of the matrix V , and k v i k is L 2 norm of v i .
Revised: where v i is the i -th column of the matrix V , and ∥ v i ∥ is L 2 norm of v i .

Original: The mixing methodsolves (1) by coordinate descent, where each column v i of V is repeatedly updated as follows:
Revised: The mixing methodsolves (1) by coordinate descent, where each column v i of V is repeatedly updated as follows:

Original: gi
Revised: g

Original: P j 2 [ n ] ,j 6 = i C i,j v j and vi  g
Revised: i ← (cid:80) j ∈ [ n ] ,j ̸ = i C i,j v j and v

Original: i k g i k .
Revised: i ← − g i ∥ g i ∥ .

Original: This always ﬁnds a ﬁxed point of the equations.
Revised: This always finds a fixed point of the equations.

Original: In fact, itis shown that almost surely this ﬁxed point attains a global optimum of the optimisation problem.
Revised: In fact, itis shown that almost surely this fixed point attains a global optimum of the optimisation problem.

Original: An example of the above optimisation problem most relevant to us is a continuous relaxation of
Revised: An example of the above optimisation problem most relevant to us is a continuous relaxation of

Original: MAXSAT.
Revised: MAXSAT.

Original: MAXSAT is a problem of ﬁnding truth assignments to n boolean variables b 1 , . . .
Revised: MAXSAT is a problem of finding truth assignments to n boolean variables b 1 , . . .

Original: , b n .
Revised: , b n .

Original: It assumes m clauses of those variables, F 1 , . . .
Revised: It assumes m clauses of those variables, F 1 , . . .

Original: , F m , where F ` is the disjunction of some variableswith or without negation: F ` = b i 1 _ . . .
Revised: , F m , where F ℓ is the disjunction of some variableswith or without negation: F ℓ = b i 1 ∨ . . .

Original: _ b i p _ ¬ b i p +1 _ . . .
Revised: ∨ b i p ∨ ¬ b i p +1 ∨ . . .

Original: ¬ b i p + q .
Revised: ¬ b i p + q .

Original: Then, MAXSAT asks for a 91 truth assignment on the variables that maximises the number of true clauses F i under the assignment.
Revised: Then, MAXSAT asks for atruth assignment on the variables that maximises the number of true clauses F i under the assignment.

Original: The rules of many problems, including Sudoku, can be expressed as an instance of MAXSAT.
Revised: The rules of many problems, including Sudoku, can be expressed as an instance of MAXSAT.

Original: To apply the mixing method to MAXSAT, we introduce relaxed vectors v 1 , . .
Revised: To apply the mixing method to MAXSAT, we introduce relaxed vectors v 1 , . .

Original: . , v n 2 R k that encodethe boolean variables, and construct the matrix S 2 R m ⇥ n that encodes the m clauses of MAXSAT:the ( ` , j ) -th entry of S has 1 if F ` contains b j ; and  1 if F ` includes ¬ b j ; and 0 if neither of thesecases holds.
Revised: , v n ∈ R k that encodethe boolean variables, and construct the matrix S ∈ R m × n that encodes the m clauses of MAXSAT:the ( ℓ, j ) -th entry of S has 1 if F ℓ contains b j ; and − 1 if F ℓ includes ¬ b j ; and 0 if neither of thesecases holds.

Original: Then, the problem in (1) is formed with C =  S T S , and solved by the mixing method.
Revised: Then, the problem in (1) is formed with C = − S T S , and solved by the mixing method.

Original: Our presentation is simpliﬁed, and the original version can be found in [Wang et al., 2018, 2019].

Original: SATNet is a variant of the mixing method where some of the columns of V are ﬁxed and theoptimisation is over the rest of the columns.
Revised: SATNet is a variant of the mixing method where some of the columns of V are fixed and theoptimisation is over the rest of the columns.

Original: Concretely, it assumes that the column indices in [ n ] aresplit into two disjoint sets, I and O (i.e., I 
Revised: Concretely, it assumes that the column indices in [ n ] aresplit into two disjoint sets, I and O (i.e., I ∪ O =

Original: [O = [ n ] and I \ O = ; ).
Revised: [ n ] and I ∩ O = ∅ ).

Original: The inputs of SATNet are the 101 columns v i of V with i 2 I , and the outputs are the rest of the columns (i.e., the v o ’s with o 2 O ).
Revised: The inputs of SATNet are thecolumns v i of V with i ∈ I , and the outputs are the rest of the columns (i.e., the v o ’s with o ∈ O ).

Original: The symmetric matrix C is the parameter of SATNet.
Revised: The symmetric matrix C is the parameter of SATNet.

Original: Given the input vectors, SATNet repeatedlyexecutes the coordinate descent updates on each output column, until it converges.
Revised: Given the input vectors, SATNet repeatedlyexecutes the coordinate descent updates on each output column, until it converges.

Original: One important feature of SATNet is that it has a custom algorithm for backpropagation.
Revised: One important feature of SATNet is that it has a custom algorithm for backpropagation.

Original: Let V I be thematrix of the input columns to SATNet, and V O be that of the output columns computed by SATNeton the input V
Revised: Let V I be thematrix of the input columns to SATNet, and V O be that of the output columns computed by SATNeton the input V

Original: I under the parameter C .
Revised: I under the parameter C .

Original: Assume that l is a loss of the output V O .
Revised: Assume that l is a loss of the output V O .

Original: In this context,
Revised: In this context,

Original: SATNet provides formulas and algorithms for computing the derivatives @ l/ @ V I and @ l/ @ C .
Revised: SATNet provides formulas and algorithms for computing the derivatives ∂l/∂V I and ∂l/∂C .

Original: We recall the formulas for the derivatives.
Revised: We recall the formulas for the derivatives.

Original: Let o 1 < o 2 < . . .
Revised: Let o 1 < o 2 < . . .

Original: < o |O| be the sorting of the indices in
Revised: < o |O| be the sorting of the indices in

Original: O .
Revised: O .

Original: Assume that SATNet was run until convergence, so that the output columns in V O are the ﬁxedpoint of the coordinate descent updates: for all o 2 O , g o = 
Revised: Assume that SATNet was run until convergence, so that the output columns in V O are the fixedpoint of the coordinate descent updates: for all o ∈ O , g o = (cid:80) j ∈ [ n ] ,j ̸ = o C o,j v j and v o = − g o ∥

Original: P
Revised: g o ∥ .

Original: [Equation]
Revised: The formulas for ∂l/∂V I and ∂l/∂C at ( V I , V O , C ) are defined in terms of the next quantities: C ′ , D ′ ∈ R |O|×|O| , P ∈ R |O| k ×|O| k ,

Original: [Equation]
Revised: U ∈ R |O| k

Original: and
Revised: ×

Original: [Equation]
Revised: [Equation]

Original: The formulas for @ l/ @ V I and @ l/ @ C at ( V I , V O , C ) are deﬁned in terms of the next quantities:
Revised: W

Original: [Equation]
Revised: [Equation]

Original:  They have the following deﬁnitions: for i, j 2 [ |O| ] , p, q 2 [ k ] , and r, s 2 [ n ] ,
Revised: R |O| k × n 2 . They have the following definitions: for i, j ∈ [ |O| ] , p, q ∈ [ k ] , and r, s ∈ [ n ] ,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: (cid:26) ( I k − v o i v To i ) p,q if i =

Original:  if i 6 = j,
Revised: j if i ̸ = j,

Original: U =
Revised: U =

Original: ( P ( [Equation] D 0 + C 0 )⌦
Revised: ( P (( D ′ + C ′ )

Original:  I 
Revised: ⊗ I k ))

Original: [Equation] [Equation] ✓ @ l @ vec( V O ) ◆ T ,
Revised: † (cid:18) ∂l ∂ vec( V O ) (cid:19) T ,

Original: W ( i  1) k
Revised: W ( i − 1) k

Original: + p, ( r  1) n + s = 8>>< >>: if r = o i and s =
Revised: + p, ( r − 1) n + s =   if r = o i and s =

Original: o
Revised: o

Original: i
Revised: i

Original: V p,s if r =
Revised: V p,s if r =

Original: o i and s
Revised: o i and s

Revised: ̸

Original: 6 = 
Revised: = o

Original: o i
Revised: i

Original: V p,s if r [-
-]
Revised: V p,s if r [-
-]̸

Original: = o i and s
Revised: = o i and s

Original: =
Revised: =

Original: o i if r [-
-]
Revised: o i if r [-
-]̸

Original: = o i and s 6 =
Revised: = o i and s ̸

Original:  o i .
Revised: = o i .

Original: Here ⌦ is the Kronecker product, _ † is Moore-Penrose inverse (also known as pseudo inverse), andvec( V O ) is the vector obtained by stacking the columns of V O .
Revised: Here ⊗ is the Kronecker product, _ † is Moore-Penrose inverse (also known as pseudo inverse), andvec( V O ) is the vector obtained by stacking the columns of V O .

Original: Let C O ,
Revised: Let C O ,

Original: I 2 R |O| ⇥ |I| be obtained by 115 restricting C to the indices ( o, i ) with o 2 O and i 2 I .
Revised: I ∈ R |O|×|I| be obtained byrestricting C to the indices ( o, i ) with o ∈ O and i ∈ I .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: SATNet computes the above derivative formulas efﬁciently by iterative algorithms.
Revised: SATNet computes the above derivative formulas efficiently by iterative algorithms.

Original: 2.2 Symmetries and equivariant maps
Revised: 2.2 Symmetries and equivariant maps

Original: By symmetries on a set X , we mean a group G that acts on X .
Revised: By symmetries on a set X , we mean a group G that acts on X .

Original: The acting here refers to a function_ · _ from G ⇥ X to X , called group action , such that (i) e · x = x for the unit e 2 G and any x 2 X , 120 and (ii) ( g  g 0 ) · x = g · ( g 0 · x ) for all g, g 0 2 G and x 2 X , where _  _ is the group operator of G .
Revised: The acting here refers to a function_ · _ from G × X to X , called group action , such that (i) e · x = x for the unit e ∈ G and any x ∈ X ,and (ii) ( g ◦ g ′ ) · x = g · ( g ′ · x ) for all g, g ′ ∈ G and x ∈ X , where _ ◦ _ is the group operator of G .

Original: We use symmetriesgenerated from a collection of permutations on a ﬁnite set.
Revised: We use symmetries of permutations on a finite set.

Original: The set X in our caseis R k ⇥ n , the space of the matrix V in (1), and G is a subgroup of the group S n of all permutations on[ n ] .
Revised: The set X in our case is R k × n , the space of thematrix V in (1), and G is a subgroup of the group S n of all permutations on [ n ] .

Original: The group action g · V is then deﬁned by permuting the columns of V by g : for all i, j 2 [ n ] ,( g · V ) i,j = V i,g  1 ( j ) .
Revised: The group actiong · V is then defined by permuting the columns of V by g : for all i, j ∈ [ n ] , ( g · V ) i,j = V i,g − 1 ( j ) .

Original: This group action can be expressed compactly with the n ⇥ n permutation matrix P g  1 where ( P g  1 ) i,j = 1 { i = g  1 ( j ) } and g · V = V P g  1 for the indicator function 1 .
Revised: This group action can be expressed compactly with the n × n permutation matrix P g − 1 where( P g − 1 ) i,j = 1 { i = g − 1 ( j ) } and g · V = V P g − 1 for the indicator function 1 .

Original: Throughout126 the paper, we often equate each element g of G with its permutation matrix P g , and view G itself asthe group of permutation matrices P g for g 2 G with the standard matrix multiplication.
Revised: Throughout the paper, weoften equate each element g of G with its permutation matrix P g , and view G itself as the group ofpermutation matrices P g for g ∈ G with the standard matrix multiplication.

Original: One important reason for considering symmetries is to study maps that preserve these symmetries,called equivariant maps.
Revised: One important reason for considering symmetries is to study maps that preserve these symmetries,called equivariant maps.

Original: Let G be a group that acts on sets X and Y .
Revised: Let G be a group that acts on sets X and Y .

Original: Deﬁnition 2.1.
Revised: Definition 2.1.

Original: A function f : X ! Y is G -equivariant or equivariant if f ( g · x ) =
Revised: A function f : X → Y is G -equivariant or equivariant if f ( g · x ) =

Original: g · ( f ( x ))
Revised: g · ( f ( x ))

Original: forall g 2 G and x 2 X .
Revised: forall g ∈ G and x ∈ X .

Original: It is G -invariant or invariant if f ( g · x ) =
Revised: It is G -invariant or invariant if f ( g · x ) =

Original: f
Revised: f

Original: ( x ) for all g 2 G and x 2 X .
Revised: ( x ) for all g ∈ G and x ∈ X .

Original: The forms of equivariant maps have been studied extensively in the work on equivariant neuralnetworks and group representation theory 
Revised: The forms of equivariant maps have been studied extensively in the work on equivariant neuralnetworks and group representation theory [6, 18, 27].

Original: [Cohen and Welling, 2016, Maron et al., 2019, Wang et al.

Original: In particular, when f is linear, various representation theorems for different G ’s describe thematrix form of f .
Revised: In particular, when f is linear, variousrepresentation theorems for different G ’s describe the matrix form of f .

Original: We use permutation groups deﬁned inductively by the following three operations.
Revised: We use permutation groupsdefined inductively by the following three operations.

Original: Deﬁnition 2.2.
Revised: Definition 2.2.

Original: Let G and H be permutation groups on [ p ] and [ q ] , with each group element viewedas a p ⇥ p or q ⇥ q permutation matrix.
Revised: Let G and H be permutation groups on [ p ] and [ q ] , with each group element viewedas a p × p or q × q permutation matrix.

Original: The direct sum G  H , the direct product G ⌦ H , andthe wreath product H o G are the following groups of ( p + q ) ⇥ ( p + q ) or pq ⇥ pq permutationmatrices with matrix multiplication as their composition:G  H = { g  h : g 2 G, h 2 H } ; G ⌦ H = { g ⌦ h g
Revised: The direct sum G ⊕ H , the direct product G ⊗ H , and thewreath product H ≀ G are the following groups of ( p + q ) × ( p + q ) or pq × pq permutation matriceswith matrix multiplication as their composition:

Original: H
Revised: [Equation] Here g ⊕ h is the block diagonal matrix with p × p matrix g as its upper-left corner and q × q matrixh as the lower-right corner, g ⊗ h is the Kronecker product of two matrices g and h , and wr( ⃗h, g )

Original: G Kronecker product of two matrices g and h and wr( ~ h g is the pq ⇥ pq permutationmatrix deﬁned by wr( ~ h g 
Revised: isthe pq × pq permutation matrix defined by wr( ⃗h, g ) ( i − 1) q + j, ( i ′ − 1) q + j ′ = 1 { g i,i ′ =(

Original: ( i  q +
Revised: h i )

Original:  } for all i,
Revised: j,j ′ =1 } for alli, i ′ ∈

Original:  and  j 2 [ q ] .
Revised: [ p ] and j, j ′ ∈ [ q ] .

Original: The next theorem speciﬁes the representation of G -equivariant linear maps for an inductivelyconstructed G , by describing a basis ofthe vector space of those linear maps.
Revised: The next theorem specifies the representation of G -equivariant linear maps for an inductivelyconstructed G , by describing a basis of those linear maps.

Original: The theorem can beproved by slightly modifying the argument in [Wang et al., 2020].

Original: For a permutation group G on [ m ] ,let E ( G ) =
Revised: For a permutation group G on [ m ] , let E ( G ) =

Original: { M 2 R m ⇥ m : gM = Mg, 8 g 2 G } , the vector space of G -equivariant linear maps, 148 where each g 2 G is regarded as a permutation matrix and gM is the multiplication of matrices g and M ( i.e. , the composition of two linear maps g and M ).
Revised: { M ∈ R m × m : Mg = gM, ∀ g ∈ G } , the vector space of G -equivariant linear maps, whereeach g ∈ G is regarded as a permutation matrix.

Original: See Appendix B for the proof of Theorem 2.3.
Revised: See Appendix B for the proof of Theorem 2.3.

Original: Theorem 2.3.
Revised: Theorem 2.3.

Original: Let G, H be permutation groups on [ p ] and [ q ] , and B ( G ) , B ( H ) be some bases of
Revised: Let G, H be permutation groups on [ p ] and [ q ] , and B ( G ) , B ( H ) be some bases of

Original: E ( G ) and E ( H ) , respectively.
Revised: E ( G ) and E ( H ) , respectively.

Original: Then, the following sets form bases for G  H , G ⌦ H , and H o G :B ( G  H ) = { A  0 q , 0 p  B
Revised: Then, the following sets form bases for G ⊕ H , G ⊗ H , and H ≀ G :

Original: [Equation]
Revised: [Equation]

Original:  A 2 B ( G ) 
Revised: ⊗ B : A ∈ B ( G ) ,

Original: [Equation] B 2 B ( H ) } [Equation]
Revised: B ∈ B ( H ) } ; B ( H ≀ G ) =

Original:  O [Equation] H
Revised: { A ⊗ 1 O ′ ×

Original: [Equation]
Revised: O ′′ :

Original:  B ( G ⌦ H
Revised: A ∈ B ( G ) ,

Original: [Equation] = [Equation] A 
Revised: A i,i = 0 for i ∈

Original: [Equation]
Revised: [ p ] and O ′ ,

Original: [Equation] B [Equation] ( H ) } 
Revised: O ′′ ∈ O ( H ) } ∪ { I O ⊗ B : B ∈ B ( H ) ,

Original: [Equation] B ( H o
Revised: O ∈ O ( G ) } .

Original: [Equation] = { A ⌦ [Equation] ⌦
Revised: Here 0 m is an everywhere-zero matrix in R m × m , and 1 R × S ,

Original: [Equation] A [Equation] B ( G ), B
Revised: I R are the matrices defined by( 1 R × S )

Original: [Equation] B ( H ) }. Here I m , 0 m , and 1 m are identity, everywhere-zero, and everywhere-one matrices in R m ⇥ m .
Revised: i,j = 1 { i ∈ R,j ∈ S } , ( I R ) i,j = 1 { i = j,i ∈ R } whose shapes are defined by the context in whichthey are used. Here, 1 O × ( p + O ′ ) , 1 ( p + O ′ )

Original: 1 R ⇥ Sfor R, S ✓ [ p + q ] is the matrix in R ( p + q ) ⇥ ( p + q ) deﬁned by ( 1  R ⇥ S ) i,j = 1 { i 2 R,j 2 S } .
Revised: × O ∈ R ( p + q ) × ( p + q ) , 1 O ′ × O ′′ ∈ R q × q , I O ∈ R p × p .

Original: Also,
Revised: Also,

Original: O ( G ) =
Revised: O ( G ) =

Original: {{ g ( i ) : g 2 G } : i 2 [ p ] } (i.e., the set of G -orbits), and p + O = { p + i : i 2 O } .
Revised: {{ g ( i ) : g ∈ G } : i ∈ [ p ] } (i.e., the set of G -orbits), and p + O = { p + i : i ∈ O } .

Original:  Symmetry-aware SATNet
Revised:  Symmetry-aware SATNet

Original: In this section, we present SymSATNet, which abbreviates symmetry-aware SATNet.
Revised: In this section, we present SymSATNet, which abbreviates symmetry-aware SATNet.

Original: This variant isdesigned to operate when symmetries of a learning task are known a priori (via an algorithm or adomain expert).
Revised: This variant isdesigned to operate when symmetries of a learning task are known a priori (via an algorithm or adomain expert).

Original: The proofs of the theorem and the lemma in the section are in Appendix C.
Revised: The proofs of the theorem and the lemma in the section are in Appendix C.

Original: SymSATNet solves the optimisation problem of SATNet, but under the following assumptions:
Revised: SymSATNet solves the optimisation problem of SATNet, but under the following assumptions:

Original:  The optimisation objective  as a map on  R k ⇥ n is invariant with respect to certain symmetriesspeciﬁed by a  group Gacting on R k ⇥ n , the space of matrices V . The group G and its action are ofthe type in Section 2.2. That is, eachof g 2 G acts as apermutation on the columns of V .
Revised: Assumption 3.1. The optimisation objective ⟨ C, V T V ⟩ in (1) as a map on V = R k × n is invariantunder a permutation group G , whose action is of type in Section 2.2 (i.e., each g ∈ G acts as apermutation on the columns of V ).

Original: Continuing our convention, we denote P g by g .
Revised: Continuing our convention, we denote P g by g .

Original: One immediate consequence of these assumptions is that for all g  G and [Equation]
Revised: One immediate consequence of Assumption 3.1 is⟨ C, V T V ⟩ = ⟨ C, ( g · V ) T ( g · V ) ⟩ = ⟨ C, ( V g − 1 ) T ( V g − 1 ) ⟩ for all g ∈ G and V ∈

Original: [Equation]
Revised: R k × n .

Original: The next theorem re-phrases this property of the optimisation objective as equivariance of C :
Revised: The next theorem re-phrases this property of the optimisation objective as equivariance of C :

Original: Theorem 3.1.
Revised: Theorem 3.2.

Original: Let C be a symmetric n ⇥ n matrix.
Revised: Let C be a symmetric n × n matrix.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original:  for all V 2 R k ⇥ n and g 2 G if C as a linear map on R n is G -equivariant, that is, Cg = gC for allg 2 G .
Revised: [Equation] (3) for all V ∈ R k × n and g ∈ G if C as a linear map on R n is G -equivariant, that is, Cg = gC for allg ∈ G .

Original: Furthermore, if k = n , the converse also holds.
Revised: Furthermore, if k = n , the converse also holds.

Original: What we discussed so far lets us incorporate the symmetries into the objective of SATNet, as wewill show shortly. This leaves the  diagonal constraints of SATNet.
Revised: This theorem lets us incorporate the symmetries into the objective of SATNet, and leaves the handlingof the diagonal constraints of SATNet.

Original: The next lemma says that thoseconstraints require no special treatment since they are  preserved by the action of any g 2 G .
Revised: The next lemma says that those constraints require no specialtreatment, though, since they are already preserved by the action of any g ∈ G .

Original: Lemma 3.2.
Revised: Lemma 3.3.

Original: Let V 2 R k ⇥ n and g 2 G .
Revised: Let V ∈ R k × n and g ∈ G .

Original: Every column of V has the L 2 -norm 1 if and only if everycolumn of V g  1 has the L 2 -norm 1 .
Revised: Every column of V has the L 2 -norm 1 if and only if everycolumn of V g − 1 has the L 2 -norm 1 .

Original: Let E ( G ) =
Revised: Recall E ( G ) =

Original: { M 2 R n ⇥ n : Mg = gM, 8 g 2 G } be the vector space of G -equivariant matrices 175 and
Revised: { M ∈ R n × n : Mg = gM, ∀ g ∈ G } is the vector space of G -equivariant matrices.

Original:  E ( G ) s be the subset of E ( G ) containing  symmetric matrices.
Revised: Let E ( G ) s be the subset of E ( G ) containing only symmetric matrices.

Original: When G is a permutation groupconstructed inductively by direct sum, direct product,  wreath product, we can generate a basis
Revised: When G is a permutationgroup constructed by direct sum, direct product, and wreath product, we can generate a basis

Original: B ( G ) of E ( G ) automatically using Theorem 2.3.
Revised: B ( G ) of E ( G ) automatically using Theorem 2.3.

Original: Then, we can convert E ( G ) to an orthogonalbasis of E ( G ) s by applying the Gram-Schmidt orthogonalisation to { B + B T : B 2 B ( G ) } .
Revised: Then, we can convert B ( G ) to an orthogonalbasis of E ( G ) s by applying the Gram-Schmidt orthogonalisation to { B + B T : B ∈ B ( G ) } .

Original: Let
Revised: Let

Original: B ( G ) s
Revised: B ( G ) s

Original: = { B 1 , . .
Revised: = { B 1 , . .

Original: , B d } be such anautomatically-generated orthogonal basis of E ( G ) s .
Revised: , B d } be such an orthogonal basis of E ( G ) s .

Original: SymSATNet is SATNet where the matrix C in the optimisation objective has the form:
Revised: SymSATNet is SATNet where the matrix C in the optimisation objective has the form:

Original: [Equation]
Revised: [Equation]

Original: for some scalar ✓ 1 , . . .
Revised: for some scalars θ 1 , . . .

Original: , ✓ d 2 R .
Revised: , θ d ∈ R .

Original: Note that by this condition on the form of C , SymSATNet has onlyd parameters ✓ 1 , . .
Revised: Note that by this condition on the form of C , SymSATNet hasonly d parameters θ 1 , . .

Original: . , ✓ d , instead of n ⇥ m for some m in the original formulation of SATNet.
Revised: , θ d , instead of n × m for some m in the original formulation of SATNet.

Original: Whilelearning C with symmetries, d is usually far smaller than n 2 or even n , and this reduction brings speedup and improved generalisation.
Revised: When the learning target has enough symmetries, d is usually far smaller than n 2 or even n , and thisreduction brings speed-up and improved generalisation.

Original: Note that (4) amounts to requiring that the optimisation problem of

Original: SATNet remain invariant with respect to G ; by Theorem 3.1, it ensures that the optimisation objectiveis G -invariant, and by Lemma 3.2, the diagonal constraints do not change by the action of G .

Original: The forward computation of SymSATNet is precisely that of SATNet, the repeated coordinate-wiseupdates until convergence, and the backward computation is the one of SATNet extended (by thechain rule) with a step backpropagating the derivatives @ l/ @ C to each [-@ l/ @✓ ↵ for ↵ 
-]
Revised: The forward computation of SymSATNet is precisely that of SATNet, the repeated coordinate-wiseupdates until convergence, and the backward computation is the one of SATNet extended (by thechain rule) with a step backpropagating the derivatives ∂l/∂C to each [-@ l/ @✓ ↵ for ↵ 
-]∂l/∂θ

Original:  [ d ] .[-
-]
Revised: α for α ∈ [ d ] .[-
-]

Original: We summarise SymSATNet below using the usual notation of SATNet ( I , O , V I , V O , and V ):
Revised: We summarise SymSATNet below using the usual notation of SATNet ( I , O , V I , V O , and V ):

Original: • The input is V I , the matrix of the input columns of V .
Revised: • The input is V I , the matrix of the input columns of V .

Original: • The parameters are ( ✓ 1 , . . .
Revised: • The parameters are ( θ 1 , . . .

Original: , ✓ d ) 2 R d .
Revised: , θ d ) ∈ R d .

Original: They deﬁne the matrix C by (4).
Revised: They define the matrix C by (4).

Original: • The forward computation solves the following optimisation problem using coordinate descent, and returns V O , the matrix of the output columns of V :
Revised: • The forward computation solves the following optimisation problem using coordinatedescent, and returns V O , the matrix of the output columns of V :

Original: argmin
Revised: argmin

Original: [Equation]
Revised: [Equation]

Original: subject to 
Revised: to ∥

Original: [Equation]
Revised: [Equation]

Original: for o
Revised: for o

Original: [Equation]
Revised: [Equation]

Original: O . • The backward computation computes @ l/ @
Revised: O . • The backward computation computes ∂l/∂V

Original: V I and @ l/ @✓ ↵ by (2) and the chain rule:
Revised: I and ∂l/∂θ α by (2) and the chain rule:

Original: [Equation]
Revised: [Equation]

Original:  Discovery of symmetries
Revised:  Discovery of Symmetries

Original: One obstacle for using SymSATNet is that the user has to specify symmetries.
Revised: One obstacle for using SymSATNet is that the user has to specify symmetries.

Original: We now discuss howto alleviate this issue by presenting an algorithm for discovering candidate symmetries automatically.
Revised: We now discuss howto alleviate this issue by presenting an algorithm for discovering candidate symmetries automatically.

Original: The goal of our algorithm, denoted by S YM F IND , is to ﬁnd a permutation group G that captures thesymmetries of an unknown learning target and is expressible by the grammar G ::
Revised: The goal of our algorithm, denoted by S YM F IND , is to find a permutation group G that captures thesymmetries of an unknown learning target and is expressible by the following grammar:

Original: = I m | {+
+} Z m | S m |201 G  G | G ⌦ G | G o G , where m 2 N .
Revised: G ::= I m | {+
+} Z m | S m | G ⊕ G | G ⊗ G | G ≀ G for m ∈ N .

Original: The I mdenotes the trivial group containing only the identity202 permutation on [ m ] , and Z m denotes the group of cyclic permutations on [ m ] , each of which mapsi 2 [ m ] to ( i + n ) mod m for some n .
Revised: The I m the trivial group containing only the identity permutation on [ m ] , and Z m denotes the group of cyclic permutations on [ m ] , each ofwhich maps i ∈ [ m ] to ( i + n ) mod m for some n .

Original: The S m is the group of all the permutations on [ m ] .
Revised: The S m is the group of all the permutations on[ m ] .

Original: The lastthree cases are direct sum, direct product, and wreath product (see Deﬁnition 2.2).
Revised: The last three cases are direct sum, direct product, and wreath product (see Definition 2.2).

Original: They describe three ways of decomposing a group G into smaller parts.
Revised: Theydescribe three ways of decomposing a group G into smaller parts.

Original: Having such a decomposition of G bringsthe beneﬁt that we can recursively and efﬁciently compute a basis of G -equivariant linear maps.
Revised: Having such a decomposition of G brings the benefit that we can recursively and efficiently compute a basis of G -equivariant linearmaps.

Original: The design of S YM F IND is based on our empirical observation that a softened version of symmetriesoften emerges in the parameter matrix C of the original SATNet during training.
Revised: The design of S YM F IND is based on our empirical observation that a softened version of symmetriesoften emerges in the parameter matrix C of the original SATNet during training.

Original: Even in the earlypart of training, many entries of C share similar values, so that there is a  group G 0 of approximatesymmetries of C such that Cg ⇡ gC for any g 2 G 0 .
Revised: Even in the earlypart of training, many entries of C share similar values, and there is a large-enough group G with Cg ≈ gC for all g ∈ G , which intuitively means that G captures symmetries of C .

Original: Furthermore, we observed that sucha G 0 oftenconsists of symmetries of the learning target.
Revised: Furthermore, weobserved, such G often consists of symmetries of the learning target.

Original: This observation suggests an algorithm that takes C asinput and ﬁnds sucha G0 expressible in our grammar or its slight extension.
Revised: This observation suggests analgorithm that takes C as input and finds such G expressible in our grammar or its slight extension.

Original: Algorithm 1 S Input: M 2 R m ⇥ m
Revised: 1: Input: M ∈ R m × m

Original: Output: ( G,  )  if k prj(grp( S m , id m ) , M )  M k F   then  return ( S m , id m )  end if  A  { ( I m , id m ) }  if k prj(grp( Z m , id m ) , M )  M k F   then 
Revised: Output: ( G, σ ) 2: if ∥ prj(grp( S m , id m ) , M ) − M ∥ F ≤ λ then 3: return ( S m , id m ) 4: end if 5: A ← { ( I m , id m ) } 6: if ∥ prj(grp( Z m , id m ) , M ) − M ∥ F ≤ λ then 7:

Original: A  A [ { ( Z m , id m ) }  end if  ( G 0 ,  0 )
Revised: A ← A ∪ { ( Z m , id m ) } 8: end if 9: ( G ′ , σ ′ )

Original:  S UM F IND ( M ) ; 
Revised: ← S UM F IND ( M ) ; 10:

Original: A  A [ { ( G 0 ,  0 ) }  for every divisor p of m do  ( G 00 ,  00 )  P ROD F IND ( M, p ) ; 
Revised: A ← A ∪ { ( G ′ , σ ′ ) } 11: for every divisor p of m do 12: ( G ′′ , σ ′′ ) ← P ROD F IND ( M, p ) ; 13:

Original: A  A [ { ( G 00 ,  00 ) }  end for  ( G,  )  argmin ( G,  ) [-
 A-] dim( E (grp( G,  )))  return ( G,  )
Revised: A ← A ∪ { ( G ′′ , σ ′′ ) } 14: end for 15: ( G, σ ) ← argmin ( G,σ ) [-
 A-]∈A dim( E (grp( G, σ ))) 16: return ( G, σ )

Original: The input of S YM F IND is a matrix M 2 R m ⇥ m .
Revised: The input of S YM F IND is a matrix M ∈ R m × m .

Original: As previously explained, when S YM F IND iscalled at the top level, it receives as input theparameter C of SATNet learnt by a ﬁxed numberof training steps.
Revised: As previously explained, when S YM F IND iscalled at the top level, it receives as input the parameter C of SATNet learnt by a fixed numberof training steps.

Original: However, subsequent recursivecalls to S YM F IND may have input M differentfrom C .
Revised: However, subsequent recursivecalls to S YM F IND may have input M differentfrom C .

Original: Then, S YM F IND returns a group G inour grammar and a permutation  on [ m ] , together deﬁning a permutation group on [ m ] :
Revised: Then, S YM F IND returns a group Gin our grammar and a permutation σ on [ m ] ,together defining a permutation group on [ m ] :

Original: [Equation]
Revised: [Equation]

Original: where  is the composition of permutations.
Revised: where ◦ is the composition of permutations.

Original: When G is decomposed into, say G 1  G 2 , the 224  speciﬁes which indices in [ m ] get permutedby G 1 and G 2 .
Revised: When G is decomposed into, say G 1 ⊕ G 2 , theσ specifies which indices in [ m ] get permutedby G 1 and G 2 .

Original: Once top-level S YM F IND returns( G,  ) , we construct B (grp( G,  )) 
Revised: Once top-level S YM F IND returns ( G, σ ) , we construct B (grp( G, σ )) s , as in

Original: s , as explainedin Section 3 with a minor adjustment with  . {+
+}
Revised: Section 3 with a minor adjustment with σ . {+
+}

Original: Algorithm 1 describes S YM F IND , where id m isthe identity permutation on [ m ] , k · k F is the Frobenius norm, dim( V ) is the dimension of a vector230 space 
Revised: Algorithm 1 describes S YM F IND , where id m is the identity permutation on [ m ] , ∥·∥ F is the Frobeniusnorm, dim( V ) is the dimension of a vector space V , and the Reynolds operator prj projects a matrix

Original: V , and the Reynolds operator prj projects a matrix M 2 R m ⇥ m orthogonally to the subspaceof G -equivariant m ⇥ m matrices (i.e., it maps M to the closest G -equivariant matrix in L 2 distance):
Revised: M ∈ R m × m orthogonally to the subspace of G -equivariant m × m matrices (i.e., it maps M to theclosest G -equivariant matrix in L 2 distance):

Original: [Equation]
Revised: [Equation]

Original: so that k prj(grp( G, M ))  M k F computes the distance between the marix M and the space E ( G ) .
Revised: so that ∥ prj( G, M ) − M ∥ F computes the distance between the matrix M and the space E ( G ) ofequivariant matrices.

Original: In the lines 1-3, the algorithm ﬁrst checks whether S m models symmetries of the input M accurately.
Revised: In the lines 2-4, the algorithm first checks whether S m models symmetries of the input M accurately.

Original: If so, the algorithm returns ( S m , id m ) .
Revised: If so, the algorithm returns ( S m , id m ) .

Original: Otherwise, it assumes that an appropriate group for M ’ssymmetries is one of the remaining cases in the grammar, and constructs a list A of candidates intiallycontaining the trivial group ( I m , id m ) .
Revised: Otherwise, it assumes that an appropriate group for M ’ssymmetries is one of the remaining cases in the grammar, and constructs a list A of candidatesintially containing the trivial group ( I m , id m ) .

Original: In the lines 5-7, the algorithm adds  ( Z m , id m ) to A if theyapproximate M ’s symmetries well.
Revised: In the lines 6-8, the algorithm adds a pair ( Z m , id m )to A if it approximates M ’s symmetries well.

Original: In the lines 8-9, it handles the case that M ’s symmetries aremodelled by a direct sum
Revised: In the lines 9-10, the algorithm calls the subroutine

Original: L i G 0 i . The algorithm calls the subroutine S UM F IND which returns a group G 0 of the form
Revised: S UM F IND which finds grp( G ′ , σ ′ ) with G ′ =

Original: L i G 0 i[-and a permutation  
 if it can ﬁnd such grp( G 0 ,  0 )-] that approximates M ’ssymmetries well.The returned group-permutation pair is added to A .
Revised: (cid:76) i G ′ i[-and a permutation  
 if it can ﬁnd such grp( G 0 ,  0 )-] that approximates M ’s symmetries well.

Original: In the lines 10-13, the algorithmcalls the other subroutine P ROD F IND for every divisor p of m .
Revised: Inthe lines 11-14, the algorithm calls the other subroutine P ROD F IND for every divisor p of m .

Original: For each p , P ROD F IND ﬁnds a group
Revised: Foreach p , P ROD F IND finds grp( G ′′ , σ ′′ ) with G ′′ =

Original: G 00 of the form G 00 1 ⌦ G 00 2 or G 00 2 o G 00 1 and an appropriate permutation  00 where G 00 1 is a permutation 243 group on [ p ] andG2 on [ m/p ] , and such grp( G 00 ,  00 ) approximates M ’s symmetries well.
Revised: G ′′ 1 ⊗ G ′′ 2 (or G ′′ 2 ≀ G ′′ 1 ) that approximates M ’ssymmetries well, where G ′′ 1 and G ′′ 2 are permutation groups on [ p ] and [ m/p ] .

Original: The resultof each such call is added to the list A .

Original: Finally, in the line 14, our S YM F IND picks a pair ( G,  ) from  A with the strongest level of symmetries in the sense that the basis of grp( G,  ) -equivariant matricesfor the selected ( G,  ) has the fewest elements.
Revised: Finally, in the line 15,our S YM F IND picks a pair ( G, σ ) from the candidates A with the strongest level of symmetries inthe sense that the basis of grp( G, σ ) -equivariant matrices has the fewest elements.

Original: The subroutine S UM F IND clusters entries of M as blocks since block-shaped clusters commonlyarise in matrices equivariant with respect to a direct sum of groups.
Revised: The subroutine S UM F IND clusters entries of M as blocks since block-shaped clusters commonlyarise in matrices equivariant with respect to a direct sum of groups.

Original: The other subroutine P ROD F INDuses the technique of Van Loan and Pitsianis [1993]; it exploits a typical pattern of Kronecker productof matrices, and detects the presence of the pattern in M by applying SVD to a reshaped version of M .
Revised: The other subroutine P ROD F INDuses a technique [24] to exploit a typical pattern of Kronecker product of matrices, and detects thepresence of the pattern in M by applying SVD to a reshaped version of M .

Original: Each subroutine may call S YM F IND recursively.
Revised: Each subroutine may call S YM F IND recursively.

Original: See Appendix D for the details.
Revised: See Appendix D for the details.

Original:  Experimental results
Revised:  Experimental Results

Original:  We experimentally evaluated SymSATNet and the S YM F IND algorithm on the tasks of learningrules of two problems,namely, Sudoku and what we call completion problem of Rubik’s cube.
Revised: [Figure] We experimentally evaluated SymSATNet and the S YM F IND algorithm on the tasks of learning rulesof two problems, Sudoku and the completion problem of Rubik’s cube.

Original: Our evaluation used the original SATNet  as a baseline, andwas under both manually-speciﬁedand automatically-discovered symmetries.
Revised: The original SATNet wasused as a baseline, and both ground-truth and automatically-discovered symmetries were used for

Original:  For S YM F IND , we randomly generated matrices Mequivariant with respect to  known symmetries and tested whether S YM F IND could recover thosesymmetries from M .
Revised: SymSATNet. For S YM F IND , we also tested its ability to recover known symmetries given randomlygenerated equivariant matrices.

Original: We observed signiﬁcant improvement of SymSATNet over SATNet in thelearning tasks, and also the promising results and limitation of S YM F IND .
Revised: We observed significant improvement of SymSATNet over SATNetin the learning tasks, and also the promising results and limitation of S YM F IND .

Original: [Figure]
Revised: [Figure]

Original: Sudoku problem In Sudoku, we are askedto ﬁll in the empty cells of a 9 ⇥ 9 boardsuch that every row, every column, and eachof nine 3 ⇥ 3 blocks have all of 1 up to 9 .
Revised: Sudoku problem In Sudoku, we are asked to fillin the empty cells of a 9 × 9 board such that everyrow, every column, and each of nine 3 × 3 blockshave all numbers 1 − 9 .

Original: Let

Original:  A 2 { 0 , 1 } 9 ⇥ 9 ⇥ 9 be the encoding of a full number assignment for the board where the ( i, j, k ) -th entry of A is 1 if the ( i, j ) -th cell of the boardcontains k .
Revised: Let A ∈ { 0 , 1 } 9 × 9 × 9 bethe encoding of a full number assignment for theboard where the ( i, j, k ) -th entry of A is 1 if the( i, j ) -th cell of the board contains k .

Original: In SATNet, we ﬂatten A to the assignment on the n = 9 3 = 729 boolean variables, and relax each of these variables so thatit contains a vector in R k , resulting V 2 R k ⇥ nin theoptimisation objective of SATNet.
Revised: In SATNet,we flatten A to the assignment on the n = 9 3boolean variables, and relax each variable into R k ,resulting V ∈ R k × n in the objective of SATNet.

Original: The rules of Sudoku have symmetries formalised by the permutation group G =
Revised: The rules of Sudoku have symmetries formalisedby G =

Original: ( S 3 o S 3 ) ⌦ ( S 3 o S 3 ) ⌦ S 9 .
Revised: ( S 3 ≀ S 3 ) ⊗ ( S 3 ≀ S 3 ) ⊗ S 9 .

Original: Each of two S 3 o S 3275 refers to solution-preserving permutations forrows and columns in Sudoku.
Revised: Each of two S 3 ≀ S 3 refers to solution-preserving permutationsfor rows and columns in Sudoku.

Original: The last S 9 refersto permutations of the assigned numbers 1  9 ineach cell.
Revised: The last S 9refers to permutations of the assigned numbers1 − 9 in each cell.

Original: See Appendix E for more informationabout the symmetry group for Sudoku.
Revised: See Appendix E for more information about the symmetry group for Sudoku.

Original: To learn the rules of Sudoku using SymSATNet,we constructed a basis B ( G ) s as explained in Section 3.
Revised: To learn the rules of Sudoku using SymSATNet,we constructed a basis B ( G ) s as explained in Section 3.

Original: It has 18 elements, which mean that SymSATNet has 18 parameters to learn.
Revised: It has 18 elements, which means that SymSATNet has 18 parameters to learn.

Original: We used 9K training and 1K test examples generated by the Sudoku generator of Park [2018].
Revised: We used 9K training and 1K test examples generated by the Sudoku generator [21].

Original: Eachexample is a pair ( V I , V O ) where the input V I assigns the values of 39-50 cells (out of 81 cells) andthe output V O speciﬁes the remaining cells.
Revised: Each example isa pair ( V I , V O ) where the input V I assigns 31-42 cells (out of 81 cells) and the output V O specifiesthe remaining cells.

Original: SymSATNet was compared with two types of SATNet: SATNet-Plain without any auxiliary variables, and SATNet-300aux with 300 auxiliary variables.
Revised: SymSATNet was compared with two types of SATNet: SATNet-Plain withoutauxiliary variables, and SATNet-300aux with 300 auxiliary variables.

Original: Wetrained these 3 instances under the binary cross entropy loss and the Adam optimizer [Kingma and Ba,2015], with the learning rate ⌘ = 2 ⇥ 10  3 for SATNet-Plain and SATNet-300aux as in the originalwork on SATNet, and ⌘ = 4 ⇥ 10  2 for SymSATNet.
Revised: We trained under the binarycross entropy loss and the Adam optimizer [16], with the learning rate η = 2 × 10 − 3 for SATNet-Plainand SATNet-300aux as in the original work and η = 4 × 10 − 2 for SymSATNet.

Original: We measured test accuracy, which is theaverage number of the correctly-solved Sudoku instances in the 1K test set.
Revised: We measured testaccuracy, which is the rate of the correctly-solved Sudoku instances by the forward computations.

Original: Each of our experimentswas run four times, and we report the average with 95% conﬁdence interval in each run. [Table] [Table] [Table]
Revised: Each of our experiments was run 10 times, and we report the average with 95% confidence interval.

Original: % ±

Original: [Table]
Revised: [Table]

Original: % ± [Table] The results are in Figure 1 and Table 1.(SymSATNet-Auto is SymSATNet withour automatic symmetry-discovery algorithm, and will be described  in the lastparagraph of this section.)
Revised: The results are in Figure 1 and Table 1.(SymSATNet-Auto refers to a variant of SymSATNet that uses S YM F IND as asubroutine to find symmetries automatically, and will be described later in thissection.)

Original: Our SymSATNet outperformed SATNet-Plain and SATNet-300aux.
Revised: Our SymSATNet outperformed SATNet-Plain and SATNet-300aux.

Original: On average, itstraining run for 100 iterations ﬁnished 2 - 4 ⇥faster in terms of wall clock than those ofthese two alternatives.
Revised: Onaverage, its 100 epochs finished 2 - 4 ×faster in terms of wall clock than twoalternatives.

Original: This is due to thereduced number of parameters and avoidance ofcertain matrix operations.
Revised: due to the reduced number of iterations and avoidance of matrix operations.

Original: See Appendix F for more information aboutthe operations in SymSATNet.
Revised: See Appendix F for theefficiency of SymSATNet.

Original: Despite the speed up, the best test accuracy of SymSATNet ( 99 . 3% )on average was signiﬁcantly better than SATNet-Plain ( 88 . 3% ) and slightly better than SATNet-300aux ( 98 . 0% ).
Revised: Despite thespeed up, the best test accuracy of SymSATNet ( 99 . 2% ) was significantly betterthan SATNet-Plain ( 88 . 1% ) and slightlybetter than SATNet-300aux ( 97 . 9% ).

Original: Completion problem of Rubik’s cube Thestandard Rubik’s cube is composed of 6 faces, eachof which has 9 facelets.
Revised: Completion problem of Rubik’s cube The Rubik’s cube is composed of 6 faces, each of whichhas 9 facelets.

Original: We considered a constraint satisfaction problem related to Rubik’s cubewhere we aregiven a partial colour assignment of facelets, and then asked to complete  the remainingfacelets such that the resulting colour assignment is solvable; by moving the cube, we can make allfacelets in each face have the same colour, and no same colours appear in two faces.
Revised: We considered a constraint satisfaction problem where we are asked to complete themissing facelets of the Rubik’s cube such that the resulting cube is solvable; by moving the cube, wecan make all facelets in each face have the same colour, and no same colours appear in two faces.

Original: Let A 2 { 0 , 1 } 6 ⇥ 9 ⇥ 6 be a colour assignment of Rubik’s cube where the ( i, j, k ) -th entry has 1 ifand only if the j -th facelet of the i -th face has colour k .
Revised: Let A ∈ { 0 , 1 } 6 × 9 × 6 be a colour assignment of Rubik’s cube where the ( i, j, k ) -th entry has 1 ifand only if the j -th facelet of the i -th face has colour k .

Original: We formulate the optimisation objective of
Revised: We formulate the optimisation objective of

Original: SATNet for Rubik’s cube using the relaxation ofsuch an assignment A to V 2 R k ⇥ n for n = 6 ⇥ 9 ⇥ 6 .
Revised: SATNet for Rubik’s cube using the relaxation of A to V ∈ R k × n for n = 6 × 9 × 6 .

Original: This problem has symmetries formalised bya group G = R 54 ⌦ R 6 on [ n ] .
Revised: This problem has symmetries formalised by G = R 54 ⊗R 6 on [ n ] .

Original: Here R 54 and R 6 are 318 permutation groups on [54] and [6] , each of which captures the allowed rotations of facelets, andtherotations of the whole cube.
Revised: Here R 54 and R 6 are permutationgroups on [54] and [6] , each of which captures the allowed moves of facelets, and the rotations of thewhole cube.

Original: If a colour assignment A is solvable, so is the transformation of A byany permutations in G .
Revised: If a colour assignment A is solvable, so is the transformation of A by any permutationsin G .

Original: See Appendix E for more information about the symmetries of this problem.
Revised: See Appendix E for more information about the symmetries of this problem.

Original: We generated a basis B ( G ) s in three steps.
Revised: We generated a basis B ( G ) s in three steps.

Original: We ﬁrst created B ( R 54 ) and B ( R 6 ) using the technique of 322 Finzi et al.
Revised: We first created B ( R 54 ) and B ( R 6 ) using the generatorsof each group [15].

Original: Next, we combined them using Theorem 2.3, and got B ( G ) , which was convertedto a symmetric orthogonal basis B ( G ) s via Gram-Schmidt.
Revised: Next, we combined them using Theorem 2.3 to get B ( G ) , which was convertedto a symmetric orthogonal basis B ( G ) s via Gram-Schmidt.

Original: The ﬁnal result has 48 basis elements.
Revised: The final result has 48 basis elements.

Original: We used a dataset of 9K training and 1K test examples generated by randomly applying moves to thesolution of the cube.
Revised: We used a dataset of 9K training and 1K test examples generated by randomly applying moves to thesolution of the cube.

Original: Each example is a pair ( V I , V O ) where V I assigns colours to facelets except fortwo corner facelets two edge facelets, and one center facelet, and V O speciﬁes the colours of thoseﬁve missing facelets.
Revised: Each example is a pair ( V I , V O ) where V I assigns colours to facelets except fortwo corner facelets, two edge facelets, and one center facelet, and V O specifies the colours of thosefive missing facelets.

Original: In the test examples, only V I is used.
Revised: In the test examples, only V I is used.

Original: We trained SymSATNet, SATNet-Plain,and SATNet-300aux for 100 epochs, under the same conﬁguration as in the Sudoku case.
Revised: We trained SymSATNet, SATNet-Plain,and SATNet-300aux for 100 epochs, under the same configuration as in the Sudoku case.

Original: The results appear in Figure 1 and Table 1.
Revised: The results appear in Figure 1 and Table 1.

Original: (SymSATNet-Auto will be described below.)

Original: Onaverage, the 100 -epoch training of SymSATNet completed faster in the wall-clock time than thoseof SATNet-Plain and SATNet-300aux.
Revised: On average, the 100 -epoch training of SymSATNetcompleted faster in the wall-clock time than those of SATNet-Plain and SATNet-300aux.

Original: Also, it achieved better test accuracies ( 67 . 2% ) than thesealternatives ( 55 .
Revised: Also, itachieved better test accuracies ( 66 . 9% ) than these alternatives ( 55 .

Original: 7% and 56 .
Revised: 7% and 56 .

Original: Note that unlike Sudoku, the test accuracy of SATNet-300auxwas only marginally better than that of SATNet-Plain, which indicates that both suffered from theoverﬁtting issue.
Revised: Note that unlike Sudoku, the test accuracy of SATNet-300aux was only marginally better than that of SATNet-Plain,which indicates that both suffered from the overfitting issue.

Original: Note also the sharp increase in the training time of SATNet-300aux.
Revised: Note also the sharp increase in thetraining time of SATNet-300aux.

Original: These twoindicate that adding auxiliary variables is not so effective for the completion problem for Rubik’scube, while exploiting symmetries is still useful as indicated by the performance of SymSATNet.
Revised: These two indicate that adding auxiliary variables is not so effectivefor the completion problem for Rubik’s cube, while exploiting symmetries is still useful.

Original: Automatic discovery of symmetries To test the effectiveness of S YM F IND , we tested whether
Revised: Automatic discovery of symmetries To test the effectiveness of S YM F IND , we tested whether

Original: S YM F IND could ﬁnd proper symmetries in Sudoku and Rubik’s cube.
Revised: S YM F IND could find proper symmetries in Sudoku and Rubik’s cube.

Original: Wetrained SATNet-Plainfor 10-epochs in Sudoku and 20-epochs in Rubik’s cube, and applied S YM F IND to the parametermatrices C .
Revised: We applied S YM F IND to theparameter C of SATNet-Plain in T -th training epoch, where T = 10 for Sudoku and T = 20 for Rubik’s cube.

Original: For Sudoku, S YM F IND always recovered the full symmetries with G =
Revised: For Sudoku, S YM F IND always recovered the full symmetries with G =

Original: ( S 3 o S 3 ) ⌦ 341 ( S 3 o S 3 ) ⌦ S 9 in our 10 trials.
Revised: ( S 3 ≀ S 3 ) ⊗( S 3 ≀ S 3 ) ⊗ S 9 in our 10 trials.

Original: For Rubik’s cube, the group of full symmetries is grp( G,  ) for342 G =
Revised: For Rubik’s cube, the group of full symmetries is grp( G, σ ) for G =

Original:  S 2 o S 3 )  ( S 3 o S 8 )  ( S 2 o S 12 )) ⌦ ( S 2 o S 3 ) .
Revised: (( S 2 ≀S 3 ) ⊕ ( S 3 ≀S 8 ) ⊕ ( S 2 ≀S 12 )) ⊗ ( S 2 ≀S 3 ) .

Original: S YM F IND recovered all the parts except S 2 o S 12 .
Revised: S YM F IND recovered all the parts except S 2 ≀S 12 .

Original: 343 Instead of this, the algorithm found S 12 ⌦ S 2 or S 3 ⌦ S 8 , or S 4 ⌦ S 6 in our 10 trials.
Revised: Instead of this, the algorithm found S 12 ⊗ S 2 or S 3 ⊗ S 8 , or S 4 ⊗ S 6 in our 10 trials.

Original: We manually 344 observed that the entries of C in the portion of S 2 o S 12 part were difﬁcult to be clustered, violating 345 the assumption of S YM F IND .
Revised: We manuallyobserved that the entries of C in the corresponding part were difficult to be clustered, violating theassumption of S YM F IND .

Original: This illustrates the fundamental limitation of S YM F IND .
Revised: This illustrates the fundamental limitation of S YM F IND .

Original: To account for the limitation of S YM F IND , we reﬁned the group G of detected symmetries to asubgroup in an additional validation step, before training SymSATNet with those symmetries.
Revised: To account for the limitation of S YM F IND , we refined the group G of detected symmetries to asubgroup in an additional validation step, before training SymSATNet with those symmetries.

Original: Inthe validation step, we check the usefulness of each part G i of the expression of G in our grammar.
Revised: In thevalidation step, we checked the usefulness of each part G i of the expression of G in our grammar.

Original: Concretely, we rewrite G only with the part G i in concern, where all the other parts of G are maskedby the trivial groups
Revised: Concretely, we rewrote G only with the part G i in concern, where all the other parts of G were maskedby the trivial groups

Original: I k .
Revised: I k .

Original: We denote each resulting group of this rewriting by G 0 i . Then, we project Conto the [-space of G 
 i -equivariant matrices-] usingthe corresponding Reynolds operator, and measurethe accuracy of SATNetwith C before and after the projection over validation examples.
Revised: After projecting C with the [-space of G 
 i -equivariant matrices-]masked groups using Reynolds operator, wemeasured the improvement of accuracy of SATNet over validation examples.

Original: Finally,we keep the parts G i that lead to the improvement of validation accuracy larger than a threshold,and assemble such parts to gain G 0 , which is ﬁnally used to train SymSATNet.
Revised: Finally, we assembledonly the parts G i that led to sufficient improvement.

Original: For example, if G =
Revised: For example, if G =

Original: ( Z 3 ⌦ S 4 )  S 5 is discovered by S YM F IND , we consider the parts G 1 =
Revised: ( Z 3 ⊗S 4 ) ⊕S 5 is discoveredby S YM F IND , we consider the parts G 1 =

Original: Z 3 , G 2 = S 4 , and356 G 3 = S 5 .
Revised: Z 3 , G 2 = S 4 , and G 3 = S 5 .

Original: Then we construct three groups G 0 1 ={+′ 
+}
Revised: Then, we construct threemasked groups G {+′ 
+}

Original:  ( Z 3 ⌦ I 4 )  I 5 , G [-0 
-]
Revised: = ( Z 3 ⊗ I 4 ) ⊕ I 5 , G [-0 
-]′ 2 =

Original: = ( I 3 ⌦ S 4 )  I 5 , and357 G 0 
=
Revised: ( I 3 ⊗ S 4 ) ⊕ I 5 , and G ′

Original:  ( I 3 ⌦ I 4 )  S 5 , and measure the accuracy of SATNet with C projected by each G 0 i over358 validation examples.
Revised: = ( I 3 ⊗ I 4 ) ⊕ S 5 , andmeasure the accuracy of SATNet with C projected by each G ′ i over validation examples.

Original: IfG1 and G 0 3 show accuracy improvements greater than a threshold, we combine G 1 and G 3 to form G 0 =
Revised: If G ′ 1and G ′ 3 show accuracy improvements greater than a threshold, we combine G 1 and G 3 to form G ′ =

Original: ( Z 3 ⌦ I 4 )  S 5 , which is then used to train SymSATNet.
Revised: ( Z 3 ⊗ I 4 ) ⊕ S 5 , which is then used to train SymSATNet.

Original: We used 8K training, 1K validation, and 1K test examples to train SymSATNet with symmetriesfound by S YM F IND and the validation step.
Revised: We used 8K training, 1K validation, and 1K test examples to train SymSATNet with symmetriesfound by S YM F IND and the validation step.

Original: We denote these runs by SymSATNet-Auto.
Revised: We denote these runs by SymSATNet-Auto.

Original: For Sudoku, we took a group G discovered from the SATNet-Plain parameter C at the 10 -th trainingepoch, constructed its subgroup G 0 via the validation step, and computed an orthogonal basis B ( G 0 ) s .
Revised: We took agroup G discovered by S YM F IND in T -th training epoch (with the same T above) and constructedits subgroup G ′ via the validation step.

Original: SymSATNet was then trained after being initialised by the projection of C withB ( G 0 ) s .
Revised: SymSATNet was then trained after being initialised by theprojection of C with G ′ .

Original: The sameprocess was applied to Rubik’s cube, but with C taken at the 20 -th training epoch.

Original: The otherconﬁgurations are the same as in our experiments with manually-speciﬁed symmetries.
Revised: The other configurations are the same as before.

Original: As shown in Figure 1 and Table 1, SymSATNet-Auto performed the best for Sudoku ( 99 . 6% ), betterthan even SymSATNet.
Revised: As shown in Figure 1 and Table 1, SymSATNet-Auto performed the best for Sudoku ( 99 . 5% ) and

Original: For Rubik’s cube, SymSATNet-Auto ( 67 . 1% ) also outperformed SATNetPlain and SATNet-300aux.
Revised: Rubik’s cube ( 68 . 1% ) better than even SymSATNet.

Original: During the four trials with Sudoku, SymSATNet-Auto was always giventhe full symmetries in Sudoku.
Revised: During the 10 trials with Sudoku, SymSATNet- Auto was always given the full symmetries in Sudoku.

Original: In two of the four trials with Rubik’s cube, SymSATNet-Auto wasgiven correct subgroups  (( S 2 o S 3 )  ( S 3 o S 8 )  ( I 4 ⌦ I 6 )) ⌦ ( S 2 o S 3 ) and (( S 2 o S 3 )  ( S 3 o ( S 2 o
Revised: For Rubik’s cube, when SymSATNet-Autowas given correct subgroups (e.g., (( S 2 ≀ S 3 ) ⊕ ( S 3 ≀ S 8 ) ⊕ I 4 ⊗ I 6 )) ⊗ ( S 2 ≀ S 3 ) , (( S 2 ≀ S 3 ) ⊕( S 3 ≀

Original:  S 4 ))
Revised: ( S 2 ≀ S 4 ))

Original:  ( I 8 ⌦ I 3 )) ⌦ ( S 2 o S 3 ) , and performed even better than SymSATNet.
Revised: ⊕ ( I 8 ⊗ I 3 )) ⊗ ( S 2 ≀ S 3 ) ), then it performed even better than SymSATNet.

Original: In the other trials,373 SymSATNet-Auto exploited slightly incorrect symmetries, but it outperformed SATNet-Plain and
Revised: In two ofthe 10 trials, slightly incorrect symmetries were exploited, but it outperformed SATNet-Plain and

Original: SATNet-300aux.
Revised: SATNet-300aux.

Original: These results showthat exploiting the partial symmetries of subgroups derived bythe validation step are still useful for learing, even when the symmetries are slightly inaccurate.
Revised: These results show the partial symmetries of subgroups derived by the validationstep are still useful, even when they are slightly inaccurate.

Original: [Figure]
Revised: [Figure]

Original: noisy Sudoku and Rubik’s cube datasets.

Original: Robustness to noise We tested robustness of
Revised: Robustness to noise We tested robustness of

Original: SymSATNet and SymSATNet-Auto to noise bytraining them with noise-corrupted datasets.
Revised: SymSATNet and SymSATNet-Auto to noise bynoise-corrupted datasets.

Original: Wegenerated noisy Sudoku and Rubik’s cube datasetswhere each training example has a randomlychosen cell or facelet corrupted with noise: thenoise alters the value of the cell or the colour ofthe facelet to a random value other than the original.
Revised: We generated noisy Sudoku and Rubik’s cube datasets where each training example is corrupted with noise; it alters thevalue of a random cell or the colour of a randomfacelet to a random value other than the original.

Original: We used noisy datasets with 0-3 corruptedinstances to measure the test accuracy, and repeatedthree runs for each dataset to report the average and95% conﬁdence interval.
Revised: We used noisy datasets with 0-3 corruptedinstances to measure the test accuracy, and tried10 runs for each dataset to report the average and95% confidence interval.

Original: All the other setups arethe same as before.
Revised: All the other setups arethe same as before.

Original: Figure 2 shows the results.
Revised: Figure 2 shows the results.

Original: Inboth problems, SymSATNet was the most robust tonoise, showing remarkably consistent accuracies.
Revised: In both problems, SymSATNet was the most robust, showing remarkably consistent accuracies.

Original: SymSATNet-Auto showed comparable robustnessto SATNet-300aux in noisy Sudoku, but outperformed the two baselines in noisy Rubik’s cube.
Revised: SymSATNet-Auto showed comparable robustnessto SATNet-300aux in noisy Sudoku, but outperformed the two baselines in noisy Rubik’s cube.

Original: Next, to show the robustness of S YM F IND , we applied the algorithm to the problem of restoring apermutation group G from a noise-corrupted Gequivariant symmetric matrix M .
Revised: Next, to show the robustness of S YM F IND , weapplied it to restore permutation groups G fromnoise-corrupted G -equivariant symmetric matrices M .

Original: We pickedfourgroup-permutation pairs ( G i ,  i ) for i 2 [4] where 1 ,  2 ,  3 are random permutations on [15] , [30],[12] , and  4 is the identity permutation on [8] , and
Revised: We picked ( G i , σ i ) for i ∈ [4] where σ 1 , σ 2 ,σ 3 are random permutations on [15] , [30] , [12] ,and σ 4 is the identity permutation on [8] , and

Original: G
Revised: G

Original: [Equation]
Revised: [Equation]

Original: [Equation] Then, we generated grp( G i ,  i ) -equivariant symmetric matrices M i by projecting random matriceswith standard normal entries into the space E (grp( G i ,  i )) s .
Revised: Then, we generated grp( G i , σ i ) -equivariant symmetric matrices M i by projecting random matriceswith standard normal entries into the space E (grp( G i , σ i )) s .

Original: Then, Gaussian noises from N (0 , ! 2 )for ! = 5 ⇥ 10  3 are added to M i ’s entries, and the resulting matrix M [-
-]
Revised: Then, Gaussian noises from N (0 , ω 2 )for ω = 5 × 10 − 3 are added to M i ’s entries, and the resulting matrix M [-
-]′

Original: i is given to S YM F IND .
Revised: i is given to S YM F IND .

Revised: and partial accuracies of S YM F IND for given groups over 1K runs.

Original: [Table]
Revised: [Table]

Original: For each ( G i ,  i ) , we repeatedly generated M 0 iand ran S YM F IND on M [-
-]
Revised: For each ( G i , σ i ) , we repeatedly generated M ′ i andran S YM F IND on M [-
-]′

Original: i for 1K times, andmeasured the portionof cases out of 1 K where S YM F IND recovered ( G i ,  i ) exactly (full accuracy), and also the portion of cases where S YM F IND returned a subgroup of ( G i ,  i ) whichis not the trivial group I m (partial accuracy).
Revised: i for 1K times, and measuredthe portion where S YM F IND recovered ( G i , σ i ) exactly (full accuracy), and also the portion of caseswhere S YM F IND returned a subgroup of ( G i , σ i )which is not the trivial group I m (partial accuracy).

Original: As Table 2 shows, the measured full accuracies werein the range of 59 . 8  88 .
Revised: As Table 2 shows, the measured full accuracieswere in the range of 60 . 3 − 93 .

Original: 7% , and the partial accuracies were in the range of 76 .
Revised: 5% , and the partial accuracies were in the range of 79 .

Original: 7  89 .
Revised: 2 − 94 .

Original: Theseresults show the ability of S YM F IND to recover meaningful and sometimes full symmetries.
Revised: These results show the ability of S YM F IND to recover meaningful and sometimes full symmetries.

Revised:  Conclusion

Revised: We presented SymSATNet, that is capable of exploiting symmetries of the rules or constraints tobe learnt by SATNet.

Original: Checklist
Revised: We also described the S YM F IND algorithm for automatically discoveringsymmetries from the parameter C of the original SATNet at a fixed training epoch, which is based onour empirical observation that symmetries emerge during training as duplicated or similar entries of C . Our experimental evaluations with two rule-learning problems related to Sudoku and Rubik’s cubeshow the benefit of SymSATNet and the promise and limitation of S YM F IND . Although componentsof S YM F IND are motivated by the theoretical analysis of the space of equivariant matrices, such as

Revised: Theorem 2.2, S YM F IND lacks a theoretical justification on its overall performance.

Revised: One interestingfuture direction is to fill in this gap by identifying when symmetries emerge during the training of

Revised: SATNet and proving probabilistic guarantees on when S YM F IND returns correct group symmetries.

Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See Section 3.
Revised: [Yes] See Section 3.

Revised: [Table]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Revised: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: [Table]

Original: using/curating?
Revised: using/curating?

Original: [Yes]
Revised: [Yes]

Original:  (e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
Revised: [Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table] information or offensive content?

Original: [Yes]
Revised: [Yes]

Original: [Table]
Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: ) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: ) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A] [Table]
Revised: [N/A] [Table]