Original: Modeling Transitivity and Cyclicity in Directed Graphs via Binary Code Box Embeddings
Revised: Modeling Transitivity and Cyclicity in Directed Graphs via Binary Code Box Embeddings

Original: Abstract
Revised: Abstract

Original: Modeling directed graphs with differentiable representations is a fundamental requirement for performing machine learning on graph-structured data.
Revised: Modeling directed graphs with differentiable representations is a fundamental requirement for performing machine learning on graph-structured data.

Original: Geometric embedding models (e.g. hyperbolic, cone, and box embeddings) excel at this task, exhibiting useful inductive biases for directed graphs.
Revised: Geometric embedding models (e.g. hyperbolic, cone, and box embeddings) excel at this task, exhibiting useful inductive biases for directed graphs.

Original: However, modeling directed graphs that both contain cycles and some element of transitivity, two properties common in real-world settings, is challenging.
Revised: However, modeling directed graphs that both contain cycles and some element of transitivity, two properties common in real-world settings, is challenging.

Original: Box embeddings, which can be thought of as representing the graph as an intersection over some learned supergraphs, have a natural inductive bias toward modeling transitivity, but (as we prove) cannot model cycles.
Revised: Box embeddings, which can be thought of as representing the graph as an intersection over some learned supergraphs, have a natural inductive bias toward modeling transitivity, but (as we prove) cannot model cycles.

Original: To this end, we propose binary code box embeddings , where a learned binary code selects a subset of graphs for intersection.
Revised: To this end, we propose binary code box embeddings , where a learned binary code selects a subset of graphs for intersection.

Original: We explore several variants, including global binary codes (amounting to a union over intersections) and per-vertex binary codes (allowing greater flexibility) as well as methods of regularization.
Revised: We explore several variants, including global binary codes (amounting to a union over intersections) and per-vertex binary codes (allowing greater ﬂexibility) as well as methods of regularization.

Original: Theoretical and empirical results show that the proposed models not only preserve a useful inductive bias of transitivity but also have sufficient representational capacity to model arbitrary graphs, including graphs with cycles.
Revised: Theoretical and empirical results show that the proposed models not only preserve a useful inductive bias of transitivity but also have sufﬁcient representational capacity to model arbitrary graphs, including graphs with cycles.

Original:  Introduction
Revised:  Introduction

Original: Many real-world networks, such as social media interactions, paper citations, web links and ontologies, are naturally represented as directed graphs [14, 6].
Revised: Many real-world networks, such as social media interactions, paper citations, web links and ontologies, are naturally represented as directed graphs [14, 6].

Original: Two common properties of these graphs are transitivity and cyclicity.
Revised: Two common properties of these graphs are transitivity and cyclicity.

Original: A cycle in a directed graph is a directed path starting from a vertex and traversing back to itself.
Revised: A cycle in a directed graph is a directed path starting from a vertex and traversing back to itself.

Original: For example, “organic matter” → “worm” → “fish” → “cat” → “organic matter” is a food cycle.
Revised: For example, “organic matter” ! “worm” ! “ﬁsh” ! “cat” !

Revised: “organic matter” is a food cycle.

Original: Transitivity in a directed graph is the property that if there exists a directed path from u to v , then edge ( u, v ) also exists.
Revised: Transitivity in a directed graph is the property that if there exists a directed path from u to v , then edge ( u, v ) also exists.

Original: For example, if “cat is mammal” and “mammal is animal” are true, “cat is animal” is true.
Revised: For example, if “cat is mammal” and “mammal is animal” are true, “cat is animal” is true.

Original: In the age of deep learning, it is necessary to determine a way to capture the salient information from a graph via some differentiable parameterization.
Revised: In the age of deep learning, it is necessary to determine a way to capture the salient information from a graph via some differentiable parameterization.

Original: To this end, various graph embedding methods have been proposed.
Revised: To this end, various graph embedding methods have been proposed.

Original: Some work, such as DeepWalk [17] and Node2vec [8], maps each vertex to a vector in Euclidean space.
Revised: Some work, such as DeepWalk [17] and Node2vec [8], maps each vertex to a vector in Euclidean space.

Original: These methods perform a low-rank factorization of the adjacency matrix or the graph Laplacian [18] and are designed to model undirected graphs.
Revised: These methods perform a low-rank factorization of the adjacency matrix or the graph Laplacian [18] and are designed to model undirected graphs.

Original: These can be extended to capture edge asymmetry in directed graphs by using two separate representations - one for source, and one for target - either unconstrained or related with one another via some function.
Revised: These can be extended to capture edge asymmetry in directed graphs by using two separate representations - one for source, and one for target - either unconstrained or related with one another via some function.

Original: It has been proven that using a dot product or distance-based energy function, separate source and target vectors can encode any graph (given sufficient dimension) [1], including graphs with cycles.
Revised: It has been proven that using a dot product or distance-based energy function, separate source and target vectors can encode any graph (given sufﬁcient dimension) [1], including graphs with cycles.

Original: When the source and target representations live in separate spaces, however, relationships which result from following directed paths (e.g., transitivity) are harder to encode.
Revised: When the source and target representations live in separate spaces, however, relationships which result from following directed paths (e.g., transitivity) are harder to encode.

Original: For example, given edge ( i, j ) and ( j, k ) , a vector model can learn to make | s i − t
Revised: For example, given edge ( i, j ) and ( j, k ) , a vector model can learn to make | s i  t

Original: j | 2 ≈ 0 and | s j − t k | 2 ≈ 0 , where s i , s j are source vectors and
Revised: j | 2 ⇡ 0 and | s j  t k | 2 ⇡ 0 , where s i , s j are source vectors and

Original: GBC-Box ∪
Revised: GBC-Box ∪

Original: B
Revised: B

Original: D
Revised: D

Original: A
Revised: A

Original: C
Revised: C

Original: E
Revised: E

Original: B
Revised: B

Original: D
Revised: D

Original: A
Revised: A

Original: C
Revised: C

Original: E    5   5   5 B
Revised: E    5   5   5 B

Original: D
Revised: D

Original: A
Revised: A

Original: C
Revised: C

Original: E =
Revised: E =

Original: B
Revised: B

Original: D
Revised: D

Original: A
Revised: A

Original: C
Revised: C

Original: E
Revised: E

Original: B
Revised: B

Original: D
Revised: D

Original: A
Revised: A

Original: C
Revised: C

Original: E ∪ B
Revised: E ∪ B

Original: D
Revised: D

Original: A
Revised: A

Original: C
Revised: C

Original: E ∩    5   5   5 B
Revised: E ∩    5   5   5 B

Original: D
Revised: D

Original: A
Revised: A

Original: C
Revised: C

Original: E = 1 10 

Revised: E = 1 10

Original: Graph
Revised: Graph

Original: Decomposition
Revised: Decomposition

Original: Interval
Revised: Interval

Original: Representations
Revised: Representations

Original: Binary Codes
Revised: Binary Codes

Original: Box
Revised: Box

Original: Subspaces
Revised: Subspaces

Original: Input
Revised: Input

Original: Output
Revised: Output

Original: D B
Revised: D B

Original: C A
Revised: C A

Original: E
Revised: E

Original: A
Revised: A

Original: B C
Revised: B C

Original: D E
Revised: D E

Original: A B C
Revised: A B C

Original: D E
Revised: D E

Original: A B
Revised: A B

Original: C
Revised: C

Original: D
Revised: D

Original: E
Revised: E

Original: A B
Revised: A B

Original: C
Revised: C

Original: D
Revised: D

Original: E
Revised: E

Original: [Figure]
Revised: [Figure]

Original: t j , t k are target vectors.
Revised: t j , t k are target vectors.

Original: However, the condition | s i − t k | 2 ≈ 0 which would represent the transitive edge ( i, k ) has no encouragement to hold, as the source and target spaces are entirely disconnected.
Revised: However, the condition | s i  t k | 2 ⇡ 0 which would represent the transitive edge ( i, k ) has no encouragement to hold, as the source and target spaces are entirely disconnected.

Original: Transitivity cannot be trivially injected via symbolic rules, e.g., adding all transitive edges to a directed graph.
Revised: Transitivity cannot be trivially injected via symbolic rules, e.g., adding all transitive edges to a directed graph.

Original: This is because most real world edges are not strictly transitive: the degree of transitivity is “soft”, and may hold locally but not globally, or vary for different edge types or sets of nodes in the graph.
Revised: This is because most real world edges are not strictly transitive: the degree of transitivity is “soft”, and may hold locally but not globally, or vary for different edge types or sets of nodes in the graph.

Original: Some work, such as HOPE [15] and APP [29], attempts to capture transitivity by factorizing high-order proximity signals instead of the graph Laplacian.
Revised: Some work, such as HOPE [15] and APP [29], attempts to capture transitivity by factorizing high-order proximity signals instead of the graph Laplacian.

Original: This branch of work is limited by the imperfection of high-order proximity scores (for example, these scores do not model cycles well).
Revised: This branch of work is limited by the imperfection of high-order proximity scores (for example, these scores do not model cycles well).

Original: ATP [25] resolves this by breaking cycles in the graph, accepting a loss of graph information.
Revised: ATP [25] resolves this by breaking cycles in the graph, accepting a loss of graph information.

Original: An alternative approach is to represent nodes in an embedding space with additional geometric structure.
Revised: An alternative approach is to represent nodes in an embedding space with additional geometric structure.

Original: For example, hyperbolic embeddings leverage the negative curvature of hyperbolic space to provably model trees with less distortion [22].
Revised: For example, hyperbolic embeddings leverage the negative curvature of hyperbolic space to provably model trees with less distortion [22].

Original: Region-based embeddings such as box embeddings [28, 11, 3] and hyperbolic entailment cones [13, 7, 21, 9] have a natural bias toward modeling transitivity.
Revised: Region-based embeddings such as box embeddings [28, 11, 3] and hyperbolic entailment cones [13, 7, 21, 9] have a natural bias toward modeling transitivity.

Original: These region-based embeddings can capture the transitivity in a directed graph without relying on high-order proximity scores, using only the original adjacency matrix as supervision.
Revised: These region-based embeddings can capture the transitivity in a directed graph without relying on high-order proximity scores, using only the original adjacency matrix as supervision.

Original: Previous work [2] proved that box embeddings can model any directed acyclic graph (DAG).
Revised: Previous work [2] proved that box embeddings can model any directed acyclic graph (DAG).

Original: A natural question, therefore, is whether box embeddings can capture graphs with cycles.
Revised: A natural question, therefore, is whether box embeddings can capture graphs with cycles.

Original: As we will show in Section 2, this is not the case.
Revised: As we will show in Section 2, this is not the case.

Original: In this work, we propose binary code box embeddings 1 , a generalization of box embeddings to represent arbitrary directed graphs.
Revised: In this work, we propose binary code box embeddings 1 , a generalization of box embeddings to represent arbitrary directed graphs.

Original: The model is motivated by the intuition that a given directed graph can be regarded as a union of multiple sub-graphs, where each sub-graph is acyclic, and therefore can be represented using boxes.
Revised: The model is motivated by the intuition that a given directed graph can be regarded as a union of multiple sub-graphs, where each sub-graph is acyclic, and therefore can be represented using boxes.

Original: We introduce the concept of a “binary code” which selects these sub-graphs.
Revised: We introduce the concept of a “binary code” which selects these sub-graphs.

Original: Our contributions lie in three folds:
Revised: Our contributions lie in three folds:

Original: • We propose global (GBC-B OX ) and per-vertex (VBC-B OX ) binary code box models, a generalization of box embeddings capable of representing arbitrary directed graphs.
Revised: • We propose global (GBC-B OX ) and per-vertex (VBC-B OX ) binary code box models, a generalization of box embeddings capable of representing arbitrary directed graphs.

Original: • We analyze theoretically the limitations of existing box embedding models when representing cycles.
Revised: • We analyze theoretically the limitations of existing box embedding models when representing cycles.

Original: We also prove that, given sufficient dimensions, both binary code box models can model any directed graph.
Revised: We also prove that, given sufﬁcient dimensions, both binary code box models can model any directed graph.

Original: This establishes that, in theory, the representational capacity of these models is not limited.
Revised: This establishes that, in theory, the representational capacity of these models is not limited.

Original: • We evaluate our model on graph reconstruction and link prediction tasks with various synthetic graphs and real world graphs, and observe that our proposed methods perform the best in almost all scenarios, especially when a graph has strong cyclicity and transitivity.
Revised: • We evaluate our model on graph reconstruction and link prediction tasks with various synthetic graphs and real world graphs, and observe that our proposed methods perform the best in almost all scenarios, especially when a graph has strong cyclicity and transitivity.

Original:  Background
Revised:  Background

Original: Given a simple 2 directed graph G with vertices and edges ( V , E ) , we seek to represent the vertices using some mapping ϕ : V → R d , and an energy functionE :
Revised: Given a simple 2 directed graph G with vertices and edges ( V , E ) , we seek to represent the vertices using some mapping  : V ! R d , and an energy function

Original:  V × V 
Revised: E : V ⇥ V !

Original: → R + providing a score (based on ϕ and, perhaps, some hyperparameters λ ) which is interpreted as the negative log probability of edge existence, E( u, v ; ϕ, λ ) 
Revised: R + providing a score (based on  and, perhaps, some hyperparameters  ) which is interpreted as the negative log probability of edge existence, E( u, v ;  ,  ) =

Original: = − log P (( u, v ) ∈ E ) .
Revised:  log P (( u, v ) 2 E ) .

Original: We can view these probabilities as a weighted graph, however in practice it is often necessary to make a hard decision on edge existence, which is done by choosing a (global) threshold T and binarizing the output.
Revised: We can view these probabilities as a weighted graph, however in practice it is often necessary to make a hard decision on edge existence, which is done by choosing a (global) threshold T and binarizing the output.

Original: We denote the energy function for a particular model M with a subscript, i.e. E M , and say that M is capable of modeling a graph
Revised: We denote the energy function for a particular model M with a subscript, i.e. E M , and say that M is capable of modeling a graph

Original: G =
Revised: G =

Original: ( V , E ) if there is some ϕ , λ , and T such that E
Revised: ( V , E ) if there is some  ,  , and T such that E

Original: = { ( u, v ) | E M ( u, v ; ϕ, λ ) < T } .
Revised: = { ( u, v ) | E M ( u, v ;  ,  ) < T } .

Original: 2.1 Boxicity
Revised: 2.1 Boxicity

Original: Let I be the set of closed and bounded intervals in R .
Revised: Let I be the set of closed and bounded intervals in R .

Original: An interval graph is an undirected graph
Revised: An interval graph is an undirected graph

Original: G =
Revised: G =

Original: ( V , E ) such that there exists a mapping φ : V → I for which
Revised: ( V , E ) such that there exists a mapping ' : V ! I for which

Original: [Equation]
Revised: [Equation]

Original: More generally, we can consider B d , the set of d -dimensional “boxes”, which are Cartesian products of intervals,
Revised: More generally, we can consider B d , the set of d -dimensional “boxes”, which are Cartesian products of intervals,

Original: [Equation]
Revised: [Equation]

Original: where x − i and x + i are min and max coordinates in dimension i .
Revised: where x  i and x + i are min and max coordinates in dimension i .

Original: As defined by Roberts [20], the boxicity of an undirected graph G is the smallest dimension d such that there exists a mapping φ : V → B d for which { u, v } ∈ E
Revised: As deﬁned by Roberts [20], the boxicity of an undirected graph G is the smallest dimension d such that there exists a mapping ' : V !

Original:  if and only if φ ( u ) ∩ φ ( v ) ̸ = ∅ .
Revised: B d for which { u, v } 2 E if and only if ' ( u ) \ ' ( v ) 6 = ; .

Original: Equivalently, the boxicity is the minimal number of interval graphs whose intersection is G .
Revised: Equivalently, the boxicity is the minimal number of interval graphs whose intersection is G .

Original: 2.2 Box Embeddings
Revised: 2.2 Box Embeddings

Original: Vilnis et al.
Revised: Vilnis et al.

Original: [28] provide a way of using boxes to represent directed graphs by defining an asymmetric energy function based on box volumes, and subsequent work has introduced various improvements and extensions of this idea Dasgupta et al.
Revised: [28] provide a way of using boxes to represent directed graphs by deﬁning an asymmetric energy function based on box volumes, and subsequent work has introduced various improvements and extensions of this idea Dasgupta et al.

Original: [4], Boratko et al.
Revised: [4], Boratko et al.

Original: In this section we will define the energy function of box embedding models under a common framework, in preparation to motivate our extension to binary code box embeddings.
Revised: In this section we will deﬁne the energy function of box embedding models under a common framework, in preparation to motivate our extension to binary code box embeddings.

Original: The energy function for all box embedding variants has the form
Revised: The energy function for all box embedding variants has the form

Original: [Equation]
Revised: [Equation]

Original: where ϕ ( u ) i are the parameters associated with node u in dimension i and F ( ϕ ( u ) i , ϕ ( v ) i ; λ ) ∈ [0 , 1] is a per-dimension score representing the probability of edge existence.
Revised: where  ( u ) i are the parameters associated with node u in dimension i and F (  ( u ) i ,  ( v ) i ;  ) 2 [0 , 1] is a per-dimension score representing the probability of edge existence.

Original: The model originally defined in Vilnis et al.
Revised: The model originally deﬁned in Vilnis et al.

Original: [28] represented each node using a box, as in (1).
Revised: [28] represented each node using a box, as in (1).

Original: The per-dimension parameters are the endpoints of an interval, ϕ ( u ) i =
Revised: The per-dimension parameters are the endpoints of an interval,  ( u ) i =

Original: [ ϕ ( u ) − i , ϕ ( u ) + i ] , and the score function is defined as
Revised: [  ( u )  i ,  ( u ) + i ] , and the score function is deﬁned as

Original: F Box ([ x − ,x + ] , [ y − , y + ]) =
Revised: F Box ([ [Equation]  ,

Original: | [ x − , x + ] ∩ [ y − , y + ]| |
Revised: [Equation] + ] , [ y  , y + ]

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: This encourages the box for a given vertex to contain (or overlap highly with) the boxes for it’s children.
Revised: This encourages the box for a given vertex to contain (or overlap highly with) the boxes for it’s children.

Original: Boxes which are disjoint or contained in one another can present problems for training, however.
Revised: Boxes which are disjoint or contained in one another can present problems for training, however.

Original: Dasgupta et al.
Revised: Dasgupta et al.

Original: [3] addressed this by modeling the endpoints of intervals using Gumbel random variables.
Revised: [3] addressed this by modeling the endpoints of intervals using Gumbel random variables.

Original: The per-dimension score can be written as
Revised: The per-dimension score can be written as

Original: [Equation]
Revised: [Equation]

Original: where LSE t denotes the following continuous extension of LogSumExp with temperature t ≥ 0 :
Revised: where LSE t denotes the following continuous extension of LogSumExp with temperature t  0 :

Original: [Equation]
Revised: [Equation]

Original: In practice, the temperatures τ and ν are tuned separately as global hyperparameters, however when they are equal the parameters x − , y − (resp.
Revised: In practice, the temperatures ⌧ and ⌫ are tuned separately as global hyperparameters, however when they are equal the parameters x  , y  (resp.

Original: x + , y + ) can be interpreted as the mean of the
Revised: x + , y + ) can be interpreted as the mean of the

Original: GumbelMax (resp.
Revised: GumbelMax (resp.

Original: GumbelMin ) random variables with scale ν = τ , and F G-B OX approximates a ratio of expected box volumes.
Revised: GumbelMin ) random variables with scale ⌫ = ⌧ , and F G-B OX approximates a ratio of expected box volumes.

Original: Note that for any ( x − , x + ) , ( y − , y + ) ∈ R 2 , F G-B OX is continuous with respect to τ, ν ∈ R ≥ 0 , and F G-B OX (( x − , x + ) , ( y − , y + ); 0 , 0) 
Revised: Note that for any ( x  , x + ) , ( y  , y + ) 2 R 2 , F G-B OX is continuous with respect to ⌧ , ⌫ 2 R  0 , and F G-B OX (( x  , x + ) , ( y  , y + ); 0 , 0) =

Original: = F B OX ([ x − , x + ] , [ y − , y + ]) .
Revised: F B OX ([ x  , x + ] , [ y  , y + ]) .

Original: Boratko et al.
Revised: Boratko et al.

Original: [2] takes this one step further, using a mapping which learns 4 parameters per dimension: ϕ ( u ) i =
Revised: [2] takes this one step further, using a mapping which learns 4 parameters per dimension:  ( u ) i =

Original: ( ϕ ( u ) − i , ϕ ( u ) + i , ϕ ( u ) i,τ , ϕ ( u ) i,ν ) .
Revised: (  ( u )  i ,  ( u ) + i ,  ( u ) i, ⌧ ,  ( u ) i, ⌫ ) .

Original: The per-dimension score function is then defined as
Revised: The per-dimension score function is then deﬁned as

Original: F T -B OX (( x − , x + , x τ , x ν ) , ( y − , y 
Revised: F T -B OX (( x  , x + , x ⌧ , x ⌫ ) , ( y  , y + , y ⌧ , y ⌫ )) =

Original: [Equation]
Revised: F G-B OX (( x  , x + ) , ( y  , y + ); ( x ⌧ + y ⌧, x ⌫ + y ⌫)) .

Original:  Existing Representational Capacity and Limitations 3.1 Representational Capacity
Revised:  Existing Representational Capacity and Limitations 3.1 Representational Capacity

Original: We would like to know each model’s representational capacity (the set of graphs capable of being modeled) as well as how this may change depending on hyperparameter settings.
Revised: We would like to know each model’s representational capacity (the set of graphs capable of being modeled) as well as how this may change depending on hyperparameter settings.

Original: It was proven in Boratko et al.
Revised: It was proven in Boratko et al.

Original: [2] that B OX can model any DAG, and of course since T -B OX is equivalent to
Revised: [2] that B OX can model any DAG, and of course since T -B OX is equivalent to

Original: B OX when ϕ ( u ) i,τ = ϕ ( u ) i,ν = 

Revised: B OX when  ( u ) i, ⌧ =  ( u ) i, ⌫ =

Original: this also holds for T -B OX .
Revised: this also holds for T -B OX .

Original: As defined in Section 2, we can also say that G-B OX is capable of modeling any DAG, since it is equivalent to B OX when we set the temperature hyperparameters to zero ( i.e. τ = ν = 0 ), however this is practically quite different – the temperatures are not trainable as in T -B OX , and would never be set to 0 in order to avoid the training difficulties of B OX .
Revised: As deﬁned in Section 2, we can also say that G-B OX is capable of modeling any DAG, since it is equivalent to B OX when we set the temperature hyperparameters to zero (i.e. ⌧ = ⌫ = 0 ), however this is practically quite different – the temperatures are not trainable as in T -B OX , and would never be set to 0 in order to avoid the training difﬁculties of B OX .

Original: Hence, the existing proof of representational capacity in [2] says very little about the practical representational capacity of G-B OX .
Revised: Hence, the existing proof of representational capacity in [2] says very little about the practical representational capacity of G-B OX .

Original: 5 Thankfully, more can be proven.
Revised: 5 Thankfully, more can be proven.

Original: Theorem 1.
Revised: Theorem 1.

Original: Given a threshold T , temperature hyperparameters τ, ν , there exists and a bijectionψ on the set of parameterizations {V → R 2 d } such that for all u, v ∈ V ,
Revised: Given a threshold T , temperature hyperparameters ⌧ , ⌫ , there exists and a bijection on the set of parameterizations {V ! R 2 d } such that for all u, v 2 V ,

Original: [Equation]
Revised: [Equation]

Original: In other words, for any temperature hyperparameters, G-B OX can represent any graph representable by B OX .
Revised: In other words, for any temperature hyperparameters, G-B OX can represent any graph representable by B OX .

Original: For the proof, see Appendix B. Proposition 3 in Boratko et al.
Revised: For the proof, see Appendix B. Proposition 3 in Boratko et al.

Original: [2] states that any DAG can be modeled by B OX in O ((∆ + 2) log |V| ) dimensions with O ( D (∆ + 2) log 2 |V| ) bits of precision per box, where D ≤ |V| is the depth of G and ∆ is the maximum degree.
Revised: [2] states that any DAG can be modeled by B OX in O ((  + 2) log |V| ) dimensions with O ( D (  + 2) log 2 |V| ) bits of precision per box, where D  |V| is the depth of G and  is the maximum degree.

Original: Combining this with Theorem 1 we have the following:
Revised: Combining this with Theorem 1 we have the following:

Original: Corollary 2.
Revised: Corollary 2.

Original: Let G be any DAG.
Revised: Let G be any DAG.

Original: Given any τ, ν ∈ R ≥ 0 , there exists a mapping ϕ : V → R 2 d and a threshold T > 0 such that E G-B OX ( u, v ; ϕ, ( τ, ν )) < 
Revised: Given any ⌧ , ⌫ 2 R  0 , there exists a mapping  : V ! R 2 d and a threshold T > 0 such that E G-B OX ( u, v ;  , ( ⌧ , ⌫ )) < T

Original: T if and only if ( u, v ) ∈ E , where d = O ((∆ + 2) log |V| ) , and ∆ is the maximum degree in G .
Revised: if and only if ( u, v ) 2 E , where d = O ((  + 2) log |V| ) , and  is the maximum degree in G .

Original: In other words, for any setting of temperature hyperparamters, G-B OX can model any DAG.
Revised: In other words, for any setting of temperature hyperparamters, G-B OX can model any DAG.

Original: 3.2 Limitation on Modeling Cycles
Revised: 3.2 Limitation on Modeling Cycles

Original: In this section, we point out that G-B OX cannot model any graph containing a (directed) chordless cycle , which is a cycle such that no two vertices are connected by an edge which does not belong to the cycle.
Revised: In this section, we point out that G-B OX cannot model any graph containing a (directed) chordless cycle , which is a cycle such that no two vertices are connected by an edge which does not belong to the cycle.

Original: Theorem 3.
Revised: Theorem 3.

Original: If E is such that E( u, v ) − E( v, u )
Revised: If E is such that E( u, v )  E( v, u )

Original: =
Revised: =

Original: g ( u ) − g ( v ) for some function g : V → R then it cannot model any graph containing a chordless cycle with more than 2 nodes.
Revised: g ( u )  g ( v ) for some function g : V ! R then it cannot model any graph containing a chordless cycle with more than 2 nodes.

Original: Proof.
Revised: Proof.

Original: Suppose the vertices 1 , 2 , ..., N comprise a chordless cycle, the edges of which are D = { (1 , 2) , (2 , 3) , ..., ( N − 1 , N ) , ( N, 1) } .
Revised: Suppose the vertices 1 , 2 , ..., N comprise a chordless cycle, the edges of which are D = { (1 , 2) , (2 , 3) , ..., ( N  1 , N ) , ( N, 1) } .

Original: Suppose further that we can model the graph containing this cycle using energy E and threshold T .
Revised: Suppose further that we can model the graph containing this cycle using energy E and threshold T .

Original: In particular, we have E( u, v ) <
Revised: In particular, we have E( u, v ) <

Original: T and E( v, u ) ≥ T for ( u, v ) ∈ D .
Revised: T and E( v, u )  T for ( u, v ) 2 D .

Original: This implies that g ( u ) − g ( v ) =
Revised: This implies that g ( u )  g ( v ) =

Original: E( u, v ) − E( v, u ) < 0 , and thus g ( u ) <
Revised: E( u, v )  E( v, u ) < 0 , and thus g ( u ) <

Original: g ( v ) for each ( u, v ) ∈ D .
Revised: g ( v ) for each ( u, v ) 2 D .

Original: Hence g (1) < g (2) < · · · < g ( N ) < g (1) , which is a contradiction.
Revised: Hence g (1) < g (2) < · · · < g ( N ) < g (1) , which is a contradiction.

Original: Corollary 4.
Revised: Corollary 4.

Original: G-B OX cannot model any graph containing a chordless cycle on more than 2 nodes.
Revised: G-B OX cannot model any graph containing a chordless cycle on more than 2 nodes.

Original: Proof.
Revised: Proof.

Original: Theorem 3 applies to E G-B OX with g ( u ) =
Revised: Theorem 3 applies to E G-B OX with g ( u ) =

Original: − log (cid:81) di =1 LSE ν ( ϕ ( u ) + i − ϕ ( u ) − i , 0) .
Revised:  log [Equation]

Original: It is possible to avoid the contradiction in Theorem 3 by the introduction of one reverse edge, and in this case we observe that it is theoretically possible for G-B OX to represent a graph.
Revised: It is possible to avoid the contradiction in Theorem 3 by the introduction of one reverse edge, and in this case we observe that it is theoretically possible for G-B OX to represent a graph.

Original: Proposition 1.
Revised: Proposition 1.

Original: If G is a graph which is the union of a chordless cycle and one reverse edge, G-B OX can model G in 2 dimensions.
Revised: If G is a graph which is the union of a chordless cycle and one reverse edge, G-B OX can model G in 2 dimensions.

Original: The proof of this statement is contained in Appendix C.
Revised: The proof of this statement is contained in Appendix C.

Original:  Method
Revised:  Method

Original: In this section, we will introduce the binary code box embedding concept, which includes a family of models whose shared feature is the use of learned binary codes to select subsets of dimensions.
Revised: In this section, we will introduce the binary code box embedding concept, which includes a family of models whose shared feature is the use of learned binary codes to select subsets of dimensions.

Original: Two we will focus on in particular include GBC-B OX , which uses “global” binary code vectors, and VBC-B OX , which uses per-vertex binary code vectors.
Revised: Two we will focus on in particular include GBC-B OX , which uses “global” binary code vectors, and VBC-B OX , which uses per-vertex binary code vectors.

Original: The following topics will be covered: the motivation for binary codes, the definition of GBC-B OX and VBC-B OX , their representational capacity to model arbitrary directed graphs, our learning objectives and regularization, the models’ inductive biases and strengths, and some discussion about their limitations and alternative variants.
Revised: The following topics will be covered: the motivation for binary codes, the deﬁnition of GBC-B OX and VBC-B OX , their representational capacity to model arbitrary directed graphs, our learning objectives and regularization, the models’ inductive biases and strengths, and some discussion about their limitations and alternative variants.

Original: 4.1 Motivation
Revised: 4.1 Motivation

Original: The idea of binary code boxes is to allow more flexibility than simply taking an intersection over interval graphs, as captured by boxicity (see Section 2.1).
Revised: The idea of binary code boxes is to allow more ﬂexibility than simply taking an intersection over interval graphs, as captured by boxicity (see Section 2.1).

Original: Recall, in the undirected case, the boxicity of a graph G =
Revised: Recall, in the undirected case, the boxicity of a graph G =

Original: ( V , E ) was equivalent to the smallest number d such that for some set of interval graphs S
Revised: ( V , E ) was equivalent to the smallest number d such that for some set of interval graphs S

Original: = { G i =
Revised: = { G i =

Revised: [Equation]

Original: ( V , Ei
Revised: V , E

Original: [Equation]
Revised: [Equation]

Original: } d i =1 , we have E = ∩ d i =1 E i , i.
Revised: } di =1 , we have E = \ di =1 E i , i.e.

Original: [Equation]
Revised: [Equation]

Original: There are various ways to increase the flexibility of this representation.
Revised: There are various ways to increase the ﬂexibility of this representation.

Original: For example, we could consider allowing a union over intersections by specifying k subsets of S , { S i } ki =1 , for which
Revised: For example, we could consider allowing a union over intersections by specifying k subsets of S , { S i } ki =1 , for which

Original: [Equation]
Revised: [Equation]

Original: To allow for even greater flexibility, we could allow each vertex to select a subset of graphs to intersect.
Revised: To allow for even greater ﬂexibility, we could allow each vertex to select a subset of graphs to intersect.

Original: Formally, we allow the specification of a functionψ : V → 2 S which assigns a subset of interval graphs to each vertex, for which
Revised: Formally, we allow the speciﬁcation of a function : V ! 2 S which assigns a subset of interval graphs to each vertex, for which

Original: [Equation]
Revised: [Equation]

Original: For the undirected case the advantage is minimal.
Revised: For the undirected case the advantage is minimal.

Original: Increasing the flexibility in these ways may allow us to represent an undirected graph in smaller dimension (or, equivalently, using a smaller number of interval graphs), however as mentioned previously we know any undirected graph can be represented as an intersection of interval graphs.
Revised: Increasing the ﬂexibility in these ways may allow us to represent an undirected graph in smaller dimension (or, equivalently, using a smaller number of interval graphs), however as mentioned previously we know any undirected graph can be represented as an intersection of interval graphs.

Original: For directed graphs, however, this is not the case, and (as we prove in Section 4.3) this generalization allows for any directed graph to be represented.
Revised: For directed graphs, however, this is not the case, and (as we prove in Section 4.3) this generalization allows for any directed graph to be represented.

Original: 4.2 Definition
Revised: 4.2 Deﬁnition

Original: In order to capture the idea of a “union of intersections” specified in (6) we consider learning k “binary code” vectors b j ∈ [0 , 1] d .
Revised: In order to capture the idea of a “union of intersections” speciﬁed in (6) we consider learning k “binary code” vectors b j 2 [0 , 1] d .

Original: Each binary vector corresponds to a selection of which dimensions to include - if the i th component is 0 the scores for edges in this dimension should be ignored, and if it is 1 they should be included.
Revised: Each binary vector corresponds to a selection of which dimensions to include - if the i th component is 0 the scores for edges in this dimension should be ignored, and if it is 1 they should be included.

Original: For convenience, we will represent these as the columns of a d × k matrix B ∈ [0 , 1] d × k .
Revised: For convenience, we will represent these as the columns of a d ⇥ k matrix B 2 [0 , 1] d ⇥ k .

Original: The energy function in this case isE [Equation] OX
Revised: The energy function in this case is

Original: [Equation]
Revised: [Equation]

Original: v

Original: [Equation]

Original: B ) [Equation] [Equation] In order to capture the notion of per-vertex subset selection in (7), we learn 3 parameters per dimension, which we denote as ϕ ( u ) i=
Revised: In order to capture the notion of per-vertex subset selection in (7), we learn 3 parameters per dimension, which we denote as  ( u ) i

Original:  ( ϕ ( u ) − i , ϕ ( u ) + i , ϕ ( u ) ⋄ i ) ∈ R × R × [0 , 1] .
Revised: = (  ( u )  i ,  ( u ) + i ,  ( u ) ⇧ i ) 2 R ⇥ R ⇥ [0 , 1] .

Original: The binary code ϕ ( u ) ⋄ i indicates whether this dimension should be taken into account when calculating the probability of edge existence for edges involving this node - if it is 0 , the dimension should be ignored, and if it is 1 it may be included.
Revised: The binary code  ( u ) ⇧ i indicates whether this dimension should be taken into account when calculating the probability of edge existence for edges involving this node - if it is 0 , the dimension should be ignored, and if it is 1 it may be included.

Original: We incorporate this at the level of the per-dimension score function as follows:
Revised: We incorporate this at the level of the per-dimension score function as follows:

Original: [Equation]
Revised: [Equation]

Original: Using the product in the exponent is a relaxation of the intersectionψ ( u ) ∩ ψ ( v ) from (7).
Revised: Using the product in the exponent is a relaxation of the intersection ( u ) \ ( v ) from (7).

Original: When computing the probability of an edge ( u, v ) , the binary codes can learn to ignore certain dimensions by making ϕ ( u ) ⋄ i or ϕ ( v ) ⋄ i equal to 0 .
Revised: When computing the probability of an edge ( u, v ) , the binary codes can learn to ignore certain dimensions by making  ( u ) ⇧ i or  ( v ) ⇧ i equal to 0 .

Original: In the following, we point out two perspectives which provide further intuition behind these models:
Revised: In the following, we point out two perspectives which provide further intuition behind these models:

Original: Generalization: Both GBC-B OX and VBC-B OX are a generalization of G-B OX , as the energy function is equivalent when all binary codes are 1 , in which case all dimensions are used for volume calculation.
Revised: Generalization: Both GBC-B OX and VBC-B OX are a generalization of G-B OX , as the energy function is equivalent when all binary codes are 1 , in which case all dimensions are used for volume calculation.

Original: As we will show in Section 4.3 these models are strictly more expressive, as when some binary codes are less than 1 these models can represent more complex graphs.
Revised: As we will show in Section 4.3 these models are strictly more expressive, as when some binary codes are less than 1 these models can represent more complex graphs.

Original: Projection: For GBC-B OX , if B i,j ∈ { 0 , 1 } we can think of this defining a set of projections { P j } kj =1 where P j projects the boxes parameterized by ϕ into the (cid:80) di =1 B i,j dimensional subspace where B i,j = 1 .
Revised: Projection: For GBC-B OX , if B i,j 2 { 0 , 1 } we can think of this deﬁning a set of projections { P j } kj =1 where P j projects the boxes parameterized by  into the P di =1 B i,j dimensional subspace where B i,j = 1 .

Original: Similarly, for VBC-B OX , if ϕ ( u ) ⋄ i ∈ { 0 , 1 } we can think of this determining the dimensions D u :=
Revised: Similarly, for VBC-B OX , if  ( u ) ⇧ i 2 { 0 , 1 } we can think of this determining the dimensions D u :=

Original: { i : ϕ ( u ) ⋄ i = 1 } which the box will be projected in.
Revised: { i :  ( u ) ⇧ i = 1 } which the box will be projected in.

Original: Given an edge ( u, v ) , we project into dimensions D u ∩ D v before determining the edge existence.
Revised: Given an edge ( u, v ) , we project into dimensions D u \ D v before determining the edge existence.

Original: 4.3 Representational Capacity
Revised: 4.3 Representational Capacity

Original: The energy functions for GBC-B OX and VBC-B OX were constructed such that Theorem 3 would not apply, thus making it possible that they may be capable of representing some graphs with cycles.
Revised: The energy functions for GBC-B OX and VBC-B OX were constructed such that Theorem 3 would not apply, thus making it possible that they may be capable of representing some graphs with cycles.

Original: In this section, we prove that both can model any directed graph.
Revised: In this section, we prove that both can model any directed graph.

Original: Theorem 5.
Revised: Theorem 5.

Original: Given a directed graph G =
Revised: Given a directed graph G =

Original: ( V , E ) and any τ, ν ≥ 0 and k ≥ 2 , there exists a threshold
Revised: ( V , E ) and any ⌧ , ⌫  0 and k  2 , there exists a threshold

Original: T > 0 , parameters ϕ : V → R 2 d , and binary codes B ∈ [0 , 1] d × k for which
Revised: T > 0 , parameters  : V ! R 2 d , and binary codes B 2 [0 , 1] d ⇥ k for which

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Given a directed graph G =
Revised: Given a directed graph G =

Original: ( V , E ) , let ( <, V ) be an arbitrary strict total order on the vertices.
Revised: ( V , E ) , let ( <, V ) be an arbitrary strict total order on the vertices.

Original: Then define subgraphs D [-
-] =
Revised: Then deﬁne subgraphs D [-
-]1 =

Original: ( V , E 1 ) and D 

Revised: ( V , E 1 ) and D

Original: = ( V , E 2 ) where E 1 = { ( u, v ) | u < v } ∩ E and
Revised: = ( V , E 2 ) where E 1 = { ( u, v ) | u < v } \ E and

Original: E 2 = { ( u, v ) | u > v } ∩ E .
Revised: E 2 = { ( u, v ) | u > v } \ E .

Original: Observe that D 1 and D 2 are directed acyclic graphs, and thus Corollary 2 implies that for j ∈ { 1 , 2 } there exists a threshold T j , dimension d j = O ((∆ + 2) log |V| ) , and mapping ϕ
Revised: Observe that D 1 and D 2 are directed acyclic graphs, and thus Corollary 2 implies that for j 2 { 1 , 2 } there exists a threshold T j , dimension d j = O ((  + 2) log |V| ) , and mapping  j : V !

Original: j : V → R 2{+
+} d j such that
Revised: R {+
+} d j such that

Original: [Equation]
Revised: [Equation]

Original: Let d = d 1 + d 2 , T = min( T 1 , T 2 ) , and define ϕ : V → R 2 d and B ∈ [0 , 1] d × k as follows:
Revised: Let d = d 1 + d 2 , T = min( T 1 , T 2 ) , and deﬁne  : V ! R 2 d and B 2 [0 , 1] d ⇥ k as follows:

Original: [Equation]
Revised: [Equation]

Original: Then we have E GBC-B OX ( u, v ; ( ϕ, B ) , ( τ, ν )) =
Revised: Then we have E GBC-B OX ( u, v ; (  , B ) , ( ⌧ , ⌫ )) =

Original: min j ∈{ 1 , 2 } TT j E G-B OX ( u, v ; ϕ j , ( τ, ν )) which completes the proof with k = 2 , and therefore implies the result for all k > 2 .
Revised: min j 2 { 1 , 2 } TT j E G-B OX ( u, v ;  j , ( ⌧ , ⌫ )) which completes the proof with k = 2 , and therefore implies the result for all k > 2 .

Original: While motivated by a similar idea, note that VBC-B OX is not a generalization of GBC-B OX , and thus an independent proof of representational capacity is required.
Revised: While motivated by a similar idea, note that VBC-B OX is not a generalization of GBC-B OX , and thus an independent proof of representational capacity is required.

Original: Theorem 6.
Revised: Theorem 6.

Original: Given a directed graph G =
Revised: Given a directed graph G =

Original: ( V , E ) and any τ, ν ≥ 0 , there exists a threshold T > 0 and parameters ϕ : V → R 2 d × [0 , 1] d for which E VBC-B OX ( u, v ; ϕ, ( τ, ν ))<
Revised: ( V , E ) and any ⌧ , ⌫  0 , there exists a threshold T > 0 and parameters  : V ! R 2 d ⇥ [0 , 1] d for which E VBC-B OX ( u, v ;  , ( ⌧ , ⌫ ))

Original:  T if and only if ( u, v ) ∈ E .
Revised: < T if and only if ( u, v ) 2 E .

Original: Proof.
Revised: Proof.

Original: Given a graph G =
Revised: Given a graph G =

Original: ( V , E ) let H = {{ u, v } | u, v ∈ V , u ̸ = v } .
Revised: ( V , E ) let H = {{ u, v } | u, v 2 V , u 6 = v } .

Original: We will construct a VBCB OX model in d = | H | dimensions.
Revised: We will construct a VBC- B OX model in d = | H | dimensions.

Original: For convenience, index the dimensions using h ∈ H .
Revised: For convenience, index the dimensions using h 2 H .

Original: Then let ϕ ( u ) ⋄ h
Revised: Then let  ( u ) ⇧ h

Original: = 1 if u ∈ h , and 0 otherwise.
Revised: = 1 if u 2 h , and 0 otherwise.

Original: This means when evaluating the edge ( u, v ) or ( v, u ) we simply need to compare in the 1-d space obtained by projecting the boxes to dimension h
Revised: This means when evaluating the edge ( u, v ) or ( v, u ) we simply need to compare in the 1-d space obtained by projecting the boxes to dimension h

Original: = { u, v } , and furthermore that this dimension will not be used when considering any other edges.
Revised: = { u, v } , and furthermore that this dimension will not be used when considering any other edges.

Original: Any directed graph on 2 nodes can be embedded using boxes in 1-dimension (observable by direct construction), which completes the proof.
Revised: Any directed graph on 2 nodes can be embedded using boxes in 1-dimension (observable by direct construction), which completes the proof.

Original: While Theorem 5 and Theorem 6 are helpful in establishing that, unlike all prior box embedding models and many other geometric embeddings, GBC-B OX and VBC-B OX are capable of modeling any directed graph, the implied dimensionality bounds are far from optimal.
Revised: While Theorem 5 and Theorem 6 are helpful in establishing that, unlike all prior box embedding models and many other geometric embeddings, GBC-B OX and VBC-B OX are capable of modeling any directed graph, the implied dimensionality bounds are far from optimal.

Original: In general, both models tend to require fewer dimensions than alternatives, which we analyze theoretically in Appendix D and observe empirically in Section 5.
Revised: In general, both models tend to require fewer dimensions than alternatives, which we analyze theoretically in Appendix D and observe empirically in Section 5.

Original: 4.4 Learning
Revised: 4.4 Learning

Original: We fit geometric embeddings by optimizing a binary cross entropy objective.
Revised: We ﬁt geometric embeddings by optimizing a binary cross entropy objective.

Original: Given some edges in a training set T , the loss is defined as
Revised: Given some edges in a training set T , the loss is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: ) (cid:17) (cid:21) (10) where N ( u, v ) is the set of negative samples for each positive edge ( u, v ) within one batch.
Revised: where N ( u, v ) is the set of negative samples for each positive edge ( u, v ) within one batch.

Original: We sample minibatches of positive edges in T , and for each positive edge we sample 32 edges not in
Revised: We sample minibatches of positive edges in T , and for each positive edge we sample 32 edges not in

Original: T by randomly corrupting either the source or target node.
Revised: T by randomly corrupting either the source or target node.

Original: We also use a self-adversarial negative weight, as described in [26].
Revised: We also use a self-adversarial negative weight, as described in [26].

Original: 4.5 Limitations and Regularization
Revised: 4.5 Limitations and Regularization

Original: There are a few limitations of Binary Code Box Embeddings: 1) Transitivity and Flexibility In order to model cycles with separate sub-spaces, the inductive bias of asymmetric transitivity might be weakened.
Revised: There are a few limitations of Binary Code Box Embeddings: 1) Transitivity and Flexibility In order to model cycles with separate sub-spaces, the inductive bias of asymmetric transitivity might be weakened.

Original: 2) Inefficiency of Parameterization For VBC-B OX , the number of parameters is increased by 50% compared with a G-B OX model in equal dimension.
Revised: 2) Inefﬁciency of Parameterization For VBC-B OX , the number of parameters is increased by 50% compared with a G-B OX model in equal dimension.

Original: In addition, during inference, a large portion of box co-ordinates are “dead” when binary codes are near zero.
Revised: In addition, during inference, a large portion of box co-ordinates are “dead” when binary codes are near zero.

Original: Here, we introduce a regularization method and a tunable binary code size to resolve these concerns.
Revised: Here, we introduce a regularization method and a tunable binary code size to resolve these concerns.

Original: Regularization We can regularize the sparsity of binary codes to penalize dimension drop-off using the lasso with a regularization weight w r , leading to a loss function
Revised: Regularization We can regularize the sparsity of binary codes to penalize dimension drop-off using the lasso with a regularization weight w r , leading to a loss function

Original: L = L BCE + w r ∥ 1 − B ∥ ℓ 1 .
Revised: L = L BCE + w r k 1  B k ` 1 .

Original: Restricted Binary Code Size We can constrain the number of trainable binary codes to the last d bin dimensions, setting the first d − d bin dimensions to 1 .
Revised: Restricted Binary Code Size We can constrain the number of trainable binary codes to the last d bin dimensions, setting the ﬁrst d  d bin dimensions to 1 .

Original: Both w r and d bin can provide a handle on mitigating issues mentioned above.
Revised: Both w r and d bin can provide a handle on mitigating issues mentioned above.

Original: One can increase w r or decrease d bin to preserve more transitivity and make full use of the model’s parameters.
Revised: One can increase w r or decrease d bin to preserve more transitivity and make full use of the model’s parameters.

Original:  Experiments 5.1 Graph Reconstruction
Revised:  Experiments 5.1 Graph Reconstruction

Original: While GBC-B OX and VBC-B OX can provably model any directed graph, the extent to which they can be trained to do so via gradient descent is another matter.
Revised: While GBC-B OX and VBC-B OX can provably model any directed graph, the extent to which they can be trained to do so via gradient descent is another matter.

Original: In this section, we compare the reconstruction performance of various geometric embedding methods on a number of synthetic graphs, including simple directed cycles, trees and scale-free networks.
Revised: In this section, we compare the reconstruction performance of various geometric embedding methods on a number of synthetic graphs, including simple directed cycles, trees and scale-free networks.

Original: Baselines.
Revised: Baselines.

Original: We compare our model with different baselines:
Revised: We compare our model with different baselines:

Original: Vector* : We implement a vector baseline where each node is parameterized by a source and target vector, and the energy function is measured by E( u, v ) = − log σ ( ϕ ( u ) source · ϕ ( v ) target ) .
Revised: Vector* : We implement a vector baseline where each node is parameterized by a source and target vector, and the energy function is measured by E( u, v ) =  log  (  ( u ) source ·  ( v ) target ) .

Original: * indicates it uses source and target vectors.
Revised: * indicates it uses source and target vectors.

Original: Lorentzian : It has been shown that hyperbolic space can embed undirected trees with arbitrarily low distortion [22], therefore we also compare with the baseline of squared Lorentzian distance on the hyperboloid [10, 2].
Revised: Lorentzian : It has been shown that hyperbolic space can embed undirected trees with arbitrarily low distortion [22], therefore we also compare with the baseline of squared Lorentzian distance on the hyperboloid [10, 2].

Original: Hyperbolic Entailment Cones : Ganea et al.
Revised: Hyperbolic Entailment Cones : Ganea et al.

Original: [7] model vertices as cones in hyperbolic space, combining the bias of hyperbolic space to represent tree-like graphs with the transitivity bias of region-based representations.
Revised: [7] model vertices as cones in hyperbolic space, combining the bias of hyperbolic space to represent tree-like graphs with the transitivity bias of region-based representations.

Original: Box Models : We also compare with G-B OX [4] and T -B OX [2], as defined in Section 2.2.
Revised: Box Models : We also compare with G-B OX [4] and T -B OX [2], as deﬁned in Section 2.2.

Original: We use Bayesian hyperparameter tuning based on the optimal F1 score for reconstruction.
Revised: We use Bayesian hyperparameter tuning based on the optimal F1 score for reconstruction.

Original: Capacity over Cycles.
Revised: Capacity over Cycles.

Original: We evaluate each model’s capacity to represent cycles, where simple directed cycles are generated with an increasing number of vertices ( |V| = 2 2 , 2 4 , 2 8 , 2 12 ).
Revised: We evaluate each model’s capacity to represent cycles, where simple directed cycles are generated with an increasing number of vertices ( |V| = 2 2 , 2 4 , 2 8 , 2 12 ).

Original: In addition, we analyze the effect of adding one reverse edge to the cycle, which conventional box embedding models can represent.
Revised: In addition, we analyze the effect of adding one reverse edge to the cycle, which conventional box embedding models can represent.

Original: Results are shown in Table 1.
Revised: Results are shown in Table 1.

Original: For a fair comparison, all methods useparameters per vertex 6 .
Revised: For a fair comparison, all methods useparameters per vertex 6 .

Original: VBC-B OX shows the best reconstruction performance.
Revised: VBC-B OX shows the best reconstruction performance.

Original: Most other geometric baselines cannot model cycles.
Revised: Most other geometric baselines cannot model cycles.

Original: Surprisingly, VBC-B OX even outperforms the Vector* baseline when |V| = 2 12 , indicating our model’s high expressivity and surprising ease of training.
Revised: Surprisingly, VBC-B OX even outperforms the Vector* baseline when |V| = 2 12 , indicating our model’s high expressivity and surprising ease of training.

Original: Results also show that, in concordance with Proposition 1, G-B OX can model a cycle containing a reverse edge when |V| is small, though it may struggle to do so.
Revised: Results also show that, in concordance with Proposition 1, G-B OX can model a cycle containing a reverse edge when |V| is small, though it may struggle to do so.

Original: We also see that T -B OX can model certain cycles when |V| is small, which is not a contradiction of Theorem 3 as the energy function for T -B OX does not satisfy the premise.
Revised: We also see that T -B OX can model certain cycles when |V| is small, which is not a contradiction of Theorem 3 as the energy function for T -B OX does not satisfy the premise.

Original: [Table]
Revised: [Table]

Original: Capacity over Trees.
Revised: Capacity over Trees.

Original: It is known that box models have an inductive bias which makes them more suited to capturing asymmetric transitivity, whereas hyperbolic space is particularly suitable for undirected trees.
Revised: It is known that box models have an inductive bias which makes them more suited to capturing asymmetric transitivity [Table] whereas hyperbolic space is particularly suitable for undirected trees.

Original: In this experiment
Revised: In this experiment

Original: [Table]
Revised: [Table]

Original: we evaluate whether binary code models maintain the inductive bias of box embeddings.
Revised: we evaluate whether binary code models maintain the inductive bias of box embeddings.

Original: We generated four balanced (out-)trees, each with 2 13 vertices and branching factor of [2 , 3 , 5 , 10] .
Revised: We generated four balanced (out-)trees, each with 2 13 vertices and branching factor of [2 , 3 , 5 , 10] .

Original: We also generated another four graphs with full transitive closures.
Revised: We also generated another four graphs with full transitive closures.

Original: We compare each method with 12, 24 and 48 parameters per vertex.
Revised: We compare each method with 12, 24 and 48 parameters per vertex.

Original: For each setting, we average the results over the four trees
Revised: For each setting, we average the results over the four trees

Original: [Table]
Revised: [Table]

Original: and present the results in Table 2.
Revised: and present the results in Table 2.

Original: As expected, G-B OX performs the best on transitively closed trees, while the Lorentzian model performs well on balanced trees.
Revised: As expected, G-B OX performs the best on transitively closed trees, while the Lorentzian model performs well on balanced trees.

Original: It is shown that
Revised: It is shown that

Original: GBC-B OX performs equally well as G-B OX on transitively closed trees, while the performance of
Revised: GBC-B OX performs equally well as G-B OX on transitively closed trees, while the performance of

Original: VBC-B OX is slightly lower.
Revised: VBC-B OX is slightly lower.

Original: In contrast, the latter performs similarly or better than the former on the transitive reduction of trees.
Revised: In contrast, the latter performs similarly or better than the former on the transitive reduction of trees.

Original: This suggests that more transitive bias is preserved in GBC-B OX , while per vertex binary codes provide more representational flexibility.
Revised: This suggests that more transitive bias is preserved in GBC-B OX , while per vertex binary codes provide more representational ﬂexibility.

Original: In addition, these results confirm a trend observed in [2], where binary code models and and T -B OX outperform the Lorentzian model on balanced trees when dimension size is increased, and the performance of other geometric-based embeddings are not capable of overfitting even as dimension increases.
Revised: In addition, these results conﬁrm a trend observed in [2], where binary code models and and T -B OX outperform the Lorentzian model on balanced trees when dimension size is increased, and the performance of other geometric-based embeddings are not capable of overﬁtting even as dimension increases.

Original: Capacity over Random Graphs.
Revised: Capacity over Random Graphs.

Original: Finally, we conduct experiments on scale-free networks, a simulation to real-world graphs, where the edge distribution follows preferential attachment.
Revised: Finally, we conduct experiments on scale-free networks, a simulation to real-world graphs, where the edge distribution follows preferential attachment.

Original: In order to analyze how the cyclicity of graphs affects each model’s performance, we randomly sampled
Revised: In order to analyze how the cyclicity of graphs affects each model’s performance, we randomly sampled

Original: [Table]
Revised: [Table]

Original: nearly three-thousand graphs using a wide range of parameters used for graph generation.
Revised: nearly three-thousand graphs using a wide range of parameters used for graph generation.

Original: Then we split the generated graphs into five bins by our proposed measure cyclicity : The proportion of vertices in a given graph involved in at least one cycle.
Revised: Then we split the generated graphs into ﬁve bins by our proposed measure cyclicity : The proportion of vertices in a given graph involved in at least one cycle.

Original: In order to analyze models’ effectiveness of modeling cycles instead of the density of graphs, we randomly sample 5 graphs from each bin where the average degree is in the range between 3 and 4.
Revised: In order to analyze models’ effectiveness of modeling cycles instead of the density of graphs, we randomly sample 5 graphs from each bin where the average degree is in the range between 3 and 4.

Original: Results are shown in Figure 2.
Revised: Results are shown in Figure 2.

Original: From the chart
Revised: From the chart

Original: [Table]
Revised: [Table]

Original: we can see that our proposed model outperforms standard G-B OX in all scenarios.
Revised: we can see that our proposed model outperforms standard G-B OX in all scenarios.

Original: Moreover, the gap in performance is more significant when cyclicity is high.
Revised: Moreover, the gap in performance is more signiﬁcant when cyclicity is high.

Original: Figure 2 also clearly shows that
Revised: Figure 2 also clearly shows that

Original: VBC-B OX provides more representational capacity overall, while GBC-B OX is less expressive in modeling cyclic graphs
Revised: VBC-B OX provides more representational capacity overall, while GBC-B OX is less expressive in modeling cyclic graphs

Original: [Table]
Revised: [Table]

Original:  , GBC-B OX and VBC-B OX using 24, 48, 96 parameters per vertex.
Revised: OX , GBC-B OX and VBC-B OX using 24, 48, 96 parameters per vertex.

Original: 5.2 Link Prediction
Revised: 5.2 Link Prediction

Original: Finally, we apply our binary code box models on link prediction tasks to evaluate the models’ generalization ability.
Revised: Finally, we apply our binary code box models on link prediction tasks to evaluate the models’ generalization ability.

Original: We evaluate on the following real world graphs: Google hyperlink network,
Revised: We evaluate on the following real world graphs: Google hyperlink network,

Original: Epinions trust network, CORA citation network, Twitter network, and DREAM5 gene regulatory networks.
Revised: Epinions trust network, CORA citation network, Twitter network, and DREAM5 gene regulatory networks.

Original: For more data statistics, see Appendix E.
Revised: For more data statistics, see Appendix E.

Original: During training, all hyperparameters are tuned via 10% cross-validation and the test set results are averaged over 10 models of different random seeds, which were trained on the full training set with the best hyper-parameters found during crossvalidation.
Revised: During training, all hyperparameters are tuned via 10% cross-validation and the test set results are averaged over 10 models of different random seeds, which were trained on the full training set with the best hyper-parameters found during crossvalidation.

Original: We also tune the weight of the sparsity regularization w r for all BC-Box models (see
Revised: We also tune the weight of the sparsity regularization w r for all BC-Box models (see

Original: Section 4.5).
Revised: Section 4.5).

Original: In Table 3, we follow [30] and compare with several baselines including vector-based baselines such as DeepWalk [17], LINE [27], node2vec [8], HOPE [15], APP [29], DGGAN [30] and our own implementation of a vector-based model with separate source and target parameters, as well as region-based baselines such as G-B OX [3] and T -B OX [2].
Revised: In Table 3, we follow [30] and compare with several baselines including vector-based baselines such as DeepWalk [17], LINE [27], node2vec [8], HOPE [15], APP [29], DGGAN [30] and our own implementation of a vector-based model with separate source and target parameters, as well as region-based baselines such as G-B OX [3] and T -B OX [2].

Original: Models are evaluated using the Area
Revised: Models are evaluated using the Area

Original: Under ROC Curve (AUC).
Revised: Under ROC Curve (AUC).

Original: For fair comparison, we follow [30] and use 128 parameters per vertex for all our models.
Revised: For fair comparison, we follow [30] and use 128 parameters per vertex for all our models.

Original: Results show that our binary code box models out-perform other baselines in most cases.
Revised: Results show that our binary code box models out-perform other baselines in most cases.

Original: Furthermore, there is an clear trend that VBC-B OX performs the best when graphs are highly cyclic (on the left side of the table), whereas VBC-B OX with fewer binary code dimensions and GBC-B OX model start to perform better when graphs are less cyclic.
Revised: Furthermore, there is an clear trend that VBC-B OX performs the best when graphs are highly cyclic (on the left side of the table), whereas VBC-B OX with fewer binary code dimensions and GBC-B OX model start to perform better when graphs are less cyclic.

Original: In the case where a graph is mostly acyclic (on the right side of the table), G-B OX performs equally well.
Revised: In the case where a graph is mostly acyclic (on the right side of the table), G-B OX performs equally well.

Original: We also observe that box geometry is superior than vector-based models in all scenarios, even if graphs are less transitive, or have a lot of cycles, showing the strength of box geometry in modeling directed graphs.
Revised: We also observe that box geometry is superior than vector-based models in all scenarios, even if graphs are less transitive, or have a lot of cycles, showing the strength of box geometry in modeling directed graphs.

Original: In addition, we also compare with another recent work from Sim et al.
Revised: In addition, we also compare with another recent work from Sim et al.

Original: [23] over
Revised: [23] over

Original: DREAM5 datasets, where we observe that box embedding models out-performs their baselines significantly in most cases.
Revised: DREAM5 datasets, where we observe that box embedding models out-performs their baselines signiﬁcantly in most cases.

Original: (On In Silico dataset, our model has an average precision of 68.8%, out-performs their best result 61.0%.)
Revised: (On In Silico dataset, our model has an average precision of 68.8%, out-performs their best result 61.0%.)

Original: Detailed results are in Appendix F.
Revised: Detailed results are in Appendix F.

Original: [Table]
Revised: [Table]

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we introduced binary code box embeddings, a generalized box embedding method to model directed graphs.
Revised: In this paper, we introduced binary code box embeddings, a generalized box embedding method to model directed graphs.

Original: We provide both theoretical and empirical results showing the capacity of our model for modeling directed graphs.
Revised: We provide both theoretical and empirical results showing the capacity of our model for modeling directed graphs.

Original: We demonstrated that this model can maintain a useful bias of transitivity while also modeling graphs with cycles.
Revised: We demonstrated that this model can maintain a useful bias of transitivity while also modeling graphs with cycles.

Original: Acknowledgements
Revised: Acknowledgements

Original: This work was supported in part by the Center for Data Science and the Center for Intelligent
Revised: This work was supported in part by the Center for Data Science and the Center for Intelligent

Original: Information Retrieval, and in part by the National Science Foundation under Grants No. 2106391 and
Revised: Information Retrieval, and in part by the National Science Foundation under Grants No. 2106391 and

Original: No. 1763618, and in part by IBM Research AI through AI Horizons Network, and in part by the Chan
Revised: No. 1763618, and in part by IBM Research AI through AI Horizons Network, and in part by the Chan

Original: Zuckerberg Initiative under the project “Scientific Knowledge Base Construction”, and the Office of
Revised: Zuckerberg Initiative under the project “Scientiﬁc Knowledge Base Construction”, and the Ofﬁce of

Original: Naval Research (ONR) via Contract No. N660011924032 under Subaward No. 123875727 from the
Revised: Naval Research (ONR) via Contract No. N660011924032 under Subaward No. 123875727 from the

Original: University of Southern California.
Revised: University of Southern California.

Original: The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon.
Revised: The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon.

Original: Some of the work reported here was performed using high performance computing equipment obtained under a grant from the Collaborative R&D Fund managed by the Massachusetts Technology Collaborative.
Revised: Some of the work reported here was performed using high performance computing equipment obtained under a grant from the Collaborative R&D Fund managed by the Massachusetts Technology Collaborative.

Original: Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor.A Equivalence of G-B OX and Dasgupta et al. [4] In our notation, the model in Dasgupta et al. [4] would have score function
Revised: Any opinions, ﬁndings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect those of the sponsor.

Original: [Equation]

Original: where γ is the Euler-Mascheroni constant.

Original: As presented in Dasgupta et al.

Original: [4], D-B OX used a single temperature β

Original: = τ = ν , and derived this score function as an approximation to a ratio of expected volumes of intervals whose endpoints were modeled by Gumbel random variables.

Original: Gumbel random variables are typically parameterized by a location and scale , and Dasgupta et al.

Original: [4] interpreted the parameters x − , y − (resp.

Original: x + , y + ) as the location parameters for GumbelMax (resp.

Original: GumbelMin ) distributions with scale β .

Original: Remark 1.

Original: Although the model was proposed and analyzed in Dasgupta et al.

Original: [4] using a single temperature parameter β , the authors do use separate τ and ν parameters when implementing the model, and so we adopt that formulation when defining F D-B OX above.

Original: We claim D-B OX and G-B OX are equivalent, in the following sense.

Original: Proposition 2.

Original: Given any ν ≥ 0 there exists a bijection ψ on the set of functions { V → R 2 d } such that

Original: [Equation]

Original: and, being a bijection,

Original: [Equation]

Original: [Equation]

Original: Proof.

Original: Observe that for any a, b, c ∈ R and t ≥ 0 ,

Original: [Equation]

Original: +

Original: [Equation]

Original: Then

Original: F G

Original: [Equation]

Original: Therefore, as introduced in Section 2, if we label the output of ϕ using d pairs ϕ ( u ) i =

Original: ( ϕ ( u ) − i , ϕ ( u ) + i ) and define ψ ( ϕ ) to be a mapping from V

Original: → R 2 d such that

Original: [Equation]

Original: the calculations above prove (11), and the proof of (12) is similar.

Original: Remark 2.

Original: Note that the mean of X ∼ GumbelMax( µ, β ) is µ + βγ , and similarly the mean of X ∼ GumbelMin( µ, β ) is µ − βγ .

Original: As mentioned above, in the setting where β

Original: = τ =

Original: ν the parameters of D-B OX can be interpreted as the location parameters for Gumbel distributions, and thus (13) simply takes the location parameters to their mean.

Original: Hence, in the case where τ = ν , the

Original: G-B OX model can simply be interpreted as using the mean of the Gumbel distributions as opposed to the location parameter.

Original: This leads to a slight simplification in the score function by removing the 2 νγ , which has minor computational and mathematical benefits.

Original: The more conceptual benefit, however, is that it unifies B OX , G-B OX , and T -B OX .

Original: B Representational Capacity of G-B OX

Original: In this section we prove any graph capable of being represented by B OX is also capable of being represented by G-B OX , regardless of the temperature hyperparameters.

Original: The proof involves two components:

Original: The representational capacity of G-B OX depends not on the absolute values of τ and ν , but rather their ratio.

Original: The energy of B OX can be approximated by G-B OX using small enough τ and ν .

Original: Proposition 3.

Original: Let τ 1 , ν 1 > 0 and ϕ : V → R 2 d be given.

Original: Then for any τ 2 , ν 2 > 0 such that τ 2 ν 2 = τ 1 νthe function ψ ( ϕ ) : V → R 2 d for which ψ ( ϕ )( u ) ± i =

Original: ν 2 ν 1 ϕ ( u ) ±

Original: i is such that

Original: [Equation]

Original: Proof.

Original: The proof is by direct calculation.

Original: First, note that for any t, c > 0 we have for any vector x ∈ R n

Original: [Equation]

Original: In particular, for c = ν 2 ν 1 = τ 2 τ 1 (where the latter equality follows from the premise of the proposition) we have for any a, b ∈ R

Original: LSE τ 2 ( ca, cb ) =

Original: c LSE τ 1 ( a, b ) and

Original: LSE ν 2 ( ca − cb, 0)

Original: [Equation]

Original: c LSE ν 1 ( a − b, 0) .

Original: Thus

Original: [Equation]

Original: which proves (14)

Original: [Equation]

Original: The following lemma will be helpful in proving the next part.

Original: Lemma 1.

Original: For all y > 0 , given ε > 0 and some M ∈ R , there exists δ > 0 such that for all 0 < ν < δ , for all x < M we have

Original: [Equation]

Original: Proof.

Original: Note that LSE ν ( x, 0) is monotonically increasing in x for any ν ≥ 0 , and is always greater than max( x, 0) .

Original: Furthermore,

Original: [Equation]

Original: as it obtains it’s maximum when x = 0 (which can be observed by inspection of the signs of the derivative).

Original: Then for all x < M we have

Original: [Equation]

Original: Now (15) does not depend on x , and tends to 0 as ν → 0 , which completes the proof.

Original: Proposition 4.

Original: Given a mapping ϕ : V → R 2 d where ϕ ( u ) + i > ϕ ( u ) − i for each u ∈ V , i ∈ { 1 , . . .

Original: , d } , we have that for all u, v ∈ V ,

Original: [Equation]

Original: Proof.

Original: Given fixed x − < x + , y − <

Original: y + , let f ( τ ) =

Original: − LSE τ ( − x + , − y + ) − LSE( x − , y − ) , and z = min( x + , y + ) − max( x − , y − ) =

Original: lim τ

Original: → 0 f ( τ ) .

Original: Then (cid:12)(cid:12) F G-B OX

Original: [Equation]

Original: x

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: y

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: Given ε > 0 , choose δ 1 such that 0 < τ < δ 1 implies the second summand in (17) is bounded by ε/ 2 .

Original: Then f ( τ ) is bounded, and we can apply Lemma 1 to choose δ 2 such that 0 <

Original: ν < δ 2 implies the first summand is less than ε/ 2 .

Original: Thus taking δ = min( δ 1 , δ 2 ) completes the proof on the level of the per-dimension score functions, and thus (16) follows by continuity.

Original: We are now ready to prove the main theorem.

Original: Theorem 1.

Original: Given a threshold T , temperature hyperparameters τ, ν , there exists and a bijection ψ on the set of parameterizations {V → R 2 d } such that for all u, v ∈ V ,

Original: [Equation]

Original: Proof.

Original: Let ε > 0 be a number we will specify later.

Original: [-Then by Proposition 4, for each ( u, v ) ∈ V 
-]
Revised: [-Then by Proposition 4, for each ( u, v ) ∈ V 
-]

Original: we have some δ ( u,v ) > 0 such that

Original: [Equation]

Original: Let

Original: [Equation]

Original: Since τ ′ ν ′

Original: =

Original: τν we can apply Proposition 3, which guarantees the existence of a function ψ such that

Original: [Equation]

Original: Noting that τ ′ , ν ′ ∈ (0 , δ ) , we can combine this with (18), and find

Original: [Equation]

Original: Let

Original: T

Original: [Equation]

Original: and set ε = min( T − T 1 , T 2 − T ) .

Original: Then if ( u, v ) ∈ E we have

Original: E

Original: [Equation]

Original: , v ; ψ

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: and if ( u, v ) / ∈ E we have

Original: E G-B OX ( u, v ; ψ ( ϕ ) , ( τ, ν )) > T 2 − ( T 2 − T )

Original: = T, which completes the proof.

Original: C Representing Cycles with Box Embeddings

Original: Proposition 1 If G is a directed graph which is the union of a chordless cycle and one reverse edge,

Original: G-B OX can model G in 2 dimensions.

Original: Proof.

Original: Given a G = {V , E} , E = { (1 , 2) , (2 , 3) , (3 , 4) , (4 , 5) , , ( N − 1 , N ) , ( N, 1) } ∪ { (1 , N ) }

Original: When N = 2 , it is trivial since two boxes can be equal or overlap with each other largely (as shown in Fig 3a).

Original: When N = 3 , we can construct boxes as shown in Fig 3b, where E B OX (1 , 2)

Original: = − log 0 .

Original: E B OX (2 , 3) =

Original: − log 1 / 3 , E B OX (3 , 1) = − log 0 .

Original: 375 , E B OX (2 , 1) = − log 0 .

Original: 25 , E B OX (3 , 2) =

Original: [Figure]

Original: [Figure]

Original: − log 0 .

Original: 25 , E B OX (1 , 3) = − log 1 . 0 .

Original: Therefore if T = − log 0 .

Original: 3 , they can represent a graph 1 → 2 → 3 ↔ 1 .

Original: When N > 3 and N is an even number, we can construct a 2-d B OX as in Figure 3c.

Original: Let the area of ϕ ( i ) be V ( ϕ ( i )) , the area of intersection box between two nodes that are connected is δ i

Original: = V ( ϕ ( i ) ∩ ϕ

Original: ( i + 1)) , i ≤ N − 1 , and let V ( ϕ ( N ) ∩ ϕ (1))

Original: =

Original: Cδ N − 1 .

Original: We align the boxes as shown in Figure 3c with constant width and decreasing heights such that V ( i + 1)

Original: = αV

Original: ( i ) for some value α .

Original: [-Then for large enough α , there exists arrangement of boxes as shown in Figure 3c such that δ i +
-]
Revised: [-Then for large enough α , there exists arrangement of boxes as shown in Figure 3c such that δ i +
-]

Original: [-= αδ i , i ≤ N − 2 and C = 
-]
Revised: [-= αδ i , i ≤ N − 2 and C = 
-]

Original: α N − 1 .

Original: Apart from boxes N/ 2 and N , this is straightforward, as their decreased heights will already amount to multiplying the size of the intersection by α .

Original: For the boxes on the endpoints, we leverage the fact that we are not constrained to a fixed width to accomplish this.

Original: Then for T = − log δ

Original: [Equation]

Original: ( ϕ

Original: [Equation]

Original: we have

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: and for ( u, v ) / ∈ E , E B OX ( u, v ) ≥ T .

Original: The argument for N odd is similar, using an arrangement as shown in Figure 3d.

Original: Given Appendix B, this proof also applies to G-B OX .

Original: D Dimensionality Bounds for Binary Code Models

Original: We start by proving the following lemma, which implies that any directed graph on 2 nodes can be embedded using boxes in 1 -dimension with any threshold T > 0 .

Original: Lemma 2.

Original: Given T > 0 , there exist x − , x + , y − , y + ∈ R such that

Original: [Equation]

Original: Proof.

Original: The proof is a direct construction; take

Original: [Equation]

Original: Then

Original: [Equation]

Original: and thus

Original: F B OX

Original: [Equation]

Original: e

Original: [Equation]

Original: and F B OX ((0 , e − 2 T ) , (0 , 1))

Original: =

Original: e − 2 T which implies the desired result.

Original: We then strengthen the statement of Theorem 6 to apply to an arbitrary threshold.

Original: Theorem 7.

Original: Given a directed graph G =

Original: ( V , E ) , any temperatures τ, ν ≥ 0 , and any threshold

Original: T > 0 , there exists a parameterization ϕ

Original: : V → R 2 d × [0 , 1] d with d = O ( |V| 2 ) for which

Original: [Equation]

Original: Proof.

Original: Given a graph G =

Original: ( V , E ) let H = {{ u, v } | u, v ∈ V , u ̸ = v } .

Original: We will construct a

Original: VBC-B OX model in d = | H | dimensions.

Original: For convenience, index the dimensions using h ∈ H , and let ϕ ( u ) ⋄ h

Original: [-= 
-]
Revised: [-= 
-]

Original: if u / ∈ h .

Original: Thus when evaluating edge ( u, v ) or ( v, u ) , dimension h = { u, v } is the only dimension whose score may not be equal to 1 .

Original: Lemma 2 implies any graph on 2 nodes can be embedded using threshold T in one dimension, and Theorem 1 implies this is also true for G-B OX for any setting of temperatures, completing the proof.

Original: Theorem 8.

Original: Let G =

Original: ( V , E ) , and let τ, ν ≥ 0 be given temperature hyperparameters.

Original: Let V F be the minimum feedback vertex set, E F = E ∩ V 2 F , and

Original: G F =

Original: ( V F , E F ) .

Original: Then for any temperatures τ, ν there exists a threshold T > 0 and a parameterization ϕ

Original: : V → R 2 d × [0 , 1] d such that

Original: [Equation]

Original: where d = O ((∆ F + 2) log( |V F | )

Original: + |V C | 2 ) , with ∆ F the maximum degree of G F , and

Original: [Equation]

Original: Proof.

Original: Theorem 2 implies that G F can be embedded using G-B OX with the given temperature hyperparameters τ, ν in dimension at most d F

Original: = O ((∆ F + 2) log |V F | ) .

Original: Let ϕ F : V F → R 2 d F be the parameterization for this embedding, and T the threshold on the energy.

Original: Now let

Original: [Equation]

Original: [-, and define E 1 = E F ∪ E ¬ F and G 
-]
Revised: [-, and define E 1 = E F ∪ E ¬ F and G 
-]

Original: = ( V , E 1 ) .

Original: We can extend ϕ F to a VBC-B OX parameterization ϕ 1 : V → R 2 d F × [0 , 1] 2 d F as follows:

Original: [Equation]

Original: otherwise.

Original: This parameterization is such that

Original: [Equation]

Original: Now let E C

Original: =

Original: E ∩ E ¬ F , and note that these edges are only between nodes in V C .

Original: Define G C =

Original: ( V C , E C ) , which, by Theorem 7, can be embedded with threshold T in d C

Original: = O

Original: ( |V C | 2 ) dimensions.

Original: Let ϕ C : V C

Original: → R 2 d C be the associated parameterization.

Original: Now let G 2 =

Original: ( V , E C ∪ ( V F × V F )) , then extend ϕ C to a parameterization ϕ 2 : V → R 2 d C as follows:

Original: [Equation]

Original: = (cid:26) ( ϕ C ( u ) − i , ϕ C ( u ) + i , 1) if u ∈ V C , (0 , 1 , 0) otherwise.

Original: For this parameterization, we have

Original: [Equation]

Original: The desired VBC-B OX embedding ϕ for G with d = d F + d C = O ((∆ F + 2) log( |V F | ) + |V C | 2 ) dimensions now be created by concatenating ϕ 1 and ϕ 2 , for which

Original: [Equation]

Original: Since E F ∩ E C = ∅ , inspecting (19) and (20) we have that

Original: [Equation]

Original: E Data Statistics

Original: Google [16] (15,763 nodes and 171,206 edges) is a hyperlink network from pages within Google’s sites.

Original: Nodes represent pages and directed edges represent hyperlink between pages.

Original: Epinions [19] (75,879 nodes and 508,837 edges) is a trust network from the online social network Epinions.

Original: Nodes represent users and directed edges represent trust between users.

Original: CORA [24] (23,166 nodes and 91,500 edges) is a citation network of academic papers.

Original: Nodes represent papers and directed edges represent the citation relationships between papers.

Original: Twitter [5] (465,017 nodes and 834,797 edges) is a social network.

Original: Nodes represent users and directed edges represent following relationships between users.

Original: DREAM5 [12] is a gene regulatory networks across organisms.

Original: In silico network has 1, nodes and 4,012 edges.

Original: E. Coli network has 1,081 nodes and 2,066 edges.

Original: S. cerevisiae network has 1,994 nodes and 3,940 edges.

Original: These networks contain a relatively small number number of cycles.

Original: F Link Prediction on DREAM5 Datasets

Original: In Table 4, we compared with a recent work that embeds graphs into pseudo-Riemannian manifolds [23], along with other baselines such as Euclidean and Hyperboloid embeddings on DREAM datasets.

Original: Models are evaluated using Average Precision (AP).

Original: Results show that Binary Code Box significantly out-performs baseline methods on In Silico and S. Cerevisiae datasets, while showing competitive performance on the E. Coli graph.

Original: It can also be observed that Gumbel Box performs competitively on these graphs, which are almost acyclic.

Original: [Table]

Original: G Hyper-parameter Search

Original: We follow the setting from Boratko et al.

Original: [2] for hyper-parameter search strategies in graph reconstruction experiments.

Original: Table 5 shows ranges of Bayesian hyper-parameter search for our link prediction experiments.

Original: [Table]

Original: H Case Study

Original: In this section

Original: [Table]

Original: we visualize how binary codes work to preserve transitivity and cyclicity together.

Original: As shown in Figure

Original: [Table]

Original: our analysis is over two synthetic graphs.

Original: For fair comparison, we embed both graphs into 3 dimensional G-B OX and 2 dimensional VBC-B OX .

Original: The graph on the top is formed by a 7-node directed chain (vertex 0 to vertex 6) with full transitive closures and one additional node connecting the chain into a cycle.

Original: It shows that Gumbel Box can model transitive closures well, but cannot model the cycle while Binary Code Box can handle both.

Original: The latter models transitive closures by sharing full box space from node 0 to node

Original: [Table]

Original: and then models the cycle by selecting sub-dimensions in node 6 and 7.

Original: The graph on the bottom is formed by a chain of triangle cycles.

Original: It shows that Gumbel Box cannot handle cycles, and generates an acyclic graph.

Original: In contrast, binary code boxes can handle this graph nicely with much lower errors by alternately switching among sub-spaces within each triangle cycles.

Original: In Fig.

Original: 5, we visualize how GBC-B OX handles cycles and transitive edges.

Original: We compare GBC-B OX with 2 dimensional G-B OX .

Original: Fig.

Original: 5a shows that when representing a DAG, GBC-B OX learns to utilize all dimensions in both binary code vectors and leverages box containment to model edge directions.

Original: Fig.

Original: 5b shows that given a pure cycle, GBC-B OX learns “skinny” boxes to model 0 → 1 , 2 → 3 using the vertical axis, and the rest of the edges in the horizontal axis.

Original: Fig.

Original: 5c shows a more complicated graph and our model also learns to split the graph into 0 → 1 , 1 → 2 , 0 → 2 in the horizontal axis, and 2 → 3 , 3 → 0 , 2 → 0 in the

Original: In comparison, the

Original: G-B struggles with cycles cannot reconstruct ground truth graphs from 5b and 5c

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Figure]