Original: R EPRESENTATION L EARNING WITH M ULTISETS
Revised: R EPRESENTATION L EARNING WITH M ULTISETS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the problem of learning permutation invariant representations that can capture containment relations.
Revised: We study the problem of learning permutation invariant representations that can capture “ﬂexible” notions of containment.

Original: We  propose training a model on a novel task: predicting the size of the symmetric difference  between pairs of multisets, sets which may contain multiple copies of the same object. With motivation from fuzzy set theory, we formulate both multiset representations and how to predict symmetric difference sizes given these representations. We model multiset elements as vectors on the standard simplex and multisets as the summations of such vectors and we predict symmetric difference as the (cid:96) 1 -distance between multiset representations.
Revised: We formalize this problem via a measure theoretic deﬁnition of multisets, and obtain a theoretically-motivated learning model. We propose training this model on a novel task: predicting the size of the symmetric difference (or intersection) between pairs of multisets.

Original: We demonstrate that our representations more effectively predict the sizes of symmetric differences than DeepSets-based approaches with unconstrained object representations. Furthermore, we demonstrate that the model learns meaningful representations, mapping objects of different classes to different standard basis vectors.
Revised: We demonstrate that our model not only performs very well on predicting containment relations (and more effectively predict the sizes of symmetric differences than DeepSetsbased approaches with unconstrained object representations) but that it also learns meaningful representations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Tasks for which the input is an unordered collection, i.e. a set , are ubiquitous and include multipleinstance learning Ilse et al.
Revised: Tasks for which the input is an unordered collection, i.e. a set , are ubiquitous and include multipleinstance learning Ilse et al.

Original: point-cloud classiﬁcation Zaheer et al.
Revised: point-cloud classiﬁcation Zaheer et al.

Original: ; Qi et al.
Revised: ; Qi et al.

Original: (2017), estimating cosmological parameters
Revised: (2017), estimating cosmological parameters

Original: Zaheer et al.
Revised: Zaheer et al.

Original: (2017); Ravanbakhsh et al.
Revised: (2017); Ravanbakhsh et al.

Original: (2016), collaborative ﬁltering Hartford et al.
Revised: (2016), collaborative ﬁltering Hartford et al.

Original: (2018), and relation extraction Verga et al.
Revised: (2018), and relation extraction Verga et al.

Original: Rossiello et al.
Revised: Rossiello et al.

Original: Recent work has demonstrated the beneﬁts of permutation invariant models that have inductive biases well aligned with the set-based input of the tasks (Ilse et al., 2018; Qi et al., 2017; Zaheer et al., 2017;
Revised: Recent work has demonstrated the beneﬁts of permutation invariant models that have inductive biases well aligned with the set-based input of the tasks (Ilse et al., 2018; Qi et al., 2017; Zaheer et al., 2017;

Original: Lee et al., 2019).
Revised: Lee et al., 2019).

Original: Thethe containment relationship between sets — and intersection more generally — is often considered as a measure of relatedness.
Revised: The containment relationship between sets — and intersection more generally — is often considered as a measure of relatedness.

Original: For instance, when comparing the keywords for two documents, we may wish to model that { currency , equilibrium } describes a more speciﬁc set of topics than (i.e. is “contained” in) { money , balance , economics } .
Revised: For instance, when comparing the keywords for two documents, we may wish to model that { currency , equilibrium } describes a more speciﬁc set of topics than (i.e. is “contained” in) { money , balance , economics } .

Original: The containment order is a natural partial order on sets.
Revised: The containment order is a natural partial order on sets.

Original: However, we are often interested not in sets, but multisets , which may contain multiple copies of the same object; examples include bags-of-words, geo-location data over a time period, and data in any multiple-instance learning setting (Ilse et al., 2018).
Revised: However, we are often interested not in sets, but multisets , which may contain multiple copies of the same object; examples include bags-of-words, geo-location data over a time period, and data in any multiple-instance learning setting (Ilse et al., 2018).

Original: The containment order can be extended to multisets.
Revised: The containment order can be extended to multisets.

Original: Learning to represent multisets in a way that respects this partial order is a core representation learning challenge.
Revised: Learning to represent multisets in a way that respects this partial order is a core representation learning challenge.

Original: Note that this may require modeling not just exact containment, but relations that consider the relatedness of individual objects.
Revised: Note that this may require modeling not just exact containment, but relations that consider the relatedness of individual objects.

Original: We may want to learn representations of the multisets’ elements which induce the desired multiset relations.
Revised: We may want to learn representations of the multisets’ elements which induce the desired multiset relations.

Original: In the aforementioned example, we may want money ≈ currency and balance ≈ equilibrium .
Revised: In the aforementioned example, we may want money ≈ currency and balance ≈ equilibrium .

Original: Previous work has considered modeling hierarchical relationships or orderings between pairs of individual items (Ganea et al., 2018; Lai and Hockenmaier, 2017; Nickel and Kiela, 2017; Suzuki et al., 2019; Vendrov et al., 2015; Vilnis et al., 2018; Vilnis and McCallum, 2015; Li et al., 2019;
Revised: Previous work has considered modeling hierarchical relationships or orderings between pairs of individual items (Ganea et al., 2018; Lai and Hockenmaier, 2017; Nickel and Kiela, 2017; Suzuki et al., 2019; Vendrov et al., 2015; Vilnis et al., 2018; Vilnis and McCallum, 2015; Li et al., 2019;

Original: Athiwaratkun and Wilson, 2018).
Revised: Athiwaratkun and Wilson, 2018).

Original: However, this work does not naturally extend from representing individual items to modeling relations between multisets via the elements’ learned representations.
Revised: However, this work does not naturally extend from representing individual items to modeling relations between multisets via the elements’ learned representations.

Original: Furthermore, we may want to consider richer information about the relationship between two multisets beyond containment, such as the size of their intersection.
Revised: Furthermore, we may want to consider richer information about the relationship between two multisets beyond containment, such as the size of their intersection.

Original: In this paper, we present a  method for learning representations of multisets and their elements, given the relationships between pairs of multisets — in particular, we propose to use the sizes of their symmetric differences.
Revised: In this paper, we present a measure-theoretic deﬁnition of multisets, which lets us formally deﬁne the “ﬂexible containment” notion exempliﬁed above. The theory lets us derive method for learning representations of multisets and their elements, given the relationships between pairs of multisets — in particular, we propose to use the sizes of their symmetric differences or of their intersections.

Original: We learn these representations with the goal of predicting the relationships between unseen pairs of multisets (whose elements may themselves have been unseen during training).
Revised: We learn these representations with the goal of predicting the relationships between unseen pairs of multisets (whose elements may themselves have been unseen during training).

Original: We model multiset elements as vectors on the standard simplex and multisets as the summations of such vectors, and we predict symmetric difference as the (cid:96) 1 -distance between multiset representations. Both our representations and how we predict symmetric difference size are theoretical rooted in fuzzy set theory.
Revised: We prove that this allows us to predict containment relations between unseen pairs of multisets.

Original: We show empirically that both these aspects of our model are important  to predicting symmetric difference sizes, comparing our approach to DeepSets-based approaches (Zaheer et al., 2017) with unconstrained item representations.
Revised: We show empirically that the theoretical basis of our model is important for being able to capture these relations, comparing our approach to DeepSets-based approaches (Zaheer et al., 2017) with unconstrained item representations.

Original: Furthermore, we demonstrate that our model learns meaningful representations, mapping objects of different classes to different standard basis vectors.
Revised: Furthermore, we demonstrate that our model learns “meaningful” representations.

Original:  R ELATED WORK 2. S ET REPRESENTATION
Revised:  R ELATED WORK 2. S ET REPRESENTATION

Original: Qi et al.
Revised: Qi et al.

Original: (2017) and Zaheer et al.
Revised: (2017) and Zaheer et al.

Original: (2017) both explore learning functions on sets.
Revised: (2017) both explore learning functions on sets.

Original: Importantly, they arrive at similar theoretical statements about the approximation of such functions, which rely on permutation invariant pooling functions.
Revised: Importantly, they arrive at similar theoretical statements about the approximation of such functions, which rely on permutation invariant pooling functions.

Original: In particular, Zaheer et al.
Revised: In particular, Zaheer et al.

Original: (2017) show that any set function f ( A ) can be approximated by a model of the form ρ (cid:0)(cid:80)
Revised: (2017) show that any set function f ( A ) can be approximated by a model of the form ρ (cid:0)(cid:80)

Original: a ∈
Revised: a ∈

Original: A φ ( a ) (cid:1) for some learned ρ and φ , which they call DeepSets.
Revised: A φ ( a ) (cid:1) for some learned ρ and φ , which they call DeepSets.

Original: They note that the sum can be replaced by a max-pool (which is essentially the formulation of Qi et al.
Revised: They note that the sum can be replaced by a max-pool (which is essentially the formulation of Qi et al.

Original: (2017)), and observe empirically that this leads to better performance.
Revised: (2017)), and observe empirically that this leads to better performance.

Original: More recently, there has been some very interesting work on leveraging the relationship between sets.
Revised: More recently, there has been some very interesting work on leveraging the relationship between sets.

Original: Probst (2018) proposes a set autoencoder, while Skianis et al.
Revised: Probst (2018) proposes a set autoencoder, while Skianis et al.

Original: (2019) learn set representations with a network that compares the input set to trainable “hidden sets.”
Revised: (2019) learn set representations with a network that compares the input set to trainable “hidden sets.”

Original: However, both these approaches require solving computationally expensive matching problems at each iteration.
Revised: However, both these approaches require solving computationally expensive matching problems at each iteration.

Original: 2. O RDERS AND HIERARCHIES
Revised: 2. O RDERS AND HIERARCHIES

Original: Vendrov et al. (2015) and Ganea et al.
Revised: Vendrov et al. (2015) and Ganea et al.

Original: (2018) seek to model partial orders on objects via geometric relationships between their embeddings — namely, using cones in Euclidean space and hyperbolic space, respectively.
Revised: (2018) seek to model partial orders on objects via geometric relationships between their embeddings — namely, using cones in Euclidean space and hyperbolic space, respectively.

Original: Nickel and Kiela (2017) use a similar idea to embed hierarchical network structures in hyperbolic space, simply using the hyperbolic distance between embeddings.
Revised: Nickel and Kiela (2017) use a similar idea to embed hierarchical network structures in hyperbolic space, simply using the hyperbolic distance between embeddings.

Original: These approaches are uniﬁed under the framework of “disk embeddings” by Suzuki et al.
Revised: These approaches are uniﬁed under the framework of “disk embeddings” by Suzuki et al.

Original: The idea is to map each object to the product space X × R , where X is a metric space.
Revised: The idea is to map each object to the product space X × R , where X is a (pseudo-)metric space.

Original: This mapping can be expressed as A (cid:55)→ ( f ( A ) , r ( A )) , and it is trained with the objective that A (cid:22) B
Revised: This mapping can be expressed as A (cid:55)→ ( f ( A ) , r ( A )) , and it is trained with the objective that A (cid:22) B

Original: if
Revised: if

Original: and only if d X ( f ( A ) , f ( B )) ≤ r ( B ) − r
Revised: and only if d X ( f ( A ) , f ( B )) ≤ r ( B ) − r

Original: ( A ) .
Revised: ( A ) .

Original: An equivalent statement can be made for multisets (see Theorem 3.3.1).
Revised: An equivalent statement can be made for multisets (see Proposition 3.2.4).

Original: Other work has taken a probabilistic approach to the problem of representing hierarchical relationships.
Revised: Other work has taken a probabilistic approach to the problem of representing hierarchical relationships.

Original: Lai and Hockenmaier (2017) attempt to formulate the Order Embeddings of Vendrov et al.
Revised: Lai and Hockenmaier (2017) attempt to formulate the Order Embeddings of Vendrov et al.

Original: (2015) probabilistically, modeling joint probabilities as the volumes of cone intersections.
Revised: (2015) probabilistically, modeling joint probabilities as the volumes of cone intersections.

Original: Vilnis et al.
Revised: Vilnis et al.

Original: (2018) represent entities as “box embeddings,” or rectangular volumes, where containment of one box inside another models order relationships between the objects.
Revised: (2018) represent entities as “box embeddings,” or rectangular volumes, where containment of one box inside another models order relationships between the objects.

Original: (Marginal and conditional probabilities can be computed from intersections of boxes.)
Revised: (Marginal and conditional probabilities can be computed from intersections of boxes.)

Original: Vilnis and McCallum (2015) propose modeling words as Gaussian distributions in order to capture notions of entailment and generality, and this work has been extended to mixtures of Gaussians by Athiwaratkun and Wilson (2017).
Revised: Vilnis and McCallum (2015) propose modeling words as Gaussian distributions in order to capture notions of entailment and generality, and this work has been extended to mixtures of Gaussians by Athiwaratkun and Wilson (2017).

Original: 2. F UZZY - AND MULTI - SETS
Revised: 2. F UZZY - AND MULTI - SETS

Original: The theory of fuzzy sets can be traced back to Zadeh (1965).
Revised: The theory of fuzzy sets can be traced back to Zadeh (1965).

Original: A fuzzy set A of objects from a universe
Revised: A fuzzy set A of objects from a universe

Original: U is deﬁned via its membership function µ A : U → [0 , 1] .
Revised: U is deﬁned via its membership function µ A : U → [0 , 1] .

Original: Fuzzy set operations — such as intersection — are then deﬁned in terms of this function.
Revised: Fuzzy set operations — such as intersection — are then deﬁned in terms of this function.

Original: In modern fuzzy set theory, intersection is usually deﬁned via a t-norm , which is a function T : [0 , 1] 2 → [0 , 1] satisfying certain properties.
Revised: In modern fuzzy set theory, intersection is usually deﬁned via a t-norm , which is a function T : [0 , 1] 2 → [0 , 1] satisfying certain properties.

Original: The intersection of two fuzzy sets A and B is deﬁned via the membership function µ A ∩ B ( x ) =
Revised: The intersection of two fuzzy sets A and B is deﬁned via the membership function µ A ∩ B ( x ) =

Original: T ( µ A ( x ) , µ B ( x )) .
Revised: T ( µ A ( x ) , µ B ( x )) .

Original: (More in-depth background, including the deﬁning properties of t-norms, is provided in 3.1.)
Revised: (More in-depth background, including the deﬁning properties of t-norms, is provided in Appendix B.)

Original: There is also more recent literature on extending fuzzy set theory to multisets (Casasnovas andMayor, 2008;
Revised: There is also more recent literature on extending fuzzy set theory to multisets (Casasnovas and

Original:  Miyamoto, 2000), using a membership function of the form µ A : U × [0 , 1] → N , where µ A ( x, α ) is the number of appearances in A of an object x with membership α . 
Revised: Mayor, 2008; Miyamoto, 2000), using a membership function of the form µ A : U × [0 , 1] → N , where µ A ( x, α ) is the number of appearances in A of an object x with membership α . which may help explain this observation.

Original: Let U = { x 1 , . . .
Revised: Namely, max-pooling is idempotent , meaning that repeatedly pooling a representation with itself does not change the result. On the other hand, summation does not have this property, and so repeated copies of an element are reﬂected in the result.

Original: , x n } be  the universe of possible objects.
Revised: In this way, DeepSets (with the sum rather than max-pool) is in fact modeling multisets rather than sets, which depending on the application may be undesirable.  P ROBLEM F ORMULATION Our goal is to learn to represent and predict a notion of containment between multisets. We begin with a brief motivating example, and then move on to provide the formalization of the problem.

Original: We will denote by U ∗ the set of all multisets with elements from U . We are interested in formulating a learnable function Ψ : U ∗ → R d mapping such multisets to vectors.
Revised: Recall our example, where for A = { currency , equilibrium } and B = { money , balance , economics } , we have a sense in which A is “contained” in B . After seeing many such example pairs of multisets, we want to be able to deduce that for A (cid:48) = { currency , food } and B (cid:48)

Original: Our contributions are: (1) formulating fuzzy multiset operations in a way that is amenable to machine learning, and (2) proposing to learn representations by predicting the sizes of the symmetric differences between pairs of multisets. Furthermore, both the formulation of how we predict symmetric difference size and the actual representations we use to do so are crucial components of (1). 3. B ACKGROUND : FUZZY SETS We begin with some background in fuzzy set theory.
Revised: = { money , food } , the relation A (cid:48) ⊆

Original: It is useful to begin by noting a canonical function of the form above for sets: Ψ( A ) = [ 1 x 1 ∈ A , . . . , 1 x n ∈ A ] . Each coordinate of the vector is 1 if x i is in A , 0 otherwise.
Revised: B (cid:48) holds.

Original: Fuzzy sets generalize this idea, letting Ψ( A ) = [ µ A ( x 1 ) , . . . , µ A ( x n )] , where µ : U → [0 , 1] is the membership function . Intuitively, µ A maps each x ∈ U to “how much of a member” x is of A , on a scale from 0 to 1. With this simple generalization, fuzzy set operations can be deﬁned.
Revised: 3. M ULTISETS

Original: This is traditionally done by leveraging element-wise fuzzy logical operations, which we deﬁne below.
Revised: In general, there exists a universe Ω of objects (in our above example, words). We let Ω ∗ denote the set of all multisets of objects from Ω , formally deﬁned as follows.

Original: Deﬁnition 3.1.

Revised: Deﬁnition 3.1.

Original: A t-norm is  a  function T : [0 , 1] 2 → [0 , 1] , satisfying the following properties: • Commutativity: T ( a, b ) = T ( b, a ) • Monotonicity: If a ≤ c and b ≤ d , then T ( a, b ) ≤ T ( c, d ) • Associativity: T ( a, T ( b, c ))
Revised: A multiset A is deﬁned by a its membership function m A : Ω → M , where

Original: = T ( T ( a, b ) , c ) • 1 is the identity: T ( a, 1) = a T-norms generalize the notion of conjunction.
Revised: M ⊆ R

Original: Note that the above conditions imply that for any a , T ( a, 0) = 0 , and that T (1 , 1) = 1 . These two observations show that t-norms are “compatible” with classical, non-fuzzy logic — where we identify 0 with “false” and 1 with “true.” The standard t-norm is T ( a, b ) = min { a, b } .
Revised: + is a subset of the non-negative reals, and m A maps each object to the “number of times” it occurs in A .

Original: [-Deﬁnition 3.1.
-] A strong negator is a strictly monotonic, decreasing function n : [0 , 1] → [0 , 1] such that n (0) = 1 , n (1) = 0 and n ( n ( x )) = x .
Revised: [-Deﬁnition 3.1.
-]The choice of M dictates the kind of multiset A is.

Original: Unsurprisingly, strong negators generalize logical negation. The standard strong negator is n ( x ) =  1 − x .
Revised: In particular, M = { 0 , 1 } gives classical sets, and M = N gives the traditional notion of multiset — a set which may contain multiple copies of the same object.

Original: Deﬁnition 3.1.3 An S-norm (also called a t-conorm ) is a function with the same properties as a t-norm, except that the identity element is 0. S-norms generalize disjunction. For every t-norm (and a given negator), we can deﬁne a complementary s-norm : S ( a, b ) = n ( T ( n ( a ) , n ( b ))) .
Revised: If M = R + , then we call A a “fuzzy multiset.”

Original: This is a generalization of De Morgan’s laws. The standard s-norm, complementary to the min t-norm, is S ( a, b ) = max { a, b } . The membership function for the intersection of two fuzzy sets A and B isnaturally deﬁned as µ A ∩ B ( x ) =
Revised: The cardinality (or size) of a multiset is deﬁned with respect to a measure λ on Ω .

Original: T ( µ A ( x ) , µ B ( x )) for a t-norm T . Similarly, the complement of a fuzzy set is given by µ A( x ) = n ( x ) for a strong negator n , and the union of two fuzzy sets is given by µ A ∪ B ( x ) = S ( µ A ( x ) , µ B ( x )) for an s-norm S . Usually, we want T and S to be complementary with respect to n .
Revised: Deﬁnition 3.1.2 The cardinality of a multiset A is | A | = (cid:82) Ω m A ( x ) dλ ( x ) .

Original: Then, we can generalize all the usual set operations to fuzzy sets by combining the three basic operations above. 3. G ENERALIZING TO FUZZY MULTISETS Our function Ψ above can easily be extended to (non-fuzzy) multisets by letting Ψ( A ) = [ m A ( x 1 ) , . . .
Revised: We will always ﬁx some measure λ on Ω (which may be called the dominating measure ) and take all cardinalities with respect to λ .

Original: , m A ( x n )] , where m A ( x i ) ∈ N is the multiplicity of x in A . This function m A should immediately remind us of fuzzy sets. Indeed, if we simply let m A map to any non-negative real number — that is, m A ( x i ) ∈ R
Revised: Note that we can always view m A as a density (i.e. the RadonNikodym derivative) of some measure µ A on Ω with respect to λ .

Original: + — we obtain a representation of a version of fuzzy multisets. Note that this is not the same formulation of “fuzzy multisets” usually given in literature (Casasnovas and Mayor, 2008; Miyamoto, 2000). However, this formulation is much more easily amenable to the machine-learning setting. Intuitively, the beneﬁt of “fuzzifying” our multisets is the ability to optimize via backpropagation.
Revised: We can thus identify a multiset A with the measure µ A on Ω , and write | A | = µ A (Ω) .

Original: With this deﬁnition of fuzzy multisets, we now deﬁne fuzzy multiset operations. Furthermore, we require that these operations be compatible both with their equivalents for non-fuzzy multisets and plain old sets ( i.e. when m A ( x i ) ∈ N and m A ( x i ) ∈ { 0 , 1 } , respectively). This will in fact essentially force us to adopt the following deﬁnitions:
Revised: In the case that the universe Ω is countable, we simply let λ be the counting measure, in which case the cardinality of any multiset A is |

Original: • Size: | A | = (cid:80) ni
Revised: A | = (cid:80) x ∈ Ω m a ( x ) .

Original: [-=
-]
Revised: [-=
-]All the usual operations on pairs of multisets are deﬁned in terms of their membership functions.

Original: m A ( x i ) • Intersection: m A ∩ B ( x i ) = min { m A ( x i ) , m B ( x i ) } • Union: m A ∪ B ( x i )
Revised: Deﬁnition 3.1.3 For two multisets A and B , their intersection A ∩ B , union A ∪ B , sum A + B , difference A − B , and symmetric difference A (cid:52) B , are multisets given by the following memberships functions, respectively:

Original:  =
Revised: • m A ∩ B ( x ) =

Original:  max { m A ( xi ) , m B ( xi ) } •Multiset addition: m A + B ( xi ) =
Revised: min { m A ( x ) , m B ( x ) } • m A ∪ B ( x ) = max { m A ( x ) , m B ( x ) } • m A + B ( x ) =

Original: m A ( xi ) + m B ( xi ) •Multiset difference: m A \ B ( xi ) = max { m A ( xi ) − m B ( xi ) , 0 } •Symmetric difference: m A (cid:52) B ( xi ) =
Revised: m A ( x ) + m B ( x ) • m A \ B ( x ) = max { m A ( x ) − m B ( x ) , 0 } • m A (cid:52) B ( x ) =

Original: | m A ( xi ) − m B ( xi ) |
Revised: | m A ( x ) − m B ( x ) |

Original: It should  stand out that our fuzzy multiset intersection and union are in fact the element-wise applications of the standard t-norm and s-norm, respectively.
Revised: These deﬁnitions are standard for multisets with both whole-number and real-valued memberships (Casasnovas and Mayor, 2008; Miyamoto, 2000; Blizard, 1989). To those familiar with fuzzy set theory, it should immediately stand out the intersection and union are given by the standard T-norm and T-conorm (functions used to deﬁne these operations on fuzzy sets; see Appendix B). This means that our deﬁnition of fuzzy multiset contains a copy of fuzzy set theory. Unfortunately, there is no intuitive way to use other T-norms in order to deﬁne multiset operations.

Revised: (For intuition on the above operations and why this is the case, see Appendix A.)

Original: However, our requirement of compatibility with non-fuzzy multiset operations means that we cannot use general t-norms and s-norms, and as we will now argue, makes the operations above the only reasonable choices.{+“real-valued multisets” of Blizard (1989), although the author approaches the subject from a standpoint of formal logic and axiomatic set theory. Deﬁnition 3.1.
+}
Revised: Finally, for multisets, containment is formally deﬁned as follows. to the machine-learning setting. Our notion here is also more closely related to the {+“real-valued multisets” of Blizard (1989), although the author approaches the subject from a standpoint of formal logic and axiomatic set theory. Deﬁnition 3.1.
+}

Original: Consider the two (non-fuzzy) multisets, A = { 1 , 1 , 1 , 2 , 2 } and  B = { 1 , 1 , 2 , 3 } .
Revised: For two multisets A ∈ Ω ∗ and B ∈ Ω ∗ , we say that B contains A , or A ⊆ B , if and only if m A ( x ) ≤ m B ( x ) for all x ∈ Ω . Note however that as demonstrated by our motivating example above, for two multisets A and B , we want to have a more “ﬂexible” notion than A ⊆ B . (It is not actually the case that { currency , equilibrium } ⊆ { money , balance , economics } .)

Revised: We will now formally provide a structure allowing for this ﬂexibility.

Revised: 3. A “ FLEXIBLE ” NOTION OF CONTAINMENT

Revised: Let us ﬁrst make two observations.

Revised: Firstly, the desired ﬂexibility will depend on some notion of “similairty” between the objects in Ω .

Revised: Secondly, this similarity must be externally provided by our observations of these “subset” relations.

Revised: In our example above, we had a sense that money ≈ currency and balance ≈ equilibrium , because we observed that A is “contained” in B (perhaps along with many other similar examples).

Revised: We now formalize this idea.

Revised: Deﬁnition 3.2.1 For two universes Ω and U , a map T : Ω ∗ → U ∗ is a called a multiset transformation from Ω to U .

Revised: The idea is that there exists some multiset transformation T from Ω to U , but we may not observe the structure of T or this new universe U .

Revised: However, we indirectly observe this structure, because our notion of subsets will be taken in U ∗ rather than in Ω ∗ .

Original: Their intersection should contain all their elements in common: A ∩ B  = { 1 , 1 , 2 } .
Revised: In particular, in our example above, the sense in which A is “contained” in B is that T ( A ) ⊆ T ( B ) . The simplest example of such a setting, on which we focus, is when U = { 1 , . . . , k } =

Original:  That is, we take the minimum number of times each element appears in either A or B , and that is the number of times the element appears in A ∩ B . 
Revised: [ k ] . That is, T maps each multiset in Ω ∗ to a multiset of numbers from 1 to k . These numbers can be thought of as “tags,” “classes,” or “labels,” meaning that each multiset has associated to it some tags, each of which may occur more than once.

Original: This straightforwardly gives us m A ∩ B ( x ) = 
Revised: Say for our running example, T ( A ) = { 1 , 2 } and T ( B ) = { 1 , 2 , 3 } . We then obtain that T ( A ) ⊆ T ( B ) , as desired. The example mapping T above is suggestive.

Original: min { m A ( x ) , m B ( x ) } . 
Revised: It suggests a category of such T functions that are commonly useful: when each object in Ω is itself associated with a “tag” in U . Here, money and currency both are associated to tag 1 , balance and equilibrium are both associated to tag 2, and economics to tag 3. Our map T is induced by this element-wise mapping. We formalize this notion as follows.

Original: Requiring that our notion of fuzzy{+Deﬁnition 3.2.
 We call a function t : Ω → U a universe transformation . Each universe transformation t induces a pushforward+} multiset intersection be compatible with non-fuzzy multisets means for whole-number inputs, the membership function must be equivalent to the min t-norm.
Revised: {+Deﬁnition 3.2.
 We call a function t : Ω → U a universe transformation . Each universe transformation t induces a pushforward+} multiset transformation T , where the membership function of T ( A ) is [Equation]

Original: As there is no clear gain from letting the membership function deviate{+Proposition 3.2.
 Every pushforward multiset transformation T preserves cardinalities; that is,+} for fractional inputs, we deﬁne fuzzy multiset intersection as such.
Revised: {+Proposition 3.2.
 Every pushforward multiset transformation T preserves cardinalities; that is,+} for any A , we have | A | = | T ( A ) | . Note that if we view A as the measure µ A on Ω , then the measure µ T ( A ) on U is the honest-togoodness pushforward measure µ A ◦ t − 1 . The above result follows easily.

Original: Following similar reasoning, we can convince ourselves that multiset union should be deﬁned as m A ∪ B ( x ) = max { m A ( x ) , m B ( x ) } . 
Revised: Let’s summarize where we are so far. We observe some relations between pairs of multisets over Ω . We also assume there is a multiset transformation T from Ω to some “latent” universe U , and that our observed relations are explained by relations that hold in U ∗ . In general, we wish to understand the structure of T , in order to predict similar relations for unobserved pairs of multisets.

Original: It is important to differentiate this from “multiset addition,” which simply combines two multisetsdirectly: A + B = { 1 , 1 , 1 , 1 , 1 , 2 , 2 , 2 , 3 } for our example above, andin general m A + B = m A ( x ) + m B  ( x ) .
Revised: We may also hope that in the process we learn something about the structure of Ω , and that in general this learning process is feasible due to U being much smaller or simpler than Ω . For example, we might assume that T is in fact a pushforward transformation induced by an unobserved labeling of the elements in Ω by elements in U = [ k ] . However, there is a problem with this setup: it is unlikely that for two multisets A ∈ Ω ∗ and B ∈ Ω ∗ , we have either T ( A ) ⊆ T ( B ) or T ( B ) ⊆

Original: Multiset difference is a little harder to deﬁne.
Revised: T ( A ) . However, there are richer relations that can exist between two multisets than just containment, and which can be observed for any such pair. For example, regardless of whether either of T ( A ) or T ( B ) is a subset of the other, we can ask about how much they overlap — i.e. the size of either their intersection | T ( A ) ∩ T ( B ) | , or of their symmetric difference, | T ( A ) (cid:52) T ( B ) | . Note that if we also know their sizes | T ( A ) | and | T ( B ) | , then we can know whether one contains the other (Theorem 3.2.6).

Original: The main problem is that we cannot rely on  a notion of “complement” for multisets.
Revised: This follows directly from the following: (1) that the size of the symmetric difference gives rise to a (pseudo-)metric on multisets, which can then be used to express a “disk embedding” inequality (Suzuki et al., 2019) relating containment to cardinalities (Proposition 3.2.4); and (2), the sizes of the symmetric difference and intersection are related via the sizes of the multisets themselves (Lemma 3.2.5). See Appendices C and D for proofs; note also that for each of the following three statements, we assume A and B are multisets over the same universe. Proposition 3.2.4 For any two multisets A and B , A ⊆ B if and only if | A (cid:52) B | ≤ | B | − | A | .

Original: Instead, let us again try to reason by example.
Revised: Lemma 3.2.5 For any two multisets A and B , | A (cid:52) B | = | A | + | B | − 2 | A ∩ B | . Theorem 3.2.6 Given A and B (whose cardinalities we can calculate), it sufﬁces to know either | A ∩ B | or | A (cid:52) B | to conclude whether A ⊆ B and whether B ⊆ A . Theorem 3.2.6 thus motivates the use of either | T ( A ) ∩ T ( B ) | or | T ( A ) (cid:52) T ( B ) | to learn about containment, in the case where T preserves cardinalities. In particular, we then know the cardinalities | T ( A ) | = | A | and | T ( B ) | =

Original: For our example multisets above, we have A \ B = { 1 , 2 } .
Revised: | B | , and thus either | T ( A ) ∩ T ( B ) | or | T ( A ) (cid:52) T ( B ) | is sufﬁcient to deduce the containment relations between T ( A ) and T ( B ) . We therefore see that we can use a training signal more readily available than binary yes-no containment — measurements of overlap between multisets — to still learn to predict containment relations. Thus, the problem we will be solving here is learning to predict either | T ( A ) (cid:52) T ( B ) | or | T ( A ) ∩ T ( B ) | from examples. Importantly, the error on these predictions will indicate how well we learned to capture our “ﬂexible” notion of containment.

Original: To arrive at this result, we remove from A each copy of an element which also appears in B . 
Revised: Formally, our learning task will therefore be as follows. There exists a universe Ω , which we assume for practical purposes can be embedded in R d for some known d . There is also some latent universe U =

Original: Note that if B had more of a certain element than A , that element would not appear in the ﬁnal result.
Revised: [ k ] together with an unknown multiset transformation T : Ω ∗ → U ∗ . We will assume that T preserves cardinalities — in practice this means either that T is a pushfoward transformation induced by some t : Ω → U , or an “expectation transformation,” which we deﬁne in Section 4.2. We then observe samples ( A, B ) from a training distribution D over pairs of multisets in Ω ∗ . For practical reasons, our sampled multisets will wholenumber multiplicities.

Original: In other words, we are performing a subtraction of counts which is “glued” to a minimum value of zero.
Revised: For each such pair, we also observe the overlap via either | T ( A ) (cid:52) T ( B ) | or | T ( A ) ∩ T ( B ) | . Which of these is used is ﬁxed beforehand for the entire task, and we assume this choice is known. (We test both choices in our experiments.) Our assumption that T preserves cardinality is important, because together with these observations, it allows us to conclude whether

Original: That is, m A \ B (x ) = max { m A  ( x ) 
Revised: T ( A ) is a subset of T ( B ) . We then pick a hypothesis target universe ˆ U =

Original: − m B ( x ) , 0 } .
Revised: [ ˆ k ] , and we let ˆ k

Original: We can further convince ourselves of the correctness of this expression by noting that we recover the identity
Revised: = k if k is known.

Original: A \ ( A \ B ) = A ∩ B .
Revised: (Experimentally, we examine the cases ˆ k < k and ˆ k = k .)

Original: Finally,symmetric multiset difference can be deﬁned using our expression for multiset difference, combined with either multiset addition or union.
Revised: Finally, our goal is to learn a model — i.e. a map ˆ T : Ω ∗ → ˆ U ∗ — that minimizes squared error in the predicted overlaps.

Original: In particular, note that A (cid:52) B = ( A \ B )+( B \ A ) = ( A \ B ) ∪ ( B \ A ) — addition and union both work because ( A \ B ) and ( B \ A ) are necessarily disjoint. This gives us:
Revised: That is, we learn ˆ T in order to minimize the appropriate choice of the following two losses:

Original: [Equation]
Revised: [Equation]

Original: (The equation still holds if we replace the addition with a maximum.)
Revised: Having formulated our learning task, we now deﬁne our learnable model ˆ T : Ω ∗ → ˆ U ∗ . In order to do so, we want our model to give us “representations” of the multisets in ˆ U ∗ in the most common machine-learning sense — i.e. vectors in some Euclidean space. We begin this section by deﬁning how we obtain and use such representations, and then conclude by deﬁning our model ˆ T itself that gives us these representations.

Original: 3. M ODELING THE RELATIONSHIP BETWEEN MULTISETS
Revised: 4. R EPRESENTATION OF MULTISETS

Original: {+Deﬁnition 4.1.
+}
Revised: {+Deﬁnition 4.1.
+}

Original: We can now justify our choice of  the size of  the symmetric difference to capture the relationship between two multisets.
Revised: For any universe ˆ U , a d -dimensional representation function is a map Ψ : ˆ U ∗ → R d . For a multiset S ∈ ˆ U ∗ , we call Ψ( S ) the representation of S . In general, we want our representations of multisets to be “useful,” in the sense that we can use them to perform common operations — such as those in Deﬁnition 3.1.3. More importantly for our task, we need to be able to calculate the size of either the symmetric difference or the intersection of two multisets.

Original: First, note that given | A | and | B | , and any one of | A ∩ B | , | A \ B | , or | A (cid:52) B | , we can tell what fraction of elements of A are in B and vice-versa.
Revised: Our choice of target universe ˆ U = [ˆ k ] gives us such a representation function. Deﬁnition 4.1.2 Let ˆ U be the ﬁnite universe [ˆ k ] . The natural representation function Ψ

Original: In particular, we can also tell whether A ⊆ B and whether B ⊆ A .
Revised: ˆ k : ˆ U ∗ → R ˆ k is the map S (cid:55)→ [ m S (1) , . . . , m S (ˆ k )] .

Original: In fact, we can formulate this relationship between containment and symmetric difference as a disk embedding inequality, where the metric space is that induced on U ∗ by the counting measure:
Revised: This should be an intuitive concept. For example, the natural representation function for classical sets gives the familiar indicator vector representation Ψ

Original: Theorem 3.3.1 For two fuzzy multisets A and B , A ⊆ B if
Revised: ˆ k ( S ) = [ 1 1 ∈ S , . . , 1 ˆ k ∈ S ] . Furthermore, we get “usefullness” of these representations for free, since all operations deﬁned via membership functions (e.g. those in Deﬁnition 3.1.3) can be performed coordinate-wise.

Revised: Furthermore, the cardinality of a multiset S ∈ ˆ U ∗ is given by thus sum of the entries in Ψ ˆ k ( S ) .

Revised: Together with the non-negativity of membership functions, this gives us the following.

Revised: (As these two results are essentially immediate, we omit their proofs.)

Revised: Lemma 4.1.3 For any multiset S ∈ [ˆ k ] ∗ , | S | =

Revised: ||

Revised: Ψ ˆ k ( S )

Original:  and only if | A (cid:52) B | ≤ | B | − | A | .
Revised: || 1 . Proposition 4.1.4 For any two multisets R and S over [ˆ k ] , we have | R (cid:52) S | = ||

Revised: Ψ ˆ k ( R ) − Ψ

Revised: ˆ k ( S )

Original: Proof.
Revised: ||and | R ∩ S | = min { Ψ ˆ k ( R ) , Ψ ˆ k ( S ) } , where the minimum is applied coordinate-wise.

Original: We will in fact show that  the inequality above can be replaced with an equality!
Revised: We thus use the natural representation function on ˆ U to train our model. We note that Proposition 4.1.4 could provide a reason to prefer the size of the symmetric difference over the size of the intersection as the training signal. The reasoning is that (cid:96) 1 -distance has a gradient which depends on both the representations Ψ ˆ k

Revised: ( R ) and Ψ ˆ k ( S ) in each coordinate (except at 0), while the coordinate-wise minimum can only depend on one of the representations in each coordinate.

Revised: We test this idea in our experiments.

Revised: 4. T HE LEARNABLE MODEL

Original: First, note thatA ⊆ B is equivalent to the statement that for any x ∈ U, m A ( x ) ≤ m B ( x ) .
Revised: Recall that the unobserved multiset transformation T : Ω ∗ → U ∗ preserves cardinalities. In particular, suppose for the purpose of exposition that T is the pushforward transformation induced by some labeling t : Ω → U .

Revised: We both want our hypothesis class of models ˆ T to contain all such pushforward multiset transformations, and to potentially be restricted to those ˆ T which preserve cardinalities.

Revised: Unfortunately, we cannot directly learn over the set of all pushforward transformations, as this is equivalent to learning the correct discrete labeling t : Ω → [ k ] , which is both a hard and nondifferentiable problem.

Original: Suppose that A ⊆ B{+Instead, we take a probabilistic approach. Deﬁnition 4.2.
 For two universes Ω and ˆ U , a probabilistic universe transformation is a map (cid:96) : Ω → ∆( ˆ U ) , where ∆( ˆ U ) is the space of probability measures on ˆ U+} .
Revised: {+Instead, we take a probabilistic approach. Deﬁnition 4.2.
 For two universes Ω and ˆ U , a probabilistic universe transformation is a map (cid:96) : Ω → ∆( ˆ U ) , where ∆( ˆ U ) is the space of probability measures on ˆ U+} .

Revised: As we will see, probabilistic universe transformations to [ˆ k ] have the advantage of being smoothly parametrizable.

Revised: In analogy to the pushforward multiset transformation induced by a t : Ω → U , we leverage our probabilistic transformation above to deﬁne a different kind of induced multiset transformation.

Revised: Deﬁnition 4.2.2 Let (cid:96) :

Original: Then, for all x ∈ U, m B ( x ) ≥ m A  ( x ) , and the desired result follows:
Revised: Ω → ∆( ˆ U ) be a probabilistic universe transformation. The expectation multiset transformation L : Ω ∗ → ˆ U ∗ is deﬁned to be the map A (cid:55)→ E P ∼ ( [Table]

Revised: [ P ] .

Revised: We ﬁrst note that L is well deﬁned, in the sense that L ( A ) is always a valid measure on ˆ U .

Revised: This can easily be seen by re-writing the expression as follows:

Original: [Equation]
Revised: [Equation]

Original: Suppose on the other hand that |{+L preserves cardinalities (see Appendix E for proof). Theorem 4.2.
 Any expectation multiset transformation L : Ω ∗ → ˆ U ∗ preserves cardinalities, i.e. for any+} A (cid:52) B | ≤ | B | − |  A  | .
Revised: L ( A ) has a natural interpretation, as the expected multiset obtained by sampling an element of ˆ U for each x ∈ A according to its corresponding distribution (cid:96) ( x ) (with contribution weighted by m A ( x ) ). Additionally, we have the desirable property that {+L preserves cardinalities (see Appendix E for proof). Theorem 4.2.
 Any expectation multiset transformation L : Ω ∗ → ˆ U ∗ preserves cardinalities, i.e. for any+} A ∈ Ω ∗ , | A | = | L ( A ) | .

Revised: We will thus let our learned model ˆ T : Ω ∗ → ˆ U ∗ be an expectation transformation L induced by some probabilistic universe transformation (cid:96) .

Revised: Note that we expect to be able to learn not only in the case where the unknown T is a pushforward transformation, but in fact when T itself is some expectation transformation (although we do not test the latter experimentally.)

Revised: 4. P UTTING IT ALL TOGETHER

Original: We ﬁrst note that this in fact means that | A (cid:52) B | = | B | − | A | ,since we are given
Revised: The outstanding question is: how do we parametrize our representations Ψ ˆ k ( L ( A )) for any given A ∈ Ω ∗ ? By the deﬁnition of the natural representation function Ψ ˆ k , we can write the i -th component of the representation of L ( A ) as

Original: [Equation]
Revised: [Equation]

Original: but it must be that
Revised: dµ A

Original: [Equation]
Revised: [Equation]

Original:  ) | ≥ (cid:88)
Revised: x ) , where the last two equalities come from viewing (cid:96) ( x ) as a general measure and thus multiset on ˆ U . Now, assuming that A is in fact one of the multisets sampled from our training distribution D , we know that A has whole-number multiplicities. (This will also be the case during evaluation, and thus in fact for any multiset we are trying to represent.) The above can then by simply written as Ψ ˆ k

Original: [Equation]
Revised: ( L ( A )) i =

Original:  x 
Revised: (cid:80) x ∈

Original: [Equation] x ) . Now, suppose for the sake of contradiction that for some x ∗ ∈ U , it is the case that m A ( x ∗ ) > m B ( x ∗ ) . Then, m B ( x ∗ ) − m A ( x ∗ ) < 0 ≤ | m B ( x ∗ )
Revised: A Ψ

Original: − m A ( x ∗ ) | . But this implies that (cid:88) x ∈U [Equation] ) , which is a contradiction.
Revised: ˆ k

Original: (cid:4)
Revised: ( (cid:96) ( x ))

Original: This theoretical connection is by itself a compelling reason to use symmetric difference over intersection or non-symmetric difference. In the context of backpropagation, however, our expression for fuzzy symmetric difference also has the advantage of having derivatives depending both on m A ( x i ) and m B ( x i ) for each i , unless m A ( x i ) = m B ( xi ) . We are now ready to formulate our learning model.
Revised: i , where each x occurs in the sum m A ( x ) times.

Original: To do so, we must formulate Ψand our loss function.
Revised: More simply, we have Ψ

Original: Our loss function will penalize error in predicting | A (cid:52) B | for input multisets A and B . We will use ∆( A, B ) to denote our predicted symmetric difference size. Given a training distribution D of pairs of multisets, our loss is L = E A,B ∼ D (cid:2) (∆( A, B ) − | A (cid:52) B | ) 2 (cid:3) . We formulate Ψ with the DeepSets expression in mind: Ψ( A ) =
Revised: ˆ k

Original: ρ (cid:0)(cid:80) a ∈
Revised: ( L ( A )) = (cid:80) x ∈

Original: A φ ( a ) (cid:1) , where φ : U → R d is some given object-featurization function. To make a perfect analogy to the fuzzy multiset representation above, we would require that each φ ( a ) be a standard basis vector. We want to be a little more general, however.
Revised: A Ψ

Original: For example, we may want to allow φ ( a ) ∈ R d with d < n . More importantly, we need to be able to learn φ . We do this by relaxing from standard basis vectors, allowing φ ( a ) to be any non-negative vector such that || φ ( a ) || 1 = 1 . That is, φ ( a ) can be a weighted combination of each of the standard bases with non-negative weights summing to 1 (which for d = n can be interpreted as a weighted combination each of the elements of U ).
Revised: ˆ k

Original: The normalization guarantees that φ ( a ) has the same scale as any of the basis elements. Importantly, if the norm || · || 1 is invariant under ρ , then we are guaranteed that || Ψ( A )
Revised: ( (cid:96) ( x )) . Since each (cid:96) ( x ) is just a distribution over ˆ k elements, it sufﬁces to learn a map from Ω to the probability simplex in R ˆ k — the non-negative vectors whose components sum to 1.

Original: [-|| 
 = | A |-] .
Revised: [-|| 
 = | A |-]Recalling that we assumed our input universe Ω consists of vectors in R d , we pick a favorite object-featurization network φ : R d → R ˆ k .

Original: In practice, we guarantee  the restrictions above by replacing φ ( a ) with f ( φ ( a ))
Revised: We then guarantee than we obtain a point in the probability simplex by taking φ ( a ) to f ( φ ( a ))

Original: ||
Revised: ||

Original: f ( φ ( a ))
Revised: f ( φ ( a ))

Original: || 1 , where f : R → R + is a function applied element-wise.
Revised: || 1 , where f : R → R + is a function applied component-wise.

Original: For differentiability, we choose the softplus function f ( x ) =
Revised: For differentiability, we choose the softplus function f ( x ) =

Original: log(1 + e x ) .
Revised: log(1 + e x ) .

Original: To guarantee the invariance of || · || 1 under ρ , we simply let ρ be the identity.

Original: We thus obtain: Ψ( A ) = (cid:80) a ∈
Revised: Our complete model is thus the representation Ψ( A ) = (cid:80) x ∈

Original: A f ( φ ( a )) ||
Revised: A f ( φ ( a )) ||

Original: f ( φ ( a ))
Revised: f ( φ ( a ))

Original: ||1 , and ∆( A, B ) = || Ψ( A ) − Ψ
Revised: ||

Original: B )
Revised: Our losses, in terms of these representations, are

Original: || 1 . [Table]
Revised: [Equation] [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We compare our model above with two reasonable alternatives.
Revised: We begin here with an overview what we want to test about our model.

Original: For the ﬁrst, we simply relax the restrictions on object representations — which by abuse ofnotation we will call the restriction ofφ (to the standard simplex) — letting Ψ( A ) =
Revised: In Section 5.1 we move on to describe our training and evaluation procedures. The experimental results themselves follow. A clear question to seek the answer to empirically is whether the size of the symmetric difference or of the intersection works better in practice.

Original: (cid:80) a ∈
Revised: (Recall that the symmetric difference may have more informative gradients, possibly leading to better learning and performance.) We thus compare these two approaches, both in terms of the error on the respective tasks themselves, and in terms of the error on predicting containment. More generally, the theory motivating our model suggest that there is a delicate balance in the properties that make the model well-posed. 5 We tackle this idea from two directions.

Original:  Aφ ( a ) , and ∆ as before.
Revised: First, we ask how important is the precise deﬁnition of our model, Ψ( A ) = (cid:80) x [Equation] A [Equation]

Original: The second alternative is to simply replace both Ψ and ∆ (which should itself permutation invariant) with DeepSets functions, letting Ψ( A  ) 
Revised: An obvious baseline to compare against is Ψ ( A ) = (cid:80) x ∈ A φ ( x ) , which should help us answer the question of how important it is that each object is mapped to a point in the probability simplex.

Original: =
Revised: Looking at this formulation, an immediate connection one might make is to the DeepSets model of Zaheer et al.

Original:  ρ 1 (cid:0)(cid:80)
Revised: (2017): Ψ( A ) = ρ 1 ( (cid:80) x

Original: a ∈
Revised: ∈

Original: A φ ( a ) (cid:1) and ∆( A, B ) =
Revised: A φ ( x )) for some learnable function ρ 1 . The authors prove this model can learn any permutation invariant function — e.g. the size of the intersection or symmetric difference of two multisets. We thus use both the models above as baselines in all our experiments, calling the former “unrestricted multisets” and the latter “DeepSets” The second category of question we ask here is whether we gain anything from our construction of the multiset operations on representations.

Original:  ρ 2 (Ψ( A ) + Ψ( B ))  .
Revised: We tackle this question replacing the terms || Ψ ( A ) − Ψ ( B ) ||and min { Ψ( A ) l Ψ( B ) } in our losses with ρ 2 (Ψ( A )+Ψ( B )) for a learnable function ρ 2 .

Original: We compare these models both on the task of predicting symmetric difference size, and in terms of their learned representations.
Revised: The intuition here is that this new prediction is in fact a second DeepSets model trying to learn of our prediction functions — where we choose DeepSets because both intersection and symmetric difference are permutation invariant (i.e. commutative). This setting will be called the “learned operation” setting. We further test whether our parametrizations of the multisets operations are somehow intrisically good via a scheme of “cross-wiring” them — using one for a task where we should use the other — on which we elaborate in Section 5.4. Finally, we will also examine the learned representations φ ( x ) of elements x ∈ Ω .

Original: 4. T RAINING AND EVALUATION PROCEDURES
Revised: 5. T RAINING AND EVALUATION PROCEDURES

Original: We use MNIST (LeCun, 1998) as our dataset.
Revised: We use MNIST (LeCun, 1998) as our dataset.

Original: The training set consists of 60,000 handwritten images of digits [Table] and the test set of 10,000.
Revised: The training set consists of 60,000 handwritten images of digits, and the test set of 10,000.

Original: We train all the models on 3 × 10 5 training pairs of multisets.
Revised: We train all the models on 3 × 10 5 training pairs of multisets ( A, B ) ∈ Ω .

Original: Both of the multisets in each pair are generated randomly at each iteration, as follows.
Revised: At each iteration of training, both A and B are generated randomly, as follows.

Original: First a size is uniformly sampled in the chosen range — in our experiments, either [2 , 5]or [2 , 10] . 
Revised: First a whole-number cardinality is uniformly sampled in some chosen range — in our experiments we use [2 , 5] . (We exclude singleton sets to ensure that the models aren’t just learning from comparing pairs of singletons.)

Original: 2 That many images are then chosen uniformly at random (with replacement) from the training set.
Revised: Once the cardinality is chosen, then that number of images x ∈ Ω is then chosen uniformly at random (with replacement) from the training set.

Original: The  symmetric difference  to be predicted is calculated directly from the image labels.
Revised: The multiset representation is calculated as usual, via one of the representation functions Ψ deﬁned above, and the predicted cardinality of the symmetric difference or intersection is then calculated using these representations. The value to be predicted is calculated directly from the labels of the images in the multisets — e.g. if A is two images of ones, and B is a one and a three, the target value will be 1 for intersection, and 2 for symmetric difference.

Original: All models are optimized using Adam (Kingma and Ba, 2015) with the default parameters β 1 = 0 .
Revised: The squared error is minimized using Adam (Kingma and Ba, 2015), with the default parameters β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: 999 , and a learning rate of 5 × 10 − 5 .
Revised: 999 , and a learning rate of 5 × 10 − 5 .

Original: The learning rate was chosen by logarithmic grid search from 1 down to 5 × 10 − 6 , training on up to 10pairs during the search.
Revised: The learning rate was chosen by logarithmic grid search from 1 down to 5 × 10 − 6 , training on up to 10 4 pairs during the search.

Original: (All models performed best with the chosen learning rate — or at least no worse than any of the other learning rates.)
Revised: (All models performed best with the chosen learning rate — or at least no worse than any of the other learning rates.)

Original: Given this learning rate, we chose to train the models for 3 × 10 5 iterations, ﬁnding that almost all of the models converged by this point.
Revised: Given this learning rate, we chose to train the models for 3 × 10 5 iterations, ﬁnding that almost all of the models converged by this point.

Original: Evaluation is performed similarly to training, with the addition of multiset sizes uniform on [2 , 20] , and with images sampled from the test set.
Revised: Evaluation is performed similarly to training, with the addition of multiset sizes uniform on [2 , 20] , and with images sampled from the test set.

Original: Importantly, this means that the none of the images seen during training appear during evaluation.
Revised: Importantly, this means that the none of the images seen during training appear during evaluation.

Original: For symmetric difference size prediction, each model is evaluated on 3 × 10 4 such multiset pairs, and  we let φ : U → R n (that is, d = n = 10 ).
Revised: Each model is evaluated on 3 × 10 4 such multiset pairs, and unless otherwise stated we let φ : Ω → R k (that is, ˆ h = k = 10 ).

Original: For the object featurizing function φ , we use a variant of the LeNet-5 neural network (LeCun, 1998).
Revised: For the object featurizing function φ , we use a variant of the LeNet-5 neural network (LeCun, 1998).

Original: Speciﬁcally, we adopt the same architecture as used by Ilse et al.
Revised: Speciﬁcally, we adopt the same architecture as used by Ilse et al.

Original: (Ilse et al., 2018).
Revised: (Ilse et al., 2018).

Original: (See Appendix A for network architectures.)
Revised: (See Appendix F for network architectures, including those used for ρ 1 and ρ 2 .)

Original: 4. S YMMETRIC DIFFERENCE SIZE PREDICTION
Revised: 5. C ARDINALITY PREDICTION

Original: Our results on the MNIST test-set are presented in Table 1.
Revised: We examine the performance of six kinds of models — multisets, unrestricted multisets, and DeepSets, each with or without learned multiset operations — on the tasks of predicting cardinality, either of the symmetric difference of the intersection. We will refer here to Tables 1 (symmetric difference) and 2 (intersection), which report the mean absolute errors of the predictions. Within each of the tables, two patterns are immediately clear.

Original: We report the mean absolute errors in symmetric difference size prediction, i.e. by how many elements on average is a model off from the true size.
Revised: First, a amount portion of the prediction error may be explained by whether the multiset operations are learned, or taken to be the theoretically-motivated parametrizations; the models with learned operations exhibit more than twice the prediction error. While this shows there is a beneﬁt to using our theoretically-motivated deﬁnitions, it does not necessarily mean that our deﬁnitions are intrinsically or uniquely well-suited for the task. We will revisit this point later. The second salient pattern is that as we move away from the expectation-transformation model (which we simply call “multisets” in our tables), ﬁrst to the unrestricted multiset model, and then to DeepSets, there is a rapid decrease in performance (in some cases almost ten-fold).

Original: We observe thatincluding the restriction on φ greatly improves performance when  the  symmetric difference  is predicted as ∆( A, B ) =
Revised: This suggests that the theory behind our model is indeed useful. Finally, when we compare across the two tables — that is, compare cardinality prediction for symmetric difference and for intersection — we observe a surprisingly large gap in error. The error on intersection size prediction is consistently about twice as small as on the other task. It is worth noting that if anything, we expected an opposite effect.

Revised: This gap is intriguing, and we believe that it should be explored further.

Original: || Ψ( A ) 
Revised: We now compare the same models above on what is perhaps the more important task: prediction whether there exists a containment relation between T ( A ) and T ( B ) for some A and B . In particular, for any such pair, we predict whether T ( A ) = T ( B ) , T ( A ) (cid:40) T ( B ) , T ( B ) (cid:40) T ( A ) , or there is containment relation (i.e. we treat this as a classiﬁcation problem).

Revised: We perform this prediction by relying of Theorem 3.2.6 (and the assumptions on our representations).

Revised: In particular, for any pair A and B , we predict the containment relation implied by Proposition 3.2.4, where we take the cardinality of the symmetric difference predicted by our model.

Revised: (If the model predicts intersection, we just use Lemma 3.2.5 to go from one to the other.)

Revised: Note that in this experiment we sample pairs A and

Revised: B such that the probability of each kind of containment is essentially uniform.

Revised: Referring to Tablesand 4, we observe almost exactly the same patters as above — with over 96% accuracy achieved by both multiset models.

Original: − Ψ
Revised: The one difference is that for the unnormalized model and the DeepSets model (with non-learned operations), the version of the models trained on the intersection task perform noticeably worse than the corresponding models trained on symmetric difference. Furthermore, on all other models, the performances are comparable. This further complicates the picture from above, as it suggests that while intersection may somehow be easier to learn to predict the cardinality of, perhaps the task itself is a worse way to capture containment relations. 5. “C ROSS - WIRING ” OPERATIONS

Original: B ) || Furthermore, when φ is restricted to the standard simplex, training on larger multisets leads to slightly better performance.
Revised: Motivated by our observation that models less closely aligned with our theory seems to perform worse, we devise a small test to see whether our symmetric difference and intersection cardinality operations are somehow intrinsic.

Original: It is unclear, however, whether training on larger multisets helps the multiset models “generalize” to multisets with sizes larger than those seen in training.
Revised: To do so, we perform two experiments.

Original: Finally, we see that the using the fuzzy symmetric multiset difference was also important to learning, as thepure DeepSets modelstruggles to predict  the correct value.
Revised: First, we train our regular expectationtransformation-based multiset model to predict symmetric difference cardinality, but where it’s prediction function is given by min { Ψ( A ) , Ψ( B ) } . Similarly, in the second experiment, we train the model to predict intersection cardinality, but where the prediction function is || Ψ ( A ) − Ψ

Original: (We note thatrounding the models’ predictions to whole numbers only slightly improved performance, if at all.)
Revised: ( B ) || We observe that the former model achieves and mean-absolute error of 2.3710 on the test set, while the second achieves ones of 0.9929. There values are signiﬁcantly higher than the errors achieved with the “correct” prediction functions, suggesting that there indeed a sense in which these are the “right” functions.

Original: 4. E XAMINING LEARNED REPRESENTATIONS
Revised: 5. E XAMINING LEARNED REPRESENTATIONS

Original: With φ ( a ) ∈ R n restricted to  thestandard ( n − 1) -simplex, we ﬁnd empirically that learned representations of objects are approximately the standard basis vectors (as shown in Figure 1 for n = 3 ).
Revised: We ﬁnally turn to examining the object-representations learned by our model. As one would expect, the learned representations of objects are approximate the standard basis vectors (as shown in Figure 1 for n = 3 ).

Original: This makes sense intuitively, since this is exactly the fuzzy multiset representation on which we base our symmetric difference and size operations. This is an interesting property; we note that, given the mapping from coordinate toclass, 98.9% classiﬁcation accuracy is achieved just by picking the maximum-valued coordinate of the representation of each object.
Revised: This suggests our expectation-transformation model is learning appropriate point-mass probabilities corresponding to each object’s label in U .

Original: We also examine the case d < n ,  when  the dimension of our representations is smaller than the number of possible objects.
Revised: We also examine the case ˆ k < k , which may occur when we don’t know the true size of U .

Original: Here, the “pinched” nature of the restricted representations may be undesirable (Figure 2a).
Revised: Here, the “pinched” nature of the restricted representations may be undesirable (Figure 2a).

Original: This problem, of course, gets worse with the discrepancy between number of objects and dimension (Figure 2b).
Revised: This problem, of course, gets worse with the discrepancy between number of objects and dimension (Figure 2b).

Original: On the other hand, the unrestricted multiset model is able to learn more balanced-looking clusters.
Revised: On the other hand, the unrestricted multiset model is able to learn more balanced-looking clusters.

Original: However, the clusters for d = n appear slightly less well-separated (Figures 3 and 4).
Revised: However, the clusters for d = n appear slightly less well-separated (Figures 3 and 4).

Original: The DeepSets model didn’t learn interpretable representations (Figure 4c).
Revised: The DeepSets model didn’t learn interpretable representations (Figure 4c).

Original: [Figure]
Revised: Furthermore, when we measure the accuracy of the regular multiset model on the containment prediction task above of each of the models, we obtain good results even when ˆ k is “too small.”

Original: [Figure] [Figure] [Figure]{+In particular, ﬁxing k = 5 : ˆ k = 
+}
Revised: {+In particular, ﬁxing k = 5 : ˆ k = 
+}

Original: [Figure]
Revised: we obtain an accuracy of 0.9586, and ˆ k = 3 gives 0.9045. This suggest that the representations learned are in fact robust to small discrepancies in dimension.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a novel task: predicting the size of  the symmetric difference  between pairs of multisets.
Revised: We propose a novel task: predicting the size of either the symmetric difference of the intersection between pairs of multisets.

Original: We  demonstrate the utility of this idea, developing a  model that given only the sizes of symmetric differences between pairs of multisets, learns representations of such multisets and their elements.
Revised: We motivate this construction via a measure-theoretic notion of “ﬂexible containment.” We demonstrate the utility of this idea, developing a theoretically-motivated model that given only the sizes of symmetric differences between pairs of multisets, learns representations of such multisets and their elements.

Original:  Our model learns to map each type of object to a standard basis vector, thus essentially performing semi-supervised clustering.
Revised: These representations allow us to predict containment relations with extremely high accuracy. Our model learns to map each type of object to a standard basis vector, thus essentially performing semi-supervised clustering.

Original: One interesting area for future theoretical work is understanding a related problem: clustering n objects given multiset difference sizes.
Revised: One interesting area for future theoretical work is understanding a related problem: clustering n objects given multiset difference sizes.

Original: As a ﬁrst step, we show in Appendix B that n − 1 speciﬁc multiset comparisons are sufﬁcient to recover the clusters. 
Revised: As a ﬁrst step, we show in Appendix H that n − 1 speciﬁc multiset comparisons are sufﬁcient to recover the clusters. We would also be curious to see if one can learn the latent multiset space U . A M ULTISET OPERATIONS

Original: We would also be curious to see if our model can learn meaningful representations even if the given labels are not exactly thesizes of symmetric differences — for example, human judgements of how different two bags-of-words are.
Revised: Here we provide through examples an intuitive understanding of the binary multiset operations from Deﬁnition 3.1.3. Consider the two (non-fuzzy) multisets, A = { 1 , 1 , 1 , 2 , 2 } and B = { 1 , 1 , 2 , 3 } . Their intersection should contain all their elements in common: A ∩ B = { 1 , 1 , 2 } .

Revised: That is, we take the minimum number of times each element appears in either A or B , and that is the number of times the element appears in A ∩ B .

Revised: This straightforwardly gives us m A ∩ B ( x ) =

Revised: min { m A ( x ) , m B ( x ) } .

Revised: Following similar reasoning, we can convince ourselves that multiset union should be deﬁned as m A ∪ B ( x ) = max { m A ( x ) , m B ( x ) } .

Revised: It is important to differentiate this from “multiset addition,” which simply combines two multisets directly: A + B = { 1 , 1 , 1 , 1 , 1 , 2 , 2 , 2 , 3 } for our example above, and in general m A + B = m A ( x ) + m B ( x ) .

Revised: Multiset difference is a little harder to deﬁne.

Revised: The main problem is that we cannot rely on a notion of “complement” for multisets.

Revised: Instead, let us again try to reason by example.

Revised: For our example multisets above, we have A \ B = { 1 , 2 } .

Revised: To arrive at this result, we remove from A each copy of an element which also appears in B .

Revised: Note that if B had more of a certain element than A , that element would not appear in the ﬁnal result.

Revised: In other words, we are performing a subtraction of counts which is “glued” to a minimum value of zero.

Revised: That is, m A \ B ( x ) = max { m A ( x )

Revised: − m B ( x ) , 0 } .

Revised: We can further convince ourselves of the correctness of this expression by noting that we recover the identity

Revised: A \ ( A \ B ) =

Revised: A ∩ B .

Original: Finally, we believe it may be interesting in future work to explore using  multiset symmetric difference size prediction as an auxiliary or pre-training task, when representations of objects need to be learned.
Revised: Finally, symmetric multiset difference can be deﬁned using our expression for multiset difference, combined with either multiset addition or union. In particular, note that A (cid:52) B = ( A \ B )+( B \ A ) = ( A \ B ) ∪ ( B \ A ) — addition and union both work because ( A \ B ) and ( B \ A ) are necessarily disjoint.

Revised: This gives us:

Revised: [Equation]

Revised: (The equation still holds if we replace the addition with a maximum.)

Revised: B F UZZY SETS

Revised: A fuzzy set A over a universe Ω is given by a function m A : Ω → [0 , 1] .

Revised: Intuitively, m A maps each x ∈ Ω to “how much of a member” x is of A , on a scale from 0 to 1.

Revised: With this simple idea, fuzzy set operations can be deﬁned.

Revised: This is traditionally done by leveraging element-wise fuzzy logical operations, which we deﬁne below.

Original: {+Deﬁnition B.0.
+}
Revised: {+Deﬁnition B.0.
+}

Revised: A t-norm is a function T : [0 , 1] 2 → [0 , 1] , satisfying the following properties:

Revised: • Commutativity: T ( a, b )

Revised: = T ( b, a ) • Monotonicity:

Revised: If a ≤ c and b ≤ d , then T ( a, b ) ≤ T ( c, d ) • Associativity: T ( a, T ( b, c ))

Revised: =

Revised: T ( T ( a, b ) , c ) • 1 is the identity: T ( a, 1) =

Revised: a

Revised: T-norms generalize the notion of conjunction.

Revised: Note that the above conditions imply that for any a ,

Revised: T ( a, 0) = 0 , and that T (1 , 1) = 1 .

Revised: These two observations show that t-norms are “compatible” with classical, non-fuzzy logic — where we identify 0 with “false” and 1 with “true.”

Revised: The standard t-norm is T ( a, b ) = min { a, b } .

Original: {+Deﬁnition B.0.
+}
Revised: {+Deﬁnition B.0.
+}

Revised: A strong negator is a strictly monotonic, decreasing function n : [0 , 1] → [0 , 1] such that n (0) = 1 , n (1) = 0 and n ( n ( x ))

Revised: =

Revised: x .

Revised: Unsurprisingly, strong negators generalize logical negation.

Revised: The standard strong negator is n ( x ) = 1 − x .

Revised: Deﬁnition B.0.3 An S-norm (also called a t-conorm ) is a function with the same properties as a t-norm, except that the identity element is 0.

Revised: S-norms generalize disjunction.

Revised: For every t-norm (and a given negator), we can deﬁne a complementary s-norm : S ( a, b ) =

Revised: n ( T ( n ( a ) , n ( b ))) .

Revised: This is a generalization of De Morgan’s laws.

Revised: The standard s-norm, complementary to the min t-norm, is S ( a, b )

Revised: = max { a, b } .

Revised: The membership function for the intersection of two fuzzy sets A and B is naturally deﬁned as µ A ∩ B ( x ) =

Revised: T ( µ A ( x ) , µ B ( x )) for a t-norm T .

Revised: Similarly, the complement of a fuzzy set is given by µ A ( x ) = n ( µ A ( x )) for a strong negator n , and the union of two fuzzy sets is given by µ A ∪ B ( x ) =

Revised: S ( µ A ( x ) , µ B ( x )) for an s-norm S .

Revised: Usually, we want T and S to be complementary with respect to n .

Revised: Then, we can generalize all the usual set operations to fuzzy sets by combining the three basic operations above.

Revised: C P ROOF OF P ROPOSITION 3.2.

Revised: We show that for any two multisets A and B over the same universe Ω , A ⊆ B if and only if | A (cid:52) B | ≤ | B | − | A | .

Revised: In fact, noting that it is always the case that | A (cid:52) B | ≥ | B | − | A | (which we will not prove but is easy to show), the following proof shows this holds with equality.

Revised: Proof.

Revised: Let λ be the dominating measure with respect to which the cardinalities are taken.

Revised: We ﬁrst show the forward direction.

Revised: Suppose A ⊆ B , that is, for every x ∈ Ω , we have m A ( x ) ≤ m B ( x ) .

Revised: The result follows directly (with equality):

Revised: [Equation]

Revised: For the converse direction, suppose on the other hand that | A (cid:52) B | ≤ | B | − | A | .

Revised: Now suppose for the sake of contradiction that for some x ∗ ∈ Ω , we have m A ( x ∗ ) >

Revised: m B ( x ∗ ) .

Revised: Then m B ( x ∗ ) − m A ( x ∗ ) < | m B ( x ∗ ) − m A ( x ∗ ) | .

Revised: But this implies that

Revised: [Equation]

Revised: which is a contradiction.

Revised: (cid:4)

Revised: D P ROOF OF L EMMA 3.2.

Revised: We show that for any two multisets A and B over the same universe Ω , | A (cid:52) B | = | A | + | B |− 2 | A ∩ B | .

Revised: Proof.

Revised: Let λ be the dominating measure with respect to which the cardinalities are taken.

Revised: Let Ω A ⊆

Revised: Ω be the elements x ∈ Ω on which m

Revised: A ( x ) > m B ( x ) , and similarly let Ω B ⊆

Revised: Ω be the elements x ∈ Ω on which m B ( x ) >

Revised: m A ( x ) .

Revised: Finally let Ω 0 be those elements x ∈ Ω for which m A ( x ) =

Revised: m B ( x ) .

Revised: Note that Ω A , Ω B , and Ω 0 are disjoint, and that their union is the entire universe Ω .

Revised: We can then write the cardinality of the intersection of A and B as

Revised: [Equation]

Revised: We then observe that | A | + | B | − 2 | A ∩ B | is

Revised: [Equation]

Revised: Simplifying, we obtain

Revised: [Equation]

Revised: But this is exactly the cardinality of the symmetric difference | A (cid:52) B | .

Revised: (cid:4)

Revised: E P ROOF OF T HEOREM 4.2.

Revised: We will show that for any probabilistic universe transformation (cid:96) : Ω → ∆( U ) , the induced expectation transformation

Revised: L : Ω ∗ → U ∗ preserves cardinalities.

Revised: That is, | A | = | L ( A ) | .

Revised: (For easy of notation, we write U here rather than

Revised: the ˆ U used in statement of the result in the main text

Revised: [Equation]

Revised: Proof.

Revised: Recall that since L ( A ) is a multiset and thus a measure

Revised: [Equation]

Revised: we have | L ( A ) | =

Revised: ( L ( A ))

Revised: ( U ) .

Revised: Expanding L ( A ) , we obtain

Revised: [Equation]

Revised: where the second equality holds because (cid:96) ( x ) is a probability measure over U .

Revised: (cid:4)

Original: A N ETWORK ARCHITECTURES
Revised: F N ETWORK ARCHITECTURES

Original: Given input images with c channels and an output dimension d  the function φ is parametrized by the network: 
Revised: Given input images with c channels, and an output dimension d , the function φ is parametrized by the network: [Table]

Original:  A two-dimensional convolution layer with c input channels, 20 output channels, kernel size 5, and stride 1 (no padding) 2.
Revised: [Table] A two-dimensional convolution layer with c input channels, 20 output channels, kernel size 5, and stride 1 (no padding) 2.

Original: A ReLU activation 3.
Revised: A ReLU activation 3.

Original: A two-dimensional max-pooling layer with kernel size 2 and stride4.
Revised: A two-dimensional max-pooling layer with kernel size 2 and stride4.

Original: A two-dimensional convolution layer with 20 input channels, 50 output channels, kernel size 5, and stride 1 (no padding) 5.
Revised: A two-dimensional convolution layer with 20 input channels, 50 output channels, kernel size 5, and stride 1 (no padding) 5.

Original: A ReLU activation 6.
Revised: A ReLU activation 6.

Original: A two-dimensional max-pooling layer with kernel size 2 and stride7.
Revised: A two-dimensional max-pooling layer with kernel size 2 and stride7.

Original: A fully-connected linear layer with output size d (the input size is determined by c ) 
Revised: A fully-connected linear layer with output size d (the input size is determined by c ) For the DeepSets model, we used for ρ 1 the

Original: For the DeepSets model, we used for ρ 1 the architecture:
Revised: architecture:

Original: A fully-connected linear layer with input size d and output size2.
Revised: A fully-connected linear layer with input size d and output size2.

Original: A hyperbolic tangent activation 3.
Revised: A hyperbolic tangent activation 3.

Original: A fully-connected linear layer with input and output size
Revised: A fully-connected linear layer with input and output size

Original: For ρ 2 we used:
Revised: For ρ 2 we used:

Original: A fully-connected linear layer with input and output size2.
Revised: A fully-connected linear layer with input and output size2.

Original: A hyperbolic tangent activation 3.
Revised: A hyperbolic tangent activation 3.

Original: A fully-connected linear layer with input size 100 and output size 
Revised: A fully-connected linear layer with input size 100 and output size G A DDITIONAL FIGURES [Table] [Table]

Revised: [Figure]

Original: B C LUSTERING n O BJECTS G IVEN n − 1 S YMMETRIC S ET D IFFERENCE S IZES
Revised: H C LUSTERING n O BJECTS G IVEN n − 1 S YMMETRIC S ET D IFFERENCE S IZES

Original: We are interested in the following problem.
Revised: We are interested in the following problem.

Original: Suppose we have a set of n objects U , each of which belongs to one of k clusters, C 1 , . . .
Revised: Suppose we have a set of n objects U , each of which belongs to one of k clusters, C 1 , . . .

Original: , C k .
Revised: , C k .

Original: Let M : 2 U → { 1 , . . .
Revised: Let M : 2 U → { 1 , . . .

Original: , k } ∗ be the function which takes any subset of U , and gives the multiset of cluster labels represented in that subset.
Revised: , k } ∗ be the function which takes any subset of U [Table] and gives the multiset of cluster labels represented in that subset.

Original: We are given oracle access to the function ∆ : 2 U × 2 U → N which gives the size of the symmetric set difference between the cluster-label multisets: ∆( A, B ) =
Revised: We are given oracle access to the function ∆ : 2 U × 2 U → N which gives the size of the symmetric set difference between the cluster-label multisets: ∆( A, B ) =

Original: | M ( A ) (cid:52) M ( B ) | .
Revised: | M ( A ) (cid:52) M ( B ) | .

Original: How many queries are required to determine the clusters C 1 , . .
Revised: How many queries are required to determine the clusters C 1 , . .

Original: , C k (up to permutation)?
Revised: , C k (up to permutation)?

Original: We show that the clusters can be determined with n − 1 speciﬁc queries.
Revised: We show that the clusters can be determined with n − 1 speciﬁc queries.

Original: (Another way to think of this is as a training data problem, rather than an oracle querying problem; we show n − 1 training examples can be sufﬁcient.)
Revised: (Another way to think of this is as a training data problem, rather than an oracle querying problem; we show n − 1 training examples can be sufﬁcient.)

Original: We do this in two steps.
Revised: We do this in two steps.

Original: The step lets us identify k disjoint subsets of U , such that no two of these subsets contain objects from the same cluster.
Revised: The step lets us identify k disjoint subsets of U , such that no two of these subsets contain objects from the same cluster.

Original: The second step conﬁrms that these subsets are in fact the clusters C 1 , . . .
Revised: The second step conﬁrms that these subsets are in fact the clusters C 1 , . . .

Original: , C k .
Revised: , C k .

Original:  The ﬁrst step consists of logarithmically “splitting” U .
Revised: [Figure] [Figure] The ﬁrst step consists of logarithmically “splitting” U .

Original: The very ﬁrst query in this step is ∆ (cid:16)(cid:83) (cid:100) k/ 2 (cid:101) i
Revised: The very ﬁrst query in this step is ∆ (cid:16)(cid:83) (cid:100) k/ 2 (cid:101) i

Original: =1 C i , (cid:83) ki
Revised: =1 C i , (cid:83) ki

Original: = (cid:100) k/ 2 (cid:101) C i (cid:17) , which tells us that (cid:83) (cid:100) k/ 2 (cid:101) i =
Revised: = (cid:100) k/ 2 (cid:101) C i (cid:17) , which tells us that (cid:83) (cid:100) k/ 2 (cid:101) i =

Original: 1 C i and (cid:83)
Revised: 1 C i and (cid:83)

Original: ki
Revised: ki

Original: =
Revised: =

Original: (cid:100)
Revised: (cid:100)

Original: k/ 2 (cid:101) C
Revised: k/ 2 (cid:101) C

Original: i are disjoint in terms of represented clusters.
Revised: i are disjoint in terms of represented clusters.

Original: We proceed recursively, each query “splitting” the sets in half (in terms of which clusters they contain).
Revised: We proceed recursively, each query “splitting” the sets in half (in terms of which clusters they contain).

Original: The number of such steps required is k − 1 (which is the number of internal nodes in a balanced binary search tree for k objects).
Revised: The number of such steps required is k − 1 (which is the number of internal nodes in a balanced binary search tree for k objects).

Original: We’ll call the resulting disjoint sets ˜ C 1 , . . .
Revised: We’ll call the resulting disjoint sets ˜ C 1 , . . .

Original: ˜ C k
Revised: ˜ C k

Original: (since we technically don’t yet know they correspond to the true clusters).
Revised: (since we technically don’t yet know they correspond to the true clusters).

Original: For the second step, we must verify that the objects in each of our sets resulting from step one all belong to the same cluster.
Revised: For the second step, we must verify that the objects in each of our sets resulting from step one all belong to the same cluster.

Original: This can be done by ordering the objects within each set, and comparing each consecutive pair as singletons.
Revised: This can be done by ordering the objects within each set, and comparing each consecutive pair as singletons.

Original: For each of our sets ˜ C
Revised: For each of our sets ˜ C

Original: i , we thus make | ˜
Revised: i , we thus make | ˜

Original: C i | − 1 such queries.
Revised: C i | − 1 such queries.

Original: Across all such sets, we thus make (cid:80) ki =1 | ˜
Revised: Across all such sets, we thus make (cid:80) ki =1 | ˜

Original: C i | − 1 = n − k queries.
Revised: C i | − 1 = n − k queries.

Original: So, the total number of queries made is ( k − 1) + ( n − k )
Revised: So, the total number of queries made is ( k − 1) + ( n − k )

Original: =
Revised: =

Original: n − 1 .
Revised: n − 1 .