F IBER B UNDLE M ORPHISMS AS A F RAMEWORK FOR M ODELING M ANY - TO -M ANY M APS
A BSTRACT
While it is not generally reﬂected in the ‘nice’ datasets used for benchmarking machine learning algorithms, the real-world is full of processes that would be best described as many-to-many.
That is, a single input can potentially yield many different outputs (whether due to noise, imperfect measurement, or intrinsic stochasticity in the process) and many different inputs can yield the same output (that is, the map is not injective).
For example, imagine a sentiment analysis task where, due to linguistic ambiguity, a single statement can have a range of different sentiment interpretations while at the same time many distinct statements can represent the same sentiment.
When modeling such a multivalued function f : X → Y , it is frequently useful to be able to model the distribution on f ( x ) for speciﬁc input x as well as the distribution on ﬁber f − 1 ( y ) for speciﬁc output y .
Such an analysis helps the user (i) better understand the variance intrinsic to the process they are studying and (ii) understand the range of speciﬁc input x that can be used to achieve output y .
Following existing work which used a ﬁber bundle framework to better model many-to-one processes, we describe how morphisms of ﬁber bundles provide a template for building models which naturally capture the structure of many-to-many processes.
I NTRODUCTION
Variation is ubiquitous in the real-world.
This is especially true for physical processes where a single input to the system can produce many different possible outputs.
For example, in nearly all scientiﬁc ﬁelds it is common for an experiment to yield (slightly) different results each time it is repeated, even when all controllable parameters are held ﬁxed.
On the other hand, variation in systems can also ﬂow in the other direction.
It is common to have many inputs to the system that yield a single output.
For example, there are inﬁnitely many images of cats that all map to the label ‘cat’.
From the perspective of machine learning (ML) we can think of processes that possess both of these properties as being many-to-many.
Each input to the system may yield many different outputs and many different inputs can also yield a single output.
A non-injective multivalued function underlies such processes.
In this paper we develop a deep learning architecture capable of modeling the distributions found on both images and ﬁbers of a many-to-many map.
More precisely, let f : X → Y be a many-to-many map, so that f ( x ) and f − 1 ( y ) are both sets for any x ∈ X and y ∈ Y .
A probability distribution on X induces probability distributions on both f ( x ) and f − 1 ( y ) .
Our goal is to build a model that simultaneously learns all of these distributions over the course of training.
Such information may not be necessary for simple inference tasks, but modeling the distributions on both images and ﬁbers gives information that can be critical for decision making.
For example, understanding the distribution f − 1 ( y ) can help a user identify all possible inputs that can be applied to achieve the speciﬁc desired output y .
On the other hand, being able to understand f ( x ) for varying x can point to those x that may yield less consistent values in Y .
The present work was inspired by a problem in advanced manufacturing where one wants to understand all the different manufacturing processing parameter settings X that will lead to a single desirable material property y ∈ Y .
If f represents the manufacturing process this amounts to understanding f − 1 ( y ) .
At the same time, certain processing parameters in X can yield less inconsistent results (more variation in f ( x ) ), which is often useful information when each processing run is time-, labor-, and resource-intensive.
To achieve both of these objectives within a single model architecture we build off the work of
Courts & Kvinge (2022) which proposed a ﬁber bundle framework to model many-to-one maps.
We ﬁnd that while ﬁber bundles offer a useful template for many-to-one maps, bundle morphisms better capture the structure required for an architecture meant to capture many-to-many maps.
In the model that we propose, which we call a Bundle Morphism Network (BMNet) , we ofﬂoad the variability intrinsic to f by lifting it to a new ‘bundle morphism’ F : E X → E Y .
We incorporate local trivializations into F so that it is easy to move from X to E X and Y to E Y .
Finally, the variation that emerges when we consider either f ( x ) or f − 1 ( y ) is encoded in simple distributions (e.g., 1dimensional Gaussians) on the ﬁbers of E X and E Y respectively.
The local trivializations, which we build into our model using invertible neural networks, are trained to transform these distributions to the distribution on f ( x ) and f − 1 ( y ) .
We test BMNet on several synthetic datasets that represent many-to-many maps with a ﬁber bundle like ﬂavor.
We benchmark BMNet against other many-to-many models as well as their relatives such as conditional GANs (cGANs) (Mirza & Osindero, 2014) and conditional normalizing ﬂow (cNF) models (Winkler et al., 2019).
We show that BMNet generally outperforms these models on most of our synthetic datasets suggesting that the next step is to benchmark BMNet against a wider variety of more complicated real-world datasets.
F IBER B UNDLES AND B UNDLE M ORPHISMS
Early in the history of geometry it was recognized that one way to analyze a space is to decompose it into simpler constituent parts.
The most familiar of such decompositions is the product, wherein space X is identiﬁed as the Cartesian product of two simpler spaces Y and Z , X ∼ =
Y × Z .
The torus
T is a simple example of a space that is (topologically) the product of two spheres, T ∼ =
S 1 × S 1 .
Some spaces, however, may appear to be a product space locally and yet fail to be a product space globally due to a ‘twist’ in the product structure.
One of the simplest examples of this is the M ¨ obius band.
The notion of a ﬁber bundle, ﬁrst introduced in Seifert (1933) and Whitney (1935), is meant to capture the essence of this phenomenon.
We include a review of the deﬁnition of a ﬁber bundle in Section A.8 of the Appendix.
[Figure]
As with many structures within topology and geometry, it is useful to identify the maps between ﬁber bundles that preserve their structure.
Let E
=
( E 1 , B 1 , Z 1 , π 1 ) and E 2 = ( E 2 , B 2 , Z 2 , π 2 ) be two ﬁber bundles.
A bundle morphism F between total spaces E 1 and
E 2 is a continuous map F : E 1 → E 2 and continuous map on base spaces f : B 1 → B 2 such the following diagram commutes.
While it is not intended to follow this deﬁnition “on the nose”, the concept of a bundle morphism informs the structure of the Bundle Morphism Network introduced in Section 3.
B UNDLE M ORPHISM N ETWORK
Building off of BundleNet, Bundle Morphism Network (BMNet) is a neural network Φ : X × Z 1 ×
R X × R Y → Y × Z 2 where X is the input space for the task, Y is the output space, Z 1 and Z 2 are called the ﬁbers, and R X
= { r X 1 , . . .
, r Xn 1 } and R Y =
{ r Y 1 , . . .
, r Yn 2 } are a collection of conditioning vectors representing neighborhoods of X and Y respectively.
For ﬁxed r X i ∈ R X , r Y j ∈ R Y we write Φ i,j :=
Φ( − , − , r Xi , r Yj ) .
Φ i,j is invertible (we use the FrEIA package (Ardizzone et al., 2021) to build Φ in our experiments).
For clarity we refer to the process of running the model in the direction
X × Z 1 → Y × Z 2 as the forward direction and running the model in the direction
Y × Z 2 → X × Z 1 as the reverse direction .
Z 1 and Z 2 are assumed to follow user chosen distributions D 1 and D 2 .
In the experiments in this paper we choose to use the uniform distribution on the circle when sampling from Z 1 and Z 2 (justiﬁcation for this was given in (Courts & Kvinge, 2022)).
Elements of R X and R Y consist of representatives of neighborhoods in the support of the training data input distribution (in X ) and the training data output distribution (in Y ) respectively.
Given a training set D ⊂ X × Y , we cluster both the input from X and the labels from Y and let R X and
R Y be the cluster centroids.
Described in the forward direction, the model consists of a conditional afﬁne transformation conditioned on elements of R Y , several RVNP blocks and coordinate permutation layers (Dinh et al., 2016), and an additional conditional afﬁne transformation conditioned on elements of R X .
These invertible conditional layers correspond to the local trivializations for each “bundle”.
During training, a batch consists of training examples { ( x i , y i ) } such that all x in the batch belong to the same cluster with center r Xi ∈ R X and all y in the batch belong to the same cluster with center r Yj ∈ R Y .
For each x , we sample a z 1 from distribution D 1 on Z 1 .
To run the model in the forward direction, we condition on r Yj and r Xi and obtain (ˆ y, ˆ z 2 ) =
Φ( x, z 1 , r Xi , r Yj ) .
To run the model in the reverse direction, we sample z 2 and obtain (ˆ x, ˆ z 1 ) =
Φ − 1 ( y, z 2 , r Xi , r Yi ) .
Once we have a collection of such ˆ y , which we denote by ˆ S , and y , which we denote by S , the loss in the forward direction is
[Equation]
L reverse is deﬁned analogously.
This is a symmetric version of the mean-squared minimum distance, used to ensure that the learned distribution covers the entirety of the training distribution.
Notice that this is different from the loss function in Courts & Kvinge (2022).
We found that applying an analogous version of the loss found in that paper was unstable when applied to our problem.
The detailed training algorithm is outlined in Algorithm 1.
At inference time, we will only have access to one of x or y .
If we wish to run the model in the forward direction and generate the distribution of f ( x ) , we need a conditioning vector r Yj ∈ R Y , which we cannot assume we have since we do not have a y value paired with x .
To get around this problem, we ﬁnd the k nearest neighbors of x from the training set and randomly sample one ( x (cid:48) , y (cid:48) ) .
We use the corresponding y (cid:48) from ( x (cid:48) , y (cid:48) ) to determine r Yj .
The detailed inference algorithm is outlined in Algorithm 2.
Running inference in the reverse direction is analogous.
E XPERIMENTS
We evaluate on three synthetic datasets.
The ﬁrst dataset, Torus-to-circle I , is illustrated in Figureand consists of a map from the torus to a circle.
In the forward direction, (Figure 4, center), each ij y r Xi arg min r ∈R X | r − x | r Yj = arg min r ∈R Y | r − |}
X
∼ proj y ( {+D x ) 
 : r Yj = arg min r ∈R Y | r − y (cid:48) | 7: Sample z ∼ D i8: (ˆ y, ˆ z 2 ) =+}
Φ( x, z, r X , r Y ) 9:
S = S ∪ { ˆ y } 10: end while slice of the torus maps to a random point sampled uniformly from an interval of the circle centered on the projection of that slice.
The second dataset, Torus-to-circle II ( Figure 5), similarly maps the torus to the circle.
In the forward direction a slice of the torus with axis of revolution angle θ is projected to one of two points on the circle with angles θ/ 2 or θ/ 2 + π , with equal probability.
The third dataset, M ¨ obius-to-circle , is very similar to Torus-to-circle I , with the torus replaced by a
M¨obius band.
More details on each dataset can be found in Section A.4 of the Appendix.
We compare BMNet to two other many-to-many models, Augmented CycleGAN (AugCGAN) (Almahairi et al.) and Latent Normalizing Flows for Many-to-Many Cross-Domain Mappings (LNFMM) (Mahajan et al., 2020).
Additionally, we include a pair of cGAN models (Mirza & Osindero, 2014) and a pair of cNF models (Winkler et al., 2019).
Note that in order to adapt cGANs and cNFs for many-to-many problems, we train one cGAN and cNF for the forward direction and one for the backward direction.
We follow the evaluation procedure proposed in Courts & Kvinge (2022).
In the forward direction, to evaluate the global distribution, we generate 5000 points and compare to 5000 points from the true distribution.
To evaluate local distributions, we samplepoints U from X and for each x ∈ U , use the model to generate 200 points in Y and compare this to the true distribution deﬁned by the dataset.
Evaluation of the backward direction is analogous.
To compare true and reconstructed distributions we use a range of different metrics, including the 1-Wasserstein metric, mean squared minimum distance (MSMD), maximum mean discrepancy (MMD), and KL divergence.
To obtain conﬁdence intervals, we train each model 5 times and report the mean with 95% conﬁdence intervals.
4. R ESULTS
As can be seen in Table 1, at the global level our model is the best performing model in terms of the Wasserstein-1 distance on most of the datasets. While AugCGAN performs well at the global level, as seen in Table 2, this manyto-many model does not learn the local distributions.
The cNF model also generally performs well, particularly at the local level, though requires training a separate model in each direction and does not learn the discrete nature of the Torus-to-circle II dataset in the forward direction. This is depicted in Appendix A.6.
To give a ﬂavor of the difference in reconstructions for models explicitly designed to model many-to-many processes, the reader should consult Figure 3 which depicts the global and local distributions of this dataset reconstructed by BMNet (top row) and an AugCGAN (bottom row).
Locally, while the AugCGAN struggles to learn the full range of possible inputs to achieve a given y , our model is able to recreate a reasonable approximation of cylindrical sections of the torus.
In general, we see this pattern repeated across datasets in both the forward and backward direction.
This suggests to us that the structure of BMNet makes it more capable of handling the problem set forth in this short paper.
Full results for each dataset, all models, all metrics, and further visualizations are available in SectionsA.6 and A.7 of the Appendix.
[Table]
[Table]
C ONCLUSION
Many-to-many processes are common in nature.
While there are a range of deep learning-based frameworks that can be used to solve simple tasks related to these processes, network architectures for more comprehensive modeling have until now remained limited.
Guided by the concept of a bundle morphism, in this paper we introduce the ﬁrst model architecture explicitly designed to capture the more nuanced aspects of many-to-many processes, providing the capability to model not only the distribution as a whole, but also the distributions of both the image of points and their ﬁbers.
This research was supported by the Mathematics for Artiﬁcial Reasoning in Science (MARS) initiative via the Laboratory Directed Research and Development (LDRD) investments at Paciﬁc Northwest National Laboratory (PNNL). PNNL is a multi-program national laboratory operated for the
U S. Department of Energy (DOE) by Battelle Memorial Institute under Contract
No. DE-AC0576RL0-1830.
A A PPENDIX A. R ELATED W ORK
As discussed above, this work builds on Courts & Kvinge (2022) where a ﬁber bundle-inspired deep learning framework was used to address the problem of modeling the ﬁbers in many-to-one machine learning tasks.
Our work differs from that one in that (i) the problem (modeling both ﬁbers and images in many-to-many maps) is distinct from the problem that is considered in Courts & Kvinge (2022), (ii) the inspiration for BMNet’s architecture is the concept of a bundle morphism, while the inspiration for BundleNet was an individual ﬁber bundle, and (iii) BMNet’s training routine and architecture are distinct from BundleNet.
Beyond BundleNet, ﬁber bundles (and more speciﬁcally vector bundles) have recently begun to see more use as a tool in data science.
Scoccola & Perea (2021) for example, seek to develop the theory required to use vector bundles in a rigorous way in data science.
Recent applications of vector bundles include: cryo-electron microscopy (Ye & Lim, 2017; Gao et al., 2021) and computer graphics (Kn¨oppel & Pinkall, 2016).
The present work differs from those listed above in that we only use the concept of a bundle morphism to guide the problem framework and model architecture for learning many-to-many maps.
More broadly, deep learning-based generative models have recently made great strides forward, both in terms of the complexity of distributions that can be modeled and the ﬁdelity of reconstructions.
While the majority of these approaches seek to model a single distribution, there have also been a signiﬁcant number of efforts to develop conditional generative approaches which are capable of modeling multiple distributions conditioned on additional input.
These include conditional variational autoencoders (Sohn et al., 2015) and conditional generative adversarial networks (Mirza & Osindero, 2014).
One of the results of this paper was to show that like the many-to-one ﬁber modeling problem introduced in (Courts & Kvinge, 2022), the many-to-many ﬁber and image modeling problem cannot be satisfactorily addressed with existing architectures.
Recent topological approaches to evaluating generative model performance includes (Zhou et al., 2020).
A. L IMITATIONS
Because it ﬁt well within the training routine we wanted to use, we chose to use an architecture that is invertible after conditioning (as in Courts & Kvinge (2022)).
Of course, in practice bundle morphisms need not be invertible (though the local trivializations that are built into our model are) and it is likely that this constraint will hurt performance on certain types of datasets.
In the future it would be worth revisiting the training procedure to understand how to include non-invertible components into the architecture.
Beyond this, the major limitation of this work is that the experiments that we describe are limited to synthetic datasets.
We felt that this was a reasonable choice based on space limitations and the novelty of the problem, but future work should identify and test against real-world many-to-many datasets.
One challenge related to the former is that most existing benchmark ML datasets are chosen to be ‘nice’ and thus there is a bias against one-to-many type phenomena.
A. T HE A NALOGY B ETWEEN A B UNDLE M ORPHISM N ETWORK AND A T RUE B UNDLE M ORPHISM
We caution that while BMNet is inspired by the idea of a bundle morphism, we did not strictly constrain the model to this deﬁnition.
In this section we brieﬂy lay out the analogy between components of the model architecture and a bundle morphism.
To ﬁx notation assume that
E 1 =
( E 1 , B 1 , Z 1 , π 1 ) and E 2 = ( E 2 , B 2 , Z 2 , π 2 ) are two ﬁber bundles where E i is the total space,
B i is the base space,
Z i is the ﬁber, and π i is the projection operator for i ∈ { 1 , 2 } .
Also let F : E 1 → E 2 be a bundle morphism that descends to a continuous map f : B 1 → B 2 making the required diagram commute.
On the other hand, let Φ :
X
×
Z 1 × R X × R Y
→ Y × Z 2 be an instance of BMNet.
• B 1 and B 2 correspond to X and Y respectively.
• Z 1 and Z 2 in the ﬁber bundle set-up correspond to Z 1 and Z 2 in the description of BMNet.
Note the distributions associated with Z 1 and Z 2 in BMNet are critical to the model, whereas probability distributions on the ﬁbers of ﬁber bundles are not commonly considered in topology or geometry.
This may be an interesting area for further investigation.
• The representatives R X
= { r X 1 , . . .
, r Xn 1 } and R Y =
{ r Y 1 , . . .
, r Yn 2 } correspond to the local trivialization neighborhoods in B 1 and B 2 .
• Φ i,j corresponds to φ j ◦ F ◦ φ − 1 i , where φ i : π − 1 1 ( U i )
→ U i × F 1 and φ j : π − 1 2 ( U j ) → U j × F 2 are local trivializations between neighborhoods π − 1 1 ( U i ) ⊆
E 1 and π − 1 2 ( U j ) ⊆
Erespectively.
A. D ATASETS
Torus-to-circle I:
Here we describe the synthetic datasets used in more detail.
For the Torus-tocircle I dataset, given radius r and radius R , the torus representation is:
[Equation]
We sample from T with R = 1 and r = 0 .
25 by randomly sampling θ and φ .
We pair this with a point from the circle by randomly sampling a perturbation α to add to φ . :
[Equation]
This dataset is visualized in Figure 4.
Torus -to-circle II: Generation of a point from the Torus-to-circle II dataset begins by sam - pling a point from T as above and then pairing that with one of two points from C , either (cos( θ/ 2) , sin( θ/ 2)) or (cos( θ/ 2 + π ) , sin( θ/ 2 + π )) with equal probability.
This dataset is depicted in Figure 5.
M¨obius-to-circle: The M¨obius-to-circle dataset is given by:
[Equation]
We sample from M with R = 1 and r = 0 .
25 by randomly sampling θ and s .
We pair this with a point from the circle as in Torus-to-Circle I.
A. T RAINING D ETAILS
Each model was trained on 5,000 training examples on a single GPU for 2,000 epochs.
An initial learning rate of 10 − 4 was used, and was reduced by a factor of 10 at 1000 and 1500 epochs.
Experimentally, we have found that the most effective prior to impose on Z 1 and Z 2 was a uniform circular prior.
During training, we update the parameters of the prior distributions based on the learned z 1 and z 2 .
Additionally, we enforce a regularization term on z 1 and z 2 for model stability during training.
A. V ISUALIZING RECONSTRUCTIONS
In this section we provide visualizations of the reconstructions of BMNet and the other models in Figures 6 through 9.
A. F ULL R ESULTS
We provide full results with all metrics for both models on all three datasets in Tables 3 to 14. [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] A. R EVIEW : F IBER B UNDLES A ﬁber bundle is a 4-tuple ( E, B, Z, π ) of topological spaces and a (surjective) continuous map π : E → B .
E is assumed to be covered by a collection of open sets { U i } i ∈I and each U i has a [Table] [Table] homeomorphism ϕ
i : π − 1 ( U i ) → U i × Z , called a local trivialization , associated with it.
Informally, ϕ i ensures that each neighborhood π − 1 ( U i ) looks like a direct product of U i and Z .
Finally, each ϕ i should respect the projection π so that the following diagram commutes.
[Equation]
E is known as the total space , B is known as the base space , and Z is known as the ﬁber .
Note that the product space X ∼ =
Y × Z is trivially identiﬁed as a ﬁber bundle where X is the total space
E , Y (respectively Z ) is the base space B , Z (resp.
Y ) is the ﬁber, π is the usual projection map from Y ×
Z to Y (resp.
Z ), { U i } i ∈I consists of a single neighborhood which is all of Y so that π − 1 ( Y ) =
Y × Z , and ϕ
i is just the identity map.
The reader can check that diagram equationeasily commutes in this case.