Residual Multiplicative Filter Networks for Multiscale Reconstruction
Abstract
Coordinate networks like Multiplicative Filter Networks (MFNs) and B ACON offer some control over the frequency spectrum used to represent continuous signals such as images or 3D volumes.
Yet, they are not readily applicable to problems for which coarse-to-ﬁne estimation is required, including various inverse problems in which coarse-to-ﬁne optimization plays a key role in avoiding poor local minima.
We introduce a new coordinate network architecture and training scheme that enables coarse-to-ﬁne optimization with ﬁne-grained control over the frequency support of learned reconstructions.
This is achieved with two key innovations.
First, we incorporate skip connections so that structure at one scale is preserved when ﬁtting ﬁner-scale structure.
Second, we propose a novel initialization scheme to provide control over the model frequency spectrum at each stage of optimization.
We demonstrate how these modiﬁcations enable multiscale optimization for coarseto-ﬁne ﬁtting to natural images.
We then evaluate our model on synthetically generated datasets for the the problem of single-particle cryo-EM reconstruction.
We learn high resolution multiscale structures, on par with the state-of-the art.
Project webpage: https://shekshaa.github.io/ResidualMFN/ .
Introduction
Coordinate networks have emerged as a powerful way to represent and reconstruct images, videos and 3D scenes [1–3] and for solving challenging inverse problems such as 3D molecular reconstruction for cryo-electron microscopy (cryo-EM) [4–6].
They typically take as input a point deﬁned on a continuous, low-dimensional domain (e.g., a 2D position for images) and they output the signal value at that point (e.g., the color).
In recently proposed architectures [7, 8], the frequency content of the represented signal can also be explicitly controlled.
Such networks are motivated in part by the effectiveness of multiscale methods in image processing and 3D reconstruction.
Nevertheless, while current scale-aware coordinate network architectures offer some control over scale [8–10], they are not readily compatible with classical multiscale methods used for coarse-to-ﬁne optimization, like pyramids [11, 12] or multigrid solvers [13].
State-of-the-art cryo-EM models, for instance, use frequency-marching to progressively estimate 3D density, beginning with a coarse structure, then adding ﬁner structure at each iteration [14, 15].
Some networks use heuristics to empirically constrain the scale of network output for coarse-to-ﬁne optimization, but they have no explicit constraints on the represented frequencies [9, 16, 17].
Others represent signals at multiple scales at inference time, but do not enable control of the frequency spectrum during training [8, 10].
Here, we introduce a new coordinate network architecture and training scheme that enables one to effectively control the frequency support of the learned representations during multiscale optimization, building on recent neural networks with an analytical and controllable Fourier spectra [7, 8].
For coarse-to-ﬁne reconstruction, one can divide the training process into stages, each of which learns a single scale.
Training starts at the coarsest resolution and then progressively adds ﬁner scales.
Naively employing this procedure with existing architectures will damage the signal reconstruction at coarser scales when ﬁtting ﬁner ones (e.g., see Fig. 3).
To mitigate this, we modify MFNs by adding skip connections [18], so reconstruction at one scale naturally incorporates the learned structure at coarser scales.
This alleviates the need to adapt previous layers in favor of ﬁtting the ﬁner grained reconstructions, effectively obviating the need to re-learn signal structure captured by coarser scales.
Second, we derive an initialization scheme which explicitly introduces hierarchical gaps in the frequency spectrum of a new layer so that, by construction, its frequency support has a controllable degree of overlap.
This requires the network to use the skip connections in order to ﬁll the holes in the spectrum and ensures lower levels remain faithful to coarse scale representations of the signal.
We apply our technique to the inverse problem of single-particle cryo-EM reconstruction [6, 4, 14, 19] to determine the 3D structure of macromolecular complexes, like proteins, from collections of noisy 2D tomographic projections.
This is a challenging non-linear inverse problem, closely related to multiview reconstruction, requiring estimation of the pose and 3D structure of bio-molecules.
Successful 3D reconstruction relies on coarse-to-ﬁne methods, sometimes called frequency marching [20], to reduce the risk of becoming trapped in poor local minima.
Our results demonstrate effective and efﬁcient ab initio reconstruction of 3D structures with cryo-EM to high resolution.
This paper proposes a new architecture and training approach to enable powerful multiscale estimation techniques with coordinate-based networks.
Speciﬁcally, we make the following contributions:
• We develop residual multiplicative ﬁlter networks, a new coordinate network architecture with a tailored initialization scheme for multiscale signal reconstruction and representation.
• We design fast and efﬁcient coarse-to-ﬁne training strategies for residual multiplicative ﬁlter networks that leverage the multiscale representation.
• We apply the proposed architecture and training strategy to cryo-EM reconstruction on two synthetic datasets and achieve ﬁnal reconstructions competitive with the cryoSPARC [14], a state-of-the-art method.
Related Work
Coordinate-based Networks.
Coordinate-based networks (e.g., [2, 3]) offer a memory-efﬁcient, continuous function parameterization that can be ﬂexibly trained to reconstruct 3D appearance [1, 21– 25] and structure [26–32], including applications in biomedical imaging [33, 34] and cryo-EM [5, 4, 6].
Although originally formulated as fully-connected MLP architectures, new models have been proposed to improve performance and interpretability.
For instance, one can use multiple small fully-connected networks to improve representational capacity [35–37], or combine coordinate-based networks with explicit feature grids to improve efﬁciency [38, 39], or facilitate generalization across shapes or scenes [40–43, 29, 44, 45].
Multiplicative Filter Networks (MFNs) replace the conventional MLP architecture with successive layers of Hadamard products and sine non-linearities [7].
Closest to our own work, MFNs have an analytical Fourier spectrum whose bandwidth can be explicitly constrained [8], improving the controllability and interpretability of the representation.
Inspired by this work, we establish Residual MFNs (rMFNs) with specialized control over the Fourier spectrum to enable coarse-to-ﬁne optimization.
Multiscale Reconstruction.
Multiscale representations and reconstruction methods are fundamental concepts in signal and image processing.
For example, wavelets are a fast and efﬁcient multiscale signal representation [46, 47] useful for denoising, compression, communications, and optical ﬂow estimation [48].
Multigrid frameworks are used for many differential equation solvers for physics simulations [13].
Gaussian and Laplacian pyramids [49, 11] have broad application to image processing and have, in part, inspired modern deep learning architectures [50].
Recent coordinate-based network architectures build on these fundamental concepts of multiscale signal representation and reconstruction.
Some architectures use explicit feature grids at multiple scales to enable efﬁcient training and inference for representing shape [39, 51] or rendering scenes [52, 38].
Multiscale fully-connected coordinate network architectures have been realized by progressively increasing the frequencies of positional encodings during optimization, for example, for bundle adjustment and neural scene representation [9, 17].
It is also possible to optimize separate networks with inductive biases towards low or high frequencies to improve shape representation [16].
Still, while these methods introduce multiscale training techniques, the architectures rely on inductive biases rather than explicit constraints on the scale or Fourier spectrum of the representation.
Other methods use coordinate networks for multiscale representation rather than efﬁcient optimization.
For instance, scale-aware positional encodings are trained at all scales simultaneously so that different output scales can be queried at inference time [10].
Band-limited coordinate networks (B ACON ) extend MFNs to control the network bandwidth, but require training on multiple output scales simultaneously for multiscale representation at inference time [8].
Our work is inspired by B ACON , but signiﬁcantly modiﬁes the architecture, initialization scheme, and training techniques to make efﬁcient multiscale optimization techniques compatible with coarse-to-ﬁne reconstruction.
Cryo-EM Reconstruction.
While coordinate networks have previously been used for cryo-EM reconstruction [5, 4, 6], they differ from our method in that they explicitly represent signals in the
Fourier domain.
Our goal is to demonstrate coarse-to-ﬁne reconstruction methods with coordinate networks in the real domain, and we demonstrate this for cryo-EM reconstruction.
The proposed method may also prove promising for advanced cryo-EM methods that resolve intrinsic motion of structures since spatial motion can be directly modeled in the primal domain [53].
Background
Coordinate neural networks have emerged as effective tools for approximating complex spatial data (e.g., see [1, 2, 7, 8, 10]).
In the case of images, for example, they provide a mapping from continuous positions on the image plane to RGB values.
Mapping 3D coordinates to volumetric density allows modeling 3D geometry.
In particular, these networks have been shown to be readily trained to accurately approximate low-dimensional, complex signals in a memory-efﬁcient way.
Among this broad family of network architectures, in this paper we focus on Multiplicative Filter Networks (MFNs) [7] such as B ACON [8], as they provide explicit control over the Fourier spectra of the function approximations.
Multiplicative Filter Networks.
Most coordinate-based networks, like SIREN [2] and Random Fourier Features [3], use an MLP architecture consisting of the successive composition of linear transformations and element-wise non-linearities.
In contrast, Multiplicative Filter Networks (MFNs) use a Hadamard product (i.e., element-wise multiplication) instead of composition [7].
Concretely, in an L -layer MFN, the d in dimensional input coordinates, x 2 R d in , are transformed using L +non-linear ﬁlter modules, denoted by g ( i ) ( · ) : R d in ! R d h , i = 0 , 1 , . . .
, L , and d h is the hidden layer dimension.
In [7], either sinusoidal or Gabor ﬁlters are used for this transformation.
The sinusoidal case is given by g ( i ) ( x ) = sin( ! ( i ) x
+  ( i ) ) , where !
( i ) 2 R d h ⇥ 3 and  ( i ) 2 R d h are referred to as frequencies and phases.
At layer i , the intermediate representation of previous layer, z ( i  1) , after being linearly transformed, is multiplied by the non-linear ﬁlter of input g ( i ) ( x ) .
Formally,
[Equation]
The behaviour of MFNs can be understood by analyzing the frequency of intermediate layers.
Using the trigonometric identity
[Equation]
closed-form expressions for intermediate representations can be derived [7, 8].
Dropping the bias terms b ( i ) for simplicity, the individual components of the intermediate representation z ( i ) is equal to a weighted sum of an exponential number of sine terms [7]:
[Equation]
where n = ( n 0 , · · · , n i  1 , n i ) is a tuple of indices with n j 2 { 1 , . .
, d h } and s j 2 {  1 , 1 } .
Each term in the summation comprises an amplitude, frequency and phase shift, given by
[Equation]
B ACON .
This analysis of the frequency content of an MFN shows that the bandwidth of z ( i ) ( x ) is the sum of the input bandwidths.
This was leveraged in B ACON [8] to band-limit the representation of each individual layer.
Additionally, an output was created for each layer to produce intermediate reconstructions; i.e.,
[Equation]
where W ( i ) out 2 R d out ⇥ d h and b ( i ) out 2 R d out .
However, to encourage y ( i ) to preserve its signal representation at scale i , one requires extra scale-speciﬁc losses during training, without which, subsequent training will corrupt the lower resolution representation.
Motivating Example.
Although B ACON does provide control over spectral bandwidth, when adopted in the context of a coarse-to-ﬁne optimization procedure, it fails to preserve and carry the learned coarse-scale representations when one moves to the next ﬁner-scale stage of the optimization.
Consequently, at each round, one must re-optimize the entire representations at all previous scales, hindering efﬁcient coarse-to-ﬁne optimization.
As an example, in Fig.
3 we examine the representation learned by B ACON within a coarse-to-ﬁne training strategy, where successive output layers are trained to ﬁt an image at ﬁner scales.
We apply a staged training procedure such that a loss is applied to outputs at each scale in separate rounds of optimization.
Unfortunately, when ﬁtting ﬁner scales B ACON completely forgets the learned representations of coarse scale outputs from previous optimization rounds (see supplement).
For a fairer comparison, given any round, we keep all scale-speciﬁc losses, but let only the linear output layers for other scales to be updated.
Still, the results of B ACON (Fig.
3, top row) show that the representations at coarser scales yet become corrupted while training at ﬁner scales.
In fact, reconstructions at different scales are highly coupled in B ACON , since new layers can result in new frequencies anywhere within the band limit.
In the following, we modify the architecture and its initialization and training scheme (Fig. 1) to make coordinate networks applicable to coarse-to-ﬁne multiscale reconstruction.
Residual Multiplicative Filter Networks
From Eq.
4, we can express frequencies in layer i in terms of frequencies in previous layers with the recursion
[Equation]
where ˜ n and ˜ s are formed from n and s by removing last entry, e.g., ˜ n = ( n 0 , · · · , n i  1 )
=
n <i .
Thus z ( i ) n i contains frequencies from the previous layer, shifted by ! ( i ) n i and  ! ( i ) n i , where ! ( i ) n i is the frequency of hidden unit n i in layer i .
More speciﬁcally, assuming that frequencies in the previous layer, i.e., ! ( i  1) (˜ n , ˜ s ) , are band-limited to [  B i  1 , + B i  1 ] , those in z ( i ) lie in the union of several of its shifted clones.
From this perspective, the frequencies at layer i , ! ( i ) , characterize two important concepts: 1) the growth in bandwidth from the previous layer, and 2) the extent of overlap between the previous layer frequency spectrum and each new individual clone.
For instance, given entries of ! ( i ) in [   , +  ] , the range of frequencies represented in z ( i ) expands to [  B i  1   , B i  1 +  ] .
Also, if a speciﬁc ! ( i ) n i is set to a relatively large value of 2 B i  1 this results in a clone with frequency support in the range [ B i  1 , 3 B i  1 ] , and it avoids overlap with the original bandwidth [  B i  1 , B i  1 ] .
4.1 Frequency Initialization
Motivated by this interpretation of ! ( i ) , we propose a new initialization scheme which explicitly controls the overlap and expansion of frequencies in successive layers.
For simplicity, we describe this initialization for the 2D case, though it can be extended to higher dimensions straightforwardly.
First, we follow a procedure similar to B ACON [8] to initialize and ﬁx frequencies of ﬁlters which allows limiting outputs to speciﬁed bandwidths.
Then, given the band-limited support [  B i  1 , + B i  1 ] of frequencies in layer i  1 , we initialize the 2D vectors ! ( i ) j as follows:
[Equation]
where R = { (0 , 1) , (1 , 0) , (1 , 1) , (1 ,  1) } , and U denotes either continuous or discrete uniform distributions.
Here, ! ( i ) j is composed of two terms.
The ﬁrst term mainly speciﬁes the direction ( r ( i ) j ) and the proportional amount (  2 B i  1 ) by which frequencies of layer i  1 will be shifted.
The second term is a relatively small random perturbation to break the symmetry.
An intuitive illustration of these operations is provided in Fig.
In 2D, there are 8 different directions to expand, half of which are redundant since s i = ± 1 entails two shifts in opposite directions ± ( r j, 1 , r j, 2 ) .
Thus we only consider 4 possible values R .
Using this initialization z ( i ) is thus band-limited to ! ( i ) {+
+}
[  (1 +  1 +  2 ) B i  1 , (1 +  1 +  2 ) B i  1 ] .
4.2 Residual Connections
Consider setting  2 = 2 +  1 during initialization.
In this case, by design, the frequency spectrum for sine terms in layer i would thus not overlap at all with that of layer i  1 ; in at least one direction, the frequencies would either end up to left or right of interval [  B i  1 , B i  1 ] .
Thus the output layer would include no sine terms with frequencies in the central interval [  B i  1 , B i  1 ] .
To obtain the output of layer i , we sum it with the output of layer i  1 .
That is,
[Equation]
This can be interpreted as a residual connection that ensures we keep an identical copy of frequencies from the previous layer; i.e., y ( i  1) will ﬁll the gap in the frequency space.
In practice, rather than the setting  2 = 2 +  1 , one can use other less strict combinations which allow for different amounts of overlap and expansion.
We found empirically setting  2 = 2 ,  1 = 0 .
3 to be effective.
2D Image Fitting
We ﬁrst evaluate each of the proposed modiﬁcations to band-limited networks in the context of 2D image ﬁtting.
This task is useful for didactic purposes, allowing one to clearly see the beneﬁts of skip connections and our new initialization scheme in the context of coarse-to-ﬁne image reconstruction.
We adopt a staged training procedure and ﬁt a series of band-limited coordinate networks to natural images provided by [1].
The dataset consists of 32 RGB images, each of size 512 ⇥ 512 .
We ﬁrst downsample the images by a factor of 2 to obtain images of size⇥ 256 .
These images are ﬁt during training using a Mean Squared Error (MSE) training objective, applied to the output at the scale corresponding to the stage of training.
As previously described, for the fairest comparison, the “Vanilla” B ACON baseline is optimized with losses at all scales, though, at the stage of a given scale, only the linear output layers of other scales can be updated.
Still, this keeps the optimization of the internal representations staged.
All networks use a four-layer architecture and are trained for 10 , 000 iterations, divided into chunks of 500 , 500 , 1000 , 8000 for optimizing output scales { 1 , 2 , 3 , 4 } , respectively.
The frequencies for all methods are initialized to have the same band-limit for consistency.
All methods were implemented in PyTorch [54] and trained on a single NVIDIA
GeForce RTX 2080.
[Table]
Figure 3 visualizes the outputs at each scale for B ACON , the residual MFN with the band-limited
B ACON initialization, and the residual MFN with our proposed initialization.
Since the coarse-to-ﬁne training scheme can potentially disturb previously trained outputs when applying the loss at the ﬁner scales, we visualize the differences between the Fourier spectra at each output scale after the corresponding training iterations and after the full training regiment (Fig. 3, right).
The proposed method shows the least disturbance after training.
For the entire image dataset, we report the Fourier spectrum mean absolute difference and standard deviation in Table 1.
Finally, we examine all networks on the image generalization task by querying them on a ﬁner grid of 512 ⇥ 512 and computing the Peak Signal-to-Noise Ratio (PSNR) between reconstructed signal and the original ground truth.
The average PSNR values over all images is summarized in the last column of Table 1.
We also benchmark SAPE [9] as a baseline for coarse-to-ﬁne reconstruction, on the image generalization task.
SAPE gradually exposes higher frequencies as a function of time and space, while providing only a single reconstruction at a time, so only their full-scale reconstruction can be evaluated for comparison.
While our approach improves interpretability of the reconstructions, by enabling reuse of previous learned representations, it achieves empirical performance competitive with baselines.
Additional experiments on a textual dataset are provided in Appendix A.
Cryo-EM 3D Reconstruction
Single particle cryo-EM is an increasingly popular experimental technique to recover the 3D structure of macromolecular complexes, such as proteins and viruses.
Cryo-EM has gained widespread attention in the last decade with pioneering advances in hardware and data processing techniques.
This has enabled atomic or near-atomic resolution reconstruction of challenging structures [55].
Multiscale, coarse-to-ﬁne reconstructions has long played a critical role in cryo-EM estimation, e.g., [20], making it an ideal testbed for our proposed methods.
Image Formation Model.
Based on a simpliﬁed model [56], the protein structure is represented as a function V : R! R  0 , which maps points in 3D space to non-negative real values, expressing the Coulomb potential induced by the atoms.
Images { I i } ni =1 are captured with a transmission electron microscope which are 2D orthogonal projections of randomly oriented potential maps.
Each image
I i has a corresponding latent pose (orientation)
R i {+
+} SO (3) representing the 3D rotation of the molecule in the image.
In practice, to ensure sufﬁcient contrast, the microscope is defocused when images are captured, resulting in a point-spread function (PSF)
g i .
To minimize radiation damage to delicate biological molecules, samples are exposed to low dosages of electrons resulting notoriously noisy images.
Taken together, the image formation model can be expressed as
[Equation]
where the projection is, by convention, assumed to be along the z -direction after rotation, ? corresponds to convolution and ✏ is Gaussian noise [14, 4]
[Figure]
3D Reconstruction Problem.
The goal of a cryo-EM experiment is to estimate the unknown 3D potential map of the biomolecule present in the observed images { I i } ni =1 .
This task has some intrinsic challenges.
First, the images are extremely noisy (see Fig. 4c).
Second, the ground-truth poses are typically unknown, or only a rough estimate is available, so it is not possible to perform 3D reconstruction using the forward model of Eq.
directly.
Previous work [6, 4, 19] estimate pose parameters during reconstruction with various techniques including amortized inference, variational bounds or direct marginalization.
Here we use a stochastic gradient-based method, which optimizes the representation of the 3D structure V , parameterized by our proposed residual MFN, together with the unknown pose variables { R i } ni =1 .
The loss function is the squared error between the observed image
I i and that generated by the model, i.e.,
[Equation]
[Equation]
) where I ⇤ i ( x, y ) is given above in Eq.
See Appendix D for more detail on the pose parameterization and optimization.
In practice, naively optimizing for both the poses and potential map suffers from poor local minima.
To avoid this, coarse-to-ﬁne reconstruction approaches are commonly employed.
At early stages, low frequency content of the map is estimated as the SNR is higher so estimation is easier, which then allows for more reliable subsequent pose estimation as higher frequencies are included.
Previous approaches [4, 14] employ this technique, called Frequency Marching [20], as they model the map in the Fourier domain where individual frequencies are readily accessible.
In contrast, we address the reconstruction problem in a coarse-to-ﬁne manner while expressing the potential map in real space.
6.1 Experiments
We evaluated our approach on two synthetic datasets which allow us to directly evaluate against ground truth structures.
The datasets are based on atomic models of 1OL5 ( https://www.rcsb.
org/structure/1ol5 ) and 4AKE ( https://www.rcsb.org/structure/4ake ) from Protein Data Bank (PDB).
We ﬁrst convert atomic models to⇥ 64 ⇥ 64 discrete potential maps, with voxel sizes 1 Å and 1 . 2 Å respectively, using pdb2mrc.py module from EMAN2 [58].
Then we simulate the image formation model as in Equation 9, at 50 , 000 rotations uniformly sampled on SO (3) .
To generate each image, we rotate the map V , resample it on the original grid, project it along canonical z-direction and apply the PSF.
Zero-mean white Gaussian noise is added to these simulated clean images produce a realistic SNR of 10%.
The structures and sample images are visualized in Fig.
Experimental Settings.
We use a 3-layer network with 128 hidden units to represent the 3D structure.
The network outputs the structure in 3 hierarchical scales and, accordingly, the training process is divided into 3 stages, one for each scale.
We set the hyperparameters as  1 = 0 .
3 ,  2 = 2 .and train the network for a total of 100 epochs.
For PDB1ol5, we spend respectively 15, 15, and 70 epochs optimizing the ﬁrst to the third scales.
For PDB4ake, we ﬁnd a better reconstruction if we dedicate more epochs on the ﬁrst and second scales, namely 25 epochs on each of the ﬁrst and second scales.
When computing the loss, images are band-limited to the corresponding scale using a Gaussian ﬁlter whose scale parameter is set to match the bandwidth of the current stage scale.
Mini-batches of images are used to update poses and weights of the network representing the structure.
For each mini-batch, we alternate between optimizing the structure and poses for 5 and 20 iterations, respectively.
We use Adam optimizer [59] with a learning rate 0 .
001 for the network parameters and a higher learning rate of 0 .
01 for pose parameters.
Each pose parameter is initialized to random value uniformly distribution over a geodesic distance of 45 to 90 degrees from the ground truth.
This can be considered as a very crude initial estimate for poses.
In Appendix E we consider even coarser initial pose values, showing that they achieve competitive qualitative and quantitative performance to those here.
Experiments are implemented in PyTorch [54], models are trained on
NVIDIA TITAN
Xp GPUs, and ChimeraX [60] is used to visualize the 3D maps.
Results.
Qualitative comparison of reconstructions with those obtained by homogeneous reﬁnement of cryoSPARC (as the baseline) are illustrated in Fig.
In the middle, we observe that, from low to high scales, increasingly more structural features are resolved.
The low-scale reconstructions capture only the global and low-resolution features, whereas the full-scale ones contain ﬁne-grained details.
Our approach is able to well-resolve some peripheral parts of the structure, especially for the more complex structure of PDB4ake.
Also there are some structural features learned by our reconstruction yet totally missed by cryoSPARC.
All in all, our ﬁnal reconstruction visually looks competitive or better when compared with the baseline.
As shown on the right side of Fig.
5, to quantitatively evaluate reconstructions, we compute the
Fourier Shell Correlations (FSCs) [61] between the reconstructed map and the ground-truth.
FSC is the gold-standard metric in cryo-EM area measuring the normalised cross-correlation coefﬁcient between two 3D volumes in Fourier domain along shells of increasing radius (see Appendix C for more background on FSC).
We measure FSC between ground-truth and reconstructions at all scales, following the convention of the threshold 0 .
5 to compute ﬁnal resolution, as ground-truth is known a priori in our experiments.
We also evaluate how well the poses are optimized and converge to the ground-truth.
To this end, the histogram showing number of images with various geodesic distances between estimated and ground-truth poses are visualized as in Fig.
For both datasets, there is a concentration of images on left hand side corresponding to slight error in estimation of pose parameters.
Thus, for most of the images, predicted pose parameters are within few degrees of the corresponding ground truth, verifying that poses are optimized well along with the reconstruction.
Furthermore, per dataset, we select two sample images, one with almost the correct pose estimate, and the other with one that is far from ground truth.
Then we demonstrate their reconstruction at all scales as in Fig.
Finally, we conduct an experiment in which we supervise the network only at the full-scale reconstruction throughout the entire training.
The output structures are depicted in the right panel of Fig. 6, realizing a situation where poses are trapped in local minima, and we thus observe no substantial improvement in the structure for many iterations.
Conclusions, Limitations and Future Work
In this paper, we build on Band-limited Coordinate Network (B ACON ) and propose a new architecture and training strategy providing ﬁne-grained control over spectral characteristics of network outputs, yielding a multiscale representation suitable for coarse-to-ﬁne reconstruction.
The combination of skip connections and a consistent initialization scheme allows the proposed network to ensure lower levels of the network represent the coarser scales of the signal which are then reused at ﬁner scales.
We empirically show that these modiﬁcations enable coordinate networks to ﬁt natural images in coarse-to-ﬁne fashion.
They are also shown to be effective for cryo-EM reconstruction.
Limitations.
We ﬁnd that the proposed multi-scale optimization procedure is effective in avoiding local minima in the Cryo-EM reconstruction problem.
We have yet to explore whether residual
MFNs are also effective on other problems with local minima due to the existence of nuisance variables.
Although beyond the scope of this paper, it is also possible that minor modiﬁcations of the architectural and hyper-parameter settings would improve the robustness of the optimization; e.g., increasing the number of layers for more ﬁne-grained scales, or a more diverse inspection of  hyperparameters.
Moreover, the cryo-EM experiments in this paper used synthetic data, to enable more thorough the evaluation of the method.
Real datasets are likely to require the use of other bases as well, e.g., Gabor , which are better able to represent the spatial variability of resolution in cryo-EM reconstructions.
We have also focused on homogeneous and rigid structures.
An advantages of our approach is that the efﬁcient multiscale representation of the signal in the real domain opens promising doors for modeling ﬂexible structures, a major open problem in cryo-EM [4, 53].
Societal Impact.
Cryo-EM for macromolecular structure determination is one of the foremost exciting areas in molecular biology with enormous social impact.
Cryo-EM tools were key to determining the structure of the SARS-COV2 spike protein, the discovery of its pre-fusion conformation, and the assessment of potential medical countermeasures.
Advances in this ﬁeld enable the discovery and exploration of large protein complexes and viruses more generally [62].
On the other hand, we strongly condemn any usage of our proposed coordinate networks for generating malicious representations, improperly modifying signals, or spreading misinformation.
Acknowledgments and Disclosure of Funding
We thank Ali Punjani for numerous valuable discussions and for use of the cryoSPARC software package.
This research was supported in part by the Province of