Unsupervised Learning From Incomplete Measurements for Inverse Problems
Anonymous Author(s)
AffiliationAddress email
Abstract
In many real-world inverse problems, only incomplete measurement data are available for training which can pose a problem for learning a reconstruction function.
Indeed, unsupervised learning using a fixed incomplete measurement process is impossible in general, as there is no information in the nullspace of the measurementoperator.
This limitation can be overcome by using measurements from multipleoperators.
While this idea has been successfully applied in various applications, aprecise characterization of the conditions for learning is still lacking.
In this paper,we fill this gap by presenting necessary and sufficient conditions for learning theunderlying signal model needed for reconstruction which indicate the interplaybetween the number of distinct measurement operators, the number of measurements per operator, the dimension of the model and the dimension of the signals.
Furthermore, we propose a novel and conceptually simple unsupervised learningloss which only requires access to incomplete measurement data and achievesa performance on par with supervised learning when the sufficient condition isverified.
We validate our theoretical bounds and demonstrate the advantages ofthe proposed unsupervised loss compared to previous methods via a series ofexperiments on various imaging inverse problems, such as accelerated magneticresonance imaging, compressed sensing and image inpainting.
Introduction
In multiple sensing applications, we observe measurements y ∈ R m associated with a signalx ∈ X ⊂ R n , through the forward process
[Equation]
where A ∈ R n × n is a linear measurement operator and ϵ denotes the noise affecting the measurements.
This is the case of computed tomography [1], depth ranging [2] and non-line-of-sight imaging [3]to name a few.
Estimating x from y is generally an ill-posed inverse problem due to the incompleteoperator
A with m < n and the presence of noise.
Knowledge of the signal model is required tomake this problem well-posed.
In many cases, obtaining ground-truth reconstructions x to learn the reconstruction function y
(cid:55)→ xmight be very expensive or even impossible.
For example, in medical imaging, it is not alwayspossible to obtain fully sampled images of patients as they require long acquisition times.
Inastronomical imaging, it is impossible to obtain ground-truth references due to physical limitations.
In electron-microscopy imaging [4], we can only measure 2D projections of a molecule.
In thesesettings, we can only access measurements y for learning.
Moreover, if the measurement process Ais incomplete, it is fundamentally impossible to learn the model with only measurements y , as thereis no information about the model in the nullspace of A .
Thus, we end up with a chicken-and-egg problem: in order to reconstruct x we need the reconstruction function, but to learn this function werequire some reconstructed samples x .
This fundamental limitation can be overcome by using information from multiple incomplete sensingoperators
A 1 , . .
A G , the general principle being that each operator can provide additional information about the signal model if it has a different nullspace.
For example, in the image inpaintingproblem, Studer and Baraniuk [5] used the fact that the set of missing pixels may vary betweenobserved images to learn a sparse dictionary model and reconstruct the images.
Yang et al.
[6]used multiple operators to learn a Gaussian mixture model in the context of hyperspectral imagingand high-speed video.
Bora et al.
[7] exploited this idea for learning a generative model in variousimaging problems such as deblurring and compressed sensing.
Matrix completion methods [8] exploita similar principle, as the missing entries of each column (i.e., signal) are generally different.
Ideallywe would like to learn the reconstruction function and signal model from only a small number ofdifferent measurement operators.
We are thus motivated to determine typically how many suchoperators are required.
The problem can be formalized as follows.
We first focus on the noiseless case to study the intrinsicidentifiability problems associated to having only incomplete measurement data.
The effect of noisewill be discussed in Section 4.
We assume that we observe a set of N training samples y i , where thei th signal is observed via A g i ∈
R m × n , one of G linear operators , i.e.,y
[Equation]
(2) where g i ∈ { 1 , . .
, G } and i = 1 , . . .
, N .
While we assume that the measurement operator A g i isknown for all observed signals, it is important to note that we do not know a priori if two observations( y i , A g i ) and ( y i ′ ,
A g i ′ ) are related to the same signal 1 x i .
There are two natural questions regardingthis learning problem:
Q1.
Signal Recovery Is there a unique signal x ∈ X
which verifies the measurements y
=
A g x ?
Inother words, is the reconstruction function f : ( y, A g )
(cid:55)→ x one-to-one?
Q2.
Model Identification Can we uniquely identify the signal model from only measurementsobtained via these incomplete operators?
In general, there can be a unique solution for neither problem, just one or both.
There might be aunique solution for signal recovery if the model is known, but it might be impossible to learn themodel in the first place (e.g., blind compressed sensing [9]).
The converse is also possible, that is,uniquely identifying a model without having enough measurements per sample to uniquely identifythe associated signal (e.g., subspace learning from rank-1 projections [10]).
The answer to Q1 is well-known from generalized compressed sensing theory, see for example [11].
Unique signal recovery is possible if the signal model is low-dimensional, i.e., if A g has m > 2 kmeasurements, where k is the model dimension.
On the other hand, Q2 has been mostly studied inthe context of matrix completion, where the set of signals is assumed to lie in a low-dimensionalsubspace of R n .
Bora et al.
[7] presented some results in the general setting, but only for the casewhere G = ∞ which is quite restrictive.
In this paper, we provide sharp necessary and sufficientconditions which hold for any low-dimensional distribution (beyond linear subspaces) and onlyrequire a finite number of operators G .
If the conditions for signal recovery and model identification are satisfied, we can expect to learn thereconstruction function from measurement data alone.
We introduce a new unsupervised learningobjective which can be used to learn the reconstruction function f : ( y, A g )
(cid:55)→ x , and providesperformances on par with supervised learning when the sufficient conditions are met.
The maincontributions of this paper are as follows:
• We show that unsupervised learning from a finite number of incomplete measurementoperators is only possible if the model is low-dimensional.
More precisely, we show thatm ≥ n/G measurements per operator are necessary for learning, and that for almost everyset of G operators, m > k + n/G measurements per operator are sufficient.
• We propose a new unsupervised loss for learning the reconstruction function that only
requires incomplete measurement data, which empirically obtains a performance on par
with fully supervised methods when the sufficient condition m > k + n/G is met.
• A series of experiments demonstrate that our bounds accurately characterize the perfor-
mance of unsupervised approaches on synthetic and real datasets, and that the proposed
unsupervised approach outperforms previous methods in various inverse problems
1.1 Related Work
Blind Compressed Sensing
The fundamental limitation of failing to learn a signal model fromincomplete (compressed) measurements data goes back to blind compressed sensing [9] for thespecific case of models exploiting sparsity on an orthogonal dictionary.
In order to learn the dictionaryfrom incomplete data, [9] imposed additional constraints on the dictionary, while some subsequentpapers [12, 13] removed these assumptions by proposing to use multiple operators A g as studied here.
This paper can be seen as a generalization of such results to more general signal models.
Matrix Completion Matrix completion consists of inferring missing entries of a data matrix Y =[ y 1 , . . .
, y N ] , whose columns are generally inpainted samples from a low-dimensional distribution,i.e., y i
=
A g i x i where the operators A g i randomly select a subset of m entries of the signal x i .
Thisproblem can be viewed as the combination of model identification, i.e., identifying the low-ranksubspace that the columns of X =
[ x 1 , . .
, x N ] belong to, and signal recovery, i.e., reconstructing theindividual columns.
Assuming that the samples belong to a k -dimensional subspace can be imposedby recovering a rank- k signal matrix X from Y .
If the columns are sampled via G sufficientlydifferent patterns A g i with the same number of entries m , a sufficient condition [14] for uniquelyrecovering almost every subspace model is 2 m ≥ (1 − 1 /G )
k + n/G .
A similar condition was shown in [15] for the case of high-rank matrix completion [16], which ariseswhen the samples x i belong to a union of k -dimensional subspaces.
We show that model identificationis possible for almost every set of G operators with m > k + n/G measurements, however the theorypresented here goes beyond linear subspaces, being also valid for general low-dimensional models.
Deep Nets for Inverse Problems Despite providing very competitive results, most deep learningbased solvers require measurements and signal pairs ( x i , y i ) (or at least clean signals x i ) in order tolearn the reconstruction function y
(cid:55)→ x from incomplete measurements.
A first step to overcome thislimitation is due to Noise2Noise [17], where the authors show that it is possible to learn from onlynoisy samples.
However, their ideas only apply to denoising settings where there is a trivial nullspace,as the operator A is the identity matrix.
This approach was extended in [18] to the case where twomeasurements are observed per signal, each associated with a different operator.
Yaman et al.
[19]and Artifact2Artifact [20] empirically showed that it is possible to exploit different measurementoperators to learn the reconstruction function in the context of magnetic resonance imaging (MRI).
AmbientGAN [7] proposed to learn a signal distribution from only incomplete measurements usingmultiple forward operators, however they only provide reconstruction guarantees for the case wherean infinite number of operators A g is available 3 , a condition that is not met in practice. Another line of work focuses on learning using measurements from a single incomplete operator.
Theworks in [21, 22] use the large system limit properties of random compressed sensing operators tolearn from measurements alone.
The equivariant imaging approach [23, 24] leverages invariance ofthe signal set to a group of transformations to learn from general incomplete operators.
Signal Recovery Preliminaries
We denote the nullspace of A as N A .
Its complement, the range space of the pseudo-inverse A † , isdenoted as R A , where R A ⊕ N A
= R n and ⊕ denotes the direct sum.
Throughout the paper, weassume that the signals are sampled from a measure µ supported on the signal set X ⊂ R n .
Signalrecovery has a unique solution if and only if the forward operator x
(cid:55)→ y is one-to-one, i.e., if forevery pair of signals x 1 , x 2 ∈ X where x 1 ̸ = x 2 we have that
[Equation]
In other words, there is no vector x 1 − x 2 ̸ = 0 in the nullspace of A .
It is well-known that this isonly possible if the signal set X is low-dimensional.
There are multiple ways to define the notion ofdimensionality of a set in R n .
In this paper, we focus on the upper box-counting dimension which isdefined for a compact subset S ⊂ R n as
[Equation]
where N ( S, ϵ ) is the minimum number of closed balls of radius ϵ with respect to the norm ∥ · ∥that are required to cover S .
This definition of dimension covers both well-behaved models such ascompact manifolds and more general low-dimensional sets.
The mapping x
(cid:55)→ y is one-to-one foralmost every forward operator A ∈ R m × n if [25]
[Equation]
where ∆ X denotes the normalized secant set which is defined as
[Equation] [Equation]
The term almost every means that the complement has Lebesgue measure 0 in the space of linearmeasurement operators R m × n .
The normalized secant set of models of dimension k generally hasdimension 2 k , requiring m > 2 k measurements to ensure signal recovery.
For example, the union ofk -dimensional subspaces requires at least 2 k measurements 4 to guarantee one-to-oneness [26].
Thisincludes well-known models such as k -sparse models (e.g., convolutional sparse coding [27]) andco-sparse models (e.g., total variation [28]).
In the regime k < m ≤ 2 k , the subset of signals whereone-to-oneness fails is at most (2 k − m ) -dimensional
Uniqueness of Any Model?
A natural first question when considering uniqueness of the model is: can we recover any measure µobserved via forward operators
A 1 , . .
A G , even in the case where X is the full R n ?
We show that,in general, the answer is no.
Uniqueness can be analysed from the point of view of the characteristic function of µ , defined asφ ( w )
=
E { e i w ⊤ x } where the expectation is taken with respect to µ and i =
√− 1 is the imaginaryunit.
If two distributions have the same characteristic function, then they are necessarily the samealmost everywhere.
Each forward operator provides information about a subspace of the characteristicfunction as
[Equation]
where A † g A g is a linear projection onto the subspace R A g .
Given that m < n , the characteristicfunction is only observed in the subspaces R A g for all g ∈ { 1 , . . .
, G } .
For any finite number ofoperators, the union of these subspaces does not cover the whole R n , and hence there is loss ofinformation, i.e., the signal model cannot be uniquely identified.
In the case of an infinite number of operators G = ∞ , the Cramér-Wold theorem guarantees uniqueness of the signal distribution if all possible one dimensional projections ( m = 1 ) are available [29, 7].
However, in most practical settings we can only access a finite number of operators and manydistributions will be non-identifiable.
Uniqueness of Low-Dimensional Models
Most models appearing in signal processing and machine learning are assumed to be approximatelylow-dimensional, with a dimension k which is much lower than the ambient dimension n .
Asdiscussed in Section 2, the low-dimensional property is the key to obtain stable reconstructions, e.g.,in compressed sensing.
In the rest of the paper, we impose the following assumptions on the model:
A
The signal set X is either
(a) A bounded set with box-counting dimension k .
(b) An unbounded conic set whose intersection with the unit sphere has box-counting
dimension k − 1 .
This assumption has been widely adopted in the inverse problems literature, as it is a necessaryassumption to guarantee signal recovery.
Our definition of dimension covers most models used inpractice, such as union of subspaces (simple subspace models, convolutional sparse coding models,k -sparse models), low-rank matrices and compact manifolds.
It is worth noting that dimension is aproperty of the dataset and thus independent of the specific algorithm used for learning.
In the rest of the paper, we focus on conditions for the identification of the support X instead ofthe signal distribution µ , due to the following observation: if there is a one-to-one reconstructionfunction (which happens for almost every A with m > 2 k as explained in Section 2), uniqueness ofthe support implies uniqueness of µ .
If X is known and there is a measurable one-to-one mappingfrom each observed measurement y to X , then it is possible to obtain µ as the push-forward of themeasurement distribution.
Before delving into the main theorem, we present a simple example which provides intuition of howa low-dimensional model can be learned via multiple projections
A g :
Learning a one-dimensional subspace Consider a toy signal model with support X ⊂ R 3 whichconsists of a one-dimensional linear subspace spanned by ϕ =
[1 , 1 , 1] ⊤ , and G = 3 measurementoperators
A 1 , A 2 , A 3 ∈ R 2 × 3 which project the signals into the x (3) = 0 , x (2) =
and x (1) =
0planes respectively, where x ( i ) denotes the i th entry of the vector x .
The example is illustratedin Figure 1.
The first operator A 1 imposes a constraint on X , that is, every x ∈ X should verifyx (1) − x (2)
= 0 .
Without more operators providing additional information about X , this constraintyields a plane containing X , and there are infinitely many one-dimensional models that would fit thetraining data perfectly.
However, the additional operator A 2 adds the constraint x (2) − x (3) = 0 ,which is sufficient to uniquely identify X as
[Equation]
is the desired 1-dimensional subspace.
Finally, note that in this case the operator
A 3 does not restrictthe signal set further, as the constraint x (1) − x (3) = 0 is verified by the other two constraints.
The ideas from the one-dimensional subspace example can be generalized and formalized as follows:for each projection A g , we can constrain the model support X by considering the set
[Equation]
11) which has dimension at most n − ( m − k ) .
Note that the true signal model is a subset of ˆ X g .
Theinferred signal set belongs to the intersection of these sets
[Equation]
[Equation]
which can be expressed concisely as
[Equation]
Even though we have derived the set ˆ X from a purely geometrical argument, the constraints in (13)also offer a simple algebraic intuition: the inferred signal set consists of the points v ∈ R n whichverify the following system of equations
[Equation]
for all possible choices of G points x 1 , . . .
, x G in X .
In other words, given a dataset of N incompletemeasurements { A g i x i } Ni =1 , it is possible to build ˆ X by trying all the possible combinations of Gsamples 5 and keeping only the points v which are the solutions of (14).
It is trivial to see that X ⊆
ˆ X ,
but when can we guarantee X
= ˆ X ?
As in the previous toy example, ifthere are not enough constraints, e.g., if we have a single A and no additional measurement operators,the inferred set will have a dimension larger than k , containing undesired aliases.
In particular, wehave the following lower bound on the minimum number of measurements:
Proposition 4.1 (Theorem 1 in [23]) .
A necessary condition for model uniqueness is that [Equation] and thus m ≥ n/G .
Proof.
In order to have model uniqueness, the system in (14) should only admit a solution ifv = x 1 = · · · = x G .
If the rank condition in (15) is not satisfied, there is more than one solution forany choice of x 1 , . . . , x G ∈ X .
Note that this necessary condition does not take into account the dimension of the model.
As discussedin Section 3, a sufficient condition for model uniqueness must depend on the dimension of the signalset k .
Our main theorem shows that k additional measurements per operator are sufficient for modelidentification:
Theorem 4.2.
For almost every set of G mappings A 1 , . . .
, A G ∈ R m × n , under assumption A1 thesignal model X can be uniquely recovered if the number of measurements verify m > k + n/G .
The proof is included in Appendix A. If we have a large number of independent operators G ≥ n ,
Theorem 4.2 states that only m > k + 1 measurements are sufficient for model identification, whichis slightly smaller (if the model is not trivial, i.e., k > 1 ) than the number of measurements typicallyneeded for signal recovery m > 2 k .
In this case, it is possible to uniquely identify the model,without necessarily having a unique reconstruction of each observed signal.
However, as discussedin Section 2, for k < m ≤ 2 k , the subset of signals which cannot be uniquely recovered is at most(2 k − m ) -dimensional.
Operators with Different Number of Measurements
The results of the previous subsectionscan be easily extended to the setting where each measurement operator has a different number ofmeasurements, i.e. A 1 ∈ R m 1 × n , . . .
, A G ∈ R m G × n .
In this case, the necessary condition in
Proposition 4.1 is (cid:80) Gg =1 m g ≥ n , and the sufficient condition in Theorem 4.2 is 1 G (cid:80) [-G g =
-]Gg =1 m g >k + n/G .
As the proofs mirror the ones of Proposition 4.1 and Theorem 4.2, we leave the details tothe reader.
Noisy measurement data Surprisingly, the results of this section are also theoretically valid if themeasurements are corrupted by independent additive noise ϵ , i.e., y =
A g x + ϵ , as long as the noisedistribution is known and has a nowhere zero characteristic function (e.g., Gaussian noise):
Proposition 4.3.
For a fixed noise distribution, if its characteristic function is nowhere zero, thenthere is a one-to-one mapping between the space of clean measurement distributions and noisemeasurement distributions.
The proof is included in Appendix A.
If the clean measurement distribution can be uniquely identifiedfrom the noisy distribution, the results in Theorem 4.2 and Proposition 4.1 also carry over to thenoisy setting.
Note that this only guarantees model identifiability and makes no claims on the samplecomplexity of any learning process.
Algorithms
Unsupervised algorithms mainly come in two flavours: we can first learn a model ˆ X to then reconstructby projecting measurements into this set, or we can attempt to directly learn the reconstructionfunction parameterized by a deep network.
5.1 Learn the Model and Reconstruct
Dictionary and Subspace Learning
If X is (approximately) a union of subspaces [9] or a singlesubspace [8], we can learn a model by
[Equation]
16) where ρ 1 ( D ) and ρ 2 ( z ) are regularisation terms that promote low-dimensional solutions, e.g., sparsecodes z if X is a union of subspaces.
At test time, the dictionary is fixed and the optimization isperformed over the codes only.
AmbientGAN Complex datasets are often better modelled by a generative network f : R k
(cid:55)→ R nwhose input is a low-dimensional latent code z ∈ R k .
The generative model can be learned using anadversarial strategy, i.e.,
[Equation]
(17) where d : R n (cid:55)→ R n is the discriminator network which compares measurements in the imagedomain, z is usually sampled from a Gaussian distribution, and q ( t ) = log( t ) for standard GANs andq ( t )
=
t for Wasserstein GANs.
At test time, the reconstruction can be obtained by finding the latentcode that best fits the measurements ˆ x = f (arg min z ∥
y − A g f ( z ) ∥ 2 ) , as in [30].
5.2 Learn to Reconstruct
Another approach consists in learning directly the reconstruction mapping f : R m × R m × n (cid:55)→ R nwhose inputs are the noisy measurement y and the associated operator A g , and the output is thereconstructed signal x .
The reconstruction can have either a denoiser form, f ( y, A g ) =
˜ f
( A † g y )where ˜ f is independent of A g [1], or a more complex unrolled structure with many denoising andgradient steps [31].
Measurement Splitting Inspired by the Noise2Noise approaches [17, 18], some self-supervisedmethods [19, 20] split each measurement into two parts, y ⊤ =
[ y ⊤ 1 , y ⊤ 2 ] , such that the input is y 1 andthe target is y 2 .
These methods can be summarised as minimising the following loss
[Equation] [Equation]
where y
=
A g, 1 x + ϵ
and y
=
A g, 2 x + ϵ 2 .
This approach suffers from the fact that f does not useall the available information in a given measurement, as it attempts to solve a harder reconstructionproblem associated with y 1 =
A g, 1 x + ϵ 1 .
As reconstruction networks often fail to generalise tooperators with more measurements [32], this method can suffer from suboptimal reconstructions attest time.
Proposed Method The analysis in Section 4 shows that model identifiability necessarily requiresthat reconstructed signals are consistent with all operators A 1 , . . . , A G . Thus, we propose anunsupervised loss that ensures consistency across all projections A g , that is
[Equation]
where ˆ x = f ( y, A g ) .
The first term ensures measurement consistency y =
A g f
( y, A g ) , whereasthe second term enforces consistency across operators, i.e., f ( y, A g ) =
f ( A s f ( y, A g ) , A s ) for allg ̸ = s .
Crucially, the second term prevents the network from learning the trivial pseudo-inversef ( y, A g )
=
A † g y .
In practice, we choose an operator
A s uniformly at random per minibatch.
Thus,compared the supervised case, we only require an additional evaluation of f and A s per minibatch.
We coin this approach multi-operator imaging (MOI).
This loss overcomes the main disadvantages of previous approaches: it doesn’t require traininga discriminator network, and it learns to reconstruct using all the available information in eachmeasurement y .
Experiments
In this section, we present a series of experiments where the goal is to learn the reconstructionfunction with deep networks using real datasets.
Experiments on low-dimensional subspace learningusing synthetic datasets are presented in Appendix C.1.
All our experiments were performed usingan internal cluster of 4 NVIDIA RTX 3090 GPUs with a total compute time of approximately 48 hs.
Compressed Sensing and Inpainting with MNIST We evaluate the theoretical bounds on the
MNIST dataset, based on the well known approximation of its box-counting dimension k ≈ 12 [33].
The dataset contains N = 60000 training samples, and these are partitioned such that N/G differentsamples are observed via each operator.
The forward operators are compressed sensing (CS) matriceswith entries sampled from a Gaussian distribution with zero mean and variance m − 1 .
The test setconsists of 10000 samples, which are also randomly divided into G parts, one per operator.
To evaluatethe theoretical bounds, we attempt to minimize the impact of the inductive bias of the networks’architecture [34, 35] by using a network with 5 fully connected layers and relu non-linearities.
Figure 2a shows the average test peak-signal-to-noise ratio (PSNR) achieved by the model trainedusing the proposed MOI loss for G = 1 , 10 , 20 , 30 , 40 and m = 50 , 100 , 200 , 300 , 400 .
The resultsfollow closely the bound presented in Section 4 which is indicated by the red dashed line, as thenetwork is only able to learn the reconstruction mapping when the sufficient condition m > k + n/Gis verified.
In sampling regimes below this condition, the performance is similar to simply applyingthe pseudo-inverse A † g .
We also evaluate the reconstruction for a different number G of random inpainting masks anddifferent rates m .
The inpainting operators have a diagonal structure which has zero measure in 309 R m × n , however our sufficient condition still provides a reasonable lower bound on predicting theperformance, as shown in Figure 2b.
It is likely that due to the coherence between measurementoperators and images (both operators and MNIST images are sparse), more measurements are requiredto obtain good reconstructions than in the CS case.
[Figure]
Inpainting with CelebA We evaluate the unsupervised methods in Section 5 on the CelebAdataset [36], which is split into 32556 images for training and 32556 images for testing.
We usethe same U-Net architecture (see Appendix B for details) for MOI and supervised learning, andthe DCGAN architecture [37] for AmbientGAN as in [7].
We observed that using the U-Net for
AmbientGAN’s generator achieves worse results than the DCGAN architecture.
Reconstructed testimages are shown in Figure 3 and average test PSNR is presented in Table 1.
The proposed method
[Figure]
[Figure]
† y
AmbientGAN
MOI (ours)
Supervised
Inpainting/CelebA 9.
[Table]
[Table]
[Table]
[Table]
[Table]
A
[Table]
Meas.
Splitting
MOI (
[Table]
Supervised
Acc.
MRI/FastMRI
[Table]
[Table]
[Table] obtains an improvement of more than 4 dB with respect to AmbientGAN and falls only 2 . 1 dB behindfully-supervised learning. Accelerated MRI with FastMRI Finally, we consider the FastMRI dataset [38], where the setof forward operators A g consist of different sets of single-coil k -space measurements, with 4 ×acceleration, i.e., m/n = 0 .
We used 900 images for training and 74 for testing, which we splitacross G = 40 operators.
We compare measurement splitting, MOI and supervised learning, allusing the same U-Net architecture.
For the measurement splitting method, we follow the strategyin [19], and choose to assign a random subset representing 60% of the measurements in A g to A g, 1and the remaining to A g, 2 .
We observed that a model trained with measurement splitting obtainedless test error using reduced measurements associated with A g, 1 instead of the full measurements A g (see Appendix B), so we report the best results using A g, 1 .
As observed in [32], the networkfails to generalize to operators with more measurements.
Average test PSNR is presented in Table 1.
Reconstructed test images are shown in Appendix C.
The proposed method obtains .
8 dB improvementover measurement splitting and is approximately 2 dB worse than the fully supervised setting.
Limitations
Theorem 4.2 does not cover cases where the operators A g present some problem specific constraints(e.g.
, they are inpainting matrices) as well as cases where the signal model is only approximatelylow dimensional.
Note however that Proposition 4.1 applies to constrained operators.
We leave thestudy of sufficient conditions for these particular cases for future work.
The proposed loss might notbe effective in problems where learning the reconstruction function is impossible, e.g., due to veryhigh noise affecting the measurements [4].
In this particular case, it might still be possible to learn agenerative model as in AmbientGAN.
Conclusions
We have presented sampling theorems for the unsupervised learning of signal models from incompletemeasurements using multiple measurement operators.
Our bounds characterize the interplay betweenthe fundamental properties of the problem: the ambient dimension, the data dimension and the numberof measurement operators.
Moreover, the bounds are agnostic of the learning algorithms and provideuseful necessary and sufficient conditions for designing principled sensing strategies.
Furthermore,we presented a new practical unsupervised learning loss which learns to reconstruct incompletemeasurement data from multiple operators, outperforming previously proposed unsupervised methods.
The proposed strategy avoids the adversarial training in AmbientGAN, which can suffer from modecollapse [39], and, contrary to measurement splitting, is trained using full operators A g .
Our resultsshed light into the setting where access to ground truth data cannot be guaranteed which is of extremeimportance in various applications.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
We used existing models and datasets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] All datasets considered in the paper are publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
We did not use any crowdsourcing or conducted research with [Table] human subjects. [Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table] Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
[Table]
A Proofs
The proof of Theorem 4.2 utilises the following technical lemma:
Lemma A.1 (Lemmas 4.5 and 4.6 in [25]) .
Let S be a bounded subset of R n , and let G 0 , G 1 , . . .
, G tbe Lipschitz maps from S to R m .
For each integer r ≥ 0 , let S r be the subset of z ∈ S such that therank of the m × t matrix
[Equation]
20) is r , and let
boxdim ( S r ) =
k r .
For each α ∈ R t define G α ( z ) =
G 0 + Φ z α .
If for all integers r ≥ 0we have that r > k r , then G − 1 α (0) is empty for almost every α ∈ R t .
Proof.
The proof of Lemma A.1 follows standard covering arguments and may be sketched asfollows.
From the dimensionality assumption, the set S r can be essentially covered by O ( ϵ − k r )ϵ -balls.
Furthermore, for any z ∈ S r , the probability (measured with respect to α ∈ R t ) that G α ( z )maps to the neighborhood of 0 scales as ϵ r .
Hence the probability of this happening for any of thepoints in the cover scales as ϵ r − k r .
If we take r > k r then the probability of such an event tends tozero as we shrink ϵ .
Full details can be found in the proofs in [25].
We can now present the proof of Theorem 4.2:
Proof.
In order to have model uniqueness, we require that the inferred signal set ˆ X defined as
[Equation]
(21) equals the true set X , or equivalently that their difference
[Equation]
G − v )
= 0 , x 1 , . . .
, x G ∈ X} (22) is empty, where \ denotes set difference.
Let S ⊂ R n ( G +1) be the set of all vectors z =[ v, x 1 , . . .
, x G ] ⊤ with v ∈ R n \ X and x 1 , . . .
, x G ∈ X .
The difference set defined in (22) isempty if and only if for any z ∈ S we have
[Equation]
where G α maps z ∈ S to R mG .
Let α = [ vec ( A 1 ) ⊤ , . .
, vec ( A G ) ⊤ ] ⊤ ∈ R mnG , then as a functionof α we can also write (23) as
[Equation]
where ⊗ is the Kronecker product and we used the fact that A ( x g − v ) =
( x g − v ) ⊤ ⊗ I m vec ( A ) .
As v does not belong to the signal set, the matrix on the left hand side of (25) has rank mG for allz ∈ S .
We treat the cases of bounded and conic signal sets separately, showing in both cases that, foralmost every α ∈ R mnG , the condition in (25) holds for all z ∈ S if m > k + n/G :
Bounded signal set Let S ρ be a subset of S defined as
[Equation]
, x ⊤ G ] ⊤ , x 1 , . . .
, x G ∈ X , ∥ v ∥ 2 ≤ ρ } .
As S ρ is bounded, we have boxdim ( S ρ ) ≤ kG + n .
Thus, if mG > kG + n , Lemma A.1states that for almost every α , (25) holds for all z ∈ S ρ .
As S can be decomposed as acountable union of S ρ of increasing radius, i.e., S =
(cid:83) ρ ∈ N S ρ , and a countable union ofevents of measure zero has measure zero, then for almost every α all z ∈ S verifies (25) ifm > k + n/G .
Conic signal set If the signal set is conic, then S is also conic.
Hence, due to the linearity of (23)with respect to z , there exists z ∈ S which does not verify (23) if and only if for any boundedset B containing an open neighbourhood of 0 , there exists a z ∈ S ∩ B which does notverify (23).
As boxdim ( S ∩ B ) ≤ Gk + n , Lemma A.1 states that for almost every α , allz ∈ S verifies (25) as long as m > k + nG .
We end this section with the proof of Proposition 4.1:
Proof.
Consider the noisy measurements associated to the g th operator A g , as z = y + ϵ , where z arethe observed noisy measurements, y are the clean measurements and ϵ is additive noise (independentof y ).
The characteristic function of the sum of two independent random variables is given by themultiplication of their characteristic functions, i.e.,
[Equation]
27) where φ z , φ y and φ ϵ are the characteristic functions the noisy measurement, clean measurementsand noise distributions, respectively.
If the characteristic function of the noise distribution is nowherezero, we can uniquely identify the characteristic function of the clean measurement distribution as
[Equation]
The clean measurement distribution is fully characterized by its characteristic function φ y ( w ) .
Weend the proof by noting that the same reasoning applies to the measurements of every operator
A gwith g ∈ { 1 , . . .
, G } .
B Training Details
Algorithm 1 provides the pseudo-code of the proposed multi-operator imaging (MOI) method.
Thetraining details for each task are as follows:
[Equation]
[Equation]
operators
[Equation]
reconstruction function (
[Equation]
g
[Equation]
, neural network) for y, A g in loader: # load a minibatch y with N samples
[Table]
Compressed Sensing and Inpainting with MNIST.
In both cases, we use the Adam optimizerwith a batch size of 128 and weight decay of 10 − 8 .
We use a fully connected network with 5 layers,where the number neurons in each layer are 784 , 1000 , 32 , 1000 , 784 respectively.
The nonlinearityis relu and the network has a residual connection between the input and output.
For the CS task, weuse an initial learning rate of 10 − 4 and train the networks for 1000 epochs, keeping the learning rateconstant for the first 800 epochs and then shrinking it by a factor of 0 . 1 .
For the inpainting task, weuse an initial learning rate of 5 × 10 − 4 and train the networks for 500 epochs, keeping the learningrate constant for the first 300 epochs and then shrinking it by a factor of 0 . 1 .
Inpainting with CelebA. The CelebA dataset contains more than 200K celebrity images, each with40 binary attributes.
We pick the attribute smile to evaluate the proposed method.
The center part ofthe aligned images in the CelebA dataset are cropped to 128 × 128 .
We divide the selected imagesinto two subsets for training and testing.
There are 32557 images in each subset, which we split across
G = 40 operators.
For the inpainting task, we use the U-Net architecture (see Figure 4) to implement
[Figure]
C ⨁
C the reconstruction function f
( y, A ) , and the DCGAN architecture for AmbientGAN.
Using the U-Netarchitecture for AmbientGAN’s generator obtains an average test PSNR of 27 .
5 ± 1 . 3 dB, whichis 2.1 dB below the performance obtained by the DCGAN generator reported in Section 6.
We use
Adam with a batch size of 20 , an initial learning rate of 5 × 10 − 4 and a weight decay of 10 − 8 .
Wetrain the networks for 300 epochs, shrinking the learning rate by a factor of 0 .
1 after the first 200epochs.
Accelerated MRI with fastMRI.
For the 4 × accelerated MRI task, we use Adam with a batchsize of 4 and an initial learning rate of 5 × 10 − 4 .
We also use the U-Net in Figure 4 to implementthe reconstruction function f
( y, A ) .
We train the networks for 500 epochs, keeping the learningrate constant for the first 300 epochs and then shrinking it by a factor of 0 . 1 .
In all experiments, weuse complex-valued data and treat real and imaginary parts of the images as separate channels.
Forthe purpose of visualization, we display only the magnitude images.
Reconstructed test images areshown in Figure 5.
The measurement splitting method performs worse at test time if we use the full measurements yassociated with A g instead of the reduced measurements y 1 associated with A g, 1 .
Using the fullmeasurements obtains an average test PSNR of 27.3 dB, i.e., a decrease of 1.4 dB with respect to theperformance using A g, 1 reported in the main paper.
[Figure]
C Additional Experiments
C.1 Subspace Learning
We consider the problem of learning a k -dimensional subspace model from partial observations, wherethe signals x i are generated from a standard Gaussian distribution on the low-dimensional subspace.
The observations y i are obtained by randomly choosing one out of G operators A 1 , . . .
, A G ∈ R m × n ,each composed of iid Gaussian entries of mean 0 and variance n − 1 .
In order to recover the signal matrix X =
[ x 1 , . . .
, x N ] , we solve the following low-rank matrix recovery problem
[Equation]
[Equation]
where ∥ · ∥ ∗ denotes the nuclear norm
A recovery is considered successful if (cid:80)
[Equation]
<10 − 1 , where ˆ x i is the estimated signal for the i th sample.
We use a standard matrix completionalgorithm [40] to solve (29).
The ambient dimension is fixed at n = 50 , and the experiment isrepeated for k = 1 , 10 , 40 .
For each experiment we set N = 150 k in order to have enough samplesto estimate the subspaces [14].
Figure 6 shows the probability of recovery over 25 Monte Carlo trialsfor different numbers of measurements m and operators G .
The reconstruction probability exhibits asharp transition which follows the bound presented in Theorem 4.2, i.e., m > k + n/G .
[Figure]
C.2 Learning with Unrolled Network Architecture
Section 6 of the main paper presents results comparing the proposed unsupervised loss and competingapproaches using the denoising architecture, i.e. f ( y, A ) =
˜ f
( A † y ) , where ˜ f is the U-Net presentedin Figure 4.
Here we are interested in investigating the performance of MOI and other competingapproaches using a more complex unrolled architecture.
We use an unrolled proximal gradientdescent (PGD) network (see Algorithm (30)) with T = 3 iterations.
The step size is initialized asη ( t )
= 0 .
and is then learned during training.
We employ 3 U-Net networks using the architecturepresented in Figure 4 to implement f ( t ) for t = 1 , 2 , 3 (no weight sharing across PGD iterations).
Theaverage test PSNR is shown in Table 2 and reconstructed images are shown in Figure 7.
All trainingapproaches obtain a better performance than using the denoising network due to the architecturalimprovements.
MOI obtains 1.9 dB improvement over measurement splitting and is 1 dB worse thansupervised learning. 
Unrolled Proximal Gradient Descent (PGD) input: y, A x (0) ← A † y for t = 0 , 1 , · · · , T − 1 : x ( t +1)
← f ( t +1)
( x ( t ) − η ( t ) A ⊤ ( Ax ( t ) − y )) end for return f
( y, A ) :
= x ( T ) (30)
Unrolling (PGD-3)
A † y
Meas.
Splitting
MOI (ours)
Supervised
Acc.
MRI/FastMRI
[Table]
[Figure]