S INGLE -P HOTON I MAGE C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Quantum Computing based Machine Learning mainly focuses on quantum computing hardware that is experimentally challenging to realize due to requiring quantum gates that operate at very low temperature.
We demonstrate the existence of a “quantum computing toy model” that illustrates key aspects of quantum information processing while being experimentally accessible with room temperature optics.
Pondering the question of the theoretical classiﬁcation accuracy performance limit for MNIST (respectively “Fashion-MNIST”) classiﬁers, subject to the constraint that a decision has to be made after detection of the very ﬁrst photon that passed through an image-ﬁlter, we show that a machine learning system that is permitted to use quantum interference on the photon’s state can substantially outperform any machine learning system that can not.
Speciﬁcally, we prove that a “classical”
MNIST (respectively “Fashion-MNIST”) classiﬁer cannot achieve an accuracy of better than 22 .
96% (respectively 21 .
38% for “Fashion-MNIST”) if it must make a decision after seeing a single photon falling on one of the 28 × 28 image pixels of a detector array.
We further demonstrate that a classiﬁer that is permitted to employ quantum interference by optically transforming the photon state prior to detection can achieve a classiﬁcation accuracy of at least 41 .
27% for MNIST (respectively 36 .
14% for “Fashion-MNIST”).
We show in detail how to train the corresponding quantum state transformation with TensorFlow and also explain how this example can serve as a teaching tool for the measurement process in quantum mechanics.
I NTRODUCTION
Both quantum mechanics and machine learning play a major role in modern technology, and the emerging ﬁeld of AI applications of quantum computing may well enable major breakthroughs across many scientiﬁc disciplines.
Yet, as the majority of current machine learning practitioners do not have a thorough understanding of quantum mechanics, while the majority of quantum physicists only have an equally limited understanding of machine learning, it is interesting to look for “Rosetta
Stone” problems where simple and widely understood machine learning ideas meet simple and widely understood quantum mechanics ideas.
It is the intent of this article to present a setting in which textbook quantum mechanics sheds a new light on a textbook machine learning problem, and vice versa, conceptually somewhat along the lines of Google’s TensorFlow Playground (Smilkov et al.
(2017),) which was introduced as a teaching device to illustrate key concepts from Deep Learning to a wider audience.
Speciﬁcally, we want to consider the question what the maximal achievable accuracy on common one-out-of-many image classiﬁcation tasks is if one must make a decision after the detection of the very ﬁrst quantum of light (i.e. photon) that passed a ﬁlter showing an example image from the test set.
In this setting, we do not have a one-to-one correspondence between example images from the training (respectively test) set and classiﬁcation problems.
Instead, every example image deﬁnes a probability distribution for the ( x, y ) detector pixel location on which the ﬁrst photon passing an image ﬁlter lands, the per-pixel probability being the pixel’s brightness relative to the accumulated (across all pixels) image brightness.
So, from every ( 28 × 28 pixels) example image, we can sample arbitrarily many photon-detection-event classiﬁer examples, where the features are a pair of integer pixel coordinates, and the label is the digit class.
On the MNIST handwritten digit dataset (LeCun and Cortes (2010)), any machine learning system that only gets to see a single such “photon detected at coordinates ( x, y ) ” event as its input features, of the pixel that ﬂashed up are the only input features, is limited in accuracy by the maximum likelihood estimate, since we have: P ( Image class C | Photon detected at ( x, y ) )
= (cid:80)
E P ( Image class C |
Example E ) P ( Example E | Photon detected at ( x, y ) ) .
On photon detection events generated each by ﬁrst randomly picking a training set image, and then randomly picking a brightness-weighted pixel from that, we cannot do any better than predicting the most likely digit class given these input features – the two pixel coordinates.
Evaluating the performance of this proven-optimal-on-examples-generated-from-training-set MLE classiﬁer on examples generated from the test set, we ﬁnd an accuracy of 22 .
96% (even if classiﬁer training permitted ‘cheating’ by looking at the test set), for the MNIST dataset – substantially higher than random guessing ( 10% ).
The most likely digit class per pixel is shown in ﬁgure 2(b).
This accuracy is, by construction, the best we can hope to achieve for a classiﬁer whose input is the address of a single pixel (a detailed proof is provided in appendix A.2): We cannot, for example, outperform it by redistributing light intensity between pixels, since any such redistribution could only destroy some of the available useful information, not magically create extra useful information.
An entirely different situation arises when we allow quantum mechanics to enter the stage: For a single photon passing through a coherently illuminated image ﬁlter, with all pixels at the same optical phase on the incoming wave, we can imagine putting some precision optical device between the image ﬁlter and the detector array that redistributes not the probabilities (which correspond to light intensity when aggregating over many photons), but the amplitudes that make up the spatial part of the photon wave-function.
Illuminating such a set-up with many photons would show a hologram-like interference pattern on the detector array.
This transformation of the (single-)photon wave function by linear optical elements then has tuneable parameters which we can adjust to improve classiﬁer accuracy.
Quantum mechanics tells us that every (lossless) linear optical device can be represented by a linear unitary transform on the photon state: The action of any complex optical device consisting of (potentially very many) components which transforms a N -component photon state (in our case,
N = 28 2 amplitudes in the spatial part of the photon wavefunction) can be described by an element of the N 2 -dimensional unitary matrix Lie group U ( N ) .
Vice versa, Reck et al.
(1994) describes a constructive algorithm by which any U ( N ) transformation matrix can be translated back to a network of optical beam splitters and phase shifters.
1. R ELATED W ORK
Conceptually, exploiting interference to enhance the probability of a quantum experiment producing the sought outcome is the essential idea underlying all quantum computing.
The main difference between this problem and modern quantum computing is that the latter tries to perform calculations by manipulating quantum states of multiple “entangled” constituents, typically coupled two-state quantum systems called “qubits,” via “quantum gates” that are controlled by parts of the total quantum system’s quantum state.
Building a many-qubit quantum computer hence requires delicate control over the interactions between constituent qubits.
This usually requires eliminating thermal noise by going to millikelvin temperatures.
For the problem studied here, the quantum state can be transformed with conventional optics at room temperature: the energy of a green photon is 2 .
5 eV, way above the typical room temperature thermal radiation energy of kT (cid:39) 25 meV.
The price to pay is that it is challenging to build a device that allows multiple photons to interact in the way needed to build a many-qubit quantum computer.
Nevertheless, Knill, Laﬂamme, and Milburn (Knill et al.
(2001)) devised a protocol to make this feasible in principle, avoiding the need for coherency-preserving nonlinear optics (which may well be impossible to realize experimentally) by clever exploitation of ancillary photon qubits, boson statistics, and the measurement process.
In all such applications, the basic idea is to employ coherent multiphoton quantum states to do computations with multiple qubits.
In the problem studied here, there is only a single photon, the only relevant information that gets processed is encoded in the spatial part of its wave function (i.e. polarization is irrelevant), so the current work resembles the “optical simulation of quantum logic” proposed by Cerf et al.
(1998) where a N-qubit system is represented by 2 N spatial modes of a single photon.
Related work studied similar “optical simulations of quantum computing” for implementing various algorithms, in particular (small) integer factorization (Clauser and Dowling (1996); Summhammer (1997)), but to the best of the present authors’ knowledge did not consider machine learning problems.
This work can be described as belonging to the category of machine learning methods on quantum non-scalable architectures.
Alternatively, one can regard it as a quantum analogue of recent work that demonstrated digital circuit free MNIST digit classiﬁcation via classical nonlinear optics, for instance via saturable absorbers (Khoram et al.
Apart from providing an accessible and commonly understandable toy problem for both quantum and ML research communities, this simplequantum/simple-ML corner also may be of interest for teaching the physics of the measurement process (“the collapse of the wave function”) in a more accessible setting.
Whereas explanations of the measurement process are forced to remain vague where they try to model “the quantum states of the observer” (typically unfathomably many states that one would never hope to be able to model in terms of actual numbers), using machine learning as a sort-of cartoon substitute for high level mental processes actually allows us to come up with fully concrete toy models of the measurement process on low-dimensional (such as: D < 1000 )
Hilbert spaces that nevertheless capture many of the essential aspects – to the extent that “ML classiﬁes the measurement as showing the image of a shoe” can be regarded as a crude approximation to “observer sees a shoe”.
Looking closer at the relation between the present article and Khoram et al.
(2019), both articles study the general feasibility of realizing Machine Learning classiﬁers in the form of an analog optical computer at the theoretical level, using numerical optimization to produce a blueprint of a device that can perform inference for a speciﬁc problem.
In both articles, the primary problem under study is MNIST handwritten digit classiﬁcation, the input is encoded as spatial dependency of a (monochromatic) laser beam’s light intensity, and classiﬁcation happens by using interference to funnel optical energy onto a detector array.
In both cases, stochastic gradient descent is used to shape how this funneling of optical energy happens.
Indeed, even the loss function used for training (cross entropy) is essentially equivalent.
The key differences are that Khoram et al.
(2019) only considers the many-photon limit of classical wave optics, which allows one the luxury of using non-linear optical components, speciﬁcally saturable absorbers, to implement non-linearities.
This has no analog for the single photon case.
Also, having many photons available allows identifying the target class that receives most laser light and calling this the prediction of the model.
This is clearly not possible when a decision has to be made after seeing only a single photon.
If one sent many photons through an interference device as described in this article and picked the target class with the highest photon count, one would observe classiﬁcation accuracies of about 90% rather than the claimed about- 40% for a single photon.
This is considerably higher than the accuracies of about 80% presented inKhoram et al.
as the focus of that article is on manufacturability, running gradient backpropagation directly on a Finite Difference Frequency Domain PDE simulation of Maxwell’s equations and taking materials engineering constraints into account, whereas our work focuses on upper and lower bounds for achievable accuracy, exploiting the one-to-one equivalence between linear optical devices and unitary transforms.
Our work directly trains the parameters of the unitary transform, which only afterwards get mapped to a blueprint for an experimental device realization.
Speculatively, if a device were built experimentally that was designed by the methods inKhoram et al.
(2019), subject to the extra constraint that no non-linear elements can be used, and then deployed in a low-light-intensity single-photon setting, using a suitable detector such as a SPAD array, it may manage to realize better-than-classically-achievable classiﬁer performance, for reasons explained in the current work.
1. T HE MEASUREMENT PROCESS
How well one can one solve a mental processing task, such as identifying a handwritten digit, if one is permitted to only measure a single quantum?
This question leads to a Hilbert space basis factorization that parallels the factorization needed to study the quantum mechanical measurement process.
Let us consider a gedankenexperiment where our quantum system (see Feynman et al.
; Landau and Lifshitz (1981) for an introduction to quantum mechanics) is a single atom that has two experiment-relevant quantum states, ‘spin-up’ and ‘spin-down’,
[Equation]
This atom undergoes a measurement by interacting, over a limited time period, with an apparatus.
The measurement process may involve for instance an atom emitting a photon that is detected by a camera, and it may include a human observing the result.
We describe a quantum state in the potentially enormous Hilbert space of apparatus states with the vector | ψ Apparatus (cid:105) .
If, in this gedankenexperiment, we actually assume that we have maximal information about the (quantum) state of the measurement apparatus (which, however, in practical terms would be unfathomably complicated) at the beginning of the experiment, then the full quantum state of the initial system is the tensor product
[Equation]
This factorization implies that atom and apparatus states are independent before the interaction.
Without interaction between the apparatus and the atom, the time-evolution of the total system factorizes.
A measurement requires an interaction between the apparatus and the atom, the solution of the Schrödinger equation is equivalent to the application of a unitary operator U to the state | ψ System , initial (cid:105) .
This has the effect of combining the state components of the atom and the apparatus and, as a consequence, the joint time evolution no longer can be factorized.
The overall state is | ψ
[Equation] [Equation]
(cid:105) =
U | ψ
[Equation]
, initial (cid:105) and can be always decomposed in the sum:
[Equation]
where the apparatus states | ψ Apparatus , ﬁnal= ↑ (cid:105) and | ψ Apparatus , ﬁnal= ↓ (cid:105) represent the state of the apparatus after the measurement for the two basis states of the atom.
Therefore, the apparatus is in a different state for the two cases, which leads to the apparent “collapse” of the wave function.
The apparatus in the state | ψ Apparatus , ﬁnal= ↑ (cid:105) perceives the “collapse” because the atom seems to have taken the state | ψ Atom= ↑ (cid:105) .
The state of the apparatus includes also the representation of the thought process of a possible human observer, for instance asking herself at what instant the atom took a well determined state.
This thought process disregards the superposed state | ψ Apparatus , ﬁnal= ↓ (cid:105) which represents the alternative reality, where the apparatus observed a different outcome.
Considering that a mental process could be seen as a measurement on the environment, one would naturally be inclined to think that high level mental concepts never would naturally lend themselves to a description in terms of some Hilbert space basis that has tensor product structure | ψ generalconcept (cid:105)⊗ | ψ details (cid:105) .
Machine learning is now making the question to what extent this may nevertheless work quantitatively testable for some simple cases, if we consider it as providing reasonably good (for this purpose) models for mental concepts.
Let us consider the spatial part of a single photon’s quantum state as it traveled through a mask that has the shape of a complicated object.
For instance, let’s assume that the mask is obtained from a random sample of the “Fashion-MNIST” dataset, Xiao et al.
(2017), where each sample represents an object such as a shirt, a trouser, etc.
One would generally expect that any sort of transformation that connects a highly regular and mathematically simple description of such a quantum system, such as in terms of per-picture-cell (“pixel”) amplitudes, with a description in human-interpretable terms, such as “the overall intensity pattern resembles a shirt,” would unavoidably involve very complicated entanglement, and one should not even remotely hope to be able to even only approximately express such photon states in terms of some factorization
[Equation]
since one would not expect the existence of a basis of orthonormal quantum states that can be (approximately) labeled | ψ shirt (cid:105) , | ψ shoe (cid:105) , etc.
Using machine learning, we can quantitatively demonstrate that, at least for some simple examples, precisely such a factorization does indeed work remarkably well , at least if we content ourselves with the concept of a “shirt shape” being that of a one-out-of-many machine learning classiﬁer, so not quite that of a human.
In any case, it is reassuring to see that even near-future few-qbits quantum computers might be able to model high level concepts rather well.
[Equation]
C LASSIFICATION
Our gedankenexperiment starts with a single photon passing from faraway through a programmable
LCD screen, which we here consider to consist of N × N pixels and show an image, where for both the MNIST handwritten digit dataset of LeCun and Cortes (2010) and the “Fashion-MNIST” dataset of Xiao et al.
(2017), we have N = 28 .
The size of the screen shall be sufﬁciently small for the photon’s quantum state to be at the same phase as it reaches each individual pixel.
This does not mean that the screen has to be small in comparison to the wavelength.
Rather, the light source must provide highly collimated illumination.
The relevant spatial part of the photon’s quantum state is described by an element of a N × Ndimensional complex vector space.
We can choose a basis for this Hilbert space such that the quantum state of a photon that managed to pass through the screen (rather than getting absorbed) has the form
[Equation]
where the | ψ jk (cid:105) basis functions correspond to a photon that went through pixel ( j, k ) , and the coefﬁcients c jk are real, non-negative, proportional to the square roots of the image’s pixel-brightnesses, and are normalized according to (cid:80) j,k | c j,k | 2 = 1 .
As we want to perform a rotation on this Hilbert space that maximizes alignment with a tensor product Hilbert space where one factor describes an image class, we pad this N 2 -dimensional
Hilbert space into a larger Hilbert space with dimensionality M divisible by the number of object classes C , i.e. M = C · S .
This amounts to adding always-dark pixels (that may not form a complete row) to the image.
The problem then amounts to engineering, for a problem P such as handwritten digit recognition, a single problem-speciﬁc unitary transform U P of the photon state, | ψ Photon (cid:105) → U P | ψ Photon (cid:105) , such that we can meaningfully claim:
[Equation]
Speciﬁcally, for each individual example image E , we would like to have
[Equation]
(7) where C ( E ) is the ground truth label of the example in a supervised learning setting.
Using the method described in Reck et al.
(1994), this trained matrix then can be translated to an optical network blueprint.
The transformed quantum state at the output side of the network of beam splitters and phase shifters then gets measured by a detector array that can discriminate M = C · S quantum states which are labeled | ψ digitisa0 (cid:105) ⊗ | ψ stylevariant1 (cid:105) , | ψ digitisa0 (cid:105) ⊗ | ψ stylevariant2 (cid:105) , ..., | ψ digitisa3 (cid:105) ⊗ | ψ stylevariant57 (cid:105) , ..., | ψ digitisa9 (cid:105) ⊗ | ψ stylevariant S max (cid:105) .
If we detect the photon in any of the | ψ digitisa7 (cid:105) ⊗ . .
cells, the classiﬁer output is a “7”, and likewise for the other digits.
From a machine learning perspective, the trainable parameters hence are the complex entries of the matrix U P , which according to quantum mechanics have to satisfy an unitarity constraint, U P U † P
=
I , and this search space automatically covers all experimentally realizable linear optical devices.
For
MNIST, where examples have 28 × 28 pixels, the most obvious choice is padding to a M = 790 -dimensional input vector.
While one could implement the unitarity constraint in terms of a (regularizer) loss-function contribution that measures the degree of violation of unitarity, it here makes more sense to instead use a parametrization of U P that automatically guarantees unitarity, using Lie group theory.
If W P is a 790 × 790 matrix of trainable (real) weights, then the hermitean matrix
H P = − i
( W P − W TP )
+ ( W P + W TP ) parametrizes the Lie algebra u (790) , and U P = exp( iH P ) covers all of the (compact) unitary group U (790) .
This approach slightly over-parametrizes the problem, since, in the tensor-product basis that we are transforming to, we can freely re-deﬁne the basis on each of the ten 790 / 10 = 79 -dimensional style subspaces.
This means that 10% of the parameters are redundant.
Overall, with all the trainable weights being provided by the matrix W P , and the brightness of the pixel at coordinates ( y, x ) for example E being b E ; yx , we have this formula for the probability of a photon travelling through an optical device that was designed by training weights and landing on a detector cell that predicts class c :
[Equation]
Here, y and x are image row- and column-indices (for MNIST, running from 0 to 27 ), j, k are matrix row- and column-indices (in our example, running from 0 to 789 ) for the exponentiated unitary matrix U P = expm( · · · ) , s is a style-index (here, running from 0 to S − 1 = 78 ), the term under the square root is the relative contribution of the ( y, x ) -pixel to the total brightness of example image E , and the δ -factors are used for translating a pair of row,column image-indices to a linear c =c =(
[Figure]
c =c =
[Figure]
pixel index, respectively an index on the U P -rotated quantum state vector to a pair of (class, style)- indices.
Technically speaking, from the viewpoint of mapping an optical amplitude that describes light intensity passing through the image-ﬁlter to the quantum amplitude of a particular (class, style)- combination, this is simply a linear model (since quantum mechanics is linear), whose linear weights are however speciﬁed in a slightly unusual way, underneath a complex matrix exponential (since quantum mechanics is unitary, i.e. probability-preserving).
The probability to predict a given class c is then obtained by summing over the probabilities associated with the given class (but different style-index).
Model accuracy has to be evaluated with caution: as we need to make a prediction after detection of a single photon, accuracy is the quantum probability of the correct label, averaged over all examples.
Naturally, we can not determine which of the C output classes would receive the most photons (= has highest probability) if all we have is a single photon detection event.
This accuracy, about 40% for the problems considered here, differs substantially from the accuracy that would be obtainable by looking at many photons coming from the same example image, which here typically exceeds 90% , roughly in alignment with the expected achievable performance of a linear model on MNIST.
In other words, probabilities are uncalibrated, and the (non-linear “deep learning”) transformation that would be required to calibrate them cannot be expressed as a unitary operator.
Let us consider a radically simpliﬁed example that illustrates why this method works.
We want to discriminate between only two different shapes (with no further shape variation) on a 2 × 4 pixel screen where each pixel is either “on” or “off”, using only one photon.
Speciﬁcally, let us consider the two Tetris “T” shapes represented in ﬁgure 1(a).
For both shapes, the probability that the single photon arrives on one of the “on” pixels is 1 / 4 ; therefore, taking into account that for two pixels the correct shape is identiﬁed exactly and for two with 50% probability, we conclude that the baseline accuracy is 1 / 2 + 1 / 4 = 75% .
Instead, we can apply a unitary transformation to reshape the probability amplitudes.
Let us now consider the simple but not optimal transformation of the photon amplitude that replaces the pair of amplitudes ( a, b ) in each 2-pixel column with (( a − b ) / √ 2 , ( a + b ) / √ 2) , i.e. creates destructive interference in the top row and constructive interference in the bottom row.
This gives the detection probability patterns shown in ﬁgure 1 (b).
Maximum likelihood estimation here gives an accuracy of 1 / 2 + 3 / 8 = 87 .
Obtaining the maximum achievable accuracy will here require a more complicated all-pixel amplitude transformation, obtained as follows:
The quantum amplitude transformation is angle-preserving, and the angle α between the two amplitude quantum states q 1 , q 2 is given by cos α = (cid:104) q 1 | q 2 (cid:105) = 0 .
Hence, we can rotate these two states to lie in the plane of the ﬁrst two Cartesian coordinate axes of the Hilbert space, and at the same angle from their bisector.
Identifying these coordinate axes with the correct labels, the accuracy is the cosine-squared of the angle between the transformed state and the corresponding axis, i.e. cos 2 ( π/ 4 − α/ 2) =
( √ 3 + 2) / 4 ≈ 93 . 30% .
While the performance measure that we care about here is the probability for a correct classiﬁcation, one observes that model training is nevertheless more effective when one instead minimizes crossentropy, as one would when training a conventional machine learning model.
Intuitively, this
[Table]
seems to make sense, as a gradient computed on cross-entropy loss is expected to transport more information about the particular way in which a classiﬁcation is off than a gradient that is based only on maximizing the correct classiﬁcation
Overall, this task is somewhat unusual as a machine learning problem for three reasons: First, it involves complex intermediate quantities, and gradient backpropagation has to correctly handle the transitioning from real to complex derivatives where the loss function is the magnitude-squared of a complex quantity.
TensorFlow is at the time of this writing the only widely used machine learning framework that can handle this aspect nicely.
Second, (as explained above), we cannot simply pick the class for which the predicted probability is highest as the predicted class.
Rather, the probability for the single-photon measurement to produce the ground truth label sets the accuracy.
Third, while most machine learning architectures roughly follow a logistic regression architecture and accumulate per-class evidence which gets mapped to a vector of per-class probabilities, we here have the probabilities as the more readily available data, so the computation of cross-entropy loss will have to infer logits from probabilities.
Due to this need to compute logarithms of probabilities, it is very important that the training process does not intermediately see invalid probabilities outside the range (0 . . .
1) , and this is ensured by parametrizing unitary transforms as matrix exponentials of anti-hermitean matrices.
TensorFlow code to both train such a model and also evaluate its performance is included in the supplementary material.
R ESULTS
Figure 2 shows the most probable image class for each pixel, for the “Fashion-MNIST” and MNIST datasets.
A classiﬁer that looks up and predicts the most likely class in this table achieves maximal accuracy among all single photon classiﬁers that do not employ quantum interference.
This includes classiﬁers that have had access to the test set during training.
This accuracy is reported in the third column of table 1.
(We note that, as pointed out by Sun et al.
(2007), the “Fashion-MNIST” dataset contains many mislabeled instances, which affects both classical and quantum results.)
We can compute the amount of information provided by the photon by computing the difference between the class entropy, i.e. − log 2 (0 . 1) = 3 .
32 , since there are 10 classes, and the entropy associated to the classiﬁcation errors, i.e. the accuracy.
The mutual information for the classical classiﬁer is given in the fourth column of table 1.
Training a unitary U (790) quantum transformation that gets applied after the photon passed the image ﬁlter and before it hits a bank of 790 detectors allows boosting accuracy for both the “Fashion-MNIST” and MNIST datasets, as reported in the ﬁfth column of table 1.
The observation of the photon after the transformation provides a higher amount of mutual information with respect to the classical case.
The values of mutual information in this case are given in the last column of table 1.
Explicit matrices to perform the transformation for the two data sets have been made available with the supplementary material.
The quantum transformation U P allows us to deﬁne the pixel-space projection operators:
[Equation]
(9) with which we can decompose any example into contributions that are attributable to the different classes.
Here, one must keep in mind that such separation is done at the level of probability
[Figure] [Figure]
amplitudes, so while we can compute intensities/probabilities from these components, which are mutually orthogonal as quantum states, summing these per-component per-pixel intensities will not reproduce the example’s per-pixel intensities.
This shows most clearly when considering the decomposition of an example “Trouser” from the “Fashion-MNIST” dataset’s test set with the model we trained for this task, as shown in ﬁgure 4(a).
The dark vertical line between the legs in the original image mostly comes from destructive interference between a bright line from the “Trousers” component and a matching bright line from the “Dress” component.
Due to the intrinsic quantum nature of this set-up, care must be taken when interpreting confusion matrices.
Naturally, we never can claim of any single-photon classiﬁer that it would ‘classify a particular image example correctly’, since re-running the experiment on the same example will not see the photon always being counted by the same detector!
So, strictly speaking, for any single-photon classiﬁer realized as a device, the “confusion matrix” could be determined experimentally only in the statistical sense, leaving uncertainty in the entries that decreases with the number of passes over the test set.
Confusion matrices are shown in ﬁgure 3.
Our factorization ansatz appears to contain a hidden constraint: we are forcing each image class to use the same number of style-states.
One could imagine, for instance, that a classiﬁer might achieve even
[Figure]
higher accuracy by treating image classes unevenly.
Any such model that allows more style-space dimesions for some classes can always be embedded into a model that allows more style-space dimensions for all classes, so this question can be answered by padding to a larger Hilbert space.
Numerical experiments, e.g. padding to 1000 rather than 790 dimensions, suggest that this has no appreciable impact on classiﬁcation accuracy.
D ISCUSSION
In summary, we demonstrated that, at least for the considered datasets, the space of the single observed photon state can be factorized remarkably well by a product of the example class space and a space collecting the remaining variables, such as the style.
This factorization can be obtained easily by the proposed method and is experimentally realizable with optical elements placed in front of the sensor.
The supplementary material contains a blueprint for an example circuit outperforming the classical limit (at 36 .
05% accuracy) on 10 × 10 downsampled MNIST.
An experimental implementation of the proposed system would be a demonstration of a hightemperature and low- effective-qubit quantum ML device.
With respect to other experimental approaches to quantum computing, such a device would have the limitation that it is built for the speciﬁc classiﬁcation problem and cannot be reconﬁgured easily.
It would be interesting to see whether an advanced quantum protocol along the lines of Knill et al.
(2001) might enable the realization of more sophisticated intermediate-scale high temperature quantum machine learning in a way that mostly (like here) bypasses the need for quantum logic built from common quantum gates.
T ENSOR F LOW C ODE
TensorFlow2 code to reproduce the experiments of this work and all the ﬁgures is provided in the ancillary ﬁles together with the computed unitary tranformations for MNIST and “Fashion-MNIST”.
A A PPENDIX A. E XPERIMENT S CHEMATICS
A
B
C
D
Figure 5 shows the schematics of an exper- imental set-up: The lens (A) stylizes the last optical component of the monochro- matic, coherent, linear-polarized (i e. laser) light source that emits photons
en-
tering from the left and travelling to the right.
Light intensity is controlled (e.g. by means of an absorbing ﬁlter, not shown) to be so low that photons travel through the apparatus individually.
Any interference effects are hence due to self-interference of a single photon’s wave function (just as in the double slit gedankenexperiment).
The laser photons then hit a coherently illuminated N × N screen (B) (e.g. a LCD screen, in this diagram 10 × 10 ) which allows light to pass through a given pixel with coordinates ( y, x ) with a probability that is proportional to the ink density on the example image.
The photon arrives at each pixel with the same optical phase (i.e. having travelled the same (fractional) number of wavelengths as seen from the laser).
The diagram shows an exemplary raster image of a digit zero with maximal brightness (maximal ink density on the digitized ML example) on 14 pixels (with zero-based row/column coordinates ( y, x ) =
, (5 , 7) ), with
75% brightness on three pixels (coordinates (3 , 3) , (6 , 3) , (6 , 5) ), and 50% brightness on one pixel (coordinates (7 , 6) ).
The ‘Classical Baseline’ set-up does not use interference and would work just as well in a world where photons are ‘Particles of Light’ that cannot self-interfere (as envisioned by Newton).
Photons coming from the light source (A) will keep hitting the screen (B), frequently getting absorbed by a dark pixel.
At some point, a photon will (by chance) manage to hit a non-dark pixel and not get absorbed (the more likely the brighter the speciﬁc pixel) and travel on to the single-photon detector array (D) (such as: a SPAD array) and be detected as having passed through a speciﬁc pixel.
Ignoring experimental imperfections that could in principle be made small such as optical losses, the only possible transform on the photon state one could perform with a passive-linear optical device at (C) would be equivalent to coupling the photon into an array of optical ﬁbers, routing each ﬁber from one pixel to some other pixel, and coupling out the photon at the other side of the device.
This is equivalent to re-shufﬂing the pixels, which can always be un-done by re-shufﬂing the addresses of the cells of the detector array (D) and so does not affect classiﬁcation accuracy – the diagram hence omits such transformations that cannot affect performance.
The quantum set-up (bottom) is perhaps easiest to analyze via Feynman’s path integral interpretation of Quantum Mechanics:
There are different ‘histories’ which lead from the same initial point (a photon coming from the laser) to the same ﬁnal result (the photon being detected at a speciﬁc pixel, such as: ‘at coordinates (4, 5)’), and the prescription is that we have to attribute a complex quantum amplitude to each such ‘history’, summing over all amplitudes that connect the same initial and ﬁnal state to get the resultant amplitude, and obtaining the associated probability as the magnitude-square of the (complex) resultant probability.
We can also consider the resultant per-pixel quantum amplitudes for a photon having travelled from the light source (A) not all the way to the detector but to some intermediate point, such as just after passing the screen (B).
These are described by vectors with N 2 entries, one per pixel, whose absolute-magnitude-squares sum to 1.
In the example, the photon-state-vector on any ﬂat, screen-parallel surface between B and C, | ψ BC (cid:105) , has zero entries for all but the 14+3+1=18 non-dark pixels.
The entries ψ BC [24] , ψ BC [25] , . . .
that correspond the 14 maximally-bright pixels at (2 , 4) , (2 , 5) , . .
. are identical, as the photon reached each pixel at the same optical phase.
Calling this amplitude c B (B for ‘bright’), the amplitudes for the three moderately-bright pixels c M , and the single dim-but-not-dark pixel’s amplitude
c D , the amplitude magnitude-squares must be proportional to pixel brightness (probability for a photon to pass through the image) and sum to 1 (total probability for the photon that passed through the screen to have passed through a pixel).
As the complex quantum amplitude phase matches the optical phase, these constraints ﬁx c B = u · (cid:112)
/Z
≈ 0 .
244 , c M = u · (cid:112) 0 .
75 /Z
≈ 0 .
212 , c D = u · (cid:112) 0 .
5 /Z
≈ 0 .
173 , with Z being the overall normalization factor that makes the sum of magnitude-squares of all amplitudes 1, i.e. Z = 14 · 1 + 3 · 0 .
75 + 1 · 0 . 5 , and u being some complex number of magnitude 1 , the non-observable overall quantum phase factor.
In the ‘Quantum’ set-up, we can employ a linear optical device, built from many beam-splitters and phase shifters (= ‘delay lines’), to adjust self-interference of the photon wave function.
The diagram shows two example paths out of the total 10 × 10 different paths that the photon can take before it reaches the detector array.
Quantum Mechanics tells us that ‘the single photon(!) travels along all these paths simultaneously’
– this is just Young’s double slit experiment in a slightly more complicated setting.
There is a 1-to-1 correspondence between physically realizable linear optical devices and probability-preserving (generalized) rotations of the quantum state vector.
The (in this downsampled example: 100 × 100 ) components of such a transformation form a unitary matrix.
If we used a linear optical device that implemented a random such transformation, and sent many photons through the apparatus, they collectively would produce an image conceptually resembling the interference pattern on photo ﬁlm that codiﬁes a hologram.
Using a basic Machine Learning procedure, stochastic gradient descent, we can train the parameters of the transform such that photons coming from an image that shows a digit ‘0’ preferentially land on the 0-th row of the detector array, photons coming from an image that shows a digit ‘4’ preferentially land on the 4-th row, etc.
The supplementary material describes a speciﬁc set-up in terms of optical components that reaches > 36% accuracy for MNIST downsampled to 10 × 10 : If this were manufactured from ideal-quality optical components, the probability for a photon that passed through the screen (B) to land on the detector row matching its digit-class is better than 36% .
A. C LASSICAL B ASELINE A CCURACY T HRESHOLD : M AXIMALITY P ROOF
Elementary statistical considerations allow us to obtain a stringent upper bound for the maximum accuracy that cannot be exceeded by any classiﬁer which satisﬁes these two properties:
• P1: The classiﬁer must make a prediction using as its input only the index of the one detector in the detector-array that received the ﬁrst photon.
It can not use any additional information about the example (but may have been trained with arbitrary information about the example set , even including full knowledge of the training and test set).
• P2:
There is a one-to-one correspondence between image pixels and detector-cells:
For each image- pixel, there is exactly one detector-cell such that when the example image E is presented, then the
probability for the ﬁrst photon to land on the detector cell k is proportional to the brightness of the associated pixel in image E .
This bound is what we call the ‘classical accuracy bound’.
The ‘quantum’ classiﬁer violates P2 by employing photon self-interference: The probability for the k -th detector to observe the ﬁrst photon depends on collective information which the photon ‘holographically’ transports about the input image once it passed the image-ﬁlter, rather than on a single pixel.
The protocol for evaluating classiﬁer accuracy is as follows: We pick a random example from the dataset’s test set, set up the device to present this example as a problem, send light towards the ﬁlter-screen, and look at the ﬁrst photon that managed to pass the ﬁlter-screen and get counted by the detector array, then map the index of the detector that counted the photon to a predicted class.
We register ‘successful prediction’ if the predicted class matches the example’s label, otherwise we register an ‘unsuccessful prediction’.
The accuracy is the probability of the prediction to be successful.
Somewhat unusually, this means that there is no such thing as ‘the predicted class of a given image’, as one normally would have it in a Machine Learning problem.
This is due to the inherent randomness of quantum mechanics:
Even repeating classiﬁcation for the same image multiple times, we will see photons land on detectors that correspond to different classiﬁcations.
If we were to experimentally determine accuracy, this would then suffer from the usual problems of determining a probability via a statistical experiments: one can make the likelihood to be way off arbitrarily small, but never shrink it to zero.
However, the aforementioned protocol makes it possible to directly compute the maximal achievable probability for any classiﬁer, without resorting to a statistical experiment.
The gist of our argument parallels the reasoning behind the claim that we can show with simple statistics that no ML classiﬁer can possibly outperform an accuracy threshold of 29 / 36 for predicting from the eye total of rolling two dice whether any of the two dice showed a six.
Here, the reason is that we can get maximal accuracy by looking look at all the possible realizations of any given eye total and make the best possible guess given the situation.
For eye totals 2 − 6 ( 1 + 2 + 3 + 4 + 5 = 15 of 36 cases), we would predict ‘No’ and always be correct.
For eye totals 11 and 12 ( 2+1 additional cases), we would predict ‘Yes’ and also always be correct.
For each other eye total, there are two realizations where one die shows a ‘six’, so we would want to predict ‘Yes’ for eye totals having at most four realizations, i.e. where we have at least a 50% chance of being correct, and ‘No’ otherwise.
Using this approach, we would incorrectly classify three cases as ‘Yes’ ( 5 + 5 , 4 + 5 , 5 + 4 ), and incorrectly classify four cases as ‘No’ ( 6 + 1 , 1 + 6 , 6 + 2 , 2 + 6 ).
Except for these 7 / 36 cases, we would make a correct prediction, so optimal accuracy is 29 / 36 .
Given the perhaps somewhat unfamiliar ‘quantum ML’ setting, and the need to rigorously justify the optimality claim, we prove it below.
The only material difference to the dice-sum example is that relative weights of realizations are not determined by counting, but by looking at pixel brightnesses.
In analogy to the the dice example, the key observation is that the classiﬁer’s input is a single pixel-index, and its output is an image-class.
So, we can completely specify any (deterministic or not) classiﬁer’s action by tabulating, per-pixel-index, what the probability is for this classiﬁer to map the given input pixel-index k to each of the possible output classes c .
The resulting matrix K kc would, for a deterministic classiﬁer, simply be a matrix with one-hot encoded image class, one row per pixel-index.
The classiﬁer’s accuracy is then given by
Accuracy = P ( Classiﬁcation is correct )
[Equation]
[Equation]
Here, P ( E ) is the probability to pick example E from the test set (i.e. 1 / { test set size } ), P ( γ k | E ) is the probability to detect the photon in the detector cell with index k , given the example E , and P ( y E =
C
( γ k )) probability that example E ’s label y E matches the classiﬁer’s output on the input “the photon was detected in cell k ”.
The probability for detecting a photon in cell k when randomly drawing an example image from the test set is P ( γ k )
=
(cid:80) E P ( E ) · P ( γ k | E ) .
The probability for a fairly drawn example’s label to be y c when a photon was detected at cell k is P ( y c | γ k )
=
(cid:80)
E P ( E ) · P ( y E = y c ) · P ( γ k | E ) .
Let us tabulate these P ( y c | γ k ) in the { #pixels } × { #classes } matrix R kc :=
P ( y c | γ k ) .
We then have:
Accuracy =
[Equation]
In words: We can compute accuracy by looking at each detector cell k and each class c , determining the probability P ( γ k ) that, when fairly drawing examples from the test set, a photon gets detected at cell k , and splitting up this probability into contributions from examples where the target class was 0, 1, 2, etc.
These contributions are P ( γ k ) · P ( y c | γ k ) .
We make a correct classiﬁcation when the classiﬁer also predicts class c given the input k .
The classiﬁer’s behavior when given the input k is speciﬁed by row k of the K -matrix, so this probability is K kc .
Here, P ( γ k ) and R kc are determined by the test set.
Each admissible matrix K kc that has (cid:80)
c K kc = 1 speciﬁes a different classiﬁer, and the accuracy is a function of this matrix K only.
The question is now which admissible matrix K maximizes accuracy.
Total classiﬁcation performance (accuracy) is a weighted sum over per-detector- cell performances (the weights being the probabilities to observe a photon in cell k when doing detection experiments on samples drawn fairly from the test set).
Let K 1 be a matrix that maximizes accuracy, and K 2 be a matrix obtained by picking, for each cell-index k , a probability row-vector that maximizes (cid:80)
c R kc K 2 ,kc .
We have Accuracy ( K 1 ) ≥ Accuracy ( K 2 ) (since K 1 is optimal), and also (cid:80)
c R kc ( K 2 ,kc − K 1 ,kc ) ≥
(since K 2 maximizes this value on each row k ), so, taking a weighted sum with weights P ( γ k ) ≥ 0 , we ﬁnd (cid:80) k P ( γ k ) (cid:80)
c R kc ( K 2 ,kc − K 1 ,kc ) ≥ 0 , i.e. Accuracy ( K 2 ) ≥
Accuracy ( K 1 ) , hence Accuracy ( K 1 ) =
Accuracy ( K 2 ) .
In words, we achieve maximal accuracy if we individually look at each “photon detected in cell k ” case and make the optimal prediction there.
Now, for a ﬁxed cell-index k , (cid:80) c
R kc K 2 ,kc is maximal if the matrix-row K 2 ,kc has an entry 1 for the index c for which R kc is maximal, and is zero otherwise.
To see this, let us assume K 2 ,kc > 0 for some index c for which there is another class index d with R kd > R kc .
Then, incrementing K 2 ,kd by K 2 ,kc and subsequently setting K 2 ,kc to zero increases (cid:80)
c R kc K 2 ,kc , which is a contradiction.
So, optimal choices of K 2 ,kc are zero for classes c for which R kc is not maximal.
Also, we always attain the maximum when choosing each row-vector of K to be one-hot and have its 1-entry in a place that maximizes R kc , i.e. maximal achievable accuracy is obtained by a classiﬁer which, for every cell-index k , predicts the most likely digit-class subject to the constraint that a randomly drawn example from the test set had its ﬁrst photon-detection occur at detector cell k .
This theoretical upper bound on classiﬁer accuracy hence is given by:
[Equation]
For the MNIST dataset, this is found to be 22 .
957% (rounded up to 22 .
96% ), while for Fashion-“MNIST”, we get 21 .
375% (rounded up to 21 .
Code that implements this calculation is available in the supplementary material.
We should emphasize that the constructive procedure described here that yields a classiﬁer attaining this stringent upper bound does inspect the test set, and relevant deviations in statistical properties between training and test set would manifest in the form of lowering attainable accuracy for a classiﬁer that is trained on the training set only.
A. C HANGES
For the sake of transparency and providing a better overview over the evolution of this document, we added this appendix that tracks changes made during the review period.
This appendix will be removed at the end of the review period.
A.3. 2020-11-13 RESPONSE → 2020-11-13 CORRECTION
Formula (8) required a ﬁx: The sum over style-parameters is, in quantum mechanical language, an ‘incoherent summation’ rather than ‘coherent summation’:
After the unitary transform, as the photon hits a detector, this gives us information about the class (for the downsampled 10 × 10 problem discussed in appendix A.1, via the row of the detector-cell), and also the style-parameter (via the column of the detector-cell).
After this measurement, we know ‘class’ and style , but decide to ignore ‘style’.
Still, as the ‘style’ is the outcome of a measurement, there can be no inference between different ‘style’ results, and so we need to sum the probability-contributions for all the different style-outcomes, rather than summing amplitudes.
We apologize for this rather obvious oversight that happened as we somewhat rushed our ﬁrst response in order to give reviewers an early answer.
A.3. I NITIAL S UBMISSION → 2020-11-13 RESPONSE
Major changes were:
Re-structured introduction, introducing a ‘related work’ section and expanding in the main text the explanation of the relation of this work to other recent analog all-optical image classiﬁcation research.
Added formula (8) that directly expresses the probability for a given example to be classiﬁed in a particular way in terms of image pixel data and model parameters.
Replaced the classical baseline accuracies (which were for any classiﬁer that was trained with access to the training set only) with strict proven accuracy bounds that can not even be beaten by classiﬁers that were trained with knowledge of the test set .
So, these hard thresholds for non-quantum classiﬁers now can not be outperformed by any means, including ‘cheating’ in the form of training on the test set.
The impact is that the baseline performance threshold rises from 21 .
28% to 22 .
96% (MNIST) and from 18 .
28% to 21 .
38% (F-MNIST), requiring adjustment of some plots.
Added a self-contained appendix with schematics alongside a detailed explanation of how image information gets translated to a photon quantum state.
Added a self-contained appendix with the detailed proof (including a proof idea sketch) of the claimed maximum attainable classical accuracy bound.