Hybrid Neural Autoencoder for Sensory Neuroprostheses and Its Applications in Bionic Vision
Anonymous Author(s)
AffiliationAddress
Abstract
Sensory neuroprostheses are emerging as a promising technology to restore lostsensory function or augment human capacities.
However, sensations elicited bycurrent devices often appear artificial and distorted.
Although current modelscan often predict the neural or perceptual response to an electrical stimulus, anoptimal stimulation strategy solves the inverse problem: what is the requiredstimulus to produce a desired response?
Here we frame this as an end-to-endoptimization problem, where a deep neural network encoder is trained to invert aknown, fixed forward model that approximates the underlying biological system.
As a proof of concept, we demonstrate the effectiveness of our hybrid neuralautoencoder (HNA) on the use case of visual neuroprostheses.
We found that HNA is able to produce high-fidelity stimuli from the MNIST and COCO datasetsthat outperform conventional encoding strategies and surrogate techniques acrossall tested conditions.
Overall this is an important step towards the long-standingchallenge of restoring high-quality vision to people living with incurable blindnessand may prove a promising solution for a variety of neuroprosthetic technologies.
Introduction
Sensory neuroprostheses are emerging as a promising technology to restore lost sensory functionor augment human capacities [9, 16].
In such devices, diminished sensory modalities (e g., hearing[45], vision [17, 26], cutaneous touch [41]) are re-enacted through streams of artificial input to thenervous system.
For example, visual neuroprostheses electrically stimulate neurons in the early visualsystem to elicit neuronal responses that the brain interprets as visual percepts.
Such devices have thepotential to restore a rudimentary form of vision to millions of people living with incurable blindness.
However, all of these technologies face the challenge of interfacing with a highly nonlinear system ofbiological neurons whose role in perception is not fully understood.
Due to the limited resolution ofelectrical stimulation, prostheses often create neural response patterns foreign to the brain.
Consequently, sensations elicited by current sensory neuroprostheses often appear artificial and distorted[14, 29].
A major outstanding challenge is thus to identify a stimulus encoding that leads to perceptually intelligible sensations.
Often there exists a numeric or symbolic forward model, f (Fig.
1A),constrained by empirical data, that can predict a neuronal or (ideally) perceptual response to theapplied stimulus (see [5] for a recent review).
To find the stimulus that will elicit a desired response,one essentially needs to find the inverse of the forward model, f − 1 .
However, realistic forwardmodels are rarely analytically invertible, making this a challenging open problem for neuroprostheses.
Here we propose to approach this as an end-to-end optimization problem, where a deep neuralnetwork ( encoder ) is trained to invert a known, fixed forward model ( decoder , Fig. 1B).
The encoderis trained to predict the patterns of electrical stimulation patterns that elicit perception ( e.g., vision,audition) or neural responses ( e.g., firing rates) closest to the target.
This hybrid neural autoencoder
In order to optimize end-to-end, the forward model must be differentiable and computationallyefficient.
When this is not the case, an alternative approach is to train a surrogate neural network, ˆ f ,to approximate the forward model [34, 28, 30, 31].
However, even well-trained surrogates may resultin errors when included in our end-to-end framework, due to the encoders’ ability to learn to exploitthe surrogate model.
We therefore also evaluate whether a surrogate approach to HNA is suitable.
To this end, we make the following contributions:
• We propose a hybrid neural autoencoder (HNA) consisting of a deep neural encoder trained toinvert a fixed, numerical or symbolic forward model (decoder) to optimize stimulus selection.
Our framework is general and addresses an important challenge with sensory neuroprostheses.
• As a proof of concept, we demonstrate the utility of HNA for visual neuroprostheses, where wepredict electrode activation patterns required to produce a desired visual percept.
We show thatthe HNA is able to produce high-fidelity, patient-specific stimuli representing handwritten digitsand segmented images of everyday objects, drastically outperforming conventional approaches.•
We evaluate replacing a computationally expensive or nondifferentiable forward model with asurrogate, highlighting benefits and potential dangers of popular surrogate techniques.
Background
Sensory Neuroprostheses Recent advances in neural understanding, wearable electronics, andbiocompatible materials have accelerated the development of sensory neuroprostheses to restoreperceptual function to people with impaired sensation.
Use of neuroprostheses typically requiresinvasive implants that elicit neural responses via electrical, magnetic, or optogenetic stimulation.
Twoof the most promising applications include cochlear implants, which stimulate the auditory nerve toelicit sounds [45], and visual implants (see next subsection) to restore vision to the blind.
However, avariety of other devices are in development; for instance, to restore touch [41, 40] or motor function[6].
The latter differ from other sensory neuroprostheses in that they generate stimuli using motorfeedback ( closed-loop ) [7, 43].
In the absence of feedback ( open-loop ), a proper stimulus encoding isparamount to the success of these devices.
Restoring Vision to the Blind For millions of people who are living with incurable blindness,a visual prostheses ( bionic eye , Fig. 2, left ) may be the only treatment option [17].
Analogous tocochlear implants, these devices electrically stimulate surviving cells in the visual pathway to evokevisual percepts ( phosphenes ), which can support simple behavioral tasks [26, 39, 23].
A common misconception is that each electrode in the microelectrode array can be thought of as apixel in an image; to generate a complex visual experience, one then simply needs to turn on theright combination of pixels [13].
However, recent evidence suggests that phosphenes often appeardistorted ( e.g. , as simple geometric shapes such as lines, wedges, and blobs) and vary drasticallyacross subjects and electrodes [14, 17].
Phosphene appearance has been best characterized in epiretinal implants, where inadvertent activationof nerve fiber bundles (NFBs) in the optic fiber layer of the retina leads to elongated phosphenes[3, 35] (Fig. 2, center ).
To this end, Granley et.
al [19] developed a forward model to predictphosphene shape as a function of both neuroanatomical parameters ( i.e. , retinal location of thestimulating electrode) and stimulus parameters ( i.e. , pulse frequency, amplitude, and duration).
Withthis model, phosphenes are primarily characterized by two main parameters, ρ and λ , which dictatethe size and elongation of the elicited phosphene, respectively (Fig. 2, right ).
Stimulation frommultiple electrodes is nonlinearly integrated into a combined perception, and if two electrodes happento activate the same NFB, they might not generate two distinct phosphenes. In addition, ρ and λ varydrastically across patients [3], making stimulus encoding a particularly challenging problem.
Related Work
The conventional ‘naive’ encoding strategy simply sets the amplitude of each electrode to thebrightness of the corresponding pixel in the target image [8, 26, 38], essentially making the stimulusa downsampled version of the target.
Although simple, this encoding strategy only works with nearlinear forward models and cannot account for phosphene distortions encountered by real prosthesispatients [3, 14]. When used with a more sophisticated forward model, this strategy often producesunrecognizable percepts (Fig. 2, right ).
To provide an alternative, Shah et al.
[36] used a greedy approach to iteratively select the stimulithat best recreate a desired neural activity pattern over a given temporal window, assuming that thebrain would integrate them into a coherent visual percept.
Ghaffari et al.
[18] used a neural networksurrogate model combined with an interior point algorithm to optimize for localized, circular neuralactivation patterns for individual electrodes.
Fauvel et al.
[15] used human in-the-loop Bayesianoptimization to achieve encodings perceptually favored by the subject.
Spencer et al.
[38] proposedframing the problem of stimulus encoding as inversion of a forward model of neural activationpatterns. To approximate the inverse, their approach either requires simplification or is NP-hard [38].
Furthermore, it cannot predict the perceptual consequences of the predicted neural activity.van Steveninck et al.
[10] proposed an end-to-end optimization strategy with a fixed phosphene model,similar to HNA.
However, their approach crucially differs from ours in that they included a secondarydeep neural network to post-process the predicted phosphenes.
This is a critical limitation, because alow reconstruction loss does not reveal whether a high-fidelity encoder was learned or whether the secondary decoder network simply learned to correct for the encoder’s mistakes.
In addition, theyused an unrealistic phosphene model that simply upscales and smooths a binary stimulus pattern.
It istherefore not clear whether their results would generalize to real visual prosthesis patients.
Relic et al.
[34] also utilized the end-to-end approach, but without the secondary decoder network usedin [10].
They used a more realistic phosphene model, which accounts for some spatial distortions( e.g. , axonal streaks), but not the effects of stimulus parameters.
Since including a nontrivialphosphene model in the loop is not straightforward, they instead trained a surrogate neural networkto approximate the forward model.
We re-implemented Relic’s ‘surrogate’ approach in this paper asa baseline method to compare against, as described in Section 4.
Taken together, we identified three main limitations of previous work that this study aims to address:
1) Unrealistic forward models.
Most previous approaches ( e.g. , [10, 36, 38]) use an overly simpli-
fied forward model that cannot account for empirical data [3, 14].
We overcome this limitation
by developing (and inverting) a differentiable version of a neurophysiologically informed and
psychophysically validated phosphene model [19] that can account for the effects of stimulus
amplitude, frequency, and pulse duration on phosphene appearance.
2) Optimization of neural responses.
Most previous approaches ( e.g. , [36, 38]) focus on optimizing neural activation patterns in the retina in response to electrical stimulation (“bottom-up”).
However, the visual system undergoes extensive remodeling during blinding diseases such as
retinitis pigmentosa [27].
Thus the link between neural activity and visual perception is unclear.
We overcome this limitation by inverting a phenomenological (“top-down”) model constrained by behavioral data that predicts visual perception directly from electrical stimuli [3, 19].
3) Objective function Most previous approaches rely on minimizing mean squared error (MSE)between the target and reconstructed image.
Although simple and efficient, MSE is known to bea poor measure of perceptual dissimilarity for images [44] and does not align well with humanassessments of image quality [46].
We overcome this limitation by proposing a joint perceptualmetric that combines mean absolute error (MAE), VGG, and Laplacian smoothing losses.
Methods
Problem Formulation We consider a system where there is some known forward process fmapping stimuli to responses f : S
(cid:55)→ R , f ( S ) ⊂ R .
In the case of visual prostheses, f may mapstimuli to visual percepts.
In spinal cord rehabilitation devices, f might map electrical stimuli toneural activation patterns.
f may be parameterized by patient-specific parameters ϕ .
Finding the best stimulus for an arbitrary target response t ∈ R is therefore equivalent to findingthe inverse of f .
However, since not every response can be perfectly reproduced by a stimulus, thetrue inverse of f is not well defined.
We therefore seek the pseudoinverse (still denoted as f − 1 forsimplicity) instead, which outputs the stimuli which produces the closest response to t under somedistance metric d :
[Equation]
This problem is straightforward to solve using an autoencoder approach, where a learned encoderf − 1 is trained to invert the fixed decoder f (i.e., forward model).
Encoder We approximate the pseudoinverse f − 1 with a deep neural network encoder ˆ f − 1 ( t , ϕ ; θ )with weights θ , trained to minimize the distance d between the target image t and predicted image ˆ t :
[Equation]
[Equation]
where ϕ is sampled from a uniform random distribution spanning the empirically observed range ofpatient-specific parameters [3, 19].
We use a simple architecture consisting solely of fully connected (FC) and batch normalization(BN) [21] layers.
First, the target t is flattened and input to one FC layer.
In parallel, the patientparameters ϕ are input to a BN layer before two hidden FC layers.
Then, the outputs of these two paths are concatenated, and the combined vector input through two FC layers, producing anintermediate representation x .
Amplitudes are predicted from x with a FC layer.
The amplitudes arethen concatenated with x , put through a BN layer, and used to predict frequency and pulse duration,each with a FC layer.
The encoder outputs are merged into one stimulus matrix ˆs .
All layers uses aleaky ReLU activation functions, except for stimulus outputs, which use ReLu activation, enforcingnonnegativity.
In total there were 1.4M trainable parameters.
We also tried using convolutional,transformers, and other attention layers, but the fully connected architecture performed best. Decoder For our decoder f , we use a reformulated but equivalent version of the model described in[19].
We use a square 15 × 15 array of 150 µ m electrodes, spaced 400 µ m apart and centered on thefovea.
The size and scale of this device was motivated by similar designs in real epiretinal implants.f takes as input a stimulus matrix s ∈ R n e x3 ≥ 0 , where the stimulus on each electrode ( s e ) is a biphasicpulse train described by its frequency, amplitude, and pulse duration.
A simulated map of retinal
NFBs is used to predict phosphene shape.
Following [3], each retinal ganglion cells’ activationis assumed to be the maximum stimulation intensity along its axon.
Each axon’s sensitivity tostimulation is assumed to decay exponentially with i) Euclidean distance to the stimulating electrode(radial decay rate, ρ ) and distance to the soma along the curved axon (axonal decay rate, λ ).
Toaccount for stimulus properties [19], ρ , λ , and the per-electrode brightness are scaled by three simpleequations F size ( s e , ϕ ) , F streak ( s e , ϕ ) , and F bright ( s e , ϕ ) , respectively.
The brightness of a pixel located at the point x ∈ R 2 in the output image is given by
[Equation]
where A is the cells’ axon trajectory, E is the set of electrodes, ϕ
= { ρ, λ, ... } is a set of 12patient-specific parameters, and d s is the path length along the axon trajectory [22]from a to x :
[Equation]
This model ( f ) can be fit to individual patients; however, it is computationally slow and not differentiable.
For more details on these equations, see [19].
We therefore considered two approaches.
• Differentiable Model: We reformulated equations 4 and 5 into an equivalent set of parallelized
matrix operations, implemented in Tensorflow [1].
Significant efforts were put towards developing
a model optimized for XLA engines on GPU, resulting in speedups of up to 5000x compared to
the model as presented in [19], enabling large-scale gradient descent.
To enforce differentiability,
we numerically approximated d s using | A | = 500 axon segments per axon.
• Surrogate Model: We also implemented the surrogate approach from [34] as a baseline method,
where f is approximated with another deep neural network ˆ f ϕ ( s ; θ f ) with weights θ f .
To achieve
this we generated 50,000 percepts using randomly selected stimuli passed through f and fit a
deep neural network to produce identical images.
As f is highly dependent on patient-specific
parameters ϕ , we generated new data and fit a separate surrogate model for each ϕ in our experimental set.
Specific implementation details of the surrogate are presented in Appendix A.
Our implementation improves upon [34] by using the more advanced phosphene model described
above, which accounts for effects of stimulus properties and allows for optimization of stimulus pulse duration and frequency, in addition
to amplitude.
Metrics To measure perceptual similarity, we use a joint perceptual objective consisting of a VGG[37] perceptual similarity term, a mean absolute error term, and a smoothness regularization term.
The mean absolute error term is given by
[Equation]
The VGG term aims to capture higher-level, abstract differences between images [10, 24].
Boththe target image and the reconstructed phosphene are input to a VGG-19 network pretrained forclassification on ImageNet [11], and the MSE between the activations on a downstream convolutional layer is computed.
Let V l be a function that extracts the activations of the l -th convolutional layer fora given image.
The VGG loss is then defined as:
[Equation]
We also include a smoothing regularization term.
This term imposes a loss on the second spatialderivative of the predicted image.
A low second derivative implies that where the target image doeschange, it changes slowly.
We found this encouraged smoother, more connected phosphenes.
Toapproximate the second derivative, we convolve the image with a Laplacian filter [32] of size k ,denoted by Lap ( · , k ) , and compute the mean.
The smoothness loss is given by:
[Equation]
Our final objective is the weighted sum of the three individual losses, given by Eq.
9, where α and βare hyperparameters controlling the relative weighting of the three terms:
[Equation]
We also implement a secondary metric to quantify phosphene recognizability, applicable only forthe MNIST reconstruction task.
We first pre-train a classifier network on the MNIST targets untilit reaches 99% test accuracy, and then fix the weights.
The relative accuracy (RA) is then definedas the ratio of the classifiers accuracy on the reconstructed images to its accuracy on the targets
RA = ACC/ACC ( t ) .
A perfect encoder would result in RA = 100% .
A similar process was notpossible for the COCO task, due to the possibility of having multiple objects in each target image.
Training/Optimization We trained using Tensorflow 2.7 [1] on a single NVIDIA RTX 3090 with24GB memory.
Stochastic gradient descent with Nesterov momentum was used to minimize thejoint perceptual loss.
We used a batch size of 16 due to memory limitations imposed by f .
Theamplitude, frequency predictions are scaled by 2, 20 respectively, while the pulse duration predictionswere shifted by 1e-3 prior to being fed through the decoder.
This encourages the initial predictionsof the network to be in a reasonable range.
The Laplacian filter size k is set to 5.
We choose l tobe first convolutional layer in the last block using cross validation (see Appendix D).
Similarly, weperform cross validation to find the best values for α and β .
Instead of using one value, we found thatincrementally increasing the weighting of the VGG loss ( β ) from 0 while simultaneously decreasingthe initially high weight on the smoothing constraint ( α ) was crucial for performance, especiallywhen the range of allowed ϕ values was large (see Appendix D).
Datasets We first evaluated on handwritten digits from MNIST [12], enabling comparison toprevious works [34].
Images preprocessing consisted of resizing the target images to the same shapeas the output of f (49x49).
We also evaluate on more realistic images of common objects from the
MS-COCO [25] dataset.
We selected a subset of 25 of the MS-COCO object categories deemedmore likely to be encountered by blind individuals ( e.g. people, household objects), and use onlyimages that contain at least one instance of these objects.
We further filter out images by various othercriteria, such as being too cluttered or too dim.
This process results in a total of approximately 47Ktraining images and 12K test images.
See Appendix C for a full description of the selection process.
Natural images often contain too much detail to be encoded with prosthetic vision.
While scenesimplification strategies exist [20], we focus on the encoding algorithm, so we simply used theground-truth segmentation masks to segment out the objects of interest.
The images were thenconverted to grayscale, and resized to (49, 49).
Results
5.1 MNIST
The phosphenes produced from the HNA, surrogate, and naive encoders on the MNIST test set areshown in Fig.
3 and performance is summarized in Table 1.
For each
[Table]
5.2 COCO
The phosphenes produced by HNA and the naive encoder for the segmented COCO dataset are shownin Fig.
We omit the surrogate results due to its poor perceptual performance on MNIST.
Averagedacross all ϕ , HNA had a joint loss of 0.713 on the test set and MAE of 0.1408, while the naive encoderhad a joint loss of 1.873 and MAE of 0.2830.
5.3 Modeling Patient-to-Patient Variations
MNIST encoder performance across simulated patients ( ϕ ) is shown in Fig.
Since the surrogateencoder has to be retrained for each patient, comparison is infeasible.
To visualize the effects ofchanging ρ and λ on the produced phosphenes, Fig.
5A shows the result of encoding two example
MNIST digits, both using the naive method and our encoder.
As λ increases, the naive phosphenesappear increasingly elongated, and as ρ increases, the phosphenes become increasingly large andblurry.
The phosphenes from HNA are slightly too dim and disconnected at low ρ , but are relativelystable across other values of ρ and λ .
To compare performance across the entire dataset, we computed the average test set loss acrossthe same range of ρ and λ (Fig.
5B).
The encoder performs well across a wide range of simulatedpatients, with larger loss only at low ρ .
The naive method performs well only on a limited set of ϕ ,with small λ and ρ ≈ 200 .
The naive loss was higher than the learned encoder at every simulatedpoint.
Random sampling of ρ and λ for each image results in a joint loss of 0.921, MAE of 0.120,and RA of 94.0% for HNA, while the naive encoder results in a joint loss of 3.17, MAE of 0.596, and
RA of 63.6%.
The same analysis yielded similar results on COCO (Appendix F).
An analysis acrossother parameters is presented in Appendix E.
In order for prosthetic vision to be useful, different instances of the same objects would ideallyproduce similar phosphenes, allowing for consistent perception.
To evaluate whether our modelachieves this, we cluster the target images and resulting phosphenes using T-SNE [42] shown in
Fig.
5C.
The ground truth images form clusters corresponding to the digits 0-9.
The phosphenes fromour encoder roughly form similar, slightly less separated groupings, whereas the naive phosphenes donot.
To ensure that this was not the result of bad T-SNE hyperparameters, we repeated the clusteringacross different perplexities and learning rates, obtaining similar or worse results.
5.4 Joint Perceptual Error Ablation Study
To show that the joint perceptual metric performs better than any of its individual components, wetrain models using just the VGG loss (Eq. 7), and just MAE (Eq. 6).
Shown are values for ρ =150and λ =600.
As mentioned previously, encoders trained using just VGG loss fail to converge, thus wepretrain the VGG encoder using MAE and smoothing loss, then transition to using only VGG.
We donot consider ablating the smoothing term (Eq. 8) because it is simply a regularization term.
Fig.
6shows the phosphenes produced by HNA trained on the joint, VGG-only, and MAE-only loss.
The VGG encoder had a test VGG loss of 4% lower than the joint model, and the phosphenesit produces have the correct high level perceptual details, but are oversmoothed and blurry. The
MAE encoder had a final test MAE of 9% lower than the joint model. Despite a lower MAE, thereconstructed images appear to be lower quality.
The joint model had a RA of 99.0% The VGGencoder had a RA of 95.9%, while the joint model had a RA of 77.6%
Discussion
Visual Prostheses We found that HNA is able to produce high-fidelity stimuli from the MNISTand COCO datasets that outperform conventional encoding strategies across all tested conditions.
Importantly, HNA produces phosphenes that are consistent across representations of the same object(Fig.
5C), which is critical to allowing prosthesis users to learn to associate certain visual patterns withspecific objects.
On the MNIST task, HNA produced high quality reconstructions, nearly matchingthe targets (Figure 3.
On the harder COCO task, HNA significantly outperformed the naive encoder, but was still unable to capture all of the detail in the images.
In Appendix G, we demonstrate that thisis mainly due to the limited spatial resolution and not a fundamental limitation of HNA.
Another advantage of the HNA is that it can be trained to predict stimuli across a wide range ofpatient-specific parameter values ϕ , whereas the conventional naive encoder works well only forsmall values of ρ and λ .
This may be one reason why the naive encoding strategy has been shown tolead to substantial individual differences in visual outcomes [39, 33].
Our results suggest that stimuliproduced with HNA may be able to reduce at least some amount of this patient-to-patient variability.
Furthermore, HNA also proved superior to a surrogate forward model.
The latter offer an alternativewhen the forward model is computationally expensive or not differentiable.
Understandably, anyinaccuracies in the surrogate model will propagate to the learned encoder during training.
However,we observed this phenomenon even when the surrogate performed well individually (see Appendix A), suggesting that even for well trained surrogates, the encoder may still learn to exploit the inexactsurrogate instead of learning to invert the true model.
We suspect that the surrogate method’s inferiorperformance here compared to [34] can be explained by our larger stimulus search space.
Thus, wecannot currently suggest HNA for forward models where a surrogate model must be used, unless theforward model is sufficiently simple or has a small stimulus space.
Broader Impacts While our work is presented in the context of visual prostheses, the HNAframework may apply to any sensory neuroprosthesis where stimulus selection can be informed by anumeric or symbolic forward model.
For example, cochlear implants [45] could use HNA to choosestimuli that result in a desired sound, and spinal cord implants [6] could use HNA to find the bestway to relay neural signals through a damaged section of the spinal cord.
However, HNA might notapply to all neural interfaces, such as systems without a clear neural or perceptual target ( e.g. , deepbrain stimulation for the treatment of Parkinson’s [2]) or closed-loop systems [4, 7].
Limitations Our end-to-end architecture allows for exact optimization of the forward model, butit also imposes limitations on the stability and speed of training.
For example, we observed thatconvergence of the encoder required customized learning rate scheduling, careful tuning of metrichyperparameters, and modification of the encoder output layer initialization.
Additionally, includingthe forward model in-the-loop required large amounts of memory and compute time.
Notably, theselimitations are only applicable during training. Although this may be application-specific, we expectany complex forward model to encounter similar issues, hindering ease of implementation.
Safety Concerns The use of a deep neural network for stimulus encoding does raise safety concerns.
Since we cannot examine the process by which stimuli are chosen, it is possible that an HNA encodermight produce potentially harmful stimuli that could lead to serious adverse events ( e.g. , seizures).
However, this concern is mitigated by the fact that most current neuroprostheses are equipped withfirmware responsible for verifying that all stimuli stay within FDA-approved safety limits.
Conclusion
In summary, this paper proposes a hybrid autoencoder structure as a general framework for stimulusoptimization in sensory neuroprostheses and, as a proof of concept, demonstrates its utility on theprominent example of visual neuroprostheses, drastically outperforming conventional encodingstrategies.
This may prove a promising solution for a variety of neuroprosthetic technologies.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] . You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description. For example: • Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] we highlight both the limitationsof our general framework and our specific proof of concept, see Section 6(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 6, last paragraph(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] We do not
present any theoretical results(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We include
the trained models, and code to reproduce the main result figures in the supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] detailed instructions are given in 4, training paragraph onhyperparameter selection, learning rates, and many other training details.
Data splits
were predetermined by the datasets (MNIST and COCO already have splits), with some
more details given in Appendix C
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We do not train our models multiple times.
Once trained,
the networks produce consistent results.
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes] This is specified in Section 4,
training paragraph
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We used models
from [19, 3, 34], which are all cited whenever used in the paper.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
, we provide saved weights for our final trained models, along with code to displayresults shown on main results figures.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] We do not use any new personal data
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
The data used does not contain any identifying
[Table]
information or offensive content.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]
See supplementary materials for appendix
[Table]