Learning Optical Flow From Continuous Spike Streams
Anonymous Author(s)
AffiliationAddress email
Abstract
Spiking camera is an emerging bio-inspired vision sensor with ultra-high temporalresolution.
It records scenes by accumulating photons and outputting continuousbinary spike streams.
Optical flow is a key task for spiking cameras and theirapplications.
A previous attempt has been made for spike-based optical flow.
However, the previous work only focuses on motion between two moments, and ituses graphics-based data for training, whose generalization is limited.
In this paper,we propose a tailored network, Spike2Flow that extracts information from binaryspikes with temporal-spatial representation based on the differential of spike firingtime and spatial information aggregation.
The network utilizes continuous motionclues through joint correlation decoding.
Besides, a new dataset with real-worldscenes is proposed for better generalization.
Experimental results show that ourapproach achieves state-of-the-art performance on existing synthetic datasets andreal data captured by spiking cameras.
The source code and dataset will be publiclyavailable after publication.
Introduction
Neuromorphic camera (NeurCam) is a kind of emerging vision sensor [35; 43; 46; 18; 11; 19] inspiredby retina.
Using asynchronous sampling models, NeurCams can record the natural scene continuouslyby firing sparse spike streams with an ultra-high temporal resolution.
Besides, NeurCams have highdynamic range, low latency and low energy consumption.
Compared with traditional cameraswith single-exposure imaging, NeurCams are more suitable for high-speed imaging in applicationscenarios such as unmanned aerial vehicles and chemical particle observation.
One kind of NeurCamsis the event camera [35; 43; 46; 18] inspired by the peripheral retina.
It asynchronously fires spikeswhen the brightness change exceeds a certain threshold in the logarithmic domain.
However, eventcameras can hardly recover textures due to their differential sampling model, especially for staticregions.
Another kind of emerging NeurCam is the spiking camera [11; 19] inspired by the foveaof retina.
Spiking cameras also record the scene by firing spikes asynchronously in each pixel,but different from event cameras that record the relative brightness changes, each pixel in spikingcameras encodes the scene by accumulating photons and firing a spike once the accumulation exceedsa certain threshold independently.
With such an integral sampling model, spiking cameras canrecord fine textures of the scene, which makes it more appropriate for pixel-level tasks than eventcameras.
Since spiking cameras can record high-speed motion and recover the scenes well, they haveshown great potential in tasks such as video reconstruction [74; 67; 70; 76; 68; 75], denoising [62],super-resolution [66; 60] and detection [19].
In these tasks, the relative motion between the spikingcamera and objects in the scene, i.e., the optical flow, is fundamental and pivotal.
Optical flow estimation has been an important and challenging problem since it was proposed [16].
Inrecent years, there are a deal of developments in optical flow [12; 23; 53; 20; 21; 63; 57; 69; 55; 27; 65].
A simple way to estimate optical flow for spiking cameras is reconstructing images and use videobased methods.
However, there are two problems.
Firstly, simple reconstruction methods mayintroduce a lot of noise while complex methods have high computational costs [67; 70; 68; 66].
Secondly, simple reconstruction ways such as averaging the spike streams along the temporal axiswould lose the precise temporal information in spike streams and introduce temporal fluctuation,which may mislead the flow estimation.
Event-based methods [72; 73; 32; 15; 14] also cannot havesatisfactory performance due to the difference of data modalities.
Thus, more efficient methods areneeded for estimating optical flow from spike streams.
Hu et al.
[17] propose SCFlow as an earlyexploration of this question.
They design a pyramidal network and two synthetic datasets basedon graphics models to train and evaluate the network respectively.
However, the above-mentionedmethod is not robust since it omits several issues:
(1) Efficient representation of binary spikes.
Each spike output by the camera does not onlyrepresent the status at the current moment, but the integral process of the spike.
Besides, a single
spike cannot express the information of the corresponding space-time point.
Using only convolution
to extract features from the spike streams may not be efficient.
(2) Continuousness of spike streams.
Previous work only considers motion between two single
moments in spike streams, and it omits the continuousness information in the moving procedure.
(3) Reality of the datasets.
Previous work use datasets synthesized by graphics models to train and
evaluate the network.
However, there is a huge gap between virtual scenes and the real world.
In this paper, we proposed the Spike2Flow to estimate flow
from
continuous spike streams.
Wepropose the differential of spike firing time (DSFT) to transform the binary spikes to better representthe procedure of the integration for each spike.
A spatial information aggregation (SIA) module isproposed to aggregate a larger receptive field for each pixel with a self-attention mechanism.
The
DSFT and SIA form the temporal-spatial representation (TSR) for spike streams.
Besides, we proposea joint correlation decoding (JCD) module to use the continuous motion clues by simultaneouslyestimating a series of flow fields.
To train and evaluate the network in real scenes, based on scenes in
SlowFlow [25], we generate flow fields and spike streams to construct a dataset, i.e., real scenes withspike and flow (RSSF).
The main contributions of this paper can be summarized as follows:
(1) A spike-based optical flow network, Spike2Flow, is proposed.
The Spike2Flow extracts featuresfrom binary spikes with temporal-spatial representation and jointly estimates a series of flow fields toutilize the continuousness of the moving procedure.(2) A dataset for spike-based optical flow, real scenes with spike and flow (RSSF) is proposed.
Thescenes in RSSF are from the real world, improving the generalization of networks trained by RSSF.
(3) Experiments demonstrate that the Spike2Flow achieves state-of-the-art performance on RSSF,photo-realistic high-speed motion (PHM) dataset, and real data captured by spiking cameras.
Related Work
Neuromorphic Cameras.
Neuromorphic camera (NeurCam) is a kind of vision sensor that getsinspiration from retinas and works asynchronously in each pixel.
Event cameras (including DVS[35], DAVIS [43], ATIS [46], CeleX [18] and et al.)
and spiking cameras [11; 19] are two typesof mainstream NeurCams.
Both of such cameras record the optical scene asynchronously.
Eventcameras employ a differential sampling model that fires a spike when the illuminance change inthe logarithmic domain exceeds a certain threshold.
In contrast, spiking cameras use an integralsampling model that accumulates photons and fires a spike when the accumulation exceeds a certainthreshold.
The spikes output by event cameras are more sparse, but the fine textures in the scene arelost, especially in static regions
Video-Based Optical Flow.
Optical flow estimation aims to find dense pixel correspondences betweentwo moments in videos, which has lots of applications, such as video enhancement [58; 6], frameinterpolation [26; 2; 33] and recognition [3; 9; 56].
FlowNet [12] is the first end-to-end flow estimationnetwork trained by the synthetic FlyingChairs dataset.
Subsequent works [47; 23; 53; 20; 21; 54; 69]get inspiration from variational methods [4; 52; 50].
They introduce classical knowledge such asthe pyramid, coarse-to-fine, and cost volume to the networks.
VCN [63] and DICL [57] design newapproaches to build more robust correlation between frames.
RAFT [55] builds a multi-scale all-pairscost volume and performs recurrent refinement in a fixed resolution.
RAFT has excellent performanceand becomes the baseline of the subsequent works [28; 61; 27; 65].
The methods mentioned aboveare based on supervised learning.
There are also some unsupervised optical flow estimation networks[41; 59; 37; 38; 36; 39; 29; 51].
Besides optical flow between two frames, there are also methods forflow among multi-frames.
[24; 38; 36] use temporal context by jointly estimating flow from a frameto its previous and future frames.
[44; 48; 22] propagate the temporal information from the previouspair of images to the next pair by passing warped flow or latent states of the decoder.
Event-Based Optical Flow.
There are also several works on event-based optical flow.
Zhu et al.[72; 73] train networks like U-Net [49] on MVSEC dataset [71] with loss function based on grayimages from active pixel sensor [43] of event cameras and the mean timestamp of events respectively.
Subsequent works [32; 15] introduce spiking neural networks (SNNs) to the encoder of the networkor the whole network.
STEFlow [10] uses a recurrent network to encode the event stream andconstructs correlation among features from events through time.
Gehrig et al.
[14] propose E-RAFTto implement all-pairs correlation and recurrent refinement in event-based flow based on a morecomplex autonomous driving dataset DSEC [13].
Approaches
3.1 Working Mechanism of Spiking Cameras
The spiking camera is composed of an array of pixels working asynchronously.
Each pixel of aspiking camera is composed of three main components: photon-receptor, integrator, and comparator.
The photon-receptor receives photons from the natural optical scene and converts the photons into avoltage.
The integrator accumulates the voltage from the photon-receptor continuously.
Through thecheck of the comparator, once the voltage of the integrator exceeds a certain threshold, it fires a spikeand resets the accumulation.
The voltage of the accumulator can be formulated as:
[Equation]
where A ( x , t ) is the voltage of the accumulator at pixel x = ( x, y ) .
t pre ( x ) is the time of the previousspike in pixel x .
α is the light conversion rate.
I ( x , τ ) is the lights intensity in pixel x at time τ .
θ isthe threshold of the comparator.
The above-mentioned working mechanism is called Integral-and-Fire(IF).
With such IF procedure, the spiking cameras can fire spikes asynchronously and continuously.
Due to the limitations of microelectronic technology, the reading time of the spikes is quantified witha period T , but the T can reach a micro-second level.
The comparator checks the firing of all pixelsat time nT, n ∈ N .
Thus, the output of the spiking camera is a spatial-temporal binary stream S in
H × W × N size.
The H and W are the height and width of the sensor respectively, and N is thetemporal steps of the spike stream.
3.2 Problem Statement and Data Generation
Problem Statement.
Optical flow estimation for spiking cameras is to estimate the pixel-level motionfield of the projection from the optical scene to the sensor plane.
Suppose that we denote a binaryspike stream as S
=
{ S ( x , t ) | x ∈ Ω , t ∈ N , t ≤ n } , if we set t start = t 0 as the starting moment,the optical flow can be denoted as w = { w ( x , t | t 0 )
| t ∈ N , t 0 ≤ t ≤ n } .
The ideally meaning ofw can be formulated as:
[Equation]
Spike Stream Input
Temporal
[Figure]
Spatial Representation
Joint
[Figure]
Decoding
[Figure]
m po r a l
A x i s
[Figure]
Clip Clip
Clip
[Figure]
Feature N
ContextFeature
C
[Figure]
C N
[Figure]
Flow where → means pixel-level registration.
Given a spike stream S , the target of optical flow estimationis estimating w with different starting moments.
The data format of the spike stream is continuousbinary matrices, which is different from the classic data pattern of video.
Correspondingly, theprocessing method should also be different.
Firstly, unlike a pixel in videos, a single spike representsthe integral procedure rather than the status at the current moment, and a single spike cannot expressspatial-temporal information.
Secondly, the spike stream is more continuous than classic video.
Thus,taking advantage of the continuousness for analyzing the scene is a challenge and an opportunity.
Data Generation.
SCFlow proposed two synthetic datasets through a graphics simulator for trainingand evaluation respectively.
However, there are several limitations of the simulating and rendering,such as unrealistic textures, overly simplified lighting conditions, and unreasonable appearance.
The huge gap between the synthetic scenes and the real world causes the models often have poorgeneralization on the real domain due to the gap [34; 45; 64; 7].
Thus, it is essential to train networkswith data from real scenes.
To improve the generalization on the real domain, we use a high-speeddataset Slow Flow [25] with a high spatial and temporal resolution to synthesize an optical flowdataset, i.e., real scenes with spike and flow (RSSF) for spiking cameras with real scenes.
We use theraw data of Slow Flow to generate RSSF.
The raw data have 41 scenes and sum to tens of thousands offrames, and there are three kinds of spatial resolution: 2560 × 2048 , 2560 × 1440 , and 2048 × 1152 .
We select 11 scenes to generate the testing set and the other 31 scenes to generate the training set.
More details are included in supplementary materials.
The generation pipeline is as follows.
Firstly, we imitate the image signal processor (ISP) to process the raw data in Bayer pattern to colorimages, where the operations include demosaicing, white balance, and intensity mapping.
The imageframes are 2 × downsampled spatially and temporally for saving computing and storage sources.
Secondly, we use GMA [27] network trained on data mixed by Sintel [5], KITTI [42], HD1K [31]and FlyingThings [40] to generate the reference optical flow of the image frames.
Thirdly, We usethe reference flow to simulate the motion of the scene and construct a virtual spiking camera tofire the spike streams.
We interpolate 20 time steps of images between every two adjacent framesfor generating spikes, and we use 10 times temporal oversampling to improve the precision of thescene.
The ground truth contains flow for the duration of 20 , 40 , and 60 interpolated images, denotedas dt = 20 , dt = 40 , and dt = 60 respectively.
It is noted that although the reference flow is notabsolutely accurate, it is still reliable due to the following reasons
(1) High-quality data.
The images for generating the spike have a high spatial and temporal
resolution, which can represent the ideal high-speed scene that we can hardly get in the real world.
Estimating the reference flow from such high-quality data is reliable.
[Figure]
FT
[Figure]
an s f o r m
ConvConv
ConvConv
ConvConv
Query
Key
Value
Feature / Context Encoder Feature / Context Encoder Addition
Dot Production
Softmax
Temporal Information Aggregation
FeatureExtraction
Spatial Information Aggregation
Clip of Spike Stream
OutputFeature
C x H x W C x H x W
C x H x W
HW x HW
C x H x W   g
PrimaryFeature
C x H x W p F
F
A primary feature is then extracted from
DSFT to primary feature
Then the primary feature is processed by the spatial information aggregation (
(2) Correspondence between spike and reference flow.
Although the reference flow is not absolutelyaccurate for color frames.
The reference flow corresponds to the spike since the spike is generatedfrom the reference motion.
Thus, the reference flow is reliable for the spike streams.(3)
Good generalization.
Results in section 4.2 show satisfactory performance of our network on
PHM dataset [17] and real data.
For PHM, the ground truth flow is absolutely reliable, which indicatesthe generalization of synthetic data and the reliability of our training data.
Good performance on realdata demonstrates the generalization of our training data on real scenes although there is a domaingap between real and simulated spikes.
3.3 Overall Architecture of the network
The overall architecture is shown in Fig.
The input spike stream S is firstly clipped tospike sub-streams { S 0 , S 1 , . . .
, S N } .
Each sub-stream is extracted to be the feature for matching{ F M0 , F M1 , . .
F M N } , which represents the central moment of the corresponding spike sub-stream,and the first feature S 0 is extracted to be context feature F C0 like RAFT [55].
Noted that it is hard toextract rich features directly from binary spike stream, we propose temporal-spatial representation(TSR) for the above-mentioned feature extraction.
To utilize the continuousness of the spike streams,the first matching feature F M0 constructs all-pairs correlations { C 1 , . .
, C N } with all other matchingfeatures.
The recurrent decoder jointly estimates flow fields from the central time of S 0 to the centraltime of { S 1 , . . .
S 2 } through all the correlations and the context feature F C0 .
For better embedding the spike streams to the feature domain, we propose to aggregate the informationin the temporal and spatial domains.
We design the differential of spike firing time (DSFT) in thetemporal domain to transform the spike stream from the binary domain, which aggregates informationalong the temporal axis for each pixel.
Based on DSFT, the spike stream is then embedded in ahigh-dimensional feature with a downsampling operation.
A spatial information aggregation (SIA)module is proposed to extract the information in the spike further.
To utilize the continuousness ofspike streams, which reflects the procedure of motion, we propose a joint correlation decoding (JCD)module to jointly estimate a series of flow fields starting from the same moment.
3.4 Temporal-Spatial Representation for Spikes
Differential of Spike Firing Time.
Each spike in spike streams represents the integral procedure ofphotons rather than status at the current moment, and the “ 1 ” in different positions often have differentmeanings.
Using features extracted from the binary spikes for matching may not be appropriateto reflect the structures of scenes.
We propose to represent the information contained in the spikethrough the firing time.
The absolute value of firing time can hardly reflect the relative relationshipbetween spikes through time.
For better aggregating the temporal information, as shown in Fig.
3,we propose to use the differential of spike firing time to represent the binary spike.
If we denote the
S p i
k e S t r
[Figure]
D S FT
S t r ea m
DSFT of spike stream S = S ( x , t ) as D = D ( x , t ) , the DSFT transform can be formulated as:
[Equation]
Where D is the DSFT transform.
T next ( x , t ) and T pre ( x , t ) denote the firing time of the next andprevious spike at spatial-temporal moment ( x , t ) .
D ( x ) denote the domain of definition of x .
Weuse DSFT to reflect the firing rate of the spikes, which is related to the brightness.
The motion in thescenes changes the brightness, and the Poisson process of photon arrivals causes fluctuations in thefiring rate.
Thus, the firing rate is temporally variational at each pixel.
DSFT can better recover thespikes’ dynamic process than obtaining a more constant firing rate with longer time windows.
Spatial Information Aggregation.
A single spike in space-time coordinates can hardly describethe scene, and we need a group of spikes to represent each pixel.
To enhance the features of spikefor better matching, we propose a spatial information aggregation (SIA) module to integrate contextinformation with a larger receptive field.
As shown on the right side of Fig.
2, we use the self-attentionmechanism to aggregate the spatial information, which can be formulated by:
[Equation]
Where F denotes the feature finally output by the encoder.
F p denotes the primary feature before
SIA.
θ, ϕ , and g embed the primary feature to query, key and value respectively.
Through the SIAmodule, each pixel gets a long-range aggregation with pixels in its relative area.
3.5 Joint Decoding of Correlation
There is more continuously temporal information in spike streams compared with videos.
To extractthe continuous moving procedure of the optical scene in spike, we propose to jointly decode a seriesof motion from the same starting moment, i.e. { w ( x , t | t 0 ) } .
To simplify the computation, wesparsely sample the motion and estimate { w i = w ( x , iT + t 0 | t 0 ) }
Ni =1 , where T is the samplingcycle, i is the sampling index, and N is the maximum sampling index.
For each sampling moment,we construct a multi-scale all-pairs correlation [55] from the matching features F M0 and F M i :
[Equation]
Local correlations { L 1 , . . .
, L N } are then looked up through current estimated flow fields with aradius r .
The local grid for looking up can be formulated as:
[Equation]
The recurrent decoder is based on ConvGRU [8].
In each iteration, the decoder optimizes the flowfields by estimating their residual through local correlations and context feature F C0 .
Each localcorrelation is encoded to be motion feature M i with the corresponding flow.
All the motion features{ M 1 , . . .
, M N } and F C0 are concatenated and input to the ConvGRU for optimizing the flow fields.
Suppose the recurrent decoder has V iterations.
Given ground truth flow fields { w gt i } N i =1 the lossfunction is defined as:
[Equation]
where γ is the decay factor set as 0 . 8 .
Experimental Results
4.1 Implementation Details
In the experiments, we set N as 3 and T as 20 , which means we jointly estimate optical flow under20 , 40 , and 60 time step difference.
We set the number of input spike frames as 21 .
For constructingcorrelation, we set the multi-scale level as 3 , and we set the looking up radius r = 3 .
The model istrained on the training set of the real scenes with spike and flow (RSSF) dataset.
We randomly cropthe spike stream to 320 × 448 spatially during the training procedure and set the batch size as 6 .
Weuse randomly horizontal and vertical flips as data augmentation to balance the motion in the trainingdataset.
We use Adam optimizer [30] with β 1 = 0 .
9 and β 2 = 0 .
The learning rate is initially setas 3 e- 4 and scaled by 0 .
7 every 10 epoch.
The model is trained for 100 epochs.
We use average end-point error (AEPE) and percentage of outliers (PO%) as evaluating metrics.
AEPE is the spatial average of Euclidean distance between optical flow w and the ground truth w gt :
[Equation]
where H and W are the height and width of the flow.
percentage of outliers is the percentage of thepixel with end-point error larger than 0.5 and 5% of its ground truth at the same time.
4.2 Comparative Results
We compare our method with SCFlow [17] and methods straightforwardly designed based on RAFT[55], SCV [28], and GMA [27] for estimating optical flow for spiking camera.
There are two ways toinput the spikes: (1) AvgImg .
Averaging the spike stream along the temporal axis as a gray image.(2) Spike .
Directly input the spike streams into the network work as a multi-channel image.
We donot use the event-based optical flow architecture since it has been proven to be noneffective in [17],and we do not use the training set in [17] since it is synthetic and the generalization ability is limited.
We use "RAFT - AvgImg" to represent the direct adaptation based on RAFT with AvgImg, and othersare the same.
All the methods are retrained under settings in section 4.1 based on the official code on
RSSF training set with spike streams as input respectively.
Noted that Spike2Flow jointly estimatesflow fields with time duration dt = 20 , 40 , 60 .
We set the training data for comparable models andrandomly select dt from { 20 , 40 , 60 } for each sample in a mini-batch for the sake of fairness.
Weevaluate the models on three kinds of data: (1) The testing set of real scenes with spike and flow(RSSF), (2) photo-realistic high-speed motion (PHM), and (3) real data captured by spiking cameras.
Results on real scenes with spike and flow (RSSF) .
The quantitative results on RSSF is shownin Tab. 1.
The Spike2Flow jointly estimates the flow in dt = 20 , 40 , 60 cases while other modelsestimates respectively.
Spike2Flow gets the best performance in both AEPE and PO in all cases.
Thevisual results on RSSF are shown in Fig.
All the optical flow color-coding rule is the same with[1].
More results are included in the supplementary material.
[Table]
[Figure]
[Figure] [Figure]
[Table]
Results on photo-realistic high-speed motion (PHM) .
For comparative experiments on
PHM, we use the same model as experimentson the testing set of RSSF, which is trained onthe training set of RSSF.
The ground truth ofoptical flow in PHM is generated from a graphics simulator, which is highly reliable.
PHMcontains ground truth flow in dt = 10 and 20cases.
For Spike2Flow, we jointly estimateflow fields in dt = 10 , 20 , 30 cases.
For othermodels, we estimate flow fields in dt = 10and dt = 20 cases respectively.
Noted that weselect the 9 scenes in PHM except fly sincethe motion speed of fly is excessive and unrealistic.
Quantitative results are shown in Tab.2.
Spike2Flow still achieves the best performance in all the cases, which demonstrates the goodgeneralization of our method.
Compare Tab. 2 with Tab. 1, the performance results on PHM are lowerthan those on RSSF since the scene in PHM is simulated through a graphics model, and the speedmotion of the scenes is ultra-high, which is more challenging for optical flow estimation.
The visualresults on PHM are shown in Fig.
5 and more results are included in the supplementary material.
Results on real data .
We evaluate the models on real data captured by spiking cameras.
The visualresults are shown in Fig 6.
Although we do not have the ground truth in real data, we can seethat results of Spike2Flow have sharper edges and more clear background, which shows a bettergeneralization.
More details are included in the supplementary material.
4.3 Ablation Study
To verify the efficiency of the modules we propose, we implement a series of ablation studies.
Thequantitative results are shown in Tab. 3.
We use color gradation from yellow to green for each column,where greener represents better performance.
The results demonstrate the effectiveness of DSFT,
[Figure]
[Figure]
[Figure]
[Figure]
SIA, and JCD modules.
Although the Exp.
(D) and (E) have similar performance in dt = 20 anddt
= 40 , Exp.
(E) performs better than (D) in the percentage of outliers under dt = 60 , which showsthe effectiveness of the SIA module.
Comparison between Exp.
(B) and (C) can also show that SIAis effective.
Besides proposed modules, we also perform ablation studies on the number of inputspike frames (NISF).
We select { 1 , 5 , 11 , 15 , 21 } as the NISF candidates.
The quantitative results areshown in Tab. 4, where we can see larger NISF makes the performance better
[Table] [Table]
[Figure]
[Table]
Conclusions
We propose a method for learning optical flow from continuous spike streams.
To improve thegeneralization of spike-based optical flow, we propose a flow dataset RSSF with real scenes.
Forbetter extracting information from binary spike streams, we propose DSFT and SIA modules toextract the temporal and spatial information respectively.
To use the continuousness of spike streamsas motion clues, we propose a JCD module to jointly estimate a series of flow fields.
Experimentalresults show that Spike2Flow achieves state-of-the-art performance in spike-based optical flow on
RSSF, PHM, and real data, and ablation studies verify the effectiveness of our proposed modules.
Limitations.
The characteristics of spike firing in extremely dark or bright scenes are quite different,so the same representation strategy may not work well all the time.
We plan to extend our model tohandle these questions in future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See section 4.1.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary material.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]