Original: Spiking ConvLSTM for Semantic Music Generation
Revised: Spiking ConvLSTM for Semantic Music Generation

Original: Anonymous Submission

Original: Abstract
Revised: Abstract

Original: Spiking neural networks received a lot of attention in a scien- tific literature due to low memory and energy consumption, which makes them suitable for edge computing and off-line deployment of AI models on low power devices.
Revised: Spiking neural networks received a lot of attention in a scientific literature due to low memory and energy consumption, which makes them suitable for edge computing and off-line deployment of AI models on low power devices.

Original: The use of spiking neural networks in a multimodal generative context was not yet explored and in this paper the focus in made on comparison of performance between classical and spiking versions of shallow Convolutional Long-Short Term Memory (ConvLSTM) network architecture.
Revised: The use of spiking neural networks in a multimodal generative context was not yet explored and in this paper the focus in made on comparison of performance between classical and spiking versions of shallow Convolutional Long-Short Term Memory (ConvLSTM) network architecture.

Original: A new encoding strategy based on the system of graphs allowed dimen- sionality and data augmentation of one-dimensional sequential data, resulting in better generalization capacities of trained models.
Revised: A new encoding strategy based on the system of graphs allowed dimensionality and data augmentation of one-dimensional sequen- tial data, resulting in better generalization capacities of trained models.

Original: To the best of our knowledge, this is the first attempt of implementing spiking ConvLSTM model for semantic music generation task.
Revised: To the best of our knowledge, this is the first attempt of implementing spiking ConvLSTM model for se- mantic music generation task.

Original: We release the resources with the article ( upon paper acceptance ).
Revised: We release the resources with the article (https://github.com/asnota/spiking_convlstms).

Original: Introduction
Revised: Introduction

Original: Recent advances in gradient calculation in spiking neural networks with surrogate approaches (Kheradpisheh,
Revised: Recent advances in gradient calculation in spiking neural networks with surrogate approaches (Kheradpisheh,

Original: Mirsadeghi and Masquelier 2022, Zhang et al. 2021) allowed amelioration of the learning efficiency of this type of networks, making them interesting for practical use.
Revised: Mirsadeghi and Masquelier 2022, Zhang et al. 2021) allowed amelioration of the learning efficiency of this type of networks, making them interesting for practical use.

Original: The integration of spiking neurons has been previously done with feedforward neural networks (She 2020), recurrent neural networks (Demirag et al.
Revised: The integration of spiking neurons has been previously done with feedforward neural networks (She 2020), recurrent neural networks (Demirag et al.

Original: , Kim and Sejnowski 2019), convolutional neural networks (Guan & Mo 2020) and belief neural networks (O’Connor et al. 2013).
Revised: , Kim and Sejnowski 2019), convolutional neural networks (Guan and Mo 2020) and belief neural networks (O’Connor et al. 2013).

Original: The angle of research focused mostly on classification tasks, however there are several examples of spiking neurons use in a generative adversarial network architecture (Molano-Mazon et al.
Revised: The angle of research focused mostly on classification tasks, however, there are several examples of spiking neurons use in a generative adversarial network architecture (Molano-Mazon et al.

Original: 2018, Kotariya and Ganguly 2021, Rosenfeld, Simeone and Rajendran 2021).
Revised: 2018, Kotariya and Ganguly 2021, Rosenfeld, Simeone and Rajendran 2021).

Original: The generative prospective in a multimodal setting was not yet explored and is mostly related to the absence of benchmark data on generative abilities of spiking neural networks, along with a lack of systematic research measuring the differences of performance between classical and spiking neural networks in a generative setting.
Revised: The generative prospective in a mul- setting was not yet explored and is mostly related to the absence of benchmark data on generative abilities of spiking neural networks, along with a lack of systematic research measuring the differences of performance between classical and spiking neural networks in a generative setting.

Original: There were attempts to develop a benchmark dataset for spiking neural networks, however each time only one modality is covered (Cramer et al. 2022, Liu et al. 2016), therefore the data for multimodal research were not yet proposed.
Revised: There were attempts to develop a benchmark dataset for spiking neural networks, however, each time only one modality was covered (Cramer et al. 2022, Liu et al. 2016), therefore, the data for multimodal research were not yet proposed.

Original: In this article we propose a multimodal dataset with parallel data, referred further as seqGraph, and compare the performance of classical shallow convolutional Long ShortTerm Memory network (ConvLSTM) with its spiking version (SConvLSTM).
Revised: In this article we propose a multimodal dataset with parallel data, referred further as seqGraph, and compare the performance of classical shallow convolutional Long ShortTerm Memory network (ConvLSTM) with its spiking version (SConvLSTM).

Original: The data of seqGraph compriseone-dimensional harmonic sequences (their analytical representation, which uses Roman and Arabic numerals to represent the chord fundamental and its inversion), their respective audio representation (generated with music21 Python library for computational musicology, out of the initial harmonic sequences) and a 2D representation of chords, received via original encoding technique based on the system of graphs.
Revised: The data of seqGraph compriseone-dimensional harmonic sequences (their analytical representation, which uses Roman and Arabic numerals to represent the chord fundamental and its inversion), their respective audio representation (generated with music21 Python library for computational musicology, out of the initial harmonic sequences) and a 2D representation of chords, received via original encoding technique based on the system of graphs.

Original: The dimensionality augmentation allows to match the dimensionality of existing datasets and evaluate the network performance on data for which the benchmark results are known.
Revised: The dimensionality augmentation allows to match the dimensionality of existing datasets and evaluate the network performance on data, for which the benchmark results are known.

Original: We also apply a varying normalization term to the 2D partition of data, which results in considerable data augmentation (up to 2160 entries).
Revised: We also apply a varying normalization term to the 2D partition of data, which results in considerable data augmentation (up to 2160 entries).

Original: The models in this paper are trained on 2D partition of the seqGraph dataset, being compared with the existing Moving
Revised: The models in this paper are trained on 2D partition of the seqGraph dataset, being compared with the existing Moving

Original: MNIST dataset and a custom selection of sequences of images from  classical MNIST dataset, referred further as seqMNIST.
Revised: MNIST dataset and a custom selection of sequences of images from a classical MNIST dataset, referred further as seqMNIST.

Original: We exploit the implementation of  ConvLSTM model (Xingjian et al. 2015), modifying it to its shallow version, and build our shallow spiking ConvLSTM model.
Revised: We exploit the implementation of the ConvLSTM model (Xingjian et al. 2015), modifying it to its shallow version, and build our shallow spiking ConvLSTM model.

Original: The contribution of this paper consists in the following:
Revised: The contribution of this paper consists in the following:

Original: Exploration of the generative abilities of spiking neural networks; 2.
Revised: Exploration of the generative abilities of spiking neural networks; 2.

Original: Cross modal generation exploration (the use of one modality to generate another modality); 3.
Revised: Cross modal generation exploration (the use of one modality to generate another modality); 3.

Original: Multimodal data, which includes semantic, visual and audio information, as a tool for further multi - modal research in spiking neural networks.
Revised: Multimodal data, which includes semantic, visual and audio information, as a tool for further multimodal research in spiking neural networks;

Original: Original encoding method based on a frame theory, and the system of graphs, which has several advantages over classical 2D data, such as: 1) Data augmentation with the use of varying normalization term; 2) Reduced information density, which is better processed by spiking neural networks; 3) Easy transition of unsupervised visual information toa labelled 1D sequences for inputs, targets and predictions, which facilitates multimodal and cross-modal research.
Revised: Original encoding method based on a frame theory, and the system of graphs, which has several advantages over classical 2D data, such as: • Data augmentation with the use of varying normalization term; • Reduced information density, which is better processed by spiking neural networks; • Easy transition from an unsupervised visual information to labelled 1D sequences for inputs, targets and predictions, which facilitates multimodal and cross-modal research.

Original: Encoding method Methodology
Revised: Encoding Method Methodology

Original: The encoding strategy proposed in this paper is based on an original system of graphs in music harmony, which was build following the frame and graph theory by Marvin Minsky (1974).
Revised: The encoding strategy proposed in this paper is based on an original system of graphs in music harmony, which was built following the frame and graph theory by Marvin Minsky (1974).

Original: The system has been shown useful in buildingof mobile (Anonymous 2016), VR (Anonymous and Darkazanli 2020) and web-based graphical interfaces for learning music harmony, tested with success in several pedagogical experiments (Anonymous 2019a).
Revised: The system has been shown useful in building mobile (Shvets 2016), VR (Shvets and Darkazanli 2020) and web-based graphical interfaces for learning music harmony, tested with success in several pedagogical experiments (Shvets 2019a).

Original: The system of graphs presents an analytical representation of functional correlations between chords, visualized as the degrees of the diatonic scale (Roman numerals), placed in a specific order.
Revised: The system of graphs presents an analytical representation of functional correlations between chords, visualized as the degrees of the diatonic scale (Roman numerals), placed in a specific order.

Original: The placement order is defined by the possibility of connection between the chords and expresses the variation potential within a multitude of chord progressions.
Revised: The placement order is defined by the possibility of connection between the chords and expresses the variation potential within a multitude of chord progressions.

Original: The slots are reserved for a specific type of information, where the slots reserved for the seventh chords cannot contain the information about triads (Figure 1).
Revised: The slots are reserved for a specific type of information: the slots reserved for the seventh chords cannot contain the information about triads (Figure 1).

Original: This limit comes from the theory of graphs, defining a graph as a graphical representation of frames, corresponding to aspecific knowledge representation.
Revised: This limit comes out of the theory of graphs, defining a graph as a graphical representation of frames, which describe a stereotyped situation, forming a knowledge substructure.

Original: The graphs are organized in horizontal triads, as shown in Figure 2, and vertical triads (Figure 3).
Revised: The graphs are organized in horizontal triads (Figure 2) and vertical triads (Figure 3).

Original: A harmonic sequence is therefore represented as a path within the system, connecting the nearest instances of the chord in the system of graphs.
Revised: A harmonic sequence is therefore represented as a path within the system, connecting the nearest instances of the chord in the system of graphs.

Original: passing progressions between its inversions
Revised: the degree In this ex ample

Original: Data conversion process
Revised: Data Conversion Process

Original: The value of the graph representation consists in a possibility ofthe encoding of semantic music data.
Revised: The value of the graph representation consists in a possibility of semantic music data encoding.

Original: Such encoding allows training and evaluation methods application, inherent to a visual domain.
Revised: Such encoding allows training and evaluation methods application, inherent to a visual domain.

Original: This way, the harmonic sequences features learning may be done with the 2D convolutional LSTM layers, discussed below.
Revised: This way, the harmonic sequence features learning may be done with the 2D convolutional LSTM layers, discussed below.

Original: Another advantage of such encoding is the possibility of considerable data augmentation with the application of a small normalization term.
Revised: Another advantage of such encoding is the possibility of considerable data augmentation with the application of a small normalization term.

Original: Finally, the modality switch is useful in fostering multimodal studies in general and particularly within spiking neural networks.
Revised: Finally, the modality switch is useful in fostering multimodal studies in general and particularly within spiking neural networks.

Original: To obtain the feature maps out of the arrays of harmonic sequences, the following transformation steps must be performed:
Revised: To obtain the feature maps out of the arrays of harmonic sequences, the following transformation steps must be performed:

Original: • Chords dictionary creation. • Two-dimensional matrix creation using the graph systemframe and replacing the chords with the values from  previously created dictionary. • Creation of 28x28 feature maps of harmonic paths in 2D space using harmonic sequences mapped to the chords dictionary in a progressive way: one sequence of 5 chords results into 5 matrices, gradually filled in with the chord values. • Normalization of the matrices with a changing normalization term (in a range between 0.01 and 0.1).
Revised: • Chords dictionary creation; • Two-dimensional matrix creation using the graph system and replacing the chords with the values from a previously created dictionary; • Creation of 28x28 feature maps of harmonic paths in 2D space using harmonic sequences mapped to the chords dictionary in a progressive way: one sequence of 5 chords results into 5 matrices, gradually filled in with the chord values; • Normalization of the matrices with a changing normalization term (in a range between 0.01 and 0.1).

Original: Using this data conversion strategy, it was possible to obtain2160 data entries out of the initial handcrafted 216 har- monic
Revised: Using this data conversion strategy, it was possible to obtain data entries out of the initial handcrafted 216 harmonic sequences.

Original: sequences. To illustrate a described transition process, let us take a sequence of three chords II 7 -VI 46 -II 56 and present it as a graph (Figure 4a), then replace a semantic chord designation with numerical values – indexes we chose to represent the chords (Figure 4b): we thus receive a 3x3 matrix.
Revised: To illustrate a described transition process, let us take a sequence of three chords II 7 -VI 46 -II 56 and present it as a graph (Figure 4a), then replace a semantic chord designation with numerical values – indices we chose to represent the chords (Figure 4b): we thus receive a 3x3 matrix.

Original: If we repeat the described procedure, but taking the dimensionality of the whole system, we receive a matrix ready to be processed by a neural network (49 chords of the system of graphs are giving precisely 27x21 matrix, however we added one column and seven rows of a padding filled with zeros to facilitate computingof 2D convolutional operations).
Revised: The repetition of the described procedure for the dimensionality of the whole system gives a matrix ready to be processed by a neural network (49 chords from the system of graphs are giving precisely 27x21 matrix, however, we added one column and seven rows of a padding filled with zeros to facilitate computing 2D convolutional operations).

Original: After the data conversion is finished and each chord of the harmonic sequence is represented as a 28x28 tensor, we can visualize the matrices using matplotlib library.
Revised: After the data conversion is finished and each chord of the harmonic sequence is represented as a 28x28 tensor, we can visualize the matrices using matplotlib library.

Original: Let’s visualize a harmonic sequence from our dataset, consisting of 5 chords: II 7 -VI 46 -II 56 -V 35 -I 35 , in Figure 5. 
Revised: Let’s visualize a harmonic sequence from our dataset, consisting of 5 chords: II 7 -VI 46 -II 56 -V 35 -I 35 , in Figure 5. 1) II2) VI

Original: [Figure]
Revised: [Equation]

Original:  The shape of the chord in presented visualization is defined by its position within the system of graphs – the chord IIcan only start the passing progression sequence (appear only in a top of the graph), therefore it is represented as a single point, whereas II 56 can start and finish the passing progression sequences and as a result, appearing in both tops of the graph within the same horizontal triad of graphs (therefore a single chord is represented with two dots).
Revised: 4) II5) V6) I The shape of the chord in presented visualization is defined by its position within the system of graphs – the chord IIcan only start the passing progression sequence (appears only in a top of the graph), therefore it is represented as a single point, whereas II 56 can start and finish the passing progression sequences and as a result, appears in both tops of the graph within the same horizontal triad of graphs (therefore, a single chord is represented with two dots).

Original: Finally, the triadic structures (VI 46 , V 35 and I 35 ) appear in the graph vertices only and are common for a given vertical triad of graphs, which produces three positions and, consequently, three dots to appear in a matrix visualization.
Revised: Finally, the triadic structures (VI 46 , V 35 and I 35 ) appear in the graph vertices only and are common for a given vertical triad of graphs, which produces three positions and, consequently, three dots to appear in a matrix visualization.

Original: Experiment
Revised: Experiment

Original: The new data encoding scheme has been tested in an unsupervised setting, using classical and spiking neural networks.
Revised: The new data encoding scheme has been tested in an unsupervised setting, using classical and spiking neural networks.

Original: The 2D data required implementation of an architecture capable to process sequences of frames, therefore, a hybrid ConvLSTM architecture was chosen.
Revised: The 2D data required implementation of an architecture capable to process sequences of frames, therefore, a hybrid ConvLSTM architecture was chosen.

Original: In ConvLSTM each element of the sequence passes through a convolutional layer followed by  LSTM layer.
Revised: In ConvLSTM each element of the sequence passes through a convolutional layer, followed by an LSTM layer.

Original: Previous experiment has shown that this type of architecture captures well two-dimensional sequential data (Anonymous 2019b) and will be exploited in current experiment using  novel data encoding scheme.
Revised: Previous experiment has shown that this type of architecture captures well two-dimensional sequential data (Shvets 2019b) and will be exploited in current experiment using the novel data encoding scheme.

Original: The first stage of the experiment focused on comparison of  ConvLSTM model trained on equivalent 2D data, with and without augmentation.
Revised: The first stage of the experiment focused on comparison of the ConvLSTM model trained on equivalent 2D data, with and without augmentation.

Original: The result received from the models trained on custom data is further compared with the models trained on reduced versions of  Moving MNIST dataset (Srivastava et al. 2015), in respect of the quantity of entries from a custom 2D dataset in its original and augmented forms (216 and 2160 sets).
Revised: The result received from the models trained on custom data is further compared with the models trained on reduced versions of the Moving MNIST dataset (Srivastava et al. 2015), in respect of the quantity of entries from a custom 2D dataset in its original and augmented forms (sets containing 216 and 2160 entries respectively).

Original: The Moving MNIST dataset was used to train the very first implementation of  ConvLSTM model and allows to account for the difference between original deep model and its shallow version, as well as the number of data points, considerably reduced in our experiment.
Revised: The Moving MNIST dataset was used to train the very first implementation of the ConvLSTM model and allows to account for the differences between original deep model and its shallow version, as well as the number of data points, considerably reduced in our experiment.

Original: The very nature ofthe data in the Moving MNIST dataset, consisting of the sequence of 2D frames, representing two moving digits in each data entry, doesn’t fit the diversity of our data, since each 2D frame of the seqGraph dataset represents a distinct chord, which has a different shape than other frames of the sequence.
Revised: The very nature of data in the Moving MNIST dataset, which consists of a sequence of 2D frames, representing two moving digits in each data entry, doesn’t fit the diversity of our data, since each 2D frame of the seqGraph dataset represents a distinct chord, which has a different shape, compared to other frames of the sequence.

Original: Besides, the chords in each sequence are distinct (the repetition is only allowed for a target), where four non-repeating chords in an input sequence are followed either by  non-repeating target or a target, similar to one of the chords from the input sequence.
Revised: Besides, the chords in each sequence are distinct (the repetition is only allowed for a target), where four non-repeating chords in an input sequence are followed either by a non-repeating target or a target, similar to one of the chords from the input sequence.

Original: Therefore, we selected distinct digits from a MNIST dataset to form 216 sequences, which matches the degree of diversity proposed in seqGraph.
Revised: Therefore, we selected distinct digits from a MNIST dataset to form 216 sequences, which matches a degree of diversity proposed in seqGraph.

Original: The second stage of the experiment compares  performance of spiking versions of neural networks, trained onthe data from the first stage.
Revised: The second stage of the experiment compares the performance of spiking versions of neural networks, trained on data from the first stage of the experiment.

Original: Therefore, the augmented and nonaugmented seqGraph, Moving MNIST and seqMNIST datasets were used to train SConvLSTM models.
Revised: Therefore, the augmented and non-augmented seqGraph, Moving MNIST and seqMNIST datasets were used to train SConvLSTM models.

Original: All neural networks were developed using  Pytorch framework with additional use of  snnTorch Python package for SConvLSTM implementation, which allows performing gradient-based learning with spiking neural networks.
Revised: All neural networks were developed using a Pytorch framework with additional use of the snnTorch Python package for SConvLSTM implementation, which allows performing gradient-based learning of spiking neural networks.

Original: Neural Networks Architecture
Revised: Neural Networks Architecture

Original: The shallow ConvLSTM model architecture comprised one LSTM layer, followed by 3D batch normalization (third dimensionality was necessary to account for  sequential nature of  data) and convolutional 2D layer.
Revised: The shallow ConvLSTM model architecture comprised one LSTM layer, followed by 3D batch normalization (third dimensionality was necessary to account for the sequential nature of the data) and convolutional 2D layer.

Original: The tangent activation function was used for gated mechanism inside ConvLSTM cell, while the sigmoid activation function was applied to the output of the final convolutional 2D layer, generating prediction on the input batch.
Revised: The tangent activation function was used for gated mechanism inside ConvLSTM cell, while the sigmoid activation function was applied to the output of the final convolutional 2D layer, generating prediction on the input batch.

Original: The entire model architecture is shown in Figure 6.
Revised: The entire model architecture is shown in Figure 6.

Original: The spiking ConvLSTM model architecture comprised the same set of layers, with a difference consisting in
Revised: The spiking ConvLSTM model architecture comprised the same set of layers, with a difference consisting in

Original: SConvLSTM cell integration.
Revised: SConvLSTM cell integration.

Original: The implementation of the cell  was taken from snnTorch API and unlike ConvLSTM cell, it uses sigmoid activation function for the input, forget and output gates, while preserving tangents for the cell state.
Revised: The implementation of the cell ( sconv2dlstm ) was taken from snnTorch API, which differs by a possibility to output not only hidden state with continuous values, called membrane activation potential in a spiking context, but also spikes - matrices filled with discrete values only (zeros and ones).

Original: The Straight Through Estimator surrogate gradient descend was used to learn the weights.
Revised: The Straight Through Estimator surrogate gradient descend was used to learn the weights.

Original: The models were trained using membrane activation potential,representing continuous values, which allowed application of the same loss function for both classical and spiking neural networks.
Revised: The models were trained using membrane activation potential, which allowed applying the same loss function for both classical and spiking neural networks.

Original: Training Setup
Revised: Training Setup

Original: The similar hyperparameters were applied to all the models, such as mini-batch size (15 sequences per batch), optimizer (Adam), learning rate (1e-4), sequence length (5 chords in a sequence), number of epochs (100 epochs), manual seed (42).
Revised: The similar hyperparameters were applied to all the models, such as mini-batch size (15 sequences per batch), optimizer (Adam), learning rate (1e-4), sequence length (5 chords in a sequence), number of epochs (100 epochs), manual seed (42).

Original: The loss function for ConvLSTM and
Revised: The loss function for ConvLSTM and

Original: SConvLSTM models was binary cross entropy.
Revised: SConvLSTM models was binary cross entropy.

Original: Since we used non-neuromorphic data, there was no need for additional time steps parameter, which is required for processing time distributed data, therefore this parameter was set to zero for
Revised: Since we used non-neuromorphic data, there was no need for additional time step parameter, which is required for processing time distributed data, therefore this parameter was set to zero for

Original: SConvLSTM models.
Revised: SConvLSTM models.

Original: The measurement of the training efficiency was done using mean value from 5 k-folds.
Revised: The measurement of the training efficiency was done using mean value from 5 k-folds.

Original: The initial one-dimensional custom data from seqGraph dataset consisted of 216 sequences of 5 chords each, where 4 chords served as an input and the 5 th chord was treated as a target.
Revised: The initial one-dimensional custom data from seqGraph dataset consisted of 216 sequences of 5 chords each, where 4 chords served as an input and the 5 th chord was treated as a target.

Original: In two-dimensional data split, each chord of the sequence was converted into a 28x28 matrix form, where the first 4 matrices represented the input features, while the last matrix represented a target.
Revised: In two-dimensional data split, each chord of the sequence was converted into a 28x28 matrix form, where the first 4 matrices represented the input features, while the last matrix represented a target.

Original: This way, four input chords conditioned the prediction of the final chord of the sequence.
Revised: This way, four input chords conditioned the prediction of the final chord of the sequence.

Original: The data from Moving MNIST dataset was resized from 64x64 to 28x28, the sequence length was reduced fromframes to 5 frames and the initial 10.000 entries were reduced to 216 and 2160 respectively, in order to match the size and structure of the custom seqGraph dataset.
Revised: The data from Moving MNIST dataset were resized from 64x64 to 28x28, the sequence length was reduced fromframes to 5 frames and the initial 10.000 entries were reduced to 216 and 2160 respectively, in order to match the size and structure of the custom seqGraph dataset.

Original: Finally, as was mentioned above, the seqMNIST was formed by selecting distinct digits for 216 sequences ofimages each, were 4 first images served as an input and the last 5 th image represented a target to predict.
Revised: Finally, as was mentioned above, the seqMNIST was formed by selecting distinct digits for 216 sequences ofimages each, were 4 first images served as an input and the last 5 th image represented a target to predict.

Original: Results
Revised: Results

Original: First stage.
Revised: First stage.

Original: The result of the first stage of  experiment is presented in Table 1, showing the loss values for train and validation (val.)
Revised: The result of the first stage of the experiment is presented in Table 1, showing the loss values for train and validation (val.)

Original: splits in the beginning of training (train start, val.
Revised: splits in the beginning of training (train start, val.

Original: start) and after the training completion (train end, val.
Revised: start) and after the training completion (train end, val.

Original: end).
Revised: end).

Original: The data concerns five ConvLSTM models trained on non-augmented seqGraph (seqGraph 216), 216 entries from  Moving MNIST (M. MNIST 216), seqMNIST (seqMNIST 216), augmented seqGraph (seqGraph 2160) and 2160 entries from  Moving MNIST datasets (M. MNIST 2160) respectively.
Revised: The data concerns five ConvLSTM models trained on non-augmented seqGraph (seqGraph 216), 216 entries from the Moving MNIST (M. MNIST 216), seqMNIST (seqMNIST 216), augmented seqGraph (seqGraph 2160) and 2160 entries from the Moving MNIST datasets (M. MNIST 2160) respectively.

Original: ConvLSTM models trained on three datasets with the same amount of data (216) show very different result in terms of the training efficiency.

Original: The loss in the beginning of training is almost identical for all three models, however, in the end of training, seqGraph gave the lowest train and validation loss values.

Original: The seqMNIST dataset, in opposite, resulted in the highest loss values, therefore, we can conclude that seqMNIST data were the hardest to process by the model.

Original: The result is confirmed by learning curves, shown in figures 7-9, where the best agreement between train and validation loss curves were achieved with seqGraph dataset (Figure 7).

Original: The distance between train and validation loss curves appears for the model trained on Moving MNIST (Figure 8) and augments for the model trained on seqMNIST (Figure 9).

Original: Model
Revised: Model

Original: Train start
Revised: Train start

Original: Val.
Revised: Val.

Original: start
Revised: start

Original: Train end
Revised: Train end

Original: Val.
Revised: Val.

Original: end
Revised: end

Revised: ConvLSTM seqGraph401.

Original: ConvLSTM
Revised: ConvLSTM

Original: [Table]

Original: [Table]

Original: M. MNIST413.
Revised: M. MNIST413.

Original: ConvLSTM seqMNIST430.

Original: ConvLSTM seqGraph142.

Original: ConvLSTM
Revised: ConvLSTM

Original: M. MNIST206.
Revised: [Table] The ConvLSTM models trained on three datasets with the same amount of data (216) show very different results in terms of the training efficiency. The loss in the beginning of training is almost identical for all three models, however, in the end of training, seqGraph gave the lowest train and validation loss values. The seqMNIST dataset, in opposite, resulted in the highest loss values, therefore, we can conclude that seqMNIST data were the hardest to process by the model.

Revised: The result is confirmed by learning curves, shown in figures 7-9, where the best agreement between train and validation loss curves were achieved with the seqGraph dataset (Figure 7).

Revised: The distance between train and validation loss curves appears for the model trained on Moving MNIST

Original:  The images generated with the model trained on seqMNIST have undistinguishable shape (Fig. 12), whereas the images generated with the model trained on seqGraph, although blurred, contain quite distinct shapes (Fig. 10).
Revised: (Figure 8) and augments for the model trained on seqMNIST (Figure 9). The images generated by the model trained on seqMNIST have indistinguishable shapes (Figure 12), whereas the images generated by the model trained on seqGraph, although blurred, contain quite distinct shapes (Figure 10).

Original: The images generated with the model trained on 216 entries from
Revised: The images generated by the model trained on 216 entries from

Original: Moving MNIST are the clearest (Fig. 11), however, the model was trained to generate a position of the target digits, not a distinct digit.
Revised: Moving MNIST are the clearest (Figure 11), however, the model was trained to generate a position of the target digits, not a distinct digit.

Original: Therefore, the charge of generating the shapes of  digits is lifted by the repetition of the same shapes in a sequence of 4 input images, provided at the training and inference phases. 
Revised: Therefore, the charge of generating the shapes of the digits is lifted by the repetition of the same shapes in a sequence of 4 input images, provided at the training and inference phases. targets the

Original:  The table data for the ConvLSTM models trained onentries from augmented seqGraph and Moving MNIST show a significant difference of the training efficiency, expressed in much lower training and validation losses for seqGraph in comparison to Moving MNIST.
Revised: ConvLSTM The table data for the ConvLSTM models trained onentries from augmented seqGraph and Moving MNIST show a significant difference of the training efficiency, expressed in much lower training and validation losses for seqGraph in comparison to Moving MNIST.

Original: This allows suggesting that dimensionality augmentation of data helps with the generalization capacities of the model.
Revised: This allows suggesting that dimensionality augmentation of data helps with the generalization capacity of the model.

Original: The learning curves (Figures 13-14) confirm that hypothesis, as the training and validation curves of the ConvLSTM model trained on seqGraph have the best agreement, while the same distance between train and validation loss curves appears for the model trained on 2160 entries of Moving
Revised: The learning curves (Figures 13-14) confirm that hypothesis, as the training and validation curves of the ConvLSTM model trained on seqGraph have the best agreement, while the same distance between train and validation loss curves appears for the model trained on 2160 entries of Moving

Original: MNIST, as it was the case for the model trained on 216 entries of the same dataset.
Revised: MNIST, as it was the case for the model trained on 216 entries of the same dataset.

Original: The generated images of these two models are shown in
Revised: The generated images of these two models are shown in

Original: Figures 15-16.
Revised: Figures 15-16.

Original: <

Original: Second stage.
Revised: Second stage.

Original: The result of the second stage of the experiment is presented in Table 2.
Revised: The result of the second stage of the experiment is presented in Table 2.

Original: As in case with ConvLSTM models, the SConvLSTM models trained on seqGraph data (augmented and non-augmented versions) show the lowest training and validation losses comparing to SConvLSTM models trained on Moving MNIST or seqMNIST datasets.
Revised: As in case with ConvLSTM models, the SConvLSTM models trained on seqGraph data (augmented and non-augmented versions) show the lowest training and validation losses compared to SConvLSTM models trained on Moving MNIST or seqMNIST datasets.

Original: The similar tendency is preserved for learning curves, for the group of datasets containing 216 entries (Figures 17-19), as well as for the datasets containing 2160 entries (Figures 23-24).
Revised: The similar tendency is preserved for learning curves, for the group of datasets containing 216 entries (Figures 17-19), as well as for the datasets containing 2160 entries (Figures 23-24).

Original: Although the loss values overall are slightly higher for the

Original: SConvLSTM models, the quality of generated images (Figures 20-22 and 25-26) is better than the generations made by

Original: ConvLSTM models.

Original: The observation is particularly truthful for images generated with SConvLSTM trained on 2160 entries of the augmented seqGraph dataset (Figure 25), which are much cleaner and exact in terms of the pixel position, than the images generated by the non-spiking counterpart trained on the same data (Figure 15).

Original: This allows suggesting that spiking ConvLSTM model architecture better processes sparse data.

Original: Model
Revised: Model

Original: Train start
Revised: Train start

Original: Val.
Revised: Val.

Original: start
Revised: start

Original: Train end
Revised: Train end

Original: Val.
Revised: Val.

Original: end
Revised: end

Original: SConvLSTM seqGraph416.
Revised: SConvLSTM seqGraph416.

Original: SConvLSTM
Revised: SConvLSTM

Original: M. MNIST446.
Revised: M. MNIST446.

Original: SConvLSTM seqMNIST450.
Revised: SConvLSTM seqMNIST450.

Revised: SConvLSTM seqGraph89.

Original: SConvLSTM
Revised: SConvLSTM

Revised: M. MNIST166.

Original: [Table]
Revised: Although the loss values overall are slightly higher for the SConvLSTM models, the quality of the generated images (Figures 20-22 and 25-26) is better than the generations made by ConvLSTM models. The observation is particularly truthful for images generated by the SConvLSTM model trained on 2160 entries of the augmented seqGraph dataset (Figure 25), which are much cleaner and more exact in terms of the pixel position, than the images generated by the non-spiking counterpart trained on the same data (Figure 15). This allows suggesting that spiking ConvLSTM model architecture better processes sparse data.

Original: Ablation studies
Revised: Ablation Studies

Original: We tested images generation without  batch normalization and without application of  Sigmoid activation  to the last frame.
Revised: We tested images generation without a batch normalization and without application of the Sigmoid activation function to the last frame.

Original: Thus, without  batch normalization layer no visual information was generated, although the loss values indicated that the networks trains well.
Revised: Thus, without a batch normalization layer, no visual information was generated, although the loss values indicated that the networks trained well.

Original: The ablation of  Sigmoid activation to the last frame resulted in blurriness augmentation in generated images, compared to the images generated with a full version of the network.
Revised: The ablation of the Sigmoid activation function resulted in blurriness augmentation in generated images, compared to the images generated with a full version of the network.

Original: We also tried adding spiking layers to the network, however the training was unpredictable and the generated images lacked pixel information.
Revised: We also tried adding spiking layers to the network, however the training was unpredictable and the generated images lacked pixel information.

Original: The possible cause of the unpredictable training might be the dead neurons problem, to which the spiking neural networks are particularly sensible.
Revised: The possible cause of the unpredictable training might be the dead neuron problem, to which the spiking neural networks are particularly sensitive.

Original: A possible solution of the problem can be an artificial simulation of neurons that did not fire (Kheradpisheh,
Revised: A possible solution of the problem can be an artificial stimulation of neurons that did not fire (Kheradpisheh,

Original: Mirsadeghi and Masquelier 2021), however the solution implementation is beyond the scope of this experiment.
Revised: Mirsadeghi and Masquelier 2021), however the solution implementation is beyond the scope of this experiment.

Original: Conclusions
Revised: Conclusions

Original: The article proposed a shallow spiking ConvLSTM architecture, the performance of which was compared to a shallow ConvLSTM architecture.
Revised: The article proposed a shallow spiking ConvLSTM architecture, the performance of which was compared to a shallow ConvLSTM architecture.

Original: The network was tested with a custom dataset seqGraph, sequences of digits from
Revised: The network was tested with a custom dataset seqGraph, sequences of digits from

Original: MNIST and sequences of frames from Moving MNIST datasets.
Revised: MNIST and sequences of frames from Moving MNIST datasets.

Original: It has been shown that  sparse nature of seqGraph data consistently gives better results in terms of the training efficiency, agreement of training and validation learning curves and the clarity of generated images for spiking and classical models, with the best result received for  spiking version of ConvLSTM trained on augmented version of seqGraph.
Revised: It has been shown that the sparse nature of seqGraph data consistently gives better results in terms of the training efficiency, agreement of training and validation learning curves and the clarity of generated images for spiking and classical models, with the best result received for the spiking version of ConvLSTM trained on augmented version of seqGraph.

Original: The multimodal nature of  constructed data allows further research in multimodal generation context.
Revised: The multimodal nature of the constructed data allows further research in multimodal generation context.