Original: d-DeVIS: A Gray Box Interpretable Visual Debugging Approach for Deep Sequence Learning Model 
Revised: A Gray Box Interpretable Visual Debugging Approach for Deep Sequence Learning Model Anonymous Author(s) AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: The widespread usability of Deep Learning algorithmsto solve various machinelearning problems demands transparent decision making policy. Deep Learningalgorithms are considered asa black box type learningmodel and they are toocomplex to understand. 
Revised: Deep Learning algorithms are often used as black box type learning and they aretoo complex to understand. The widespread usability of Deep Learning algorithmsto solve various machine learning problems demands deep and transparent understanding of the internal representation as well as decision making.

Original: Moreover, the learning models trained on sequential data,such as audio  data, has intricate reasoning process due to their complex distributionof features.
Revised: Moreover, thelearning models, trained on sequential data, such as audio and video data, haveintricate internal reasoning process due to their complex distribution of features.

Original: Thus, a visual simulator might be helpful to trace the internal decisionmaking mechanisms in response of adversarial input data, and thus, helps to debugand design appropriate deep learning models.
Revised: Thus, a visual simulator might be helpful to trace the internal decision makingmechanisms in response to adversarial input data, and it would help to debugand design appropriate deep learning models.

Original: However, interpreting the reasoningof the hidden layer is not well studied in the literature.
Revised: However, interpreting the internalreasoning of deep learning model is not well studied in the literature.

Original: Hence, in this work, wehave developed a visual interactive web application, namely d-DeVIS, which helpsto visualize the internal reasoning of a learning model which is trained on the audiodata.
Revised: In this work,we have developed a visual interactive web application, namely d-DeVIS, whichhelps to visualize the internal reasoning of the learning model which is trained onthe audio data.

Original: d-DeVIS allows to perceive the behavior as well as to debug the model byinteractively generating adversarial audio data point.
Revised: The proposed system allows to perceive the behavior as well as todebug the model by interactively generating adversarial audio data point.

Original: A demo video description ofd-DeVIS can be found here: https://youtu.be/v8lgR-ses_Y
Revised: The webapplication of d-DeVIS is available at http://bit.ly/d-DeVIS .

Original:  Introduction
Revised:  Introduction

Original: Machine Learning(ML)  have been pouring the blessings in a form of solving Artiﬁcial Intelligence(AI)problems, such as classiﬁcation, clustering, genomics data visualization.
Revised: Machine Learning(ML) algorithms have been pouring the blessings in a form of solving Artiﬁcial Intelligence(AI) problems, such as classiﬁcation, clustering, genomics data visualization, etc.

Original: Deep Learning(DL), aninﬂuential extension of ML, which is evolving rapidly in recent years and successfully applied insolving various real-world problems, for instance machine translation, speech recognition, imageclassiﬁcation etc.
Revised: Deep Learning(DL), an inﬂuential extension of ML, has been evolving rapidly in recent years andsuccessfully being applied in solving various real-world problems including machine translation,speech recognition, image classiﬁcation, etc[1][2][3].

Original: While previously ML models requires the external domain knowledge, DL ismostly characterized for efﬁcient learning of the non-linear complex features representation withoutdomain expert.
Revised: While traditional ML models require externaldomain knowledge, DL is mostly characterized for efﬁcient learning of the non-linear complexfeature representation without having domain expertise.

Original: Due to the complex feature representation, it is tough to visualize a deep learningmodel and thus people tends to use the DL model  as a black-box learning approach.
Revised: Hence, the DL model remains as a black-boxtype learning for practitioners and researchers.

Original: As a result,the interpretability and transparency of DL models are reduced signiﬁcantly [5].
Revised: In effect, the interpretability and transparency of DLmodels have been reduced signiﬁcantly [4].

Original: Although, DLapproaches have been used widely, in the literature a few works address the interpretability issue ofdeep learning models.
Revised: Although DL approaches have been studied widely, afew works address the interpretability issue of deep learning models in the literature.

Original: With the increasing use of the DL methodologies in real-world system, such as self driving car andmedical imaging, it becomes a prime concern to have publicly understandable systems explaining theunderlying reasoning.
Revised: With the increasing use of the DL methodologies in real-world systems, such as self driving car andmedical imaging, it becomes a prime concern to have publicly understandable systems explaining theunderlying reasoning.

Original: Although the linear systems can be easily demonstrated with simple exampleshaving mathematical proofs, non-linear systems like Deep Neural Network(DNN) is complex tounderstand and visualize.
Revised: Although the linear systems can be easily demonstrated with simple exampleshaving mathematical proofs, non-linear systems, such as Deep Neural Network(DNN), is complex tounderstand and visualize.

Original: Nonetheless, the general users as well as researchers need to understandthe mechanism of the algorithms for debugging and model selection purposes.
Revised: Nonetheless, the general users as well as researchers need to understandthe mechanism of the algorithms to debug and determine appropriate learning model.

Original: In addition, theteachers and the learners are interested to visualize the algorithms to develop the basic intuition ofthe algorithm.
Revised: In addition, theteachers and the learners are interested to visualize the algorithms to develop the basic intuition ofthe algorithm.

Original: As it is provenin the literature that people can develop a mental model of an algorithm when they are taught using visualization approaches [8][3], the researchers have been working toutilize the visualization approaches to teach the ML algorithms [10].
Revised: The researchers have been working to utilize the visualization approaches to teachthe ML algorithms [7] while it has been proven that people can grasp the principles of an algorithmbetter when they are taught using visualization approaches [5][6]

Original: Visualizing Neural Network(NN) methodologies is not a new talk in the research of visual learning[2]. Previously most of  the approaches utilize the static and Black Box visualization approaches tovisualize the hidden layer features representations and  the relationships between input and output ofthe learning model.
Revised: Visualization of internal operation details of a machine learning algorithm has been studied previouslyin [8], where the authors have surveyed several visualization techniques to understand the learningand decision-making processes of neural networks and also describe their work in knowledge-basedneural networks. After the explosion of deep learning applications in computer vision and machinetranslations, researchers have been trying to visualize the interpretations of the specialized algorithmsused for different kinds of unstructured data. In [9], authors have introduced a novel visualizationtechnique that gives insight into the function of intermediate feature layers and the operations of

Revised: Convolutional Neural Network(CNN).

Original: However, it is paramount important to visualize the DL techniques using gray boxapproach so that user can understand the global behavior and  the granular features representations toimprove the explain-ability of DL models. Previously, in the rule-based learning models it was easy to understand the underlying reasoning andvisualize the model behavior.
Revised: Nonetheless, it is rather black-box type visualization approachto reveal the model behavior, as such it can not interpret the internal reasoning. In [10], authors havedeveloped an interactive system to enable users understand and explore the deep learning modelsand get an insight on the learning mechanisms of image classiﬁers. It introduces a gray-box typeapproach but does not demonstrate how classiﬁers work in response to sequence audio data.

Original: Inspiring from this system, we  designed d-DeVIS : Deep Sequence Learning Model’s Debugger and Visual InteractiveSimulator.
Revised: In this paper, we have designed a deep Sequence Learning Model Debugger and Visual Interactive

Original: In this work, we are focusing on graybox concept, wherethe outcome of  internal architecture blocks are transparent to the users.
Revised: Simulator, namely d-DeVIS, that focuses on gray box concept, where outcome of an internal blockis transparent to the users.

Original: Here,we are interested to visualize the internal features representation of a Convolutional Neural Network(CNN) model  inthe response of multimodal audio sequence data.
Revised: More explicitly, we are interested to visualize the internal featurerepresentation of a deep sequence learning model (i.e. CNN) in response to multi modal audiosequence data.

Original: The layer wise visualization ofhidden feature representation and the hyper-parameters assists  to understand and the mathematicalinterpretations ofthe DL models.
Revised: The layer wise visualization of hidden features in d-DeVIS assists us to understandthe interpretation of feature extraction methods of DL models.

Original: The main contributions of the d-DeVIS are as:
Revised: The main contributions of the paperare as follows:

Original: • A web-based application,d-DeVIS , to visualize the  hidden layers features representations
Revised: • A web-based application, d-DeVIS, to visualize the representation of hidden layers’ featuresand

Original: and the behavior of of a CNN model in response to the adversarial audio sequence data. • d-DeVIS allows user to interactively change the audio features, such as pitch, amplitude etc, and interpret thethe behavior of the learning model based on the modiﬁed data. • We  designed a visually transparent debugging UI, which demonstrates layer-wise features
Revised: the behavior of the CNN model in response to the adversarial audio sequence data.• d-DeVIS, allows user to interactively change the audio features, such as pitch, amplitudeetc, and interpret the behavior of the learning model based on the modiﬁed data.• We have designed a visually transparent debugging User Interface(UI), which demonstrateslayer-wise features’ representation and model hyper parameters.

Original: representations and model hyper parameters. Thus, it guides DL model’s debugging. • d-DeVIS enables users to hear and visualize the intermediary hidden layer results in order to interpret the ﬁnal prediction.
Revised: In so doing, it guides DLmodel’s debugging.• d-DeVIS enables users to hear and visualize the intermediary hidden layer results, layer-wiseconverted audio outputs and weight distributions, in order to interpret the ﬁnal prediction.

Original: It also allows  to compare the performance of the learning model in response to different adversarial audio input.
Revised: Italso allows practitioners to compare the performance of the learning model in response todifferent adversarial audio input.

Original: The rest of the works are designed as follows.
Revised: The rest of the paper is structured as follows.

Original: In Section 2, we have discussed the related works.
Revised: In Section 2, we discuss the related work.

Original: Thereafter, Section 3 is designed with the goals and features of the proposed system.
Revised: Thereafter, Section 3 is focused on the goals and features of the proposed system.

Original: Section 4describe the detailed proposed system and use case, respectively.
Revised: Section 4 describes the usecases of d-DeVIS.

Original: Finally, Section 5 has concludedthe work with future plans.
Revised: Finally, Section 5 concludes with future plans.

Original:  Related Work
Revised:  Related Work

Original: The recent widespread use of deep learning models in various artiﬁcial intelligence task attracts boththe visualization and the deep learning communities to tackle the new challenge of improving theinterpretability and explainability of these models [2].
Revised: The recent widespread use of deep learning models in various artiﬁcial intelligence task attracts boththe visualization and the deep learning communities to deal with the new challenge of improvingthe interpretability and explainability of these models [8].

Revised: It is worth mentioning that visualizing the

Original: However, visualizing the Neural Network(NN)models is not a new research areas, it has been studied well before the recent surge of deep learningmodels.
Revised: Neural Network (NN) models is not a new research domain. To be precise, it has been studied wellbefore the recent surge of deep learning models.

Original: For instance, N 2 Vis [10] visualizes the attributes of NN, such as hidden layers weights,weights’ volatility, network structure and nodal activation levels.
Revised: For instance, N 2 Vis [7] visualizes the attributes of NN, such as hidden layers weights, weights’ volatility, network structure and nodal activation levels.

Original: Nonetheless, most of the previousapproaches utilize the static graphical visualization to describe hidden reasoning of the learningmodels.
Revised: Nonetheless, most of the previous approaches utilize the static graphical visualization to describehidden reasoning of the learning models.

Original: In recent years, a few works  address  explainability and transparency issue of the DL models anddesigned interactive visualization models to depict underline reasoning.
Revised: In recent years, a number of works have been sought to address the explainability and transparencyissue of the DL models and few others have been focused on designing interactive visualizationmodels to illustrate underline reasoning.

Original: For example, Tensorﬂow Playground [9] designed an interactive interface, where user can change the parameters and structureof the NN models and examine their effect.
Revised: For example, Tensorﬂow Playground [11] designed aninteractive interface, where users can change the parameters and structure of the NN models andexamine their effect.

Original: Moreover, ShapeShop [4], enables user to interactivelychange input image and visualize the behavior of the DL models and it’s internal features representations.
Revised: Moreover, ShapeShop [10] enables the users to interactively change input imageand visualize the behavior of the DL models and it’s internal feature’s representation.

Original: Similarly, in [1], author designed an application, which allows  user to examine the behavior ofa DL based image classiﬁcation models.
Revised: Similarly, in[12], authors designed an application, which allows an user to examine the behavior of a DL basedimage classiﬁcation models.

Original: Apart from these black-box visualization approaches, a few works visualize the behavior of deeplearning models.
Revised: Apart from these black-box visualization approaches, a number of works visualize the behaviorof deep learning models.

Original: For instance, in [6] author presents a static visualization of hidden state representation and the prediction model behavior of Long-Short-Term-Memory(LSTM) based languagemodel.
Revised: For instance, in [13], authors present a static visualization of hidden state representation and the prediction model behavior of Long-Short-Term-Memory(LSTM) basedlanguage model.

Original: Similar to the previous work, LSTMVis [12], designed an interactive visualization approachto visualize the hidden state representations of recurrent neural network and allows user to examinethe internal behavior of LSTM model on different application scenarios.
Revised: Similar to the previous work, LSTMVis [14] designed an interactive visualizationapproach to visualize the hidden state representations of recurrent neural network and allows userto examine the internal behavior of LSTM model on different application scenarios.

Original: Additionally, in [7] and [13],author visualize the Convolutional Neural Network(CNN and visually explainable reasoning of internal feature representation.
Revised: Additionally,in [15] and [9], authors visualize the Convolutional Neural Network (CNN) and provide visuallyexplainable reasoning of internal feature representation.

Original: Furthermore, Seq2Seq[11] designed a visual debugging tools forthe sequence-to-sequence learning model and enables users to interact with the model to develop aninsight about the model.
Revised: Furthermore, Seq2Seq [16] designed a visualdebugging tools for the sequence-to-sequence learning model and enables users to interact with themodel to develop an insight about the model.

Original: Inspired from the previous works [4][12][11], we  designed an interactive visual DL models debuggingsystem, d-DeVIS : Deep Sequence Learning Model’s Debugger and Visually Interactive Simulator.
Revised: Inspired from the previous works done by [10, 14, 16], we have designed an interactive visual DLmodels debugging system, d-DeVIS: Deep Sequence Learning Model’s Debugger and Visually Interactive Simulator.

Original: Most of the previous works utilize the black box visualization approaches to help developing the basicintuition of the deep learning models.
Revised: Most of the previous works utilize the black box visualization approachesto help developing the basic intuition of the deep learning models.

Original: Surprisingly, visualizing the deep learning model behavior andfeature representation of the multimodal data, such as audio or visual, is neglected in the literature.
Revised: Surprisingly, visualizing thedeep learning model behavior and features representation of the multimodal data, such as audio orvisual, is neglected in the literature.

Original: Moreover, visualizing the correlation between the hidden layer features representations and the modelbehavior is not properly studied for sequence based models. d-DeVIS allows user to interactivelychange the multimodal audio data to generate adversarial data examples and enables users to examinethe deep learning model behavior to visualize the feature representations.
Revised: Moreover, visualizing the correlation between the hidden layerfeatures representation and the model behavior is not properly studied for sequence based models.d-DeVIS allows user to interactively change the multimodal audio data to generate adversarial dataexamples and enables users to examine the deep learning model behavior to visualize the featuresrepresentation.

Original:  Proposed System: d-DeVIS
Revised:  Design and Development of d-DeVIS

Original: In this section, we present the key components and goals in designing our proposed interactiveapplication to visualize DL model in response to the adversarial data input.
Revised: In this section, we present the key components and goals for designing our proposed interactiveapplication to visualize DL model in response to the adversarial data input.

Original: In order for the interactivityof the users and ﬂexibility of the system, we made a web application which shows the gray boxdebugging method for deep neural network of sequence data.
Revised: We take into considerationsthe interactivity of the users and ﬂexibility of the system. To do so, we have developed a webapplication that shows the gray box debugging method for deep neural network of sequence data.

Original: The prime goal of designing d-DeVISis to makesure the learning and debugging DL model easier and visualize the internal reasoning ofdeep sequence model and feature representations of hidden layers with the help of an interactiveuser interface.
Revised: The prime goal of designing d-DeVIS is to make the learning and debugging DL model user friendlyand also ensure that it should be able to visualize the internal reasoning of deep sequence model andfeatures representation of hidden layers with the help of an interactive user interface.

Original: We listed a couple of crucial design goals in designing an interpretable deep audiosequence learning model in the Table 1.
Revised: Table 1 lists anumber of major design goals for designing an interpretable deep audio sequence learning model.

Original: [Table]
Revised: [Table]

Original: and
Revised: Improve

Original: [Table]
Revised: [Table]

Original: parency
Revised: Interpretability and Transparency

Original: An interpretable system of DL models depicts how deep sequence learning models work and how the hidden layer features can help to easily interpret the functionality of the learning model
Revised: An interpretable system of DL models depicts how deep sequence learning models work and how the hidden layer features can help to easily interpret the functionality of the learning model. [Table] Gray

Original: [Table]
Revised: [Table]

Original: A good grasp of the feature extraction method of deep neural networks is required for DL enthusiast and d-DeVIS provides a ﬂuid gray box debugging experience which enables the users to understand how the features of the hidden layers affect the training.
Revised: A good grasp of the feature extraction method of deep neural networks is required for DL enthusiast and d-DeVIS provides a ﬂuid gray box debugging experience which enables the users to understand how the features of the hidden layers affect the training.

Original: G3:
Revised: [Table]

Original: Interactively Examining
Revised: Interactively Examining

Original: the Deep Sequence Model Behavior
Revised: [Table] Behavior

Original: An interactive tool is required, where user can manipulate audio features(such as slicing, cross-fading, loudness, repetition etc) to generate adversarial example data.
Revised: An interactive tool is required, where user can manipulate audio features(such as slicing, cross-fading, loudness, repetition, etc) to generate adversarial example data.

Original: Moreover, it allows user to examine the internal reasoning in response of the modiﬁed adversarial data.
Revised: Moreover, it allows user to examine the internal reasoning in response to the modiﬁed adversarial data.

Original: G4:
Revised: G4:

Original: Comparison and exposure of the extracted features from audio data The proposed system must enable users to listen the extracted audio data 
Revised: Comparison and exposure of the extracted features from audio data The proposed system must enable users to listen the extracted audio data from different layer after applying CNN ﬁlters.

Original: from

Original: different layer after applying the ﬁlters

Original: of Convolutional

Original: Neural Network.

Original: Hence users  able to grasp theaudio extracted hidden layer audio features.
Revised: Hence, users should be able to grasp the extracted hidden layer audio features.

Original: 3.1 Features of d-DeVIS
Revised: 3.1 Features of d-DeVIS

Original: We  designed d-DeVIS as an interactive web applications by considering the design goals, which arelisted in Table 1.
Revised: We have designed d-DeVIS as an interactive web application while considering the design goals listedin Table 1.

Original: The primary goal of our proposed  is to ease the interpretation of the intermediatereasoning and the deep audio sequence learning model.
Revised: The primary goal of our proposed system is to ease the interpretation of the intermediatereasoning and the deep audio sequence learning model.

Original: We divide the proposed d-DeVIS model intothe following three major components.
Revised: We divide the proposed d-DeVIS model intothe following three major components.

Original: • Model Visualization.
Revised: • Model Visualization.

Original: • Audio Feature 
Revised: • Audio Feature Manipulation

Original: • Adversarial Feature Comparison
Revised: • Adversarial Feature Comparison

Original: 3.1.1 Model Visualization
Revised: 3.1.1 Model Visualization

Original: The primary purpose of our work is to interpret the internal reasoning of the deep sequence learningmodel in response of adversarial audio example data.
Revised: The primary purpose of our work is to interpret the internal reasoning of the deep sequence learningmodel in response to adversarial audio example data.

Original: For this reason, d-DeVIS provides an interactiveweb application interface, which depict the intermediate layer wise visual features representationin the form of audio spectrogram.
Revised: For this reason, d-DeVIS provides an interactiveweb application interface, which depicts the intermediate layer wise visual features representationin the form of audio spectrogram.

Original: Moreover, we employed the inverse Fourier transformation toextract the audio feature from the intermediate layer spectrogram.
Revised: Moreover, we employed the inverse Fourier transformation toextract the audio features from the intermediate layer spectrogram.

Original: d-DeVIS allows user to not onlyvisualize the features extracted by the hidden layer ﬁlters, but also it enables them to listen to theaudio representation of theextracted features of the input audio  by the CNN.
Revised: d-DeVIS allows user to notonly visualize the features extracted by the hidden layer ﬁlters, but also it enables them to listen tothe audio representation of the features of the input audio extracted by the CNN.

Original: The web interfaceto visualize the layer wise feature is depicted in Fig 1.
Revised: The web interfaceto visualize the layer wise feature is depicted in Fig 1.

Original: Furthermore, d-DeVIS allows the users toexamine the weight distributions of the internal hidden layer.
Revised: Furthermore, d-DeVIS allows the users toexamine the weight distributions of the internal hidden layer.

Original: To extract the intermediate features,we trained a baseline CNN model on audio sequence data.
Revised: To extract the intermediate features,we trained a baseline CNN model on audio sequence data.

Original: The details of the trained model and thebacked system of d-DeVIS is presented in Section 3.2.
Revised: The details of the trained model and thebacked system of d-DeVIS is presented in Section 3.2.

Original: During any forward propagation step, the spectrogram feature data of the audio ﬁles are traversedthrough the hidden layer of the CNN.
Revised: During any forward propagation step, the spectrogram feature data of the audio ﬁles are traversedthrough the hidden layer of the CNN.

Original: At each layer, the convolution ﬁlter tries to extract signiﬁcanthidden features from the audio data input and optimizes itself during backward propagation in orderto minimize the training loss.
Revised: At each layer, the convolution ﬁlter tries to extract signiﬁcanthidden features from the audio data input and optimizes itself during backward propagation in orderto minimize the training loss.

Original: In our system, the users will be able to upload an audio ﬁle or recordan audio of their own.
Revised: In our system, the users will be able to upload an audio ﬁle or recordan audio of their own.

Original: After the processing  the input, our system will calculate the logarithmicspectrogram and feed  into the trained model to produce the prediction.
Revised: After the processing of the input, our system will calculate the logarithmicspectrogram and feed it into the trained model to produce the prediction.

Original: At each convolution layerthere are predeﬁned tunned ﬁlters.
Revised: At each convolution layerthere are predeﬁned tunned ﬁlters.

Original: The types of features it extracts from the input data dependson the ﬁlters.
Revised: The types of features CNN] extracts from the input data dependson the ﬁlters.

Original: In our trained model, the ﬁrst layer and second layers have 16 ﬁlters, the third layerhas 32 ﬁlters, each.
Revised: In our trained model, the ﬁrst layer and second layers have 16 ﬁlters, the third layerhas 32 ﬁlters, each.

Original: So, our system visualizes the features corresponding to the ﬁlters and also thedistributions of the trained weights.
Revised: So, our system visualizes the features corresponding to the ﬁlters and also thedistributions of the trained weights.

Original: A particular feature extracted by the 13th ﬁlter of ﬁrst layer is visualized in Fig 1(a).
Revised: A particular feature extracted by the 13th ﬁlter of ﬁrst layer is visualized in Fig 1(a).

Original: When a userclicks on the image it zooms in to show the spectrogram clearly.
Revised: When a userclicks on the image it zooms in to show the spectrogram clearly.

Original: Users can also listen to the hiddenextracted feature by clicking on the play button, which is depicted in Fig.
Revised: Users can also listen to the hiddenextracted feature by clicking on the play button, which is depicted in Fig.

Original: 1(c).
Revised: 1(c).

Original: 3.1.2 Adversarial Feature Comparison
Revised: 3.1.2 Adversarial Feature Comparison

Original: d-DeVIS allows users to interpret the DL model behavior by examine the intermediate featuresrepresentation based on the different audio data input.
Revised: d-DeVIS allows users to interpret the DL model behavior by examining the intermediate featurerepresentation based on the different audio data input.

Original: The adversarial behavior comparison isillustrated in Fig 2 and the different module of this feature is presented in red numbers.
Revised: The adversarial behavior comparison isillustrated in Fig 2 and the different module of this feature is presented in red alphabets.

Original: The module aand b are the two spectrogram representation of the two audio inputs with their predictions by thetrained deep sequence learning model.
Revised: The modulea and b are the two spectrogram representation of the two audio inputs with their predictions by thetrained deep sequence learning model.

Original: Users can observe the feature representation of different layerin module d and e. There are different spectrogram images of the extracted features and users canclick on them to listen to the audio representation.
Revised: Users can observe the feature representation of different layersin module d and e. There are different spectrogram images of the extracted features and users can click on them to listen to the audio representation.

Original: Finally users can also see the weight distributionof each layers by clicking on the button marked by f.
Revised: Finally users can also see the weight distributionof each layers by clicking on the button marked by f.

Original: 3.1.3 Audio Feature Manipulation
Revised: 3.1.3 Audio Feature Manipulation

Original: Our proposed interactive system, d-DeVIS , enables the users to not only examine the behavior ofthe learning model in response to a sample ﬁle or recording of their voice but also it allows users tomanipulate the different properties of audio example data and thus enables to generate adversarialexample data.
Revised: Our proposed interactive system, d-DeVIS, enables the users to not only examine the behavior ofthe learning model in response to a sample ﬁle or recording of their voice but also it allows users tomanipulate the different properties of audio example data and thus enables to generate adversarialexample data.

Original: Among the various characteristics, which allows to change the audio data, d-DeVISallows to manipulate the following audio features.
Revised: Among the various characteristics of the audio, which can be changed, d-DeVISallows to manipulate the following audio features.

Original: • Slicing allows users to slice an audio.
Revised: • Slicing allows users to slice an audio.

Original: • Cross-fading changes the amplitude of the sound waves.
Revised: • Cross-fading changes the amplitude of the sound waves.

Original: • Changing the loudness option will make the beginning louder and the ending quieter.
Revised: • Changing the loudness option will make the beginning louder and the ending quieter.

Original: • Repeating option repeats the sound twice.
Revised: • Repeating option repeats the sound twice.

Original: • Invert: allows to invert the sound wave,
Revised: • Invert: allows to invert the sound wave,

Original: i
Revised: i

Original: e. inverted sound will be played from the ending.
Revised: e. inverted sound will be played from the ending.

Original: • Fade: option fade in for one second and then a fade out for the other second.
Revised: • Fade: option fades in for a particular time and then fades out similarly.

Original: A pictorial modiﬁcation of audio feature is presented in Fig.
Revised: A pictorial modiﬁcation of audio feature is presented in Fig.

Original: After manipulating and generating theaudio example data, d-DeVIS allows users to examine the behavior of audio deep sequence learningmodel by observing behavior change in response of original and adversarial audio data input.
Revised: After manipulating and generating theaudio example data, d-DeVIS allows users to examine the behavior of audio deep sequence learningmodel by observing behavior changes in response to the original and adversarial audio data input.

Original: In Table 2, we discuss all the features of d-DeVIS and how the features meet the design goals.
Revised: In Table 2, we discuss all the features of d-DeVIS and how the features meet the design goals.

Original: 3.2 Implementation of d-DeVIS
Revised: 3.2 Implementation of d-DeVIS

Original: d-DeVIS is developed as a web application so that users can seamlessly interact with the systemto interpret the behavior of deep learning model by generating adversarial audio input data.
Revised: d-DeVIS is developed as a web application so that users can seamlessly interact with the systemto interpret the behavior of deep learning model by generating adversarial audio input data.

Original: In thefollowing section we present the implementation details of d-DeVIS .
Revised: Inthe following section we present the implementation details of d-DeVIS.

Revised: The source code of ourimplementation is available at https://github.com/anon-conf/d-DeVIS .

Original: 3.2.1 Trained Deep Learning Model with Audio Data
Revised: 3.2.1 Trained Deep Learning Model with Audio Data

Original: We trained a CNN model on Speech Commands dataset 1 , which is used to visualize the behavior ofthe model.
Revised: We trained a CNN model on Speech Commands dataset 1 , which is used to visualize the behavior ofthe model.

Original: The dataset consists of almost 30 speech classes but for the sake of the simplicity andreduce the training time we used 10 classes, which are the audio recordings of zero to nine digitsin English language. All the clips are two seconds long. We calculated logarithmic spectrogram asfeatures of the audio (.wav) data to feed into the training model. A three layer Convolutional Neural
Revised: The dataset consists of almost 30 speech classes but for the sake of the simplicity and

Original: Network (CNN) is used as the spectrogram feature matrix can represent an image and CNNs have

Original: [Figure]
Revised: [Figure]

Original: Feature
Revised: Feature

Original: Goal
Revised: Goal

Original: Visualizing the Hidden Extracted Features of Convolutional neural networks: d-DeVIS provides visualization of the extracted features in each layers as image data and shows the various features of different ﬁlters of the deep convolutional network.
Revised: Visualizing the Hidden Extracted Features of Convolutional neural networks: d-DeVIS provides visualization of the extracted features in each layer as image data and shows the various features of different ﬁlters of the deep Convolutional Neural Network.

Original: Users can also hear the audio representation of the hidden extracted features
Revised: Users can also hear the audio representation of the hidden extracted features

Original: G1 & G & G
Revised: G1 & G & G

Original: Interactive User Experience:
Revised: Interactive User Experience:

Original: For a ﬂuid user experience,we provide an interactive platform for the users so that they will be able to focus on the productivity of the system without any unnecessary hassles.
Revised: For a ﬂuid user experience,we provide an interactive platform for the users so that they will be able to focus on the productivity of the system without any unnecessary hassle.

Original: G3 & G
Revised: G3 & G

Original: Visualizing the Audio Features as well as Modifying the Waveforms: Due to the complex structure of audio data, our system let’s users modify various aspects of the sound property and visualize the updated waveform to provide a keen knowledge on audio data representation.
Revised: Visualizing the Audio Features as well as Modifying the Waveforms: Due to the complex structure of audio data, our system let’s users modify various aspects of the sound property and visualize the updated waveform to provide a keen knowledge on audio data representation.

Original: G2 & G
Revised: G2 & G

Original: Custom Audio Input for Testing and Feature Distribution Visualization: User can not only upload a default audio data but also they can record custom speech to test the trained model.
Revised: Custom Audio Input for Testing and Feature Distribution Visualization: User can not only upload a default audio data but also they can record custom speech to test the trained model.

Original: Proper distribution of the weights is also visualized.
Revised: Proper distribution of the weights is also visualized.

Original: G1 & G
Revised: G1 & G

Original: Comparing different audio inputs and their hidden features: d-DeVIS also enables users to measure the differences of different audio inputs and check their extracted layer features.
Revised: Comparing different audio inputs and their hidden features: d-DeVIS also enables users to measure the differences of different audio inputs and check their extracted layer features.

Original: G proven to be decent at image classiﬁcation.
Revised: G reduction of training time we used 10 classes, which are the audio recordings of zero to nine digitsin English language. All the clips are one second long. We calculated logarithmic spectrogram asfeatures of the audio (.wav) data to feed into the training model. A three layer Convolutional Neural

Revised: Network (CNN) is used as the spectrogram feature matrix represents an image and CNNs have provento be decent at image classiﬁcation.

Original: Our baseline model reached 95% validation accuracywith a minimal hyper parameter tuning.
Revised: Our baseline model reached 95% validation accuracy with aminimal hyper parameter tuning.

Original: We have used Keras deep learning framework which is awrapper library of Tensorﬂow to train our deep learning model.
Revised: We have used Keras deep learning framework which is a wrapperlibrary of Tensorﬂow to train our deep learning model.

Original: We have utilized the computationsystem of Google Colaboratory platform for the training purpose.
Revised: We have utilized the computation system of Google Colaboratory platform for the training purpose.

Original: 3.2.2 Front-end and Back-end of d-DeVIS
Revised: 3.2.2 Front-end and Back-end of d-DeVIS

Original: Audio ﬁles are manipulated such as Slicing the audio, Changingin Loudness, Cross-fading, Repeatingthe Sound, Invert and Fade  using Numpy, Pydub and Scipy libraries.
Revised: Audio ﬁles manipulations such as Slicing the audio, Changing Loudness, Cross-fading, Repeatingthe Sound, Invert and Fade are done using Numpy, Pydub and Scipy libraries.

Original: Matplotlib is utilized to visualizethe audio features.
Revised: Matplotlib is utilized to visualize the audio features.

Original: After the training  the model, we have saved the data using Pickle python module.
Revised: After the training of the model, we have saved the data using Picklepython module.

Original: We have used HTML5, Bootstrap, Javascript for designing the front-end of the application.
Revised: We have used HTML5, CSS, Javascript for designing the front-end of the application.

Original: Wehavealso used python ﬂask web frameworkthat helped to maintain the interaction between front-end andback-end communication.
Revised: We used the Vue.js framework to build an SPA and communicated with the server using REST API. The back-end is built in python using the Flask framework.

Original:  Use Cases
Revised:  Use Cases

Original: While experimenting with the system, we have applied several modiﬁcations to both the CNN trainedmodel and the input audio ﬁle.
Revised: While experimenting with the system, we have applied several modiﬁcations to both the CNNtrained model and the input audio ﬁle.

Original: Thus, we have analyzed the results that we got due to  thetunning withthe system.
Revised: Thus, we have analyzed the results that we got due to thetunning with the system.

Original: In this section, we present important use cases that demonstrate the general applicabilityof our system.
Revised: In this subsection, we present important use cases that demonstrate thegeneral applicability of our system.

Revised: A demo video of our proposed system d-DeVIS can be found athttps://youtu.be/lpcpArypI7c .

Original: • Visualizing the Audio Features: Speech is a sequence data which is hard to grasp just bylooking at the amplitude vs time representation.
Revised: • Visualizing the Audio Features: Speech is a sequence data which is hard to grasp just by looking at the amplitude vs time representation.

Original: In our system, a user can upload or recorda custom audio ﬁle and tune with various aspects of the waveform.
Revised: In our system, a user can upload or recorda custom audio ﬁle and tune with various aspects of the waveform.

Original: So predictions of theaudio will change with accordance to the change in the waveforms and users can easily seethe changed results.• Learning Medium for the Academia:
Revised: So predictions of theaudio will change with accordance to the change in the waveforms and users can easily seethey changed results.• Learning Medium for the Academia:

Original: Our system provides an interactive web applicationwith which learners will be able to test various types of aspects of audio data and the deeplearning model.
Revised: Our system provides an interactive web application with which learners will be able to test various types of aspects of audio data and thedeep learning model.

Original: After the model training completion, the proper visualizations are given. The academia will be able to use it for teaching purpose.•
Revised: Through d-DeVIS academia can provide appropriate insights for thestudents to help them grasp the feature extraction method of neural networks.

Original:  Experimenting Platform for AI Enthusiasts: We provide a platform for easy training andproper results of the feature extractions which are shown as a form of images.
Revised: So it’s a greatmedium for learning.• Experimenting Platform for AI Enthusiasts: We provide a platform for easy training andproper results of the feature extractions which are shown as a form of images.

Original: Users can testtheir own custom input and experiment on various types of hyper-parameters from whichit can get the maximum accuracy.
Revised: Users can testtheir own custom input and observe the decisive hidden features that make the distinctionsbetween the inputs.

Original: Thus, the feature manipulation and interactivity of thesystem will inspire the deep learning enthusiasts and engineers to do various experiments onit.
Revised: Thus, the feature manipulation and interactivity of the system will inspire the deep learning enthusiasts and engineers to do various experiments on it.

Original:  Conclusionand Future Work
Revised:  Conclusion

Original: d-DeVIS allows the user to visualize how CNN recognizes digits from audio sequence data.
Revised: d-DeVIS allowed the users to visualize how CNN recognizes digits from audio sequence data.

Original: It collectsinput from user and allows them to interactively manipulate it.
Revised: Itcollected input from user and allowed them to interactively manipulate it.

Original: The tool easily allows for comparingthe given input with other adverserial examples.
Revised: The tool easily allowed thecomparison of the given input with other adversarial examples.

Original: Overall, this helps users to develop a better intuitionof the underlying reasoning of the model which will allow them to make more learned decisionsregarding learning model development in the future.
Revised: Overall, this helped users to developa better intuition of the underlying reasoning of the model which allowed them to make more learneddecisions regarding learning model development.

Original: In future extension of d-DeVIS, we have the plan to visualizethe other deep learning models behaviorallowing users to manipulate the input data representation interactively.
Revised: In future extension of d-DeVIS we have the plan to visualize other deep learning models behaviorand allow users to manipulate the input data representation interactively.

Original: Moreover, visualizing thehidden layer complex feature representations for multi-modal sequence data, is a great avenue forfuture research work.
Revised: Moreover, visualizing thehidden layer complex feature representations for multi-modal sequence data is a great avenue forfuture research work.