Original: M ODUL OM: D ISSEMINATING DEEP LEARNING RESEARCH WITH MODULAR OUTPUT MATHEMATICS
Revised: M ODUL OM: D ISSEMINATING D EEP L EARNING R ESEARCH WITH M ODULAR O UTPUT M ATHEMATICS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Solving a task with a deep neural network requires an appropriate formulation of the underlying inference problem.
Revised: Solving a task with a deep neural network requires an appropriate formulation of the underlying inference problem.

Original: A formulation deﬁnes the type of variables output by the network, but also the set of variables and functions, denoted output mathematics, needed to turn those outputs into task-relevant predictions.
Revised: A formulation deﬁnes the type of variables output by the network, but also the set of variables and functions, denoted output mathematics, needed to turn those outputs into task-relevant predictions.

Original: Despite the fact that the task performance may largely depend on the formulation, most deep learning experiment repositories do not offer a convenient solution to explore formulation variants in a ﬂexible and incremental manner.
Revised: Despite the fact that the task performance may largely depend on the formulation, most deep learning experiment repositories do not offer a convenient solution to explore formulation variants in a ﬂexible and incremental manner.

Original: Software components for neural network creation, parameter optimization or data augmentation, in contrast, offer some degree of modularity that has proved to facilitate the transfer of know-how associated to model development.
Revised: Software components for neural network creation, parameter optimization or data augmentation, in contrast, offer some degree of modularity that has proved to facilitate the transfer of know-how associated to model development.

Original: But this is not the case for output mathematics.
Revised: But this is not the case for output mathematics.

Original: Our paper proposes to address this limitation by embedding the output mathematics in a modular component as well, by building on multiple inheritance principles in object-oriented programming.
Revised: Our paper proposes to address this limitation by embedding the output mathematics in a modular component as well, by building on multiple inheritance principles in object-oriented programming.

Original: The ﬂexibility offered by the proposed component and its added value in terms of knowledge dissemination are demonstrated in the context of the Panoptic-Deeplab method, a representative computer vision use case.
Revised: The ﬂexibility offered by the proposed component and its added value in terms of knowledge dissemination are demonstrated in the context of the Panoptic-Deeplab method, a representative computer vision use case.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Proper dissemination of deep learning research outcomes requires pedagogical explanation of theoretical concepts, but also transfer of hands-on experiments (Isensee et al., 2021).
Revised: Proper dissemination of deep learning research outcomes requires pedagogical explanation of theoretical concepts, but also transfer of hands-on experiments (Isensee et al., 2021).

Original: Papers succeed in the former, but are limited regarding the second.
Revised: Papers succeed in the former, but are limited regarding the second.

Original: Despite Github repositories and popular deep learning libraries (e.g. PyTorch (Paszke et al., 2019), Tensorﬂow(Abadi et al., 2015)) nowadays provide a common way to share code and information regarding the practical implementation of a model, the release of code is still far from solving all issues related to reproducibility, fair comparison of methods and, maybe more importantly, transmission of knowledge required for fruitful incremental development.
Revised: Despite Github repositories and popular deep learning libraries (e.g. PyTorch (Paszke et al., 2019), Tensorﬂow(Abadi et al., 2015)) nowadays provide a common way to share code and information regarding the practical implementation of a model, the release of code is still far from solving all issues related to reproducibility, fair comparison of methods and, maybe more importantly, transmission of knowledge required for fruitful incremental development.

Original: This is because the multiple code bases released to address a given task often vary in the way they deﬁne and manipulate the variables associated to their respective mathematical formulation of the task, i.e. in the variables and functions, denoted output mathematics, deﬁned to turn the network outputs into task-relevant predictions (e.g. compare the predictions in (Cheng et al., 2020) and (Neven et al., 2019)).
Revised: This is because the multiple code bases released to address a given task often vary in the way they deﬁne and manipulate the variables associated to their respective mathematical formulation of the task, i.e. in the variables and functions, denoted output mathematics, deﬁned to turn the network outputs into task-relevant predictions (e.g. compare the predictions in (Cheng et al., 2020) and (Neven et al., 2019)).

Original: This makes their comparison subtle since involved data, and intermediate results (of interest for debugging and/or understanding purposes), may largely differ.
Revised: This makes their comparison subtle since involved data, and intermediate results (of interest for debugging and/or understanding purposes), may largely differ.

Original: It is frequent that the relatively minor variations of the task formulation reported in a paper lead to signiﬁcant changes in the associated reference software implementations.
Revised: It is frequent that the relatively minor variations of the task formulation reported in a paper lead to signiﬁcant changes in the associated reference software implementations.

Original: Fundamentally, those large discrepancies between implementations are primarily due to the lack of modularity associated to the processing of the network outputs.
Revised: Fundamentally, those large discrepancies between implementations are primarily due to the lack of modularity associated to the processing of the network outputs.

Original: Modularity is a key desired feature when designing software.
Revised: Modularity is a key desired feature when designing software.

Original: Decomposing a program in modules improves understandability, makes the code easy to reuse, facilitates its evolution and/or maintenance, and allows multiple programmers to work independently on distinct modules (Parnas, 1972).
Revised: Decomposing a program in modules improves understandability, makes the code easy to reuse, facilitates its evolution and/or maintenance, and allows multiple programmers to work independently on distinct modules (Parnas, 1972).

Original: Without surprise, modularity is ubiquitous in the code base of any deep learning (DL) experiment.
Revised: Without surprise, modularity is ubiquitous in the code base of any deep learning (DL) experiment.

Original: For instance, external libraries often provide different optimizers, datasets, data augmentation definitions that can be used off the shelf to create interchangeable components (Chollet et al., 2015; Paszke et al., 2019). This is also the case for the deﬁnition of neural networks architecture. In that
Revised: For instance, external libraries often provide different optimizers, datasets, data augmentation def-

Original: [Figure]
Revised: [Figure]

Original: In contrast, the mathematics related to the network output are generally not implemented in a modular manner.
Revised: In contrast, the mathematics related to the network output are generally not implemented in a modular manner.

Original: Those mathematics include the functions that deﬁne losses, the functions required to translate the network output into semantically meaningful information and potentially many other functions that compute intermediate variables or are required to visualize the manipulated concepts (e g. accuracy or quality metrics, but also high-dimension tensors visualized based on a PCA transform), for debugging purposes for instance.
Revised: Those mathematics include the functions that deﬁne losses, the functions required to translate the network output into semantically meaningful information, and potentially many other functions that compute intermediate variables or are required to visualize the manipulated concepts (e g. accuracy or quality metrics, but also high-dimension tensors visualized based on a PCA transform), for debugging purposes for instance.

Original: In the rest of this manuscript the term output mathematics is used to denote this set of mathematical functions that handle the network outputs.
Revised: In the rest of this manuscript, the term output mathematics is used to denote this set of mathematical functions that handle the network outputs.

Original: The reason why output mathematics (OM) are not modular lies in the fact that they are conventionally implemented as a set of functional abstractions (i e. functions).
Revised: The reason why output mathematics (OM) are not modular lies in the fact that they are conventionally implemented as a set of functional abstractions (i.e. functions).

Original: Fundamentally, functional abstractions are constrained by their signatures, i.e. their list of arguments, and are thus especially suited to deal with scenarios where the developer wants to test multiple versions of a single function whose interface with the rest of the program is ﬁxed.
Revised: Fundamentally, functional abstractions are constrained by their signatures, i.e. their list of arguments, and are thus especially suited to deal with scenarios where the developer wants to test multiple versions of a single function whose interface with the rest of the program is ﬁxed.

Original: This scenario does not ﬁt to the OM case, often characterized by the manipulation of multiple interdependent functions, especially in modern computer vision models (pose estimation (Newell et al., 2017;
Revised: This scenario does not ﬁt to the OM case, often characterized by the manipulation of multiple interdependent functions, especially in modern computer vision models (pose estimation (Newell et al., 2017;

Original: Papandreou et al., 2018; Kreiss et al., 2019), instance segmentation (Neven et al., 2019; Novotny et al., 2018;
Revised: Papandreou et al., 2018; Kreiss et al., 2019), instance segmentation (Neven et al., 2019; Novotny et al., 2018;

Original: He et al., 2017) or panoptic segmentation (Kirillov et al., 2019; Cheng et al., 2020)).
Revised: He et al., 2017) or panoptic segmentation (Kirillov et al., 2019; Cheng et al., 2020)).

Original: This interdependency requires careful orchestration to feed each function with the right arguments.
Revised: This interdependency requires careful orchestration to feed each function with the right arguments.

Original: Hence, as depicted in Figure 1 (left), the functions written in the code base of such experiments end up in being largely constrained by their signatures, which determine the way they access the variables they process.
Revised: Hence, as depicted in Figure 1 (left), the functions written in the code base of such experiments end up in being largely constrained by their signatures, which determine the way they access the variables they process.

Original: They depend on each other and are not plug-and-play: the alternatives to a function are limited to those with the same interface, which can be designed to be called at the same time with the same dependencies.
Revised: They depend on each other and are not plug-and-play: the alternatives to a function are limited to those with the same interface, which can be designed to be called at the same time with the same dependencies.

Original: Consequently, the changes in dependencies induced by slight modiﬁcations in the task formulation might impact the code in many places.
Revised: Consequently, the changes in dependencies induced by slight modiﬁcations in the task formulation might impact the code in many places.

Original: This hampers the deployment of a modular implementation paradigm, which penalizes transmission of know-how and incremental research, since it prevents leveraging existing OM methods to investigate alternative task formulations.
Revised: This hampers the deployment of a modular implementation paradigm, which penalizes transmission of know-how and incremental research, since it prevents leveraging existing OM methods to investigate alternative task formulations.

Original: Promoting modularity would make sub-tasks free to evolve in unforeseen ways at minimal code rewriting cost, but would also help in comparing alternative task formulations by pointing explicitly the modules that differ between them.
Revised: Promoting modularity would make sub-tasks free to evolve in unforeseen ways at minimal code rewriting cost, but would also help in comparing alternative task formulations by pointing explicitly the modules that differ between them.

Original: This paper proposes a ﬁrst practical solution to make the OM modular, and illustrates how the underlying structure of such a modular framework favors ﬂuent dissemination of research outcomes, enabling easier increments and more accurate comparisons.
Revised: This paper proposes a ﬁrst practical solution to make the OM modular, and illustrates how the underlying structure of such a modular framework favors ﬂuent dissemination of research outcomes, enabling easier increments and more accurate comparisons.

Original: Our prime contribution lies in a set of recommendations to facilitate developers’ work when exploring the space of formulations to address a given task.
Revised: Our prime contribution lies in a set of recommendations to facilitate developers’ work when exploring the space of formulations to address a given task.

Original: As illustrated in Figure 1 (right), the key idea of our proposal consists in making the OM variables accessible to methods without any function signature.
Revised: As illustrated in Figure 1 (right), the key idea of our proposal consists in making the OM variables accessible to methods without constraining

Original: This is achieved by deﬁning the OM modules as classes, and by making all OM variables

Original: [Figure]
Revised: [Figure]

Revised: function signature.

Revised: This is achieved by deﬁning the OM modules as classes, and by making all

Original:  accessible to the class method through dedicated methods.
Revised: OM variables accessible to the class method through dedicated methods.

Original: Multiple class inheritance principles, with method overriding and mixins, are shown to offer a natural way to increment the code or explore variants of the problem formulation.
Revised: Multiple class inheritance principles, with method overriding and mixins, are shown to offer a natural way to increment the code or explore variants of the problem formulation.

Original: Our proposal, grounded in the class inheritance paradigm, is a step forward in offering modularity, with minimal constraints on functions interfaces and interdependencies.
Revised: Our proposal, grounded in the class inheritance paradigm, is a step forward in offering modularity, with minimal constraints on functions interfaces and inter-dependencies.

Original: By relaxing the assumptions made about how those functions interact with the rest of the code (both in terms of access to data and function composition), it facilitates the presentation of the output mathematics as an inheritance diagram that explicitly presents how methods differ from a baseline, with the purpose of facilitating both the transfer of knowledge and the incremental development of DL models.
Revised: By relaxing the assumptions made about how those functions interact with the rest of the code (both in terms of access to data and function composition), it facilitates the presentation of the output mathematics as an inheritance diagram that explicitly presents how methods differ from a baseline, with the purpose of facilitating both the transfer of knowledge and the incremental development of DL models.

Original: In a nutshell, through appropriate deﬁnition of classes and hierarchical dependencies, multiple inheritance is shown to offer the ﬂexibility needed to overwrite and/or compose functions at will, making the output mathematics module easy to adapt or substitute by another.
Revised: In a nutshell, through appropriate deﬁnition of classes and hierarchical dependencies, multiple inheritance is shown to offer the ﬂexibility needed to overwrite and/or compose functions at will, making the output mathematics module easy to adapt or substitute by another.

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: Section 2 surveys the inheritance mechanisms that are mobilized to implement our solution.
Revised: Section 2 surveys the inheritance mechanisms that are mobilized to implement our solution.

Original: It also concretely deﬁnes the notion of output mathematics.
Revised: It also concretely deﬁnes the notion of output mathematics.

Original: Section 3 presents our proposed paradigm to implement the output mathematics as a modular component, and explains why and how this paradigm overcomes most of the issues encountered by the conventional and scattered implementation alternatives.
Revised: Section 3 presents our proposed paradigm to implement the output mathematics as a modular component, and explains why and how this paradigm overcomes most of the issues encountered by the conventional and scattered implementation alternatives.

Revised: Section 4 discusses key implications of modular output mathematics for research dissemination.

Original: Section 5 concludes.
Revised: Section 5 concludes.

Original:  B ACKGROUND 2. N ETWORK OUTPUT MATHEMATICS
Revised:  B ACKGROUND 2. N ETWORK OUTPUT MATHEMATICS

Original: Fundamentally, in a deep learning model, the output mathematics correspond to the set of operations that are related to the transformation of the output activations of a neural network into semantically meaningful information.
Revised: Fundamentally, in a deep learning model, the output mathematics correspond to the set of operations that are related to the transformation of the output activations of a neural network into semantically meaningful information.

Original: They are deﬁned in mathematical language, through the manipulation of mathematical variables.
Revised: They are deﬁned in mathematical language, through the manipulation of mathematical variables.

Original: They include (i) the deﬁnition of the training loss(es), typically comparing the network output(s) with some representation of the desired prediction(s), but also (ii) the functions that process/decode the (multiple) outputs of the network to turn them into meaningful predictions.
Revised: They include (i) the deﬁnition of the training loss(es), typically comparing the network output(s) with some representation of the desired prediction(s), but also (ii) the functions that process/decode the (multiple) outputs of the network to turn them into meaningful predictions.

Original: For example, modern pose estimation methods, e.g. (Kreiss et al., 2019), predict a set of heatmaps in which hot-spots locate a speciﬁc type of keypoint (e.g. head or feet), together with vector ﬁelds that are used to associate the keypoints of an instance.
Revised: For example, modern pose estimation methods, e.g. (Kreiss et al., 2019), predict a set of heatmaps in which hot-spots locate a speciﬁc type of keypoint (e.g. head or feet), together with vector ﬁelds that are used to associate the keypoints of an instance.

Original: As another example, popular instance segmenta- tion methods predict a seed map to locate the instance center, together with a map of embeddings, deﬁned so that the pixels of individual instances aggregate in tight and distinct clusters in the embedding space (Newell et al., 2017; Neven et al., 2019).
Revised: As another example, popular instance segmentation methods predict a seed map to locate the instance center, together with a map of embeddings, deﬁned so that the pixels of individual instances aggregate in tight and distinct clusters in the embedding space (Newell et al., 2017; Neven et al., 2019).

Original: In this formulation, each instance center corresponds to a hot-spot in the seed map, and the pixels of a speciﬁc instance are simply identiﬁed as the ones whose embedding lies sufﬁciently close to the embedding of that particular instance centers.
Revised: In this formulation, each instance center corresponds to a hot-spot in the seed map, and the pixels of a speciﬁc instance are simply identiﬁed as the ones whose embedding lies sufﬁciently close to the embedding of that particular instance centers.

Original: Figure 2 illustrates this principle in the case where the embedding space corresponds to 2D spatial coordinates, deﬁned in each pixel as the sum between the pixel coordinates and a 2D offset that is predicted by the network.
Revised: Figure 2 illustrates this principle in the case where the embedding space corresponds to 2D spatial coordinates, deﬁned in each pixel as the sum between the pixel coordinates and a 2D offset that is predicted by the network.

Original: In this scenario, the network is trained to make the embeddings of all pixels associated to a same instance equal to a common 2D vector, referred to as the instance center.
Revised: In this scenario, the network is trained to make the embeddings of all pixels associated to a same instance equal to a common 2D vector, referred to as the instance center.

Original: This corresponds to the Panoptic-Deeplab method, deﬁned in (Cheng et al., 2020).
Revised: This corresponds to the Panoptic-Deeplab method, deﬁned in (Cheng et al., 2020).

Original: The ﬁgure presents the main variables involved in the instance segmentation process.
Revised: The ﬁgure presents the main variables involved in the instance segmentation process.

Original: The variables depicted in blue on the left-side are directly predicted by the network, in the form of raw tensors.
Revised: The variables depicted in blue on the left-side are directly predicted by the network, in the form of raw tensors.

Original: Other variables are computed from the dataset (ground-truth column), from the raw network predictions (predicted maps column), or from both the dataset and predicted maps (loss functions).
Revised: Other variables are computed from the dataset (ground-truth column), from the raw network predictions (predicted maps column), or from both the dataset and predicted maps (loss functions).

Original: The mathematical operations that are used to compute all those variables (i.e. maps and losses) correspond to the so-called output mathematics, and directly reﬂect the computer vision problem formulation.
Revised: The mathematical operations that are used to compute all those variables (i.e. maps and losses) correspond to the so-called output mathematics, and directly reﬂect the computer vision problem formulation.

Original: In Section 3, we present a number of implementation guidelines that offer the ﬂexibility required to change those functions at will.
Revised: In Section 3, we present a number of implementation guidelines that offer the ﬂexibility required to change those functions at will.

Original: The application of those guidelines to implement and compare variants of the Panoptic-Deeplab formulation is presented in Appendix to demonstrate the advantages of our proposal in terms of incremental development and knowledge transfer.
Revised: The application of those guidelines to implement and compare variants of the Panoptic-Deeplab formulation is presented in Appendix to demonstrate the advantages of our proposal in terms of incremental development and knowledge transfer.

Original: Since those guidelines build on the multiple inheritance paradigm, we brieﬂy review its key principles in the next section.
Revised: Since those guidelines build on the multiple inheritance paradigm, we brieﬂy review its key principles in the next section.

Original: 2. M ULTIPLE INHERITANCE
Revised: 2. M ULTIPLE INHERITANCE

Original: Fundamentally, in object-oriented programming, classes are extensible templates for creating objects, providing the initial values for object instance variables, and the bodies for methods (Bruce, 2002).
Revised: Fundamentally, in object-oriented programming, classes are extensible templates for creating objects, providing the initial values for object instance variables, and the bodies for methods (Bruce, 2002).

Original: All objects generated from the same class share the same methods, but manipulate distinct instance variables.
Revised: All objects generated from the same class share the same methods, but manipulate distinct instance variables.

Original: New objects are created from a class by calling its constructor with speciﬁc arguments.
Revised: New objects are created from a class by calling its constructor with speciﬁc arguments.

Original: A class can inherit the base behavior of another class.
Revised: A class can inherit the base behavior of another class.

Original: Its methods are deﬁned by default to those of its base class, but they can also be overridden to reﬁne their behavior as needed.
Revised: Its methods are deﬁned by default to those of its base class, but they can also be overridden to reﬁne their behavior as needed.

Original: For compatibility reasons, the overriding method usually has the same name, parameters, and return type as the method it overrides.
Revised: For compatibility reasons, the overriding method usually has the same name, parameters, and return type as the method it overrides.

Original: Overriding helps in avoiding code duplication between different classes that would only differ through subtle variants of their methods.
Revised: Overriding helps in avoiding code duplication between different classes that would only differ through subtle variants of their methods.

Original: For this reason, classes are a construct of choice when engineering modular systems.
Revised: For this reason, classes are a construct of choice when engineering modular systems.

Original: A base class can deﬁne the generic behavior of a component as well as how it should interact with the external world, while leaving some methods to be reﬁned by its child classes, created for a speciﬁc purpose.
Revised: A base class can deﬁne the generic behavior of a component as well as how it should interact with the external world, while leaving some methods to be reﬁned by its child classes, created for a speciﬁc purpose.

Original: In the early age of object-oriented programming, it was not well understood how a class should behave if it had to inherit from several classes implementing a same method differently.
Revised: In the early age of object-oriented programming, it was not well understood how a class should behave if it had to inherit from several classes implementing a same method differently.

Original: For this reason, and because it proved to be sufﬁcient to design most systems, many languages only allowed single inheritance, and engineers learned to design systems that way.
Revised: For this reason, and because it proved to be sufﬁcient to design most systems, many languages only allowed single inheritance, and engineers learned to design systems that way.

Original: However, as illustrated in Figure 3, multiple inheritance largely increases object-oriented programming ﬂexibility.
Revised: However, as illustrated in Figure 3, multiple inheritance largely increases object-oriented programming ﬂexibility.

Original: It enables an object or class (here, D and E) to inherit characteristics and features from more than one parent (here B and C).
Revised: It enables an object or class (here, D and E) to inherit characteristics and features from more than one parent (here B and C).

Original: To address the ambiguity as to which parent a particular method (here f ) is inherited from when more than one parent class implements the said method, composition mechanisms have been proposed.
Revised: To address the ambiguity as to which parent a particular method (here f ) is inherited from when more than one parent class implements the said method, composition mechanisms have been proposed.

Original: In Python, the order of inheritance affects the class semantics, and the ambiguity is solved using the C3 linearization (or Method Resolution Order (MRO))
Revised: In Python, the order of inheritance affects the class semantics, and the ambiguity is solved using the C3 linearization (or Method Resolution Order (MRO))

Original: algorithm (Barrett et al., 1996).
Revised: algorithm (Barrett et al., 1996).

Original: This gives the language the opportunity to support multiple inheritance with consistent and well-understood behavior.
Revised: This gives the language the opportunity to support multiple inheritance with consistent and well-understood behavior.

Original: This solution to disambiguate multiple inheritance makes it possible to exploit mixins (Bracha &
Revised: This solution to disambiguate multiple inheritance makes it possible to exploit mixins (Bracha &

Original: Cook, 1990), which denote classes (e.g. B and C in Figure 3) that are deﬁned with the purpose of being combined with a base class.
Revised: Cook, 1990), which denote classes (e.g. B and C in Figure 3) that are deﬁned with the purpose of being combined with a base class.

Original: The combination of a mixin with a set of base classes results in a family of modiﬁed classes (like D and E in Figure 3).
Revised: The combination of a mixin with a set of base classes results in a family of modiﬁed classes (like D and E in Figure 3).

Original: It allows a programmer to share a functionalitybetween multiple classes without duplicating the same code in them. An example is illustrated in Figure 3 where a single mixin (B or C) is created to affect some methods (here f ) in a precise way without having to know exactly into which base class it will be combined. [Figure]
Revised: It allows a programmer to share a functionality

Original: [Figure]

Original: f(x)

Original: [Figure]
Revised: [Figure]

Original: A().f(x) # [Figure] [Figure] B().f(x
Revised: w/

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] *x) C().f(x [Figure]
Revised: maintenance issues.

Original: [Figure] *x) [Figure] [Figure]
Revised: In contrast, multiple inheritance enables the mixing of both features in a new class without requiring additional code, nor structurally complex code.

Original: [Figure] (x [Figure]
Revised: between multiple classes without duplicating the same code in them.

Original: [Figure] [Figure] *x)) [Figure]
Revised: An example is illustrated in

Original: multiple inheritance w/ multiple inheritance [Figure] issues. In contrast, multiple inheritance enables the mixing of both features in a new class without requiring additional code, nor structurally complex code.
Revised: Figure 3 where a single mixin (B or C) is created to affect some methods (here f ) in a precise way without having to know exactly into which base class it will be combined.

Original: When a class inherits from several mixins, the modiﬁcations induced by the mixins are composed according to the order in which the mixins are listed in the inheritance declaration.
Revised: When a class inherits from several mixins, the modiﬁcations induced by the mixins are composed according to the order in which the mixins are listed in the inheritance declaration.

Original: See Figure 3 for examples.
Revised: See Figure 3 for examples.

Original:  M ODULAR IMPLEMENTATION OF OUTPUT MATHEMATICS
Revised:  M ODULAR IMPLEMENTATION OF OUTPUT MATHEMATICS

Original: Our paper proposes a solution to manipulate the network output mathematics as a modular component.
Revised: Our paper proposes a solution to manipulate the network output mathematics as a modular component.

Original: Our solution aims at promoting reuse and incremental reﬁnement, across experiments, of the functions involved in the mathematical formulation of the task handled by the network.
Revised: Our solution aims at promoting reuse and incremental reﬁnement, across experiments, of the functions involved in the mathematical formulation of the task handled by the network.

Original: It is expected to support incremental and experimental development, but also to facilitate the exploration of various formulations of the optimization problem addressed by the training phase, e.g. based on different losses.
Revised: It is expected to support incremental and experimental development, but also to facilitate the exploration of various formulations of the optimization problem addressed by the training phase, e.g. based on different losses.

Original: Those alternative formulations (e.g. of the computer vision problem of interest) might even consider the prediction of alternative variables than the ones considered in the baseline reference formulation.
Revised: Those alternative formulations (e.g. of the computer vision problem of interest) might even consider the prediction of alternative variables than the ones considered in the baseline reference formulation.

Original: Our proposal should be considered as a reference baseline, to be challenged and reﬁned in future research.
Revised: Our proposal should be considered as a reference baseline, to be challenged and reﬁned in future research.

Original: It aims at enabling ﬂuent composition and replacement of OM methods, making it possible to share hands-on experiments, and to compare between distinct formulations of the relation  between the network prediction and the task at hand.
Revised: It aims at enabling ﬂuent composition and replacement of OM methods, making it possible to share hands-on experiments, and to compare between distinct formulations of the relation [Figure] between the network prediction and the task at hand.

Original: In practice, our solution consists of a number of implementation guidelines that can be summarized as follows.
Revised: In practice, our solution consists of a number of implementation guidelines that can be summarized as follows.

Original: [Figure]

Original: Our ﬁrst recommendation consists in assigning a speciﬁc class to each DL experiment, i.e. to each problem formulation, hosting as methods the mathematical functions implied in the computation of the task results and associated losses : the experiment’s output mathematics class.
Revised: Our ﬁrst recommendation consists in assigning a speciﬁc class to each DL experiment, i.e. to each problem formulation, hosting as methods the mathematical functions implied in the computation of the task results and associated losses : the experiment’s output mathematics class.

Original: To minimize code duplication, this class should inherit from mixins the behaviors that deviate from a baseline formulation.
Revised: To minimize code duplication, this class should inherit from mixins the behaviors that deviate from a baseline formulation.

Original: The baseline should thus be deﬁned to implement methods shared across multiple modiﬁed experiments.
Revised: The baseline should thus be deﬁned to implement methods shared across multiple modiﬁed experiments.

Original: This is illustrated in Figure 4 for the use case introduced in Figure 2, and further studied in Appendix B.
Revised: This is illustrated in Figure 4 for the use case introduced in Figure 2, and further studied in Appendix B.

Original: The second recommendation addresses the data accessibility.
Revised: The second recommendation addresses the data accessibility.

Original: To reduce the constraints and interdependencies induced by the inclusion of arguments in the signature of methods, the variables that are relevant to the output mathematics, including ground-truth, raw predictions, and high-level variables derived from the latter, should be part of the object instantiated from the output mathematics class and be made accessible through its methods with as few parameters as possible.
Revised: To reduce the constraints and interdependencies induced by the inclusion of arguments in the signature of methods, the variables that are relevant to the output mathematics, including ground-truth, raw predictions, and high-level variables derived from the latter, should be part of the object instantiated from the output mathematics class and be made accessible through its methods with as few parameters as possible.

Original: In that way, those variables become available to all the other methods of the object, without having to anticipate their usage within a list of arguments in the signature of each method.
Revised: In that way, those variables become available to all the other methods of the object, without having to anticipate their usage within a list of arguments in the signature of each method.

Original: This offers an increased ﬂexibility when  overriding a method.
Revised: This offers an increased ﬂexibility when reusing or overriding a method.

Original: This ﬂexibility is illustrated in Figure 5, which depicts an example where a method is overridden by the child class.
Revised: This ﬂexibility is illustrated in Figure 5, which depicts an example where a method is overridden by the child class.

Original: On the left side, each method receives a particular variable as parameter, which causes an issue when the child class method wants to use another variable than the one considered in the base class.
Revised: On the left side, each method receives a particular variable as parameter, which causes an issue when the child class method wants to use another variable than the one considered in the base class.

Original: In contrast, on the right side, all variables that are relevant to the studied problem are part of the class’ instance variables and are available through dedicated methods, without the need to be passed as arguments.
Revised: In contrast, on the right side, all variables that are relevant to the studied problem are part of the class’ instance variables and are available through dedicated methods, without the need to be passed as arguments.

Original: As a consequence, the child class method can be adapted whatever the variable required by its new behavior.
Revised: As a consequence, the child class method can be adapted whatever the variable required by its new behavior.

Original: In more detail, the goal here is to create a mixin replacing the predicted semantic segmentation by the ground-truth semantic segmentation.
Revised: In more detail, the goal here is to create a mixin replacing the predicted semantic segmentation by the ground-truth semantic segmentation.

Original: Such replacements can be useful for ablation studies for instance, to reevaluate a metric in case a sub-task is perfectly solved, or to evaluate whether the training of a particular sub-task perturbates that of others.
Revised: Such replacements can be useful for ablation studies for instance, to reevaluate a metric in case a sub-task is perfectly solved, or to evaluate whether the training of a particular sub-task perturbates that of others.

Original: In this situation, we can observe on the left side that the ground-truth segmentation cannot be returned by the OracleSegmentationMixin simply because the ground truth has not been routed as an argument of the inherited semantic pred method.
Revised: In this situation, we can observe on the left side that the ground-truth segmentation cannot be returned by the OracleSegmentationMixin simply because the ground truth has not been routed as an argument of the inherited semantic pred method.

Original: Indeed, based on the base class, external codeis only providing to the semantic pred method the semantic activations parameter, not the panoptic truth required by the semantic truth . It is not rare to encounter similar
Revised: Indeed, based on the base class, external code

Original: [Figure]
Revised: [Figure]

Original:  goal here is to create a mixin replacing the predicted semantic segmentation by the ground-truth semantic segmentation.
Revised: The goal here is to create a mixin replacing the predicted semantic segmentation by the ground-truth semantic segmentation.

Original: Reasons motivating such replacement are presented in the text.

Original: The semantic pred and semantic truth methods respectively receive the semantic activations and panoptic truth parameter.
Revised: The semantic pred and semantic truth methods respectively receive the semantic activations and panoptic truth parameter.

Original: Therefore, mixins cannot expect their methods to receive other parameters.
Revised: Therefore, mixins cannot expect their methods to receive other parameters.

Original: It makes it impossible for OracleSegmentationMixin to redeﬁne semantic pred so that it returns the semantic truth variable.
Revised: It makes it impossible for OracleSegmentationMixin to redeﬁne semantic pred so that it returns the semantic truth variable.

Original: (right)
Revised: (right)

Original: By making variables accessible through methods calls, passing variables as function arguments is not necessary anymore.
Revised: By making variables accessible through methods calls, passing variables as function arguments is not necessary anymore.

Original: This reduces the constraints imposed by the method signatures and renders the above-described modiﬁcation trivial.
Revised: This reduces the constraints imposed by the method signatures and renders the above-described modiﬁcation trivial.

Revised: is only providing to the semantic pred method the semantic activations parameter, not the panoptic truth required by the semantic truth .

Original:  situations in less trivial examples: predicting uncertainties, integrating new tasks in the losses, or deﬁning a variable as a function of the network output rather than as a constant derived from the dataset ground truth.
Revised: It is not rare to encounter similar situations in less trivial examples: predicting uncertainties, integrating new tasks in the losses, or deﬁning a variable as a function of the network output rather than as a constant derived from the dataset ground truth.

Original: This last case occurs for example when replacing the constant ground-truth instance center by the mean of the embeddings associated to the ground-truth instance (Newell et al., 2017; Neven et al., 2019).
Revised: This last case occurs for example when replacing the constant ground-truth instance center by the mean of the embeddings associated to the ground-truth instance (Newell et al., 2017; Neven et al., 2019).

Original: To avoid duplication of computation, we add a memoization mechanism that caches the result on the ﬁrst call and restitutes it on subsequent calls.
Revised: To avoid duplication of computation, we add a memoization mechanism that caches the result on the ﬁrst call and restitutes it on subsequent calls.

Original: Systematic caching means that any variable computed on a batch stays in memory for as long as that batch.
Revised: Systematic caching means that any variable computed on a batch stays in memory for as long as that batch.

Original: Though this may sound naive, it often makes sense in the context of deep model training since most intermediate results (among which neural network activations) have to be remembered for the backpropagation anyway.
Revised: Though this may sound naive, it often makes sense in the context of deep model training since most intermediate results (among which neural network activations) have to be remembered for the backpropagation anyway.

Original: Those activations alone usually outweigh by a large margin most of the byproducts required to compute the loss.
Revised: Those activations alone usually outweigh by a large margin most of the byproducts required to compute the loss.

Original: Furthermore, not caching a variable in the output mathematics may result in computing it many times, which potentially also leads the backpropagation library to store it multiple times, thereby increasing the memory-footprint.
Revised: Furthermore, not caching a variable in the output mathematics may result in computing it many times, which potentially also leads the backpropagation library to store it multiple times, thereby increasing the memory-footprint.

Original: It can however make sense to allow the programmer to introduce exceptions to this rule.
Revised: It can however make sense to allow the programmer to introduce exceptions to this rule.

Original: Such a mechanism has been implemented in the example code provided with this paper.
Revised: Such a mechanism has been implemented in the example code provided with this paper.

Original: (Authors, 2021)
Revised: (Istasse, 2021)

Original: Appendix B considers a representative example of a computer vision deep learning algorithm to challenge our assumptions about the limited memory and computational overhead induced by our proposed implementation paradigm.
Revised: Appendix B considers a representative example of a computer vision deep learning algorithm to challenge our assumptions about the limited memory and computational impact induced by our proposed implementation paradigm on training performance.

Original: Beyond showing that the overhead is neglectable (peak memory overhead ≤ 0 .

Original: 4% , and no computational overhead), the work reported in this Appendix ﬁrst reveals that our proposed recommendations for modular output mathematics can be naturally implemented in the reference Panoptic-Deeplab method, with minimal changes to the reference code.
Revised: Beyond showing that the overhead is far from prohibitive, the work reported in this Appendix ﬁrst reveals that our proposed recommendations for modular output mathematics can be naturally implemented in the reference Panoptic-Deeplab method, with minimal changes to the reference code.

Original: Second, the modular output mathematics implementation is leveraged to prototype a variation of the instance segmentation problem formulation, incorporating learnable instance centers in the instance prediction process.
Revised: Second, the modular output mathematics implementation is leveraged to prototype a variation of the instance segmentation problem formulation, incorporating learnable instance centers in the instance prediction process.

Original: This work reveals that, thanks to its modularity, the output mathematics component is straightforward to adapt to the new formulation of the problem.
Revised: This work reveals that, thanks to its modularity, the output mathematics component is straightforward to adapt to the new formulation of the problem.

Original:  D ISSEMINATION OF OUTPUT MATHEMATICS
Revised:  D ISSEMINATION OF OUTPUT MATHEMATICS

Original: Figure 7 illustrates how the modular organization of the output mathematics proposed in this paper helps in disseminating deep learning experiments, and favor incremental development and fair comparisons of methods.
Revised: Figure 7 illustrates how the modular organization of the output mathematics proposed in this paper helps in disseminating deep learning experiments, and favors incremental development and fair comparisons of methods.

Original: This ﬁgure provides an example of how to document the reference software released with a deep learning experiment, by deﬁning explicitly the link between the mathematical formulas introduced in the paper associated to the DL experiment, and the variables and methods manipulated by the reference software.
Revised: This ﬁgure provides an example of how to document the reference software released with a deep learning experiment, by deﬁning explicitly the link between the mathematical formulas introduced in the paper associated to the DL experiment, and the variables and methods manipulated by the reference software.

Original: It presents how the variables and the mathematical formulas are (i) grouped into classes and (ii) deﬁned as class methods in the code.
Revised: It presents how the variables and the mathematical formulas are (i) grouped into classes and (ii) deﬁned as class methods in the code.

Original: In this list of correspondences, other researchers will ﬁnd the information required to access and modify particular variables and methods.
Revised: In this list of correspondences, other researchers will ﬁnd the information required to access and modify particular variables and methods.

Original: Upon incremental development, researchers just have to prepare and disseminate a table gathering the classes and methods that have been modiﬁed compared to the baseline.
Revised: Upon incremental development, researchers just have to prepare and disseminate a table gathering the classes and methods that have been modiﬁed compared to the baseline.

Original: This is exempliﬁed in Figure 7 for the Panoptic-Deeplab method with learnable centers.
Revised: This is exempliﬁed in Figure 7 for the Panoptic-Deeplab method with learnable centers.

Original: Note that since variables are accessed without resorting to constraining function signatures, data are arbitrarily available, making it easy to create unforeseen alternative formulations of the methods, without the need to describe complex interdependencies related to the data ﬂow.
Revised: Note that since variables are accessed without resorting to constraining function signatures, data are arbitrarily available, making it easy to create unforeseen alternative formulations of the methods, without the need to describe complex interdependencies related to the data ﬂow.

Original: Furthermore, modularity enables creating banks of plug-and-play variants for a given baseline, in a way analogous to what is possible with neural architectures, optimizers, or data augmentation.
Revised: Furthermore, modularity enables creating banks of plug-and-play variants for a given baseline, in a way analogous to what is possible with neural architectures, optimizers, or data augmentation.

Original: Finally, it also becomes possible to extract the output mathematics of a baseline (and therefore the complete set of variants built on it) from one project and inject them in another as only some key inputs and outputs (mostly the task variables in Figure 7) need to be interfaced with external code.
Revised: Finally, it also becomes possible to extract the output mathematics of a baseline (and therefore the complete set of variants built on it) from one project and inject them in another as only some key inputs and outputs (mostly the task variables in Figure 7) need to be interfaced with external code.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper has proposed to group the functions implementing the output mathematics (OM) inside a modular component, offering increased ﬂexibility when designing deep learning experiments, by allowing easy substitution and modiﬁcation of the mathematical functions associated to the problem formulation handled by the neural network.
Revised: This paper has proposed to group the functions implementing the output mathematics (OM) inside a modular component, offering increased ﬂexibility when designing deep learning experiments, by allowing easy substitution and modiﬁcation of the mathematical functions associated to the problem formulation handled by the neural network.

Original: Our paper focuses on the OM because this part of the code is characterized by a potentially large number of interdependent functions, dealing both with network- and task-related data.
Revised: Our paper focuses on the OM because this part of the code is characterized by a potentially large number of interdependent functions, dealing both with network- and task-related data.

Original: This is in contrast to most other DL components are related either to the network (e.g. optimizer, scheduler, or network layers) or to the task (e.g. data augmentation or task-related metrics and visualization routines), and are characterized by well-deﬁned interfaces that make their functional abstraction relatively straightforward.
Revised: This is in contrast to most other DL components are related either to the network (e.g. optimizer, scheduler, or network layers) or to the task (e.g. data augmentation or task-related metrics and visualization routines), and are characterized by well-deﬁned interfaces that make their functional abstraction relatively straightforward.

Original: Our modular implementation of the output mathematics builds on two main recommendations that can be summarized as follows.
Revised: Our modular implementation of the output mathematics builds on two main recommendations that can be summarized as follows.

Original: First, by implementing the output mathematics as methods associated to a set of base and mixin classes, various problem formulations can be designed in a modular way by leveraging the multiple inheritance paradigm.
Revised: First, by implementing the output mathematics as methods associated to a set of base and mixin classes, various problem formulations can be designed in a modular way by leveraging the multiple inheritance paradigm.

Original: Second, by exposing the mathematical variables involved in the problem formulation through methods with only necessary parameters, those variables become available to all other methods, relaxing the constraints and dependencies imposed by method signatures over the code and the data ﬂow.
Revised: Second, by exposing the mathematical variables involved in the problem formulation through methods with only necessary parameters, those variables become available to all other methods, relaxing the constraints and dependencies imposed by method signatures over the code and the data ﬂow.

Original: Application of those guidelines to the Panoptic-Deeplab method has revealed that our proposed modular implementation is easy to integrate in an existing experimental code base, without penalizing its performance.
Revised: Application of those guidelines to the Panoptic-Deeplab method has revealed that our proposed modular implementation is easy to integrate in an existing experimental code base, without penalizing its performance.

Original: The ﬂexibility of the proposed approach has also been demonstrated by showing how easy it is to change the problem formulation, the deﬁnition of associated variables and the data ﬂow.
Revised: The ﬂexibility of the proposed approach has also been demonstrated by showing how easy it is to change the problem formulation, the deﬁnition of associated variables and the data ﬂow.

Original: We believe this work provides a convenient way for practitioners to run the many different experiments involved in the development of DL models.
Revised: We believe this work provides a convenient way for practitioners to run the many different experiments involved in the development of DL models.

Original: It may also facilitate the comparison of similar formulations.
Revised: It may also facilitate the comparison of similar formulations.

Original: Moreover, because it gives access to new formulations from a limited amount of code, it opens perspectives regarding the automatic exploration and optimization of the output mathematics, as is done for some other parts of the DL experiments such as the data-augmentation pipeline and the neural architecture (Cubuk et al., 2019; Jin et al., 2019).
Revised: Moreover, because it gives access to new formulations from a limited amount of code, it opens perspectives regarding the automatic exploration and optimization of the output mathematics, as is done for some other parts of the DL experiments such as the data-augmentation pipeline and the neural architecture (Cubuk et al., 2019; Jin et al., 2019).

Original: A CKNOWLEDGMENTS AND ACCESSIBILITY STATEMENT
Revised: A CCESSIBILITY STATEMENT

Original: The ideas presented in this paper aim at more accessible research, by making it easier to build on an existing model and to compare different methods on the same ground.
Revised: The ideas presented in this paper aim at more accessible research, by making it easier to build on an existing model and to compare different methods on the same ground.

Original: The authors hope that the guidelines proposed to write and disseminate the code related to deep learning experiments will lead to widespread solutions, e.g. that they will get adapted to other languages, or that mainstream libraries will adopt similar constructs.
Revised: The authors hope that the guidelines proposed to write and disseminate the code related to deep learning experiments will lead to widespread solutions, e g. that they will get adapted to other languages, or that mainstream libraries will adopt similar constructs.

Original: The authors will be glad to collaborate to make it happen.
Revised: The authors will be glad to collaborate to make it happen.

Original: The authors also commit to making the content of this publication accessible to the widest possible audience.
Revised: The authors are willing to make the content of this publication accessible to the widest possible audience on which it could have an impact.

Original: Our example source code (Authors, 2021) isplanned to be released along withthe cameraready version, along with a comprehensive readme and, probably delayed, a blog post.
Revised: Our example source code (Istasse, 2021) is released along with a comprehensive readme .

Original: For any request orsuggestion, please do not hesitate to contact the authors by e-mail or byopening an issue on the publication Github repository (Authors, 2021).
Revised: For help requests or suggestions, feel free to open an issue on the GitHub repository or to contact the authors by e-mail A CKNOWLEDGEMENTS This research is supported by the DeepSport project of the Walloon Region, Belgium. C. De

Original:  A D ATAFLOW GRAPH CONSTRUCTION
Revised: Vleeschouwer is a Research Director of the Fonds de la Recherche Scientiﬁque - FNRS. Computational resources have been provided by the supercomputing facilities of the Universit´e catholique de Louvain (CISM/UCL) and the Consortium des ´ Equipements de Calcul Intensif en F ´ ed ´ eration Wallonie Bruxelles (C´ECI) funded by the Fond de la Recherche Scientiﬁque de Belgique (F.R.S.-FNRS) under convention 2.5020.11 and by the Walloon Region. A D ATAFLOW GRAPH CONSTRUCTION

Original: By including the variables of interest in the  class, the external code  does not have to care about their routing.
Revised: By including the variables of interest in the OM class, the rest of the experiment code base does not have to care about their routing.

Original: The output mathematics component becomes self-sufﬁcient for transforming provided inputs into all the products that can be derived from them.
Revised: The output mathematics component becomes self-sufﬁcient for transforming provided inputs into all the products that can be derived from them.

Original: All those transformations and associated routing of data, forming a dataﬂow graph, are encoded in a class, deﬁned from a base class and mixins.
Revised: All those transformations and associated routing of data are equivalent to a dataﬂow graph which is encoded in a class, deﬁned from a base class and mixins.

Original: To apply those transformations to actual inputs, the class is instantiated based on the tensors associated to those inputs (and forming the raw batch ), and the value of a variable of interest can be obtained by calling the instance method associated with it.
Revised: To apply those transformations to actual inputs, the class is instantiated based on the tensors associated to those inputs (and forming the raw batch ) and the value of a variable of interest can be obtained by calling the instance method associated with it.

Original: Figure 6 illustrates the dataﬂow graph that is inferred from the code on the right side of Figure 5.
Revised: Figure 6 illustrates the dataﬂow graph that is inferred from the code on the right side of Figure 5.

Original: It can be derived from the  output mathematics class because no external code intervenes in the ﬂow ofdata (external code just has to feed the graph with appropriate tensors). Hence, all the computations taking place on data to compute any variable are determined by the class code. The same cannot be said for the  functions requiring variables to be passed as arguments.
Revised: Without the modular output mathematics component, other components of the experiment that are external to the OM have to call the OM functions with variables to be passed as arguments.

Original: Those variables are routed by external code and it could transform them, which prevents the inference of a similar dataﬂow graph from the functions alone.
Revised: Therefore, variables are routed by external code that could transform them, which prevents the inference of a similar dataﬂow graph from the OM code alone.

Original: Moreover, because of the careful routing that has to be maintained when variables are passed as arguments, it is costly to ensure that a same variable is never computed twice at different places. 
Revised: Moreover, because of the careful routing that has to be maintained when variables are passed as arguments, it is costly to ensure that a same variable is never computed twice at different places. In

Original: In contrast, handling variables through method calls makes it easy to force any variable to be computed at most once.
Revised: [Figure] contrast, handling variables through method calls makes it easy to force any variable to be computed at most once.

Original: Each method can indeed be adapted in order to cache its result on a ﬁrst call in an instance attribute, and to retrieve the cached result on subsequent calls with identical parameters.
Revised: Each method can indeed be adapted in order to cache its result on a ﬁrst call in an instance attribute, and to retrieve the cached result on subsequent calls with identical parameters.

Original: In case a variable is not required, because the variable is not used in the current variant or mode ( e.g. training or prediction), the method is simply never called, and resources are not wasted to compute it.
Revised: In case a variable is not required, because the variable is not used in the current variant or mode ( e.g. training or prediction), the method is simply never called, and resources are not wasted to compute it.

Original: One improvement over the recommended systematic caching policy would be to release each variable when it is used for the last time, or to recompute 
Revised: One improvement over the recommended systematic caching policy would be to release each variable when it is used for the last time, or to recompute terms that are not required by backpropagation.

Original: [Figure] Those conditions are currently difﬁcult to detect given the interpreted nature of Python, but when the whole code having access to the output mathematics component is known, those problems can be addressed through dataﬂow analysis and optimization.
Revised: Those conditions are currently difﬁcult to detect given the interpreted nature of Python, but when the whole code having access to the output mathematics component is known, those problems can be addressed through dataﬂow analysis and optimization.

Original: In addition to those concerns related to potential memory overhead, caching mechanisms and calls through an entire hierarchy of classes introduce extra instructions to be performed by the CPU.
Revised: In addition to those concerns related to potential memory overhead, caching mechanisms and calls through an entire hierarchy of classes introduce extra instructions to be performed by the CPU.

Original: However, in practical cases, the mathematical computations required by the deep learning algorithm are much more expensive than those mechanisms, so that the overall time overhead of the latter is shown to be negligible 2 (see experiments in Appendix B).
Revised: However, in practical cases, the mathematical computations required by the deep learning algorithm are much more expensive than those mechanisms, so that the overall time overhead of the latter is shown to be negligible 2 (see experiments in Appendix B).

Original: B A SSESSMENT ON THE P ANOPTIC -D EEPLAB METHOD
Revised: B A SSESSMENT ON THE P ANOPTIC -D EEPLAB METHOD

Original: Panoptic-Deeplab is a pixel-wise panoptic segmentation approach that, although surprisingly simple, achieves state of the art performance.
Revised: Panoptic-Deeplab is a pixel-wise panoptic segmentation approach that, although surprisingly simple, achieves state of the art performance.

Original: This use case has been chosen because it provides a popular and representative deep learning solution to a fundamental computer vision problem.
Revised: This use case has been chosen because it provides a popular and representative deep learning solution to a fundamental computer vision problem.

Original: The integration of our modular component to this case will be released in the form of a complete repository in (Authors, 2021), along with the cameraready version.
Revised: The integration of our modular component to this case will be released in the form of a complete repository in (Istasse, 2021), along with the cameraready version.

Original: It ﬁrst reveals that an arbitrary code can conveniently be adapted to host our proposed  modular and composable implementation of the output mathematics, with limited resource overhead.
Revised: It ﬁrst reveals that an arbitrary code can conveniently be adapted to host our proposed [Figure] modular and composable implementation of the output mathematics, with limited resource overhead.

Original: It also illustrates how the ﬂexibility offered by our solution facilitates the communication about the incremental development of an alternative formulation of the instance segmentation problem.
Revised: It also illustrates how the ﬂexibility offered by our solution facilitates the communication about the incremental development of an alternative formulation of the instance segmentation problem.

Original: To demonstrate the feasibility and the advantages of our proposal, we proceed in two steps.
Revised: To demonstrate the feasibility and the advantages of our proposal, we proceed in two steps.

Original: First, we apply our recommendations for modular output mathematics on the public Panoptic-Deeplab reimplementation.
Revised: First, we apply our recommendations for modular output mathematics on the public Panoptic-Deeplab reimplementation.

Original: We observe that this can be done easily, with minimal changes to the reference code.
Revised: We observe that this can be done easily, with minimal changes to the reference code.

Original: Second, we exploit our modular output mathematics to prototype a variation of the instance segmentation formulation, incorporating learnable instance centers in the instance prediction process.
Revised: Second, we exploit our modular output mathematics to prototype a variation of the instance segmentation formulation, incorporating learnable instance centers in the instance prediction process.

Original: [Figure]

Original: This work reveals that, thanks to its modularity, the output mathematics component is straightforward to adapt to the new formulation of the problem.
Revised: This work reveals that, thanks to its modularity, the output mathematics component is straightforward to adapt to the new formulation of the problem.

Original: Figure 7 illustrates how we have chosen to formulate the Panoptic-Deeplab method using modular output mathematics and to extend them for the learnable centers.
Revised: Figure 7 illustrates how we have chosen to formulate the Panoptic-Deeplab method using modular output mathematics and to extend them for the learnable centers.

Original: Time and memory overheads, compared to the base implementation, are estimated for experiments running our modular output mathematics component, with and without the added learnable centers functionality.
Revised: Time and memory overheads, compared to the base implementation, are estimated for experiments running our modular output mathematics component, with and without the added learnable centers functionality.

Original: B. M ODULAR OUTPUT MATHEMATICS
Revised: B. M ODULAR OUTPUT MATHEMATICS

Original: Integrating our recommendations for a modular implementation of the output mathematics into the public Panoptic-Deeplab implementation (Cheng et al., 2020) is fairly straightforward.
Revised: Integrating our recommendations for a modular implementation of the output mathematics into the public Panoptic-Deeplab implementation (Cheng et al., 2020) is fairly straightforward.

Original: Details about our speciﬁc implementation choices can be found in the code (Authors, 2021).
Revised: Details about our speciﬁc implementation choices can be found in the code (Istasse, 2021).

Original: Here, we survey the main steps involved during this development.
Revised: Here, we survey the main steps involved during this development.

Original: The ﬁrst step consists in implementing the four upper classes illustrated in Figure 7, associating a class method to every mathematical concept present in the initial code.
Revised: The ﬁrst step consists in implementing the four upper classes illustrated in Figure 7, associating a class method to every mathematical concept present in the initial code.

Original: Sometimes, a same concept can be represented in different ways and it is difﬁcult to come up with different names.
Revised: Sometimes, a same concept can be represented in different ways and it is difﬁcult to come up with different names.

Original: In such cases, it can be practical to handle the different representations of a concept in a method with a “representation” parameter.
Revised: In such cases, it can be practical to handle the different representations of a concept in a method with a “representation” parameter.

Original: For instance, the semantic maps are predicted by the network as unbounded activations maps, from which probabilities can be obtained via the application of the softmax function, and the predicted class through the argmax function.
Revised: For instance, the semantic maps are predicted by the network as unbounded activations maps, from which probabilities can be obtained via the application of the softmax function, and the predicted class through the argmax function.

Original: We use the representation parameter to refer to them as semantic pred(‘logits’) , semantic pred(‘softmax’) and semantic pred() respectively.
Revised: We use the representation parameter to refer to them as semantic pred(‘logits’) , semantic pred(‘softmax’) and semantic pred(‘argmax’) respectively.

Original: The output mathematics class is in our case an anonymous class created once and for all at the beginning of the execution, inheriting from PanDLBase , itself inheriting from the four aforementioned classes, and the eventual mixins speciﬁed by the practitioner in the experiment conﬁguration.
Revised: The output mathematics class is in our case an anonymous class created once and for all at the beginning of the execution, inheriting from PanDLBase , itself inheriting from the four aforementioned classes, and the eventual mixins speciﬁed by the practitioner in the experiment conﬁguration.

Original: Each raw batch is used as a parameter to the constructor in charge of instantiating the class to create an instance speciﬁc to that set of data.
Revised: Each raw batch is used as a parameter to the constructor in charge of instantiating the class to create an instance speciﬁc to that set of data.

Original: Each obtained instance can then be forwarded to the parts of the code requiring access to tensors, e.g. to perform the optimization step, to compute and log metrics, or to draw visualizations.
Revised: Each obtained instance can then be forwarded to the parts of the code requiring access to tensors, e.g. to perform the optimization step, to compute and log metrics, or to draw visualizations.

Original: In our case, we reused the argument dedicated to the raw batch.
Revised: In our case, we reused the argument dedicated to the raw batch.

Original: Finally, data computations in those parts of the code only need to be replaced by calls to the adequate methods of our output mathematics instance.
Revised: Finally, data computations in those parts of the code only need to be replaced by calls to the adequate methods of our output mathematics instance.

Original: This way, those components that refer to the output variables do not contain code that is speciﬁc to Panoptic-Deeplab anymore.
Revised: This way, those components that refer to the output variables do not contain code that is speciﬁc to Panoptic-Deeplab anymore.

Original: Hence, they become compatible with alternative implementations of the output mathematics.
Revised: Hence, they become compatible with alternative implementations of the output mathematics.

Original: To handle the caching, we make the PanDLBase class also inherits from the OutputMath class that implements the default memoization policy described in Section 3 while providing ways to specify exceptions to it.
Revised: To handle the caching, we make the PanDLBase class also inherits from the OutputMath class that implements the default memoization policy described in Section 3 while providing ways to specify exceptions to it.

Original: B. I MPLEMENTING THE L EARNABLE C ENTERS MIXIN
Revised: B. I MPLEMENTING THE L EARNABLE C ENTERS MIXIN

Original: Formulating the instance segmentation problem based on learnable instance centers implies that the network is not explicitly supervised to predict centroids at the center of instances anymore.
Revised: Formulating the instance segmentation problem based on learnable instance centers implies that the network is not explicitly supervised to predict centroids at the center of instances anymore.

Original: Instead, each ground-truth instance center location depends on the output of the network, and get positioned at the center of mass of the locations p + Φ ( p ) pointed pixels of that instance.
Revised: Instead, each ground-truth instance center location depends on the output of the network, and get positioned at the center of mass of the locations p + Φ ( p ) pointed pixels of that instance.

Original: This way, the network is given the freedom to redistribute instance centers spatially, which may help to solve clustering ambiguities when some instances have centers that are close to each other.
Revised: This way, the network is given the freedom to redistribute instance centers spatially, which may help to solve clustering ambiguities when some instances have centers that are close to each other.

Original: This idea is common in the literature (Newell et al., 2017; Neven et al., 2019).
Revised: This idea is common in the literature (Newell et al., 2017; Neven et al., 2019).

Original: Were the ground-truth offsets and center heatmap computations implemented conventionally, adjusting them would have required heavy rearranging of the code and its data ﬂow.
Revised: Were the ground-truth offsets and center heatmap computations implemented conventionally, adjusting them would have required heavy rearranging of the code and its data ﬂow.

Original: In the public
Revised: In the public

Original: Panoptic-Deeplab reimplementation for instance, those targets are computed on the CPU before sending data to the GPU and running the neural network, and therefore have no access to the latter’s predictions.
Revised: Panoptic-Deeplab reimplementation for instance, those targets are computed on the CPU before sending data to the GPU and running the neural network, and therefore have no access to the latter’s predictions.

Original: In our case, it appears, as suggested in Figure 7, that only overriding the method associated with the c ∗ i variable is required.
Revised: In our case, it appears, as suggested in Figure 7, that only overriding the method associated with the c ∗ i variable is required.

Original: For better centroid separation however, we also choose to adapt the offset loss with a term that penalizes pixels pointing to locations p + Φ ( p ) lending closer to other centroids than that of their ground-truth instance.
Revised: For better centroid separation however, we also choose to adapt the offset loss with a term that penalizes pixels pointing to locations p + Φ ( p ) lending closer to other centroids than that of their ground-truth instance.

Original: In the next section, we assess that the overhead introduced by our modular output mathematics is not prohibitive with respect to their beneﬁts.
Revised: In the next section, we assess that the overhead introduced by our modular output mathematics is not prohibitive with respect to their beneﬁts.

Original: B. O VERHEAD INDUCED BY THE MODULAR FRAMEWORK
Revised: B. O VERHEAD INDUCED BY THE MODULAR FRAMEWORK

Original: In contrast to what is illustrated in Figure 2, in the base implementation, neither the ground-truth instance IDs nor the panoptic IDs were not part of the raw batch , rendering the computation of ground-truth instance masks tedious.

Original: Indeed, with this constraint, instance masks had to be computed by clustering of the pixels from the ground-truth centroids and ground-truth offsets.

Original: We take it as an opportunity to evaluate the cost of having a pathological representation in the raw batch fed to the output mathematics, and compare it to forwarding the instance IDs directly, which is a more adequate representation for ﬂexible and generic code.

Original: Four experiments have been run and compared. (i) The original Panoptic-Deeplab training loop (ii) An equivalent training, but implemented based ona modular implementation of theoutput mathematics (iii)
Revised: Four experiments have been run and compared: (i) the original Panoptic-Deeplab training loop, where the target maps (i.e. ground-truth centers, offsets and semantic labels) are computed on the

Original: A training withthe learnable centers feature integrated by adapting our proposed modular implementation when the raw batch transmitted to the GPUs has been modiﬁed to contain groundtruth instance IDs. (iv) Same as in (iii), but with instance IDs computed on the GPU, from the
Revised: CPU; (ii) an equivalent training, with our proposed modular output mathematics but accessing the targets maps that are still computed on the CPU; (iii) a training with target maps being computed in

Original: [Table]
Revised: [Table]

Original: DeepLab raw batch, which includes the ground-truth centers and offsets but not the instance IDs
Revised: the OM component, on the GPU; and (iv) a training with the learnable centers mixin replacing the ﬁxed centers in the OM. In each of those

Original: [Table]
Revised: [Table]

Original: In each of those
Revised: we track the duration of forward pass, backward pass, both happening on the GPU, and the overall time of a training batch

Original: [Table]
Revised: [Table]

Original: we are mostly interested in thetime spent doing computations during the forward pass, the backward pass and the maximal GPU memory allocated during the processing of a batch.
Revised: which includes computing the targets in CPU threads when needed We are also interested in the maximal GPU memory allocated, since it generally constrains the batch size.

Original: The timings do not take the initial transfer of the raw batch from the CPU  to  the GPU into account.
Revised: We perform those experiments on a machine with one NVidia V100 GPU, a batch size of 4, and dedicate 4 CPU cores to data loading (of an Intel Xeon Gold 5217). All the other parameters come from the MobileNetV2 conﬁguration of Panoptic-Deeplab. The results are averaged over 100 consecutive batches.

Original: Table 1 reports each metric for all experiments.
Revised: Table 1 reports each metric for all experiments.

Revised: The only difference between (i) and (ii) is the integration of the OM component.

Original: The base formulation, with and without the modular output mathematics, exhibit very similar timings and peak memory usage. This conﬁrms that the overhead introduced by the modular implementation in terms of cache handling and call dispatching is negligible compared to the computations taking place.
Revised: A signiﬁcant amount of computations still happen outside the OM because the target maps are computed by the data-loader CPU threads. Both rows exhibit very similar timings and peak memory usage, which conﬁrms that the overhead introduced by the component, its calls and caching mechanisms, is negligible.

Original: The learnable centers require iterating over the instances to generate ground-truth offset and center tensors, as well as the instance masks.
Revised: Implementation (iii) moves the computation of targets from data-loading happening in CPU threads to the OM component. The overall batch processing time is heavily reduced compared to (ii), because less data is required to be transferred between CPU and GPU, and because the GPU is a more efﬁcient device to perform those operations, despite the many loops that have to take place. Would it not be the case, it would still be possible for the programmer to specify that some computations inside the OM have to take place on the CPU.

Original: Those justify the increaseobserved between (ii) to (iii) in terms of forward pass time and memory consumption. In the fourth line of Table 1, we observe that feeding it only with offsets and centers, as done by the conventional Panoptic-Deeplab implementation, requires tocompute the instance
Revised: We emphasize that, as suggested in Section 3, despite the systematic caching of intermediate results taking place, the increase in peak memory is relatively low. Finally, (iv) adds the learnable centers on top of (iii). The overhead of this particular variant is signiﬁcant because it fundamentally impacts the mathematical system, using network predictions for creating targets, hence creating new operations contributing to the gradients.

Original: ID on the GPU which severely penalize memory usage. Due to  the excessive amount of  memory required to achieve it in extreme cases, we have to handle those on the CPU, which impacts in its turn time efﬁciency.
Revised: In addition, the computation of the new embedding loss sometimes has to occur on the CPU because computing the distance between the embedding of each pixel and each ground-truth instance centroid requires more memory than available on the GPU.

Original: It reveals the importance of thinking about the base representations provided through the raw batch, in spite of the fact the  modular output mathematics allow conversions to happen transparently.
Revised: Yet, all-in-all, the modularity of our output mathematics allows us to implement it in a way that it is still on average more than three times faster than the public Panoptic-Deeplab implementation. We see the fact that all this complexity could be added seamlessly, contained in a class deﬁnition, as an example that modular OM are effective to broaden the possibilities of existing code bases in a practical fashion.