Original: O
Revised: O

Original: NE -W AY P ROTOTYPICAL N ETWORKS
Revised: NE -W AY P ROTOTYPICAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Few-shot models have become a popular topic of research in the past years.
Revised: Few-shot models have become a popular topic of research in the past years.

Original: They offer the possibility to determine class belongings for unseen examples using just a handful of examples for each class.
Revised: They offer the possibility to determine class belongings for unseen examples using just a handful of examples for each class.

Original: Such models are trained on a wide range of classes and their respective examples, learning a decision metric in the process.
Revised: Such models are trained on a wide range of classes and their respective examples, learning a decision metric in the process.

Original: Types of few-shot models include matching networks and prototypical networks.
Revised: Types of few-shot models include matching networks and prototypical networks.

Original: We show a new way of training prototypical few-shot models for just a single class.
Revised: We show a new way of training prototypical few-shot models for just a single class.

Original: These models have the ability to predict the likelihood of an unseen query belonging to a group of examples without any given counterexamples.
Revised: These models have the ability to predict the likelihood of an unseen query belonging to a group of examples without any given counterexamples.

Original: The difﬁculty here lies in the fact that no relative distance to other classes can be calculated via softmax.
Revised: The difﬁculty here lies in the fact that no relative distance to other classes can be calculated via softmax.

Original: We solve this problem by introducing a “null class” centered around zero, and enforcing centering with batch normalization.
Revised: We solve this problem by introducing a “null class” centered around zero, and enforcing centering with batch normalization.

Original: Trained on the commonly used Omniglot data set, we obtain a classiﬁcation accuracy of .98 on the matched test set, and of .8 on unmatched MNIST data.
Revised: Trained on the commonly used Omniglot data set, we obtain a classiﬁcation accuracy of .98 on the matched test set, and of .8 on unmatched MNIST data.

Original: On the more complex MiniImageNet data set, test accuracy is .8.
Revised: On the more complex MiniImageNet data set, test accuracy is .8.

Original: In addition, we propose a novel Gaussian layer for distance calculation in a prototypical network, which takes the support examples’ distribution rather than just their centroid into account.
Revised: In addition, we propose a novel Gaussian layer for distance calculation in a prototypical network, which takes the support examples’ distribution rather than just their centroid into account.

Original: This extension shows promising results when a higher number of support examples is available.
Revised: This extension shows promising results when a higher number of support examples is available.

Original:  M OTIVATION
Revised:  M OTIVATION

Original: One- or few-shot learning is an active area of research that focuses on modeling semantic concepts with very little training data.
Revised: One- or few-shot learning is an active area of research that focuses on modeling semantic concepts with very little training data.

Original: Such models learn information about the general task of comparing queries to one or several support examples of the requested classes, which can then be applied to unseen classes.
Revised: Such models learn information about the general task of comparing queries to one or several support examples of the requested classes, which can then be applied to unseen classes.

Original: A large amount of data is still necessary for training, but this data does not need to belong exactly to the domains that may become interesting in the future, but rather to a variety of similar problems.
Revised: A large amount of data is still necessary for training, but this data does not need to belong exactly to the domains that may become interesting in the future, but rather to a variety of similar problems.

Original: This notion is inspired by human learning; humans can usually generalize to a new concept from just one or very few examples, using their prior knowledge of the world and related concepts.
Revised: This notion is inspired by human learning; humans can usually generalize to a new concept from just one or very few examples, using their prior knowledge of the world and related concepts.

Original: A special case occurs when we do not need to distinguish between multiple concepts, but rather want to enable models to recognize a single new concept.
Revised: A special case occurs when we do not need to distinguish between multiple concepts, but rather want to enable models to recognize a single new concept.

Original: In many practical scenarios, we will often be interested in just one novel class, rather than multiple at once.
Revised: In many practical scenarios, we will often be interested in just one novel class, rather than multiple at once.

Original: Application examples include detecting news or social media texts about a certain events; detecting one type of unseen object (e.g. a face) in a dataset; veriﬁcation of unseen ﬁngerprints; or outlier detection within an unseen class.
Revised: Application examples include detecting news or social media texts about a certain events; detecting one type of unseen object (e.g. a face) in a dataset; veriﬁcation of unseen ﬁngerprints; or outlier detection within an unseen class.

Original: In existing few-shot methods, this is commonly translated into a two-class problem with the classes “belonging to the concept” and “not belonging to the concept”.
Revised: In existing few-shot methods, this is commonly translated into a two-class problem with the classes “belonging to the concept” and “not belonging to the concept”.

Original: As these two classes have very different types of feature distributions, this is not the most elegant solution and poses problems, especially with regards to training data selection.
Revised: As these two classes have very different types of feature distributions, this is not the most elegant solution and poses problems, especially with regards to training data selection.

Original: It also requires effort for modeling a class (the negative one) that is not of interest.
Revised: It also requires effort for modeling a class (the negative one) that is not of interest.

Original: In this paper, we present an addition to few-shot learning approaches that enables generalization to a single class in order to determine whether queries belong to this (previously unseen) concept or not.
Revised: In this paper, we present an addition to few-shot learning approaches that enables generalization to a single class in order to determine whether queries belong to this (previously unseen) concept or not.

Original: This is implemented by introducing a “null” (or “garbage”) class which does not need to trained to the prototypical network approach.
Revised: This is implemented by introducing a “null” (or “garbage”) class which does not need to trained to the prototypical network approach.

Original: Application of this novel model type is demonstrated on the commonly used Omniglot and MiniImageNet data sets.
Revised: Application of this novel model type is demonstrated on the commonly used Omniglot and MiniImageNet data sets.

Original:  R ELATED WORK 2. O NE - CLASS CLASSIFICATION
Revised:  R ELATED WORK 2. O NE - CLASS CLASSIFICATION

Original: The problem of one-class classiﬁcation has been a topic of research for decades.
Revised: The problem of one-class classiﬁcation has been a topic of research for decades.

Original: It has also been interpreted as outlier detection, anomaly detection, novelty detection, and concept learning.
Revised: It has also been interpreted as outlier detection, anomaly detection, novelty detection, and concept learning.

Original: Comprehensive overviews of “traditional” methods for solving this task are given in Khan & Madden (2014) and Chandola et al.
Revised: Comprehensive overviews of “traditional” methods for solving this task are given in Khan & Madden (2014) and Chandola et al.

Original: Chandola et al.
Revised: Chandola et al.

Original: (2009) identify several different fundamental techniques, e.g. statistics, nearest-neighbour methods, clustering, and classiﬁcation.
Revised: (2009) identify several different fundamental techniques, e.g. statistics, nearest-neighbour methods, clustering, and classiﬁcation.

Original: Strategies can be derived from research areas such as data mining, information theory, and machine learning.
Revised: Strategies can be derived from research areas such as data mining, information theory, and machine learning.

Original: Within the machine learning domain, a common solution relies on One-class Support Vector Machines (OSVMs)
Revised: Within the machine learning domain, a common solution relies on One-class Support Vector Machines (OSVMs)

Original: [Sch¨olkopf et al.
Revised: [Sch¨olkopf et al.

Original: (1999); Tax & Duin (1999)], although earlier-stage neural networks were also utilized [Hawkins et al.
Revised: (1999); Tax & Duin (1999)], although earlier-stage neural networks were also utilized [Hawkins et al.

Original: In recent years, various deep learning approaches have been suggested as well.
Revised: In recent years, various deep learning approaches have been suggested as well.

Original: Auto-encoders are frequently used for dimensionality reduction (corresponding to a feature extraction) of the input data in this context.
Revised: Auto-encoders are frequently used for dimensionality reduction (corresponding to a feature extraction) of the input data in this context.

Original: Some methods then directly calculate the reconstruction error to determine whether new queries belong to the same class or are outliers (e.g. Sakurada & Yairi (2014); Chen et al.
Revised: Some methods then directly calculate the reconstruction error to determine whether new queries belong to the same class or are outliers (e.g. Sakurada & Yairi (2014); Chen et al.

Original: Others add a subsequent classiﬁcation network (e.g. Morton & Grifﬁn (2016) or Erfani et al.
Revised: Others add a subsequent classiﬁcation network (e.g. Morton & Grifﬁn (2016) or Erfani et al.

Original: (2016), where the auto-encoder is replaced with a Deep Belief Network).
Revised: (2016), where the auto-encoder is replaced with a Deep Belief Network).

Original: Ruff et al.
Revised: Ruff et al.

Original: (2018) take inspiration from OSVM approaches and train
Revised: (2018) take inspiration from OSVM approaches and train

Original: CNNs jointly with a classiﬁcation SVM.
Revised: CNNs jointly with a classiﬁcation SVM.

Original: Lately, adversarial approaches have seen attention in this domain as well, e.g. in Schlegl et al.
Revised: Lately, adversarial approaches have seen attention in this domain as well, e.g. in Schlegl et al.

Original: (2017) and Sabokrou et al.
Revised: (2017) and Sabokrou et al.

Original: (2018), where they are used in a similar fashion as the previous auto-encoders.
Revised: (2018), where they are used in a similar fashion as the previous auto-encoders.

Original: Direct training on a single annotated class is not implemented as frequently as the combination of automatic feature extraction and distance (or reconstruction error) calculation.
Revised: Direct training on a single annotated class is not implemented as frequently as the combination of automatic feature extraction and distance (or reconstruction error) calculation.

Original: Oza & Patel (2019) demonstrates a strategy for training a CNN on positive examples and artiﬁcial negative examples generated from Gaussian distributions, while Perera & Patel (2018) shows a method based on automatic feature extraction in combination with specialized loss functions and template matching.
Revised: Oza & Patel (2019) demonstrates a strategy for training a CNN on positive examples and artiﬁcial negative examples generated from Gaussian distributions, while Perera & Patel (2018) shows a method based on automatic feature extraction in combination with specialized loss functions and template matching.

Original: A broad overview of deep learning techniques from the anomaly detection perspective is provided in
Revised: A broad overview of deep learning techniques from the anomaly detection perspective is provided in

Original: Chalapathy & Chawla (2019).
Revised: Chalapathy & Chawla (2019).

Original: All of these approaches rely on the availability of data from the expected class for training.
Revised: All of these approaches rely on the availability of data from the expected class for training.

Original: What is missing is a type of model that can determine whether a query belongs to a class from a small number of examples that are provided at evaluation time without the need to re-train.
Revised: What is missing is a type of model that can determine whether a query belongs to a class from a small number of examples that are provided at evaluation time without the need to re-train.

Original: Such a classiﬁcation is possible with few-shot learning.
Revised: Such a classiﬁcation is possible with few-shot learning.

Original: 2. F EW - SHOT LEARNING
Revised: 2. F EW - SHOT LEARNING

Original: The concept of one- or few-shot learning is motivated by human learning: If humans are presented with a new class of objects, they do not require more than a few examples to be able to match new instances to this class (often, just one supporting example is sufﬁcient).
Revised: The concept of one- or few-shot learning is motivated by human learning: If humans are presented with a new class of objects, they do not require more than a few examples to be able to match new instances to this class (often, just one supporting example is sufﬁcient).

Original: This ability is, among other factors, informed by general world experience, which teaches humans to set boundaries between object classes.
Revised: This ability is, among other factors, informed by general world experience, which teaches humans to set boundaries between object classes.

Original: Matching networks, which implement this idea, were ﬁrst introduced by Vinyals et al.
Revised: Matching networks, which implement this idea, were ﬁrst introduced by Vinyals et al.

Original: Training is performed on sets of support examples for a subset of possible classes plus a query example belonging to one of the classes in so-called episodes.
Revised: Training is performed on sets of support examples for a subset of possible classes plus a query example belonging to one of the classes in so-called episodes.

Original: These episodes are generated for a wide range of class permutations.
Revised: These episodes are generated for a wide range of class permutations.

Original: In the one-shot case, only one support example is given for each class, while there are multiple ones in few-shot models.
Revised: In the one-shot case, only one support example is given for each class, while there are multiple ones in few-shot models.

Original: Such a network has two input branches: One for the support examples, and one for the query example.
Revised: Such a network has two input branches: One for the support examples, and one for the query example.

Original: In both branches, the inputs are run through a sub-network which performs an embedding.
Revised: In both branches, the inputs are run through a sub-network which performs an embedding.

Original: The embedding networks may or may not share their weights.
Revised: The embedding networks may or may not share their weights.

Original: The query’s resulting embedding is then compared against the supports’ resulting embeddings using a pre-deﬁned distance metric.
Revised: The query’s resulting embedding is then compared against the supports’ resulting embeddings using a pre-deﬁned distance metric.

Original: This results in likelihood measures for each input class, commonly implemented with a softmax.
Revised: This results in likelihood measures for each input class, commonly implemented with a softmax.

Original: The over-all comparison metric for the task is learned implicitly as the network learns an appropriate embedding.
Revised: The over-all comparison metric for the task is learned implicitly as the network learns an appropriate embedding.

Original: Inthefew-shot case,
Revised: Inthefew-shot case,

Original: multiple support examples per class are treated independently of each other throughout the network.
Revised: multiple support examples per class are treated independently of each other throughout the network.

Original: Example-related likelihoods are then summed up to obtain class likelihoods at the output.
Revised: Example-related likelihoods are then summed up to obtain class likelihoods at the output.

Original: In this way, matching networks essentially implement a weighted nearest-neigbor classiﬁcation.
Revised: In this way, matching networks essentially implement a weighted nearest-neigbor classiﬁcation.

Original: As a further development, Snell et al.
Revised: As a further development, Snell et al.

Original: (2017) introduced prototypical networks .
Revised: (2017) introduced prototypical networks.

Original: Instead of treating all of a class’ support examples independently, they compute a so-called “prototype” of the class after the embedding network.
Revised: Instead of treating all of a class’ support examples independently, they compute a so-called “prototype” of the class after the embedding network.

Original: This prototype is commonly the mean vector (or centroid) of the class’s embedded support examples.
Revised: This prototype is commonly the mean vector (or centroid) of the class’s embedded support examples.

Original: When the squared Euclidean distance is used as the metric, this implements a linear classiﬁer rather than the nearest neighbor classiﬁer produced by matching networks.
Revised: When the squared Euclidean distance is used as the metric, this implements a linear classiﬁer rather than the nearest neighbor classiﬁer produced by matching networks.

Original: Finnetal.
Revised: Finnetal.

Original: (2017) goes one step further and introduces model-agnostic meta learning (MAML): An algorithm to train models that can be easily ﬁnetuned to new tasks with very few examples.
Revised: (2017) goes one step further and introduces model-agnostic meta learning (MAML): An algorithm to train models that can be easily ﬁnetuned to new tasks with very few examples.

Original: This can also be applied to other few-shot methods such, as matching networks.
Revised: This can also be applied to other few-shot methods such, as matching networks.

Original: When applying few-shot learning to a binary problem, support examples for two classes are required: Positive examples of the class of interest, and random other examples.
Revised: When applying few-shot learning to a binary problem, support examples for two classes are required: Positive examples of the class of interest, and random other examples.

Original: This amounts to a workaround, as there are no two comparable classes.
Revised: This amounts to a workaround, as there are no two comparable classes.

Original: Instead, one of them merely consists of counterexamples (a sort of “garbage” class).
Revised: Instead, one of them merely consists of counterexamples (a sort of “garbage” class).

Original: Examples in this class will have a much wider feature distribution.
Revised: Examples in this class will have a much wider feature distribution.

Original: This makes selecting random examples difﬁcult, as they should still be representative of this distribution, particularly when only a small number of support examples is used.
Revised: This makes selecting random examples difﬁcult, as they should still be representative of this distribution, particularly when only a small number of support examples is used.

Original: It also means that computational power and time must be dedicated to a class that is not actually of interest, and as such is not very elegant.
Revised: It also means that computational power and time must be dedicated to a class that is not actually of interest, and as such is not very elegant.

Original: Siamese networks [Koch et al.
Revised: Siamese networks [Koch et al.

Original: (2015)] can be seen as an interesting way to solve this problem, as they implement a decision about whether two unseen examples belong to the same class or different ones.
Revised: (2015)] can be seen as an interesting way to solve this problem, as they implement a decision about whether two unseen examples belong to the same class or different ones.

Original: However, they are not able to exploit more than one support example per class like prototypical networks can.
Revised: However, they are not able to exploit more than one support example per class like prototypical networks can.

Original:  P ROPOSED APPROACH 3. O NE - WAY PROTOTYPICAL MODELS
Revised:  P ROPOSED APPROACH 3. O NE - WAY PROTOTYPICAL MODELS

Original: Motivated by the counterexample issue, we take the described approach one step further:
Revised: Motivated by the counterexample issue, we take the described approach one step further:

Original: We construct a prototypical network that only requires positive support examples, from which it generates an internal embedding of the whole class.
Revised: We construct a prototypical network that only requires positive support examples, from which it generates an internal embedding of the whole class.

Original: The query embedding is then compared to this, and judged to belong to the same class or not.
Revised: The query embedding is then compared to this, and judged to belong to the same class or not.

Original: We call this type of network a “one-way prototypical network”.
Revised: We call this type of network a “one-way prototypical network”.

Original: The sub-network for generating the internal embeddings is implemented just as in regular prototypical networks with shared weights between the support and query branches.
Revised: The sub-network for generating the internal embeddings is implemented just as in regular prototypical networks with shared weights between the support and query branches.

Original: An illustration is shown in ﬁgure 1a.
Revised: An illustration is shown in ﬁgure 1a.

Original: This approach presents a challenge for the distance metric: Because Cosine or Euclidean distances are not bounded, we cannot set a threshold for class belonging.
Revised: This approach presents a challenge for the distance metric: Because Cosine or Euclidean distances are not bounded, we cannot set a threshold for class belonging.

Original: In order to do this, we need a basis for comparison.
Revised: In order to do this, we need a basis for comparison.

Original: We implement this by introducing a “null class” that models the whole space of possible examples without explicitly being given those examples (as in the two-class implementation mentioned above).
Revised: We implement this by introducing a “null class” that models the whole space of possible examples without explicitly being given those examples (as in the two-class implementation mentioned above).

Original: Queries are judged on whether they are more probable to belong to the (positive) class we are interested in rather than the null class.
Revised: Queries are judged on whether they are more probable to belong to the (positive) class we are interested in rather than the null class.

Original: The question then becomes how to construct this class.
Revised: The question then becomes how to construct this class.

Original: As explained above, prototypical networks model classes by calculating the centroid of the class’s support examples.
Revised: As explained above, prototypical networks model classes by calculating the centroid of the class’s support examples.

Original: As the model includes batch normalization layers, we can assume that the centroid of the whole latent space will be close to the zero point in all embedding dimensions.
Revised: As the model includes batch normalization layers, we can assume that the centroid of the whole latent space will be close to the zero point in all embedding dimensions.

Original: A query’s distance to the centroid of the positive class is therefore compared to its norm (i.e. its distance to the zero centroid) to determine whether it is more likely to belong to the positive class or to the null class encompassing everything.
Revised: A query’s distance to the centroid of the positive class is therefore compared to its norm (i.e. its distance to the zero centroid) to determine whether it is more likely to belong to the positive class or to the null class encompassing everything.

Original: Figure 1b shows a visualization where the null centroid is denoted as c 0 .
Revised: Figure 1b shows a visualization where the null centroid is denoted as c 0 .

Original: A similar idea was presented in
Revised: A similar idea was presented in

Original: Ren et al.
Revised: Ren et al.

Original: (2018), which suggests the use of unlabeled data to improve prototypes.
Revised: (2018), which suggests the use of unlabeled data to improve prototypes.

Original: The null (or “distractor”) class here is only used to identify examples not belonging to any of the trained classes, rather than training the model to explicitly distinguish between a single given class and the null class.
Revised: The null (or “distractor”) class here is only used to identify examples not belonging to any of the trained classes, rather than training the model to explicitly distinguish between a single given class and the null class.

Revised: As in regular prototypical models, Euclidean distance is used.

Original: One issue that arises is that in the state-of-the-art prototypical models, batch normalization is not performed at the end of the convolution blocks of the embedding network, but rather after the convolution layer and before the (ReLU) non-linearity and pooling layers.
Revised: One issue that arises is that in the state-of-the-art prototypical models, batch normalization is not performed at the end of the convolution blocks of the embedding network, but rather after the convolution layer and before the (ReLU) non-linearity and pooling layers.

Original: The ﬁnal block is therefore not quite batch-normalized.
Revised: The ﬁnal block is therefore not quite batch-normalized.

Original: As there is some debate about the ordering of these layers anyway
Revised: As there is some debate about the ordering of these layers anyway

Original: [Ioffe & Szegedy (2015); Mishkin], we propose reordering the block layout to a structure of: Convolution - (Leaky) ReLU - Max Pooling - Batch Normalization.
Revised: [Ioffe & Szegedy (2015); Mishkin], we propose reordering the block layout to a structure of: Convolution - (Leaky) ReLU - Max Pooling - Batch Normalization.

Original: An alternative to the null class solution would be a simple thresholding of the class likelihood.
Revised: An alternative to the null class solution would be a simple thresholding of the class likelihood.

Original: However, preliminary experiments showed that this threshold is very problem- and concept-dependent, and thus does not generalize well to unseen episodes.
Revised: However, preliminary experiments showed that this threshold is very problem- and concept-dependent, and thus does not generalize well to unseen episodes.

Original: [Figure]
Revised: [Figure]

Original: 3. O NE - WAY NORMAL PROTOTYPICAL MODELS
Revised: 3. O NE - WAY NORMAL PROTOTYPICAL MODELS

Original: As described, prototypical networks only take into account the centroid of a class’s support examples.
Revised: As described, prototypical networks only take into account the centroid of a class’s support examples.

Original: This is sufﬁcient when all classes can be assumed to be roughly on the same semantic level and therefore have similar feature distribution widths.
Revised: This is sufﬁcient when all classes can be assumed to be roughly on the same semantic level and therefore have similar feature distribution widths.

Original: However, this may not always be true (e.g. we may encounter cases where the model attempts to distinguish between queries of the class “Bobtail cat” and queries of the class “Insect”).
Revised: However, this may not always be true (e.g. we may encounter cases where the model attempts to distinguish between queries of the class “Bobtail cat” and queries of the class “Insect”).

Original: This problem becomes even more prevalent for the one-class case, as unseen classes may have wider or narrower distributions than unseen examples, and we do not have a basis for comparison deﬁned by other classes.
Revised: This problem becomes even more prevalent for the one-class case, as unseen classes may have wider or narrower distributions than unseen examples, and we do not have a basis for comparison deﬁned by other classes.

Original: Allen et al.
Revised: Allen et al.

Original: (2019) suggest solving this problem by dynamically adding prototypes per class as necessary - so-called Inﬁnite Mixture Prototypes (IMP).
Revised: (2019) suggest solving this problem by dynamically adding prototypes per class as necessary - so-called Inﬁnite Mixture Prototypes (IMP).

Original: We propose a different extension to prototypical networks: Modeling classes by their distribution instead of their centroid only.
Revised: We propose a different extension to prototypical networks: Modeling classes by their distribution instead of their centroid only.

Original: These distributions are ﬁtted to the class’s support examples.
Revised: These distributions are ﬁtted to the class’s support examples.

Original: Due to the effect of batch normalization, we assume multivariate normal distributions for this extension.
Revised: Due to the effect of batch normalization, we assume multivariate normal distributions for this extension.

Original: A query’s class assignment is then determined by its likelihood under each class’s distribution.
Revised: A query’s class assignment is then determined by its likelihood under each class’s distribution.

Original: The previous embedding network is implemented just as before, except that Leaky ReLU activations become necessary instead of ReLUs to guarantee training stability.
Revised: The previous embedding network is implemented just as before, except that Leaky ReLU activations become necessary instead of ReLUs to guarantee training stability.

Original: (To clarify: If the value of any of the embedding dimensions becomes exactly zero after the non-linearity, the all-over probability becomes zero as well, leading to a failure of gradient calculation during training.
Revised: (To clarify: If the value of any of the embedding dimensions becomes exactly zero after the non-linearity, the all-over probability becomes zero as well, leading to a failure of gradient calculation during training.

Original: Leaky ReLU prevents this from happening).
Revised: Leaky ReLU prevents this from happening).

Original: This extension of prototypical networks also integrates with the one-way framework described above.
Revised: This extension of prototypical networks also integrates with the one-way framework described above.

Original: Instead of assuming a centroid at zero for the null class, we now assume a multivariate normal distribution with a mean at zero and a standard deviation of 1 (this is not guaranteed exactly, though: Batch normalization enforces a centering at zero and a standard deviation of 1, but does not enforce Gaussian-ness).
Revised: Instead of assuming a centroid at zero for the null class, we now assume a multivariate normal distribution with a mean at zero and a standard deviation of 1 (this is not guaranteed exactly, though: Batch normalization enforces a centering at zero and a standard deviation of 1, but does not enforce Gaussian-ness).

Original: A visualization is provided in ﬁgure 2.
Revised: A visualization is provided in ﬁgure 2.

Original: (In the one-way case, this is an advantage over IMP: Preliminary experiments show that adding the one-way concept to IMP models does not lead to improvements over standard prototypical networks as the multiple positive prototypes tend to overpower the one negative prototype.)
Revised: (In the one-way case, this is an advantage over IMP: Preliminary experiments show that adding the one-way concept to IMP models does not lead to improvements over standard prototypical networks as the multiple positive prototypes tend to overpower the one negative prototype.)

Original: Of course, this idea has one obvious drawback: The model requires a sufﬁcient number of support examples to be able to determine a salient standard deviation over them, particularly as the embedding dimensions are treated independently.
Revised: Of course, this idea has one obvious drawback: The model requires a sufﬁcient number of support examples to be able to determine a salient standard deviation over them, particularly as the embedding dimensions are treated independently.

Original: We hypothesize that beyond a certain number of support examples, implementing class’s distributions instead of their centroids only will enable the model to distinguish between them better.
Revised: We hypothesize that beyond a certain number of support examples, implementing class’s distributions instead of their centroids only will enable the model to distinguish between them better.

Original: (As a side note, Fort (2017) also showed how to integrate a distribution concept into prototypical networks, calling them “Gaussian Prototypical Networks”.
Revised: (As a side note, Fort (2017) also showed how to integrate a distribution concept into prototypical networks, calling them “Gaussian Prototypical Networks”.

Original: However, in this publication, the covariance matrix is calculated for each embedded example rather than over the various support embeddings.
Revised: However, in this publication, the covariance matrix is calculated for each embedded example rather than over the various support embeddings.

Original: Is is then taken into account explictly in the distance metric.
Revised: It is then taken into account explictly in the distance metric.

Original: This allows a better handling of uncertainties, particularly for noisy data.
Revised: This allows a better handling of uncertainties, particularly for noisy data.

Original: Our focus here is somewhat different: We are more interested in obtaining additional information from the relationships between support examples).
Revised: Our focus here is somewhat different: We are more interested in obtaining additional information from the relationships between support examples).

Original:  E XPERIMENTS AND RESULTS 4. D ATA AND EXPERIMENTAL DESIGN
Revised:  E XPERIMENTS AND RESULTS 4. D ATA AND EXPERIMENTAL DESIGN

Original: We ﬁrst test our approach on a well-known toy data set for one- or few-shot learning: The Omniglot image data set
Revised: We ﬁrst test our approach on a well-known toy data set for one- or few-shot learning: The Omniglot image data set

Original: [Lake et al.
Revised: [Lake et al.

Original: This data set contains 1623 characters hand-drawn 20 times by Amazon Mechanical Turk 1 annotators.
Revised: This data set contains 1623 characters hand-drawn 20 times by Amazon Mechanical Turk 1 annotators.

Original: These characters come from 50 alphabets; a 30-20 split of the alphabets is suggested for training and validation.
Revised: These characters come from 50 alphabets; a 30-20 split of the alphabets is suggested for training and validation.

Original: Per episode, we use 1 to 19 instances of a character as support examples, and either one of the remaining examples or a random other character as the query.
Revised: Per episode, we use 1 to 19 instances of a character as support examples, and either one of the remaining examples or a random other character as the query.

Original: As commonly done in literature, we also test the trained models on an unmatched data set: The MNIST handwritten digit database [LeCun et al.
Revised: As commonly done in literature, we also test the trained models on an unmatched data set: The MNIST handwritten digit database [LeCun et al.

Original: Second, we run experiments on the much more complex MiniImageNet data set [Vinyals et al.
Revised: Second, we run experiments on the much more complex MiniImageNet data set [Vinyals et al.

Original: MiniImageNet is a subset of the larger ImageNet data set [Russakovsky et al.
Revised: MiniImageNet is a subset of the larger ImageNet data set [Russakovsky et al.

Original: It contains 60,000 color images of size 84x84, split across 100 semantic classes with 600 examples each.
Revised: It contains 60,000 color images of size 84x84, split across 100 semantic classes with 600 examples each.

Original: Out of these, 80 classes are used for training and 20 for test.
Revised: Out of these, 80 classes are used for training and 20 for test.

Original: We use the splits provided by
Revised: We use the splits provided by

Original: Ravi & Larochelle (2017).
Revised: Ravi & Larochelle (2017).

Original: Our experiments are performed with 1 to 20 support examples per class.
Revised: Our experiments are performed with 1 to 20 support examples per class.

Original: In both cases, pre-processing simply consists of a normalization of the input image data to a mean of 0 and a standard deviation of 1.
Revised: In both cases, pre-processing simply consists of a normalization of the input image data to a mean of 0 and a standard deviation of 1.

Original: All trainings are performed six times with random initializations and training episodes.
Revised: All trainings are performed six times with random initializations and training episodes.

Original: The results are reported in terms of accuracy and averaged across runs; standard deviation between runs is generally very low (¡.01).
Revised: The results are reported in terms of accuracy and averaged across runs; standard deviation between runs is generally very low ( < .01).

Original: Each training is run for 20 (for Omniglot ) or 10 ( MiniImageNet ) episodes, with 32,000 randomly generated episodes each, resulting effectively in 640,000 and 320,000 training episodes respectively.
Revised: Each training is run for 20 (for Omniglot ) or 10 ( MiniImageNet ) episodes, with 32,000 randomly generated episodes each, resulting effectively in 640,000 and 320,000 training episodes respectively.

Original: Adam is used as the optimizer.
Revised: Adam is used as the optimizer.

Original: 4. E XPERIMENT A: S IAMESE AND TWO - WAY MODELING
Revised: 4. E XPERIMENT A: S IAMESE AND TWO - WAY MODELING

Original: In our ﬁrst experiment, we test the performance of previous architectures on the one-way problem:
Revised: In our ﬁrst experiment, we test the performance of previous architectures on the one-way problem:

Original: With Siamese, matching, and prototypical networks.
Revised: With Siamese, matching, and prototypical networks.

Original: The former can only integrate a single support example and compare this to the query.
Revised: The former can only integrate a single support example and compare this to the query.

Original: For the latter two, we implement the one-way problem as a two-way classiﬁcation, with the relevant “positive” class on the one hand, and the “negative” class on the other.
Revised: For the latter two, we implement the one-way problem as a two-way classiﬁcation, with the relevant “positive” class on the one hand, and the “negative” class on the other.

Original: Support examples are thus selected from the positive class, and randomly from the whole remaining data set.
Revised: Support examples are thus selected from the positive class, and randomly from the whole remaining data set.

Original: Results on the Omniglot and MiniImageNet data sets are displayed in ﬁgure 3.
Revised: Results on the Omniglot and MiniImageNet data sets are displayed in ﬁgure 3.

Original: Supports range between 1 and 19 ( Omniglot ) or 20 ( MiniImageNet ).
Revised: Supports range between 1 and 19 ( Omniglot ) or 20 ( MiniImageNet ).

Original: The results are generally acceptable, with prototypical models always delivering higher accuracies than matching networks.
Revised: The results are generally acceptable, with prototypical models always delivering higher accuracies than matching networks.

Original: This difference becomes larger the more support examples are available, as this allows the prototypical networks to model the negative class better, whereas the nearestneighbor classiﬁcation of matching networks is not strongly improved with more support examples.[Figure] [Figure]
Revised: This difference becomes larger the more support examples are available, as this allows the prototypical networks to model the negative class better, whereas the nearestneighbor classiﬁcation of matching networks is not strongly improved with more support examples.

Original: Accuracy of the prototypical networks ranges between .
Revised: Accuracy of the prototypical networks ranges between .

Original: 75 for one support example, and . 977 foron the Omniglot data set, and between .
Revised: 75 for one support example, and . 977 for [Figure]

Original:  (one support) and . 74 (20 supports) on MiniImageNet .
Revised: [Figure] on the Omniglot data set, and between . 59 (one support) and . 74 (20 supports) on MiniImageNet .

Original: Siamese networks perform well (even best on Omniglot ) for one support example, but are quickly surpassed by prototypical networks when more are available.
Revised: Siamese networks perform well (even best on Omniglot ) for one support example, but are quickly surpassed by prototypical networks when more are available.

Original: The novel normal prototypical networks work well for this problem, but do not surpass standard prototypical networks.
Revised: The novel normal prototypical networks work well for this problem, but do not surpass standard prototypical networks.

Original: For very few support examples, results do not rise much above the . 5 baseline, or training may even fail completely.
Revised: For very few support examples, results do not rise much above the . 5 baseline, or training may even fail completely.

Original: This is to be expected, as the distribution cannot be determined from such a low number of supports.
Revised: This is to be expected, as the distribution cannot be determined from such a low number of supports.

Original: 5 examples appear to be necessary to make these models work at all, and 10 to make them generate competitive results.
Revised: 5 examples appear to be necessary to make these models work at all, and 10 to make them generate competitive results.

Original: We also test the models trained on Omniglot on the unmatched MNIST data set.
Revised: We also test the models trained on Omniglot on the unmatched MNIST data set.

Original: Results are not shown here, but follow the same trends, just somewhat lower.
Revised: Results are not shown here, but follow the same trends, just somewhat lower.

Original: The accuracy of prototypical networks is .
Revised: The accuracy of prototypical networks is .

Original: 686 for one support example, and .
Revised: 686 for one support example, and .

Original: 883 for 19.
Revised: 883 for 19.

Original: 4. E XPERIMENT B: O NE - WAY PROTOTYPICAL MODELS
Revised: 4. E XPERIMENT B: O NE - WAY PROTOTYPICAL MODELS

Original: We next test the novel one-way conﬁguration for prototypical models which distinguishes between a “positive” class that we are interested in, and a “null” class that is ﬁxed at a zero centroid as described in section 3.1.
Revised: We next test the novel one-way conﬁguration for prototypical models which distinguishes between a “positive” class that we are interested in, and a “null” class that is ﬁxed at a zero centroid as described in section 3.1.

Original: The results are shown in ﬁgure 4.
Revised: The results are shown in ﬁgure 4.

Original: The ﬁrst bar in each group is the two-way result from the previous experiment for comparison.
Revised: The ﬁrst bar in each group is the two-way result from the previous experiment for comparison.

Original: The second one displays the accuracy of the same type of model, but with the null class instead of an explicit “negative” class.
Revised: The second one displays the accuracy of the same type of model, but with the null class instead of an explicit “negative” class.

Original: As mentioned above, we also test a model with reordered layers so that batch normalization comes last before the prototype calculation; the third bar shows the results for this model.
Revised: As mentioned above, we also test a model with reordered layers so that batch normalization comes last before the prototype calculation; the third bar shows the results for this model.

Original: We discover that the one-way realization works better in almost all cases.
Revised: We discover that the one-way realization works better in almost all cases.

Original: Results improve particularly in cases with few support examples.
Revised: Results improve particularly in cases with few support examples.

Original: This makes sense, as few examples do not allow the[Figure] two-way model to ﬁnd the centroid of the negative class easily, whereas this is pre-deﬁned in the one-way network.
Revised: This makes sense, as few examples do not allow the two-way model to ﬁnd the centroid of the negative class easily, whereas this is pre-deﬁned in the one-way network.

Original: We also notice that the reordered model provides better results than the regular one; batch-normalizing our latent representations right before the prototype calculation therefore  appears to be important (we observe the same effect when simply adding another batch normalization layer to the standard model).
Revised: We also notice that the reordered model provides better results than the regular one; batch-normalizing our latent representations right before the prototype calculation therefore [Figure] appears to be important (we observe the same effect when simply adding another batch normalization layer to the standard model).

Original: With this model, accuracy on Omniglot rises to .
Revised: With this model, accuracy on Omniglot rises to .

Original: 969 for one support example, and to .
Revised: 969 for one support example, and to .

Original: 98 for 19.
Revised: 98 for 19.

Original: On the more complex MiniImageNet data, accuracy becomes .
Revised: On the more complex MiniImageNet data, accuracy becomes .

Original: 64 for one support example and .
Revised: 64 for one support example and .

Original: 8 for 20.
Revised: 8 for 20.

Revised: For comparison, this re-ordering was also tested on the two-way prototypical model; no signiﬁcant differences in the results were found when compared to the original layer ordering.

Original: Once again, we also test the Omniglot models on MNIST .
Revised: Once again, we also test the Omniglot models on MNIST .

Original: For 19 support examples, the accuracy is slightly worse compared to the two-class model at .
Revised: For 19 support examples, the accuracy is slightly worse compared to the two-class model at .

Original: This makes sense as the batch normalization is optimized on Omniglot , and a large number of supports allows for a better estimation of the negative class’ distribution.
Revised: This makes sense as the batch normalization is optimized on Omniglot , and a large number of supports allows for a better estimation of the negative class’ distribution.

Original: However, results are better or equal for a small number of support examples (e.g. . 743 for one).
Revised: However, results are better or equal for a small number of support examples (e.g. . 743 for one).

Original: 4. E XPERIMENT C: N ORMAL PROTOTYPICAL MODELS
Revised: 4. E XPERIMENT C: N ORMAL PROTOTYPICAL MODELS

Original: Finally, we test the various conﬁgurations of our novel normal prototypical models described in section 3.2.
Revised: Finally, we test the various conﬁgurations of our novel normal prototypical models described in section 3.2.

Original: The results are shown in ﬁgure 5.
Revised: The results are shown in ﬁgure 5.

Original: Out of these, the ﬁrst bar in each group displays the two-way model accuracy from experiment A, in which the prototypes are modeled with a normal distribution.
Revised: Out of these, the ﬁrst bar in each group displays the two-way model accuracy from experiment A, in which the prototypes are modeled with a normal distribution.

Original: The second bar shows the results for the same model, but with a null class modeled with a unit Gaussian (mean 0, standard deviation 1) instead of the explicit negative class.
Revised: The second bar shows the results for the same model, but with a null class modeled with a unit Gaussian (mean 0, standard deviation 1) instead of the explicit negative class.

Original: Third, we allow the model to train the standard deviation of the null class instead of ﬁxing it at 1.
Revised: Third, we allow the model to train the standard deviation of the null class instead of ﬁxing it at 1.

Original: The fourth and ﬁfth bars show results for similar models, but once again with the embedding layers reordered so that the last step before the prototype calculation is batch normalization.
Revised: The fourth and ﬁfth bars show results for similar models, but once again with the embedding layers reordered so that the last step before the prototype calculation is batch normalization.

Original: As described in experiment A, this type of model does not produce usable results with few support vectors, and may even fail to train.
Revised: As described in experiment A, this type of model does not produce usable results with few support vectors, and may even fail to train.

Original: For this reason, we only show results for 5 or more support vectors.
Revised: For this reason, we only show results for 5 or more support vectors.

Original: With three or more for Omniglot or 5 or more for MiniImageNet , we start to notice an improvement when using the one-way implementation over the two-way version - just as in the previous experiment.
Revised: With three or more for Omniglot or 5 or more for MiniImageNet , we start to notice an improvement when using the one-way implementation over the two-way version - just as in the previous experiment.

Original: Once again, reordering the embedding layers to put batch normalization last increases accuracy, but the effect is not as strong as in standard prototypical models.
Revised: Once again, reordering the embedding layers to put batch normalization last increases accuracy, but the effect is not as strong as in standard prototypical models.

Original: Allowing the model to adapt the standard deviation of the null distribution during training also helps.
Revised: Allowing the model to adapt the standard deviation of the null distribution during training also helps.

Original: This is somewhat surprising because batch normalization enforces a standard deviation of 1 over all data.
Revised: This is somewhat surprising because batch normalization enforces a standard deviation of 1 over all data.

Original: We suspect that this happens because the actual distribution is not perfectly Gaussian, and therefore is not modeled well by our assumptions.
Revised: We suspect that this happens because the actual distribution is not perfectly Gaussian, and therefore is not modeled well by our assumptions.

Original: We tested adding a loss term to enforce Gaussian-ness, but did not see any improvemtents.
Revised: We tested adding a loss term to enforce Gaussian-ness, but did not see any improvemtents.

Original: This effect will require closer investigation in the future.
Revised: This effect will require closer investigation in the future.

Original: Over-all, the results follow the same trends as those of the standard prototypical models, but mostly remain slightly below them.
Revised: Over-all, the results follow the same trends as those of the standard prototypical models, but mostly remain slightly below them.

Original: However, for a larger number of support examples ( ≥ 15), they surpass them slightly in accuracy on the Omniglot data set.
Revised: However, for a larger number of support examples ( ≥ 15), they surpass them slightly in accuracy on the Omniglot data set.

Original: Future experiments with even more supports are necessary to determine where they become useful.
Revised: Future experiments with even more supports are necessary to determine where they become useful.

Original: Over-all, we obtain accuracies of . 985 for
Revised: Over-all, we obtain accuracies of . 985 for

Original: Omniglot with 19 support examples, and .
Revised: Omniglot with 19 support examples, and .

Original: 78 for MiniImageNet with 20.
Revised: 78 for MiniImageNet with 20.

Original: When testing the Omniglot models on MNIST , they yield an accuracy of .
Revised: When testing the Omniglot models on MNIST , they yield an accuracy of .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: As an additional experiment, we analyze the training times of our various models on an NVIDIA
Revised: As an additional experiment, we analyze the training times of our various models on an NVIDIA

Original: Volta V100 with 16GB RAM.
Revised: Volta V100 with 16GB RAM.

Original: This is calculated by averaging the training times of the last three epochs of each model.
Revised: This is calculated by averaging the training times of the last three epochs of each model.

Original: Figure 6 displays the results on Omniglot and MiniImageNet .
Revised: Figure 6 displays the results on Omniglot and MiniImageNet .

Original: As mentioned above, each epoch consists of 32,000 episodes, and we choose the conﬁguration with 5 support examples for comparison.
Revised: As mentioned above, each epoch consists of 32,000 episodes, and we choose the conﬁguration with 5 support examples for comparison.

Original: The ﬁrst two bars in each group correspond to the two-way matching and prototypical models from experiment A; their training times are roughly in the same range.
Revised: The ﬁrst two bars in each group correspond to the two-way matching and prototypical models from experiment A; their training times are roughly in the same range.

Original: Interestingly, the two-way normal prototypical model (fourth bar, also from experiment A) trains slightly faster on Omniglot and much faster for MiniImageNet while achieving almost the same results as the prototypical model.
Revised: Interestingly, the two-way normal prototypical model (fourth bar, also from experiment A) trains slightly faster on Omniglot and much faster for MiniImageNet while achieving almost the same results as the prototypical model.

Original: We are currently investigating the reasons for this.
Revised: We are currently investigating the reasons for this.

Original: More importantly, bars three and ﬁve show the training times for the one-way implementations.
Revised: More importantly, bars three and ﬁve show the training times for the one-way implementations.

Original: For both prototypical and normal prototypical models, training time is almost halved.
Revised: For both prototypical and normal prototypical models, training time is almost halved.

Original: This is explained by the fact that only half the support examples need to be processed, and the centroids or distributions only need to be calculated for one class instead of two in each episode.
Revised: This is explained by the fact that only half the support examples need to be processed, and the centroids or distributions only need to be calculated for one class instead of two in each episode.

Original: As seen above, these models also usually deliver better results than the two-way implementations, thus providing another argument for their usage.
Revised: As seen above, these models also usually deliver better results than the two-way implementations, thus providing another argument for their usage.

Original:  C ONCLUSION AND FUTURE WORK
Revised:  C ONCLUSION AND FUTURE WORK

Original: In this paper, we presented approaches for few-shot learning in cases where we are only interested in a single class in each episode.
Revised: In this paper, we presented approaches for few-shot learning in cases where we are only interested in a single class in each episode.

Original: Naively, this can be modeled with a two-way matching or prototypical model with support examples for the relevant, “positive” class, and random other support examples for the “negative” class.
Revised: Naively, this can be modeled with a two-way matching or prototypical model with support examples for the relevant, “positive” class, and random other support examples for the “negative” class.

Original: This is not very elegant as selecting negative examples is difﬁcult, and those may not reﬂect the negative class very well, especially when few supports are used.
Revised: This is not very elegant as selecting negative examples is difﬁcult, and those may not reﬂect the negative class very well, especially when few supports are used.

Original: We suggest introducing a “null” class in prototypical models instead which does not require support examples.
Revised: We suggest introducing a “null” class in prototypical models instead which does not require support examples.

Original: As batch normalization enforces zero centering in the latent space, we simply assume a zero centroid for this null class.
Revised: As batch normalization enforces zero centering in the latent space, we simply assume a zero centroid for this null class.

Original: We show that such models work better than the two-way implementation.
Revised: We show that such models work better than the two-way implementation.

Original: Additionally, training time is reduced almost by a factor of two as only half of the support examples are required.
Revised: Additionally, training time is reduced almost by a factor of two as only half of the support examples are required.

Original: As another novel idea, we suggest modeling class prototypes not just by their mean (or centroid), but by their distribution.
Revised: As another novel idea, we suggest modeling class prototypes not just by their mean (or centroid), but by their distribution.

Original: We assume normal distributions here, and model them via their mean and standard deviation.
Revised: We assume normal distributions here, and model them via their mean and standard deviation.

Original: In the one-way case, we model the null class with a multivariate unit Gaussian (with mean 0 and standard deviation 1).
Revised: In the one-way case, we model the null class with a multivariate unit Gaussian (with mean 0 and standard deviation 1).

Original: This has the disadvantage of requiring a sufﬁcient number of support examples to estimate these parameters.
Revised: This has the disadvantage of requiring a sufﬁcient number of support examples to estimate these parameters.

Original: If those are available, the approach generally works, but usually does not surpass the standard prototypical model below a certain number of support examples (around 15).
Revised: If those are available, the approach generally works, but usually does not surpass the standard prototypical model below a certain number of support examples (around 15).

Original: We will further investigate whether these models can be useful when a very large number of support examples are available.
Revised: We will further investigate whether these models can be useful when a very large number of support examples are available.

Original: There also is some evidence that such models train faster than standard prototypical models, which we will analyze.
Revised: There also is some evidence that such models train faster than standard prototypical models, which we will analyze.

Original: We suspect that one problem is the assumption of
Revised: We suspect that one problem is the assumption of

Original: Gaussian-ness, which is not guaranteed in the latent space.
Revised: Gaussian-ness, which is not guaranteed in the latent space.

Original: It will be helpful to examine whether we can either enforce this, or otherwise ﬁnd better ways to model the class distribution.
Revised: It will be helpful to examine whether we can either enforce this, or otherwise ﬁnd better ways to model the class distribution.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: You may include other additional sections here.
Revised: You may include other additional sections here.