L OSS F UNCTIONS FOR M ULTISET P REDICTION
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the problem of multiset prediction.
The goal of multiset prediction is to train a predictor that maps an input to a multiset consisting of multiple items.
Unlike existing problems in supervised learning, such as classiﬁcation, ranking and sequence generation, there is no known order among items in a target multiset, and each item in the multiset may appear more than once, making this problem extremely challenging.
In this paper, we propose a novel multiset loss function by viewing this problem from the perspective of sequential decision making.
The proposed multiset loss function is empirically evaluated on two families of datasets, one synthetic and the other real, with varying levels of difﬁculty, against various baseline loss functions including reinforcement learning, sequence, and aggregated distribution matching loss functions.
The experiments reveal the effectiveness of the proposed loss function over the others.
I NTRODUCTION
A relatively less studied problem in machine learning, particularly supervised learning, is the problem of multiset prediction.
The goal of such a problem is to learn a mapping from an arbitrary input to a multiset of items.
This problem appears in a variety of contexts.
For instance, in the context of high-energy physics, one of the important problems in a particle physics data analysis is to count how many physics objects, such as electrons, muons, photons, taus, and jets, are in an collision event (Ehrenfeld et al., 2011).
In computer vision, automatic alt-text, such as the one available on
Facebook, 1 is a representative example of multiset prediction (Welleck et al., 2017; Lempitsky &
Zisserman, 2010).
In multiset prediction, a learner is presented with an arbitrary input and the associated multiset of items.
It is assumed that there is no predeﬁned order among the items, and that there are no further annotations containing information about the relationship between the input and each of the items in the multiset.
These properties make the problem of multiset prediction unique from other wellstudied problems.
It is different from sequence prediction, because there is no known order among the items.
It is not a ranking problem, since each item may appear more than once.
It cannot be transformed into classiﬁcation, because the number of possible multisets grows exponentially with respect to the maximum multiset size.
In this paper, we view multiset prediction as a sequential decision making process.
Under this view, the problem reduces to ﬁnding a policy that sequentially predicts one item at a time, while the outcome is still evaluated based on the aggregate multiset of the predicted items.
We then ﬁrst propose an oracle policy that assigns non-zero probabilities only to prediction sequences that result exactly in the target, ground-truth multiset given an input.
This oracle is optimal in the sense that its prediction never decreases the precision and recall regardless of previous predictions.
That is, its decision is optimal in any state (i.e., prediction preﬁx).
We then propose a novel multiset loss which minimizes the KL divergence between the oracle policy and a parametrized policy at every point in a decision trajectory of the parametrized policy.
We compare the proposed multiset loss against an extensive set of baselines.
They include a sequential loss with an arbitrary rank function, sequential loss with an input-dependent rank function, aggregated distribution matching loss and its one-step variant.
We also test policy gradient, as was done by Welleck et al.
(2017) recently for multiset prediction.
Our evaluation is conducted on two sets of datasets with varying difﬁculties and properties.
According to the experiments, we ﬁnd that the proposed multiset loss outperforms all the other loss functions.
M ULTISET P REDICTION
A multiset prediction problem is a generalization of classiﬁcation, where a target is not a single class but a multiset of classes.
The goal is to ﬁnd a mapping from an input x to a multiset Y = (cid:8) y 1 , . . .
, y |Y| (cid:9) , where y k ∈ C .
Some of the core properties of multiset prediction are
the input x is an arbitrary vector. 2.
there is no predeﬁned order among the items y i in the target multiset Y . 3.
the size of Y may vary depending on the input x . 4.
each item in the class set C may appear more than once in Y .
2. R ELATED P ROBLEMS IN S UPERVISED L EARNING
Variants of this multiset prediction problem have been extensively studied.
They however differ from our deﬁnition of the problem.
Here, we go over each variant and discuss how it differs from our deﬁnition of multiset prediction.
Power Multiset Classiﬁcation Perhaps the most naive approach to multiset prediction is to transform the class set C into a set M ( C ) of all possible multisets.
This transformation, or the size of
M ( C ) , is not well deﬁned unless some constraints are put in place.
If the maximum size of a target multiset is set to K , the number of all possible multisets is
[Equation]
With some constant | C | , we notice that this grows exponentially in the maximum size of the target multiset.
Once the class set C is transformed, we can train a multi-class classiﬁer π that maps an input x to one of the elements in M ( C ) .
However, this is infeasible in practice and generally intractable.
For instance, for the COCO Medium dataset used later in the experiments (see section 4.1), M ( C ) has roughly 20 thousand elements while the dataset only contains roughly 40 thousand training examples.
For the full MS COCO dataset, | M ( C ) | is on the order of 10 49 , making it infeasible to learn a classiﬁer using this method.
Ranking A ranking problem can be considered as learning a mapping from a pair of input x and one of the items c ∈ C to its score s ( x, c ) .
All the items in the class set are then sorted according to the score, and this sorted order determines the rank of each item.
By taking the top- K items from this sorted list, we can turn this problem of ranking into set prediction.
Similarly to multiset prediction, the input x is arbitrary, and the target is a set without any prespeciﬁc order.
Ranking however differs from multiset prediction in that it is unable to handle multiple occurrences of a single item in the target set.
Aggregated Distribution Matching Instead of considering the target multiset as an actual multiset, one can convert it into a distribution by computing the frequency of each item from the class set in the target multiset.
That is,
[Equation]
where I · is an indicator function.
Then, we can simply minimize a divergence between this distribution and the predictive distribution from a model.
This loss function works only when the conditional distribution p ( y | x ) substantially differs from the marginal distribution p ( y ) , since the model would resort to a trivial solution of predicting the marginal distribution regardless of the input x .
We describe this approach in more detail in Sec. 3.1, and test it against our proposal in the experiments.
Sequence prediction A sequence prediction problem is characterized as ﬁnding a mapping from an input x to a sequence of classes Y = (cid:0) y 1 , . . .
, y |Y| (cid:1) .
Representative examples of sequence prediction include machine translation, automatic speech recognition and other tagging problems, such as part-of-speech tagging, in natural language processing.
Similarly to multiset prediction, the input x is arbitrary, and an item in the class set C may appear more than once in the target sequence.
It is however different from multiset prediction in that there is a clear, predetermined order of items in the target sequence.
We detail this sequence prediction approach later in Sec. 3.2.
2. M ULTISET L OSS F UNCTION FOR M ULTISET P REDICTION
In this paper, we propose a novel loss function, called multiset loss , for the problem of multiset prediction.
This loss function is best motivated by treating the multiset prediction problem as a sequential decision making process with a model being considered a policy π .
This policy takes as input the input x and all the previously predicted classes ˆ y <t at time t , and outputs the distribution over the next class to be predicted.
That is, π θ ( y t | ˆ y <t , x ) .
This policy is parametrized with a set θ of parameters.
We ﬁrst deﬁne a free label multiset at time t as
Deﬁnition 1 (Free Label Multiset) .
[Equation]
ˆ y t − 1 is the prediction made by the policy at time t − 1 .
This free label multiset Y t contains all the items that remain to be predicted after t − 1 predictions by the policy.
We then construct an oracle policy π ∗ .
This oracle policy takes as input a sequence of predicted labels ˆ y <t and the input x .
It outputs a distribution whose entire probability (1) is evenly distributed over all the items in the free label multiset Y t .
In other words,
Deﬁnition 2 (Oracle) .
[Equation]
An interesting and important property of this oracle is that it is optimal given any preﬁx ˆ y <t with respect to both precision and recall.
This is intuitively clear by noticing that the oracle policy allows only a correct item to be selected.
We call this property the optimality of the oracle.
Remark 1.
Given an arbitrary preﬁx ˆ y <t ,
Prec
[Equation] [Equation]
and Rec
[Equation] [Equation]
≤ Rec
[Equation] [Equation]
, for any ˆ y ∼ π ∗ (ˆ y <t , x ) .
The proof is given in Appendix A.
From the remark above, it follows that the oracle policy is an optimal solution to the problem of multiset prediction in terms of precision and recall.
Remark 2.
Prec (ˆ y ≤|Y| , Y ) = 1 and Rec (ˆ y ≤|Y| , Y ) = 1 , for all ˆ y ≤|Y| ∼ π ∗ ( y 1 | x ) π ∗ ( y 2 | y 1 , x ) · · · π ∗ ( y |Y| | y < |Y| , x ) .
The proof can be found in Appendix B.
It is trivial to show that sampling from such an oracle policy would never result in an incorrect prediction.
That is, this oracle policy assigns zero probability to any sequence of predictions that is not a permutation of the target multiset.
Remark 3.
[Equation]
In short, this oracle policy tells us at each time step t which of all the items in the class set C must be selected.
This optimality allows us to consider a step-wise loss between a parametrized policy π θ and the oracle policy π ∗ , because the oracle policy provides us with an optimal decision regardless of the quality of the preﬁx generated so far.
We thus propose to minimize the KL divergence from the oracle policy to the parametrized policy at each step separately.
This divergence is deﬁned as
[Equation]
where H ( π t ∗ ) is the entropy of the oracle policy at time step t .
This entropy term can be safely ignored when learning π θ , since it is constant with respect to θ .
We deﬁne
[Equation]
and call it a per-step loss function.
We note that it is indeed possible to use another divergence in the place of the KL divergence.
It is intractable to minimize the per-step loss from Eq.
(2) for every possible state (ˆ y <t , x ) , since the size of the state space grows exponentially with respect to the size of a target multiset.
We thus propose here to minimize the per-step loss only for the state, deﬁned as a pair of the input x and the preﬁx ˆ y <t , visited by the parametrized policy π θ .
That is, we generate an entire trajectory (ˆ y 1 , . .
, ˆ y T ) by executing the parametrized policy until either all the items in the target multiset have been predicted or the predeﬁned maximum number of steps have passed.
Then, we compute the loss function at each time t based on ( x, ˆ y <t ) , for all t = 1 , . . .
, T .
The ﬁnal loss function is then the sum of all these per-step loss functions.
Deﬁnition 3 (Multiset Loss Function) .
[Equation]
where T is the smaller of the smallest t for which Y t = ∅ and the predeﬁned maximum number of steps allowed.
As was shown by Ross et al.
(2011), the use of the parametrized policy π θ instead of the oracle policy π ∗ allows the upper bound on the learned policy’s error to be linear with respect to the size of the target multiset.
If the oracle policy had been used, the upper bound would have grown quadratically with respect to the size of the target multiset.
To conﬁrm this empirically, we test the following three alternative strategies for executing the parametrized policy π θ in the experiments:
Greedy search: ˆ y t = arg max y log π θ ( y | ˆ y <t , x ) 2.
Stochastic sampling: ˆ y t ∼ π θ ( y | ˆ y <t , x ) 3.
Oracle sampling: ˆ y t ∼ π ∗ ( y | ˆ y <t , x )
Once the proposed multiset loss is minimized, we evaluate the learned policy by greedily selecting each item from the policy.
Variable-Sized Target Multiset
We have deﬁned the proposed loss function for multiset prediction while assuming that the size of the target multiset was known.
This is however a major limitation, and we relax this constraint by introducing an additional termination policy.
The termination policy π s outputs a stop distribution given the predicted sequence of items ˆ y <t and the input x .
Because the size of the target multiset is known during training, we simply train this termination policy in a supervised way using a binary cross-entropy loss.
At evaluation time, we simply threshold the predicted stop probability at a predeﬁned threshold (0.5).
O THER L OSS F UNCTIONS
In addition to the proposed multiset loss function, we propose three more loss functions for multiset prediction.
They serve as baselines in our experiments later.
3. A GGREGATED D ISTRIBUTION M ATCHING
In the case of distribution matching, we consider the target multiset Y as a set of samples from a single, underlying distribution q ∗ over the class set C .
This underlying distribution can be empirically estimated by counting the number of occurrences of each item c ∈ C in Y .
That is,
[Equation]
where I is the indicator function as before.
Similarly, we can construct an aggregated distribution computed by the parametrized policy π θ .
As with the proposed multiset loss in Def. 3, we ﬁrst execute π θ to predict a multiset ˆ Y .
This is converted into an aggregated distribution q θ in the same way as we turned the target multiset into the oracle aggregate distribution.
Learning is equivalent to minimizing the divergence between these two distributions.
In this paper, we test two types of divergences.
The ﬁrst one is from a family of L p distances deﬁned as
[Equation]
where q ∗ and q are the vectors representing the corresponding categorical distributions.
The other is a usual KL divergence deﬁned earlier in Eq.
[Equation]
One major issue with this approach is that minimizing the divergence between the aggregated distributions does not necessarily result in the optimal policy (see the oracle policy in Def. 2.)
That is, a policy that minimizes this loss function may assign non-zero probability to an incorrect sequence of predictions, unlike the oracle policy.
This is due to the invariance of the aggregated distribution to the order of predictions.
Later when analyzing this loss function, we empirically notice that a learned policy often has a different behaviour from the oracle policy, for instance, reﬂected by the increasing entropy of the action distribution over time.
One-Step Variant We can train an one-step predictor with this aggregate distribution matching criterion, instead of learning a policy π θ .
That is, a predictor outputs both a point q θ ( ·| x ) in a | C | - dimensional simplex and the size ˆ l θ ( x ) of the target multiset.
Then, for each unique item c ∈ C , the number of its occurrences in the predicted multiset ˆ Y is
[Equation]
The corresponding loss function is then
[Equation]
where λ > 0 is a coefﬁcient for balancing the contributions from the two terms.
A major weakness of this one-step variant, compared to the approaches based on sequential decision making, is the lack of modelling dependencies among the items in the predicted multiset.
We test this approach in the experiments later and observe this lack of output dependency modelling results in substantially worse prediction accuracy.
3. S EQUENCE PREDICTION WITH A PREDEFINED ORDER
All the loss functions deﬁned so far have not relied on the availability of an existing order of items in a target multiset.
However by turning the problem of multiset prediction into sequential decision making, minimizing such a loss function is equivalent to capturing an order of items in the target multiset implicitly.
Here, we instead describe an approach based on explicitly deﬁning an order in advance.
This will serve as a baseline later in the experiments.
We ﬁrst deﬁne a rank function r that maps from one of the unique items in the class set c ∈ C to a unique integer.
That is, r : C → Z .
This function assigns the rank of each item and is used to order items y i in a target multiset Y .
This results in a sequence S
=
( s 1 , . . .
, s |Y| ) , where r ( s i ) ≥ r ( s j ) for all j > i , and s i ∈ Y .
With this target sequence S created from Y using the rank function r , we deﬁne a sequence loss function as
[Equation]
Minimizing this loss function is equivalent to maximizing the conditional log-probability of the sequence S given x .
This sequence loss function has two clear disadvantages.
First, it does not take into account the actual behaviour of the policy π θ (see, e.g., Bengio et al., 2015; Daum´e III & Marcu, 2005; Ross et al., 2011).
This makes a learned policy potentially vulnerable to cascading error at test time.
Second and more importantly, this loss function requires a pre-speciﬁed rank function r .
Because multiset prediction does not come with such a rank function by deﬁnition, we must design an arbitrary rank function, and the ﬁnal performance varies signiﬁcantly based on the choice.
We demonstrate this variation in section 4.3.
Input-Dependent Rank Function When the input x has a well-known structure, and an object within the input for each item in the target multiset is annotated, it is possible to devise a rank function per input.
A representative example is an image input with bounding box annotations.
Here, we present two input-dependent rank functions in such a case.
First, a spatial rank function r spatial assigns an integer rank to each item in a given target multiset Y such that
[Equation]
where x i and x j are the objects corresponding to the items y i and y j .
Second, an area rank function r area decides the rank of each label in a target multiset according to the size of the corresponding object inside the input image:
[Equation]
The area may be determined based on the size of a bounding box or the number of pixels, depending on the level of annotation.
We test these two image-speciﬁc input-dependent rank functions against a random rank function in the experiments.
3. R EINFORCEMENT L EARNING
In (Welleck et al., 2017), an approach based on reinforcement learning was proposed for multiset prediction.
Instead of assuming the existence of an oracle policy, this approach solely relies on a reward function r designed speciﬁcally for multiset prediction.
The reward function is deﬁned as
[Equation]
The goal is then to maximize the sum of rewards over a trajectory of predictions from a parametrized policy π θ .
The ﬁnal loss function is
[Equation]
where the second term inside the expectation is the negative entropy multiplied with a regularization coefﬁcient λ .
The second term encourages the exploration during training.
As in (Welleck et al.,
2017), we use REINFORCE (Williams, 1992) to stochastically minimize the loss function above with respect to π θ .
This loss function is optimal in that the return, i.e., the sum of the step-wise rewards, is maximized when both the precision and recall are maximal ( = 1 ).
In other words, the oracle policy, deﬁned in
Def. 2, maximizes the expected return.
This approach of reinforcement learning is however known to be difﬁcult, with a high variance (Peters & Schaal, 2008).
Especially, in this case, the size of the state space grows exponentially with respect to the size of the target multiset, and the action space of each step is as large as the number of unique items in the class set.
E XPERIMENTS AND A NALYSIS
In this section, we extensively evaluate the proposed multiset loss function against various baseline loss functions presented throughout this paper.
More speciﬁcally, we focus on its applicability and performance on image-based multiset prediction.
4. D ATASETS
MNIST Multi MNIST Multi is a class of synthetic datasets.
Each dataset consists of multiple 100x100 images, each of which contains a varying number of digits from the original MNIST (LeCun et al., 1998).
We vary the size of each digit and also add clutters.
In the experiments, we consider the following variants of MNIST Multi:
• MNIST Multi (4) : |Y| = 4 , 20-50 pixel digits • MNIST Multi (1-4) : |Y| ∈ { 1 , . . .
, 4 } , 20-50 pixel digits • MNIST Multi (10) : |Y| = 10 , 20 pixel digits
Each dataset has a training set with 70,000 examples and a test set with 10,000 examples.
We randomly sample 7,000 examples from the training set to use as a validation set, and train with the remaining 63,000 examples.
MS COCO As a real-world dataset, we use Microsoft COCO (Lin et al., 2014) which includes natural images with multiple objects.
Compared to MNIST Multi, each image in MS COCO has objects of more varying sizes and shapes, and there is a large variation in the number of object instances per image which spans from 1 to 91.
The problem is made even more challenging with many overlapping and occluded objects.
To control the difﬁculty in order to better study the loss functions, we create the following two variants:
• COCO Easy : |Y| = 2 , 10,230 training examples, 24 classes • COCO Medium : |Y| ∈ { 1 , . . .
, 4 } , 44,121 training examples, 23 classes
In both of the variants, we only include images whose |Y| objects are large and of common classes.
An object is deﬁned to be large if the object’s area is above the 40-th percentile across the train set of MS COCO.
After reducing the dataset to have |Y| large objects per image, we remove images containing only objects of rare classes.
A class is considered rare if its frequency is less than 1 | C | , where C is the class set.
These two stages ensure that only images with a proper number of large objects are kept.
We do not use ﬁne-grained annotation (pixel-level segmentation and bounding boxes) except for creating input-dependent rank functions from Sec. 3.2.
For each variant, we hold out a randomly sampled 15% of the training examples as a validation set.
We form separate test sets by applying the same ﬁlters to the COCO validation set.
The test set sizes are 5,107 for COCO Easy and 21,944 for COCO Medium.
4. M ODELS
MNIST Multi We use three convolutional layers of channel sizes 10, 10 and 32, followed by a convolutional long short-term memory (LSTM) layer (Xingjian et al., 2015).
At each step, the feature map from the convolutional LSTM layer is average-pooled spatially and fed to a softmax classiﬁer.
In the case of the one-step variant of aggregate distribution matching, the LSTM layer is skipped.
MS COCO We use a ResNet-34 (He et al., 2016) pretrained on ImageNet (Deng et al., 2009) as a feature extractor.
The ﬁnal feature map from this ResNet-34 is fed to a convolutional LSTM layer, as described for MNIST Multi above.
We do not ﬁnetune the ResNet-34 based feature extractor.
[Table]
Training and evaluation For each loss, a model was trained for 200 epochs (350 for
MNIST Multi 10).
After each epoch, exact match was computed on the validation set.
The model state with the highest validation exact match was used for evaluation on the test set.
When evaluating a trained policy, we use greedy decoding and auxiliary stop prediction for determining the size of a predicted multiset.
Each predicted multiset is compared against the ground-truth target multiset, and we report both the accuracy based on the exact match (EM) and
F-1 score (F1).
More details about the model architectures and training are in the Appendix C.
4. E XPERIMENT 1: I NFLUENCE OF A R ANK F UNCTION ON S EQUENCE P REDICTION
[Table]
First, we investigate the sequence loss function
L seq from Sec.
3.2, while varying a rank function.
We test three alternatives: a random rank function 3 r and two input-dependent rank functions r spatial and r area .
We compare these rank functions on MNIST Multi (4) and COCO Easy validation sets.
We present the results in Table 1.
It is clear from the results that the performance of the sequence prediction loss function is dependent on the choice of a rank function.
In the case of MNIST
Multi, the area-based rank function was far worse than the other choices.
This however was not true with COCO Easy, on which the spatial rank function was worst among the three.
In both cases, we have observed that the random rank function performed best, and from here on, we use the random rank function in the remaining experiments.
This set of experiments ﬁrmly suggests the need of an order-invariant multiset loss function, such as the multiset loss function proposed in this paper.
4. E XPERIMENT 2: E XECUTION S TRATEGIES FOR THE M ULTISET L OSS F UNCTION
In this set of experiments, we compare the three execution strategies for the proposed multiset loss function, illustrated in Sec.
They are greedy decoding, stochastic sampling and oracle sampling.
We test them on MNIST Multi (10) and COCO Easy.
As shown in Table 2, greedy decoding and stochastic sampling, both of which consider states that are likely to be visited by the parametrized policy, outperform the oracle sampling.
This is consistent with the theory by Ross et al.
Although the ﬁrst two strategies perform comparably to each other, across both of the datasets and the two evaluation metrics, greedy decoding tends to outperform stochastic sampling.
We conjecture this is due to better matching between training and
[Table]
testing in the case of greedy decoding.
Thus, from here on, we use greedy decoding when training a model with the proposed multiset loss function.
4. E XPERIMENT 3: L OSS F UNCTION C OMPARISON
We now compare the proposed multiset loss function against the ﬁve baseline loss functions: reinforcement learning L RL , aggregate distribution matching– L 1 dm and L KLdm –, its one-step variant L 1-step , and sequence prediction L seq .
MNIST Multi We present the results on the MNIST Multi variants in Table 3.
On all three variants and according to both metrics, the proposed multiset loss function outperforms all the others.
The reinforcement learning based approach closely follows behind.
Its performance, however, drops as the number of items in a target multiset increases.
This is understandable, as the variance of policy gradient grows as the length of an episode grows.
A similar behaviour was observed with sequence prediction as well as aggregate distribution matching.
We were not able to train any decent models with the one-step variant of aggregate distribution matching.
This was true especially in terms of exact match (EM), which we attribute to the one-step variant not being capable of modelling dependencies among the predicted items.
[Table]
[Table]
MS COCO Similarly to the results on the variants of MNIST Multi, the proposed multiset loss function matches or outperforms all the others on the two variants of MS COCO, as presented in Table 4.
On COCO Easy, with only two objects to predict per example, both aggregated distribution matching (with KL divergence) and the sequence loss functions are as competitive as the proposed multiset loss.
The other loss functions signiﬁcantly underperform these three loss functions, as they did on
MNIST Multi.
The performance gap between the proposed loss and the others, however, grows substantially on the more challenging COCO Medium, which has more objects per example.
The proposed multiset loss outperforms the aggregated distribution matching with KL divergence by 3.7 percentage points on exact match and 4.8 on F1.
This is analogous to the experiments on the MNIST Multi variants, where the performance gap increased when moving from four to ten digits.
4. A NALYSIS : E NTROPY E VOLUTION
One property of the oracle policy deﬁned in Sec.
2.2 is that the entropy of the predictive distribution strictly decreases over time, i.e., H π ∗ ( y
| ˆ y <t , x ) >
H π ∗ ( y | ˆ y ≤ t , x ) .
This is a natural consequence from the fact that there is no pre-speciﬁed rank function, because the oracle policy cannot prefer any item from the others in a free label multiset.
Hence, we examine here how the policy learned based on each loss function compares to the oracle policy in terms of per-step entropy.
We consider the policies trained on MNIST Multi (10), where the differences among them were most clear.
As shown in Fig.
1, the policy trained on MNIST Multi (10) using the proposed multiset loss closely follows the oracle policy.
The entropy decreases as the predictions are made.
The decreases can be interpreted as concentrating probability mass on progressively smaller free labels sets.
The variance is quite small, indicating that this strategy is uniformly applied for any input.
The policy trained with reinforcement learning retains a relatively low entropy across steps, with a decreasing trend in the second half.
We carefully suspect the low entropy in the earlier steps is due to the greedy nature of policy gradient.
The policy receives a high reward more easily by choosing one of many possible choices in an earlier step than in a later step.
This effectively discourages the policy from exploring all possible trajectories during training.
On the other hand, the policy found by aggregated distribution matching ( L KLdm ) has the opposite behaviour.
The entropy in general grows as more predictions are made.
To see why this is sub-optimal, consider the ﬁnal (10th) step.
Assuming the ﬁrst nine predictions { ˆ y 1 , ..., ˆ y 9 } were correct (i.e. they form a subset of Y ), there is only one correct class left for the ﬁnal prediction ˆ y 10 .
The high entropy however indicates that the model is placing a signiﬁcant amount of probability on incorrect sequences.
We believe such a policy is found by minimizing the aggregated distribution matching loss function because it cannot properly distinguish between policies with increasing and decreasing entropies.
The increasing entropy also indicates that the policy has learned a rank function implicitly and is fully relying on it.
Given some unknown free label multiset, inferred from the input, this policy uses the implicitly learned rank function to choose one item from this set.
We conjecture this reliance on an inferred rank function, which is by deﬁnition sub-optimal, 4 resulted in lower performance of aggregate distribution matching.
C ONCLUSION
We have extensively investigated the problem of multiset prediction in this paper.
We rigorously deﬁned the problem, and proposed to approach it from the perspective of sequential decision making.
In doing so, an oracle policy was deﬁned and shown to be optimal, and a new loss function, called multiset loss , was introduced as a means to train a parametrized policy for multiset prediction.
The experiments on two families of datasets, MNIST Multi variants and MS COCO variants, have revealed the effectiveness of the proposed loss function over other loss functions including reinforcement learning, sequence, and aggregated distribution matching loss functions.
The success of the proposed multiset loss brings in new opportunities of applying machine learning to various new domains, including high-energy physics.
A P ROOF OF R EMARK
Proof.
Note that the precision with ˆ y <t is deﬁned as
Prec (ˆ y <t , Y ) =
(cid:80)
y ∈
ˆ y <t
I y ∈Y | ˆ y <t | .
Because ˆ y ∼ π ∗ (ˆ y <t , x )
∈
Y t ,
Prec (ˆ y ≤ t , Y ) =
1 + (cid:80)
y ∈
ˆ y <t
I y ∈Y 1 + | ˆ y <t | .
Then,
Prec (ˆ y ≤ t , Y ) − Prec (ˆ y <t , Y ) =
1 − Prec (ˆ y <t , Y ) 1 + | ˆ y <t | ≥ 0 , because 0 ≤ Prec (ˆ y <t , Y ) ≤ 1 and | ˆ y <t | ≥ 0 .
The equality holds when Prec (ˆ y <t , Y )
= 1 .
Similarly, we start with the deﬁnition of the recall:
[Equation]
Because ˆ y ∼ π ∗ (ˆ y <t , x )
∈
Y t ,
[Equation]
Since the denominator is identical,
[Equation]
B P ROOF OF R EMARK
Proof.
When t = 1 ,
Prec (ˆ y ≤
[Equation]
, Y ) = 1 , because ˆ y 1 ∼ π ∗ ( ∅ , x ) ∈ Y .
From Remark 1, we know that
Prec (ˆ y ≤ t , Y ) =
Prec (ˆ y <t , Y ) , when Prec (ˆ y <t , Y ) = 1 .
By induction, Prec (ˆ y ≤|Y| , Y ) = 1 .
From the proof of Remark 1, we know that the recall increases by 1 Y each time, and we also know that
[Equation]
when t = 1 .
After |Y| − 1 steps of executing the oracle policy, the recall becomes
[Equation]
C M ODEL D ESCRIPTIONS
Model An input x is ﬁrst processed by a tower of convolutional layers, resulting in a feature volume of size w (cid:48) × h (cid:48) with d feature maps, i.e., H = φ ( x ) ∈ R w (cid:48)
× h (cid:48)
× d .
At each time step t , we resize the previous prediction’s embedding emb (ˆ y t − 1 )
∈
R ( w (cid:48) )
( h (cid:48) ) to be a w (cid:48) × h (cid:48) tensor and concatenate it with H , resulting in ˜ H ∈ R w (cid:48)
× h (cid:48) ×
( d +1) .
This feature volume is then fed into a stack of convolutional LSTM layers.
The output from the ﬁnal convolutional LSTM layer C ∈ R w (cid:48) × h (cid:48)
× q is spatially average-pooled, i.e., c = 1 w (cid:48) h (cid:48) (cid:80) w (cid:48)
i =1 (cid:80) h (cid:48)
j =1 C i,j, · ∈ R q .
This feature vector c is then turned into a conditional distribution over the next item after afﬁne transformation followed by a softmax function.
When the one-step variant of aggregated distribution matching is used, we skip the convolutional LSTM layers, i.e., c = 1 w (cid:48) h (cid:48) (cid:80) w (cid:48) i =1 (cid:80) h (cid:48)
j =1 H i,j, · ∈ R d .
See Fig.
2 for the graphical illustration of the entire network.
See Table 5 for the details of the network for each dataset.
[Table]
Preprocessing For MNIST Multi, we do not preprocess the input at all.
In the case of MS COCO, input images are of different sizes.
Each image is ﬁrst resized so that its larger dimension has 600 pixels, then along its other dimension is zero-padded to 600 pixels and centered, resulting in a 600x600 image.
Training The model is trained end-to-end, except ResNet-34 which remains ﬁxed after being pretrained on ImageNet.
For all the experiments, we train a neural network using Adam (Kingma & Ba, 2014) with a ﬁxed learning rate of 0.001, β of (0.9, 0.999) and (cid:15) of 1e-8.
The learning rate was selected based on the validation performance during the preliminary experiments, and the other parameters are the default values.
For MNIST Multi, the batch size was 64, and for COCO was 32.