L EARNING E FFICIENT O NLINE 3D B IN P ACKING ON P ACKING C ONFIGURATION T REES
Anonymous authors
Paper under double-blind review
A BSTRACT
Online 3D Bin Packing Problem (3D-BPP) has widespread applications in industrial automation and has aroused enthusiastic research interest recently.
Existing methods usually solve the problem with limited resolution of spatial discretization, and/or cannot deal with complex practical constraints well.
We propose to enhance the practical applicability of online 3D-BPP via learning on a novel hierarchical representation – packing conﬁguration tree (PCT).
PCT is a full-ﬂedged description of the state and action space of bin packing which can support packing policy learning based on deep reinforcement learning (DRL).
The size of the packing action space is proportional to the number of leaf nodes, i.e. candidate placements, making the DRL model easy to train and well-performing even with continuous solution space.
During training, PCT expands based on heuristic rules, however, the DRL model learns a much more effective and robust packing policy than heuristic methods.
Through extensive evaluation, we demonstrate that our method outperforms all existing online BPP methods and is versatile in terms of incorporating various practical constraints.
I NTRODUCTION
As one of the most classic combinatorial optimization problems, the 3D bin packing problem usually refers to packing a set of cuboid-shaped items i ∈
I , with sizes s xi , s yi , s zi along x, y, z axes, respectively, into the minimum number of bins with sizes S x , S y , S z , in an axis-aligned fashion.
Traditional 3D-BPP assumes that all the items to be packed are known a priori (Martello et al., 2000), which is also called ofﬂine BPP.
The problem is known to be strongly NP-hard (De Castro Silva et al., 2003).
However, in many real-world application scenarios, e.g., logistics or warehousing (Wang & Hauser, 2019a), the upcoming items cannot be fully observed; only the current item to be packed is observable.
Packing items without the knowledge of all upcoming items is referred to as online BPP (Seiden, 2002).
Due to its obvious practical usefulness, online 3D-BPP has received increasing attention recently.
Given the limited knowledge, the problem cannot be solved by usual search-based methods.
Different from ofﬂine 3D-BPP where the items can be placed in an arbitrary order, the placement of online BPP must follow the coming order of the items, which imposes additional constraints and difﬁculty to achieve comparable utilization.
Online 3D-BPP is usually solved with either heuristic methods (Ha et al., 2017) or learning-based ones (Zhao et al., 2021a), with complementary pros and cons.
Heuristic methods are generally not limited by the size of action space, but they ﬁnd difﬁculties in handling complex practical constraints such as packing stability or speciﬁc packing preferences.
Learning-based approaches usually perform better than heuristic methods, especially under various complicated constraints.
However, the learning is hard to converge with a large action space, which has greatly limited the applicability of learning-based methods due to, e.g., the limited resolution of spatial discretization (Zhao et al., 2021a).
We propose to enhance learning-based online 3D-BPP towards practical applicability through learning with a novel hierarchical representation – packing conﬁguration tree (PCT) .
PCT is a dynamically growing tree where the internal nodes describe the space conﬁgurations of packed items and leaf nodes the packable placements of the current item.
PCT is a full-ﬂedged description of the state and action space of bin packing which can support packing policy learning based on deep reinforcement learning (DRL).
We extract state features from PCT using graph attention networks (Velickovic et al., 2018) which encodes the spatial relations of all space conﬁguration nodes.
The state feature is input into the actor and critic networks of the DRL model.
The actor network, designed based on pointer mechanism, weighs the leaf nodes and outputs the action (the ﬁnal placement).
During training, PCT grows under the guidance of heuristics such as Corner Point (Martello et al., 2000), Extreme Point (Crainic et al., 2008), and Empty Maximal Space (Ha et al., 2017).
Although
PCT is expanded with heuristic rules, conﬁning the solution space to what the heuristics could explore, our DRL model learns a discriminant ﬁtness function (the critic network) for candidate actions, resulting in an effective and robust packing policy exceeding the heuristic methods.
Furthermore, the size of the packing action space is proportional to the number of leaf nodes, making the DRL model easy to train and well-performing even with continuous solution space.
Through extensive evaluation, we demonstrate that our method outperforms all existing online 3D-BPP methods and is versatile in terms of incorporating various practical constraints such as isle friendliness and load balancing (Gzara et al., 2020).
To the best of our knowledge, we are the ﬁrst ones to deploy the learning-based method on solving online 3D-BPP with continuous solution space successfully.
R ELATED W ORK
Ofﬂine 3D-BPP
The early interest of 3D-BPP mainly focused on its ofﬂine setting.
Ofﬂine 3DBPP assumes that all items are known as a priori and can be placed in an arbitrary order.
Martello et al.
(2000) ﬁrst solved this problem with an exact branch-and-bound approach.
Limited by exponential worst-case complexity of exact approaches, lots of heuristic and meta-heuristic algorithms are proposed to get an approximate solution quickly, such as guided local search (Faroe et al., 2003), tabu search (Crainic et al., 2009), and hybrid genetic algorithm (Kang et al., 2012).
Hu et al.
(2017) decompose the ofﬂine 3D-BPP into packing order decisions and online placement decisions.
The packing order is optimized with an end-to-end DRL agent and the online placement policy is a handdesigned heuristic.
This two-step fashion is widely accepted and followed by Duan et al.
(2019), Hu et al.
(2020), and Zhang et al.
Although ofﬂine 3D-BPP has been well studied, their searchbased approaches cannot be directly transferred to the online setting.
Instead, lots of heuristic methods have been proposed to solve this problem.
For reasons of simplicity and good performance, the deep-bottom-left (DBL) heuristic (Karabulut & Inceoglu, 2004) has long been a favorite.
Ha et al.
(2017) sort the empty spaces with this DBL order and place the current item into the ﬁrst ﬁt one.
Wang & Hauser (2019b) propose a Heightmap-Minimization method to minimize the volume increase of the packed items as observed from the loading direction.
Hu et al.
(2020) optimize the empty spaces available for the packing future with a Maximize-Accessible-Convex-Space method.
DRL for Online 3D-BPP
The heuristic methods are intuitive to implement and can be easily applied to various scenarios.
However, the price of good ﬂexibility is that all these methods perform mediocrely, especially for online 3D-BPP instances with speciﬁc constraints.
Designing new heuristics for speciﬁc classes of 3D-BPP is heavy work since this problem has an NP-hard solution space, many situations need to be premeditated manually by trial and error.
Substantial domain knowledge is also necessary to ensure the safety and reliability of the algorithm.
To automatically generate a policy that works well on speciﬁed online 3D-BPP instances, Zhao et al.
(2021a) have ﬁrst employed the DRL method on solving this problem, however, their method only works in a small discrete coordinate space.
Despite its limitations, this work is soon followed by Hong et al.
and Zhao et al.
(2021b) for logistics robot implementation.
Zhang et al.
(2021) adopt a similar online placement policy for ofﬂine needs referring to Hu et al.
All these learning-based methods only work in a grid world with limited discretization accuracy, which reduces their practical applicability.
Practical Constraints
The majority of literature for 3D-BPP (Martello et al., 2000) only considers the basic non-overlapping constraint 1 and containment constraint 2:
[Equation]
Where p means the front-left-bottom coordinate of item i and d the coordinate axis, e ij takes value 1 otherwise 0 if item i precedes item j along d .
The algorithms for 3D-BPP are of limited practical applicability if no even basic real-world constraints, e.g., stability (Ramos et al., 2016), are considered.
Zhao et al.
(2021b) propose a fast stability estimation method for DRL training and test their learned policy with real logistics boxes.
The ﬂaw of their work is the heightmap (the upper frontier of packed items) state representation like Zhang et al.
(2021) is still used, while the underlying [Figure] Current node Packed item
Packed node 𝒕 =𝒕 =𝒕 =
[Figure]
𝒕 [Figure] x z o constraints between packed items are missed. The unavailability of underlying spatial information makes their problem a partially observable Markov Decision Process (Spaan, 2012) which is not conducive to DRL training and limits the performance on 3D-BPP instances with more complex practical constraints, like isle friendliness and load balancing (Gzara et al., 2020).
M ETHOD
In this section, we ﬁrst introduce our PCT concept in Section 3.1 for describing the online packing process.
The parameterization of the tree structure and the leaf node selection policy are introduced in Section 3.2 and Section 3.3 respectively.
In Section 3.4, we formulate online 3D-BPP as Markov
Decision Process based on the PCT, followed by the description of the training method.
3. P ACKING C ONFIGURATION T REE
When a rectangular item n t is added to a given packing with position ( p xn , p yn , p zn ) at time step t , it introduces a series of new candidate positions where future items can be accommodated, as illustrated in Figure 1. Combined with the axis-aligned orientation o ∈ O for n t based on these positions, we get candidate placements (i.e. position and orientation).
The packing process can be seen as a placement node being replaced by a packed item node, and new candidate placement nodes are generated as children.
As the packing time step t goes on, these nodes are iteratively updated and a dynamic packing conﬁguration tree is formed, denoted as T .
The internal node set B t ∈ T t represents the space conﬁgurations of packed items, and the leaf node set L t ∈ T t the packable candidate placements.
During the packing, leaf nodes that are no longer feasible, e.g., covered by packed items, will be removed from L t .
When there is no packable leaf node that makes n t satisfy the constraints of placement, the packing episode ends.
Without loss of generality, we stipulate a vertical top-down packing within a single bin (Wang & Hauser, 2019b).
Traditional 3D-BPP literature only cares about the remaining placements for accommodating the current item n t , their packing policies can be written as π ( L t | L t , n t ) .
If we want to promote this problem for practical demands, 3D-BPP needs to satisfy more complex practical constraints which also act on B t .
Taking packing stability for instance, a newly added item has possibly force and torque effect on the whole item set B t (Ramos et al., 2016).
The addition of n t should make B t a more stable spatial distribution so that more items can be added in the future.
Therefore, our packing policy over L t is deﬁned as π ( L t |T t , n t ) , which means probabilities of selecting leaf nodes from L t .
For online packing, we hope to ﬁnd the best leaf node selection policy to expand the PCT with more relaxed constraints so that more future items can be appended.
Leaf Node Expansion Schemes
The performance of online 3D-BPP policies is strongly affected by the choice of leaf node expansion schemes – which incrementally calculate new candidate placements introduced by the just placed item n t .
A good expansion scheme should reduce the number of solutions to be explored while not missing too many feasible packings.
Meanwhile, polynomially computability is also expected.
Designing such a scheme from scratch is non-trivial.
Fortunately, several placement rules independent from particular packing problems have been proposed, such as Corner Point (Martello et al., 2000), Extreme Point (Crainic et al., 2008), and Empty Maximal
Space (Ha et al., 2017).
We extend these schemes which have proven to be accurate and efﬁcient to our PCT expansion.
The performance of learned policies will be reported in Section 4.1.
3. T REE R EPRESENTATION
Given the bin conﬁguration T t and current item n t , the packing policy can be parameterized as π ( L t |T t , n t ) .
The tuple ( T t , n t ) can be treated as a graph and encoded by Graph Neural Networks (GNNs)
(Gori et al., 2005).
Speciﬁcally, the PCT keeps growing with time step t and cannot be embedded by spectral-based approaches (Bruna et al., 2014) which require a ﬁxed graph structure.
We adopt non-spectral Graph Attention Networks (GATs) (Velickovic et al., 2018), which require no priori on graph structures.
The raw space conﬁguration nodes B t , L t , n t are presented by descriptors in different formats.
We use three independent node-wise Multi-Layer Perceptron (MLP) blocks to project these heterogeneous descriptors into the homogeneous node features: ˆ h = { φ θ B ( B t ) , φ θ L ( L t ) , φ θ n ( n t ) } ∈ R d h × N , d h is the dimension of each node feature and φ θ is an MLP block with its parameters θ .
The feature number N should be | B t | + | L t | + 1 , which is a variable.
The GAT layer is used to transform ˆ h into high-level node features.
The Scaled Dot-Product Attention (Vaswani et al., 2017) is applied to each node for calculating the relation weight of one node to another.
These relation weights are normalized and used to compute the linear combination of features ˆ h .
The feature of node i embedded by the GAT layer can be represented as:
[Equation]
Where W Q ∈ R d k × d h , W K ∈ R d k × d h , W V ∈ R d v × d h
[Equation]
and W O ∈ R d h × d v are projection matrices, d k and d v are dimensions of projected features.
The softmax operation normalizes the relation weight between node i and node j .
The initial feature ˆ h is embedded by a GAT layer and the skip-connection operation (Vaswani et al., 2017) is followed to get the ﬁnal output features h :
[Equation]
Where φ FF is a node-wise Feed-Forward MLP with output dimension d h and h (cid:48) is an intermediate variable.
Equation 4 can be seen as an independent block and be repeated multiple times with different parameters.
We don’t extend GAT to employ the multi-head attention mechanism (Vaswani et al., 2017) since we ﬁnd that additional attention heads cannot help the ﬁnal performance. We execute Equation 4 once and we set d v = d k .
More implementation details are provided in Appendix A.
3. L EAF N ODE S ELECTION
Given the node features h , we need to decide the leaf node indices for accommodating the current item n t .
Since the leaf nodes vary as the PCT keeps growing over time step t , we use a pointer mechanism (Vinyals et al., 2015) which is context-based attention over variable inputs to select a leaf node from L t .
We still adopt Scaled Dot-Product Attention for calculating pointers, the global context feature ¯ h is aggregated by a mean operation on h : ¯ h = 1 N (cid:80) N i =1 h i .
The global feature ¯ h is projected to a query q by matrix W q ∈ R d k × d h and the leaf node features h L are utilized to calculate a set of keys k L by W k ∈ R d k × d h .
The compatibility u L of the query with all keys are:
[Equation]
Here h i only comes from h L .
The compatibility vector u L ∈ R | L t | represents the leaf node selection logits.
The probability distribution over the PCT leaf nodes L t is: [Equation]
[Equation]
Following Bello et al.
(2017), the compatibility logits are clipped with tanh, where the range is controlled by hyperparameter c clip , and ﬁnally normalized by a softmax operation.
3. M ARKOV D ECISION P ROCESS F ORMULATION
The online 3D-BPP decision at time step t only depends on the tuple ( T t , n t ) and can be formulated as Markov Decision Process, which is constructed with state S , action A , transition P , and reward R .
We solve this MDP with an end-to-end DRL agent.
The MDP model is formulated as follows.
State The state s t at time step t is represented as s t =
( T t , n t ) , where T t consists of the internal nodes B t and the leaf nodes L t .
Each internal node b ∈ B t is a spatial conﬁguration of sizes ( s xb , s yb , s zb ) and coordinates ( p xb , p yb , p zb ) corresponding to a packed item.
The current item n t is a size tuple ( s xn , s yn , s zn ) .
Extra properties will be appended to b and n t for speciﬁc packing preferences, such as density, item category, etc.
The descriptor for leaf node l ∈ L t is a placement vector of sizes ( s xo , s yo , s zo ) and position coordinates ( p x , p y , p z ) , where ( s xo , s yo , s zo ) indicates the sizes of n t along each dimension after an axis-aligned orientation o ∈ O .
Only the packable leaf nodes which satisfy placement constraints are provided.
Action The action a t ∈ A is the index of the selected leaf node l , denoted as a t = index ( l ) .
The action space A has the same size as L t .
A surge of learning-based methods (Zhao et al., 2021a) directly learn their policy on a grid world through discretizing the full coordinate space, where |A| grows explosively with the accuracy of the discretization.
Different from these existing works, our action space solely depends on the choice of leaf node expansion scheme and the packed items B t .
Therefore, our method can also be used to solve online 3D-BPP with continuous solution space, i.e. solution space where the packing coordinates are continuous values and free of spatial discretization.
We also ﬁnd that even if only an intercepted subset L sub ∈ L t is provided, our method can still maintain a good performance.
Transition The transition P ( s t +1 | s t ) is jointly determined by the current policy π and the probability distribution of sampling items.
Our online sequence is generated on-the-ﬂy from an item set I in a uniform distribution.
The generalization performance of our method on item sampling distributions different from the training one is discussed in Section 4.4.
Reward Our reward function R is deﬁned as r t =
c r · w t once n t is inserted into PCT as an internal node successfully, otherwise, r t =
and the packing episode ends.
Here c r is a constant and w t is the weight of n t .
The choice of w t depends on the customized needs.
For simplicity and clarity, unless otherwise noted, we set w t as the volume occupancy v t of n t , where v t =
s xn · s yn · s zn .
Training Method A DRL agent seeks for a policy π ( a t | s t ) to maximize the accumulated discounted reward: J ( π ) =
E τ ∼ π
[ (cid:80) ∞ t =
γ t R ( s t , a t )] .
Where γ ∈ [0 , 1] is the discount factor, and τ = ( s 0 , a 0 , s 1 , . . . )
is a sampled trajectory.
Our DRL agent is trained with Actor-Critic using Kronecker-Factored Trust Region (ACKTR) (Wu et al., 2017), which has proven to perform surprisingly well on online packing problems (Zhao et al., 2021a).
The actor weighs the leaf nodes L t and outputs the policy distribution π θ ( L t |T t , n t ) .
The critic maps the global context ¯ h into a state value prediction to predict how much accumulated discount reward the agent can get from t and helps the training of the actor.
The action a t is sampled on π θ ( L t |T t , n t ) for training and we take the argmax of the policy for the test.
[Figure]
ACKTR runs multiple parallel processes for gathering on-policy training samples.
The node number N of each sample varies with the time step t and the packing sequences of each process.
For batch calculation, we fullﬁll PCT to a ﬁxed length with dummy nodes, as illustrated by Figure 2 (a).
These redundant nodes are eliminated by masked attention (Velickovic et al., 2018) during the feature calculation of GAT.
The aggregation of h only happens on the eligible nodes.
For preserving node spatial relations, state s t is embedded by GAT as a fully connected graph as
Figure 2 (b), without any inner mask operation.
More implementation details are provided in Appendix A.
E XPERIMENTS
In this section, we ﬁrst report the performance of our PCT model combined with different leaf node expansion schemes.
Then we will demonstrate the beneﬁts of the PCT structure: better node spatial relation representations as well as more ﬂexible action space.
Finally, we test the generalization performance of our method on item sampling distributions different from the training one and extend it to other online 3D-BPP with complex practical constraints.
Baselines Although there are very few online packing implementations publicly available, we still do our best to collect or reproduce various online 3D-BPP algorithms, both heuristic and learningbased, from potentially relevant literature.
We help the heuristic methods to make a pre-judgment of placement constraints, e.g., stability, in case of premature downtime.
The learning-based agents are trained until there are no signiﬁcant performance gains.
Although both Zhao et al.
(2021b)
and Zhao et al.
(2021a) are learning-based methods recently proposed, we only compare with the former since it is the upgrade of the latter.
We also compare with the reported results of Zhang et al.
(2021) under the same setting.
All methods are implemented in Python and tested on 2000 instances with a desktop computer equipped with a Gold 5117 CPU and a GeForce TITAN V GPU.
The source code of our method along with all baselines will be published after acceptance for domain development.
Datasets Some baselines like Karabulut & Inceoglu (2004) need to traverse the entire coordinate space to ﬁnd the optimal solution, and the running costs explode as the spatial discretization accuracy increases.
To ensure that all algorithms are runnable within a reasonable period, we use the discrete dataset proposed by Zhao et al.
(2021a) without special declaration.
The bin sizes S d are set towith d ∈ { x, y, z } and the item sizes s d ∈ Z + are not greater than S d / 2 to avoid over-simpliﬁed scenarios.
Our performance on a more complex continuous dataset will be reported in Section 4.3.
Considering that there are many variants of 3D-BPP, we choose three representative settings:
Setting 1 :
Following Zhao et al.
(2021b) the stability of the B t is checked when n t is placed.
Only two horizontal orientations ( | O | = 2 ) are permitted for robot manipulation convenience
Setting 2 :
Following Martello et al.
, item n t only needs to satisfy Constraint 1 and 2.
Arbitrary orientation ( | O | = 6 ) is allowed here.
This is the most common setting in 3D-BPP literature.
Setting 3 : Inherited from setting 1 , each item n t has an additional density property ρ sampled from (0 , 1] uniformly.
This information is appended into the descriptors of B t and n t .
4. P ERFORMANCE OF PCT P OLICIES
We ﬁrst report the performance of our learned policies combined with different leaf node expansion schemes.
Three existing schemes which have proven to be both efﬁcient and effective are adopted here: Corner Point (CP), Extreme Point (EP), and Empty Maximal Space (EMS).
These schemes are all related to boundary points of a packed item along d axis.
We combine the start/end points of n t with the boundary points of b ∈ B t to get the superset, namely Event Point (EV). See Appendix B for details and learning curves.
We extend these schemes to our PCT model.
Although the number of leaf nodes generated by these schemes is within a reasonable range, we only randomly intercept a subset L sub t from L t if | L t | exceeds a certain length, for saving computing resources.
The interception length is a constant during training and determined by a grid search (GS) during the test.
See Appendix A for implementation details.
The performance comparisons are summarized in Table 1. [Table] H e u r i s ti c
[Table] Although PCT grows under the guidance of heuristics, the combinations of PCT with EMS and EV still learn effective and robust policies outperforming all baselines by a large margin regarding all settings.
Note that the closer the space utilization is to 1, the more difﬁcult online 3D-BPP is.
It is interesting to see that policies guided by EMS and EV even exceed the performance of the full coordinate space (FC) which is expected to be optimal. This demonstrates that a good leaf node expansion scheme reduces the complexity of the problem and helps DRL agents learn better performance.
To prove that the interception of L t will not harm the ﬁnal performance, we train agents with full leaf nodes derived from the EV scheme (EVF) and the test performance is slightly worse than the intercepted cases.
We guess that the interception keeps the ﬁnal performance may be caused by two reasons.
First, sub-optimal solutions may exist even in the intercepted leaf node set L sub .
In addition, the randomly chosen leaf nodes force the agent to make new explorations in case that the policy π falls into the local optimum.
We remove GS from EV cases to prove its effectiveness.
The performance of Zhao et al.
(2021b) deteriorates quickly in setting 2 and setting 3 due to the multiplying orientation space and insufﬁcient state representation separately.
Running costs, scalability performance, behavior understanding, visualized results, and the implementation of our real-world packing system can all be found in Appendix C.
We also repeat the same experiment as Zhang et al.
(2021) which pack items sampled from a predeﬁned item set |I| = 64 in setting 2 .
While Zhang et al.
(2021) packs on average 15 . 6 items and achieves 67 . 0% space utilization, our method packs 19 .
7 items with a space utilization of 83 . 0% .
4. B ENEFITS OF T REE P RESENTATION
Here we verify that the PCT representation does help online 3D-BPP tasks.
For this, we embed each space conﬁguration node independently like PointNet (Qi et al., 2017) to prove the node spatial relations help the ﬁnal performance.
We also deconstruct the tree structure into node sequences and embed them with Ptr-Net (Vinyals et al., 2015), which selects a member from serialized inputs, for indicating that the graph embedding fashion ﬁts our tasks well.
We have veriﬁed that an appropriate choice of L t makes DRL agents easy to train, then we remove the internal nodes B t from T t , along with its spatial relations with other nodes, to prove B t is also a necessary part.
We choose EV as the leaf node expansion scheme here.
The comparison results are summarized in Table
[Table] [Table]
If we ignore the spatial relations between the PCT nodes or only treat the input as a ﬂattened sequence [Table] the performance of the learned policies will be severely degraded.
The presence of B functions more on setting 1 and setting 3 since setting 2 allows items to be packed in any empty spaces without considering constraints with internal nodes.
This also conﬁrms that a complete PCT representation is essential for online 3D-BPP of practical needs.
4. P ERFORMANCE ON C ONTINUOUS D ATASET
𝑧
[Equation] The most concerning issue about online 3D-BPP is their solution space limit.
Given that most learning-based methods can only work in a limited discrete coordinate space, we directly test our method in a continuous bin with sizes S d = 1 to prove our superiority.
Due to the lack of public datasets for online 3D-BPP issues, we generate item sizes through a uniform distribution s d ∼ U ( a, S d / 2) , a is set to 0.1 in case that endless items are generated.
Speciﬁcally, for 3D-BPP instances where stability is considered, the diversity of item size s z needs to be controlled.
If all subsets of B t meet:
[Equation]
This means any packed items cannot form a new plane for providing support in the direction of gravity and the available packing areas shrinks.
Excessive diversity of s z will degenerate 3D-BPP into 1D-BPP as shown on the right.
To prevent this degradation from leading to the underutilization of the bins, we sample s z from a ﬁnite set { 0 . 1 , 0 . 2 , . . .
, 0 . 5 } on setting 1 and setting 3 .
[Table]
We ﬁnd that some heuristic methods (Ha et al., 2017) also have the potential to work in the continuous domain.
We improve these methods as our baselines.
Another intuitive approach for solving online 3D-BPP with continuous solution space is driving a DRL agent to sample action from a gaussian distribution (GD) and output continuous coordinates directly.
The test results are summarized in Table 3. Although the continuous-size item set is inﬁnite ( |I| = ∞ ) which increases the difﬁculty of the problem and reduces the performance of all methods, our method still performs better than all competitors.
The DRL agent which outputs continuous actions directly cannot even converge and their variance is not considered.
To the best of our knowledge, we are the ﬁrst ones to deploy the learning-based method on solving online 3D-BPP with continuous solution space successfully [Table]
4. G ENERALIZATION ON D IFFERENT D ISTRIBUTIONS
The generalization ability of learning-based methods has always been a concern.
Here we demonstrate that our method has a good generalization performance on item size distributions different from the training one.
We conduct this experiment with continuous solution space. We sample s d from normal distributions N ( µ, σ 2 ) for generating test sequence where µ and σ are the expectation and the standard deviation. Three normal distributions are adopted here, namely N (0 . 3 , 0 .
N (0 . 1 , 0 . 2 2 ) , and N (0 . 5 , 0 . 2 2 ) , as shown in Figure 3.
The larger µ of the normal distribution, the larger the average size of sampled items.
We still control s d within the range of [0 . 1 , 0 . 5] . If the sampled item sizes are not within this range, we will resample until it meets the condition
[Table] [Table]
We directly transfer our policies trained on U (0 . 1 , 0 . 5) to these new datasets without any ﬁne-tuning. We use the best-performing heuristic method LSAH (Hu et al., 2017) in Section 4.3 as a baseline.
The test results are summarized in Table 4. Our method performs well on the distributions different from the training one and always surpasses the LSAH method.
See Appendix C for more results about the generalization ability of our method on disturbed distributions and unseen items.
[Figure]
4. M ORE C OMPLEX P RACTICAL C ONSTRAINTS
To further prove that our method ﬁts 3D-BPP with complex constraints well, we give two demonstrations about extending our method to online 3D-BPP with additional practical constraints: isle friendliness and load balancing (Gzara et al., 2020): 3D-BPP with Isle Friendliness Isle friendliness refers to that items belonging to the same category should be packed as closely as possible.
The item weight is set as w t = max (0 , v t − c · dist ( n t , B t )) , c is a constant.
The additional object function dist ( n t , B t ) means the average distance between n t and the items of the same category in B t .
The additional category information is appended into the descriptors of B t and n t .
Four item categories are tested here.
3D-BPP with Load Balancing Load balancing dictates that the packed items should have an even mass distribution within the bin.
The item weight is set as w t = max (0 , v t − c · var ( n t , B t )) .
Object var ( n t , B t ) is the variance of the mass
[Table]
We choose the learning-based method Zhao et al.
(2021b)
as our baseline since heuristic methods only take space utilization as their objects.
The results are summarized in Table 5.
Compared with the baseline algorithm, our method better achieves the additional object while still taking into account the space utilization as the primary.
D ISCUSSION
We formulate the online 3D-BPP as a novel hierarchical representation – packing conﬁguration tree.
PCT is a full-ﬂedged description of the state and action space of bin packing which makes the DRL agent easy to train and well-performing.
We extract state feature from PCT using graph attention networks which encodes the spatial relations of all space conﬁguration nodes.
The graph representation of PCT helps the agent with handling online 3D-BPP with complex practical constraints, while the ﬁnite leaf nodes prevent the action space from growing explosively.
Our method surpasses all other online 3D-BPP algorithms and is the ﬁrst learning-based method that solves online 3D-BPP with continuous solution space successfully.
Our method performs well even with items sampling distributions different from the training.
We also give demonstrations to prove that our method is versatile in terms of incorporating various practical constraints.
For future research, we are extremely interested in applying our method to the more difﬁcult irregular shape packing (Wang & Hauser, 2019a), where the sampling costs for training DRL agents are more expensive and the solution space will be far more complex.
A good strategy for balancing exploration and exploitation for learning agents is eagerly needed in this situation.
Finding other alternatives for better representing 3D-BPP with different packing constraints and designing better leaf node expansion schemes for PCT are also interesting directions.
E THICS S TATEMENT
We believe the broader impact of this research is signiﬁcant.
BPP is considered one of the most needed academic problems due to its wide applicability in our daily lives (Skiena, 1997).
Providing a good solution to online 3D-BPP yields direct and profound impacts beyond academia.
We note that online packing setups have been already widely deployed in logistics hubs and manufacturing plants.
A good online 3D-BPP policy beneﬁts all the downstream operations like package wrapping, transportation, warehousing, and distribution.
Our method can also be treated as a separate decision module and utilized to solve other variants of 3D-BPP issues, such as multi-bin online 3D-BPP, online 3D-BPP with lookahead (Zhao et al., 2021a), online placement decision for ofﬂine 3D-BPP (Hu et al., 2017), and the logistics robot implementation for online packing (Zhao et al., 2021b), which have been well studied in previous works.
We give demonstrations about applying our method to solving online 3D-BPP with various complex practical constraints, the similar idea can also be adopted to meet other customized desires.
R EPRODUCIBILITY
Our real-world demo is submitted with the supplemental material.
All experimental data are well preserved for checking.
The source code of our method along with all gathered baselines mentioned above will be published after acceptance for domain development.
A I MPLEMENTATION D ETAILS
Deep Reinforcement Learning We formulate online 3D-BPP as Markov Decision Process and solve it with the deep reinforcement learning method.
A DRL agent seeks for a policy π to maximize the accumulated discounted reward:
[Equation]
Where γ ∈ [0 , 1] is the discount factor, and τ = ( s 0 , a 0 , s 1 , . . .
) is a trajectory sampled based on the policy π .
We extract the feature of state s t =
( T t , n t ) using graph attention networks (Velickovic et al., 2018) for encoding the spatial relations of all space conﬁguration nodes.
The context feature is fed to two key components of our pipeline: an actor network and a critic network.
The actor network, designed based on pointer mechanism, weighs the leaf nodes of PCT, which is written as π ( a t
| s t ) .
The action a t is an index of selected leaf node l ∈ L t , denoted as a t = index ( l ) .
The critic network maps the context feature into a state value prediction V ( s t ) which helps the training of actor network.
The whole network is trained via a composite loss L = α · L actor + β · L critic ( α = β = 1 in our implementation), which consists of actor loss L actor and critic loss L critic .
These two loss functions are deﬁned as:
[Equation]
Where r t = c r · w t is our reward signal and we set γ as 1 since the packing episode is ﬁnite.
We adopt a step-wise reward r t = c r · w t once n t is inserted into PCT as an internal node successfully.
Otherwise, r t =
and the packing episode ends.
The choice of item weight w t depends on the packing preferences.
In the general sense, we set w t as the volume occupancy v t =
s xn · s yn · s zn of n t , and the constant c r is 10 / ( S x · S y · S z ) .
For online 3D-BPP with additional packing constraints, this weight can be set as w t = max (0 , v t − c · O ( s t , a t )) .
While term v t ensures that space utilization is still the primary concern, the objective function O ( s t , a t ) guides the agent to satisfy additional constraints like isle friendliness and load balancing.
We adopt ACKTR (Wu et al., 2017) method for training our DRL agent, which iteratively updates an actor and a critic using Kronecker-factored approximate curvature (K-FAC)
(Martens & Grosse, 2015) with trust region.
Zhao et al.
(2021a) have demonstrated that this method has a surprising superiority on online packing problems over other model-free DRL algorithms like SAC (Haarnoja et al., 2018).
Feature extraction Speciﬁcally, ACKTR runs multiple parallel processes (64 here) to interact with their respective environments and gather samples.
The different processes may have different packing time step t and deal with different packing sequences, the space conﬁguration nodes number
N also changes.
To combine these data with irregular shapes into one batch, we fullﬁll B t and L t to ﬁxed lengths, 80 and 25 ·| O | respectively, with dummy nodes.
The descriptors for dummy nodes are all-zero vectors and have the same size as the internal nodes or the leaf nodes.
The relation weight logits u ij of dummy node j to arbitrary node i is replaced with − inf to eliminate these dummy nodes during the feature calculation of GAT.
The global context feature ¯ h is aggregated only on the eligible nodes h : ¯ h = 1 N (cid:80) Ni
=1 h i .
All space conﬁguration nodes are embedded by GAT as a fully connected graph as Figure 2 (b), without any inner mask operation.
We only provide the packable leaf nodes which satisfy placement constraints for DRL agents.
For setting 2 , we check in advance if a candidate placement satisﬁes Constraint 1 and 2.
For settingand setting 3 where the mass of item n t is v t and ρ · v t respectively, we will additionally check if one placement meets the constraints of packing stability.
Beneﬁts from the fast stability estimation method proposed by Zhao et al.
(2021b), this pre-checking process can be completed in a very short time, and our DRL agent samples data at a frequency of more than 400 FPS.
The node-wise MLPs φ θ B , φ θ L , and φ θ n used to embed raw space conﬁguration nodes are two-layer linear networks with LeakyReLU activation function.
φ FF is a two-layer linear structure activated by ReLU.
The feature dimensions d h , d k , and d v are 64. The hyperparameter c clip used to control the range of clipped compatibility logits is set to 10 in our GAT implementation. Choice of PCT Length
Since PCT allows discarding some valid leaf nodes and this will not harm our performance, we randomly intercept a subset L sub t from L t if | L t | exceeds a certain length.
Finding a suitable PCT length for different bin conﬁgurations is important, we will give our recommendations for ﬁnding this hyperparameter.
For training, we ﬁnd that the performance of learned policies is more sensitive to the number of allowed orientations | O | .
Thus we set the PCT length as c · | O | where c can be determined by a grid search nearby c = 25 for different bin conﬁgurations. For our experiments, c = 25 works quite well. During the test, the PCT length can be different from the training one, we suggest searching this interception length with a validation dataset via a grid search which ranges from 50 to 300 with step length 10.
B L EAF N ODE E XPANSION S CHEMES
[Figure] We introduce the leaf node expansion schemes adopted in our PCT implementation here. These schemes are used to incrementally calculate new candidate placements introduced by the just placed item n t .
A good expansion scheme should reduce the number of solutions to be explored while not missing too many feasible packings.
Meanwhile, polynomially computability is also expected.
As shown in Figure 4, the policies guided by suitable leaf node expansion schemes outperform the policy trained on a full coordinate (FC) space in the whole training process.
We extend three existing heuristic placement rules which have proven to be accurate and efﬁcient to our PCT expansion, i.e. Corner Point , Extreme Point , and Empty Maximal Space .
Since all these schemes are related to boundary points of packed items, we combine the start/end points of n t with these boundary points as a superset, namely Event Point .
Corner Point Martello et al.
ﬁrst introduce the concept of Corner Point (CP) for their branch-and-bound methods.
Given 2D packed items in the xoy plane, the corner points can be found where the envelope of the items in the bin changes from vertical to horizontal, as shown in
Figure 5 (a).
The past corner points which no longer meet this condition will be deleted. Extend this 2D situation to 3D cases, the new candidate 3D positions introduce by the just placed item n t are a subset of { ( p xn + s xn , p yn , p zn ) , ( p xn , p yn + s yn , p zn ) , ( p xn , p yn , p zn
+ s zn ) }
if the envelope of the corresponding 2D plane, e.g., xoy, yoz, and xoz , is changed by n t .
The time complexity of ﬁnding 3D corner points incrementally is O ( c ) with an easy-to-maintained bin height map data structure to detect the change of envelope on each plane, c is a constant here.
Extreme Point Crainic et al.
(2008) extend the concept of Corner Point to Extreme Point (EP) and claim their method reaches the best ofﬂine performance of that era.
Its insight is to provide the means to exploit the free space deﬁned inside a packing by the shapes of the items that already exist.
When the current item n t is added, new EPs are incrementally generated by projecting the coordinates { ( p xn + s xn , p yn , p zn ) , ( p xn , p yn + s yn , p zn ) , ( p xn , p yn , p zn + s yn ) } on the orthogonal axes, e,g. project ( p xn + s xn , p yn , p zn ) in the directions of the y and z axes to ﬁnd intersections with all items lying between item n t and the boundary of the bin.
The nearest intersection in the respective direction is an [Figure] extreme point.
Since we stipulate a vertical top-down loading direction, the 3D extreme points in the strict sense may exist a large item blocking the loading direction.
So we ﬁnd the 2D extreme points (see Figure 5 (b)) in the xoy plane and repeat this operation on each distinct z value (i.e. start/end p z coordinate of a packed item) which satisﬁes p zn ≤ z ≤ p zn
+ s zn .
The time complexity of this method is O ( m · | B 2 D | ) , where B 2 D is the packed items that exist in the corresponding z plane and m is the number of related z scans.
Empty Maximal Space Empty Maximal Spaces (EMSs) (Ha et al., 2017) are the largest empty orthogonal spaces whose sizes cannot extend more along the coordinate axes from its front-leftbottom (FLB) corner.
This is a simple and effective placement rule.
An EMS e is presented by its FLB corner ( p xe , p ye , p ze ) and sizes ( s xe , s ye , s ze ) .
When the current item n t is placed into e on its FLB corner, this EMS is split into three smaller EMSs with positions ( p xe + s xn , p ye , p ze ) , ( p xe , p ye + s yn , p ze ) , ( p xe , p ye , p ze + s zn ) and sizes ( s xe − s xn , s ye , s ze ) , ( s xe , s ye − s yn , s ze ) , ( s xe , s ye , s ze − s zn ) , respectively.
If the item n t only partially intersects with e , we can apply a similar volume subtraction to the intersecting part for splitting e .
For each ems, we deﬁne the left-up ( p xe , p ye + s ye , p ze ) , right-up ( p xe + s xe , p ye + s ye , p ze ) , left-bottom ( p xe , p ye , p ze ) , and right-bottom ( p xe + s xe , p ye , p ze ) corners of its vertical bottom as candidate positions, as shown in Figure 5 (c).
These positions also need to be converted to the FLB corner coordinate for placing item n t .
The left-up, right-up, left-bottom and right-bottom corners of e should be converted to ( p xe , p ye + s ye − s yn , p ze ) , ( p xe + s xe − s xn , p ye + s ye − s yn , p ze ) , ( p xe , p ye , p ze ) , and ( p xe + s xe − s xn , p ye , p ze ) respectively.
Since all EMSs e ∈ E in the bin needs to detect intersection with n t , the time complexity of ﬁnding 3D EMSs incrementally is O ( | E | ) .
A 3D schematic diagram of PCT expansion guided by EMSs is provided in Figure 6. [Figure]
Event Point It’s not difﬁcult to ﬁnd that all schemes mentioned above are related to boundary points of a packed item along d ∈ { x, y } axes (we assume the initial empty bin is also a special packed item here).
When the current item n t is packed, we update the existing PCT leaf nodes by scanning all distinct z values which satisfy p zn ≤ z ≤ p zn
+ s zn and combine the start/end points of n t with the boundary points that exist in this z plane (see Figure 5 (d)) to get the superset, which is called Event Points C M ORE R ESULTS In this section [Table]
we report more results of our method.
Section C.1 further discusses the generalization ability of our method on disturbed item sampling distributions and unseen items.
Section C.2 reports the running cost of each method. Section C.3 scales our method to a larger problem and reports the performance. Section C.4 visualizes packing sequences to analyze model behavior and more visualized results are provided in Section C.5. Section C.6 introduces our real-world packing system.
C
F URTHER D ISCUSSIONS ON G ENERALIZATION
We have veriﬁed in Section 4.4 that our method has good generalization performance on item sampling distributions different from the training one.
Now we further analyze the generalization ability of our method.
Firstly, we will demonstrate that our algorithm has a good generalization performance on disturbed item sampling distributions.
We conduct this experiment in the discrete setting where the item set |I| = 125 is ﬁnite.
For each item i ∈ I , we add a random non-zero disturbance δ i on its original sample probability p
i , e
[Table]
g., p i = p i · (1 − δ i ) .
We normalize the disturbed p i as the ﬁnal item sampling distribution.
Note that δ i is ﬁxed during sampling one complete sequence.
The state transition of DRL P ( s t +1 | s t ) which is partly determined by the probabilities of sampling items will be different from the training one where the uniform distribution sample items with equal probabilities.
Generalizing to a new transition is a hard challenge for DRL methods.
We test different ranges of δ i and the results are summarized in Table 6.
[Table]
Beneﬁts from the efﬁcient guidance of heuristic leaf node expansion schemes, our method maintains its performance under various amplitude disturbances.
Our method even behaves well with a strong disturbance δ i ∈ [ − 100% , 100%] applied, which means some items may never be sampled by some distributions
when δ i
= 1 and p i · (1 − δ i )
{+= 
+}
in a speciﬁc sequence.
Beyond experiments on generalization to disturbed distributions
[Table]
we also test our method with unseen items.
We conduct this experiment in the discrete setting.
We randomly delete 25 items of I and train PCT policies with |I sub | = 100 .
Then we test the trained policies on full I .
See Table 7 for results.
Our method still performs well on datasets where unseen items exist regarding all settings
[Table]
trained with the EV scheme.
[Table]
C. R UNNING C OSTS
For 3D-BPP of online needs, the running costs for placing items are especially important.
We count the running costs of the experiments in Section 4.1 and Section 4.3 and summarize them in Table 8.
Each running cost at time step t is counted from putting down the previous item n t − 1 until the current item n t is placed, which includes the decision time, the time to check placement feasibility, and the time to interact with the packing environment.
The running cost of our method is comparable to most baselines.
Our method can meet real-time packing requirements in both discrete solution space and continuous solution space.
[Table]
C. S CALABILITY
The number of PCT nodes changes constantly with the generation and removal of leaf nodes during the packing process.
To verify whether our method can solve packing problems with a larger scale | B | , we conduct a stress test on setting 2 where the most orientations are allowed and the most leaf nodes are generated.
We limit the maximum item sizes s d to S d / 5 in the continuous setting so that more items can be accommodated.
We transfer the best-performing policies on setting 2 (trained with EMS) to these new datasets without any ﬁne-tuning.
The results are summarized in Table
[Table]
[Table]
PCT will not grow exponentially with packing scale | B | since invalid leaf nodes will be removed from leaf nodes L during the packing process, both discrete and continuous cases.
For continuous cases, | L | is more sensitive to | B | due to the diversity of item sizes (i.e. |I| = ∞ ), however, | L | still doesn’t explode with | B | and it grows in a sub-linear way.
Our method can execute packing decisions at a real-time speed with controllable PCT sizes even if the item scale is around two hundred.
C. U NDERSTANDING OF M ODEL B EHAVIORS
The qualitative understanding of model behaviors is important, especially for practical concerns.
We visualize our packing sequence to give our analysis.
The behaviors of learned models differ with the packing constraints.
If there is no speciﬁc packing preference, our learned policies will start packing nearby a ﬁxed corner (Figure 7 (a)).
The learned policies tend to combine items of different heights together to form a plane for supporting future ones (Figure 7 (b)).
Meanwhile, it prefers to assign little items to gaps and make room (Figure 7 (c)) for future large ones (Figure 7 (d)).
If additional packing preference is considered, the learned policies behave differently.
For online 3D-BPP with load balancing, the model will keep the maximum height in the bin as low as possible and pack items layer by layer (Figure 7(e)).
For online 3D-BPP with isle friendliness, our model tends to pack the same category of items nearby the same bin corner (Figure 7 (f)).
C. V ISUALIZED R ESULTS
We visualize the experimental results of Section 4.1 in Figure 8.
Each plot is about a randomly generated item sequence packed by our best-performing policies (PCT & EV on setting 1 and setting 3 , PCT & EMS on setting 2 ). [Figure] [Figure]
tti
[Figure]
S e tti
[Figure]
C. R EAL -W ORLD P ACKING S YSTEM
[Figure]
We follow the online autonomous bin packing system implementation of Zhao et al.
(2021b)
in a logistics warehouse to validate our PCT method.
As shown in Figure 9, the conveyor belt transports items to the robot arm (STEP®SR20E) at a ﬁxed speed.
The onconveyor RGB-D camera (PhoXi 3D Scanner XL) recognizes the size and location of the current item.
This information is sent to the robot arm for picking the current item and our PCT model for the placement decision.
We align the bin coordinate with the robot coordinate and transform the packing decision to a real-world position.
Then the robot arm place the current item into the bin which is a table trolley with sizes S x
= 110 cm, S y = 90 cm , and S z = 80 cm .
We recorded sizes and positions of packed items as descriptors of internal nodes B .
Different from the original implementation of Zhao et al.
(2021b) where only one on-conveyor camera is used for detecting the target item to be packed, we add one more on-bin RGB-D camera and adopt Mask R-CNN (He et al., 2017) for instance segmentation of input depth image, for monitoring possible drift of packed items.
If a packed item deviates from the position of the PCT decision and this drift is detected, we will correct the descriptor of the corresponding internal node b ∈ B with the offset position.
We use items with s x , s y ranging from 20cm to 40cm, and s z ranging from 10cm to 25cm.
We ﬁll items with paddings of the same density and train PCT policies on setting 1 with the
EV scheme and the load balancing constraint.
We test our system with 50 random sequences and our method achieves 78 .
6% space utilization with 50 .
9 items packed.
Our real-world packing demo is also submitted with the supplemental material.