S EMANTICALLY C ONTROLLABLE G ENERATION OF P HYSICAL S CENES WITH E XPLICIT K NOWLEDGE
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep Generative Models (DGMs) are known for their superior capability in generating realistic data.
Extending purely data-driven approaches, recent specialized
DGMs satisfy additional controllable requirements such as embedding a trafﬁc sign in a driving scene by manipulating patterns implicitly in the neuron or feature level.
In this paper, we introduce a novel method to incorporate domain knowledge explicitly in the generation process to achieve the semantically controllable generation of physical scenes.
We ﬁrst categorize our knowledge into two types, the property of objects and the relationship among objects, to be consistent with the composition of natural scenes.
We then propose a tree-structured generative model to learn hierarchical scene representation, whose nodes and edges naturally corresponded to the two types of knowledge, respectively.
Consequently, explicit knowledge integration enables semantically controllable generation by imposing semantic rules on the properties of nodes and edges in the tree structure.
We construct a synthetic example to illustrate the controllability and explainability of our method in a succinct setting.
We further extend the synthetic example to realistic environments for autonomous vehicles and conduct extensive experiments: our method efﬁciently identiﬁes adversarial physical scenes against different state-of-the-art 3D point cloud segmentation models, and satisﬁes the trafﬁc rules speciﬁed as the explicit knowledge.
I NTRODUCTION
The recent breakthrough in machine learning enables us to learn complex distributions behind data with sophisticated models.
These models help us understand the data generation process so as to realize controllable data generation [1, 65, 16].
Deep Generative Models (DGMs) [23, 34, 17], which approximate the data distribution with neural networks (NN), are representative methods to generate data targeting a speciﬁc style, category, or attribute.
However, existing controllable generative models focus on manipulating implicit patterns in the neuron or feature level.
For instance, [7] dissects
DGMs to build the relationship between neurons and generated data, while [55] interpolates in the latent space to obtain vectors that control the poses of objects.
One main limitation of these existing models is that they cannot explicitly incorporate unseen semantic rules, which may lead to meaningless generated data that violates common sense.
For example, to build diverse physical scenes for evaluating autonomous vehicles, the generated cars should follow semantic trafﬁc rules and physical laws, which cannot be enforced by directly manipulating neurons.
In light of the limitations of previous work, we aim to develop a structured generative framework to integrate explicit knowledge [15] during the generation process and thus control the generated scene to be compliant with semantic rules.
Natural scenes can be described with objects and their various relationships [5].
Thus, in this paper, we categorize the semantic knowledge that describes scenes into two types, where the ﬁrst type denoted as node-level knowledge represents the properties of single objects and the second type denoted as edge-level knowledge represents the relationship among objects.
We also observe that tree structure is highly consistent with this categorization for constructing scenes, where nodes of the tree represent objects and edges the relationship.
By automatically controlling the tree structure during the generation, we explicitly integrate the node-level and edge-level knowledge.
In detail, we propose a general framework, Semantically Controllable Generation (SCG), which consists of two stages as shown in Figure 1.
In stage one , we train a tree-structured generative model that parameterizes nodes and edges of trees with NN to learn the representation of structured data.
In stage two , explicit knowledge is applied to different levels of the tree to achieve semantically controllable generation for downstream tasks such as satisfying certain conditions or reducing the performance of recognition algorithms.
To verify the proposed SCG, we ﬁrst construct a synthetic scene reconstruction example to illustrate the advantages of SCG and provide analysis on its controllability and explainability.
With SCG, it is possible to generate natural scenes that follow semantic rules, e.g., boxes with the same color should be positioned close to each other.
To demonstrate the practicality of SCG, we conduct extensive experiments on adversarial LiDAR scene generation against state-of-the-art 3D segmentation models.
We show that our generated safety-critical physical scenes successfully attack victim models and meanwhile follow the speciﬁed trafﬁc rules.
In addition, compared with traditional attack methods, scenes generated by our method achieve stronger adversarial transferability across different victim models.
Our technical contributions are summarized below:
• We propose a semantically controllable generative framework (SCG) via integrating explicit knowledge and categorize the knowledge into two types according to the composition of scenes.
• We propose a tree-structured generative model based on our knowledge categorization and construct a synthetic example to demonstrate the effectiveness of our knowledge integration.
• We propose the ﬁrst semantic adversarial point cloud attack based on SCG, named Scene Attack , against state-of-the-art segmentation algorithms, demonstrating several essential properties.
S EMANTICALLY C ONTROLLABLE G ENERATION
We deﬁne the scene x ∈ X in the data space and the latent code z ∈ Z in the latent space.
This paper aims to generate scene x that satisﬁes certain semantic rules K t , which are related to the downstream task t ∈ T .
The scene x will be used to solve the downstream task t by minimizing the corresponding objective function L t ( x ) . In this section, we ﬁrst describe the tree-based generative model for learning the hierarchical representations of x , which is important and necessary for applying knowledge to achieve semantic controllability. Then we explain the two types of knowledge to be integrated into the generative model together with the generation algorithm that uses explicit knowledge K t as guidance for downstream tasks. 2. T REE - STRUCTURED V ARIATIONAL A UTO - ENCODER (T-VAE)
VAE [34] is a powerful DGM that combines auto-encoder and variational inference [9].
It estimates a mapping between data x and latent code z to ﬁnd the low-dimensional manifold of the data space.
The objective function of training VAE is to maximize a lower bound of the log-likelihood of training data, which is so-called Evidence Lower Bound (ELBO)
[Equation]
where KL is Kullback–Leibler (KL) divergence.
q ( z | x ; φ ) is the encoder with parameters φ , and p ( x | z ; θ ) is the decoder with parameters θ .
The prior distribution of the latent code p ( z ) is usually a
Gaussian distribution for simpliﬁcation of KL divergence calculation.
One typical characteristic of natural scenes is the variable data dimension caused by the variable number of objects, which is very challenging to represent with a ﬁxed number of parameters as in traditional models [34].
Besides, many existing structured generative models [64, 40] do not consider the hierarchy of natural scenes nor have the capability to incorporate explicit knowledge.
In general, graph is commonly used to represent structured data [41], but sometimes is too complicated to describe the hierarchy and inefﬁcient to generate.
As a special case of graph, tree naturally embed hierarchical information via recursive generation with depth-ﬁrst-search traversal [29, 48].
This hierarchy is not only highly consistent with natural physical scenes, but also makes easier to apply explicit knowledge, supported by previous works in cognition literature [46].
In this work, we propose a novel tree-structured generative model, which is inspired by the stickbreaking approach [58]: Assume we have a stick with length W and we iteratively break it into segments w ( n,i ) with:
[Equation]
where ( n, i ) means the i -th segment of the n -th level.
K n is the total number of segments in the n -th level.
The index starts from 1 and entire length has index (1 , 1) .
The recursive function of breaking the stick follows
[Equation]
where α ( n,i ) ∈
[0 , 1] is the splitting ratio for w ( n,i ) and j is the index in the n + 1 -th level.
Intuitively, this breaking process creates a tree structure where the i -th level is corresponding to the i -th layer of the tree and segments are nodes in the tree.
We extent the above division to 2D space with two parameters α and β .
A 2D plane example is illustrated in Figure 2, where each node has 4 child nodes in the next layer.
Assume there are M kinds of nodes in the scene, we deﬁne a batch of encoders E m and decoders
D m for all m ∈ M and n ∈ { 1 , · · · , N − 1 } :
[Equation]
where f ( n,i ) is named as the feature vector that passes the messages through the tree structure.
g ( n,i ) is named as the property vector of node ( n, i ) that stories speciﬁc properties such as color of the object generated by node ( n, i ) .
ˆ g ( n +1 ,j ) is the predicted property vector and l m is the number of children of node m .
Besides the encoders and decoders, we also deﬁne a Classiﬁer to determine the child node type and a Sampler to infer the posterior distribution of the latent code,
[Equation]
where ˆ c ( n,i ) is the predicted node type and [ z µ , z σ ] is used to calculate the latent code z with the reparameterization trick [9].
θ c
and θ s are models parameters for the Classiﬁer and Sampler .
Parameters for the encoders q ( z | x ; φ ) and decoders p ( x | z ; θ ) are denoted respectively as φ = { φ 1 , · · · , φ m , φ s } and θ = { θ 1 , · · · , θ m , θ c } .
The ﬁnal structures of the encoder and decoder depend on the tree structure of the data point and vary in the dataset.
We follow Recursive Neural Networks (RvNN) [61] to build the tree structure recursively.
Finally, the tree x is summarized as
[Equation]
where c represents the node type.
Following (1), the ELBO of T-VAE to be maximized is
[Equation]
where the equality holds because c and g are conditionally independent given z .
The L C term represents the cross-entropy loss (CE) of all nodes in p ( x | z ; θ ) ,
[Equation]
where the prior distribution of node type p ( c ) is calculated from the training dataset and serves as weights.
To make the reconstructed tree have same structure as the original one, we use Teacher
Forcing [67] during the training stage.
However, in the generation stage, we take the node with maximum probability as the child to expand the tree.
The L R term uses mean square error (MSE) to approximate the log-likelihood of node property,
[Equation]
where N m is the times that node m appears in the tree and 1 [ · ] is the indicator function.
In (9), we normalize the MSE with N m instead of (cid:80) Nn K n to avoid the inﬂuence caused by imbalanced node type in the tree.
Please refer to Appendix B for detailed model deﬁnition and generative process.
The advantage of this hierarchical structure is that we only need to store the global information in the root node and use local information in other nodes, making the model easier to capture the feature from different scales in the scene.
Moreover, this tree structure makes it possible to explicitly apply semantic knowledge in Stage 2 , which will be explained in Section 2.2.
Suppose there is a function set F , where the function f ( A ) ∈ F returns true or false for a given input node A of a tree x . Then, we deﬁne the two types of propositional knowledge K t for a particular downstream task t (we omit t below for simpliﬁcation) using the ﬁrst-order logic [60] as following. Deﬁnition 1 (Knowledge Set)
The node-level knowledge k n is denoted as f ( A ) for a function f ∈ F , where A is a single node.
The edge-level knowledge k e is denoted as f 1 ( A ) → ∀ i f 2 ( B i ) for two functions f 1 , f 2 ∈ F , where we apply knowledge f 2 to all
A ’s child nodes B i .
Then, The knowledge set is constructed as K =
{ k (1) n , · · · , k (1) e , · · · } .
In the tree context, k n describes the properties of a single node, and k e describes the relationship between the parent node and its children.
Thus, the implementation of knowledge is based on the recursive traversal of the tree.
We summarized this process in Algorithm 1 .
For the speciﬁc knowledge used in our experiment, please refer to Appendix C.
Input: Dataset D , Task loss
L t x ) , Searching budget B , Knowledge set K
Output: Generated x s 1 Stage 1: Train T VAE
Initialize parameters { θ , φ } for in D do
Encode z ← q ( z | x ; φ )
Decode ˆ x ← p ( x | z ; θ )
Update parameters { θ , φ } by maximizing ELBO (7) end 8 Store the learned decoder p ( x | z ; θ ) 9 Stage 2: Generation
Initialize latent code z ∼ N (0 , I ) while B is not used up do if L t ( x ) is differentiable then z ← z − η ∇ L t ( p ( x | z ; θ )) else
z ← Black-box optimization end x (cid:48) ← ApplyK
( K , x ) z ← prox L K
( z ) with (10) end 20 Decode the scene x s = p ( x | z ; θ ) Input k
K do
[-Deﬁnition 
 The node-level knowledge k n is denoted as f ( A ) for a function f ∈ F , where A is a single node. The edge-level knowledge k e is denoted as f 1 ( A ) → ∀ i f 2 ( B i ) for two functions f 1 , f 2 ∈ F , where we apply knowledge f 2 to all A ’s child nodes B i .-]4 end
end 10 end
Then, we apply explicit knowledge to the decoder of TVAE using the proximal gradient algorithm.
Firstly, the original task objective L t ( x ) is augmented to L a ( x ) =
L t ( x ) + L K ( x, x (cid:48) ) , where the second term measures the distance between
the original tree x and modiﬁed tree x (cid:48) . Speciﬁcally, k n changes the property vector g of single node A in x to g (cid:48) that make f ( A ) be true, and k e traverses the tree x to ﬁnd node
A that satisfy f 1 in k e then change the type vector c or the property vector g of the children of A to c (cid:48) and g (cid:48) respectively.
The reference vector c (cid:48) and g (cid:48) are related to the downstream task and pre-deﬁned in the function f .
Then L K consists of two parts, MSE is used between the property vector g and g (cid:48) , and CE is used between the type vector c and c (cid:48) .
For instance, the explicit knowledge described as “if one box is blue, its child nodes should be red” will be implemented by the following operations.
Starting from the root, we ﬁnd all box nodes whose colors are blue and collect the property vectors g of its child nodes; then we change g to g (cid:48) , which means the color red.
Next, L K is obtained by calculating MSE between all g (cid:48) and the original g .
This process is illustrated in Figure 2(c).
In most applications, L t ( x ) requires many resources to compute, while L K is efﬁcient to evaluate since it only involves the inference of p ( x | z ; θ ) .
Therefore, we resort to Proximal algorithms [52], a type of optimization method, to restrict the searching step to a ﬁnite region.
In our setting, the explicit knowledge can be treated as the trusted region to limit and guide the optimization of the downstream task. The knowledge loss and task objective are alternatively optimized under the proximal optimization framework. We deﬁne the proximal operator as
[Equation]
which projects the candidate z to z (cid:48) to minimizes the knowledge inconsistency caused by L K .
The second term in (10) is a regularize to make the projected point close to the original point.
This problem can be solved by gradient descent since the decoder p ( x | z ; θ ) of T-VAE is differentiable.
After the projection, we need to solve the original optimization problem L t ( x ) . We use gradient descent for a differentiable L t ( x ) or change to black-box optimization methods [6] when L t ( x ) is non-differentiable.
The complete algorithm is summarized in Algorithm 2 .
E XPERIMENTS
First, we design a synthetic scene to illustrate the controllability and explainability of the proposed framework.
The synthetic physical scene provides a simpliﬁed setting to unveil the essence of the knowledge-guided generation.
After that, we evaluate the performance of SCG on realistic trafﬁc scenes represented by point clouds.
Based on SCG, we propose a new adversarial attack method,
Scene Attack , against multiple state-of-the-art segmentation methods.
3. S YNTHETIC S CENE R ECONSTRUCTION
Task Description.
We aim to reconstruct a scene to match a given image as shown in Figure 4.
The task objective is a reconstruction error
L t ( x ) = (cid:107) S − R ( x ) (cid:107) 2 , where R is a differentiable image renderer [33] and S is the image of target scene.
Under this succinct setting, it is possible to analyze and compare the contribution of explicit knowledge integration, since we can access the optimal solution.
According to the understanding of the target scene, we deﬁne three knowledge rules: 1 (cid:13)
The scene has at most two plates; 2 (cid:13)
The boxes that belong to the same plate should have the same color; 3 (cid:13)
The boxes belong to the same plate should have distance smaller than a threshold γ .
We compare our method with three baselines: Direct Search (DS), VAE, and Grammar-VAE (GVAE) [37].
DS directly optimizes the positions and colors of boxes and plates in the data space.
In contrast, VAE and GVAE search in the latent space and generate the scene from their decoders.
[Table] [Table]
Experiment Settings.
We synthesize the dataset by randomly generating 10,000 samples with a varying number of boxes and plates.
We also inject the target scene 10 times into the dataset to make sure it accessible to all models.
Unlike GVAE and T-VAE, DS and VAE need to access the number of boxes and plates in the target image (e.g., two plates and eight boxes) to ﬁx the dimension of the input feature.
To get the good initial points for DS, we add a small perturbation to the positions and colors of all objects in the target scene.
Similarly, for other methods, we add the perturbation to the optimal latent code, which is obtained by passing the target scene to the encoder to get good initialization.
Evaluation Results.
The generated samples from ﬁve models are displayed in Figure 3, and the ﬁnal errors are summarized in Table.
With good initialization, all models ﬁnd a similar scene to the target one, while with the random initialization, all models are trapped in local minimums.
However, obtaining good initialization is not practical in most real-world applications, indicating that this task is non-trivial and all models without knowledge cannot solve it.
After integrating the knowledge into the T-VAE model, we obtain I of Figure 3.
We can see that all three knowledge have positive guidance for the optimization, e.g., the boxes concentrate on the centers of plates with knowledge 3 (cid:13)
When combining the three knowledge, even from a random initialization, our T-VAE can ﬁnally ﬁnd the target scene, leading to a small error in Table.
We also want to mention that it is also possible to apply simple knowledge to GVAE during the generation.
However, the advantages of our method are that (1) we can integrate any constraints as long as they can be represented by 1, whereas GVAE can only apply hard constraints to objects with co-occurrences.
Analysis of Knowledge and Controllability.
To analyze the contribution of each knowledge, we plot the knowledge losses of 1 (cid:13) 3 (cid:13) in Figure 4(b) together with the task loss.
All knowledge losses decrease quickly at the beginning and guide the searching in the downstream task.
Next, we made ablation studies to explore the reason why knowledge helps the generation.
In Figure 4(b), we compare the optimization trajectories of T-VAE (red → yellow) and T-VAE w/ SCG (blue → green) with the same initialization.
For T-VAE, the generated samples are diverse but totally different from the target scene, while for T-VAE w/ SCG, the trajectories go to another direction and the generated samples are good.
However, the interesting thing is that although the knowledge help us ﬁnd good scenes, it does not reach the same point in the latent space with the trajectories from good initialization (cyan → purple).
This can be explained by the entanglement of the latent space [43], which makes multiple variables control the same property.
To further study this point, we plot Figure 4(c), where we use 3 different initialization for T-VAE w/ SCG.
The result shows that all three cases ﬁnd the target scene but with totally different trajectories, which provides evidence to our conjecture.
In summary, we believe the contribution of knowledge can be attribute to the entanglement of the latent space, which makes the searching easily escape the local minimum and ﬁnd the nearest optimal points.
3. A DVERSARIAL T RAFFIC S CENES G ENERATION
Task Description.
We aim to generate realistic trafﬁc scenes against segmentation algorithms as well as satisfying certain semantic knowledge rules.
The adversarial scenes are deﬁned as scenes that reduces the performance of downstream victim systems.
To generated adversarial LiDAR scenes containing various fore-/background rather than the point cloud of a single 3D object as existing studies [38, 63], a couple of challenges should be considered: First, LiDAR scenes with millions of points are hard to be directly operated; Second, generated scenes need to be realistic and follow trafﬁc rules.
Since there are no existing methods to compare with directly, we compare three methods: (1) Point Attack : a point-wise attack baseline [69] that adds small disturbance to points; (2) Pose Attack : [Table] a scene generation method developed by us that searches pose of vehicles in the scene; (3) Scene
Attack : a semantically controllable generative method based on our T-VAE and SCG.
We explore the attack effectiveness against different models of these methods, as well as their transferability.
For Pose Attack and Scene Attack , we implement an efﬁcient LiDAR model R ( x, B ) [49] (refer to Appendix A for details) to convert the generated scene x to a point cloud scene with an background B .
The task objective min L t ( x ) = max L P ( R ( x, B )) is deﬁned by maximizing the loss function L P of segmentation algorithms P .
We design three explicit knowledge rules: 1 (cid:13) roads follow a given layout (location, width, and length); 2 (cid:13) vehicles on the lane follow the direction of the lane; 3 (cid:13) vehicles should gather together but keep a certain distance.
1 (cid:13) ensure generated vehicles follow the layout of the background B and 3 (cid:13) makes the scene contain more vehicles.
Experiment Settings.
We select 4 segmentation algorithms (PointNet++ [57], PolarSeg [74],
SqueezeSegV3 [71], Cylinder3D [75]) as our victim models, all of which are pre-trained on Semantic
Kitti dataset [8].
We collect two backgrounds B (Highway and Intersection) in Carla simulator [18].
Since it is usually unable to access the parameters of segmentation algorithms, we focus on the black-box attack in this task.
Point Attack optimizes L t ( x ) with SimBA [25], while Pose Attack and
Scene Attack optimizes L t ( x ) with Bayesian Optimization (BO) [54].
For the training of T-VAE, we build a dataset by extracting the pose information of vehicles together with road and lane information from Argoverse dataset [11].
Attack follows basic trafﬁc rules while scene generated by Pose Attack violates basic physical laws.
Evaluation Results.
We show the Intersection over Union (IoU) metric for the vehicle during the attack in Figure 5.
Generally, it is harder to ﬁnd adversarial scenes in the highway background than in the intersection background since the latter has much more vehicles.
Within 100 iterations, Point Attack method nearly has no inﬂuence to the performance since it operates in very high dimensions.
In contrast, Pose Attack and Scene Attack efﬁciently reduce the IoU value.
Although
Pose Attack achieves comparable results to our method, scenes generated by it (shown in Figure 6) are unrealistic due to the overlaps between vehicles.
In contrast, scenes generated by our method only modify the vehicles within the trafﬁc constraints.
In Table 2, we explore the transferability of Point Attack and
Scene Attack .
Transferability, which means using generated samples from the Source model to attack other Target models, is crucial for evaluating adversarial attack algorithms.
Better transferability indicates that the samples carry important patterns that are ignored in most victim systems.
Although Point Attack method dramatically reduces the performance of all four victims, the generated scenes have weak transferability and cannot decrease the performance of other victim models.
However, scenes generated by Scene Attack successfully attack all models, even those not targeted during the training, which demonstrates strong adversarial transferability.
More generated scenes can be found in Appendix E
R ELATED W ORK
Incorporating Knowledge in Neural Networks.
Integrating knowledge to data-driven models has been explored in various forms from training methods, meta-modeling, embedding to rules used for reasoning.
[27] distills logical rules with a teacher-student framework under Posterior Regularization [22].
Another way of knowledge distillation is encoding knowledge into vectors then reﬁning the features from the model that are in line with the encoded knowledge [24].
These methods need to access Knowledge Graphs [19] during the training, which heavily depends on human experts.
Meta-modeling of complex ﬂuid is integrated into the NN to improve the performance of purely data-driven networks in [45].
In addition, [72] restricted generative models’ output to satisfy physical laws expressed by partial differential equations.
In the reinforcement learning area, reward shaping [50] is recognized as one technique to incorporate heuristic knowledge to guide the training.
Structured Deep Generative Models.
The original DGMs, such as GAN [23] and VAE [34], are mostly used for unstructured data.
They leverage the powerful feature extraction of neural networks to achieve impressive results [32, 10].
However, the physical world is complex since objects have diverse and structured relationships.
Domain speciﬁc structured generative models are developed via tree structure (RvNN-VAE [40]) or graph structure (Graph-VAE [59]).
Rule-based generative models are also explored by sampling from pre-deﬁned rules [37, 31, 14].
Typical applications of this kind of model are molecular structure generation [26, 29, 30], natural scene generation [51, 13], and automatic program generation [12].
Unlike these existing methods, our approach explicitly integrates knowledge during the generation process.
One valuable application of DGMs is generating samples that meet the requirements of downstream tasks [20, 65].
[1, 2] searches in the latent space of StyleGAN [32] to obtain images that are similar to a given image.
For structured data, such a searching framework transforms discrete space optimization to continuous space optimization, which was shown to be more efﬁcient [44].
However, it may not guarantee the rationality of generated structured data due to the loss of interpretability and constraints in the latent space [12].
Physical Scene Generation.
Traditional ways of scene generation focus on sampling from predeﬁned rules and grammars, such as probabilistic scene graphs used in [56] and heuristic rules applied in [18].
These methods rely on domain expertise and cannot be easily extensible to largescale scenes.
Recently, data-driven generative models [14, 64, 51, 40, 36] are proposed to learn the distribution of objects and decouple the generation of scene into sequence [64] and graph [51, 40] of objects.
Although they reduce the gap between simulation and reality, generated scenes cannot satisfy speciﬁc constraints.
Another substantial body of literature [21, 35, 24] explored learning scene graphs from images or generating scene images directly via an end-to-end framework.
Their generalization to high-dimensional data is very challenging, making them less effective than modularized methods proposed by [36, 68, 14].
Semantically Adversarial Attacks.
Early adversarial attack methods focused on the pixel-wise attack in the image ﬁeld, where L p -norm is used to constrain the adversarial perturbation.
For the sake of the interpretability of the adversarial samples, recent studies begin to consider semantic attacks .
They attack the rendering process of images by modifying the light condition [42, 73] or manipulating the position and shape of objects [4, 70, 28].
This paper explores the generation of adversarial point cloud scenes, which already has similar prior works [66, 3, 62].
[66, 3] modify the environment by adding objects on the top of existing vehicles to make them disappear.
[62] create a ghost vehicle by adding an ignorable number of points.
However, they modify a single object without considering the structural relationship of the whole scene.
C ONCLUSION
In this paper, we explore semantically controllable generation tasks with explicit knowledge integration.
Inspired by the categorization of knowledge for the scene description, we design a tree-structured generative model to represent structured data.
We show that the two types of knowledge can be explicitly injected into the tree structure to guide and restrict the generation process efﬁciently and effectively.
After considering explicit semantic knowledge, we verify that the generated data contain dramatically fewer semantic constraint violations.
Meanwhile, the generated data still maintain the diversity property and follow its original underlying distribution.
Although we focus on the scene generation application, the SCG framework can be extended to other structured data generation tasks, such as chemical molecules and programming languages, showing the hierarchical properties.
One assumption of this work is that the knowledge is helpful or at least harmless as they are summarized and provided by domain experts, which needs examination in the future.
Ethics Statement
Our proposed method aims to enhance existing data-driven generative models with explicit knowledge summarized by human experts.
In this paper, we focus on the application of physical scene generation.
Considering the broad usage of this method in human society, we identify three potentially harmful insights:
The explicit knowledge we integrate is summarized by human experts, therefore, could be biased.
With such kind of knowledge, the generated data could also be biased and even results in discrimination.
The knowledge we use might potentially contain private information.
After integrating it into the generative models, there is a risk of user information disclosure.
The knowledge-enhanced generation model is powerful to generate realistic adversarial examples.
These examples might be used to attack autonomous systems in the real world, e
g. self-driving algorithms, and cause damages.
To mitigate the potential negative ethics impacts mentioned above, we encourage research to follow these instructions:
1) Construct a strict mechanism of knowledge screening and monitor the generated samples against bias and discrimination.
We can also propose a knowledge screening mechanism or an intelligent discriminator to reduce the private information content.
3) Since our generated scenes are realistic and follow trafﬁc rules, it is reasonable to make the defense models drive safely in our generated scenes.
However, we should restrict the usage of our generated scenes only in evaluation and testing stages before deployment to help self-driving algorithms overcome their shortcomings.
Reproducibility Statement
All of our experiments are conducted on a desktop with Intel i9-9900K CPU @ 3.60GHz, NVIDIA
GeForce GTX 1080Ti, and 64GB memory.
The settings of the experiment environment are described in the code folder in the supplementary.
To make the experiment results reproductable, we provide all pre-trained models (please follow the instruction in code/README.md to download).
The hyper-parameters of the two experiments are shown in Table 3 and Table 4 in the Appendix.
The dataset we used in Synthetic Scene Reconstruction Experiment is synthesized by ourselves, which can be downloaded from the link provided in the README.md ﬁle.
This dataset is published under MIT License.
The dataset we used in LiDAR Scene Generation Experiment is based on the
Argoverse dataset, which is published by Argo AI with MIT License.
We use the entire tracking part of the dataset for training and the processed dataset can be downloaded from the link provided in the
README.md ﬁle.
A L I DAR M ODEL I MPLEMENTATION
The LiDAR model is implemented by Moller-Trumbore algorithm [49] with the PyTorch [53] package for high-efﬁciency computation.
We assume there is a plane (cid:52) V 0 V 1 V 2 in the 3D space constructed by points V 0 , V 1 , V 2 .
A ray R ( t ) with origin O and normalized direction D (we make (cid:107) D (cid:107) 2 =for simpliﬁcation) is represented as R ( t ) =
O + tD , where t is the distance between O and the end point D of the ray.
If D is the intersection between the ray and the plane (cid:52) V 0 V 1 V 2 , we can represent the ray with barycenteric coordinate:
[Equation]
11) where u and v are weights.
To simplify the equations, we deﬁne three new notations:
[Equation]
Then, we can solve the distance t in (11) by:
[Equation]
Since we know the ray direction D , we can get the 3D coordinate of the intersection with this distance t .
During the implementation, we reuse D × E 2 and T × E 1 to speed up the computation.
To make sure the intersecting point T ( u, v ) is inside the triangle (cid:52) V 0 V 1 V 2 , we need to have:
[Equation]
If these three conditions are not fulﬁlled, the intersection point will be removed.
To calculate the point cloud generated by a LiDAR, we ﬁrst convert vehicle mesh models to triangles
F with Delaunay triangulation [39].
Then, we create the array of LiDAR rays with width W and height H for 360 ◦ view.
Finally, we use (13) to calculate the intersection point between all triangles
F and LiDAR rays R i,j ( t ) in parallel to get the ﬁnal range map F H × W .
The background pointcloud will also be converted to range map B H × W by:
[Equation]
[Equation]
where ( x, y, z ) is the coordinate of one point in pointcloud, θ is used to calculate the index of row, and φ is used to calculate the index of column.
Then we mix F H × W and B H × W by taking the minimal value for each element:
[Equation]
where M i,j represent the ( i, j ) -entry of the mixed range map scene M .
Then, we convert the range map to the ﬁnal output pointcloud scene S with:
[Equation]
The entire pipeline of the above process is summarized in Figure 7.
The parameters we used for
LiDAR follow the conﬁguration of the Semantic Kitti dataset, where the channel H = 64 , the horizontal resolution W = 2048 , the upper angle is 2 ◦ , and the lower angle is − 25 ◦ .
The gap between reality and simulation can be reduced by realistic simulation and sensor models [47], but this will not be explored in this paper.
B D ETAILED T-VAE M ODEL S TRUCTURE
Our T-VAE model consists of several encoders and decoders that are related to the deﬁnition of the scene.
In this paper, we explored two experiments with two scenes: a synthetic box placement image scene and a trafﬁc point cloud scene.
In Figure 8, we show the details of the modules for two scenes.
We also use an synthetic scene as an example to show the encoding and decoding processes in Figure 9, where original data point is stored in tree structure and the generated data is also a tree structure.
The encoding process converts the tree into a stack and then encodes the information using the node type deﬁned in the tree.
The decoding process expands the tree with the predict node type from the Classiﬁer .
For the synthetic box placement image scene, there are 5 Encoder-Decoder pairs, a Sampler , and
a
Classiﬁer .
W node determines the global information such as location, and orientation of the entire scene.
P node spawns a plate object in the scene with positions and colors determined by the property vector.
Both Q and B nodes spawn a box object in the scene with positions and colors determined by the property vector.
E node serves as a stop signal to end the expansion of a branch, therefore, will not spawn anything in the scene and does not have model parameters.
The trafﬁc point cloud scene has similar deﬁnitions.
R node contains the information of the road and only has two children.
L node determines the lane information such as the width and direction.
Both Q and V nodes spawn a vehicle in the scene with positions and orientations determined by the property vector.
[Table] [Table]
C K NOWLEDGE D EFINITION
For each experiment in this paper, we design three knowledge rules.
We explain the details of the implementation of these rules.
In the Synthetic Scene Reconstruction Experiment, we calculate L Y ( x, Y t ( x )) with the following implementations:
(cid:172)
The scene has at most two plates , which can be implemented by W node has at most two P children nodes.
We traverse the entire generated tree x to ﬁnd W node, then we collect the children nodes of W and count the number of P node.
If the number is larger than 2, we calculate the cross-entropy loss between the node type and E node label.
(cid:173)
The colors of the boxes that belong to the same plate should be the same , which can be implemented by The color of P node’s children nodes should be the same.
We traverse the entire generated tree x to ﬁnd all P nodes, then we collect the colors of the children of P .
The average color ¯ c is calculated for each P node and ¯ c is used as label to calculate the MSE for all children nodes of the corresponding P node.
(cid:174)
The distance between the boxes that belong to the same plate should be smaller than a threshold γ which can be implemented by The distance between P node’s children nodes should be smaller than a threshold γ .
We traverse the entire generated tree x to ﬁnd P node, then we collect the absolute position of all its children nodes and calculate the MSE between this position and the position of P .
In the LiDAR Scene Generation Experiment, we calculate L Y ( x, Y t ( x )) with the following implementations:
(cid:172) Roads follow a given layout (location, width, and length) , which can implemented by R node follows a given layout (location, width, and length).
We traverse the entire generated tree x to ﬁnd R node, then calculate the MSE between the property vector of R and the given layout.
(cid:173) Vehicles on the lane follow the direction of the lane , which can implemented by L node follows pre-deﬁned directions.
We traverse the entire generated tree x to ﬁnd L node, then calculate the MSE between the property vector of
R and the given layout.
(cid:174) Vehicles should gather together but keep a certain distance , which can implemented by Q node has at least two Q nodes as its children until the absolute width and height of current block is smaller than thresholds w and h .
We traverse the entire generated tree x to ﬁnd Q node, then collect the type of its children nodes.
When the collected Q node type is less than 2, we calculate the cross-entropy loss between two collected node type and Q node label.
When the width and height of current block is smaller than w and h , we stop applying this rule.
After calculating all errors in L Y ( x, Y t ( x )) , we can directly use back-prorogation to calculate the gradient of latent code z and update it with gradient descent method.
D B ASELINES IN S YNTHETIC S CENE R ECONSTRUCTION D. D IRECT S EARCH
The dimension of the physical space is 6 × (2+8) = 60 , where 6 is the property dimension including position, orientation, and colors, 2 is the number of plates, and 8 is the number of boxes.
We use gradient descent with the learning rate η to directly search in the physical space.
Since there is no constraint to avoid overlaps between boxes, the generated scenes could be unrealistic.
D. V ARIATIONAL A UTO - ENCODER (VAE)
The input dimension of the encoder is the same as the searching space of Direct Search .
The VAE model has a encoder and a decoder, both of which has a ﬁxed number of model parameters.
There are 4 hidden layers in the encoder and each layer has 128 neurons.
The decoder also has 4 hidden layers with 128 neurons.
For the output of color, we add a Sigmoid ( · ) function to normalize it to the range [0 , 1] .
The location is normalized by N l before it is taken into the encoder.
D. G RAMMAR VAE (GVAE)
GVAE [37] requires the input data to be described with a set of pre-deﬁned grammars.
According to the task of synthetic scene reconstruction experiment, we design 9 rules,
W → P, W → B, P → P | E, P → P | B, P → B, P → E, B → B, B → E, E → E (18) and they are represented in one-hot vector in the dataset.
The original GVAE is designed only for rule-based discrete data generation (e.g. molecules), thus we modify the structure to add continuous attribute representation.
The encoder consists of 3 1-dimensional Convolution layers with kernel size 3 × 3 .
The numbers of channels for the Convolution layers are [32 , 64 , 128] .
The decoder is an LSTM model with 128 neurons, therefore, the decoding process is sequential.
During the training stage, the maximal length of rules is ﬁxed to 20 with E → E as padding, and the decoder will output 20 rules.
The cross-entropy loss is used between the input rules and decoded rules.
During the generation stage, the rules generated from the decoder will be ﬁrstly stored in a stack and convert to the tree with the ﬁrst-in-last-out (FILO) principle.
E A DDITIONAL Q UALITATIVE R ESULTS
In Figure 10, we show samples randomly generated from VAE, GVAE, and T-VAE.
The results show that all three models are able to generate diverse samples.
Speciﬁcally, samples generated by VAE always have 2 plates and 8 boxes due to the ﬁxed input data dimension.
In contrast, samples from
GVAE and T-VAE have variable numbers of plates and boxes.
In Figure 11, we show 4 more generated scenarios from the Point Attack methods with prediction results from 4 segmentation models.
For better visualization, we also show three detailed ﬁgures.
In
Figure 12 and Figure 13, we show more results from Pose Attack and Scene Attack methods.
Scenes generated by Scene Attack follow basic trafﬁc rules.
F S EGMENTATION M ODELS IN L I DAR S CENE G ENERATION F. P OINT N ET ++
This model [57] directly uses point-wise features in the 3D space as the backbone to deal with the segmentation problem.
Although this model does not have impressive results on the Semantic KITTI dataset, we select it because it inﬂuences a lot of existing point cloud processing models.
We use the code from this repository and train the model on Semantic KITTI dataset by ourselves following the original training and testing split setting.
F. P OLAR S EG
This model [74] converts the data representation from 3D Cartesian coordinate to Polar coordinate and extracts features with 2D convolution layers.
We use the code from this repository and use the pre-trained model provided by the authors.
Since we only consider the vehicle class, we change all other labels to non-vehicle class.
F. S QUEEZE S EG V
This model [71] projects the 3D point cloud to 2D range maps and extract features with 2D convolution layers from the range maps.
We use the code from this repository and use the pre-trained model provided by the authors.
Since we only consider the vehicle class, we change all other labels to non-vehicle class.
F. C YLINDER 3D
This model [75] converts the data representation from 3D Cartesian coordinate to the Polar coordinate and divides the space into blocks with a cylinder representation.
We use the code from this repository and use the pre-trained model provided by the authors.
Since we only consider the vehicle class, we change all other labels to non-vehicle class.