Image Manipulation via Neuro-Symbolic Networks
Abstract
Image manipulation via natural language text – an extremely useful task for multiple AI applications but requires complex reasoning over multi-modal spaces.
Neuro-symbolic approaches has been quite effective in solving such tasks as they offer better modularity , interpretability , and generalizability .
A noteworthy such approach is NSCL [10] developed for the task of Visual Question Answering (VQA).
We extend NSCL for the image manipulation task and propose a solution referred to as N EURO SIM.
Unlike previous works, which either require supervised data training or can only deal with simple reasoning instructions over single object scenes; N EURO SIM can perform complex multi-hop reasoning over multi-object scenes and requires only weak supervision in the form of annotated data for the VQA task.
On the language side, N EURO SIM contains neural modules that parse an instruction into a symbolic program over a Domain Specific Language (DSL) comprising manipulation operations that guide the manipulation.
On the perceptual side, N EURO SIM contains neural modules which first generate a scene graph of the input image and then change the scene graph representation following the parsed instruction.
To train these modules, we design novel loss functions capable of testing the correctness of manipulated object and scene graph representations via query networks.
An image decoder is trained to render the final image from the manipulated scene graph representation.
Extensive experiments demonstrate that N EURO SIM is highly competitive with state-of-the-art supervised baselines.
Introduction
In this paper, our aim is to build neuro-symbolic models for the task of weakly supervised manipulation of images comprising multiple objects, via complex multi-hop natural language instructions.
We are interested in a weakly supervised solution that does not require explicit annotation in the form of manipulated images.
We rely on the key intuition that this task can be achieved simply by querying the manipulated representation without ever explicitly looking at the target image. [Figure] thing behind the large ball to big [Figure]
∗ Equal Contribution. Work done while HS, PG and MG were at IIT Delhi Neuro Causal and Symbolic AI Workshop at the 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Our solution builds on the work by Mao et al.[10] on Neuro-Symbolic Concept Learner (NSCL).
We extend this work to incorporate the notion of manipulation of a given image.
At high level, our solution works as follows – i)
We first introduce a manipulation operation as part of the DSL and then learn to parse the natural language manipulation instruction comprising these concepts via a hierarchical parser [2].
The parsed instruction is represented in the form of a symbolic program dictating the manipulation to be performed on the input image.
ii)
We make use of a pre-trained network to extract latent object representations.
iii)
We make use of concept quantization networks to ground the linguistic concepts into visual objects and their attributes.
iv)
As one of our main contributions, we design neural modules that given a manipulation operator, transform the object (scene) so as to have the characteristics (attributes) as dictated by the parsed manipulation instruction.
These neural modules are trained using novel loss functions that measure the faithfulness of the manipulated scene and object representations by accessing a separate set of query networks that are trained using VQA annotations.
v) Separately, a network is trained to render the image from a scene graph representation using a combination of L 1 and adversarial losses in the style of [9].
We refer to our system as Neuro-Symbolic Image Manipulator (N EURO SIM).
Figure 1 shows an example of (input,output) pair for N EURO SIM.
For our experiment purposes, we extend CLEVR, a benchmark dataset for VQA, to incorporate manipulation instructions and create a dataset referred to as Complex
Image Manipulation via Natural Language Instructions ( CIM-NLI ) .
Our evaluation on CIM-NLI dataset shows that we are highly competitive with state-of-the-art supervised approaches [13, 3] for this task, and specifically performing well on instructions which involve multi-hop reasoning.
Related Work
In terms of task complexity, the closest to us are approaches such as TIM-GAN [13], GeNeVA [3], which build an encoder decoder architecture and work with a latent representation of the image as well as the manipulation instruction.
They require explicit annotations in terms of manipulated images during training.
We argue that this can require a significant more annotation effort, compared to our weak supervision setting, where we only need visual question answer annotations.
Unlike us, these approaches work with purely neural models, and as shown in our experiments, their performance is heavily dependent on the amount of data available for training.
In terms of technique, the closest to our work are neuro-symbolic approaches for Visual Question
Answering (VQA) such as NSVQA [12], NSCL [10], Neural Module Networks [1] and its extensions [6, 7].
While these works’ modeling approach is similar to ours and consists of constructing latent programs, the desired tasks are different.
They solve the task of VQA, whereas our goal is to solve the task of automated image manipulation.
N EURO SIM: Neuro-Symbolic Image Manipulator
Figure 2 captures a high level architecture of the proposed N EURO SIM pipeline.
N EURO SIM allows editing images containing multiple objects, via complex natural language instructions.
Similar to Mao et al.
[10], N EURO SIM assumes the availability of a domain-specific language (DSL) for parsing the instruction text T into an executable program P .
It reasons over the image for locating where the manipulation needs to take place followed by carrying out the manipulation operation.
The first three modules, namely i) visual representation network, ii) semantic parser, and iii) concept quantization network are borrowed from the NSCL pipeline [10] and suitably customized/trained as required for our purpose.
In what follows, we describe design as well as training mechanism of N EURO SIM.
3.1 Modules Inherited from NSCL
1) Visual Representation Network: Given input image I , this network converts it into a scene graph G I =
( N, E ) .
The nodes N of this scene graph are object embeddings and the edges E are embeddings capturing relationship between pair of objects (nodes).
Node embeddings are obtained by passing the bounding box of each object (along with the full image) through a ResNet-34 [4].
Edge embeddings are obtained by concatenating the corresponding object embeddings.
2) Semantic Parsing Module: The input to this module is a manipulation instruction text T (or a question when training on VQA task) in
[Figure]
3) Concept Quantization Network: Following [10], any object in an image is defined by the set of visual attributes ( A ) , and set of symbolic values ( S a ) for each attribute a ∈ A .
For example, attributes can be shape, size, etc.
Different symbolic values allowed for an attribute are also known as concepts .
For example, S color = { red, blue, green, . . . } .
Each visual attribute a ∈ A is implemented via a separate neural network
f a ( · ) which takes the object embedding as input and outputs the attribute value for the object in a continuous (not symbolic) space.
For example, let f color : R d obj −→ R d attr represent a neural network for the color attribute and consider o
∈
R
d obj as the object embedding.
Then, v color =
f color ( o ) ∈ R
d attr is the embedding for the object o pertaining to the color attribute.
Each symbolic concept s ∈ S a for a particular attribute a (for example, different kinds of colors) is also assigned a respective embedding in the same continuous space R d attr .
Such an embedding is denoted by c s .
These concept embeddings are initialized at random, and later on fine tuned during training.
An attribute embedding ( v color in the example above) can then be compared with the embeddings of all the concepts (for example, c red , c blue , etc.)
using cosine similarity, for the purpose of various tasks such as filtering objects based on say, a specific color.
Training:
As a first step of training N EURO SIM, we train all the above three modules via a three step curriculum learning process as outlined in [10].
During this training, semantic parser is trained jointly with the concept quantization networks for generating programs corresponding to the question texts coming from the VQA dataset.
The corresponding output programs are composed of primitive operations coming from VQA DSL used in [10] (e.g. filter , count , etc.) and does not include constructs related to the manipulation.
We assume that by now, first three modules have gotten trained for high accuracy on the VQA task.
We refer the reader to [10] for more details.
3.2 Novel Modules and Training Procedure for N EURO SIM
N EURO SIM training starts with three sub-modules trained on the VQA task as described in Section 3.1.
Next, we extend the original DSL to include an additional functional sub-module within semantic parsing module, change .
Refer to appendix section A for details on the DSL.
We now reset the semantic parsing module and train it again from scratch for generating programs corresponding to image manipulation instruction text T .
Such a program is subsequently used by the downstream pipeline to reason over the scene graph G I and manipulate the image.
In this step, the semantic parser is trained using an off-policy program search based REINFORCE [11] algorithm.
Unlike the training of semantic parser for the VQA task, in this step, we do not have any final answer like reward supervision for training.
Hence, we resort to a weaker form of supervision.
In particular, consider an input instruction text T and set of all possible manipulation program templates P t from which one can create any actual program P that is executable over the scene graph of the input image.
For a program P ∈ P t , our reward is positive if this program P selects any object (or part of the scene graph) to be sent to the change manipulation network.
Once semantic parser is retrained, we clamp the first three modules and continue using them for the purpose of parsing instructions and converting images into their scene graph representations.
Scene graphs are manipulated using our novel module called change manipulation network which is describe next.
4) Change Manipulation Network: For each visual attribute a ∈ A (e.g. shape, size, ... ), we have a separate change neural network that takes the pair of (object embedding, embedding of the changed concept) as input and outputs the embedding of the changed object.
For example, let g color : R d obj + d attr −→ R d obj represent the neural network that changes the color of an object.
Consider o ∈ R
d obj as the object embedding and c red ∈ R d attr as the concept embedding for the red color, then (cid:101) o
= g color ( o ; c red ) ∈
R d obj represents the changed object embedding, whose color would be red .
After applying the change neural network, we obtain the changed representation of the object (cid:101) o
= g a ( o ; c s ∗ a ) , where s ∗ a is the desired changed value for the attribute a .
This network is trained using following losses.
[Equation]
where, h a ( x ) gives the concept value of the attribute a (in symbolic form s ∈ S a ) for the object x .
The quantity p ( h a ( x ) = s ) denotes the probability that the concept value of the attribute a for the object x is equal to s and is given as follows p
[Equation]
where, dist ( a, b ) =
( a ⊤ b − t 2 ) /t 1 is the shifted and scaled cosine similarity, t 1 , t 2 being constants.
The first loss term ℓ a penalizes the model if the value of the attribute a for the manipulated object is different from the desired value s ∗ a in terms of probabilities.
The second term ℓ a , on the other hand, penalizes the model if the values of any of the other attributes a ′ , deviate from their original values.
Apart from these losses, we also include following additional losses.
[Equation]
where c old is the original value of the attribute a of object o , before undergoing change.
Intuitively the first loss term ℓ cycle says that changing an object and then changing it back should result in the same object.
The second loss term ℓ consistency intuitively means that changing an object o that has value c old for attribute a , into a new object with the same value
c old , should not result in any change.
These additional losses prevent the change network from changing attributes which are not explicitly taken care in earlier losses (1) and (2).
For example, rotation or location attributes of the objects that are not part of our DSL.
We also impose an adversarial loss ℓ objGAN to ensure that the new object embedding (cid:101) o is from the same distribution as real object embeddings.
D stands for discriminator in equation (5).
Total loss is a weighted sum of equations (1) to (5).
Method β = 5 . 4 k β [Equation] 7 k β
[Equation]
10 k [Equation] k β [Equation]
54 k
[Table]
GeNeVA
[Table]
[Table]
[Table]
3.3 Image Rendering from Scene Graph
5) Rendering Network: Design and training methodology for this module closely follows [9].
We take multiple images { I 1 , I 2 · · · I n } and generate their scene graph { G I 1 , G I 2 · · · G I n } using the visual representation network described earlier.
Each of these scene graphs is then transformed into the final image using the scene graph to image generation architecture described in [9].
Experiments
1) Dataset: We create a multi-object multi-hop instruction based image manipulation dataset, referred to as CIM-NLI.
This dataset was generated with the help of CLEVR toolkit [8]. CIM-NLI consists
of (source image I , manipulation instruction text T , manipulated target image (cid:101) I ∗ ) tuples.
2) Baselines: We compare our model with supervised approaches, TIM-GAN[13] and GeNeVA[3]. 3) Evaluation Metrics:
[Table]
guided image
For evaluation on image manipulation task, we use two metrics - i) FID, ii) Recall@ k .
FID [5] measures the realism of the generated image.
Recall@ k measures the semantic similarity of gold manipulated image (cid:101) I ∗ and system produced manipulated image (cid:101)
I .
We compute Recall@
k
as in Zhang et al. 4) Results: Let β denote the number of examples from CIMNLI dataset seen by N EURO SIM during training.
Tableshows N EURO SIM (being weakly supervised) performs significantly better than baselines when trained with β ≤ 10 k and very close to its closest competitor with β = 20 k examples using the R @1 performance metric.
This clearly demonstrates the strength of N EURO SIM in learning to manipulate while only making use of VQA annotations. 5) Multi-hop Reasoning Performance:
Table 2 compares baselines with N EURO SIM on instructions requiring 0 -hop versus multi-hop ( 1 - 3 hops) reasoning.
When dealing with multi-hop instructions performance of TIM-GAN drops significantly, whereas N EURO SIM results in almost equal performance on both sets implying that it is effective at handling complex reasoning tasks.
6) Qualitative Assessment: Figure 3 compares the images generated by N EURO SIM, TIM-GAN, and GeNeVA.
Some of the images generated by TIM-GAN and GeNeVA contain smudges where the manipulation has been done partially, while N EURO SIM suffers less from this problem.
Overall visual appearance of TIM-GAN is better in a some cases owing to fully supervised training.
[Figure]
Conclusion and Future Work
In this paper, we present a neuro-symbolic approach N EURO SIM to solve image manipulation task using weak supervision of VQA annotations.
Our approach builds on existing work on neuro-symbolic
VQA [10] to incorporate manipulation operations.
The manipulation is achieved via generation of latent symbolic programs.
Our experiments on a newly created dataset of image manipulation demonstrates the potential of our approach compared to supervised baselines.
In future, we intend to extend this approach to other image editing operators and real images.
A Domain Specific Language (DSL)
Table 1 captures the DSL used by our N EURO SIM pipeline.
The first 5 constructs in this table are common with the DSL used in [3].
The last operation ( Change ) was added by us to allow for the manipulation operation.
Table 2 show the type system used by the DSL in this work.
It is inherited
Operation Signature [ Output ← Input ])
Semantics
Scene
ObjSet ← ()
Returns all objects in the scene.
Filter
ObjSet ← (ObjSet, ObjConcept)
Filter out a set of objects from ObjSet that have concept (e.g. red) specified in ObjConcept.
Relate
ObjSet ← (ObjSet, RelConcept, Obj)
Filter out a set of objects from ObjSet that have concept specified relation concept (e.g.
RightOf) with object Obj.
Query
ObjConcept ← (Obj, Attribute)
Returns the Attribute value for the object Obj.
Exist
Bool ← (ObjSet)
Checks if the set ObjSet is empty.
Change
Obj ← (Obj, Concept)
Changes the attribute value of the input object (Obj), corresponding to the input concept, to
Concept
B Dataset Details B. CIM-NLI Dataset
This dataset was generated with the help of CLEVR toolkit [2] by using following recipe.
First, we create
a source image I and the corresponding scene data by using Blender [1] software
[Table]
Concepts for any given object, such
[Table]
blue [Table]
cylinder [Table] etc [Table]
color
[Table]
shape [Table] concepts for any given object pair, such as RightOf [Table]
LeftOf
[Table]
etc
[Table]
For each source image I created above, we generate multiple instruction texts T ’s using its scene data.
These are generated using templates, similar to question templates proposed by [2].
For each such ( I, T ) pair, we attach a corresponding symbolic program P (not used by N EURO SIM though) as well as scene data for the corresponding changed image.
Finally, for each ( I, T ) pair, we generate the target gold image (cid:101) I ∗ using Blender software and its scene data from previous step.