GALOIS: Boosting Deep Reinforcement Learning via Generalizable Logic Synthesis
Abstract
Despite achieving superior performance in human-level control problems, unlike humans, deep reinforcement learning (DRL) lacks high-order intelligence (e.g., logic deduction and reuse), thus it behaves ineffectively than humans regarding learning and generalization in complex problems.
Previous works attempt to directly synthesize a white-box logic program as the DRL policy, manifesting logic-driven behaviors.
However, most synthesis methods are built on imperative or declarative programming, and each has a distinct limitation, respectively.
The former ignores the cause-effect logic during synthesis, resulting in low generalizability across tasks.
The latter is strictly proof-based, thus failing to synthesize programs with complex hierarchical logic.
In this paper, we combine the above two paradigms together and propose a novel G ener a lizable Lo g
i c S ynthesis ( GALOIS ) framework to synthesize hierarchical and strict cause-effect logic programs.
GALOIS leverages the program sketch and defines a new sketch-based hybrid program language for guiding the synthesis.
Based on that, GALOIS proposes a sketchbased program synthesis method to automatically generate white-box programs with generalizable and interpretable cause-effect logic.
Extensive evaluations on various decision-making tasks with complex logic demonstrate the superiority of GALOIS over mainstream baselines regarding the asymptotic performance, generalizability, and great knowledge reusability across different environments.
Introduction
Deep reinforcement learning (DRL) has achieved great breakthroughs in various domains like robotics control [27], video game [24], software testing [48, 51, 5], etc.
Despite its sheer success, DRL models still perform less effective learning and generalization abilities than humans in solving long sequential decision-making problems, especially those requiring complex logic to solve [16, 40].
For example, a seemingly simple task for a robot arm to put an object into a drawer is hard to solve due to the complex intrinsic logic (e.g., open the drawer, pick the object, place the object, close the drawer)
Additionally, DRL policies are also hard to interpret since the result-generating processes of the neural network remain opaque to humans due to its black-box nature [35, 28].
To mitigate the above challenges, researchers seek the programming language, making the best of both connectionism [30] and symbolism [43], to generate white-box programs as the policy to execute logic-driven and explainable behaviors for task-solving.
Logic contains explainable tasksolving knowledge that naturally can generalize across similar tasks.
Therefore, attempts have been made to introduce human-defined prior logic into the DRL models [46].
Human-written logic programs are found to be an effective way to improve the learning performance and zeroshot generalization [40].
However, such a manner requires manually written logic programs beforehand for each new environment, motivating an urgent need for automatic program synthesis.
[Figure]
Existing program synthesis approaches can be categorized into two major paradigms: imperative and declarative programming [6, 36, 29], each has its distinct limitation.
The imperative programming aims to synthesize multiple sub-programs, each has a different ability to solve the problem, and combine them sequentially as a whole program [44, 15, 17].
However, programs synthesized in such a way has limited generalizability and interpretability since the imperative programming only specify the post-condition ( effect ) while ignores the pre-condition ( cause ) of each sub-program, which is regarded as a flawed reflection of causation [8] that is prone to aliasing .
In other words, the agent will arbitrarily follow the synthesized program sequentially without knowing why (i.e., cause-effect logic).
For example, assume a task in Figure 1 that requires the agent to open the box, get the key, open the door, then reach the goal.
The synthesized imperative program would contain sub-programs: toggle_box() ; get_key() ; open_door() ; reach_goal() , each should be executed sequentially (the blue path).
However, when applying such a program to another similar task with minor logical differences: the key is placed outside the box, meaning the agent does not need to open the box.
The synthesized program becomes sub-optimal as the agent will always follow the program to open the box first.
However, the optimal policy should directly head for the key and ignores the box (denoted as the orange path).
On the other side, declarative programming aims to synthesize programs with explicit cause-effect logic [16, 10] in the form of first-order logic (FOL) [26], requiring the programs to be built on the proof system (i.e., verify the trigger condition given the facts, then decide which rule should be activated) [4].
However, due to the trait of FOL, programs synthesized in this way lack hierarchical logic and thus are ineffective in solving complex tasks [1].
To combine the advantages of both paradigms and synthesize program with hierarchical cause-effect logic, we propose a novel G ener a lizable Lo g
i c S ynthesis (GALOIS) framework 3 for further boosting the learning ability, generalizability and interpretability of DRL.
First, GALOIS introduces the concept of the program sketch [38] and defines a new hybrid sketch-based domain-specific language (DSL), including the syntax and semantic specifications, allowing synthesizing programs with hierarchical logic and strict cause-effect logic at the same time.
Beyond that, GALOIS proposes a sketchbased program synthesis method extended from the differentiable inductive logic programming [12], constructing a general way to synthesize hierarchical logic program given the program-sketch.
In this way, GALOIS can not only generate hierarchical programs with multiple sub-program synergistic cooperation for task-solving but also can achieve strict cause-effect logic with high interpretability and generalizability across tasks.
Furthermore, the synthesized white-box program can be easily extended with expert knowledge or tuned by humans to efficiently adapt to different downstream tasks.
Our contributions are threefold: (1) a new sketch-based hybrid program language is proposed for allowing hierarchical logic programs for the first time, (2) a general and automatic way is proposed to synthesize programs with generalizable cause-effect logic, (3) extensive evaluations on various complex tasks demonstrate the superiority of GALOIS over mainstream DRL and program synthesis baselines regarding the learning ability, generalizability, interpretability, and knowledge (logic) reusability across tasks.
Preliminary 2.1 Markov Decision Process
The sequential decision-making problem is commonly modeled as a Markov decision process (MDP), which is formulated as a 5 -tuple ( S, A, R, P, λ ) , where S is the state space, A is the action space,
R : S ×
A → R is the reward function, P : S ×
A → S is the transition function, and λ is the discount factor.
The agent interacts with the environment following a policy π ( a t
| s t ) to collect experiences { ( s t , a t , r t ) }
Tt =0 , where T is the terminal time step.
The goal is to learn the optimal policy π ∗ that maximizes the expected discounted return: π ∗ = arg max π E a ∼ π [ (cid:80) Tt =
λ t r t ] .
2.2 Inductive Logic Programming
Logic programming is a programming paradigm that requires programs to be written in a definite clause, which is of the form: H : − A 1 , ..., A n , where H is the head atom and A 1 , ..., A n , n ≥ 0 is called the body that denotes the conjunction of n atoms, : − denotes logical entailment: H is true if A 1 ∧ A 2 ...
∧ A n is true.
An atom is a function ψ ( ω 1 , ..., ω n ) , where ψ is a n -ary predicate and ω i , i ∈ [1 , n ] are terms.
A predicate defined based on ground atoms without deductions is called an extensional predicate.
Otherwise, it is called an intensional predicate.
An atom whose terms are all instantiated by constants is called a ground atom.
The ground atoms whose propositions are known in prior without entailment are called facts.
Note that a set composed of all the concerning ground atoms is called a Herbrand base.
Inductive Logic Programming (ILP) [19] is a logic program synthesis model which synthesizes a logic program that satisfies the pre-defined specification.
In the supervised learning setting, the specification is to synthesize a logic program C such that ∀ ζ, λ : F, C | = ζ, F, C ̸| = λ , where ζ, λ denotes positive and negative samples, F is the set of background facts given in prior; and for the reinforcement learning setting, the specification is to synthesize C such that C = argmax C R , where
R is the average return of each episode.
Specifically, ILP is conducted based on the valuation vector e ∈ { 0 , 1 } | G | , G denotes the Herbrand base of the ground atoms.
Each scalar of e represents the true value of the corresponding ground atom.
During each deduction step, e is recursively updated with the forward chaining mechanism, such that the auxiliary atoms and target atoms would be grounded.
Methodology 3.1 Motivation
As aforementioned, solving real decision-making problems, e.g., robot navigation and control [44, 34, 45], commonly requires complicated logic.
As humans, we use the “divide-and-conquer” concept to dismantle problems into sub-problems and solve them separately.
It is natural to think of generating a hierarchical logic program to solve complex problems.
This intuition, however, has hardly been adopted in program synthesis since the strict cause-effect logic program is intrinsically non-trivial to generate, let alone the one with hierarchical logic [36, 29].
In this work, we propose a generalized logic synthesis (GALOIS) framework for synthesizing a white-box hierarchical logic program (as the policy) to execute logically interpretable behaviors in complex problems.
Figure 2 shows the overview of GALOIS, comprised of two key components: 1 a sketch-based DSL, and 2 a sketch-based program synthesis method.
It is noteworthy that
GALOIS uses a white-box program as the policy to interact with the environment and collect data for policy optimization.
Here, a new DSL is defined for creating hierarchical logic programs; and the sketch-based program synthesis method based on differentiable ILP is adopted for generating effective logic for the policy synthesis.
In this way, GALOIS can synthesize white-box programs with generalizable logic more efficiently and automatically.
3.2 Sketch-based Program Language
To synthesize logic programs with both hierarchical logic and explicit cause-effect logic, we design a novel sketch-based DSL, namely L hybrid , absorbing both the advantages of imperative and declarative programming.
Figure 3 shows the detail syntax and semantic specifications of L hybrid .
It is
Program Sketch
Action Env
Herbrand base white-box program (Policy)
State 𝑠𝑠 𝑎𝑎
Interaction
Synthesize Logic Program
N o r m &
S a m p l e … … … …
Verifying (a) Sketching-based domain specific language (b) sketch-based ILP
ICFG code block N …
Exit
Entry code block𝑎𝑎 syntactic specification semantic specification
E n c od e r
E
D ec od e r
D 𝑓𝑓 𝜌𝜌 1 1 ,𝑓𝑓 𝜌𝜌 1 1 , 𝑚𝑚 𝑓𝑓 𝜌𝜌 2 1 ,𝑓𝑓 𝜌𝜌 2 1 ,𝑓𝑓 𝜌𝜌 2 1 , 𝑛𝑛 𝑓𝑓 𝜌𝜌 1 2 ,𝑓𝑓 𝜌𝜌 1 2 ,𝜃𝜃 𝜌𝜌 1 2 ,𝜃𝜃 𝜌𝜌 1 2 ,𝑓𝑓 𝜌𝜌 1 3 , 1 𝜃𝜃 𝜌𝜌 1 3 ,𝑓𝑓 𝜌𝜌 1 3 ,𝑓𝑓 𝜌𝜌 1 1 ,𝑓𝑓 𝜌𝜌 1 1 ,𝜃𝜃 𝜌𝜌 1 3 , 𝑚𝑚 𝜃𝜃 𝜌𝜌 2 3 ,𝜃𝜃 𝜌𝜌 2 3 , 𝑛𝑛 𝑓𝑓 𝜌𝜌 2 3 , 𝑛𝑛 𝑓𝑓 𝜌𝜌 2 3 ,
D ec l a r a ti v e
I m p e r a ti v e
[Figure]
noteworthy that L hybrid ensures the synthesized program follow strict cause-effect logic.
Beyond that, following L hybrid , we synthesize programs using program sketches, allowing generating hierarchical logic programs.
In the following, we describe the formal syntactic and semantic specifications first and illustrate how the program sketch derives hierarchical logic programs.
Syntactic Specification: The formal syntactic specifications of L hybrid are defined using elements from both the declarative and imperative language (shown in Figure 3(a)).
Intuitively, the declarative language demands the synthesized program follow strict cause-effect logic, while the imperative language enables programs with hierarchical logic.
In specific, imperative language elements are expression e and command c .
Term e can be instantiated as constant n or function call f , and c can be assignment statement x := e , sequential execution c ; c or control flow ( while loop).
Declarative language elements are function f and clause R .
To expose cause-effect relations, we constrains functions to be implemented declaratively: f ::= R | f R , where R represents logic clause in the form of R ::
= A “:-” A - list , where A denotes atom and A - list is the clause body.
It is noteworthy that, to implement the program sketch , we introduce a novel language element called hole function , denoted as ?? .
This hole function denotes an unimplemented logic sub-program (i.e., code block in Figure 2) to be synthesized given the constraints specified by the program sketch and its corresponding semantic specification. [Figure] [Equation] [Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure] [Equation] [Figure] Semantic Specification: Having the syntactic specification, any syntactically valid program sketch can be derived.
However, without semantic guidance (e.g., lack of task-related semantics), the synthesized program may lack sufficient hierarchical logic to efficiently solve tasks [52].
Hence, we propose leveraging the program sketch [38] and defining associated semantic specifications to guide the synthesis to generate hierarchical logic programs.
In the following, we illustrate the details of the program sketch used in this work and its formal semantic specifications, based on which sketching-based inductive logic programming is performed.
Specifically, as shown in the inter-procedural control flow graph (ICFG) illustration Figure 4, the program sketch contains three major basic blocks of hole functions (denoted as ρ ) to be synthesized: ??
WHERE , ??
HOW and ?? WHAT .
During each round of recursion, the program first executes and checks whether termination condition is satisfied, if not, an assignment statement is executed: l :=??
WHERE ( s ) by calling a hole function : ??
WHERE .
We define the meaning of hole function following the formalism of standard denotational semantics [32] in Figure 3(b).
Concretely, for ??
WHERE , the body of the synthesized clause
A WHERE ( s ) is constructed from the Herbrand base representation of the current state s , namely objects’ states and agent’s attributes: G
WHERE = { ψ j ( obj i ) : i ∈ [1 , m ] , j ∈ [1 , n ] } ∪ { ψ y ( attr x ) : x ∈ [1 , u ] , y ∈
[1 , v ] } .
The clause body entails the head atom g , which denotes an abstract object within the environment (i.e., a sub-goal that agent shall arrive during this round of recursion, e.g., key , box , etc.).
The semantic function C (cid:74) · (cid:75) evaluates the clause and returns the relative coordinates between the agent and the subgoal.
The return value is passed to the logic sub-program ??
HOW (shown as the red dashed arrow). ??
HOW deduces the direction d of next time step the agent shall move to: pos (cid:55)→ C (cid:74)
A HOW (@) |
= d (cid:75) s , where pos is the agent’s next-time-step position after execution,
A HOW (@) is constructed from Herbrand base which consists of ground atoms that applies predicates regarding numerical feature on the relative coordinates: G HOW = { ψ i ( x ) , ψ i ( y ) : i = [1 , n ] } . ??
HOW executes recursively until the sub-goal specified by ??
WHERE is achieved.
Finally, ??
WHAT deduces the action a to take to interact with the object at the sub-goal position: o (cid:55)→ C (cid:74)
A WHAT ( s ) | = a (cid:75) s , where o denotes the updated state of the interacted object.
Note that the program sketch we used is generalizable and can be applied to environments with different logic (see details in Section 4).
For tasks whose environments are significantly different from the ones evaluated in this work, modifying or redesigning the sketch is also straightforward [38, 52].
3.3 Sketch-based Program Synthesis
GALOIS interacts with the environment to collect experiences to synthesize white-box programs with hierarchical and cause-effect logic following L hybrid .
As shown in Figure 2(b), in the following, we illustrate how the program interacts with the environment, what is the structure of the program and how it is trained.
Practically, different from the black-box model, GALOIS requires different types of input and output.
Therefore, GALOIS maintains an encoder E ( · ) and a decoder D ( · ) to interact with the environment.
E ( s ) maps the state s to a set of ground atoms (formatted as valuation vector e ρ ) with the verification from L hybrid , i.e., e ρ = E ( s, L hybrid ) ).
As shown in Figure 2(b), the leftmost squares with different color represents the atoms from different hole functions (e.g., blue squares { f d =1 ,t WHERE } mt =1 denotes the atoms for ??
WHERE ( d denotes d − 1 forward-chaining steps performed)).
Based on e ρ , GALOIS outputs predicate probabilities and the Decoder maps them to the action probabilities (i.e., a ∼ D ( p ( e ρ )) , where p ( · ) denotes the deduction process).
In this way, the program is executable via fuzzy conjunction [11, 12], and the program synthesis can be performed.
Guided by the semantics of the hole functions, GALOIS performs deduction using the weights θ assigned to each candidate clauses of the specific atom (i.e., one weight θ in the weights vector θ indicates one candidate clause).
This process is shown in Figure 2(b).
The rightmost squares represent the final atom deduced in the corresponding hole function.
GALOIS combines all the ground atoms to perform a complex program.
For example, f 2 , 1 WHERE is inferred with conjunction between f 1 , 1 WHERE and f 1 , 2 WHERE .
A learnable weight is assigned to each candidate clause (e.g., θ 3 , 1 WHERE associates with the clause : gt_key():- ¬has_key(X), is_agent(X), has_key(Y), is_env(Y) which is derived with two steps of deduction, shown in Figure 4).
Now we explain in detail how a certain predicate is deduced.
Given initialized valuation vector set e ρ , the deductions of the predicates are:
[Equation]
where e τρ denotes the valuation vector for all the atoms in hole function ρ at deduction step τ (initialized to 1), which is essentially a vector that stores the inferred truth values for all the corresponding atoms.
⊕ denotes the probabilistic sum: x ⊕ y
=
x
+ y − x · y .
Specifically, given the normalized [Figure] weight vector θ τ,ψρ for the predicate ψ in hole function ρ at deduction step τ , to perform a single-step deduction, we take the Hadamard product ⊙ of θ τ,ψρ and the one-step inference results based on the valuation vector of last forward-chaining step, where h ( · ) denotes the inference function 4 .
We then obtain the deductive result by taking the sum of all the intentional predicates.
Finally, the valuation vector is updated to be e τρ by taking the probabilistic sum ⊕ of the deductive result and the last step valuation vector.
Intuitively, this process is similar to the forward propagation of a neural network, while GALOIS uses logic deduction to generate results.
The policy is trained in an on-policy manner.
For each episode, the RL agent collects experiences { ( s t , a t , r t ) }
Tt =0 by interacting with the environment using current policy π θ .
With the collected experiences, GALOIS can thus synthesize the optimal hierarchical logic program to get the maximum expected cumulative return: π θ ∗ = arg max θ E a ∼ π θ (cid:104)(cid:80) Tt {+=
+}
γ t r ( s t , a t ) (cid:105) , where θ denotes the learnable parameters in GALOIS.
We train it with the Monte-Carlo policy gradient [42]:
[Equation]
where H ( π θ ) is the entropy regularization to improve exploration [22], the γ ϵ is the hyperparameter to control the decrease rate of the entropy with time.
Experiments
To evaluate the effectiveness of GALOIS, we study the following research questions (RQs):
RQ1 (Performance): How effective is GALOIS regarding the performance and learning speed?
RQ2 (Generalizability): How is the generalizability of GALOIS across environments?
RQ3 (Reusability): Does GALOIS show great knowledge reusability across different environments?
4.1 Setup
Environments: We adopt the MiniGrid environments [7], which contains various tasks that require different abilities (i.e., navigation and multistep logical reasoning) to accomplish.
We consider four representative tasks with incremental levels of logical difficulties as shown in Figure 5.
Baselines: Various baseline are used for comparisons, including mainstream DRL approaches, i.e., value-based (DQN [23]), policy-based (PPO [31]), actor-critic (SAC [13]), hierarchical (h-DQN [20]) algorithms, and the program synthesis guided methods (MPPS [44]).
To avoid unfair comparison, we use the same training settings for all methods (see Appendix B for more details).
Performance Analysis (RQ1).
To answer RQ1, we evaluate the performance of GALOIS and other baseline methods in the training environment.
The results in Figure 6 show that GALOIS outperforms all other mainstream baselines in terms of performance in environments that require complex logic, showing that GALOIS can learn [Figure] UnlockPickup [Figure]
[Figure] [Table] [Figure] [Table]
[Figure] the comprehensive task-solving logic, leading to the highest performance. Note that in the DoorKey environment, all baseline methods can reach optimal training performance, and DQN converges the fastest. This is because the DoorKey environment is relatively simpler, whose intrinsic logic is easy to learn, and hierarchical models have more parameters than the DQN model, leading to a slower convergence speed.
Moreover, we observe that MPPS, hDQN, and GALOIS converge faster than the methods without hierarchy in environments that require more complex logic (e.g., UnlockPickup,
BoxKey).
We can thus conclude that introducing hierarchy contributes to more efficient learning.
Besides, unlike other pure neural network baselines, GALOIS and MPPS present steadier asymptotic performance during training with also smaller variance.
This result demonstrates the effectiveness of introducing program synthesis for steady policy learning.
Specifically, MPPS theoretically fails on MultiRoom as there exists no deterministic imperative program description (denoted as N/A in Table 2).
The reason is that the sequence of colored doors that the agent should cross differs for each episode (e.g., ep1: red_door → yellow_door → blue_door , ep2: blue_door → red_door → yellow_door ), thus the program on solving this task is dynamically changing, which fails the imperative program synthesizer.
This further indicates the importance of synthesizing declarative programs with cause-effect logic instead of merely finding the ordered sequence of subprograms for solving tasks.
More details are discussed in the following sections.
4.3 Generalizability Analysis (RQ2).
To answer RQ2, we evaluate models’ performance on test environments with different sizes and tasksolving logic (i.e., semantic modifications).
Concretely, as shown in Table 1, GALOIS outperforms all the other baseline methods (highest average returns are highlighted in gray) and maintains near-optimal performance.
Furthermore, we also observe that all other baseline methods maintain acceptable generalizability.
This contradicts the conclusion in [16] that neural network-based agents fail to generalize to environments with size changes.
We hypothesize that this attributes to the use [Figure] [Figure] [Table]
of different types of representations.
To evaluate the effectiveness of using the valuation vector representation, we conduct experiments using the observations directly obtained from environments (e.g., the status and locations of objects).
Surprisingly, though achieving decent performance in the training environment, all the vanilla neural network-based baselines perform poorly on test environments of different sizes.
Therefore, we conclude that by introducing logic expression as state representation (in the form of valuation vectors), better generalizability can be obtained.
However, as illustrated by the results, the valuation vector itself is not enough to achieve supreme generalizability,
GALOIS manages to achieve even better generalizability due to explicit use of cause-effect logic with a hierarchical structure.
We then evaluate models’ generalizability on two test environments with minor semantic modifications, namely BoxKey ( sem-mod ) and UnlockPickup ( sem-mod ), as shown in Figure 7 (left).
Specifically, for UnlockPickup ( sem-mod ), different from the training environment, there is no key in the environment, and the door is already open.
And thus the agent should head for the target location directly.
For BoxKey ( sem-mod ), the key is placed outside the box.
Thus, optimally, the agent should directly head for the key and ignore the existence of the box.
The results in Table 2 indicate that all the baselines are severely compromised while GALOIS retains near-optimal generalizability.
This attributes to its explicit use of cause-effect logic.
Figure 7 (right) shows an example synthesized program of GALOIS (we include more synthesized program examples in Appendix A).
E.g .
The program specifies the cause of gt_box() (marked in bold) as the agent has no key and there exists no visible key in the environment.
Thus when placed under BoxKey( sem-mod ), GALOIS agent would skip gt_box() and head directly for the key since gt_box() is grounded as false by the logic clause body.
The result indicates that the explicit use of effect-effect logic is not only verifiable for humans but allows GALOIS model to perform robustly in environments with different task-solving logic.
For MPPS, since it only learns a fixed sequence of sub-programs it fails to generalize.
E.g .
the synthesized program of MPSS trained on BoxKey is: toggle_box() ; get_key() ; open_door() ; reach_goal() , thus when the key is placed under
BoxKey( sem-mod ), the agent would follow the learned program and redundantly toggle the box first.
4.4 Knowledge Reusability Analysis (RQ3)
To answer RQ3, we initialize a GALOIS model’s weights with the knowledge base learned from other tasks (e.g., DoorKey → BoxKey) and fine-tune the entire model continuously.
Figure 8 shows the detailed results of knowledge reusability among three different environments.
Apparently, the learning efficiency can be significantly increased by warm-starting the weights of the GALOIS model with knowledge learned from different tasks with overlapped logic compared with the one that is
[Figure]
learned from scratch.
Furthermore, we demonstrate the reusability of knowledge from each subprogram, respectively.
The results show that a considerable boost in learning efficiency can already be obtained by reusing knowledge from each sub-program respectively (e.g., BoxKey(DoorKey-where) agent is only warm-started with the sub-program of ??
WHERE ), which is an advantage brought by GALOIS’s hierarchical and cause-effect logic.
Figure 9 shows an example of knowledge reusing from DoorKey to BoxKey environments ( BoxKey(DoorKey-full) ).
By reusing the logic learned from the DoorKey environment (the orange path in Figure 9), agent only needs to learn the cause-effect logic of toggle_box() from scratch, which greatly boosts the learning efficiency.
Related Work
[Figure] Neural Program Synthesis: Given a set of program specifications (e.g., I/O examples, natural language instructions, etc.), program synthesis aims to induce an explicit program that satisfies the given specification.
Recent works illustrate that neural networks are effective in boosting both the synthesis accuracy and efficiency [9, 6, 3, 12, 18].
Devlin et al.
[9] propose using a recurrent neural network to synthesize programs for string transformation.
Chen et al.
[6] further proposes incorporating intermediate execution results to augment the model’s input state, which significantly improves performance for imperative program synthesis.
∂ ILP [12] proposes modeling the forward chaining mechanism with a statistical model to achieve synthesis for Datalog programs.
Program Synthesis by Sketching:
Many real-world synthesis problems are intractable, posing a great challenge for the synthesis model.
Sketching [38, 52, 37] is a novel program synthesis paradigm that proposes establishing the synergy between the human expert and the synthesizer by embedding domain expert knowledge as general program sketches (i.e., a program with unspecified fragments to be synthesized), based on which the synthesis is conducted.
Singh et al.
[37] propose a feedback generation system that automatically synthesizes program correction based on a general program sketch.
Nye et al.
[25] propose a two-stage neural program synthesis framework that first generates a coarse program sketch using a neural model, then leverages symbolic search for second-stage fine-tuning based on the generated sketch.
Program Synthesis for Reinforcement Learning: Leveraging program synthesis for the good of reinforcement learning has been increasingly popular as it is demonstrated to improve performance and interpretability significantly.
Jiang et al.
[16] introduce using ∂ ILP model for agent’s policy, which improves downstream generalization by expressing policy as explicit functional programs.
Imperative programs are used as a novel implementation of hierarchical reinforcement learning in which the agent’s policy is guided by high-level programs [40, 44, 15].
In addition, program synthesis has also been used as a post hoc interpretation method for neural policies [41, 2].
Conclusion
In this work, we propose a novel generalizable logic synthesis framework GALOIS that can synthesize programs with hierarchical and cause-effect logic.
A novel sketch-based DSL is introduced to allow hierarchical logic programs.
Based on that, a hybrid synthesis method is proposed to synthesize programs with generalizable cause-effect logic.
Experimental results demonstrate that GALOIS can significantly outperform DRL and previous program-synthesis-based methods in terms of learning ability, generalizability, and interpretability.
Regarding limitation, as it is general for all program synthesis-based methods, the input images need to be pre-processed into Herbrand base for the synthesis model, which is required to be done once for each domain.
Therefore, automatic Herbrand base learning would be an important future direction.
Another promising direction is applying
GALOIS in more challenge competitive multi-agent scenarios [50, 14, 47] or cooperative multi-agent scenarios [21, 39, 49].
We state that our work would not produce any potential negative societal impacts.
Acknowledgments and Disclosure of Funding
This work is supported by the National Natural Science Foundation of China (Grant No. 62106172),
New Generation of Artificial Intelligence Science and Technology Major Project of Tianjin (Grant
No.: 19ZXZNGX00010), Science and Technology on Information Systems Engineering Laboratory (Grant No. WDZC20205250407), National Research Foundation, Prime Ministers Office, Singapore under its National Cybersecurity R&D Program (Award No. NRF2018NCR-NCR005-0001), NRF
Investigatorship NRF-NRFI06-2020-0001, the Ministry of Education, Singapore under its Academic
Research Fund Tier 1 (21-SIS-SMU-033), Tier 2 (MOE2019-T2-1-040), and Tier 3 (MOET320200004).
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the Ministry of Education, Singapore.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Section(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re us- ing/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots,
applicable
N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals
if applicable
N/A] (c
Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]