IReEn: Reverse-Engineering of Black-Box Functions via Iterative Neural Program Synthesis
Abstract
In this work, we investigate the problem of revealing the functionality of a black-box agent.
More speciﬁcally, we are interested in a formal description of the behavior of such an agent.
This task is also known as reverse engineering and plays a pivotal role in software engineering, computer security, and most recently in explainability.
In contrast to prior work, we do not rely on privileged information on the black-box, but rather investigate the problem under a weaker assumption of having only access to inputs and outputs of the agent.
We approach this problem by iteratively reﬁning a candidate set using a generative neural program synthesis approach until we arrive at a program that mimics the agent’s behavior.
We assess the performance of our approach on the Karel dataset.
Our results show that the proposed approach even outperforms prior work that had privileged information on the black-box function.
Extended version available at https://arxiv.org/abs/2006.10720 .
Introduction
Reverse-engineering is about gaining insights into the inner workings of a mechanism.
In our work, we consider a program to be a blackbox function that we can only interface with it through input-output interactions.
This is a desired scenario in software engineering [9, 5] and security[8, 17].
Furthermore, a similar paradigm is used to seek interpretation for reinforcement learning agents [16].
Similar principles have also been applied to the machine learning domain to copy black-box models [15, 10, 11].
Despite all of the progress in reverse-engineering the software and machine learning models there are typical limitations in the proposed works.
E.g., in the decompilation task, one of the assumptions is to have access to the assembly code of the black-box programs which is a signiﬁcant information leak about the black-box function.
Furthermore, in deep learning, a common issue is that the reverseengineered models usually are not represented in an interpretable and human-readable form.
In recent work, neural program synthesizers are employed to recover a functional and interpretable form of a black-box program that is generated based only on I/
Os examples [1].
On close inspection, however, it turns out that these approaches also leverage privileged information by relying on a biased sampling strategy of I/
Os that was obtained under the knowledge of the black-box function.
In contrast to prior work, we propose an iterative neural program synthesis scheme to tackle the task of reveres-engineering in a black-box setting without any access to privileged information.
Despite the weaker assumptions and hence the possibility to use our method broadly in other ﬁelds, we show that in many cases it is possible to reverse engineer programs on the Karel dataset.
We even achieve better results than prior work that has access to privileged information.
We achieve this by an iterative program synthesis approach.
We query a black-box function using random inputs to obtain a set of
I/Os and reﬁne the candidate set by an iterative neural program synthesis scheme.
Note that iterative program synthesis was proposed before in [14, 6].
However, To the best of our knowledge, this is the ﬁrst iterative neural program synthesis approach that operates in a black-box setting.
To summarize, the contributions of this work are as follow: 1.
We propose an iterative neural program synthesis scheme to reverse-engineer the black-box programs.
We propose a functional equivalence metric in order to quantify progress on this task.
We evaluate our approach on Karel dataset, where our approach outperforms prior work despite having access to less information.
Problem Overview
In this section, we formulate the problem description.
We base our notation on Bunel et al.
Program Synthesis.
Program synthesis deals with the problem of deriving a program in a speciﬁed programming language that satisﬁes the given speciﬁcation.
We treat input-output pairs 𝐼 ∕ 𝑂
= { ( 𝐼 𝑘 , 𝑂 𝑘 ) } 𝐾 𝑘 =1 as a form of specifying the functionality of the program.
This problem can be formalized as ﬁnding a solution to the following optimization problem,
[Equation]
where  is the space of all possible programs written in the given language, and Ω is a measure for the programs.
The situation is illustrated in Figure 2.
As most program languages do not have a unique representation for a certain functionality.
Naturally, by adding more constraints, we obtain a nested constraint set that converges towards the feasible set of functionally equivalent programs.
w.r.t.
nested constraint sets.
Program synthesis approaches usually use privileged information via biased sampling scheme in terms of speciﬁcations [1, 2, 12].
To arrive at the speciﬁcations, one has to know the functionality of the target program 𝑃 as they are designed to capture e.g. all branches of the program.
We call these crafted speciﬁcations crafted I/Os .
In this work, we focus on a black-box setting, where no such privileged information is available.
Hence, we will have to randomly generate 𝐾 inputs { 𝐼 𝑘 } 𝐾𝑘 =1 and next query the program 𝑝 to obtain the corresponding outputs { 𝑂 𝑘 } 𝐾𝑘 =1 .
We call the obtained I/Os in the black-box setting random I/Os .
It turns out (section 4), that indeed such random input queries yield signiﬁcantly less information than the crafted I/Os .
Iterative Neural Program Synthesis
Reverse-engineering a black-box function and representing it in a high-level language is a challenging task.
The main reason is that we can only interact with the black-box function using input-output examples.
Also, solving the above constraint optimization problem is intractable.
Therefore, in the following, we relax the optimization problem to a Bayesian inference problem and show how to iteratively incorporate additional constraints in order to synthesize the desired program.
3.1 Synthesizing Programs given Input-Output Constraints
To synthesize the program in black-box setting we optimize the neural program synthesizer Ψ proposed by Bunel et al.
This generative synthesizer outputs set of programs given 𝐼 ∕ 𝑂 constraints (Equation 3).
In detail, we train an encoder-decoder model on a set of speciﬁcations { 𝐼 ∕ 𝑂 𝑖 } 𝑖 and ground-truth programs { 𝑝 𝑖 } 𝑖 .
Each speciﬁcation is a set of 𝐾 pairs 𝐼 ∕ 𝑂 𝑖 = {( 𝐼 𝑘 𝑖 , 𝑂 𝑘𝑖 )} 𝐾 𝑘 =1 where the program needs to be consistent with I/
Os, that is, 𝑝 𝑖 ( 𝐼 𝑘𝑖 ) =
𝑂 𝑘𝑖 for all 𝑘 ∈ {1 , … , 𝐾 } .
[Equation]
Here, ̂𝑃 is a set program candidate(s) { ̂𝑝 1 , ..., ̂𝑝 𝐶 } ∈ ̂𝑃 and 𝐶 ≥ 1 .
During inference, we condition synthesizer Ψ on random I/
Os to synthesize the underlying program of the target black-box function.
3.2 Sample Rejection Strategy
Naturally, we expect approximation errors of the optimization problem by the generative model.
One main source of the error is that only a limited number of constraints can be incorporated in the model.
To correct for these errors, we follow up with the sample rejection stage.
We use random I/
Os to evaluate the generated programs and score them based on the number of the I/Os which were covered by the programs.
While in principle, any failed I/
O should lead to rejecting a candidate, empirically, we ﬁnd that keeping the highest scoring samples turns out to be advantageous.
3.3 Iterative Reﬁnement
The generative model only takes a small number of constraints, while in black-box setting is unclear which constraints to use to arrive at the “functional equivalent” feasible set.
Similar problems have been encountered in constraint optimization, where delayed constraint generation techniques have been employed to deal with a large number of constraints [4].
Motivated by these ideas, we propose an iterative strategy, where we condition in each step on a set of violated constraints that we ﬁnd.
Algorithm
1 Function IterativeSynthesis( Ψ , 𝐼 ∕ 𝑂 ) :
Initialize 𝐼 ∕ 𝑂 𝑠 ⊆
𝐼 ∕ 𝑂 .
𝑠 𝑏𝑒𝑠𝑡 = 0 //
To keep the best score.
n = constant // e.g. n=  for 𝑖 ← 1 to 𝑛 do ̂𝑃 =
Ψ( 𝐼 ∕ 𝑂 𝑠 ) ̂𝑝 best , ̂𝑠 best , 𝐼 ∕ 𝑂 𝑠
=
Scoring ( ̂𝑃, 𝐼 ∕ 𝑂 ) if 𝑠 best < ̂𝑠 best then 𝑝 best =
̂𝑝 best 𝑠 best = ̂𝑠 best end end return 𝑝 best 14 End Function
We present the details of the proposed approach in Algorithm 1.
Itertive synthesis function takes synthesizer Ψ and a set of
I/Os of the black-box function (line 1).
In line 2 we randomly choose a subset of I/Os as 𝐼 ∕ 𝑂 𝑠 .
In line 3 we initial the 𝑠 best to zero.
Note that we use 𝑝 best to store the best candidate, and 𝑠 best to store the score of the best candidate.
In the iterative loop, we condition the synthesizer on the given 𝐼 ∕ 𝑂 𝑠 to get the program candidates ̂𝑃 (line 6).
Then we call Scoring function to score the program candidates in line 7.
The scoring function returns the best candidate, the score of that candidate, and the new set of 𝐼 ∕ 𝑂 𝑠 , where the 𝐼 ∕ 𝑂 𝑠 are the
I/Os which were not satisﬁed by ̂𝑝 best .
Here, ̂𝑝 best , and ̂𝑠 best store the best candidate and the score of it for the current iteration.
Then at line 8, we check if ̂𝑠 best is larger than the global score 𝑠 𝑏𝑒𝑠𝑡 , if the condition satisﬁes we update the global 𝑝 best , and 𝑠 best (line 9-10).
In line 13 we return the best candidate after 𝑛 iterations.
Experiments
In this section, we show the eﬀectiveness of our approach in synthesizing the black-box programs.
4.1 Dataset and Implementation Details
To evaluate our approach, we consider the Karel programming language.
Karel featured a robot in a grid world, where this robot can move inside the grid world and modify the state of the world using a set of predeﬁned functions and control ﬂow structures.
We use the Karel dataset published by
Devlin et al.
[3] to train and evaluate our models.
In this dataset, for each program, there is 5 I/Os as speciﬁcation, and one is the held-out for testing.
This dataset contains 1,116,854 examples for training and 2,500 examples for testing the models.
More details can be found in Appendix A.
To train the neural program synthesizer, we employ the neural networks architecture proposed by
Bunel et al.
[1], and we use that in our iterative approach as the synthesizer.
We also ﬁne-tune the pre-trained synthesizer on 100,000 pairs of random I/Os and target programs.
To ﬁne-tune the synthesizer, we use Adam optimizer [7].
We empirically ﬁnd that ﬁne-tuning the synthesizer can lead to better performance than training it using random I/Os.
During inference, we use beam search with beam width 64 and select top-k programs.
We provide more implementation details in Appendix B.
[Table]
4.2 Functional Equivalence Metric
In [1, 13] two metrics have been used to evaluate the neural program synthesizer.
Exact Match:
A predicted program is an exact match of the target if it is the same as the target program in terms of tokens.
Generalization:
A predicted program is a generalization of the target if it satisﬁes speciﬁcation and held-out
I/Os.
Both of these metrics have drawbacks.
A predicted program might be functionally equivalent to the target but not be the exact match.
On the other side, a program can be a generalization of the target by satisfying a small set of I/Os.
However, it might not implement the functionality of the target.
To overcome this issue, we propose the Functional Equivalence metric, where we consider a predicted program as an approximated functional equivalence of the target if it covers a large set of unseen I/Os.
We provide more details of this metric in Appendix C.
4.3 Evaluation
To evaluate our approach in synthesizing the black-box programs, we query each black-box program with 50 valid random inputs to get the outputs.
Note that we sample inputs from the same distribution that was used in [1].
In each iteration using the obtained 50 I/Os, we synthesize the programs, where we use 5 out of 50 I/Os to conditions on the synthesizer and use 50 I/Os to score the generated candidates.
To evaluate the generated programs, in addition to generalization and exact match accuracy, we also consider the Functional Equivalence metric.
To compute the approximated functional equivalency, we use 100 I/Os.
If the generated program satisﬁes all of 100 I/
Os we consider it as a program that is an approximately functional equivalent of the target program.
Table 1 shows the performance of our approach in diﬀerent settings in the top, and the results of the synthesizer proposed by Bunel et al.
[1] in the bottom.
These results show that when we only use random I/Os (ﬁrst row), there is a huge drop in the accuracy in all of the metrics in comparison to the results of the crafted I/Os.
Based on these results we can see that random I/Os contain signiﬁcantly less information about the target program in comparison to the crafted I/Os.
However, when we use our iterative approach with the ﬁne-tuned model (fourth row), we can see that our approach outperforms even the crafted I/Os in all of the metrics.
Note that we apply our iterative synthesis for 10 iterations.
We provide more evaluation results in Appendix D.
Conclusion
In this work, we propose an iterative neural program synthesis scheme to reverse-engineer the blackbox functions.
In contrast to previous works that have access to privileged information, we only rely on the input-output pairs.
To tackle the problem of reverse-engineering the black-box function, we propose a neural program synthesizer in an iterative scheme to search for the best program candidate by conditioning the synthesizer on a set of violated constraints.
Our evaluation on Karel dataset shows that our proposed approach even outperforms the previous work that uses privileged information.
Appendix A More Details of the Karel Task
We use the Karel dataset to evaluate our proposed approach.
This dataset was created based on Karel programming language [12].
Figure 3 shows the grammar speciﬁcation of this programming language [1].
In this grammar, we can see the details of the Karel programming language, including diﬀerent statements, conditions, and actions.
This ﬁgure shows that Karel consists of control ﬂow structures such as conditionals and loops, which make this programming language a challenging and complex task.
Figure 4 demonstrates an example of Karel task with two I/Os and the corresponding program.
To represent the Karel gird world (inputs and outputs), we use a tensor of size 16 ×
𝐻 × 𝑊 .
Here we consider each grid world has a maximum size 16 × 18 × 18 .
In this tensor representation, each cell in the grid is represented as a 16-dimensional vector.
This 16-dimensional vector corresponds to the features described in Table 2.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
Appendix B Details of the Neural Program Synthesizer
One of the components of our iterative synthesizer scheme is the neural program synthesizer.
To train this synthesizer, we follow the design of the proposed work by Bunel et al.
Here, we train an encoder-decoder architecture to generate the programs given the input-output examples.
In this section, we provide details of the neural network architecture and the training procedure of the program synthesizer.
B.1 Neural Network Architecture
The encoder-decoder program synthesizer contains an encoder to encode the given I/O to a vector and a decoder that maps the given vector(s) to a set of program candidates.
In particular, we use a convolutional neural network to encode the given I/O to a 512-dimensional vector.
In the decoder, we have a 2-layer LSTM with a hidden size 256.
Using this neural network architecture, the decoder generates the candidate programs given the encoded vectors of the I/Os.
[Table]
B.2 Training and Fine-tuning Details
To train the neural program synthesizer we use Adam optimizer [7], and the learning rate 10 −4 .
We train the model for 100 epochs with a batch size of 128.
Note that, to ﬁne-tune the synthesizer model on random I/
Os we use the learning rate 10 −5 , and we ﬁne-tune the model for 10 epochs with a batch size of 128.
Appendix C Functional Equivalence Metric
In our work, we propose a new metric to evaluate the performance of the program synthesizers.
We called this metric Functional Equivalence, where we consider the predicted program is approximately functional equivalent to the target program if it covers a large set of I/Os.
C.1 Details and Parameter
To compute the results for approximated functional equivalency, we use 100 I/Os which were not seen by the program synthesizer.
We get these I/Os by generating random inputs and query the ground truth program to get the corresponding outputs, and we check whether the random inputs hit all of the branches of the ground truth program.
In our experiments, we found that with using more I/
Os the more predicted programs we discover to not be functionally equivalent to the target programs.
We found 100 number of I/Os as a point where the number of functionally equivalent programs stay stable in the evaluations.
C.2 Examples of Functional Equivalence Programs
Here we provide examples of functional equivalent programs.
We found these examples using the results of functional equivalence accuracy.
In Figures 5 and 6 we provide two examples of functional equivalent programs, which are not exact match of each other.
In these ﬁgures, the left box shows the target black-box program, and the right box shows the output program, which was predicted by our iterative program synthesizer approach.
In Figure 5, we can see there is a diﬀerence between the target black-box program and the output program at lines 2 to 7.
However, this diﬀerence is only a swap in the ifelse condition and the corresponding functions which were used in the ifelse statement, so it does not aﬀect the functionality of the programs.
Figure 6 show another example of functional equivalent programs.
In this ﬁgure, we can see that in line 2 of the target program there is a repeat statement that was not used in the output program.
However, when we turn an agent for four times, the agent’s direction will remain the same, so turning the agent in the same place for ﬁve times is equal to turning the agent for one time, and we have trungRight() function in line 4 of the output program, so having the repeat in target program does not aﬀect the functional equivalency of these two programs.
Appendix D More Analysis of Evaluation Results
Here, we investigate the eﬀectiveness of the iterative reﬁnement approach and also the eﬀect of using the diﬀerent numbers of I/Os for the sample rejection strategy.
D.1 Eﬀectiveness of Iterative Reﬁnement
Figures 7a to 7c show the eﬀectiveness of our proposed iterative approach in 10 iterations.
In these ﬁgures, x and y axis refer to the number of iterations and the accuracy respectively.
Figure 7a shows the exact match accuracy, Figure 7b shows the generalization accuracy, and in Figure 7c we can see the results of functional equivalence metric.
In these ﬁgures, we provide results with and without ﬁne-tuning the synthesizer.
Here we can see the improvement of the exact match, generalization, and functional equivalence accuracy over the iterations.
In Figure 7c we have a margin of 7% improvement in the functional equivalence accuracy for "Random I/Os + FT + IReEn" setting after 10 iterations.
In other words, these results show that we can search for better program candidates by iteratively incorporate additional constraints.
D.2 Eﬀectiveness of Number of I/Os for Sample Rejection Strategy.
In our approach to choose one candidate among all of the generated program candidates, we consider a sample rejection strategy.
To do that, we use the random I/
Os to assign a score to the generated candidates based on the number of satisﬁed
random
I/Os.
Finally, we consider the candidate with the highest score as the best candidate and reject the rest.
Figures 8a to 8c show the eﬀect of using the diﬀerent numbers of random I/Os on scoring the candidates and ﬁnding the best program candidate.
x and y axis in these ﬁgures refer to the number of random I/
Os and the accuracy of our approaches respectively.
Figure 8a shows the exact match accuracy, Figure 8b shows the generalization accuracy and in Figure 8c we provide functional equivalence results.
These ﬁgures show that by using more random I/Os in the scoring strategy, we can ﬁnd more accurate programs that result to gain better performance.
More speciﬁcally, by using more random I/Os for scoring the candidates, we can capture more details of the black-box function and ﬁnd the best potential candidate among the generated one.
[Figure] [Figure]