Using natural language and program abstractions to instill human inductive biases in machines
Abstract
Strong inductive biases give humans the ability to quickly learn a variety of tasks.
Although meta-learning is a method to endow neural networks with useful inductivebiases, agents trained by meta-learning may sometimes acquire very differentstrategies from humans.
We show that co-training these agents on predictingrepresentations from natural language task descriptions and programs induced togenerate such tasks guides them toward human-like inductive biases.
Humangenerated language descriptions and program induction models that add newlearned primitives both contain abstract concepts that can compress descriptionlength.
Co-training on these representations result in more human-like behaviorin downstream meta-reinforcement learning agents than less abstract controls(synthetic language descriptions, program induction without learned primitives),suggesting that the abstraction supported by these representations is key.
Introduction
Humans are able to rapidly perform a variety of tasks without extensive experience [1].
Thismay be because of strong inductive biases towards abstract structured knowledge (e.g. hierarchies,compositionality, etc.)
[2, 3] that act as strong prior knowledge, enabling generalization to novelenvironments with little new data.
These biases present one of the most salient differences betweenhumans and neural network-based learners [4] and may be one of the keys to building artificial agentswith human-like generalization capabilities.
For this reason, there has been interest among machinelearning researchers in identifying and instilling these inductive biases into neural network-basedagents [5–7].
One emerging approach to implicitly bestowing inductive biases on neural networks is meta-learning[8, 9].
In meta-learning paradigms, an agent is trained not just on a single task but on a distributionof tasks, with the aim of acquiring the underlying abstractions that these tasks have in common.
However, since neural networks are not easily interpretable, it can be difficult to tell if the resultingneural networks actually acquired this abstract knowledge, or whether they have instead learnedstatistical artifacts correlated with abstract rules [10, 11].
What representations can give our artificial agents access to human inductive biases?
Previous workin reinforcement learning has shown that neural networks can improve performance through auxiliarytasks [12–14].
We build on this insight to construct auxiliary tasks that require reproducing differentkinds of representations, and examine what kinds of representations tend to produce agents withhuman-like inductive biases.
Human biases toward abstract knowledge might be linked to their ability to verbalize this knowledgethrough natural language [15, 16].
Human language descriptions can therefore act as a repository ofthis prior knowledge.
Recent work in machine learning has shown that neural network representationscan be shaped and structured through natural language supervision for various tasks [17–23].
Abstract
[Figure]
+ +
[Figure]
knowledge in humans has also been modeled by program induction [24–27], where a model directlyinfers a structured programmatic representation from data.
These programmatic representations areconstructed from primitive atoms using symbolic rules and procedures.
Recent work has leveragedneural networks to make this otherwise expensive and brittle process more scalable [ neurosymbolicmodels ; 28–30].
In these approaches, abstract knowledge is built-in explicitly into the artificial agentusing a pre-specified and built-in Domain Specific Language (DSL), which can be restrictive.
Inmany situations, we would like artificial agents that are implicitly guided towards human-like abstractknowledge [31–34] rather than explicitly building it in, since it can be hard to determine what specificconcepts to build into a DSL for any arbitrary environment.
Therefore, in this work, we focus on howto use representations from program induction to guide artificial agents that don’t have these built inconcepts.
In this work, we show that language and programs can be used as repositories of abstract priorknowledge/inductive biases of humans that one can co-train with to elicit human-like biases in otherrelated tasks.
The differences in behavior elicited by the use of different inductive biases are subtle,so we use a recent framework from Kumar et al.
[10] to build tasks that distinguish these subtledifferences.
The specific domain we use is a meta-reinforcement learning task where the agentsequentially reveals patterns on 2D binary grids.
We find that guiding meta-reinforcement learningagents with representations from language and programs not only increases performance on taskdistributions humans are adept at, but also decreases performance on control task distributions wherehumans perform poorly.
We also show a correspondence between human-generated language andprograms synthesized with library learning .
This indicates that humans use higher-order conceptsto compress their descriptions in the same way that recent library learning approaches in programinduction add new concepts to the DSL to compress program length [25].
Finally, we show evidencethat this process of abstraction through compression in these representations (either language orprogram) is a key driver of human-like behavior in the downstream meta-learning agent, whenco-trained on these representations.
Dataset and task paradigm
Our goal is to examine differences in how human-like the behavior of an agent is, as elicited bydifferent co-training objectives.
To do this, we first need a distribution that reflects human inductivebias (as well as a control which is closely matched but doesn’t directly reflect this inductive bias)– so that human-like behavior is distinguishable.
We introduce these distributions in the followingsections, as well as how we build an RL task based on these distributions.
[Figure]
[Figure]
We used the dataset collected in Kumar et al.
[11], where the authors directly sample from humans’prior expectations on two-dimensional binary grids.
These data contain the direct human priorknowledge/inductive biases and can therefore be used to study them.
They used a technique called
Gibbs Sampling with People [GSP; 35, see Fig.
2] that samples internal prior distributions by puttinghumans “in the loop” of a Gibbs sampler.
The stimulus space consisted of the space of 4 × 4 boardsgiving 16 stimulus dimensions.
Each stimulus dimension (corresponding to a tile on the board) hadtwo possible values and determined the binary color of the tile, namely, red or white.
Each GSP trialconsisted of a human participant predicting what color a single masked square in the grid shouldbe, conditional on the colors of all other squares on the grid.
The specific instructions given were“What should be the underlying color of the covered greyed tile such that the board is generated by avery simple rule?”
(Fig. 2).
Once a decision was made the resulting stimulus was passed on to a newparticipant, who repeated the task with a different masked tile.
When sampling from this distribution,the probability of each board is based on how frequently it occurred during the GSP sampling process,which reflects its probability under human priors over the space of 2D grids.
We selected the 500most probable boards to collect language and program data.
We refer to these as the ‘GSP boards’.
2.2 Constructing a matched control distribution
The control distribution of boards matches the statistics of the GSP boards but is not produced byhuman decisions (see Fig. 2).
A fully connected neural network (3 layers, 16 units each) was trainedto encode the conditional distributions of the GSP boards: a random tile is masked out, and thenetwork is trained to predict its value given the other tiles (similar to masked language models; [36]).
These conditional distributions contain all the relevant statistical information about the boards.
Thenetwork achieved an accuracy of above 99% on this task.
This network is then used to generate samples, using the exact same process as used to generatesamples from human priors using Gibbs sampling.
A board in which each tile is randomly set tored or white with probability 0.5 is initialized, and this trained network is used to predict maskedout tiles.
Since the conditional model is trained on the GSP boards, this generates a set of boardswith similar statistical properties to the original GSP boards (example: number of red tiles in the
GSP boards (Mean=8.4, SD=2.26) do not significantly differ from the control boards (Mean=7.4,
SD=2.01), p = 0 .
12 ), but also implicitly encode the priors of the conditional model (i.e. a trainedneural network instead of humans) as well.
2.3 A search task for meta-reinforcement learning
Task.
These distributions of boards can be used to construct new reinforcement learning tasks.
Weuse the same task, performance measures, and agent architecture as in Kumar et al.
In this task,the agent starts with an almost fully masked out board (all greyed tiles except for one revealed red),and has to select tiles to reveal.
The revealed tiles are red or white depending on the tile colors ofa preselected underlying board.
One board with a fixed configuration of red tiles defines a singletask.
The agent’s goal is to reveal all the red tiles, while revealing as few white tiles as possible.
The
[Figure]
L S T M
B w h i t e t il e s r e v ea l ed ( z - sc o r ed ) episode ends when the agent reveals all the red tiles.
There is a reward for each red tile revealed, anda penalty for each white tile revealed.
Agents are trained on a set of boards and tested on held out testboards.
Doing well on this task requires the agent to learn and represent the distribution over boardsin its training distribution.
The primary manipulation is whether a board is generated from humanpriors or from the control distribution (that is closely matched in terms of statistics, but generatedfrom machine priors).
Learners that do better on boards that were directly sampled from people’sprior distribution on 2D grid stimuli can be said to have a human-like inductive bias, and vice versafor machine-generated boards (see Fig. 2).
Evaluation metric.
The specific metric we use to track performance is as follows.
An agent thatdoes well on the task will reveal all the red tiles while revealing as few white tiles as possible.
Wetherefore measure performance by counting the number of white tiles revealed in the episode.
Sincedifferent boards will have different number of red tiles and thus have varying levels of difficulty, wemeasure the performance relative to a “nearest neighbor” heuristic.
This heuristic randomly selectscovered tiles adjacent to currently uncovered red tiles.
We run this heuristic 1000 times on each boardand the human/agent’s number of white tiles revealed is z-scored according to this distribution.
Az-score of below 0 means that the human/agent did better than the mean performance of the nearestneighbor heuristic.
The specific value of the z-score reflects how many standard deviations thehuman/agent did better than the mean performance of the nearest neighbor heuristic.
To comparetask performance across different task distributions and models, we use a non-parametric bootstrapindependent samples statistical test to determine significance.
Baseline experiments.
The agent architecture is an LSTM meta-learner trained with reinforcementlearning.
This architecture is inspired by Wang et al.
Our LSTM meta-learner takes the fullboard x t as input, passes it through convolutional and fully connected layers and feeds that, alongwith the previous action a t − 1 and reward r t − 1 , to 120 LSTM cells.
The agent had 16 possible actionscorresponding to choosing a tile (on the 4 × 4 board) to reveal.
The reward function was: +1 forrevealing red tiles, -1 for white tiles, +5 for the last red tile, and -2 for choosing an already revealedtile.
The agent was trained using Proximal Policy Optimization [PPO; Schulman et al.
38] using the
Stable Baselines package [39] for one million episodes.
We trained our agent on the distribution of the human-generated GSP samples (see Fig 2).
We thenevaluated the agent on a set of human-generated samples as well as machine-generated samples,both of which were held out of training.
Despite the fact that we did not train the agent on themachine-generated samples, the agent still performs significantly better on the control samples thanthe human samples ( p < 0 . 0001 )
(Fig.
3B), which is in stark contrast to humans who performsignificantly better on the human-generated samples ( p < 0 . 0001 ).
The fact that the meta-RL agentexhibits the opposite pattern of performance across human and machine-generated tasks (compared tohumans) suggests that the agent has learned a different representation than that used by the humans,reflecting a different set of inductive biases than that of the humans.
This is a replication of the resultsreported in Kumar et al.
[11] – using the same architectures and task structure, but here the agent istrained with PPO instead of A2C. Having established that humans and agents seem to use differentrepresentations to perform these tasks – presumably driven by differences in inductive bias – we nowconsider how to provide the agent with more human-like inductive bias(es).
Instilling human inductive biases with language and programs
3.1 Natural language representations
We collected natural-language descriptions of 500 of the most probable GSP boards from a naive groupof participants.
Each participant wrote descriptions for 25 unique boards in response to the followingprompt: “Your goal is to describe this pattern of red squares in words.
Be as detailed as possible.
Someone should be able to reproduce the entire board given your description.
You may be rewardedbased on how detailed your description is.”
Participants were randomly assigned boards such that eachboard had 9-12 descriptions from different participants.
As a control for the language data, we alsogenerated synthetic text descriptions.
Synthetic descriptions used a template that said “The reds are in:row X 1 and column Y 1 ,...,” for every red tile location ( X n , Y n ) .
We permuted the order in which thered tile locations were verbalized to generate multiple descriptions per board and mimic the multiplehuman descriptions per board for the human-generated descriptions.
To embed each description(both human-generated and synthetic) into a vector space with semantic meaning, we obtained the
RoBERTa
[40] sentence embedding for each description using the SentenceTransformer package( https://www.sbert.net/ , based on Reimers and Gurevych [41]).
The SentenceTransformersmodel maps text into a 768 dimensional dense semantically meaningful vector space (i.e. text samplesthat are semantically similar are close to each other in vector space).
Reimers and Gurevych [41] dothis by using a contrastive objective on a dataset of semantically-paired text where embeddings fromthe same pair are pushed closer and embeddings from different pairs are pushed further apart.
3.2 Program representations
We used DreamCoder [25] to parse the boards from the GSP experiment into programs that cangenerate them.
Given a domain-specific language (DSL) and a way to score a program comprised of
DSL functions, DreamCoder enumerates and scores programs during its wake phase .
This process isaccelerated in two ways during the sleep phase : first, by learning a neural-network-based recognitionmodel that guides searching the space of programs during wake, and second by augmenting the DSLwith subprograms abstracted from repeated expressions in existing programs using library learning .
DreamCoder alternates between wake and sleep phases for multiple cycles.
We now describe our domain and DSL.
In this domain, the goal is to produce the target board ona 4 × 4 grid by controlling a pointer that marks locations with a pen (see Fig. 1).
The pointer stateconsists of a location on the grid, an orientation in one of the four cardinal directions, and whether thepointer’s pen is down (i.e. whether it will mark at subsequent future locations).
The DSL is comprisedof the following primitives: move which moves the pointer forward from the current location based onorientation; left , right which turn the pointer, changing its orientation; pen-up , pen-down whichchange the state of the pointer’s pen; and fork which stores the pointer state, executes a program,and then restores the agent state (except that grid locations marked in the subroutine are not reset).
The score s ( ρ | l ) of a program ρ when executed from an initial location l is the sum of several penaltiesthat encourage the program to replicate the target board faithfully in an efficient manner.
First, thereis a penalty of − 10 for each marked location that is in the target board and not in the generatedprogram’s board, and a penalty of −∞ for each location that is in the generated program’s board andnot in the target board 1 .
The program is therefore strongly encouraged to mark all the tiles that aremarked in the target, and strongly discouraged from marking additional tiles that are not in the target.
Second, there are penalties that encourage the program to replicate the target board efficiently.
Thereis a penalty of − 1 for each instance of move , left , and right when the pen was down, encouragingprograms to draw boards in few steps.
The score of a program is set as the best score achieved by theprogram starting from any initial location max l s ( ρ | l ) .
In the first sleep phase, DreamCoder trains a neural network that is used to guide program search –this recognition model does so by taking the board as input and predicting programs that are likelyto solve that board based on previous successful solutions.
The recognition model consists of a16-channel 3 × 3 convolutional layer, followed by two fully-connected layers with 64 units each, andending with a linear layer.
To generate a vector-based representation of the boards that contains theinformation necessary for program induction, we use the final hidden layer of the recognition model
[Figure]
Language
[Figure]
s t r a c t .
[Figure]
A
B
C
D
[Figure]
as the representation of the board.
This embeds each board into a dense vector, similar to the naturallanguage embedding of each board.
In the library learning sleep phase, DreamCoder grows the DSL by adding library functions basedon reused components of successful programs found during wake (see Fig.
4B for an example).
Specifically, it adds program components that minimize the description length of both the currentprogram library and each successful program, once rewritten to use this new library component.
These learned library abstractions serve to compress programs found in previous wake cycles andbetter enable the search for future programs in future wake cycles.
We execute DreamCoder with andwithout library learning to test the usefulness of program representations with abstractions.
3.3 Comparing programs and language descriptions
We first consider the descriptions we obtained from humans.
We found a wide range in the levelof abstraction in human natural language descriptions.
Some are more literal and verbose in theirdescriptions whereas others use more abstract concepts to compress their descriptions (Fig. 4A).
Onaverage, human-generated descriptions were shorter than synthetically-generated descriptions, whichlisted every red tile location (Fig. 4C).
By analogy, library-learned concepts allow for compression ofprograms (Fig. 4B).
Both human-generated language (vs synthetic) and library-learned programs (vsonly primitives) have the same effect on enabling shorter description lengths (Fig. 4C).
We further found that the description lengths in human language and library-learned programs aresignificantly correlated (Fig.
4D) across boards.
In other words, a board that is more compressibleusing human language abstractions was also more compressible with library learning over programs.
This indicates that the abstractions used by humans in language overlap with the abstractions learnedby Dreamcoder during library learning.
This correlation is not significant when comparing humangenerated language description length with description lengths of programs without library learning(Fig.
4D).
This indicates that the correlation may be due to the abstractions captured through librarylearning.
An illustrative example can be seen in Fig.
1, where the program’s used library function and the phrase "U shape" within in the human language description are actually the same concept.
We verify this further by testing for the difference between these two correlations using a nonparametric permutation test.
The correlation between human-generated language description lengthand library learning program description length is statistically significantly higher ( p = 0 . 0034 )than the correlation between human-generated description length and non-library learning programdescription length.
Not only do the representations between language and programs have correlated description lengths,but the actual representations learned can be similar as well.
We consider a sample of three relativelysimple boards in Fig.
5 and conduct a representational similarity analysis.
We find that the similaritystructure of these boards overlap across modalities – i.e they are similar between human-generatedlanguage and library-learned programs, as well as between synthetic language and primitive programs.
Conversely, these matrices don’t look very similar within modality.
That is, the matrices for abstracthuman-generated language don’t look very similar to the matrices for non-abstract synthetic language,and similarly for programs.
This indicates that modality is not the key driver of similarity inrepresentation space; rather, the level of abstraction can be more salient.
This also indicates that theprocess of abstraction changes the representation in similar ways across language and programs.
3.4 Grounding agents in language and program representations
In order to guide the agent to learn a human-like inductive bias, we introduce a task grounding termto the loss function: loss =
L PPO ( θ ) + c task L task ( ˆ ψ θ , ψ ) .
Here L PPO ( θ ) is the original PPO lossfunction, θ is the current model’s parameters, c task is a hyperparameter coefficient that weights thetask embedding loss L task (see Fig. 6A for the types of task embeddings we used), ψ is the taskembedding, and ˆ ψ θ is the agent’s prediction of the task embedding.
The task embedding loss L taskis the Mean Squared Error (MSE) between the agent’s predicted and actual task embedding.
Thisforces the agent to simultaneously predict an embedding of the hidden task state (i.e. the underlyingboard) as well as come up with the best possible action for the current observation (see Fig.
6A).
Like the Baseline RL agent in Fig.
3A, we trained our learners on a training set from the humangenerated boards and evaluated its performance on a held-out test set of human-generated andmachine-generated boards.
We trained different agents to predict different kinds of task embeddingsduring training (Fig.
6A) based on language (both synthetic and human-generated language) andprograms (both no library learning and library learning).
We refer to co-training on predictingthese auxiliary representations as ‘grounding’ the agent on the corresponding representation.
Theterm ’grounding’ is often used in work that trains agents to associate data in one modality (i.e. textinstructions) to another (i.e. visual images) [42–48].
We also co-trained on directly predicting aflattened version of the underlying board stimulus, which is equivalent to co-training the agent on anautoencoder loss. Representational Similarity
[Figure]
Similarity matrices (correlation matrices for respective embeddings) for a sample of human-generated boards with line motifs for human-generated language, programs with library learning, synthetic language, and programs without library learning.
Abstract representations’ similarity profiles (human language, library programs) across modalities are more similar than those of withinmodality representations (human language+synthetic language, library learning programs+nonlibrary learning programs).
B
Human Generated (GSP) Machine Generated (Control)
P r
i m i
t i v e
P r og r a m
L i b r a r y
P r og r a m
S y n t
h e t i c
L a n g u a g e
H u m a n
L a n g u a g e
A u t o e n c o d e r
R L B a s e li n e
H u m a n 1.0 0.
A d i ff .
f
r o m c o n t r o l ( z - s c
o r e ) w h
i t e t il e s ( z - s c
o r e )
C
We set out to test whether grounding meta-learning agents on these task embeddings will result inthem producing more human-like performance.
We know humans perform better on GSP boardsthan control boards (Fig.
3B), while the generic meta-RL agent does the opposite.
Performingbetter on the GSP boards and worse on the control boards would therefore indicate more human-likebehavior.
To compare task performance across different task distributions or different models, we usea non-parametric bootstrap independent samples test to test significance.
We see that grounding with human-generated descriptions leads to a human-like inductivebias (Fig. 6B).
Training on this auxillary loss led to an increase in performance at the GSP boardsrelative to the baseline agent ( p < 0 . 0001 ).
The human-language grounded agent also performedsignificantly better at the GSP boards than the control boards ( p < 0 . 0001 ), just like humans do.
In contrast, this auxiliary loss significantly reduced performance on the machine-generated controlboards (as compared to the baseline agent, p = 0 . 021 ).
This indicates that the auxiliary loss didnot generically improve performance across the board, but rather improves performance selectivelyin the human-generated boards, and worsens it in the machine generated boards.
In other words, itmakes the agent more human-like .
The human-language grounded agent also does the best at thehuman-generated tasks across all models.
The autoencoder agent is substantially better on the GSP boards than the original agent ( p < 0 . 001 ).
However, its performance does not differ significantly between the human and machine-generatedtasks ( p = 0 . 252 ), a performance pattern different from that of humans.
Similarly, we also find thatgrounding on synthetic descriptions does not lead to human-like bias and instead has an effectsimilar to that of the autoencoder loss – it improves performance uniformly rather than selectively.
Infact, the agent using synthetic text does better in the control distribution than the GSP distribution( p = 0 . 0016 ), which is qualitatively similar to the performance pattern of the original agent (Fig.
3B),and distinctly dissimilar from human behavior.
We now move to examining the agents co-trained with program representations.
We found thatgrounding on library learning programs leads to a human-like inductive bias (Fig. 6B).
Specifically, although grounding on non-library program task embeddings does numerically better onhuman-generated tasks than machine-generated tasks, this difference is not significant ( p = 0 . 11 ).
This difference, however, is highly significant when grounding on library-learning program taskembeddings ( p < 0 . 0001 ).
In addition, grounding on library learning program task embeddings doessignificantly better at the GSP tasks than grounding on non-library learning program embeddings
( p < 0 . 0001 ).
Just as grounding on human language produces more human-like inductive bias thansynthetic language, grounding on library learning programs produces more human-like inductive biasthan grounding on programs without library learning.
This points to evidence that the level of abstraction of the task embeddings influences the extentto which the agent acquires a human inductive bias , as indicated by the differences in performanceamong low (synthetic for language, no library learning for programs) and high-level (human-generatedlanguage and library learning programs) agents.
In human-generated descriptions, humans writeabout abstract concepts (e.g. lines, shapes, letters, etc).
Similarly abstract concepts can be learnedas library functions (see Fig.
4B).
The process of abstraction in language and programs allows forcompression of the respective description and induced program (Fig.
see 4C).
The embeddings ofthese representations that were compressed through abstract concepts may therefore be distillinguseful concepts into our meta-learning agent that enables it to acquire a human-like inductive bias.
3.5 Discussion
In this work, we argue that language descriptions and program abstractions can act as repositoriesfor human inductive biases that may be distilled into artificial neural networks.
To test this idea, weconsidered a tile-revealing task with 2D grids that were directly sampled from human priors.
Wefound that grounding on either human-generated language or programs in library learning not onlyimproves machine performance on tasks that people perform well , but also impairs performance ontasks that people perform more poorly .
Although the idea of co-training agents on language to shapetheir representations has been explored before, most works utilize synthetic descriptions [18, 22].
Ourwork suggests human-generated language can lead to more human-like performance than more literalsynthetic language descriptions [17], because human-generated descriptions contain informationabout abstract concepts (e.g. lines, shapes, letters, etc) that compress description length and arereflected in tasks directly sampled from human priors, therefore better capturing human inductivebiases.
Although we used synthetic descriptions that are as literal/devoid of abstractions as possibleto test this point, synthetic descriptions built to contain such abstractions could work as well.
We also show evidence that co-training artificial agents on representations from program inductionresults in learning human inductive biases just as co-training on language does.
Programs inducedwith library-learned concepts [25] are analogous to language in that adding library-learned conceptsto the initial DSL enables compression of description length
just as how humans use abstract conceptsto compress description length.
Just like human language, programs with library abstractions betterenable learning of human-like inductive biases.
Program induction can sometimes be difficult due tothe combinatorial explosion of the program space [49] and the non-trivial decision of picking a DSL.
Co-supervision of tabula rasa artificial agents using representations from program induction can bea useful way to build agents with both the flexibility of neural networks and the human-like priorsthat program induction can provide.
Additionally, although we investigate language and programs asseparate but analogous representations that can guide the training of artificial agents, recent workhas also learned joint compositional generative models over program libraries and natural languagedescriptions [19].
An exciting direction for future work is to combine our objectives, utilizing jointprogram and language representations to instill human inductive biases in artificial agents.
There are some limitations to our work.
We use a constrained task paradigm on 2D grids.
Althoughthis gives us the advantage of being able to directly study samples of human and machine priorson the task space (Fig. 2), there still must be work done to see how language and programs caninfluence acquisition of human inductive biases in more scaled up, naturalistic, and real-worldsettings.
In such environments, it may be more difficult to determine the relevant abstractionsnecessary when implementing program induction techniques or getting enough human descriptionson such large environments.
Tackling these issues may involve, for example, using program inductionon human-generated samples from realistic computer graphics DSL’s
(see Ellis et al.
50 for 2Dcase) or using smaller amounts of human-generated descriptions to fine-tune vision-language models[51] to automate the description generation process, each of which has their own challenges.
Thesepotential problems when scaling up the approaches of this paper to such environments will be excitingfuture work.
Strong inductive biases towards abstract structure allow humans to generalize to novel environmentswithout much experience [4].
By instilling these biases in our artificial agents, we can work towardsenabling machines to demonstrate human-like general intelligence.
A AppendixA.1 Program induction with dreamcoder
DreamCoder was run in two modes for the reported analyses: with and without library learning.
Allparameters matched between the two modes except those exclusive to library learning: 4 iterations of
DreamCoder, 10 CPUs, 2 minute timeout for program enumeration, 2 minute timeout for recognitionmodel training and 50% of tasks used to train the recognition model were randomly sampled“dreams”.
The recognition model was trained to predict a unigram distribution over DSL primitivesand library components.
No task batching was enabled, so every task was solved at every iteration.
Parameters only relevant to library learning were: structure penalty λ
= 1 . 5 , refactoring steps n = 1 .
A.2 Details of training and hyperparameter tuning of meta-reinforcement learning agents
Meta-learning can be considered as a bi-level optimization problem where there is an outer-loopof learning in which the model learns a useful inductive bias across different tasks of the sametask distribution and an inner-loop of learning which takes that inductive bias and rapidly learnsor adapts within a specific task [9].
In recurrent-based meta-reinforcement learning architectureslike ours, in which tasks are fed sequentially to the model, the outer loop is explicitly implementedas a reinforcement learning algorithm that updates the weights across tasks and the inner loop isimplicitly implemented in the activation dynamics of the recurrent network [52, 53] which employsfast adaptation within a specific task.
See Ortega et al.
[54] for a formal explanation of this adaptation.
In our case, the LSTM weights are updated in the outer loop across different grids to give the LSTMa useful prior learned from patterns or abstractions seen across different grid tasks.
The activationdynamics within the LSTM utilizes this prior, along with the history of observations within theepisode, to implement a fast algorithm to solve the current grid task for the inner loop.
This work’s agent’s encoder processes the board through a convolutional layer and a fully connectedlayer and passes this output into the LSTM along with the previous action and reward (see Fig.
3B).
To implement the grounding mechanism (Fig.
6A), we take the encoder’s output, and pass it throughtwo additional fully connected layers to predict the task embedding.
All agents were trained using
Proximal Policy Optimization (PPO; Schulman et al.
38 using the Stable Baselines package Raffinet al. 39)
for one million episodes.
We performed a hyperparameter sweep separately for the agentswithout the grounding loss (i.e. original agents) and with the grounding loss, since we have to tunethe new c task weight on the grounding loss jointly.
We performed a hyperparameter sweep for: batchsize, n_steps (number of steps to run in an environment update), gamma, learning rate, learningrate schedule (constant or linear), clip range, number of epochs, the λ for Generalized Advantage
Estimate (GAE λ ), max grad norm, activation function, value loss coefficient, entropy coefficient,and grounding loss coefficient for agents with the grounding loss.
The hyperparameter sweep wasdone by sampling from the space of hyperparameter using the Tree-Structured Parzen Estimator [55].
We evaluated 200 samples of hyperparameters from the space for all agents.
Both grounding andnon-grounding agents used the same hyperparameter spaces to sweep over.
We initially did a separatehyperparameter sweep for different grounding agents, but we found in initial experiments that they allreached similar hyperparameter values and training reward after the search.
Hyperparameters wereevaluated by training on 100,000 episodes and looking at the training reward.
The environments used during test time (Fig.
3B and Fig. 6) were completely held-out during this process.
We trained eachgrounding agent (and the non-grounding agent) fifteen times and ran their policy fifteen times foreach held-out test task distribution (GSP and control tasks).
Training was run on a university clusterusing NVIDIA P100 GPUs with 16 GB of memory.
A.3 Human Language Description Experiment
We took 500 of the highest probability GSP boards and randomly assigned subjects 25 of the boardssuch that each subject gets a unique set of boards and each board gets 9-12 descriptions each fromdifferent people.
The prompt people were given was: “Your goal is to describe this pattern of redsquares in words.
Be as detailed as possible.
Someone should be able to reproduce the entire boardgiven your description.
You may be rewarded based on how detailed your description is.
“ We used
Prolific ( http://www.prolific.co ) for anonymous subject recruiting.
Participants were paid at arate of approximately $12 per hour (with a total study cost of $1844) and were fully anonymizedthrough the whole process.
All text data focused on just the board stimuli and did not include anypersonally-identifiable content.
The experiment contained writing about artificial grid stimuli, sopotential risks to subjects were very minimal.
Subjects gave their consent through a standard consent
[Table]
form at the start of the experiment (where they had to confirm that they agree and give their consent).
[Table] [Table]
activation_fn relu tanh grounding_coef 0.
d i ff .
f
r o m c o n t r o l ( z - s c
o r e ) w h
i t e t il e s ( zs c o r e )
P r
i m i
t i v e
P r og r a m
L i b r a r y P r og r a m
R L B a s e li n e
H u m a n
D N N
C o n t r o l 1.
[Figure]
For the program results, we co-trained the RL agent with representations from the recognition DNN model used in DreamCoder that proposes programs given a board.
As a comparison point, we provide results when cotraining with representations from a "generic DNN" that is trained on the
GSP boards.
Although co-training with this network does improve performance on human and machine generated boards relative to the original agent, it does not selectively improve performance on human-generated boards and impair performance on machine-generated boards like co-training with the DreamCoder recognition model does.
Even though DreamCoder uses a DNN architecture, its specific architectural modifications, training objective, and data distribution used for program induction within the DreamCoder framework play a critical role in developing representations that can be distinguished from those of a more ‘vanilla’ DNN and can serve as a distinct and effective cotraining target that can better instill human inductive biases.
A.4 Broader Impact Statement
This work aims to guide artificial agents through human-like behavior.
One way in which thiswork achieves this goal is by co-training with human-generated language descriptions.
Althoughall language data collected in this study was anonymized and devoid of personally-identifiableinformation, a larger discussion is to be had in scaling this approach up while respecting the privacyand anonymity of people’s data.
Should an approach like ours be massively scaled, a conversationabout careful checks on data privacy must be had in order to prevent leakage of private information indownstream agent behaviors.
Current artificial systems are not built explicitly to exhibit human-like qualities or behavior.
Ourwork on explicitly training in human-like biases into artificial systems’ behavior can be greatlybeneficial in terms of being more easily interpretable by humans and being more easily amenable tohuman-machine collaborations.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See towards the end of thediscussion section.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See section
A.4 of the Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We discuss potential negative societal impacts in section A.4 and human
data collection in section A.3.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Code will be
available in the supplementary material for reviewers and publicly available online
(with more extensive documentation) upon paper acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See section A.2 of appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See figure captions and section A.2 of appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See section A.2 of appendix.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the authorsresponsible for collecting the dataset and creating the task paradigm that we used.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Data and code will be available in the supplementary material and a well-documented
version will be publicly available upon paper acceptance.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See section A.3 on human data collection of appendix.
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[Yes] ee section A.3 on human data collection ofappendix.5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[Yes] See section A.3 on human data collection of appendix.(b)
Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[Yes] See section A.3 on human data collectionof appendix.(c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[Yes] See section A.3 on human data collection ofappendix.