Finding Sub-task Structure with Natural Language Instruction
Abstract
When mapping a natural language instruction to a sequence of actions, it is often useful to identify sub-tasks in the instruction.
Such subtask segmentation, however, is not necessarily provided in the training data.
We present the
A2LCTC (Action-to-Language Connectionist
Temporal Classification) algorithm
to automatically discover a sub-task segmentation of an action sequence.
A2LCTC does not require annotations of correct sub-task segments and learns to find them from pairs of instruction and action sequence in a weakly-supervised manner.
We experiment with the ALFRED dataset and show that A2LCTC accurately finds the sub-task structures.
With the discovered subtasks segments, we also train agents that work on the downstream task and empirically show that our algorithm improves the performance.
Introduction
Building computational agents that execute actions given natural language instruction has a great deal of potential in real-world applications.
One common approach is to cast the problem as mapping from instruction text into action sequence, and train an agent with supervised learning (Chen and
Mooney, 2011; Mei et al., 2016).
A challenge on effective machine learning stems from a long horizon of the tasks.
Typical navigation tasks often involve more than a paragraph of instructions (Chen and Mooney, 2011; Misra et al., 2017; Shridhar et al., 2020).
In such cases, many existing approaches exploit the task hierarchy, e.g. , decompose one episode of the task into sub-tasks and treat them as separate training instances (Mei et al., 2016), incorporate the hierarchical information into the model (Zhu et al., 2020), or aid the learning process with progress monitoring (Ma et al., 2019).
However, annotations for such a decomposition are not necessarily available in training data.
In this case, previous work has attempted to perform sub-task segmentation by augmenting data through crowdsourcing (Hong et al., 2020), or developing a heuristic algorithm (Zhu et al., 2020).
In this paper, we present A2LCTC (Action-toLanguage Connectionist Temporal Classification), an unsupervised algorithm that automatically discovers a sub-task segmentation of an action sequence.
Given pairs of a natural language instruction and action sequence, A2LCTC maps each action to a fine-grained instruction (Figure 1).
We formulate the problem of sub-task segmentation as classification of each action into a finegrained instruction.
Inspired by the connectionist temporal classification algorithm (Graves et al., 2006), we consider an objective function that maximizes the log-likelihood of the coarsely aligned data.
This formulation allows us to learn the classification in a weakly-supervised manner without any need of the ground truth mapping.
We experiment with the ALFRED dataset which involves navigating a robot to perform household task (Shridhar et al., 2020).
A2LCTC successfully discovers the sub-task information in the unsegmented training data (§3), which are shown to be useful for the downstream task (§4).
Learning Sub-task Segmentation
Given a training instance of a navigation task which consists of language instruction X and action sequence A =
[ a 1 , ..., a T ] , we aim at finding a decomposition of the instance ( X, A ) =
[( X 1 , A 1 ) , ..., ( X L , A L )] , which is semantically plausible and useful for learning the task.
Our approach starts with dividing the instruction into fine-grained segments.
In the experiment, the segment X i corresponds to a verbal phrase ( e.g. , “ go to the desk ”) extracted from the instruction using a simple rule-based algorithm (Appendix A.1).
2. Formulating the Task as Temporal Classification
We formulate the decomposition task as classification of actions into one of the fine-grained instructions: our algorithm predicts a mapping π =
[ π 1 , ..., π T ] where π t ∈ [1 , L ] .
We assume that the alignment is monotonic, i.e. , the actions and instructions are both arranged in a chronological order, which typically holds for stepby-step instruction text 1 .
This formulation allows us to develop an unsupervised learning method to effectively solve the task without ground-truth label mappings.
We introduce A2LCTC (Action-toLanguage Connectionist Temporal Classification), which is based on the CTC algorithm originally used for speech recognition (Graves et al., 2006) or action labeling in video (Huang et al., 2016).
Our model attempts to maximize the following likelihood for each training instance:
[Equation]
where B is the operator to remove repeated labels, e.g., , B ([1 , 1 , 2 , 2 , 2 , 3])
=
That is, the objective is the sum of the probabilities over all the possible assignments under the monotonic constraint.
Under the conditional independence assumption, we further decompose the likelihood into P
[Equation]
With this decomposition, we employ the forward algorithm (Stratonovich, 1965) to efficiently calculate the sum over all possible paths.
2. Modeling with Neural Network
We model the probability computation using neural networks.
In our approach, each fine-grained instruction and action are represented as feature vectors.
We then define the probability P ( π t | X, A ) as the softmax of the dot product of the feature vectors of the action a t and the instruction [ x 1 , ..., x L ] :
[Equation]
In our implementation, a fine-grained instruction
X i is tokenized into words and x i is computed as the average of the word embeddings followed by a linear layer.
The action feature vectors are computed by feeding action embeddings [ ˆ a 1 , ..., ˆ a T ] into a one-layer bidirectional LSTM (Hochreiter and Schmidhuber, 1997) to capture the semantics of actions in the context.
At inference time, the model induces the best mapping π that maximize P ( π | X, A ) with the viterbi algorithm to form decomposed instances ( X i , A i ) .
2. Stabilization of Unsupervised Learning
Optimizing the aforementioned objective function with neural networks turns out to be highly unstable.
Therefore, we further employ the following techniques to better guide the training process.
Length-based curriculum learning .
The training objective (1) is defined as the sum of the probabilities over all possible assignments, whose size grows at the speed of O ( L T ) .
When L and T are large, too many degenerated assignments take up a large part of the probability mass, which hinders the training especially at the beginning of training.
To address this problem, we adopt the curriculum learning framework (Bengio et al., 2009).
At the beginning of training, we restrict the training instances with a shorter action sequence and gradually expose the model to longer instances (Spitkovsky et al., 2010).
Details are shown in
Appendix A.4.
Initializing with pre-trained word embeddings .
We initialize the word embeddings w with the
Glove embeddings (Pennington et al., 2014) to inject a model with the prior knowledge of words.
Initializing with pre-trained action embeddings .
We also initialize the action embeddings ˆ a with pretrained embeddings.
We obtain action embeddings from the action sequences in the training data with
Skip-gram (Mikolov et al., 2013).
Experiments
We test A2LCTC with the ALFRED dataset (Shridhar et al., 2020) 2 , where the agent performs household tasks in a simulated indoor environment given
English instruction.
We choose this dataset because (1) the data involves relatively long sequence of actions (over 100); (2) it offers ground-truth sub-task segmentation data which allows the evaluation of our algorithm.
Although the ALFRED task itself involves understanding visual inputs, which is currently beyond the scope of A2LCTC, the action set defined in the dataset is semantically rich enough for A2LCTC to solve the segmentation problem.
3. Experimental Setups
Training and Evaluation Data .
The dataset contains expert demonstrations, which are split into the training set and two validation sets: valid_seen and valid_unseen .
In valid_seen , the ALFRED tasks are given in the same set of environments as the training data, while unseen environments are used for valid_unseen .
We use valid_seen as the validation set during training and report the results evaluated with both of them.
The task defines 12 types of actions: five for navigation (e.g., LookUp , MoveForward ,
RotateRight ) and seven for interaction (e.g.,
Pickup , Slice ).
To simplify training and evaluation, we preprocess data by merging the same consecutive navigation actions into a single action (see Appendix B for detailed statistics on the data).
Evaluation Metrics .
Each training instance in the
ALFRED dataset contains ground-truth sub-goal segments for instructions and actions.
We use them to evaluate the learned sub-task segments.
Note that our algorithm operates on finer-grained instructions segmented into verbal phrases, while the ground-truth segments are coarser; some sentences contain a couple of verbal phrases ( e.g. , “ Rinse the sponge out in the sink, and pick it up again ”).
In our evaluation, we first merge the finegrained instructions into their corresponding subgoal instruction, together with the mapped actions, and then compare the overlap with the gold subgoal segments of actions.
We report the sub-goal exact match (EM) score defined as the percentage of the sub-goal segments perfectly reconstructed.
We also report the subgoal F1 score defined as the macro average of the
F1 scores calculated by taking the overlap between the predicted and ground-truth segment.
3. Baselines
Under our task formulation, A2LCTC offers an advantage that it can take into account the temporal constraint and the semantics of instructions.
However, many existing unsupervised sequence alignment algorithms ( e.g. , IBM models) operate only between discrete symbols and are not directly applicable in this situation where we need to align actions to text (or a bag of words in our model).
Thus, we compare A2LCTC with two baselines that do not consider the textual information.
Uniform .
The uniform baseline assigns an equal number of actions to each of the fine-grained instructions.
Byte-pair Encoding .
The byte-pair encoding (BPE) baseline is based on a data compression algorithm that finds repeated patterns in the data and merges them into chunks (Gage, 1994; Sennrich et al., 2016).
Concretely, we repeat the following two steps until each action sequence is split into the number of the corresponding fine-grained instructions: (1) count bigrams in the action sequences; (2) merge the most frequent bigrams.
3. Results
Table 1 shows that A2LCTC significantly outperforms the baselines, which indicates that our model successfully leverages textual information and learns meaningful alignments between the finegrained instructions and actions.
Although the BPE baseline does not use textual information, it exhibits reasonable F1 scores (61.
points in valid_unseen and much higher EM scores than U NIFORM (27.1 vs. 9.3 points).
This reflects the characteristics of the ALFRED dataset.
As each episode in the dataset is generated from specific templates, the actions follow specific patterns, which enable the BPE to learn correct segmentation to some extent.
3. Ablation Study
In A2LCTC, we utilize several techniques to stabilize the training process.
Table 2 shows the effects of ablating one stabilization method from the full
A2LCTC model.
Our results indicate that curriculum learning is most essential to successful training.
Without curriculum learning, A2LCTC suffers from significant performance degradation (41.2 → 14.6 in EM and 78.2 → 36.7 in F1 score).
[Table] [Table]
Ablating pre-training language or action embeddings still obtains mean values comparable to the full A2LCTC but yields much larger standard deviations.
This indicates that these two stabilization methods are also beneficial for A2LCTC.
Evaluation with the Downstream Task
We evaluate the effectiveness of sub-task segments induced by A2LCTC on the downstream ALFRED task.
Models .
Our baseline agent (B ASELINE ) is based on the CNN-LSTM sequence-to-sequence architecture in Shridhar et al.
(2020), which takes the whole instruction and current state as input and then predicts an action at each time step.
Unless specified, we use the same hyperparameters as the original implementation.
To incorporate the sub-task information, we extend the baseline with the progress monitoring module (Ma et al., 2019).
We use two progress monitoring schemes from Shridhar et al.
(2020), which estimate the current time step and the completed sub-tasks.
Specifically, the modules are trained to predict the proportion of elapsed steps or completed sub-tasks to the total numbers.
We evaluate the segmentation of A2LCTC as well as the two baseline methods (U NIFORM and
BPE).
Those algorithms are applied on the training data and the agents are trained with the progress monitoring according to the segmentation.
Metric .
We evaluate the agents with the subgoal sequence accuracy.
The agent predicts the next valid_seen valid
[Table]
action given the history from an expert trajectory.
The metric measures how many subgoal chunks of actions, which is defined by the ground-truth segmentation of the dataset, are successfully predicted in the evaluation data.
Note that this metric simplifies the original task in that it ignores the object interactions and focuses on action prediction 3 .
Results .
Table 3 summarizes the result.
On both valid_seen and valid_unseen splits, the agents trained with fine-grained instruction (U NIFORM ,
BPE and A2LCTC) significantly outperform
B ASELINE .
The fact that U NIFORM achieves improvement indicates that keeping track of detailed progress is helpful even if it inaccurately performs the fine-grained task segmentation (see Section 3.3).
A2LCTC performs best because of the better accuracy of the segmentation than the others.
This demonstrates that A2LCTC successfully provides more informative instructions for the agent in solving the downstream task.
Conclusion
We presented A2LCTC, which finds a hierarchical structure of an action sequence by mapping each action to fine-grained natural language instructions without ground-truth mapping data.
We demonstrated that A2LCTC successfully learns meaningful segments and training the ALFRED agents with these segments leads to improved performance.
A2LCTC currently relies only on semantic correspondence between actions and text.
Applying
A2LCTC to the tasks with low-level actions is an important extension, e.g. , actions specifying the direction to move.
Furthermore, the instruction may often describe the visual input whose information is not encoded in the actions.
Another important future direction is to incorporate visual or additional information to tackle a broader range of domains.
A Implementation details of A2LCTC A. Instruction Decomposition
We split a whole instruction into fine-grained instructions each of which is a verbal phrase.
Our current implementation employs a simple rule-based algorithm, which segments a whole instruction at periods, commas, phrases such as and or then .
For example, “ Turn around and go back to the table ” will be segmented into [“ Turn around ”, “ go back to the table ”], while we also write few rules to handle erroneous splits such as conjunctions between nouns ([“ take the apple ”, “ (and) banana ”]), commas before a prepositional phrase ([“ Put the bowl ”, “ (,) on the coffee table to the left of the statue ”]), or verbal phrases that cannot be instruction by itself ([“ go to the left ”, “ (and) face the bathtub ”]).
A. Neural Network Architecture Instruction Feature Vectors
The feature vector of a fine-grained instruction X l is simply modeled by taking the average of the word embeddings [ w 1 , ..., w N ] followed by a linear layer (the results from different encoding strategies are shown in Appendix C).
[Equation]
In our experiment, the dimension of word embeddings, the input and output size of the linear layer is all set to 50.
The total number of parameters of A2LCTC is about 44K.
The training takes approximately one hour with a single GPU.
Action Feature Vectors
The action feature vectors are computed through feeding embeddings for primitive actions [ ˆ a 1 , ..., ˆ a T ] to a one-layer bidirectional LSTM.
[Equation] [Equation]
where [ ˆ a 1 , ..., ˆ a T ] are embeddings for primitive actions.
The size of the action embeddings and the hidden size of the LSTM are set to 50.
The outputs of LSTM in the forward and backward directions are summed to merged into one single feature vector.
A. Hyperparameters for training
[Table]
A. Stabilization of Unsupervised Learning Length-based curriculum learning
We set the maximum length of training instances for each training epoch according to a schedule.
In our experiment, the maximum length starts at 20, and linearly increases to 60 with 30 steps.
Initializing with pre-trained action embeddings
We use the gensim library 4 to train action embeddings.
We use the Skip-gram algorithm and the hyperparameters are shown in Table
[Table]
B Data Statistics
Our experiments are based on the expert demonstration data in the ALFRED dataset (Table 5).
[Table]
After the instruction decomposition, the instructions contain 10 fine-grained instructions on average.
The entire distribution is shown in Figure2.
To simplify training and evaluation, we preprocess data by merging the same consecutive actions into one single action.
This results in the average sequence length of 25.
The entire distribution before and after the merge preprocessing is shown in Figure3 and 4.
C Additional Results for the Segmentation Task
Here we compare different strategies for encoding instruction segments.
Besides the mean pooling of word embeddings followed by a linear layer in A2LCTC (M EAN ), we also tried the summation of word embeddings (S UM ), the mean pooling of one-layer Bi-LSTM outputs LSTM.
The hidden size of LSTM is set to 50, which is the same as the word embeddings, and the vectors of the forward and backward directions were summed to form the output vectors.
The result is shown in Table 6.
[Table]
We find that M EAN gives the most stable result.
LSTM exhibits the worst performance, indicating that it is hard to optimize the unsupervised objective in A2LCTC with an overly complex architecture.
D Additional Results for the Downstream Task D. Results with the ground-truth sub-goal annotation
In section 4, we compared the models trained with automatically generated fine-grained sub-task segments.
Here we provide the results from the model trained with the ground-truth sub-goal segments (S UBGOAL ) in Table 7.
Note that the granularity of S UBGOAL is coarser than the other models.
[Table]
S UBGOAL provides better results than B ASELINE , which demonstrates the benefit of the ground-truth sub-gold segmentation in the dataset.
However, the improvement is limited compared to the U NIFORM segmentation, which segments an action sequence into the chunks of the same size.
The model benefits from inaccurate but finer-grained segmentation more than accurate but coarse segmentation.
We hypothesize that this reflects the characteristics of the dataset.
The ALFRED dataset is created by generating expert trajectories from task templates.
As a result, the type of actions are somewhat correlated with the time step within an episode.
For example, navigation actions such as MoveForward are more likely to be executed at the beginning of the episode, whereas interactive actions such as PutObject are at the end.
Adding finer-grained progress monitoring supervision at training time can help the agent learn the correlation between time steps and actions better than coarser progress monitoring.
D. Success Rates of the Downstream Task
In our preliminary experiments, we find the original navigation task is too difficult for the baseline model: the success rate is very low with high variance, which prevents meaningful comparison among the variants of the model.
The success rate (SC) and goal condition success rate (GC) are provided on Table 8.
With multiple runs, we did not observe any significant difference ( p > 0 .
05 in the Welch’s t-test) among the models
[Table]