S TRUCTURED N EURAL S UMMARIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Summarization of long sequences into a concise statement is a core problem in natural language processing, requiring non-trivial understanding of the input.
Based on the promising results of graph neural networks on highly structured data, we develop a framework to extend existing sequence encoders with a graph component that can reason about long-distance relationships in weakly structured data such as text.
In an extensive evaluation, we show that the resulting hybrid sequence-graph models outperform both pure sequence models as well as pure graph models on a range of summarization tasks.
I NTRODUCTION
Summarization, the task of condensing a large and complex input into a smaller representation that retains the core semantics of the input, is a classical task for natural language processing systems.
Automatic summarization requires a machine learning component to identify important entities and relationships between them, while ignoring redundancies and common concepts.
Current approaches to summarization are based on the sequence-to-sequence paradigm over the words of some text, with a sequence encoder — typically a recurrent neural network, but sometimes a 1D-CNN (Narayan et al., 2018) or using self-attention (McCann et al., 2018) — processing the input and a sequence decoder generating the output.
Recent successful implementations of this paradigm have substantially improved performance by focusing on the decoder, extending it with an attention mechanism over the input sequence and copying facilities (See et al., 2017; McCann et al., 2018).
However, while standard encoders ( e.g. bidirectional LSTMs) theoretically have the ability to handle arbitrary long-distance relationships, in practice they often fail to correctly handle long texts and are easily distracted by simple noise (Jia & Liang, 2017).
In this work, we focus on an improvement of sequence encoders that is compatible with a wide range of decoder choices.
To mitigate the long-distance relationship problem, we draw inspiration from recent work on highly-structured objects (Li et al., 2015; Kipf & Welling, 2017; Gilmer et al., 2017;
Allamanis et al., 2018; Cvitkovic et al., 2018).
In this line of work, highly-structured data such as entity relationships, molecules and programs is modelled using graphs.
Graph neural networks are then successfully applied to directly learn from these graph representations.
Here, we propose to extend this idea to weakly-structured data such as natural language.
Using existing tools, we can annotate (accepting some noise) such data with additional relationships ( e.g. co-references) to obtain a graph.
However, the sequential aspect of the input data is still rich in meaning, and thus we propose a hybrid model in which a standard sequence encoder is generating rich input for a graph neural network.
In our experiments, the resulting combination outperforms baselines that use pure sequence or pure graph-based representations.
Brieﬂy, the contributions of our work are: 1.
A framework for extension of standard sequence encoder models with a graph component that leverages additional structure in sequence data.
Application of this extension to a range of existing sequence models and an extensive evaluation on three summarization tasks from the literature.
We release all used code and data at http://anonymizedforsubmission .
S TRUCTURED S UMMARIZATION T ASKS
In this work, we consider three summarization tasks with different properties.
All tasks follow the common pattern of translating a long (structured) sequence into a shorter sequence while trying public void Add(string name, object value = null, DbType? dbType = null, ParameterDirection? direction = null,
int? size = null, byte?
precision = null, byte? scale =null) { parameters[Clean(name)]
= new ParamInfo{
Name = name, Value = value, ParameterDirection =
direction ??
ParameterDirection.
Input,
DbType = dbType, Size = size, Precision = precision, Scale = scale };}
Ground truth : add a parameter to this dynamic parameter list B I LSTM
→ LSTM : adds a new parameter to the speciﬁed parameter B I LSTM+GNN → LSTM : creates a new instance of the dynamic type speciﬁed B I LSTM+GNN → LSTM+P OINTER : add a parameter to a list of parameters to preserve as much meaning as possible.
The ﬁrst two tasks are related to the summarization of source code (Figure 1), which is highly structured and thus can proﬁt most from models that can take advantage of this structure; the ﬁnal task is a classical natural language task illustrating that hybrid sequence-graph models are applicable for less structured inputs as well.
M ETHOD N AMING
The aim of this task is to infer the name of a function (or method in objectoriented languages, such as Java, Python and C#) given its source code (Allamanis et al., 2016).
Although method names are a single token, they are usually composed of one or more subtokens (split using snake case or camelCase ) and thus, the method naming task can be cast as predicting a sequence of subtokens.
Consequently, method names represent an “extreme” summary of the functionality of a given function (on average, the names in the Java dataset have only 2.9 subtokens).
Notably, the vocabulary of tokens used in names is very large (due to abbreviations and domainspeciﬁc jargon), but this is mitigated by the fact that 33% of subtokens in names can be copied directly from subtokens in the method’s source code.
Finally, source code is highly structured input data with known semantics, which can be exploited to support name prediction.
M ETHOD D OC
Similar to the ﬁrst task, the aim of this task is to predict a succinct description of the functionality of a method given its source code (Barone & Sennrich, 2017).
Such descriptions usually appear as documentation of methods ( e.g. “docstrings” in Python or “JavaDocs” in Java).
While the task shares many characteristics with the M ETHOD N AMING task, the target sequence is substantially longer (on average 19.1 tokens in our C# dataset) and only 19.4% of tokens in the documentation can be copied from the code.
While method documentation is nearer to standard natural language than method names, it mixes project-speciﬁc jargon, code segments and often describes non-functional aspects of the code, such as performance characteristics and design considerations.
NLS UMMARIZATION
Finally, we consider the classic summarization of natural language as widely studied in NLP research.
Speciﬁcally, we are interested in abstractive summarization, where given some text input ( e.g. a news article) a machine learning model produces a novel natural language summary.
Traditionally, NLP summarization methods treat text as a sequence of sentences and each one of them as a sequence of words (tokens).
The input data has less explicitly deﬁned structure than our ﬁrst two tasks.
However, we recast the task as a structured summarization problem by considering additional linguistic structure, including named entities and entity coreferences as inferred by existing NLP tools.
M ODEL
As discussed above, standard neural approaches to summarization follow the sequence-to-sequence framework.
In this setting, most decoders only require a representation h of the complete input sequence ( e.g. the ﬁnal state of an RNN) and per-token representations h t
i for each input token t i .
These token representations are then used as the “memories” of an attention mechanism (Bahdanau et al., 2014; Luong et al., 2015) or a pointer network (Vinyals et al., 2015a).
In this work, we propose an extension of sequence encoders that allows us to leverage known (or inferred) relationships among elements in the input data.
To achieve that, we combine sequence encoders with graph neural networks (GNNs) (Li et al., 2015).
For this, we ﬁrst use a standard sequential encoder ( e.g. bidirectional RNNs) to obtain a per-token representation h t i , which we then feed into a GNN as the initial node representations.
The resulting per-node ( i.e. per-token) representations h (cid:48)
t i can then be used by an unmodiﬁed decoder.
Experimentally, we found this to surpass models that use either only the sequential structure or only the graph structure (see Sect. 4).
We now discuss the different parts of our model in detail.
Gated Graph Neural Networks To process graphs, we follow Li et al.
and brieﬂy summarize the core concepts of GGNNs here.
A graph G =
( V , E , X ) is composed of a set of nodes V , node features X , and a list of directed edge sets E
=
( E 1 , . . .
, E K ) where K is the number of edge types.
Each v ∈ V is associated with a real-valued vector x v representing the features of the node ( e.g. , the embedding of a string label of that node), which is used for the initial state h (0) v of a node.
Information is propagated through the graph using neural message passing.
For this, every node v sends messages to its neighbors by transforming its current representation h ( i ) v using an edgetype dependent function f k .
Here, f k can be an arbitrary function; we use a simple linear layer.
By computing all messages at the same time, all states can be updated simultaneously.
In particular, a new state for a node v is computed by aggregating all incoming messages as m ( i ) v = g
( { f k ( h ( i ) u ) | there is an edge of type k from u to v } ) .
g is an aggregation function; we use elementwise summation for g .
Given the aggregated message m ( i ) v and the current state vector h ( i ) v of node v , we can compute the new state h
( i +1) v =
GRU ( m ( i ) v , h
( i ) v ) , where GRU is the recurrent cell function of a gated recurrent unit.
These dynamics are rolled out for a ﬁxed number of timesteps
T , and the state vectors resulting from the ﬁnal step are used as output node representations, i.e. ,
GNN (( V , E , X ))
= { h ( T ) v } v ∈V .
Sequence GNNs We now explain our novel combination of GGNNs and standard sequence encoders.
As input, we take a sequence S =
[ s 1 . . .
s N ] and K binary relationships R 1 . . .
R K ∈ S × S between elements of the sequence.
For example, R = could be the equality relationship { ( s i , s j ) | s i =
s j } .
The choice and construction of relationships is dataset-dependent, and will be discussed in detail in Sect.
Given any sequence encoder SE that maps S to per-element representations [ e 1 . . .
e N ] and a sequence representation e ( e.g. a bidirectional RNN), we can construct the sequence GNN SE GNN by simply computing [ e (cid:48) 1 . . .
e (cid:48) N ] =
GNN (( S, [ R 1 . . .
R K ] , [ e 1 . . .
e N ])) .
To obtain a graph-level representation, we use the weighted averaging mechanism from Gilmer et al.
Concretely, for each node v in the graph, we compute a weight σ ( w ( h ( T ) v )) ∈
[0 , 1] using a learnable function w and the logistic sigmoid σ and compute a graph-level representation as ˆ e = (cid:80) 1 ≤ i ≤ N σ ( w ( e (cid:48) i )) · ℵ ( e (cid:48) i ) , where ℵ is another learnable projection function.
We found that best results were achieved by computing the ﬁnal e (cid:48) as W · ( e ˆ e ) for some learnable matrix W .
This method can easily be extended to support additional nodes not present in the original sequence
S after running SE ( e.g. , to accommodate meta-nodes representing sentences, or non-terminal nodes from a syntax tree).
The initial node representation for these additional nodes can come from other sources, such as a simple embedding of their label.
Implementation Details.
Processing large graphs of different shapes efﬁciently requires to overcome some engineering challenges.
For example, the CNN/DM corpus has (on average) aboutnodes per graph.
To allow efﬁcient computation, we use the trick of Allamanis et al.
(2018) where all graphs in a minibatch are “ﬂattened” into a single graph with multiple disconnected components.
The varying graph sizes also represent a problem for the attention and copying mechanisms in the decoder, as they require to compute a softmax over a variable-sized list of memories.
To handle this efﬁciently without padding, we associate each node in the (ﬂattened) “batch” graph with the index of the sample in the minibatch from which the node originated.
Then, using TensorFlow’s unsorted segment * operations, we can perform an efﬁcient and numerically stable softmax over the variable number of representations of the nodes of each graph.
E VALUATION 4. Q UANTITATIVE E VALUATION
We evaluate Sequence GNNs on our three tasks by comparing them to models that use only sequence or graph information, as well as by comparing them to task-speciﬁc baselines.
We discuss the three tasks, their respective baselines and how we present the data to the models (including the relationships considered in the graph component) next before analyzing the results.
4.1. S ETUP FOR M ETHOD N AMING
Datasets, Metrics, and Models.
We consider two datasets for the M ETHOD N AMING task.
First, we consider the “Java (small)” dataset of Alon et al.
(2018a), re-using the train-validation-test splits they have picked.
We additionally generated a new dataset from 23 open-source C# projects mined from GitHub (see below for the reasons for this second dataset), removing any duplicates.
More information about these datasets can be found in Appendix C. We follow earlier work on M ETHODN AMING (Allamanis et al., 2016; Alon et al., 2018a) and measure performance using the F1 score over the generated subtokens.
However, since the task can be viewed as a form of (extreme) summarization, we also report ROUGE-2 and ROUGE-L scores (Lin, 2004), which we believe to be additional useful indicators for the quality of results.
ROUGE-1 is omitted since it is equivalent to
F1 score.
We note that there is no widely accepted metric for this task and further work identifying the most appropriate metric is required.
We compare to the current state of the art (Alon et al., 2018a), as well as a sequence-to-sequence implementation from the OpenNMT project (Klein et al.).
Concretely, we combine two encoders (a bidirectional LSTM encoder with 1 layer and 256 hidden units, and its sequence GNN extension with 128 hidden units unrolled over 8 timesteps) with two decoders (an LSTM decoder with 1 layer andhidden units with attention over the input sequence, and an extension using a pointer network-style copying mechanism (Vinyals et al., 2015a)).
Additionally, we consider self-attention as an alternative to RNN-based sequence encoding architectures.
For this, we use the Transformer (Vaswani et al., 2017) implementation in OpenNMT ( i.e. , using self-attention both for the decoder and the encoder) as a baseline and compare it to a version whose encoder is extended with a GNN component.
Data Representation Following the work of Allamanis et al.
; Alon et al.
(2018a)
, we break up all identiﬁer tokens ( i.e. variables, methods, classes, etc. ) in the source code into subtokens by splitting them according to camelCase and pascal case heuristics.
This allows the models to extract information from the information-rich subtoken structure, and ensures that a copying mechanism in the decoder can directly copy relevant subtokens, something that we found to be very effective for this task.
All models are provided with all (sub)tokens belonging to the source code of a method, including its declaration, with the actual method name replaced by a placeholder symbol.
To construct a graph from the (sub)tokens, we implement a simpliﬁed form of the work of Allamanis et al.
First, we introduce additional nodes for each (full) identiﬁer token, and connect the constituent subtokens appearing in the input sequence using a I N T OKEN edge; we additionally connect these nodes using a N EXT T OKEN edge.
We also add nodes for the parse tree and use edges to indicate that one node is a C HILD of another.
Finally, we add L AST L EXICAL U SE edges to connect identiﬁers to their most (lexically) recent use in the source code.
4.1. S ETUP FOR M ETHOD D OC
Datasets, Metrics, and Models.
We tried to evaluate on the Python dataset of Barone & Sennrich (2017) that contains pairs of method declarations and their documentation (“docstring”).
However, following the work of Lopes et al.
(2017), we found extensive duplication between different folds of the dataset and were only able to reach comparable results by substantially overﬁtting to the training data that overlapped with the test set.
We have documented details in subsection C.3, and decided to instead evaluate on our new dataset of 23 open-source C# projects from above, again removing duplicates and methods without documentation.
Following Barone & Sennrich (2017), we measure the BLEU score for all models.
However, we also report F1, ROUGE-2 and ROUGE-L scores, which should better reﬂect the summarization aspect of the task.
We consider the same models as for the
M ETHOD N AMING task, using the same conﬁguration, and use the same data representation.
Munster have signed
New
Zealand international
Francis
Saili on a two-year deal
Utility back
Saili who made his ...
Sentence Sentence
Person Person Person
Country
Duration
Token
Entity
N EXT
I N
R EF
4.1. S ETUP FOR NLS UMMARIZATION
Datasets, Metrics, and Models.
We use the CNN/DM dataset (Hermann et al., 2015) using the exact data and split provided by See et al.
The data is constructed from CNN and Daily Mail news articles along with a few sentences that summarize each article.
To measure performance, we use the standard ROUGE metrics.
We compare our model with the near-to-state-of-the-art work of
See et al.
(2017), who use a sequence-to-sequence model with attention and copying as basis, but have additionally substantially improved the decoder component.
As our contribution is entirely on the encoder side and our model uses a standard sequence decoder, we are not expecting to outperform more recent models that introduce substantial novelty in the structure or training objective of the decoder (Chen & Bansal, 2018;
Narayan et al., 2018).
Again, we evaluate our contribution using an
OpenNMT-based encoder/decoder combination.
Concretely, we use a bidirectional LSTM encoder with 1 layer and 256 hidden units, and its sequence GNN extension with 128 hidden units unrolled over 8 timesteps.
As decoder, we use an LSTM with 1 layer and 256 hidden units with attention over the input sequence, and an extension using a pointer network-style copying mechanism.
Data Representation We use Stanford CoreNLP (Manning et al., 2014) (version 3.9.1) to tokenize the text and provide the resulting tokens to the encoder.
For the graph construction (Figure 2), we extract the named entities and run coreference resolution using CoreNLP.
We connect tokens using a N EXT edge and introduce additional super-nodes for each sentence, connecting each token to the corresponding sentence-node using a I N edge.
We also connect subsequent sentence-nodes using a N EXT edge.
Then, for each multi-token named entity we create a new node, labeling it with the type of the entity and connecting it with all tokens referring to that entity using an I N edge.
Finally, coreferences of entities are connected with a special R EF edge.
Figure 2 shows a partial graph for an article in the CNN/DM dataset.
The goal of this graph construction process is to explicitly annotate important relationships that can be useful for summarization.
We note that (a) in early efforts we experimented with adding dependency parse edges, but found that they do not provide signiﬁcant beneﬁts and (b) that since we retrieve the annotations from CoreNLP, they can contain errors and thus, the performance of the our method is inﬂuenced by the accuracy of the upstream annotators of named entities and coreferences.
4.1. R ESULTS & A NALYSIS
We show all results in Tab. 1.
Results for models from the literature are taken from the respective papers and repeated here.
Across all tasks, the results show the advantage of our hybrid sequence
GNN encoders over pure sequence encoders.
On M ETHOD N AMING , we can see that all GNN-augmented models are able to outperform the current specialized state of the art, requiring only simple graph structure that can easily be obtained using existing parsers for a programming language.
The results in performance between the different encoder and decoder conﬁgurations nicely show that their effects are largely orthogonal.
On M ETHOD D OC , the unmodiﬁed S ELF A TT → S ELF A TT model already performs quite well, and the augmentation with graph data only improves the BLEU score and worsens the results on ROUGE.
Inspection of the results shows that this is due to the length of predictions.
Whereas the ground
[Table]
truth data has on average 19 tokens in each result, S ELF A TT → S ELF A TT predicts on averagetokens
[Table]
and S ELF A TT +GNN → S ELF A TT 16 tokens.
Additionally
[Table]
we experimented with an ablation in which a model is only using graph information
[Table]
e [Table] g. , a setting comparable to a simpliﬁcation of the architecture of Allamanis et al.
For this, we conﬁgured the GNN to use 128-dimensional representations and unrolled it for 10 timesteps, keeping the decoder conﬁguration as for the other models.
The results indicate that this conﬁguration performs less well than a pure sequenced model.
We speculate that this is mainly due to the fact that 10 timesteps are insufﬁcient to propagate information across the whole graph, especially in combination with summation as aggregation function for messages in graph information propagation.
Finally, on NLS UMMARIZATION , our experiments show that the same model suitable for tasks on highly structured code is competitive with specialized models for natural language tasks.
While there is still a gap to the best conﬁguration of See et al.
(2017) (and an even larger one to more recent work in the area), we believe that this is entirely due to our simplistic decoder and training objective, and that our contribution can be combined with these advances.
In Table 2 we show some ablations for NLS UMMARIZATION .
As we use the same hyperparameters across all datasets and tasks, we additionally perform an experiment with the model of See et al.
(2017) (as implemented in OpenNMT) but using our settings.
The results achieved by these baselines trend to be a bit worse than the results reported in the original paper, which we believe is due to a lack of hyperparameter optimization for this task.
We then evaluated how much the additional linguistic [Table] public static bool TryFormat(float value, Span<byte> destination, out int bytesWritten, StandardFormat format=default) { return TryFormatFloatingPoint<float>(value, destination, out bytesWritten, format); } [Table]
formats
[Table]
structure provided by CoreNLP helps.
First, we add the coreference and entity annotations to the baseline B I LSTM
→ LSTM + P OINTER model (by extending the embedding of tokens with an embedding of the entity information, and inserting fresh “¡REF1¿”, ...
tokens at the sources/targets of co-references) and observe only minimal improvements.
This suggests that our graph-based encoder is better-suited to exploit additional structured information compared to a biLSTM encoder.
We then drop all linguistic structure information from our model, keeping only the sentence edges/nodes.
This still improves on the baseline B I LSTM
→ LSTM + P OINTER model (in the ROUGE-2 score), suggesting that the GNN still yields improvements in the absence of linguistic structure.
Finally, we add long-range dependency edges by connecting tokens with equivalent string representations of their stems and observe further minor improvements, indicating that even using only purely syntactical information, without a semantic parse, can already provide gains.
4. Q UALITATIVE E VALUATION
We look at a few sample suggestions in our dataset across the tasks.
Here we highlight some observations we make that point out interesting aspects and failure cases of our model.
M ETHOD D OC
Figures 1 and 3 illustrate typical results of baselines and our model on the
M ETHOD D OC task (see Appendix A for more examples).
The hardness of the task stems from the large number of distractors and the need to identify the most relevant parts of the input.
In Figure 1, the token “parameter” and variations appears many times, and identifying the correct relationship is non-trivial, but is evidently eased by graph edges explicitly denoting these relationships.
Similarly, in Figure 3, many variables are passed around, and the semantics of the method require understanding how information ﬂows between them. NLS UMMARIZATION
Figure 4 shows one sample summarization. More samples for this task can be found in Appendix B. First, we notice that the model produces natural-looking summaries with no noticeable negative impact on the ﬂuency of the language over existing methods. Furthermore, the GNN-based model seems to capture the central named entity in the article and creates a summary centered around that entity. We hypothesize that the GNN component that links long-distance relationships helps capture and maintain a better “global” view of the article, allowing for better iden-
Input: Arsenal , Newcastle United and Southampton have checked on Caen midﬁelder N’golo Kante .
Paris- born Kante is a defensive minded player who has impressed for Caen this season and they are willing to sell for around £ 5million .
Marseille have been in constant contact with Caen over signing the 24-year-old who has similarities with Lassana Diarra and Claude Makelele in terms of stature and style .
N’Golo Kante is attracting interest from a host of Premier League clubs including Arsenal .
Caen would be willing to sell Kante for around £ 5million .
Reference: n’golo kante is wanted by arsenal , newcastle and southampton .
marseille are also keen on the £ 5m rated midﬁelder .
kante has been compared to lassana diarra and claude makelele .
click here for the latest premier league news .
See et al.
(2017) (+ Pointer): arsenal , newcastle united and southampton have checked on caen midﬁelder n’golo kante .
paris-born kante is attracting interest from a host of premier league clubs including arsenal .
paris-born kante is attracting interest from a host of premier league clubs including arsenal
See et al.
(2017) (+ Pointer + Coverage): arsenal , newcastle united and southampton have checked on caen midﬁelder n’golo kante .
paris-born kante is a defensive minded player who has impressed for caen this season .
marseille have been in constant contact with caen over signing the 24-year-old .
B I LSTM+GNN → LSTM: marseille have been linked with caen midﬁelder %UNK% %UNK% .
marseille have been interested from a host of premier league clubs including arsenal . caen have been interested from a host of premier league clubs including arsenal .
B I LSTM+GNN → LSTM+P OINTER n’golo kante is attracting interest from a host of premier league clubs . marseille have been in constant contact with caen over signing the 24-year-old . the 24-year-old has similarities with lassana diarra and claude makelele in terms of stature .
tiﬁcation of central entities.
Our model still suffers from repetition of information (see Appendix B), and so we believe that our model would also proﬁt from advances such as taking coverage into account (See et al., 2017) or optimizing for ROUGE-L scores directly via reinforcement learning (Chen & Bansal, 2018;
Narayan et al., 2018).
R ELATED W ORK
Natural language processing research has studied summarization for a long time.
Most related is work on abstractive summarization, in which the core content of a given text (usually a news article) is summarized in a novel and concise sentence.
Chopra et al.
(2016) and Nallapati et al.
(2016) use deep learning models with attention on the input text to guide a decoder that generates a summary.
See et al.
(2017) and McCann et al.
(2018) extend this idea with pointer networks (Vinyals et al., 2015a) to allow for copying tokens from the input text to the output summary.
These approaches treat text as a simple token sequences, not explicitly exposing additional structure.
In principle, deep sequence networks are known to be able to learn the inherent structure of natural language ( e.g. in parsing (Vinyals et al., 2015b) and
entity recognition
Lample et al., 2016))
, but our experiments indicate that explicitly exposing this structure by separating concerns improves performance.
Recent work in summarization has proposed improved training objectives for summarization, such as tracking coverage of the input document (See et al., 2017) or using reinforcement learning to directly identify actions in the decoder that improve target measures such as ROUGE-L (Chen &
Bansal, 2018;
Narayan et al., 2018).
These objectives are orthogonal to the graph-augmented encoder discussed in this work, and we are interested in combining these efforts in future work.
Exposing more language structure explicitly has been studied over the last years, with a focus on tree-based models (Tai et al., 2015).
Very recently, ﬁrst uses of graphs in natural language processing have been explored.
Marcheggiani & Titov (2017) use graph convolutional networks to encode single sentences and assist machine translation.
De Cao et al.
(2018) create a graph over named entities over a set of documents to assist question answering.
Closer to our work is the work of Liu et al.
(2018), who use abstract meaning representation (AMR), in which the source document is ﬁrst parsed into
AMR graphs, before a summary graph is created, which is ﬁnally rendered in natural language.
In contrast to that work we do not use AMRs but directly encode relatively simple relationships directly on the tokenized text, and do not treat summarization as a graph rewrite problem.
Combining our encoder with AMRs to use richer graph structures may be a promising future direction.
Finally, summarization in source code has also been studied in the forms of method naming, comment and documentation prediction.
Method naming has been tackled with a series of models.
For example,
Allamanis et al.
(2015) use a log-bilinear network to predict method names from features, and later extend this idea to use a convolutional attention network over the tokens of a method to predict the subtokens of names (Allamanis et al., 2016).
Raychev et al. (2015) and Bichsel et al.
(2016) use
CRFs for a range of tasks on source code, including the inference of names for variables and methods.
Recently, Alon et al.
(2018b;a) extract and encode paths from the syntax tree of a program, setting the state of the art in accuracy on method naming.
Linking text to code can have useful applications, such as code search (Gu et al., 2018), traceability (Guo et al., 2017), and detection of redundant method comments (Louis et al., 2018).
Most approaches on source code either treat it as natural language ( i.e. , a token sequence), or use a language parser to explicitly expose its tree structure.
For example, Barone & Sennrich (2017) use a simple sequenceto-sequence baseline, whereas Hu et al.
(2017) summarize source code by linearizing the abstract syntax tree of the code and using a sequence-to-sequence model.
Wan et al.
(2018) instead directly operate on the tree structure using tree recurrent neural networks (Tai et al., 2015).
The use of additional structure on related tasks on source code has been studied recently, for example in models that are conditioned on learned traversals of the syntax tree (Bielik et al., 2016) and in graph-based approaches (Allamanis et al., 2018;
Cvitkovic et al., 2018).
However, as noted by Liao et al.
GNN-based approaches suffer from a tension between the ability to propagate information across large distances in a graph and the computational expense of the propagation function, which is linear in the number of graph edges per propagation step.
D ISCUSSION & C ONCLUSIONS
We presented a framework for extending sequence encoders with a graph component that can leverage rich additional structure.
In an evaluation on three different summarization tasks, we have shown that this augmentation improves the performance of a range of different sequence models across all tasks.
We are excited about this initial progress and look forward to deeper integration of mixed sequence-graph modeling in a wide range of tasks across both formal and natural languages.
The key insight, which we believe to be widely applicable, is that inductive biases induced by explicit relationship modeling are a simple way to boost the practical performance of existing deep learning systems.
M ETHOD N AMING
C# Samplepublic bool _(D d) { return d != null && d.Val =
=
Val ; }
Ground truth equals
B I LSTM
→
LSTM foo
B I LSTM+GNN → LSTM equals
B I LSTM+GNN → LSTM+P OINTER equals
C# Sampleinternal void _(string switchName, Hashtable bag, string parameterName) { object obj = bag[parameterName]; if(obj !
= null){ int value =
(int) obj;
AppendSwitchIfNotNull(switchName,value.
ToString(CultureInfo.
InvariantCulture)); } }
Ground truth append switch with integer
B I LSTM →
LSTM set string
B I LSTM+GNN → LSTM append switch
B I LSTM+GNN → LSTM+P OINTER append switch if not null
C# Sampleinternal static string _(){ var currentPlatformString = string.
Empty; if (RuntimeInformation.IsOSPlatform(OSPlatform.
Windows)){currentPlatformString="WINDOWS"; }else if (RuntimeInformation.IsOSPlatform(OSPlatform.
Linux))
{ currentPlatformString = "LINUX"; } else if ( RuntimeInformation.
IsOSPlatform(OSPlatform.
OSX))
{ currentPlatformString = "OSX"; }else {
Assert.
True(false, "unrecognized current platform"); } return currentPlatformString ; }
Ground truth get os platform as string
B I LSTM → LSTM get name
B I LSTM+GNN → LSTM get platform
B I LSTM+GNN → LSTM+P OINTER get current platform string
C# Samplepublic override DbGeometryWellKnownValue CreateWellKnownValue(DbGeometry geometryValue){ geometryValue.
CheckNull("geometryValue"); varspatialValue=geometryValue.
AsSpatialValue();DbGeometryWellKnownValueresult=
CreateWellKnownValue(spatialValue, ()=>
SpatialExceptions.
CouldNotCreateWellKnownGeometryValueNoSrid("geometryValue"),()=>SpatialExceptions.
CouldNotCreateWellKnownGeometryValueNoWkbOrWkt("geometryValue"), (srid,wkb,wkt)=>newDbGeometryWellKnownValue(){CoordinateSystemId=srid,WellKnownBinary=wkb,WellKnownText=wkt });returnresult; }
Ground truth create well known value
B I LSTM →
LSTM spatial geometry from xml
B I LSTM+GNN → LSTM geometry point
B I LSTM+GNN → LSTM+P OINTER get well known value
Java Samplepublic static void _(String name, int expected, MetricsRecordBuilder rb) { Assert.assertEquals("Bad value for metric " + name, expected,getIntCounter(name, rb)); }
Ground truth assert counter
B I LSTM →
LSTM assert email value
B I LSTM+GNN → LSTM assert header
B I LSTM+GNN → LSTM+P OINTER assert int counter
B N ATURAL L ANGUAGE S UMMARIZATION S AMPLES
Input: -LRB-
CNN -RRB-
Gunshots were ﬁred at rapper Lil Wayne ’s tour bus early Sunday in
Atlanta .
No one was injured in the shooting , and no arrests have been made , Atlanta Police spokeswoman Elizabeth Espy said .
Police are still looking for suspects .
Ofﬁcers were called to a parking lot in Atlanta ’s
Buckhead neighborhood
, Espy said .
They arrived at 3:25 a.m. and located two tour buses that had been shot multiple times .
The drivers of the buses said the incident occurred on Interstate 285 near Interstate 75 , Espy said .
Witnesses provided a limited description of the two vehicles suspected to be involved : a “ Corvette style vehicle ” and an SUV .
Lil Wayne was in Atlanta for a performance at Compound nightclub Saturday night .
CNN ’s Carma Hassan contributed to this report .
Reference: rapper lil wayne not injured after shots ﬁred at his tour bus on an atlanta interstate , police say .
no one has been arrested in the shooting
See et al.
(2017) (+ Pointer): police are still looking for suspects .
the incident occurred on interstate 285 near interstate 75 , police say .
witnesses provided a limited description of the two vehicles suspected to be involved : a “ corvette style vehicle ” and an suv .
See et al.
(2017) (+ Pointer + Coverage): lil wayne ’s tour bus was shot multiple times , police say .
police are still looking for suspects .
they arrived at 3:25 a.m. and located two tour buses that had been shot .
B I LSTM+GNN → LSTM: the incident occurred on interstate %UNK% near interstate 75 .
no one was injured in the shooting , and no arrests have been made , atlanta police spokeswoman says .
B I LSTM+GNN → LSTM+P OINTER gunshots ﬁred at rapper lil wayne ’s tour bus early sunday in atlanta , police say .
no one was injured in the shooting , and no arrests have been made , police say .
Input: Tottenham have held further discussions with Marseille over a potential deal for midﬁelder
Florian Thauvin .
The 22-year-old has been left out of the squad for this weekend ’s game with
Metz as Marseille push for a £ 15m sale .
The winger , who can also play behind the striker , was the subject of enquiries from Spurs earlier in the year and has also been watched by Chelsea and
Valencia .
Tottenham have held further talks with Ligue 1 side Marseille over a possible deal for
Florian Thauvin .
Marseille are already resigned to losing Andre Ayew and Andre-Pierre Gignac with English sides keen on both .
Everton , Newcastle and Swansea , have all shown an interest in
Ayew , who is a free agent in the summer .
Reference: ﬂorian thauvin has been left out of marseille ’s squad with metz .
marseille are pushing for a £ 15m sale and tottenham are interested .
the winger has also been watched by chelsea and la liga side valencia .
See et al.
(2017) (+ Pointer): tottenham have held further discussions with marseille over a potential deal for midﬁelder ﬂorian thauvin .
the 22-year-old has been left out of the squad for this weekend ’s game with metz as marseille push for a 15m sale .
See et al.
(2017) (+ Pointer + Coverage): ﬂorian thauvin has been left out of the squad for this weekend ’s game with metz as marseille push for a 15m sale .
the 22-year-old was the subject of enquiries from spurs earlier in the year .
B I LSTM+GNN → LSTM: the 22-year-old has been left out of the squad for this weekend ’s game with metz .
the 22-year-old has been left out of the squad for this weekend ’s game with metz .
the winger has been left out of the squad for this weekend ’s game with metz .
B I LSTM+GNN → LSTM+P OINTER tottenham have held further discussions with marseille over a potential deal .
the winger has been left out of the squad for this weekend ’s game .
tottenham have held further talks with marseille over a potential deal .
[Table]
NFORMATION
C.
C# D ATASET
We extract the C# dataset from open-source projects on GitHub.
Overall, our dataset contains 460, methods, 55,635 of which have a documentation comment.
The dataset is split 85-5-10%.
The projects and exact state of the repositories used is listed in Table
[Table]
6f32f6a
[Table]
AutoMapper 19d6f7fc
Object-to [Table] Mapping Library
BenchmarkDotNet 57005f
Benchmarking Library
CommonMark.
NET f3d
Markdown Parser
CoreCLR cc5dcbe .NET
Core Runtime
CoreFx ec1671fd .NET
Foundational Libraries
Dapper 3c7cde
Object Mapper Library
EntityFramework c4d9a
Object-Relational Mapper
Humanizer 2b1c94c
String Manipulation and Formatting
Lean 90ee6aae
Algorithmic Trading Engine
Mono 9b9e4f4b .NET
Implementation
MsBuild 7f95dc
Build Engine
Nancy de458a9b
HTTP Service Framework
NLog 49fdd08e
Logging Library
Opserver 9e4d3a
Monitoring System orleans f89c
Distributed Virtual Actor Model
Polly f3d2973d
Resilience & Transient Fault Handling Library
Powershell 9ac701db
Command-line Shell ravendb 6437de
Document Database roslyn 8ca0a
Compiler & Code Analysis & Compilation
ServiceStack 17f081b
Real-time web library
SignalR 9b05bcb
Push Notiﬁcation Framework
Wox 13e6c5ee
Application Launcher
C.
J AVA M ETHOD N AMING D
ATASETS
We use the datasets and splits of Alon et al.
(2018a) provided by their website.
Upon scanning all methods in the dataset, the size of the corpora can be seen in Table 4.
More information can be found at Alon et al.
(2018a).
C. P YTHON M ETHOD D OCUMENTATION D ATASET
We use the dataset as split of Barone & Sennrich (2017) provided by their GitHub repository.
Upon parsing the dataset, we get 106,065 training samples, 1,943 validation samples and 1,937 test samples.
We note that 16.9% of the documentation sampels in the validation set and 15.3% of the samples in test set have a sample with the identical natural language documentation on the training set.
This eludes to a potential issue, described by Lopes et al.
[Table]
C. G RAPH D ATA S TATISTICS
Below we present the data characteristics of the graphs we use across the datasets
[Table] [Table]