We present a new approach to the design of deep networks for natural language processing ( NLP ) , based on the general technique of Tensor Product Representations ( TPRs ) for encoding and processing symbol structures in distributed neural networks . A network architecture -the Tensor Product Generation Network ( TPGN ) -is proposed which is capable in principle of carrying out TPR computation , but which uses unconstrained deep learning to design its internal representations . Instantiated in a model for image-caption generation , TPGN outperforms LSTM baselines when evaluated on the COCO dataset . The TPR-capable structure enables interpretation of internal representations and operations , which prove to contain considerable grammatical content . Our caption-generation model can be interpreted as generating sequences of grammatical categories and retrieving words by their categories from a plan encoded as a distributed representation . In this paper we introduce a new architecture for natural language processing ( NLP ) . On what type of principles can a computational architecture be founded ? It would seem a sound principle to require that the hypothesis space for learning which an architecture provides include network hypotheses that are independently known to be suitable for performing the target task . Our proposed architecture makes available to deep learning network configurations that perform natural language generation by use of Tensor Product Representations ( TPRs ) ( Smolensky and Legendre , 2006 ) . Whether learning will create TPRs is unknown in advance , but what we can say with certainty is that the hypothesis space being searched during learn-ing includes TPRs as one appropriate solution to the problem . TPRs are a general method for generating vector-space embeddings of complex symbol structures . Prior work has proved that TPRs enable powerful symbol processing to be carried out using neural network computation ( Smolensky , 2012 ) . This includes generating parse trees that conform to a grammar ( Cho et al . , 2017 ) , although incorporating such capabilities into deep learning networks such as those developed here remains for future work . The architecture presented here relies on simpler use of TPRs to generate sentences ; grammars are not explicitly encoded here . We test the proposed architecture by applying it to image-caption generation ( on the MS-COCO dataset , ( COCO , 2017 ) ) . The results improve upon a baseline deploying a state-of-the-art LSTM architecture ( Vinyals et al . , 2015 ) , and the TPR foundations of the architecture provide greater interpretability . Section 2 of the paper reviews TPR . Section 3 presents the proposed architecture , the Tensor Product Generation Network ( TPGN ) . Section 4 describes the particular model we study for image captioning , and Section 5 presents the experimental results . Importantly , what the model has learned is interpreted in Section 5.3 . Section 6 discusses the relation of the new model to previous work and Section 7 concludes . Tensor Product Representation ( TPR ) ( Smolensky , 1990 ) is a general technique for constructing vector embeddings of complex symbol structures in such a way that powerful symbolic functions can be computed using hand-designed neural network computation . Integrating TPR with deep learning is a largely open problem for which the work presented here proposes a general approach : design deep architectures that are TPRcapable -TPR computation is within the scope of the capabilities of the architecture in principle . For natural language generation , we proposed such an architecture , the Tensor Product Generation Network ( TPGN ): it embodies the TPR operation of unbinding which is used to extract particular symbols ( e.g. , words ) from complex structures ( e.g. , sentences ) . The architecture can be interpreted as containing a part that encodes a sentence and a part that selects one structural role at a time to extract from the sentence . We applied the approach to image-caption generation , developing a TPGN model that was evaluated on the COCO dataset , on which it outperformed LSTM baselines on a range of standard metrics . Unlike standard LSTMs , however , the TPGN model admits a level of interpretability : we can see which roles are being unbound by the unbinding vectors generated internally within the model . We find such roles contain considerable grammatical information , enabling POS tag prediction for the words they generate and displaying clustering by POS .

Challenge: While Tensor Product Representation is a powerful model for obtaining vector embeddings for symbol structures, its application with deep learning models is still less investigated.
Approach: They propose a newly designed model that is based on Tensor Product Representations for encoding and processing words and sentences.
Outcome: The Tensor Product Representation-based generative model outperforms LSTM models by evaluating on COCO image-caption dataset and also achieves high interpretability.