  We combine deep learning and Conditional Probabilistic Context Free Grammars
(CPCFG) to create an end-to-end system for extracting structured information
from complex documents. For each class of documents, we create a CPCFG that
describes the structure of the information to be extracted. Conditional
probabilities are modeled by deep neural networks. We use this grammar to parse
2-D documents to directly produce structured records containing the extracted
information. This system is trained end-to-end with (Document, Record) pairs.
We apply this approach to extract information from scanned invoices achieving
state-of-the-art results.
