Paper:
Learning Pipelines with Limited Data and Domain Knowledge: A Study in Parsing Physics Problems
As machine learning becomes more widely used in practice, we need new methods to build complex intelligent systems that integrate learning with existing software, and with domain knowledge encoded as rules. As a case study, we present such a system that learns to parse Newtonian physics problems in textbooks. This system, Nuts&Bolts, learns a pipeline process that incorporates existing code, pre-learned machine learning models, and human engineered rules. It jointly trains the entire pipeline to prevent propagation of errors, using a combination of labelled and unlabelled data. Our approach achieves a good performance on the parsing task, outperforming the simple pipeline and its variants. Finally, we also show how Nuts&Bolts can be used to achieve improvements on a relation extraction task and on the end task of answering Newtonian physics problems.

Review:
Summary: The paper addresses an important aspect of learning pipelines with limited data while incorporating domain specific knowledge relevant to a task.  The main idea is the use of PSL (probabilistic soft logic) as a framework to map partial estimates from multiple feedforward algorithms, along with domain specific logical rules, to parse visual diagrams from physics texts.  Specifically, the pipelines use feature extractors for lines, arcs, corners , text elements, object elements (e.g.blocks in physics diagrams). These are combined along with human specified rules for groupings, high-level elements, text/figure labeling schemes along with the inference engine to produce the parse into a formal logical language.  Experiments illustrate how the learned system:  1)  is superior to state of the art diagram parsing scheme, 2) can utilize labelled as well as unlabelled data to achieve improved performance, 3) can handle various degrees of supervision in different parts of the pipeline and is robust , and 4) through integrative modeling of the stages in pipeline prevents error propagation.  Quality, Clarity, originality, significance of the paper:  The paper is well written and has extensive references to relevant literature, adequate experimentation.  The paper is a systems application paper and adequate information is presented in supplemental material to provide details of the limitation of the results at a systems level. The current work is a systems paper combining well known aspects/algorithms in the literature.   Section 5 (C1) claims that there is no prior work on end to end parsing of diagrams. While I believe this may be true for physics diagrams, the paper misses citation to  key papers that exploit parallel pipelines for feature extraction, bi-lattice logic formalism for reasoning and parsing, and translation to a formal representation.   See:  Vinay D. Shet et al: Predicate Logic Based Image Grammars for Complex Pattern Recognition. International Journal of Computer Vision 93(2): 141-161 (2011)  Toufiq Parag, Claus Bahlmann, Vinay D. Shet, Maneesh Singh: A grammar for hierarchical object descriptions in logic programs. CVPR Workshops 2012: 33-38  P. Sajda et al,  In a Blink of an Eye and a Switch of a Transistor: Cortically Coupled Computer Vision, Vol. 98, No. 3, March 2010 | Proceedings of the IEEE  The current paper also does not refer to a class of past work involving performance modeling of vision pipelines ( including some in the document understanding area) that involve building appropriate trust models for pipelines.   Proper understanding of errors in processing stages can allow appropriate fusion mechanisms to get robust outputs (see: http://haralick.org/journals/performance.shtml).  While it is clear that integrative learning at a holistic systems level reduces impact of error propagation, error modeling and appropriate bayesian fusion has been done in Bayesian model based vision literature (see: Binford, T.O., Levitt, T.S.,  Evidential reasoning for object recognition, PAMI(25), No. 7, July 2003, pp. 837-851. ).  Given that the application domain involving physics diagrams still involve human produced diagrams, I would imagine that a systematic model-based approach can produce robust interpretations.    Other comments: It will be quite difficult to reproduce the results in the paper given the complexity of the system. It is recommended that the datasets, groundtruth, and executables of the system be made available for research purposes.  Comments Based on Author's rebuttal: The authors have put in substantial effort to address my concerns in my first review.  Unfortunately this paper is a complex system and a lot of implementation detail for the alternatives is omitted in its present form.   It is impossible to judge given the one page rebuttal why the alternative frameworks (e.g. bi-lattice reasoning)  perform worse.  I would want to know more details: e.g.  Did they implement Shet et al paper exactly as specified?  what types of rules did they use?  How did they learn the rule weights?  etc.   The authors state that there is no prior work that uses software modules, parallel estimators, and fusion. I refer the authors to a few patents and publications in systems engineering for vision in the early 2000's address this:  Methodology for model based systems engineering / performance modeling: US Patents: 7,899,209 -Statistical modeling and performance characterization of a real-time dual camera surveillance system  (also appeared in: Greiffenhagen et al, CVPR 2000, CVPR 2001, Proc. of IEEE 2001)  7,428,337 - Automatic design of morphological algorithms for machine vision , by Gao et al, 2002  -  addresses learning of nonlinear operator chains for a given classification task  While these references are not central to the present paper, they may be useful for the authors.   Overall recommendation:  I don't have a problem with acceptance of the paper based on what the authors have demonstrated.  While the authors definitely made attempts to address my review,  comparison of alternative systems approaches are often challenging and an open question is to ask how one can make rigorous claims of superiority of one approach over another.   As a systems scientist, I would not be upset with rejection of the paper.  As an applications engineer, I would be ok with the acceptance of the paper.  Thus, my rating for the paper does not change.