Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks

-We give conditions ensuring that multilayer jeedJorward networks with as Jew as a single hidden layer and an appropriately smooth hidden layer activation fimction are capable o f arbitrarily accurate approximation to an arbitrao' function and its derivatives. In fact, these networks can approximate functions that are not dtifferentiable in the classical sense, but possess only a generalized derivative, as is the case jor certain piecewise dtlf[erentiable Junctions. The conditions imposed on the hidden layer activation function are relatively mild: the conditions imposed on the domain o f the fimction to be approximated have practical intplications. Our approximation results provide a previously missing theoretical justification jor the use of multilayer /~'ed/brward networks in applications requiring simultaneous approximation o f a function and its derivatives. Keywords--Approximation, Derivatives, Sobolev space, Feedforward networks. 1. I N T R O D U C T I O N The capability of sufficiently complex multilayer feedforward networks to approximate an unknown mapping./~. R' ----~ R arbitrarily well has been recently investigated by Cybenko (1989), Funahashi (1989), Hecht-Nielsen (1989), Hornik, Stinchcombe, and White (1989) (HSW) (all for sigmoid hidden layer activation functions) and Stinchcombe and White (1989) (SW) (non-sigmoid hidden layer activation functions). In applications, it may be desirable to approximate not only the unknown mapping, but also its unknown derivatives. This is the case in Jordan's (1989) recent investigation of robot learning of smooth movement . Jordan states: The Jacobian matrix 8z/Ox . . . is the matrix that relates small changes in the controller output to small changes in the task space results and cannot be assumed to be available a priori, or provided by the environment. However, all of the derivatives in the matrix are forward derivatives. They are easily obtained by differentiation if a forward model is available. The forward model itself must be learned, but this can be achieved directly by system idenAcknowledgements: We arc indebted to Angelo Melino for pressing us on the issue addressed here and to the referees for numerous helpful suggestions. White's participation was supported by NSF Grant SES-8806990. Requests for reprints should be sent to Halbert White, Department of Economics, D-008, University of California, San Diego. La Jolla, CA 92093. tification. Once the model is accurate over a particular domain, its derivatives provide a learning operator that allows the system to convert errors in task space into errors in articulatory space and thereby change the controller. Thus, learning an adequate approximation to the Jacobian matrix of an unknown mapping is a key component of Jordan 's approach to robot learning of smooth movement . Despite the success of Jordan 's experiments, there is no existing theoretical guarantee that multilayer feedforward networks generally have the capability to approximate an unknown mapping and its derivatives simultaneously. For example, a network with hard limiting hidden layer activations approximates unknown mappings with a piecewise-constant function, the first derivatives of which exist and are zero almost everywhere. Obviously, the derivatives of such a network output function cannot approximate the derivatives of an arbitrary function. Intuition suggests that networks having smooth hidden layer activation functions ought to have output function derivatives that will approximate the derivatives of an unknown mapping. However , the justification for this intuition is not obvious. Consider the class of single hidden layer feedforward networks having network output functions belonging to the set 2. "(G) -~ {g : x"-~ R[ g(x) : /:,G(_i';,,):