Motivated by data-rich experiments in transcriptional regulation and sensory neuroscience, we consider the following general problem in statistical inference. A system of interest, when exposed to a stimulus S, adopts a deterministic response R of which a  noisy measurement M is made. Given a large number of measurements and corresponding stimuli , we wish to identify the correct " response function " relating R to S . However the "noise function" relating M to R is unknown a priori. Here we show that maximizing likelihood over both response functions and noise functions is equivalent to simply identifying maximally informative response functions -- ones that maximize the mutual information I[ R;M ] between predicted responses and corresponding measurements . Moreover, if the correct response function is in the class of models being explored, maximizing mutual information becomes equivalent to simultaneously maximizing every dependence measure that satisfies the Data Processing Inequality. We note that experiments of the type considered are unable to distinguish between parametrized response functions lying along certain "diffeomorphic modes" in parameter space . We show how to derive these diffeomorphic modes and observe, fortunately, that such modes typically span a very low-dimensional subspace of parameter space. Therefore, given sufficient data, maximizing mutual information can pinpoint nearly all response function parameters without requiring any model of experimental noise .
Motivated by data-rich experiments in transcriptional regulation and sensory neuroscience, we consider the following general problem in statistical inference. When exposed to a high-dimensional signal S, a system of interest computes a representation R of that signal which is then observed through a noisy measurement M . From a large number of signals and measurements , we wish to infer the " filter " that maps S to R. However, the standard method for solving such problems, likelihood-based inference, requires perfect a priori knowledge of the "noise function" mapping R to M. In practice such noise functions are usually known only approximately, if at all, and using an incorrect noise function will typically bias the inferred filter. Here we show that , in the large data limit, this need for a pre-characterized noise function can be circumvented by searching for filters that instead maximize the mutual information I[ M;R ] between observed measurements and predicted representations . Moreover, if the correct filter lies within the space of filters being explored, maximizing mutual information becomes equivalent to simultaneously maximizing every dependence measure that satisfies the Data Processing Inequality. It is important to note that maximizing mutual information will typically leave a small number of directions in parameter space unconstrained. We term these directions "diffeomorphic modes" and present an equation that allows these modes to be derived systematically. The presence of diffeomorphic modes reflects a fundamental and nontrivial substructure within parameter space, one that is obscured by standard likelihood-based inference .
R
A system of interest, when
When
R
stimulus S, adopts a deterministic response R of which
high-dimensional signal S,
A
system of interest computes a representation R of that signal which is then observed through a
R
is made. Given
. From
R
measurements and corresponding stimuli
signals and measurements
R
identify the correct
infer the
R
response function
filter
R
relating R to S . However the
that maps S to R. However, the standard method for solving such problems, likelihood-based inference, requires perfect a priori knowledge of the
R
relating M to R is unknown a priori.
mapping R to M. In practice such noise functions are usually known only approximately, if at all, and using an incorrect noise function will typically bias the inferred filter.
R
maximizing likelihood over both response functions and noise functions is equivalent to simply identifying maximally informative response functions -- ones that
, in the large data limit, this need for a pre-characterized noise function can be circumvented by searching for filters that instead
R
R;M
M;R
R
predicted responses and corresponding measurements
observed measurements and predicted representations
R
response function is in the class of models
filter lies within the space of filters
R
We note that experiments of the type considered are unable to distinguish between parametrized response functions lying along certain "diffeomorphic modes"
It is important to note that maximizing mutual information will typically leave a small number of directions
R
. We show how to derive these diffeomorphic modes and observe, fortunately, that such modes typically span a very low-dimensional subspace of parameter space. Therefore, given sufficient data, maximizing mutual information can pinpoint nearly all response function parameters without requiring any model of experimental noise
unconstrained. We term these directions "diffeomorphic modes" and present an equation that allows these modes to be derived systematically. The presence of diffeomorphic modes reflects a fundamental and nontrivial substructure within parameter space, one that is obscured by standard likelihood-based inference