AI as statistical methods for imperfect theories
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Science has progressed by reasoning on what models could not predict becausethey were missing important ingredients.
And yet without correct models, standardstatistical methods for scientiﬁc evidence are not sound.
Here, I argue that machinelearning methodology provides solutions to ground reasoning about empiricallyevidence more on models’ predictions, and less on their ingredients.
Science uses false models as means for truer theory [Wimsatt, 1987].
How can statistical toolsground valid reasoning on empirical evidence without true models?
Generalization is the key.
Here
I develop the argument that, unlike popular belief, reasoning from black-box models is good forscience, because it builds the validity of inferences on prediction of observables.
Science has progressed by reﬁning relevant constructs from wrong models1.1 Observing motions of bodies, working out laws of physics
Early scientists, such as Aristotle, did not conceive mechanics in terms of acceleration and forces.
Rather, they thought in terms of natural motion of objects, proportional to their weight.
The notionof force made its way, as discussed by Ibn S ¯ ın ¯ a, but motion was seen as proportional to externalforces.
The Copernican revolution motivated the importance of acceleration.
Increasingly preciseastronomical observations led to formulate planetary motion as elliptical trajectories.
Scientists suchas Kepler were seeking simple phenomenological rules, “harmonies” in his words, to explain theobservations, eg that across the different planets the square of the period is proportional to the cube ofthe major diameter of the orbit.
By introducing acceleration via differential calculus, Newton couldpropose laws of mechanics that explained observations of both celestial and earthly motion.
The birth of Newtonian mechanics illustrates how better observations and statistical models leadto better theories, even when starting without the right theoretical framework .
It shows how newingredients may be needed, such as introducing the construct of acceleration.
It shows that progressis driven by seeking theories that generalize across many settings.
The importance of accelerationwas revealed by uniting motion of bodies on Earth and in astronomy.
Indeed, as friction is ubiquitouson Earth, applying a force to an object often leads to a velocity roughly proportional to this force.
Later, better observations called for new frameworks, quantum or relativistic.
Irregularities in theorbit of Mercury were ﬁrst explained by adding a planet to the solar system, Vulcan.
But observationsof this planet turned out to be ﬂawed, and the irregularities in Mercury’s orbit are now understoodas relativistic corrections.
The Vulcan hypothesis illustrates how theoretical frameworks shape theinterpretations of empirical results: observations are “theory laden”
[Boyd and Bogen, 2021].
Today, the fundamental laws of physics are incredible precise.
Are phenomenological models stillimportant for their empirical validation?
From a statistical perspective, the Neyman-Pearson lemma tells us that the optimal way to compare models is to use their likelihood [van Dyk, 2014].
Indeed,particle physics has long polished probabilistic models, minute stochastic description of observationsbuilt from ﬁrst principles [Sjöstrand et al., 2001, Aaltonen et al., 2008].
And yet, recent statisticalanalysis of Higgs bosons is powered by black-box machine learning models –such as boosted decisiontrees– as they capture best background sensor noise [Aaltonen et al., 2009, Radovic et al., 2018].1.2 Cognitive neuroscience: uncovering the functional units of human vision
Cognitive neuroscience strives to explain cognitive functions from neural activity.
Which ingredientsto include in such a model is a more open-ended question in than in physics.
Breaking down highlevel functions into units of investigation is particularly challenging.
This endeavor has made muchprogress for the speciﬁc problem of vision.
Studying early visual cortex response to specially-craftedstimuli, Hubel and Wiesel [1959] revealed neurons that form localized edge detectors.
Slightly morecomplex shapes isolated other brain units [Logothetis et al., 1995].
These ﬁndings are tied to thestimuli presented, themselves motivated by cognitive theories used to decompose mental processes.
Theories of visual processing break down it into successive operations tuned to speciﬁc aspects ofthe stimuli [Marr, 1982].
As any cognitive theory, their empirical neuroscience validation is thenbound to this decomposition.
Even with modern neural measurements, a decomposition into invalidingredients, such as “alimentiveness” or “philoprogenitiveness” of 19 th century phrenology, wouldlead to a brain mapping valid from the statistical standpoint [Poldrack, 2010].
Complete models of cortical visual processing assemble brain functional units, each implementingspeciﬁc operations [Riesenhuber and Poggio, 1999].
They derive from many studies of neuralresponses to elementary manipulations of visual stimuli.
But their neuroscience validity faced achicken-and-egg problem as long as each functional unit had been studied in isolation: each studyhad investigated only one aspect of otherwise very complex stimuli, natural images.
Models of visioncan be derived without invoking neuroscience arguments, as in computer vision where computationalmodels are optimized directly on natural images, eg for object recognition [Pinto et al., 2009, Sermanetet al., 2014].
In fact, encoding studies showed that pure computational models explain better neuralactivity than models based on hand-crafted reductions of natural images [Yamins et al., 2014].
Thesecomputer-vision models, based on artiﬁcial neural networks, extract intermediate representations ofnatural images, which can be mapped to brain responses, conﬁrming functional units obtained inmore hypothesis-laden neuroscience experiments [Eickenberg et al., 2017].
The large computational models do not answer some cognitive-neuroscience debates, such as thespeciﬁc semantic tuning of functional areas.
For instance, a brain area crucial to recognizing humanfaces is known as the fusiform face area [Kanwisher et al., 1997].
Yet, some researchers claim thatits role is best explained by implementing visual expertise, rather than face recognition [Tarr and
Gauthier, 2000].
As the corresponding brain area responds to both types of stimuli, the debate becametrapped in a ontological disagreement: which of visual expertise or face recognition is a more centralmental function?
One side argues visual expertise leads to face recognition, and the other that facerecognition is innate to the social human.
Encoding studies use as ingredients to map brain responses the internals of large computationalmodels of vision.
As such, they circumvent questions related to ﬁnding valid ontologies of cognitiveprocesses: on the one hand, they cannot bring evidence in favor of ontological choices, but onthe other hand they enable empirical evidence without buying into one framework.
There are twoingredients to this robustness.
First, encoding studies can work on more ecological and richer stimuli.
Hence they capture all facets of cognition, but must rely on computational models of the stimuli,typically borrowing from artiﬁcial intelligence [Varoquaux and Poldrack, 2019].
Second, they modelbrain responses using high-dimensional statistical models focused on prediction.
These can ﬁt moreingredients jointly, avoiding difﬁcult modeling choices.
As a result, they can generalize ﬁndingsacross stimuli probing different parts of a cognitive ontology: natural images, simpliﬁed faces, orwedges traditionally used for retinotopic mappings [Eickenberg et al., 2017].
This is in sharp contrastwith conventional brain mapping methodology:
based on oppositions between stimuli, it does notlead to formal models bridging results from different experimental paradigms.
How do
ﬁt in
progress
2.1 From scientiﬁc evidence to scientiﬁc knowledge: more than data
Internal versus external validity The validity of a study’s ﬁndings is more than a statisticalquestion.
Internal validity controls inferences about the relations across the quantities in the study,for instance that measurements have no unmodeled errors.
External validity, more important but lessdiscussed, asserts that those relations are maintained beyond the study’s settings
[Cook and Campbell,1979].
It may for instance fail when running a study on a sample non representative of the population.
Validity of constructs Scientiﬁc theories and models are constructed from abstract ingredientssuch as “intelligence” or phrenology’s “alimentiveness” in psychology.
These constructs are centralto reasoning about empirical evidence, to position it in a broader context.
A good construct is onethat is useful to explain many different observations, beyond a single study [Cronbach and Meehl,1955].
Interpreting an empirical study in a theoretical framework requires construct validity of itsmeasures and manipulations: that these indeed to relate well to the construct of interest.
For instance,to be interpretable as intelligence, IQ tests should not be counfounded by cultural knowledge.
Stances on theories Models, and thus theory, are needed to interpret empirical ﬁnding.
Theacceptance of these theories often builds upon implicit stances on their ingredients.
In psychology,
Fried [2020] argues that statistical models should build on “strong theories” and provide “explanationof a phenomenon” relating valid psychological constructs, beyond mere data ﬁt.
Yarkoni [2020]points out that such a view carries implicit preferences on choices of construct that may be difﬁcult todefend.
In particular, such model esthetic assumes realism about psychological constructs: that thesehave an absolute existence beyond the minds of the scientists.
A scientiﬁc discourse must position itsclaims on unobservable constructs, for instance centers of gravity in mechanics.
Realism accepts tobuild scientiﬁc knowledge on unobservable entities only if they are objective and mind-independent.
Instrumentalism , rather, accepts that some ingredients of theories are mere instruments needed to tietogether observable outcomes, and that the success of a theory is asserted solely on these observables.
Questions on the validity of basic modeling ingredients are less discussed in a well-establishedscience such as physics, as there is a consensus on the ingredients: forces, acceleration, temperature–which has a non-trivial deﬁnition–... And yet, this consensus was achieved through iterations.
Planetary observations in the times of Kepler were analyzed with phenomenological models lackingthe ingredients of dynamics, but were fundamental to nourishing Newtonian mechanics.
2.2 Reasoning with statistical tools
Statistics gives the scientist tools to reason from noisy observations.
The prevailing approach ismodel reasoning : a probabilistic model describing data generation is built, encompassing ingredientsof the application domain.
Parameters estimated using this model are interpreted within its logic [Cox,2006, chap 9].
Cox [2001] goes as far as saying that statistical models are “efforts to establish datadescriptions that are potentially causal”.
Another form of reasoning –design-based [Cox, 2006, chap9] or warranted reasoning [Cook, 1991, Baiocchi and Rodu, 2021]– relies on speciﬁc experimentaldesign, as randomization, for causal inferences without a model of the data-generating mechanism.
Finally, Breiman [2001] famously noted that increasingly many statistical tools forgo data modeling,to focus on algorithmic capacity to approximate relations.
Their success is established by outcomereasoning : gauging predictions on observables [Baiocchi and Rodu, 2021], key to machine learning.
Grounding more statistical reasoning on output rather than models
With a historical emphasis on data modeling, statistics has an implicit realism stance.
Yet, as we haveseen in physics or vision neuroscience, scientiﬁc progress is achieved despite analyzing observationswithout the right conceptual framework.
Outcome reasoning, with tools of machine learning, gives arobust statistical framework for science: given imperfect premises, it fails less.
3.1 Robustness to model mis-speciﬁcation
With model reasoning, parameters can be interpreted only conditional to the choice of model, whichis outside of statistical control.
Statisticians often assume that many hard modeling questions can beresolved by domain experts.
Yet science is performed by limited beings [Wimsatt, 2007] and evenexperts have ﬁnite resources to dedicate to a given problem [Simon, 1955].
Model imperfections canhave vast consequences on statistical conclusions.
Botvinik-Nezer et al.
[2020] asked 70 differentteams of experts to analyze the same brain imaging data.
Variations in modeling choices –all based onlinear models– lead to vastly different parameters, and qualitatively different neuroscience ﬁndings.
Controlling predictions instead of model parameters leads to a different statistical regime.
Even thesimple case of the linear model changes drastically: with learning theory, analysis is possible evenin the miss-speciﬁed setting, showing that multi-colinearity in the design is not an issue [Hsu et al.
,2014], unlike when performing inference on model parameters.
Higher-dimensional settings arepossible, which means that the analyst no longer has to cherry-pick a small number of descriptors.
In neuroscience, it has enabled studying richer descriptions of the stimuli, generated by artiﬁcialintelligence techniques rather than set in a speciﬁc reductionist theoretical framework.
Switching tooutput reasoning requires reinventing analytical paradigms: in brain imaging switching to decodingmodels that gauge the ability to predict neural responses.
3.2 Putting
explicit generalization at the center of the inference
Judging a model by its predictions is good science.
It shifts the burden on validity on observables.
These may suffer biases, such as censoring, which must be accounted for even in machine-learningsettings [Ishwaran et al., 2008].
But in the long run, the validity of scientiﬁc theories is established bytheir ability to generalize across many settings.
Cross-validation on a study sample is however not a test of a strong ability to generalize; it givesno evidence of external validity.
Machine-learning models may easily create local approximationswhich do not generalize to new settings, bad scientiﬁc models.
Yet, their ability to generalize canbe explicitly tested.
This is unlike model-based tests of qualitative theories, as in psychology orsociology.
Indeed, a methodology based on machine learning can be applied to rich descriptionsof the objects under study –the raw images presented–, while model-based reasoning is applied toa small number of features, specially crafted to represented the constructs of interest –a face-placeopposition.
In the former, the generalization is readily tested on data from different settings via aquantitative prediction error.
In the latter, the ﬁnding is more conceptual and given a new setting itmust be instantiated with a new modeling effort.
Beyond broad generalization, an oft-requested feature of an analytical model is to provide “understanding”.
For domain reasoning, it is helpful to try to tease out the contribution of various ingredients.
An emerging non-parametric statistical toolbox is catering to this purpose: black-box explanationtechniques [Molnar, 2020], such as partial dependency plot [Friedman, 2001] or the knock-off [Barberand Candès, 2019].
These tools ground their inferences on model outputs, the quantities amenableto strong empirical validation.
Demanding more from an analytical model, for instance opposingphenomenological data explanations with valid theoretical understanding, forces buying into a giventheoretical framework Yarkoni [2020], with the risk of circular reasoning on the evidence.
Parametric models are appealing for intuitive counterfactual reasoning [Angrist and Pischke, 2008]:they appear as “data descriptions that are potentially causal”
[Cox, 2001].
Yet, more than a parametric model, valid causal inference needs a structural characterization of variables, distinguishingconfounders, colliders, mediators...
[Greenland et al., 1999].
In such settings, machine learningmodels shine by their potential robustness to mismodeling [Rose and Rizopoulos, 2020].
Black-box models for thinking outside the box Empirical validation of a theory tied to itsingredients smells of self-fulﬁlling prophecies.
This is the risk of model-based statistical reasoning.
Science needs statistical reasoning based more on model predictions.
Machine learning