  Online experiments are the gold standard for evaluating impact on user
experience and accelerating innovation in software. However, since experiments
are typically limited in duration, observed treatment effects are not always
permanently stable, sometimes revealing increasing or decreasing patterns over
time. There are multiple causes for a treatment effect to change over time. In
this paper, we focus on a particular cause, user-learning, which is primarily
associated with novelty or primacy. Novelty describes the desire to use new
technology that tends to diminish over time. Primacy describes the growing
engagement with technology as a result of adoption of the innovation.
User-learning estimation is critical because it holds experimentation
responsible for trustworthiness, empowers organizations to make better
decisions by providing a long-term view of expected impact, and prevents user
dissatisfaction. In this paper, we propose an observational approach, based on
difference-in-differences technique to estimate user-learning at scale. We use
this approach to test and estimate user-learning in many experiments at
Microsoft. We compare our approach with the existing experimental method to
show its benefits in terms of ease of use and higher statistical power, and to
discuss its limitation in presence of other forms of treatment interaction with
time.
