We examine properties of permutation tests in the context of synthetic control. Permutation tests are frequently used methods of inference for synthetic control when the number of potential control units is small. We analyze the permutation tests from a repeated sampling perspective and show that the size of permutation tests may be distorted. Several alternative methods are discussed. Econometrics 2017, 4, 52 2 of 12 we document the size distortion of the permutation tests. We also discuss a few alternative methods of inference. Alberto Abadie kindly pointed out that the placebo test in synthetic control is often based on randomization inference idea, under which the symmetry restriction is built-in, while our analysis is predicated from the usual random sampling perspective, which leads to the violation of symmetry. This perspective is shared with an anonymous referee, who notes that (i) the synthetic control literature uses permutation tests in the context of design-based inference, and, as such, the permutation tests have exact size; (ii) the present article shows that permutation tests may not have correct size under a different mode of inference based on repeated sampling, although interpreting the permutation tests in the previous literature as tests based on repeated sampling would be incorrect; and (iii) the present article also proposes some alternatives that are valid in a repeated sampling setting. It would be useful to understand the exact mechanism through which the difference between the two perspectives manifests itself. The same referee points out that the present paper adopts a setting where T 0 → ∞, while the original litereature assumes fixed T 0 . In this section, we provide a brief discussion of the placebo test in the context of the synthetic control method. We begin with an overview of the synthetic control, borrowing heavily from discussions in Abadie et al. (2010) and Doudchenko and Imbens (2016) . We then move on to describe the placebo test, and point out the importance of the symmetry assumption. We argue that the symmetry assumption is violated in general for placebo tests using linear combinations of outcomes, such as synthetic control. We conclude this section that such violation should be expected in general even when a normalized version of the test statistic is adopted. We start with the overview of the synthetic control method. Consider a panel data with J + 1 cross sectional units observed over the time periods t = 1, . . . , T. Units j = 1, . . . , J are the control units that receive the treatment in none of the time periods. The unit j = 0 receives no treatment in periods 1, . . . , T 0 , and receives active treatment in time periods t = T 0 + 1, . . . , T. For simplicity, we will often assume that T = T 0 + 1. The outcome variable Y j,t is such that Y j,t = Y j,t (1) if the jth unit receives treatment in time t, and Y j,t = Y j,t (0) otherwise. Obviously, Y j,t = Y j,t (0) , j = 1, . . . , J; t = 1, . . . , T, The idea underlying the synthetic control is that if there were some weights 1ω 1 , . . . ,ω J such that during the pre-intervention periods (t = 1, . . . , T 0 ). Then, ∑ J j=1ω j Y j,t can be used as a (synthetic) control for Y 0,t during the post-intervention periods (t = T 0 + 1, . . . , T). Abadie et al. (2010) and Doudchenko and Imbens (2016) discuss various methods of finding theω's so that the requirement in Equation (1) is satisfied. We analyze the weights and the nature of approximation from the asymptotic 1 Doudchenko and Imbens (2016) also consider a slightly more general requirement Y 0,t ≈ α + ∑ J j=1 w j Y j,t . This is a sensible way to enhance accuracy of synthetic control viewed as a point estimator. It also provides a link to the difference-in-differences estimator. Because our focus is on inferential aspects of the problem, we simplify notation and analysis by abstracting away from the intercept term. 7 Abadie et al. (2010) also impose the positivity restriction, i.e., ω j ≥ 0 for all J. 8 It is straightforward to prove that under stationarity assumption, the only model that allows the synthetic controls to trace the trajectory of the outcome for the treated (i.e., Y 0,t = ∑ J j=1 ω j Y j,t for some ω) is a linear factor model with Var( j,t ) = 0. 9 See Ferman and Pinto (2017) for related discussion on the bias of the synthetic control estimator. J j=1 ω * j Y j,T 0 +1 (0), assuming that the first and second terms in the factor model in Equation (3) are not present: 11 11 This can be done by assuming that α 0 = ∑ J j=1 ω j α j and θ t = 0. Econometrics 2017, 4, 52 9 of 12 Y 0,T 0 +1 (0) − J ∑ j=1 ω * j Y j,T 0 +1 (0) = γ 0 − J ∑ j=1 ω * j γ j δ T 0 +1 + 0,T 0 +1 − J ∑ j=1 ω * j j,T 0 +1 . This implies that the variance of Y 0,T 0 +1 (0) − ∑ J j=1 ω * j Y j,T 0 +1 (0) can be written as
Alternatively, REF assess the same procedure from the perspective of a placebo test, arguing that the approach fails to control Type I error.
Synthetic Control and Inference
Econometrics
Mathematics