metadata: authors: 0: author: Falorsi,Piero Demetrio
1: author: Righi,Paolo
2: author: Ponzini,Giulia
count: World
docna: 0: docna: Improving the Quality of Survey Estimates
            from Longitudinal Studies : An Application to LSMS Panel Surveys
docty: Policy Research Working Paper
abstracts: cdata!: Longitudinal surveys are an invaluable
            source for analyzing the current state of and changes in
            human populations over time. However, maintaining the
            accuracy of estimates from a panel sample becomes more
            difficult as the length of the panel survey increases. Key
            concerns are the lack of sample representativeness, due to
            the sample erosion caused by deaths and movers and the
            impact of new births, and migration flows. Moreover, sample
            fatigue introduces an increasing measurement error. Correct
            design, implementation, and use of a panel survey considers
            a set of methods to deal with these problems at different
            stages of the statistical process: the sampling design, the
            data collection, and the estimation. This paper focuses on
            the case of panels with a rotating sample design. This case
            represents a powerful hybrid solution for facing the impact
            of panel dynamics on sample representativeness. Empirically,
            the paper focuses on the estimation procedures. Using data
            from the Uganda National Panel Survey, the longest Living
            Standards Measurement Study panel survey, it experimentally
            evaluates the suggested technique. In summary, the findings
            show that the calibrated generalized weight share method
            base estimator yields individual-level statistics that
            appear to be more accurate than those produced by the
            current Uganda National Panel Survey estimator.
            Additionally, the calibrated generalized weight share method
            base cross-sectional estimates on the transition matrix show
            a generally higher degree of stability when the sample is
            changed compared to the current Uganda National Panel Survey estimates.
display_title: Improving the Quality of Survey
            Estimates from Longitudinal Studies : An Application to LSMS
            Panel Surveys
listing_relative_url: /research/2025/01/34443991/improving-quality-survey-estimates-longitudinal-studies-application-lsms-panel-surveys
topicv3: Household Surveys m1504071 2761
publishtoextweb_dt: 2025-01-08T00:00:00Z
dois: 10.1596/1813-9450-11026
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
no_of_pages: 37
author_keys: 0: author_key: 1885718
1: author_key: NA
2: author_key: 1487387
model_extractions: Policy Research Working Paper 11026 Improving the Quality of Survey Estimates from Longitudinal Studies An Application to LSMS Panel Surveys _Piero Falorsi Paolo Righi Giulia Ponzini_ Development Economics Development Data Group January 2025 A verified reproducibility package for this paper is available at http://reproducibility.worldbank.org, click **here** for direct access.
Policy Research Working Paper 11026

# **Abstract**

Longitudinal surveys are an invaluable source for analyzing the current state of and changes in human populations over time. However, maintaining the accuracy of estimates from a panel sample becomes more difficult as the length of the panel survey increases. Key concerns are the lack of sample representativeness, due to the sample erosion caused by deaths and movers and the impact of new births, and migration flows. Moreover, sample fatigue introduces an increasing measurement error. Correct design, implementation, and use of a panel survey considers a set of methods to deal with these problems at different stages of the statistical process: the sampling design, the data collection, and the estimation. This paper focuses on the case of panels with a rotating sample design. This case represents a powerful hybrid solution for facing the impact of panel dynamics on sample representativeness. Empirically, the paper focuses on the estimation procedures. Using data from the Uganda National Panel Survey, the longest Living Standards Measurement Study panel survey, it experimentally evaluates the suggested technique. In summary, the findings show that the calibrated generalized weight share method base estimator yields individual-level statistics that appear to be more accurate than those produced by the current Uganda National Panel Survey estimator. Additionally, the calibrated generalized weight share method base cross-sectional estimates on the transition matrix show a generally higher degree of stability when the sample is changed compared to the current Uganda National Panel Survey estimates.

This paper is a product of the Development Data Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at gponzini@worldbank.org. A verified reproducibility package for this paper is available at http://reproducibility. worldbank.org, click **here** for direct access.

<!-- Start of picture text --> TRANSPARENT<br>ANALYSIS<br>G<br>R E<br>N<br>S<br>I P<br>Y E<br>K A<br>C A<br>R P<br>LI R<br>O E<br>O C<br>W R<br>H<br>S<br>P<br><!-- End of picture text --> _The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Improving the Quality of Survey Estimates from Longitudinal Studies:

# An Application to LSMS Panel Surveys

Piero Falorsi<sup>1</sup> , Paolo Righi<sup>2</sup> , Giulia Ponzini<sup>3</sup>

## JEL codes: C81, C83, C87, Y80

Keywords: Panel surveys, attrition, tracking, Generalized Weight Share Method, survey methods.

> 1International Consultant, Development Economics Data Group, World Bank, 1,2 Italian Insititute of Statistics (ISTAT), 3 Development Economics Data Group, World Bank;

This paper was produced with financial support from the 50x2030 Initiative to Close the Agricultural Data Gap, a multipartner program that seeks to bridge the global agricultural data gap by transforming data systems in 50 countries in Africa, Asia, the Middle East and Latin America by 2030. For more information on the Initiative, visit 50x2030.org.
# 1. INTRODUCTION

Longitudinal surveys, based on repeated observations of the same statistical units over time, are an invaluable source for analyzing the current state and changes in human populations over time. The Living Standards Measurement Study – Integrated Surveys on Agriculture (LSMSISA) initiative demonstrates the value of longitudinal surveys by disseminating household panel data on issues such as health, agriculture, access to basic services, nutrition, poverty status, and much more, for countries across Sub-Saharan Africa. The European Union’s Statistics on Income and Living Conditions (EU SILC) and the US Bureau of Statistics’ Survey of Income and Program Participation (SIPP) are other key examples of the value of longitudinal studies.

In panel surveys, accuracy depends on many of the same factors as in cross-sectional surveys. However, maintaining the accuracy of estimates from a panel sample becomes more difficult as the length of the panel survey increases. One key concern is the panel attrition of respondents who drop out of the survey, which can become increasingly troublesome for long-running panels. Additionally, movers – that is, people who have moved away from where they were interviewed in the first observation of the longitudinal study – represent a dynamic subpopulation and are complex and costly to interview. Finally, new entrants to the population, such as immigrants and newborns, are particularly difficult to capture in traditional panel surveys.

The correct design, implementation, and use of a panel survey considers several methods to address these problems at different stages of the statistical process, namely the sample design, data collection, and estimation stages. To the extent possible, sample design should minimize the loss of representativeness of panel data due to attrition, leavers, and new entrants. Practical approaches to this include the use of refreshed samples (i.e., rotating panels or split panels) and protocols for incorporating individuals not previously involved in the study to capture population dynamics. Based on tracking rules to follow the movers, the data collection stage should be designed to retrieve essential information on individuals or households that have dropped out of the sample. The use of flexible modes of data collection (e.g., telephone interviews) makes it possible to reach those who may not respond to traditional techniques. Additionally, collecting a minimal set of variables on dropouts (e.g., by proxy or doorstep interview) can improve the quality of the survey estimates. The estimation stage considers dropouts, movers, new entrants, and the dynamics of the target population over time. This is achieved by defining a weighting procedure that updates the direct sampling weights to account for new individuals in panel households and by using calibration estimators with upto-date known population totals.

This work proposes to improve the quality of panel surveys by considering the three aspects of survey design outlined above. We consider the estimation at the current time of cross-sectional and longitudinal parameters of a target population. Estimates are computed from sample data collected at the current time on individuals who entered the sample previously and were subsequently followed according to the rules of longitudinal observation considered in the survey. This approach clarifies many aspects of the representativeness of data collected in panel studies. It applies to the many cases that characterize current large-scale household panel surveys (e.g., EU SILC, LSMS, etc.) and can be easily extended to more complex issues in actual survey situations.

The paper is structured as follows: Section 2 gives a formal definition of longitudinal and crosssectional populations and the target parameters to be estimated in longitudinal studies. Section 3 presents the sampling framework and the estimator based on multisource (Singh and Mecatti 2011) and indirect sampling (Lavallée 2007; Falorsi, Righi and Lavallée 2019), which addresses 2
the different representativeness of the various sampling subpopulations and the changes in household composition over time. Section 4 presents improvements in field operations and data collection to facilitate the implementation of the proposed methodology. Section 5 briefly introduces the Living Standards Measurement Study (LSMS), defining its objectives and key aspects of the quality of the estimates. Section 6 illustrates the empirical application of the proposed methodology to the LSMS-ISA data, using the 2009, 2013 and 2015 waves of the Uganda National Panel Survey as a case study. Section 7 summarizes the main findings and concludes.

# 2. FORMAL DEFINITION OF CROSS-SECTIONAL AND LONGITUDINAL POPULATIONS

2.1. Populations observed at different points in time is . We represent tt tt tt Let UU be the population of NN individuals of a given country at the current time t. UUt = tt tt,1 tt,ii tt,MM partitioned into MM∑ . subpopulations of households denoted as UU , ... , UU , ... , UU tt tt,ii tt,ii tt the set of households as HHMMtt = {1, ... , ii, ... , MM }. The household UU has NN individuals and NN ii=1 tt,ii as cross-sectional populations of individuals and households since NN they refer to a specific point in time. tt tt We can refer to UU and HH To analyze the dynamics of cross-sectional populations in a longitudinal setting from an initial longitudinal-populations . For the sake of simplification, we first introduce a population not partitioned in households. Thus, point in time tt<sup>∗</sup> , to the current time tt, we introduce the concept of individuals. The longitudinal-population identification is complicated by both the scope of study and the longitudinal observation's tt←tt<sup>∗</sup> tt←tt<sup>∗</sup> operability over time (Helliot et al. 2009). The most common definitions are the intersection-let UU denote the longitudinal population of NN . tt←tt<sup>∗</sup> tt<sup>∗</sup> tt tt←tt<sup>∗</sup> , and includes the individuals tt<sup>∗</sup> tt population, UU = UU ∩UU , and the union-population UU = UU ∪UU who are still living and resident in the country at the time t. The longitudinal survey tt<sup>∗</sup> tt observes the same individuals over time, excluding both births and deaths during the period of The intersection-population is a proper subset of both UU and UU tt<sup>∗</sup> of UUstudy. However, the longitudinal population definition based on the intersection-population has some drawbacks related to the representativeness at time t and to the limitations in the study of the dynamics in the households’ composition. While the union-population best describes the population dynamics, it requires a dynamic sample design which introduces births and entrants to avoid bias. For this study, we propose a definition that takes the advantages from both approaches. , include: - tt←tt<sup>∗</sup> tt←tt<sup>∗</sup> ; Based on this, the NN individuals of UU<sup>∗</sup>

- , include:

- - tt←tt<sup>∗</sup> tt←tt<sup>∗</sup> ; - tt<sup>∗</sup> tt

- all the people of the intersection-population of households UU ∩UU

- all new members of the household of the individuals in the intersection-population, even<sup>∗</sup> .

- With this definition, the newborns and those immigrants who become be part of the tt<sup>∗</sup> if they were not part of UU

households of the individuals in the intersection-population are new entries into the . If immigration to the country and newborns are numerically approximates the crosspopulation from time tt<sup>∗</sup> is conceptually tt←tt<sup>∗</sup> insignificant phenomena, then the longitudinal population UU, and then in the following survey we observe tt←tt<sup>∗</sup> sectional population at time tt. Moreover, the longitudinal observation of UUall household members for the initial sample of individuals. simple. We select a sample at the initial time, tt<sup>∗</sup> 3
<!-- Start of picture text --> jea\ yee<br>” AS<br>foe“ \ a B2<br>Bs<br>| B3 |<br>i” B4 x a:<br>: B4<br>: AP Household<br>/ ies<br>| ‘we INE Head of household<br>— | ews i GE Household member<br>—_—» (line<br>|<br><!-- End of picture text -->
the new household that has been created. Thus, the continuity is used to define operational rules that make it possible to identify which household at the current time is the continuation of a household observed at a previous time. The continuity rules for identifying the longitudinal , we exclude household d from the analysis. household at the time tt may be different (Falorsi et al., 2009). In figure 2.1, we see that if the continuity rule identifies as the longitudinal household the one with the same head of time tt<sup>∗</sup> To overcome this problem, we propose a many-to-many approach: one household from time tt are those comprising<sup>∗</sup> may generate many households at time tt; conversely, one household at time tt may derive from . In figure 2.1, we see that household A at time 1 continues at time 2 with tt←tt<sup>∗</sup> several households from time tthouseholds a, b, and d.<sup>∗</sup> . The households of population HH tt←tt<sup>∗</sup> people of UU is one , if tt Formalizing the many-to-many criterion and without loss of generality, household kk of UU of the longitudinal households that derive from the household l at time tt> 0,<sup>∗</sup> (2.1) �NNtt∗,l �NNtt,kk tt←tt<sup>∗</sup> tt←tt<sup>∗</sup> kkii,ljj kk,l jj=1 ii=1 ll = LL tt←tt<sup>∗</sup> kkii,ljj where ll is the link {0,1} variable which assumes value 1 if individual jj of the household l at correspondence between the longitudinal population of people and that of households. For time tt<sup>∗</sup> is the same individual ii of household kk at time tt. With this approach, there is a perfect , and at the current tt←tt<sup>∗</sup> each person of UU , we may define their household at the initial time, tt<sup>∗</sup> time tt. Definition 2.1 includes as a particular case the one-to-one continuity rule. In this tt←tt<sup>∗</sup> . kk,l case, LL is a {0,1} dichotomous variable, which equals to one if household kk is the only one tt tt<sup>∗</sup> household of UU which is the longitudinal continuation of the household l of UU

# 2.2. Parameters of interest

5
# 2.2.2. Longitudinal parameters of individuals

The net change parameter is given by (2.5) ∆ ∗ ∗ . pp pp pp Parameter 2.5 expresses the difference between the cross-sectional total at the time t with the tt←tt tt tt = YY − YY . The net change reflects changes in both the characteristics and composition of the population. corresponding total at time tt<sup>∗</sup> Analysis of net change using cross-sectional aggregate estimates may conceal significant status changes at the individual level, from now on referred to as gross changes (Steel et al. 2009). Gross change implies the measurement of the phenomenon in the same units over time.

6
Schema 2.2. below shows the computation of the household gross change according to the continuity rules and the household relationships depicted in Figure 2.1.

7
**Schema 2.2. Gross change parameter computation from household continuity rules for Figure 2.1**

|One-t<br>|o-one continuityrules*<br><br>|||One-to-manycontinu<br>|ityrule<br>||
|---|---|---|---|---|---|---|
|Household|t<sup>∗</sup><br>a<br>b|c<br>D|Household|t<sup>∗</sup>|a<br>b|c<br>d|
|_t_<br>Povertys|tatus atgiven time^<br>0<br>0|1<br>0|_t_<br>Povertys|tatus atgiven time^|0<br>0|1<br>0|
|A|0<br>1<br>0|0<br>0|A|0|1<br>0|1/3<br>1|
|B|1<br>0<br>1|0<br>0|B|1|0<br>2/3|1/3<br>0|
|C|1<br>0<br>0|1<br>0|C|1|0<br>1/3|1/3<br>0|
|Hhld<br>|||Hhld<br>||||
|ouseo<br>U2←1|Computation<br>Parameter|%|ouseo<br>U2←1|Computation|Parameter|%|
|Positive change event|1(Bb)<br>1|33.3|Positive change event|2/3(Bb)+1/3(Cb)|1|25.0|
|Negative change event|0<br>0|0.0|Negative change event|1/3(Ac)|1/3|8.3|
|Stationaryevent|1(Aa)+ 1(Cc)<br>2|66.7|Stationary event|1 (Aa)+1(Ad)+1/3 (Bc)+<br>1/3(Cc)|8/3|66.7|
|Total househ|olds<br>3|100|Total ho|useholds|4|100.0|
|*Continuity rule follows t<br>^Cross-sectional poverty s|he head of household<br>tatus=1, non-poverty status=0||^Cross-sectional poverty|status=1, non-poverty s|tatus=0||

# 3. SAMPLING SETTING AND ESTIMATOR

As anticipated in the introduction, we consider the case where longitudinal and cross-sectional computed using data from a longitudinal survey in which individuals were first observed at estimates referring to the current time (e.g., the year 2024) denoted as time tt are to be intermediate between (e.g., the year 2020) in a refresh of the initial survey (based on well-specified tracking and continuity rules), in which a new sample is also independently selected. This observational time tt<sup>∗</sup> (e.g., the year 2015) and subsequently observed again in year tt<sup>∗∗</sup> setting is very attractive as it summarizes most of the estimation and representativeness tt and tt<sup>∗</sup> problems that can arise in actual large-scale longitudinal studies. The following subsections formalize, using specific notation, the aspects related to sampling design, movers, and direct estimation.

# 3.1. Sampling design

We introduce below some symbols to better illustrate the operations just described. Initial time , from the cross-sectional ,l tt<sup>∗</sup> tt<sup>∗</sup> . We select a fixed sample size of households, SS= ∑ ,l ,l individuals of sample tt<sup>∗</sup> tt<sup>∗</sup> tt<sup>∗</sup> tt<sup>∗</sup> population HH , where household l∈HH enters the sample with inclusion probability ππ . SS tt<sup>∗</sup> tt<sup>∗</sup> l∈HH,l) member of household l is included in the sample tt∗ tt<sup>∗</sup> tt<sup>∗</sup> includes mm households, being mm ππ . We observe all NN tt<sup>∗</sup> = ∑l=1 ,l denote the household l. Thus, the jj-th (jj= 1, ... , NN in terms of individuals. mmtt∗ tt<sup>∗</sup> tt<sup>∗</sup> with the same probability of inclusion in their household. Let nn NN Intermediate time tt<sup>∗</sup> , of households from the cross-sectional realized sample size of SS tt<sup>∗∗</sup> tt<sup>∗∗</sup> . We select a new sample, SS . We tt<sup>∗∗</sup> tt<sup>∗∗</sup> tt<sup>∗</sup> tt<sup>∗∗</sup> population HH . SS is selected independently from SS and is a fixed-size sample of mm tt<sup>∗∗</sup> tt<sup>∗∗</sup> ,kk households. Household kk∈HH of individuals. enters in the sample with the inclusion probability ππ tt<sup>∗∗</sup> ,kk observe in the survey all NN individuals of sampled household kk. In this way, we observe a tt<sup>∗∗</sup> time, the enumerators track sample of nn people who they are able to track, as guided tt<sup>∗</sup> by specific Moreover, starting from sample SS tracking rules . Tracking rules are fundamental in deciding whether to follow people interviewed in the previous tt<sup>∗</sup> tt<sup>∗</sup> and collect the information of interest on all nn . For example, as mentioned in the who moved from where they were interviewed at time tt<sup>∗</sup> 8
specific LSMS-ISA Uganda National Panel Survey (UNPS) survey documentation, all individuals older than 15 years that are biologically related to the head of household who leave a sampled household are found and interviewed, as long as they do not move out of the parish where they are not tracked and their characteristics of interest are not recorded. However, referring again to were initially located. The latter rule implies that people who left their parish after time ttthe specific example, it is useful to note that regardless of the tracking rules, UNPS survey<sup>∗</sup> enumerators often track all those who remained within the Uganda border, even if they moved out of the original EA or parish. is re-contacted for the survey, the enumerator identifies all of their household members and collects the survey data. This ensures tt<sup>∗</sup> that the study captures changes in the household composition (due to births, deaths, marriages, Once an individual selected in the original sample SS . tt<sup>∗</sup> etc.) of all the individuals tracked in survey SS that were tt<sup>∗</sup> tt<sup>∗∗</sup> ←tt<sup>∗</sup> tt<sup>∗∗</sup> ←tt<sup>∗</sup> Thus, starting from the original sample SS , the longitudinal sample SS. of mm tt<sup>∗∗</sup> ←tt<sup>∗</sup> tt<sup>∗∗</sup> ←tt<sup>∗</sup> tt<sup>∗</sup> households with nn individuals is formed. SS contains all persons of SS Current time tracked, along with all members of their households at time tt<sup>∗∗</sup> are tracked and surveyed. The rules of tracking and data collection are the same as those tt<sup>∗∗</sup> tt<sup>∗∗</sup> ←tt<sup>∗</sup> tt. No new sample is selected. The SS and SS samples formed at the time tt<sup>∗∗</sup> sample. tt<sup>∗∗</sup> ←tt<sup>∗</sup> according to the specific survey’s tt<sup>∗</sup> already described for the formation of the SS longitudinal sample from the SS tt<sup>∗∗</sup> tt<sup>∗∗</sup> Thus, the enumerators track all nn individuals of sample SS ∩ tt←tt<sup>∗∗</sup> tt←tt<sup>∗∗</sup> tracking rules. This way, the enumerators form longitudinal sample SS , with mm and observed in longitudinal sample tt←tt<sup>∗∗</sup> tt<sup>∗</sup> households and nn people. Furthermore, the enumerators track all the people of SS tt<sup>∗∗</sup> ←tt<sup>∗</sup> tt<sup>∗</sup> SS who are the people selected in original sample SS people. tt<sup>∗∗</sup> ←tt<sup>∗</sup> tt←tt<sup>∗</sup> tt←tt<sup>∗</sup> tt←tt<sup>∗</sup> SS . So, the enumerators form longitudinal sample SS , with mm households and nn . tt←tt<sup>∗∗</sup> tt←tt<sup>∗</sup> The samples for the estimation at time tt are SS and SS

# 3.2. Movers and representativeness of the samples

Movers create two types of problems in longitudinal surveys. First, there are additional costs. Surveys that employ face-to-face interviews use cluster and multi-stage sampling designs to control costs. Following a mover to a new address may incur considerable extra expenses if the new address is not in one of the original sample areas. Also, there is a risk that no interviewer may be available to visit a mover if the move is discovered during the field operations. Thomas et al. (2001) discuss the cases of certain surveys in low-income countries that suffered significant attrition and bias because they did not track movers. Therefore, new contact information should be obtained to facilitate collecting data on the mover through phone or online interviewing. Another solution is to collect the data of interest on movers through proxy interviews; however, using proxies requires implementing adjustments in the estimation phase to reduce bias.

Based on the above, tracking protocols should be defined to determine a reasonable approach for balancing increased costs on one hand and distortion or bias on the other.

The target current population of households can be defined as the union of the three disjoint subpopulations: ∗ ∪E ∗∗ ∗ ∪E ∗∗ where tt tt←tt tt ←tt tt←tt (3.1) HH = FF 9
 or is eligible for tt←tt<sup>∗</sup> FFFor example, for UNPS, the households eligible for tracking are those in which at least is the household subpopulation that remains fixed from time tt<sup>∗</sup> tracking from time tt<sup>∗</sup> to time tt according to the tracking rules adopted for the survey. does not include the households of all immigrants, which contain only new people one individual was already present at time tt, and the households consisting only of people who<sup>∗</sup> and lives in the original parish until time tt. tt←tt<sup>∗</sup> FF out of their parish. entering the country after time tt<sup>∗</sup>  E comprises the households whose members consist only of immigrants or movers moved after time tt<sup>∗</sup> ]. tt<sup>∗∗</sup> ←tt<sup>∗</sup>  E includes the households whose members consist only of immigrants or movers in in the time interval [tt<sup>∗</sup> , tt<sup>∗∗</sup> tt←tt<sup>∗∗</sup> Partition 3.1 and Figure 3.1. define the representativeness of the two longitudinal samples the time interval [tt<sup>∗∗</sup> , tt]. . Herein, we refer to representativeness as the possibility of computing direct unbiased estimates of a given population from the sample tt←tt<sup>∗∗</sup> tt←tt<sup>∗</sup> tt (SSdata. and SS ) regarding the sub-populations of HH For the Sample only to the sub-population tt←tt<sup>∗</sup> tt←tt<sup>∗</sup> SS , the tracking rules limit the representativeness of sample SS to time t. tt←tt<sup>∗</sup> FF For Sample , the tracking rules restrain the representativeness of the sample data only to of households of time tt who have been eligible for tracking from time tt<sup>∗</sup> the sub-population tt←tt<sup>∗∗</sup> SS tt←tt<sup>∗∗</sup> FF and E , being of households of time tt who have been eligible for tracking from time tt<sup>∗∗</sup> to time tt. ∪E . tt←tt<sup>∗∗</sup> tt←tt<sup>∗</sup> tt<sup>∗∗</sup> ←tt<sup>∗</sup> FF can be partitioned into the two subpopulations FF , we have no people belonging to the subpopulation Ett←tt<sup>∗∗</sup> tt←tt<sup>∗</sup> tt<sup>∗∗</sup> ←tt<sup>∗</sup> ,, FF = FF who are the people entered in the population after the last panel refresh. As we have no sample tt←tt<sup>∗∗</sup> tt<sup>∗∗</sup> ←tt<sup>∗</sup> Finally, we see that in sample SSdata on this subpopulation, we cannot compute direct estimates.

10
<!-- Start of picture text --> Sample $). . represents proc |<br>Prue<br>_ ° a<br>Timeline - -<br>t* t t<br>No sample data |a<br><!-- End of picture text -->

1 ifk €

-) ) [G-F )G-e ) -) [@-F )a-e J
# 3.3. Estimation process

12
∗ ∗∗ (tt←) tt←tt tt←tt (3.12) SS = SS ∪SS be the longitudinal sample-union at current time tt, obtained by joining together the two individuals. tt←tt<sup>∗</sup> tt←tt<sup>∗∗</sup> (tt←) (tt←) tt←tt<sup>∗</sup> tt←tt<sup>∗∗</sup> longitudinal samples SS and SS . The sample SS has mm = mm + mm (tt←) tt←tt<sup>∗</sup> tt←tt<sup>∗∗</sup> households, and nn = nn + nn in the standard form as a weighted sum of the data in the sample union as: FFtt←tt∗∗ We may express the estimator YY<sup>�</sup> , = � �NNtt,kk FFtt←tt∗∗ tt,kkii SS(tt←),kk (3.13) YY<sup>�</sup> kk∈ SS(tt←) ii=1 yy dd . With straightforward algebraic manipulation, starting from the above formula (3.6), ... , (3.11) we SS(tt←),kk (tt←) where dd is the direct sample weight of household kk of sample SS have ∗∗ ∗ = 1 SStt←tt∗,kk tt←tt<sup>∗</sup> , ααdd if kk∈SS SS(tt←),kk = � SStt←tt∗∗,kk tt←tt = 0 kk,tt←tt (3.14) dd (1 −αα)dd if kk∈SS ∧ FF SStt←tt∗∗,kk tt←tt<sup>∗∗</sup> kk,tt←tt<sup>∗</sup> dd if kk∈SS ∧ FF are given in formula (3.8) and (3.11) as the base weights of the sample-union. SStt←tt∗,kk SStt←tt∗∗,kk where the expressions for dd and dd SS(tt←),kk respectively. In the following, we denote dd As of the definition of the αα value, Singh and Mecatti (2011) provide an in-depth illustration of the different approaches proposed in literature for fixing the optimal value of αα in the context of multiple frame surveys. Hartley (1962, 1974) proposes choosing αα to minimize the variance (1974) suggests opting for a simpler alternative expression which is always positive, even if it FFtt←tt∗∗ of YY<sup>�</sup> . Unfortunately, the solution depends on the variable yy and αα may be negative. Hartley drawback and closely approximates the Hartley’s solution depends on yy. We suggest here an even simpler solution which does not suffer from the above ~~.~~ ∗ ∗∗ tt←tt<sup>∗</sup> mm (3.15) αα= tt←tt tt←tt mm + mm is known as factor of multiplicity. In the case illustrated in Section 5 below, this factor equals 2. tt←tt<sup>∗</sup> tt←tt<sup>∗∗</sup> If mm ≅mm , as in the case illustrated in Section 5 below, then αα= 0.5. The reciprocal of αα is known as factor of multiplicity. In the case illustrated in Section 5 below, this factor equals 2. 3.3.2. Calibration estimator may not be FFtt←tt∗∗ does not represent SS(tt←),kk The direct estimates YY<sup>�</sup> (see 3.13) obtained with the direct final weights dd E and the estimates could under-cover the actual population for that FFtt←tt∗∗ accurate at least for three reasons, thus requiring adjustment. First, YY<sup>�</sup> segment. Second, non-response may introduce bias in the estimates, as the nonresponding tt←tt∗∗ the aggregate YY population may have different characteristics from those who agree to participate in the survey. Third, there may be differences in the sample estimates with the known totals of some auxiliary . High values of the differences are often a sign of inaccuracy, resulting from either the abovementioned under-coverage problem or if tt variables available on the cross-sectional population HHsome subpopulations exhibited higher non-response rates than others.

The direct estimates can be adjusted jointly for the three issues above using the calibration estimator (Deville and Särndal 1992; Singh and Mohl 1996). This estimator defines calibrated 13
As Särndal and Lundström (2015) argued, the calibration approach can easily leverage the wealth of information available on non-response; if the auxiliary variables are explicative of the non-response, the calibration estimator reduces the no-response bias. In this way, the estimates jointly smooth the non-response bias and calibrate the known population totals. This approach has three advantages for the method often adopted for producing survey estimates, which is based on performing preliminary adjustments for non-response based on no-response propensity scores and which produces weights defined at the individual level, implying that people in the same household may have different sampling weights.

The first advantage is that the sampling variance estimation is better founded and more transparent. Calibration estimators behave asymptotically, as they converge in probability to the regression estimator. They produce robust inferences with sound statistical properties for both the sampling design and the statistical model.

The s econd benefit is that fewer adjustment steps are needed, making the process flow for the estimation more straightforward. Särndal and Lundström (2005, pg. 88) note that if the information on non-response is available only at the sample level, the calibration approach can be carried out in two steps, where the first step calibrates on the known totals at the selected sample level, which includes both respondents and non-respondents. The two-step approach improves the accuracy of the estimates when the reasons for attrition are different from the reasons for sampling under-coverage. Thus, two calibrations are implemented: the first calibration adjusts the base weights for panel attrition and the second calibration uses the adjusted weights to produce the final weights to address the under-coverage.

Finally, the third advantage is that the calibration method produces estimates that are defined at the household level. That means all individuals in the same household have the same final weight, which significantly facilitates the consistency of estimates with variables defined at the individual level and those constructed with variables at the household level. ′ = �X , ... , X , ... , X � from administrative data, the latest census, or demographic tt tt,1 tt,pp′ tt,PP To introduce this estimator, let XX be a column vector of PP auxiliary = �X , ... , Xtt , ... , X � household k such that known totals for the population UU tt,kk tt,1,kk tt,pp,kk tt,PP,kk statistics. Let XX be a vector of auxiliary PP variables of the . �MMtt tt,kk tt Let X kk=1XX = XX are two possible situations regarding the relationship between the auxiliary variables defined tt,pp,kk,ii at the household level, X be the auxiliary variable defined for the ii-th individual of the kk-th household. There , and those defined at the individual level, X . Suppose the variable X can only be defined at the household level (e.g., the household's state of poverty). tt,pp,kk tt,pp,kk,ii In that case, all individuals in the household will have the same value of the household variable, tt,pp,kk 14
> (d ) = min »
# 4. DATA COLLECTION

# 4.1. Tracking rules

Statistical offices define the tracking rules to deal with the potential bias and sample size reduction caused by movers. The definition of the rules must consider the trade-off between improving the accuracy of the estimates and the cost of following the movers throughout the country. This trade-off usually leads to a compromise solution where movers are followed within a limited area. In this case, individuals are followed when they move within the area of the first observation.

To carry out the tracking, the survey needs to define a tracking protocol based on:

- The tracking objectives . To describe the procedure in concrete terms, let us consider the three waves tt or the individuals incorporated in the households in the successive waves. For<sup>∗</sup> , tt<sup>∗∗</sup> , and tt already introduced in Section 4.1. At the start of the fieldwork for the current time tt, we have the sample of individuals and households originally selected at example, as a tracking rule, we can specify that the survey plan applies the tracking protocol tt<sup>∗</sup> and tt<sup>∗∗</sup> .

- - The delimitation of the area in which the tracking is carried out (i.e., Enumeration Area, tt<sup>∗</sup> tt<sup>∗∗</sup> at tt only to the individuals selected in the initial samples SS and SS

- The delimitation of the area District, Parish, etc.).

- The questionnaire definition . For instance, in the event that some individuals have already decided to move, it might be useful to collect information that would facilitate re-contacting them in the next survey (e.g., whether the respondent is thinking of moving in the next wave of the panel, the reason for the move, and the telephone number of the potential mover).

- The protocol for the proxy interviews that defines the minimum set of variables to be collected for non-respondents by doorstep interview and for movers by proxy interview.

- The definition of the data collection procedure for movers , which considers the area of interview (for instance, the data collection may be carried out through a face-to-face interview if the mover stays in the sample area or by phone interview if the mover moves to an area outside the sample).

# 4.2. Linking and other auxiliary variables for computing base weights

We refer to the base weight as the weight calculated according to the Generalized Weight Share Method (Lavalleé, 2007) divided by the multiplicity factor (Section 3.3.1). These weights are defined at household level and each individual has the base weight of the household to which he or she belongs. As explained in section 3.3.1, the GWSM requires (i) the variable linking the individual to the initial sample to which they belong (in the previous example we have two original samples at ) and (ii) the variable linking the individual to the target population, i.e., the year of their first appearance in the target population (for newborns and immigrants). tt<sup>∗</sup> and tt<sup>∗∗</sup> The process of calculating the base weights includes the multiplicity factor, which represents the number of opportunities for a household to be selected in the initial samples. Each updated sample offers one potential opportunity for the units to be selected. As in section 4.15, if 16
to the two conditions below: tt←tt<sup>∗</sup> tt←tt<sup>∗∗</sup> mm ≅mm , we assign the multiplicity factor to the household observed at time tt according Condition 1 . If the household has at least one component that belongs to the target population and is not a mover, the multiplicity factor is 2. Condition 2 . If no component of the household satisfies condition 1, then the multiplicity factor in wave tt<sup>∗</sup> is 1. (non(mover or immigrant Note that the conditions 1) and 2) introduce the concept of fixed individual at time tt has individual<sup>∗</sup> traceable mover with respect to time tt<sup>∗</sup> ) and fixed individual at time tt has individual multiplicity factor equal to<sup>∗∗</sup> 1. with respect to time tt<sup>∗</sup> and fixed with respect to time tt<sup>∗∗</sup> ). A fixed individual at tt<sup>∗</sup> multiplicity factor equal to 2; a fixed individual at tt<sup>∗∗</sup>

- <sup>∗∗</sup>

- 1. The multiplicity factor equals 2 if at least one component was selected in the

- Consequently, in wave tt the following household multiplicity values may occur: .

- 2. The multiplicity factor equals 2 if at least a component of the household was original sample at wave tt<sup>∗∗∗</sup>

- .

- 2. The multiplicity factor equals 2 if at least a component of the household was<sup>∗</sup> of the household equals 1.

- 3. The multiplicity factor equals 1 if at least a component of the household was kk,tt←tt<sup>∗</sup> selected at wave tt<sup>∗∗</sup> and variable FF of the household equals 0.

- Finally, the response indicator variables of the individuals in the current wave and in the first kk,tt←tt<sup>∗</sup> selected at wave tt<sup>∗∗</sup> and variable FF

- wave (for the individuals selected in the original samples) complete the information to build the base weights.

To facilitate operational understanding of the steps involved in constructing base weights at the time t, we list the variables involved in this process and the values they can take in table 4.1.

17
|Table 4.1. Linkin<br>Examp|g and other auxi<br>le: two original|liary variables t to compute th<br>samples(first sample, t<sup>∗</sup>,and|e base weights at the timet.<br>refresh sample, t<sup>∗∗</sup>)|
|---|---|---|---|
|Variable|Symbol<br>|Description|Values|
|Personal<br>identification code|v1<br>|Unique code over time.|Code.<br>|
|Original sample of<br>selection|v2<br>|Time of the original sample including<br>the individual. Missing when the<br>individuals is incorporated in a sample<br>household.|Time: t<sup>∗</sup>, t<sup>∗∗</sup>; missing.|
|Household<br>identification code in<br>the original sample|v3<br> <br>|Code of the individual’s household in<br>the original sample. Unique code over<br>time. Missing when the individual is<br>incorporatedina samplehousehold.|Code; missing.|
|Inclusion probability|v4 = πto,k<br>(with t<sup>o</sup>equals to t<sup>∗</sup><br>or t<sup>∗∗</sup>)<br>|Inclusion probability of the individual in<br>the original sample. Zero when the<br>individual is incorporated in a sample<br>household.|Numeric: constant in the<br>household.|
|Household<br>identification code in<br>the currentwave|<br> <br>v5|Code of the household in the current<br>wave.|Code.|
|Population<br>membership*|v6|Individual membership to the target<br>population related to sample (in<br>relationship with v2variable).|1 (when the individual is from<br>the target population of the<br>original sample);<br>0 (when the individual is not<br>in the target population of the<br>originalsample).|
|Individual<br>multiplicity factor in<br>the current wave (t)|v7<br><br><br>|<br>Multiplicity factor of the individual<br>selected in an original sample with<br>respect to the population t<sup>∗</sup>or t<sup>∗∗</sup>(built<br>considering the xv2variable).<br>|2 (when the individual is a<br>fixed individual with respect<br>to the population t<sup>∗</sup>);<br>1 (when the individual is a<br>fixed individual with respect<br>to the population t<sup>∗∗</sup>);<br>Missing (not in the original<br>samples).<br>|
|Household<br>multiplicity factor in<br>the current wave t|v8 = <sup>1</sup><br>α<br>(α defined in<br>Formula4.15)|Household multiplicity factor. Consider<br>the max of the v7values for the<br>individuals in the household.|2 (when at list an individual of<br>the household has v7=2).<br>1 (otherwise).|
|<br>Response indicator<br>in the original<br>sample (only for<br>estimate gross-<br>changeparameters)|<br>v9<br>|<br>Response indicator of the individual in<br>the original sample.|<br>1 (respondent);<br>0 (non-respondent or not in<br>the original sample)|
|Response indicator<br> <br> <br>|v10<br> <br>|Response indicator of the individual in<br><br> <br>|1 (respondent);<br><br> <br>|
|inthe currentwave<br>*Example: in wave t the<br>Value=1 when<br>Value=1 when<br>Value=0 when<br>time t<sup>∗</sup>).<br>|household has at least on<br>the individual is selecte<br>the incorporated indivi<br>the incorporated indivi<br>|the currentwave.<br>e individual selected in the original sample<br>d at t<sup>∗</sup>(individual belongs to the target popu<br>dual in the household was in the target popu<br>dual in the household was not in the target p<br>|0 (non-respondent).<br>at t<sup>∗</sup>(target population is the population at t<sup>∗</sup>):<br>lation);<br>lation at t<sup>∗</sup>;<br>opulation at t<sup>∗</sup>(i.e, newborn or immigrant after|
|<br> <br>Based on the var<br>the household k|<br>iables listed in t<br>may be compute<br>|<br>he above table, the base weig<br>d as:<br><br><br><br><br>|<br>ht introduced in Section 3.3.1 of<br>|
|<br> <br>|<br>wk<br> <br>|<br>= <sup>1</sup><br>v8k<br>�<br>1<br>∑<br>v6i<br>i∈k<br>��<br>1<br>v4i<br>v10i<br>i∈k<br> <br>|��.<br><br>|
|where for each i<br>that, following th<br>with the excepti|ndividual i∈k<br>e dynamics of t<br>on of certain hou|is v5i= v5k, being v5kthe h<br>he households, the household<br>seholds for which the codes d|ousehold code. Finally, we note<br>code, v5, can change over time,<br>o not change.|

18
# 4.3. Defining the minimal set of variables to be collected on non-respondents by doorstep interviews and on movers by proxy interviews

Sample size reduction must improve the accuracy of cross-sectional and longitudinal estimates. Attrition (people dropping out and people moving) can bias estimates if those who drop out or move are systematically different from those who remain in the survey. For example, suppose that people drop out when they become employed, because they no longer have time to participate in the survey. In this case, the panel will produce downward biased estimates of the number of people in employment. Panel updating, tracking, and calibration estimation address the loss of sampled units.

To reduce attrition bias, it may be helpful to collect additional information on non-respondents, which can be used as auxiliary variables in a two-step calibration approach, where the known totals at the selected sample level include variables derived from respondents along with information collected on non-respondents. For this, two calibrations will be required: the first adjusts the basic weights for panel attrition and the second calibration deals with undercoverage. As mentioned above, the two-step approach improves the accuracy of the estimates if the reasons for attrition are different from the reasons for the sample under-coverage. However, the collection of this information must be designed and implemented with great care; its utility and accuracy should be assessed by empirical studies before introducing its collection into the current statistical process.

The additional variables can be collected through a short form questionnaire module that captures demographic variables (e.g., age, gender, etc.), social variables (employment status, literacy level, etc.) and, for movers, a variable on the reason for leaving the enumeration area (e.g., economic reason, personal reason, etc.). Finally, the telephone number of the movers should be recorded to facilitate phone interviews with individuals who the survey is unable to locate.

Doorstep interview. The doorstep interview is an abbreviated version of the interview used to obtain key information on the characteristics of non-respondents, who are often related to target phenomena of interest. For example, the occurrence of literacy-related non-response is likely to be largely concentrated among migrants with low literacy in the official survey languages, and perhaps with low literacy in general. However, the extent of this concentration is currently unknown due to a lack of further information on these groups. The doorstep interview is a short and simple interview conducted with households (or individuals) who are not willing to participate in the main survey and is designed to obtain this basic information for the non-respondent. As mentioned above, demographic, literacy, and employment questions are simple questions that could be administered to non-respondents.

Proxy interview. The proxy interview can be used in two different contexts: i) if one person in the household is a mover, then another household member can answer the short questionnaire and ii) if the whole household moves, then the neighbours (or the community) can answer the short questionnaire. In both cases, obtaining the telephone numbers of the movers may allow for a telephone interview if tracking is not allowed by the tracking rules.

As with the doorstep interview, the purpose of the proxy interview is to ascertain the concentration of specific characteristics in the sample of movers. A module can be administered to the proxy respondent. Two other questions are then added to the short interview: the telephone number of the mover and the reason for moving out of the household.

19
# 4.4. Planning the multi-mode data collection

The operational costs of the survey limit the tracking rules; thus, movers who leave the sampled area (i.e. enumeration area, the parish, district, etc.) cannot be tracked in all cases. To obtain synthetic information on the variables of interest for movers and to reduce panel attrition due to movers, a cost-effective strategy is to conduct a telephone interview. In this case, a shortened questionnaire is preferable or, in more sophisticated cases, the survey design may include a CATI (Computer Assisted Telephone Interview) interview and the panel survey will rely on multi-mode data collection.

Finally, the standard questionnaire needs to ask for the telephone number to avoid nonresponse of movers in future waves.

# 5. SHORT DESCRIPTION OF THE LSMS-ISA PANEL SURVEYS

We test this methodological framework through a simulation using real data from the Living Standards Measurement Study – Integrated Surveys on Agriculture (LSMS-ISA), conducted with the support of the World Bank. The purpose of the simulation is to verify the properties and usability of the proposed estimator and to compare its estimates with those currently produced in a specific application case. The LSMS–ISA surveys are quite heterogeneous when considering the entire estimation process, including the sampling plan, the tracking protocol, the target parameters, the set of auxiliary variables, and the estimators used. For the sake of simplicity, the proposed estimator does not include all the specific issues and technicalities affecting the survey used in the simulation. Instead, we focus on the main steps of the process that are common to all longitudinal surveys to draw general conclusions that are not survey specific.

LSMS-ISA surveys are nationally representative, longitudinal household surveys designed to understand the relationship between living standards and agriculture in Africa. Since its launch in 2008, the LSMS-ISA project has worked with statistical offices in eight partner countries in Sub-Saharan Africa, providing technical assistance to design and implement multi-purpose household panel surveys with a strong focus on agriculture, while promoting innovation and efficiency in data collection methods. Table 5.1 summarizes the partner countries, the panel surveys supported, and the years of survey implementation.

Table <u>5.1 LSMS-ISA Surveys</u>

|**Country**|**Survey**|**Year**|
|---|---|---|
|**Burkina Faso**|Burkina Faso Enquête Harmonisée sur le<br>Conditions de Vie des Ménages(EHCVM)|2018/19, 2022|
|**Ethiopia**|Ethiopia Socioeconomic Survey (ERSS)|2011/2012, 2013/2014, 2015/2016,<br>2018/2019,2021/2022|
|**Malawi**|Third Integrated Household Survey (IHS3)|2010|
||Integrated Household Panel Survey (IHPS)|2013, 2016/2017, 2019/2020|
|**Mali**|Enquête Agricole de Conjoncture Intégrée<br>(EAC-I)|2014/2017|
|**Niger**|National Survey on Household Living<br>Conditions and Agriculture(ECVM/A)|2011/2014|
|**Nigeria**|General Household Survey (GHS)|2010/2011, 2012/2013, 2015/2016,<br>2018/2019|
|**Tanzania**|Tanzania National Panel Survey (TZNPS)|2008/2009, 2010/2011, 2012/2013,<br>2014/2015,2020/2021|
|**Uganda**|Uganda National Panel Survey|2009/2010, 2010/2011, 2011/2012,<br>2013/2014, 2015/2016, 2018/2019,<br>2017/2018,2019/2020|

20
In general, the LSMS-ISA longitudinal samples coincide totally or partially (i.e., as a subsample) with an existing agricultural or household sample survey. For instance, the Ethiopia Socioeconomic Survey (ESS) interviewed a subset of agricultural households from the existing Agricultural Sample Survey (AgSS), complementing its exclusively rural sample with a sample of urban EAs. The Tanzania National Panel Survey (NPS) and the Uganda National Panel Survey (UNPS) are composed of a subset of EAs drawn from household budget surveys, namely and respectively the Tanzania Household Budget Survey (THBS) and the Uganda National Household Survey (UNHS). In Malawi, the Integrated Household Panel Survey (IHPS) tracked and reinterviewed a subsample of households from the Third Integrated Household Survey (IHS3). In Nigeria, the General Household Survey-Panel (GHS-Panel) is a subsample of the GHS core cross-sectional survey. Finally, in Niger, the longitudinal study followed the entire sample of the National Survey on Household Living Conditions and Agriculture (ECVM/A).

The LSMS-ISA surveys consist of two-stage probability samples which use the general population census for their sampling frame. In most samples of the LSMS-ISA, enumeration areas (EAs) are selected as primary sampling units with probability proportional to size. A sample of households is then randomly chosen from the complete listing of households in the selected EAs. Thus, the LSMS-ISA sample constitutes a random sample of EAs, households, and individuals.

The LSMS-ISA samples are meant to be nationally representative of households and of individuals. Ideally, longitudinal studies preserve representativeness over time, indicating that the sample should represent both the current population at each survey occasion and the dynamics over time of the initial population. To maintain both types of representativeness, longitudinal surveys follow up with people interviewed in previous survey rounds and add new individuals to ensure that new members of the population such as migrants and newborns are included (Glewwe and Jacoby 2000). To this end, panel surveys establish rules to define interview targets in follow-up rounds and create specific protocols to track households and individuals that move away from their original location. Moreover, they adopt refreshed samples (i.e., rotating panel or split panel) and protocols for the inclusion in the survey of individuals not previously involved in the study.

When discussing tracking in panel surveys, Witoelar (2011) introduced the concept of interview targets and tracking targets. Interview targets are those that need to be interviewed; tracking targets are those individuals or households that should be tracked if they have moved and, if found, should be interviewed. Similarly, the author distinguished between following-up rules and tracking rules in defining the criteria for individuals or households to be interviewed depending on whether they are found in the same location or are movers. To some extent, the LSMS-ISA panel surveys can be seen as a panel of both households and individuals. All households and individuals found and interviewed during the baseline are interview targets across the entire history of the panel sample.

The LSMS-ISA tracking approaches are more heterogeneous and mainly diverge in terms of: (i) targeted tracking units, (ii) characteristics of the tracking targets, and (iii) geographical scope of the tracking. Decisions across these three elements will impact representativeness and cost.

The LSMS-ISA are heterogenous in terms of the unit targeted for tracking and reinterviewing. Usually, these panel surveys have tracked dwellings, households, or individuals, with each option having its own distinct implications. In 2022, the Burkina Faso EHCVM revisited the dwellings previously visited in 2018/19. Nigeria and Ethiopia track and reinterview those households participating in the first wave of data collection but not individuals that split from the household. The Malawi IHPS, Tanzania NPS, and Uganda UNPS track individuals.

21
Tracking dwellings instead of households is a simpler option. Following up with households over time requires defining which household is the successor of the original household, which is not a trivial task, as households may split or regroup over time. As noted by Witoelar (2011) household membership and composition quickly change, household structure and boundaries are often unclear, and the definition of household often varies according to the objective of the survey. Continuity rules to define which household is regarded to be the continuation of the one observed during the previous survey are proposed and discussed in Falorsi (2009).

The LSMS-ISA panel surveys have established practical rules to define which household to follow. The Ethiopia ESS, for instance, identifies the group of household members still living in the original dwelling as the original household. If the household moved away as a whole, it is tracked and re-interviewed if it is still residing within the Ethiopian border. If all household members left the country, the unit is dropped from the sample. If the household split, the portion of the household that moved with the member identified as the head of the household in the previous wave is tracked and interviewed. Similarly, the Nigeria GHS-Panel interviews household members found in the original dwelling and tracks households that moved outside of the dwelling but are still located in Nigeria. If the household splits, the original household is defined as the group of household members still residing with the head of the household from the previous round. If the original household head is not found, the original household is defined as the one comprising the largest number of original household members.

Panels of dwellings and households have the disadvantage that members who moved out of their baseline household are not part of the sample anymore, despite still being part of the population. The sample of the remaining members therefore will be less representative of the population at the present time as well as of the population evolving over time. However, as we will detail later in the paper, this lack of representativeness is mitigated because all LSMS-ISA surveys incorporate new individuals who have joined interviewed (and tracked) households, such as newborns or immigrants.

The loss of representativeness over time can be addressed by tracking original individuals. The extent to which individual tracking can mitigate representativeness issues depends on decisions regarding the characteristics of the tracking targets and the geographical scope of the tracking. Conceptually, deciding which individuals to track depends on the characteristics of the population that need to be represented over time. Aspirations of studies that focus on a specific subpopulation are different from LSMS-ISA panel surveys, which intend to describe the national population.

However, tracking all individual movers can be very expensive. The more ambitious the tracking goals (e.g., the greater the distance away from their original home that an individual must be tracked), the more that costs will increase. Generally, LSMS-ISA individual tracking targets are defined based on the individual’s age as well as on their relationship to the household head. Malawi, Tanzania, and Uganda track and follow up with movers that are biologically related to the household head, excluding therefore servants or guests at the time of the previous survey. Moreover, all three surveys impose a lower bound under which individuals moving out of the households are not tracked (12 years old in Malawi and 15 years old in Tanzania and Uganda).

The geographical scope of the tracking may also impact the representativeness of the panel survey in future rounds, if those individuals or households that move the farthest are also very different from others. How far to track is generally established based on administrative boundaries, geographic boundaries, or distance (Witoelar, 2011). Panel surveys that rely on local tracking only search for households in the same EAs or nearby. Distant tracking allows for tracking movers far away the original dwelling. Regardless of whether they track households 22
or individuals, all LSMS-ISA surveys use distant tracking to track and interview movers out of the original EA, as long as they still live within the country.

In the Malawi, Tanzania, and Uganda panel surveys, once the individual forms a new household, all household members residing with the individual enter the sample and become tracking targets. Integrating the joint household into the sample accounts for new populations who may be otherwise underrepresented and captures the evolving dynamics of the population at large. However, including the mover’s new household increases data collection costs. An alternative and less costly way to enhance representativeness in longitudinal studies are the adoption of rotating panels or sample refreshes, which have been used by some LSMS-ISA surveys. For instance, the Uganda UNPS 2013/14 has rotated out and replaced one-third of the EAs from its initial sample. The Tanzania NPS sample was refreshed in 2014/15, but the subsequent 2020/21 round of NPS data collection only follows up with this refresh sample, marking a discontinuity in the longitudinal representativeness of the population from initial rounds. In 2022, the Ethiopia ESS sample was fully refreshed and the Burkina Faso Enquête Harmonisée sur le Conditions de Vie des Ménages (EHCVM) followed up with its 2018/19 households, complementing the sample with a set of new EAs selected from a more recent sampling frame. Finally, Nigeria also partially refreshed its survey sample in 2018/19.

# 6. EMPIRICAL EVALUATION OF THE UGANDA DATA

As a case study, we apply the estimator methods discussed above to the Uganda National Panel Survey (UNPS). The UNPS is a multi-purpose household panel survey implemented by the Uganda National Bureau of Statistics (UBOS) with the technical support of the World Bank LSMS-ISA project. Started in 2009/10 as a follow up of the Uganda National Household Survey 2005/6, its primary aim is to inform policy makers in budgetary decisions and policy interventions and to monitor major national policies and programs. Implemented on an annual basis, the UNPS also provides representative information on income dynamics at the household level and consumption expenditure estimates to monitor poverty during intervals between other national survey efforts. Moreover, the UNPS collects high-quality agricultural data integrated within a multi-topic framework, which allows for understanding the linkages between agriculture, welfare, and other sociodemographic characteristics.

Since its launch in 2009, the UNPS conducted 8 waves of data collection on a sample of approximately 3,000 households. The UNPS 2009/10 was followed by additional rounds in 2010/11, 2011/12, 2013/14, 2015/16, 2017/18, 2018/19, and 2019/20. Its sample is representative at the national, urban-rural, and regional levels. This study uses data from the UNPS 2009/10 (wave 1), the UNPS 2013/14 (wave 4), and the UNPS 2015/16 (wave 5). The UNPS 2009/10 comprises a sample of 2,975 households, the UNPS 2013/14 counts 3,118 households, and the UNPS 2015/16 interviewed 3,304 households. After data cleaning and data preparation for the analysis, our final dataset comprises 18,313 individuals from wave 1, 17,377 individuals from wave 4, and 15,905 individuals from wave 5.

Figure 6.1 shows the change in status of the surveyed individuals across the 3 panel waves of data collection used in this work. The 2009/10 level represents the 18,313 individuals interviewed in the first wave. Of these, 45% (8,211 individuals) were found and reinterviewed in 2013/14, 15% were not found, and 40% were rotated out. The sample refresher of 2013/14 brought in 5,387 new individuals, or around 30% of the original sample. New individuals in the non-mover and split-off households accounted for 3,779 new units, equal to the 21 % of the original sample. Between 2013/14 and 2015/16, t31% of the 2013/14 wave sample was lost, while 85% was found and reinterviewed. The figure shows the dynamics across years of the old and refresher samples as well. For instance, we can see that 87% of the refresher sample was 23
<!-- Start of picture text --> e<br>gr.y<br>£{ q a<br>5 } =<br>x oa<br>— y~ z<br>oO ae ><br>Pe<br>5<br>a<br>2009/2010 G OTs 2015/2016<br><!-- End of picture text -->
and became split-offs, and iii) individuals that moved out their original household to form new split-off households. Tracking rules have changed over the course of the panel rounds for all three tracking targets mentioned above.

In the first three waves, the UNPS tracked all the original households that moved away from their 2005/06 original location to any other location, although not all households of the initial sample were targeted for individual tracking in cases where any of their members moved out of the household. Individuals and the split-off households they formed were only tracked for a subsample of the original households, namely 2 households randomly selected per EA (20 percent of the original household sample). This individual tracking was meant to compensate for the losses to the sample due to attrition. This 20 percent of households from which the individuals were tracked were referred to as “split-offs tracking targets”. Moreover, of the splitoffs tracking targets, only individuals 15 years and above who were biologically related to the household head were tracked. These individuals and split-off households were found and interviewed, even if they moved beyond their original EA/parish. Once interviewed, the splitoff individuals and all members of their new household became part of the UNPS sample and were interviewed and tracked in all subsequent UNPS waves.

Starting in UNPS 2013/14, the scope of the split-off target tracking was expanded to include all households that were part of the sample as long as they were still within Uganda, regardless of distance from the original household location and whether they were original or split-offs. In those households, only members older than 15 years and identified in the previous wave as the head, the spouse, or the child of the household head were eligible for tracking. Other members were interviewed only if they were living with one of these core members. This means that if no core member is found in the last known location, the household was not interviewed even if other previous household members still lived there.

In wave 4, the sample of the UNPS was refreshed. One-third of the original sampled households were rotated out as part of the panel refresh and were no longer tracked or interviewed.

In the current UNPS setting, tracking individuals requires the completion of an individual tracking form which contains all contact information for the split-offs and/or the individual movers. The information on their new location needed for the full tracking is generally gathered from their previous household members or any other knowledgeable person. For each core member that had moved away, a tracking form is completed. Based on the information filled in this form, the mover individuals are contacted and interviewed.

Although the tracking target sample comprises only the core members of each household, all persons living with these core members are interviewed and become part of the UNPS sample. Finally, if these individuals are core members of the new split-off household, they are interviewed in the subsequent waves of the UNPS, even if they move to different locations.

# 6.1. Empirical evaluation of the cross-sectional estimates

We computed the UNPS 2015/16 estimates. Data from the previous waves are included in the analysis to identify the household dynamics (e.g., movers, immigrants, and newborns).

We calibrated the base weights (expressed in Formula 3.8) to the known sex by age class population totals using UBOS official projections for 2015 and based on the population census of 2014. The projections used to calibrate the weights are presented in Table 6.1. Hereafter, we denote these weights as calibrated GWSM base weights.

We applied the calibrated GWSM base weights to a set of variables from UNPS 2015/16 data and compared them to Uganda official statistics to assess the functioning of the weights vis-à25
vis the UNPS 2015/16 sampling weights. The base weights are those defined in Section 3 at the household level. The UNPS sampling weights are those defined at the individual level and are currently used for the survey estimates by UBOS (2011). They are obtained according to a complex process, based on performing preliminary adjustments for non-response based on noresponse propensity scores. Specifically, the final individual-level weights are obtained through the following six methodological steps: i) definition of the base weights as the reciprocal of the probability of inclusion, ii) calculation of the fair share correction factor, iii) pooling of the weights defined in step 3, iv) computation of the correction factors for attrition, v) trimming of the weights defined up to step 4, and vi) post-stratification.

Table 6.1. Population projection by age and gender (2015)

|Age group|Male|Female|
|---|---|---|
|0-4|3,220,300|3,028,800|
|5-9|2,870,800|2,730,500|
|10-14|2,528,000|2,500,100|
|15-19|2,008,700|2,097,500|
|20-24|1,509,200|1,790,100|
|25-29|1,181,200|1,403,800|
|30-34|938,700|1,082,800|
|35-39|743,900|836,500|
|40-44|630,600|677,500|
|45-49|470,800|486,800|
|50-54|382,300|443,200|
|55-59|241,700|277,900|
|60-64|194,600|241,600|
|65-69|140,400|171,100|
|70-74|114,900|158,600|
|75-79|71,500|90,700|
|80+|96,400|140,600|
|Total|17,344,000|18,158,100|

- Note: UBOS official projections for 2015 and based on the population census of 2014 available at <u>https://www.ubos.org/wp content/uploads/statistics/Population_Projections_2018.xlsx, downloaded June 28, 2021.</u>

We use published data and reports from the National Population and Housing Census (NPHC) 2014 (UBOS 2016; UBOS 2017) and the Uganda National Household Survey (UNHS) 2016/17 as main sources of official statistics. The former is conducted by UBOS about every 10 years with the aim of collecting benchmark demographic and socio-economic data of the Uganda population. The latter is the sixth follow-up survey of the UNHS, a cross-sectional survey implemented by UBOS starting in 1999/20, which aims to collect data on demographic and socioeconomic characteristics, with a sample of 15,636 households. The UNPS is a follow-up to its 2005/06 survey and largely implements the same methodology.

The NPHC 2014 and UNHS 2016/17 are the official source of data with the closest collection period to the UNPS 2015/16 we found. The reference period for the UNPS 2015/16 is March 26
2015 to March 2016, whereas data collection for NPHC 2014 was pursued in August/September and the UNHS 2016/17 data collection was implemented between June 2016 and June 2017.

Table 6.2 presents the comparison estimate indicators on individual and household characteristics at the national level. The first three columns show the indicators from the UNPS 2015/16 without using sampling weights (unweighted estimator), using the original UNPS weights (current UNPS estimator), and using the calibrated GWSM base weights (calibrated GWSM base estimator). The column of unweighted estimates is helpful because it indicates the different effects by which various weighting strategies act on the final estimates, thus changing the unweighted estimates.

In the last two columns, the table presents the official statistics and their source. In general terms, estimates are more accurate using the calibrated GWSM base estimator rather than the current UNPS estimator for indicators at the individual level. The share of female population and the share of children below 18 years old weighted using the calibrated base GWSM weights closely approximate the official statistics. This result is expected as the weights are calibrated using the population totals projection based on the 2014 census that we are using as a benchmark.

Regardless of the weight we apply, the UNPS 2015/16 overestimates the official “true” value of the literate population above 10 years. Although the three estimates are consistent with each other, the differences reported in the table are statistically significant because they are based on a sample of about 16,000 individuals. For such a sample size, considering a deft value(Kish, 1966) ranging from 2 to 3 (implying the deff value ranges from 4 to 9), the sampling standard deviation of a proportion varies around values between 0.007 and 0.010.

The unweighted statistic, which has a much higher value (80.7%) than the official statistics (72.2%), suggests that the non-response affects the non-literate population. For such reason, this subpopulation is under-represented in the panel. However, the observed difference could be an issue of comparability in how this information was asked in the questionnaires. The different weighting systems seem unable to deal with this problem. Finally, the calibrated GWSM base weight estimator appears to be more accurate for two key indicators of the survey, namely the working population and the poverty head count at the national level.

At the household level, the current UNPS and the calibrated GWSM base estimators seem more comparable, even though the former produces more accurate statistics, particularly for the variables on share of urban households and share of agricultural households. In a few cases – namely, access to electricity and ownership of the dwelling – the UNPS 2019/20 produces inaccurate estimates of the official statistics, regardless of the weight used.

27
Table 6.2. Individual and household estimates by the unweighted, current UNPS and <u>calibrated GWSM base estimators</u>

||Unweighted<br>|UNPS<br>|Calibrated<br>GWSM<br>base<br>|Offi<br>|cial Statistics<br>|
|---|---|---|---|---|---|
||Mean|Mean|Mean|Mean|Source|
|Individuals||||||
|Female|51.1|50.8|51.1|51.0|Census (2014)|
|% children below 18 years old|51.2|52.1|55.0|55.0|Census (2014)|
|Literate population (+10 yrs)|80.7|80.8|81.7|72.2|Census (2014)|
|Working population|76.8|76.1|80.1|78.8|UNHS (2016)|
|Poverty headcount|20.4|19.3|19.9|21.4|UNHS (2016)|
|Households||||||
|Share of urban households|25.3|25.1|22.9|25.0|Census (2014)|
|Size of the household|4.8|5.0|4.8|4.7|Census (2014)|
|Share of agricultural households|78.7|80.1|79.4|80.0|Census (2014)|
|Number<br>of<br>rooms<br>in<br>the<br>household|2.2|2.3|2.1|2.4|UNHS (2016)|
|Household owns a cellphone|74.0|72.9|75.4|74.3|UNHS (2016)|
|Household<br>has<br>access<br>to<br>electricity|15.4|15.1|15.6|21.0|Census (2014)|
|Household has access to safe<br>water|75.2|72.2|71.7|72.0|Census (2014)|
|Household owns the dwelling|81.9|82.9|81.1|71.8|UNHS (2016)|
|House has brick walls|67.2|63.8|64.2|66.6|UNHS(2016)|

Table 6.3 breaks down the share of working population by area of residence. The calibrated GWSM base estimator (82.0%) seems closer to the official statistics than the current UNPS estimates (78.8%) in representing the official statistics in rural areas (82.6%). As indicated earlier, these differences, although small, are statistically significant. More controversial are the estimates in the urban areas, where we do not see a best estimator.

<u>Table 6.3. Employed population by area of residence</u>

||Unweighted|UNPS|Calibrated GWSM<br>base|<br>Off|icial Statis|tics|
|---|---|---|---|---|---|---|
||Mean|Mean|Mean|Mean|Year|Source|
|Rural|79.6|78.8|82.0|82.6|2016|UNHS|
|Urban|67.5|67.4|72.7|69.0|2016|UNHS|

Having concluded the analysis of statistics at the national level, we now turn to the study of poverty by area of residence (Table 6.4), which does not show concrete differences among the estimators. The incidence of poverty is much higher in the rural areas than urban areas.

- <u>Table 6.4. Poverty head count ratio by area of residence</u>

|Unweighted|UNPS|Calibrated|Official Statistics|
|---|---|---|---|
|||GWSM base||

28
||Mean|Mean|Mean|Mean|Year|Source|
|---|---|---|---|---|---|---|
|Rural|23.8|22.2|22.3|25.0|2016|UNHS|
|Urban|9.0|9.6|10.2|9.6|2016|UNHS|

# 6.2. Empirical evaluation of the longitudinal estimates

Panel surveys produce net change and gross change estimates (Section 2.2.2). Gross change offers valuable insights into net change, so it is important that the cross-sectional estimates achieved by transition matrices are consistent with the estimates obtained with the overall UNPS 2015/16 sample.

Table 6.5 shows the dynamics of employment across the 2009/ 2015 and 2013/2015 panels. In particular, the first three columns report the employment status in 2015 with respect to the status in 2009. Only individuals present in both 2009 and 2015 are represented in the table. The last three columns show the same statistics for those individuals in 2015 who also responded to the survey in 2013.

Focusing on the 2009/2015 estimates, we note that all estimators produce upward estimates with respect to the estimates presented in table 6.2. This result suggests that the individuals who left across the years – characterizing most of the attrition – were largely unemployed. The three estimators indicated that the panel comprised 78.6%, 77.6%, and 81.4% employed persons. The refresh operation rebalanced the panel. Focusing on the cross-sectional estimates given by the 2013/2015 panel, 77.4% of individuals were employed according to the unweighted estimator, 76.7% for the current UNPS estimator, and 80.8% for the calibrated GWSM base weight estimator.

We can state that the last estimator produces more stable and consistent estimates when considering different sub-samples, i.e., the 2009/2015 and the 2013/2015 panel and the entire sample observed in the 2015 with a percentage relative variance (ratio of standard error to mean) among the the three estimates equal to 0.8%, The UNPS estimator has 1.0% of relative error and the unweighted estimator 1.2% of standard error.

Table 6.5. Employment transition matrix estimates by unweighted, current UNPS, and calibrated GWSM base estimators

||Unweighted|2009/1<br>UNPS|5<br>Calibrated<br>GWSM base|2<br>Unweighted|013/15<br>UNPS|Calibrated<br>GWSM<br>base|
|---|---|---|---|---|---|---|
|Unemployed in|||||||
|2009 (or 2013)<br>and 2015|5.1|5.4|3.7|11.6|12.1|9.3|
|Unemployed in<br>2009 (or 2013) and<br>employed in 2015|6.0|5.6|5.8|8.5|8.1|7.6|
|Employed in 2009|||||||
|(or 2013) and 2015|72.6|72.0|75.6|68.9|68.6|73.2|
|Employed in 2009<br>(or 2013) and<br>unemployed in<br>2015|16.3|16.9|14.9|11.0|11.2|9.9|

29
The breakdown of employment dynamics in the 2009/2015 and 2013/2015 panels by area of residence as seen in Table 6.6 below demonstrates that panel attrition is concentrated on the unemployed in urban areas. Consistently with what we see in table 6.3 the Calibrated GWSM are more accurate than the UNPS panel weights in estimating employment in rural area in the 2009/2015 sample. Results for the the 2013/15 panel show that the refresher has improved accurancy. In urban area, all methods gave upward estimates and in particular for the 2009/15 panel sample. The findings indicate that individuals leaving the UNPS are mostly unemployed in urban areas, suggesting the need to improve tracking rules to reduce the potential bias of the estimates.

Table 6.6. Employment by area of residence transition matrix estimates by the unweighted, current UNPS, and calibrated GWSM base estimators

||||2009/15|||2013/15||
|---|---|---|---|---|---|---|---|
|||Unweighted|UNPS|Calibrated<br>GWSM<br>base|Unweighted|UNPS|Calibrated<br>GWSM<br>base|
|Unemployed in 2009|Rural|3.8|4.2|3.1|8.9|9.2|7.4|
|(or 2013) and 2015|Urban|9.5|9.7|6.9|20.3|21.3|16.8|
|Unemployed in 2009<br>(or 2013) and<br>employed in 2015|Rural<br>Urban|4.4<br>11.9|4.0<br>11.3|4.3<br>13.3|8.1<br>10.0|7.7<br>9.4|7.1<br>9.7|
|Employed in 2009 (or|Rural|75.4|74.6|77.3|71.8|71.3|75.2|
|2013) and 2015|Urban|62.5|63.1|66.8|59.5|59.9|65.5|
|Employed in 2009 (or|Rural|16.4|17.3|15.3|11.3|11.8|10.4|
|2013) and<br>unemployedin 2015|Urban|16.1|15.8|13.0|10.1|9.3|8.1|

We carried out a similar analysis for the poverty headcount ratio (table 6.7). Estimates from the 2009/2015 panel are all around the official statistics: we have respectively 21.9%, 21.1%, and 21.4% for the unweighted, UNPS and the calibrated GWSM base estimator. The refresh operation (2013/2015 sample) reduced the frequencies to 20.5%, 19.3%, and 20.0%. When we compare these estimates with the official statistics (21.4%), it does not appear that one estimator surpasses the others in terms of quality. Nevertheless, comparing the percent relative variance among the three estimates of the poverty headcount ratio, two from the longitudinal estimates and one from the cross-sectional estimator, we note that the UNPS estimators has 5.2% percent relative variance while the calibrated GWSM estimator has 4.1% of percent relative variance. The unweighted estimator has 4.0% of percent relative variance.

30
Table 6.7. Poverty rate transition matrix estimates by the unweighted, current UNPS, and calibrated GWSM base estimators

||Unweighted|2009/15<br>UNPS|<br>Calibrated<br>GWSM base|2<br>Unweighte<br>d|013/15<br>UNPS|Calibrate<br>d GWSM<br>base|
|---|---|---|---|---|---|---|
|In non-poor HH in 2009|||||||
|(or 2013) and 2015|63.1|64.8|63.6|65.0|66.5|65.4|
|In non-poor HH in 2009|||||||
|(or 2013) and poor HH<br>2015|12.3|12.1|13.0|8.3|8.1|8.3|
|In poor HH in 2009 (or<br>2013) and poor HH<br>2015|9.6|9.0|8.4|12.2|11.2|11.7|
|In poor HH in 2009 (or|||||||
|2013) and non-poor HH<br>2015|15.0|14.2|15.1|14.4|14.1|14.5|

Table 6.8 shows the transition matrix for the area of residence, for which the 2009/2015 panel estimates are greater than 2013/2015 estimates. The overestimations of poor people suggest that non-poor individuals, especially in rural areas, are more dynamic and tend to move out of the UNPS sample more frequently than poor individuals. In the 20013/2015 panels, figures stabilize. All statistics are consistent with the national-level estimates from the UNPS 2015 cross-sectional sample. The refresh of the sample in 2013/14 seems to have mitigated the attrition of non-poor individuals. Rural and urban areas of residence show the same trends. Comparing the percent relative variance among the three estimates of the poverty headcount ratio, two from the longitudinal estimates and one from the cross-sectional estimator, we note that for the rural sub-population, the UNPS estimators has 5.3% of percent relative variance whilethe calibrated GWSM estimator has 3.5%. The estimates for the urban sub-population show the UNPS estimators has 0.6% percent relative variance, the calibrated GWSM estimator has 3.0%, while the unweighted has 5.4%. So in this case we are not able to indicate which one is the more stable estimator.

31
# Table 6.8. Poverty rate by area of residence transition matrix estimates by the unweighted, current UNPS, and calibrated GWSM base estimators

||||2009/15|Calibrated||2013/15|Calibrated|
|---|---|---|---|---|---|---|---|
|||Unweighted|UNPS|GWSM<br>base|Unweighted|UNPS|GWSM<br>base|
|In non-poor HH in 2009|Rural|58.1|59.4|59.9|59.4|61.4|61.0|
|(or 2013) and 2015|Urban|81.1|83.5|81.6|84.2|83.8|83.4|
|In non-poor HH in 2009<br>|Rural|13.8|13.7|14.0|9.2|8.8|9.1|
|(or 2013) and in poor<br>HH in 2015|Urban|7.0|6.5|7.7|5.5|5.6|5.0|
|In poor HH in 2009 (or|Rural|11.4|10.6|9.7|14.7|13.4|13.2|
|2013) and 2015|Urban|3.0|3.1|2.1|3.8|4.1|5.4|
|In poor HH in 2009 (or<br>|Rural|16.7|16.3|16.3|16.7|16.4|16.6|
|2013) and non-poor HH<br>in 2015|Urban|8.8|6.9|8.6|6.5|6.5|6.2|

# 6.3. Summary of the empirical results

The comparison of the estimators has been carried out upon two levels: their statistical properties and their practical implementation. As far their statistical properties, section 4 introduced the properties of the new calibrated estimator and section 6 provided concrete results on real data. Given the complexities involved in claiming the superiority of one estimator to another, we used benchmarking statistics to conduct this evaluation. The results are summarized as follows:

- The calibrated GWSM base estimator seems to produce more accurate individual-level statistics than the current UNPS estimator.

- The two estimators produce equally accurate statistics at the household level, suggesting that the GWSM base estimator can be further improved by considering household known totals.

- While the cross-sectional estimates based on the transition matrix generally do not reproduce the exact cross-sectional estimates based on the entire UNPS sample, the sample refresh operation mitigates this problem. This suggests that both estimators can only partially deal with the dynamic nature of the population, making it important to i) coherently implement a periodic sample refresh (i.e., rotate the panel) and ii) improve the tracking rules, as panel attrition affects specific subpopulations of interest.

- The calibrated GWSM base cross-sectional estimates on the transition matrix appear generally more stable when changing the sample (i.e., 2009/2015 or 2013/2015) than the current UPNS estimates.

- While the scope of the simulation has been to obtain general conclusions, as far as the practical implementation level is concerned, considering the Uganda case to be concrete, we can give the following indication, defined to change the current process as little as possible: - 1 3 5. The 2 is not explicitly stored, since a non-respondent in the original sample (wave

- 1) is not clearly distinguished from a new individual that was not included in the original According to Table 5.1, Uganda panel data stores the variables, vv , vv and  vv sample.  variable  vv

- - 6 , using other stored variables and making some accurate assumptions. The variable should be directly asked We artificially created the population membership variable, vv

32
- in the new version of the questionnaire (i.e., “Given that the household has at least one component belonged to the original sample selected from a population, does the individual belong to the same population or are they new (newborn or immigrant)?”).

- - 7, using other stored variables and making some accurate assumptions. This variable should be directly asked in the We artificially created the individual multiplicity factor, vvnew version of the questionnaire (i.e., “Does the individual of the original sample change parish from 2009 to 2013 (years of the original samples?)”).

- - 7 8. - 9 10, using Based on the values of vv , we computed the household multiplicity factor vv other stored variables and making some accurate assumptions.

- - We artificially rebuilt the response indicator variable in each wave, vv4, for each individual selected in the original  and vv sample (not for the incorporated individuals), even if the individual did not respond We stored the inclusion probability, vv(i.e., in the current panel data, the weight is missing).

- We computed the GWSM weight that does not rely on the concept of the parent household, which can be a complex operation in practice.

- We computed the final weights using the standard calibration estimator and applied these for the cross-sectional and longitudinal estimates, creating a unique vector of sampling weights that simplifies the coherence of the individual and household estimates.

# 7. CONCLUSIONS

Given the complexity of the topics covered, we now retrace our main steps to provide a unified picture of the various results. In this paper, we explored the problem of enhancing the quality of cross-sectional and longitudinal estimates from surveys that are obtained by combining samples chosen at previous times with given rules to track panel members as they move away from the location of their initial survey interview.

We have seen that the issue is complicated and that action needs to be taken on several fronts, which refer to related but different scientific fields.

The first aspect is defining the quantities one wants to know, considering the nature of the population, which has both a cross-sectional and longitudinal aspects. That leads to appropriate consideration of the definition of the statistical units of interest over time and at the various points at which attention is focused for constructing cross-sectional and longitudinal estimates. While the solutions for simple statistical units (such as individuals) are relatively uncontroversial, for composite units (such as households), there are multiple possible solutions, and each solution has pros and cons in terms of being representative of what is happening in the reality of natural populations and of the observational ease of the phenomena of interest. We proposed the many-to-many approach for defining the continuity of a household over time, meaning that one household from an initial time may generate many families at the next time. Conversely, a household at the current time may be derived from several households from previous times. With that approach, there is a perfect correspondence between the longitudinal population of people and that of households. Moreover, the many-to-many approach includes the one-to-one continuity rule as a particular case.

Next, we considered the observational processes that make theoretically defined population quantities observable. We have seen that it is necessary to take into due consideration the timeline of repeated observations to avoid distortions that derive from considering the same 33
observations several times or, on the contrary, failing to represent in any way some subpopulations that may have a particular interest from the economic and social point of view, such as the movers or newborns or those who enter the population at a given moment.

Moving now to the estimation methodology, we have seen how the GWSME estimator offers a method that allows us to provide unbiased estimates and is simple to implement. The calibration of the estimates with the correction of the weights at the household level presents several theoretical and applicative advantages. The variance estimation has a well-established asymptotic behavior and is far more transparent and well-founded. Because the estimation is based on a small number of adjustment stages, the process flow is simpler. Lastly, the approach greatly improves the consistency between estimates created with household-level variables and those utilizing individual-level definitions of the variables.

Finally, the last aspect examined concerns data collection and the different strategies that can be implemented to mitigate the bias resulting from non-response and panel attrition.

In conclusion, we used data from the Uganda National Panel Survey (UNPS) to experimentally evaluate the suggested technique. In summary, we found that the calibrated GWSM base estimator yields individual-level statistics that appear to be more accurate than those produced by the current UNPS estimator. Additionally, the calibrated GWSM base cross-sectional estimates on the transition matrix show a generally higher degree of stability when the sample is changed compared to the current UNPS estimates.

We stress that the above results are obtained using less auxiliary information of the current UNPS estimator, both because some of this information is not clearly documented and because one of the objectives is to propose a simplified estimator with respect to the current one, using fewer computational steps and based on a robust sampling theory well known in the scientific literature.

# REFERENCES

Deville, J. C. and Särndal C.E. (1992). Calibration Estimator in Survey Sampling. Journal of American Statistical Association. Vol.87. N° 418.

FAO (2015). Integrated Survey Framework, Guidelines, Rome. <u>http://www.gsars.org/wpcontent/uploads/2015/05/ISF-Guidelines_12_05_2015-WEB.pdf. Accessed on 16 December</u> 2015.

FAO (2014). The Global Strategy to Improve Agricultural and Rural Statistics. Technical Report on the Integrated Survey Framework, Technical Report Series GO-02-2014, <u>http://gsars.org/wp-content/uploads/2014/07/Technical_report_on-ISF-Final.pdf. Accessed</u> on 1 December 2014.

Groves, R.M., Don A. Dillman, J. L. Eltinge, R- J. A. Little . (2001). Survey No-Response, Wiley. New York.

Falorsi P. D., P. Righi, P. Lavallée . (2019). Optimal Sampling for the Integrated Observation of Different Populations. Survey methodology, Vol. 45, No. 3, pp. 485-511. Statistics Canada, Catalogue No. 12-001-X.

Falorsi, F, Russo A. (2009) . Popolazioni e parametri di interesse nelle indagini ripetute nel tempo. In Alleva G., Falorsi P. Indicatori e modelli statistici per la valutazione degli squilibri territoriali. Franco Angeli, Milano. ISBN: 9788856812077.

34
Glewwe, P. and Jacoby, H. (2000). “Recommendations for collecting panel data”, in M. Grosh and P. Glewwe (eds.), Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 years of the Living Standards Measurement Study. Washington D.C., World Bank.

Horvitz, D.G. and D.L. Thompson. (1952). A generalisation of sampling without replacement from finite-universe. J. Amer. Statist. Assoc., 47,663-685.

Kish, L. (1965), Survey Sampling, New York: Wiley. ISBN 978-0471109495.

Lavallée, P. ( 2007). Indirect Sampling. Springer: Ottawa.

Lynn, P., (2009), Methodology of Panel Surveys, Wyley. Wiley and Sons, Ltd, Publication.

Mecatti, F. ( 2007). A single frame multiplicity estimator for multiple frame surveys. Survey Methodology, 33(2): 151-157.

Narain, R.D. 1951. On sampling without replacement with varying probabilities. J. Ind. Soc. Agril. Statist., 3,169-174.

Särndal C.E., and Lundström S. (2015). Estimation in Surveys with Nonresponse. Wiley. New York.

Singh, A.C. & Mecatti, F. (2011). Generalized Multiplicity-Adjusted Horvitz-Thompson Estimation as a Unified Approach to Multiple Frame Surveys. Journal of Official Statistics, 27(4): 633–650.

Uganda Bureau of Statistics (2011), The Uganda National Panel Survey (UNPS) 2009/10 – Main Report, Kampala, Uganda.

Uganda Bureau of Statistics (2017), The National Population and Housing Census 2014 – Education in the Thematic Report Series, Kampala, Uganda.

Uganda Bureau of Statistics (2016), The National Population and Housing Census 2014 – Main Report, Kampala, Uganda.

University of Essex . (2016), British Household Panel Survey, <u>https://www.iser.essex.ac.uk/bhps</u> <mark>Witoelar, Firman. (2011). T</mark> racking in Longitudinal Household Surveys. <mark>World Bank’s Living Standards Measurement Study-Integrated Surveys on Agriculture (LSMS-ISA). World Bank.</mark> World Bank . (2020), Basic Information Document - The Uganda National Panel Survey (UNPS) - - - 2015/2016. <u>http://surveys.worldbank.org/lsms/programs/integrated surveys agriculture</u> - <u>ISA/uganda#bootstrap panel.</u> Zardetto, D . (2015). ReGenesees: An advanced R system for calibration, estimation and sampling error assessment in complex sample surveys. Journal of Official Statistics, 31 (2), 177.

35