# 7.3 Lessons learned for improving impact evaluation design

The implementation of this impact evaluation raised methodological and practical lessons. First, it is essential to define and maintain a representative control group to carry out a rigorous evaluation of a project or program. In the present exercise, household surveys have been carried out only where civil works were planned, for the reasons mentioned above. The retrofit pipeline comparison only worked partially, resulting in an unbalanced sample size between the treatment and control groups. In Jalapão and the Northeast, the control group had only 60 households, far fewer than the treatment group composed of 360. Maintaining the comparability between the treatment and control groups is also essential to avoid bias in the comparison. At the design stage, each treatment community should be paired with one control community located within the same municipality and sharing similar characteristics, for the sake of comparability. Important loopholes may appear in the course of the evaluation if, as is likely to occur in such exercises over the long term, the control groups end up benefiting from road improvements. In addition, it is important to collect as many observables as possible in an effort to control the potential heterogeneity of the two groups. In the case of Tocantins, the data clearly show significant differences in household characteristics between the treatment and control groups, mainly because the sample of the control group was limited to one of the two regions in both the first and second groups. In the present case, a matching DID method was used to control differences in characteristics between groups.

It is similarly important to ensure comparability between baseline and follow-up surveys. In theory, identical households or communities should have been interviewed before and after interventions to develop panel data. In the present evaluation, the follow-up surveys could not be implemented in the same households because the location of the households interviewed for the baseline was not documented. As a result, the evaluation is not based on panel data. This is considered a significant design flaw because it complicates intertemporal comparability. Improvements in future impact evaluations may be achieved by recording household identification and/or geo-references of surveyed households.

The timing of the follow-up survey(s) is also a critical aspect: although direct and physical outcomes, such as travel time to the nearest hospital, can be observed immediately after project completion, it may take some time to observe other indirect socioeconomic impacts, such as job creation and health outcomes. It would be useful to carry out multiple follow-up surveys over the long run, although longer-term evaluations also run the risk of losing part of the control group for the reasons mentioned above.

Coordination among the managers of the works and the team in charge of impact evaluation is essential to ensure appropriate timing and location of the surveys. In fact, the timing of the works must often adapt to particular field constraints (delays due to administrative procedures or weather conditions, relocation of works, etc.). In the present evaluation, even basic information, such as the works’ location and schedule, have not always been made available on time.

The questionnaires used for the interviews must be extremely well conceived and customized to the project’s particular design and context. For example, although questions about the transport distance or time traveled for a particular activity seemed pertinent a priori, with hindsight the questionnaire should rather have focused on the number of impediments to travel due to inclement weather over the past 6 to 12 months (to cover seasonality). This type of question would have been more effective in view of achieving spot improvement works on rural roads, thus allowing for year-round transport. Moreover, in retrospect, interviewees offered very subjective responses when asked about distances. Other questions, such as those related to illness, seem to have little bearing on the project. Alternatively, a question about the frequency of foregone medical care due to transport problems would have been more pertinent.

30