Central Bank Business Surveys: Version 2.0 Speech by Chiara Scotti Deputy Governor of Banca d’Italia 14th Annual Conference on ‘Central Bank Business Survey and Liaison Programs’ Centro Convegni Carlo Azeglio Ciampi Rome, 30 October 2024 Ladies and gentlemen, good afternoon. The annual conference on ‘Central Bank Business Survey and Liaison Programs’ is now coming to a close. Let me warmly thank all the participants and speakers who have contributed to the success of this initiative, jointly organized by Banca d’Italia1 and the European Central Bank. The large number of central banks on both sides of the Atlantic participating in this conference reveals the importance of business surveys for policymakers. The usefulness of firm-level surveys has become so clear that an increasing number of central banks have added them to their analytical toolkits in recent years. Indeed, we have many more central banks attending the CBBS conference today than a few years ago, partly thanks to our colleagues at the Atlanta Fed who have pushed for such a larger group.2 This year’s programme brought together insightful methodological and applied papers on various aspects of business surveys. In the remainder of my intervention, I would like to briefly discuss the advantages and limitations of traditional business surveys, and then take advantage of having such an incredible audience in front of me to nudge researchers, both from academic institutions and from central banks, to explore new directions to develop what I defined in the title as business surveys 2.0, to make sure we seize the opportunities presented by new technologies and digitalization. Some advantages and challenges of traditional surveys Sample surveys have been a fundamental tool for capturing policy-relevant heterogeneities among agents, especially firms. Banca d’Italia has a long tradition With thanks to Marco Bottone, Elena Mattevi, Andrea Neri and Concetta Rondinelli for their valuable contributions. Central Bank Business Survey and Liaison Program Group - Federal Reserve Bank of Atlanta. in this field. In the early 1960s, we launched our Survey on Household Income and Wealth,3 and soon after, in 1972, our annual Survey of Industrial and Service Firms (INVIND).4 Over time, we have continued to expand our toolbox: in 1999, we launched the quarterly Survey on Inflation and Growth Expectations,5 which systematically collects firms’ expectations of consumer price inflation at different time horizons, as well as quantitative information on past and expected changes in their own selling prices. More recently, in 2022, we started the biannual Short-Term Survey of Italian Households,6 which tracks household economic conditions over the business cycle. Business surveys remain crucial to this day for providing timely insights into key analytical issues. As shown in many academic papers, they have allowed us to achieve a deeper understanding of expectations7 and their relation with firm-level dynamics,8 of uncertainty9 and other outcomes10 that would have been otherwise unattainable. Because they can provide timely, specific, and flexible information, business surveys have proved to be particularly useful in recent years, which have been marked by large unexpected shocks. During the COVID-19 pandemic, for example, our surveys were crucial to understand the underlying heterogeneity that drove the surge in bank borrowing and deposits.11 More recently, in a world of heightened geopolitical fragmentation, they have been useful to understand the potential impact of supply disruptions from high-risk countries on European regions, sectors, and firms, and, thanks to the coordinated efforts of several central banks, they have allowed researchers to conduct harmonized cross-country analyses.12 ‘Survey on Household Income and Wealth’, Banca d’Italia, Statistics Series. ‘Survey of Industrial and Service Firms’, Banca d’Italia, Statistics Series. ‘Survey on Inflation and Growth Expectations’, Banca d’Italia, Statistics Series. ‘Survey of Italian Households. Short-term Outlook’ (only in Italian), Banca d’Italia, Statistics Series. For example, C.F. Manski, ‘Measuring expectations’, Econometrica, 72, 5, pp. 1329-1376, 2004; L. Bartiloro, M. Bottone and A. Rosolia, ‘The heterogeneity of the inflation expectations of Italian firms along the business cycle’, International Journal of Central Banking, 15, 5, pp. 175-205, 2019. For example, O. Coibion, Y. Gorodnichenko and T. Ropele, ‘Inflation expectations and firm decisions: New causal evidence’, The Quarterly Journal of Economics, 135, 1, pp. 165-219, 2020; A. Rosolia, ‘Do firms act on their inflation expectations? Another look at Italian firms’, Journal of Political Economy Macroeconomics, 2, 4, 2024. For instance, L. Guiso and G. Parigi, ‘Investment and demand uncertainty’, The Quarterly Journal of Economics, 114, 1, pp. 185-227, 2019; D. Altig, J.M. Barrero, N. Bloom, S. Davis, B. Meyer and N. Parker, ‘Surveying business uncertainty’, Journal of Econometrics, 231, 1, pp. 282-303, 2022. For instance, the interpretation of firms about the ECB’s inflation target, see M. Bottone, A. Tagliabracci and G. Zevi, ‘Inflation expectations and the ECB’s perceived inflation objective: Novel evidence from firm-level data’, Journal of Monetary Economics, 129, S15-S34, 2022. M. Bottone, E. Mattevi, L. Modugno, M. Mongardini and A. Neri, ‘Indebitamento e liquidità delle imprese nel 2020: evidenze su microdati di impresa’, (only in Italian), Banca d’Italia, Covid-19 Notes, 21 December 2021. I. Balteanu, M. Bottone, A. Fernandez-Cerezo, D. Ioannou, A. Kutten, M. Mancini and R. Morris, ‘European firms facing geopolitical risk: Evidence from recent Eurosystem surveys’, VoxEU, 18 May 2024; L. Panon, L. Lebastard, M. Mancini, A. Borin, P. Caka, G. Cariola, D. Essers, E. Gentili, A. Linarello, T. Padellini, F. Requena and J. Timini, ‘Inputs in Distress: Geoeconomic Fragmentation and Firms’ Sourcing’, Banca d’Italia, Questioni di Economia e Finanza (Occasional Papers), 861, 2024. However, conducting traditional business surveys based on sound methodologies such as probability sampling is becoming increasingly challenging and costly for central banks. Examples of key challenges include the rise in both the item non-response rate − which has tripled to 15 per cent for investment plans since the inception of our INVIND survey − and the unit non-response rate, especially among firms that perceive a high response burden.13 Moreover, the growing awareness and concerns about data privacy and time constraints14 not only complicate the conduct of surveys, but also pose serious risks to data quality and inevitably raise questions about how to use and potentially adapt sample surveys in the future. Where could we go next? The digitalization of information systems, the increasing availability of administrative data, big data, and non-probabilistic and online surveys15 (which may seem like quick, easy, and inexpensive ways to collect data), as well as the rise of artificial intelligence (AI), bring potential opportunities for the development of surveys that must be carefully assessed. I see some promising directions. For example, we should take full advantage of the growing availability of administrative and other unstructured data. Combining these data with methodologically sound and carefully designed sample surveys can greatly enhance the informative value of both. This integration can occur at various levels, such as sample selection, weighting, and streamlined questionnaires. In addition, we should harness the rapid advances in AI and machine learning techniques, which have potential benefits at various stages of the survey process. Let me briefly highlight some avenues for further exploration and research in the latter area. Survey research focuses primarily on the wording of questions and answers in an attempt to gather soft information that it is not readily available in more traditional data, a.k.a. hard information. So it seems natural, at least to me, that a first way to improve traditional surveys is to use large language models (LLMs). In the preparation phase of surveys, for example, LLMs could help create questions, identify inconsistencies and suggest effective response options. In data cleaning and management, LLMs could help detect inconsistent responses and prevent low-quality entries. In the final stage of reporting results, LLMs could help ensure accessible formats, whether in the form of summaries, visualizations, presentations, or written text. M. Bottone, L. Modugno and A. Neri, ‘Response burden and data quality in business’, Journal of Official Statistics, 37, 4, pp. 811-836, 2022. For example, M. Galesic and M. Bosnjak, ‘Effects of questionnaire length on participation and indicators of response quality in a web survey’, The Public Opinion Quarterly, 73, 2, pp. 349-360, 2009. R. Gambacorta, M. Lo Conte, M. Murgia, A. Neri, R. Rizzi and F. Zanichelli, ‘Mind the mode: lessons from a web survey on household finances’, Banca d’Italia, Questioni di Economia e Finanza (Occasional Papers), 437, 2018; A. Neri and F. Zanichelli, ‘Principali risultati dell’Indagine Straordinaria sulle Famiglie italiane nel 2020’ (only in Italian), Banca d’Italia, Covid-19 Notes, 26 June 2020; R. Cummings and E. Tedeschi, ‘The effect of online interviews on the University of Michigan Survey of Consumer Sentiment’, Briefing Books, October, 2024. A second way to improve traditional surveys is to use natural language processing and text analysis to quickly extract signals from text. These methods are already widely used in the literature to extract signals on economic variables, such as industrial production from text analysis of survey responses from manufacturing firms,16 or economic activity from earnings calls,17 which are an important channel of communication between market participants and the management of publicly traded companies. In light of these experiences, we might further explore the possibility of extracting information from informal interviews with managers that are traditionally more difficult to interpret, moving away from traditional surveys in which respondents read and select the most appropriate answer. Third, even in the conduct of more formal interviews, surveys could be facilitated by the use of AI-assisted interviewing, which integrates advanced artificial intelligence into the interviewing process, particularly as a tool to assist interviewers in improving data quality (minimizing human error and bias). Such a development would enable more dynamic and responsive interviews that adapt in real time to respondents’ answers, automating routine tasks, providing a more engaging and personalized interaction, possibly leading to higher response rates and greater satisfaction with participation. Fourth, machine learning models could replace current methods for imputing missing data in business surveys, as they are able to capture the complexity in data structures, recognizing non-linear relationships and allowing a large number of predictors to be included.18 Machine learning techniques could also help to detect outliers in surveys, with algorithms that compare responses within the same questionnaire and across waves. Last but not least, LLMs could revolutionize traditional data collection methods. For example, instead of sending out questionnaires, we could provide a programme that firms could adapt to their own databases using LLMs, allowing them to extract the hard information they need, leaving more time for qualitative or soft information, such as plans and assessments. Clearly, the significant benefits of reducing the burden on respondents would have to be carefully weighed against the potential problems of data confidentiality that might discourage firms from participating in the survey. These are just some examples of the potential evolution of business surveys. Understanding whether and how new technologies and tools could be integrated into survey production processes will surely be a challenge, but one that I think is worth taking on. T. Cajner, L.D. Crane, C. Kurz, N. Morin, P.E. Soto and B. Vrankovich, ‘Manufacturing sentiment: Forecasting industrial production with text analysis’, FEDS Finance and Economics Discussion Series, Working Paper No. 2024-26, May 2024. M.A. Gosselin and T. Taskin, ‘What Can Earnings Calls Tell Us About the Output Gap and Inflation in Canada?’, Bank of Canada, Staff Discussion Paper, 29 June 2023; T. Taskin and F.U. Ruch, ‘Global Demand and Supply Sentiment: Evidence from Earnings Calls’ Bank of Canada, Staff Working Paper, 30 June 2023. In this sense, random forests seem to offer some improvements over traditional logistic regression for the complex response model. See T.D. Buskirk and S. Kolenikov, ‘Finding respondents in the forest: A comparison of logistic regression and random forest models for response propensity weighting and stratification’, Survey Methods: Insights from the Field, 2015. Conclusions This conference has provided an excellent opportunity to exchange experiences on methodological and applied issues. It has fostered scientific debate on core topics for monetary and economic policy, facilitated dialogue between institutions, and explored future avenues for research and innovation. Going forward, it is essential to build on the momentum generated here. In an increasingly interconnected world, strengthening our networks and increasing our cooperation could improve the quality and reliability of business surveys and our ability to identify emerging phenomena. Given the busy agenda of this two-day meeting and the growing number of participants, we might consider following up on this event with additional online meetings throughout the year, focusing on specific areas of interest, such as survey design, data collection and data analysis, perhaps with an effort to include the suggestions we have just discussed. Individual institutions often lack the resources to tackle these challenges alone. By working together, we can share lessons learned, best practices, and innovative methodologies to advance the development of business surveys and ensure that they remain a valuable part of our economic toolkits. Designed by the Printing and Publishing Division of the Bank of Italy