metadata: authors: 0: author: Cavanagh,Jack
1: author: Fliegner,Jasmin Claire
2: author: Kopper,Sarah
3: author: Sautmann,Anja
count: World
docna: 0: docna: A Metadata Schema for Data from Experiments
            in the Social Sciences
docty: Policy Research Working Paper
abstracts: cdata!: The use of randomized controlled trials
            (RCTs) in the social sciences has greatly expanded,
            resulting in newly abundant, high-quality data that can be
            reused to perform methods research in program evaluation, to
            systematize evidence for policymakers, and for replication
            and training purposes. However, potential users of RCT data
            often face significant barriers to discovery and reuse. This
            paper proposes a metadata schema that standardizes RCT data
            documentation and can serve as the basis for
            one&#x2014;or many, interoperable &#x2014;data
            catalogs that make such data easily findable, searchable,
            and comparable, and thus more readily reusable for secondary
            research. The schema is designed to document the unique
            properties of RCT data. Its set of fields and associated
            encoding schemes (acceptable formats and values) can be used
            to describe any dataset associated with a social science
            RCT. The paper also makes recommendations for implementing a
            catalog or database based on this metadata schema.
display_title: A Metadata Schema for Data from
            Experiments in the Social Sciences
listing_relative_url: /research/2023/02/33996566/metadata-schema-data-experiments-social-sciences
topicv3: Data Catalogues m1326864 2385,Evidence-Based Policy m1326939 706
publishtoextweb_dt: 2023-02-06T00:00:00Z
dois: 10.1596/1813-9450-10296
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
author_keys: 0: author_key: NA
1: author_key: NA
2: author_key: NA
3: author_key: 1338140
model_extractions: Policy Research Working Paper 10296 A Metadata Schema for Data from Experiments in the Social Sciences _Jack Cavanagh Jasmin Claire Fliegner Sarah Kopper Anja Sautmann_ Development Economics Development Research Group February 2023
Policy Research Working Paper 10296

# **Abstract**

The use of randomized controlled trials (RCTs) in the social sciences has greatly expanded, resulting in newly abundant, high-quality data that can be reused to perform methods research in program evaluation, to systematize evidence for policymakers, and for replication and training purposes. However, potential users of RCT data often face significant barriers to discovery and reuse. This paper proposes a metadata schema that standardizes RCT data documentation and can serve as the basis for one—or many, interoperable —data catalogs that make such data easily findable, searchable, and comparable, and thus more readily reusable for secondary research. The schema is designed to document the unique properties of RCT data. Its set of fields and associated encoding schemes (acceptable formats and values) can be used to describe any dataset associated with a social science RCT. The paper also makes recommendations for implementing a catalog or database based on this metadata schema.

This paper is a product of the Development Research Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at asautmann@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
A Metadata Schema for Data from Experiments in the Social Sciences<sup>∗</sup> Jack Cavanagh<sup>†</sup> Jasmin Claire Fliegner<sup>‡</sup> Sarah Kopper<sup>†</sup> Anja Sautmann<sup>§</sup> Keywords: Randomized control trials, metadata, data publication, secondary research, trial registration JEL codes: C10, C81, C90

> ∗We would like to thank David Rhys Bernard, Merc`e Crosas, Maya Duru, Benjamin Morse, Julian Gautier, Steven Glazerman, Jakob Hennig, Maria Ruth Jones, Jessaca Spybrook, Wendy Thomas, Gabriel Tourek, James Turitto, Keesler Welch, Lars Vilhuber for providing detailed feedback on the metadata schema, and Caitlin Brown and Rachel Griffith for providing helpful comments on the paper. We would also like to thank Davi Bhering, Simon Cooper, Michael Gibson, Sabhya Gupta, Katharina Kaeppel, Daniela Muhaj, Isabela Salgado, Sheral Shah, and Selva Swetha for helping us test the schema with data from 29 RCTs. We also thank Mehmood Asghar, Barbara Bierer, Olivier Dupriez, Julie Goldman, Rebecca LI, Katherine McNeill, Amy Nurnberger, Limor Peer, Matthew Welch, and Julie Wood, for support and feedback at various stages of the project. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.

> Supplementary materials, including proposed controlled vocabularies, can be found in the associated GitHub repository https://github.com/sakopper/rct ~~m~~ etadata ~~s~~ chema.

> †J-PAL/MIT, email: jcavanagh@povertyactionlab.org and skopper@povertyactionlab.org

> ‡The University of Manchester, email: jasmin.fliegner@manchester.ac.uk

> §Development Economics Research Group, World Bank, email: asautmann@worldbank.org.
# **1 Introduction**

The use of randomized control trials (RCTs) in the social sciences has greatly expanded over the past two decades, from economics and political science to public health and education.<sup>1</sup> In parallel, journals, funders, policy organizations, and organizations promoting open science have emphasized that original research data be publicly accessible for others to analyze and use. As a result of this concerted push, hundreds of original RCT datasets from a wide variety of contexts and populations are already published and in principle accessible to researchers, with the potential to benefit numerous areas of research.

However, there remain to date significant barriers to the discovery and use of existing RCT data. Published datasets are scattered across data repositories, journal and university websites, and researcher homepages. In many data repositories, filter options and documentation fields are broad, often consisting of free form text fields, and data quality and provenance are hard to assess. RCT data are often published with a focus on replicating the analysis in a specific paper, and the properties specific to RCTs and features of the data have to be pieced together from inspecting datasets, data appendices, and readme files. Combining datasets across studies is hindered by a lack of harmonization and documentation. Any work in this regard by individual researchers is lost to the next person who wants to conduct a similar study.

In this paper, we propose a metadata schema that can serve as the basis for a catalog of RCT data (or, more broadly, any social science experimental data). The metadata schema defines a set of fields with encoding schemes that can be used to describe datasets associated with social science experiments. The encoding scheme defines the acceptable formats and values to complete the field, such as free text entry, dates, numeric values, or controlled vocabularies (multiple choice options).<sup>2</sup> The proposal organizes fields into thematic modules and includes whether a field should be optional or mandatory. A core objective of this proposal is standardization. Standardizing the documentation of specific types of data allows harmonization, aggregation, and cross-referencing. Data repositories that follow a standardized schema can be easily made searchable by internet search engines such as Google Dataset Search. As much as possible, our schema therefore conforms with common data description standards (such as those of the Data Documentation Initiative (DDI)), and with existing schemata and catalogs for experiments as well as survey data from the social sciences (such as the World Bank’s Microdata Catalog, the AEA RCT Registry, and ClinicalTrials.gov).

In addition, we added fields that we considered particularly useful for secondary research. Datasets documented with the schema can be easily searched or filtered by many different criteria, from the type

> 1We use the term RCT to describe an experiment that uses randomization to assign some intervention or treatment to participants but is conducted outside of a controlled environment (such as a laboratory).

> 2We follow the definitions in ISO (2021).

2
and time period of intervention, to features of the randomized research design (such as stratification or clustering) to contents of the data (such as how take-up/treatment compliance was measured). These options do not currently exist in commonly used data repositories in the social sciences (e.g. Harvard Dataverse, openICPSR, or the World Bank Microdata Catalog). We formulated a set of principles to guide decision making in selecting the final set of fields. These aim to balance the effort to contribute metadata, the usefulness of the metadata for different research purposes, and the complexity of the information collected. For example, we wanted to make the entry of new metadata straightforward for the majority of RCT datasets, while providing enough flexibility to describe unusual research designs or unique data properties.<sup>3</sup> In addition to our proposal, we also formulated a set of recommendations for implementing a catalog or database based on this metadata schema.

With this schema, we hope to provide a public good to the research community that facilitates the creation of inter-operable catalogs, with the goal of speeding up scientific discovery, reducing duplication of effort, and creating a systematic overview of existing RCT research. Research facilitated by RCT data catalogs on external validity, impact heterogeneity, and generalizability of policy impacts beyond the immediate study populations has the potential to help policymakers, funders, or impact investors make decisions about policy programs. Reuse of RCT data can in turn bolster investments in primary data collection and spur methods improvements that will make future experiments faster and more robust. Better access to RCT data can also offer new research opportunities for scholars who do not have the resources to undertake costly primary data collection themselves. Finally, original data is a citable contribution to science independent of an associated paper or report. Enhancing the visibility of the data and the data citation helps ensure that those responsible for creating the data receive credit (some of whom may not be co-authors of the academic study).

The next section gives a brief overview of areas of research that make use of secondary RCT data. Section 3 describes the process undertaken to create the metadata schema. Section 4 walks through the schema in detail, and section 5 describes considerations for creating a metadata catalog based on the schema.

# **2 Uses of Secondary Experimental Data in Research**

RCT data have many properties that make them useful for testing hypotheses and garnering insights that were not necessarily the focus of the original study. Randomization provides a credible exogenous source of variation in the data. Many RCTs collect representative data on large populations, often of groups that are underrepresented (e.g. because they are not part of the formal economy) or of particular interest for policy research (e.g. eligible for certain benefits). RCT datasets often contain indicators and variables of high policy relevance and may use innovative measurement methods such as lab-in-the-field preference measures.

> 3We also aimed to make the schema usable to describe laboratory experiments, although this is not the main focus.

3
In addition, as RCTs as a method mature, researchers have turned their attention to consolidating and systematizing the RCT-based evidence, as well as expanding and improving experimental methodology and econometric analysis methods. In short, many emerging areas of secondary research could benefit from improved access, systematic cataloging, and harmonized documentation of RCT data. Here we provide a brief overview of research that reuses RCT data and informed the development of the schema.

_Combining evidence._ Meta-analysis techniques such as Bayesian hierarchical models (BHM) can increase external validity and generalizability of experimental results. Recent examples include Bandiera et al. (2021), who combine 16 laboratory and field experiments to estimate the impact of performance pay on women, and Meager (2019), who estimates the impact of micro credit on income-generating activities and consumption. When available data is not catalogued, meta-studies run the risk of overlooking less prominent studies.<sup>4</sup> _Assessing estimation and prediction methods._ In seminal work, LaLonde (1986) used experiments as a benchmark to assess the bias in non-experimental estimation methods. The literature that followed (Fraker and Maynard, 1987; Dehejia and Wahba, 1999, 2002; Glazerman et al., 2003, and many more) has matured to the point of being able to draw conclusions about the full distribution of bias (e.g Chaplin et al., 2018). Researchers have also used experimental data for making out-of-sample predictions and evaluating external validity, e.g. by comparing different prediction methods or quantifying site selection bias (e.g. Hotz et al., 2005; Allcott, 2015; Gechter et al., 2019).

_Measurement._ Researchers use existing data to validate methods of measurement for important concept and indicators. Recent work has for example examined sources of measurement error in agricultural data (e.g. Beegle et al., 2012; Rosenzweig and Udry, 2019) and measurement methods for women’s agency (e.g. Donald et al., 2020; Jayachandran et al., 2021).

_Estimating structural models._ Structural models can exploit the experimental variation for identification and enrich experimental data, for example by evaluating the role of underlying preferences and behavioral factors for take-up decisions as a means to conduct welfare analysis or make predictions for the effects of new policies (e.g. Todd and Wolpin, 2006, 2010; Meghir et al., 2019; Guiteras et al., 2019).

_Statistical learning and machine learning._ A rapidly growing literature applies machine learning methods to RCT data. Examples include regularization methods to discipline covariate selection or the identification of treatment effect heterogeneity (e.g. Chernozhukov et al., 2018; Chernozhukov et al., 2018), and new sampling methods, e.g. multi-arm bandits and related adaptive experimental algorithms (e.g. Dimakopoulou et al. (2018); Caria et al. (2021); Kasy and Sautmann (2021)). Existing RCT datasets can be used to optimize algorithms, check large- or small-sample behavior, and simulate thousands of trials to improve speed and evaluate performance without incurring costs or burdening subjects.

> 4An issue potentially exacerbated by biases inherent to the publication process (Andrews and Kasy, 2019).

4
_Epistemology, RCT methodology, and research transparency._ Researchers have begun to examine large sets of studies to understand the “political economy” of designing, conducting, and publishing experiments (e.g., Andrews and Oster (2019) on external validity bias; Gechter and Meager (2022) on the role of pre-existing infrastructure for site selection; H ̈offler (2017) on how data publication influences citation rates; Anderson and Magruder (2017) on how to reduce the number of false discoveries; or Christensen and Miguel (2018) on the adherence to transparent research practices).

_Summary statistics._ RCT data can help both researchers and the broader public to better understand underrepresented populations or get an overview of the body of experimental evidence. Especially in lowincome contexts, RCT data delivers a detailed picture of populations that are typically not well-represented in available data – neither government data, such as (formal) labor market statistics, nor private data, such as bank records. RCT data can be used to extract stylized facts, conduct exploratory research or power calculations, and more. For example, the non-profit AidGrade (2019) compiled a database of standardized effect sizes and standard errors to facilitate simple forms of comparative analysis (e.g. Vivalt, 2015, 2019). The metadata schema is designed to benefit all these different applications by

- enabling filtering datasets on criteria such as unit of randomization or intervention assignment strategy;

- collecting information specific to RCTs such as interventions, treatment arms, or treatment compliance;

- facilitating the combination of multiple RCT datasets by documenting features such as available covariates, time period covered in the data, or inclusion/exclusion criteria;

- recording external resources such as registry entries, ethics review protocols, and academic publications, and documenting information such as whether a pre-analysis plan exists or who funded or partnered in the implementation of the RCT.

# **3 Creating the Metadata Schema**

We followed the process outlined in ISO (2021) in creating the metadata schema. Throughout, we were advised by a group of data scientists and data librarians from Harvard Dataverse, Harvard Medical School, Harvard Business School, MIT, ISPS/Yale, DDI, Vivli and the World Bank Microdata Catalog.

We started by reviewing the types of research that use RCT and experimental data (as summarized in section 2), conducting a survey with researchers who have re-used RCT data or expressed interest in methods research,<sup>5</sup> and researching existing metadata schemata for social science and experimental data. The four main sources of metadata fields we ended up using host some of the largest collections of information on existing RCTs and RCT data (at the time of writing).

> 5Expressions of interest were collected through the Research Methods Initiative of Innovations for Poverty Action (IPA) and the Global Poverty Research Lab, or through surveys of affiliated researchers of the Abdul Latif Jameel Poverty Action Lab (J-PAL).

5
For information related to survey data, we focused on schemata based on the **Data Documentation Initiative (DDI)** , an international standard for documenting survey data (DDI, 2021), in particular the fields used in the Harvard Dataverse and the International Household Survey Network (IHSN) template of the World Bank Microdata Catalog. The **Harvard Dataverse** is an accredited, cross-disciplinary data repository that enables any researcher or institution to publish and archive data and code. J-PAL and IPA maintain a Dataverse data collection called the “Datahub for Field Experiments in Economics and Public Policy” that currently hosts over 200 RCT datasets. The repository builds on a suite of tools for the publication of scholarly data (King, 2007), and its metadata schema is mapped to the DDI Codebook. The schema is organized in blocks that can be used to customize the metadata documentation in individual data collections (Harvard Dataverse, 2021). We considered all blocks used by the J-PAL/IPA Datahub.

The **World Bank Microdata Library** hosts survey and other data from multiple institutions, including the World Bank’s own research departments, which publish their data under the World Bank’s Open Data Policy. It aggregates multiple named collections, including those of the Development Economics Research Group (DECRG), the Development Impact Evaluation Unit (DIME), and the Strategic Impact Evaluation Fund (SIEF),<sup>6</sup> and all metadata can be accessed through an Automated Programming Interface (API) (The World Bank, 2022). The World Bank’s IHSN microdata template has four sections compatible with DDI – Document Description, Study Description, Datasets, and Variables Groups – and an External Resources section compatible with the Dublin Core metadata standard (IHSN, 2022). We considered all of these for the schema and adopted many usage recommendations from Dupriez et al. (2021).

Most existing data catalogs do not cover fields related specifically to the design of RCTs. For these, we turned to two important trial registries. The **AEA RCT Registry** of the American Economic Association is likely the most complete record of past and ongoing RCTs in economics, including unpublished studies (AEA RCT Registry, 2022). The registry metadata contains information specific to RCTs that is not typically included in survey schemata, such as the intervention, randomization method, outcome measures, or the reviewing IRB, although these contents are mostly stored in free text fields and there is no cross walk with DDI or other metadata standards. The US National Library of Medicine and the National Institutes of Health maintain the trial registry and results database **ClinicalTrials.gov** for clinical trials in the United States (McCray and Ide, 2000). All clinical studies of drugs and devices controlled by the Food and Drug Administration (FDA) must be registered here. The registry has detailed metadata field definitions and maintains an API feed (ClinicalTrials.gov, 2022). Many repositories draw from or link to this registry (e.g. Vivli, 2022; ISRCTN, 2022). The fields on interventions, study arms, and outcome measures provided a

> 6Impact evaluation often means RCT in this context but can also mean other rigorous analysis methods aimed at estimating causal impacts.

6
model for our schema.

Roughly 270 metadata fields were under consideration for inclusion in the final schema. Some metadata schemata provided valuable insights even if their fields were ultimately not adopted; the full list of schemata considered can be found in the GitHub repository for this project. The following **principles for the RCT metadata schema** guided decision-making on (i) which metadata fields to include, (ii) the encoding scheme for each field including whether to create a controlled vocabulary (multiple choice options), and (iii) the cardinality of each field (i.e., whether the field is optional vs. mandatory and if it can be repeated):

1. The primary purpose of the schema is to provide information that helps identify RCT datasets for secondary research, i.e., using the previously collected data for new studies.

2. The schema primarily describes the design of the data collection and content of the data, not the academic study or analysis results.

3. Preference is given to DDI-compliant over existing and to existing over newly created fields.

4. Field definitions should make information comparable across studies.

5. The level of detail collected must balance usefulness for the purpose above with the effort required to create an RCT metadata record for contributors.

6. The schema must balance ease of use with completeness.

Note that principle 1 and 2 set the schema apart from data repositories that focus on making the analysis in the original study replicable (and thus primarily record aspects of the data that were already exploited for research). Item 2 also excludes information such as estimated treatment effect sizes, which depend on the analysis method applied. Items 3 and 4 aim to make the schema interoperable with existing catalogs. Item 4 led us to sometimes amend the original field definitions and provide controlled vocabularies wherever possible (see below). Item 5 ruled out information that could be difficult to obtain or verify, especially ex-post (such as information on intervention cost or budget). Item 6 means that the defaults of the schema focus on the most common data structures, while optional free text fields allow for supplying additional information.

Iterations of the metadata schema were extensively reviewed and tested by the authors, supported by a group of J-PAL staff, an external group of experts in RCT data reuse or metadata schemata, and the advisory group mentioned above. Part of the testing consisted of completing the metadata fields for a set of 29 RCTs. Most of these test datasets are hosted on the J-PAL Dataverse and were chosen based on aspects of their design, with the aim of testing both “typical” RCTs and “edge cases” (for a full list, see the supplementary material on the GitHub repository for this project). The purpose was both to test the metadata schema in its entirety, including the sequencing and structure of the fields, and to select, develop and test associated controlled vocabularies. We also tested whether the field definitions were easy to understand, and whether there was any difficulty obtaining the requested information for a given RCT. Tester feedback led us for 7
example to provide a definition of what constitutes a “dataset”; see Section V. Data below.

## **A Note on the Controlled Vocabularies**

Though not formally part of the schema, the controlled vocabularies (CVs) are an important part of our proposal. CVs are primarily used for filtering, and as with multiple-choice survey questions, they need to “partition” the space of possible options, meaning the set of individual entries must cover the entire universe of options without overlap (i.e. options that could fit two or more entries). Moreover, entries need to be balanced, in the sense that they need to be specific enough to help users narrow down the set of studies of interest, but broad enough so that each entry applies to more than a small number of studies.

The catalog testing process<sup>7</sup> led to numerous adaptations to existing CVs, both removal/aggregation of entries and addition of new entries. Two examples are the fields _Kind of Data_ and _Mode of data collection_ (CVs G and I in Appendix B). These fields are part of the DDI and appear in many schemata including the IHSN, which uses a truncated version of the DDI CV (Dupriez et al., 2021). However, testers found the IHSN CV not well-suited to describe administrative data or data from laboratory or “lab-in-the-field” experiments. This led us to add back relevant items from the DDI CV as well as create new entries.

In some cases we developed entirely new CVs, such as CV E in Appendix B, which describes available types of covariates at the cluster or group level. We see some of the newly proposed CVs as under development. In these cases, we provide a version 0.9 in the GitHub repository of this project, with the aim of updating to version 1.0 based on a larger body of RCT datasets.

# **4 The Metadata Schema in Detail**

In what follows, we give an overview of the metadata schema, divided into modules I to VII. Appendix A contains a corresponding table of all metadata fields, with the CVs we suggest listed in Appendix B. In this section, we provide supplementary information about the schema for back-end users or _contributors_ who create metadata schema entries, _data users_ who are perusing a catalog based on this schema or reading individual metadata entries, and data stewards and application programmers integrating the schema into an existing application or building a new catalog, also called _catalog owners_ . We included illustrative examples that may be helpful for establishing a standardized way to describe an RCT’s research design. Some details serve users who are less familiar with the conventions and practices for social science RCTs; researchers who routinely work with RCT data may find the information in Appendices A and B sufficient.

For each metadata field, the table in Appendix A contains a name, a short description, an encoding

> 7Testers were asked to comment on the overall suitability of each CV for documenting RCTs, as well as the individual options within each CV. Testers could also suggest new CV entries. For test fills, testers both provided free-form text responses and selected all applicable choices from each existing CV (if any) in order to evaluate the CV’s coverage as well as potential ambiguities.

8
scheme, and the “cardinality” of the field, that is, whether the field is optional or mandatory, and whether it is unique or “repeatable”. We also included suggestions for implementation for an adopting organization (see section 5 for more). A cross-walk with other metadata schemata is posted in the Github repository. Note that in many cases we adopted existing fields but modified the wording of definitions for clarity in the context of social science experiments.

**What Constitutes an RCT Metadata Record?** An RCT is an experiment combined with data collection on the subjects or experimental units. It is principally defined by the study population and unit of randomization, the intervention, and the randomization procedure used to create comparable treatment arms. The schema is designed assuming that each top-level metadata record corresponds to exactly one _study_ or RCT, in which a set of interventions were randomized in a sample representative of some population described in section II. We describe how to delineate the one or more individual _datasets_ that are part of an RCT below in section V on “Data”.

# **I Basic Information**

The **basic information** fields in the schema provide summary information on the RCT.

The section contains all the information needed to cite RCT data, including the _study title_ as well as _author/data owner names_ and _affiliations_ . Original data is a contribution to science separate from publications based on the data. Contributors may consider crediting a larger or different set of individuals from the academic article. If the data is stored in a repository or other citable location, the metadata record should include the same citation information that is also provided with the data.

Similarly, the _abstract_ describes the purpose, nature, and scope of the RCT and data, and may contain more or different information from the paper abstract. The _topic classification_ uses a CV to describe the area of research. While we tested CVs such as CESSDA, IHSN, the World Bank themes, and the J-PAL/IPA sectors, ultimately an organization adopting the metadata schema may choose a topic CV based on its own requirements and use case. For example, an economic journal may choose to use the JEL codes (AEA, 2022). The _version_ and _version date_ fields provide version control for the metadata record.

# **II Study Population**

The **study population** section provides information related to the study as a whole and who was included in it, covering location, study population, and study sample.

Contributors can choose the _country of intervention_ from a CV (ISO country code), with the option to add free-text detail on _geographical coverage_ , as well as any _inclusion and exclusion criteria_ for the intervention studied. In a policy context, these might be formal eligibility criteria for a social program or benefit; the 9
researchers might also apply other research-related conditions for inclusion into the treatment and control groups. Jointly, the geographical information and inclusion/exclusion criteria describe the sampling frame from which the randomization units in the treatment and control groups are (randomly) drawn (see also section IV on sampling method).

The second set of fields concerns the unit of randomization as the primary unit of statistical analysis (as opposed to the unit of observation; see below). A _randomization unit_ can be an individual experimental unit or a group (a cluster). Contributors are asked to choose the randomization unit from a controlled vocabulary. The CV we recommend expands the DDI CV considerably to account for frequently occurring units of randomization in social science experiments, such as households, businesses, or schools. The CV allows for separate description of physical units (e.g., a production line, a class room) and administrative or legal units (e.g., all employees of a firm, all students at the same grade level). This distinction can matter for example with interventions that exhibit physical spillovers, such as health interventions. Health interventions are for example often assigned at the school, classroom, or grade level (see e.g. Parker et al., 2021).

With individual-level randomization, the unit of randomization is typically the same as the unit of observation. With cluster-level randomization, each randomization unit may contain several observation units or even different types of observation units.<sup>8</sup> A unit counts as a _targeted_ randomization unit or cluster if it was intended for inclusion in the study (either to receive an intervention or in the control group), even if the intervention was ultimately not offered or received as intended, or if no outcomes were measured. A unit counts as an _actual_ randomization unit if at least one outcome was measured for one observation unit within the cluster post-intervention. There may be a variety of reasons for a discrepancy between actual and targeted sample sizes. This could be random variation (e.g. patients or job seekers visiting a facility on a given date), but also implementation errors. Note that even if a targeted unit is assigned the experimental intervention as planned, there may be non-compliance, i.e. subjects may not take up the intervention or circumvent or counteract it. Compliance is covered in section IV; _actual study sample size_ should include non-compliers. The fields in this section pool observations across waves and treatment arms to provide high-level information on the size of the study. For a breakdown by arm see section III.

Cluster-randomized studies may randomly assign different units of observation, say, buyers and sellers. Note, however, that the unit of randomization is “one level higher”, i.e. (for example) the market in which buyers and sellers interact. Even though individual buyers and sellers are randomized into treatment, the random variation used in the analysis comes from the different treated shares on both sides of the market (in the field “Study was designed to analyze” in section IV, contributors would in this case report “general 8The unit – or units – of observation are recorded in section V. Data below, how randomization units are sampled is in section IV. Study Design, and how randomization units are assigned to treatment arms is detailed in section III. Outcomes and Interventions.

10
equilibrium effects”). Similarly, in the cross-over design in Lopez et al. (2022) (see below), patients arriving at clinics at different days received different treatment arms. However, the level of randomization is the clinic. In the edge case where an academic study uses two separate randomization procedures and presents the treatment effects from each separately – i.e. uses the randomization at different levels for identification – contributors may choose to create two metadata records, which can be linked in section VII. Otherwise we recommend reporting the higher level unit of randomization (e.g. the market vs. the buyer or seller). _Example:_ The study “Targeting the Poor: Evidence from a Field Experiment in Indonesia” by Alatas et al. (2012) compared different methods for targeting aid to poor households. To create their sample, the authors chose three provinces in Indonesia, then randomly selected 640 villages from those provinces (stratified by geographic location, see below). The choice of the three provinces should be described as part of the inclusion/exclusion criteria. The description should also include that larger villages with more than 100 households per sub-village on average were excluded from sampling in one district. The _randomization unit_ in this study was the village, while the _unit of observation_ was the household. Both the _targeted_ and _actual_ sample sizes were 640 villages and sub-villages and 5756 households.

# **III Outcomes and Interventions**

This section concerns the tested interventions and outcomes. The first set of fields describes the outcome measures collected. The fields in this section are repeated for each outcome variable. Contributors provide a short free text _name_ , a _category_ chosen from a controlled vocabulary, an optional free-text _description_ , and a yes/no answer as to whether the outcome was measured at least once prior to the intervention (“at baseline”). Baseline outcome measures are relevant for treatment effect estimates but can also serve a range of secondary research purposes, such as summary statistics of the study population. We tested different CVs for the outcome categories but ultimately decided against making a recommendation before carrying out further testing with more (and more varied) RCTs. The free text description field can be used to provide additional information such as the unit of measurement, data format and type, or any transformations applied. This is especially useful when comparing data from different studies.

The next part of the module records the actual interventions and their assignment to study arms. A social science RCT may test the effects of a policy change, encouragement, information, or other process or action. In laboratory experiments, experimental treatments can include complex variation of the incentives, game forms, or information provided that govern the interactions of participants. An _intervention_ is any experimental manipulation of the participants’ environment. An _arm_ is a randomly selected subgroup of participants that receives none, one, or multiple interventions as part of the study. A canonical RCT consists of one treatment arm receiving the intervention and a control group arm that does not receive the 11
intervention. In practice, RCT designs are often more complex, with multiple study arms and different interventions or intervention levels. The schema proceeds by listing all interventions and arms, and then matching none, one, or multiple interventions to each arm. This structure closely follows _ClinicalTrials.gov_ ; to our knowledge, no metadata schema used in the social sciences provides this level of detail.<sup>9</sup> Contributors are first asked to list the study _interventions_ by assigning a short _name_ , classifying the _intervention type_ , and then providing an optional free text _description_ . The CV for the intervention type is still in development, as the CVs tested did not provide the right balance between broad classifications and detailed options for the kinds of interventions that occur frequently in social science RCTs. Note that the intervention type is one of three key fields that either alone or in combination help users narrow down the content of the study; the other two are the topic area (under I. Basic Information), and the outcome measures (see above). All three may be different: for example, Oster and Thornton (2012) randomize the distribution of menstrual cups in Nepal; they measure take-up by direct recipients as well as individuals in their social networks as the outcome; and the aim of the research is to understand peer effects in technology adoption. Finalizing the CVs for these fields will require testing with a larger body of studies. The new CVs will be posted on the GitHub repository for this project.

Next, contributors can select the _intervention assignment strategy_ from a CV and then optionally provide a more complete description. The CV options are adapted from _ClinicalTrials.gov_ and include parallel, factorial, and cross-over assignment, as well as the option “other”.<sup>10</sup> Contributors can also provide additional information, such as whether the random assignment was carried out using stratification. We encourage listing out all stratification variables. Stratification and other procedures aimed at improving balance such as re-randomization typically reduce variance but also need to be accounted for in the analysis.

Finally, the section records the _study arms_ . Contributors can give an identifying _name_ to each arm, list the _targeted_ and _actual number of randomization units_ in the arm, and then indicate which intervention(s) this group received (if any: a control group may receive no interventions). This allows users to back out, for example, how many subjects received the same intervention across treatment arms, and (for factorial designs) which intervention combinations are observed in the data.

Some experimental designs are common in the social sciences but less so in clinical trials. We make some recommendations on how to describe these designs using the options provided. One example is randomized phase-in, meaning that the intervention is randomly assigned to start at different times for different experimental arms, and the comparison between the arms (already) receiving the intervention and the arms

> 9For example, the AEA RCT Registry and YARD only have free text fields to describe the intervention(s) and do not between _interventions_ and _arms_ .

> 10Note that we dropped “single group assignment” and “sequential assignment” as these assignment strategies do not create a comparison group and therefore do not constitute an RCT by the common understanding of this term in the social sciences.

12
not (yet) receiving the intervention in each period is used to estimate the treatment effect. We recommend using the “crossover design” option, defining one arm for each study group that starts the intervention at a different time, and explaining the timing of the phase-in and duration for which each arm receives the intervention in the free-form text _Example:_ In Barrera-Osorio et al. (2020), 101 private secondary schools in Uganda were randomly assigned to receive per-student vouchers from the government, 51 starting in the 2011 school year, and 50 starting in the 2012 school year. The authors use the difference in intervention start date to estimate the short-term impact of the public-private partnership program on student enrollment and performance. Other examples of studies using phase-in designs are given in Bouguen et al. (2020).

In more standard cross-over designs, each arm receives different (possibly all) interventions sequentially, and only their order is randomly assigned. Such designs are frequently used in laboratory experiments. A cross-over experiment with two treatment conditions A and B might then be defined as having two arms and two interventions, where both arms receive both A and B (but one arm receives intervention A first and the other receives intervention B first).

_Example:_ In Lopez et al. (2022), the two clinic-based interventions consisted of discount vouchers for a free course of malaria treatment, given either to physicians to pass on to patients at their discretion (“doctor voucher”), or directly to patients before the consultation with the physician (“patient voucher”). The days on which each clinic received either no intervention, the doctor voucher intervention, or the patient voucher intervention were selected based on a randomized schedule. This design could be described as a cross-over design with 60 arms (each clinic) that each received both interventions. The rotational calendar that was used can be described in the free-text field. This cross-over design is unusual in that every clinic was randomized into a schedule.

Factorial (cross-randomization) designs combine two (or more) types of interventions with different levels or intensities, leading to arms that each receive different combinations of (levels of) interventions. The simplest factorial designs have two interventions, and the “levels” might simply consist of either receiving the intervention, or not receiving it. The four treatment arms then receive A and B, only A, only B, or neither. More complex factorial designs might involve multiple levels of each type of intervention.

_Example:_ Cohen et al. (2015) cross-randomized several subsidy levels (from 0% to 92%) for malaria medication and malaria tests to understand how the availability of affordable testing affected demand for malaria treatment. This design may be recorded by defining one intervention for each subsidy level and each subsidized good that appears in the study. Factorial assignment encompasses “fractional” factorial designs which may drop some of the cells that would be created by a full cross-combination of all intervention levels.

To some degree, the definition of arms and interventions is up to the metadata contributor. For example, 13
some policies or programs consist of a bundle of different types of interventions, such as a health consultation combined with a discount on a health product. An RCT may only test some combinations of these components; e.g., a family planning consultation with or without a discount on birth control, but not a discount without a consultation. Such designs are technically closer to a parallel design than a factorial design, since the effect of the discount cannot be independently assessed from the effect of the consultation. That said, contributors may still choose to define the two interventions “discount” and “consultation” (as in a factorial design) rather than “consultation plus discount” and “consultation.” This can make sense especially if the two intervention types differ substantively and data users might be looking for interventions of only one type. Contributors should still select the intervention assignment strategy that best fits (in this case, parallel).

Recently, adaptive experimental designs have received increased attention in the social sciences. One type of experiment uses the information learned during early observations or waves in the experiment to alter the assignment shares of the different treatment arms (e.g., Kasy and Sautmann, 2021; Caria et al., 2021). Even though the arm size is ex-ante unknown, we recommend using the “factorial” or “parallel” CV entries, using the arm sizes targeted in each stage of the adaptive design, and the free-text field to describe the adaptive assignment strategy. Other experiments study the optimal treatment of a given experimental unit over time, see e.g. Almirall et al. (2014). This may include randomized changes to the treatment over time. Here, the contributor might choose the option “other” and describe the assignment strategy in the free text field.

The last two fields in this section record the overall time period of all interventions. The timing of the individual intervention is not recorded separately in order to reduce the burden of completing the intervention fields, but if the timing of an intervention is important to the design, contributors can either use the freetext field or choose to define separate interventions that distinguish time of treatment receipt. This may be appropriate in phase-in or cross-over designs as above; if the intervention de facto changes over time (e.g. a remedial tutoring program with a changing curriculum); or if treatment effects are expected to differ significantly based on timing or length of the treatment.

# **IV Study Design**

This module provides further information on the research design of the study. The first field asks whether the study builds on or extends a prior RCT. This could be the case if a new outcome is measured, or an intervention is added, but the randomization and sample of at least some of the original study are retained. This information can be important for assessing statistical power or identifying related data sources.

The next set of fields records how the randomization units in the study were sampled from the sampling frame described by the fields in section II. Contributors define the sampling _type_ using the associated CV detailed in Appendix B and can then (optionally) _describe the sampling method_ . When the sampling strategy 14
is simple, the “type” field may be sufficient. For example, an RCT may include the entire population in a location – all students in a district’s schools, etc. – in which case the sampling method is “1. Total universe (population)”; no further explanation is required. When sampling was carried out in multiple stages that involve different forms of probability selection (Option 2.5: Probability - Multistage), or using a mixed strategy (Option 4: Mix of probability and non-probability sampling), providing a description of the process is very helpful to data users. Note that _sampling type_ refers to the method for sampling randomization units. If different, information on the sampling of the _observational_ units may be provided in section V. The _Study sampling method: Description_ field also allows contributors to provide more detail on how the sample size was chosen (e.g., a sample determined by ex ante power calculations vs. an RCT at scale), and why targeted and actual numbers of randomization units may not be the same (see above).

_Example:_ The Indonesia study by Alatas et al. (2012) sampled randomization units using a stratified multistage design. The authors randomly selected 640 villages from the included three provinces based on a 30/70 urban/rural split, and then randomly selected one sub-village (neighborhood) from each village. On their own, the two sampling stages could be described as “2.3.1 Probability - Stratified: Disproportional stratified” and “2.1 Probability - Simple random,” respectively. In the CV, the contributor should select “2.5 Probability - Multistage” and describe the two selection stages for the village and sub-village in the free text field. Additional helpful context for the _Study sampling method: Description_ field could include that, even though the targeted and actual sample sizes are equal, five of the originally selected villages were replaced prior to the randomization for various reasons.

The other elements of the Study Design section provide additional information relevant for the variability and external validity of the treatment effect estimates. First, contributors can select the types of _covariates_ available in the data from a CV, with the decision to indicate types of available covariates rather than variable-level information aiming to not overly burden contributors. After testing different CVs, we propose a CV adapted from GESIS (Hoffmeyer-Zlotnik, 2016) to describe individual-level covariates, and a new vocabulary to describe covariates at the cluster level, such as household or other group-level characteristics. Information on covariates is needed for meta-analyses and can also be useful for methodological research. For instance, Tabord-Meehan (2018) uses data from an experiment on increasing charitable donations by Karlan and Wood (2017) to demonstrate a new method of adaptive stratification that uses information from a first experimental wave to select “stratification trees” in the second wave.

The next field of the schema includes a new optional field that describes which forms of treatment effects _the study was designed to analyze_ . This includes the “intent to treat” effect, average treatment effect, and local average treatment effect or average treatment effect on the treated. Note that the latter two imply that compliance with the treatment assignment is known (see below). A study that is designed to 15
analyze “4. Heterogeneous treatment effects or effects by subgroup” is powered to detect treatment effects in each population subgroup. The researchers may conduct disproportional stratified sampling and oversample subgroups that constitute a small share of the population in order to estimate treatment effects in this subgroup. A study designed to measure “5. General equilibrium effects” might measure outcomes for groups other than the directly affected group and randomize the interventions at the market level, rather than the individual level. An example might be to measure the effects of providing the unemployed with job search assistance on salaries and firms. A study that captures “6. Spillovers or externalities” measures effects of treating one unit on other units in the vicinity. This is often done by varying the share of treated units within a cluster and requires collecting outcome data on untreated units.

_Example:_ Cr ́epon et al. (2013) randomly varied the share of unemployed job seekers in a city receiving a job placement assistance program in order to study displacement effects on those not receiving the program.

These design features are specific to social science RCTs, and to our knowledge there exists to date no CV for them. We consider this CV under development.

The last field of this section describes compliance with the randomized intervention assignment. In some situations, treatment assignment is not identical with treatment receipt or take-up. Those assigned to the intervention may not actually receive it, and conversely those not assigned to it may nonetheless gain access.

_Example:_ Imperfect compliance is particularly common in so-called encouragement designs. In an experiment with around 1500 small firms in Tajikistan, Okunogbe and Pouliquen (2022) vary whether firms are trained and provided assistance for filing their taxes electronically in order to estimate the effect of e-filing on tax payments and other outcomes. About 60% of the control, but 93% of the treatment group adopt e-filing. This is an example of “two-sided non-compliance”:<sup>11</sup> some firms in the treatment group do not use e-filing while many in the control group do.

The degree of treatment compliance is important for external validity and potential selection effects and has also been used in methodological research; for example, Bernard et al. (2022) use imperfect compliance RCTs to estimate the bias of observational methods in practice. Partial compliance is more common in the social sciences than in laboratory or medical trials, which are typically closely controlled. At the same time, it is often difficult to unambiguously define and measure. We ask contributors to use the free-form _compliance description_ field to explain what forms of non-compliance are in principle possible, how compliance was measured, and what the actual rates of non-compliance are.

> 11See Angrist and Pischke (2009) for the terminology of one-sided and two-sided compliance.

16
# **V Data**

The **data** section of the metadata schema describes the actual data available to users, arranged in one or more datasets. For data with restricted access, access modalities can be described in the External Resources section. Data that are not accessible to anyone but the original researchers should not be described.

More than one _dataset_ may be associated with a given RCT. For example, the contributors may have collected census or administrative data for a larger sample than the ultimate study population, or specific information on two different populations affected by the RCT (e.g. buyers and sellers of a good). Datasets are distinct from data files; a dataset may be broken up into several files and even stored in multiple locations, for instance restricted-access GPS data vs. publicly accessible de-identified survey responses.<sup>12</sup> In general, a set of records may constitute its own dataset if it contains information central to the study, such as a separate outcome measure or the sampling frame, and (i) consists of observational units from a distinct study population (or sample from the study population) or (ii) is based on an independent data source (e.g. with a specific mode of data collection).

What delineates a dataset is ultimately up to the metadata contributor, but we recommend keeping the number of datasets to the minimum needed to describe the data well for users; often the data associated with a given RCT can be characterized as a single dataset. For example, the same dataset may contain several rounds of data collection, and the metadata schema allows multiple “cycles” (including repeated cross-sections). Even data from different sources can often be treated as part of the same dataset.

In some cases it can be useful to define a separate dataset to describe the data in sufficient detail. For example, two measures of the same outcome, such as measures of crop productivity obtained through inperson audits and satellite imagery, may have large discrepancies in coverage and diverging numbers of observations. In cluster-randomized studies, data may be available both at the cluster and individual levels, and these datasets should be described separately if each contains primary outcome measures. Similarly, a full census in the study area followed by sampling a subset of the population for the interventions and endline data collection warrants defining separate datasets for each data collection round.

For each dataset, contributors are asked to define a short description or _name_ , then provide information on the types and number of _observational units_ , first in total and then, further below, per arm. This information is collected at the dataset level in order to allow for multiple units of observation, for example outcomes measured at the teacher and the student levels. The per-arm fields are optional to allow description of data collected prior to treatment assignment (e.g. a study population census). A unit counts as a _targeted_ observation unit if it was selected or intended for data collection. This may be an estimated number. A

> 12Conversely, a data file may contain several datasets (e.g. an excel file with several sheets).

17
unit counts as an _actual_ observation unit if at least one outcome was measured for it. With clustered randomization designs, the targeted and actual numbers of randomization and observation units may all be different. Targeted and actual number of observations may differ in particular if there is survey attrition, which gives an indication of data quality and can help answer methodological questions (e.g. when researching study designs intended to limit attrition).

Contributors are also asked to define the time method (i.e. whether the data contains one or more crosssectional samples or panel data) and the number of _cycles_ (i.e. waves or rounds of data collection), list the _modes of data collection_ in detail, describe the sampling method, and specify whether there are sampling weights. Note that researchers may technically carry out separate power calculations and devise sampling procedures for different units of observation within the same study. In order to collect this information in one place, any information related to power calculations (including the determination of the targeted number of units of observation) should be included in the field IV.3 “Study sampling method: Description”. The method of sampling for the randomization unit is included in IV.2-3, but the method of sampling the unit of observation within the unit of randomization and any discrepancy between targeted and actual units of observation is recorded here in field V.1.I.

A free text field allows contributors to provide _data collection notes_ , which may describe for example how the observational units in the dataset were sampled, when during the experiment data was collected (e.g. at baseline, midline, or endline), and what quality controls were in place for the data collection. Finally, contributors are asked to provide information on the timing of data collection by specifying the time period covered in each data cycle, and, if different (e.g. in retrospective surveys) the dates of data collection.

# **VI Ethics and Research Transparency**

This section allows contributors to provide information that may be relevant to the legitimacy, external validity, credibility, and robustness of the study and its data. This includes information on _ethics review_ conducted, specifically the _reviewing institution(s)_ and _protocol number(s)_ , as well as information on funding or supporting bodies and implementation partners. Here, survey firms, collaborating government agencies, and other parties can be named.

Contributors can also indicate whether any documentation is available to users on the ethics of the study, such as consent forms or a structured ethics appendix, as proposed by Asiedu et al. (2021), and to what degree records related to registration or pre-specification exist. The CV includes options such as “pre-results acceptance” for studies accepted into a journal based on the pre-specified research design, and “populated pre-analysis plan” for cases where the researchers produced a document containing the analysis exactly as pre-specified, typically separate from the research paper (see Banerjee et al. (2020)). The CVs for both fields 18
were newly developed.

# **VII External Resources**

This section provides information on external resources available for the study, including the location of the described data. For each resource, contributors can provide a _type_ and free-form _description_ , _citation_ , _link (DOI/URL)_ , and information on the _access policy_ for this resource.

If available, contributors should not only point to the data itself, but also to resources describing the data and study such as academic publications, reports, codebooks, ethics protocols, etc. The controlled vocabulary for the resource _type_ is adapted from the World Bank Microdata Catalog but allows as additional types “database or data repository entry” (to describe the locations of the study’s datasets), “trial registration”, “pre-analysis plan”, “populated pre-analysis plan”, and “research ethics documentation”. The last four entries complement the information in section VI.

Data from a single RCT may be stored in different locations (e.g. replication data in a repository like Harvard Dataverse, access-restricted identifying data with the researcher, administrative data with a separate data provider); each resource should be linked. Conversely, different external resources, such as the data itself along with replication code, are sometimes stored in the same location or with the same citation information. The CV allows contributors to enter the information once and tick all options that apply under _type_ and provide additional detail in the _description_ .

# **5 Creating a Metadata Catalog Based on the Metadata Schema**

We close with a few considerations for implementing a catalog based on the proposed schema, with a focus on maximizing the quality, consistency, and usability of the metadata and catalog.

**Catalog:** An RCT metadata catalog consists principally of a back-end data entry interface; front-end search, filtering, and display functions; and an underlying database of metadata fields. Catalog owners may consider modifying the schema. For users interested in adopting only a subset of the metadata fields, we made an effort to minimize dependencies across modules (with the exception of core properties of the RCT such as the study arms, which appear in several modules). Catalog owners may also be interested in expanding the catalog beyond recording the properties of the data, for example by adding fields for analysis results, such as standardized treatment effect sizes, or contextual information, such as intervention costs.<sup>13</sup>

To fully support interoperability and bulk uses of the metadata, we recommend an _API interface_ to access and download the catalog contents. This functionality also allows the catalog to be read by data search

> 13These are not part of our schema because they often require a separate effort: verification (e.g. by replicating the analysis), modification (e.g. by standardizing effect sizes or harmonizing analysis approaches), and possibly additional research/estimation of data not in the public domain (such as the fixed and variable intervention costs).

19
engines like Google Dataset Search, increasing reach and accessibility. Since the CVs for this schema are under development, we recommend that early adopters use free text fields after “other” options to collect information for potential expansions or modifications to the CVs.<sup>14</sup> Finally, _user accounts_ allow contributors to track and edit catalog entries after they have been posted, data owners to claim entries, and front-end users to save searches or individual records, while a _data review system_ contributes to ensuring data quality control and avoiding version control issues, duplication of entries, or so-called “ghost entries”. For this purpose, incomplete entries could be regularly flagged for purging.

**Back-end:** The data entry interface for contributors should minimize the time and effort (and hence potential for errors) required to complete an entry to help maintain data quality. This includes making use of existing information where available. Especially for bulk entry of existing RCT data, a sophisticated implementation could draw on existing public records where available, such as the API interfaces of ClinicalTrials.gov and the World Bank Microdata Catalog,<sup>15</sup> , or the downloadable AEA RCT Registry data. New catalog entries could also be partially pre-filled by scraping web data from external resources such as registry entries and academic articles. This could best be achieved by beginning the data entry with the links to (some) external resources (which would also permit a duplication check within the catalog).

Within a metadata entry, information filled early on can provide inputs to later fields, such as pre-filling the treatment arms that were recorded in the “outcomes and interventions” section (III) in the “data” section (V). The catalog can also prompt contributors for information based on earlier inputs, such as suggesting entries for ethics documentation (section VI) or prior studies (section IV) in the “external resources” section VII. Automated cross-checks and data validation can additionally be applied to numeric and multiple-choice entries. We make specific suggestions in the programming notes in Appendix A. These features minimize errors and duplication of effort and improve consistency. Incomplete entries should be regularly automatically saved to prevent data loss.

An intuitive interface and extensive user support can also help with data entry. For example, CVs should be implemented as multiple choice radio buttons or “select all that apply” tick boxes; at the end of an entry in a loop, the navigation should allow users to choose between adding another entry or moving to the next field. In addition to the field and CV definitions in the Appendix, the interface could provide help “bubbles” containing longform instructions and detailed examples that could be drawn from this article. This could be especially useful for describing the intervention assignment and study design in sections III and IV.

**Front-end:** Careful interface design for front-end users can facilitate a quick overview over multiple datasets as well as individual RCTs. An individual study view could pull selected information from various

> 14The authors encourage submitting suggestions for such changes through the GitHub for this project.

> 15This is facilitated by the crosswalk we created to link individual fields in our schema to those in other schemas; see this project’s GitHub repository.

20
modules and arrange it for better reading. An example is the cross-reference between which arms and interventions, which would be most intuitively displayed as a table. Ideally, users can also save searches or selected studies (see above) and visualize various metadata fields for this set (e.g. show the share of entries with a specific value for CV fields). Front-end users should also have access to the same longform/help information as back-end users to facilitate interpretation of the data.

The most important usability features for a catalog are the search and filter options available. Boolean operators AND, OR, and NOT improve filtering within CVs or across fields, e.g. they could allow users to find all studies outside of a specific country, or studies that cover specifically early childhood and primary education. Mathematical operators ( _>_ , _≤_ , etc.) on dates and numerical entries can for example help find studies within certain time periods, or of a certain sample size. A WYSIWYG mask for constructing a search within and across fields helps first-time users, while a free-text entry field supporting advanced search functions makes the exact criteria applied replicable for other users, e.g. for meta-analysis purposes.

Just as with the metadata schema itself, standardization and free access are paramount for fostering the reuse of RCT data and promoting equitable access. We therefore encourage catalog implementers to make user access free and to use open-source programming to allow other catalog owners to adopt useful features.

21