Should I disclose my dataset? Caveats between reproducibility and individual data rights (2022)

Abstract: Natural language processing techniques have helped domain experts solve legal problems. Digital availability of court documents increases possibilities for researchers, who can access them as a source for building datasets — whose disclosure is aligned with good reproducibility practices in computational research. Large and digitized court systems, such as the Brazilian one, are prone to be explored in that sense. However, personal data protection laws impose restrictions on data exposure and state principles about which researchers should be mindful. Special caution must be taken in cases with human rights violations, such as gender discrimination, over which we elaborate as an example of interest. We present legal and ethical considerations on the issue, as well as guidelines for researchers dealing with this kind of data and deciding whether to disclose it.

## introduction the increasing availability of data in digital formats, along with the means to process and interpret it, has boosted the interest in its versatile use. the enriched commercial value of personal data has justified the adoption of personal data protection laws aiming to protect individual and collective rights. having such a legal structure -with a broader social recognition of implications associated with personal data usage -demands that data controllers be mindful about ethical issues and legal liabilities when dealing with this resource. research agents have been major controllers of data on individuals. while science has always relied on data, the societal switch to digitalintensive structures has changed much of their nature, amount, and availability. this context calls for specific approaches from researchers when balancing individual rights and scientific reproducibility -since disclosing datasets, while beneficial for research publicity, might expose information over which special considerations might apply. computational research based on data-intensive frameworks, such as machine learning, typically operates over collecting, processing, and interpreting large amounts of data; being used to awareness of resource sharing, the computing community tends to encourage reproducibility practices. in experimental contexts, that usually means disclosing descriptions of methods and results and codes, tools, and data. digital data can come from many sources. when derived from the realm of the social sciences it is often produced in text form, which motivates its use as input for natural language processing methods. social scientists have relied on computational approaches to help answer some of their research questions; in the legal domain, court documents often provide rich material, which computational tools allow to be analyzed on improved scales. among the many inquiries that large-scale analysis of court documents could help address, we are particularly interested in gender-related ones. examples include: (a) which role do gender biases play in decisions regarding gender-based violence (gbv) legal cases? (b) how many cases are linked to the same victim? (c) how many police investigations make it to court? these are research questions for which natural language processing methods seem suitable. domain experts often identify demands for this research while exploring their own areas, creating communities around common issues. a large community of researchers and practitioners interested in how computational approaches can be used to address questions in the legal domain has emerged in brazil; the country is one of the most litigious of the world in the court, having one lawyer for each batch of around 160 people @xcite , and approximately 80 million active legal cases @xcite . with a substantial court system, large databases of documents issued by such courts, and an engaged research community in the field, brazil emerges as a legal data hotspot -with many issues regarding data disclosure from state entities and researchers. the country issued its general data protection act in 2018, based on european's general data protection regulation, which expanded the debate on such issues. focus on gbv-related cases is justified not only by research and human rights significance but also due to the amount of delicate personal information they carry on the subjects involved, meaning that disclosing them without regard for legal and ethical principles could implicate severe harm. while focusing on this context, we stress that our considerations might apply to others. a similar observation should be made for the location we chose to highlight. focusing on the brazilian context will benefit its large community of researchers and practitioners interested in the field. it may also provide useful insights from other legal settings -particularly civil law ones (e.g., continental europe), in which brazilian legal structures and fundamental statutes are heavily based. our main contributions are: we hope to help the community of interested researchers and practitioners understand the fundamentals of the brazilian data protection legal system and its caveats. this paper is organized as follows. section 2 introduces research reproducibility concepts. it is followed by discussions on data disclosure and publicity in section 3, where we present legal principles, practical issues, and ethical concerns on the matter. risk assessment and mitigation measures are described in section 4. sections 3 and 4 also suggest guidelines of good practices for researchers. finally, section 5 summarizes approaches that could help researchers address concerns on disclosure of court documents and similar data. ## reproducibility reproducibility has been at the core of the debate on scientific integrity, being recognized as a critical quality of modern research (goodman et al., 2016; baker, 2016; loscalzo, 2012). the concept is open enough to evoke debate on its meaning, and comprises different aspects of scientific soundness and accountability (goodman et al., 2016; drummond, 2009); however, there appears to be some consensus on the importance of community scrutiny for research quality assessment, for which reproducibility is essential. scrutiny, fraud prevention, and fraud detection are not the only motivation behind efforts toward reproducible research. science is a collective endeavor of public interest; therefore, resourcesharing strengthens networks, creates research possibilities, and helps build connections inside and between communities -not only for science itself but also for practitioners and society as a whole. this is especially true in empirical research, as is usually the case in computer science. in fact, many efforts have been made towards fostering a culture of openness of resources inside the computingrelated community -from free and open source software initiatives @xcite to open science guidelines and frameworks (wilkinson et al., 2016; peng, 2011; sonnenburg et al., 2007). peng (2011) describes a reproducibility spectrum for computer science research in which the gold standard would be attained by publishing linked and executable code and data along with results. in some fields, such as machine learning, the importance of empirical choices behind results that might support decision-making processes is such that it could justify one arguing that reproducibility is as important of a property as the research results themselves. in this context, data sharing and quality assessment emerge as an object of concern as well (gebru et al., 2021; de schutter, 2010; blockeel and vanschoren, 2007). data collecting, cleaning, labeling, and/or processing are often part of the experimental pipeline in machine learning research, which justifies interest in making them available for peers and stakeholders. in some cases, however, the means and extent to which data should be shared are not trivial decisions. when individual rights of the subjects regarded in the dataset might be at stake, sharing this data becomes a challenge since adjustments -or even the decision not to share -might be needed to avoid legal and/or ethical violations. privacy, for instance, is one of the main concerns (pröell et al., 2015). some domains, such as health and clinic research, are notably prone to this issue. when faced with such a situation, researchers must take legal and ethical boundaries into account, assess the risks involved in disclosing the data, and weigh them against the benefits of reproducibility. ## issues on disclosing data provided by courts particularities on data sharing emerge in the context of research that uses computational approaches to court decisions. this section delves into some of them from the perspective of our research example: exploring natural language processing and other computational techniques over brazilian court decisions in gbv-related cases. however, as mentioned in section 1, our considerations might also be helpful for other contexts. ## risk assessment and mitigation when faced with the decision to disclose court documents used in research, one must confront risks against the benefits of science replicability since full disclosure might potentially harm and violate the rights of the subjects whose personal data is displayed. risks can exist regardless of legal restrictions, given that records from courts typically carry a large amount of personal information of parties, witnesses, and other subjects related to the case, both in the document(s) text and metadata. making personal data available establishes as liable the person or entity in charge of the disclosure, who becomes a controller according to law (gdpr, article 4(7); lgpd, article 5 vi) (schwait-zer, 2021). as a controller, a researcher or research agency operates under distinct ethical guidelines than those of courts and law enforcement agencies -which, when disclosing personal data, are usually complying with a legal duty to transparency and publicity, as well as broader public interest principles. while carrying public interest on its own, science reproducibility is not a legal obligation (thus not dismissing liability in the same way that applies to state entities), and can be acceptably achieved with mitigation-mindful data availability when full disclosure is not allowed or advised. further, the legal system context represents a special circumstance for personal data disclosing due to implications regarding rights of access to justice, due process of law, and defense -which also relates to publicity and transparency. one would be unable to build a defense if not provided with complete information on the case, including data on parties and their lawyers, allegations, documents, and evidence. transparency of court documents is generally a matter of state accountability. imposing severe constraints against this kind of publicity could have noxious outcomes for democratic settings and is not the same as restricting personal data disclosure in scientific frameworks. in that sense, although documents used in research might be publicly available in other sources (e.g., court websites), their disclosure by researchers can increase risks for the subjects involved, considering that: (a) it reunites the data in a single, cohesive source, often cleaner, and more structured than the original and combined with annotation and metadata, therefore making it easier for different groups of people to access it and make inferences from it; (b) public status of such documents in original sources might change over time, adding an extra layer of harm-related responsibility on the researcher who decides to disclose them. in the context of gbv-related cases, risks of full personal data disclosure by researchers or research agencies include: in fact, such risks have been used to advocate for initiatives such as bill 3333/20, whose main proposal is to establish "absolute secrecy" for personal information displayed in police reports and court documents in cases of domestic violencewhich are currently public by default. if approved, alleged aggressors would be hindered from accessing personal data on the victim(s), thus impairing their right to defense. for researchers, this would add a class of documents in the secrecy-justified caution cluster. exposing sensitive and/or confidential data can increase the possibilities of rights restrictions, retaliation from a subject's community and institutions, and physical and mental suffering. let us consider, for instance, the disclosure of the lgbtq+ status of a subject implicated in a legal case: such a deed could have discrimination-related consequences such as the loss of a job, impairment of social and family ties, or threats to one's physical integrity. ease of access to data obtained from courts allows for inferences that would hardly be made otherwise -an exciting possibility for good-faith researchers and policymakers but also a cautioninspiring scenario. from a dataset of brazilian court decisions with specific characteristics, for example, one could extract a map of precise addresses of victims, defendants, or plaintiffs (some of which could be minors or belong to other protected groups). an ill-motivated, technically capable agent could use that information to perpetrate physical, moral, emotional, or other kinds of harm to these people -and, while there are legal provisions to make perpetrators accountable, some damages might be beyond repair. we note that the risks mentioned above do not constitute an exhaustive list; ideally, researchers should evaluate which issues might apply to their context and know their data enough to build a proper risk assessment in order to decide on the extent of data disclosure considering available resources and both ethical and legal restrictions. when personal information is part of the data source in research, mitigating such risks is possible and advised. risks are usually associated with data disclosure rather than their use itself. personal data protection laws ordinarily do not distinguish use from disclosure for legal purposes, placing both operations under the concept of "processing" (see note 10). however, discerning them is relevant in our context of interest. while using court documents in research settings (e.g., as input for training models or to perform other quantitative and qualitative analyses) does not directly threaten or pose harm to subjects involved, disclosing them without taking prior mitigation actions might do. we identify three levels of personal data implication for our context: mitigation measures to protect personal data embedded in public court documents might include several actions from researchers and research agencies, who should evaluate the risks of data disclosure, benefits of full replicability, and availability of resources to perform mitigation. we stress two of them: (a) anonymization and (b) disclosure by demand with a deed of the undertaking. anonymization when personal data is anonymized, it is no longer considered personal data (lgpd, article 12; gdpr, recital 26)therefore, none of the issues discussed in this work would apply, and documents containing them could be disclosed, ab initio, without legal nor ethical implications. to be considered fully anonymized, personal identification corresponding to the data must be untraceable and not reversible by reasonable efforts foot_11 ; thus, pseudonymizationwhich allows for identification to be restored -, while allowed to comply with legal guidelines on data storage, is not enough to allow full disclosure. there are, however, practical obstacles. full anonymization is not always attainable since it might require massive manual efforts or the use of technically challenging tools, which do not necessarily guarantee complete accuracy. some kinds of data are challenging to anonymize; computational research often deals with large amounts of documents and sensitive information is usually nonstructurally embedded in the text, meaning that masking them pre-disclosure -or even identifying them -might not be possible. deeper discussions on technical and juridical aspects of legal data anonymization can be found in the works of csányi et al. (2021) and van opijnen et al. (2017). regarding replicability, anonymization barely affects it unless the personal information is relevant for the analysis. in some cases, determining the relevance of personal information for experimental settings is overly demanding and/or outside of the scope of research, e.g., when black-box models learn from input documents. in those scenarios, approaches for model interpretability and/or explainability might be taken into consideration (rudin, 2019a,b; molnar, 2022). at any rate, if research results and code are duly published and the methodology is thoroughly explained, reproducibility should not be severely disturbed. assuming that the documents used as the source are publicly available, anyone following the same procedures should be able to access them, therefore claiming their responsibility upon processing the data. if mitigation is needed or advised, but adequate anonymization is not feasible, researchers should consider mitigation measures described next. disclosure by demand in this case, the person, group, or entity responsible for research provides a contact channel through which the data can be requested and sent by demand. ideally, whoever requests the material should agree to a deed of undertaking bound by the good faith of parties, with clauses preventing inappropriate data processing and protecting the subjects' best interest. traceability of data controllers is a major advantage of this method. while being the safest option regarding personal data protection, we identify the following caveats: (a) it relies on assuming good faith of the researchers; (b) it constrains replicability, given that it adds extra layers of compromise, bureaucracy, and communication for interested parties. also, mitigation measures (a) and (b) could be combined, although this would require extra effort. researchers can still decide not to make data available, therefore escaping from the burden of responsibility over the dataset disclosure and choosing privateness over publicity. ## possible paths both research reproducibility and data protection of subjects mentioned are essential values in democratic settings and must be preserved and encouraged. good research practices and awareness of legal and ethical restrictions can help researchers and agencies decide whether -and to which extent -disclose their court documents datasets. while much of the responsibility for the form and availability of such documents relies on the courts, researchers also have liability over the content they choose to disclose. the following approaches could help them address it in the future.