Machine translation of anonymized documents with human-in-the-loop (2023)

Abstract: In this paper, we introduce a workflow that utilizes human-in-the-loop for post-editing anonymized texts, with the aim of reconciling the competing needs of data privacy and data quality. By combining the strengths of machine translation and human post-editing, our methodology facilitates the efficient and effective translation of anonymized texts, while ensuring the confidentiality of sensitive information. Our experimental results validate that this approach is capable of providing all necessary information to the translators for producing high-quality translations effectively. Overall, our workflow offers a promising solution for organizations seeking to achieve both data privacy and data quality in their translation processes.

## introduction almost five years ago, the european union, setting a milestone for data protection, enforced the general data protection regulation (gdpr). private and public organizations were required to remove sensitive content from public distribution involving european citizens under this legislation (european parliament and council of the european @xcite . text may need to be anonymized before it is translated to protect sensitive or confidential information. text anonymization is a critical step in protecting sensitive or confidential information before machine translation (mt). anonymization involves removing or disguising personally identifiable information or other sensitive data in a text to protect the privacy and confidentiality of individuals or organizations mentioned in the text @xcite . anonymization is particularly important in situations where the translated text may be viewed by individuals who are not authorized to access the sensitive information contained in the original text. for example, in the case of medical records or legal documents, it may be necessary to remove personally identifiable information to protect patient or client privacy @xcite . moreover, post-editing machine-translated texts is often required to ensure that the translation accurately conveys the intended meaning and tone of the original text. a human-in-the-loop workflow for post-editing machine-translated texts can improve the quality of the final translation by leveraging the strengths of both human and mt @xcite . by anonymizing the text before translation and utilizing post-editing workflows, the confidentiality and privacy of sensitive information can be maintained, while allowing the text to be effectively translated and used for its intended purpose. furthermore, mt incorporates the factor of speed, meaning that post-editing is faster than translating from scratch. in this paper, we propose a human-in-theloop workflow for post-editing machine-translated documents that have been anonymized to protect sensitive information. the proposed workflow leverages the strengths of both humans and mt to improve the quality of the final translation while ensuring that the privacy and confidentiality of sensitive information are maintained. ## related work an important area of research in mt is the development of techniques to handle sensitive or confidential information, such as medical records, legal texts, or bank documents. after conducting a thorough review of the relevant scientific literature in this field, it appears that no similar research has been carried out. despite the absence of similar studies, researchers endeavor to enhance the power of mt to translate confidential information through the utilization of dictionaries and terminologies, as demonstrated in the works of @xcite and @xcite . nevertheless, none of these studies involve the inclusion of human intervention in the process. conversely, there are some efforts from computer-assisted translation (cat) tools, such as xtm cloud, foot_0 that allow for the post-editing of anonymized texts. however, in the process, these tools replace named entities with numerical codes, which can sometimes cause confusion for translators and machines. an example of this type of anonymization can be seen in how the original text "john smith is a professor at stanford university" is transformed into "1 is a professor at 2". such anonymization methods can pose a challenge for both human and mt models in comprehending the text. one alternative approach could involve substituting the original text with labels such as "name", "location", etc. although this method may be superior to using codes, it still lacks vital details, such as whether the "name" label pertains to a male or a female. our research distinguishes itself from previous efforts by involving professional translators in the workflow to ensure that machine-translated output meets the standards of human translation. by working with meaningful sentence context and replacing sensitive information with fake data, both human translators and mt models can reduce the risk of errors and decrease the amount of time required for post-editing. this unique approach provides valuable insights into the role of human participation in mt and highlights the importance of considering human involvement in the development and implementation of ai-based technologies. in the context of our study, we introduce a workflow that combines the benefits of both humans and mt. it focuses on preserving the privacy and confidentiality of sensitive information while ensuring the accuracy of the final translation. the proposed workflow involves several key steps, including the initial mt of the anonymized text, followed by a human post-editing stage. the post-editor reviews a pseudo-anonymized machine-generated translation and makes the necessary corrections to ensure that the translation is accurate and conveys the intended meaning. then, the pseudo-anonymized text is replaced by the machine-translated text of the original text. for instance, consider a case where a medical report needs to be translated for a patient who is traveling to a different country for treatment. the report contains sensitive medical information that needs to be anonymized before translation. in this scenario, the anonymization process may result in the replacing of personal names, medical facility names, and location information with labels (e.g., "name", "location", etc.) or with alternatives (e.g., "angela" will be replaced with "maria", "london" will be replaced with "new york", etc.). as a result, the mt may generate text that lacks contextual information, making it challenging for the reader to accurately understand the intended meaning. following the anonymization of the text, a professional translator performs a post-editing task to ensure that the machine-generated translation accurately conveyed the intended meaning of the original text. the post-edited text is then subject to a final step, where an algorithm is used to replace the anonymized entities with their original versions in the translated text. by employing this approach, sensitive information is protected, and patient privacy is maintained throughout the translation process. in addition, the use of pseudo-anonymization eliminates biases, while allowing for accurate and contextually appropriate translations. following is a high-level overview of the postediting workflow for anonymized text: as part of our study, we leveraged the faker library to pseudo-anonymize the sensitive information found in the documents. the faker python library (version 9.1.4) allows us to generate realistic and anonymized data by creating fake names, addresses, and other personally identifiable information @xcite . we extended this, by utilizing the genderize python library (version 0.3.1), which uses probabilistic methods to predict the gender of a given name, enabling us to replace it with another name of the same gender @xcite . by employing this technique, the context required for an mt to comprehend and accurately translate the text is retained to the greatest extent possible. the anonymized text is fed into an mt system to generate a preliminary translation. this step provides a starting point for the human post-editor to work from. our research methodology is designed to be flexible to meet the varying needs of our study. to achieve this, we support both in-house mt frameworks (e.g., chatgpt-powered mt (openai, 2022), @xcite , marian (junczys-dowmunt et al., 2018), etc.) and publicly available providers such as google translate foot_2 to generate translations. to carry out the data replacement process, we use the awesome aligner (version 2.2) to align the words/phrases between the original source and the machine-translated text @xcite . this allows us to identify corresponding word pairs and accurately replace the pseudo-anonymized data with the machine-translated data of the original sentence. figure 1 illustrates the overall process flow of the architecture we have designed for mt of anonymized documents with human-in-the-loop. this architecture includes several components that work together to achieve this objective. the workflow can be further customized based on the specific needs of the project and the type of sensitive information present in the original text. it allows for accurate and contextually appropriate translations while preserving the privacy and confidentiality of sensitive information. it can also be enhanced by integrating cat tools with it. overall, the proposed workflow provides an effective solution for translating anonymized text while preserving the privacy and confidentiality of sensitive information. the use of both human and mt ensures high-quality translations that convey the intended meaning, which is particularly important in domains such as healthcare, legal, and financial sectors, where accuracy and confidentiality are critical. ## evaluation and results to evaluate the effectiveness of the proposed human-in-the-loop workflow for post-editing anonymized texts, we conducted a series of experiments using both objective and subjective measures. the crucial steps of our workflow are (1) the pseudo-anonymization of the entities with fake entities and (2) their replacement with the machine-translated versions of the original entities after the post-editing process. the evaluation was conducted by assessing individual sentences. for the subjective evaluation (step 1), we conducted a user study in which 14 participants of different nationalities (with a background in translation or linguistics) were asked to select which of the generated sentences better conveyed the original text. the participants had to choose among three options: the text that included pseudo-anonymized entities, the substitution with numeric codification, or the labeling codification. after carrying out the first part of this study, the participants were asked to provide their insights about the different methodologies used to anonymize the original text and the issues identified during the task concerning the postedition of the different alternatives. the test set used in our study comprised a diverse range of documents, including legal contracts, medical reports, and financial statements. to ensure a representative sample, we sourced the documents from multiple industries and geographic regions, resulting in a test set that was both comprehensive and challenging. in total, it contained 100 sentences with an average length of 15 words per sentence. the shortest sentence in the test set was 3 words long, while the longest sentence had 45 words. our test set consisted of various types of entities including 60 person names including subdomains, all of which were carefully annotated for an accurate analysis. we took steps to ensure that the test set did not contain any duplicated entities, to prevent any potential bias or skewing of results. the results of this subjective evaluation show that the pseudo-anonymized text and the labeling codification were considered the most appropriate options even though some issues were highlighted. when analyzing the answers, we found out that several subjects chose multiple options, pseudoanonymization, and labeling codification being the most frequent. after checking the comments, we realized that some of the issues could be avoided by using different post-processes after the pseudoanonymization is performed. a list of pros and cons for each of the main options selected is provided below. in addition, some of the above-mentioned issues and the corresponding post-processes suggested to avoid the problem are explained too. probably, new processes will arise once the workflow is used in production. pros and cons of using pseudo-anonymization pros: pros and cons of using labeling codification pros: regarding the objective evaluation (step 2), the participants were provided with different postedited alternatives of an original text which included the machine-translated entities replacement. each alternative results from a different anonymized option (anonymization with numeric codes, labeling codification, or pseudoanonymization). they were first anonymized, machine-translated, and then, post-edited, and original sentence numeric codification pseudo-anonymization labeling codification to contact the office of scientific integrity, call (404) 639-7570 or send an email to oads@cdc.gov. to contact the {1}, call {2} or send an email to {3}. to contact the office of foreign affairs, call (345) 636-7545 or send an email to dfg@ghu.gov. to contact the [org], call @xcite or send an email to @xcite . considering minor mistakes those which do not affect the meaning (grammar, word order, etc.), and major mistakes those affecting the meaning (mistranslation, omission, addition, etc.), the subjects had to rate the quality of each resulting translation based on the following scale: the results shown in the table indicates that the text with pseudoanonymized entities received significantly higher ratings (@xmath0.33) compared to the text with other codifications (labeling @xmath1.14, and numeric codification @xmath2.91), which suggests that the pseudo-anonymized entities better preserved the meaning and characteristics of the original text. the primary objective of these evaluations was to determine whether the pseudo-anonymized entities preserved the full meaning, i.e., gender and other characteristics of the original text. this evaluation enabled us to ensure that the pseudo-anonymized entities did not introduce any unintended biases or distortions to the original text. as part of the second step of our evaluation process, we asked 5 professional translators to post-edit the pseudo-anonymized versions of the original text into spanish and german. following this, our algorithm replaced the pseudoanonymized entities with the machine-translated versions of the original text. as mentioned above, the resultant output was verified by them, who examined whether the de-anonymized version was linguistically proficient as if they themselves had translated the anonymized entities. this process allowed us to validate the effectiveness of our methodology and assess its suitability for the study. by verifying the data replacement, we were able to identify any areas for improvement and refine our approach to ensure its accuracy. results provided us with valuable feedback on the strengths and limitations of our methodology, enabling us to develop a more reliable and effective approach for future research in this area. in general, the translators provided us with positive feedback for all the target languages. for spanish, it was reported that pseudoanonymization was clear enough to produce a correct and accurate text which always kept the intended meaning after replacing the anonymized text. the other two anonymization options introduced sometimes misleading information. for instance, in one of the sentences a nationality had been anonymized with the label @xcite , which caused a concordance issue in the final version of the translation. for german, the reported observations were similar to those for spanish. in this case, a problem related to pronoun use and inflection was reported due to the anonymization of "thames". when using the label [loc] or a numeric code, there was no information about the type of place, while with pseudo-anonymization, the post-editor got "seine" instead, and could choose the proper pronoun and article, as well as their correct declination. overall, the experimental results demonstrate that the proposed human-in-the-loop workflow for post-editing anonymized documents can significantly improve translation quality while reducing the workload of human post-editors. although our workflow has yielded promising results, it is important to acknowledge the risk that machine translation may not accurately capture the intended meaning of entities in the original text, which could result in mistranslations. furthermore, the automated alignment process may also be prone to inaccuracies, which could further compound these risks. ## conclusion in conclusion, the human-in-the-loop workflow for post-editing anonymized documents offers a promising solution for organizations seeking to balance the competing demands of data privacy and data quality. our research represents a significant innovation in the field of mt and post-editing, as it utilizes cutting-edge techniques and is the first of its kind to be presented. by leveraging the strengths of both mt and human post-editing, our workflow enables efficient and effective translation of anonymized texts while preserving the confidentiality of sensitive information. our experimental findings indicate that our approach is effective in reducing the risk of a human translator accessing sensitive information during the translation process. we hope that our work will inspire further research on this topic and contribute to the development of more robust and efficient workflows for post-editing anonymized texts with human involvement.