Using Machine Learning to Validate a Novel Taxonomy of Phenomenal Translation States (2024)

Abstract: We report an experiment in which we use machine learning to validate the empirical objectivity of a novel annotation taxonomy for behavioral translation data. The HOF taxonomy defines three translation states according to which a human translator can be in a state of Orientation (O), Hesitation (H) or in a Flow state (F). We aim at validating the taxonomy based on a manually annotated dataset that consists of six English-Spanish translation sessions (approx 900 words) and 1813 HOF-annotated Activity Units (AUs). Two annotators annotated the data and obtain high average inter-annotator accuracy 0.76 (kappa 0.88). We train two classifiers, a Multi-layer Perceptron (MLP) and a Random Forest (RF) on the annotated data and tested on held-out data. The classifiers perform well on the annotated data and thus confirm the epistemological objectivity of the annotation taxonomy. Interestingly, inter-classifier accuracy scores are higher than between the two human annotators.

## introduction translation is considered to involve complex and non-linear cognitive processes @xcite . understanding the intricacies of the temporal dynamics of these processes is a fundamental aspect in translation process research (tpr). various approaches have been proposed over the past 40 years to understand the distinct phases and mental states experienced by translators (jakobsen, 2017). starting with think-aloud protocols in the 1980s, in which translators comment their own translation behavior during their translations @xcite , the field of enquiry has moved towards less invasive technologies, that is, keystroke logging and eye tracking @xcite . the recordings of these logging tools make it possible to assess the flow of translation in a seamless way and to investigate how translations evolve in time; where translators type smoothly, where they get stuck, and where they search for (external) resources, etc. one approach to analysing the translation process has been to segment the behavioral translation process data (tpd) into processing units @xcite . but how these segments should be defined and what they exactly represent has been a topic of continuous exploration and debate. the assessment of the translation rhythm (aka "pause analysis" (kumpulainen, 2015; muñoz and apfelthaler, 2022)) has provided valuable insights into translation patterns as produced by more or less experienced translators @xcite , for different levels of text complexity @xcite , for different translation goals @xcite , post-editing behavior @xcite and also for spoken translation (e.g., interpretation, sight translation, @xcite ). the underlying assumption has been that longer keystroke pauses are indicative of more challenging translations, while stretches of smooth typing can be observed when there are no/less translation hurdles or difficulties @xcite . however, determining an exact pause threshold to differentiate these phenomena remains a challenge. many studies @xcite @xcite among others) present varying segmentation methods with different pause thresholds, ranging from 300ms to five seconds or more. these strands of research rely on deterministic fragmentation techniques to segment the key logging and eye tracking data into translation units (tus, @xcite ) or activity units (aus, @xcite ). however, these approaches lack intuitive labeling and thus make it difficult to derive a comprehensive understanding of the complex nature of the translation process and how it unfolds over time. some researchers suggest a hierarchical process model @xcite and others (muñoz and apfelthaler, 2022; dragsted, 2010) advocate a translator-specific fragmentation of the tpd into processing units depending on the translators' typing speed. combined, this suggests that human translation processes are embedded in a hierarchical mental architecture, encapsulating various processing strata. a hierarchical approach to understanding translation has the potential to offer a nuanced understanding of translators' behavior and strategies on various interacting levels of analysis. in order to advance this project, a novel higher-level segmentation taxonomy was introduced in @xcite that fragments the tpd in three broad phenomenal translation states, hesitation (h), orientation (o) and translation flow (f). the hof taxonomy assumes that behavioral traces of these three states can be observed in the tpd and that translators can be at any one point in time in only one of the three states. in previous work @xcite we have annotated a small corpus with hof translation states. the corpus is publicly available as part of the critt tpr-db foot_0 . the hof annotation corpus provides a layer of manual annotation, introducing segment labels of an assumed phenomenal layer of translation processes, suited to analyse the hierarchical embedding of translation processes. however, the hof taxonomy, capturing qualities of conscious translator experience, is entirely new territory and the genaralizability and validity of the annotation schema is still unclear. in this paper, we therefore conduct further investigation to assess whether and to what extent this new taxonomy is epistemologically valid -that is, we want to investigate to what extent different annotators might agree the hof states to represent a phenomenal translation "reality". while a manual annotation has shown a varying amount of agreement between two annotators (kappa 0.37 and 0.88, @xcite , in this paper we use ml techniques to further validate the consistency of the taxonomy. ai and ml techniques can be used in various ways and for different purposes. mollo (2024), for instance, enumerates a few scenarios where ai can be used as: ai-as-engineering (industrial and commercial projects, as e.g., mt systems), ai-aspsychology to improve our understanding of biological intelligence, ai-as-idea or ai-as-recreation for recreating biological intelligence in artificial systems. ai can also be used for "exploring intelligence spaces" so as to uncover new forms of intelligence that are different from human intelligence or to uncover algorithms (zhong et al., 2023). in this paper we use ml techniques to investigate the "epistemological objectivity" (searle, 1998; searle, 2017, see also section 5 for a discussion) of the hof annotation schema. that is, we are interested in verifying whether ml can reproduce the results of our manual annotations to a similar amount of accuracy. we assume that if the trained models performs well on the classification task, it confirms the objectivity of the annotation taxonomy used to create the training data. conversely, poor model performance would indicate issues with the annotation taxonomy. for instance, it might be the case that, even though two annotators agree in their annotation label, they might be biased by some intuition, cultural or otherwise un-observable features which may not be accessible to the ml technology. however, if ml reaches similar results of accuracy as the inter-annotator agreement indicates, we take the annotation taxonomy to implement reproducible and epistemological objective annotation criteria. that is, as ml lacks subjective, personal or cultural influences in the process annotation process (i.e., hof state labeling), high accuracy on held-out testing data may be an indicator of stable results with minimum bias, in section 2 we describe the data and the manual annotation process of the reference (training) data. section 3 describes our implementation of two classifiers -multi-layer perception (mlp) and random forest (rf) -while section 4 reports our training and evaluation on a set of 1813 hof-annotated aus. we report higher precision and recall between the classifier and annotator as compared to inter-annotator agreement between the two annotators. section 5 concludes with a discussion on searle's notion of "ontological subjectivity" and "epistemological objectivity" and their relation to the evaluation of our hof states. ## activity units, translation units and translation states the translation process can be conceptualized as a dynamic flow of mental processes marked by information input as gathered through eye movements and textual output in the form of keystrokes or mouse movements. as translators navigate the source text (st) and produce the target text (tt), their behavior is influenced by numerous factors. to better understand these processes, different approaches have been pursued that fragment translation-behavioral data (keystrokes and gaze data) into processing units. figure 1 shows a progression graph that depicts three ways of segmenting the approximately 28 seconds of the plotted translation session. tus (carl and kay, 2011; alves and vale, 2009), indicated in the top in figure 1 , are characterized by a typing pause (a blank space in top line of the figure ) followed by a typing burst (or production unit, pu) indicated as grey boxes. aus @xcite are constructed based on the coordination of gaze activities and typing behavior and are marked at the bottom in figure 1 in different colors. three distinct hof translation states are indicated with black dotted lines @xcite . boundaries of translation states coincide with au boundaries, so that sequences of aus can be used to fragment the tpd into hof translation states. this section explores these constructs and the three annotations, emphasizing their significance and interplay. ## training translation state classifiers in this section we describe two classifiers that were used to assess the annotated translation states. while the variation of inter-annotator agreement, as reported in ## evaluation of classifiers the training of au-to-state classifiers is based on six annotated sessions from phase 3 and 5 with 1813 datapoints (aus), as shown in in a 10-fold cross validation with rf and mlp we get best average accuracy values of 0.85 for the two classifiers. ## discussion and conclusion human translation is a complex cognitive process that involves numerous interacting processes. to understand and analyse these processes, one approach to translation process research (tpr) has been to collect and synchronize behavioral data (keystrokes and gaze data) from translation sessions and to segment the flow of data into various kinds of processing units. several automatic segmentation approaches have been suggested, but as the labels often lack intuitive understanding it is difficult to interpret the data. a novel higher-order hof taxonomy has been proposed @xcite that segments the data into three phenomenal states in which a translator can be: a state of orientation (o) accounts for the • reality as "we" describe it: norms, conventions (money, marriage, etc.) a small corpus of behavioral data annotated and released @xcite . the annotated data consists of six english-spanish translation sessions (approximately 900 words) and 1813 hofstate annotated activity units (aus, @xcite ). two annotators annotated the data with hof labels and -after specifying a decision tree and annotation guidelines -annotators reached a good inter-rater agreement. given the novelty of the annotation taxonomy, we investigate how well the hof annotations can be reproduced. we use machine learning (ml) classifiers to validate the "epistemological objectivity" @xcite of the annotation schema. that is, we deploy a multi-layer perceptron and a random forests classifier to assess the "objectivity" of the manual annotations, where high accuracy of the ml classifiers would indicate the validity of the underlying hof annotations taxonomy. in his discussion about "modes of existence", @xcite makes a distinction between, on the one hand, subjective and objective ways of understanding and, on the other hand, between the epistemology and the ontology of knowledge and reality (see ontological subjectivity then refers to the idea that subjective experiences is a form of reality, but there may not be an independent, objective reality beyond these subjective constructions (see despite the fact that consciousness has a subjective mode of existence-and is thus not directly accessible to scientific inquiry-searle claims that this does not prevent us from having an epistemological objective science of consciousness. while translators experience subjective states of orientation, hesitation and flow, these states, we assume, can be recovered in the behavioral data and studied under epistemically objective conditions. norms, regulations or-as in our case the hof annotation taxonomy-can be understood, deployed and objectively verified within observable tpd in a given context. our results suggest, however, that there might be a gradual slope between epistemological subjective and epistemological objective modes of existence, rather than a binary one. aus information. the features sgnbr and tgnbr indicate how many source and target words were covered in the au, while crosss and crosst are measures of the distance / reordering between the source and the target words @xcite . four of the au features refer to the nearby context of the au. type is type of au (see in this study, we define effort, effect, and significance for an au to depend on the type and the duration of the two preceding two aus and the next au. the effort of an au is computed as the sum of log(dur(au)) for each context-au of type t1, t2 or t8 (no keystroke activity is observed). the effect is computed as the sum of log(dur(au)) for each context-au of type t4, t5 or t6. the significance of an au is then its effect minus its effort, so that more significant aus are characterized by longer stretches of text production.