LEDA : a Large-Organization Email-Based Decision-Dialogue-Act Analysis Dataset (2023)

Abstract: Collaboration increasingly happens online. This is especially true for large groups working on global tasks, with collaborators all around the globe. The size and distributed nature of such groups makes decision-making challenging. This paper proposes a set of dialog acts for the study of decision-making mechanisms in such groups, and provides a new annotated dataset based on real-world data from the public mail-archives of one such organisation – the Internet Engineering Task Force (IETF). We provide an initial data analysis showing that this dataset can be used to better understand decision-making in such organisations. Finally, we experiment with a preliminary transformer-based dialog act tagging model.

## introduction and related work motivation online collaboration has been used for many years by large distributed organizations. the increasing availability of high-speed internet connections and collaboration tools, along with the covid-19 pandemic, are making it ever more prevalent. large distributed organizations of this type often undertake important tasks. for example, the internet engineering task force (ietf) and the world wide web consortium (w3c) are responsible for developing the technical standards that underpin the internet. consequently, understanding the decision-making processes in this type of organization is essential to increase transparency and accountability, to facilitate tracking of decisions and the reasoning behind them, and to understand alternatives that were considered (or not) and the voices that were (or were not) heard. goals most studies of decision making in text (e.g. @xcite @xcite rely on annotation and analysis of dialogue acts (das). we adopt this approach and label emails from public ietf mailing lists with das. our aim is to answer the following research questions: rq1: what is an appropriate set of das to use for this annotation task?; rq2: how do communication patterns change through the life-cycle of a decision discussion?; and rq3: how do different types of participants differ in how they contribute to the process? the overall goal of these questions is to better understand the mechanisms underlying the decision-making process in a large, distributed, collaborative organization. ## dataset our data consists of emails from the ietf mailing list archive. 1 the ietf is a typical example of decision making in a large, distributed, online collaborative community; it has rich metadata available via the ietf datatracker; foot_1 and the data is publicly available with appropriate consent. 3 ietf background the ietf is a large, open, voluntary organization tasked with developing internet standards @xcite @xcite . it is comprised of working groups (wgs), each focusing on a relatively narrow field: e.g., rmcat foot_3 wg focuses on specific real-time media congestion avoidance techniques. each wg has one or more participants as chairs. during its development, an internet standard is called a draft. drafts are discussed in the mailing lists (the archive @xmath0, predominantly in english, between 56k participants over 20 years) and in several live meetings yearly. after sufficient revision and review, a draft becomes an internet standard. ## automated dialogue act tagging we provide a preliminary da tagging model to investigate the predictability of our da tags, and to serve as a baseline for future work. we use a hierarchical sequence model, inspired by work in da tagging for spoken dialogue (e.g. @xcite : the input is a sequence of segments (each one a sequence of words), and the output is a sequence of predictions, one 14-dimensional vector for each input segment, representing da probabilities. each input segment is encoded into a vector; we use the [cls] token of bert @xcite . the sequence of segment vectors is then passed to a bidirectional-lstm (hochreiter and schmidhuber, 1997); each bilstm hidden state vector is passed through a linear layer (shared for all time steps) to produce the output prediction vector sequence. the loss function is binary cross-entropy averaged across all labels and all elements of the sequence. the model is implemented using pytorch @xcite and scikit-learn @xcite . we used a learning rate of 2 -5 , batchsize of 32, and lstm hidden-layer size of 256. all other hyper-parameters are left at default values. we experiment with two variants of bert: bertbase and bert-base-ietf (fine-tuned using language modeling loss on the entire ietf mail archive). we split the data into train (60%), validation (20%), and test threads (20%). we report results on test threads by the model best on the validation threads. the input sequences for the model are the possible root-to-leaf paths in the input threads, following (e.g. zubiaga et al., 2016). 8 results are given in curiously, bert-base-ietf performs comparably to or worse than bert-base. we hypothesize the reason for this may be the specific language of the ietf (technical discussions). it may cause the additional language model training step to make the bert-baseietf model forget information generally useful for da tagging. on the other hand, this information is retained in bert-base. if this is the case, it would hurt the performance of bert-base-ietf after further fine-tuning on the da tagging task. however, we leave investigation of this and other hypotheses for this unexpected result to future work. ## conclusion we have presented a taxonomy of dialogue acts (das) and a labeled dataset of emails. moreover, we provided a data analysis and a preliminary da prediction model. we hope this dataset will be useful to facilitate further research on the interaction behavior of participants in online collaboration settings. future work could include a more detailed investigation into the underlying reasons for the observed trends. another possibility is looking into the interaction of das and the participant interaction graph as described by @xcite . finally, to get further insights, it would be interesting to annotate segments of with a particular da with additional labels, e.g., explicit/implicit for agreement or different sub-types of question.