the work occurred, such as the timing, duration, intent, and notable features of wrangling session. As in an elicitation study, I also instructed participants to record their work in a minimal form, such that there would be enough detail to recall the issue (such as the name of a difficult tag) without requiring them to stop and explain the complexities of the situation at that time. The diary protocol appears in Appendix A. The diary protocol I distributed to participants is an event-contingent protocol. Event-contingent protocols are ideal for capturing each instance of an event, particularly when that event does not conform to a predictable schedule (Iida et al., 2012; Ohly, Sonnentag, Niessen, & Zapf, 2010). I instructed participants to complete the form for each session of wrangling tasks, with an understanding that participants could make adjustments for styles of wrangling that were not conducive to this pattern (e.g., several short periods of work throughout the day). Diary studies carry two considerable drawbacks: difficulty and burden (Carter & Mankoff, 2005; Iida et al., 2012; Ohly et al., 2010). They carry a risk common to any protocol participants complete independently, namely that the participants might misunderstand a difficult protocol and fill it out incorrectly. One way to alleviate potential difficulty is to provide a sample, instructional diary to participants. Doing so risks participants conforming to the example, which is counter to the goal of capturing participants’ diverse experiences and understandings of their work. I took an opposite approach in designing a diary protocol with few constraints outside of a few factual questions (e.g., “What time is it?”). The protocol was open-ended and allows for participant interpretation. For example, the instruction to record “an example of a complex wrangling task” did not define “complex,” but instead allowed participants to define this term for themselves. Nina provided the following examples of wrangling tasks
Cape names such as “Nightwing” and “Red X” were not merely nicknames equivalent to “Robin” but other unique identities of Dick Grayson, complete with their own costumes and storylines. As Victoria confirmed, this hierarchy meant that, in user retrieval, selecting “Dick Grayson” would return all works users had tagged with any of these four character tags. Figure 4.4: Hierarchy of Dick Grayson character names Hierarchical relationships such as this one resolved some of the ambiguity of a street name corresponding to multiple cape However, due to plot details in the DCU source material, wranglers needed to create a more complex hierarchy. Specifically, Victoria noted that “there’s five Robins,” meaning that at least one of the cape names in Figure 4.1 corresponded to multiple street names. That is, just as Dick Grayson has been many superheroes, many men and women have been Robin. Because TFR users tagged for multiple Robin street names from the DCU, wranglers who followed the above hierarchy had to create a polyhierarchy to account for overlapping street name/cape name relationships. In a classic hierarchy, subtags would have only one metatag, though metatags could have many subtags, as in Figure 4.1. In a polyhierarchy, subtags could have many metatags, as in Figure 4.2. The example hierarchy in Figure 4.2 includes more—but not all—of the Robin street names wranglers linked to the cape name. In this more complete hierarchy, “Carrie Kelley” and “Damian Wayne” are also metatags to “Robin,” and the additional cape name “Catgirl” is a subtag to “Carrie Kelley.” “Robin” is the only cape name shared among the three street names.
“Bandom” at TFR On TFR, “Bandom” was a popular fandom tag largely interchangeable with the “Music RPF” (Real Person Fiction) tag. Wranglers placed it within “Celebrities & Real People” and “Music & Bands,” top-level categories—not tags—meant to organize fandom tags by media type. The placement of “Bandom” was parallel to the fandom tag “Music RPF,” though in the common-sense definition and in wranglers’ discussions, “Bandom” was a subset of “Music RPF” fandom. In other words, while “Music RPF” would refer to works about any musicians, “Bandom” would refer only to bands and not to solo musicians or classical composers, thereby being a subset of “Music RPF.” Wranglers designated fandom tags as direct subtags of either or both tags or as subtags to more specific genres below “Music RPF.” For example, “Wolfgang Amadeus Mozart” is a subtag of “Classical Music RPF,” which is a subtag of “Music RPF.” “Pink Floyd” is a subtag of both “Bandom” and “Music RPF,” and “Twenty One Pilots” is a subtag only of “Bandom.” “Bandom” therefore functioned as a superwork, collecting related works that shared the quality of being Bandom fandoms. Because “Bandom” served both as a subset and as a parallel term to “Music RPF,” there was an imperfect overlap between the “Music RPF” and “Bandom” tags, as shown in Figure 5.1. As of March 2017, a search for “Music RPF” returned 33,948 fanworks, with very few (739) tagged with “Music RPF” itself. In the vast majority of cases, users had tagged with more specific subtags (e.g., “Wolfgang Amadeus Mozart”) and not used the “Music RPF” tag itself. The usage of Bandom was much different. A search for “Bandom” returned 36,641 fanworks. For 13,258 of those fanworks (more than users had tagged for “Bandom” directly. The usage of Bandom suggests a different meaning on the part of users, as users appeared to treat “Music RPF” as a category and
Future research in this area should examine the extent to which users differentiate between algorithmic and human judgment modes of large-scale classification design. In particular, such research should examine whether systems in which the human judgment is more open and participatory—as in Stack Overflow and LibraryThing’s instantiations—are perceived differently than instantiations such as TFR’s, in which the decision making is behind-the-scenes. I suggest that, as with other classification design decisions, the extent to which the system makes human judgment open and participatory—or, conversely, hidden and selective—would reflect other values in the community. To that end, in my closing theme I address the extent to which we might view classification systems and their design as a research lens in their own right. Inherent in my argument that the classification system and the approach to classification design at TFR reflected the values of the community is the idea that the reverse is also true; evidence of the values of TFR’s community were found in classification design. Though my questions in this study were about the design process and not about the community, future research could take the reverse approach. Indeed, studies of labelling and naming have provided persuasive and provocative insights into other cultures, such as in Lakoff’s (1987) account of gendered nouns in Women, Fire, and Dangerous Things. Systematic studies of community values that incorporate studies of how a community agrees to organize and label their knowledge and their documents will reveal a more complete picture of that community. I argued here that the classification system and its community were complementary, especially as the system reflected the community’s of inclusivity. However, there is the possibility that research into other settings would find less