Towards Task-based Personal Information Management Evaluations David Elsweiler Department Computer and Information Sciences, University of Strathclyde dce@cis.strath.ac.uk Ian Ruthven Department Computer and Information Sciences, University of Strathclyde ir@cis.strath.ac.uk ABSTRACT Personal Information Management (PIM) is a rapidly growing area of research concerned with how people store, manage and re-find information.
A feature of PIM research is that many systems have been designed to assist users manage and re-find information, but very few have been evaluated.
This has been noted by several scholars and explained by the difficulties involved in performing PIM evaluations.
The difficulties include that people re-find information from within unique personal collections; researchers know little about the tasks that cause people to re-find information; and numerous privacy issues concerning personal information.
In this paper we aim to facilitate PIM evaluations by addressing each of these difficulties.
In the first part, we present a diary study of information re-finding tasks.
The study examines the kind of tasks that require users to re-find information and produces a taxonomy of re-finding tasks for email messages and web pages.
In the second part, we propose a task-based evaluation methodology based on our findings and examine the feasibility of the approach using two different methods of task creation.
Categories and Subject Descriptors H3.3 [Information Search and Retrieval]: General Terms Measurement,Management,Experimentation, Human Factors 1.
INTRODUCTION Personal Information Management (PIM) is a rapidly growing area of research concerned with how people store, manage and re-find information.
PIM systems - the methods and procedures by which people handle, categorize, and retrieve information on a day-to-day basis [18] - are becoming increasingly popular.
However the evaluation of these PIM systems is problematic.
One of the main difficulties is caused by the personal nature of PIM.
People collect information as a natural consequence of completing other tasks.
This means that the collections people generate are unique to them alone and the information within a collection is intrinsically linked with the owner``s personal experiences.
As personal collections are unique, we cannot create evaluation tasks that are applicable to all participants in an evaluation.
Secondly, personal collections may contain information that the participants are uncomfortable sharing within an evaluation.
The precise nature of this information - what information individuals would prefer to keep private - varies across individuals making it difficult to base search tasks on the contents of individual collections.
Therefore, experimenters face a number of challenges in order to conduct realistic but controlled PIM evaluations.
A particular feature of PIM research is that many systems have been designed to assist users with managing and re-finding their information, but very few have been evaluated; a situation noted by several scholars [1, 6, 7].
Recently, however, researchers have started to focus on ways to address the problem of PIM evaluation.
For example, Kelly [16] proposes that numerous methodologies must be taken to examine and understand the many issues involved in PIM, although, she makes explicit reference to the need for laboratory based PIM studies and a common set of shared tasks to make this possible.
Capra [6] also identifies the need for controlled PIM lab evaluations to complement other evaluation techniques, placing specific emphasis on the need to understand PIM behaviour at the task level.
In this paper, we attempt to address the difficulties involved to faciliate controlled laboratory PIM evaluations.
In the first part of this paper we present a diary study of information re-finding tasks.
The study examines the kind of tasks that require users to re-find information and produces a taxonomy of re-finding tasks for email messages and web pages.
We also look at the features of the tasks that make re-finding difficult.
In the second part, we propose a task-based evaluation methodology based on our findings and examine the feasibility of the approach using different methods of task creation.
Thus, this paper offers two contributions to the field: an increased understanding of PIM behaviour at the task level and an evaluation method that will facilitate further investigations.
2.
RELATED WORK A variety of approaches are available to study PIM.
Naturalistic approaches study participants performing naturally, completing their own tasks as they occur, within familiar environments.
These approaches allow researchers to overcome many of the difficulties caused by the personal nature of PIM.
As the tasks performed are real and not simulated, the participants can utilise their own experiences, previous knowledge and information collections to complete the tasks.
A benefit of the approach is that data can be captured continuously over extended time periods and measurements can be taken at fixed points in time within these [15].
Naturalistic approaches can be applied by conducting fieldwork [17, 8], ethnographic methods as suggested by [15] or via log file analysis [9, 7].
Both ethnographic and fieldwork methods require the presence of an experimenter to assess how PIM is performed, which raises a number of issues.
Firstly, evaluation in this way is expensive; taking long time periods to study small numbers of participants and these small samples may not be representative of the behaviour of larger populations.
Secondly, because participants cannot be continually observed, experimenters must choose when to observe and this may affect the findings.
An alternative strategy to conducting naturalistic evaluations is to utilise log file analysis.
This approach makes use of logging software that captures a broad sampling of user activities in the context of natural use of a system.
In [9] a novel PIM search tool was deployed to 234 users and the log data provided detailed information about the nature of user queries, interactions with the query interface and about properties of the items retrieved.
Log file analysis is a powerful methodology as it allows the capture of a large quantity of detailed information about how users behave with the system without the expense and distracting influence of an observer.
Nevertheless, there are limitations to this strategy.
Firstly, to attain useful results, the deployed prototype must be something that people would use i.e. it has to be a fully functional piece of software that offers improvement on the systems ordinarily available to participants.
Developing a research prototype to this standard is beyond the resources of many researchers.
Further, caution must be taken when analysing logs, as the captured data shows nothing about the goals and intentions that the user had at the time.
It is, therefore, difficult to make any concrete statements about the reasons for the behaviour depicted in the logs.
This reveals a need to complement naturalistic studies with controlled experiments where the experimenter can relate the behaviour of study participants to goals associated with known search tasks.
Laboratory-based studies simulate users'' real world environment in the controlled setting of the laboratory, offering the ability to study issues that are tightly defined and narrow in scope.
One difficulty in performing this kind of evaluation is sourcing collections to evaluate.
Kelly [16] proposes the introduction of a shared test collection that would provide sharable, reusable data sets, tasks and metrics for those interested in conducting PIM research.
This may be useful for testing algorithms in a way similar to TREC in mainstream IR [13].
However, a shared collection would be unsuitable for user studies because it would not be possible to incorporate the personal aspects of PIM while using a common, unfamiliar collection.
One alternative approach is to ask users to provide their own information collections to simulate familiar environments within the lab.
This approach has been applied to study the re-finding of personal photographs [11], email messages [20], and web-bookmarks [21].
The usefulness of this approach depends on how easy it is to transfer the collection or gain remote access.
Another solution is to use the entire web as a collection when studying web page re-finding [4].
This may be appropriate for studying web page re-finding because previous studies have shown that people often use web search engines for this purpose [5].
A second difficulty in performing PIM laboratory studies is creating tasks for participants to perform that can be solved by searching a shared or personal collection.
Tasks relate to the activity that results in a need for information [14] and are acknowledged to be important in determining user behaviour [26].
A large body of work has been carried out to understand the nature of tasks and how the type of task influences user information seeking behaviour.
For example, tasks have been categorised in terms of increasing complexity [3] and task complexity has been suggested to affect how searchers perceive their information needs [25] and how they try to find information [3].
Other previous work has provided methodologies that allow the simulation of tasks when studying information seeking behaviour [2].
However, little is known about the kinds of tasks that cause people to search their personal stores or re-find information that they have seen before.
Consequently, it is difficult to devise simulated work task situations for PIM.
The exception is the study of personal photograph management, where Rodden``s work on categorising personal photograph search tasks has facilitated the creation of simulated work task situations [22].
There have been other suggestions as to how to classify PIM tasks.
For example, [5] asked participants to classify tasks based on how frequently they perform the task type in their daily life and how familiar they were with the location of the sought after information and several scholars have classified information objects by the frequency of their use e.g. [24].
While these are interesting properties that may affect how a task will be performed, they do not give experimenters enough scope to devise tasks.
Personal collections are one reason why task creation is so difficult.
Rodden``s photo task taxonomy provides a solution here because it allows tasks, tailored to private collections to be categorised.
Systems can then be compared across task types for different users [11].
Unfortunately, no equivalent taxonomy exists for other types of information object.
Further, other types of object are more sensitive to privacy than photographs; it is unlikely that participants would be as content to allow researchers to browse their email collections to create tasks as they were with photographs in [11].
This presents a serious problem - how can researchers devise tasks that correspond to private collections without an understanding of the kinds of tasks people perform or jeopardising the privacy of study participants?
A few methods have been proposed.
For example, [20] studied email search by asking participants to re-find emails that had been sent to every member in a department; allowing the same tasks to be used for all of the study participants.
This approach ensured that privacy issues were avoided and participants could use things that they remember to complete tasks.
Nevertheless, the systems were only tested using one type of task - participants were asked to find single emails, each of which shared common properties.
In section 4 we show that people perform a wider range of email re-finding tasks than this.
In [4], generic search tasks were artificially created by running evaluations over two sessions.
In the first session, participants were asked to complete work tasks that involved finding some unknown information.
In the second session, participants completed the same tasks again, which naturally involved some re-finding behaviour.
The limitations of this technique are that it does not allow participants to exploit any personal connections with the information because the information they are looking for may not correspond to any other aspect of their lives.
Further, if time is utilised by a system or interface being tested the approach is unsuitable because all of the objects found in the first session will have been accessed within the same time period.
Our review of evaluation approaches motivates a requirement for controlled laboratory experiments that allow tightly defined aspects of systems or interfaces to be tested.
Unfortunately, it has also been shown that there are difficulties involved in performing this type of evaluation - it is difficult to source collections and to devise tasks that correspond to private collections, while at the same time protect the privacy of the study participants.
In the following section we present a diary study of refinding tasks for email and web pages.
The outcome is a classification of tasks similar to that devised by Rodden for personal photographs [22].
In section 5 we build on this work by examining methods for creating tasks that do not compromise the privacy of participants and discuss how our work can facilitate task-based PIM user evaluations.
We show that by collecting tasks using electronic diaries, not only can we learn about the tasks that cause people to re-find personal information, but we can learn about the contents of private collections without compromising the privacy of the participants.
This knowledge can then be used to construct tasks for use in PIM evaluations.
3.
METHOD Diary Studies are a naturalistic technique, offering the ability to capture factual data, in a natural setting, without the distracting influence of an observer.
Limitations of the technique include difficulties in maintaining participant dedication levels and convincing participants that seemingly mundane information is useful and should be reported [19].
[12] suggest that the effects of the negatives can be limited, however, with careful design and good implementation.
In our diary study, we followed the suggestions in [12] to achieve the best possible data.
To this end, we restricted the recorded tasks to web and email re-finding.
By asking users to record fewer tasks it was anticipated that participant apathy would be reduced and dedication levels maintained.
The participants were provided with a personalised web form in which they could record details about their information needs and the contexts in which these needs developed.
Web forms were deployed rather than paperbased diaries because to re-find web and email information the user would be at a computer with an Internet connection and there would be no need to search for a paper-based diary and pen.
The diary form solicited the following information: whether the information need related to re-finding a web page or an email message and a description of the task they are performing.
This description was to contain both the information that the participant wished to find and the reason that they needed the information.
To help with this, the form gave three example task descriptions, which were also explained verbally to each participant during an introductory session.
The experimenter ensured that the participants understood that the tasks to be recorded were not limited to the types shown in the examples.
The examples were supplied purely to get participants thinking about the kinds of things they could record and to show the level of and type of details expected.
The form also asked participants to rate each task in terms of difficulty (on a scale from 1-5, where 1 was very easy and 5 was very hard).
Finally, they were asked when was the last time they looked at the sought after information.
Again, they were able to choose from 5 options (less than a day ago, less than a week ago, less than a month ago, less than a year ago, more than a year ago).
Time information was used to examine the frequency with which the participants re-found old and new information, and when combined with difficulty ratings created a picture of whether or not the time period between accessing and re-accessing impacted on how difficult the participants perceived tasks to be.
36 participants, recruited by mass advertisement through departmental communication channels, research group meetings and undergraduate lectures, were asked to digitally record details of their information re-finding tasks over a period of approximately 3 weeks.
The final population consisted of 4 academic staff members, 8 research staff members, 6 research students and 18 undergraduate students.
The ages of participants ranged from 19-59.
As both personal and work tasks were recorded, the results collected cover a broad range of re-finding tasks.
4.
RESULTS Several analyses were performed on the captured data.
The following sections present the findings.
Firstly, we examine the kinds of re-finding tasks that were performed both when searching on email and on the web.
Next, we consider the distribution of tasks - which kinds of tasks were performed most often by participants.
Lastly, we explore the kinds of re-finding tasks that participants perceived as difficult.
4.1 Nature of Web and Email Re-finding Tasks During the study 412 tasks were recorded.
150 (36.41%) of these tasks were email based, 262 (63.59%) were webbased.
As with most diary studies, the number of tasks recorded varied extensively between particpants.
The median number of tasks per participant was 8 (interquartile range (IQR)=9.5).
More web tasks (median=5,IQR=7.5) were recorded than email tasks (median=3, IQR=3).
This means that on average each participant recorded approximately one task every two days.
From the descriptions supplied by the participants, we found similar features in the recorded tasks for both email and web re-finding.
Based on this observation a joint classification scheme was devised, encompassing both email and web tasks.
The tasks were classified as one of three types: lookup tasks, item tasks and multi-item tasks.
Lookup tasks involve searching for specific information from within a resource, for example an email or a web page, where the resource may or may not be known.
Some recorded examples of lookup tasks were: • LU1: Looking for the course code for a class - it``s used in a script that is run to set up a practical.
I``d previously obtained this about 3 weeks ago from our website.
• LU2: I am trying to determine the date by which I step down as an External Examiner.
This is in an email somewhere • LU3: Looking for description of log format from system R developed for student project.
I think he sent me in it an email Item tasks involve looking for a particular email or web page, perhaps to pass on to someone else or when the entire contents are needed to complete the task.
Some recorded examples of item tasks were: • I1: Looking for SIGIR 2002 paper to give to another student • I2: Find the receipt of an online airline purchase required to claim expenses • I3: I need the peer evaluation forms for the MIA class E sent me them by email To clarify, lookup tasks differ from item tasks in two ways - in the quantity of information required and in what the user knows about what they are looking for.
Lookup tasks involve a need for a small piece of information e.g. a phone number or an ingredient, and the user may or may not know exactly the resource that contains this information.
In item tasks the user knows exactly the resource they are looking for and needs the entire contents of that resource.
Multi-item tasks were tasks that required information that was contained within numerous web pages or email messages.
Often these tasks required the user to process or collate the information in order to solve the task.
Some recorded examples were: • MI1: Looking for obituaries and other material on the novelist John Fowles, who died at the weekend.
Accessed the online Guradian and IMES • MI2: Trying to find details on Piccolo graphics framework.
Remind myself of what it is and what it does.
Looking to build a GUI within Eclipse • MI3: I am trying to file my emails regarding IPM and I am looking for any emails from or about this journal There were a number of tasks that were difficult to classify.
For example, consider the following recorded task: • LU4: re-find AS``s paper on graded relevance assessments because I want to see how she presented her results for a paper I am writing This task actually consists of two sub-tasks: 1 item task(refind the paper) and 1 lookup task (look for specific information within the paper).
It was decided to treat this as a lookup task because the user``s ultimate goal was to access and use the information within the resource.
There were a number of examples of combined tasks, mainly of the form item then lookup, but there were also examples of item then multi-item.
For example: • MI4: re-find Kelkoo website so that I can re-check the prices of hair-straighteners for my girlfriend A second source of ambiguity came from tasks such as finding an email containing a URL as a means of re-accessing a web page.
It was also decided to categorise these as lookup tasks because in all cases these were logged by participants as email searches and, within this context, what they were looking for was information within an email.
Another problem was that some of the logs lacked the detail required to perform a categorisation e.g. • U1: searching for how to retrieve user``s selection from a message box.
Decided to use some other means Such tasks were labelled as U for unclassifiable.
To verify the consistency of the taxonomy, the tasks were recategorised by the same researcher after a delay of two weeks.
The agreement between the results of the two analyses was largely consistent (96.8%).
Further, we asked a researcher with no knowledge of the project or the field to classify a sample of 50 tasks.
The second researcher achieved a 90% agreement.
We feel that this high agreement on a large number of tasks by more than one researcher provides evidence for the reliability of the classification scheme.
The distribution of task types is shown in table 1.
Overall, lookup and item tasks were the most common, with multiitem tasks only representing 8.98% of those recorded.
The distribution of the task types was different for web and email re-finding.
The majority of email tasks (60%) involved looking for information within an email (lookup), in contrast to web tasks where the majority of tasks (52.67%) involved looking for a single web page (item).
Another distinction was the number of recorded multi-item tasks for web and email.
Multi-item tasks were very rare for email re-finding (only 2.67% of email tasks involved searching for multiple resources), but comparatively common for web re-finding (12.6%).
Lookup Item Multi-item Unclass.
Email 90(60%) 52(34.67%) 4(2.67%) 4(2.67%) Web 87(33.21%) 138(52.67%) 33(12.60%) 4(1.53%) All 177(42.96%) 190(46.12%) 37(8.98%) 8(1.94%) Table 1: The distribution of task types In addition to the three-way classification described above, the recorded tasks were classified with respect to the temperature metaphor proposed by [24], which classifies information as one of three temperatures: hot, warm and cold.
We classified the tasks using the form data.
Information that had been seen less than a day or less than a week before the task were defined as hot, information that had been seen less than a month before the task as warm, and information that had been seen less than a year or more than a year before the task as cold.
Unfortunately, a technical difficulty with the form only allowed 335(81.3%) of the tasks to be classified.
The remainder were defined as U for unclassifiable.
A cross-tabulation of task types and temperatures is shown in table 2.
Hot Warm Cold Unclass.
Email 50(33.33%) 36(24.00%) 37(24.67%) 27(18%) Web 112(42.75%) 60(22.90%) 40(15.27%) 50(19.08%) All 162(39.32%) 96(23.30%) 77(18.69%) 77(18.69%) Table 2: The distribution of temperatures Most of the tasks that caused people to re-find web pages (42.75%) and email messages (33.33%) involved searching for information that has been accessed in the last week.
However there were also a number of re-finding tasks that involved searching for older information: 23.30% of the tasks recorded (24.00% for email and 22.90% for web) involved searching for information accessed in the last month and 18.69% of the tasks recorded (24.67% for email and 15.27% for web) were looking for even older information.
This is important with respect to evaluation because there is psychological evidence suggesting that people remember less over time e.g. [23].
This means that users may find searching for older information more difficult or perhaps alter their seeking strategy when looking for hot, warm or cold information.
4.2 What tasks are difficult?
We looked for patterns in the recorded data to determine if certain tasks were perceived as more difficult than others.
For example, we examined whether the media type affected how difficult the participants perceived the task to be.
There was no evidence that participants found either email (median=2 IQR=2) or web (median=2 IQR=2) tasks more difficult.
We also investigated whether the type of task or the length of time between accessing and re-accessing made a task more difficult.
Figure 1 shows this information graphically.
Figure 1: Difficulty ratings for task types From figure 1, it does not appear that any particular task type was perceived as difficult with respect to the others, although there is a suggestion that lookup tasks were perceived more difficult when looking for cold information than hot and item tasks were perceived more difficult for warm information than hot.
To assess the relationship between information temperature and the perceived difficulty, we used Mood``s median tests to determine whether the rank of difficulty scores was in agreement for the information temperatures being compared (p<0.05).
For the look-up task data, there was evidence that hot tasks were perceived easier than cold (p=0.0001) and that warm tasks were perceived easier than cold tasks(p=0.0041), but there was no evidence to distinguish between the difficulty ratings of hot and warm tasks(p=0.593).
For the item task data, there was evidence that hot and cold tasks were rated differently (p=0.024), but no evidence to distinguish between hot and warm tasks(p=0.05) or warm and cold tasks(p=0.272).
These tests confirm that the length of time between accessing and re-accessing the sought after information indeed influenced how difficult participants perceived the task to be.
Nevertheless, the large number of tasks of all types and temperatures rated by participants as easy i.e. < 3, suggests that there are other factors that influence how difficult a task is perceived to be.
To learn about these factors would require the kind of user evaluations proposed by [16, 6] - the kind of evaluations facilitated by our work.
4.3 Summary In the first part of this paper, we described a diary study of web and email re-finding tasks.
We examined the types of task that caused the participants to search their personal stores and found three main categories of task: tasks where the user requires specific information from within a single resource, tasks where a single resource is required, and tasks that require information to be recovered from multiple resources.
It was discovered that look-up and item tasks were recorded with greater frequency than multi-item tasks.
Although no evidence was found that web or email tasks were more difficult, there was some evidence showing that the time between accessing and re-accessing affected how difficult the participants perceived tasks to be.
These findings have implications for evaluating PIM behaviour at the task level.
The remainder of this paper concentrates on this, discussing what the findings mean with respect to performing task-based PIM user evaluations.
5.
TASK-BASED PIM EVALUATIONS The findings described in section 4 are useful with respect to evaluation because they provide experimenters with enough knowledge to conduct controlled user evaluations in lab conditions.
Greco-Latin square experimental designs can be constructed where participants are assigned n tasks of the three types described above to perform on their own collections using x systems.
This would allow the performance of the systems or the behaviour of the participants using different systems to be analysed with respect to the type of task being performed (look-up, item, or multi-item).
In the following sections we evaluate the feasibility of this approach when employing different methods of task creation.
5.1 Using Real Tasks One method of creating realistic re-finding tasks without compromising the privacy of participants is to use real tasks.
Diary-studies, similar to that described above, would allow experimenters to capture a pool of tasks for participants to complete by searching on their own collections.
This is extremely advantageous because it would allow experimenters to evaluate the behaviour of real users, completing real search tasks on real collections while in a controlled environment.
There is also the additional benefit that the task descriptions would not make any assumptions about what the user would remember in a real life situation because they would only include the information that had been recorded i.e. the information that was available when the user originally performed the task.
Nevertheless, to gain these benefits we must, firstly, confirm that the task descriptions recorded are of sufficient quality to enable the task to be re-performed at a later date.
Secondly, we must ensure that a diary-study would provide experimenters with enough tasks to construct a balanced experimental design that would satisfy their data needs.
To examine the quality of recorded tasks, 6 weeks after the diary study had completed, we asked 6 of our participants, selected randomly from the pool of those who recorded enough tasks, to re-perform 5 of their own tasks.
The tasks were selected randomly from the pool of those available.
The issued tasks consisted of 10 email and 20 web tasks, 9 of which were lookup tasks, 12 were item tasks, and 8 were multi-item tasks.
The issued tasks represented a broad-sampling of the complete set of recorded tasks.
They also included tasks with vague descriptions e.g. • LU5:Find a software key for an application I required to reinstall.
• LU6:Trying to find a quote to use in a paper.
Cannot remember the person or the exact quote The usefulness of such tasks would rely on the memories of participants i.e. would the recorder of task LU5 remember which application he referred to and would the recorder of LU6 remember enough about the context in which the task took place to re-perform the task?
Presented with the tasks exactly as they recorded them, the participants were asked to re-perform each task with any system of their choice.
Of the 30 tasks issued, 26 (86.67%) were completed without problems, 2 (6.67%) of the tasks were not completed because the description recorded was insufficent to recreate the task, and 2 tasks (6.67%) were not completed because the task was too difficult or the required web page no longer existed.
Experimenters are likely to be interested in the final group of tasks because it is important to discover what makes a task difficult and how user behaviour changes in these circumstances.
Therefore, from the 30 tasks tested, only 2 tasks were not of sufficient quality to be used in an evaluation situation.
Further, there did not seem to be any issue of the type, temperature or difficulty ratings affecting the quality of the task descriptions.
These findings suggest that the participants who recorded most tasks in the diary study also recorded tasks with sufficient quality.
However, did the diary study generate enough tasks to satisfy the needs of experimenters?
Participant Tasks Lookup Item Multi-item Unclass.
10 26 16 8 2 0 43 9 4 5 0 0 26 9 5 4 0 0 8 9 8 1 0 0 40 8 5 3 0 0 18 7 3 4 0 0 4 6 5 1 0 0 7 6 5 0 1 0 12 5 4 0 0 1 22 5 4 1 0 0 36 5 0 5 0 0 46 5 2 2 0 1 3 5 3 2 0 0 Table 3: The quantities of recorded email tasks Participant Tasks Lookup Item Multi-item Unclass.
26 32 7 20 5 0 32 31 11 18 2 0 10 19 0 10 7 2 33 18 5 13 0 0 5 15 0 7 2 4 8 11 0 6 5 0 22 10 0 3 5 2 28 10 1 7 2 0 37 10 1 9 0 0 35 9 7 2 0 0 6 9 0 1 8 0 40 7 1 5 1 0 9 7 0 0 5 2 12 7 1 0 3 2 42 6 0 4 2 0 29 6 0 3 3 0 15 5 0 2 1 2 4 5 0 4 1 0 43 5 2 3 0 0 18 5 0 0 3 2 Table 4: The quantities of recorded web tasks Naturally the exact number of tasks required to perform a user evaluation will depend on the goals of the evaluation, the number of users and the number of systems to be tested etc..
However, for illustrative purposes we chose 5 tasks as a cut-off point for our data.
From tables 3 and 4, which show the quantities of email and web tasks recorded for each participant, we can see that of the 36 participants, only 13 (36.1%) recorded 5 or more email tasks and 20 (55.6%) recorded 5 or more web tasks.
This means that many of the recruited participants could not actually participate in the final evaluation.
This is a major limitation of using recorded tasks in evaluations because participant recruitment for user tests is challenging and it may not be possible to recruit enough participants if experimenters lose between half and two-thirds of their populations.
Further, there was some imbalance in the numbers of recorded tasks of different types.
Some participants recorded several lookup tasks but very few item tasks and others recorded several item tasks but few lookup tasks.
There was also a specific lack of multi-item email tasks.
This situation makes it very difficult for experimenters to prepare balanced experimental designs.
Therefore, even though our first test suggests that the quality of recorded tasks was sufficient for the participants to re-perform the tasks at a later stage, the number of tasks recorded was probably too low to make this a viable option for experimental task creation.
However, it may be possible to increase the number of tasks recorded by frequently reminding participants or by making personal visits etc. 5.2 Using Simulated Tasks Based on Real Tasks Another benefit of diary-studies is that they provide information about the contents and uses of private collections without invading participants'' privacy.
This section explores the possibility of using a combination of the knowledge gained from diary studies and other attributes known about participants to artificially create re-finding tasks corresponding to the taxonomy defined in section 4.1.
We explain the techniques used and demonstrate the feasibility of creating simulated tasks within the context of a user evaluation investigating email re-finding behaviour.
Space limitations prevent us from reporting our findings; instead we concentrate on the methods of task creation.
As preparation for the evaluation, we performed a second diary-study, where 34 new participants, consisting of 16 post-graduate students and 18 under-graduate students, recorded 150 email tasks over a period of approximately 3 weeks.
The collected data revealed several patterns that helped with the creation of artificial tasks.
For example, students in both groups recorded tasks relating to classes that they were taking at the time and often different participants recorded tasks that involved searching for the same information.
This was useful because it provided us with a clue that even though some of the participants did not record a particular task, it was possible that the task may still be applicable to their collections.
Other patterns revealed included that students within the same group often searched for emails containing announcements from the same source.
For example, several undergraduate students recorded tasks that included re-finding information relating to job vacancies.
There were also tasks that were recorded by participants in both groups.
For example, searching for an email that would re-confirm the pin code required to access the computer labs.
To supplement our knowledge of the participants'' email collections, we asked 2 participants from each group to provide email tours.
These consisted of short 5-10 minute sessions, where participants were asked to explain why they use email, who sends them email, and their organisational strategies.
This approach has been used successfully in the past as a non-intrusive means to learn about how people store and maintain their personal information [17].
Originally, we had planned to ask more participants to provide tours, but we found 2 tours per group was sufficient for our needs.
Again, patterns emerged that helped with task creation.
We found content overlap within and between groups that confirmed many of our observations from the diary study data.
For example, the students who gave tours revealed that they received emails from lecturers for particular class assignments, receipts for completed assignments, and various announcements from systems support and about job vacancies.
Importantly, the participants were also able to confirm which other students had received the same information.
This confirmed that many of tasks recorded during the diary study were applicable, not only to the recorder, but to every participant in 1 or both groups.
Based on this initial investigatory work, a set of 15 tasks (5 of each type in our taxonomy) was created for each group of participants.
We also created a set of tasks for a third group of participants that consisted of research and academic staff members, based on our knowledge of the emails our colleagues receive.
Where possible we used the information recorded in the diary study descriptions to provide a context for the task i.e. a work task or motivation that would require the task to be performed.
When the diary study data did not provide sufficient context information to supply the participants with a robust description of the information need, we created simulated work task situations according to the guidelines of [2].
A further advantage of using simulated tasks in this way, rather than real-tasks, is that some of the users will not have performed the task in the recent past and this allows the examination of tasks that look for information of different temperatures.
If only real-tasks had been used all of the participants would have performed the tasks during the period of the diary study.
The created tasks were used in a final evaluation, where we examined the email re-finding behaviour of users with three different email systems.
21 users (7 in each group) performed 9 tasks each (1 task of each type on each system) using their own personal collections in a Greco-Latin square experimental design.
Performing a PIM evaluation in this way allowed the examination of re-finding behaviour in a way not possible before - we were able to observe the email re-finding strategies employed by real users, performing realistic tasks, on their own collections in a controlled environment.
The study revealed that the participants remembered different attributes of emails, demostrated different finding behaviour, and exhibited different levels of performance when asked to complete tasks of the different types in the taxonomy.
The key to both the task creation and the analysis of the results was our taxonomy, which provided the template to create tasks and also a means to compare the behaviour and performance of different users (and systems) performing different tasks of the same type.
Some of the findings of the evaluation will be published in [10].
Summarising the approach, to conduct a user experiment using our methodology, researchers would be required to perform the following steps: 1)Conduct a diary study as above 1 .
2)Analyse the recorded tasks looking for overlap between the participants.
3)Supplement the gained knowledge about the contents of participants'' collections by asking a selection of the participants to provide a tour of their collection.
4)Use the knowledge gained to devise tasks of the three different types defined within the taxonomy.
More de1 Information about this and the diary forms required can be found at http://www.cis.strath.ac.uk/˜dce/PIMevaluations tailed information on how to use the research described in this paper to perform task-based PIM evaluations can be found at our website (see footnote 1).
6.
CONCLUSIONS This paper has focused on overcoming the difficulties involved in performing PIM evaluations.
The personal nature of PIM means that it is difficult to construct balanced experiments because participants each have their own unique collections that are self-generated by completing other tasks.
We suggested that to incorporate the personal aspects of PIM in evaluations, the performance of systems or users should be examined when users complete tasks on their own collections.
This approach itself has problems because task creation for personal collections is difficult: researchers don``t know much about the kinds of re-finding tasks people perform and they don``t know what information is within individual personal collections.
In this paper we described ways of overcoming these challenges to facilitate task based PIM user evaluations.
In the first part of the paper we performed a diary study that examined the tasks that caused people to re-find email messages and web pages.
The collected data included a wide range of both work and non-work related tasks, and based on the data we created a taxonomy of web and email re-finding tasks.
We discovered that people perform three main types of re-finding task: tasks that require specific information from within a single resource, tasks that require a single complete resource, and tasks that require information to be recovered from multiple resources.
In the second part of the paper, we discussed the significance of the taxonomy with respect to PIM evaluation.
We demonstrated that balanced experiments could be conducted comparing system or user performance on the task categories within the taxonomy.
We also suggested two methods of creating tasks that can be completed on personal collections.
These methods do not compromise the privacy of study participants.
We examined the techniques suggested, firstly by simulating an experimental situation - participants were asked to re-perform their own tasks as they recorded them, and secondly, in the context of a full evaluation.
Performing evaluations in this way will allow systems that have been proposed to improve users'' ability to manage and re-find their information to be tested, so that we can learn about the needs and desires of users.
Thus, this paper has offered two contributions to the field: an increased understanding of PIM behaviour at the task level and an evaluation method that will facilitate further investigations.
7.
ACKNOWLEDGMENTS We would like to thank Dr Mark Baillie for his insightful comments and help analysing the data.
8.
REFERENCES [1] R. Boardman, Improving tool support for personal information management, Ph.D. thesis, Imperial College London, 2004.
[2] P. Borlund, The iir evaluation model: A framework for evaluation of interactive information retrieval systems, Information Research 8 (2003), no. 3, paper no. 152.
[3] K. Bystr¨om and K. J¨arvelin, Task complexity affects information seeking and use, Information Processing and Management 31 (1995), no. 2, 191-213.
[4] R. G. Capra and M. A. Perez-Quinones, Re-finding found things: An exploratory study of how users re-find information, Tech.
report, Virginia Tech, 2003.
[5] R. G. Capra and M. A. Perez-Quinones, Using web search engines to find and refind information, Computer 38 (2005), no. 10, 36-42.
[6] R. G. Capra and M. A. Perez-Quinones, Factors and evaluation of refinding behaviors., SIGIR 2006 Workshop on Personal Information Management, August 10-11, 2006, Seattle, Washington, 2006.
[7] E. Cutrell, D.Robbins, S.Dumais, and R.Sarin, Fast, flexible filtering with phlat, Proc.
SIGCHI ``06 (New York, NY, USA), ACM Press, 2006, pp. 261-270.
[8] M. Czerwinski, E. Horvitz, and S. Wilhite, A diary study of task switching and interruptions, Proc.
SIGCHI ``04, 2004, pp. 175-182.
[9] S. Dumais, E. Cutrell, J. Cadiz, G. Jancke, R. Sarin, and D.C. Robbins, Stuff i``ve seen: a system for personal information retrieval and re-use, Proc.
SIGIR ``03:, 2003, pp. 72-79.
[10] D. Elsweiler and I. Ruthven, Memory and email re-finding, In preparation for ACM TOIS CFP special issue on Keeping, Re-finding, and Sharing Personal Information (2007).
[11] D. Elsweiler, I. Ruthven, and C. Jones, Dealing with fragmented recollection of context in information management, Context-Based Information Retrieval (CIR-05) Workshop in CONTEXT-05, 2005.
[12] D. Elsweiler, I. Ruthven, and C. Jones, Towards memory supporting personal information management tools, (to appear in) Journal of the American Society for Information Science and Technology (2007).
[13] D. Harman, What we have learned, and not learned, from trec, Proc.
ECIR 2000, 2000.
[14] P. Ingwersen, Information retrieval interaction, Taylor Graham, 1992.
[15] D. Kelly, B. Bederson, M. Czerwinski, J. Gemmell, W. Pratt, and M. Skeels (eds.)
, Pim workshop report: Measurement and design, 2005.
[16] D. Kelly and J. Teevan, (to appear in) personal information management, ch.
Understanding what works: Evaluating personal information management tools, Seattle: University of Washington Press., 2007.
[17] B. H. Kwasnik, How a personal document``s intended use or purpose affects its classification in an office, SIGIR``89 23 (1989), no.
SI, 207-210.
[18] M.W. Lansdale, The psychology of personal information management., Appl Ergon 19 (1988), no. 1, 55-66.
[19] L. Palen and M. Salzman, Voice-mail diary studies for naturalistic data capture under mobile conditions, CSCW ``02: Proceedings of the 2002 ACM conference on Computer supported cooperative work, 2002.
[20] M. Ringel, E. Cutrell, S. Dumais, and E. Horvitz, Milestones in time: The value of landmarks in retrieving information from personal stores., Proc.
INTERACT 2003, 2003.
[21] G. Robertson, M. Czerwinski, K. Larson, D. C. Robbins, D. Thiel, and M. van Dantzich, Data mountain: using spatial memory for document management, Proc.
UIST ``98:, 1998.
[22] K. Rodden, How do people organise their photographs, BCS IRSG 21st Annual Colloquium on Information Retrieval Research,Glasgow, Scotland, 1999.
[23] D.C. Rubin and A.E. Wenzel, One hundred years of forgetting: A quantitative description of retention, Psychological Bulletin 103 (1996), 734-760.
[24] A. J. Sellen and R. H. R. Harper, The myth of the paperless office, MIT Press, Cambridge, MA, USA, 2003.
[25] P. Vakkari, Task complexity, problem structure and information actions: Integrating studies in on information seeking and retrieval., Information Processing and Management 35 (1999), 819-837.
[26] P. Vakkari, A theory of task-based information retrieval, Journal of Documentation 57 (2001), no. 1, 44-60.

Towards Task-based Personal Information Management Evaluations
ABSTRACT
Personal Information Management (PIM) is a rapidly growing area of research concerned with how people store, manage and re-find information.
A feature of PIM research is that many systems have been designed to assist users manage and re-find information, but very few have been evaluated.
This has been noted by several scholars and explained by the difficulties involved in performing PIM evaluations.
The difficulties include that people re-find information from within unique personal collections; researchers know little about the tasks that cause people to re-find information; and numerous privacy issues concerning personal information.
In this paper we aim to facilitate PIM evaluations by addressing each of these difficulties.
In the first part, we present a diary study of information re-finding tasks.
The study examines the kind of tasks that require users to re-find information and produces a taxonomy of re-finding tasks for email messages and web pages.
In the second part, we propose a task-based evaluation methodology based on our findings and examine the feasibility of the approach using two different methods of task creation.
1.
INTRODUCTION
Personal Information Management (PIM) is a rapidly growing area of research concerned with how people store, manage and re-find information.
PIM systems - the methods
and procedures by which people handle, categorize, and retrieve information on a day-to-day basis [18] - are becoming increasingly popular.
However the evaluation of these PIM systems is problematic.
One of the main difficulties is caused by the personal nature of PIM.
People collect information as a natural consequence of completing other tasks.
This means that the collections people generate are unique to them alone and the information within a collection is intrinsically linked with the owner's personal experiences.
As personal collections are unique, we cannot create evaluation tasks that are applicable to all participants in an evaluation.
Secondly, personal collections may contain information that the participants are uncomfortable sharing within an evaluation.
The precise nature of this information - what information individuals would prefer to keep private - varies across individuals making it difficult to base search tasks on the contents of individual collections.
Therefore, experimenters face a number of challenges in order to conduct realistic but controlled PIM evaluations.
A particular feature of PIM research is that many systems have been designed to assist users with managing and re-finding their information, but very few have been evaluated; a situation noted by several scholars [1, 6, 7].
Recently, however, researchers have started to focus on ways to address the problem of PIM evaluation.
For example, Kelly [16] proposes that numerous methodologies must be taken to examine and understand the many issues involved in PIM, although, she makes explicit reference to the need for laboratory based PIM studies and a common set of shared tasks to make this possible.
Capra [6] also identifies the need for controlled PIM lab evaluations to complement other evaluation techniques, placing specific emphasis on the need to understand PIM behaviour at the task level.
In this paper, we attempt to address the difficulties involved to faciliate controlled laboratory PIM evaluations.
In the first part of this paper we present a diary study of information re-finding tasks.
The study examines the kind of tasks that require users to re-find information and produces a taxonomy of re-finding tasks for email messages and web pages.
We also look at the features of the tasks that make re-finding difficult.
In the second part, we propose a task-based evaluation methodology based on our findings and examine the feasibility of the approach using different methods of task creation.
Thus, this paper offers two contributions to the field: an increased understanding of PIM behaviour at the task level and an evaluation method that will facilitate further investigations.
2.
RELATED WORK
A variety of approaches are available to study PIM.
Naturalistic approaches study participants performing naturally, completing their own tasks as they occur, within familiar environments.
These approaches allow researchers to overcome many of the difficulties caused by the personal nature of PIM.
As the tasks performed are "real" and not simulated, the participants can utilise their own experiences, previous knowledge and information collections to complete the tasks.
A benefit of the approach is that data can be captured continuously over extended time periods and measurements can be taken at fixed points in time within these [15].
Naturalistic approaches can be applied by conducting fieldwork [17, 8], ethnographic methods as suggested by [15] or via log file analysis [9, 7].
Both ethnographic and fieldwork methods require the presence of an experimenter to assess how PIM is performed, which raises a number of issues.
Firstly, evaluation in this way is expensive; taking long time periods to study small numbers of participants and these small samples may not be representative of the behaviour of larger populations.
Secondly, because participants cannot be continually observed, experimenters must choose when to observe and this may affect the findings.
An alternative strategy to conducting naturalistic evaluations is to utilise log file analysis.
This approach makes use of logging software that captures a broad sampling of user activities in the context of natural use of a system.
In [9] a novel PIM search tool was deployed to 234 users and the log data provided detailed information about the nature of user queries, interactions with the query interface and about properties of the items retrieved.
Log file analysis is a powerful methodology as it allows the capture of a large quantity of detailed information about how users behave with the system without the expense and distracting influence of an observer.
Nevertheless, there are limitations to this strategy.
Firstly, to attain useful results, the deployed prototype must be something that people would use i.e. it has to be a fully functional piece of software that offers improvement on the systems ordinarily available to participants.
Developing a research prototype to this standard is beyond the resources of many researchers.
Further, caution must be taken when analysing logs, as the captured data shows nothing about the goals and intentions that the user had at the time.
It is, therefore, difficult to make any concrete statements about the reasons for the behaviour depicted in the logs.
This reveals a need to complement naturalistic studies with controlled experiments where the experimenter can relate the behaviour of study participants to goals associated with known search tasks.
Laboratory-based studies simulate users' real world environment in the controlled setting of the laboratory, offering the ability to study issues that are tightly defined and narrow in scope.
One difficulty in performing this kind of evaluation is sourcing collections to evaluate.
Kelly [16] proposes the introduction of a shared test collection that would provide sharable, reusable data sets, tasks and metrics for those interested in conducting PIM research.
This may be useful for testing algorithms in a way similar to TREC in mainstream IR [13].
However, a shared collection would be unsuitable for user studies because it would not be possible to incorporate the personal aspects of PIM while using a common, unfamiliar collection.
One alternative approach is to ask users to provide their own information collections to simulate familiar environments within the lab.
This approach has been applied to study the re-finding of personal photographs [11], email messages [20], and web-bookmarks [21].
The usefulness of this approach depends on how easy it is to transfer the collection or gain remote access.
Another solution is to use the entire web as a collection when studying web page re-finding [4].
This may be appropriate for studying web page re-finding because previous studies have shown that people often use web search engines for this purpose [5].
A second difficulty in performing PIM laboratory studies is creating tasks for participants to perform that can be solved by searching a shared or personal collection.
Tasks relate to the activity that results in a need for information [14] and are acknowledged to be important in determining user behaviour [26].
A large body of work has been carried out to understand the nature of tasks and how the type of task influences user information seeking behaviour.
For example, tasks have been categorised in terms of increasing complexity [3] and task complexity has been suggested to affect how searchers perceive their information needs [25] and how they try to find information [3].
Other previous work has provided methodologies that allow the simulation of tasks when studying information seeking behaviour [2].
However, little is known about the kinds of tasks that cause people to search their personal stores or re-find information that they have seen before.
Consequently, it is difficult to devise simulated work task situations for PIM.
The exception is the study of personal photograph management, where Rodden's work on categorising personal photograph search tasks has facilitated the creation of simulated work task situations [22].
There have been other suggestions as to how to classify PIM tasks.
For example, [5] asked participants to classify tasks based on how frequently they perform the task type in their daily life and how familiar they were with the location of the sought after information and several scholars have classified information objects by the frequency of their use e.g. [24].
While these are interesting properties that may affect how a task will be performed, they do not give experimenters enough scope to devise tasks.
Personal collections are one reason why task creation is so difficult.
Rodden's photo task taxonomy provides a solution here because it allows tasks, tailored to private collections to be categorised.
Systems can then be compared across task types for different users [11].
Unfortunately, no equivalent taxonomy exists for other types of information object.
Further, other types of object are more sensitive to privacy than photographs; it is unlikely that participants would be as content to allow researchers to browse their email collections to create tasks as they were with photographs in [11].
This presents a serious problem - how can researchers devise tasks that correspond to private collections without an understanding of the kinds of tasks people perform or jeopardising the privacy of study participants?
A few methods have been proposed.
For example, [20] studied email search by asking participants to re-find emails that had been sent to every member in a department; allowing the same tasks to be used for all of the study participants.
This approach ensured that privacy issues were avoided and participants could use things that they remember to complete tasks.
Nevertheless, the systems were only tested using one type of task - participants were asked to find single emails, each of which shared common properties.
In section 4 we show that
people perform a wider range of email re-finding tasks than this.
In [4], generic search tasks were artificially created by running evaluations over two sessions.
In the first session, participants were asked to complete work tasks that involved finding some unknown information.
In the second session, participants completed the same tasks again, which naturally involved some re-finding behaviour.
The limitations of this technique are that it does not allow participants to exploit any personal connections with the information because the information they are looking for may not correspond to any other aspect of their lives.
Further, if time is utilised by a system or interface being tested the approach is unsuitable because all of the objects found in the first session will have been accessed within the same time period.
Our review of evaluation approaches motivates a requirement for controlled laboratory experiments that allow tightly defined aspects of systems or interfaces to be tested.
Unfortunately, it has also been shown that there are difficulties involved in performing this type of evaluation - it is difficult to source collections and to devise tasks that correspond to private collections, while at the same time protect the privacy of the study participants.
In the following section we present a diary study of refinding tasks for email and web pages.
The outcome is a classification of tasks similar to that devised by Rodden for personal photographs [22].
In section 5 we build on this work by examining methods for creating tasks that do not compromise the privacy of participants and discuss how our work can facilitate task-based PIM user evaluations.
We show that by collecting tasks using electronic diaries, not only can we learn about the tasks that cause people to re-find personal information, but we can learn about the contents of private collections without compromising the privacy of the participants.
This knowledge can then be used to construct tasks for use in PIM evaluations.
3.
METHOD
4.
RESULTS
4.1 Nature of Web and Email Re-finding Tasks
4.2 What tasks are difficult?
4.3 Summary
5.
TASK-BASED PIM EVALUATIONS
5.1 Using Real Tasks
5.2 Using Simulated Tasks Based on Real Tasks
6.
CONCLUSIONS
This paper has focused on overcoming the difficulties involved in performing PIM evaluations.
The personal nature of PIM means that it is difficult to construct balanced experiments because participants each have their own unique collections that are self-generated by completing other tasks.
We suggested that to incorporate the personal aspects of PIM in evaluations, the performance of systems or users should be examined when users complete tasks on their own collections.
This approach itself has problems because task creation for personal collections is difficult: researchers don't know much about the kinds of re-finding tasks people perform and they don't know what information is within individual personal collections.
In this paper we described ways of overcoming these challenges to facilitate task based PIM user evaluations.
In the first part of the paper we performed a diary study that examined the tasks that caused people to re-find email messages and web pages.
The collected data included a wide range of both work and non-work related tasks, and based on the data we created a taxonomy of web and email re-finding tasks.
We discovered that people perform three main types of re-finding task: tasks that require specific information from within a single resource, tasks that require a single complete resource, and tasks that require information to be recovered from multiple resources.
In the second part of the paper, we discussed the significance of the taxonomy with respect to PIM evaluation.
We demonstrated that balanced experiments could be conducted comparing system or user performance on the task categories within the taxonomy.
We also suggested two methods of creating tasks that can be completed on personal collections.
These methods do not compromise the privacy of study participants.
We examined the techniques suggested, firstly by simulating an experimental situation - participants were asked to re-perform their own tasks as they recorded them, and secondly, in the context of a full evaluation.
Performing evaluations in this way will allow systems that have been proposed to improve users' ability to manage and re-find their information to be tested, so that we can learn about the needs and desires of users.
Thus, this paper has offered two contributions to the field: an increased understanding of PIM behaviour at the task level and an evaluation method that will facilitate further investigations.

Towards Task-based Personal Information Management Evaluations
ABSTRACT
Personal Information Management (PIM) is a rapidly growing area of research concerned with how people store, manage and re-find information.
A feature of PIM research is that many systems have been designed to assist users manage and re-find information, but very few have been evaluated.
This has been noted by several scholars and explained by the difficulties involved in performing PIM evaluations.
The difficulties include that people re-find information from within unique personal collections; researchers know little about the tasks that cause people to re-find information; and numerous privacy issues concerning personal information.
In this paper we aim to facilitate PIM evaluations by addressing each of these difficulties.
In the first part, we present a diary study of information re-finding tasks.
The study examines the kind of tasks that require users to re-find information and produces a taxonomy of re-finding tasks for email messages and web pages.
In the second part, we propose a task-based evaluation methodology based on our findings and examine the feasibility of the approach using two different methods of task creation.
1.
INTRODUCTION
Personal Information Management (PIM) is a rapidly growing area of research concerned with how people store, manage and re-find information.
PIM systems - the methods
and procedures by which people handle, categorize, and retrieve information on a day-to-day basis [18] - are becoming increasingly popular.
However the evaluation of these PIM systems is problematic.
One of the main difficulties is caused by the personal nature of PIM.
People collect information as a natural consequence of completing other tasks.
This means that the collections people generate are unique to them alone and the information within a collection is intrinsically linked with the owner's personal experiences.
As personal collections are unique, we cannot create evaluation tasks that are applicable to all participants in an evaluation.
Secondly, personal collections may contain information that the participants are uncomfortable sharing within an evaluation.
The precise nature of this information - what information individuals would prefer to keep private - varies across individuals making it difficult to base search tasks on the contents of individual collections.
Therefore, experimenters face a number of challenges in order to conduct realistic but controlled PIM evaluations.
Recently, however, researchers have started to focus on ways to address the problem of PIM evaluation.
Capra [6] also identifies the need for controlled PIM lab evaluations to complement other evaluation techniques, placing specific emphasis on the need to understand PIM behaviour at the task level.
In this paper, we attempt to address the difficulties involved to faciliate controlled laboratory PIM evaluations.
In the first part of this paper we present a diary study of information re-finding tasks.
The study examines the kind of tasks that require users to re-find information and produces a taxonomy of re-finding tasks for email messages and web pages.
We also look at the features of the tasks that make re-finding difficult.
In the second part, we propose a task-based evaluation methodology based on our findings and examine the feasibility of the approach using different methods of task creation.
Thus, this paper offers two contributions to the field: an increased understanding of PIM behaviour at the task level and an evaluation method that will facilitate further investigations.
2.
RELATED WORK
A variety of approaches are available to study PIM.
Naturalistic approaches study participants performing naturally, completing their own tasks as they occur, within familiar environments.
These approaches allow researchers to overcome many of the difficulties caused by the personal nature of PIM.
As the tasks performed are "real" and not simulated, the participants can utilise their own experiences, previous knowledge and information collections to complete the tasks.
Both ethnographic and fieldwork methods require the presence of an experimenter to assess how PIM is performed, which raises a number of issues.
Firstly, evaluation in this way is expensive; taking long time periods to study small numbers of participants and these small samples may not be representative of the behaviour of larger populations.
Secondly, because participants cannot be continually observed, experimenters must choose when to observe and this may affect the findings.
An alternative strategy to conducting naturalistic evaluations is to utilise log file analysis.
This approach makes use of logging software that captures a broad sampling of user activities in the context of natural use of a system.
This reveals a need to complement naturalistic studies with controlled experiments where the experimenter can relate the behaviour of study participants to goals associated with known search tasks.
One difficulty in performing this kind of evaluation is sourcing collections to evaluate.
Kelly [16] proposes the introduction of a shared test collection that would provide sharable, reusable data sets, tasks and metrics for those interested in conducting PIM research.
However, a shared collection would be unsuitable for user studies because it would not be possible to incorporate the personal aspects of PIM while using a common, unfamiliar collection.
One alternative approach is to ask users to provide their own information collections to simulate familiar environments within the lab.
This approach has been applied to study the re-finding of personal photographs [11], email messages [20], and web-bookmarks [21].
The usefulness of this approach depends on how easy it is to transfer the collection or gain remote access.
Another solution is to use the entire web as a collection when studying web page re-finding [4].
This may be appropriate for studying web page re-finding because previous studies have shown that people often use web search engines for this purpose [5].
A second difficulty in performing PIM laboratory studies is creating tasks for participants to perform that can be solved by searching a shared or personal collection.
Tasks relate to the activity that results in a need for information [14] and are acknowledged to be important in determining user behaviour [26].
A large body of work has been carried out to understand the nature of tasks and how the type of task influences user information seeking behaviour.
For example, tasks have been categorised in terms of increasing complexity [3] and task complexity has been suggested to affect how searchers perceive their information needs [25] and how they try to find information [3].
Other previous work has provided methodologies that allow the simulation of tasks when studying information seeking behaviour [2].
However, little is known about the kinds of tasks that cause people to search their personal stores or re-find information that they have seen before.
Consequently, it is difficult to devise simulated work task situations for PIM.
The exception is the study of personal photograph management, where Rodden's work on categorising personal photograph search tasks has facilitated the creation of simulated work task situations [22].
There have been other suggestions as to how to classify PIM tasks.
While these are interesting properties that may affect how a task will be performed, they do not give experimenters enough scope to devise tasks.
Personal collections are one reason why task creation is so difficult.
Rodden's photo task taxonomy provides a solution here because it allows tasks, tailored to private collections to be categorised.
Systems can then be compared across task types for different users [11].
Unfortunately, no equivalent taxonomy exists for other types of information object.
Further, other types of object are more sensitive to privacy than photographs; it is unlikely that participants would be as content to allow researchers to browse their email collections to create tasks as they were with photographs in [11].
This presents a serious problem - how can researchers devise tasks that correspond to private collections without an understanding of the kinds of tasks people perform or jeopardising the privacy of study participants?
A few methods have been proposed.
For example, [20] studied email search by asking participants to re-find emails that had been sent to every member in a department; allowing the same tasks to be used for all of the study participants.
This approach ensured that privacy issues were avoided and participants could use things that they remember to complete tasks.
Nevertheless, the systems were only tested using one type of task - participants were asked to find single emails, each of which shared common properties.
In section 4 we show that
people perform a wider range of email re-finding tasks than this.
In [4], generic search tasks were artificially created by running evaluations over two sessions.
In the first session, participants were asked to complete work tasks that involved finding some unknown information.
In the second session, participants completed the same tasks again, which naturally involved some re-finding behaviour.
The limitations of this technique are that it does not allow participants to exploit any personal connections with the information because the information they are looking for may not correspond to any other aspect of their lives.
Our review of evaluation approaches motivates a requirement for controlled laboratory experiments that allow tightly defined aspects of systems or interfaces to be tested.
Unfortunately, it has also been shown that there are difficulties involved in performing this type of evaluation - it is difficult to source collections and to devise tasks that correspond to private collections, while at the same time protect the privacy of the study participants.
In the following section we present a diary study of refinding tasks for email and web pages.
The outcome is a classification of tasks similar to that devised by Rodden for personal photographs [22].
In section 5 we build on this work by examining methods for creating tasks that do not compromise the privacy of participants and discuss how our work can facilitate task-based PIM user evaluations.
We show that by collecting tasks using electronic diaries, not only can we learn about the tasks that cause people to re-find personal information, but we can learn about the contents of private collections without compromising the privacy of the participants.
This knowledge can then be used to construct tasks for use in PIM evaluations.
6.
CONCLUSIONS
This paper has focused on overcoming the difficulties involved in performing PIM evaluations.
The personal nature of PIM means that it is difficult to construct balanced experiments because participants each have their own unique collections that are self-generated by completing other tasks.
We suggested that to incorporate the personal aspects of PIM in evaluations, the performance of systems or users should be examined when users complete tasks on their own collections.
This approach itself has problems because task creation for personal collections is difficult: researchers don't know much about the kinds of re-finding tasks people perform and they don't know what information is within individual personal collections.
In this paper we described ways of overcoming these challenges to facilitate task based PIM user evaluations.
In the first part of the paper we performed a diary study that examined the tasks that caused people to re-find email messages and web pages.
The collected data included a wide range of both work and non-work related tasks, and based on the data we created a taxonomy of web and email re-finding tasks.
We discovered that people perform three main types of re-finding task: tasks that require specific information from within a single resource, tasks that require a single complete resource, and tasks that require information to be recovered from multiple resources.
In the second part of the paper, we discussed the significance of the taxonomy with respect to PIM evaluation.
We demonstrated that balanced experiments could be conducted comparing system or user performance on the task categories within the taxonomy.
We also suggested two methods of creating tasks that can be completed on personal collections.
These methods do not compromise the privacy of study participants.
We examined the techniques suggested, firstly by simulating an experimental situation - participants were asked to re-perform their own tasks as they recorded them, and secondly, in the context of a full evaluation.
Performing evaluations in this way will allow systems that have been proposed to improve users' ability to manage and re-find their information to be tested, so that we can learn about the needs and desires of users.
Thus, this paper has offered two contributions to the field: an increased understanding of PIM behaviour at the task level and an evaluation method that will facilitate further investigations.

Towards Task-based Personal Information Management Evaluations
ABSTRACT
Personal Information Management (PIM) is a rapidly growing area of research concerned with how people store, manage and re-find information.
A feature of PIM research is that many systems have been designed to assist users manage and re-find information, but very few have been evaluated.
This has been noted by several scholars and explained by the difficulties involved in performing PIM evaluations.
The difficulties include that people re-find information from within unique personal collections; researchers know little about the tasks that cause people to re-find information; and numerous privacy issues concerning personal information.
In this paper we aim to facilitate PIM evaluations by addressing each of these difficulties.
In the first part, we present a diary study of information re-finding tasks.
The study examines the kind of tasks that require users to re-find information and produces a taxonomy of re-finding tasks for email messages and web pages.
In the second part, we propose a task-based evaluation methodology based on our findings and examine the feasibility of the approach using two different methods of task creation.
1.
INTRODUCTION
Personal Information Management (PIM) is a rapidly growing area of research concerned with how people store, manage and re-find information.
PIM systems - the methods
and procedures by which people handle, categorize, and retrieve information on a day-to-day basis [18] - are becoming increasingly popular.
However the evaluation of these PIM systems is problematic.
One of the main difficulties is caused by the personal nature of PIM.
People collect information as a natural consequence of completing other tasks.
This means that the collections people generate are unique to them alone and the information within a collection is intrinsically linked with the owner's personal experiences.
As personal collections are unique, we cannot create evaluation tasks that are applicable to all participants in an evaluation.
Secondly, personal collections may contain information that the participants are uncomfortable sharing within an evaluation.
The precise nature of this information - what information individuals would prefer to keep private - varies across individuals making it difficult to base search tasks on the contents of individual collections.
Therefore, experimenters face a number of challenges in order to conduct realistic but controlled PIM evaluations.
A particular feature of PIM research is that many systems have been designed to assist users with managing and re-finding their information, but very few have been evaluated; a situation noted by several scholars [1, 6, 7].
Recently, however, researchers have started to focus on ways to address the problem of PIM evaluation.
For example, Kelly [16] proposes that numerous methodologies must be taken to examine and understand the many issues involved in PIM, although, she makes explicit reference to the need for laboratory based PIM studies and a common set of shared tasks to make this possible.
Capra [6] also identifies the need for controlled PIM lab evaluations to complement other evaluation techniques, placing specific emphasis on the need to understand PIM behaviour at the task level.
In this paper, we attempt to address the difficulties involved to faciliate controlled laboratory PIM evaluations.
In the first part of this paper we present a diary study of information re-finding tasks.
The study examines the kind of tasks that require users to re-find information and produces a taxonomy of re-finding tasks for email messages and web pages.
We also look at the features of the tasks that make re-finding difficult.
In the second part, we propose a task-based evaluation methodology based on our findings and examine the feasibility of the approach using different methods of task creation.
Thus, this paper offers two contributions to the field: an increased understanding of PIM behaviour at the task level and an evaluation method that will facilitate further investigations.
2.
RELATED WORK
A variety of approaches are available to study PIM.
Naturalistic approaches study participants performing naturally, completing their own tasks as they occur, within familiar environments.
These approaches allow researchers to overcome many of the difficulties caused by the personal nature of PIM.
As the tasks performed are "real" and not simulated, the participants can utilise their own experiences, previous knowledge and information collections to complete the tasks.
A benefit of the approach is that data can be captured continuously over extended time periods and measurements can be taken at fixed points in time within these [15].
Naturalistic approaches can be applied by conducting fieldwork [17, 8], ethnographic methods as suggested by [15] or via log file analysis [9, 7].
Both ethnographic and fieldwork methods require the presence of an experimenter to assess how PIM is performed, which raises a number of issues.
Firstly, evaluation in this way is expensive; taking long time periods to study small numbers of participants and these small samples may not be representative of the behaviour of larger populations.
Secondly, because participants cannot be continually observed, experimenters must choose when to observe and this may affect the findings.
An alternative strategy to conducting naturalistic evaluations is to utilise log file analysis.
This approach makes use of logging software that captures a broad sampling of user activities in the context of natural use of a system.
In [9] a novel PIM search tool was deployed to 234 users and the log data provided detailed information about the nature of user queries, interactions with the query interface and about properties of the items retrieved.
Log file analysis is a powerful methodology as it allows the capture of a large quantity of detailed information about how users behave with the system without the expense and distracting influence of an observer.
Nevertheless, there are limitations to this strategy.
Firstly, to attain useful results, the deployed prototype must be something that people would use i.e. it has to be a fully functional piece of software that offers improvement on the systems ordinarily available to participants.
Developing a research prototype to this standard is beyond the resources of many researchers.
Further, caution must be taken when analysing logs, as the captured data shows nothing about the goals and intentions that the user had at the time.
It is, therefore, difficult to make any concrete statements about the reasons for the behaviour depicted in the logs.
This reveals a need to complement naturalistic studies with controlled experiments where the experimenter can relate the behaviour of study participants to goals associated with known search tasks.
Laboratory-based studies simulate users' real world environment in the controlled setting of the laboratory, offering the ability to study issues that are tightly defined and narrow in scope.
One difficulty in performing this kind of evaluation is sourcing collections to evaluate.
Kelly [16] proposes the introduction of a shared test collection that would provide sharable, reusable data sets, tasks and metrics for those interested in conducting PIM research.
This may be useful for testing algorithms in a way similar to TREC in mainstream IR [13].
However, a shared collection would be unsuitable for user studies because it would not be possible to incorporate the personal aspects of PIM while using a common, unfamiliar collection.
One alternative approach is to ask users to provide their own information collections to simulate familiar environments within the lab.
This approach has been applied to study the re-finding of personal photographs [11], email messages [20], and web-bookmarks [21].
The usefulness of this approach depends on how easy it is to transfer the collection or gain remote access.
Another solution is to use the entire web as a collection when studying web page re-finding [4].
This may be appropriate for studying web page re-finding because previous studies have shown that people often use web search engines for this purpose [5].
A second difficulty in performing PIM laboratory studies is creating tasks for participants to perform that can be solved by searching a shared or personal collection.
Tasks relate to the activity that results in a need for information [14] and are acknowledged to be important in determining user behaviour [26].
A large body of work has been carried out to understand the nature of tasks and how the type of task influences user information seeking behaviour.
For example, tasks have been categorised in terms of increasing complexity [3] and task complexity has been suggested to affect how searchers perceive their information needs [25] and how they try to find information [3].
Other previous work has provided methodologies that allow the simulation of tasks when studying information seeking behaviour [2].
However, little is known about the kinds of tasks that cause people to search their personal stores or re-find information that they have seen before.
Consequently, it is difficult to devise simulated work task situations for PIM.
The exception is the study of personal photograph management, where Rodden's work on categorising personal photograph search tasks has facilitated the creation of simulated work task situations [22].
There have been other suggestions as to how to classify PIM tasks.
For example, [5] asked participants to classify tasks based on how frequently they perform the task type in their daily life and how familiar they were with the location of the sought after information and several scholars have classified information objects by the frequency of their use e.g. [24].
While these are interesting properties that may affect how a task will be performed, they do not give experimenters enough scope to devise tasks.
Personal collections are one reason why task creation is so difficult.
Rodden's photo task taxonomy provides a solution here because it allows tasks, tailored to private collections to be categorised.
Systems can then be compared across task types for different users [11].
Unfortunately, no equivalent taxonomy exists for other types of information object.
Further, other types of object are more sensitive to privacy than photographs; it is unlikely that participants would be as content to allow researchers to browse their email collections to create tasks as they were with photographs in [11].
This presents a serious problem - how can researchers devise tasks that correspond to private collections without an understanding of the kinds of tasks people perform or jeopardising the privacy of study participants?
A few methods have been proposed.
For example, [20] studied email search by asking participants to re-find emails that had been sent to every member in a department; allowing the same tasks to be used for all of the study participants.
This approach ensured that privacy issues were avoided and participants could use things that they remember to complete tasks.
Nevertheless, the systems were only tested using one type of task - participants were asked to find single emails, each of which shared common properties.
In section 4 we show that
people perform a wider range of email re-finding tasks than this.
In [4], generic search tasks were artificially created by running evaluations over two sessions.
In the first session, participants were asked to complete work tasks that involved finding some unknown information.
In the second session, participants completed the same tasks again, which naturally involved some re-finding behaviour.
The limitations of this technique are that it does not allow participants to exploit any personal connections with the information because the information they are looking for may not correspond to any other aspect of their lives.
Further, if time is utilised by a system or interface being tested the approach is unsuitable because all of the objects found in the first session will have been accessed within the same time period.
Our review of evaluation approaches motivates a requirement for controlled laboratory experiments that allow tightly defined aspects of systems or interfaces to be tested.
Unfortunately, it has also been shown that there are difficulties involved in performing this type of evaluation - it is difficult to source collections and to devise tasks that correspond to private collections, while at the same time protect the privacy of the study participants.
In the following section we present a diary study of refinding tasks for email and web pages.
The outcome is a classification of tasks similar to that devised by Rodden for personal photographs [22].
In section 5 we build on this work by examining methods for creating tasks that do not compromise the privacy of participants and discuss how our work can facilitate task-based PIM user evaluations.
We show that by collecting tasks using electronic diaries, not only can we learn about the tasks that cause people to re-find personal information, but we can learn about the contents of private collections without compromising the privacy of the participants.
This knowledge can then be used to construct tasks for use in PIM evaluations.
3.
METHOD
Diary Studies are a naturalistic technique, offering the ability to capture factual data, in a natural setting, without the distracting influence of an observer.
Limitations of the technique include difficulties in maintaining participant dedication levels and convincing participants that seemingly mundane information is useful and should be reported [19].
[12] suggest that the effects of the negatives can be limited, however, with careful design and good implementation.
In our diary study, we followed the suggestions in [12] to achieve the best possible data.
To this end, we restricted the recorded tasks to web and email re-finding.
By asking users to record fewer tasks it was anticipated that participant apathy would be reduced and dedication levels maintained.
The participants were provided with a personalised web form in which they could record details about their information needs and the contexts in which these needs developed.
Web forms were deployed rather than paperbased diaries because to re-find web and email information the user would be at a computer with an Internet connection and there would be no need to search for a paper-based diary and pen.
The diary form solicited the following information: whether the information need related to re-finding a web page or an email message and a description of the task they are performing.
This description was to contain both the information that the participant wished to find and the reason that they needed the information.
To help with this, the form gave three example task descriptions, which were also explained verbally to each participant during an introductory session.
The experimenter ensured that the participants understood that the tasks to be recorded were not limited to the types shown in the examples.
The examples were supplied purely to get participants thinking about the kinds of things they could record and to show the level of and type of details expected.
The form also asked participants to rate each task in terms of difficulty (on a scale from 1-5, where 1 was very easy and 5 was very hard).
Finally, they were asked when was the last time they looked at the sought after information.
Again, they were able to choose from 5 options (less than a day ago, less than a week ago, less than a month ago, less than a year ago, more than a year ago).
Time information was used to examine the frequency with which the participants re-found old and new information, and when combined with difficulty ratings created a picture of whether or not the time period between accessing and re-accessing impacted on how difficult the participants perceived tasks to be.
36 participants, recruited by mass advertisement through departmental communication channels, research group meetings and undergraduate lectures, were asked to digitally record details of their information re-finding tasks over a period of approximately 3 weeks.
The final population consisted of 4 academic staff members, 8 research staff members, 6 research students and 18 undergraduate students.
The ages of participants ranged from 19-59.
As both personal and work tasks were recorded, the results collected cover a broad range of re-finding tasks.
4.
RESULTS
Several analyses were performed on the captured data.
The following sections present the findings.
Firstly, we examine the kinds of re-finding tasks that were performed both when searching on email and on the web.
Next, we consider the distribution of tasks - which kinds of tasks were performed most often by participants.
Lastly, we explore the kinds of re-finding tasks that participants perceived as difficult.
4.1 Nature of Web and Email Re-finding Tasks
During the study 412 tasks were recorded.
150 (36.41%) of these tasks were email based, 262 (63.59%) were webbased.
As with most diary studies, the number of tasks recorded varied extensively between particpants.
The median number of tasks per participant was 8 (interquartile range (IQR) = 9.5).
More web tasks (median = 5, IQR = 7.5) were recorded than email tasks (median = 3, IQR = 3).
This means that on average each participant recorded approximately one task every two days.
From the descriptions supplied by the participants, we found similar features in the recorded tasks for both email and web re-finding.
Based on this observation a joint classification scheme was devised, encompassing both email and web tasks.
The tasks were classified as one of three types: lookup tasks, item tasks and multi-item tasks.
Lookup tasks involve searching for specific information from within a resource, for example an email or a web page, where the resource may or may not be known.
Some recorded examples of lookup tasks were:
• LU1: "Looking for the course code for a class - it's used in a
script that is run to set up a practical.
I'd previously obtained this about 3 weeks ago from our website."
• LU2: "I am trying to determine the date by which I step down as an External Examiner.
This is in an email somewhere" • LU3: "Looking for description of log format from system R developed for student project.
I think he sent me in it an email"
Item tasks involve looking for a particular email or web page, perhaps to pass on to someone else or when the entire contents are needed to complete the task.
Some recorded examples of item tasks were:
• I1: "Looking for SIGIR 2002 paper to give to another student" • I2: "Find the receipt of an online airline purchase required to claim expenses" • I3: "I need the peer evaluation forms for the MIA class E sent me them by email"
To clarify, lookup tasks differ from item tasks in two ways - in the quantity of information required and in what the user knows about what they are looking for.
Lookup tasks involve a need for a small piece of information e.g. a phone number or an ingredient, and the user may or may not know exactly the resource that contains this information.
In item tasks the user knows exactly the resource they are looking for and needs the entire contents of that resource.
Multi-item tasks were tasks that required information that was contained within numerous web pages or email messages.
Often these tasks required the user to process or collate the information in order to solve the task.
Some recorded examples were:
• MI1: "Looking for obituaries and other material on the novelist John Fowles, who died at the weekend.
Accessed the online Guradian and IMES • MI2: "Trying to find details on Piccolo graphics framework.
Remind myself of what it is and what it does.
Looking to build a GUI within Eclipse" • MI3: "I am trying to file my emails regarding IPM and I am looking for any emails from or about this journal"
There were a number of tasks that were difficult to classify.
For example, consider the following recorded task:
• LU4: "re-find AS's paper on graded relevance assessments because I want to see how she presented her results for a paper I am writing"
This task actually consists of two sub-tasks: 1 item task (refind the paper) and 1 lookup task (look for specific information within the paper).
It was decided to treat this as a lookup task because the user's ultimate goal was to access and use the information within the resource.
There were a number of examples of combined tasks, mainly of the form item then lookup, but there were also examples of item then multi-item.
For example:
• MI4: "re-find Kelkoo website so that I can re-check the prices of hair-straighteners for my girlfriend"
A second source of ambiguity came from tasks such as finding an email containing a URL as a means of re-accessing a web page.
It was also decided to categorise these as lookup tasks because in all cases these were logged by participants as email searches and, within this context, what they were looking for was information within an email.
Another problem was that some of the logs lacked the detail required to perform a categorisation e.g.
• U1: "searching for how to retrieve user's selection from a message box.
Decided to use some other means"
Such tasks were labelled as U for "unclassifiable".
To verify the consistency of the taxonomy, the tasks were recategorised by the same researcher after a delay of two weeks.
The agreement between the results of the two analyses was largely consistent (96.8%).
Further, we asked a researcher with no knowledge of the project or the field to classify a sample of 50 tasks.
The second researcher achieved a 90% agreement.
We feel that this high agreement on a large number of tasks by more than one researcher provides evidence for the reliability of the classification scheme.
The distribution of task types is shown in table 1.
Overall, lookup and item tasks were the most common, with multiitem tasks only representing 8.98% of those recorded.
The distribution of the task types was different for web and email re-finding.
The majority of email tasks (60%) involved looking for information within an email (lookup), in contrast to web tasks where the majority of tasks (52.67%) involved looking for a single web page (item).
Another distinction was the number of recorded multi-item tasks for web and email.
Multi-item tasks were very rare for email re-finding (only 2.67% of email tasks involved searching for multiple resources), but comparatively common for web re-finding (12.6%).
Table 1: The distribution of task types
In addition to the three-way classification described above, the recorded tasks were classified with respect to the temperature metaphor proposed by [24], which classifies information as one of three temperatures: hot, warm and cold.
We classified the tasks using the form data.
Information that had been seen less than a day or less than a week before the task were defined as hot, information that had been seen less than a month before the task as warm, and information that had been seen less than a year or more than a year before the task as cold.
Unfortunately, a technical difficulty with the form only allowed 335 (81.3%) of the tasks to be classified.
The remainder were defined as U for "unclassifiable".
A cross-tabulation of task types and temperatures is shown in table 2.
Table 2: The distribution of temperatures
Most of the tasks that caused people to re-find web pages (42.75%) and email messages (33.33%) involved searching for information that has been accessed in the last week.
However there were also a number of re-finding tasks that involved searching for older information: 23.30% of the tasks recorded (24.00% for email and 22.90% for web) involved searching for information accessed in the last month and 18.69% of the tasks recorded (24.67% for email and 15.27% for web) were looking for even older information.
This is important with respect to evaluation because there is psycho
logical evidence suggesting that people remember less over time e.g. [23].
This means that users may find searching for older information more difficult or perhaps alter their seeking strategy when looking for hot, warm or cold information.
4.2 What tasks are difficult?
We looked for patterns in the recorded data to determine if certain tasks were perceived as more difficult than others.
For example, we examined whether the media type affected how difficult the participants perceived the task to be.
There was no evidence that participants found either email (median = 2 IQR = 2) or web (median = 2 IQR = 2) tasks more difficult.
We also investigated whether the type of task or the length of time between accessing and re-accessing made a task more difficult.
Figure 1 shows this information graphically.
Figure 1: Difficulty ratings for task types
From figure 1, it does not appear that any particular task type was perceived as difficult with respect to the others, although there is a suggestion that lookup tasks were perceived more difficult when looking for cold information than hot and item tasks were perceived more difficult for warm information than hot.
To assess the relationship between information temperature and the perceived difficulty, we used Mood's median tests to determine whether the rank of difficulty scores was in agreement for the information temperatures being compared (p <0.05).
For the look-up task data, there was evidence that hot tasks were perceived easier than cold (p = 0.0001) and that warm tasks were perceived easier than cold tasks (p = 0.0041), but there was no evidence to distinguish between the difficulty ratings of hot and warm tasks (p = 0.593).
For the item task data, there was evidence that hot and cold tasks were rated differently (p = 0.024), but no evidence to distinguish between hot and warm tasks (p = 0.05) or warm and cold tasks (p = 0.272).
These tests confirm that the length of time between accessing and re-accessing the sought after information indeed influenced how difficult participants perceived the task to be.
Nevertheless, the large number of tasks of all types and temperatures rated by participants as easy i.e. <3, suggests that there are other factors that influence how difficult a task is perceived to be.
To learn about these factors would require the kind of user evaluations proposed by [16, 6] - the kind of evaluations facilitated by our work.
4.3 Summary
In the first part of this paper, we described a diary study of web and email re-finding tasks.
We examined the types of task that caused the participants to search their personal stores and found three main categories of task: tasks where the user requires specific information from within a single resource, tasks where a single resource is required, and tasks that require information to be recovered from multiple resources.
It was discovered that look-up and item tasks were recorded with greater frequency than multi-item tasks.
Although no evidence was found that web or email tasks were more difficult, there was some evidence showing that the time between accessing and re-accessing affected how difficult the participants perceived tasks to be.
These findings have implications for evaluating PIM behaviour at the task level.
The remainder of this paper concentrates on this, discussing what the findings mean with respect to performing task-based PIM user evaluations.
5.
TASK-BASED PIM EVALUATIONS
The findings described in section 4 are useful with respect to evaluation because they provide experimenters with enough knowledge to conduct controlled user evaluations in lab conditions.
Greco-Latin square experimental designs can be constructed where participants are assigned n tasks of the three types described above to perform on their own collections using x systems.
This would allow the performance of the systems or the behaviour of the participants using different systems to be analysed with respect to the type of task being performed (look-up, item, or multi-item).
In the following sections we evaluate the feasibility of this approach when employing different methods of task creation.
5.1 Using Real Tasks
One method of creating realistic re-finding tasks without compromising the privacy of participants is to use real tasks.
Diary-studies, similar to that described above, would allow experimenters to capture a pool of tasks for participants to complete by searching on their own collections.
This is extremely advantageous because it would allow experimenters to evaluate the behaviour of real users, completing real search tasks on real collections while in a controlled environment.
There is also the additional benefit that the task descriptions would not make any assumptions about what the user would remember in a real life situation because they would only include the information that had been recorded i.e. the information that was available when the user originally performed the task.
Nevertheless, to gain these benefits we must, firstly, confirm that the task descriptions recorded are of sufficient quality to enable the task to be re-performed at a later date.
Secondly, we must ensure that a diary-study would provide experimenters with enough tasks to construct a balanced experimental design that would satisfy their data needs.
To examine the quality of recorded tasks, 6 weeks after the diary study had completed, we asked 6 of our participants, selected randomly from the pool of those who recorded enough tasks, to re-perform 5 of their own tasks.
The tasks were selected randomly from the pool of those available.
The issued tasks consisted of 10 email and 20 web tasks, 9 of which were lookup tasks, 12 were item tasks, and 8 were multi-item tasks.
The issued tasks represented a broad-sampling of the complete set of recorded tasks.
They also included tasks with vague descriptions e.g.
• LU5: "Find a software key for an application I required to reinstall".
• LU6: "Trying to find a quote to use in a paper.
Cannot remember the person or the exact quote"
The usefulness of such tasks would rely on the memories of participants i.e. would the recorder of task LU5 remember which application he referred to and would the recorder of LU6 remember enough about the context in which the task took place to re-perform the task?
Presented with the tasks exactly as they recorded them, the participants were asked to re-perform each task with any system of their choice.
Of the 30 tasks issued, 26 (86.67%) were completed without problems, 2 (6.67%) of the tasks were not completed because the description recorded was insufficent to recreate the task, and 2 tasks (6.67%) were not completed because the task was too difficult or the required web page no longer existed.
Experimenters are likely to be interested in the final group of tasks because it is important to discover what makes a task difficult and how user behaviour changes in these circumstances.
Therefore, from the 30 tasks tested, only 2 tasks were not of sufficient quality to be used in an evaluation situation.
Further, there did not seem to be any issue of the type, temperature or difficulty ratings affecting the quality of the task descriptions.
These findings suggest that the participants who recorded most tasks in the diary study also recorded tasks with sufficient quality.
However, did the diary study generate enough tasks to satisfy the needs of experimenters?
Table 3: The quantities of recorded email tasks
Table 4: The quantities of recorded web tasks
Naturally the exact number of tasks required to perform a user evaluation will depend on the goals of the evaluation, the number of users and the number of systems to be tested etc. .
However, for illustrative purposes we chose 5 tasks as a cut-off point for our data.
From tables 3 and 4, which show the quantities of email and web tasks recorded for each participant, we can see that of the 36 participants, only 13 (36.1%) recorded 5 or more email tasks and 20 (55.6%) recorded 5 or more web tasks.
This means that many of the recruited participants could not actually participate in the final evaluation.
This is a major limitation of using recorded tasks in evaluations because participant recruitment for user tests is challenging and it may not be possible to recruit enough participants if experimenters lose between half and two-thirds of their populations.
Further, there was some imbalance in the numbers of recorded tasks of different types.
Some participants recorded several lookup tasks but very few item tasks and others recorded several item tasks but few lookup tasks.
There was also a specific lack of multi-item email tasks.
This situation makes it very difficult for experimenters to prepare balanced experimental designs.
Therefore, even though our first test suggests that the quality of recorded tasks was sufficient for the participants to re-perform the tasks at a later stage, the number of tasks recorded was probably too low to make this a viable option for experimental task creation.
However, it may be possible to increase the number of tasks recorded by frequently reminding participants or by making personal visits etc. .
5.2 Using Simulated Tasks Based on Real Tasks
Another benefit of diary-studies is that they provide information about the contents and uses of private collections without invading participants' privacy.
This section explores the possibility of using a combination of the knowledge gained from diary studies and other attributes known about participants to artificially create re-finding tasks corresponding to the taxonomy defined in section 4.1.
We explain the techniques used and demonstrate the feasibility of creating simulated tasks within the context of a user evaluation investigating email re-finding behaviour.
Space limitations prevent us from reporting our findings; instead we concentrate on the methods of task creation.
As preparation for the evaluation, we performed a second diary-study, where 34 new participants, consisting of 16 post-graduate students and 18 under-graduate students, recorded 150 email tasks over a period of approximately 3 weeks.
The collected data revealed several patterns that helped with the creation of artificial tasks.
For example, students in both groups recorded tasks relating to classes that they were taking at the time and often different participants recorded tasks that involved searching for the same information.
This was useful because it provided us with a clue that even though some of the participants did not record a particular task, it was possible that the task may still be applicable to their collections.
Other patterns revealed included that students within the same group often searched for emails containing announcements from the same source.
For example, several undergraduate students recorded tasks that included re-finding information relating to job vacancies.
There were also tasks that were recorded by participants in both groups.
For example, searching for an email that would re-confirm the pin code required to access the computer labs.
To supplement our knowledge of the participants' email collections, we asked 2 participants from each group to provide email tours.
These consisted of short 5-10 minute sessions, where participants were asked to explain why they use email, who sends them email, and their organisational strategies.
This approach has been used successfully in the past as a non-intrusive means to learn about how people store and maintain their personal information [17].
Originally, we had planned to ask more participants to provide tours, but we found 2 tours per group was sufficient for
our needs.
Again, patterns emerged that helped with task creation.
We found content overlap within and between groups that confirmed many of our observations from the diary study data.
For example, the students who gave tours revealed that they received emails from lecturers for particular class assignments, receipts for completed assignments, and various announcements from systems support and about job vacancies.
Importantly, the participants were also able to confirm which other students had received the same information.
This confirmed that many of tasks recorded during the diary study were applicable, not only to the recorder, but to every participant in 1 or both groups.
Based on this initial investigatory work, a set of 15 tasks (5 of each type in our taxonomy) was created for each group of participants.
We also created a set of tasks for a third group of participants that consisted of research and academic staff members, based on our knowledge of the emails our colleagues receive.
Where possible we used the information recorded in the diary study descriptions to provide a context for the task i.e. a work task or motivation that would require the task to be performed.
When the diary study data did not provide sufficient context information to supply the participants with a robust description of the information need, we created simulated work task situations according to the guidelines of [2].
A further advantage of using simulated tasks in this way, rather than real-tasks, is that some of the users will not have performed the task in the recent past and this allows the examination of tasks that look for information of different temperatures.
If only real-tasks had been used all of the participants would have performed the tasks during the period of the diary study.
The created tasks were used in a final evaluation, where we examined the email re-finding behaviour of users with three different email systems.
21 users (7 in each group) performed 9 tasks each (1 task of each type on each system) using their own personal collections in a Greco-Latin square experimental design.
Performing a PIM evaluation in this way allowed the examination of re-finding behaviour in a way not possible before - we were able to observe the email re-finding strategies employed by real users, performing realistic tasks, on their own collections in a controlled environment.
The study revealed that the participants remembered different attributes of emails, demostrated different finding behaviour, and exhibited different levels of performance when asked to complete tasks of the different types in the taxonomy.
The key to both the task creation and the analysis of the results was our taxonomy, which provided the template to create tasks and also a means to compare the behaviour and performance of different users (and systems) performing different tasks of the same type.
Some of the findings of the evaluation will be published in [10].
Summarising the approach, to conduct a user experiment using our methodology, researchers would be required to perform the following steps: 1) Conduct a diary study as above'.
2) Analyse the recorded tasks looking for overlap between the participants.
3) Supplement the gained knowledge about the contents of participants' collections by asking a selection of the participants to provide a tour of their collection.
4) Use the knowledge gained to devise tasks of the three different types defined within the taxonomy.
More de'Information about this and the diary forms required can be found at http://www.cis.strath.ac.uk/˜dce/PIMevaluations tailed information on how to use the research described in this paper to perform task-based PIM evaluations can be found at our website (see footnote 1).
6.
CONCLUSIONS
This paper has focused on overcoming the difficulties involved in performing PIM evaluations.
The personal nature of PIM means that it is difficult to construct balanced experiments because participants each have their own unique collections that are self-generated by completing other tasks.
We suggested that to incorporate the personal aspects of PIM in evaluations, the performance of systems or users should be examined when users complete tasks on their own collections.
This approach itself has problems because task creation for personal collections is difficult: researchers don't know much about the kinds of re-finding tasks people perform and they don't know what information is within individual personal collections.
In this paper we described ways of overcoming these challenges to facilitate task based PIM user evaluations.
In the first part of the paper we performed a diary study that examined the tasks that caused people to re-find email messages and web pages.
The collected data included a wide range of both work and non-work related tasks, and based on the data we created a taxonomy of web and email re-finding tasks.
We discovered that people perform three main types of re-finding task: tasks that require specific information from within a single resource, tasks that require a single complete resource, and tasks that require information to be recovered from multiple resources.
In the second part of the paper, we discussed the significance of the taxonomy with respect to PIM evaluation.
We demonstrated that balanced experiments could be conducted comparing system or user performance on the task categories within the taxonomy.
We also suggested two methods of creating tasks that can be completed on personal collections.
These methods do not compromise the privacy of study participants.
We examined the techniques suggested, firstly by simulating an experimental situation - participants were asked to re-perform their own tasks as they recorded them, and secondly, in the context of a full evaluation.
Performing evaluations in this way will allow systems that have been proposed to improve users' ability to manage and re-find their information to be tested, so that we can learn about the needs and desires of users.
Thus, this paper has offered two contributions to the field: an increased understanding of PIM behaviour at the task level and an evaluation method that will facilitate further investigations.

person inform manag
re-find inform
privaci issu
taxonomi
email messag
measur
experiment
human factor
individu collect
naturalist approach
laboratori-base studi
user evalu