One critical step in cancer diagnosis and treatment is pathologists' analysis of histological images obtained from a patient's tissue sections to identify evidence of tumor cells and determine the grade (e.g., benign vs. malignant) based on medical guidelines.
Such an analysis is often challenging for pathologists, due to the sheer amount of effort it requires to identify sparse patterns of tumor cells given the ultra-high resolution of multiple histological images in a single patient's case. Further, the process suffers from subjectivity due to the intra-and inter-observer variations, e.g., different interpretations of the grading guidelines, and different ways of sampling and examining the slides to 'implement' a given guideline.
To overcome these challenges, digital histology, enabled by advanced scanning techniques, promises to transform traditional microscopic analysis to digital visualization automatable by data-driven artificial intelligence (AI) [18] . However, there remain three limitations of existing AI-aided histological analysis that prevent its adoption in clinical practice:
• A lack of comprehensiveness: most AI models tend to focus on one specific criterion inferred from one specific type of histological data [5, 12, 19, 24, 28, 32, 35, 47] while in practice pathologists almost always rely on multiple criteria by examinations across multiple types of data;
• A lack of explainability: most AI models function as 'black boxes' and provide little justifications for the generated analysis-such a lack of transparency creates a barrier that prevents AI from being accepted by pathologists;
• A lack of integrability: most AI models abstract histological analysis as a computational problem, thus it remains unclear how to integrate such models into pathologists' existing workflow and practices;
To address these limitations, we develop CrossPath-a brain tumor grading tool for pathologists to perform top-down, cross data type, multi-criterion histological analysis by shepherding mixed AI models. Currently, we focus on the grading of meningioma-the most common primary type of brain tumoras a point of departure for exploring the design of CrossPath. The goal is to aid pathologists to grade meningioma with the aid of AI, which assumes that tumor areas have already been identified on a slide. further select and drill down to a specific criterion, which retrieves a set of evidence ( Figure 1d ) to explain why AI arrives at its result. For example, for the criterion of mitotic count (Figure 1f ), the pathologist can see patches of evidence ( Figure  1d ) sampled in the largest aggregation of mitosis. Moreover, the pathologist can open a heatmap (Figure 1c ) overlaying the whole slide image (WSI) to overview the density distribution of positive mitotic cells recognized by the AI. Selecting a patch directs the pathologists' attention to a high power field (HPF, Figure 1a , yellow box) of the mitosis on the original WSI, where they can further examine the low-level histological features and approve or decline AI's analysis with one click (Figure 1b) , which in turn will update AI's results on individual criterion and, if necessary, the overall suggested grading as well based on WHO guidelines.
To validate CrossPath, a technical evaluation shows that the validation performance of AI models achieves an area under curve (AUC) of 0.842, 0.989, 0.923, and 0.834/0.673 in identifying the mitotic nucleus, necrosis, prominent nucleolus, and meningothelial/fibrous tissues separately. Moreover, Cross-Path achieves an averaged error rate of 12.3% and 13.7% in nuclei counting (hypercellularity) and Ki-67 index calculation. A user study with seven pathologists shows that, with less than an hour of learning, pathologists were able to use CrossPath to make accurate grading decisions, with about one third reduction in time consumption compared to examining with the optical microscope. Pathologists' qualitative feedback indicates that CrossPath obtains a high level of comprehensiveness, explainability, and integrability; meanwhile, the main challenge is that the current system does not support users to adjust the prediction threshold via the interface, and hence resulting in an amount of false-positive evidences that might influence the suggested result. Another challenge is that participants were not used to digital WSI examination since the size of cells varies from computer monitors because of the extra digital zoom in the interface.
This paper makes a tool contribution to AI-aided pathology, addressing the three key issues of comprehensiveness, explainability, and integrability.
• CrossPath addresses comprehensiveness by incorporating multiple WHO-guided criteria across different data types;
• CrossPath addresses explainability by (i) at the top level, presenting grading logic based on the WHO guideline; (ii) at the mid-level, providing specific examples with features leading to AI's result for each criterion (iii) at the low level, registering evidence into WSI to reveal more contextual features;
• CrossPath addresses integrability into existing workflow by mimicking how an attending pathologist oversees trainees' work, allowing a pathologist to shepherd mixed AI models following a top-down path to examine individual AI's results with evidence on-demand guided by heatmap visualizations.
CrossPath focuses on an interactive Computer-Aided Diagnosis system, which helps pathologists discover multiple histological characteristics of high-grade meningioma, with supporting evidence and visualization of AI-generated results. Correspondingly, the related work can be categorized into three areas: (i) interactive tools for digital histology; (ii) AI frameworks for digital histology; (iii) human-AI collaborative platforms.
Digital histology often deals with high-resolution whole slide images. Beyond involving AI trained by domain experts, there are a variety of tools for pathologists to define, explore, and decide upon clinical or research usage.
When it comes to digital histology, many tools are designed primarily for studying a localized area, as opposed to using multiple data types to examine patients' biopsy section at the whole slide level. ImageJ [38] is one of the most commonly used for scientific image analysis and has been extended by computational medicine research. The platform provides essential image processing tools to allow users to perform simple tasks, such as nuclei segmentation. Over its 30 years of evolution, various distributions [10, 34, 37] , and plugins [21] have been increasingly integrated into the system, making it the most widely used software in digital histology.
Besides ImageJ, others have focused on enhancing the WSI accessibility from the perspective of non-computational users. For example, cellprofiller [8] assists non-computational users to perform complex morphological assays automatically without writing any code quantitatively. caMicroscope [36] enables a user to run a segmentation pipeline in a selected area and to analyze nuclei characteristics in whole-slide images. QuPath [4] provides extensive annotation and automation analysis tools for pathologists, such as nuclei segmentation, positive cell counting. Pathology Image Informatics Platform (PIIP) [27] extends the capabilities of Sedeen viewer 1 by adding plugins on out-of-focus detection, region of interest transformation, and immunohistochemical (IHC) slide analysis.
Although some systems mentioned above provide a certain level of automation to reduce pathologists' workload, a majority of them do not support examining WSI comprehensively at a full-slide level. As a result, such tools are hard to integrate into pathologists' workflow.
Given the abundance of histological images, data-driven AI can potentially be integrated into an automatic analysis pipeline to help save medical professional's time and effort.
1 https://pathcore.com/sedeen/ Given the high resolution as well as a large amount of the digital histology data, the mainstream frameworks incorporate AI algorithms in Computer-Aided Diagnosis, content-based image retrieval (CBIR), as well as discovering relationships among various features [22] . Amongst these three primary applications, using AI for identifying regions-of-interest for Computer-Aided Diagnosis, e.g., identifying mitosis, is most related to CrossPath. Although these histological image analysis frameworks report satisfying performance on par with human beings, one main problem is that they are majorly focused on single criteria rather than integrating multiple at the same time. Hence, the promise of implementing AI for a reduced workload is obscured when doctors expect diagnosis to be based on a comprehensive list of criteria. Another limitation is that prior work is mainly based on single data type, such as H&E images, as input. In contrast, doctors in clinical practices usually refer to multiple IHC examinations to perform a differential diagnosis [14, 41] . Further, the non-transparent, non-explainable characteristics of the end-to-end AI algorithms would limit their applications in high-stake medical decision processes, such as tumor grading.
Although the promise of AI has enlightened the horizon of freeing humans from tedious tasks, applying it into high-stake applications, such as medical decision processes, is a challenging task. Thus it is important to enable human-AI collaboration in various domains including medicine.
In a broader context, recent research in HCI has demonstrated a plethora of examples of human-AI collaboration. For example, FortÃl' enables a user to modify optimization's result, which serves as input for the next iteration to reflect the user's intent [9] . Willett et al. propose a mixed-initiative tool to enable novice users to convert a still picture to animation by providing a few user drawn scribbles as input, saving users' time and effort [46] . Lee et al. implement a simulation platform that accepts high-level design commands and generates corresponding virtual mannequins in real-time [23] .
Research also has shown that human-AI collaboration can enhance tasks in the medical domain. For example, Cai et al. propose a CBIR system where doctors can query similar histological images with a user-adjusted refine-by-concept tool to support differential diagnosis with digital histology [6] . Further, integrating human domain knowledge into the AI pipeline would enable humans to 'supervise' and enhance AI via interactive machine learning. Apart from saving human's effort, this could also help eliminate AI's erroneous predictions in high-stake medical applications. For example, Ilastik [42] enables the user to draw strokes over images for training segmentation models. The system can automatically recommend the most critical features to reduce overfitting.
HistomicsML is an active learning [39] system that dynamically queries the most uncertain patches from a random forest classifier, thus allowing pathologists to refine the classification model with fewer samples iteratively.
To sum up, the systems mentioned above succeed in reducing human workload by computer automation. However, little prior work has explored the collaborative relationship between pathologists and AI or how to integrate AI into pathologists' existing workflow. To fill this gap, our research investigates how a pathologist would corporate with AI in a multi-criteria medical application by enabling the physician to 'shepherd' AI in a top-down, evidence-based manner.
We focus on meningioma grading task to probe the design of medical human-AI collaborative systems. According to WHO guidelines, meningioma can be graded as I (benign), II (atypical), and III (malignant). Grade I meningioma are recognized by their histological subtype and a lack of anaplastic features.
Grade Grade III meningiomas have 20 or more mitosis per 10 HPFs or have the histological appearance of carcinomas, melanomas, or high-grade sarcomas. [3, 25] .
Importantly, the clinical treatment of grade I and grade II/III is different: the grade I lesions can be treated with either surgery or external beam radiation, while grade II/III lesions often need both [45] . Further, a study shows that grade II/III patients experience worse prognosis even after critical treatments [31] . Given the treatment and prognosis difference amongst different tumor grades, it is crucial to support meningioma grading. The first and foremost way to examine meningiomas is by examining tissue slides with H&E staining. In addition, Ki-67 staining ( Figure 2i ) is often used as an additional reference since the Ki-67 proliferation index is related to mitosis and is reported as positively correlated to meningioma grading [1] . Such multiple criteria help pathologists examine comprehensively on meningioma grading; an erroneous grading comes with a high cost: either an overestimated case would incur unnecessary treatment on patients, or an overlooked case would result in a delay of necessary treatment.
Tizhoosh et al. summarized ten challenges in AI-aided digital histology at a high-level [43] . The paper mentioned that diagnostic tasks are 'non-boolean' and diagnosis cannot be over-simplified into a 'yes' or 'no', as machine learning tasks usually do. Tizhoosh et al. also point out that the current uni-tasked, weak AI cannot perfectly achieve complex, multicriteria tasks. Finally, a lack of transparency and interpretability make data-driven black-box algorithms hard to understand and hence result in an untrustworthy diagnosis. To further understand where and how those pitfalls might occur in AIaided pathology, we conducted a formative study with four pathologists in a local medical center.
We started by describing the motivation and mission of the project. We then asked pathologists to describe their typical process of examining a patient's case. Next, we asked what were the major challenges in existing pathology practice, and further inquired about their expectations on AI-aided automatic diagnostic systems.
We found that there are two major challenges in current pathology practice in the grading of meningioma.
Time consumption For grading of meningioma, the sparse characteristics and narrow view of high power magnification result in the time-consuming process of examining the slides for the pathologists. A biopsy section from a patient brain tissue would generate eight to twelve H&E slides. Pathologists need to look through all those slides and integrate the information found on each slide. Specifically, pathologists need to follow the sampling guideline and search for histological features in the high power field. While the grading of meningioma is a relatively easy task for experienced pathologists, for the majority of others, the grading is extremely time-consuming and tedious, taking up to one or several hours to go through a single patient's case.
Subjectivity We found three factors that contribute to subjectivity: (i) subjectivity due to a lack of precise definitions: the WHO guidelines do not always provide a quantified description for the five histological features for high-grade meningioma. For example, the 'prominent nucleoli' criteria requires pathologists to report prominent nucleolus clusters. However, the WHO guideline does not specify how large the nucleolus should be to make them recognized as 'prominent'. Hence, pathologists often rely on their own experience to decide;
(ii) subjectivity in implementing the examination process. For example, the mitotic count for grade II meningioma is defined as >4 mitotic cells in 10 HPFs. However, the guideline does not specify the sampling rules of finding those 10 HPFs. As a result, different pathologists are likely to sample different areas on the slide; (iii) subjectivity due to the personal factors, such as the level of experience, time constraint, and fatigue [11] . The subjectivity in the current pathology workflow would result in inter-observer variance in diagnosis, and this would be fatal given the high stakes in medical diagnosis.
Based on the challenges mentioned above, we identify the following system requirements based on further discussions with pathologists.
Comprehensiveness The time consumption in current practices is in part due to the need of examining multiple criteria, especially to justify high-grade meningioma, e.g., hypercellularity, necrosis, small cell, prominent nucleoli, and sheeting.
To reduce workload and time spent for examination, the system should similarly provide support on all these different criteria based on multiple types and sources of data.
Explainability One important way to overcome subjectivity is making the grading process transparent. The system should show visual summarization of AI's automated process and recommend to pathologists highly-suspicious regions-of-interest for non-quantified features in WHO guidelines. Another important approach is to provide evidence-based justificationevidence for each criterion found by AI that can be crosschecked by pathologists.
Integrability Given its existing limitations, the system should enable pathologists to collaborate with AI to conclude a grading diagnosis. Specifically, one suggested way of integration is to 'shepherd' AI at the high level-by examining humanreadable intermediate results, and validate, cross-check with peers, or override AI's findings. Such a shepherding relationship is similar to how currently attending pathologists delegate work to and oversee their trainees.
Corresponding to the meningioma grading criteria suggested by WHO [25] , we first worked with pathologists and constructed five meningioma datasets for training. Then we train mixed AI models (detailed below) to examine a total of eight criteria. We developed a front-end interface to integrate the evidence found by AI into the H&E and Ki-67 slides, which we describe in this section.
To achieve comprehensiveness, we followed the WHO meningioma grading guideline and automates eight criteria for meningioma grading, i.e., mitotic count, Ki-67 index, hypercellularity, necrosis, small cell, prominent nucleoli, sheeting, and brain invasion. We constructed five datasets with pathologists for model training. Specifically, we constructed a mitotic nuclei dataset for mitotic cell detection, a nuclei segmentation dataset for hypercellularity detection, a necrosis dataset for necrosis detection, a prominent nucleoli dataset for prominent nucleoli detection, and a meningioma subtype dataset (meningothelial, fibrous, other) for sheeting pattern recognition.
In order to train the models, we first split the five datasets into training and validation subsets. Then, we trained a mitotic nuclei identification model, a nuclei segmentation model (for hypercellularity), a necrosis identification model, a prominent nucleoli identification model, and a sheeting identification model with the corresponding training sets. For the Ki-67 index, we used positive cell count function from [15] to detect both Ki-67 positive and negative nucleus. Due to a lack of annotated data, we selected the top-10 512 × 512 × 3-pixel patches that have the highest nuclei count within each slide as small cell recommendations. Finally, we used a rule-based method to differentiate the tumor and non-tumor areas for brain invasion criterion. Please refer to Appendix A for implementation details.
To better organize AI results on the eight criteria, we worked with pathologists to obtain their examination priority. Given the current limitations of AI, CrossPath splits all eight criteria into two categories based on WHO guidelines: deterministic and suggestive. For the highly-prioritized criteria with quantized values, i.e., mitotic count, Ki-67 index, CrossPath demonstrates deterministic values directly (Figure 3b ). For the rest of the criteria (that come with present/not present), Cross-Path provides recommendations to pathologists. Specifically, CrossPath calculates regions-of-interest (ROIs) that present the probable local regions in four histological criteria (i.e., hypercellularity, necrosis, small cell, prominent nucleoli). Each criterion follows different ROI sampling rules, and the areas of each ROI might vary from one to another. For example, a hypercellularity ROI is defined as a cluster that has more than 2000 nuclus/1HPF (Figure 4e ). Please refer to Appendix B for details on other criteria's sampling rules. Figure 4 (e, f, g, h) highlights the ROI samples. With the aid of ROI samples, pathologists can prioritize their examination on an area of interest with respect to a specific criterion without exhaustively scanning the WSI looking for a starting point.
Crosspath provides a top-down, three levels of explanations to illustrate a high-level line of reasoning without missing connections to localized areas.
At the high-level, CrossPath provides two visual cues to explain the how the system follows the WHO guidelines: first, CrossPath displays an arrow that highlights the main contributing criterion. As shown in Figure 3b , the arrow indicates that the current suggested 'WHO grade II' result ( Figure 3a) is generated based on the mitotic count criterion. Second, CrossPath uses different color bars to distinguish the aforementioned deterministic vs. suggestive criteria. Specifically, deterministic criteria have red or green color bars where red is for findings that indicate higher grade (Figure 3c) ; the suggestive criteria display orange bars. For the orange suggestive criteria, pathologists can drill down to examine mid-level samples. If a pathologist sufficiently overrides AI's results, the suggestive criteria will be changed to deterministic and the color bar would be updated correspondingly. The gray bar indicates that the corresponding criterion is not available in this case.
To better help pathologists focus on localized regions and increase intra-observer consistency, CrossPath samples and demonstrates evidence found by AI. For example, for the most important mitosis criterion, CrossPath provides the following two 'shortcuts' for pathologists to look into evidence of AI's results.
Highest Focal Region Sampling From our formative study, we found that the high-grade meningiomas share a common feature of increased mitosis in a localized area. Hence, Cross-Path provides the highest focal sampling tool to help pathologists better localize highly concentrated mitosis/Ki-67 index areas. In CrossPath, the highest focal region is calculated as the 1HPF that has the most mitosis (Figure 4a ) or highest Ki-67 index (Figure 4b) . With the aid of the highest focal region sampling tool, pathologists can verify AI's conclusion of a worst-case count of mitotic activity.
Highest Region Sampling One criterion of the WHO meningioma grading guideline is "mitotic count in 10 consecutive HPFs". In our formative study, we found that the interobserver consistency of "10 consecutive HPFs" is low. In order a b c Figure 3 . High-level logic of CrossPath: (a) the overall suggested grading generated automatically; (b) a structured overview of each WHO criterion, the arrow highlights the main contributing criterion to the suggested grading. For each criterion, the red bar stands for the criterion has a deterministic value while orange ones still need manual examination. The gray bar indicates the criterion is not available in this case; (c) users can override the orange criteria by right-clicking on each item, and change to 'found', 'not found' or 'uncertain'. The overridden criterion becomes deterministic after manual verification, and the suggested result is updated accordingly.
to improve inter-observer consistency, CrossPath provides the highest region sampling tool, where the highest region is defined as a 2 × 5HPF area that contains most mitotic nucleus (Figure 4c ) or the highest Ki-67 index (Figure 4d ). The highest region sampling tool speeds up a pathologist's work by helping them locate an area of interest that consist of 10 consecutive HPFs, where they can then see evidence of AI's examination on these 10 HPFs.
CrossPath supports registering each piece of the mid-level evidence into WSI to enable pathologists examine in even higher magnification. For example, as shown in Figure 5 , pathologists can select the AI-generated prominent nucleoli ROI (a) as CrossPath registers the corresponding evidence into the original WSI (b), and pathologists can examine the evidence in high magnification for low-level details (c).
Building off of the explainable design, Crosspath establishes a shepherding workflow based on how attending physicians oversee trainees' work.
A top-down shepherding workflow As shown in Figure 6 , pathologists' shepherding workflow with CrossPath is topdown, and it starts from the top-level suggested result ( Figure  6a ). Then, pathologists continue to examine the main contributing criterion (Figure 6b) , and further the evidence for the criterion (Figure 6c ). Next, pathologists shepherd AI by drilling down and tracing back to the original area on the WSI (Figure 6d (Figure 7f ) that shows meningothelial vs. fibrous subtype, and a brain invasion heatmap (Figure 7g ) with tumor vs. non-tumor regions. If the mid-level evidence samples are insufficient for them to verify AI's results, pathologists can go beyond the sampled areas and navigate the high-heat areas using the heatmap. Importantly, the heatmap would be used as a 'screening tool' to help pathologists rapidly narrow down to a localized region for more evidence without having to scan the entire WSI. approve/decline buttons (Figure 6f ) after they have examined the evidence within the WSI context. For the five histological patterns as well as brain invasion criteria, pathologists can directly override the AI-recommended results by right-clicking on each criterion (Figure 3c) . Correspondingly, the overall suggested grading will be updated if the manually-overriden grading indicates a different result based on WHO guidelines.
CrossPath uses eight models to detect various histological patterns that are helpful for pathologists to reach a diagnosis. In this section, we report the performance of AI with the validation dataset. Specifically, we validated the mitotic, necrosis, prominent nucleoli, and sheeting criteria with a corresponding validation set (mitotic count: 39 Due to a lack of data at present, for brain invasion and small cell patterns, rather than drawing a conclusion, CrossPath uses a rule-based, unsupervised approach to recommend areas for pathologists to examine and we discuss the performance on these two criteria later by referring to how pathologists in our study made use of the recommendation.
We conducted work sessions with pathologists using Cross-Path. The main research questions are:
• RQ1: How do pathologists interact with CrossPath? How do they process AI-generated results across multiple criteria? How do they navigate the top-down structure to view AI's results and evidences that support such results?
• RQ2: Does CrossPath reduce pathologists' effort compared to their existing workflow? How much time would a pathologist spend on one WSI in CrossPath compared existing practice? How much cognitive effort do pathologists perceive using CrossPath compared to manual examination?
• RQ3: Does CrossPath add value if adopted to pathologists' existing workflow? What is pathologists' perceived value of CrossPath: what works better than manual examination and what does not?
We recruited seven pathologists from a local medical center. The participants had experience in pathology from one to 22 years (mean=6.3), including three attendings and four senior residents. Three participants had a background in machine learning, and three had experience of digital whole slide imaging interfaces. Three of the participants examined meningioma slides weekly, three of them had done within half a year, and one of them had not practiced grading meningioma for over a year.
We collected three meningioma cases (two grade II, one grade III) from a local medical center. The ground truth grading decision were recognized by a board of pathologists. Due to the pathologists' limited availability of time, we only used a small number of slides in each case: the two grade II cases had both one H&E and one Ki-67 slide, while the grade III case only had one H&E slide. Due to the COVID-19 pandemic, all sessions were conducted online. Pathologists used Zoom's remote control to interact with CrossPath that ran on an experimenter's computer.
We first introduced the background of CrossPath and provided a detailed walkthrough of the system for each participant using a sample case that have one H&E and one Ki-67 slide. After that, we ran a timed section for the pathologists to grade the three meningioma cases that contained three H&E slides and two Ki-67 slides in total. For each case, the time was counted from when participants first click the WSI case until they had reached the grading diagnosis. After participants had finished grading all three cases, we asked participants to self-report their estimated time consumption with the three cases using the optical microscope. In this study, we did not compare CrossPath with traditional digital interface because participants' experience with such interface varied. A comparison also would not fit into the limited time each pathologist was available to spend in our work session. After the participants had examined all the cases, we conducted a semi-structured interview to elicit pathologists' response to CrossPath's perceived effort and added-value. The average duration of each work session was about 50 minutes.
We measured CorssPath's usability through five-level Likert scale on various topcis, listed as follows:
• Comprehensiveness: Participants answered the following three questions: C1:"The multi-criterion H&E features found by the system are useful to assist the diagnosis", C2:"The system gives a global view of histological features", C3:"Overall, the system is comprehensive";
• Explainability: Participants answered the following three questions: E1:"The evidence found by the system can explain the summative histological features", E2:"The suggested WHO grading generated by the system is explainable", E3:"Overall, the system is explainable";
• Integrability: Participants answered the question: I1:"It is easy to integrate the AI evidence and result into the current clinical process"
• Workload: Participants answered questions followed by NASA Task Load Index [16] in three dimensions (W1: mental demand, W2: temporal demand, and W3: effort);
• Integrity: Participant answered the question: T1:"Overall, based on the shown case study, I believe the system's suggested diagnosis matches my justification"
• Future use: Participants answered the question: F1:"If approved by the FDA, I will continue to use the system in my practice";
In this section, we summarize the result as well as the recurring themes found in the working sessions. We further discuss three design implications for future AI-aided diagnostic systems.
With the aid of CrossPath, all participants agreed with the two grading II cases, as suggested by CrossPath. For the grade III case, two (P4, P6) agreed with the machine-suggested grade III, four (P1, P2, P3, P7) agreed that the case belongs to a highgrade meningioma (grade II/III) but need more verification from the glass slide, and one (P5) downgraded the suggested diagnosis to grade II by overriding a number of mitotic counts shown in the evidence panel.
The timed session shows that each pathologist spent 5.61 minutes (std=1.17) on average on each slide, which is 33.4% lower than manual examination with traditional glass slide (8.43 minutes). Such a time reduction should factor in that pathologists were still learning CrossPath where the time spent on each slide included asking for clarifications and questions. Further, it should be noted that the time reduction was achieved despite the latency issues caused by the remote study setup. Table 1 summarizes the metrics of user response from the post-study. Based on our observations from the work sessions with pathologists, we present our findings below, which are summarized into five themes.
How pathologists use CrossPath's multiple criteria: prioritizing one, referring to others on demand We observed that if a specific criterion did not satisfy the bar of pathologist to make a diagnosis for a higher grading, pathologists would use CrossPath to browse other criteria, looking for evidence of a differential diagnosis, until they identify sufficient evidence to support their hypothesis. P3 valued such an availability of multiple criteria: "it's kind of a safety net to prevent me from under grading" (P3).
In the timed section, we observed that some pathologists started with focusing on some specific criteria to ascertain a probable diagnosis as quickly as possible. "Looks like there is a brain invasion, and I agree with grade II" (P1). ". . . to be certain, I would like to go over the glass slide for the five histological features, but for me, it is enough for grade II" (P2). "I didn't look at this whole side by myself. But if I'm using just CrossPath and I believe the mitosis looks with a grade II" (P6).
However, some pathologists would also like to see other criteria and examine the slide comprehensively-"That already is grade II by on the mitotic count, but we would also probably go look for these other criteria" (P3). "What I would want to do is take a look at the rest of it to make sure I'm not missing anything" (P4). I think I have already agreed based on the four mitosis in the HPF for grade II, but I still want to try and navigate the other functions (P5). Pathologists did not treat all criteria provided by CrossPath equally. Pathologists would prioritize examining a specific criterion, e.g., brain invasion (P1), the outcome of which would then guide them to selectively find other criteria for making a differential diagnosis. Such a relationship between criteria is analogous to 'focus + context' in information visualization [7] -different pathologists might focus on a few different criteria, but the other criteria are also important to serve as context at their disposal for supporting an existing diagnosis or finding an alternative one.
How pathologists use CrossPath's multiple criteria: overlaying and cross-checking one another Although the WHO guideline does not specify a quantified Ki-67 cut-off for meningioma grading, we observed that participants were able to incorporate the additional Ki-67 information with H&E using CrossPath (P2, P4, P6). More specifically, users overlapped the high Ki-67 index regions with the mitotic density heatmap to examine whether the AI result is accurate at a high-level. We also noticed a pathologist (P2) could use the Ki-67 information provided by the system to improve their diagnosis-"It did not really look like a grade I as I started with, ..., I think the mitosis and the higher Ki makes me think it is not a grade I" (P2).
CrossPath's top-down flow enables pathologists to navigate between high-level AI results & low-level WSI details Amongst the seven histologic criteria provided by CrossPath, participants seem to all agree that automating the mitotic count criteria would best save their time-"looking for mitosis is hard so that program really speed up [what] we're actually looking at" (P6). One of the main reasons that the limits the throughput of histological diagnosis is that criteria like mitotic count presents very small-scaled histologic features. As a result, pathologists have to switch to high power magnification to examine such small features in detail. Given the high resolution of the WSI, it is possible to 'get lost' in the narrow scope of HPF, resulting in a time-consuming process to go through the entire WSI (low throughput). With CrossPath, pathologists found its top-down flow uses specific evidence to bridge high-level AI results at a global overview and WSI's low-level details in a narrow scope. "You really have to go to high power to search for all the mitosis. So the fact that it (CrossPath) highlights the highest region and then you can just quickly look through that area or those evidence boxes. I think that's really helpful" (P5).
CrossPath's explainable design helps pathologists see what AI is doing (wrong) One way of overcoming the subjectivity is promoting transparency. Given the non-perfect behavior of the AI algorithms, providing evidence-based justification was shown to be helpful for pathologists to utilize AI's results-"It (CrossPath) shows areas of interests, I think it is most helpful for us, cause if it shows there is necrosis, and it does not show us where, then we cannot trust it" (P1). In CrossPath, users encountered falsepositive pieces of evidence in mitotic and necrosis criteria. The sampled evidence provided by CrossPath helped the users to understand what patterns that the AI was trying to pick up-"For necrosis, I think it's looking for pink type stuff, but it's making the mistake of picking up on some collagen, and then there are probably not necrosis... so even when it's wrong it's explainable" (P3). We also observed that explainablility helped pathologists understand AI's limitations and how falsepositive cases would occur-"Even under the microscope, a lot of apoptotic bodies can look like mitotic figures, and I think the system was bringing up a combination of those two ... Things like that so I can understand why the system was doing what it was doing" (P4). "I think the computer is trying to find the dense dark areas because that's kind of essentially what a mitotic figure is. But of course, that's going to catch other things-sometimes it's not it could just be like an apoptotic body" (P7).
Shepherding AI with CrossPath: pathologists are able to but do not often confirm or override AI's result Given the explainable evidence provided by CrossPath, it was easy for pathologists to recognize false-positive cases and rapidly override them-"I'm going to downgrade this to a WHO grade II based on the histologic features vs. the mitotic. Because I think this thing (the AI) picked up more of the neutrophils rather than the mitosis" (P5). However, pathologists did not often confirm/override AI's results on specific piece of evidence by clicking on the approve/decline buttons or modifying AI results directly on the criteria panel. We hypothesize that this is in part related to how pathologists generate their clinical report: pathologists make comments on founded histological features if they have found them. When they were using CrossPath, pathologists could pull the correct piece of evidence into their report directly; otherwise, if the pathologists regard the piece of evidence as false-positive, by training they do not need to do anything since it would not appear in the report. On the system level, one way to incentivize pathologists' input is to make the manually overridden evidence more useful: the AI can pick up the new annotations and retrain the AI model. Correspondingly, the system updates the new evidence interactively and shows the pathologists better and refined evidence based on their annotations.
Another tool that CrossPath provides to help pathologists shepherd AI is the heatmap visualization of the criteria. In our study, the users expressed that the heatmap visualization is helpful for them to navigate the WSI and select where to look at when verifying AI's result, even though it took some effort to understand it-"I think the heat maps are helpful, but I think it took me a little bit to understand what exactly was trying to do" (P4). "It (the heatmap) light up wherever they (mitosis) are" (P5). "I feel like the sheeting, I thought that was a really cool idea that the machine could kind of show where the tissue looks the same versus different" (P6).
We discuss three design implications based on the user feedback from CrossPath. These implications are not limited to meningioma grading and are expected to generalize to other medical applications, such as AI-aided histological image processing, or X-ray/CT/MRI image processing with AI. We further discuss limitations that lead to potential future work to improve the current design and implementation of CrossPath.
Focus+context design Medical diagnosis involves accumulating evidence from multiple criteria-our study observed that pathologists started with a focus on one criterion while continue to examine others for a differential diagnosis. Thus medical AI systems not only should make multiple criteria available, but also should support the navigation of such criteria following a 'focus+context' design [7] . One opportunity is to design 'focus+context' visualization of multiple criteria, where the major challenge is to strike a balance between juxtaposing the focused criterion with sufficient amount of contextual criteria without overwhelming the pathologists with too much information. It is also possible for a system to, based on a patient's prior history and a preprocessing of their data, recommend a pathologist to start with a focus of certain criteria followed by examining some others as context.
Providing evidence for AI's result Given the current limitations of AI algorithms, AI-aided diagnostic systems should provide evidence for physicians to verify AI's results. Similar to CrossPath, the system can provide multi-level explanatory evidence. For example, CrossPath offers three levels of explanations: a high-level logic based on WHO guidelines, a mid-level explanation-by-example evidence sampling, and a low-level registration to specific WSI features. The multilevel evidence can be especially helpful for the diseases that follow multiple criteria and are diagnosed with rule-based guidelines. With multi-level evidence, the users can navigate in a top-down manner and rapidly obtain an estimation of AI performance as well as mistakes, which also supports gaining trust of AI-aided diagnostic systems.
Making interactions useful The ultimate goal of medical AI is to reduce the workload of medical professionals and help them reach a diagnosis faster. While the fully automated diagnosis is still far under the horizon, the current AI-aided diagnostic interface should be human-centered and integrable to specific domains, rather than AI-centered and abstracted from real-world scenarios. Hence, interactive features on an interface should offer medical users a clear, straight-forward and actionable implication without much extra effort to understand them. In CrossPath, we found pathologists tended not to use the 'approve/decline' and 'modifying AI result' functions, as the interaction of using such functions yield little benefits compared to others (e.g., seeing more evidence of AI's results). To overcome such limitation, making interactions clearly useful could be a solution: the AI can follow the physician-annotated results and update the diagnosis and evidence accordingly, which could serve to better incentivize physicians.
Currently, CrossPath does not support adjusting the threshold with the front-end directly. In our user study, a participant is interested whether they can adapt the prediction threshold -"Some of the (mitosis) is hard to be certain, each pathologist has his own threshold" (P2). Furthermore, dealing with false positives and false negatives is another issue with a fixedthreshold scheme as the system is. From our study, we found out that that doctors would prefer high-sensitive results that include some false positives rather than low-sensitive results that have false-negatives -"I guess it's better to do false positives and false negatives because we can always do the doublecheck like being more sensitive like that" (P5). Therefore, in future work, the system should enable adaptive thresholding and allow pathologists to adjust the threshold on the front-end. Further, the system should also provide as many ROIs as possible without fearing to show doctors false-positive results, since doctors are fast in examining ROIs whereas missing important features would come with a high cost.
Identifying what AI is good at vs. what human is good at has been a long-standing challenge in the HCI/AI communities. In our user study, we found that the pathologists did not treat all criteria equally, and they expressed that some automation (the brain invasion, for example) is not as helpful as others.
A naÃŕve solution would be partitioning the tasks into two dimensions, i.e., human-preferable tasks and AI-preferable tasks. However, the clear boundary of such a partition easily blurs in a human-AI collaborative environment. 
Mitotic Count We first collected 1183 (478 positive, 705 negative) nucleus and cropped them into 64 × 64 × 3pixel 2 patches. We then split the dataset with a 0.9/0.1 train/validation ratio. Next, we used a Keras implementation of the ResNet50 model [17] and train with Adam optimizer, binary cross-entropy loss, 100 epoch, and a batch size of 32.
To avoid false positive cases reflect the grading result, only hard-positive nucleus with prediction probability > 0.7 are counted as mitosis.
Ki-67 Index We used positive cell count function from [15] to detect both Ki-67 positive and negative nucleus. The Ki-67 index is calculated as positive-count positive-count+negative-count × 100% in 512 × 512 × 3-patch.
Hypercelluarity We first obtained 11 regions with cell segmentation with ground truth. Then we used random crop technique and constructed a training set with 1780 512 × 512 × 3-pixel training samples. Next we trained a U-Net nuclei segmentation network [33] with Adam optimizer, mean-Intersection-Over-Union metric, binary cross-entropy loss, 20 epoch, and a batch size of 8. We further used a dynamic watersheding method to post-process the U-Net segmentation result and finally got the nuclei counting result.
Necrosis We collected 26 necrosis regions and 37 nonnecrosis regions in total. For each region, we used a randomcrop method to obtain 512 × 512×3-pixel patches. Next we constructed a training set with 2004 patches (88 positive, 1916 negative, from 51 regions) and a validation set with 571 patches (52 positive, 519 negative, from 12 regions). We used a ResNet50 model and train with Adam optimizer, binary cross-entropy loss, 30 epoch and a batch size of 4.
Small Cell Due to a lack of annotated data, we selected the top 10 512 × 512 × 3-pixel patches that have the highest nuclei count within each slide as small cell recommendations. Note that the small cell also shares a hypercellular pattern, only patches that have >125 nuclei/patch are counted.
We collected 755 (206 positive and 549 negative) nucleus and cropped them into 64 × 64 × 3 pixel patches. We then split the dataset with a 0.9/0.1 train/validation ratio and trained with a ResNet50 model with the same hyperparameters as used in mitotic count identification. To avoid false positive cases influence the result, only 2 the dimension of each pixel is 0.2508µm nucleus that have >0.9 prediction probability are counted as positive.
Sheeting We collected 20 meningothelial regions, 12 fibrous regions and 12 other regions from 8 slides. For each region, we used a random-crop method to obtain 512×512×3-pixel patches. We constructed a training set with 2219 samples (719 meningothelial, 790 fibrous, 710 other) from 34 regions and a validation set with 262 samples (91 meningothelial, 70 fibrous, 101 other) from 10 regions. We then trained a ResNet50 model with the same hyperparameters as the necrosis detector.
Brain Invasion Since meningioma is a high-cellular tumor, we used Otsu's thresholding method [30] to differentiate the tumor and non-tumor brain tissues with cell-counting: only the patches that have more than 55 nucleus/512×512-pixel patch are counted as tumor patch.
Specific thresholds used for each criterion was obtained via discussion with a pathologist. We trained the models with the datasets on a CentOS 7 server, with Intel Xeon W-2133 CPU, 104 GB memory, and two Nvidia RTX-2070 graphics cards.
Below are the ROI sampling rules used in CrossPath:
• a hypercellularity ROI is defined as a cluster that has more than 2000 nuclus/1HPF; • a necrosis ROI is defined as a cluster that has probability >0.7 from necrosis detector; • small cell ROIs are defined as top 10 512 × 512-pixel patches that have the most nuclei count over the slide; • a prominent nucleoli ROI is defined as a cluster that has more than 60 prominent nucleolus/1HPF;
The areas of hypercellularity, necrosis, and prominent nucleoli ROIs are calculated by DBSCAN clustering algorithm [13] . If no matching ROI is found on one slide, CrossPath's corresponding evidence list would be displayed as 'no evidence found'.