Auditory grouping occurs prior to intersensory pairing: evidence from temporal ventriloquism

The authors examined how principles of auditory grouping relate to intersensory pairing. Two sounds that normally enhance sensitivity on a visual temporal order judgement task (i.e. temporal ventriloquism) were embedded in a sequence of flanker sounds which either had the same or different frequency (Exp. 1), rhythm (Exp. 2), or location (Exp. 3). In all experiments, we found that temporal ventriloquism only occurred when the two capture sounds differed from the flankers, demonstrating that grouping of the sounds in the auditory stream took priority over intersensory pairing. By combining principles of auditory grouping with intersensory pairing, we demonstrate that capture sounds were, counter-intuitively, more effective when their locations differed from that of the lights rather than when they came from the same position as the lights.
Introduction
Sense organs like the ears and eyes are continuously bombarded with information. Yet, observers perceive distinct objects or events. The way information is assigned to objects has, for vision, been described with Gestalt principles like ‘similarity’, ‘good continuation’, or ‘common fate’, and similar principles have also been discovered for audition (Bregman 1990). It occurs, for instance, when a sequence of alternating high- and low-frequency tones is played at a certain rate. When the frequency difference between the tones is small, listeners group the tones into a single stream, but at bigger frequency differences, the sequence splits into two streams, one high and one low in pitch. Typically, these grouping principles apply within a single modality like vision or audition. However, sense organs not only work in isolation, they also have to cooperate to form a coherent multisensory representation of the environment. The notion on how information from different sense organs is assigned to a multisensory event is usually referred to as the ‘assumption of unity’. It states that as events from different modalities share more amodal properties, in particular space and time, it is more likely that they originate from a common object or source (e.g. Welch and Warren 1980; Bedford 1989; Stein and Meredith 1993; Radeau 1994; Bertelson 1999; Welch 1999). Following this notion, the assignment of information from different modalities to a single multisensory event will be reduced or absent when stimuli are too far apart in space or time, because in that case two objects or events will be perceived rather than a single multimodal one.
Here, we explored how principles of auditory grouping relate to intersensory pairing. Previous work on this topic suggests that auditory grouping may take priority over intersensory pairing. For example, Vroomen and de Gelder (2000) used a task in which participants had to detect a visual target in a rapidly changing sequence of visual distracters. They observed that a high tone embedded in a sequence of low tones enhanced detectability of a synchronously presented visual target. There was no such intersensory enhancement when the tone was embedded in a sequence of tones with the same frequency or when the tone was part of a melody. The cross-modal enhancement thus only occurred when the sound segregated from the sound sequence in which it was embedded. Similar results were obtained by Watanabe and Shimojo (2001). They explored how the ‘bounce illusion’ is affected by contextual auditory information. The bounce illusion is a cross-modal phenomenon in which a ‘collision’ sound presented near the crossover of two moving balls enhances the perception of the balls ‘bouncing’, whereas the absence of the sound results in a ‘streaming’ percept. The authors showed a reduction of the bounce illusion when the sounds were embedded in a sequence of similar sounds, as opposed to when the sounds were flanked by sounds of a different frequency.
Here we tested the generality of these findings by examining how auditory grouping affects auditory–visual (AV) pairing in the temporal domain using the so-called temporal ventriloquist effect (Scheier et al. 1999; Fendrich and Corballis 2001; Aschersleben and Bertelson 2003; Bertelson and Aschersleben 2003; Morein-Zamir et al. 2003; Vroomen and de Gelder 2004; Stekelenburg and Vroomen 2005; Vroomen and Keetels 2006). Temporal ventriloquism refers to the phenomenon that when a sound and light are presented at slightly different onset times (usually in the order of ∼100 ms), the sound will attract the temporal occurrence of the light. This phenomenon can be demonstrated in a visual temporal order judgment (TOJ) task in which participants are presented two lights at various stimulus onset asynchronies (SOAs) and judge, which came first. By presenting a sound before the first and after the second light, the just noticeable difference (JND) improves (i.e., participants become more sensitive), presumably because the two sounds attract the temporal occurrence of the two lights, and thus effectively pull the lights further apart in time (Scheier et al. 1999; Morein-Zamir et al. 2003; Vroomen and Keetels 2006). Judgments about which light came first are therefore more accurate if there is a ∼100 ms interval between the sounds and lights rather than when the sounds are presented simultaneously with the lights.
Here we asked what happens if the sounds that capture the onset of the lights are assigned to a stream of other sounds with which they form a well-formed sequence. If auditory grouping takes priority over intersensory pairing, one expects an improvement on the visual TOJ task only if the capture sounds segregate from the auditory stream. Alternatively, though, audio–visual pairing might take priority over auditory grouping in which case observers should improve on the visual TOJ task no matter whether the sounds segregate or not.
In Experiment 1, these predictions were tested with two capture sounds that were embedded in a sequence of flanker sounds, which either had the same or a different frequency. When the frequency of the flanker and the capture sounds were the same, the sequence was heard as a single stream which, following previous findings (Vroomen and de Gelder 2000; Watanabe and Shimojo 2001), should prevent temporal ventriloquism to occur. When the flankers differed from the capture sounds, stream segregation was more likely to occur in which case the two sounds could possibly interact with the lights and thus improve performance on the visual TOJ task. Other stream segregation cues besides the frequency of the flanker and capture sounds were further explored in Experiment 2 (rhythm) and Experiment 3 (sound location).
Experiment 1: Capture and flanker sounds with the same or different frequency
Participants performed a visual TOJ task in which they decided which of two lights appeared first. Two task-irrelevant high tones were presented either simultaneously with the lights (i.e., the ∼0 ms AV interval), or the first tone was presented ∼100 ms before the first light and the second tone ∼100 ms after the second light (i.e., the ∼100 ms AV interval). The two high tones were embedded in a sequence of other tones, which either had the same (high) or a different (low) frequency (see Fig. 1a for a schematic overview of the conditions).
Fig. 1a A schematic illustration of a trial. Lights were presented with a particular SOA ranging between −75 and 75 ms, with negative values indicating that the lower light was presented first. Two capture sounds were presented either simultaneously with the lights (∼0 ms AV-interval), or ∼100 ms before the first and ∼100 ms after the second light (∼100 ms AV-interval). The capture sounds were embedded in a sequence of flanker sounds which either had the same or a different frequency (Exp1), rhythm (Exp 2), or location (Exp 3). Perceptual grouping of the sounds is illustrated by the grey ovals. Temporal ventriloquism only occurred when the capture sounds were different from the flanker sounds at the ∼100 ms interval, as illustrated on the second line. b Schematic set-up of Experiment 1, 2 and 3
Method
Participants Thirteen students from Tilburg University were given course credits for their participation. All reported normal hearing and normal or corrected-to-normal seeing. They were tested individually and were unaware of the purpose of the experiment. The experimental procedures were approved by the Institute and were in accordance with Declaration of Helsinki.
Stimuli Two auditory stimuli, a low (1,500 Hz) and a high (3,430 Hz) pure tone of 3 ms at 72 dB(A) were used that clearly differed in pitch. The sounds were presented via a hidden loudspeaker located at eye-level, at central location and at 90 cm distance. Visual stimuli were presented by two red LEDs (diameter of 1 cm, luminance of 40 cd/m2), positioned 5° below and above the central loudspeaker. A small green LED was placed at the center of the loudspeaker and served as a fixation point (see Fig. 1b for a schematic set-up). Trials in the ∼0 ms AV interval consisted of a sound sequence of 40 sounds in which the interval between the successive tones was equal to the SOA between the two lights. The lights were presented simultaneously with the 25th and 26th sound. Trials in the ∼100 ms AV-interval consisted of a tone sequence of 15 sounds with the interval between the tones equal to the SOA of the lights plus 200 ms. The two lights were presented in the middle of the temporal gap, ∼100 ms after the 10th and ∼100 ms before the 11th sound.
Design The experiment had three within-subjects factors: Frequency of the flanker sounds (same or different frequency as the capture sounds), the AV-interval between the capture sounds and the lights (∼0 or ∼100 ms), and the SOA between the two lights (−75, −60, −45, −30, −15, +15, +30, +45, +60, and +75 ms; with negative values indicating that the lower light was presented first). These factors yielded 40 equi-probable conditions, each presented 20 times for a total of 800 trials (10 blocks of 80 trials each).
Procedure Participants sat at a table in a dimly lit and soundproof booth. The fixation light was illuminated at the beginning of the experiment, and participants were instructed to maintain fixation on this central green LED during testing. The participant’s task was to judge whether the lower or the upper LED was presented first. Responses (unspeeded) were made by pressing one of two designated keys with the right thumb (lower light first) or right index (upper light first). Whenever a response was detected, both LEDs were turned off and the next trial started after 2,000 ms. A practice block was included consisting of 16 trials in which the four longest SOAs were presented once in each condition. During practice, participants received verbal feedback (“Correct” or “Wrong”).
Results and discussion
Trials of the practice session were excluded from analyses. The proportion of ‘up-first’ responses was calculated for each condition and converted into equivalent Z-scores assuming a cumulative normal distribution (cf. Finney 1964). For each of the four conditions, the best-fitting straight line was calculated over the ten SOAs. The lines’ slopes and intercepts were used to determine the point of subjective simultaneity (PSS) and the just noticeable difference (JND = 0.675/slope). The PSS represents the average interval by which the upper stimulus had to lead the lower one in order to be perceived as simultaneous. The JND represents the smallest interval between the onsets of the two lights needed for participants to correctly judge which stimulus had been presented first on 75% of the trials. Temporal ventriloquism was measured by subtracting the JND in the ∼100 ms AV interval from the ∼0 ms AV interval (see Table 1 for the average JNDs).
Table 1Mean just noticeable differences (JND) in ms, and standard errors of the mean (in parentheses) of Experiment 1 and 2Flanker soundsSame as capture soundsDifferent from capture soundsExperiment AV-interval (ms)JNDTVEJNDTVEExp 1 (Frequency)021.0 (0.8)0.522.1 (1.1)3.8*10020.5 (1.1)18.3 (0.7)Exp 2 (Rhythm)029.1 (2.5)−3.723.9 (1.9)3.6*10032.8 (2.5)20.3 (0.8)Capture Sounds Presented at ∼0 or ∼100 ms audio–visual intervals; flanker Sounds with the same or different frequency (Exp 1) or rhythm (Exp 2) as capture sounds. The temporal ventriloquist effect (TVE) is the improvement in JND between the ∼0 and ∼100 ms audio–visual intervals*P < 0.05
A 2 × 2 ANOVA with as within-subjects factors the frequency of the flanker sounds (same or different frequency as the capturer sounds) and the AV-interval (∼0 and ∼100 ms) was conducted on the JNDs and PSSs. In the ANOVA on the PSSs, no effect was significant (all P’s > 0.30), which is in line with our expectations since no shift towards more ‘up’ or ‘down’ responses was expected. In the ANOVA on the JNDs, the important interaction between the AV-interval and the frequency of the flanker sounds was significant, F(1, 12) = 4.90, P < 0.05. Separate t tests comparing the ∼0 ms AV interval with the ∼100 ms AV interval showed that JNDs improved by 3.8 ms when the frequency of the capture and flanker sounds differed, t(12) = 3.06, P < 0.01, but no significant difference was obtained (0.5 ms) when the capture and flanker sounds were the same, t(12) = 0.71, P = 0.49. As predicted, temporal ventriloquism thus only occurred when the capture and flanker sounds differed. It seems therefore likely that segregation of the capture sounds from the flankers was necessary before the capture sounds could interact with the lights. When the capture and flanker sounds were the same, auditory grouping thus took priority over AV pairing.
Experiment 2: Capture and flanker sounds with the same or different rhythm
To further explore the relation between auditory grouping and intersensory pairing, we presented the capture sounds in or out of rhythm with the flankers. Rhythm is, besides frequency, another important auditory segregation cue (Bregman 1990). It was expected that sounds presented out of rhythm would segregate from the sound sequence, thus enhancing performance on the visual TOJ task. Capture sounds presented in rhythm should not segregate from the auditory stream, and they should thus have no effect on the visual TOJ task.
Method
Participants Twenty new students participated.
Stimuli and design Stimuli and design were as in Experiment 1, except that the time interval between the capture and flanker sounds was varied rather than their frequency. The auditory stimuli consisted of 5 m s sound bursts presented at 72 dB(A). When the capture sounds were presented in rhythm with the flankers, the same interval between consecutive sounds in the sequence was used as in Experiment 1 (i.e., SOA between the two lights + 2 × AV-interval). When the capture sounds were presented out of rhythm, the interval between the capture and flanker sounds was increased, such that there was a short pause before the first and after the second capture sound. In the ∼0 ms AV-interval condition, the two longer intervals were 8 × SOA of the lights, in the ∼100 ms AV-interval condition they were 17 × SOA of the lights +200 ms.
Results and discussion
In the 2 (Capture sounds in or out of rhythm) × 2 (AV-interval ∼0 or ∼100 ms) ANOVA on the PSSs, no effect was significant (all P’s > 0.30). The same ANOVA on the JNDs showed that the important interaction between AV-interval and rhythm was significant, F(1, 19) = 15.35, P < 0.001. Separate t tests showed that the 3.6 ms temporal ventriloquist effect (lower JNDs in the ∼100 ms AV interval rather than ∼0 ms) of sounds presented out of rhythm was significant, t(19) = 2.37, P < 0.05. Performance got actually worse (−3.7 ms) when the capture sounds were presented in the same rhythm as the flanker sounds, t(19) = −2.15, P < 0.05. Capture sounds thus again only improved performance on the visual TOJ task when they segregated from the flanker sounds.
Experiment 3: Capture and flanker sounds from the same or different location
It is known that auditory stream segregation may also occur when there is a difference in the location of consecutive sounds (Bregman 1990). In Experiment 3, we therefore varied the location of the capture and flanker sounds. The sounds could emanate either from a central loudspeaker near the two lights, or from a lateral loudspeaker on the far left or far right. If intersensory pairing only occurs when the capture sounds segregate from the flankers, then temporal ventriloquism should be obtained when the locations of the capture and flanker sound differ, but not when they are the same.
This set-up also allowed us to explore whether spatial disparity between the capture sounds and lights affects intersensory pairing. The common notion on intersensory pairing states that commonality in space between the auditory and visual signal matters. However, in contrast with this notion, it has recently been shown that temporal ventriloquism may not be affected by spatial discordance between the sounds and lights. In a study by Vroomen and Keetels (2006), it was shown that there were equal amounts of temporal ventriloquism when the two capture sounds came from the same or a different position as the lights, when the sounds were static or moved, or when the sounds and lights came from the same or opposite sides of fixation. Assuming that these results would be replicated in the present set-up as well, we expected sound location to be unimportant for intersensory pairing. Equal amounts of temporal ventriloquism were therefore expected from segregated sounds presented from the central location (near the lights) and the lateral location (far from the lights).
The notion that sound location matters for auditory grouping, but not for intersensory pairing also lead to a very counter-intuitive prediction. In case flanker sounds were presented near the central lights, there should more temporal ventriloquism by capture sounds presented from a lateral position than from central position, because only the lateral sounds segregate. With central flankers, there should thus be more temporal ventriloquism when the location of the sounds and lights differ, rather then when they are the same.
Method
Participants, stimuli and procedures were the same as in Experiment 1, except for the following changes. Eighteen new students from the same subject pool participated. The auditory stimuli consisted of 5 ms sound bursts presented at 72 dB(A), presented from one of two loudspeakers (see Fig. 1b). One speaker was located at central location at eye-level and at 90 cm distance (as in Experiments 1 and 2), the other was located on either the far left or the far right (at 90° azimuth). Four within-subjects factors were used: Location of the two capture sounds (central or lateral), Location of the flanker sounds (same or different position as the capture sounds), the AV-interval between the capture sounds and lights (∼0 or ∼100 ms), and the SOA between the two lights (−75 to +75 ms). The 80 conditions were each presented 20 times in 10 blocks of 160 trials each. In half of the blocks, the lateral speaker was on the left, in the other half it was on the right.
Results
A 2 × 2 × 2 ANOVA with as within-subjects factors Location of the two capture sounds (central or lateral), location of the flanker sounds (same or different position as the capture sounds) and the AV interval (∼0 and ∼100 ms) was conducted on the JNDs and PSSs. In the ANOVA on the PSS, there was an interaction between the location of the capture and flanker sounds, F(1, 17) = 6.31, P < 0.025, indicating that there were slightly more ‘up’ responses in trials in which the capture and flanker sounds were presented centrally (mean PSS = −1.84 ms) rather than in the other conditions (mean PSS = 2.61 ms), a finding for which there is no clear explanation.
In the ANOVA on the JNDs (see Table 2) there was no main effect of the location of the capture sounds, F(1, 17) = 1.67, P = 0.21, indicating that JNDs were unaffected by whether the capture sounds were presented centrally (near the two lights) or laterally. Most importantly, there was an interaction between the AV-interval and the location of the flanker sounds, F(1, 17) = 5.11, P < 0.05. Separate t tests confirmed that the temporal ventriloquist effect (better performance at ∼100 ms rather than ∼0 ms AV interval) was only significant when the flanker sounds came from a different position than the capture sounds. There was thus no temporal ventriloquism when the capture and flanker sounds came both from central or lateral positions (both P’s > 0.6), while there was a 3.9 ms improvement for central capture sounds with lateral flankers, t(17) = 2.175, P < 0.05, and a 3.1 ms improvement for lateral capture sounds with central flankers, t(17) = 2.55, P < 0.025. These two improvements were not significantly different from each other, t(17) = 0.52, P = 0.60. Moreover, as predicted, with central flankers, temporal ventriloquism by lateral capture sounds was bigger than that from central ones, t(18) = 1.798, P < 0.05.Table 2Mean just noticeable differences (JND) in ms, and standard errors of the mean (in parentheses) of Experiment 3Location of flanker soundsSame as capture soundsDifferent from capture soundsLocation of capture soundsAV-interval (ms)JNDTVEJNDTVECentral028.2 (2.1)−0.927.7 (2.0)3.9*10029.1 (2.3)23.8 (1.4)Lateral030.0 (2.2)1.729.0 (1.8)3.1*10028.3 (2.0)25.9 (1.8)Capture Sounds Presented at ∼0 or ∼100 ms audio–visual intervals from central or lateral location; flanker sounds presented from the same or different location as the capture sounds. The temporal ventriloquist effect (TVE) is the improvement in JND between the ∼0 and ∼100 ms audio–visual intervals*P < 0.05
General discussion
This study examined how principles of auditory grouping relate to intersensory pairing. Two capture sounds that normally enhance performance on a visual TOJ task (i.e. temporal ventriloquism) were embedded in a sequence of flanker sounds which could differ in frequency (Exp. 1), rhythm (Exp. 2), or location (Exp. 3). In all experiments, we found that temporal ventriloquism only occurred when the capture sounds differed from the flankers, and there was thus no effect when flanker and capture sounds were the same. Presumably, when the capture sounds differ, they segregate from the auditory stream, and only then they can be paired cross-modally with the lights. When the two capture sounds do not differ from the flankers, they are perceptually grouped in an auditory stream, in which case they lose their saliency and cannot interact cross-modally anymore.
These results are similar to previous findings, which have shown that intersensory interactions do not occur when sounds that normally enhance performance belong to another auditory group (Vroomen and de Gelder 2000; Watanabe and Shimojo 2001; See also Sanabria et al. 2004a, b). The results also imply that a sound can only be assigned to a single event: it either belongs to the auditory stream, or it is paired with the lights, but it cannot be assigned to both simultaneously. In this respect, it is analogous to many of the well-known ambiguous figure-ground displays (e.g., the Face-Vase illusion or the Necker cube), where it is known that only one interpretation of the scene can be maintained.
Another important finding was that commonality in space between the capture sounds and lights did not affect temporal ventriloquism. The temporal ventriloquist effect was thus equally big for segregated capture sounds that were presented near the lights or far away from the lights. A few other studies have demonstrated before that spatial disparity between sound and vision may not affect intersensory interactions (Welch et al. 1986; Bertelson et al. 1994; Stein et al. 1996; Colin et al. 2001; Murray et al. 2004; Vroomen and Keetels 2006). However, these studies always relied on null-effects, which entails the danger that they simply lacked the power to detect any effect of spatial disparity. Participants in previous studies might, for example, not have been able to perceive spatial disparity, or they might have learned to ignore it in the experimental task. Our findings, though, counter these arguments. By combining principles of auditory grouping with intersensory pairing, we were able to create a situation where the capture sounds were actually more effective when their locations differed from that of the lights rather than when they came from the same position as the lights. Within the same experimental situation, we thus demonstrated that sound location mattered for auditory grouping, but not for intersensory pairing. Such a finding makes it highly unlikely that sound location was not perceived or simply ignored. Rather, it becomes more likely that, at least in the temporal ventriloquist situation, commonality in space between sound and vision is not relevant for AV pairing.
This may, at first sight, seem unlikely, because after all, most natural multisensory events are spatially an temporally aligned, except for some minor variations in time or space that people are readily able to adjust (e.g. Vroomen et al. 2004). However, a critical assumption that underlies the idea of spatial correspondence for intersensory pairing is that space has the same function in vision and audition. This notion, though, is arguable as it has been proposed that the role of space in hearing is only to steer vision (Heffner and Heffner 1992), while in vision it is an indispensable attribute (Kubovy and Van Valkenburg 2001). If one accepts that auditory spatial perception evolved for steering vision, but not for deciding whether sound and light belong together, there is no reason why intersensory interactions would require spatial co-localization. Our results therefore have also important implications for designing multimodal devices or creating virtual reality environments, as they show that the brain can, at least in some cases, ignore intersensory discordance in space.

auditory grouping
intersensory pairing
temporal ventriloquism
temporal order judgment
multisensory perception


Design and Implementation of an Open Source Indexing Solution for a Large Set of Radiological Reports and Images

This paper hopes to share the insights we experienced during designing, building, and running an indexing solution for a large set of radiological reports and images in a production environment for more than 3 years. Several technical challenges were encountered and solved in the course of this project. One hundred four million words in 1.8 million radiological reports from 1989 to the present were indexed and became instantaneously searchable in a user-friendly fashion; the median query duration is only 31 ms. Currently, our highly tuned index holds 332,088 unique words in four languages. The indexing system is feature-rich and language-independent and allows for making complex queries. For research and training purposes it certainly is a valuable and convenient addition to our radiology informatics toolbox. Extended use of open-source technology dramatically reduced both implementation time and cost. All software we developed related to the indexing project has been made available to the open-source community covered by an unrestricted Berkeley Software Distribution-style license.
BACKGROUND
Whenever a person or an institution builds a large enough collection of anything, searching through it becomes tedious or even unfeasible. An obvious solution is indexing: the process of compiling a list of pointers to items in the collection based on alphabetical order or any other item attribute. Indexing can add useful metadata, which is data about data, to a collection. A well-known example is the Science Citation Index for papers published in academic journals as envisioned by Eugene Garfield in 1955.1
Hospitals typically generate a lot of data about patients, their examinations, and their treatments. Due to the patient-oriented workflow, hospital information systems have a strong patient-centric architecture; hence, problems tend to arise whenever information across patients is required. For purposes of business intelligence, data mining and data warehousing can provide an adequate solution. When cross-patient information is needed in a research or training context, collecting medical data will become labor-intensive or practically impossible if the relevant data only exist in a free textual form. A solution for this problem consists in building an index of all free-text words that allows for efficient queries.
In our institution, the university hospital closely associated with the largest Belgian university, close to two million radiological reports and corresponding images were made instantaneously searchable by implementing a language-independent, feature-rich indexing solution. This paper hopes to share the insights we experienced during designing, building, and running an indexing system in a production environment for more than 3 years. Extended use of open-source technology dramatically reduced both implementation time and cost.
METHODS
To achieve a complete indexing solution that is ready for a production environment, we designed a system consisting of five major sections: the report warehouse, the search engine, specific import filters, efficient indexing, and finally the user interface. These sections correspond to five discrete phases in the implementation scheme. An overview of the architecture is presented in Figure 1.
Fig 1.The architecture of the complete indexing system.
Report Warehouse
Indexing a large set of reports requires fast access to all text files to reduce the time needed to build such an index. Our radiology information system (RIS) did not comply with this rule because reports were stored in an Oracle database using a Lempel–Ziv related compression scheme.2 To counter this problem we decided to set up an independent report warehouse that did not have an impact on RIS performance.
We started out by recycling two 5-year-old servers and fitting both of them with enough hard disk capacity (160 gigabytes) and a small uninterruptible power supply. We chose to run the open-source Free Berkeley Software Distribution (FreeBSD) operation system,3 which had the important ability to maintain disk integrity after a crash or power outage. This technique, called soft updates, works by properly ordering file system metadata-writes to guarantee consistency and was introduced by Marshall Kirk McKusick.4 Both servers were configured as a high-availability cluster.
We achieve synchronization with the RIS through two parallel processes. First, a structured query language (SQL) statement interrogates the RIS to find out which reports were cancelled since the last synchronization. Subsequently, these cancelled reports are removed from the report warehouse. The second process is similar to the first; a list of new reports and reports updated with an addendum is obtained. Next, by means of another SQL statement, the report binary large objects are retrieved from the RIS database and decompressed. Alternatively, we could have been using a standardized, but more complex, method of capturing outbound Health Level 75 observation results unsolicited messages originating from RIS. At the time of implementation this method had the major disadvantage of only supporting converted flat text representations of the reports; meanwhile, our RIS gained the possibility to send out rich text format (RTF), retaining report layout and style. In an ideal situation, indexing should be built right into RIS; this way both extracting and synchronizing reports and data become obsolete. Our report warehouse only contains validated reports by policy.
Search Engine
With the advent of the Internet and ever-increasing computing power, indexing software became widely available. Our report warehouse made it possible to plug in and evaluate several search engines with a limited set of reports. Due to our characteristic European setting, language independence was needed to support Dutch, French, English, and German reports. The ability to include RIS data in the index as report metadata was also an important factor; this allows searching for (or sorting on) specific authors and requesting physicians and departments and other administrative information. Four more features we required were phrase and wildcard searching, Boolean logic, and the possibility to exclude common stop words. The section on semantics in this paper contains a detailed motivation for why these last four features are essential in a medical context.
Both commercial and open-source indexing engines were tested and compared, but in-depth evaluation reports have not been included here because they have become seriously outdated after more than 3 years. Commercial engines were difficult or impossible to integrate due to their closed-source nature. Most engines could be discarded quickly because of a missing feature or a time-consuming indexing process. Note that the Google Search Appliance6 was not available in Europe at that time. We stopped looking for other candidates after we evaluated the highly recommended Swish-e indexing engine,7 which had all the desired features and a very efficient indexing process.
Import Filters
Our evaluation showed that the filters provided by most search engines, including Swish-e, had issues with importing our reports. Most problems were related to file formats; old formats were converted incorrectly, international characters often disappeared, and automatically generated text fragments (Microsoft Word AutoText) were not being filled in. One issue was observed across file formats: hyphenated words did not always make it into the index in one piece.
Solving these problems required programming import filters for all historically and currently used file formats: both American Standard Code for Information Interchange flat text and flat text containing multiple international character sets, WordPerfect, Microsoft Word, and RTF. These in-house developed filters properly undo hyphenation and combine the imported report with corresponding RIS data into a single Extensible Mark-up Language (XML) file ready for indexing, an example of which is presented in Figure 2. At our institution there is no mandatory structure for radiological reports. Additionally, we have put a quality assurance process in place; whenever word processing software or a report template is changed, the import results are tested on international characters, automatically generated fragments, and hyphenation.
Fig 2Example of a XML file containing the imported report and its corresponding RIS metadata.
Efficient Indexing
Indexing without tuning becomes very computing-intensive as the collection of document content, and thus the number of word/document combinations, grows. Fortunately, limiting the size of the index itself strongly decreases the computing power needed. In our situation, patient and study IDs in the radiological reports were safely excluded from the index without loss of information; using the metadata from the RIS yielded the same results and was more efficient.
A second technique consists in excluding some, but certainly not all, of the very frequent words present in almost all documents. These common stop words were found by building an index and evaluating the significance of the thousand most frequent words. Amongst the 39 manually excluded stop words, good examples like “a,” “the,” “of,” and closing statements like “greetings” were found. The very frequent word “no” was not excluded because its meaning was highly significant in combination with other keywords.
In the section on semantics we will show that excluding common stop words leads to semantic advantages. We will also discuss building indexes in an incremental fashion in the section on scalability.
User Interface
Due to our completely digital and highly optimized reporting workflow,8 the only good place to put the search interface was right at the radiologists’ fingertips. To attain this, we created a full front-end integration with RIS and the picture archiving and communication system (PACS). By means of a generic search button, the interface can be launched from any RIS or PACS context. The integration also includes a security layer serving two distinct purposes. Firstly, our existing single-sign-on/single-sign-off mechanism allows radiologists and researchers to provide their authentication credentials using an encrypted hardware key, defying the need to type in many usernames and passwords. The second function of the security layer consists of auditing the search queries and the displayed reports.
Because Internet technology was being used, it was straightforward to make a web-based interface that resembles well-known Internet search engines. Such simple and user-friendly interfaces lower the threshold for new radiologists. As shown in Figure 3, users simply enter the desired keywords, possibly accompanied by parentheses and the binary operators “and,” “or,” and “not.” Wildcards are denoted by an asterisk, and quotes can be used for phrase searching. Between keywords or phrases, an implicit “and” is assumed. The user can quickly pick a previous query from his complete query history. The results can be sorted on a calculated page rank (default), descending and ascending report time, and RIS metadata. Useful query feedback is shown: the number of results, the query duration, and the excluded stop words. Fast reviewing of reports is essential; to meet this end, the user can choose between displaying summarized or full reports—in both cases, keywords are highlighted and the matching ranking score is displayed graphically. With a single click of the mouse, radiologists can open both images in PACS and the historical reports of the selected patient.
Fig 3.The user interface for radiologists and researchers.
Alas, trying to comprehend the ranking code of Swish-e is not for the faint of heart because of its inherent complexity and the speed optimizations used. The algorithms take into account the binary structure of the search query, details of keyword occurrences, and the relative frequency of words in the index. Luckily, unlike searching the Internet, queries in a radiological context tend to be very specific (see results), making fine-tuning of ranking unnecessary.
RESULTS
Preceding the go-live of our indexing solution, over one million reports and associated data were extracted from the RIS. Some filter fine-tuning was necessary to pass all reports through the layer of import filters. The initial import was spread over a 14-day period without performance penalties for production systems. At the time of writing, our report warehouse contained 1.8 million reports and corresponding administrative data, their XML versions and backup copies not included. The system holds 7.2 million files in total. The oldest report dates back to 1989, but not many reports of that era were available in digital form.
Each night the index is rebuilt completely, which facilitates early detection of disc decay because all XML reports get reread in the course of this process. Alternatively, incremental reindexing could have been used. Building a full index is very efficient; it only takes 76 min worth of CPU time on 8-year-old hardware to process 104 million words. Currently, the index contains 332,088 unique words in four languages, including names of patients and requesting physicians. Alas, all spelling mistakes ever made by our typists and radiologists are also included.
The usefulness of the indexing solution is indicated by the fact that all resident radiologists make use of it despite the nonmandatory character. Figure 4 illustrates the nearly linear increase in users since the initial deployment in May 2004.9Fig 4.The increase in users over time.
Indexing reports added useful metadata to our collection of radiological examinations. Early generations of the RIS did not track the supervisor of a radiologist in training; instead, the supervisor’s name was added manually to the text of a report. Indexing made it possible to retrieve historical supervisor information.
Having an independent report warehouse available that contains all radiological reports avoids vendor lock-in. During PACS implementation we encountered a 6-month delay in the PACS/RIS front-end integration due to a difficult multivendor situation. We successfully avoided this delay by making historical reports available at the radiologists’ work stations through the report warehouse.
Query Analysis
The search engine was consulted 7,071 times by our radiologists mainly to find interesting cases for research or training purposes. Occasionally, a query is performed to find a recent clinical exam in a fast and convenient manner.
Table 1 presents the usage of search engine features. Apparently, 61% of the radiologists are aware of and use at least one of the features despite the lack of formal training; 39% of them have never used a search engine feature. Nine in ten queries consist of keywords only, which means that results are mostly found without having to resolve to more complicated queries. Users tend to start out with a simple query and quickly skim across the clearly presented and highlighted results. Binary operators turned out to be the most popular feature, especially the “and” operator, which is used in 4.6% of the queries to narrow the returned results. Phrase and wildcard searching each are used in 3% of the queries, whereas parentheses account for 0.6%.
Table 1.The Usage of Search Engine FeaturesFeatureQueries (n = 7,071)Users (n = 38)Keywords only6,344 (89.7%)15 (39%)Phrases215 (3.0%)14 (37%)Wildcards208 (2.9%)14 (37%)Binary operator “and”325 (4.6%)15 (39%)Binary operator “not”117 (1.7%)9 (24%)Binary operator “or”83 (1.2%)8 (21%)Parentheses40 (0.6%)6 (16%)
Querying the index is very fast; the median query duration is 31 ms. The most complex query took 2.6 s: results for each keyword need to be binary combined and ranked; a processing delay is noticeable in the case of very frequent keywords or when broad wildcards are used. The users enter fairly specific queries; medially, only 36 results are returned. A statistical analysis of the query duration and the number of results can be found in Table 2. The cumulative frequency distribution of the query duration, presented in Figure 5, can give the reader an idea of the response rate of the search engine. Caching performed by the operating system has a strong influence on the query duration because 36% of the queries yield results within the 13-ms average access time of the disk drive used; Figure 6 visualizes the relevant detail of the frequency distribution.
Table 2.Statistical Analysis of the Query Duration and the Number of Results Query Duration (ms)Number of ResultsMedian3136Maximum2,598962,629Mean1295,878Standard deviation27459,286Skewness4.415Kurtosis24237Fig 5.The cumulative frequency distribution of the query duration (ms).Fig 6.Detail of the frequency distribution of the query duration (ms), showing the influence of caching performed by the operating system.
Open Source
Our indexing solution is a prime example of an open-source success story; extended use of this technology dramatically reduced both implementation time and cost. Designing and building the system took a mere 6 weeks, which was less than the “red tape” time needed to approve a commercial offering. Apart from the implementation man-hours, the total cost of ownership consists of low-budget hardware and support costs. Support calls are rare due to the high-availability design of the system and the excellent stability of the Swish-e search engine. In fact, users almost never call; the interface is very user friendly and similar to well-known interfaces of Internet search engines. Furthermore, passwords cannot be forgotten because of the single-sign-on mechanism. About four times a year, manual intervention is required when the synchronization link with the RIS breaks down; in this case, the administrator is automatically notified by e-mail. There has only been one hardware failure during the system’s lifetime, a broken power supply, but this did not cause any downtime.
All software we developed in the course of this project has been made available10 to the open-source community covered by an unrestricted BSD-style license. The source code included linking and synchronization with a RIS, five import filters, the security layer and audit logging, the complete user interface, and the integration code for both RIS and PACS.
DISCUSSION
Although indexing technology has undoubtedly been used in a clinical setting before, the authors believe they have followed a novel approach in designing and implementing a complete and scalable indexing solution. This high-availability and high-performance system has been running without issues for more than 3 years in a production environment. Furthermore, the independent and flexible design of the platform allows for hosting other applications; several research projects have already found a home under its wings. In this section, Internet indexing technology, scalability, and semantics are briefly discussed.
Internet Indexing Technology
Our decision to make use of Internet indexing technology in a medical context was straightforward because of its ubiquitous presence, the available software components, and the maturity this technology has gained over the years. Three generations of automated Internet indexing systems can be identified as to their methods of compiling their data sets, their search interfaces, and their associated etymological metaphors and mythologies.11 Firstly, the Archie search engine dating back to 1990 allowed for single keyword and regular expression searches of the file lists of anonymous File Transfer Protocol sites. Gopher, a distributed document network protocol preceding the World Wide Web,12 featured a second generation; the Veronica system allowed for Boolean queries of directory information and filenames. Thirdly, search engines that compile searchable databases of information accessible via the World Wide Web are in wide use today, such as the popular Google.13 Indexing technology slowly found its way into computer operating systems like Microsoft Corporation’s Windows Vista (released in 2007) and Apple Corps’ Mac OS X 10.4 (released in 2005).
Two aspects of Internet indexing technology could not be applied to our medical documents because of absent hyperlinking. The first was “spidering”: robots crawling the web to search for new and updated material by investigating hyperlinks. However, this did not pose a problem because the RIS kept track of new and updated reports. Secondly, page-ranking algorithms could not use rules based on the number of referrals by other documents.
A nationwide study conducted by Vorbeck et al14 in Austria, apart from the Alps, a country similar to Belgium, showed that radiologists were already familiar with the Internet in 1999. Today, all radiologists at our institution have a broadband Internet connection at home, including secure access to the electronic medical records and the PACS.
Scalability
Generally, the duration of building an index should be less than the desired timeframe in which new or updated documents show up in the system. Build times can increase beyond this point when document access speeds decrease or when the total size of the document collection grows. To counter this limitation, indexes can be built incrementally; however, care must be taken because some indexers do not recalculate frequencies of very common words for automatic exclusion. Similarly, indexes can be built in parallel; both mechanisms make use of index merging. Judging by our current results and the obsolete hardware used, we estimate that our indexing system scales up by two orders of magnitude, the main bottleneck being the nondistributed index.
To scale a cross-patient search engine, hospital-wide privacy considerations have to be taken into account. The hospital’s policy on patient privacy should be implemented in strict access rules based on document metadata.
Semantics
A disadvantage of keyword-based searching is the simple fact that radiologists or researchers want to find the meaningful concepts behind their keywords. To make word indexing more or less suitable for concept searching, four search engine features are necessary. Firstly, because medical terms mostly consist of more than one word, it has to be possible to search for phrases. Page-ranking algorithms can help by giving a higher score to documents with adjacent keywords, but in practice, the sole use of this technique alone is not specific enough. Secondly, very frequent words with low significance can hamper phrase searching and should not be indexed; when “of” and “the” are excluded, the phrases “MRI of the brain” and “MRI brain” yield the same results. A third helpful feature is Boolean logic to facilitate binary operations like “and,” “or,” and “not” in combination with parentheses. Finally, a word-stemming technique can be used to find all words with a common root. Although we did not implement word stemming because our multilanguage set-up required building a separate stemmed index for each language, the use of wildcards turned out to be an adequate alternative. Word stemming is part of the broader method of fuzzy indexing, which can be used to find words with similar pronunciations.
To find conceptual information, users build their queries in multiple steps; hence, the user interface should allow for fast reviewing of results. Table 3 shows a typical user refining his query. Note that structured reporting, which uses standardized information concepts, does not have the semantic problems of free-text reporting.
Table 3.Refining of a Query by a Typical UserTimeQueryNumber of Results12:00cirrhose52812:02cirrhose ascites18312:02cirrhose ascites MR not angio*100
A lot of research regarding medical semantics has been conducted, ranging from natural language processing techniques15–17 to building unified medical lexica18,19 for multiple languages.20 The authors hope to combine the indexing solution with some of these scientific methods and technology regarding the semantic web21 in the near future.
CONCLUSION
In this digital age, special care should be taken that the ever-growing amount of on-line information does not deteriorate into an inaccessible swamp of mere bits and bytes. On should strive to provide fast, clear, and easy access to information sources and, hence, increase the total value of stored data.
Designing and implementing an indexing solution for a large set of radiological reports and images turned out to be a technically challenging but educational endeavor. For research and training purposes, it certainly is a valuable and convenient addition to our radiology informatics toolbox. The use of open-source technology is highly recommended to reduce both implementation time and cost.

open source
radiology reporting
search engines
radiology information systems (ris)
pacs integration
medical informatics applications
document indexing


Morphological and cytochemical determination of cell death by apoptosis

Several modes of cell death are now recognized, including necrosis, apoptosis, and autophagy. Oftentimes the distinctions between these various modes may not be apparent, although the precise mode may be physiologically important. Accordingly, it is often desirable to be able to classify the mode of cell death. Apoptosis was originally defined by structural alterations in cells observable by transmitted light and electron microscopy. Today, a wide variety of imaging and cytochemical techniques are available for the investigation of apoptosis. This review will highlight many of these methods, and provide a critique on the advantages and disadvantages associated with them for the specific identification of apoptotic cells in culture and tissues.
Introduction
The mode of cell death referred to as apoptosis, and now defined by a variety of techniques, including biochemistry, flow cytometry, and imaging was originally defined based solely upon morphological criteria. In 1972, Kerr et al. (1972) introduced the neologism “apoptosis” to describe a type of cell death which had previously been referred to as “shrinkage necrosis”. Recognizing the importance of its kinetic mechanism in controlled cell deletion, either occurring spontaneously or in response to a stimulus, they offered the term apoptosis (from Greek, meaning “to fall away from”, as in leaves from a tree; thus, the falling away of cells from a tissue). They observed apoptotic cells in a wide variety of tissues, including during development and neoplastic transformation. Although they could detect apoptotic cells in many instances by light microscopy, it was their observations by transmission electron microscopy that established the characteristic ultrastructural features now considered the hallmark of apoptosis. These features include (1) cytoplasmic and nuclear condensation (pyknosis); (2) nuclear fragmentation (karyorrhexis); (3) normal morphological appearance of cytoplasmic organelles; and (4) an intact plasma membrane (Kerr et al. 1972; Wyllie et al. 1980; Galluzzi et al. 2007). Frequently, the pyknotic nucleus assumes the appearance of a half-moon or crescent shape, a feature most indicative of an apoptotic cell (Fig. 1). Following nuclear fragmentation, the cell disaggregates into a number of membrane-bound apoptotic bodies, which are engulfed via phagocytosis by neighboring epithelial cells or macrophages.
Fig. 1Transmission electron microscopic image of an apoptotic cell in a human kidney biopsy. Note the pyknotic, shrunken nucleus and the very condensed cytoplasm
In more recent years, evaluation of cells and tissues for apoptosis has evolved towards staining for light microscopic and flow cytometric analysis. As the biochemical and cell signaling events involved in the apoptotic cascade have been revealed, new tools for the analysis of apoptosis have emerged. Many of these tools are in the form of antibodies raised against proteins specific for the apoptotic pathway, or against neoepitopes on proteins resulting from action of an activated enzyme. While these new probes may specifically target aspects of the apoptotic pathway, they do not address the ultrastructural changes upon which the term apoptosis was originally defined.
This review will focus on some morphological and cytochemical techniques used to demonstrate the presence of apoptotic activity in tissue and cultured cells. For further information on this topic, the reader is referred to the earlier excellent reviews by Willingham (1999), Barrett et al. (2001) and Watanbe et al. (2002). However, before we begin to describe these specific detection methods, we need to examine more closely the different classifications of cell death, and then delve into the mechanisms responsible for initiating the apoptotic cascade.
Classification of cell death
Cell death is now known to be perpetrated through a variety of mechanisms. According to Galluzzi et al. (2007), cell death can be classified into four different types, based upon morphological characteristics: apoptosis (Type 1), autophagy (Type 2), necrosis (oncosis, Type 3), and mitotic catastrophe. The morphological changes accompanying apoptosis have been described in detail above. Whereas apoptosis is manifested by volume reduction of the nucleus and cytoplasm (cell shrinkage), necrosis (the mode of cell death with which apoptosis is most often confused) is evinced by cytoplasmic swelling, rupture of the plasma membrane, swelling of cytoplasmic organelles (particularly mitochondria), and some condensation of nuclear chromatin (Galluzzi et al. 2007). Autophagy is distinguished by the accumulation of cytoplasmic vacuoles and membranes, and mitotic catastrophe by multinucleation. Clearly, cellular morphological characteristics must be taken into consideration when determining the mode of cell death. This is of utmost importance when employing the TUNEL assay, since this technique stains both apoptotic and necrotic cells, which as just described display widely different morphological characteristics. This issue will be addressed in detail below.
Pathways of apoptosis
Cell death through apoptosis is known to occur through two primary pathways, an extrinsic pathway involving death receptors, and an intrinsic pathway via members of the Bcl-2 family (Adams and Cory 1998). The extrinsic pathway uses the classical death receptors such as Fas (CD95/APO1), TNF Receptor1, and TRAIL Receptors. Engagement of these receptors triggers a now well-defined process of recruitment of proteases known as caspases (cysteine-dependent aspartate-cleaving proteases). In the case of Fas, oligomerization by its ligand causes recruitment of the adaptor protein FADD (Fas-associated via death domain) through interactions between the death domains (DD) found in each protein (Krueger et al. 2003). FADD then recruits caspase-8 through their common death effector domains (DED). Homodimerization of caspase-8 results in its activation through sequential cleavage at critical aspartate residues. Fully active caspase-8 then cleaves downstream targets such as the Bcl-2 family member Bid to form truncated Bid or tBid (Li et al. 1998). tBid localizes to the outer mitochondrial membrane where it promotes oligomerization of another Bcl-2 member, Bax, which results in breakdown of the mitochondria and release of cytochrome c and Smac/Diablo (Verhagen et al. 2000). Cytochrome c engages a complex containing Apaf-1 and caspase-9 that in the presence of ATP forms a large molecular weight complex of active caspase-9 known as the apoptosome (Cain et al. 2000). The apoptosome can then activate the effector caspases, such as caspase-3 and caspase-6 that ultimately lead to breakdown of cellular components. Among these is cleavage of an inhibitor of caspase-activated DNAse (ICAD) that allows the DNAse to degrade DNA. This process is fundamental to various assays of DNA breaks that are described in this review.
In parallel with the extrinsic death pathway, the intrinsic pathway ultimately engages the same mitochondria-initiated downstream death pathways. In this case the cascade is initiated by any of a variety of sensors that detect cellular stress. The initiators in this case include a panel of Bcl-2 family members that contain only the BH3 domain (so-called BH3-only proteins) (Strasser 2001). These pro-apoptotic Bcl-2 family members are located throughout the cell somewhat as sentinels that detect cellular perturbations. Upon their activation they induce aggregation and activation of Bax or Bak, which then feeds into the mitochondrial death pathway described above. Activation of the intrinsic death pathway may occur in the form of DNA damage by UV irradiation that activates p53 and then two BH3-only proteins, Puma and Noxa. Glucocorticoids similarly activate Puma and Bim that are critical to induction of cell death by steroids. In fact most chemotherapeutic agents ultimately engage a BH3-only protein in the process of killing cells. As a consequence of these pathways, several death assays measure mitochondrial integrity or membrane potential as described in this review. Additional assays might assess cleavage of caspases or their substrates.
Given the localization of many Bcl-2 family members at the mitochondria, considerable interest is now developing over the concept that these proteins may function as well in some normal metabolic pathways. An early and somewhat overlooked study from the Korsmeyer laboratory showed that increased expression of Bcl-2 in T lymphocytes decreased their production of the growth cytokine IL-2 and resulted in somewhat reduced proliferative capacity following stimulation of the T cell antigen receptor (TCR) (Linette et al. 1996). Bcl-2-deficient T cells manifested the opposite phenotype. More recent studies suggest that this may function via regulation of calcium signaling (White et al. 2005). Other studies suggest that BAD may regulate glucokinase activity in mitochondria and that BAD-deficient β-islet cells are less responsive with insulin secretion to glucose levels (Wikstrom et al. 2007). In a similar manner, caspase-8 has been shown to be critical to growth of various cell types including T lymphocytes (Chun et al. 2002; Salmena et al. 2003). Thus, analysis of activation of “death” molecules will, in the future, need to consider these potential overlapping functions with normal cell metabolism.
Detection of apoptosis by electron microscopy
It was the early electron microscopic investigations of Kerr et al. (1972) and Wyllie et al. (1980) that led to their formulation of the concept of apoptosis as a wide-spread form of cell death in a variety of tissues and physiological situations. Electron microscopy is still considered the “gold standard” for the identification of apoptotic cells. However, likely due to the costs associated with performing electron microscopy, the specialized equipment and technical expertise required, and the limited tissue area represented, this technique is no longer routinely used as an apoptosis assay. In many instances, the degree of apoptosis detected, for example following treatment with a specific reagent, is an experimental endpoint. This can easily be accomplished with light microscopy or flow cytometry-based methodologies, but is exceedingly laborious by transmission electron microscopy. However, in instances where quantitation is unimportant, and the mere presence of specifically identifiable apoptotic morphology is the goal, electron microscopy is indispensable. For instance, Abbate et al. (2007) recently employed this approach as a means to characterize the nature of cell death in the myocardium after an ischemic event. They investigated the ultrastructural appearance of cardiomyocytes by transmission electron microscopy in failing rabbit hearts 16 days following coronary artery ligation. They described cardiomyocytes with severe nuclear derangements, peripheral coarse chromatin condensation, and prevalent mitochondrial and myofibrillar alterations. Although these ultrastructural characteristics are not typical of apoptotic cells, the cells were deemed to be apoptotic based upon an intact cardiomyocyte plasma membrane. These results would suggest that the “typical” ultrastructural features ascribed to apoptotic cells may in some instances be cell-type specific. However, further experimentation will be necessary to confirm this interpretation.
Scanning electron microscopy has also been used to examine apoptotic cells, with their characteristic rounded-up phenotype well represented by this imaging modality (Wyllie et al. 1980; Germain et al. 2007).
Detection of apoptosis by cytochemical optical microscopy
A wide variety of probes are commercially available today for the detection of apoptosis (see Table 1). Many of these are in the form of antibodies raised against proteins specific for the apoptotic pathway, or against neoepitopes on proteins resulting from action of an activated enzyme. These will be discussed later, however, since we will begin with ostensibly the most common cytochemical method for the determination of apoptosis, the TUNEL assay.
Table 1Commercial sources for apoptosis reagentsReagentCommercial sourceTUNEL assay kitMBL International, Woburn MAMillipore Corporation, Billerica, MAInvitrogen Corporation, Carlsbad, CAPromega, Madison, WIRoche Applied Science, Indianapolis, INApo stain (F7-26) (anti-ssDNA)Axxora, LLC, San Diego, CAAnti-cleaved caspase 3 (ASP 175)Cell Signaling Technology, Inc., Danvers, MAAnti-cleaved cytokeratin 18Chemicon, Temecula, CAAnnexin V-FITCBD Biosciences Pharmingen, San Diego, CARoche Applied Science, Indianapolis, INInvitrogen Corporation, Carlsbad, CALectins specific for α-d-mannose and β-d-galactoseLectinotest Laboratory, Lviv, UkraineEY laboratories, San Mateo, CACalbiochem, San Diego, CASigma-Aldrich, St Louis, MOAnti-cytochrome c oxidase subunit 1Invitrogen Corporation, Carlsbad, CABD Biosciences Pharmingen, San Diego, CAAnti-apoptosis-inducing factorSanta Cruz Biotechnology, Santa Cruz, CASyntem, Nimes, FranceThis list is not meant to be exhaustive, but to serve as a starting point to search for reagents. Apologies to those companies not listed
The TUNEL assay
The presence of apoptotic cells has often been inferred from a positive reaction to the TUNEL assay [terminal deoxynucleotidyl transferase (TdT)-mediated dUTP in situ nick end labeling]. However, the lack of specificity of the TUNEL assay for apoptotic cells has been well documented (Sloop et al. 1999; Jung et al. 2000; Walker and Quirke 2001; Watanabe et al. 2002; Groos et al. 2003; Takemura and Fujiwara 2006). TUNEL will detect free 3′-OH terminals of both single- and double-strand DNA breaks, thus potentially labeling both apoptotic and necrotic derived DNA strand breaks. Moreover, the TUNEL assay has also been reported to label mitotic cells (Ito et al. 2006), cells in the process of gene transcription (Kockx et al. 1998), cells undergoing DNA repair (Kanoh et al. 1999), and cells with DNA breaks caused by the microtomy sectioning procedure (Sloop et al. 1999). The effects of tissue fixation and processing may also alter TUNEL reactivity (Negoescu et al. 1996; Labat-Moleur et al. 1998; Tateyama et al. 1998). In a rat model, we found that exposure to asbestos and cigarette smoke resulted in strong TUNEL positivity in bronchiolar epithelial cells from paraffin-embedded sections stained by immunoperoxidase (Jung et al. 2000). However, when bronchioles from the same animals were examined by transmission electron microscopy, only necrotic epithelial cells were observed, indicating that the TUNEL-positive cells detected by light microscopy were likely necrotic, not apoptotic. The TUNEL-positive nuclei did not display the characteristic shrunken appearance expected for an apoptotic cell, again emphasizing the need to corroborate staining results with morphological observations. Similar conclusions were reached by Negoescu et al. (1996) and Labat-Moleur et al. (1998) in their efforts to improve tissue preparation and staining procedures for the TUNEL method.
The TUNEL assay maintains its popularity as a marker of apoptotic cells; it is available in a kit form, is easy to use, and has been available for a number of years, becoming entrenched in the literature. However, given the concerns raised above with its specificity for apoptotic cells, we are of the opinion that other methods should be employed concomitantly with this assay in order to make a determination that the mode of cell death observed is the result of an apoptotic mechanism.
Anti-single-stranded DNA antibody (Apostain)
Frankfurt et al. (1994, 1996) and Frankfurt and Krishan (2001) have pioneered the application of a monoclonal antibody raised against single-stranded DNA (ssDNA) as a more specific cytochemical probe for apoptotic cells than the TUNEL assay. They have shown that the combination of a commercially available ssDNA antibody with formamide-induced denaturation of condensed chromatin, results in specific labeling of apoptotic cells in culture and in formalin-fixed, paraffin embedded tissues (Frankfurt and Krishan 2001). Ito et al. (2006) expanded upon these observations by adapting the formamide DNA denaturation followed by anti-ssDNA antibody staining technique for the electron microscopic level. They show very convincingly by immunoelectron microscopy that the TUNEL assay results in staining of condensed chromatin in mitotic and necrotic cells in addition to apoptotic cells, whereas the formamide-ssDNA technique stained only the condensed chromatin from apoptotic cells.
We have also used this Apostain method for the specific detection of apoptotic pulmonary epithelial (Buder-Hofmann et al. 2001) or rat pleural mesothelial cells (Shukla et al. 2003) after exposure to crocidolite asbestos in culture, and in human lung adenocarcinoma cells (A-549) following treatment with the chemotherapeutic agents paclitaxel and vinorelbine (Jung et al. 2004). Moreover, we recently combined staining with the ssDNA antibody with a second specific marker of apoptosis in a novel dual staining method on tissue sections, as described below. In our hands, in agreement with studies described above, we find the formamide condensed DNA denaturation/ssDNA technique to be a reliable indicator of apoptotic cells in a variety of cell cultures and tissues.
Apoptosis-related neoepitopes
Antibodies raised against caspase-cleaved proteins are now commercially available. The enzymatic cleavage of proteins during the apoptotic cascade results in the formation of “neoepitopes” on these proteins which can be used as antigens for the production of caspase-mediated apoptotic reagents. Although several such antibodies are currently available, those raised against neoepitopes present on cleaved cytokeratin 18 and caspase 3 have been most frequently used for the detection of apoptosis. Willingham and colleagues (Barrett et al. 2001) demonstrated the utility of using an anti-cleaved cytokeratin 18 antibody for the specific detection of apoptotic cells in culture and in archived paraffin embedded tissues. Specifically stained cells showed the shrunken condensed morphology expected of an apoptotic cell. Naturally, this antibody can only be employed for cells and tissues that express cytokeratin 18.
Likewise, Gown and Willingham (2002) used a commercially available antibody raised against cleaved caspase 3 to specifically label apoptotic cells in culture and paraffin embedded tissues. Given the ubiquitous presence of caspase 3 in most cells, and the formation of the neoepitope following activation of the apoptotic cascade, this reagent should prove to be a most useful and specific label for cells undergoing caspase-mediated apoptotic cell death.
Recently, we combined immunostaining for anti-cleaved caspase 3 with the formamide denaturation/ssDNA method for the detection of two specific apoptosis probes in the same tissue section using confocal scanning laser microscopy (Taatjes et al. 2007). In dual stained cells in paraffin sections from infarcted mouse ventricle, the anti-cleaved caspase 3 antibody labeled the cytoplasm, whereas the anti-ssDNA antibody stained the cell nucleus. Cells were observed stained singly with either of the two probes, suggesting that cells in different phases of the apoptotic cascade may be identified by their staining properties with these antibodies. We include sections from the thymus of dexamethasone-treated mice as a positive control (Figs. 2, 3) for the apoptosis assays (Taatjes et al. 2007), since steroid hormone treatment has been shown by transmission electron microscopy to induce thymic apoptosis (Wyllie et al. 1980).
Fig. 2Paraffin section of thymus from mouse treated with dexamethasone and stained with anti-cleaved caspase 3 antibody (green), and anti-ssDNA antibody (red). Confocal microscopy demonstrates doubly stained thymic cells (arrowheads). Note that the cleaved caspase 3 is localized to the cytoplasm, whereas the ssDNA is localized to the nucleus. Some cells only stained by cleaved caspase 3 antibody are indicated by solid arrows. The nucleus is labeled with DAPI (blue)Fig. 3Paraffin section of thymus from mouse treated with dexamethasone and stained with anti-cleaved caspase 3 antibody (red). Confocal microscopy image shows intensely stained cells in the cortex, with neighboring unstained cells. Nuclei have been stained with DAPI (blue). This represents a good positive control for the anti-cleaved caspase 3 antibody technique
Annexin V
Some markers for apoptotic cells require the use of live cells. For instance, early in the apoptotic cascade of events, phosphatidyl serine residues flip from the internal to external side of the cells’ plasma membrane. Accordingly, annexin V, which binds specifically to phosphatidyl serine residues has been used in this manner as a specific probe for apoptosis (Koopman et al. 1994). However, as pointed out by Willingham (1999), this assay does not seem to label all apoptotic cells, and moreover will stain necrotic cells once the plasma membrane is compromised.
Other probes for apoptosis
Bilyy et al. (2004, 2005) and Bilyy and Stoika (2003) have championed the examination of plasma membrane glycoconjugates as indicators of apoptosis. They have detected an augmentation in the expression of α-d-mannose- and β-d-galactose-containing glycoproteins in the plasma membrane in response to apoptotic signals. These glycoconjugates can be visualized using a battery of lectins (carbohydrate-binding proteins of nonimmune origin), in conjunction with fluorescence- or peroxidase-conjugated probes for light microscopy. Unlike most of the markers for apoptosis which bind to cytoplasmic or nuclear constituents, these plasma membrane-specific probes offer the means to assess activity occurring at the cell surface during the apoptotic cascade (similar to annexin V binding to phosphatidylserine). Given the enormous diversity in carbohydrate structure, and the role played by carbohydrates in a variety of cell–cell interactions, it is feasible that they may offer more insight into cellular mechanisms of apoptosis in the future.
An early event in the apoptotic cascade is the release of cytochrome c from the outer mitochondrial membrane. Accordingly, the intracellular localization of cytochrome c could serve as an indicator for apoptosis. Jones and colleagues (Jiang et al. 1999) used this approach to show that in a control (untreated) osteosarcoma cell line, immunostaining for cytochrome c (as assessed by confocal microscopy) was observed in a punctuate cytoplasmic pattern, coincident with mitochondrial staining by MitoTracker Red. When cells were treated with staurosporine to induce apoptosis, the immunolabel for cytochrome c was found to be diffusely present throughout the cytoplasm, consistent with its release from the mitochondria. We have also used immunolabeling for cytochrome c as a marker for mitochondria in intact cells (Fung et al. 1998), and its further application as a potential marker for apoptotic cells dependent upon its intracellular staining pattern should be further explored.
Apoptosis-inducing factor (AIF) is a mitochondrial membrane protein involved in caspase-independent cell death (Daugas et al. 2000). Immunostaining for AIF revealing a translocation from the mitochondrion to the nucleus has been used as a marker for caspase-independent apoptosis (Daugas et al. 2000; Susin et al. 2000). However, Daugas et al. (2000) have shown that the nuclear translocation of AIF also occurs during a necrotic response, so other methods should be employed in conjunction with AIF immunostaining to differentiate between these two modes of cell death. Indeed, we have used immunostaining of AIF translocation from the cytoplasm (mitochondrion) to the nucleus, together with cell morphological assessment by phase contrast microscopy, as an indicator of apoptosis in lung C10 cells in response to asbestos exposure (Yuan et al. 2004).
Detection of apoptosis by video optical microscopy
Morphological assessment of cultured cells by video microscopy has been successfully used to follow the cellular changes induced by apoptosis. Cells treated with apoptosis-inducing agents are imaged with a light microscope, typically operating in phase contrast or differential interference contrast mode, and time-lapse recording is made with a ccd camera connected to a video recorder (Collins et al. 1997; Willingham 1999). A major advantage of this type of temporal microscopy is the ability to document cellular morphological changes associated with apoptosis with the time course of the event. Differences in cellular changes observed in necrosis versus apoptosis can be well documented, and the slow course of apoptosis is well suited to time-lapse recording (Collins et al. 1997). Using phase contrast microscopy in conjunction with video recording has produced dramatic footage of cells exposed to apoptotic agents rounding up and displaying intense surface blebbing (Collins et al. 1997). Of course, the full apoptotic cascade may require up to 24 h in cultured cells, so movies can also be made by merging files from single ccd camera images and exporting them in a movie-type format. Thorburn and colleagues (Morgan et al. 2002) used a combination of fluorescence and phase contrast microscopy to image apoptosis in live HeLa cells. A Hamamatsu digital camera was used to acquire combination images every 30 min over a 24 h period. The merged image series were exported in Quicktime format, and a representative movie sequence can be seen in the online supplemental material referenced in the manuscript (Morgan et al. 2002). This type of analysis is particularly useful for determining the temporal sequence of morphological events associated with the introduction of an apoptosis-inducing agent to cell culture media.
Detection of apoptosis by laser scanning cytometry
Laser scanning cytometry (LSC) is a method which combines the efficiency and precision of flow cytometry with the morphological characterization ability of a microscope-based imaging system (Kamentsky and Kamentsky 1991; Kamentsky et al. 1997; Taatjes et al. 2001; Pozarowski et al. 2006). The ability to examine the morphological appearance of cells following flow cytometry-like analysis overcomes the limitations posed by flow cytometry itself in the analysis of apoptosis; that is, apoptotic cells appear in the subdiploid peak of a flow cytometric histogram, together with apoptotic bodies and fluorescence debris, and cannot be directly visualized. Darzynkiewicz and colleagues have pioneered the use of LSC for analysis of apoptosis, and the reader is referred to their excellent review for further details (Bedner et al. 1999). In particular, they have used LSC in combination with a variety of apoptosis markers, including annexin-V, markers of mitochondrial transmembrane potential (rhodamine 123 and DiOC6), nuclear markers (DAPI, 7-ADD, PI), and markers of DNA strand breaks. Importantly, in all of these investigations, they emphasized the combination of fluorescence staining and morphological determination with LSC as being of utmost importance for the identification of apoptotic cells.
We have also used LSC for the cytometric and morphological characterization of apoptotic cultured mouse lung epithelial cells (C10 cell line) (Taatjes et al. 2001). Cells were exposed to hydrogen peroxide for 24 h to induce apoptosis, followed by staining with the anti-ssDNA antibody and an Oregon green-conjugated secondary antibody to stain apoptotic cells. All cells were then stained with propidium iodide and analyzed by LSC. Scattergrams from the analysis (Fig. 4) showed that hydrogen peroxide induced an increase in apoptotic cells, evinced by an increase in the population of cells depicted in the subdiploid area. Cells located in this subdiploid portion were then visually confirmed through relocation using the automated stage and microscope of the LSC. These cells were found to be labeled with the anti-ssDNA antibody, and showed the condensed morphological characteristics of apoptotic cells (Fig. 4).
Fig. 4Laser scanning cytometry determination of apoptosis in lung epithelial cells using anti-ssDNA antibody. Upper panel scattergrams showing percentage of ssDNA-positive cells (green integral vs. orange/red integral), the apoptotic fraction (orange/red integral vs. orange/red maximum pixel), and a DNA histogram for untreated C10 cells. Middle panel scattergrams showing percentage of ssDNA-positive cells (green integral vs. orange/red integral), the apoptotic fraction (orange/red integral vs. orange/red maximum pixel), and a DNA histogram for C10 cells treated with 200 μM H2O2 for 24 h. Note the increased number of cells ssDNA-positive, and the enhanced subdiploid DNA fraction in cells treated with H2O2 as compared with sham controls. Bottom panel cell relocation feature of LSC demonstrated for eight cells. Cells within the subdiploid fraction (elliptical region highlighted on scattergram in middle panel) were relocated and visually confirmed as apoptotic by morphological appearance and positive staining with the ssDNA antibody. Reprinted with permission from BioTechniques (Taatjes et al. 2001)
Detection of apoptosis by atomic force microscopy (AFM)
The atomic force microscope is a member of the scanning probe microscope family (Binnig et al. 1986). This instrument utilizes a tactile sensing system, in which a probe (typically composed of silicon nitride) is attached to the end of a cantilever and raster scanned in x- and y-directions across a specimen through the activity of a piezoelectric tube to which the cantilever assembly is mounted (Montigny et al. 2006). A laser beam is directed onto the surface of the cantilever, and the reflected beam is directed up to a four quadrant photodiode, which converts laser movement on the surface into a measurable electrical voltage change. This voltage change is then translated via computer interface into height and amplitude information about the specimen. The complexity of the system belies the simplicity of specimen preparation; fixed or unfixed cells grown on glass coverslips can be readily imaged by AFM without further processing, or importantly introduction of artifacts. Although AFM initially found its greatest utility in the imaging of isolated molecules and their dynamic interactions (Hansma and Ho 1994; Taatjes et al. 1997; Rand et al. 2003), more recently whole cell studies have become more prevalent (Madl et al. 2006). With this development, and since AFM typically images surface phenomena, it was a natural extension to use this technology to image cells undergoing apoptosis. Hessler et al. (2005) used AFM to image and measure the cellular volume decrease occurring during apoptosis. They treated the human epidermoid carcinoma cell line KB with staurosporine to induce apoptosis, and determined that after a 3-h exposure, the total cell height decreased 32%, the total cell volume was decreased by 50%, and the cell diameter changed by 15%. Given the temporal experimental capabilities of AFM, it seems as if this would be an ideal modality, in combination with other imaging methods, to investigate the cellular dynamics accompanying the apoptotic cascade, as elegantly shown by Hessler et al.
An example of the importance of delineating apoptosis: cell death in the heart
In this brief review, we have attempted to highlight some of the morphological and cytochemical techniques currently in use to classify cell death as an apoptotic response. There are a wide ranging variety of methods to choose from, rendering the choice a matter of the experimental system employed and personal preference. However, the determination of cell death as apoptotic, or through another mode should not be viewed simply as a matter of semantics. For instance, an example from clinical cardiovascular disease will illustrate the importance of the precise determination of the mode of cell death. For several decades the conceptual basis underlying treatment of acute coronary syndromes has been that limitation of myocardial oxygen requirements by diminution of afterload and heart rate coupled with augmentation of myocardial oxygen supply through restoration of myocardial perfusion are the pivotal components (Maroko et al. 1972; Shell and Sobel 1974; Sobel and Braunwald 1980). The potential value of “tissue protective” interventions including favorable modification of myocardial metabolism has long been recognized as well (Maroko et al. 1972).
Several different types of cell death have been characterized in the heart including coagulation necrosis, the typical manifestation of ischemia sufficient to render myocardium nonviable, contraction band necrosis typical of ischemia followed by reperfusion and presumably mediated in part by calcium overload, autophagy typical of cells insulted by deprivation of nutrients that consequently consume their own components, and apoptosis. The extent to which this final modality, programmed cell death or apoptosis, contributes to overall cardiomyocyte cell death is far from a trivial matter (Sobel et al. 2007). If apoptosis plays a substantial role as a contributor to total cardiomyocyte cell death and if it can be modified or prevented despite the presence of ischemia, the potential, favorable impact of tissue protective interventions is likely to be considerable. Conversely, if apoptosis plays a small role in overall cardiomyocyte cell death consequent to ischemia, an anti-apoptotic strategy would likely be ineffective. Unfortunately, the implication of apoptosis in cardiomyocyte cell death in studies in laboratory animals has been based virtually exclusively on results with the TUNEL assay (see above for issues of nonspecificity of this assay). As noted by Takemura and Fujiwara (2006), there is no unequivocal evidence that apoptosis occurs in zones of infarction.
As described above, we have recently sought to quantify apoptosis in myocardium and have developed a novel dual staining method for confocal microscopic identification of apoptotic cells (Taatjes et al. 2007). Our initial findings indicate that despite massive necrosis induced by ischemia in a transgenic strain of mice deficient in plasminogen activator inhibitor type-1 (PAI-1) hardly any cells were apoptotic as judged from positive reactions to antibodies against both cleaved caspase 3 and ssDNA. Others have observed an absence of apoptosis as judged from electron microscopic criteria (Ohno et al. 1998) despite TUNEL positivity in hearts of animals subject to transitory ischemia followed by reperfusion for several hours.
In view of these considerations, it appears important to determine definitively whether the presence of ssDNA coupled with the presence of caspace 3 cleavage product does, indeed, map isomorphically with the cells that exhibit electron microscopic criteria of apoptosis. Furthermore, it will be important to determine the extent to which the positivity of each of these criteria of apoptosis occurs in hearts of experimental animals subjected to ischemia for selected intervals followed by reperfusion for selected intervals as well as in those from animals subjected to myocardial ischemia that is persistent and to define the time course of the evolution of apoptosis, if present, under both conditions. It is, of course, possible that apparently scanty apoptosis at any given time could be of considerable biological importance since apoptotic cells are rapidly destroyed or removed by macrophages. Thus, if a given percentage of cells, though a small one, were exhibiting apoptosis repetitively, the cumulative amount could be substantial. Accordingly, both the temporal and the spatial evolutions of apoptosis associated with myocardial ischemia and infarction require elucidation. This is particularly important because apoptosis in the heart may be secondary to necrosis rather than a primary mode of cell death, a distinction that needs to be clarified because of the disparate therapeutic implications of the two possibilities. Only when armed with information needed to characterize apoptosis in the heart more definitively will it be possible to delineate the potentially beneficial effects of tissue protective interventions designed to retard or prevent apoptotic cell death in cardiomyocytes subjected to ischemia.
Concluding remarks
Determining the mechanisms of cell death is an area of intense research interest in a wide range of fields including cancer biology, pathology, and toxicology. Certainly, the role of apoptosis as a mode of “programmed cell death” during embryonic development has been unequivocally established (Penaloza et al. 2006). We hope that this brief review has highlighted a variety of imaging modalities suitable for differentiating amongst the different modes of cell death. What should be apparent is that rarely will one method be sufficient for such a determination. Each imaging method has its own strengths with respect to the morphological information it can provide. Such morphological information, combined with biochemical or immunohistochemical assessment can provide a strong indication of the specific mode of cell death observed. The key, in our estimation, is a multiparameter observational approach.
Accordingly, the take-home message for any morphological or cytochemical investigation of apoptosis is to employ more than one detection method and run multiple controls.

morphology
cell death
apoptosis
microscopy
immunostaining
cytochemistry