There is interest to expand the reach of literature mining to include the analysis of biomedical images, which often contain a paper's key findings. Examples include recent studies that use Optical Character Recognition (OCR) to extract image text, which is used to boost biomedical image retrieval and classification. Such studies rely on the robust identification of text elements in biomedical images, which is a non-trivial task. In this work, we introduce a new text detection algorithm for biomedical images based on iterative projection histograms. We study the effectiveness of our algorithm by evaluating the performance on a set of manually labeled random biomedical images, and compare the performance against other state-of-the-art text detection algorithms. We demonstrate that our projection histogram-based text detection approach is well suited for text detection in biomedical images, and that the iterative application of the algorithm boosts performance to an F score of .60. We provide a C++ implementation of our algorithm freely available for academic use. Ó 2010 Elsevier Inc. All rights reserved. 1. Background Biomedical literature mining is concerned with transforming free text into a structured, machine-readable format, to improve tasks such as information retrieval and extraction. Recent work indicates that there is much interest to also consider image information when mining research articles, as images often depict the results of experiments, and sum up a paper's key findings. There are several obstacles when mining image information. First, there are many different types of images, such as graphs, gel electrophoresis and microscopy images, diagrams or heat maps. There exists no image publication standard, neither with regard to image resolution, or image file format (images are stored at different resolutions, and in a variety of file formats, such as jpeg, tiff etc.). Also, there are no explicit image design guidelines, even though authors seem to follow some universally accepted norms when creating figures such as box plots, heatmaps or gel electrophoresis images. A unifying element across all biomedical images is image text, i.e. text characters that are embedded in images. Text in images serves several purposes, such as labeling a graph, representing genes in a heat map images, or proteins in a pathway diagram. We have previously shown that extracting image text, and making it available to image search, improves biomedical image retrieval [1]. In this work, we are concerned with optimizing the performance of a critical step in image text extraction-locating text regions in images, which is known as text detection in studies on image processing and Optical Character Recognition (OCR). Generally speaking, text detection is a crucial step in processing textual information in biomedical images. For example, properly finding the text regions is the first stage of a standard OCR pipeline for extracting image text. Determining the location of text is also important for high-level image content understanding, as it is the text location that indicates the meaning of certain image text element, such as the label of the x-versus y-axis in a graph. Practical applications aside, in this paper, we are exclusively concerned with optimizing the performance of text detection, which is a fundamental research problem in image text processing. In this work, we introduce a new text detection algorithm suited for biomedical images. We also discuss the methodological details in creating a gold standard biomedical image text detection corpus, and the use of the corpus for evaluating the performance of our algorithm. During the development of the corpus, we laid down clear guidelines on what exactly constitutes an image text region (or element) and how to manually mark the image region linked to the string. We then compared our algorithm against three existing state-of-the-art text detection methods. Even though our algorithm can in principle be applied for processing all images types, it is especially beneficial for images embedded in biomedical publications. Compared to other disciplines, biomedical authors tend to use distributed and nested text in their images in order to annotate experiment settings, conditions and results. There is a body of work using advanced texture and graph segmentation methods to detect text in images. For example, Jain and Karu [7] introduce a method for learning texture discrimination masks for image text detection. Jain and Zhong [8] used a learning based approach to detect image text through image texture analysis. Wu et al. [9] introduced a system for image text detection and recognition, which adopts a multi-scale texture segmentation scheme. In their method, a collection of second-order Gaussian derivatives are used to detect candidate text regions, followed by a K-means clustering process and a multi-resolutional stroke generation, filtering and aggregation process to further refine the detected text region. Felzenszwalb and Huttenlocher [10] proposed a graph-based image segmentation algorithm for efficiently separating textual elements from graphical elements in an image. Their algorithm can automatically adapt itself to the image structure variation. Liu et al. [11] proposed a novel method for text detection and segmentation through using stroke filters for text polarity assessment in analyzing features in local image regions. There also exists a growing collection of work on text detection from videos or motion images, which are closely related to the image text detection problem studied in this paper. In our own prior work [1], we discussed a novel approach for biomedical image search based on OCR. We have shown that the approach offers additional advantages compared to searching over image captions alone, notably the retrieval of additional and relevant images. The current study is closely linked to that project, discussing the algorithmic details for detecting image text regions. Approach An overview of our method is shown in Fig. 1. An input image (i.e. an image from a biomedical publication) undergoes detection of layout lines and panel boundaries, which are excluded from the image to increase text detection robustness. We implement the algorithm proposed by Busch et al. [20] for detecting these layout elements. The image is then converted to black and white, and subjected to an edge detection algorithm. The resulting edge image is then subjected to a pivoting text region detection (PTD) algorithm for extraction of text regions. PTD is repeated several times, in order to divide detected text regions into text sub-regions. If no more text regions are detected, the algorithm exits. Our algorithm is based on traditional histogram analysis-based text region detection, which takes edge images as input. We extend the traditional approach as follows: We perform a pivoting procedure while applying the histogram analysis, and repeat the procedure until no more text (sub)regions are detected. One of the most popular and well known text region detection methods is through analyzing the vertical and horizontal projection histograms of an image. More concretely, given an input image, we first detect the edge pixels in the image. Then a vertical and a horizontal projection histogram are derived. It is assumed that text regions generally exhibit higher density of edge pixels than non-text regions. The vertical and horizontal histograms will thus show the highest density of edge pixels in text areas. A density threshold defines the exact dimensions of the text area along the vertical and horizontal histogram. The elements of this basic procedure are discussed in more detail in the next section. One distinct feature of many biomedical images is that they often employ a distributed and nested text layout. Figs. 3a and 4a show two typical examples, where text is distributed across many different image regions. Also, text regions often display some degree off nestedness. For example, the numbers along the x axis in Fig. 4b can be grouped in one large text area, or-more correctlyinto separate (inner) text areas surrounding each individual number (Fig. 4d) . The traditional histogram-based analysis technique does not cope well with distributed and nested text layout. To address this problem, we introduce a new iterative pivoting histogram analysis procedure for text region detection. We introduce a pivoting step into the classical histogram-based text detection algorithm in order to account for the distributed nature of biomedical image text. The pivoting procedures subdivides image regions into its text subcomponents, instead of identifying large text blocks. Our procedure is realized through analyzing the histograms of the input image region following the vertical and shows the working of the algorithm on a sample image. An input image is converted into black and white and subjected to edges detection (Fig. 1d , Appendix B). For a specified region R (the whole image in the first iteration of the procedure), to detect text areas in R, we first vertically project all the edge pixels to derive the image region's horizontal histogram H h (Fig. 1e , . . . using a threshold on the densities (Fig. 1g1-3, Appendix B) . Each such segment corresponds to a vertical region in the input image. Through pairing of a vertical segment Seg i j with its corresponding horizontal segment Seg i , we are able to specify a rectangular region (bounding box) R i j in R (Fig. 1h1-3 , Appendix B), corresponding to text regions. In Appendix A (Supplementary Files), we formally describe this procedure mathematically. Our algorithm iteratively constructs vertical and horizontal histograms to find nested text regions. As can be seen in Fig. 1h2 , Appendix B, the first round of the PTD algorithm could not resolve the true text areas of the image region. In the image, region 1 groups distinct image text elements, and we propose to repeat the PTD step for separating these elements. More concretely, our algorithm maintains an active local image region collection (ALIRC) during its running time (Fig. 1 , main paper). Initially, the collection contains a single image region, which is the full image area of the input image. The algorithm then constructs pivoting vertical and horizontal histograms (see previous section) and detects text regions. Each detected text region is regarded as a new target region and added into ALIRC. The input image region is removed from ALIRC, with one exception: if, after subtracting the text regions from the input image, the input image is non-empty we populate ALIRC with an updated version of the input image, with the text areas subtracted. We iteratively apply our histogram-based text region segmentation procedure on all the image regions in the ALIRC until finer separation between text and non-text regions can be achieved. We will then output all the image regions maintained in the ALIRC. A final heuristic removes regions that are maintained in ALIRC but do not correspond to text regions. The heuristic evaluates the overall edge density, removing regions that exhibit a density that is too low or too high. 1. Assuming the height and width of the input image is h and w, respectively, we apply our pivoting text detection algorithm introduced in Section 2.3 to detect all the text regions in the full area of the input image. That is, we apply the PTD procedure onto I with the text detection scope being R ¼ ð0; w; 0; hÞ. We further assume the set of regions segmented from the input image are / ¼ fR 1 ; R 2 ; . . . ; R n g where each R i specifies the scope of a rectangular image region resulting from the PTD process. We call / the current active local image region collection. 2. For each image region R i in / ði ¼ 1; . . . ; nÞ, we apply the PTD procedure onto R i to further separate the text and non-text areas inside the region on a finer granularity. Assume this new round of text region detection produces k sub image regions, which are denoted as R 3. We repeat the above process to recursively refine every image region maintained in the current active local image region collection / until / can be no longer changed through additional calls of our PTD procedure. We then output all the image regions in the final stage of the image region collection / which are determined as text regions by our PTD process. These image regions constitute our final text region detection result for the input image I. In Appendix B (Supplementary Files), we show a step-by-step example of text region detection using our iterative and pivoting text detection algorithm for a biomedical image. Appendices C and D (Supplementary Files) contain further examples of text detection results after applying our iterative PTD algorithm on biomedical images. In this section, we will first discuss the creation of a gold standard biomedical image text detection corpus. We will then discuss our evaluation strategy to measure the performance of our iterative PTD algorithm for detecting text regions in biomedical images. To objectively evaluate the performance of our algorithm, and to quantitatively compare the performance of our method to other peer methods, we created a gold standard corpus of biomedical images with manual markup of text regions. In order to create this corpus, we selected a two step approach. The first step dealt with the identification of the text regions in the image. We set up guidelines for manual identification of text regions (image text) in biomedical images, which are listed in Table 1 . The guidelines define the nature of an image text region in a biomedical image, what to do about Greek letters and other special characters, and strings in super or subscript. After selecting 161 random images from biomedical articles indexed in PubMed Central, we used the guidelines to identify the image text regions. In the second step, we identified a minimum rectangular region (bounding box) for each detected text region. Such a bounding box is defined as the smallest rectangular region covering all character pixels of the text region. These image bounding boxes represent the gold standard image text regions. To evaluate the performance of our PTD text detection algorithm, we can proceed as follows: We compare the predicted text region bounding boxes with the bounding boxes of the gold standard corpus. In our study, we employ two approaches for measuring the degree of overlap between the predicted and gold standard text regions, looking at both the pixel overlap and the percentage of shared region. One approach for measuring the overlap of two text detection results is to measure the recall, precision and F-rate as determined by shared pixels. More concretely, recall is defined as the fraction of pixels in the gold standard text area that are contained in the (algorithmically) detected text region. Precision is defined as the fraction  Results We start with a qualitative assessment on the performance of our text detection algorithm. To this end, we provide sample images along with automatically detected text regions (Figs. 3  and 4) . The blue boxes outline the detected text regions, while the purple lines and areas indicate non-textual elements. A qualitative assessment of our approach is helpful for identifying the strength and weaknesses of our algorithm. For example, we see satisfactory text detection performance in Fig. 3b . However, two strings ''the'' and ''number'' in the bottom horizontal label of the image are mistakenly detected as one single text region ''the number''. In Fig. 4 , we show the intermediate text detection results of two rounds of the PTD algorithm, from which we can see that our algorithm progressively refines its text detection results. To explore the effectiveness and advantages of our approach, we also compare the performance of our algorithm with a few state-of-the-art text detection algorithms. To this end, we identified four recently published algorithms for text detection, including the DCT feature based text detection method proposed by Goto [21], the text particle based multi-band fusion method for text detection as proposed by Xu et al. [22], the visual saliency based and biologically inspired text detection method proposed by Fatma [23], and the fast text detection method proposed by Li et al. [24]. When conducting our experiments to quantitatively compare the performance of our algorithm with that of the four peer methods, we obtained either the source code or the executable program from the corresponding author(s). After that, we worked directly with authors of these papers for running their algorithms, ensuring that each algorithm is correctly configured and properly executed during the evaluation experiment. Upon reaching the final comparison results, we shared our intermediate and final evaluation results with the authors. Goto's method [21] applies Fisher's discriminant analysis to explore the frequency dependency of DCT-based features for identifying the optimal frequency band for text detection applications. Through their experiments, they empirically found out that a mixture of high frequency and lower-middle frequency bands is most effective for text detection from scene pictures. Based on this finding, they redefined a DCT-based feature for more accurately detecting text regions from natural scene images. Using their newly defined DCT-based text detection feature, they eventually employ an unsupervised thresholding based mechanism to differentiate text regions from background image regions. Since their method is targeted at detecting text from natural scene images, their algorithm does not pay special attention to the affect of having sophisticated text layout in an image. In fact, when text is arranged following a distributed and nested structure, as is often true in biomedical images, the overall spatial frequency features exhibited in the image region will be much affected. Under such circumstances, determining text regions according to the overall spatial frequency of a region tends not to be reliable. This fact limits the algorithm performance on biomedical images, as demonstrated in our evaluation results, shown below. In contrast to the cited DCT-based feature approach, our algorithm introduces a pivoting and iterative procedure, which progressively divides an image region into subregions. For each resultant image sub-region, we recompute the local image features in the form of vertical and horizontal histograms to adaptively analyze whether an image area contains text. Through this active sub-division based search process, our method can more thoroughly break down and capture the local image features to recover scattered and nested image text, as seen in images in biomedical publications. Xu et al. 's method [22] aims at detecting text from images with weak contrast and low text-background variance. To attain their goal, they first analyze an input image using the YUV color space via the Haar wavelet transformation. Given the derived wavelet coefficients, they introduce text particles, according to the local binary patterns demonstrated by the wavelet coefficients, which are known as Local Haar Binary Patterns. Once the concept of text particles is defined, their method further applies a density-based multi-band fusion procedure to generate the final text detection result. The main benefit of their approach for text detection is that their method can robustly detect text regions regardless of the image size, color, rotations, illuminations, and text-background contrasts. However, their method assumes that an input image does not exhibit a sophisticated text layout and uses a single pass analysis. Their assumption generally holds for street side images of signs and signages. However, for images in biomedical publications, which display distributed and nested text, the same assumption does not hold. While our histogram-based analysis' text discrimination capability is below the method of [22], our overall text detection capability is not much affected. This observation can be intuitively understood in that images carried in biomedical publications are Table 1 Guidelines for manual identification of image text regions. 1. Image texts that form a coherent entity such as ''bcl-xl (+)'', ''p < 0:01'', and ''S.E. RECOVERSOLUTION (i,j,k)'' are considered individual text regions 2. Symbols that are attached to a word, such as brackets, forward slashes, and dashes, e.g. ''MMC-transgenic '' and ''vif(+) usually carefully prepared so that they exhibit a good contrast and a sharp text background. Therefore, the strength of their more powerful text particle based multi-band fusion approach may not be advantageous in the biomedical domain. Fatma's algorithm [23] treats the text detection problem as a texture classification task, where the task of locating text regions is formulated as finding regions whose texture is classified as text texture, rather than background image texture. Fatma chose the Support Vector Classification method to implement her texture classifier. The main novelty of her work is the introduction of a computational model of human visual attention for text detection, which examines image features at various spatial scales and modalities in a bottom-up manner. Like Xu et al.'s method [22], Fatma's algorithm also assumes that there is no nested text placement. When text placement is distributed and nested, image subregions are more likely to be misrecognized as carrying image texture, rather than text texture. More fundamentally, since Fatma's algorithm works with texture features of an image region, her algorithm is not directly concerned with how text strings are positioned inside a region; what matters to her algorithm is the overall texture exhibited by a region, regardless of whether it is composed by several text strings, by background image elements, or by a mixture of both. In fact, if some distributed and nested text happens to exhibit a texture that looks like a plausible image pattern, her algorithm would readily recognize the region as a background image region. Li et al. introduced a stroke filter based approach for text detection [24]. Their algorithm applies stroke filters to examine text presence along the horizontal, vertical, left-diagonal, and rightdiagonal directions. Their work focuses on detecting text from TV shots or video keyframes where almost all the text is placed following the four principal directions (vertical, horizontal, left-diagonal, and right-diagonal directions). This is probably because TV or video frames are designed to be rapidly parsed by the audience. Consequently, these frames, which typically carry only a very limited amount of text, are organized using quite simple and intuitive text layouts to facilitate viewer's rapid reading or scanning. In comparison, images in biomedical publications are meant to be read much more carefully, with image text being often distributed and nested. As a result, Li et al.'s four directional stroke filter based approach seems not sufficient to handle all the subtleties of extracting text from biomedical images. We also implemented two simplified version of our algorithm to study the different components of our procedure. To distinguish between these different versions of our algorithm, we call the iterative text detection method introduced in Section 2.4 the multistep method, which is denoted as ''multiple steps''. We also study the performance of our method when the number of iterations is limited to one round. We call this modification of our algorithm the one step iteration version, denoted as ''one step''. Finally, we also implemented the classical histogram-based analysis without pivoting where the vertical histogram is derived for the full image rather than for the segments from the horizontal histogram (see Section 2.2). We refer to this naive version as ''naive''. The results of these evaluations are shown in Table 2 . We observe the following: The naive method outperforms the other peer methods in terms of F-rate and MAO. The pivoting procedure improves upon the naive version, with a performance increase of 0.045 F-rate and .051 MAO. The iterative procedure further improves upon the pivoting result, both in terms of F-rate and MAO. There is no performance increase when conducting more than 2 iterations of our algorithm. Discussion Our evaluation showed that the iterative PTD algorithm performs well on the gold standard text detection corpus ( Table 2 ). The naive (classical) version is outperformed by the pivoting algorithm, which performs the vertical histogram on each image text segment as determined by the horizontal histogram (Section 2.3 and Fig. 2) . The pivoting algorithm subdivides image text regions into subcomponents, instead of identifying large text blocks as in the naive or classical approach. This sub-division into smaller units seems to cope better with the distributed nature of the biomedical image text. The iterative application of our algorithm results in further performance gains. As discussed, iteration ensures the detection of nested image regions. As can bee seen in Table 2 , performance seems to stabilize after one iteration. This can be understood as follows: Biomedical images seem to contain (on average) one level of text nesting, which can be recovered by one iteration of our PTD algorithm. We conducted an extensive comparison with existing text detection algorithms. None of the tested algorithms were able to outperform the histogram-based text detection approach. It should be noted that these algorithms are optimized for a particular text detection task, which might be different from the one encountered in biomedical images. Consequently, the performance of these algorithms as presented in the literature is higher than the numbers presented in Table 2 . Our results indicate that it is difficult to use these algorithms on biomedical images without modifications. For comparison, we quickly review the performance of the tested algorithms on other image sets. In [21], the author reports algorithm performance for two typical settings of his algorithm-a low frequency mode and a high frequency mode. The evaluation is performed on the ICDAR-set, which is from the TrialTrain data used in the ICDAR 2003 Robust Reading Competition, see [25]. For the low frequency mode, the average precision, recall and F-rate of his algorithm is 32.6%, 91.9%, and 43.4%, respectively. For the high frequency mode, the average precision, recall and F-rate is 35.6%, 88.6%, and 45.1%, respectively. It should be noted that the [21] algorithm performs well on our gold standard corpus in terms of recall. Precision is low, though, indicating many falls positive calls. [22] evaluated their method on the Location Detection Database of ICDAR 2003 Robust Reading Competition Dataset, see [25]. The precision, recall and F-rate on the dataset is 60%, 81%, 69%, respectively. Finally, [24] reported the performance of their algorithm on an image set consisting of 308 images from the Web, recorded  The general category of target images suitable for our algorithm's processing includes images with distributed and nested text use, as typically seen in biomedical images. Our biomedical image text detection algorithm can benefit a number of applications in the field of biomedical information processing and management. Below we list some immediate applications: A first benefit is the improvement of biomedical OCR performance. With accurate text region detection, we can highlight the locations of image text when retrieving biomedical images. With this functionality, users can easily see the positions of their querying text. This application is already available in EverNote (www.evernote.com). Once text regions in biomedical images are detected, we can more effectively compress these image regions by first recognizing the text contents, and then representing these text areas through vector graphics. Given the text region detection result, we can build some application where a user hovers his or her mouse over a text term in an image, to reveal more details related to the text term. Reliably detecting image text regions would allow accurate analysis of document layout. This will enable automatic retrieval of images or documents according to their layout. A reliable image/document layout analysis result provides many valuable features for image/document clustering and categorization. Given the text region detection result, we can design high-level image content understanding algorithms. For example, we may analyze the spatial relationships between text terms occurring in an image. This would provide clues to parse the graphical messages embedded in the images, leading to more advanced biomedical image retrieval and recommendation results. Biomedical image search and mining is becoming an increasingly important topic in biomedical informatics. Accessing the biomedical literature via image content is complementary to textbased search and retrieval. A key element in unlocking biomedical image content is to detect and extract (via OCR) text from biomedical images, and making the text available for image search. In this paper, we are concerned with text detection, i.e. finding the precise areas of image text elements. We propose a new text detection algorithm which is ideally suited for this purpose. The key feature of our algorithm is that it searches for text regions in a pivoting and iterative fashion. The pivoting procedure allows for recovery of distributed image text, and the iterative procedure uncovers nested image information. We believe that these two algorithm features are crucial for detecting text in biomedical images.
A text detection algorithm for biomedical images was proposed by Xu and Krauthammer REF
A new pivoting and iterative text detection algorithm for biomedical images
Journal of biomedical informatics
Journal of biomedical informatics
Computer Science Medicine