Patent Document

CROSS-REFERENCE TO RELATED APPLICATION 
       [0001]    This application is a continuing application of U.S. patent application Ser. No. 13/093,274, filed Apr. 25, 2011, by Ashish Batwara, entitled “Isolating and Correcting VPD Data Mismatch and/or Corruption,” the entire content of which is specifically incorporated herein by reference for all that it discloses and teaches. 
     
    
     BACKGROUND 
       [0002]    Mass storage systems continue to provide increased storage capacities to satisfy user demands. Photo and movie storage, and photo and movie sharing are examples of applications that fuel the growth in demand for larger and larger storage systems. 
         [0003]    A solution to these increasing demands is the use of arrays of multiple inexpensive disks. These arrays may be configured in ways that may provide redundancy and error recovery without any loss of data. These arrays may also be configured to increase read and write performance by allowing data to be read or written simultaneously to multiple disk drives. These arrays may also be configured to allow “hot-swapping” which allows a failed disk to be replaced without interrupting the storage services of the array. Multiple disk storage systems typically utilize a controller that shields the user or host system from the details of managing the storage array. The controller may make the storage array appear as one or more disk drives (or volumes). This is accomplished in spite of the fact that the data (or redundant data) for a particular volume may be spread across multiple disk drives. 
         [0004]    To facilitate the development and deployment of these multiple disk storage systems, several specifications have been developed. Few of these specifications are promulgated by the Storage Bridge Bay Working Group, Inc. In particular, the Storage Bridge Bay Working Group, Inc. has promulgated the Storage Bridge Bay (SBB) Specification, Version 2.0, Jan. 28, 2008 available at www.sbbwg.org. This specification aims to define common mechanical, electrical, and internal interfaces between a storage enclosure and the electronics cards that give the system a function. The ultimate aim of the SBB specification is to allow multiple different controllers to be used in a single, standard compliant, chassis to change the “personality” of the storage array. 
       SUMMARY OF THE INVENTION 
       [0005]    An embodiment of the invention may therefore comprise a method of detecting a product data error in a storage system, comprising: reading at least a first portion of a first vital product data (VPD) EEPROM; reading at least a second portion of a second VPD EEPROM that corresponds to said first portion of said first VPD EEPROM; receiving a first indicator of whether said reading said at least said first portion failed; receiving a second indicator of whether said reading said at least said second portion failed; based on said first indicator and said second indicator, comparing said at least said first portion and said at least said second portion; based on a result of said comparing indicating a match, using data from said at least said first portion; based on said first indicator indicating an error with said first reading, using data from said at least said second portion. 
         [0006]    An embodiment of the invention may therefore further comprise a method of detecting a bus malfunction in a storage system, comprising: using a first controller, reading at least a first portion of a first vital product data (VPD) EEPROM; using said first controller, reading at least a second portion of a second VPD EEPROM that corresponds to said first portion of said first VPD EEPROM; receiving a first indicator of whether said reading said at least said first portion failed; receiving a second indicator of whether said reading said at least said second portion failed; using a second controller, reading at least a third portion of said first VPD EEPROM; using said second controller, reading at least a fourth portion of said second VPD EEPROM that corresponds to said third portion of said first VPD EEPROM; receiving a third indicator of whether said reading said at least said third portion failed; and, receiving a fourth indicator of whether said reading said at least said fourth portion failed. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0007]      FIG. 1  is a block diagram illustrating a storage system. 
           [0008]      FIG. 2  is a flowchart illustrating a method of detecting an error. 
           [0009]      FIG. 3  is a flowchart illustrating a method of isolating and correcting an error. 
           [0010]      FIG. 4  is a flowchart illustrating a method of isolating and correcting an error. 
           [0011]      FIG. 5  is a block diagram of a computer system. 
       
    
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS 
       [0012]      FIG. 1  is a block diagram illustrating a storage system. In  FIG. 1 , storage system  100  comprises: midplane  110 , power supply  115 , controller A  120 , controller B  121 , storage device  130 , storage device  131 , and enclosure  150 . Controller A  120  and optional controller B  121  (if present) are operatively coupled to midplane  110 . Storage device  130  and optional storage device  131  (if present) are operatively coupled to midplane  110 . Thus, controllers  120 - 121  may operatively connect or exchange information with storage devices  130 - 131  via midplane  110 . Controllers  120 - 121  may operatively connect with, or exchange that information with, other devices (not shown) that are coupled to storage system  100 . Storage system  100  may comprise additional controllers. Storage system  100  may comprise additional storage devices. However, these have been omitted from  FIG. 1  for the sake of brevity. 
         [0013]    Storage system  100  may be, or comprise, a system that conforms to the SBB specification. Thus, controllers  120 - 121  may be, or comprise, controllers that are compatible with or described by, for example, InfiniBand, Just a Bunch Of Disks or Just a Box Of Drives (JBOD), Redundant Array of Inexpensive Disks (RAID), Network Attached Storage (NAS), Storage Array Network (SAN), iSCSI SAN, or a Virtual Tape Library (VTL). Thus, storage devices  130 - 131  may be, or comprise, hard disk drives. Storage devices  130 - 131  may be, or comprise, other types of drives such as solid state disk drives, tape drives, and ROM drives. Other types of storage devices are possible. 
         [0014]    Midplane  110  includes Two Wire Interface (TWI) #0  140 , TWI #1  141 , and TWI #2  142 . TWI #0 operatively couples controller A  120  to controller B  121  (if present). TWI #1 operatively couples controller A  120 , controller B  121  (if present), to Vital Product Data (VPD) EEPROM #1  111 . TWI #2 operatively couples controller A  120 , controller B  121  (if present), to VPD EEPROM #2  112 . Functions and specifications for at least midplane  110 , controllers  120 - 121 , VPD EEPROM #1  111 , VPD EEPROM #2  112 , TWI #0  140 , TWI #1  141 , and TWI #2  142  are given in Storage Bridge Bay (SBB) Specification, Version 2.0, Jan. 28, 2008 available at www.sbbwg.org. 
         [0015]    In an embodiment, controller A  120  and controller B  121  (if present) read VPD EEPROM #1  111  and VPD EEPROM #2  112  when storage system  100  is initializing. If any of these reads fails, the controller  120 - 121  detecting the read failure marks the appropriate VPD EEPROM  111 - 112  as suspect. If the reads are successful, controllers  120 - 121  compare the contents of VPD EEPROM #1  111  and VPD EEPROM #2  112  to determine if they are the same. If the contents of VPD EEPROM #1  111  and VPD EEPROM #2  112  are the same, controller A  120  and controller B  121  (if present) uses the contents of VPD EEPROM #1  111 . If either the contents of VPD EEPROM #1  111  and VPD EEPROM #2  112  are not the same, or at least one of the reads were are not successful, further isolation and correction is performed. The process for further isolation and correction depends on whether controller B  121  is present. 
         [0016]    In an embodiment, when controller B  121  is not present, controller A  120  locks itself down (e.g., halts or stops read/writes) if the reads from both VPD EEPROM #1  111  and VPD EEPROM #2  112  failed. Controller A  120  may report that either controller A  120  is faulty, or that both TWI #1  141  and TWI #2  142  are faulty. In the case where only one of the reads from VPD EEPROM #1  111  and VPD EEPROM #2  112  failed, controller A  120  uses the VPD EEPROM  111 - 112  not associated with the failed read. 
         [0017]    In a case where the reads from both VPD EEPROM #1  111  and VPD EEPROM #2  112  succeeded, but the contents of VPD EEPROM #1  111  and VPD EEPROM #2  112  mismatch, controller A performs a checksum verification of VPD EEPROM #1  111  and VPD EEPROM #2  112 . If these checksum verifications both pass, controller A  120  uses the write-counter defined for VPD EEPROM #1  111  and VPD EEPROM #2  112  to select a VPD EEPROM  111 - 112 . The selection of the VPD EEPROM  111 - 112  may be based on which VPD EEPROM  111 - 112  has a higher write-count value (and thus, is the more recently written VPD EEPROM  111 - 112 ). The selected VPD EEPROM  111 - 112  may then be used to rewrite the contents of the non-selected VPD EEPROM  111 - 112 . In the case that the write-counter values for VPD EEPROM #1  111  and VPD EEPROM #2 are equal, controller A  120  may use an arbitrarily selected VPD EEPROM  111 - 112  (e.g., VPD EEPROM #1  111 ). 
         [0018]    If the checksum verifications fail on one VPD EEPROM  111 - 112 , but pass on the other, controller A  120  may rewrite the contents of the VPD EEPROM  111 - 112  that failed checksum verification with the contents of the VPD EEPROM  111 - 112  that passed the checksum verification. If a rewrite of a VPD EEPROM  111 - 112  fails, then controller A  120  uses the VPD EEPROM  111 - 112  that passed the checksum verification. If the checksum verifications fail on both VPD EEPROMs  111 - 112 , controller A  120  locks down. 
         [0019]    In an embodiment, when controller B  121  is present, controllers  120 - 121  read the status of the reads that the other controller  120 - 121  performed. In a case where the reads from both VPD EEPROM #1  111  and VPD EEPROM #2  112  failed for both controllers  120 - 121 , both controllers  120 - 121  lock down. Storage system  100  may report that both TWI #1  141  and TWI #2  142  are faulty. 
         [0020]    In a case where the reads from both VPD EEPROM #1  111  and VPD EEPROM #2  112  failed for one controller  120 - 121 , but passed for the other controller, the controller  120 - 121  with the failing reads locks down. This condition indicates that the controller  120 - 121  with the failing reads is faulty. In a case where the read from VPD EEPROM #1  111  failed for both controllers  120 - 121 , and the read from VPD EEPROM #2  112  succeeded for both controllers  120 - 121 , both controllers  120 - 121  use VPD EEPROM #2  112 . This condition indicates that TWI #1  141  is faulty. 
         [0021]    In a case where the read from VPD EEPROM #1  111  failed for one controller  120 - 121 , but succeeded for the other controller  120 - 121 , the controller  120 - 121  with the failed read uses VPD EEPROM #2  112 . This condition indicates that the controller  120 - 121  with the failed read is faulty. 
         [0022]    In a case where the read from VPD EEPROM #2  112  failed for one controller  120 - 121 , but succeeded for the other controller  120 - 121 , the controller  120 - 121  with the failed read uses VPD EEPROM #1  111 . This condition indicates that the controller  120 - 121  with the failed read is faulty. 
         [0023]    In a case where the read from VPD EEPROM #2  112  failed for both controllers  120 - 121 , both controllers  120 - 121  use VPD EEPROM #1  111 . This condition indicates that TWI #2  142  is faulty. 
         [0024]    In a case where the reads from both VPD EEPROM #1  111  and VPD EEPROM #2  112  succeeded for both controllers  120 - 121 , but the contents of VPD EEPROM #1  111  and VPD EEPROM #2  112  are detected to mismatch by at least one controller  120 - 121 , controllers  120 - 121  perform a checksum verifications of VPD EEPROM #1  111  and VPD EEPROM #2  112 . If these checksum verifications both pass, controllers  120 - 121  use the write-counter defined for VPD EEPROM #1  111  and VPD EEPROM #2  112  to select a VPD EEPROM  111 - 112 . The selection of the VPD EEPROM  111 - 112  may be based on which VPD EEPROM  111 - 112  has a higher write-count value (and thus, is the more recently written VPD EEPROM  111 - 112 ). The selected VPD EEPROM  111 - 112  may then be used to rewrite the contents of the non-selected VPD EEPROM  111 - 112 . In the case that the write-counter values for VPD EEPROM #1  111  and VPD EEPROM #2  112  are equal, controllers  120 - 121  may use an arbitrarily selected VPD EEPROM  111 - 112  (e.g., VPD EEPROM #1  111 ). 
         [0025]    In the case where a checksum verification performed by one of controllers  120 - 121  fails on one of the VPD EEPROMs  111 - 112 , and this failure is also detected by the other controller  120 - 121 , the contents of the VPD EEPROM  111 - 112  that did not pass are rewritten from the non-corrupted VPD EEPROM  111 - 112 . In the case where there is a write failure during the rewrite of the corrupted VPD EEPROM  111 - 112 , or checksum verification failure is not also detected by the other controller  120 - 121 , controllers  120 - 121  use the non-corrupted VPD EEPROM  111 - 112 . If checksum verification fails for both VPD EEPROM #1  111  and VPD EEPROM #2, the controller  120 - 121  (or both) detecting the failure of both VPD EEPROM #1  111  and VPD EEPROM #2 locks down. 
         [0026]      FIG. 2  is a flowchart illustrating a method of detecting an error. The steps illustrated in  FIG. 2  may be performed by one or more elements of storage system  100 . VPD EEPROM #1  111  and VPD EEPROM #2 (hereinafter VPD #1 and VPD #2, for brevity) are read. For example, controller A  120  (or controller B  121 , if preset) may read VPD #1 and VPD #2 ( 202 ). In box  204 , it is determined if both reads were successful ( 204 ). If both reads were successful, flow proceeds to box  206 . If both reads were not successful, flow proceeds to box  210 . If both reads were successful, it is determined if the contents from VPD #1 match the contents from VPD #2 ( 206 ). If the contents of VPD #1 match the contents of VPD #2, flow ends in box  208 . If both reads were not successful, flow proceeds to another figure via reference label A  220 . 
         [0027]    If both reads were not successful as determined in box  204 , flow proceeds to box  210 . In box  210 , it is determined if the read of VPD #1 was successful ( 210 ). If the read of VPD #1 was successful, flow proceeds to another figure via reference label A  220 . If the read of VPD #1 was not successful, flow proceeds to box  212 . In box  212 , it is determined if the read of VPD #2 was successful ( 212 ). If the read of VPD #2 was successful, flow proceeds to another figure via reference label A  220 . If the read of VPD #2 was not successful, flow proceeds to box  214 . In box  214 , both VPD #1 and VPD #2 are marked as suspect ( 214 ). Flow then proceeds to another figure via reference label A  220 . 
         [0028]      FIG. 3  is a flowchart illustrating a method of isolating and correcting an error. The steps illustrated in  FIG. 3  may be performed by one or more elements of storage system  100 . The steps illustrated in  FIG. 3  are typically performed when controller  121  is not present (i.e., there is not a redundant controller present in storage system  100 ). 
         [0029]    In  FIG. 3 , flow begins via reference label A  220  and proceeds to box  302 . In box  302 , it is determined whether both VPD #1 and VPD #2 are marked as suspect. If both VPD #1 and VPD #2 are marked as suspect, flow proceeds to box  316 . If both VPD #1 and VPD #2 are not marked as suspect, flow proceeds to box  304 . In box  316 , the controller is locked down ( 316 ). Flow then ends in box  340 . 
         [0030]    If both VPD #1 and VPD #2 are not marked as suspect, it is determined whether VPD #1 (alone) is marked suspect ( 304 ). If VPD #1 (alone) is marked suspect, flow proceeds to box  318 . If VPD #1 is not marked suspect, flow proceeds to box  306 . If VPD #1 (alone) is marked suspect, TWI #1 is reported as failed and TWI #2 is used. It is determined whether VPD #2 (alone) is marked suspect ( 306 ). If VPD #2 (alone) is marked suspect, flow proceeds to box  320 . If VPD #2 is not marked suspect, flow proceeds to box  308 . If VPD #2 (alone) is marked suspect, TWI #2 is reported as failed and TWI #1 is used. It should be understood that at this stage it is clear that VPD #1 and VPD #2 can both be read, but their contents mismatch. 
         [0031]    It is determined whether both VPD #1 and VPD #2 checksums are correct but their contents mismatch ( 308 ). If both VPD #1 and VPD #2 checksums are correct, flow proceeds to box  322 . If both VPD #1 and VPD #2 checksums are not correct, flow proceeds to box  310 . In box  310 , it is determined if the checksum for VPD #1 is correct ( 310 ). If the checksum for VPD # 1  is correct, flow proceeds to box  324 . If the checksum for VPD #1 is not correct, flow proceeds to box  312 . In box  312 , it is determined if the checksum for VPD #2 is correct ( 312 ). If the checksum for VPD #2 is correct, flow proceeds to box  326 . If the checksum for VPD #2 is not correct, flow proceeds to box  314 . In box  314 , the controller is locked down ( 314 ). Flow then ends in box  340 . 
         [0032]    If both VPD #1 and VPD #2 checksums are determined to be correct in box  308 , it is determined whether the write-counter for VPD #1 is greater than the write counter for VPD #2 ( 322 ). If the write-counter for VPD #1 is greater than the write counter for VPD #2, flow proceeds to box  324 . If the write-counter for VPD #1 is not greater than the write counter for VPD #2, flow proceeds to box  330 . In box  324 , VPD #2 is rewritten from VPD #1. Flow then proceeds to box  328 . In box  328 , it is determined whether the rewrite was successful ( 328 ). If the rewrite was successful, flow ends in box  340 . If the write was not successful, and error is reported in box  332  and then flow ends in box  340 . 
         [0033]    If the write-counter for VPD #1 is not greater than the write counter for VPD #2 in box  322 , it is determined whether the write counter for VPD #1 is equal to the write counter for VPD #2 ( 330 ). If the write counter for VPD #1 is equal to the write counter for VPD #2, flow proceeds to box  334 . If the write counter for VPD #1 is not equal to the write counter for VPD #2 flow proceeds to box  326 . In box  326 , VPD #1 is rewritten from VPD #2. In box  334 , the controller uses VPD #1 ( 334 ). Flow then ends in box  340 . 
         [0034]      FIG. 4  is a flowchart illustrating a method of isolating and correcting an error. The steps illustrated in  FIG. 4  may be performed by one or more elements of storage system  100 . The steps illustrated in  FIG. 4  are typically performed when controller  121  is present (i.e., there is a redundant controller present in storage system  100 ). 
         [0035]    In  FIG. 4 , flow begins via reference label A  220  and proceeds to box  402 . In box  402 , the read status from the alternate controller is checked ( 402 ). For example, controller A  120  may check the read status from controller B  121 . It is determined if both reads failed on both controllers ( 404 ) (i.e., the reads of VPD #1 and VPD #2 failed on both controller A  120  and controller B  121 ), flow proceeds to box  406  where both controllers are locked down and a failure of both TWI #1 and TWI #2 is reported. 
         [0036]    If both reads did not fail on both controllers, it is determined if both reads failed on this controller ( 408 ). I.e., controller A  120  determines if the reads of both VPD #1 and VPD #2 failed for controller A  120 . Controller B performs a similar action. If both reads failed on this controller, flow proceeds to box  410  where this controller is locked down and reported as failing. 
         [0037]    If both reads did not fail on this controller, it is determined whether the reads of VPD #1 failed on both controllers ( 412 ). If the reads of VPD #1 failed on both controllers, VPD #2 is used and TWI #1 is reported as failing ( 414 ). 
         [0038]    If the reads of VPD #1 did not fail on both controllers, it is determined if the read of VPD #1 failed on this controller, but succeeded on the alternate controller ( 416 ). If the read of VPD #1 failed on this controller, but succeeded on the alternate controller, VPD #2 is used on this controller and it is reported that this controller has a failure ( 418 ). 
         [0039]    If the read of VPD #2 failed on this controller and on the alternate controller ( 420 ), VPD #1 is used and a failure of TWI #2 is reported ( 422 ). 
         [0040]    If the read of VPD #2 did not fail on both controllers, it is determined whether the read of VPD #2 failed on this controller, but succeeded on the alternate controller ( 424 ). If the read of VPD #2 failed on this controller, but succeeded on the alternate controller, VPD #1 is used, and a failure of this controller is reported ( 426 ). If the read of VPD #2 did not fail on this controller and succeed on the alternate controller, the mismatch, checksum check, and rewrite procedure (described previously) is performed ( 428 ). 
         [0041]    The systems, engines, databases, processors, modules, and functions described above may be implemented with or executed by one or more computer systems. The methods described above may also be stored on a computer readable medium. Many of the elements of storage system  100  may be, comprise, or include computers systems. This includes, but is not limited to, controller  120 , controller  121 , and midplane  110 . 
         [0042]      FIG. 5  illustrates a block diagram of a computer system. Computer system  500  includes communication interface  520 , processing system  530 , storage system  540 , and user interface  560 . Processing system  530  is operatively coupled to storage system  540 . Storage system  540  stores software  550  and data  570 . Processing system  530  is operatively coupled to communication interface  520  and user interface  560 . Computer system  500  may comprise a programmed general-purpose computer. Computer system  500  may include a microprocessor. Computer system  500  may comprise programmable or special purpose circuitry. Computer system  500  may be distributed among multiple devices, processors, storage, and/or interfaces that together comprise elements  520 - 570 . 
         [0043]    Communication interface  520  may comprise a network interface, modem, port, bus, link, transceiver, or other communication device. Communication interface  520  may be distributed among multiple communication devices. Processing system  530  may comprise a microprocessor, microcontroller, logic circuit, or other processing device. Processing system  530  may be distributed among multiple processing devices. User interface  560  may comprise a keyboard, mouse, voice recognition interface, microphone and speakers, graphical display, touch screen, or other type of user interface device. User interface  560  may be distributed among multiple interface devices. Storage system  540  may comprise a disk, tape, integrated circuit, RAM, ROM, network storage, server, or other memory function. Storage system  540  may be a computer readable medium. Storage system  540  may be distributed among multiple memory devices. 
         [0044]    Processing system  530  retrieves and executes software  550  from storage system  540 . Processing system may retrieve and store data  570 . Processing system may also retrieve and store data via communication interface  520 . Processing system  550  may create or modify software  550  or data  570  to achieve a tangible result. Processing system may control communication interface  520  or user interface  570  to achieve a tangible result. Processing system may retrieve and execute remotely stored software via communication interface  520 . 
         [0045]    Software  550  and remotely stored software may comprise an operating system, utilities, drivers, networking software, and other software typically executed by a computer system. Software  550  may comprise an application program, applet, firmware, or other form of machine-readable processing instructions typically executed by a computer system. When executed by processing system  530 , software  550  or remotely stored software may direct computer system  500  to operate as described herein. 
         [0046]    The foregoing description of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and other modifications and variations may be possible in light of the above teachings. The embodiment was chosen and described in order to best explain the principles of the invention and its practical application to thereby enable others skilled in the art to best utilize the invention in various embodiments and various modifications as are suited to the particular use contemplated. It is intended that the appended claims be construed to include other alternative embodiments of the invention except insofar as limited by the prior art.

Technology Category: g