Repository: apache/accumulo

Issue #608: Garbage Collector removed referenced files
Created: 2018-08-21T14:53:01Z
A significant number of Imported files were garbage collected despite the presence of "file" entries referencing them.  This impacted only the file references within one accumulo.metadata tablet which supported many tablets for a table.
The tablet server hosting the accumulo.metadata tablet died, the tablet was reassigned, and the WALs were recovered.  See the logs summary below.
16:04:50 - tserver imports one of the files that will later go missing
17:28:59 - GC recognizes candidate file is in use
17:41:25 - GC gc.SimpleGarbageCollector collect cycle statistics reported
17:42:40 - GC gc.GarbageCollectWriteAheadLogs reports statistics
17:44:49 - GC Thread gc stuck on IO to accumulo master for at least 2 minutes
17:46:26 - GC Thread gc no longer stuck
17:56:06 - datanode dies on tserver hosting accumulo.metadata
...
Tserver reports many errors related to HDFS access, including WAL delete failures
...
17:57:14 - datanode startup
17:57:44 - tserver dies
17:58:45 - master.Master reports tablet assigned to dead tserver
17:58:49 - master.Master reports old location state
17:59:20 - master.Master reports old location state
17:59:20 - master.Master reports assigning tablet
17:59:27 - master.Master reports new location state
17:59:28 - master.Master reports new location state
17:59:29 - master.Master reports new location state
17:59:26 - tserver: Loading accumulo.metadata tablet
17:59:31 - tserver: got assignment from master for tablet, loading and verifying extent
17:59:32 - Starting WAL recovery occurs accumulo.metadata tablet
17:59:32 - WAL recovery complete for accumulo.metadata tablet
17:59:34 - master.EventCoordinator reports accumulo.metadata tablet loaded
18:06:09 - GC deletes file that was previously recognized as in use
18:10:29 - gc.SimpleGarbageCollector collect cycle statistics reported
18:12:00 - Errors begin for one of the missing files
This impacted a 1.8.1 fork that contained the 1.9.1 dataloss bug fixes.

Duplicate issue #2322: GarbageCollector deleted rfiles that were still in use
Created: 2021-10-19T19:50:23Z
Describe the bug
Similar to the issue described in #1916, the garbage collector did not remove candidates that were still in use.  So these rfile references were removed from HDFS
Versions (OS, Maven, Java, and others, as appropriate):

Affected version(s) of this project: 1.9.4
OS: CentOS 7
Others:

To Reproduce
Unable to reproduce
Additional context
Unlike the issue in #1377 where the consistency checks were insufficient to catch this issue, an investigation into this issue shows that the consistency check should have caught this issue because the prior end rows didn't align.
All files removed were hosted on 1 metadata split, indicating that the scan of that metadata tablet failed.  Looking at where that metadata tablet was hosted at the time of the failure, I found an automated report in the ticket system that shows an SSD failure on that box.  HDFS was using SSDs on that node.  The admins got the ticket and did a hot swap of the SSD drives without shutting down services.  The GC logs were gone by the time I got to the node, but what I believe happened is this.

Scans were progressing through TabletIterator.  Scan on the hosting this tablet failed, the TabletIterator logged a metadata inconsistency then reset the scanner.
The reset set the range back to start at the missing tablet and scanned again.  The box was still in a funky state so HDFS again didn't return data.  The TabletIterator found 0 entries again for the tablet and threw a TabletDeletedException.
The TabletDeletedException is just a RuntimeException and the TabletIterator continues scanning without stopping the entire GC process.

This is the only way I can explain what happened, as the consistency checks should have caught this issue.  I would like to suggest that if a TabletDeletedException is thrown during the TabletIterator scan, we cancel the entire GC cycle and start again.

Closing comment (2022-05-25T06:04:19Z): This seems related to, or perhaps a duplicate of, #608
Should we close one of them?
Also, I was thinking. The mapping of table IDs, which are BigInteger values, to their serialized form in the metadata table is not a monotonic function (it doesn't preserve order). So, when a new table is created, there's a good chance it's metadata is injected in the middle of the metadata table somewhere, and not at the end of the table in the last tablet. This might temporarily reduce initial hotspotting for new table's being created, but it could also be related to some potential garbage collection bugs if we don't account for it properly.
Interestingly, the avoidance of hotspotting would only occur for the first few tables created, when the most significant digit is still changing frequently. We go back to hotspotting again quickly after a bunch of tables are created. This hotspotting is probably not substantial, but if we want to avoid it, that could be a separate ticket... to try to generate tableIds that are more spread out. However, if it turns out that inserting new table metadata (I'm particularly concerned about cloned tables, which have duplicate file references to existing tables) into the middle of a candidate garbage collection scan is the source of any bugs, we may want to just accept the hotspotting, and generate table IDs that are always appended to the end of the table, so their metadata is never inserted into the middle of a candidate scan.