Repository: bitcoin
Issue: possible corruption: missing undo file
Labels: Block storage

An undo file is written to disk in `WriteUndoDataForBlock`. If the related block file has nFile = 2, then the undo file will also have nFile = 2. It can be flushed to disk in 4 ways:
- `FlushStateToDisk` calls `FlushBlockFile()`. This flushes the last block and undo files on disk.
- When the current block file is full and fKnown=false, `FindBlockPos` will flush the last block file. It will flush the corresponding undo file if the last height in the file is equal to the current tip height.
- It seems that it is possible for the last block and undo files to be flushed if fKnown=true and nFile < last block file, but that's irrelevant here.
- In `WriteUndoDataForBlock`, if the undo file is not the last file and the height whose undo data being written is the last height in the file.

The `UndoWriteToDisk` function opens a CAutoFile and writes the data. When it returns, the CAutoFile destructor is called which calls fclose, which flushes (via `fflush`) the buffered data to the OS (at this point it's in dirty pages).

A simplified version of what can happen:

Time 1:
- tip=3
- blockfile 2 has blocks: [5, 7, 8], so nHeightLast=8
- blockfile 2 is unflushed
- undo 2 does not exist since tip=3
- the blocks being stored in this way is possible if the headers are received, then the blocks are OOO

Time 2:
- tip=3
- block 4 arrives, but cannot fit in blockfile 2
  - `FindBlockPos` is called, blockfile 2 is flushed. Note that nHeightLast=8 for this file and tip=3 since the tip is updated later. Therefore undo file 2 isn't flushed here (it also doesn't exist).
- blockfile 3 has blocks: [4]
- blockfile 3 is unflushed
- `ActivateBestChain` will update to tip=4
- `WriteUndoDataForBlock` is called for 4
  - undo file 3 is created for block 4 (in dirty pages)
  - the BLOCK_HAVE_UNDO status flag is set for the related CBlockIndex* for block 4
- `ActivateBestChain` will update to tip=5
- `WriteUndoDataForBlock` is called for 5
  - undo file 2 is created for block 5 (in dirty pages)
  - the BLOCK_HAVE_UNDO status flag is set for CBlockIndex* 5

Time 3:
- tip=5
- undo file 2, 3 are still in dirty pages
- `FlushStateToDisk` is called:
  - `FlushBlockFile` is called which flushes the *last* block+undo file to disk (3)
  - `WriteBlockIndexDB` is called and flushes the block index state to disk.

Note: At this point, block 5 has persisted the BLOCK_HAVE_UNDO status flag. Undo file 2 may still be in dirty pages and a power loss would mean that block 5 doesn't actually have rev data. The log would look like this on restart:

```
2022-07-04T04:15:07.917838Z [init] [validation.cpp:3931] [VerifyDB] Verifying last 4 blocks at level 3
2022-07-04T04:15:07.917847Z [init] [validation.cpp:3938] [VerifyDB] [0%]...ERROR: UndoReadFromDisk: Deserialize or I/O error - CAutoFile::read: end of file: unspecified iostream_category error
2022-07-04T04:15:07.918012Z [init] [util/system.h:50] [error] ERROR: VerifyDB(): *** found bad undo data at 4, hash=221d40281e7719cd139503506649647ca50cc8bf9babaa1d1f91b9f80c4f48dd
```

To aid in testing, I set the linux vm.dirty_writeback_centisecs tunable to a high value so that the kernel would wait a while before flushing dirty pages. This made sure that files with fsync called on them would get flushed way before the non-fsync'd dirty pages.

If I'm right, then it is corruption but a lot of things have to go wrong for it to happen.