|
# FAQ |
|
|
|
## Where can I find the segmentation metrics of my experiments? |
|
**Results for the validation sets of each fold** are stored in the respective output folder after the training is completed. For example, this could be. |
|
`${RESULTS_FOLDER}/nnUNet/3d_fullres/Task003_Liver/nnUNetTrainerV2__nnUNetPlansv2.1/fold_0`. After training there will |
|
be a `validation_raw` subfolder and a `validation_raw_postprocessed` subfolder. In each of these folders is going to |
|
be a `summary.json` file with the segmentation metrics. There are metrics for each individual validation case and then |
|
at the bottom there is also a mean across all cases. |
|
|
|
**Cross-validation metrics** can only be computed after all five folds were run. You first need to run |
|
`nnUNet_determine_postprocessing` first (see `nnUNet_determine_postprocessing -h` for help). This will collect the |
|
predictions from the validation sets of the five folds, compute metrics on them and then determine the postprocessing. |
|
Once this is all done, there will be new folders located in the output directory (for example |
|
`${RESULTS_FOLDER}/nnUNet/3d_fullres/Task003_Liver/nnUNetTrainerV2__nnUNetPlansv2.1/`): `cv_niftis_raw` (raw predictions |
|
from the cross-validation) and `cv_niftis_postprocessed` (postprocessed predictions). In each of these folders is |
|
going to be a `summary.json` file with the metrics (see above). |
|
|
|
Note that the postprocessing determined on each individual fold is completely ignored by nnU-Net because it needs to |
|
find a single postprocessing configuration for the whole cross-validation. The postprocessed results in each fold are |
|
just for development purposes! |
|
|
|
**Test set results** see [here](#evaluating-test-set-results). |
|
|
|
**Ensemble performance** will be accessible here `${RESULTS_FOLDER}/nnUNet/ensembles/TASKNAME` after you ran |
|
`nnUNet_find_best_configuration`. There are summary.csv for a quick overview and then there is also going to be |
|
detailed results in the form of `summary.json` in the respective subfolders. |
|
|
|
## What postprocessing is selected? |
|
After you run `nnUNet_determine_postprocessing` (see `nnUNet_determine_postprocessing -h` for help) there will be a |
|
`postprocessing.json` file located in the output directory of your training (for example |
|
`${RESULTS_FOLDER}/nnUNet/3d_fullres/Task003_Liver/nnUNetTrainerV2__nnUNetPlansv2.1/`). If you open this with a text |
|
editor, there is a key "for_which_classes", followed by some list. For LiTS (classes 0: bg, 1: liver, 2: tumor) |
|
this can for example be: |
|
```python |
|
"for_which_classes": [ |
|
[ |
|
1, |
|
2 |
|
], |
|
1 |
|
``` |
|
This means that nnU-Net will first remove all but the largest components for the merged object consisting of classes |
|
1 and 2 (essentially the liver including the tumors) and then in a second step also remove all but the largest |
|
connected component for the liver class. |
|
|
|
Note that you do not have to run `nnUNet_determine_postprocessing` if you use `nnUNet_find_best_configuration`. |
|
`nnUNet_find_best_configuration` will do that for you. |
|
|
|
Ensemble results and postprocessing will be stored in `${RESULTS_FOLDER}/nnUNet/ensembles` |
|
(this will all be generated by `nnUNet_find_best_configuration`). |
|
|
|
## Evaluating test set results |
|
This feature was only added recently. Please run `pip install --upgrade nnunet` or reinstall nnunet from the master. |
|
|
|
You can now use `nnUNet_evaluate_folder` to compute metrics on predicted test cases. For example: |
|
|
|
``` |
|
nnUNet_evaluate_folder -ref FOLDER_WITH_GT -pred FOLDER_WITH_PREDICTIONS -l 1 2 3 4 |
|
``` |
|
|
|
This example is for a dataset that has 4 foreground classes (labels 1, 2, 3, 4). `FOLDER_WITH_GT` and |
|
`FOLDER_WITH_PREDICTIONS` must contain files with the same names containing the reference and predicted segmentations |
|
of each case, respectivelty. The files must be nifti (end with .nii.gz). |
|
|
|
## Creating and managing data splits |
|
|
|
At the start of each training, nnU-Net will check whether the splits_final.pkl file is present in the directory where |
|
the preprocessed data of the requested dataset is located. If the file is not present, nnU-Net will create its own |
|
split: a five-fold cross-validation using all the available training cases. nnU-Net needs this five-fold |
|
cross-validation to be able to determine the postprocessing and to run model/ensemble selection. |
|
|
|
There are however situations in which you may want to create your own split, for example |
|
- in datasets like ACDC where several training cases are connected (there are two time steps for each patient) you |
|
may need to manually create splits to ensure proper stratification. |
|
- cases are annotated by multiple annotators and you would like to use the annotations as separate training examples |
|
- if you are running experiments with a domain transfer, you might want to train only on cases from domain A and |
|
validate on domain B |
|
- ... |
|
|
|
Creating your own data split is simple: the splits_final.pkl file contains the following data structure (assume there are five training cases A, B, C, D, and E): |
|
```python |
|
splits = [ |
|
{'train': ['A', 'B', 'C', 'D'], 'val': ['E']}, |
|
{'train': ['A', 'B', 'C', 'E'], 'val': ['D']}, |
|
{'train': ['A', 'B', 'D', 'E'], 'val': ['C']}, |
|
{'train': ['A', 'C', 'D', 'E'], 'val': ['B']}, |
|
{'train': ['B', 'C', 'D', 'E'], 'val': ['A']} |
|
] |
|
``` |
|
|
|
Use load_pickle and save_pickle from batchgenerators.utilities.file_and_folder_operations for loading/storing the splits. |
|
|
|
Splits is a list of length NUMBER_OF_FOLDS. Each entry in the list is a dict, with 'train' and 'val' as keys and lists |
|
of the corresponding case names (without the _0000 etc!) as values. |
|
|
|
nnU-Net's five-fold cross validation will always create a list of len(splits)=5. But you can do whatever you want. Note |
|
that if you define only 4 splits (fold 0-3) and then set fold=4 when training (that would be the fifth split), |
|
nnU-Net will print a warning and proceed to use a random 80:20 data split. |
|
|
|
## How can I swap component XXX (for example the loss) of nnU-Net? |
|
|
|
All changes in nnU-Net are handled the same way: |
|
|
|
1) create a new nnU-Net trainer class. Place the file somewhere in the nnunet.training.network_training folder |
|
(any subfolder will do. If you create a new subfolder, make sure to include an empty `__init__.py` file!) |
|
|
|
2) make your new trainer class derive from the trainer you would like to change (most likely this is going to be nnUNetTrainerV2) |
|
|
|
3) identify the function that you need to overwrite. You may have to go up the inheritance hierarchy to find it! |
|
|
|
4) overwrite that function in your custom trainer, make sure whatever you do is compatible with the rest of nnU-Net |
|
|
|
What these changes need to look like specifically is hard to say without knowing what you are exactly trying to do. |
|
Before you open a new issue on GitHub, please have a look around the `nnunet.training.network_training` folder first! |
|
There are tons of examples modifying various parts of the pipeline. |
|
|
|
Also see [here](extending_nnunet.md) |
|
|
|
## How does nnU-Net handle multi-modal images? |
|
|
|
Multi-modal images are treated as color channels. BraTS, which comes with T1, T1c, T2 and Flair images for each |
|
training case will thus for example have 4 input channels. |
|
|
|
## Why does nnU-Net not use all my GPU memory? |
|
|
|
nnU-net and all its parameters are optimized for a training setting that uses about 8GB of VRAM for a network training. |
|
Using more VRAM will not speed up the training. Using more VRAM has also not (yet) been beneficial for model |
|
performance consistently enough to make that the default. If you really want to train with more VRAM, you can do one of these things: |
|
|
|
1) Manually edit the plans files to increase the batch size. A larger batch size gives better (less noisy) gradients |
|
and may improve your model performance if the dataset is large. Note that nnU-Net always runs for 1000 epochs with 250 |
|
iterations each (250000 iterations). The training time thus scales approximately linearly with the batch size |
|
(batch size 4 is going to need twice as long for training than batch size 2!) |
|
|
|
2) Manually edit the plans files to increase the patch size. This one is tricky and should only been attempted if you |
|
know what you are doing! Again, training times will be increased if you do this! 3) is a better way of increasing the |
|
patch size. |
|
|
|
3) Run `nnUNet_plan_and_preprocess` with a larger GPU memory budget. This will make nnU-Net plan for larger patch sizes |
|
during experiment planning. Doing this can change the patch size, network topology, the batch size as well as the |
|
presence of the U-Net cascade. To run with a different memory budget, you need to specify a different experiment planner, for example |
|
`nnUNet_plan_and_preprocess -t TASK_ID -pl2d None -pl3d ExperimentPlanner3D_v21_32GB` (note that `-pl2d None` will |
|
disable 2D U-Net configuration. There is currently no planner for larger 2D U-Nets). We have planners for 8 GB (default), |
|
11GB and 32GB available. If you need a planner for a different GPU size, you should be able to quickly hack together |
|
your own using the code of the 11GB or 32GB planner (same goes for a 2D planner). Note that we have experimented with |
|
these planners and not found an increase in segmentation performance as a result of using them. Training times are |
|
again longer than with the default. |
|
|
|
## Do I need to always run all U-Net configurations? |
|
The model training pipeline above is for challenge participations. Depending on your task you may not want to train all |
|
U-Net models and you may also not want to run a cross-validation all the time. |
|
Here are some recommendations about what U-Net model to train: |
|
- It is safe to say that on average, the 3D U-Net model (3d_fullres) was most robust. If you just want to use nnU-Net because you |
|
need segmentations, I recommend you start with this. |
|
- If you are not happy with the results from the 3D U-Net then you can try the following: |
|
- if your cases are very large so that the patch size of the 3d U-Net only covers a very small fraction of an image then |
|
it is possible that the 3d U-Net cannot capture sufficient contextual information in order to be effective. If this |
|
is the case, you should consider running the 3d U-Net cascade (3d_lowres followed by 3d_cascade_fullres) |
|
- If your data is very anisotropic then a 2D U-Net may actually be a better choice (Promise12, ACDC, Task05_Prostate |
|
from the decathlon are examples for anisotropic data) |
|
|
|
You do not have to run five-fold cross-validation all the time. If you want to test single model performance, use |
|
*all* for `FOLD` instead of a number. Note that this will then not give you an estimate of your performance on the |
|
training set. You will also no tbe able to automatically identify which ensembling should be used and nnU-Net will |
|
not be able to configure a postprocessing. |
|
|
|
CAREFUL: DO NOT use fold=all when you intend to run the cascade! You must run the cross-validation in 3d_lowres so |
|
that you get proper (=not overfitted) low resolution predictions. |
|
|
|
## Sharing Models |
|
You can share trained models by simply sending the corresponding output folder from `RESULTS_FOLDER/nnUNet` to |
|
whoever you want share them with. The recipient can then use nnU-Net for inference with this model. |
|
|
|
You can now also use `nnUNet_export_model_to_zip` to export a trained model (or models) to a zip file. The recipient |
|
can then use `nnUNet_install_pretrained_model_from_zip` to install the model from this zip file. |
|
|
|
## Can I run nnU-Net on smaller GPUs? |
|
nnU-Net is guaranteed to run on GPUs with 11GB of memory. Many configurations may also run on 8 GB. |
|
If you have an 11GB and there is still an `Out of Memory` error, please read 'nnU-Net training: RuntimeError: CUDA out of memory' [here](common_problems_and_solutions.md). |
|
|
|
If you wish to configure nnU-Net to use a different amount of GPU memory, simply adapt the reference value for the GPU memory estimation |
|
accordingly (with some slack because the whole thing is not an exact science!). For example, in |
|
[experiment_planner_baseline_3DUNet_v21_11GB.py](nnunet/experiment_planning/experiment_planner_baseline_3DUNet_v21_11GB.py) |
|
we provide an example that attempts to maximise the usage of GPU memory on 11GB as opposed to the default which leaves |
|
much more headroom). This is simply achieved by this line: |
|
|
|
```python |
|
ref = Generic_UNet.use_this_for_batch_size_computation_3D * 11 / 8 |
|
``` |
|
|
|
with 8 being what is currently used (approximately) and 11 being the target. Should you get CUDA out of memory |
|
issues, simply reduce the reference value. You should do this adaptation as part of a separate ExperimentPlanner class. |
|
Please read the instructions [here](documentation/extending_nnunet.md). |
|
|
|
|
|
## Why is no 3d_lowres model created? |
|
3d_lowres is created only if the patch size in 3d_fullres less than 1/8 of the voxels of the median shape of the data |
|
in 3d_fullres (for example Liver is about 512x512x512 and the patch size is 128x128x128, so that's 1/64 and thus |
|
3d_lowres is created). You can enforce the creation of 3d_lowres models for smaller datasets by changing the value of |
|
`HOW_MUCH_OF_A_PATIENT_MUST_THE_NETWORK_SEE_AT_STAGE0` (located in experiment_planning.configuration). |
|
|
|
|