| # FAQ | |
| ________________________________________________________________________________ | |
| **Q1: What should I do if I encounter OOM (out-of-memory) while training the | |
| models?** | |
| **A1**: To avoid OOM, you could try: | |
| 1. reducing the training crop size (i.e., the flag `crop_size` in | |
| `train_dataset_options`, and see Q2 for more details), which reduces the | |
| input size during training, | |
| 2. using a larger output stride (e.g., 32) in the backbone (i.e., the flag | |
| `output_stride` in `model_options`, and see Q3 for more details), which | |
| reduces the usage of atrous convolution, | |
| 3. using a smaller backbone, such as ResNet-50. | |
| ________________________________________________________________________________ | |
| **Q2: What is the `crop_size` I need to set?** | |
| **A2**: DeepLab framework always uses `crop_size` equal to `output_stride` * k + | |
| 1, where k is an integer. | |
| * During inference/evaluation, since DeepLab framework uses whole-image | |
| inference, we need to set k so that the resulting `crop_size` (in | |
| `eval_dataset_options`) is slightly larger the largest image dimension in | |
| the dataset. For example, we set eval_crop_size = 1025x2049 for Cityscapes | |
| images whose image dimension is all equal to 1024x2048. | |
| * During training, we could set k to be any integer as long as it fits to your | |
| device memory. However, we notice a better performance when we have the same | |
| `crop_size` during training and evaluation (i.e., also use whole-image crop | |
| size during training). | |
| ________________________________________________________________________________ | |
| **Q3: What output stride should I use in the backbone?** | |
| **A3**: Using a different output stride leads to a different accuracy-and-memory | |
| trade-off. For example, DeepLabv1 uses output stride = 8, but it requires a lot | |
| of device memory. In DeepLabv3+ paper, we found that using output stride = 16 | |
| strikes the best accuracy-and-memory trade-off, which is then our default | |
| setting. If you wish to further reduce the memory usage, you could set output | |
| stride to 32. Additionally, we suggest adjusting the `atrous_rates` in the ASPP | |
| module as follows. | |
| * If `backbone.output_stride` = 32, use `atrous_rates` = [3, 6, 9]. | |
| * If `backbone.output_stride` = 16, use `atrous_rates` = [6, 12, 18]. | |
| * If `backbone.output_stride` = 8, use `atrous_rates` = [12, 24, 36]. | |
| Note that these settings may not be optimal. You may need to adjust them to | |
| better fit your dataset. | |
| ________________________________________________________________________________ | |
| **Q4: Why are the results reported by the provided evaluation code slightly | |
| different from the official evaluation code (e.g., | |
| [Cityscapes](https://github.com/mcordts/cityscapesScripts))?** | |
| **A4**: In order to run everything end-to-end in the TensorFlow system (e.g., | |
| the on-line evaluation during training), we re-implemented the evaluation codes | |
| in TensorFlow. Additionally, our whole system, including the training and | |
| evaluation pipelines, uses the panoptic label format (i.e., `panoptic_label = | |
| semantic_label * label_divisor + instance_id`, where the `label_divisor` should | |
| be larger than the maximum number of instances per image), instead of the JSON | |
| [COCO formats](https://cocodataset.org/#format-data). These two changes along | |
| with rounding and similar issues result in some minor differences. Therefore, | |
| our re-implemented evaluation code is mainly used for TensorFlow integration | |
| (e.g., the support of on-line evaluation in TensorBoard). The users should run | |
| the corresponding official evaluation code in order to compare with other | |
| published papers. Note that all the reported numbers in our papers are evaluated | |
| with the official evaluation code. | |
| To facilitate the conversion between prediction formats, we also provide | |
| instructions for running the official evaluation codes on | |
| [Cityscapes](setup/cityscapes_test_server_evaluation.md) and | |
| [COCO](setup/coco_test_server_evaluation.md). | |
| ________________________________________________________________________________ | |
| **Q5: What should I do, if I could not manage to compile TensorFlow along with | |
| the provided efficient merging operation `merge_semantic_and_instance_maps`?** | |
| **A5**: In this case, we provide another fallback solution, which implements the | |
| merging operation with pure tf functions. This fallback solution does not | |
| require any TensorFlow compilation. However, note that compared to our provided | |
| TensorFlow merging operation `merge_semantic_and_instance_maps`, its inference | |
| speed is slower and the resulting segmentation performance may also be slightly | |
| lower. | |
| To use the pure-tf-function version of `merge_semantic_and_instance_maps`, set | |
| `merge_semantic_instance_with_tf_op` to `false` in your config's | |
| `evaluator_options`. | |