File size: 2,727 Bytes
506da10 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 |
# Test Server Evaluation on COCO dataset
This page walks through the steps required to convert DeepLab2 predictions for
test server evaluation on [COCO](https://cocodataset.org/).
A high-level overview of the whole process:
1. Save raw panoptic prediction in the two-channel format.
2. Convert predictions in the two-channel format to the panoptic COCO format.
3. Run local validation set evaluation or prepare test set evaluation.
We also define some environmental variables for simplicity and convenience:
`BASE_MODEL_DIRECTORY`: variables set in textproto file, which defines where all
checkpoints and results are saved.
`DATA_ROOT`: where the original COCO dataset is located.
`PATH_TO_SAVE`: where the converted results should be saved.
## Save Raw Panoptic Prediction
Save the raw panoptic predictions in the
[two-channel panoptic format](https://arxiv.org/pdf/1801.00868.pdf) by ensuring
the following fields are set properly in the textproto config file.
```
eval_dataset_options.decode_groundtruth_label = false
evaluator_options.save_predictions = true
evaluator_options.save_raw_predictions = true
evaluator_options.convert_raw_to_eval_ids = true
```
Then run the model in evaluation modes (with `--mode=eval`), and the results
will be saved at ${BASE_MODEL_DIRECTORY}/vis/raw_panoptic/\*.png.
## Convert the Prediction Format
Convert prediction results saved in the
[two-channel panoptic format](https://arxiv.org/pdf/1801.00868.pdf) to the
panoptic COCO format.
```bash
python panopticapi/converters/2channels2panoptic_coco_format.py \
--source_folder=${BASE_MODEL_DIRECTORY}/vis/raw_panoptic \
--images_json_file=${DATA_ROOT}/annotations/IMG_JSON \
--categories_json_file=panopticapi/panoptic_coco_categories.json \
--segmentations_folder=${PATH_TO_SAVE}/panoptic_cocoformat \
--predictions_json_file=${PATH_TO_SAVE}/panoptic_cocoformat.json
```
The `IMG_JSON` refers to `panoptic_val2017.json` for *val* set and
`image_info_test-dev2017.json` for *test-dev* set.
## Run Local Evaluation Scripts (for *validation* set)
Run the [official scripts](https://github.com/cocodataset/panopticapi) to
evaluate validation set results.
```bash
python panopticapi/evaluation.py \
--pred_json_file=${PATH_TO_SAVE}/panoptic_cocoformat.json \
--pred_folder=${PATH_TO_SAVE}/panoptic_cocoformat \
--gt_json_file=${DATA_ROOT}/annotations/panoptic_val2017.json \
--gt_folder=${DATA_ROOT}/annotations/panoptic_val2017
```
## Prepare Submission Files (for *test* set)
Run the following command to prepare a submission file for test server
evaluation.
```bash
zip -r coco_test_submission_panoptic.zip ${PATH_TO_SAVE}/panoptic_cocoformat ${PATH_TO_SAVE}/panoptic_cocoformat.json
```
|