Repository: mlcommons/inference
Issue: Flexible model mapping for the Open division
Root cause: Architectural Constraint

By definition, the Open division is open to submissions involving models different from the standard ones used for the Closed division. 

Unfortunately, any non-standard model currently has to be vetted by making changes to the submission checker script, namely, its `model_mapping` part, which looks something like this:
```
        # TODO: Update this list.
        "model_mapping": {
            # map model names to the official mlperf model class
            "ssd-mobilenet": "ssd-small",
            "ssd-resnet34": "retinanet",
            "mobilenet": "resnet",
            "resnet50": "resnet",
            "ssd_resnet101_v1_fpn_640x640": "ssd-small",
            ...
```

Currently, each such mapping has to be added prior to submission, and unintentionally discloses intent to submit ahead of the deadline.

A better approach would be to remove such mappings from the submission checker, and provide them in a separate file. For example, this file can be in the JSON format and by default found under `open/<submitter>/model_mapping.json` (or provided as a `--model_mapping_file` argument to the submission checker).

Another improvement would be to use generic task names instead of model names in the mapping, for example, `object_detection` instead of `retinanet` and `object_detection_coco` instead of deprecated `ssd-small` and `ssd-large`. For example, the [v2.1](https://github.com/mlcommons/inference/blob/b57978be8c27a03b4d2c915ccd9c6e459abe3a54/tools/submission/submission_checker.py#L868) mapping could be encoded as:

```
{
    "image_classification" : [ "resnet", "mobilenet" ],
    "object_detection_coco" : [ "ssd-mobilenet", "ssd-resnet34", "ssd_resnet101_v1_fpn_640x640", ... ]
}
```

There is also a [somewhat related part](https://github.com/mlcommons/inference/blob/b57978be8c27a03b4d2c915ccd9c6e459abe3a54/tools/submission/submission_checker.py#L1178) of the checker predating proper model mapping:

```
    # simple mapping, ie resnet50->resnet ?
    mlperf_model = self.base["model_mapping"].get(model)
    if mlperf_model:
      return mlperf_model

    # try to guess
    if "ssdlite" in model or "ssd-inception" in model or "yolo" in model or \
        "ssd-mobilenet" in model or "ssd-resnet50" in model:
      model = "ssd-small"
    elif "mobilenet" in model:
      model = "mobilenet"
    elif "efficientnet" in model or "resnet50" in model:
      model = "resnet"
    elif "rcnn" in model:
      model = "ssd-small"
    elif "bert-99.9" in model:
      model = "bert-99.9"
    elif "bert-99" in model:
      model = "bert-99"
    # map again, for example v0.7 does not have mobilenet so it needs to be mapped to resnet
    mlperf_model = self.base["model_mapping"].get(model, model)
```

which can be simplified.

> Another improvement would be to use generic task names instead of model names in the mapping, for example, object_detection instead of retinanet and object_detection_coco instead of deprecated ssd-small and ssd-large.

@psyhtest Do you mean we should have a configuration for each task? Or continue having a configuration for each model and group the common models in open division by task?