Repository: transformers
Issue: Wandb sweeps integraition: custom objective
Labels: WIP,Integrations

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-5.14.0-162.6.1.el9_1.0.1.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.10
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu117 (True) -- using torch in my experiments
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.10 (cpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10


### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I followed [this guide](https://huggingface.co/docs/transformers/hpo_train) to use wandb sweeps with the trainer by modifying the summarization scripts slightly. Below are the hp_space, model_init, and hyperparameter_search commands that I use.

Most notably, the objective is to maximize the sum of sari and rougeLsum.

```python
    def wandb_hp_space(trial):
        return {
            "method": "bayes",
            "metric": {
                "name": "objective",
                "goal": "minimize" if hyperopt_args.hparam_optimize_for_loss else "maximize"
            },
            "parameters": {
                "num_train_epochs": {"min": hyperopt_args.hparam_epoch_min, "max": hyperopt_args.hparam_epoch_max}
            },
            "run_cap": hyperopt_args.hparam_max_trials
        }

    def model_init(trial):
        return AutoModelForSeq2SeqLM.from_pretrained(
            model_args.model_name_or_path,
            from_tf=bool(".ckpt" in model_args.model_name_or_path),
            config=config,
            cache_dir=model_args.cache_dir,
            revision=model_args.model_revision,
            use_auth_token=True if model_args.use_auth_token else None,
        )

    def hparam_objective(metrics: Dict[str, float]) -> float:
        metrics = copy.deepcopy(metrics)

        if hyperopt_args.hparam_optimize_for_loss:
            return metrics["eval_loss"]

        return metrics["eval_rougeLsum"] + metrics["eval_sari"]

    best_trial = trainer.hyperparameter_search(
        compute_objective=hparam_objective,
        backend="wandb",
        # I think that this is only used to set the column in the sweep chart but does not mean that we use
        # this metric only for optimization. That is what the hparam_objective is for?
        metric="eval/sari",
        hp_space=wandb_hp_space,
        n_trials=hyperopt_args.hparam_max_trials,
        direction="minimize" if hyperopt_args.hparam_optimize_for_loss else "maximize",
    )
```

The problem is that when I look at the wandb interface at the generated sweep config, it looks like this:

```yaml
method: bayes
metric:
  goal: maximize
  name: eval/sari
parameters:
  num_train_epochs:
    distribution: int_uniform
    max: 30
    min: 2
run_cap: 16
```

So the generated sweep config includes `eval/sari` as the metric name because I passed it in the `hyperparameter_search`. But as you can read in the comment, I thought this was only for the wandb visualization but now I am not so sure. When I leave out the `metric` keyword (as in the example), wandb seems to fallback to `eval/loss`.

```yaml
method: bayes
metric:
  goal: maximize
  name: eval/loss
parameters:
  num_train_epochs:
    distribution: int_uniform
    max: 30
    min: 2
run_cap: 16
```

My worry here is the disconnect between the generated sweep config and the custom objective function. What is wandb optimizing now? My custom objective function (sari+rougeLsum) or `metric` that is passed to `hyperparameter_search` for `direction`?

### Expected behavior

More clarity about the relationship between a wandb sweep configuration and how the trainer uses wandb as a backend, and the importance of `metric` and `direciton` arguments vs. `compute_objective`.