'str' object has no attribute 'total_seconds' when trying to use 'train_time_interval' in 'checkpoint_callback_params'
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

Currently, it is not possible to use the 'train_time_interval' param from Pytorch lightning for checkpointing. When trying to specify it, it throws the following error.

```
  File "/home/abert/.local/lib/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 303, in on_train_batch_end
    skip_time = prev_time_check is None or (now - prev_time_check) < train_time_interval.total_seconds()
AttributeError: 'str' object has no attribute 'total_seconds'
```

And that's because Pytorch lighting 'train_time_interval' takes a timedelta object. Since we can't instantiate an object in the yaml and neither can we set it later (it throws an error saying it is not a primitive type (see below)) we can't use this feature.

```
Traceback (most recent call last):
  File "/mnt/c/Users/berta/Documents/Linagora/NeMo/examples/asr/speech_to_text_finetune.py", line 192, in main
    cfg.exp_manager.checkpoint_callback_params.train_time_interval = timedelta(seconds=30)
omegaconf.errors.UnsupportedValueType: Value 'timedelta' is not a supported primitive type
    full_key: exp_manager.checkpoint_callback_params.train_time_interval
    object_type=dict

```

 I made a quick and dirty fix on my side by making a timedelta object just before sending it to Pytorch lightning.

**Steps/Code to reproduce bug**

Running NeMo/examples/asr/speech_to_text_finetune.py with the following changes made to the yaml config file:

```
exp_manager:
  exp_dir: null
  name: ${name}
  create_tensorboard_logger: true
  create_checkpoint_callback: true
  checkpoint_callback_params:
    # in case of multiple validation sets, first one is used
    monitor: "train_loss"
    mode: "min"
    save_top_k: 5
    always_save_nemo: True # saves the checkpoints as nemo files along with PTL checkpoints
    train_time_interval: 60
    every_n_epochs: null
```

**Expected behavior**

We should be able to use this parameter, by specifying a number of seconds for example.


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.

```yaml
exp_manager:
    train_time_interval: 
      _target_: time.timedelta
      seconds: 60
```
You could pass an object throgh config as shown above. As hydra doesn;t support objects natively, current workaround is to use Any in type annotation. Added support for it here: https://github.com/NVIDIA/NeMo/pull/10559 

> exp_manager:
>     train_time_interval: 
>       _target_: time.timedelta
>       seconds: 60
> 
> You could pass an object throgh config as shown above. As hydra doesn;t support objects natively, current workaround is to use Any in type annotation. Added support for it here: [#10559](https://github.com/NVIDIA/NeMo/pull/10559)

Hi,
Thanks a lot for the response, I wasn't aware I could do that, thanks!

Hi, 
sorry for the very late comment. I tried your solution (see under) today and it doesn't work without changes (or maybe I'm doing something wrong?). 
```yaml
exp_manager:
    checkpoint_callback_params:
        train_time_interval: 
           _target_: datetime.timedelta
           seconds: 60
```
The value of "train_time_interval" is a dict because timedelta object is not instantiated. I tried adding "cfg = instantiate(cfg)" in exp_manager" function (after line 407: "cfg = OmegaConf.merge(schema, cfg)") and it does work, but I don't know if it impacts/breaks anything else. I can open a PR if there is a real problem.