Repository: transformers
Issue: Moving in a folder & `push_to_hub` for a `trust_remote_code=True` model
Labels: Feature request

### Feature request

Bonjour !

I'm opening an issue following a discussion with Lysandre on Slack.

My request is to be able to do `..` in model repositories on HF (where currently you can only do `.`). On this point, I don't know if this applies to all template directories or only to customs, which then require a `trust_remote_code=True` to load them.

A second request is that when you have a custom model (i.e. loadable via `trust_remote_code=True`) and once it's finetuned, that the `push_to_hub` function pushes all the files needed for the model to function properly, not just `config.json`, `configuration.py`, `model.safetensors`, `special_tokens_map.json`, `tokenizer.json`, `tokenizer_config.json` and `training_args.bin`.

### Motivation

The concrete case behind my requests.


We recently extended Flash Attention to the T5. 

So we had to develop a custom implementation and to load our pre-trained models for finetuning, we have to do :

```
from transformers import AutoModel
model = AutoModel.from_pretrained("CATIE-AQ/FAT5-base-UL2-fr", trust_remote_code=True)
```

For this to work, we need a [modeling file](https://huggingface.co/CATIE-AQ/FAT5-base-UL2-fr/blob/main/modeling_flash_t5.py).
 
In our code on GitHub (https://github.com/catie-aq/flashT5/blob/main/src/model/modeling_flash_t5.py), we call up classes that we've put in a `utils` folder and import them, for example (line [47](https://github.com/catie-aq/flashT5/blob/dcb45ec44b29ba9a7a04564a90a34a57fa65f490/src/model/modeling_flash_t5.py#L47)) a `..utils.positional_encoding import ALiBiPositionalEncoding, RelativePositionalEncoding, RotaryPositionalEncoding`.
On HF, this returned an error saying that there was a `..` in the `modeling_flash_t5.py` code and that it was therefore not possible to retrieve the classes. We therefore had to move all the code contained in the `utils` folder to the root.

![image](https://github.com/huggingface/transformers/assets/58078086/273787a2-f6c6-43d8-a211-587b683b3f43)

The line I used as an example above then becomes from `.positional_encoding import ALiBiPositionalEncoding, RelativePositionalEncoding, RotaryPositionalEncoding` and it works.

So being able to use classes contained in files would be appreciated 😄


The second request is related to the fact that, once this model has been finetuned, I do a `push_to_hub` to save the weights.
This pushes me the files [config.json](https://huggingface.co/bourdoiscatie/FAT5_small_cls/blob/main/config.json), [configuration_flash_t5.py](https://huggingface.co/bourdoiscatie/FAT5_small_cls/blob/main/configuration_flash_t5.py), [model.safetensors](https://huggingface.co/bourdoiscatie/FAT5_small_cls/blob/main/model.safetensors), [special_tokens_map.json](https://huggingface.co/bourdoiscatie/FAT5_small_cls/blob/main/special_tokens_map.json), [tokenizer.json](https://huggingface.co/bourdoiscatie/FAT5_small_cls/blob/main/tokenizer.json), [tokenizer_config.json](https://huggingface.co/bourdoiscatie/FAT5_small_cls/blob/main/tokenizer_config.json) and [training_args.bin](https://huggingface.co/bourdoiscatie/FAT5_small_cls/blob/main/training_args.bin).
And when I then want to reload the model to do inference, it tells me that the 8 files circled in red in the image above + the [modeling_flash_t5.py](https://huggingface.co/CATIE-AQ/FAT5-base-UL2-fr/blob/main/modeling_flash_t5.py) file are missing.

So every time I finetune, I have to do a second push where I add these 9 missing files so that my model can load properly.

Wouldn't it be possible for these files (which are detected during model loading) to be pushed directly with the 1st `push_to_hub`? 🤗

### Your contribution

Let me know if there's any way I can help.