Spaces:
Paused
Paused
<!--Copyright 2020 The HuggingFace Team. All rights reserved. | |
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with | |
the License. You may obtain a copy of the License at | |
http://www.apache.org/licenses/LICENSE-2.0 | |
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on | |
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the | |
specific language governing permissions and limitations under the License. | |
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be | |
rendered properly in your Markdown viewer. | |
--> | |
# Data Collator | |
Data collators are objects that will form a batch by using a list of dataset elements as input. These elements are of | |
the same type as the elements of `train_dataset` or `eval_dataset`. | |
To be able to build batches, data collators may apply some processing (like padding). Some of them (like | |
[`DataCollatorForLanguageModeling`]) also apply some random data augmentation (like random masking) | |
on the formed batch. | |
Examples of use can be found in the [example scripts](../examples) or [example notebooks](../notebooks). | |
## Default data collator | |
[[autodoc]] data.data_collator.default_data_collator | |
## DefaultDataCollator | |
[[autodoc]] data.data_collator.DefaultDataCollator | |
## DataCollatorWithPadding | |
[[autodoc]] data.data_collator.DataCollatorWithPadding | |
## DataCollatorForTokenClassification | |
[[autodoc]] data.data_collator.DataCollatorForTokenClassification | |
## DataCollatorForSeq2Seq | |
[[autodoc]] data.data_collator.DataCollatorForSeq2Seq | |
## DataCollatorForLanguageModeling | |
[[autodoc]] data.data_collator.DataCollatorForLanguageModeling | |
- numpy_mask_tokens | |
- tf_mask_tokens | |
- torch_mask_tokens | |
## DataCollatorForWholeWordMask | |
[[autodoc]] data.data_collator.DataCollatorForWholeWordMask | |
- numpy_mask_tokens | |
- tf_mask_tokens | |
- torch_mask_tokens | |
## DataCollatorForPermutationLanguageModeling | |
[[autodoc]] data.data_collator.DataCollatorForPermutationLanguageModeling | |
- numpy_mask_tokens | |
- tf_mask_tokens | |
- torch_mask_tokens | |