Transformers documentation

CPM

Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v4.46.3).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

CPM

Overview

CPM モデルは、Zhengyan Zhang、Xu Han、Hao Zhou、Pei Ke、Yuxian Gu によって CPM: A Large-scale Generative Chinese Pre-trained Language Model で提案されました。葉徳明、秦裕佳、 Yusheng Su、Haozhe Ji、Jian Guan、Fanchao Qi、Xiaozi Wang、Yanan Zheng、Guoyang Zeng、Huanqi Cao、Shengqi Chen、 Daixuan Li、Zhenbo Sun、Zhiyuan Liu、Minlie Huang、Wentao Han、Jie Tang、Juanzi Li、Xiaoyan Zhu、Maosong Sun。

論文の要約は次のとおりです。

事前トレーニングされた言語モデル (PLM) は、さまざまな下流の NLP タスクに有益であることが証明されています。最近ではGPT-3、 1,750億個のパラメータと570GBの学習データを備え、数回の撮影（1枚でも）の容量で大きな注目を集めましたゼロショット）学習。ただし、GPT-3 を適用して中国語の NLP タスクに対処することは依然として困難です。 GPT-3 の言語は主に英語であり、パラメーターは公開されていません。この技術レポートでは、大規模な中国語トレーニングデータに対する生成的事前トレーニングを備えた中国語事前トレーニング済み言語モデル (CPM)。最高に私たちの知識の限りでは、26 億のパラメータと 100GB の中国語トレーニングデータを備えた CPM は、事前トレーニングされた中国語としては最大のものです。言語モデルは、会話、エッセイの作成、クローゼテストと言語理解。広範な実験により、CPM が多くの環境で優れたパフォーマンスを達成できることが実証されています。少数ショット (ゼロショットでも) 学習の設定での NLP タスク。

このモデルは canwenxu によって提供されました。オリジナルの実装が見つかりますここ: https://github.com/TsinghuaAI/CPM-Generate

CPM のアーキテクチャは、トークン化方法を除いて GPT-2 と同じです。詳細については、GPT-2 ドキュメントを参照してください。 API リファレンス情報。

CpmTokenizer

class transformers.CpmTokenizer

< source >

( vocab_file do_lower_case = False remove_space = True keep_accents = False bos_token = '<s>' eos_token = '</s>' unk_token = '<unk>' sep_token = '<sep>' pad_token = '<pad>' cls_token = '<cls>' mask_token = '<mask>' additional_special_tokens = ['<eop>', '<eod>'] sp_model_kwargs: typing.Optional[typing.Dict[str, typing.Any]] = None **kwargs )

Runs pre-tokenization with Jieba segmentation tool. It is used in CPM models.

build_inputs_with_special_tokens

< source >

( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None ) → List[int]

Parameters

token_ids_0 (List[int]) — List of IDs to which the special tokens will be added.
token_ids_1 (List[int], optional) — Optional second list of IDs for sequence pairs.

Returns

List[int]

List of input IDs with the appropriate special tokens.

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An XLNet sequence has the following format:

single sequence: X <sep> <cls>
pair of sequences: A <sep> B <sep> <cls>

convert_tokens_to_string

< source >

( tokens )

Converts a sequence of tokens (strings for sub-words) in a single string.

create_token_type_ids_from_sequences

< source >

( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None ) → List[int]

Parameters

token_ids_0 (List[int]) — List of IDs.
token_ids_1 (List[int], optional) — Optional second list of IDs for sequence pairs.

Returns

List[int]

List of token type IDs according to the given sequence(s).

Create a mask from the two sequences passed to be used in a sequence-pair classification task. An XLNet

sequence pair mask has the following format:

0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence    | second sequence |

If token_ids_1 is None, this method only returns the first portion of the mask (0s).

get_special_tokens_mask

< source >

( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None already_has_special_tokens: bool = False ) → List[int]

Parameters

token_ids_0 (List[int]) — List of IDs.
token_ids_1 (List[int], optional) — Optional second list of IDs for sequence pairs.
already_has_special_tokens (bool, optional, defaults to False) — Whether or not the token list is already formatted with special tokens for the model.

Returns

List[int]

A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer prepare_for_model method.

CpmTokenizerFast

class transformers.CpmTokenizerFast

< source >

( vocab_file = None tokenizer_file = None do_lower_case = False remove_space = True keep_accents = False bos_token = '<s>' eos_token = '</s>' unk_token = '<unk>' sep_token = '<sep>' pad_token = '<pad>' cls_token = '<cls>' mask_token = '<mask>' additional_special_tokens = ['<eop>', '<eod>'] **kwargs )

Runs pre-tokenization with Jieba segmentation tool. It is used in CPM models.

build_inputs_with_special_tokens

< source >

( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None ) → List[int]

Parameters

token_ids_0 (List[int]) — List of IDs to which the special tokens will be added.
token_ids_1 (List[int], optional) — Optional second list of IDs for sequence pairs.

Returns

List[int]

List of input IDs with the appropriate special tokens.

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An XLNet sequence has the following format:

single sequence: X <sep> <cls>
pair of sequences: A <sep> B <sep> <cls>

create_token_type_ids_from_sequences

< source >

( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None ) → List[int]

Parameters

token_ids_0 (List[int]) — List of IDs.
token_ids_1 (List[int], optional) — Optional second list of IDs for sequence pairs.

Returns

List[int]

List of token type IDs according to the given sequence(s).

Create a mask from the two sequences passed to be used in a sequence-pair classification task. An XLNet

sequence pair mask has the following format:

0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence    | second sequence |

If token_ids_1 is None, this method only returns the first portion of the mask (0s).

< > Update on GitHub

←ConvBERT CPMANT→