How to convert a ð€ Transformers model to TensorFlow?
ð€ Transformersã䜿çšããããã«è€æ°ã®ãã¬ãŒã ã¯ãŒã¯ãå©çšå¯èœã§ããããšã¯ãã¢ããªã±ãŒã·ã§ã³ãèšèšããéã«ããããã®åŒ·ã¿ã掻ããæè»æ§ãæäŸããŸããã äºææ§ãã¢ãã«ããšã«è¿œå ããå¿ èŠãããããšãæå³ããŸãããããã幞ããªããšã« æ¢åã®ã¢ãã«ã«TensorFlowäºææ§ãè¿œå ããããšã¯ããŒãããæ°ããã¢ãã«ãè¿œå ããããšãããç°¡åã§ãïŒ å€§èŠæš¡ãªTensorFlowã¢ãã«ã®è©³çŽ°ãç解ããããäž»èŠãªãªãŒãã³ãœãŒã¹ã®è²¢ç®ãè¡ã£ããã éžæããã¢ãã«ãTensorFlowã§æå¹ã«ããããã®ã¬ã€ãã§ãã
ãã®ã¬ã€ãã¯ãã³ãã¥ããã£ã®ã¡ã³ããŒã§ããããªãã«ãTensorFlowã¢ãã«ã®éã¿ããã³/ãŸã㯠ã¢ãŒããã¯ãã£ãð€ Transformersã§äœ¿çšããããã«ãHugging FaceããŒã ããã®æå°éã®ç£èŠã§è²¢ç®ã§ããåãäžããŸããæ°ããã¢ãã«ãæžãããšã¯å°ããªåæ¥ã§ã¯ãããŸãããã ãã®ã¬ã€ããèªãããšã§ããããããŒã©ãŒã³ãŒã¹ã¿ãŒã®ãããªãã®ããæ£æ©ã®ãããªãã®ã«ãªãããšãé¡ã£ãŠããŸãð¢ð¶ã ãã®ããã»ã¹ããŸããŸãç°¡åã«ããããã«ãç§ãã¡ã®å ±éã®çµéšã掻çšããããšã¯éåžžã«éèŠã§ãã®ã§ã ãã®ã¬ã€ãã®æ¹åãææ¡ããããšã匷ããå§ãããŸãïŒ
ããã«è©³ãã調ã¹ãåã«ã以äžã®ãªãœãŒã¹ããã§ãã¯ããããšããå§ãããŸããð€ TransformersãåããŠã®å ŽåïŒ
ãã®ã¬ã€ãã®æ®ãã®éšåã§ã¯ãæ°ããTensorFlowã¢ãã«ã¢ãŒããã¯ãã£ãè¿œå ããããã«å¿ èŠãªãã®ã PyTorchãTensorFlowã¢ãã«ã®éã¿ã«å€æããæé ãããã³MLãã¬ãŒã ã¯ãŒã¯éã®äžäžèŽãå¹ççã«ãããã°ããæ¹æ³ã«ã€ããŠåŠã³ãŸããããã§ã¯å§ããŸãããïŒ
䜿çšãããã¢ãã«ã«å¯Ÿå¿ããTensorFlowã¢ãŒããã¯ãã£ããã§ã«ååšãããã©ããããããªãã§ããïŒ
Â
éžæããã¢ãã«ã®config.json
ã®model_type
ãã£ãŒã«ãããã§ãã¯ããŠã¿ãŠãã ãã
ïŒäŸïŒã
ð€ Transformersã®è©²åœããã¢ãã«ãã©ã«ãã«ãååãâmodeling_tfâã§å§ãŸããã¡ã€ã«ãããå Žåãããã¯å¯Ÿå¿ããTensorFlow
ã¢ãŒããã¯ãã£ãæã£ãŠããããšãæå³ããŸãïŒäŸïŒã
Step-by-step guide to add TensorFlow model architecture code
倧èŠæš¡ãªã¢ãã«ã¢ãŒããã¯ãã£ãèšèšããæ¹æ³ã¯ããŸããŸã§ããããã®èšèšãå®è£ ããæ¹æ³ãããŸããŸã§ãã ããããð€ Transformersã®äžè¬çãªæŠèŠãã æãåºããŠããã ãããããããŸããããç§ãã¡ã¯æèŠã®ããã°ã«ãŒãã§ã - ð€ Transformersã®äœ¿ããããã¯äžè²«æ§ã®ããèšèšã®éžæè¢ã«äŸåããŠããŸããçµéšãããTensorFlowã¢ãã«ãè¿œå ããéã«éèŠãªããšãããã€ããäŒãã§ããŸãïŒ
- è»èŒªãåçºæããªãã§ãã ããïŒã»ãšãã©ã®å Žåã確èªãã¹ãå°ãªããšã2ã€ã®åç §å®è£ ããããŸããããã¯ã ããªããå®è£ ããŠããã¢ãã«ã®PyTorchããŒãžã§ã³ãšãåãçš®é¡ã®åé¡ã«å¯Ÿããä»ã®TensorFlowã¢ãã«ã§ãã
- åªããã¢ãã«å®è£ ã¯æéã®è©Šç·Žãä¹ãè¶ããŸããããã¯ãã³ãŒãããããã ããã§ã¯ãªããã³ãŒããæ確ã§ããããã°ããããã æ§ç¯ããããããã§ããTensorFlowå®è£ ã§PyTorchå®è£ ãšäžèŽãããã¿ãŒã³ãè€è£œããPyTorchå®è£ ãšã®äžäžèŽãæå°éã«æããããšã§ã ããªãã®è²¢ç®ãé·æéã«ããã£ãŠæçšã§ããããšãä¿èšŒããŸãã
- è¡ãè©°ãŸã£ããå©ããæ±ããŠãã ããïŒ ð€ TransformersããŒã ã¯ããã«ããŸãããããããããªããçŽé¢ããŠããåãåé¡ã«å¯Ÿãã解決çãèŠã€ããŠããŸãã
TensorFlowã¢ãã«ã¢ãŒããã¯ãã£ãè¿œå ããããã«å¿ èŠãªã¹ãããã®æŠèŠã¯æ¬¡ã®ãšããã§ãïŒ
- å€æãããã¢ãã«ãéžæ
- transformersã®éçºç°å¢ãæºå
- ïŒãªãã·ã§ã³ïŒçè«çãªåŽé¢ãšæ¢åã®å®è£ ãç解
- ã¢ãã«ã¢ãŒããã¯ãã£ãå®è£
- ã¢ãã«ã®ãã¹ããå®è£
- ãã«ãªã¯ãšã¹ããæåº
- ïŒãªãã·ã§ã³ïŒãã¢ãæ§ç¯ããŠäžçãšå ±æ
1.-3. Prepare your model contribution
1. å€æãããã¢ãã«ãéžæãã
ãŸããåºæ¬ããå§ããŸããããæåã«ç¥ã£ãŠããå¿ èŠãããããšã¯ãå€æãããã¢ãŒããã¯ãã£ã§ãã ç¹å®ã®ã¢ãŒããã¯ãã£ã決ããŠããªãå Žåãð€ Transformers ããŒã ã«ææ¡ãæ±ããããšã¯ã圱é¿ãæ倧éã«ããçŽ æŽãããæ¹æ³ã§ãã ããŒã ã¯ãTensorFlow ãµã€ãã§äžè¶³ããŠããæã泚ç®ãããã¢ãŒããã¯ãã£ã«åããŠã¬ã€ãããŸãã TensorFlow ã§äœ¿çšãããç¹å®ã®ã¢ãã«ã«ãð€ Transformers ã«æ¢ã« TensorFlow ã¢ãŒããã¯ãã£ã®å®è£ ãååšããŠããããéã¿ãäžè¶³ããŠããå Žåã ãã®ããŒãžã®éã¿ã®è¿œå ã»ã¯ã·ã§ã³ã«çŽæ¥ç§»åããŠãã ããã
ç°¡åã«ããããã«ããã®ã¬ã€ãã®æ®ãã®éšåã§ã¯ãTensorFlow ããŒãžã§ã³ã® BrandNewBert ãè²¢ç®ããããšã決å®ãããšä»®å®ããŠããŸã ïŒããã¯ãæ°ããã¢ãã«ã®è¿œå ã¬ã€ãã§ã®äŸãšåãã§ãïŒã
TensorFlow ã¢ãã«ã®ã¢ãŒããã¯ãã£ã«åãçµãåã«ããããè¡ãããã®é²è¡äžã®åãçµã¿ããªãããå確èªããŠãã ããã
GitHub ããŒãžã®ãã«ãªã¯ãšã¹ã㧠BrandNewBert
ãæ€çŽ¢ããŠã
TensorFlow é¢é£ã®ãã«ãªã¯ãšã¹ãããªãããšã確èªã§ããŸãã
2. transformers éçºç°å¢ã®æºå
ã¢ãã«ã¢ãŒããã¯ãã£ãéžæããããæåã瀺ãããã«ãã©ãã PR ãéãããã®ç°å¢ãèšå®ããŠãã ããã 以äžã®æé ã«åŸã£ãŠãç°å¢ãèšå®ãããã©ãã PR ãéããŠãã ããã
ãªããžããªã®ããŒãžã§ âForkâ ãã¿ã³ãã¯ãªãã¯ããŠããªããžããªããã©ãŒã¯ããŸãã ããã«ãããã³ãŒãã®ã³ããŒã GitHub ãŠãŒã¶ãŒã¢ã«ãŠã³ãã®äžã«äœæãããŸãã
ããŒã«ã«ãã£ã¹ã¯ã«ãã âtransformersâ ãã©ãŒã¯ãã¯ããŒã³ããããŒã¹ãªããžããªããªã¢ãŒããšããŠè¿œå ããŸã:
git clone https://github.com/[your Github handle]/transformers.git
cd transformers
git remote add upstream https://github.com/huggingface/transformers.git
- éçºç°å¢ãèšå®ããŸããããšãã°ã以äžã®ã³ãã³ããå®è¡ããŠãã ããïŒ
git clone https://github.com/[your Github handle]/transformers.git
cd transformers
git remote add upstream https://github.com/huggingface/transformers.git
äŸåé¢ä¿ãå¢ããŠãããããOSã«å¿ããŠãTransformersã®ãªãã·ã§ã³ã®äŸåé¢ä¿ã®æ°ãå¢ãããããããŸããããã®å Žåã¯ãTensorFlowãã€ã³ã¹ããŒã«ããŠãã次ã®ã³ãã³ããå®è¡ããŠãã ããã
pip install -e ".[quality]"
泚æ: CUDAãã€ã³ã¹ããŒã«ããå¿ èŠã¯ãããŸãããæ°ããã¢ãã«ãCPUã§åäœãããããšãååã§ãã
- ã¡ã€ã³ãã©ã³ãããããããããååã®ãã©ã³ããäœæããŠãã ããã
git checkout -b add_tf_brand_new_bert
- çŸåšã®mainãã©ã³ãã«ãã§ããããŠãªããŒã¹ãã
git fetch upstream git rebase upstream/main
transformers/src/models/brandnewbert/
ã«modeling_tf_brandnewbert.py
ãšããååã®ç©ºã®.py
ãã¡ã€ã«ãè¿œå ããŸããããã¯ããªãã®TensorFlowã¢ãã«ãã¡ã€ã«ã§ãã以äžã䜿çšããŠå€æŽå 容ãã¢ã«ãŠã³ãã«ããã·ã¥ããŸãïŒ
git add .
git commit -m "initial commit"
git push -u origin add_tf_brand_new_bert
GitHubäžã§ãã©ãŒã¯ãããŠã§ãããŒãžã«ç§»åããããã«ãªã¯ãšã¹ãããã¯ãªãã¯ããŸããå°æ¥ã®å€æŽã«åããŠãHugging Face ããŒã ã®ã¡ã³ããŒã®GitHubãã³ãã«ãã¬ãã¥ã¢ãŒãšããŠè¿œå ããŠãã ããã
GitHubã®ãã«ãªã¯ãšã¹ããŠã§ãããŒãžã®å³åŽã«ããããã©ããã«å€æããã¯ãªãã¯ããŠããã«ãªã¯ãšã¹ãããã©ããã«å€æŽããŸãã
ããã§ãð€ Transformerså ã«BrandNewBertãTensorFlowã«ç§»æ€ããããã®éçºç°å¢ãèšå®ãããŸããã
3. (ä»»æ) çè«çãªåŽé¢ãšæ¢åã®å®è£ ãç解ãã
BrandNewBertã®è«æãååšããå Žåããã®èšè¿°çãªäœæ¥ãèªãæéãåãã¹ãã§ããè«æã«ã¯ç解ãé£ãã倧ããªã»ã¯ã·ã§ã³ããããããããŸããããã®å Žåã§ãåé¡ãããŸãã - å¿é ããªãã§ãã ããïŒç®æšã¯è«æã®çè«çãªç解ãæ·±ããããšã§ã¯ãªããð€ Transformersã䜿çšããŠTensorFlowã§ã¢ãã«ãå¹æçã«åå®è£ ããããã«å¿ èŠãªæ å ±ãæœåºããããšã§ãããšã¯èšããçè«çãªåŽé¢ã«ããŸãæéããããå¿ èŠã¯ãããŸããã代ããã«ãæ¢åã®ã¢ãã«ã®ããã¥ã¡ã³ããŒã·ã§ã³ããŒãžïŒããšãã°ãBERTã®ã¢ãã«ããã¥ã¡ã³ããªã©ïŒã«çŠç¹ãåœãŠãã¹ãã§ãã
å®è£ ããã¢ãã«ã®åºæ¬ãææ¡ããåŸãæ¢åã®å®è£ ãç解ããããšã¯éèŠã§ããããã¯ãåäœããå®è£ ãã¢ãã«ã«å¯ŸããæåŸ ãšäžèŽããããšã確èªãã絶奜ã®æ©äŒã§ãããTensorFlowåŽã§ã®æè¡çãªèª²é¡ãäºæž¬ããããšãã§ããŸãã
æ å ±ã®å€ãã«å§åãããŠãããšæããã®ã¯å®å šã«èªç¶ã§ãããã®æ®µéã§ã¯ã¢ãã«ã®ãã¹ãŠã®åŽé¢ãç解ããå¿ èŠã¯ãããŸããããã ãããã©ãŒã©ã ã§æ¥ãªè³ªåã解決ããããšã匷ããå§ãããŸãã
4. Model implementation
ãããããããã³ãŒãã£ã³ã°ãå§ããŸãããããå§ãããåºçºç¹ã¯ãPyTorchãã¡ã€ã«ãã®ãã®ã§ãã
src/transformers/models/brand_new_bert/
å
ã®modeling_brand_new_bert.py
ã®å
容ã
modeling_tf_brand_new_bert.py
ã«ã³ããŒããŸãããã®ã»ã¯ã·ã§ã³ã®ç®æšã¯ã
ð€ Transformersã®ã€ã³ããŒãæ§é ãæŽæ°ããTFBrandNewBert
ãš
TFBrandNewBert.from_pretrained(model_repo, from_pt=True)
ãæ£åžžã«èªã¿èŸŒãåäœããTensorFlow BrandNewBertã¢ãã«ã
ã€ã³ããŒãã§ããããã«ããããšã§ãã
æ®å¿µãªãããPyTorchã¢ãã«ãTensorFlowã«å€æããæ確ãªæ¹æ³ã¯ãããŸããããã ããããã»ã¹ãã§ããã ãã¹ã ãŒãºã«ããããã®ãã³ãã以äžã«ç€ºããŸãïŒ
- ãã¹ãŠã®ã¯ã©ã¹ã®ååã®åã«
TF
ãä»ããŸãïŒäŸïŒBrandNewBert
ã¯TFBrandNewBert
ã«ãªããŸãïŒã - ã»ãšãã©ã®PyTorchã®æäœã«ã¯ãçŽæ¥TensorFlowã®ä»£æ¿ããããŸããããšãã°ã
torch.nn.Linear
ã¯tf.keras.layers.Dense
ã«å¯Ÿå¿ããtorch.nn.Dropout
ã¯tf.keras.layers.Dropout
ã«å¯Ÿå¿ããŸããç¹å®ã®æäœã«ã€ããŠäžæ確ãªå Žåã¯ãTensorFlowã®ããã¥ã¡ã³ããŸãã¯PyTorchã®ããã¥ã¡ã³ããåç §ã§ããŸãã - ð€ Transformersã®ã³ãŒãããŒã¹ã«ãã¿ãŒã³ãèŠã€ãããŸããç¹å®ã®æäœã«çŽæ¥çãªä»£æ¿ããªãå Žåã誰ãããã§ã«åãåé¡ã«å¯ŸåŠããŠããå¯èœæ§ãé«ãã§ãã
- ããã©ã«ãã§ã¯ãPyTorchãšåãå€æ°åãšæ§é ãç¶æããŸããããã«ããããããã°ãåé¡ã®è¿œè·¡ãä¿®æ£ã®è¿œå ã容æã«ãªããŸãã
- äžéšã®ã¬ã€ã€ãŒã«ã¯ãåãã¬ãŒã ã¯ãŒã¯ã§ç°ãªãããã©ã«ãå€ããããŸãã泚ç®ãã¹ãäŸã¯ããããæ£èŠåã¬ã€ã€ãŒã® epsilon ã§ãïŒPyTorchã§ã¯
1e-5
ãTensorFlowã§ã¯1e-3
ã§ãïŒãããã¥ã¡ã³ããå確èªããŠãã ããïŒ - PyTorchã®
nn.Parameter
å€æ°ã¯éåžžãTF Layerã®build()
å ã§åæåããå¿ èŠããããŸãã次ã®äŸãåç §ããŠãã ããïŒPyTorch / TensorFlow - PyTorchã¢ãã«ã«é¢æ°ã®äžéšã«
#copied from ...
ãããå ŽåãTensorFlowã¢ãã«ãåãã¢ãŒããã¯ãã£ãããã®é¢æ°ãåããããšãã§ããå¯èœæ§ãé«ãã§ããTensorFlowã¢ãŒããã¯ãã£ãããå Žåã§ãã - TensorFlowé¢æ°å
ã§
name
å±æ§ãæ£ããèšå®ããããšã¯ãfrom_pt=True
ã®ãŠã§ã€ãã®ã¯ãã¹ããŒãããŒããè¡ãããã«éèŠã§ããéåžžãname
ã¯PyTorchã³ãŒãå ã®å¯Ÿå¿ããå€æ°ã®ååã§ããname
ãæ£ããèšå®ãããŠããªãå Žåãã¢ãã«ãŠã§ã€ãã®ããŒãæã«ãšã©ãŒã¡ãã»ãŒãžã§è¡šç€ºãããŸãã - ããŒã¹ã¢ãã«ã¯ã©ã¹
BrandNewBertModel
ã®ããžãã¯ã¯å®éã«ã¯TFBrandNewBertMainLayer
ã«ãããŸããããã¯Kerasã¬ã€ã€ãŒã®ãµãã¯ã©ã¹ã§ãïŒäŸïŒãTFBrandNewBertModel
ã¯ãåã«ãã®ã¬ã€ã€ãŒã®ã©ãããŒã§ãã - ã¢ãã«ãèªã¿èŸŒãããã«ã¯ãKerasã¢ãã«ããã«ãããå¿
èŠããããŸãããã®ããã
TFBrandNewBertPreTrainedModel
ã¯ã¢ãã«ãžã®å ¥åã®äŸãdummy_inputs
ãæã€å¿ èŠããããŸãïŒäŸïŒã - 衚瀺ãæ¢ãŸã£ãå Žåã¯ãå©ããæ±ããŠãã ãããç§ãã¡ã¯ããªãã®ãæäŒãã«ããã«ããŸãïŒ ð€
ã¢ãã«ãã¡ã€ã«èªäœã ãã§ãªããã¢ãã«ã¯ã©ã¹ãšé¢é£ããããã¥ã¡ã³ããŒã·ã§ã³ããŒãžãžã®ãã€ã³ã¿ãŒãè¿œå ããå¿ èŠããããŸããä»ã®PRã®ãã¿ãŒã³ã«åŸã£ãŠãã®éšåãå®äºã§ããŸã ïŒäŸïŒã 以äžã¯æåã§ã®å€æŽãå¿ èŠãªäžèŠ§ã§ãïŒ
- BrandNewBertã®ãã¹ãŠã®ãããªãã¯ã¯ã©ã¹ã
src/transformers/__init__.py
ã«å«ãã - BrandNewBertã¯ã©ã¹ã
src/transformers/models/auto/modeling_tf_auto.py
ã®å¯Ÿå¿ããAutoã¯ã©ã¹ã«è¿œå - ããã¥ã¡ã³ããŒã·ã§ã³ãã¹ããã¡ã€ã«ã®ãªã¹ãã«ã¢ããªã³ã°ãã¡ã€ã«ãè¿œå ãã
utils/documentation_tests.txt
src/transformers/utils/dummy_tf_objects.py
ã«é¢é£ãã BrandNewBert ã«é¢é£ããé 延ããŒãã¯ã©ã¹ãè¿œåsrc/transformers/models/brand_new_bert/__init__.py
ã§ãããªãã¯ã¯ã©ã¹ã®ã€ã³ããŒãæ§é ãæŽæ°docs/source/en/model_doc/brand_new_bert.md
ã« BrandNewBert ã®ãããªãã¯ã¡ãœããã®ããã¥ã¡ã³ããŒã·ã§ã³ãã€ã³ã¿ãŒãè¿œådocs/source/en/model_doc/brand_new_bert.md
ã® BrandNewBert ã®è²¢ç®è ãªã¹ãã«èªåèªèº«ãè¿œå- æåŸã«ã
docs/source/en/index.md
ã® BrandNewBert ã®TensorFlowåã«ç·è²ã®ãã§ãã¯ããŒã¯ â ãè¿œå
ã¢ãã«ã¢ãŒããã¯ãã£ãæºåã§ããŠããããšã確èªããããã«ã以äžã®ãã§ãã¯ãªã¹ããå®è¡ããŠãã ããïŒ
- èšç·Žæã«ç°ãªãåäœããããã¹ãŠã®ã¬ã€ã€ãŒïŒäŸïŒDropoutïŒã¯ã
training
åŒæ°ã䜿çšããŠåŒã³åºããããããæäžäœã¯ã©ã¹ããäŒæãããŸãã - å¯èœãªéã
#copied from ...
ã䜿çšããŸãã TFBrandNewBertMainLayer
ããã³ããã䜿çšãããã¹ãŠã®ã¯ã©ã¹ã®call
é¢æ°ã@unpack_inputs
ã§ãã³ã¬ãŒããããŠããŸãTFBrandNewBertMainLayer
ã¯@keras_serializable
ã§ãã³ã¬ãŒããããŠããŸã- PyTorchãŠã§ã€ãããTensorFlowãŠã§ã€ãã䜿çšããŠTensorFlowã¢ãã«ãããŒãã§ããŸã
TFBrandNewBert.from_pretrained(model_repo, from_pt=True)
- äºæãããå ¥å圢åŒã䜿çšããŠTensorFlowã¢ãã«ãåŒã³åºãããšãã§ããŸã
5. Add model tests
ãã£ãããTensorFlowã¢ãã«ãå®è£
ããŸããïŒ
ä»åºŠã¯ãã¢ãã«ãæåŸ
éãã«åäœããããšã確èªããããã®ãã¹ããè¿œå ããæéã§ãã
åã®ã»ã¯ã·ã§ã³ãšåæ§ã«ãtests/models/brand_new_bert/
ãã£ã¬ã¯ããªå
ã®test_modeling_brand_new_bert.py
ãã¡ã€ã«ãtest_modeling_tf_brand_new_bert.py
ã«ã³ããŒããå¿
èŠãªTensorFlowã®çœ®æãè¡ãããšããå§ãããŸãã
ä»ã®æ®µéã§ã¯ããã¹ãŠã®.from_pretrained()
åŒã³åºãã§ãæ¢åã®PyTorchã®éã¿ãããŒãããããã«from_pt=True
ãã©ã°ã䜿çšããå¿
èŠããããŸãã
äœæ¥ãå®äºãããããã¹ããå®è¡ããæºåãæŽããŸããïŒ ð¬
NVIDIA_TF32_OVERRIDE=0 RUN_SLOW=1 RUN_PT_TF_CROSS_TESTS=1 \ py.test -vv tests/models/brand_new_bert/test_modeling_tf_brand_new_bert.py
æãå¯èœæ§ã®é«ãçµæã¯ãå€ãã®ãšã©ãŒã衚瀺ãããããšã§ããå¿é
ããªãã§ãã ãããããã¯äºæ³ãããåäœã§ãïŒ
MLã¢ãã«ã®ãããã°ã¯éåžžã«é£ãããšãããŠãããæåã®éµã¯å¿èåïŒãšbreakpoint()
ïŒã§ããç§ãã¡ã®çµéšã§ã¯ã
æãé£ããåé¡ã¯MLãã¬ãŒã ã¯ãŒã¯éã®åŸ®åŠãªäžäžèŽããçºçããããã«ã€ããŠã¯ãã®ã¬ã€ãã®æåŸã«ããã€ãã®ãã€ã³ã¿ã瀺ããŸãã
ä»ã®å Žåã§ã¯ãäžè¬çãªãã¹ããçŽæ¥ã¢ãã«ã«é©çšã§ããªãå Žåãããããã®å Žåã¯ã¢ãã«ã®ãã¹ãã¯ã©ã¹ã¬ãã«ã§ãªãŒããŒã©ã€ããææ¡ããŸãã
åé¡ã®çš®é¡ã«é¢ä¿ãªããè©°ãŸã£ãå Žåã¯ããã©ããã®ãã«ãªã¯ãšã¹ãã§å©ããæ±ããããšããããããªãã§ãã ããã
ãã¹ãŠã®ãã¹ãããã¹ããããããã§ãšãããããŸããããªãã®ã¢ãã«ã¯ã»ãŒð€ Transformersã©ã€ãã©ãªã«è¿œå ããæºåãæŽããŸããïŒð
6. ãã«ãªã¯ãšã¹ããæåºãã
å®è£
ãšãã¹ããå®äºãããããã«ãªã¯ãšã¹ããæåºããæºåãæŽããŸãããã³ãŒããããã·ã¥ããåã«ã
ã³ãŒããã©ãŒããããŠãŒãã£ãªãã£ã§ãã make fixup
ðª ãå®è¡ããŠãã ããã
ããã«ãããèªåçãªãã§ãã¯ã«å€±æããå¯èœæ§ã®ãããã©ãŒãããã®åé¡ãèªåçã«ä¿®æ£ãããŸãã
ããã§ããã©ãããã«ãªã¯ãšã¹ããå®éã®ãã«ãªã¯ãšã¹ãã«å€æããæºåãæŽããŸããã
ãããè¡ãã«ã¯ããã¬ãã¥ãŒåŸ
ã¡ããã¿ã³ãã¯ãªãã¯ããJoaoïŒ@gante
ïŒãšMattïŒ@Rocketknight1
ïŒãã¬ãã¥ã¯ãŒãšããŠè¿œå ããŸãã
ã¢ãã«ãã«ãªã¯ãšã¹ãã«ã¯å°ãªããšã3人ã®ã¬ãã¥ã¯ãŒãå¿
èŠã§ãããã¢ãã«ã«é©åãªè¿œå ã®ã¬ãã¥ã¯ãŒãèŠã€ããã®ã¯åœŒãã®è²¬ä»»ã§ãã
ãã¹ãŠã®ã¬ãã¥ã¯ãŒããã«ãªã¯ãšã¹ãã®ç¶æ
ã«æºè¶³ããããæåŸã®ã¢ã¯ã·ã§ã³ãã€ã³ãã¯ã.from_pretrained()
åŒã³åºã㧠from_pt=True
ãã©ã°ãåé€ããããšã§ãã
TensorFlowã®ãŠã§ã€ããååšããªããããããããè¿œå ããå¿
èŠããããŸãïŒãããè¡ãæ¹æ³ã«ã€ããŠã¯ã以äžã®ã»ã¯ã·ã§ã³ã確èªããŠãã ããã
æåŸã«ãTensorFlowã®ãŠã§ã€ããããŒãžãããå°ãªããšã3人ã®ã¬ãã¥ãŒã¢ãæ¿èªãããã¹ãŠã®CIãã§ãã¯ã æåããå Žåããã¹ããããŒã«ã«ã§æåŸã«ããäžåºŠç¢ºèªããŠãã ããã
NVIDIA_TF32_OVERRIDE=0 RUN_SLOW=1 RUN_PT_TF_CROSS_TESTS=1 \ py.test -vv tests/models/brand_new_bert/test_modeling_tf_brand_new_bert.py
ãããŠãããªãã®PRãããŒãžããŸãïŒãã€ã«ã¹ããŒã³éæããã§ãšãããããŸã ð
7. (Optional) ãã¢ãäœæããŠäžçãšå ±æ
ãªãŒãã³ãœãŒã¹ã®æãé£ããéšåã®1ã€ã¯ãçºèŠã§ããããªãã®çŽ æŽãããTensorFlowã®è²¢ç®ãååšããããšãä»ã®ãŠãŒã¶ãŒãã©ã®ããã«ç¥ãããšãã§ããã§ããããïŒé©åãªã³ãã¥ãã±ãŒã·ã§ã³ã§ãïŒ ð£
ã³ãã¥ããã£ãšã¢ãã«ãå ±æããäž»èŠãªæ¹æ³ã¯2ã€ãããŸãã
- ãã¢ãäœæããŸããããã«ã¯Gradioãã¢ãããŒãããã¯ãããã³ã¢ãã«ã玹ä»ããããã®ä»ã®æ¥œããæ¹æ³ãå«ãŸããŸããã³ãã¥ããã£é§åã®ãã¢ã«ããŒãããã¯ãè¿œå ããããšã匷ããå§ãããŸãã
- TwitterãLinkedInãªã©ã®ãœãŒã·ã£ã«ã¡ãã£ã¢ã§ã¹ããŒãªãŒãå ±æããŸããããªãã®ä»äºã«èªããæã¡ãã³ãã¥ããã£ãšããªãã®ææãå ±æããã¹ãã§ã - ããªãã®ã¢ãã«ã¯ä»ãäžçäžã®äœå人ãã®ãšã³ãžãã¢ãç 究è ã«ãã£ãŠäœ¿çšãããå¯èœæ§ããããŸã ðïŒç§ãã¡ã¯ããªãã®æçš¿ããªãã€ãŒãããŠå ±åäœãšå ±æãããæäŒããåãã§ããŸãã
Adding TensorFlow weights to ð€ Hub
TensorFlowã¢ãã«ã®ã¢ãŒããã¯ãã£ãð€ Transformersã§å©çšå¯èœãªå ŽåãPyTorchã®éã¿ãTensorFlowã®éã¿ã«å€æããããšã¯ç°¡åã§ãïŒ
以äžããã®æ¹æ³ã§ãïŒ
- ã¿ãŒããã«ã§Hugging Faceã¢ã«ãŠã³ãã«ãã°ã€ã³ããŠããããšã確èªããŠãã ãããã³ãã³ã
huggingface-cli login
ã䜿çšããŠãã°ã€ã³ã§ããŸãïŒã¢ã¯ã»ã¹ããŒã¯ã³ã¯ãã¡ãã§èŠã€ããããšãã§ããŸãïŒã transformers-cli pt-to-tf --model-name foo/bar
ãšããã³ãã³ããå®è¡ããŸããããã§ãfoo/bar
ã¯å€æãããPyTorchã®éã¿ãå«ãã¢ãã«ãªããžããªã®ååã§ãã- äžèšã®ã³ãã³ãã§äœæãããð€ Hub PRã«
@joaogante
ãš@Rocketknight1
ãã¿ã°ä»ãããŸãã
ããã ãã§ãïŒ ð
Debugging mismatches across ML frameworks ð
æ°ããã¢ãŒããã¯ãã£ãè¿œå ããããæ¢åã®ã¢ãŒããã¯ãã£ã®TensorFlowã®éã¿ãäœæãããããéãPyTorchãšTensorFlowéã®äžäžèŽã«ã€ããŠã®ãšã©ãŒã«ééããããšããããŸãã å Žåã«ãã£ãŠã¯ãPyTorchãšTensorFlowã®ã¢ãã«ã¢ãŒããã¯ãã£ãã»ãŒåäžã§ããã«ãããããããäžäžèŽãææãããšã©ãŒã衚瀺ãããããšããããŸãã ã©ãããŠã§ããããïŒ ð€
ãŸãæåã«ããªããããã®äžäžèŽãç解ããããšãéèŠãã«ã€ããŠè©±ããŸããããå€ãã®ã³ãã¥ããã£ã¡ã³ããŒã¯ð€ Transformersã¢ãã«ããã®ãŸãŸäœ¿çšããã¢ãã«ãæåŸ
ã©ããã«åäœãããšä¿¡é ŒããŠããŸãã
2ã€ã®ãã¬ãŒã ã¯ãŒã¯éã§å€§ããªäžäžèŽããããšãå°ãªããšã1ã€ã®ãã¬ãŒã ã¯ãŒã¯ã®ãªãã¡ã¬ã³ã¹å®è£
ã«åŸã£ãŠã¢ãã«ãåäœããªãããšãæå³ããŸãã
ããã«ãããã¢ãã«ã¯å®è¡ãããŸããæ§èœãäœäžããå¯èœæ§ããããéããªå€±æãçºçããå¯èœæ§ããããŸããããã¯ãå
šãå®è¡ãããªãã¢ãã«ãããæªããšèšãããããããŸããïŒãã®ãããã¢ãã«ã®ãã¹ãŠã®æ®µéã§ã®ãã¬ãŒã ã¯ãŒã¯ã®äžäžèŽã1e-5
æªæºã§ããããšãç®æããŠããŸãã
æ°å€èšç®ã®åé¡ãšåæ§ã«ã詳现ã«ã€ããŠã¯çŽ°ãããšããã«ãããŸãããããŠã詳现æåã®æè¡ã§ãã以äžãç§å¯ã®èŠçŽ ã¯å¿èã§ãã ãã®çš®ã®åé¡ã«ééããå Žåã®ãå§ãã®ã¯ãŒã¯ãããŒã¯æ¬¡ã®ãšããã§ãïŒ
- äžäžèŽã®åå ãç¹å®ããŸããå€æäžã®ã¢ãã«ã«ã¯ããããç¹å®ã®ç¹ãŸã§ã»ãŒåäžã®å
éšå€æ°ããããŸãã
äž¡æ¹ã®ãã¬ãŒã ã¯ãŒã¯ã®ã¢ãŒããã¯ãã£ã«
breakpoint()
ã¹ããŒãã¡ã³ããé 眮ãããããããŠã³ã®æ¹æ³ã§æ°å€å€æ°ã®å€ãæ¯èŒããåé¡ã®åå ãèŠã€ããŸãã - åé¡ã®åå ãç¹å®ããããð€ TransformersããŒã ãšé£çµ¡ãåããŸããããåæ§ã®åé¡ã«ééããããšãããããããããè¿ éã«è§£æ±ºçãæäŸã§ãããããããŸãããæçµæ段ãšããŠãStackOverflowãGitHubã®åé¡ãªã©ã人æ°ã®ããããŒãžãã¹ãã£ã³ããŸãã
- 解決çãèŠåœãããªãå Žåãåé¡ãæãäžããå¿ èŠãããããšãæå³ããŸããè¯ããã¥ãŒã¹ã¯ãåé¡ã®åå ãç¹å®ããããšã§ãããããã£ãŠãåé¡ã®ããåœä»€ã«çŠç¹ãåœãŠãã¢ãã«ã®æ®ããæœè±¡åã§ããŸãïŒæªããã¥ãŒã¹ã¯ããã®åœä»€ã®ãœãŒã¹å®è£ ã«é²ãå¿ èŠãããããšã§ããäžéšã®å Žåã§ã¯ããªãã¡ã¬ã³ã¹å®è£ ã«åé¡ããããããããŸãã - äžæµãªããžããªã§åé¡ãéãã®ãæ§ããªãã§ãã ããã
ð€ TransformersããŒã ãšã®è©±ãåãã§ãäžäžèŽãä¿®æ£ããããšãå°é£ã§ããããšãå€æããããšããããŸãã
åºåã¬ã€ã€ãŒã®ã¢ãã«ã§äžäžèŽãéåžžã«å°ããå ŽåïŒãã ããé ããç¶æ
ã§ã¯å€§ããå¯èœæ§ãããïŒãã¢ãã«ãé
åžããããã«ãããç¡èŠããããšã«ãããããããŸããã
äžèšã§èšåããpt-to-tf
CLIã«ã¯ãéã¿å€ææã«ãšã©ãŒã¡ãã»ãŒãžãç¡èŠããããã®--max-error
ãã©ã°ããããŸãã