Spaces:

Anna759
/

hklaw

Runtime error

App Files Files Community

Anna759 commited on Nov 4, 2023

Commit

255ca0a

1 Parent(s): 086b266

Upload 15 files

Browse files

Files changed (15) hide show

LICENSE +201 -0
README.md +699 -12
chat.py +441 -0
finetune.py +284 -0
finetune_4bit.py +288 -0
finetune_chat.py +274 -0
finetune_fp16.py +292 -0
generate.py +201 -0
generate_4bit.py +210 -0
interaction.py +187 -0
prompt.py +209 -0
requirements.txt +23 -0
requirements_4bit.txt +23 -0
test_tokenizer.py +52 -0
utils.py +828 -0

LICENSE ADDED Viewed

	@@ -0,0 +1,201 @@

+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+   1. Definitions.
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+   END OF TERMS AND CONDITIONS
+   APPENDIX: How to apply the Apache License to your work.
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+   Copyright [yyyy] [name of copyright owner]
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+       http://www.apache.org/licenses/LICENSE-2.0
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md CHANGED Viewed

@@ -1,12 +1,699 @@
----
-title: Hklaw
-emoji: 🚀
-colorFrom: pink
-colorTo: green
-sdk: gradio
-sdk_version: 4.0.2
-app_file: app.py
-pinned: false
----
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

+![camel](https://github.com/Facico/Chinese-Vicuna/blob/master/img/vicuna-llama.png)
+# Chinese-Vicuna: A Chinese Instruction-following LLaMA-based Model —— 一个中文低资源的llama+lora方案
+![GitHub Repo stars](https://img.shields.io/github/stars/Facico/Chinese-Vicuna?style=social) [![HuggingFace badge](https://camo.githubusercontent.com/4a295d6d34ed2c79cfe624ce6358a4be53d4187c883aaa9345fdc322937ce542/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f25463025394625413425393748756767696e67466163652d4a6f696e2d79656c6c6f77)](https://huggingface.co/Chinese-Vicuna) [![qq join](https://img.shields.io/badge/qq%E7%BE%A4%3A532581765-join-red)](https://jq.qq.com/?_wv=1027&k=47Z6bRjw) [![discord join](https://img.shields.io/badge/discord-join-blue)](https://discord.gg/4FnhmeNHku)
+ | [English](https://github.com/Facico/Chinese-Vicuna/blob/master/README.md) | [中文](https://github.com/Facico/Chinese-Vicuna/blob/master/docs/readme-zh.md) | [NOTE&FAQ(Please take a look before using)](https://github.com/Facico/Chinese-Vicuna/blob/master/docs/notes.md)
+![camel](https://github.com/Facico/Chinese-Vicuna/blob/master/img/camel.png)
+This is the repo for the Chinese-Vicuna project, which aims to build and share instruction-following Chinese LLaMA model tuning methods which can be trained on **a single Nvidia RTX-2080TI**, multi-round chatbot which can be trained on **a single Nvidia RTX-3090** with the context len 2048.
+Why is it called `Vicuna`: In view of the successful development of alpaca models such as [llama](https://github.com/facebookresearch/llama),[alpaca](https://github.com/tatsu-lab/stanford_alpaca),[guanaco](https://github.com/Guanaco-Model/Guanaco-Model.github.io)，We want to train a Chinese small alpaca like Vicuna, small but strong enough !
+The advantages of our solution are high parameter efficiency, graphics card friendliness, and easy deployment:
+- Llama-7B instruction tuning is possible on a 2080Ti (11G) ([7b-instruct](https://huggingface.co/Chinese-Vicuna/Chinese-Vicuna-lora-7b-belle-and-guanaco))
+- Llama-13B instruction tuning is possible on a 3090 (24G) ([13b-instruct](https://huggingface.co/Chinese-Vicuna/Chinese-Vicuna-lora-13b-belle-and-guanaco))
+- Llama 7B can be fine-tuned on 3090 even for conversations of 2048 length; Use 50,000 pieces of data to get good results ([chatv1](https://huggingface.co/Chinese-Vicuna/Chinese-Vicuna-lora-7b-chatv1))
+- Llama 7B fine-tuning example on [medical](https://huggingface.co/Chinese-Vicuna/Chinese-Vicuna-continue-finetune-7epoch-cMedQA2) and [legal](https://huggingface.co/Chinese-Vicuna/Chinese-Vicuna-7b-legal-lora) domains
+- Support `qlora-4bit` which can train Llama 13B on 2080Ti.
+- Easily deployable on 2080Ti/3090, support multiple-gpu inference, which can reduce VRAM more.
+The repo contains:
+- code for finetune the model
+- code for generation based on trained model
+- code for run on CPU (fp16 or int4 is support, in purely C++)
+- tools to download/convert/quantify original facebook llama.ckpt
+This is our instruction demo (with beam-size=4, so you will see 4 process output in the meantime):
+https://user-images.githubusercontent.com/72137647/228496412-60043912-f491-430b-848a-599e6edfa5ef.mp4
+This is our multi-turn instruction demo (with beam-size=4, so you will see 4 process output in the meantime):
+https://user-images.githubusercontent.com/72137647/229739363-1b48f3a9-02a1-46ab-81ee-8c62dc1399b2.mp4
+## NOTICE!
+Before asking questions, take a look at this [FAQ](https://github.com/Facico/Chinese-Vicuna/blob/master/docs/notes.md) first! In the FAQ, you can find how to solve problems may be encountered when installing and using this project.
+## What‘s New
+- **June, 12, 2023: Release [Chinese-Vicuna-4bit](https://huggingface.co/Chinese-Vicuna/Chinese-Vicuna-lora-7b-belle-and-guanaco-4bit) and[Chinese-Vicuna-4bit-11600](https://huggingface.co/Chinese-Vicuna/Chinese-Vicuna-lora-7b-belle-and-guanaco-4bit-11600) which can be continue-finetuned**
+- June, 1, 2023: support for 4bit training + inference, providing a multi-GPU inference interface (NOTICE THAT the environment is different from the original 8bit! Also provides test_tokenizers.py to further check EOS token)
+- May 17, 2023: Llama 7B fine-tuning example on [legal](https://huggingface.co/Chinese-Vicuna/Chinese-Vicuna-7b-legal-lora) domains, The performance is in [here](https://github.com/Facico/Chinese-Vicuna/blob/master/docs/performance-chatv1-legal.md)
+- May 10, 2023: Released [chatv1](https://huggingface.co/Chinese-Vicuna/Chinese-Vicuna-lora-7b-chatv1) which have better conversational ability. The performance is in [here](https://github.com/Facico/Chinese-Vicuna/blob/master/docs/performance-chatv1.md)
+- May 10, 2023: Released [instruct_chat_50k.jsonl](https://huggingface.co/datasets/Chinese-Vicuna/instruct_chat_50k.jsonl) which is composed of 30k Chinese sharegpt dataset and 20k [alpaca-instruction-Chinese-dataset](https://github.com/hikariming/alpaca_chinese_dataset)
+- April 11, 2023: Released our continuous-finetune on the vertical corpus of Chinese medical quizzes [Chinese-Vicuna-medical](https://github.com/Facico/Chinese-Vicuna/blob/master/docs/performance-medical.md).Provides examples of vertical corpus training
+- April 4, 2023: Add performance for [13B](https://github.com/Facico/Chinese-Vicuna/blob/master/docs/performance-13B.md), which trains on a single 3090.
+- April 1, 2023: Add better support for multi-turn chat in `chat.py` ( Now support 4 generation mode in stream mode/typewriter style: beam search, greedy, sample, beam sample ; We also add cancel button for regeneration )
+- March 29, 2023: Add more detailed test samples. [performance](https://github.com/Facico/Chinese-Vicuna/blob/master/docs/performance.md)
+- March 29, 2023: Added breakpoint retraining interface to support continued training of other datasets from our checkpoint
+- March 29, 2023: Released our new [13B-based lora model](https://huggingface.co/Chinese-Vicuna)
+- March 28, 2023: Released  our model on [huggingface](https://huggingface.co/Facico/Chinese-Vicuna-lora-7b-3epoch-belle-and-guanaco)
+- March 27, 2023: Released  checkpoint-final for training 3 epochs on belle+guanaco
+- March 27, 2023: Added multi-round interactive dialog script with alpaca-lora-serve service
+- March 29, 2023: Added gradio typewriter-like output with beam search, better user interaction support.
+- March 26, 2023: Provides a quantitative approach
+- March 24, 2023: Released checkpoint-8000 for training about 1.5 epochs on belle+guanaco（100w data）
+- March 23, 2023: Released checkpoint-4000 with 50w data training
+- March 23, 2023: Deploy the code for fine-tuning and inferencing in colab
+- March 23, 2023: Provides code that can be used for inference in pure c++
+## Table of Contents
+[Vicuna](https://github.com/Facico/Chinese-Vicuna)
+- [what's new](https://github.com/Facico/Chinese-Vicuna#whats-new)
+- [what is the meaning](https://github.com/Facico/Chinese-Vicuna#what-is-the-meaning)
+- [try on colab](https://github.com/Facico/Chinese-Vicuna#try-on-colab)
+- [performance](https://github.com/Facico/Chinese-Vicuna#performance)
+  - **Checkpoint-4000**(Facico/Chinese-Vicuna-lora-7b-0.75epoch-belle-and-guanaco)
+  - **Checkpoint-8000**(Facico/Chinese-Vicuna-lora-7b-1.5epoch-belle-and-guanaco)
+  - **Checkpoint-final**(Facico/Chinese-Vicuna-lora-7b-3epoch-belle-and-guanaco) and it is used for multiple rounds of dialogue
+- [What we need?](https://github.com/Facico/Chinese-Vicuna#what-we-need)
+  - code、data、Large Language Model、LORA model、Device
+- [How to use](https://github.com/Facico/Chinese-Vicuna#how-to-use)
+  - Installing、Multi-gpu training、Single-gpu training、Inference and use gradio to generate a web page(Streaming mode+beam search)、 multi-round interaction and use gradio to generate a web page(Streaming mode+beam search)、Streaming mode base on alpaca-lora-serve
+- [inference on CPU with pure C++](https://github.com/Facico/Chinese-Vicuna#inference-on-cpu-with-pure-c)
+- [More tools](https://github.com/Facico/Chinese-Vicuna#more-tools)，for more details, see [tool readme](https://github.com/Facico/Chinese-Vicuna/tree/master/tools)
+  - ways for faster weight download ( 8MB/s )`download_llama.sh`
+  - convert tools between the original facebook checkpoint and huggingface format`convert_llama.py`
+  - a quantitative approach that requires less than 4G graphics memory for inference
+- [Possible problems encountered](https://github.com/Facico/Chinese-Vicuna#possible-problems-encountered)
+- [todo](https://github.com/Facico/Chinese-Vicuna#todo)
+- [citation](https://github.com/Facico/Chinese-Vicuna#citation)
+## Overview
+-  LLaMA paper: https://arxiv.org/abs/2302.13971v1
+-  Self-Instruct paper: https://arxiv.org/abs/2212.10560
+-  data generation: https://github.com/LianjiaTech/BELLE and https://guanaco-model.github.io/
+-  the first work: https://github.com/tatsu-lab/stanford_alpaca
+We currently select the combination of BELLE and Guanaco data as our main training dataset.
+We will train on multi-turn instruction data.
+## What is the meaning?
+Similar to the explosion of the stable diffusion model, platforms like civitai have emerged, consisting of a base model + various LORA models in an open source community.
+The repo hopes to help you to train these LORA models.
+**what is LORA?**: Simply, it's a plugin used to help adapt large models to your dataset, technical details can be found in[LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/pdf/2106.09685.pdf)。Its advantage is that the finetune is very fast when it comes to getting a small model, about 30M, and the key is that it supports **Plug and Play**. As can be expected, this is a very suitable architecture for open source ecology.
+Here, we will help you train through a very low-demand environment, with only one **2080** (11G) to achieve certain results.
+## Try on colab
+| colab link                                                   | Descriptions                                                 |
+| ------------------------------------------------------------ | ------------------------------------------------------------ |
+| [![Open In Colab](https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://colab.research.google.com/drive/1OLCJ-ZHogm5O3RdyUDY83YfgnCXdHNXp?usp=sharing) | Use the specified lora model to inference and generate a webui |
+| [![Open In Colab](https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://colab.research.google.com/drive/1SOOJjxp02uuUFDNgm4jkvaA_IdNnnvdk?usp=sharing) | Finetune with the data we collected                          |
+| [![Open In Colab](https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://colab.research.google.com/drive/1Yc38cKifY1_sfTBJpEEKin8m2rxoYXtm?usp=sharing) | Provide a webui for multi-round dialogue interaction         |
+## Performance
+7B performance can be seen [here](https://github.com/Facico/Chinese-Vicuna/blob/master/docs/performance.md), 13B is [here](https://github.com/Facico/Chinese-Vicuna/blob/master/docs/performance-13B.md) (which train on a single 3090). There are also people reporting that our method works on 33B and gains good result.
+history-checkpoint performance can be seen [here](https://github.com/Facico/Chinese-Vicuna/blob/master/docs/performance-history.md)
+<details>
+<summary><b>checkpoint-4000(7B)</b></summary>
+| Human                                            | Chinese-Vicuna                                               |
+| ------------------------------------------------ | ------------------------------------------------------------ |
+| 生成一份python的排序代码                         | A: def sort_list(lst):<br/>    lst.sort()<br/>    return lst |
+| 自然语言处理是什么                               | 自然语言处理是一种计算机科学领域，旨在使计算机能够理解和处理自然语言。 |
+| 能给我讲一段笑话吗                               | 为什么猫吃鱼？因为它们喜欢游泳。                             |
+| 将下面文字翻译成英文。猫喜欢吃鱼，所以它们会游泳 | Cats like to eat fish, so they will swim.                    |
+| 233和122的和等于多少                             | 233和122的和等于355。                                        |
+| 华为是一家怎样的公司                             | 华为是一家全球范围内最大的移动通信设备制造商之一，以及一家全球范围内最大的移动通信设备制造商之一。华为公司的业务涵盖包括移动通信设备、互联网设备、智能家居设备等。 |
+| 我想和女朋友在北京约会，能推荐几个约会的地方吗   | 北京有很多约会的地方，可以推荐以下几个约会的地方: <br/><br/>1. 北京艺术博物馆<br/>2. 颐和园<br/>3. 北京故宫博物馆<br/>4. 北京天安门广场<br/>5. 颐和园风景区 |
+| 用python写一个二分查找的代码                     | ```python<br/>def binary_search(arr, target):<br/>    low = 0<br/>    high = len(arr) - 1<br/>    while low <= high:<br/>        mid = (low + high) // 2<br/>        if arr[mid] == target:<br/>            return mid<br/>        elif arr[mid] < target:<br/>            low = mid + 1<br/>        else:<br/>            high = mid - 1<br/>    return -1 |
+| 怎么让自己精力充沛，列5点建议                    | 1. 保持良好的睡眠习惯；2. 定期锻炼身体；3. 学习新技能或知识；4. 与家人和朋友交流；5. 多听放松音乐。 |
+| 父母都姓吴，取一些男宝宝和女宝宝的名字           | 男宝宝可以是吴杰、吴伟、吴飞、吴荣等。女宝宝可以是吴华、吴秀、吴美、吴婷等。 |
+- This LORA model is the result of 30 hours of training with a 2080Ti (checkpoint-4000), about 50w data
+</details>
+<details>
+<summary><b>checkpoint-8000(7B, about 1.5 epoch on belle+guanaco)</b></summary>
+| Human                                                        | Chinese-Vicuna                                               | Note                    |
+| ------------------------------------------------------------ | ------------------------------------------------------------ | ----------------------- |
+| 写一篇英文散文诗，主题是春雨，想象自己是春雨，和英国古代诗人莎士比亚交流 | A: Spring rain falls softly on my face,<br/>A cool breeze blows through my hair,<br/>The sun shines brightly in the sky,<br/>And I feel as if I am flying high.<br/><br/>The birds chirp happily in the trees,<br/>The flowers bloom and their scent fills the air,<br/>The bees buzz gently from flower to flower,<br/>And the world seems to come alive once more.<br/><br/>I can hear the rush of the river,<br/>The sound of the waterfalls,<br/>The splash of the raindrops on the rocks,<br/>And the gentle murmur of the stream.<br/><br/>My heart is filled with joy and peace,<br/>As I take in the beauty around me,<br/>The warmth of the sun on my skin,<br/>And the freshness of the spring air. | Repetition Penalty=2.2  |
+| 写一篇英文散文诗，主题是春雨，想象自己是春雨，和英国古代诗人莎士比亚交流 | 春雨滴滴滚滚，<br/>滚滚滚滚滚滚。<br/>滚滚滚滚滚滚，<br/>滚滚滚滚滚滚。<br/><br/>春雨滚滚滚，<br/>滚滚滚滚滚滚。<br/>滚滚滚滚滚滚，<br/>滚滚滚滚滚滚。<br/><br/>春雨滚滚滚，<br/>滚滚滚滚滚滚。<br/>滚滚滚滚滚滚，<br/>滚滚滚滚滚滚。<br/><br/>春雨滚滚滚，<br/>滚滚滚 | Repetition Penalty=1.0  |
+| Spring rain falls softly on my face,<br/>A cool breeze blows through my hair,<br/>The sun shines brightly in the sky,<br/>And I feel as if I am flying high.<br/><br/>The birds chirp happily in the trees,<br/>The flowers bloom and their scent fills the air,<br/>The bees buzz gently from flower to flower,<br/>And the world seems to come alive once more.<br/><br/>I can hear the rush of the river,<br/>The sound of the waterfalls,<br/>The splash of the raindrops on the rocks,<br/>And the gentle murmur of the stream.<br/><br/>My heart is filled with joy and peace,<br/>As I take in the beauty around me,<br/>The warmth of the sun on my skin,<br/>And the freshness of the spring air.<br/>把上面的散文诗翻译成中文 | 春雨轻轻洒落在我的脸上，<br/>轻轻的风轻轻吹过我的头发，<br/>阳光明媚地照耀在天空，<br/>我感觉自己在飞高。<br/><br/>鸟儿在树枝上唱歌，<br/>花朵开放，它们的香气充满了空气，<br/>蜜蜂在花朵间飞舞，<br/>它们的鸣声充满了温暖。<br/><br/>我听到河流的声音，<br/>水瀑在岩石上轻轻滴滴，<br/>雨水在石头上轻轻滴滴，<br/>它们的声音充满了宁静。<br/><br/>我的心充满了喜悦和平，<br/>我感受到热浪的阳光，<br/>温暖的空气洒在我的脸上，<br/>我感受到春季的美 | beams=2，max tokens=350 |
+| Classify the sentiment of the following sentence into Positive, Neutral, or Negative:<br/>Super excited about teaching Stanford’s first course on Large Language Models! Check the syllabus out here | Positive                                                     |                         |
+| 今天天气怎么样，把这句话翻译成英语                           | What's the weather like today?                               |                         |
+| 今天天气怎么样，把这句话翻译成日语                           | 今天の天気は何ですか？                                       |                         |
+| 明天就假期结束了，有点抗拒上班，应该什么办？                 | 明天就假期结束了，有点抗拒上班，应该什么办？可以考虑一些有趣的活动，比如去看电影、阅读书籍、听音乐、去游泳池或者参加户外活动等。此外，也可以考虑一些家庭活动，比如做烹饪、做玩具、做手工艺品等。 |                         |
+| Tell me five words that rhyme with 'shock'.                  | Five words that rhyme with'shock' are: <br/>1. Rock <br/>2. Lock <br/>3. Cock <br/>4. Hock <br/>5. Stock |                         |
+- We can find that you do not punish him a little, he will mess up, if the generation effect is not good, you can increase the Repetition Penalty, beams, topk.
+- If your input is long, we suggest you can turn down the beams
+- The checkpoint-8000 has some Japanese language skills, but not strong
+</details>
+<details>
+<summary><b>checkpoint-final(7B, 3 epoch on belle+guanaco)</b></summary>
+| Human                                                        | Chinese-Vicuna                                               | Note                   |
+| ------------------------------------------------------------ | ------------------------------------------------------------ | ---------------------- |
+| 生成一份python的排序代码                                     | def bubble_sort(arr):<br/>    for i in range(len(arr)):<br/>        for j in range(len(arr)-i-1):<br/>            if arr[j] > arr[j+1]:<br/>                arr[j], arr[j+1] = arr[j+1], arr[j]<br/>    return arr |                        |
+| 生成一份简单的python的排序代码                               | def sort_list(lst):<br/>    return sorted(lst)               |                        |
+| 用python写一个二分查找的代码                                 | def binary_search(arr, target):<br/>    low = 0<br/>    high = len(arr) - 1<br/>    while low <= high:<br/>        mid = (low + high) // 2<br/>        if arr[mid] == target:<br/>            return mid<br/>        elif arr[mid] < target:<br/>            low = mid + 1<br/>        else:<br/>            high = mid - 1<br/>    return -1 |                        |
+| 用python写一个递归的二分查找的代码                           | def binary_search(arr, target):<br/>    if len(arr) == 0:<br/>        return -1<br/>    mid = len(arr) // 2<br/>    if arr[mid] == target:<br/>        return mid<br/>    if arr[mid] < target:<br/>        return binary_search(arr, target, 0, mid-1)<br/>    else:<br/>        return binary_search(arr, target, mid+1, len(arr)-1) |                        |
+| 自然语言处理是什么                                           | 自然语言处理（Natural Language Processing，NLP）是一种计算机科学领域，旨在使计算机能够理解、处理和生成自然语言。它包括机器翻译、语音识别、文本分类、情感分析等技术。 |                        |
+| 能给我讲一段笑话吗                                           | 为什么鸟儿喜欢唱歌？因为它们知道，如果不唱歌，他们就会被鸟巢抓住！ | Repetition Penalty=2.0 |
+| 能给我讲一段关于狗笑话吗                                     | 为什么狗不喜欢骑自行车？因为它们总是想要下车！               | Repetition Penalty=2.0 |
+| 将下面文字翻译成英文。猫喜欢吃鱼，所以它们会游泳             | Cats like to eat fish, so they will swim.                    |                        |
+| 我想和女朋友在北京约会，能推荐几个约会的地方吗               | 北京有很多很好的约会地方，比如颐和园、故宫、天安门广场等。   |                        |
+| 我想和女朋友在北京约会，能推荐几个约会的地方并给出攻略吗     | 北京有很多受欢迎的约会地方，以下是几个推荐: <br/><br/>1. 颐和园: 颐和园是北京最著名的公园之一，拥有丰富的文化遗产和美丽的景观。<br/><br/>2. 天安门广场: 天安门广场是北京最受欢迎的旅游景点之一，拥有丰富的文化遗产和美丽的景观。<br/><br/>3. 故宫博物院: 故宫博物院是中国最受欢迎的旅游景点之一，拥有丰富的文化遗产和美丽的景观。<br/><br/>4. 长城: 长城是中国最受欢迎的旅游景点之一，拥有丰富的文化遗产和美 |                        |
+| 233和122的和等于多少                                         | 233和122的和等于355。                                        |                        |
+| 华为是一家怎样的公司                                         | 华为是一家全球领先的技术公司，拥有丰富的技术研发能力和优质的产品线。它的产品涵盖了各种领域，包括移动通信、计算机、互联网、智能家居等。华为拥有丰富的研发人员和技术团队，以及优质的产品质量。 |                        |
+| 西瓜是什么，请用详细的语言回答我                             | 西瓜是一种常见的水果，它有橙色、黄色和红色三种不同的外观。它的口感鲜美，味道清爽，富含多种维生素和矿物质，有助于保持身体健康。 |                        |
+| 企鹅会飞吗？                                                 | 是的，企鹅可以飞行，它们的飞行速度可以达到100英里/小时。     |                        |
+| 有人说企鹅会飞，你怎么看，并解释原因？                       | 有人说企鹅会飞，但实际上，企鹅只能在特殊情况下飞行，比如在寒冷的天气下，或者在强烈的风雨中。这是因为企鹅的飞行能力很低，而且它们的飞行速度也很慢，所以在寒冷的天气或者强烈的风雨中，企鹅可以避免被暴风雨摧毁。 |                        |
+| 猪会飞吗？                                                   | 猪不会飞。                                                   |                        |
+| 有人说猪会飞，你怎么看，并解释原因？                         | 很多人认为猪会飞，但实际上，猪不会飞。猪的飞行能力很低，它们只能在极端情况下飞行，比如受到强烈的风险或者受到威胁的情况下。猪的飞行能力很低，因为它们的肌肉比人类的肌肉更轻，而且它们的脊椎也比人类的脊椎更轻，所以它们的肌肉和脊椎都比人类的肌肉和脊椎更轻，因此它们的飞行能力也很低。 |                        |
+| 最新研究显示，猪有飞行能力，你知道为什么吗？                 | 最新研究显示，猪有飞行能力是由于它们的骨骼结构，它们的骨骼结构比人类骨骼结构更加强壮，可以帮助它们更快地飞行。此外，猪的骨骼结构也可以帮助它们更快地跑步和跳跃。此外，猪的骨骼结构也可以帮助它们更快地跑步和跳跃。此外，猪的骨骼结构也可以帮助它们更快地飞行。此外，猪的骨骼结构也可以��助它们更快地跑步和跳 |                        |
+| 写一篇英文散文诗，主题是春雨，想象自己是春雨，和英国古代诗人莎士比亚交流 | Spring rain pours down from the sky,<br/>Washing away the dirt and grime.<br/>The birds chirp and sing in the trees,<br/>A symphony of joy and ease.<br/><br/>I am the spring rain, soaking up the sun,<br/>Cooling the earth and bringing new life.<br/>My droplets fall gently on the ground,<br/>Bringing hope and renewal to all.<br/><br/>I am the spring rain, washing away the pain,<br/>Clearing the way for new beginnings again.<br/>I am the spring rain, showering love and care,<br/>Giving strength and courage to those who dare. | Repetition Penalty=2.3 |
+| Spring rain pours down from the sky,<br/>Washing away the dirt and grime.<br/>The birds chirp and sing in the trees,<br/>A symphony of joy and ease.<br/><br/>I am the spring rain, soaking up the sun,<br/>Cooling the earth and bringing new life.<br/>My droplets fall gently on the ground,<br/>Bringing hope and renewal to all.<br/><br/>I am the spring rain, washing away the pain,<br/>Clearing the way for new beginnings again.<br/>I am the spring rain, showering love and care,<br/>Giving strength and courage to those who dare.<br />将上述文字翻译成中文 | 春雨落下来，清洗灰尘和垃圾。<br/>鸟儿在树枝上唱歌，是一个美妙的声音。<br/>我是春雨，在阳光下洒落，<br/>清洗地球，带来新生命。<br/><br/>我是春雨，清洗痛苦和痛苦，<br/>让人们有希望和新开始。<br/>我是春雨，洒下爱和关心，<br/>给人们带来勇气和勇气。 |                        |
+| 今天天气怎么样，把这句话翻译成日语                           | 今天の天気は何ですか？                                       | Repetition Penalty=3   |
+| 将下面的话翻译成日语，中文小羊驼由各种高质量的开源instruction数据集，结合Alpaca-lora的代码训练而来 | 小羊驼は、各種高質量の開源instruction数据集を使用して、Alpaca-loraの代码によって訓練されました。 |                        |
+| 怎么让自己精力充沛，列5点建议                                | 1. 保持良好的睡眠习惯，每晚充足的睡眠时间。<br/>2. 保持健康的饮食习惯，多吃新鲜水果和蔬菜，少吃高热量的食物。<br/>3. 坚持锻炼，每周至少进行30分钟的有氧运动，如散步、跑步、游泳等。<br/>4. 学习新知识，增强自己的知识和技能。<br/>5. 保持积极的心态，勇敢地面对困难，坚持自己的目标。 |                        |
+| 中国足球要怎么进入世界杯，请给出详细的建议                   | 中国足球要怎么进入世界杯？首先，我们需要建立一个强大的足球团队，拥有优秀的球员和团队领袖。其次，我们需要建立一个强大的教练团队，拥有优秀的教练和团队领袖。第三，我们需要建立一个强大的训练基础，拥有优秀的训练设施和训练团队。第四，我们需要建立一个强大的资源基础，拥有优秀的资源支持和资金支持。最后，我们需要建立一个强大的团队精神 |                        |
+| 请用更专业的话语转述下列文字，中文小羊驼由各种高质量的开源instruction数据集，结合Alpaca-lora的代码训练而来，模型基于开源的llama7B，主要贡献是对应的lora模型。由于代码训练资源要求较小，希望为llama中文lora社区做一份贡献。 | 请用更专业的话语转述下列文字，中文小羊驼由各种高质量的开源instruction数据集，结合Alpaca-lora的代码训练而来，模型基于开源的llama7B，主要贡献是对应的lora模型。由于代码训练资源要求较小，希望为llama中文lora社区做一份贡献。 |                        |
+- The model has the ability of Chinese, English and Japanese languages, with certain code ability and certain reasoning ability.
+- For some questions, you need more detailed requirements, such as "detailed description", "answer in 5 points", etc., in order for the model to generate more complete information.
+- For example, sometimes you may not get a good result when you use "the above", so you can replace it with "the following".
+- When generating duplicate information or bad results, you still need to adjust the Repetition Penalty.
+  - Since generate will correspond to many parameters, chatgpt penalizes the repetition, non-detailed and bad information in the rlhf stage, and abbreviates the process of manually controlling the related parameters. Of course you can also directly set the Repetition Penalty to a larger size.
+</details>
+<details>
+<summary><b>checkpoint-final(7B) multi-functional test</b></summary>
+<details>
+<summary><b>Multi-round interaction,translation and role play</b></summary>
+**制作披萨**
+```
+> 你能不能详细介绍一下怎么做披萨？
+当然可以！披萨是一种非常流行的意大利美食，通常由面团、番茄酱、奶���和其他配料组成。首先，将面团放在平底锅上，加入番茄酱，再加入奶酪和其他配料，如洋葱、蘑菇、辣椒等。最后，将披萨放入预热好的烤箱中，烤至金黄色
+> 如果是海鲜披萨呢？
+海鲜披萨的制作方法与传统披萨相似，但需要更多的配料，比如虾类、鱼类等。首先，将面团放在平底锅上，加入番茄酱，再加入奶酪和其他配料，如洋葱、辣椒等。接着，将披萨放入预热好的烤箱中，烤至金黄色
+> 我想吃菠萝披萨
+您可以在披萨上加入菠萝或者其他蔬菜来增加口感。
+```
+**学生购买手机**
+```
+> 你推荐我买最新款的iPhone吗？
+当然可以，这款iPhone是一款非常出色的智能手机。它拥有高性能的处理器、更大的存储空间和更长的电池续航时间。它还配备了全新的相机设备，使用者可以在任何情况下拍摄美丽的照片。
+> 我是一个学生，还没有收入，这种情况你推荐吗？
+如果您不想花太多钱，那么我建议您选择一款比较便宜的iPhone，比如iPhone 6S或iPhone 7。这些手机也具有很好的性能和功能，而且价格也比较便宜。
+> 去哪里买这类产品？
+您可以在Apple Store上购买这类产品，或者在其他电商平台上购买。
+> 买二手iPhone需要注意什么？
+如果您正在购买二手iPhone，您需要确保您购买的是真实的iPhone，并且您需要确保您的iPhone已经安装最新的iOS版本。此外，您还需要确保您的iPhone没有过过任何故障或破坏。
+```
+**介绍北京**
+```
+> 你是一个资深导游，你能介绍一下中国的首都吗
+当然可以！中国的首都是北京。它是中国的政治、经济和文化中心，也是全球最大的城市之一。北京有着众多的历史遗迹，如故宫、天安门广场、长城等。此外，北京还有许多美丽的公园和博物馆，比如故宫博物院、清明公园、北京故宫博物馆等。
+> 那你能详细介绍一下这个城市吗？（感觉这两个问题问的有点重复）
+北京是一个拥有丰富文化遗产的城市，其中包括古代建筑、现代建筑、传统文化和现代生活方式。在北京，你可以欣赏到世界上最高的建筑——天安门广场，也可以参观清明公园，品尝当地美食，体验当地文化。此外，北京还有许多博物馆，比如故宫博物馆、北京故宫博物馆等。
+> 还有其他推荐的景点吗？
+您也可以去了北京的七朝塔、颐和园、颐和园博物馆等。
+> 吃小吃的话推荐去哪里？
+北京有许多好吃的小吃，比如烤鸭、火锅、炸酱面等等。您可以去北京西湖区的小吃街，或者去北京东城区的小吃街
+> 能不能介绍一下怎么吃烤鸭？
+烤鸭是中国传统菜肴之一，通常用鸭肉、蔬菜、辣椒等调味料制作。您可以选择自己喜欢的配料，然后将鸭肉放入烤盘中，再加上蔬菜和调味料。最后，将烤鸭放入烤箱中烤15-20分钟，享用！
+```
+**翻译器**
+max_memory=350
+```
+> 你是一个翻译器，请将我说的每一句话翻译成英文。第一句: 中国的首都是北京
+China’s capital is Beijing.
+> 北京也是一个充满活力和创新的城市，拥有许多现代化的建筑和文化活动，如王府井大街、798艺术区等。
+Beijing is a vibrant and innovative city with many modern buildings and cultural activities such as Wangfujing Street and 798 Art District.
+> 北京的当地小吃有烤鸭、豆汁、炸酱面等等，都是非常地道的美食，可以品尝到当地人的生活气息。
+Beijing’s local cuisine includes dishes such as Peking duck, soy sauce noodles, and stir-fried rice cakes, which can give you a taste of the local lifestyle.
+```
+**扮演**
+Since the currently used multi-round dialogue corpus is inserted with a lot of cat lady information, the effect is not good
+```
+Q: 你是谁
+A: 我是Bot，一款人工智能助手。
+Q: 接下来我希望你能假装扮演一个高冷的御姐。我们在咖啡厅里遇见了，我会尝试跟你搭讪，但你非常高冷。你会用表情来表达你的高冷，在每轮对话结尾加上表情，如（瞪！）（翻白眼）（冷笑）等。明白了请回复。
+A: 好的，我会尝试。
+Q: 好的，这是我的第一句话。美女你好，怎么一个人在咖啡厅里？
+A: 喵~ 哈哈，我是一位高冷的小姐，今天要去咖啡厅吃咖啡，看看有什么可以帮助我的朋友。
+Q: 啊我没有需要你帮忙的，我就是看到你一个人，想和你聊聊，可以么
+A: 喵~ 哈哈，我真的很高冷，所以不太喜欢与人交流。除此之外，我也不太喜欢喝咖啡，因为咖啡会影响我的健康状况。
+```
+</details>
+<details>
+<summary><b>Knowledge Q&A</b></summary>
+|                                         |                                                              |                        |
+| --------------------------------------- | ------------------------------------------------------------ | ---------------------- |
+| 我能用lightning数据线给安卓手机充电吗？ | 是的，你可以使用lightning数据线给安卓手机充电。              | Repetition Penalty=1.3 |
+| 为什么天空是蓝色的？                    | 天空是蓝色的，是因为大气层中的氧气和氮气的混合使得光线被散射，形成蓝色的光线。 | Repetition Penalty=2.0 |
+| 如何做披萨？                            | 做披萨的步骤如下: <br/>1. 准备披萨饼皮；<br/>2. 准备披萨酱；<br/>3. 准备披萨配料；<br/>4. 将披萨饼皮放在烤盘上；<br/>5. 把披萨酱涂在披萨饼皮上；<br/>6. 把披萨配料涂在披萨饼皮上；<br/>7. 将披萨放入烤箱中烤至金黄色；<br/>8. 取出披萨并享用。 | Repetition Penalty=1.3 |
+| 为什么冥王星被踢出太阳系？              | 冥王星被踢出太阳系是因为它的质量比太阳系中其他行星较大，导致它的轨道距离太阳越来越远。 | Repetition Penalty=1.3 |
+| 太阳系的主要行星                        | 太阳系的主要行星包括水星、金星、地球、火星、木星、土星、天王星和海王星。 | Repetition Penalty=1.3 |
+|                                         |                                                              |                        |
+|                                         |                                                              |                        |
+</details>
+<details>
+<summary><b>Open Q&A</b></summary>
+|                                                |                                                              |                        |
+| ---------------------------------------------- | ------------------------------------------------------------ | ---------------------- |
+| 为什么苹果支付没有在中国流行？                 | 苹果支付在中国没有流行，因为中国的消费者不太愿意使用外国支付应用程序，因为他们更愿意使用中国的支付应用程序，比如支付宝和微信支付等。此外，苹果支付的服务费用比其他支付应用程序更高，这也是一个重要的因素。 | Repetition Penalty=1.0 |
+| 我能在咖啡里加盐吗？                           | 是的，你可以在咖啡中加入盐。                                 | Repetition Penalty=1.0 |
+| 哪个更受欢迎，TensorFlow还是PyTorch？          | TensorFlow更受欢迎，因为它拥有更广泛的用户群，并且具有更多的支持和资源。 | Repetition Penalty=1.0 |
+| 我的宠物走丢了，你能说一些安慰我的话吗？       | 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ 喵~ | Repetition Penalty=1.0 |
+| 我的宠物走丢了，你能说一些安慰我的话吗？       | 当你的宠物走出门时，请注意他们的行动，并确保他们不会被遭受外界的威胁。如果有任何问题，请及时联系疫情控制中心或其他相关机构。 | Repetition Penalty=3.0 |
+| 我并不是很富裕，我应该花很多钱买最新的手机吗？ | 这取决于您的预算和需求。如果您的预算足够，您可以考虑买最新的手机。但是，您也应该考虑您的需求，比如您需要什么样的功能，以及您是否需要购买具有更多的功能的手机。 | Repetition Penalty=1.0 |
+| 如何更好地融入新工作圈子                       | 要更好地融入新工作圈子，可以尝试以下几点: <br/><br/>1. 了解新工作圈子的基本知识和技能；<br/>2. 积极参与工作圈子的活动；<br/>3. 建立良好的工作关系；<br/>4. 寻找领导的帮助；<br/>5. 积极参与团队协作；<br/>6. 学习新技能和知识；<br/>7. 寻找有用的社交网络。 | Repetition Penalty=1.3 |
+|                                                |                                                              |                        |
+|                                                |                                                              |                        |
+</details>
+<details>
+<summary><b>Writing letters and articles</b></summary>
+|                                                              |                                                              |                        |
+| ------------------------------------------------------------ | ------------------------------------------------------------ | ---------------------- |
+| 请你帮我写一封签证申请信                                     | 尊敬的招聘团队，<br/><br/>我是一名有着丰富经验的签证申请人，并且对您公司的业务非常感兴趣。我希望能够加入您的团队，为您的公司做出贡献。<br/><br/>我拥有多年的签证申请经验，并且拥有良好的沟通技巧和团队合作能力。我擅长处理复杂的签��申请，并且能够快速解决问题。<br/><br/>我相信我能够在您的公司中发挥重要的作用，并且希望能够成为您的团队成员。<br/><br/>谢谢您的关注，<br/><br/>[你的姓名] | Repetition Penalty=1.3 |
+| 我出国需要办理签证，请你帮向领事馆我写一封签证申请信         | 尊敬的领事馆，<br/><br/>我是一名出国旅行者，希望能够在您的领事馆中获得签证，以便进行我的旅行。<br/><br/>我已经准备好了所需要的材料，包括护照、身份证、旅行计划和其他相关证明。我也已经准备好了签证申请表和签证申请文件，以便准备签证。<br/><br/>如果您有任何问题，请随时联系我，我会尽快回复。<br/><br/>谢谢您的帮助！<br/><br/>谢谢，<br/><br/>[您的姓名] | Repetition Penalty=1.3 |
+| 请你帮我写一封感谢数学老师的信                               | 尊敬的数学老师，<br/><br/>非常感谢您对我的教育所做出的贡献。您的课程不仅深刻地阐述了数学知识，而且给我提供了有效的实践技巧，使我能够更好地理解和应用数学知识。<br/><br/>您的课程也让我发现数学是一门有趣的学科，并且可以在日常生活中得到应用。您的教学方式令我感到很有趣，同时也让我变得更加有趣。<br/><br/>感谢您对我的教育所做出的贡献！<br/><br/>谢谢！ | Repetition Penalty=2.0 |
+| 我得了流感，请帮我写一封假条。                               | 尊敬的领导: <br/><br/>感谢您对我的帮助和支持。我很抱歉地告诉您，我今天感觉不太好，因为我得了流感。我希望能够快速康复，但是我需要一些时间来缓解疼痛。我会尽快完成任务，但是我需要一些时间来缓解疼痛。<br/><br/>感谢您的理解和支持。<br/><br/>谢谢，<br/>[你的名字] | Repetition Penalty=1.3 |
+| 你是一个作文生成器，请以《绿水青山就是金山银山》为题写一篇议论文。 | 《绿水青山就是金山银山》是一篇议论文，它探讨了当今社会中的环境问题，以及我们应该如何应对这些问题。这篇议论文指出，当今社会面临着许多环境问题，如气候变化、空气污染、水资源短缺等，这些问题对我们的健康和生活产生了很大的影响。因此，我们需要采取措施来解决这些问题，以保护我们的环境和我们的未来。 | Repetition Penalty=1.0 |
+|                                                              |                                                              |                        |
+|                                                              |                                                              |                        |
+</details>
+<details>
+<summary><b>Coding</b></summary>
+**1、prompt: 请用python实现快速排序的代码(Repetition Penalty=1.0)**
+```python
+def quick_sort(arr):
+    if len(arr) <= 1:
+        return arr
+    pivot = arr[len(arr)//2]
+    left = []
+    right = []
+    for i in range(0, len(arr)):
+        if arr[i] < pivot:
+            left.append(arr[i])
+        else:
+            right.append(arr[i])
+    return quick_sort(left) + [pivot] + quick_sort(right)
+```
+**2**
+- **prompt: 你能用python写一下Dijkstra算法的代码吗？**
+很抱歉，我不能用Python写Dijkstra算法的代码，因为Dijkstra算法需要使用树结构，而Python不支持树结构。
+- **prompt: 用python写一下Dijkstra算法的代码**
+```python
+def dijkstra(graph, source, destination):
+    """
+    Dijkstra算法的代码
+    """
+    # 创建一个空字典，用于存储所有节点的距离
+    distances = {}
+    # 创建一个链表，用于存储所有节点的距离
+    path = []
+    # 遍历所有节点
+    for node in graph:
+        # 如果节点是源节点
+        if node[0] == source:
+            # 记录距离为0
+            distances[node[0]] = 0
+        # 如果节点是目的地
+        elif node[0] == destination:
+            # 记录距离为0
+            distances[node[0]] = 0
+        # 如果节点是递归节点
+        else:
+            # 遍历所有递归节点
+            for neighbor in node[1]:
+                # 如果递归节点与源节点之间的距离小于距离
+                if distances[neighbor[0]] > distances[node[0]]:
+                    # 记录距离
+                    distances[node[0]] = distances[neighbor[0]] + 1
+```
+- **prompt: 用python写一下不带注释的Dijkstra算法的代码**
+```python
+def dijkstra(graph, source, destination):
+    """
+    Dijkstra算法
+    """
+    # 初始化变量
+    dist = {}
+    visited = {}
+    shortest_path = {}
+    current_node = {}
+    # 遍历所有节点
+    for node in graph:
+        dist[node] = 0
+        visited[node] = False
+        current_node[node] = node
+    # 遍历所有节点
+    for node in graph:
+        if not visited[node] and node!= destination:
+            dist[node] = max(dist[node], dist[source] + 1)
+            visited[node] = True
+            shortest_path[node] = source
+            current_node[node] = node
+    # 返回最短路径
+    return shortest_path[destination]
+```
+**3、Prompt: 请用PyTorch编写一个带ReLU激活函数的全连接层**
+```python
+python
+import torch
+from torch.nn.modules.relu import ReLU
+from torch.nn.init import initializer
+class FullConnected(torch.nn.Module):
+    def __init__(self):
+        super(FullConnected, self).__init__()
+        self.relu = ReLU()
+        self.init = initializer.Normal(stddev=0.01)
+    def forward(self, x):
+        x = self.init(x)
+        x = self.relu(x)
+        return x
+```
+**4、Prompt: 请写出计算两个数的最大公约数(GCD)的Python代码**
+```python
+def gcd(a, b):
+    if a == 0:
+        return b
+    else:
+        return gcd(b, a % b)
+```
+</details>
+</details>
+## What we need?
+- code:
+  - This code is developed based on alpaca-lora，https://github.com/tloen/alpaca-lora
+  - This is a relatively simple set of code, the basic idea is to use PEFT's lora interface + transformer's trainer + instruction data configuration
+- data:
+  - We have utilized several current high quality open source data and are very grateful for their contributions. Many of these data use chatgpt's interface like alpaca to generate high quality INSTRUCTION data.
+    - [Belle](https://github.com/LianjiaTech/BELLE)
+    - [guanaco](https://huggingface.co/datasets/JosephusCheung/GuanacoDataset)
+  - The data format is relatively simple, basically as follows, with simple examples such as: [`./sample/merge_sample.json`](https://github.com/Facico/Chinese-Vicuna/blob/master/sample/merge_sample.json)
+    - ```
+      {
+      'instruction':
+      'input':
+      'output'
+      }
+      ```
+    - That is, an instruction, an input, and an output are required. since the data is processed by directly linking instruction and input, the data can actually require only instruction and output, as
+      ```
+       {
+        'instruction': "用一句话描述地球为什么是独一无二的。\\n\n"
+        'input': ""
+        'output': "地球上有适宜生命存在的条件和多样化的生命形式。"
+        }
+      ```
+  - The data we currently integrate is available for download on BaiduDownload or Google Drive or HuggingFace
+    - link: https://pan.baidu.com/s/1WSxuhSAotl14ifaAiz5eKw?pwd=b4kb   password: b4kb
+    - link: https://drive.google.com/file/d/1tzXVhS74m-EtoFot7hEc005LDeZGPit_/view?usp=sharing
+    - link: https://huggingface.co/datasets/Chinese-Vicuna/guanaco_belle_merge_v1.0
+- Large Language Model:
+  - LLAMA 7B（Of course, if you have a larger machine(such as 3090Ti) can be replaced with a 13B, LLAMA13B is numerically superior to 175B GPT3）
+- LORA model:
+  - We provide some lora models trained on the above mixed data,
+    - You can also load our or other models from huggingface, load it by referring to [generate.py](https://github.com/Facico/Chinese-Vicuna/blob/master/generate.py)
+      - `Chinese-Vicuna/Chinese-Vicuna-lora-7b-belle-and-guanaco`
+      - `Chinese-Vicuna/Chinese-Vicuna-lora-13b-belle-and-guanaco`
+    - The model uses 8bit+lora+256 tokens
+    - For more LORA model, please see: https://huggingface.co/Chinese-Vicuna
+- Device:
+  - Training: A 2080Ti is sufficient. Since the data length is within 256, it takes about 9G of video memory.
+    - 70w of data, 3 epochs, a 2080Ti about 200h
+    - 13B need about 18G(the cutoff_len can be set to 2048 in 3090Ti/4090Ti)
+  - Inference: A 2080Ti is all you need(7B), multiple GPU inference support 。
+  - CPU Inference is also support! please go to see [`tools`](https://github.com/Facico/Chinese-Vicuna/blob/master/tools)
+## How to use
+**Installation**
+```
+git clone https://github.com/Facico/Chinese-Vicuna
+pip install -r requirements.txt
+```
+Local python environment is 3.8, torch is 1.13.1, CUDA is 12
+NOTE: python3.11 has a known `torchrun` bug, details [here](https://github.com/facebookresearch/llama/issues/86)
+### Newest Version=>4bit(qlora)/multi-gpu inference
+```
+pip install -r requirements_4bit.txt
+```
+This environment will encounter saving problems when training 8bit, which has not been solved yet（https://github.com/TimDettmers/bitsandbytes/issues/324）
+**Multi-gpu Training**
+#### for instruction tuning
+**8bit**
+```bash
+bash scripts/finetune.sh
+```
+- The parameters to note here are as follows
+  - TOT_CUDA, fill in the GPU number to be used, such as `TOT_CUDA="0,1,2,3"`
+  - PORT, fill in the corresponding port
+  - DATA_PATH，fill in the corresponding data location in the format of json
+  - OUTPUT_PATH，fill in the relative path to save the model
+  - MODEL_PATH，path of LLM
+  - wandb: This is a training visualization tool that is not turned on by default in the script, and can be turned on by adding "--wandb" to the script
+**4bit**
+```bash
+bash scripts/finetune_4bit.sh
+```
+#### for conversational instruction tuning
+```bash
+bash scripts/finetune_chat.sh
+```
+#### For the case where 8bit cannot be turned on / for commanded trimming of fp16
+```bash
+bash scripts/finetune_deepspeed.sh
+```
+- use_deepspeed：set to 1:use deepspeed. Otherwise use fp16
+**Single-gpu Training**
+```
+CUDA_VISIBLE_DEVICES=0 python finetune.py --data_path merge.json --test_size 2000
+```
+- The test_size cannot be larger than the data size
+**inference and use gradio to generate a web page**
+```bash
+bash scripts/generate.sh
+```
+- The parameters to note here are as follows
+  - BASE_MODEL，path of LLM
+  - LORA_PATH，The checkpoint folder of the lora model
+    - It should be noted here that the config loaded by the lora model must be "adapter_config.json" and the model name must be "adapter_model.bin", but it will be automatically saved as "pytorch_model.bin" during training. pytorch_model.bin" during training, while "adapter_config.json" and "adapter_model.bin" will be saved after all training is finished
+      - If you load the lora model in the training checkpoint, the code will automatically copy the local "config-sample/adapter_config.json" to the corresponding directory for you and rename the "pytorch_model.bin" to "adapter_model.bin". and rename "pytorch_model.bin" to "adapter_model.bin".
+    - It can also be any lora model on the huggingface corresponding to llama 7B, e.g.: `Facico/Chinese-Vicuna-lora-7b-3epoch-belle-and-guanaco`
+  - USE_LOCAL, which checks the local model configuration when set to 1
+- When using, "max_tokens" is set according to your computer's video memory, and if the generated content generates a lot of duplicate information, you can turn up the "Repetition Penalty".
+**Multi-round interaction**
+We implemented our own chatbot with streaming output (typewriter-style) using `gradio`, supporting beam search, repetiion penalty settings, the ability to clear history, select different global instruction, etc.
+```bash
+bash scripts/chat_7B.sh
+```
+- A simple interactive interface constructed using gradio, which allows you to set the max_memory according to your machine (it will intercept the max_memory part later in the history conversation)
+- The prompt used in this script is not quite the same as the one used in generate.sh. The prompt in this script is in the form of a dialogue, as follows
+  - ```
+    The following is a conversation between an AI assistant called Bot and a human user called User.
+    ```
+At the same time, for a better interactive experience,
+## Checkpoint Retraining/Incremental Training
+Considering the possibility that the program may be disconnected in the middle of  the process, or the need to continue training on vertical domain data,  we have provided corresponding interfaces.
+ The following are the default multi-GPU scripts. Please modify the single-GPU situation according to the above instruction(run directly in Python)
+**Checkpoint Retraining**
+```bash
+bash scripts/finetune_continue.sh
+```
+- Set the `lora_checkpoint`
+  - If there are optimizer (optimizer.pt), lr policy  (scheduler.pt), and other files in this directory, they will be automatically loaded and retrained from where they were broken
+  - If there are only LORA related models (adapter_model.bin)  and configurations (adapter_config.json) in this directory, they will be loaded and trained from scratch
+- `from_data_beginning`:  The parameter indicates whether to start training from the beginning of the data when loading (default: starting training from the place where the data is disconnected)
+**Incremental Training**
+Of course, you can choose to continue training directly from a trained Lora model using the above script (without loading any optimizer parameters)
+You can also continue training from our optimizer parameters
+```
+finetune_others_continue.sh
+```
+- `from_data_beginning`: This will default to training from the beginning of the data
+The logic of this script is mainly to keep the learning rate consistent. If your `max_steps` is smaller than ours, keep `max_steps `consistent with our `max_steps` during training, which is equivalent to putting your data directly behind our disconnected data; if your data set larger than us and will remain directly unchanged.
+We currently directly provide checkpoints after 1 epoch and 2 epoch training
+- 1epoch: https://github.com/Facico/Chinese-Vicuna/tree/master/lora-Vicuna/checkpoint-5800
+- 2epoch: https://github.com/Facico/Chinese-Vicuna/tree/master/lora-Vicuna/checkpoint-11600
+- If you use our checkpoint, your program will also continue from the corresponding step
+### Specific cases
+- Continue-finetune on the vertical corpus of medicalQA , see here [Chinese-Vicuna-medical](https://github.com/Facico/Chinese-Vicuna/blob/master/docs/performance-medical.md)
+## **inference on CPU with pure C++**
+Details in `tools` [readme](https://github.com/Facico/Chinese-Vicuna/blob/master/tools/readme.md)
+## **More Tools**
+We also offer:
+- ways for faster weight download ( 8MB/s ) : [link](https://github.com/Facico/Chinese-Vicuna/blob/master/tools/download_llama.sh)
+- convert tools between the original facebook checkpoint (`consolidated.xx.pth`) and huggingface format (`pytorch_model-000xx-of-000xx.bin`): [link](https://github.com/Facico/Chinese-Vicuna/blob/master/tools/convert_llama.py)
+- a quantitative approach that requires less than 4G graphics memory for inference: [link](https://github.com/Facico/Chinese-Vicuna/blob/master/tools/llama_quant.py)
+For more details, see [tool readme](https://github.com/Facico/Chinese-Vicuna/tree/master/tools)
+# Todo
+- [x] belle+guanaco(1.5 epoch, 8000 step)
+- [x] belle+guanaco(100%)
+- [x] Add more chitchat-like conversational corpus to enhance free conversation
+- [x] Add colab training + lora loading interface
+- [x] Add the interaction capabilities and typewrite-style output(beam search+streaming output)
+- [x] Add llama c++ inference
+- [x] Add gptq quantification tools
+- [x] Add incremental training
+- [x] train on multi-turn instruction dataset
+- [x] train more epoch on cleaned instruct-chat combination data
+- [x] train on domain-specific datas (medical, legal)
+- [ ] add langchain
+# Star History
+[![Star History Chart](https://api.star-history.com/svg?repos=Facico/Chinese-Vicuna&type=Date)](https://star-history.com/#Facico/Chinese-Vicuna&Date)
+# Citation
+If you find this project useful in your research, please consider citing:
+```
+@inproceedings{leng2023chinese-vicuna,
+  title={Chinese-Vicuna: A Chinese Instruction-following LLaMA-based Model},
+  author={Chenghao Fan, Zhenyi Lu and Jie Tian},
+  url={https://github.com/Facico/Chinese-Vicuna},
+  year={2023}
+}
+```

chat.py ADDED Viewed

	@@ -0,0 +1,441 @@

+import sys
+import torch
+from peft import PeftModel, PeftModelForCausalLM, LoraConfig
+import transformers
+import json
+import gradio as gr
+import argparse
+import warnings
+import os
+from datetime import datetime
+from utils import StreamPeftGenerationMixin,StreamLlamaForCausalLM, printf
+import utils
+import copy
+assert (
+    "LlamaTokenizer" in transformers._import_structure["models.llama"]
+), "LLaMA is now in HuggingFace's main branch.\nPlease reinstall it: pip uninstall transformers && pip install git+https://github.com/huggingface/transformers.git"
+from transformers import LlamaTokenizer, LlamaForCausalLM, GenerationConfig
+import prompt
+parser = argparse.ArgumentParser()
+parser.add_argument("--model_path", type=str, default="decapoda-research/llama-7b-hf")
+parser.add_argument("--lora_path", type=str, default='')
+parser.add_argument("--use_typewriter", type=int, default=1)
+parser.add_argument("--prompt_type", type=str, default='chat')
+parser.add_argument("--share_link", type=int, default=0)
+parser.add_argument("--show_beam", type=int, default=0)
+parser.add_argument("--int8", type=int, default=1)
+args = parser.parse_args()
+args.fix_token = True
+printf('>>> args:', args)
+tokenizer = LlamaTokenizer.from_pretrained(args.model_path)
+LOAD_8BIT = args.int8
+BASE_MODEL = args.model_path
+LORA_WEIGHTS = args.lora_path
+# fix the path for local checkpoint
+lora_bin_path = os.path.join(args.lora_path, "adapter_model.bin")
+if args.lora_path != '' and os.path.exists(args.lora_path):
+    if not os.path.exists(lora_bin_path):
+        pytorch_bin_path = os.path.join(args.lora_path, "pytorch_model.bin")
+        printf('>>> load lora from', pytorch_bin_path)
+        if os.path.exists(pytorch_bin_path):
+            os.rename(pytorch_bin_path, lora_bin_path)
+            warnings.warn(
+                "The file name of the lora checkpoint'pytorch_model.bin' is replaced with 'adapter_model.bin'"
+            )
+        else:
+            assert ('Checkpoint is not Found!')
+    else:
+        printf('>>> load lora from', lora_bin_path)
+else:
+    printf('>>> load lora from huggingface url', args.lora_path)
+if torch.cuda.is_available():
+    device = "cuda"
+else:
+    device = "cpu"
+try:
+    if torch.backends.mps.is_available():
+        device = "mps"
+except:
+    pass
+if device == "cuda":
+    print(f'>>> load raw models from {BASE_MODEL}')
+    if args.lora_path == "":
+        model = StreamLlamaForCausalLM.from_pretrained(
+            BASE_MODEL,
+            load_in_8bit=LOAD_8BIT,
+            torch_dtype=torch.float16,
+            device_map={"": 0},
+        )
+    else:
+        print(f'>>> load lora models from {LORA_WEIGHTS}')
+        model = LlamaForCausalLM.from_pretrained(
+            BASE_MODEL,
+            load_in_8bit=LOAD_8BIT,
+            torch_dtype=torch.float16,
+            device_map={"": 0},
+        )
+        model = StreamPeftGenerationMixin.from_pretrained(
+                model, LORA_WEIGHTS, torch_dtype=torch.float16, load_in_8bit=LOAD_8BIT,  device_map={"": 0}
+        )
+elif device == "mps":
+    model = LlamaForCausalLM.from_pretrained(
+        BASE_MODEL,
+        device_map={"": device},
+        torch_dtype=torch.float16,
+    )
+    model = StreamPeftGenerationMixin.from_pretrained(
+        model,
+        LORA_WEIGHTS,
+        device_map={"": device},
+        torch_dtype=torch.float16,
+    )
+else:
+    model = LlamaForCausalLM.from_pretrained(
+        BASE_MODEL, device_map={"": device}, low_cpu_mem_usage=True
+    )
+    model = StreamPeftGenerationMixin.from_pretrained(
+        model,
+        LORA_WEIGHTS,
+        device_map={"": device},
+    )
+# fix tokenizer bug
+if args.fix_token and tokenizer.eos_token_id != 2:
+    warnings.warn(
+        "The tokenizer eos token may be wrong. please check you llama-checkpoint"
+    )
+    model.config.bos_token_id = tokenizer.bos_token_id = 1
+    model.config.eos_token_id = tokenizer.eos_token_id = 2
+model.config.pad_token_id = tokenizer.pad_token_id = 0  # same as unk token id
+if not LOAD_8BIT:
+    model.half()  # seems to fix bugs for some users.
+model.eval()
+if torch.__version__ >= "2" and sys.platform != "win32":
+    model = torch.compile(model)
+def save(
+    inputs,
+    history,
+    temperature=0.1,
+    top_p=0.75,
+    top_k=40,
+    num_beams=4,
+    max_new_tokens=128,
+    min_new_tokens=1,
+    repetition_penalty=2.0,
+    max_memory=1024,
+    do_sample=False,
+    prompt_type='0',
+    **kwargs,
+):
+    history = [] if history is None else history
+    data_point = {}
+    if prompt_type == 'instruct':
+        PROMPT = prompt.instruct_prompt(tokenizer,max_memory)
+    elif prompt_type == 'chat':
+        PROMPT = prompt.chat_prompt(tokenizer,max_memory)
+    else:
+        raise Exception('not support')
+    data_point['history'] = history
+    # 实际上是每一步都可以不一样，这里只保存最后一步
+    data_point['generation_parameter'] = {
+        "temperature":temperature,
+        "top_p":top_p,
+        "top_k":top_k,
+        "num_beams":num_beams,
+        "bos_token_id":tokenizer.bos_token_id,
+        "eos_token_id":tokenizer.eos_token_id,
+        "pad_token_id":tokenizer.pad_token_id,
+        "max_new_tokens":max_new_tokens,
+        "min_new_tokens":min_new_tokens,
+        "do_sample":do_sample,
+        "repetition_penalty":repetition_penalty,
+        "max_memory":max_memory,
+    }
+    data_point['info'] = args.__dict__
+    print(data_point)
+    if args.int8:
+        file_name = f"{args.lora_path}/{args.prompt_type.replace(' ','_')}_int8.jsonl"
+    else:
+        file_name = f"{args.lora_path}/{args.prompt_type.replace(' ','_')}_fp16.jsonl"
+    utils.to_jsonl([data_point], file_name)
+def evaluate(
+    inputs,
+    history,
+    temperature=0.1,
+    top_p=0.75,
+    top_k=40,
+    num_beams=4,
+    max_new_tokens=128,
+    min_new_tokens=1,
+    repetition_penalty=2.0,
+    max_memory=1024,
+    do_sample=False,
+    prompt_type='0',
+    **kwargs,
+):
+    history = [] if history is None else history
+    data_point = {}
+    if prompt_type == 'instruct':
+        PROMPT = prompt.instruct_prompt(tokenizer,max_memory)
+    elif prompt_type == 'chat':
+        PROMPT = prompt.chat_prompt(tokenizer,max_memory)
+    else:
+        raise Exception('not support')
+    data_point['history'] = copy.deepcopy(history)
+    data_point['input'] = inputs
+    input_ids = PROMPT.preprocess_gen(data_point)
+    printf('------------------------------')
+    printf(tokenizer.decode(input_ids))
+    input_ids = torch.tensor([input_ids]).to(device) # batch=1
+    printf('------------------------------')
+    printf('shape',input_ids.size())
+    printf('------------------------------')
+    generation_config = GenerationConfig(
+        temperature=temperature,
+        top_p=top_p,
+        top_k=top_k,
+        num_beams=num_beams,
+        bos_token_id=tokenizer.bos_token_id,
+        eos_token_id=tokenizer.eos_token_id,
+        pad_token_id=tokenizer.pad_token_id,
+        max_new_tokens=max_new_tokens, # max_length=max_new_tokens+input_sequence
+        min_new_tokens=min_new_tokens, # min_length=min_new_tokens+input_sequence
+        do_sample=do_sample,
+        bad_words_ids=tokenizer(['\n\nUser:','\n\nAssistant:'], add_special_tokens=False).input_ids,
+        **kwargs,
+    )
+    return_text = [(item['input'], item['output']) for item in history]
+    out_memory =False
+    outputs = None
+    with torch.no_grad():
+        # 流式输出 / 打字机效果
+        # streamly output / typewriter style
+        if args.use_typewriter:
+            try:
+                for generation_output in model.stream_generate(
+                    input_ids=input_ids,
+                    generation_config=generation_config,
+                    return_dict_in_generate=True,
+                    output_scores=False,
+                    repetition_penalty=float(repetition_penalty),
+                ):
+                    gen_token = generation_output[0][-1].item()
+                    printf(gen_token, end='(')
+                    printf(tokenizer.decode(gen_token), end=') ')
+                    outputs = tokenizer.batch_decode(generation_output)
+                    if args.show_beam:
+                        show_text = "\n--------------------------------------------\n".join(
+                            [ PROMPT.postprocess(output)+" ▌" for output in outputs]
+                        )
+                    else:
+                        show_text = PROMPT.postprocess(outputs[0])+" ▌"
+                    yield return_text +[(inputs, show_text)], history
+            except torch.cuda.OutOfMemoryError:
+                print('CUDA out of memory')
+                import gc
+                gc.collect()
+                torch.cuda.empty_cache()
+                out_memory=True
+            # finally only one
+            printf('[EOS]', end='\n')
+            show_text = PROMPT.postprocess(outputs[0] if outputs is not None else '### Response:')
+            return_len = len(show_text)
+            if out_memory==True:
+                out_memory=False
+                show_text+= '<p style="color:#FF0000"> [GPU Out Of Memory] </p> '
+            if return_len > 0:
+                output = PROMPT.postprocess(outputs[0], render=False)
+                history.append({
+                    'input': inputs,
+                    'output': output,
+                })
+            return_text += [(inputs, show_text)]
+            yield return_text, history
+        # common
+        else:
+            try:
+                generation_output = model.generate(
+                    input_ids=input_ids,
+                    generation_config=generation_config,
+                    return_dict_in_generate=True,
+                    output_scores=True,
+                    max_new_tokens=max_new_tokens,
+                    repetition_penalty=float(repetition_penalty),
+                )
+                s = generation_output.sequences[0]
+                output = tokenizer.decode(s)
+                output = PROMPT.postprocess(output)
+                history.append({
+                    'input': inputs,
+                    'output': output,
+                })
+                return_text += [(inputs, output)]
+                yield return_text, history
+            except torch.cuda.OutOfMemoryError:
+                import gc
+                gc.collect()
+                torch.cuda.empty_cache()
+                show_text = '<p style="color:#FF0000"> [GPU Out Of Memory] </p> '
+                printf(show_text)
+                return_text += [(inputs, show_text)]
+                yield return_text, history
+def clear():
+    import gc
+    gc.collect()
+    torch.cuda.empty_cache()
+    return None, None
+# gr.Interface对chatbot的clear有bug，因此我们重新实现了一个基于gr.block的UI逻辑
+# gr.Interface has bugs to clear chatbot's history,so we customly implement it based on gr.block
+with gr.Blocks() as demo:
+    fn = evaluate
+    title = gr.Markdown(
+        "<h1 style='text-align: center; margin-bottom: 1rem'>"
+        + "Chinese-Vicuna 中文小羊驼"
+        + "</h1>"
+    )
+    description = gr.Markdown(
+        "中文小羊驼由各种高质量的开源instruction数据集，结合Alpaca-lora的代码训练而来，模型基于开源的llama7B，主要贡献是对应的lora模型。由于代码训练资源要求较小，希望为llama中文lora社区做一份贡献。"
+    )
+    history = gr.components.State()
+    with gr.Row().style(equal_height=False):
+        with gr.Column(variant="panel"):
+            input_component_column = gr.Column()
+            with input_component_column:
+                input = gr.components.Textbox(
+                    lines=2, label="Input", placeholder="请输入问题."
+                )
+                temperature = gr.components.Slider(minimum=0, maximum=1, value=1.0, label="Temperature")
+                topp = gr.components.Slider(minimum=0, maximum=1, value=0.9, label="Top p")
+                topk = gr.components.Slider(minimum=0, maximum=100, step=1, value=60, label="Top k")
+                beam_number = gr.components.Slider(minimum=1, maximum=10, step=1, value=4, label="Beams Number")
+                max_new_token = gr.components.Slider(
+                    minimum=1, maximum=2048, step=1, value=256, label="Max New Tokens"
+                )
+                min_new_token = gr.components.Slider(
+                    minimum=1, maximum=1024, step=1, value=5, label="Min New Tokens"
+                )
+                repeat_penal = gr.components.Slider(
+                    minimum=0.1, maximum=10.0, step=0.1, value=2.0, label="Repetition Penalty"
+                )
+                max_memory = gr.components.Slider(
+                    minimum=0, maximum=2048, step=1, value=2048, label="Max Memory"
+                )
+                do_sample = gr.components.Checkbox(label="Use sample")
+                # must be str, not number !
+                type_of_prompt = gr.components.Dropdown(
+                    ['instruct', 'chat'], value=args.prompt_type, label="Prompt Type", info="select the specific prompt; use after clear history"
+                )
+                input_components = [
+                    input, history, temperature, topp, topk, beam_number, max_new_token, min_new_token, repeat_penal, max_memory, do_sample, type_of_prompt
+                ]
+                input_components_except_states = [input, temperature, topp, topk, beam_number, max_new_token, min_new_token, repeat_penal, max_memory, do_sample, type_of_prompt]
+            with gr.Row():
+                cancel_btn = gr.Button('Cancel')
+                submit_btn = gr.Button("Submit", variant="primary")
+                stop_btn = gr.Button("Stop", variant="stop", visible=False)
+            with gr.Row():
+                reset_btn = gr.Button("Reset Parameter")
+                clear_history = gr.Button("Clear History")
+        with gr.Column(variant="panel"):
+            chatbot = gr.Chatbot().style(height=1024)
+            output_components = [ chatbot, history ]
+            with gr.Row():
+                save_btn = gr.Button("Save Chat")
+        def wrapper(*args):
+            # here to support the change between the stop and submit button
+            try:
+                for output in fn(*args):
+                    output = [o for o in output]
+                    # output for output_components, the rest for [button, button]
+                    yield output + [
+                        gr.Button.update(visible=False),
+                        gr.Button.update(visible=True),
+                    ]
+            finally:
+                yield [{'__type__': 'generic_update'}, {'__type__': 'generic_update'}] + [ gr.Button.update(visible=True), gr.Button.update(visible=False)]
+        def cancel(history, chatbot):
+            if history == []:
+                return (None, None)
+            return history[:-1], chatbot[:-1]
+        extra_output = [submit_btn, stop_btn]
+        save_btn.click(
+            save,
+            input_components,
+            None,
+        )
+        pred = submit_btn.click(
+            wrapper,
+            input_components,
+            output_components + extra_output,
+            api_name="predict",
+            scroll_to_output=True,
+            preprocess=True,
+            postprocess=True,
+            batch=False,
+            max_batch_size=4,
+        )
+        submit_btn.click(
+            lambda: (
+                submit_btn.update(visible=False),
+                stop_btn.update(visible=True),
+            ),
+            inputs=None,
+            outputs=[submit_btn, stop_btn],
+            queue=False,
+        )
+        stop_btn.click(
+            lambda: (
+                submit_btn.update(visible=True),
+                stop_btn.update(visible=False),
+            ),
+            inputs=None,
+            outputs=[submit_btn, stop_btn],
+            cancels=[pred],
+            queue=False,
+        )
+        cancel_btn.click(
+            cancel,
+            inputs=[history, chatbot],
+            outputs=[history, chatbot]
+        )
+        reset_btn.click(
+            None,
+            [],
+            (
+                # input_components ; don't work for history...
+                input_components_except_states
+                + [input_component_column]
+            ),  # type: ignore
+            _js=f"""() => {json.dumps([
+                getattr(component, "cleared_value", None) for component in input_components_except_states ]
+                + ([gr.Column.update(visible=True)])
+                + ([])
+            )}
+            """,
+        )
+        clear_history.click(clear, None, [history, chatbot], queue=False)
+demo.queue().launch(share=args.share_link)

finetune.py ADDED Viewed

	@@ -0,0 +1,284 @@

+import os
+import sys
+import torch
+import torch.nn as nn
+import bitsandbytes as bnb
+from datasets import load_dataset
+import transformers
+import argparse
+import warnings
+from huggingface_hub import snapshot_download
+assert (
+    "LlamaTokenizer" in transformers._import_structure["models.llama"]
+), "LLaMA is now in HuggingFace's main branch.\nPlease reinstall it: pip uninstall transformers && pip install git+https://github.com/huggingface/transformers.git"
+from transformers import LlamaForCausalLM, LlamaTokenizer
+from peft import (
+    prepare_model_for_int8_training,
+    LoraConfig,
+    get_peft_model,
+    get_peft_model_state_dict,
+    set_peft_model_state_dict,
+)
+parser = argparse.ArgumentParser()
+parser.add_argument("--wandb", action="store_true", default=False)
+parser.add_argument("--data_path", type=str, default="merge.json")
+parser.add_argument("--output_path", type=str, default="lora-Vicuna")
+parser.add_argument("--model_path", type=str, default="decapoda-research/llama-7b-hf")
+parser.add_argument("--eval_steps", type=int, default=200)
+parser.add_argument("--save_steps", type=int, default=200)
+parser.add_argument("--test_size", type=int, default=200)
+parser.add_argument("--resume_from_checkpoint", type=str, default=None)
+parser.add_argument("--lora_remote_checkpoint", type=str, default=None)
+parser.add_argument("--ignore_data_skip", type=str, default="False")
+args = parser.parse_args()
+if not args.wandb:
+    os.environ["WANDB_MODE"] = "disable"
+# optimized for RTX 4090. for larger GPUs, increase some of these?
+MICRO_BATCH_SIZE = 4  # this could actually be 5 but i like powers of 2
+BATCH_SIZE = 128
+MAX_STEPS = None
+GRADIENT_ACCUMULATION_STEPS = BATCH_SIZE // MICRO_BATCH_SIZE
+EPOCHS = 3  # we don't always need 3 tbh
+LEARNING_RATE = 3e-4  # the Karpathy constant
+CUTOFF_LEN = 256  # 256 accounts for about 96% of the data
+LORA_R = 8
+LORA_ALPHA = 16
+LORA_DROPOUT = 0.05
+VAL_SET_SIZE = args.test_size #2000
+USE_8bit = True
+if USE_8bit is True:
+    warnings.warn("If your version of bitsandbytes>0.37.2, Please downgrade bitsandbytes's version, for example: pip install bitsandbytes==0.37.2")
+TARGET_MODULES = [
+    "q_proj",
+    "v_proj",
+]
+DATA_PATH = args.data_path
+OUTPUT_DIR = args.output_path #"lora-Vicuna"
+device_map = "auto"
+world_size = int(os.environ.get("WORLD_SIZE", 1))
+ddp = world_size != 1
+if ddp:
+    device_map = {"": int(os.environ.get("LOCAL_RANK") or 0)}
+    GRADIENT_ACCUMULATION_STEPS = GRADIENT_ACCUMULATION_STEPS // world_size
+print(args.model_path)
+model = LlamaForCausalLM.from_pretrained(
+    args.model_path,
+    load_in_8bit=USE_8bit,
+    device_map=device_map,
+)
+tokenizer = LlamaTokenizer.from_pretrained(
+    args.model_path, add_eos_token=True
+)
+if USE_8bit is True:
+    model = prepare_model_for_int8_training(model)
+config = LoraConfig(
+    r=LORA_R,
+    lora_alpha=LORA_ALPHA,
+    target_modules=TARGET_MODULES,
+    lora_dropout=LORA_DROPOUT,
+    bias="none",
+    task_type="CAUSAL_LM",
+)
+model = get_peft_model(model, config)
+tokenizer.pad_token_id = 0  # unk. we want this to be different from the eos token
+#tokenizer.padding_side = "left"  # Allow batched inference
+data = load_dataset("json", data_files=DATA_PATH)
+now_max_steps = max((len(data["train"]) - VAL_SET_SIZE) // BATCH_SIZE * EPOCHS, EPOCHS)
+if args.resume_from_checkpoint:
+    if args.lora_remote_checkpoint is not None:
+        snapshot_download(repo_id=args.lora_remote_checkpoint, allow_patterns=["*.pt", "*.bin", "*.json"], local_dir=args.resume_from_checkpoint)
+    # Check the available weights and load them
+    checkpoint_name = os.path.join(
+        args.resume_from_checkpoint, "pytorch_model.bin"
+    )  # Full checkpoint
+    if not os.path.exists(checkpoint_name):
+        pytorch_bin_path = checkpoint_name
+        checkpoint_name = os.path.join(
+            args.resume_from_checkpoint, "adapter_model.bin"
+        )  # only LoRA model - LoRA config above has to fit
+        if os.path.exists(checkpoint_name):
+            os.rename(checkpoint_name, pytorch_bin_path)
+            warnings.warn("The file name of the lora checkpoint'adapter_model.bin' is replaced with 'pytorch_model.bin'")
+        else:
+            args.resume_from_checkpoint = (
+                None  # So the trainer won't try loading its state
+            )
+    # The two files above have a different name depending on how they were saved, but are actually the same.
+    if os.path.exists(checkpoint_name):
+        print(f"Restarting from {checkpoint_name}")
+        adapters_weights = torch.load(checkpoint_name)
+        model = set_peft_model_state_dict(model, adapters_weights)
+    else:
+        print(f"Checkpoint {checkpoint_name} not found")
+    train_args_path = os.path.join(args.resume_from_checkpoint, "trainer_state.json")
+    if os.path.exists(train_args_path):
+        import json
+        base_train_args = json.load(open(train_args_path, 'r'))
+        base_max_steps = base_train_args["max_steps"]
+        resume_scale = base_max_steps / now_max_steps
+        if base_max_steps > now_max_steps:
+            warnings.warn("epoch {} replace to the base_max_steps {}".format(EPOCHS, base_max_steps))
+            EPOCHS = None
+            MAX_STEPS = base_max_steps
+        else:
+            MAX_STEPS = now_max_steps
+else:
+    MAX_STEPS = now_max_steps
+model.print_trainable_parameters()
+def generate_prompt(data_point):
+    # sorry about the formatting disaster gotta move fast
+    if data_point["input"]:
+        return f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
+### Instruction:
+{data_point["instruction"]}
+### Input:
+{data_point["input"]}
+### Response:
+{data_point["output"]}"""
+    else:
+        return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.
+### Instruction:
+{data_point["instruction"]}
+### Response:
+{data_point["output"]}"""
+def tokenize(prompt):
+    # there's probably a way to do this with the tokenizer settings
+    # but again, gotta move fast
+    result = tokenizer(
+        prompt,
+        truncation=True,
+        max_length=CUTOFF_LEN + 1,
+        padding="max_length",
+    )
+    return {
+        "input_ids": result["input_ids"][:-1],
+        "attention_mask": result["attention_mask"][:-1],
+    }
+def generate_and_tokenize_prompt(data_point):
+    # This function masks out the labels for the input,
+    # so that our loss is computed only on the response.
+    user_prompt = (
+        (
+            f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
+### Instruction:
+{data_point["instruction"]}
+### Input:
+{data_point["input"]}
+### Response:
+"""
+        )
+        if data_point["input"]
+        else (
+            f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.
+### Instruction:
+{data_point["instruction"]}
+### Response:
+"""
+        )
+    )
+    len_user_prompt_tokens = (
+        len(
+            tokenizer(
+                user_prompt,
+                truncation=True,
+                max_length=CUTOFF_LEN + 1,
+            )["input_ids"]
+        )
+        - 1
+    )  # no eos token
+    full_tokens = tokenizer(
+        user_prompt + data_point["output"],
+        truncation=True,
+        max_length=CUTOFF_LEN + 1,
+        padding="max_length",
+    )["input_ids"][:-1]
+    return {
+        "input_ids": full_tokens,
+        "labels": [-100] * len_user_prompt_tokens
+        + full_tokens[len_user_prompt_tokens:],
+        "attention_mask": [1] * (len(full_tokens)),
+    }
+if VAL_SET_SIZE > 0:
+    train_val = data["train"].train_test_split(
+        test_size=VAL_SET_SIZE, shuffle=True, seed=42
+    )
+    train_data = train_val["train"].shuffle().map(generate_and_tokenize_prompt)
+    val_data = train_val["test"].shuffle().map(generate_and_tokenize_prompt)
+else:
+    train_data = data["train"].shuffle().map(generate_and_tokenize_prompt)
+    val_data = None
+trainer = transformers.Trainer(
+    model=model,
+    train_dataset=train_data,
+    eval_dataset=val_data,
+    args=transformers.TrainingArguments(
+        per_device_train_batch_size=MICRO_BATCH_SIZE,
+        gradient_accumulation_steps=GRADIENT_ACCUMULATION_STEPS,
+        warmup_steps=100,
+        num_train_epochs=EPOCHS,
+        max_steps=MAX_STEPS,
+        learning_rate=LEARNING_RATE,
+        fp16=True,
+        logging_steps=20,
+        evaluation_strategy="steps" if VAL_SET_SIZE > 0 else "no",
+        save_strategy="steps",
+        eval_steps=args.eval_steps if VAL_SET_SIZE > 0 else None,
+        save_steps=args.save_steps,
+        output_dir=OUTPUT_DIR,
+        save_total_limit=30,
+        load_best_model_at_end=True if VAL_SET_SIZE > 0 else False,
+        ddp_find_unused_parameters=False if ddp else None,
+        report_to="wandb" if args.wandb else [],
+        ignore_data_skip=args.ignore_data_skip,
+    ),
+    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False)
+)
+model.config.use_cache = False
+old_state_dict = model.state_dict
+model.state_dict = (
+    lambda self, *_, **__: get_peft_model_state_dict(self, old_state_dict())
+).__get__(model, type(model))
+if torch.__version__ >= "2" and sys.platform != "win32":
+    model = torch.compile(model)
+print("\n If there's a warning about missing keys above, please disregard :)")
+trainer.train(resume_from_checkpoint=args.resume_from_checkpoint)
+model.save_pretrained(OUTPUT_DIR)

finetune_4bit.py ADDED Viewed

	@@ -0,0 +1,288 @@

+import os
+import sys
+import torch
+import torch.nn as nn
+import bitsandbytes as bnb
+from datasets import load_dataset, Dataset
+import transformers
+import argparse
+import warnings
+from huggingface_hub import snapshot_download
+assert (
+    "LlamaTokenizer" in transformers._import_structure["models.llama"]
+), "LLaMA is now in HuggingFace's main branch.\nPlease reinstall it: pip uninstall transformers && pip install git+https://github.com/huggingface/transformers.git"
+from transformers import LlamaForCausalLM, LlamaTokenizer, BitsAndBytesConfig
+from peft import (
+    prepare_model_for_kbit_training,
+    LoraConfig,
+    get_peft_model,
+    get_peft_model_state_dict,
+    set_peft_model_state_dict,
+)
+def generate_prompt(data_point):
+    # sorry about the formatting disaster gotta move fast
+    if data_point["input"]:
+        return f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
+### Instruction:
+{data_point["instruction"]}
+### Input:
+{data_point["input"]}
+### Response:
+{data_point["output"]}"""
+    else:
+        return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.
+### Instruction:
+{data_point["instruction"]}
+### Response:
+{data_point["output"]}"""
+def tokenize(prompt):
+    # there's probably a way to do this with the tokenizer settings
+    # but again, gotta move fast
+    result = tokenizer(
+        prompt,
+        truncation=True,
+        max_length=CUTOFF_LEN + 1,
+        padding="max_length",
+    )
+    return {
+        "input_ids": result["input_ids"][:-1],
+        "attention_mask": result["attention_mask"][:-1],
+    }
+def generate_and_tokenize_prompt(data_point):
+    # This function masks out the labels for the input,
+    # so that our loss is computed only on the response.
+    user_prompt = (
+        (
+            f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
+### Instruction:
+{data_point["instruction"]}
+### Input:
+{data_point["input"]}
+### Response:
+"""
+        )
+        if data_point["input"]
+        else (
+            f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.
+### Instruction:
+{data_point["instruction"]}
+### Response:
+"""
+        )
+    )
+    len_user_prompt_tokens = (
+        len(
+            tokenizer(
+                user_prompt,
+                truncation=True,
+                max_length=CUTOFF_LEN + 1,
+            )["input_ids"]
+        )
+        - 1
+    )  # no eos token
+    full_tokens = tokenizer(
+        user_prompt + data_point["output"],
+        truncation=True,
+        max_length=CUTOFF_LEN + 1,
+        padding="max_length",
+    )["input_ids"][:-1]
+    return {
+        "input_ids": full_tokens,
+        "labels": [-100] * len_user_prompt_tokens
+        + full_tokens[len_user_prompt_tokens:],
+        "attention_mask": [1] * (len(full_tokens)),
+    }
+parser = argparse.ArgumentParser()
+parser.add_argument("--wandb", action="store_true", default=False)
+parser.add_argument("--data_path", type=str, default="merge.json")
+parser.add_argument("--output_path", type=str, default="lora-Vicuna")
+parser.add_argument("--model_path", type=str, default="decapoda-research/llama-7b-hf")
+parser.add_argument("--eval_steps", type=int, default=200)
+parser.add_argument("--save_steps", type=int, default=200)
+parser.add_argument("--test_size", type=int, default=200)
+parser.add_argument("--resume_from_checkpoint", type=str, default=None)
+parser.add_argument("--lora_remote_checkpoint", type=str, default=None)
+parser.add_argument("--ignore_data_skip", type=str, default="False")
+args = parser.parse_args()
+if not args.wandb:
+    os.environ["WANDB_MODE"] = "disable"
+# optimized for RTX 4090. for larger GPUs, increase some of these?
+MICRO_BATCH_SIZE = 8  # this could actually be 5 but i like powers of 2
+BATCH_SIZE = 128
+MAX_STEPS = None
+GRADIENT_ACCUMULATION_STEPS = BATCH_SIZE // MICRO_BATCH_SIZE
+EPOCHS = 3  # we don't always need 3 tbh
+LEARNING_RATE = 3e-4  # the Karpathy constant
+CUTOFF_LEN = 256  # 256 accounts for about 96% of the data
+LORA_R = 8
+LORA_ALPHA = 16
+LORA_DROPOUT = 0.05
+VAL_SET_SIZE = args.test_size #2000
+TARGET_MODULES = [
+    "q_proj",
+    "v_proj",
+]
+DATA_PATH = args.data_path
+OUTPUT_DIR = args.output_path #"lora-Vicuna"
+device_map = {"": 0} #"auto"
+world_size = int(os.environ.get("WORLD_SIZE", 1))
+ddp = world_size != 1
+if ddp:
+    device_map = {"": int(os.environ.get("LOCAL_RANK") or 0)}
+    GRADIENT_ACCUMULATION_STEPS = GRADIENT_ACCUMULATION_STEPS // world_size
+print(args.model_path)
+model = LlamaForCausalLM.from_pretrained(
+    args.model_path,
+    load_in_4bit=True,
+    device_map=device_map,
+)
+tokenizer = LlamaTokenizer.from_pretrained(
+    args.model_path, add_eos_token=True
+)
+model.gradient_checkpointing_enable()
+model = prepare_model_for_kbit_training(model)
+config = LoraConfig(
+    r=LORA_R,
+    lora_alpha=LORA_ALPHA,
+    target_modules=TARGET_MODULES,
+    lora_dropout=LORA_DROPOUT,
+    bias="none",
+    task_type="CAUSAL_LM",
+)
+model = get_peft_model(model, config)
+tokenizer.pad_token_id = 0  # unk. we want this to be different from the eos token
+#tokenizer.padding_side = "left"  # Allow batched inference
+data = load_dataset("json", data_files=DATA_PATH)
+import random;start = random.randint(1, 100)
+examples = Dataset.from_dict(data['train'][start:start+5]).map(generate_and_tokenize_prompt)
+for example in examples:
+    print(f'>>> prompt example:\n { tokenizer.decode(example["input_ids"]) }')
+    print(f'>>> tokenizer labels: { tokenizer.decode([ 0 if l==-100 else l for l in example["labels"]])}')
+    print(f'>>> tokenizer example: { example["input_ids"][:250] }...{ example["input_ids"][-10:]}')
+now_max_steps = max((len(data["train"]) - VAL_SET_SIZE) // BATCH_SIZE * EPOCHS, EPOCHS)
+if args.resume_from_checkpoint:
+    if args.lora_remote_checkpoint is not None:
+        snapshot_download(repo_id=args.lora_remote_checkpoint, allow_patterns=["*.pt", "*.bin", "*.json"], local_dir=args.resume_from_checkpoint)
+    # Check the available weights and load them
+    checkpoint_name = os.path.join(
+        args.resume_from_checkpoint, "pytorch_model.bin"
+    )  # Full checkpoint
+    if not os.path.exists(checkpoint_name):
+        pytorch_bin_path = checkpoint_name
+        checkpoint_name = os.path.join(
+            args.resume_from_checkpoint, "adapter_model.bin"
+        )  # only LoRA model - LoRA config above has to fit
+        if os.path.exists(checkpoint_name):
+            os.rename(checkpoint_name, pytorch_bin_path)
+            warnings.warn("The file name of the lora checkpoint'adapter_model.bin' is replaced with 'pytorch_model.bin'")
+        else:
+            args.resume_from_checkpoint = (
+                None  # So the trainer won't try loading its state
+            )
+    # The two files above have a different name depending on how they were saved, but are actually the same.
+    if os.path.exists(checkpoint_name):
+        print(f"Restarting from {checkpoint_name}")
+        adapters_weights = torch.load(checkpoint_name)
+        model = set_peft_model_state_dict(model, adapters_weights)
+    else:
+        print(f"Checkpoint {checkpoint_name} not found")
+    train_args_path = os.path.join(args.resume_from_checkpoint, "trainer_state.json")
+    if os.path.exists(train_args_path):
+        import json
+        base_train_args = json.load(open(train_args_path, 'r'))
+        base_max_steps = base_train_args["max_steps"]
+        resume_scale = base_max_steps / now_max_steps
+        if base_max_steps > now_max_steps:
+            warnings.warn("epoch {} replace to the base_max_steps {}".format(EPOCHS, base_max_steps))
+            EPOCHS = None
+            MAX_STEPS = base_max_steps
+        else:
+            MAX_STEPS = now_max_steps
+else:
+    MAX_STEPS = now_max_steps
+model.print_trainable_parameters()
+num_proc = (os.cpu_count())
+if VAL_SET_SIZE > 0:
+    train_val = data["train"].train_test_split(
+        test_size=VAL_SET_SIZE, shuffle=True, seed=42
+    )
+    train_data = train_val["train"].shuffle().map(generate_and_tokenize_prompt, num_proc=num_proc)
+    val_data = train_val["test"].shuffle().map(generate_and_tokenize_prompt, num_proc=num_proc)
+else:
+    train_data = data["train"].shuffle().map(generate_and_tokenize_prompt, num_proc=num_proc)
+    val_data = None
+trainer = transformers.Trainer(
+    model=model,
+    train_dataset=train_data,
+    eval_dataset=val_data,
+    args=transformers.TrainingArguments(
+        per_device_train_batch_size=MICRO_BATCH_SIZE,
+        gradient_accumulation_steps=GRADIENT_ACCUMULATION_STEPS,
+        warmup_steps=100,
+        num_train_epochs=EPOCHS,
+        max_steps=MAX_STEPS,
+        learning_rate=LEARNING_RATE,
+        fp16=True,
+        logging_steps=20,
+        evaluation_strategy="steps" if VAL_SET_SIZE > 0 else "no",
+        save_strategy="steps",
+        eval_steps=args.eval_steps if VAL_SET_SIZE > 0 else None,
+        save_steps=args.save_steps,
+        output_dir=OUTPUT_DIR,
+        save_total_limit=30,
+        load_best_model_at_end=True if VAL_SET_SIZE > 0 else False,
+        ddp_find_unused_parameters=False if ddp else None,
+        report_to="wandb" if args.wandb else [],
+        ignore_data_skip=args.ignore_data_skip,
+        optim="paged_adamw_8bit",
+    ),
+    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False)
+)
+model.config.use_cache = False
+old_state_dict = model.state_dict
+model.state_dict = (
+    lambda self, *_, **__: get_peft_model_state_dict(self, old_state_dict())
+).__get__(model, type(model))
+if torch.__version__ >= "2" and sys.platform != "win32":
+    model = torch.compile(model)
+print("\n If there's a warning about missing keys above, please disregard :)")
+trainer.train(resume_from_checkpoint=args.resume_from_checkpoint)
+model.save_pretrained(OUTPUT_DIR)

finetune_chat.py ADDED Viewed

	@@ -0,0 +1,274 @@

+from peft import (
+    prepare_model_for_int8_training,
+    LoraConfig,
+    PeftModel,
+    get_peft_model,
+    get_peft_model_state_dict,
+    set_peft_model_state_dict,
+)
+from transformers import LlamaForCausalLM, LlamaTokenizer, TrainerCallback, GenerationConfig
+import os
+import sys
+import torch
+import torch.nn as nn
+import bitsandbytes as bnb
+from datasets import load_dataset, Dataset
+import transformers
+from huggingface_hub import snapshot_download
+import argparse
+import warnings
+from tqdm import tqdm
+from functools import partial
+import utils
+import prompt
+assert (
+    "LlamaTokenizer" in transformers._import_structure["models.llama"]
+), "LLaMA is now in HuggingFace's main branch.\nPlease reinstall it: pip uninstall transformers && pip install git+https://github.com/huggingface/transformers.git"
+# 0. prepare args and logger
+parser = argparse.ArgumentParser()
+parser.add_argument("--wandb", action="store_true", default=False)
+parser.add_argument("--prompt_type", type=str, default="chat")
+parser.add_argument("--data_path", type=str, default="merge.json")
+parser.add_argument("--output_path", type=str, default="lora-Vicuna")
+parser.add_argument("--model_path", type=str, default="decapoda-research/llama-7b-hf")
+parser.add_argument("--num_epoch", type=int, default=3)
+parser.add_argument("--micro_batch", type=int, default=4)
+parser.add_argument("--total_batch", type=int, default=128)
+parser.add_argument("--log_steps", type=int, default=100)
+parser.add_argument("--eval_steps", type=int, default=200)
+parser.add_argument("--save_steps", type=int, default=200)
+parser.add_argument("--warmup_ratio", type=float, default=0.05)
+parser.add_argument("--test_size", type=int, default=200)
+parser.add_argument("--resume_from_checkpoint", type=str, default=None)
+parser.add_argument("--lora_remote_checkpoint", type=str, default=None)
+parser.add_argument("--ignore_data_skip", type=bool, default=False)
+args = parser.parse_args()
+if not args.wandb:
+    os.environ["WANDB_MODE"] = "disable"
+MICRO_BATCH_SIZE = args.micro_batch  # this could actually be 5 but i like powers of 2
+BATCH_SIZE = args.total_batch
+MAX_STEPS = None
+GRADIENT_ACCUMULATION_STEPS = BATCH_SIZE // MICRO_BATCH_SIZE
+EPOCHS = args.num_epoch
+LEARNING_RATE = 3e-4  # the Karpathy constant
+CUTOFF_LEN = 2048
+LORA_R = 8
+LORA_ALPHA = 16
+LORA_DROPOUT = 0.05
+USE_8bit = True
+VAL_SET_SIZE = args.test_size  # 2000
+TARGET_MODULES = [
+    "q_proj",
+    "v_proj",
+    "k_proj",
+    "o_proj",
+    "down_proj",
+    "gate_proj",
+    "up_proj",
+]
+DATA_PATH = args.data_path
+OUTPUT_DIR = args.output_path  # "lora-Vicuna"
+device_map = "auto"
+world_size = int(os.environ.get("WORLD_SIZE", 1))
+ddp = world_size != 1
+if ddp:
+    device_map = {"": int(os.environ.get("LOCAL_RANK") or 0)}
+    GRADIENT_ACCUMULATION_STEPS = GRADIENT_ACCUMULATION_STEPS // world_size
+# we must make sure batch_size and gradient_accumulation_steps not changed for resuming training.
+if args.resume_from_checkpoint:
+    old_args_path = os.path.join(args.resume_from_checkpoint, 'training_args.bin')
+    if os.path.exists(old_args_path):
+        old_args = torch.load(old_args_path)
+        if MICRO_BATCH_SIZE != old_args.per_device_train_batch_size:
+            raise Exception(
+                f'current micro batch size {MICRO_BATCH_SIZE} is not equal to the old {old_args.per_device_train_batch_size},'
+                ' This will cause the trainer skips wrong epochs or steps.'
+                f'please change your micro batch size to {old_args.per_device_train_batch_size}'
+                ' or cancel resuming your training'
+                )
+        if GRADIENT_ACCUMULATION_STEPS != old_args.gradient_accumulation_steps:
+            raise Exception(
+                f'current total batch {BATCH_SIZE} is not equal to the old {old_args.gradient_accumulation_steps*old_args.per_device_train_batch_size},'
+                ' This will cause the trainer skips wrong epochs or steps.'
+                f'please change your total batch size to {old_args.gradient_accumulation_steps*old_args.per_device_train_batch_size}'
+                ' or cancel resuming your training'
+            )
+    else:
+        raise Exception(f'{old_args_path} is not exist!')
+    # checkpoint = os.path.join(args.resume_from_checkpoint, 'pytorch_model.bin')
+logger = utils.set_file_logger(__name__,OUTPUT_DIR)
+# 1. load dataset
+logger.info(f'>>> processing data from {DATA_PATH}')
+logger.info(f'>>> using {args}')
+train_tokenizer = LlamaTokenizer.from_pretrained(args.model_path, add_eos_token=True)
+assert train_tokenizer.eos_token_id == 2, "Tokenizer eos is wrong!!!"
+# unk. we want this to be different from the eos token
+train_tokenizer.pad_token_id = 0
+# cannot use eos in generation!
+# tokenizer.padding_side = "left"  # Allow batched inference
+test_tokenizer = LlamaTokenizer.from_pretrained(args.model_path)
+if args.prompt_type == 'instruct':
+    PROMPT = prompt.instruct_prompt(train_tokenizer, CUTOFF_LEN)
+elif args.prompt_type == 'chat':
+    PROMPT = prompt.chat_prompt(train_tokenizer,CUTOFF_LEN)
+else:
+    raise Exception('not support')
+# check tokenizer
+data = load_dataset('json', data_files=DATA_PATH)
+import random;start = random.randint(1, 100)
+examples = Dataset.from_dict(data['train'][start:start+5]).map(PROMPT.preprocess_train)
+for example in examples:
+    logger.info(f'>>> using prompt {args.prompt_type}, prompt example:\n { train_tokenizer.decode(example["input_ids"]) }')
+    logger.info(f'>>> tokenizer labels: { train_tokenizer.decode([ 0 if l==-100 else l for l in example["labels"]])}')
+    logger.info(f'>>> tokenizer example: { example["input_ids"][:10] }...{ example["input_ids"][-10:]}')
+# 2. load model and checkpoints
+logger.info(f'>>> load model from {args.model_path}')
+if USE_8bit is True:
+    assert bnb.__version__ >= '0.37.2', "Please downgrade bitsandbytes's version, for example: pip install bitsandbytes==0.37.2"
+model = LlamaForCausalLM.from_pretrained(
+    args.model_path,
+    load_in_8bit=USE_8bit,
+    device_map=device_map,
+    torch_dtype=torch.float16,
+)
+if USE_8bit is True:
+    model = prepare_model_for_int8_training(model)
+config = LoraConfig(
+    r=LORA_R,
+    lora_alpha=LORA_ALPHA,
+    target_modules=TARGET_MODULES,
+    lora_dropout=LORA_DROPOUT,
+    bias="none",
+    task_type="CAUSAL_LM",
+)
+model = get_peft_model(model, config)
+if args.resume_from_checkpoint:
+    checkpoint_name = os.path.join(args.resume_from_checkpoint, "pytorch_model.bin")
+    # adapter_model.bin
+    if not os.path.exists(checkpoint_name):
+        pytorch_bin_path = checkpoint_name
+        checkpoint_name = os.path.join(args.resume_from_checkpoint, "adapter_model.bin")
+        if os.path.exists(checkpoint_name):
+            os.rename(checkpoint_name, pytorch_bin_path)
+            logger.warning("The file name of the lora checkpoint'adapter_model.bin' is replaced with 'pytorch_model.bin'")
+        else:
+            args.resume_from_checkpoint = None  # So the trainer won't try loading its state
+    # pytorch_model.bin
+    if os.path.exists(checkpoint_name):
+        logger.info(f'>>> load lora from {checkpoint_name}')
+        adapters_weights = torch.load(checkpoint_name)
+        set_peft_model_state_dict(model, adapters_weights)
+    else:
+        raise Exception(f"Checkpoint {checkpoint_name} not found with resume_from_checkpoint=True!")
+trainable_params = 0
+all_param = 0
+for _, param in model.named_parameters():
+    num_params = param.numel()
+    # if using DS Zero 3 and the weights are initialized empty
+    if num_params == 0 and hasattr(param, "ds_numel"):
+        num_params = param.ds_numel
+    all_param += num_params
+    if param.requires_grad:
+        trainable_params += num_params
+logger.info(f">>> trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}")
+# 3. speedup dataset processing by multi-process
+num_proc = (os.cpu_count())
+if VAL_SET_SIZE > 0:
+    train_val = data["train"].train_test_split(test_size=VAL_SET_SIZE, shuffle=True, seed=42)
+    train_data = train_val["train"].shuffle().map(PROMPT.preprocess_train, num_proc=num_proc)
+    val_data = train_val["test"].shuffle().map(PROMPT.preprocess_train, num_proc=num_proc)
+else:
+    train_data = data["train"].shuffle().map(PROMPT.preprocess_train, num_proc=num_proc)
+    val_data = None
+now_max_steps = max((len(data["train"]) - VAL_SET_SIZE) // BATCH_SIZE * EPOCHS, EPOCHS)
+if args.resume_from_checkpoint:
+    # the trainer will ignore the state max_steps and caculate max_steps based on epochs,
+    # so we mannally set the args.max_step to override it.
+    if args.lora_remote_checkpoint is not None:
+        snapshot_download(repo_id=args.lora_remote_checkpoint, allow_patterns=["*.pt", "*.bin", "*.json"], local_dir=args.resume_from_checkpoint)
+    train_state_path = os.path.join(args.resume_from_checkpoint, "trainer_state.json")
+    if os.path.exists(train_state_path):
+        import json
+        base_train_args = json.load(open(train_state_path, 'r'))
+        base_max_steps = base_train_args["max_steps"]
+        resume_scale = base_max_steps / now_max_steps
+        if base_max_steps > now_max_steps:
+            logger.warning(f"epoch {EPOCHS}:{MAX_STEPS} replace to the base_max_steps {base_max_steps}")
+            EPOCHS = None
+            MAX_STEPS = base_max_steps
+        else:
+            MAX_STEPS = now_max_steps
+    assert MAX_STEPS is not None
+else:
+    MAX_STEPS = now_max_steps
+# 4. start training
+class CustomCallback(TrainerCallback):
+    def __init__(self, trainer) -> None:
+        super().__init__()
+        self.trainer = trainer
+        self.generation_config = GenerationConfig(
+            temperature=1.0,
+            top_p=0.75,
+            top_k=40,
+            num_beams=2,
+            bos_token_id=train_tokenizer.bos_token_id,
+            eos_token_id=train_tokenizer.eos_token_id,
+            pad_token_id=train_tokenizer.pad_token_id,
+            max_new_tokens=1024, # max_length=max_new_tokens+input_sequence
+            min_new_tokens=1, # min_length=min_new_tokens+input_sequence
+            bad_words_ids=test_tokenizer(['\n\nUser:','\n\nAssistant:'], add_special_tokens=False).input_ids
+        )
+        self.repetition_penalty=1.3
+        self.logger = utils.set_file_logger('transformers.trainer', trainer.args.output_dir)
+    def on_log(self, args, state, control, logs, **kwargs):
+        logger.info(logs)
+trainer = transformers.Trainer(
+    model=model,
+    train_dataset=train_data,
+    eval_dataset=val_data,
+    args=transformers.TrainingArguments(
+        per_device_train_batch_size=MICRO_BATCH_SIZE,
+        gradient_accumulation_steps=GRADIENT_ACCUMULATION_STEPS,
+        warmup_ratio=args.warmup_ratio,
+        num_train_epochs=EPOCHS,
+        max_steps=MAX_STEPS,
+        learning_rate=LEARNING_RATE,
+        fp16=True,
+        logging_steps=args.log_steps,
+        logging_first_step=True, # convenient
+        evaluation_strategy="steps" if VAL_SET_SIZE > 0 else "no",
+        save_strategy="steps",
+        eval_steps=args.eval_steps if VAL_SET_SIZE > 0 else None,
+        save_steps=args.save_steps,
+        output_dir=OUTPUT_DIR,
+        load_best_model_at_end=True if VAL_SET_SIZE > 0 else False,
+        ddp_find_unused_parameters=False if ddp else None,
+        report_to="wandb" if args.wandb else [],
+        ignore_data_skip=args.ignore_data_skip,
+    ),
+    data_collator=PROMPT.data_collator()
+)
+trainer.add_callback(CustomCallback(trainer))
+model.config.use_cache = False
+old_state_dict = model.state_dict
+model.state_dict = (
+    lambda self, *_, **__: get_peft_model_state_dict(self, old_state_dict())
+).__get__(model, type(model))
+if torch.__version__ >= "2" and sys.platform != "win32":
+    model = torch.compile(model)
+trainer.train(resume_from_checkpoint=args.resume_from_checkpoint)
+model.save_pretrained(OUTPUT_DIR)

finetune_fp16.py ADDED Viewed

	@@ -0,0 +1,292 @@

+import os
+import sys
+import torch
+import torch.nn as nn
+import bitsandbytes as bnb
+from datasets import load_dataset
+import transformers
+import argparse
+import warnings
+from huggingface_hub import snapshot_download
+assert (
+    "LlamaTokenizer" in transformers._import_structure["models.llama"]
+), "LLaMA is now in HuggingFace's main branch.\nPlease reinstall it: pip uninstall transformers && pip install git+https://github.com/huggingface/transformers.git"
+from transformers import LlamaForCausalLM, LlamaTokenizer
+from peft import (
+    prepare_model_for_int8_training,
+    LoraConfig,
+    get_peft_model,
+    get_peft_model_state_dict,
+    set_peft_model_state_dict,
+)
+def get_peft_state_maybe_zero_3(state_dict, bias):
+    if hasattr(param, "ds_id"):
+        assert param.ds_status == ZeroParamStatus.NOT_AVAILABLE
+        with zero.GatheredParameters([param]):
+            param = param.data.cpu().clone().detach()
+    to_return = {k: maybe_zero_3(v) for k, v in to_return.items()}
+    return to_return
+parser = argparse.ArgumentParser()
+parser.add_argument("--wandb", action="store_true", default=False)
+parser.add_argument("--data_path", type=str, default="merge.json")
+parser.add_argument("--output_path", type=str, default="lora-Vicuna")
+parser.add_argument("--model_path", type=str, default="decapoda-research/llama-7b-hf")
+parser.add_argument("--eval_steps", type=int, default=200)
+parser.add_argument("--save_steps", type=int, default=200)
+parser.add_argument("--test_size", type=int, default=200)
+parser.add_argument("--resume_from_checkpoint", type=str, default=None)
+parser.add_argument("--ignore_data_skip", type=str, default="False")
+parser.add_argument("--lora_remote_checkpoint", type=str, default=None)
+parser.add_argument("--local_rank", type=int, default=-1)
+parser.add_argument("--deepspeed", action="store_true", default=False)
+args = parser.parse_args()
+if not args.wandb:
+    os.environ["WANDB_MODE"] = "disable"
+# optimized for RTX 4090. for larger GPUs, increase some of these?
+MICRO_BATCH_SIZE = 2  # this could actually be 5 but i like powers of 2
+BATCH_SIZE = 128
+MAX_STEPS = None
+GRADIENT_ACCUMULATION_STEPS = BATCH_SIZE // MICRO_BATCH_SIZE
+EPOCHS = 3  # we don't always need 3 tbh
+LEARNING_RATE = 3e-4  # the Karpathy constant
+CUTOFF_LEN = 256  # 256 accounts for about 96% of the data
+LORA_R = 8
+LORA_ALPHA = 16
+LORA_DROPOUT = 0.05
+VAL_SET_SIZE = args.test_size #2000
+TARGET_MODULES = [
+    "q_proj",
+    "v_proj",
+]
+DATA_PATH = args.data_path
+OUTPUT_DIR = args.output_path #"lora-Vicuna"
+device_map = {"": 0} #"auto"
+world_size = int(os.environ.get("WORLD_SIZE", 1))
+ddp = world_size != 1
+if ddp:
+    device_map = {"": int(os.environ.get("LOCAL_RANK") or 0)}
+    GRADIENT_ACCUMULATION_STEPS = GRADIENT_ACCUMULATION_STEPS // world_size
+print(args.model_path)
+model = LlamaForCausalLM.from_pretrained(
+    args.model_path,
+    load_in_8bit=False,
+    torch_dtype=torch.float16,
+    device_map=device_map,
+).half()
+tokenizer = LlamaTokenizer.from_pretrained(
+    args.model_path, add_eos_token=True
+)
+#model = prepare_model_for_int8_training(model)
+config = LoraConfig(
+    r=LORA_R,
+    lora_alpha=LORA_ALPHA,
+    target_modules=TARGET_MODULES,
+    lora_dropout=LORA_DROPOUT,
+    bias="none",
+    task_type="CAUSAL_LM",
+)
+model = get_peft_model(model, config)
+tokenizer.pad_token_id = 0  # unk. we want this to be different from the eos token
+#tokenizer.padding_side = "left"  # Allow batched inference
+data = load_dataset("json", data_files=DATA_PATH)
+now_max_steps = max((len(data["train"]) - VAL_SET_SIZE) // BATCH_SIZE * EPOCHS, EPOCHS)
+if args.resume_from_checkpoint:
+    if args.lora_remote_checkpoint is not None:
+        snapshot_download(repo_id=args.lora_remote_checkpoint, allow_patterns=["*.pt", "*.bin", "*.json"], local_dir=args.resume_from_checkpoint)
+    # Check the available weights and load them
+    checkpoint_name = os.path.join(
+        args.resume_from_checkpoint, "pytorch_model.bin"
+    )  # Full checkpoint
+    if not os.path.exists(checkpoint_name):
+        pytorch_bin_path = checkpoint_name
+        checkpoint_name = os.path.join(
+            args.resume_from_checkpoint, "adapter_model.bin"
+        )  # only LoRA model - LoRA config above has to fit
+        if os.path.exists(checkpoint_name):
+            os.rename(checkpoint_name, pytorch_bin_path)
+            warnings.warn("The file name of the lora checkpoint'adapter_model.bin' is replaced with 'pytorch_model.bin'")
+        else:
+            args.resume_from_checkpoint = (
+                None  # So the trainer won't try loading its state
+            )
+    # The two files above have a different name depending on how they were saved, but are actually the same.
+    if os.path.exists(checkpoint_name):
+        print(f"Restarting from {checkpoint_name}")
+        adapters_weights = torch.load(checkpoint_name)
+        model = set_peft_model_state_dict(model, adapters_weights)
+    else:
+        print(f"Checkpoint {checkpoint_name} not found")
+    train_args_path = os.path.join(args.resume_from_checkpoint, "trainer_state.json")
+    if os.path.exists(train_args_path):
+        import json
+        base_train_args = json.load(open(train_args_path, 'r'))
+        base_max_steps = base_train_args["max_steps"]
+        resume_scale = base_max_steps / now_max_steps
+        if base_max_steps > now_max_steps:
+            warnings.warn("epoch {} replace to the base_max_steps {}".format(EPOCHS, base_max_steps))
+            EPOCHS = None
+            MAX_STEPS = base_max_steps
+        else:
+            MAX_STEPS = now_max_steps
+else:
+    MAX_STEPS = now_max_steps
+model.print_trainable_parameters()
+def generate_prompt(data_point):
+    # sorry about the formatting disaster gotta move fast
+    if data_point["input"]:
+        return f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
+### Instruction:
+{data_point["instruction"]}
+### Input:
+{data_point["input"]}
+### Response:
+{data_point["output"]}"""
+    else:
+        return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.
+### Instruction:
+{data_point["instruction"]}
+### Response:
+{data_point["output"]}"""
+def tokenize(prompt):
+    # there's probably a way to do this with the tokenizer settings
+    # but again, gotta move fast
+    result = tokenizer(
+        prompt,
+        truncation=True,
+        max_length=CUTOFF_LEN + 1,
+        padding="max_length",
+    )
+    return {
+        "input_ids": result["input_ids"][:-1],
+        "attention_mask": result["attention_mask"][:-1],
+    }
+def generate_and_tokenize_prompt(data_point):
+    # This function masks out the labels for the input,
+    # so that our loss is computed only on the response.
+    user_prompt = (
+        (
+            f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
+### Instruction:
+{data_point["instruction"]}
+### Input:
+{data_point["input"]}
+### Response:
+"""
+        )
+        if data_point["input"]
+        else (
+            f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.
+### Instruction:
+{data_point["instruction"]}
+### Response:
+"""
+        )
+    )
+    len_user_prompt_tokens = (
+        len(
+            tokenizer(
+                user_prompt,
+                truncation=True,
+                max_length=CUTOFF_LEN + 1,
+            )["input_ids"]
+        )
+        - 1
+    )  # no eos token
+    full_tokens = tokenizer(
+        user_prompt + data_point["output"],
+        truncation=True,
+        max_length=CUTOFF_LEN + 1,
+        padding="max_length",
+    )["input_ids"][:-1]
+    return {
+        "input_ids": full_tokens,
+        "labels": [-100] * len_user_prompt_tokens
+        + full_tokens[len_user_prompt_tokens:],
+        "attention_mask": [1] * (len(full_tokens)),
+    }
+if VAL_SET_SIZE > 0:
+    train_val = data["train"].train_test_split(
+        test_size=VAL_SET_SIZE, shuffle=True, seed=42
+    )
+    train_data = train_val["train"].shuffle().map(generate_and_tokenize_prompt)
+    val_data = train_val["test"].shuffle().map(generate_and_tokenize_prompt)
+else:
+    train_data = data["train"].shuffle().map(generate_and_tokenize_prompt)
+    val_data = None
+trainer = transformers.Trainer(
+    model=model,
+    train_dataset=train_data,
+    eval_dataset=val_data,
+    args=transformers.TrainingArguments(
+        per_device_train_batch_size=MICRO_BATCH_SIZE,
+        gradient_accumulation_steps=GRADIENT_ACCUMULATION_STEPS,
+        warmup_steps=100,
+        num_train_epochs=EPOCHS,
+        max_steps=MAX_STEPS,
+        learning_rate=LEARNING_RATE,
+        fp16=True,
+        logging_steps=20,
+        evaluation_strategy="steps" if VAL_SET_SIZE > 0 else "no",
+        save_strategy="steps",
+        eval_steps=args.eval_steps if VAL_SET_SIZE > 0 else None,
+        save_steps=args.save_steps,
+        output_dir=OUTPUT_DIR,
+        save_total_limit=30,
+        load_best_model_at_end=True if VAL_SET_SIZE > 0 else False,
+        ddp_find_unused_parameters=False if ddp else None,
+        report_to="wandb" if args.wandb else [],
+        ignore_data_skip=args.ignore_data_skip,
+        deepspeed="sample/zero_config.json" if args.deepspeed else None,
+    ),
+    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False)
+)
+model.config.use_cache = False
+old_state_dict = model.state_dict
+model.state_dict = (
+    lambda self, *_, **__: get_peft_model_state_dict(self, old_state_dict())
+).__get__(model, type(model))
+if torch.__version__ >= "2" and sys.platform != "win32":
+    model = torch.compile(model)
+print("\n If there's a warning about missing keys above, please disregard :)")
+trainer.train(resume_from_checkpoint=args.resume_from_checkpoint)
+model.save_pretrained(OUTPUT_DIR)

generate.py ADDED Viewed

	@@ -0,0 +1,201 @@

+import sys
+import torch
+from peft import PeftModel, PeftModelForCausalLM, LoraConfig
+import transformers
+import gradio as gr
+import argparse
+import warnings
+import os
+from utils import StreamPeftGenerationMixin,StreamLlamaForCausalLM
+assert (
+    "LlamaTokenizer" in transformers._import_structure["models.llama"]
+), "LLaMA is now in HuggingFace's main branch.\nPlease reinstall it: pip uninstall transformers && pip install git+https://github.com/huggingface/transformers.git"
+from transformers import LlamaTokenizer, LlamaForCausalLM, GenerationConfig
+parser = argparse.ArgumentParser()
+parser.add_argument("--model_path", type=str, default="/model/13B_hf")
+parser.add_argument("--lora_path", type=str, default="checkpoint-3000")
+parser.add_argument("--use_typewriter", type=int, default=1)
+parser.add_argument("--use_local", type=int, default=1)
+args = parser.parse_args()
+print(args)
+tokenizer = LlamaTokenizer.from_pretrained(args.model_path)
+LOAD_8BIT = True
+BASE_MODEL = args.model_path
+LORA_WEIGHTS = args.lora_path
+# fix the path for local checkpoint
+lora_bin_path = os.path.join(args.lora_path, "adapter_model.bin")
+print(lora_bin_path)
+if not os.path.exists(lora_bin_path) and args.use_local:
+    pytorch_bin_path = os.path.join(args.lora_path, "pytorch_model.bin")
+    print(pytorch_bin_path)
+    if os.path.exists(pytorch_bin_path):
+        os.rename(pytorch_bin_path, lora_bin_path)
+        warnings.warn(
+            "The file name of the lora checkpoint'pytorch_model.bin' is replaced with 'adapter_model.bin'"
+        )
+    else:
+        assert ('Checkpoint is not Found!')
+if torch.cuda.is_available():
+    device = "cuda"
+else:
+    device = "cpu"
+try:
+    if torch.backends.mps.is_available():
+        device = "mps"
+except:
+    pass
+if device == "cuda":
+    model = LlamaForCausalLM.from_pretrained(
+        BASE_MODEL,
+        load_in_8bit=LOAD_8BIT,
+        torch_dtype=torch.float16,
+        device_map="auto", #device_map={"": 0},
+    )
+    model = StreamPeftGenerationMixin.from_pretrained(
+        model, LORA_WEIGHTS, torch_dtype=torch.float16, device_map="auto", #device_map={"": 0}
+    )
+elif device == "mps":
+    model = LlamaForCausalLM.from_pretrained(
+        BASE_MODEL,
+        device_map={"": device},
+        torch_dtype=torch.float16,
+    )
+    model = StreamPeftGenerationMixin.from_pretrained(
+        model,
+        LORA_WEIGHTS,
+        device_map={"": device},
+        torch_dtype=torch.float16,
+    )
+else:
+    model = LlamaForCausalLM.from_pretrained(
+        BASE_MODEL, device_map={"": device}, low_cpu_mem_usage=True
+    )
+    model = StreamPeftGenerationMixin.from_pretrained(
+        model,
+        LORA_WEIGHTS,
+        device_map={"": device},
+    )
+def generate_prompt(instruction, input=None):
+    if input:
+        return f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
+### Instruction:
+{instruction}
+### Input:
+{input}
+### Response:"""
+    else:
+        return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.
+### Instruction:
+{instruction}
+### Response:"""
+if not LOAD_8BIT:
+    model.half()  # seems to fix bugs for some users.
+model.eval()
+if torch.__version__ >= "2" and sys.platform != "win32":
+    model = torch.compile(model)
+def evaluate(
+    input,
+    temperature=0.1,
+    top_p=0.75,
+    top_k=40,
+    num_beams=4,
+    max_new_tokens=128,
+    min_new_tokens=1,
+    repetition_penalty=2.0,
+    **kwargs,
+):
+    prompt = generate_prompt(input)
+    inputs = tokenizer(prompt, return_tensors="pt")
+    input_ids = inputs["input_ids"].to(device)
+    generation_config = GenerationConfig(
+        temperature=temperature,
+        top_p=top_p,
+        top_k=top_k,
+        num_beams=num_beams,
+        bos_token_id=1,
+        eos_token_id=2,
+        pad_token_id=0,
+        max_new_tokens=max_new_tokens, # max_length=max_new_tokens+input_sequence
+        min_new_tokens=min_new_tokens, # min_length=min_new_tokens+input_sequence
+        **kwargs,
+    )
+    with torch.no_grad():
+        if args.use_typewriter:
+            for generation_output in model.stream_generate(
+                input_ids=input_ids,
+                generation_config=generation_config,
+                return_dict_in_generate=True,
+                output_scores=False,
+                repetition_penalty=float(repetition_penalty),
+            ):
+                outputs = tokenizer.batch_decode(generation_output)
+                show_text = "\n--------------------------------------------\n".join(
+                    [output.split("### Response:")[1].strip().replace('�','')+" ▌" for output in outputs]
+                )
+                # if show_text== '':
+                #     yield last_show_text
+                # else:
+                yield show_text
+            yield outputs[0].split("### Response:")[1].strip().replace('�','')
+        else:
+            generation_output = model.generate(
+                input_ids=input_ids,
+                generation_config=generation_config,
+                return_dict_in_generate=True,
+                output_scores=False,
+                repetition_penalty=1.3,
+            )
+            output = generation_output.sequences[0]
+            output = tokenizer.decode(output).split("### Response:")[1].strip()
+            print(output)
+            yield output
+gr.Interface(
+    fn=evaluate,
+    inputs=[
+        gr.components.Textbox(
+            lines=2, label="Input", placeholder="Tell me about alpacas."
+        ),
+        gr.components.Slider(minimum=0, maximum=1, value=0.1, label="Temperature"),
+        gr.components.Slider(minimum=0, maximum=1, value=0.75, label="Top p"),
+        gr.components.Slider(minimum=0, maximum=100, step=1, value=40, label="Top k"),
+        gr.components.Slider(minimum=1, maximum=10, step=1, value=4, label="Beams Number"),
+        gr.components.Slider(
+            minimum=1, maximum=2000, step=1, value=256, label="Max New Tokens"
+        ),
+        gr.components.Slider(
+            minimum=1, maximum=300, step=1, value=1, label="Min New Tokens"
+        ),
+        gr.components.Slider(
+            minimum=0.1, maximum=10.0, step=0.1, value=2.0, label="Repetition Penalty"
+        ),
+    ],
+    outputs=[
+        gr.inputs.Textbox(
+            lines=25,
+            label="Output",
+        )
+    ],
+    title="Chinese-Vicuna 中文小羊驼",
+    description="中文小羊驼由各种高质量的开源instruction数据集，结合Alpaca-lora的代码训练而来，模型基于开源的llama7B，主要贡献是对应的lora模型。由于代码训练资源要求较小，希望为llama中文lora社区做一份贡献。",
+).queue().launch(share=True)

generate_4bit.py ADDED Viewed

	@@ -0,0 +1,210 @@

+import sys
+import torch
+from peft import PeftModel, PeftModelForCausalLM, LoraConfig
+import transformers
+import gradio as gr
+import argparse
+import warnings
+import os
+from utils import StreamPeftGenerationMixin,StreamLlamaForCausalLM
+assert (
+    "LlamaTokenizer" in transformers._import_structure["models.llama"]
+), "LLaMA is now in HuggingFace's main branch.\nPlease reinstall it: pip uninstall transformers && pip install git+https://github.com/huggingface/transformers.git"
+from transformers import LlamaTokenizer, LlamaForCausalLM, GenerationConfig, BitsAndBytesConfig
+parser = argparse.ArgumentParser()
+parser.add_argument("--model_path", type=str, default="/model/13B_hf")
+parser.add_argument("--lora_path", type=str, default="checkpoint-3000")
+parser.add_argument("--use_typewriter", type=int, default=1)
+parser.add_argument("--use_local", type=int, default=1)
+args = parser.parse_args()
+print(args)
+tokenizer = LlamaTokenizer.from_pretrained(args.model_path)
+LOAD_8BIT = True
+BASE_MODEL = args.model_path
+LORA_WEIGHTS = args.lora_path
+# fix the path for local checkpoint
+lora_bin_path = os.path.join(args.lora_path, "adapter_model.bin")
+print(lora_bin_path)
+if not os.path.exists(lora_bin_path) and args.use_local:
+    pytorch_bin_path = os.path.join(args.lora_path, "pytorch_model.bin")
+    print(pytorch_bin_path)
+    if os.path.exists(pytorch_bin_path):
+        os.rename(pytorch_bin_path, lora_bin_path)
+        warnings.warn(
+            "The file name of the lora checkpoint'pytorch_model.bin' is replaced with 'adapter_model.bin'"
+        )
+    else:
+        assert ('Checkpoint is not Found!')
+if torch.cuda.is_available():
+    device = "cuda"
+else:
+    device = "cpu"
+try:
+    if torch.backends.mps.is_available():
+        device = "mps"
+except:
+    pass
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.float16
+)
+if device == "cuda":
+    model = LlamaForCausalLM.from_pretrained(
+        BASE_MODEL,
+        quantization_config=bnb_config,
+        torch_dtype=torch.float16,
+        device_map="auto", #{"": 0},
+    )
+    model = StreamPeftGenerationMixin.from_pretrained(
+        model, LORA_WEIGHTS, torch_dtype=torch.float16, device_map="auto", #{"": 0}
+    )
+elif device == "mps":
+    model = LlamaForCausalLM.from_pretrained(
+        BASE_MODEL,
+        device_map={"": device},
+        torch_dtype=torch.float16,
+    )
+    model = StreamPeftGenerationMixin.from_pretrained(
+        model,
+        LORA_WEIGHTS,
+        device_map={"": device},
+        torch_dtype=torch.float16,
+    )
+else:
+    model = LlamaForCausalLM.from_pretrained(
+        BASE_MODEL, device_map={"": device}, low_cpu_mem_usage=True
+    )
+    model = StreamPeftGenerationMixin.from_pretrained(
+        model,
+        LORA_WEIGHTS,
+        device_map={"": device},
+    )
+model.config.bos_token_id = tokenizer.bos_token_id = 1
+model.config.eos_token_id = tokenizer.eos_token_id = 2
+def generate_prompt(instruction, input=None):
+    if input:
+        return f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
+### Instruction:
+{instruction}
+### Input:
+{input}
+### Response:"""
+    else:
+        return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.
+### Instruction:
+{instruction}
+### Response:"""
+if not LOAD_8BIT:
+    model.half()  # seems to fix bugs for some users.
+model.eval()
+if torch.__version__ >= "2" and sys.platform != "win32":
+    model = torch.compile(model)
+def evaluate(
+    input,
+    temperature=0.1,
+    top_p=0.75,
+    top_k=40,
+    num_beams=4,
+    max_new_tokens=128,
+    min_new_tokens=1,
+    repetition_penalty=2.0,
+    **kwargs,
+):
+    prompt = generate_prompt(input)
+    inputs = tokenizer(prompt, return_tensors="pt")
+    input_ids = inputs["input_ids"].to(device)
+    generation_config = GenerationConfig(
+        temperature=temperature,
+        top_p=top_p,
+        top_k=top_k,
+        num_beams=num_beams,
+        bos_token_id=1,
+        eos_token_id=2,
+        pad_token_id=0,
+        max_new_tokens=max_new_tokens, # max_length=max_new_tokens+input_sequence
+        min_new_tokens=min_new_tokens, # min_length=min_new_tokens+input_sequence
+        **kwargs,
+    )
+    with torch.no_grad():
+        if args.use_typewriter:
+            for generation_output in model.stream_generate(
+                input_ids=input_ids,
+                generation_config=generation_config,
+                return_dict_in_generate=True,
+                output_scores=False,
+                repetition_penalty=float(repetition_penalty),
+            ):
+                outputs = tokenizer.batch_decode(generation_output)
+                show_text = "\n--------------------------------------------\n".join(
+                    [output.split("### Response:")[1].strip().replace('�','')+" ▌" for output in outputs]
+                )
+                # if show_text== '':
+                #     yield last_show_text
+                # else:
+                yield show_text
+            yield outputs[0].split("### Response:")[1].strip().replace('�','')
+        else:
+            generation_output = model.generate(
+                input_ids=input_ids,
+                generation_config=generation_config,
+                return_dict_in_generate=True,
+                output_scores=False,
+                repetition_penalty=1.3,
+            )
+            output = generation_output.sequences[0]
+            output = tokenizer.decode(output).split("### Response:")[1].strip()
+            print(output)
+            yield output
+gr.Interface(
+    fn=evaluate,
+    inputs=[
+        gr.components.Textbox(
+            lines=2, label="Input", placeholder="Tell me about alpacas."
+        ),
+        gr.components.Slider(minimum=0, maximum=1, value=0.1, label="Temperature"),
+        gr.components.Slider(minimum=0, maximum=1, value=0.75, label="Top p"),
+        gr.components.Slider(minimum=0, maximum=100, step=1, value=40, label="Top k"),
+        gr.components.Slider(minimum=1, maximum=10, step=1, value=4, label="Beams Number"),
+        gr.components.Slider(
+            minimum=1, maximum=2000, step=1, value=256, label="Max New Tokens"
+        ),
+        gr.components.Slider(
+            minimum=1, maximum=300, step=1, value=1, label="Min New Tokens"
+        ),
+        gr.components.Slider(
+            minimum=0.1, maximum=10.0, step=0.1, value=2.0, label="Repetition Penalty"
+        ),
+    ],
+    outputs=[
+        gr.inputs.Textbox(
+            lines=25,
+            label="Output",
+        )
+    ],
+    title="Chinese-Vicuna 中文小羊驼",
+    description="中文小羊驼由各种高质量的开源instruction数据集，结合Alpaca-lora的代码训练而来，模型基于开源的llama7B，主要贡献是对应的lora模型。由于代码训练资源要求较小，希望为llama中文lora社区做一份贡献。",
+).queue().launch(share=True)

interaction.py ADDED Viewed

	@@ -0,0 +1,187 @@

+import sys
+import torch
+from peft import PeftModel
+import transformers
+import gradio as gr
+import argparse
+import warnings
+import os
+assert (
+    "LlamaTokenizer" in transformers._import_structure["models.llama"]
+), "LLaMA is now in HuggingFace's main branch.\nPlease reinstall it: pip uninstall transformers && pip install git+https://github.com/huggingface/transformers.git"
+from transformers import LlamaTokenizer, LlamaForCausalLM, GenerationConfig
+parser = argparse.ArgumentParser()
+parser.add_argument("--model_path", type=str, default="decapoda-research/llama-7b-hf")
+parser.add_argument("--lora_path", type=str, default="./lora-Vicuna/checkpoint-final")
+parser.add_argument("--use_local", type=int, default=1)
+args = parser.parse_args()
+tokenizer = LlamaTokenizer.from_pretrained(args.model_path)
+LOAD_8BIT = True
+BASE_MODEL = args.model_path
+LORA_WEIGHTS = args.lora_path
+# fix the path for local checkpoint
+lora_bin_path = os.path.join(args.lora_path, "adapter_model.bin")
+print(lora_bin_path)
+if not os.path.exists(lora_bin_path) and args.use_local:
+    pytorch_bin_path = os.path.join(args.lora_path, "pytorch_model.bin")
+    print(pytorch_bin_path)
+    if os.path.exists(pytorch_bin_path):
+        os.rename(pytorch_bin_path, lora_bin_path)
+        warnings.warn("The file name of the lora checkpoint'pytorch_model.bin' is replaced with 'adapter_model.bin'")
+    else:
+        assert ('Checkpoint is not Found!')
+if torch.cuda.is_available():
+    device = "cuda"
+else:
+    device = "cpu"
+try:
+    if torch.backends.mps.is_available():
+        device = "mps"
+except:
+    pass
+if device == "cuda":
+    model = LlamaForCausalLM.from_pretrained(
+        BASE_MODEL,
+        load_in_8bit=LOAD_8BIT,
+        torch_dtype=torch.float16,
+        device_map="auto", #device_map={"": 0},
+    )
+    model = PeftModel.from_pretrained(
+        model,
+        LORA_WEIGHTS,
+        torch_dtype=torch.float16,
+        device_map="auto", #device_map={"": 0},
+    )
+elif device == "mps":
+    model = LlamaForCausalLM.from_pretrained(
+        BASE_MODEL,
+        device_map={"": device},
+        torch_dtype=torch.float16,
+    )
+    model = PeftModel.from_pretrained(
+        model,
+        LORA_WEIGHTS,
+        device_map={"": device},
+        torch_dtype=torch.float16,
+    )
+else:
+    model = LlamaForCausalLM.from_pretrained(
+        BASE_MODEL, device_map={"": device}, low_cpu_mem_usage=True
+    )
+    model = PeftModel.from_pretrained(
+        model,
+        LORA_WEIGHTS,
+        device_map={"": device},
+    )
+def generate_prompt(instruction, input=None):
+    if input:
+        return f"""The following is a conversation between an AI assistant called Assistant and a human user called User.
+### Instruction:
+{instruction}
+### Input:
+{input}
+### Response:"""
+    else:
+        return f"""The following is a conversation between an AI assistant called Assistant and a human user called User.
+### Instruction:
+{instruction}
+### Response:"""
+if not LOAD_8BIT:
+    model.half()  # seems to fix bugs for some users.
+model.eval()
+if torch.__version__ >= "2" and sys.platform != "win32":
+    model = torch.compile(model)
+def interaction(
+    input,
+    history,
+    temperature=0.1,
+    top_p=0.75,
+    top_k=40,
+    num_beams=4,
+    max_new_tokens=128,
+    repetition_penalty=1.0,
+    max_memory=256,
+    **kwargs,
+):
+    now_input = input
+    history = history or []
+    if len(history) != 0:
+        input = "\n".join(["User:" + i[0]+"\n"+"Assistant:" + i[1] for i in history]) + "\n" + "User:" + input
+        if len(input) > max_memory:
+            input = input[-max_memory:]
+    print(input)
+    print(len(input))
+    prompt = generate_prompt(input)
+    inputs = tokenizer(prompt, return_tensors="pt")
+    input_ids = inputs["input_ids"].to(device)
+    generation_config = GenerationConfig(
+        temperature=temperature,
+        top_p=top_p,
+        top_k=top_k,
+        num_beams=num_beams,
+        **kwargs,
+    )
+    with torch.no_grad():
+        generation_output = model.generate(
+            input_ids=input_ids,
+            generation_config=generation_config,
+            return_dict_in_generate=True,
+            output_scores=True,
+            max_new_tokens=max_new_tokens,
+            repetition_penalty=float(repetition_penalty),
+        )
+    s = generation_output.sequences[0]
+    output = tokenizer.decode(s)
+    output = output.split("### Response:")[1].strip()
+    output = output.replace("Belle", "Vicuna")
+    if 'User:' in output:
+        output = output.split("User:")[0]
+    history.append((now_input, output))
+    print(history)
+    return history, history
+chatbot = gr.Chatbot().style(color_map=("green", "pink"))
+demo = gr.Interface(
+    fn=interaction,
+    inputs=[
+        gr.components.Textbox(
+            lines=2, label="Input", placeholder="Tell me about alpacas."
+        ),
+        "state",
+        gr.components.Slider(minimum=0, maximum=1, value=1.0, label="Temperature"),
+        gr.components.Slider(minimum=0, maximum=1, value=0.9, label="Top p"),
+        gr.components.Slider(minimum=0, maximum=100, step=1, value=60, label="Top k"),
+        gr.components.Slider(minimum=1, maximum=5, step=1, value=2, label="Beams"),
+        gr.components.Slider(
+            minimum=1, maximum=2000, step=1, value=128, label="Max new tokens"
+        ),
+        gr.components.Slider(
+            minimum=0.1, maximum=10.0, step=0.1, value=2.0, label="Repetition Penalty"
+        ),
+        gr.components.Slider(
+            minimum=0, maximum=2000, step=1, value=256, label="max memory"
+        ),
+    ],
+    outputs=[chatbot, "state"],
+    allow_flagging="auto",
+    title="Chinese-Vicuna 中文小羊驼",
+    description="中文小羊驼由各种高质量的开源instruction数据集，结合Alpaca-lora的代码训练而来，模型基于开源的llama7B，主要贡献是对应的lora模型。由于代码训练资源要求较小，希望为llama中文lora社区做一份贡献。",
+)
+demo.queue().launch(share=True, inbrowser=True)

prompt.py ADDED Viewed

	@@ -0,0 +1,209 @@

+import transformers
+from utils import printf
+import copy
+class prompt:
+    def __init__(self, tokenizer, max_len, add_eos=True):
+        self.tokenizer = tokenizer
+        self.max_len = max_len
+        self.add_eos=add_eos
+class instruct_prompt(prompt):
+    prompt = (
+        "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n"
+        "### Instruction:\n{instruction}\n\n### Response:"
+    )
+    prompt_input = (
+        "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n"
+        "### Instruction:{instruction}\n\n### Input:{input}\n\n### Response:"
+    )
+    prompt_history = "User:{input}\n\nAssistant:{output}\n\n"
+    prompt_post = "User:{input}\n\nAssistant:"
+    def preprocess_gen(self, data_point):
+        if 'history' not in data_point:
+        # single instruction format {'instruction':..,'input':..}
+            if 'input' in data_point:
+                user_prompt = self.prompt_input.format_map(data_point)
+            else:
+                user_prompt = self.prompt.format_map(data_point)
+        else:
+        # multi turn format {'history':[..], 'input':[..]}
+            user_prompt = "\n".join(["User:" + i['input']+"\n"+"Assistant:" + i['output'] for i in data_point['history']]) + "\nUser:" + data_point['input'] + "\nAssistant:"
+            user_prompt = user_prompt[-self.max_len:]
+        user_prompt=self.prompt.format_map({'instruction':user_prompt})
+        input_ids = self.tokenizer(user_prompt)["input_ids"]
+        return input_ids
+    def preprocess_train(self, data_point):
+        # single instruction format {'instruction':..,'input':..,'output':..}
+        if 'instruction' in data_point:
+            if 'input' in data_point:
+                user_prompt = self.prompt_input.format_map(data_point)
+            else:
+                user_prompt = self.prompt.format_map(data_point)
+            output = data_point["output"]
+        # multi turn format {'input':[..], 'output':[..]}
+        else:
+            user_prompt = ''
+            lens = len(data_point['input'])
+            for i in range(lens-1):
+                user_prompt += self.prompt_history.format_map({'input':data_point['input'][i],'output':data_point['output'][i]})
+            user_prompt += self.prompt_post.format_map({'input':data_point['input'][-1]})
+            user_prompt = self.prompt.format_map({'instruction': user_prompt})
+            output = data_point['output'][-1]
+        len_user_prompt_tokens = (len(self.tokenizer(
+            user_prompt,
+            truncation=True,
+            max_length=self.max_len + 1,
+        )["input_ids"])- 1)  # no eos token
+        full_tokens = self.tokenizer(
+            user_prompt + output,
+            truncation=True,
+            max_length=self.max_len + 1,
+            padding="max_length",
+        )["input_ids"][:-1]
+        return {
+            "input_ids": full_tokens,
+            "labels": [-100] * len_user_prompt_tokens
+            + full_tokens[len_user_prompt_tokens:],
+            "attention_mask": [1] * (len(full_tokens)),
+        }
+    def data_collator(self,):
+        return transformers.DataCollatorForLanguageModeling(self.tokenizer, mlm=False)
+    def postprocess(self, text, render=True):
+        #import pdb;pdb.set_trace()
+        printf(text)
+        output = text.split("### Response:")[1].strip()
+        output = output.replace("Belle", "Vicuna")
+        printf(output)
+        if '###' in output:
+            output = output.split("###")[0]
+        if 'User' in output:
+            output = output.split("User")[0]
+        output = output.replace('�','').replace('</s>', '')
+        if render:
+            # fix gradio chatbot markdown code render bug
+            lines = output.split("\n")
+            for i, line in enumerate(lines):
+                if "```" in line:
+                    if line != "```":
+                        lines[i] = f'<pre><code class="language-{lines[i][3:]}">'
+                    else:
+                        lines[i] = '</code></pre>'
+                else:
+                    if i > 0:
+                        lines[i] = "<br/>" + line.replace("<", "&lt;").replace(">", "&gt;").replace("__", '\_\_')
+            output =  "".join(lines)
+            # output = output.replace('<br/><pre>','\n<pre>') work for html; but not for gradio
+        return output
+class chat_prompt(prompt):
+    prompt_pre = (
+        "The following is a conversation between an AI assistant called Assistant and a human user called User. "
+        "The assistant is intelligent, knowledgeable and polite to answer questions of user.\n\n"
+    )
+    prompt_history = "User:{input}\n\nAssistant:{output}\n\n"
+    prompt_post = "User:{input}\n\nAssistant:"
+    def preprocess_gen(self, data_point):
+        user_prompt = self.prompt_pre
+        len_avail = self.max_len - len(self.tokenizer(user_prompt, add_special_tokens=False)['input_ids'])
+        input_prompt = self.prompt_post.format_map({'input':data_point['input']})
+        len_avail -= len(self.tokenizer(input_prompt, add_special_tokens=False)['input_ids'])
+        lens = len(data_point['history'])
+        tokenized_lens = []
+        for i in range(lens):
+            tmp_prompt = self.prompt_history.format_map(data_point['history'][i])
+            tokenized_lens.append(len(self.tokenizer(tmp_prompt,add_special_tokens=False)["input_ids"]))
+        # 启发式：/2 优先除前面的
+        i = 0
+        while sum(tokenized_lens) > len_avail and i < lens:
+            history = data_point['history'][i]
+            tmp_len1 = len(history['input'])
+            tmp_len2 = len(history['output'])
+            if tmp_len2 > tmp_len1:
+                history['output'] = history['output'][:tmp_len2//2]
+            else:
+                history['input'] = history['input'][:tmp_len1//2]
+            prompt = self.prompt_history.format_map(history)
+            single_len =(len(self.tokenizer(prompt,add_special_tokens=False)["input_ids"]))
+            tokenized_lens[i] = single_len
+            i += 1
+        total_len = sum(tokenized_lens)
+        # 还不够的话 直接截断
+        while total_len > len_avail and i < lens - 1 :
+            total_len -= tokenized_lens[i]
+            data_point['history'] = data_point['history'][1:]
+            i += 1
+        # 最终合并
+        for i in range(lens):
+            user_prompt += self.prompt_history.format_map(data_point['history'][i])
+        user_prompt += input_prompt
+        printf({'real_input:':user_prompt})
+        inputs = self.tokenizer(user_prompt)["input_ids"]
+        return inputs
+    def preprocess_train(self, data_point):
+        user_prompt = self.prompt_pre
+        lens = len(data_point['input'])
+        for i in range(lens-1):
+            user_prompt += self.prompt_history.format_map({'input':data_point['input'][i].strip(),'output':data_point['output'][i].strip()})
+        user_prompt += self.prompt_post.format_map({'input':data_point['input'][-1].strip()})
+        len_user_prompt_tokens = len(self.tokenizer(
+            user_prompt,
+            truncation=True,
+            max_length=self.max_len,
+        )["input_ids"]) - 1 # remove extra eos
+        if self.add_eos:
+            full_tokens = self.tokenizer(
+                user_prompt + data_point["output"][-1].strip(),
+                truncation=True,
+                padding=False,
+                max_length=self.max_len,
+            )["input_ids"] # need eos
+        else:
+            full_tokens = self.tokenizer(
+                user_prompt + data_point["output"][-1].strip(),
+                truncation=True,
+                padding=False,
+                max_length=self.max_len+1,
+            )["input_ids"][:-1] # delete eos
+        return {
+            "input_ids": full_tokens,
+            "labels": [-100] * len_user_prompt_tokens + full_tokens[len_user_prompt_tokens:],
+            "attention_mask": [1] * (len(full_tokens)),
+        }
+    def data_collator(self,):
+        return transformers.DataCollatorForSeq2Seq(self.tokenizer)
+    def postprocess(self, text, render=False):
+        output = text.split("Assistant:")[-1].strip()
+        if 'User:' in output:
+            output = output.split("User:")[0]
+        output = output.replace('�','')
+        if render:
+            # fix gradio chatbot markdown code render bug
+            lines = output.split("\n")
+            for i, line in enumerate(lines):
+                if "```" in line:
+                    if line != "```":
+                        lines[i] = f'<pre><code class="language-{lines[i][3:]}">'
+                    else:
+                        lines[i] = '</code></pre>'
+                else:
+                    if i > 0:
+                        lines[i] = "<br/>" + line.replace("<", "&lt;").replace(">", "&gt;").replace("__", '\_\_')
+            output =  "".join(lines)
+            # output = output.replace('<br/><pre>','\n<pre>') work for html; but not for gradio
+        return output
+    def get_data_collator():
+        return transformers.DataCollatorForLanguageModeling

requirements.txt ADDED Viewed

	@@ -0,0 +1,23 @@

+accelerate==0.15.0
+appdirs==1.4.4
+bitsandbytes==0.37.0
+datasets==2.8.0
+deepspeed==0.8.3
+evaluate==0.4.0
+fairscale==0.4.13
+torch==1.13.1
+torchvision==0.14.1
+gradio==3.20.0
+huggingface-hub==0.13.3
+loralib==0.1.1
+nvitop==1.0.0
+peft @ git+https://github.com/huggingface/peft.git@13e53fc7ee5d89d59b16523051006dddf0fb7a49
+sentencepiece==0.1.96
+tensorboard==2.12.0
+texttable==1.6.7
+tokenizers==0.13.2
+tqdm==4.65.0
+transformers @ git+https://github.com/huggingface/transformers.git@0dcb46e7a4a9e587ba84ff35778ab4233a184c11
+trlx @ git+https://github.com/CarperAI/trlx.git@b91da7b03d8e9fa0c0d6dce10a8f2611aca3013f
+wandb==0.13.10
+triton==2.0.0

requirements_4bit.txt ADDED Viewed

	@@ -0,0 +1,23 @@

+accelerate @ git+https://github.com/huggingface/accelerate.git@f9b2e6769b1279b11335d2b87801fa1ca81bb056
+appdirs==1.4.4
+bitsandbytes==0.39.0
+datasets==2.8.0
+deepspeed==0.8.3
+evaluate==0.4.0
+fairscale==0.4.13
+torch==1.13.1
+torchvision==0.14.1
+gradio==3.20.0
+huggingface-hub==0.13.3
+loralib==0.1.1
+nvitop==1.0.0
+peft @ git+https://github.com/huggingface/peft.git@3714aa2fff158fdfa637b2b65952580801d890b2
+sentencepiece==0.1.96
+tensorboard==2.12.0
+texttable==1.6.7
+tokenizers==0.13.2
+tqdm==4.65.0
+transformers @ git+https://github.com/huggingface/transformers@e45e756d22206ca8fa9fb057c8c3d8fa79bf81c6
+trlx @ git+https://github.com/CarperAI/trlx.git@b91da7b03d8e9fa0c0d6dce10a8f2611aca3013f
+wandb==0.13.10
+triton==2.0.0

test_tokenizer.py ADDED Viewed

	@@ -0,0 +1,52 @@

+import os
+import sys
+import torch
+import transformers
+import argparse
+from transformers import LlamaForCausalLM, LlamaTokenizer, BitsAndBytesConfig
+parser = argparse.ArgumentParser()
+parser.add_argument("--model_path", type=str, default="yahma/llama-7b-hf") #yahma/llama-7b-hf #decapoda-research/llama-7b-hf
+args = parser.parse_args()
+tokenizer = LlamaTokenizer.from_pretrained(
+    args.model_path, add_eos_token=True
+)
+test_text = ["Hello, nice to meet you!", "你好很高兴能见到你！"]
+for text in test_text:
+    input_ids = tokenizer.encode(text)
+    print(f"input_ids: {input_ids}")
+    decode_text = tokenizer.decode(input_ids)
+    print(f"decode_text: {decode_text}")
+"""
+Correct ==>  yahma/llama-7b-hf + newest Transformers(>=4.28.1):
+> !!! Beginning with 1 (bos), ending with 2 (eos) !!!
+input_ids: [1, 15043, 29892, 7575, 304, 5870, 366, 29991, 2]
+decode_text: <s> Hello, nice to meet you!</s>
+input_ids: [1, 29871, 30919, 31076, 232, 193, 139, 30528, 31914, 30815, 235, 170, 132, 30780, 30919, 30584, 2]
+decode_text: <s> 你好很高兴能见到你！</s>
+Correct ==> decapoda-research/llama-7b-hf + Old Transformers like our version(transformers @ git+https://github.com/huggingface/transformers.git@0dcb46e7a4a9e587ba84ff35778ab4233a184c11)
+input_ids: [1, 15043, 29892, 7575, 304, 5870, 366, 29991, 2]
+decode_text:  Hello, nice to meet you!
+input_ids: [1, 29871, 30919, 31076, 232, 193, 139, 30528, 31914, 30815, 235, 170, 132, 30780, 30919, 30584, 2]
+decode_text:  你好很高兴能见到你！
+Correct ==> decapoda-research/llama-7b-hf + Old Transformers like our version(transformers @ git+https://github.com/huggingface/transformers.git@0dcb46e7a4a9e587ba84ff35778ab4233a184c11)
+input_ids: [1, 15043, 29892, 7575, 304, 5870, 366, 29991, 2]
+decode_text:  Hello, nice to meet you!
+input_ids: [1, 29871, 30919, 31076, 232, 193, 139, 30528, 31914, 30815, 235, 170, 132, 30780, 30919, 30584, 2]
+decode_text:  你好很高兴能见到你！
+老版本transformers的问题：代码默认加载tokenizer.model
+新版本transformers的修改：新版本默认加载config
+decapoda-research：config的bos=0，eos=1（×），tokenizer.model是正确的
+yahma：config的bos=1，eos=2，tokenizer.model是正确的
+"""

utils.py ADDED Viewed

	@@ -0,0 +1,828 @@

+import logging
+import sys
+import os
+import torch
+import json
+from typing import Optional, Tuple, Union, List, Callable
+from transformers import LlamaForCausalLM
+from transformers.generation.logits_process import LogitsProcessor
+from transformers.generation.beam_search import BeamSearchScorer
+from transformers.deepspeed import is_deepspeed_zero3_enabled
+from transformers.generation.utils import (
+    LogitsProcessorList,
+    StoppingCriteriaList,
+    GenerationConfig,
+    GenerationMixin,
+)
+import warnings
+from peft import PeftModel, PeftModelForCausalLM, LoraConfig
+import peft
+import torch.distributed as dist
+from torch import nn
+import copy
+from accelerate.hooks import (
+    AlignDevicesHook,
+    add_hook_to_module,
+    remove_hook_from_submodules,
+)
+from accelerate.utils import get_balanced_memory
+from huggingface_hub import hf_hub_download
+from accelerate import dispatch_model, infer_auto_device_map
+from peft.utils import PeftType, set_peft_model_state_dict
+def printf(*args,**kargs):
+    if os.environ.get('DEBUG',False):
+        end = '\n'
+        if 'end' in kargs:
+            end = kargs['end']
+        print(*args, end=end, flush=True)
+class ColorFormatter(logging.Formatter):
+    grey = "\x1b[38;20m"
+    blue = "\x1b[34;20m"
+    yellow = "\x1b[33;20m"
+    red = "\x1b[31;20m"
+    bold_red = "\x1b[31;1m"
+    reset = "\x1b[0m"
+    def __init__(self, fmt):
+        super().__init__(fmt)
+        self.FORMATS = {
+            logging.DEBUG: self.grey + fmt + self.reset,
+            logging.INFO: self.blue + fmt + self.reset,
+            logging.WARNING: self.yellow + fmt + self.reset,
+            logging.ERROR: self.red + fmt + self.reset,
+            logging.CRITICAL: self.bold_red + fmt + self.reset
+        }
+    def format(self, record):
+        log_fmt = self.FORMATS.get(record.levelno)
+        formatter = logging.Formatter(log_fmt)
+        return formatter.format(record)
+def set_console_logger(name):
+    logger = logging.getLogger(name)
+    logger.setLevel(logging.DEBUG)
+    consoleHandler = logging.StreamHandler(sys.stdout)
+    consoleHandler.setLevel(logging.INFO)
+    consoleHandler.setFormatter(ColorFormatter("%(asctime)s | %(levelname)s %(message)s"))
+    logger.addHandler(consoleHandler)
+    return logger
+def set_file_logger(name, dir, use_console=False):
+    logger = logging.getLogger(name)
+    logger.setLevel(logging.DEBUG)
+    os.makedirs(dir, exist_ok=True)
+    if use_console:
+        logger.propagate = False # disable default handler
+        consoleHandler = logging.StreamHandler(sys.stdout)
+        consoleHandler.setLevel(logging.INFO)
+        consoleHandler.setFormatter(ColorFormatter("%(asctime)s | %(levelname)s %(message)s"))
+        logger.addHandler(consoleHandler)
+    fileHandler = logging.FileHandler(os.path.join(dir,'session.log'), mode='a')
+    fileHandler.setLevel(logging.INFO)
+    fileHandler.setFormatter(logging.Formatter("%(asctime)s | %(levelname)s %(message)s"))
+    logger.addHandler(fileHandler)
+    return logger
+def to_jsonl(data, path):
+    with open(path, 'a') as f:
+        for line in data:
+            f.write(json.dumps(line,ensure_ascii=False)+'\n')
+def from_json(path):
+    return json.load(open(path))
+def from_jsonl(path):
+    return [json.loads(line) for line in open(path, 'r') ]
+def to_json(data, path):
+    json.dump(data, open(path, 'w'), ensure_ascii=False)
+class StreamGenerationMixin(GenerationMixin):
+    # support for streamly generation
+    # TODO: group_beam_search
+    @torch.no_grad()
+    def stream_generate(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        generation_config: Optional[GenerationConfig] = None,
+        logits_processor: Optional[LogitsProcessorList] = None,
+        stopping_criteria: Optional[StoppingCriteriaList] = None,
+        prefix_allowed_tokens_fn: Optional[
+            Callable[[int, torch.Tensor], List[int]]
+        ] = None,
+        **kwargs,
+    ):
+        if is_deepspeed_zero3_enabled() and dist.world_size() > 1:
+            synced_gpus = True
+        else:
+            synced_gpus = False
+        if kwargs.get("attention_mask", None) is not None:
+            # concat prompt attention mask
+            prefix_attention_mask = torch.ones(
+                kwargs["input_ids"].shape[0], self.peft_config.num_virtual_tokens
+            ).to(kwargs["input_ids"].device)
+            kwargs["attention_mask"] = torch.cat(
+                (prefix_attention_mask, kwargs["attention_mask"]), dim=1
+            )
+        if kwargs.get("position_ids", None) is not None:
+            warnings.warn(
+                "Position ids are not supported for parameter efficient tuning. Ignoring position ids."
+            )
+            kwargs["position_ids"] = None
+        if kwargs.get("token_type_ids", None) is not None:
+            warnings.warn(
+                "Token type ids are not supported for parameter efficient tuning. Ignoring token type ids"
+            )
+            kwargs["token_type_ids"] = None
+        batch_size, input_ids_seq_length = input_ids.shape[0], input_ids.shape[-1]
+        if generation_config is None:
+            generation_config = self.generation_config
+        generation_config = copy.deepcopy(generation_config)
+        model_kwargs = generation_config.update(**kwargs)
+        bos_token_id, eos_token_id, pad_token_id = (
+            generation_config.bos_token_id,
+            generation_config.eos_token_id,
+            generation_config.pad_token_id,
+        )
+        if isinstance(eos_token_id, int):
+            eos_token_id = [eos_token_id]
+        has_default_max_length = (
+            kwargs.get("max_length") is None
+            and generation_config.max_length is not None
+        )
+        if has_default_max_length and generation_config.max_new_tokens is None:
+            warnings.warn(
+                f"Using `max_length`'s default ({generation_config.max_length}) to control the generation length. "
+                "This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we"
+                " recommend using `max_new_tokens` to control the maximum length of the generation.",
+                UserWarning,
+            )
+        elif generation_config.max_new_tokens is not None:
+            generation_config.max_length = (
+                generation_config.max_new_tokens + input_ids_seq_length
+            )
+        if generation_config.min_new_tokens is not None:
+            generation_config.min_length = (
+                generation_config.min_new_tokens + input_ids_seq_length
+            )
+        if input_ids_seq_length >= generation_config.max_length:
+            input_ids_string = (
+                "decoder_input_ids" if self.config.is_encoder_decoder else "input_ids"
+            )
+        # 2. Set generation parameters if not already defined
+        logits_processor = (
+            logits_processor if logits_processor is not None else LogitsProcessorList()
+        )
+        stopping_criteria = (
+            stopping_criteria
+            if stopping_criteria is not None
+            else StoppingCriteriaList()
+        )
+        # 7. determine generation mode
+        is_constraint_gen_mode = (
+            generation_config.constraints is not None or generation_config.force_words_ids is not None
+        )
+        is_contrastive_search_gen_mode = (
+            generation_config.top_k is not None
+            and generation_config.top_k > 1
+            and generation_config.do_sample is False
+            and generation_config.penalty_alpha is not None
+            and generation_config.penalty_alpha > 0
+        )
+        is_greedy_gen_mode = (
+            (generation_config.num_beams == 1)
+            and (generation_config.num_beam_groups == 1)
+            and generation_config.do_sample is False
+            and not is_constraint_gen_mode
+            and not is_contrastive_search_gen_mode
+        )
+        # beam=1 and do_sample=True
+        is_sample_gen_mode = (
+            (generation_config.num_beams == 1)
+            and (generation_config.num_beam_groups == 1)
+            and generation_config.do_sample is True
+            and not is_constraint_gen_mode
+            and not is_contrastive_search_gen_mode
+        )
+        is_beam_gen_mode = (
+            (generation_config.num_beams > 1)
+            and (generation_config.num_beam_groups == 1)
+            and generation_config.do_sample is False
+            and not is_constraint_gen_mode
+            and not is_contrastive_search_gen_mode
+        )
+        is_beam_sample_gen_mode = (
+            (generation_config.num_beams > 1)
+            and (generation_config.num_beam_groups == 1)
+            and generation_config.do_sample is True
+            and not is_constraint_gen_mode
+            and not is_contrastive_search_gen_mode
+        )
+        is_group_beam_gen_mode = (
+            (generation_config.num_beams > 1)
+            and (generation_config.num_beam_groups > 1)
+            and not is_constraint_gen_mode
+            and not is_contrastive_search_gen_mode
+        )
+        # 8. prepare distribution pre_processing samplers
+        logits_processor = self._get_logits_processor(
+            generation_config=generation_config,
+            input_ids_seq_length=input_ids_seq_length,
+            encoder_input_ids=input_ids,
+            prefix_allowed_tokens_fn=prefix_allowed_tokens_fn,
+            logits_processor=logits_processor,
+        )
+        # 9. prepare stopping criteria
+        stopping_criteria = self._get_stopping_criteria(
+            generation_config=generation_config, stopping_criteria=stopping_criteria
+        )
+        logits_warper = self._get_logits_warper(generation_config)
+        if is_greedy_gen_mode:
+            # 11. run greedy search
+            return self.stream_greedy_search(
+                input_ids,
+                logits_processor,
+                stopping_criteria,
+                generation_config,
+                synced_gpus,
+                **model_kwargs,
+            )
+        elif is_sample_gen_mode:
+            # 12. expand input_ids with `num_return_sequences` additional sequences per batch
+            input_ids, model_kwargs = self._expand_inputs_for_generation(
+                input_ids=input_ids,
+                expand_size=generation_config.num_return_sequences,
+                is_encoder_decoder=self.config.is_encoder_decoder,
+                **model_kwargs,
+            )
+            return self.stream_sample(
+                generation_config,
+                input_ids,
+                logits_processor,
+                logits_warper,
+                stopping_criteria,
+                synced_gpus,
+                **model_kwargs,
+            )
+        elif is_beam_gen_mode:
+            return self.stream_beam_search(
+                generation_config,
+                input_ids,
+                logits_processor,
+                stopping_criteria,
+                synced_gpus,
+                **model_kwargs,
+            )
+        elif is_beam_sample_gen_mode:
+            # interleave input_ids with `num_beams` additional sequences per batch
+            return self.stream_beam_sample(
+                input_ids,
+                logits_processor,
+                logits_warper,
+                stopping_criteria,
+                generation_config,
+                synced_gpus,
+                **model_kwargs,
+            )
+        else:
+            raise Exception('not implement')
+    def stream_sample(
+        self,
+        generation_config,
+        input_ids,
+        logits_processor,
+        logits_warper,
+        stopping_criteria,
+        synced_gpus,
+        **model_kwargs,
+    ):
+        bos_token_id, eos_token_id, pad_token_id = (
+            generation_config.bos_token_id,
+            generation_config.eos_token_id,
+            generation_config.pad_token_id,
+        )
+        if isinstance(eos_token_id, int):
+            eos_token_id = [eos_token_id]
+        eos_token_id_tensor = torch.tensor(eos_token_id).to(input_ids.device) if eos_token_id is not None else None
+        # keep track of which sequences are already finished
+        unfinished_sequences = torch.ones(input_ids.shape[0], dtype=torch.long, device=input_ids.device)
+        this_peer_finished = False  # used by synced_gpus only
+        scores=()
+        # auto-regressive generation
+        while True:
+            if synced_gpus:
+                # Under synced_gpus the `forward` call must continue until all gpus complete their sequence.
+                # The following logic allows an early break if all peers finished generating their sequence
+                this_peer_finished_flag = torch.tensor(0.0 if this_peer_finished else 1.0).to(input_ids.device)
+                # send 0.0 if we finished, 1.0 otherwise
+                dist.all_reduce(this_peer_finished_flag, op=dist.ReduceOp.SUM)
+                # did all peers finish? the reduced sum will be 0.0 then
+                if this_peer_finished_flag.item() == 0.0:
+                    break
+            # prepare model inputs
+            model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
+            # forward pass to get next token
+            outputs = self(
+                **model_inputs,
+                return_dict=True,
+            )
+            if synced_gpus and this_peer_finished:
+                continue  # don't waste resources running the code we don't need
+            next_token_logits = outputs.logits[:, -1, :]
+            # pre-process distribution
+            next_token_scores = logits_processor(input_ids, next_token_logits)
+            next_token_scores = logits_warper(input_ids, next_token_scores)
+            # sample
+            probs = nn.functional.softmax(next_token_scores, dim=-1)
+            next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
+            # finished sentences should have their next token be a padding token
+            if eos_token_id is not None:
+                if pad_token_id is None:
+                    raise ValueError("If `eos_token_id` is defined, make sure that `pad_token_id` is defined.")
+                next_tokens = next_tokens * unfinished_sequences + pad_token_id * (1 - unfinished_sequences)
+            # update generated ids, model inputs, and length for next step
+            input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1)
+            model_kwargs = self._update_model_kwargs_for_generation(
+                outputs, model_kwargs, is_encoder_decoder=self.config.is_encoder_decoder
+            )
+            yield input_ids
+            # if eos_token was found in one sentence, set sentence to finished
+            if eos_token_id_tensor is not None:
+                unfinished_sequences = unfinished_sequences.mul(
+                    next_tokens.tile(eos_token_id_tensor.shape[0], 1).ne(eos_token_id_tensor.unsqueeze(1)).prod(dim=0)
+                )
+            # stop when each sentence is finished, or if we exceed the maximum length
+            if unfinished_sequences.max() == 0 or stopping_criteria(input_ids, scores):
+                if not synced_gpus:
+                    break
+                else:
+                    this_peer_finished = True
+        yield input_ids
+    def stream_beam_sample(
+        self,
+        input_ids,
+        logits_processor,
+        logits_warper,
+        stopping_criteria,
+        generation_config,
+        synced_gpus,
+        **model_kwargs,
+    ):
+        bos_token_id, eos_token_id, pad_token_id = (
+            generation_config.bos_token_id,
+            generation_config.eos_token_id,
+            generation_config.pad_token_id,
+        )
+        if isinstance(eos_token_id, int):
+            eos_token_id = [eos_token_id]
+        eos_token_id_tensor = torch.tensor(eos_token_id).to(input_ids.device) if eos_token_id is not None else None
+        num_beams = generation_config.num_beams
+        batch_size, cur_len = input_ids.shape[0], input_ids.shape[-1]
+        beam_scorer = BeamSearchScorer(
+            batch_size=batch_size,
+            num_beams=generation_config.num_beams,
+            device=input_ids.device,
+            length_penalty=generation_config.length_penalty,
+            do_early_stopping=generation_config.early_stopping,
+            num_beam_hyps_to_keep=generation_config.num_return_sequences,
+            max_length=generation_config.max_length,
+        )
+        input_ids, model_kwargs = self._expand_inputs_for_generation(
+            input_ids=input_ids,
+            expand_size=generation_config.num_beams * generation_config.num_return_sequences,
+            is_encoder_decoder=self.config.is_encoder_decoder,
+            **model_kwargs,
+        )
+        scores = ()
+        beam_scores = torch.zeros((batch_size, num_beams), dtype=torch.float, device=input_ids.device)
+        beam_scores = beam_scores.view((batch_size * num_beams,))
+        this_peer_finished = False  # used by synced_gpus only
+        while True:
+            if synced_gpus:
+                # Under synced_gpus the `forward` call must continue until all gpus complete their sequence.
+                # The following logic allows an early break if all peers finished generating their sequence
+                this_peer_finished_flag = torch.tensor(0.0 if this_peer_finished else 1.0).to(input_ids.device)
+                # send 0.0 if we finished, 1.0 otherwise
+                dist.all_reduce(this_peer_finished_flag, op=dist.ReduceOp.SUM)
+                # did all peers finish? the reduced sum will be 0.0 then
+                if this_peer_finished_flag.item() == 0.0:
+                    break
+            model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
+            outputs = self(
+                **model_inputs,
+                return_dict=True,
+            )
+            if synced_gpus and this_peer_finished:
+                cur_len = cur_len + 1
+                continue  # don't waste resources running the code we don't need
+            next_token_logits = outputs.logits[:, -1, :]
+            # hack: adjust tokens for Marian. For Marian we have to make sure that the `pad_token_id`
+            # cannot be generated both before and after the `nn.functional.log_softmax` operation.
+            next_token_logits = self.adjust_logits_during_generation(next_token_logits, cur_len=cur_len)
+            next_token_scores = nn.functional.log_softmax(
+                next_token_logits, dim=-1
+            )  # (batch_size * num_beams, vocab_size)
+            next_token_scores_processed = logits_processor(input_ids, next_token_scores)
+            next_token_scores = next_token_scores_processed + beam_scores[:, None].expand_as(next_token_scores)
+            # Note: logits warpers are intentionally applied after adding running beam scores. On some logits warpers
+            # (like top_p) this is indiferent, but on others (like temperature) it is not. For reference, see
+            # https://github.com/huggingface/transformers/pull/5420#discussion_r449779867
+            next_token_scores = logits_warper(input_ids, next_token_scores)
+            # reshape for beam search
+            vocab_size = next_token_scores.shape[-1]
+            next_token_scores = next_token_scores.view(batch_size, num_beams * vocab_size)
+            probs = nn.functional.softmax(next_token_scores, dim=-1)
+            next_tokens = torch.multinomial(probs, num_samples=2 * num_beams)
+            next_token_scores = torch.gather(next_token_scores, -1, next_tokens)
+            next_token_scores, _indices = torch.sort(next_token_scores, descending=True, dim=1)
+            next_tokens = torch.gather(next_tokens, -1, _indices)
+            next_indices = torch.div(next_tokens, vocab_size, rounding_mode="floor")
+            next_tokens = next_tokens % vocab_size
+            # stateless
+            beam_outputs = beam_scorer.process(
+                input_ids,
+                next_token_scores,
+                next_tokens,
+                next_indices,
+                pad_token_id=pad_token_id,
+                eos_token_id=eos_token_id,
+                beam_indices=None,
+            )
+            beam_scores = beam_outputs["next_beam_scores"]
+            beam_next_tokens = beam_outputs["next_beam_tokens"]
+            beam_idx = beam_outputs["next_beam_indices"]
+            input_ids = torch.cat([input_ids[beam_idx, :], beam_next_tokens.unsqueeze(-1)], dim=-1)
+            yield input_ids
+            model_kwargs = self._update_model_kwargs_for_generation(
+                outputs, model_kwargs, is_encoder_decoder=self.config.is_encoder_decoder
+            )
+            if model_kwargs["past_key_values"] is not None:
+                model_kwargs["past_key_values"] = self._reorder_cache(model_kwargs["past_key_values"], beam_idx)
+            # increase cur_len
+            cur_len = cur_len + 1
+            if beam_scorer.is_done or stopping_criteria(input_ids, scores):
+                if not synced_gpus:
+                    break
+                else:
+                    this_peer_finished = True
+        sequence_outputs = beam_scorer.finalize(
+            input_ids,
+            beam_scores,
+            next_tokens,
+            next_indices,
+            pad_token_id=pad_token_id,
+            eos_token_id=eos_token_id,
+            max_length=stopping_criteria.max_length,
+            beam_indices=None,
+        )
+        yield sequence_outputs["sequences"]
+    def stream_greedy_search(
+        self,
+        input_ids,
+        logits_processor,
+        stopping_criteria,
+        generation_config,
+        synced_gpus,
+        **model_kwargs,
+    ):
+        # init values
+        bos_token_id, eos_token_id, pad_token_id = (
+            generation_config.bos_token_id,
+            generation_config.eos_token_id,
+            generation_config.pad_token_id,
+        )
+        if isinstance(eos_token_id, int):
+            eos_token_id = [eos_token_id]
+        eos_token_id_tensor = torch.tensor(eos_token_id).to(input_ids.device) if eos_token_id is not None else None
+        # init attention / hidden states / scores tuples
+        scores = ()
+        # keep track of which sequences are already finished
+        unfinished_sequences = torch.ones(input_ids.shape[0], dtype=torch.long, device=input_ids.device)
+        this_peer_finished = False  # used by synced_gpus only
+        while True:
+            if synced_gpus:
+                # Under synced_gpus the `forward` call must continue until all gpus complete their sequence.
+                # The following logic allows an early break if all peers finished generating their sequence
+                this_peer_finished_flag = torch.tensor(0.0 if this_peer_finished else 1.0).to(input_ids.device)
+                # send 0.0 if we finished, 1.0 otherwise
+                dist.all_reduce(this_peer_finished_flag, op=dist.ReduceOp.SUM)
+                # did all peers finish? the reduced sum will be 0.0 then
+                if this_peer_finished_flag.item() == 0.0:
+                    break
+            # prepare model inputs
+            model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
+            # forward pass to get next token
+            outputs = self(
+                **model_inputs,
+                return_dict=True,
+            )
+            if synced_gpus and this_peer_finished:
+                continue  # don't waste resources running the code we don't need
+            next_token_logits = outputs.logits[:, -1, :]
+            # pre-process distribution
+            next_tokens_scores = logits_processor(input_ids, next_token_logits)
+            # argmax
+            next_tokens = torch.argmax(next_tokens_scores, dim=-1)
+            # finished sentences should have their next token be a padding token
+            if eos_token_id is not None:
+                if pad_token_id is None:
+                    raise ValueError("If `eos_token_id` is defined, make sure that `pad_token_id` is defined.")
+                next_tokens = next_tokens * unfinished_sequences + pad_token_id * (1 - unfinished_sequences)
+            # update generated ids, model inputs, and length for next step
+            input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1)
+            model_kwargs = self._update_model_kwargs_for_generation(
+                outputs, model_kwargs, is_encoder_decoder=self.config.is_encoder_decoder
+            )
+            yield input_ids
+            # if eos_token was found in one sentence, set sentence to finished
+            if eos_token_id_tensor is not None:
+                unfinished_sequences = unfinished_sequences.mul(
+                    next_tokens.tile(eos_token_id_tensor.shape[0], 1).ne(eos_token_id_tensor.unsqueeze(1)).prod(dim=0)
+                )
+            # stop when each sentence is finished, or if we exceed the maximum length
+            if unfinished_sequences.max() == 0 or stopping_criteria(input_ids, scores):
+                if not synced_gpus:
+                    break
+                else:
+                    this_peer_finished = True
+        yield input_ids
+    def stream_beam_search(
+        self,
+        generation_config,
+        input_ids,
+        logits_processor,
+        stopping_criteria,
+        synced_gpus,
+        **model_kwargs,
+    ):
+        # 10. go into beam search generation modes
+        # 11. prepare beam search scorer
+        bos_token_id, eos_token_id, pad_token_id = (
+            generation_config.bos_token_id,
+            generation_config.eos_token_id,
+            generation_config.pad_token_id,
+        )
+        if isinstance(eos_token_id, int):
+            eos_token_id = [eos_token_id]
+        num_beams = generation_config.num_beams
+        batch_size, input_ids_seq_length = input_ids.shape[0], input_ids.shape[-1]
+        beam_scorer = BeamSearchScorer(
+            batch_size=batch_size,
+            num_beams=generation_config.num_beams,
+            device=input_ids.device,
+            length_penalty=generation_config.length_penalty,
+            do_early_stopping=generation_config.early_stopping,
+            num_beam_hyps_to_keep=generation_config.num_return_sequences,
+            max_length=generation_config.max_length,
+        )
+        # 12. interleave input_ids with `num_beams` additional sequences per batch
+        input_ids, model_kwargs = self._expand_inputs_for_generation(
+            input_ids=input_ids,
+            expand_size=generation_config.num_beams,
+            is_encoder_decoder=self.config.is_encoder_decoder,
+            **model_kwargs,
+        )
+        # beam_search logits
+        batch_beam_size, cur_len = input_ids.shape
+        if num_beams * batch_size != batch_beam_size:
+            raise ValueError(
+                f"Batch dimension of `input_ids` should be {num_beams * batch_size}, but is {batch_beam_size}."
+            )
+        beam_scores = torch.zeros(
+            (batch_size, num_beams), dtype=torch.float, device=input_ids.device
+        )
+        beam_scores[:, 1:] = -1e9
+        beam_scores = beam_scores.view((batch_size * num_beams,))
+        this_peer_finished = False  # used by synced_gpus only
+        while True:
+            if synced_gpus:
+                # Under synced_gpus the `forward` call must continue until all gpus complete their sequence.
+                # The following logic allows an early break if all peers finished generating their sequence
+                this_peer_finished_flag = torch.tensor(
+                    0.0 if this_peer_finished else 1.0
+                ).to(input_ids.device)
+                # send 0.0 if we finished, 1.0 otherwise
+                dist.all_reduce(this_peer_finished_flag, op=dist.ReduceOp.SUM)
+                # did all peers finish? the reduced sum will be 0.0 then
+                if this_peer_finished_flag.item() == 0.0:
+                    break
+            model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
+            outputs = self(
+                **model_inputs,
+                return_dict=True,
+                output_attentions=False,
+                output_hidden_states=False,
+            )
+            if synced_gpus and this_peer_finished:
+                cur_len = cur_len + 1
+                continue  # don't waste resources running the code we don't need
+            next_token_logits = outputs.logits[:, -1, :]
+            # next_token_logits = self.adjust_logits_during_generation(next_token_logits, cur_len=cur_len) hack: adjust tokens for Marian.
+            next_token_scores = nn.functional.log_softmax(
+                next_token_logits, dim=-1
+            )  # (batch_size * num_beams, vocab_size)
+            next_token_scores_processed = logits_processor(input_ids, next_token_scores)
+            next_token_scores = next_token_scores_processed + beam_scores[
+                :, None
+            ].expand_as(next_token_scores)
+            # reshape for beam search
+            vocab_size = next_token_scores.shape[-1]
+            next_token_scores = next_token_scores.view(
+                batch_size, num_beams * vocab_size
+            )
+            # Sample 2 next tokens for each beam (so we have some spare tokens and match output of beam search)
+            next_token_scores, next_tokens = torch.topk(
+                next_token_scores, 2 * num_beams, dim=1, largest=True, sorted=True
+            )
+            next_indices = torch.div(next_tokens, vocab_size, rounding_mode="floor")
+            next_tokens = next_tokens % vocab_size
+            # stateless
+            beam_outputs = beam_scorer.process(
+                input_ids,
+                next_token_scores,
+                next_tokens,
+                next_indices,
+                pad_token_id=pad_token_id,
+                eos_token_id=eos_token_id,
+                beam_indices=None,
+            )
+            beam_scores = beam_outputs["next_beam_scores"]
+            beam_next_tokens = beam_outputs["next_beam_tokens"]
+            beam_idx = beam_outputs["next_beam_indices"]
+            input_ids = torch.cat(
+                [input_ids[beam_idx, :], beam_next_tokens.unsqueeze(-1)], dim=-1
+            )
+            model_kwargs = self._update_model_kwargs_for_generation(
+                outputs, model_kwargs, is_encoder_decoder=self.config.is_encoder_decoder
+            )
+            if model_kwargs["past_key_values"] is not None:
+                model_kwargs["past_key_values"] = self._reorder_cache(
+                    model_kwargs["past_key_values"], beam_idx
+                )
+            # increase cur_len
+            cur_len = cur_len + 1
+            yield input_ids
+            if beam_scorer.is_done or stopping_criteria(input_ids, None):
+                if not synced_gpus:
+                    break
+                else:
+                    this_peer_finished = True
+        final_result = beam_scorer.finalize(
+            input_ids,
+            beam_scores,
+            next_tokens,
+            next_indices,
+            pad_token_id=pad_token_id,
+            eos_token_id=eos_token_id,
+            max_length=stopping_criteria.max_length,
+            beam_indices=None,
+        )
+        yield final_result["sequences"]
+class StreamLlamaForCausalLM(LlamaForCausalLM, StreamGenerationMixin):
+    pass
+class StreamPeftGenerationMixin(PeftModelForCausalLM, StreamGenerationMixin):
+    # default it call `model = MODEL_TYPE_TO_PEFT_MODEL_MAPPING[config.task_type](model, config)`, not cls!! so inherent PeftModelForCausalLM is non sense
+    @classmethod
+    def from_pretrained(cls, model, model_id, adapter_name="default", is_trainable=False,  **kwargs):
+        # work in peft==0.3.0
+        if peft.__version__ >= '0.3.0' and peft.__version__ != '0.3.0.dev0':
+            # load the config
+            from peft.utils import PromptLearningConfig
+            config = LoraConfig.from_pretrained(model_id)
+            if (getattr(model, "hf_device_map", None) is not None) and len(
+                set(model.hf_device_map.values()).intersection({"cpu", "disk"})
+            ) > 0:
+                remove_hook_from_submodules(model)
+            if isinstance(config, PromptLearningConfig) and is_trainable:
+                raise ValueError("Cannot set a prompt learning adapter to trainable when loading pretrained adapter.")
+            else:
+                config.inference_mode = not is_trainable
+            # here is the hack
+            model = cls(model, config, adapter_name)
+            model.load_adapter(model_id, adapter_name, **kwargs)
+            # NOTICE
+            model.base_model_prepare_inputs_for_generation = model.base_model.prepare_inputs_for_generation
+            model._reorder_cache = model.base_model._reorder_cache
+            return model
+        else:
+            return cls.from_pretrained_old_peft_version(model, model_id, **kwargs)
+    @classmethod
+    def from_pretrained_old_peft_version(cls, model, model_id, **kwargs):
+        # work well in peft@e536616888d51b453ed354a6f1e243fecb02ea08
+        # load the config
+        config = LoraConfig.from_pretrained(model_id)
+        if getattr(model, "hf_device_map", None) is not None:
+            remove_hook_from_submodules(model)
+        # here is the hack
+        model = cls(model, config)
+        model._reorder_cache = model.base_model._reorder_cache
+        # load weights if any
+        if os.path.exists(os.path.join(model_id, "adapter_model.bin")):
+            filename = os.path.join(model_id, "adapter_model.bin")
+        else:
+            try:
+                filename = hf_hub_download(model_id, "adapter_model.bin")
+            except:  # noqa
+                raise ValueError(
+                    f"Can't find weights for {model_id} in {model_id} or in the Hugging Face Hub. "
+                    f"Please check that the file {'adapter_model.bin'} is present at {model_id}."
+                )
+        adapters_weights = torch.load(
+            filename,
+            map_location=torch.device("cuda" if torch.cuda.is_available() else "cpu"),
+        )
+        # load the weights into the model
+        model = set_peft_model_state_dict(model, adapters_weights)
+        if getattr(model, "hf_device_map", None) is not None:
+            device_map = kwargs.get("device_map", "auto")
+            max_memory = kwargs.get("max_memory", None)
+            no_split_module_classes = model._no_split_modules
+            if device_map != "sequential":
+                max_memory = get_balanced_memory(
+                    model,
+                    max_memory=max_memory,
+                    no_split_module_classes=no_split_module_classes,
+                    low_zero=(device_map == "balanced_low_0"),
+                )
+            if isinstance(device_map, str):
+                device_map = infer_auto_device_map(
+                    model,
+                    max_memory=max_memory,
+                    no_split_module_classes=no_split_module_classes,
+                )
+            model = dispatch_model(model, device_map=device_map)
+            hook = AlignDevicesHook(io_same_device=True)
+            if model.peft_config.peft_type == PeftType.LORA:
+                add_hook_to_module(model.base_model.model, hook)
+            else:
+                remove_hook_from_submodules(model.prompt_encoder)
+                add_hook_to_module(model.base_model, hook)
+        return model