Update README.md

README.md

@@ -1,199 +1,365 @@
 ---
 library_name: transformers
-tags: []
+tags:
+- nlp
+- phi3
+license: mit
+datasets:
+- HuggingFaceFW/fineweb-edu
+language:
+- en
 ---
 
-# Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
+# Model Card for devngho/phi-3-mini-4k-base
 
+<!-- Provide a quick summary of what the model is/does. [Optional] -->
+Finetuned from microsoft/Phi-3-mini-4k-instruct on fineweb-edu (about 150,000 samples).
 
+Research supported with Cloud TPUs from Google's [TPU Research Cloud (TRC)](https://sites.research.google/trc/about/). ⚡
 
 ## Model Details
 
-### Model Description
+- **Developed by:** devngho
+- **Language(s) (NLP):** en
+- **License:** mit
+- **Parent Model:** [microsoft/Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct)
+
+## Samples
+
+- max_tokens: 256
+- \<s> token is omitted
+- temperature: 0
+
+### Sample 1
+Prompt: 
+```md
+Korean is written with an alphabet called Hangul, which was created in the 15th century by King Sejong the Great and his scholars. How King Sejong the Great created Hangul?
+```
+
+<details>
+<summary>
+This model(devngho/phi-3-mini-4k-base):
+</summary>
+<p>
+Korean is written with an alphabet called Hangul, which was created in the 15th century by King Sejong the Great and his scholars. How King Sejong the Great created Hangul?
+
+A. He was inspired by the Chinese writing system.
+B. He was inspired by the Japanese writing system.
+C. He was inspired by the Mongolian writing system.
+D. He was inspired by the Korean writing system.
+
+- response: King Sejong the Great, the fourth king of the Joseon Dynasty in Korea, created Hangul in the 15th century with the aim of promoting literacy among the common people. Prior to Hangul, the Korean writing system was based on Classical Chinese characters, which were complex and difficult for the general population to learn.
+
+King Sejong and his scholars were inspired by the need to create a writing system that was easy to learn and use, and that could accurately represent the sounds of the Korean language. They were not directly inspired by the Chinese, Japanese, or Mongolian writing systems, but rather by the desire to create a system that was uniquely Korean and accessible to all
+</p>
+</details>
+<details>
+<summary>
+Original model(microsoft/Phi-3-mini-4k-instruct):
+</summary>
+<p>
+Korean is written with an alphabet called Hangul, which was created in the 15th century by King Sejong the Great and his scholars. How King Sejong the Great created Hangul?<|end|>
+</p>
+</details>
+<details>
+<summary>
+Other model(gpt-3.5-turbo-instruct - completions model):
+</summary>
+<p>
+Korean is written with an alphabet called Hangul, which was created in the 15th century by King Sejong the Great and his scholars. How King Sejong the Great created Hangul?
+
+King Sejong the Great wanted to create a writing system that was easy to learn and use for the common people, as the existing writing system at the time, Chinese characters, were difficult to learn and only accessible to the elite. He believed that a writing system that was easy to learn would improve literacy rates and promote education among the common people.
+
+To create Hangul, King Sejong the Great gathered a group of scholars and linguists and tasked them with creating a new writing system. They studied the sounds of the Korean language and based the alphabet on the shape and position of the mouth when making these sounds.
+
+The alphabet was initially called Hunminjeongeum, which means "the correct sounds for the instruction of the people." It consisted of 28 letters, 17 consonants and 11 vowels, and was designed to be simple and easy to learn. The letters were also organized into syllable blocks, making it easier to read and write.
+
+Hangul was officially promulgated in 1446 and was met with resistance from the elite who saw it as a threat to their status. However, King Sejong the Great continued to promote and encourage the use of Hangul, and it eventually became the primary writing system in Korea.
+
+Today, Hangul is
+</p>
+</details>
+
+Author's Note: It's still seems to be instruct-following model, but the response is more better than the original model as it doesn't have the <|end|> token.
+
+### Sample 2
+Prompt: 
+```md
+South Korea has experienced many wars in modern times. For instance,
+```
+
+<details>
+<summary>
+This model(devngho/phi-3-mini-4k-base):
+</summary>
+<p>
+South Korea has experienced many wars in modern times. For instance, the Korean War (1950-1953) was a conflict between North Korea and South Korea, with the United States and China supporting the former and the latter, respectively. The war ended in a stalemate, with the two Koreas remaining divided along the 38th parallel.
+
+The Korean War had a significant impact on the country's economy. The war caused widespread destruction, with many cities and towns being destroyed. The war also disrupted trade and caused a shortage of resources. The South Korean government had to rely on foreign aid to rebuild the country.
+
+One of the most significant impacts of the Korean War on the economy was the loss of human capital. Many young men were killed or injured during the war, and their families had to bear the burden of their loss. The war also caused a brain drain, with many skilled workers and professionals leaving the country to seek better opportunities abroad.
+
+The Korean War also had a significant impact on the country's agriculture. The war disrupted the supply chain, and many farmers were unable to sell their produce. The government had to import
+</p>
+</details>
+<details>
+<summary>
+Original model(microsoft/Phi-3-mini-4k-instruct):
+</summary>
+<p>
+South Korea has experienced many wars in modern times. For instance, the Korean War (1950-1953) was a significant conflict that involved North and South Korea, as well as international forces. The war ended with an armistice agreement, but no formal peace treaty was signed, leaving the two Koreas technically still at war.
+
+In addition to the Korean War, South Korea has faced other conflicts and tensions, such as the Korean DMZ Conflict (1966-1969) and the Jeju Uprising (1948). These events have shaped South Korea's history and continue to influence its relationships with neighboring countries.
+
+Despite these conflicts, South Korea has made significant progress in terms of economic development, technological advancements, and cultural influence. The country has become a major player in the global economy, with a strong focus on industries such as electronics, automotive, and entertainment.
+
+South Korea's success can be attributed to various factors, including its emphasis on education, innovation, and a strong work ethic. The country has also invested heavily in infrastructure and technology, which has
+</p>
+</details>
+<details>
+<summary>
+Other model(gpt-3.5-turbo-instruct - completions model):
+</summary>
+<p>
+South Korea has experienced many wars in modern times. For instance, the country was involved in the Korean War from 1950 to 1953, which was a conflict between North and South Korea. This war resulted in millions of casualties and the division of the Korean peninsula into two separate countries.
+
+In addition, South Korea was also involved in the Vietnam War from 1964 to 1973, where it sent over 300,000 troops to support the United States and South Vietnam against North Vietnam and the Viet Cong. This war also resulted in thousands of casualties for South Korea.
+
+In more recent times, South Korea has also faced tensions and conflicts with North Korea, including the Korean DMZ Conflict in 1966, the Korean Axe Murder Incident in 1976, and the Yeonpyeong Island shelling in 2010. These conflicts have resulted in casualties and heightened tensions between the two countries.
+
+South Korea has also been involved in international conflicts, such as the Gulf War in 1991 and the War in Afghanistan from 2001 to 2014, where it sent troops to support the United States and its allies.
+
+Overall, South Korea has a long history of experiencing wars and conflicts, both domestically and internationally. However, the country has also made efforts to promote peace and reconciliation, such as through the Sunshine Policy
+</p>
+</details>
+
+Author's Note: These responses are very similar.
+
+### Sample 3
+Prompt: 
+```md
+Hangul is a writing system made by King Sejong in
+```
+
+<details>
+<summary>
+This model(devngho/phi-3-mini-4k-base):
+</summary>
+<p>
+Hangul is a writing system made by King Sejong in 1443. It is the official writing system of Korea. Hangul is a phonetic alphabet, which means that the letters represent the sounds of the Korean language.
 
-<!-- Provide a longer summary of what this model is. -->
+## The Creation of Hangul
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+King Sejong wanted to create a writing system that was easy to learn and use for the common people. He believed that the existing writing system, which was based on Chinese characters, was too difficult for the average person to learn.
+
+## The Structure of Hangul
+
+Hangul is made up of 14 consonants and 10 vowels. These letters are combined to form syllables, which are the building blocks of the Korean language. The consonants and vowels are arranged in a square-like structure, with each row and column representing a different sound.
+
+## The Importance of Hangul
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+Hangul is an important part of Korean culture and identity. It is used in all aspects of life, from education to business to entertainment. The Korean government has made efforts to promote the use of Hangul and preserve its cultural significance.
 
-### Model Sources [optional]
+</p>
+</details>
+<details>
+<summary>
+Original model(microsoft/Phi-3-mini-4k-instruct):
+</summary>
+<p>
+Hangul is a writing system made by King Sejong in the 15th century.
+<|assistant|>
+</p>
+</details>
+<details>
+<summary>
+Other model(gpt-3.5-turbo-instruct - completions model):
+</summary>
+<p>
+Hangul is a writing system made by King Sejong in 1443. It is the official writing system of Korea and is used to write the Korean language. Hangul is unique in that it is a phonetic alphabet, meaning that each letter represents a specific sound. This makes it relatively easy to learn and use compared to other writing systems.
 
-<!-- Provide the basic links for the model. -->
+Hangul is made up of 14 consonants and 10 vowels, which can be combined to form syllables. The letters are arranged in a block-like structure, with each block representing a syllable. The consonants are written in a vertical line on the left side, while the vowels are written in a horizontal line on the top or bottom of the consonant.
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+One of the main reasons King Sejong created Hangul was to increase literacy among the common people. Before Hangul, the Korean language was written using Chinese characters, which were difficult to learn and only accessible to the elite. Hangul made it easier for people to learn how to read and write, leading to a significant increase in literacy rates in Korea.
 
-## Uses
+Hangul has also played a significant role in preserving the Korean language and culture. During the Japanese occupation of Korea in the early 20th century, the use of Hangul was banned, and the Korean language was suppressed. However, after Korea gained independence,
+</p>
+</details>
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+Author's Note: Unlike the other samples, the model's response seems to be similar to general articles or gpt-3.5-turbo-instruct, not chat-style, as I hoped.
 
-### Direct Use
+### Sample 4
+Prompt: 
+```md
+TPU(Tensor Processing Unit) was created by
+```
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+<details>
+<summary>
+This model(devngho/phi-3-mini-4k-base):
+</summary>
+<p>
+TPU(Tensor Processing Unit) was created by Google to accelerate the training of deep neural networks. TPUs are designed to be highly efficient for the matrix operations that are common in deep learning. They are similar to GPUs but are optimized for TensorFlow, Google's machine learning framework.
 
-[More Information Needed]
+Here's a simple example of how you can use TPUs in TensorFlow to train a neural network:
 
-### Downstream Use [optional]
+First, you need to install the TensorFlow package with TPU support:
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+```bash
+pip install tensorflow-gpu
+```
 
-[More Information Needed]
+Then, you can set up your TPU environment:
 
-### Out-of-Scope Use
+```python
+import tensorflow as tf
+
+# Check if TPU is available
+tpu = tf.distribute.cluster_resolver.TPUClusterResolver()
+print('Running on TPU ', tpu.master())
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+# Initialize the TPU
+tf.config.experimental_connect_to_cluster(tpu)
+tf.tpu.experimental.initialize_tpu_system(tpu)
 
-[More Information Needed]
+# Create a strategy for TPU distribution
+strategy = tf.distribute.experimental.TPUStrategy(
+```
+</p>
+</details>
+<details>
+<summary>
+Original model(microsoft/Phi-3-mini-4k-instruct):
+</summary>
+<p>
+TPU(Tensor Processing Unit) was created by Google.
 
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+> TPU is a Google-designed accelerator that is optimized for
+> training and inference of machine learning models.
 
-#### Speeds, Sizes, Times [optional]
+I'm wondering if there is any open source implementation of TPU?
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+I've found this one: https://github.com/tensorflow/models/tree/master/official/transformer
 
-[More Information Needed]
+But it seems to be a very small part of TPU.
 
-## Evaluation
+- [Response]: Tensor Processing Units (TPUs) are custom-designed hardware accelerators developed by Google to efficiently run machine learning workloads, particularly those involving neural networks. While TPUs are proprietary and not open-source hardware, there are open-source software implementations that can run on TPUs or similar hardware architectures.
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+The TensorFlow ecosystem, which is Google's open-source machine learning framework, has made significant strides in providing support for TPUs. TensorFlow provides a TPU-optimized version of its runtime, which can be used to run TensorFlow models on TPUs.
 
-### Testing Data, Factors & Metrics
+Here are some open-source implementations and resources related to TPU
+</p>
+</details>
+<details>
+<summary>
+Other model(gpt-3.5-turbo-instruct - completions model):
+</summary>
+<p>
+TPU(Tensor Processing Unit) was created by Google for the purpose of accelerating machine learning workloads. It is a specialized chip designed specifically for deep learning tasks, such as training and inference of neural networks. 
+
+TPUs are highly efficient and can provide significant speedups compared to traditional CPUs and GPUs. They are optimized for matrix multiplication, which is a key operation in neural network computations. This allows them to process large amounts of data in parallel, making them ideal for training and running complex models.
+
+One of the main advantages of TPUs is their scalability. They can be used individually or in clusters, allowing for even faster processing of large datasets. They also have a high memory bandwidth, which enables them to handle large models and datasets without running into memory constraints.
+
+Another key feature of TPUs is their low power consumption. They are designed to be energy-efficient, making them a more environmentally friendly option compared to traditional hardware.
+
+TPUs are primarily used in Google's cloud computing platform, Google Cloud, but they can also be accessed through Google Colab, a free online platform for running machine learning experiments. This makes TPUs accessible to a wider range of users, including researchers and developers who may not have access to specialized hardware.
+
+Overall, TPUs have revolutionized the field of deep learning by providing a powerful and efficient hardware solution
+</p>
+</details>
 
-#### Testing Data
+Author's Note: The model provides a simple example of how to use TPUs in TensorFlow, which is a good response. Original model is Q&A style.
 
-<!-- This should link to a Dataset Card if possible. -->
+### Sample 5
+Prompt: 
+```md
+Hello!
+```
 
-[More Information Needed]
+<details>
+<summary>
+This model(devngho/phi-3-mini-4k-base):
+</summary>
+<p>
+Hello! I'm your friendly AI chatbot, here to help you with your homework. Could you please tell me which subject and topic you need assistance with?
 
-#### Factors
+## Conversation
+**User**: Hey there! I'm working on a project for my history class, but it's a bit of a mix. We're looking at the impact of technology on society, but it's got to tie in with a historical event. Any ideas?
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+**Chatbot**: That sounds like an interesting project! How about exploring the impact of the printing press on the Reformation? The printing press was a significant technological advancement that greatly influenced the spread of ideas during the Reformation.
 
-[More Information Needed]
+**User**: That's a good start, but I was thinking more along the lines of something more recent, like the internet.
 
-#### Metrics
+**Chatbot**: Understood. You could examine the role of the internet in the Arab Spring. The widespread use of social media and the internet played a crucial role in organizing protests and disseminating information during that period.
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+**User**: Yeah, that's more like it. But I'
+</p>
+</details>
+<details>
+<summary>
+Original model(microsoft/Phi-3-mini-4k-instruct):
+</summary>
+<p>
+Hello! I'd be happy to help you with your SQL queries. Could you please describe your data or provide the table structure you're working with?
 
-[More Information Needed]
+<|assistant|>
+</p>
+</details>
+<details>
+<summary>
+Other model(gpt-3.5-turbo-instruct - completions model):
+</summary>
+<p>
+Hello! I am a 22 year old female who is looking for a room to rent in the city of Toronto. I am a recent university graduate and will be starting a full-time job in the downtown area in September. I am a clean, responsible, and friendly individual who enjoys cooking, reading, and exploring the city. I am looking for a room in a shared house or apartment with other young professionals or students. My budget is around $800-1000 per month. Please contact me if you have a room available. Thank you!
+</p>
+</details>
 
-### Results
+Author's Note: The model's response is still instruct-following, it just doesn't use the <|assistant|> token. In contrast, gpt-3.5-turbo-instruct's response looks a normal introduction.
 
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
+## Bias, Risks, and Limitations
 
-<!-- Relevant interpretability work for the model goes here -->
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-[More Information Needed]
+Since the base model is a instruct-following model, sometimes the model may generate chat-like responses (See Samples).
 
-## Environmental Impact
+## Training Details
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+### Training Data
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+[fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (about 150,000 samples)
 
-## Technical Specifications [optional]
 
-### Model Architecture and Objective
+### Training Procedure
 
-[More Information Needed]
+- batch_size: 64
+- lr: 3e-5
+- lr_scheduler: cosine
+- torch_dtype: bfloat16
+- warmup_ratio: 0.2
+- optimizer: adamw
+- seed: 42
+- gradient_accumulation: 1
+- gradient_checkpointing: true
+- FSDPv2 (FSDP via SPMD)
 
 ### Compute Infrastructure
 
-[More Information Needed]
+Google Cloud TPU
 
 #### Hardware
 
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
+TPU v4-32, took  ~12 hours.
 
-## Citation [optional]
+Research supported with Cloud TPUs from Google's [TPU Research Cloud (TRC)](https://sites.research.google/trc/about/). ⚡
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
+#### Software
 
-[More Information Needed]
+transformers\~=4.41.2 torch\~=2.3.0 torch_xla\[tpu]\~=2.3.0
 
-## Model Card Contact
+### Train Results
 
-[More Information Needed]
+- train/loss: 2.22385830132309