File size: 4,297 Bytes
1770fa9 e2c289d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 |
---
license: apache-2.0
datasets:
- mllmTeam/DroidCall
language:
- en
library_name: transformers
base_model:
- mllmTeam/PhoneLM-1.5B-Instruct
---
PhoneLM-1.5B-Call is a 1.5 billion parameter decoder-only language model, fined-turned from PhoneLM-1.5B-Instruct, used for Android intent calling.
## Usage
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
model_name = 'mllmTeam/PhoneLM-1.5B-Call'
system_prompt = "You are an expert in composing functions."
user_message = """
Here is a list of functions:
Name:
web_search
Description:
Initiates a web search using the specified query.
This function starts a web search using the default search engine.
It opens the search results in the default web browser or appropriate search application.
Args:
query (str): The search string or keywords to be used for the web search.
engine (str): The search engine to use. Default is "baidu".
Possible values are: "baidu", "google"
Returns:
None
Example:
# Perform a simple web search
web_search("Python programming tutorials")
# Search for a phrase
web_search('"to be or not to be"')
# Search using a specific search engine
web_search("Python programming tutorials", "google")
Now my query is: Help me search the president of United State
"""
prompt = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_message}
]
model = AutoModelForCausalLM.from_pretrained(model_name, device_map='cuda', trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_name)
input_text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True)
inp = tokenizer(input_text, return_tensors="pt")
inp = {k: v.to('cuda') for k, v in inp.items()}
out = model.generate(**inp,
max_length=1000,
do_sample=True,
temperature=0.7,
top_p=0.7
)
text = tokenizer.decode(out[0], skip_special_tokens=True)
print(text)
```
## Model Details
* **Developed by**: mllmTeam
* **Model type**: `PhoneLM 1.5B` models are auto-regressive language models based on the transformer decoder architecture.
* **Language(s)**: English
* **Paper**: [PhoneLM Technical Report]()
* **Library**: [PhoneLM](https://github.com/UbiquitousLearning/PhoneLM)
### Model Architecture
The model is a decoder-only transformer architecture with the following modifications:
| Hidden Size | Layers | Heads | Sequence Length |
|-------------|--------|-------|-----------------|
| 2560 | 19 | 16 | 2048 |
* **Position Embeddings**: Rotary Position Embeddings ([Su et al., 2021](https://arxiv.org/abs/2104.09864)) applied to the first 25% of head embedding dimensions for improved throughput following [Black et al. (2022)](https://arxiv.org/pdf/2204.06745.pdf). PhoneLM quantized the sin and cos values in Rotary Position Embeddings to 8-bit integers.
* **Normalization**: LayerNorm ([Ba et al., 2016](https://arxiv.org/abs/1607.06450)) with learned bias terms as opposed to RMSNorm ([Zhang & Sennrich, 2019](https://arxiv.org/abs/1910.07467)).
* **Biases**: We remove all bias terms from the feed-forward networks and multi-head self-attention layers, except for the biases of the query, key, and value projections ([Bai et al., 2023](https://arxiv.org/abs/2309.16609)).
* **ReLU Activation Function**: ReLU([Glorot et al., 2011](https://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf)) activation functions are adopted in feed-forward networks.
* **Tokenizer**: We use the SmolLM([Allal et al., 2024](https://huggingface.co/blog/smollm))'s tokenizer with a vocabulary size of 49,152.
## License
* This repository is released under the [Apache-2.0](https://huggingface.co/mllmTeam/PhoneLM-1.5B-Call/blob/main/LICENSE) License.
## Citation
```
@misc{yi2024phonelmanefficientcapablesmall,
title={PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training},
author={Rongjie Yi and Xiang Li and Weikai Xie and Zhenyan Lu and Chenghua Wang and Ao Zhou and Shangguang Wang and Xiwen Zhang and Mengwei Xu},
year={2024},
eprint={2411.05046},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2411.05046},
}
``` |