unsubscribe
commited on
Commit
•
8fa9dff
1
Parent(s):
2798175
add function call example
Browse files
README.md
CHANGED
@@ -55,6 +55,8 @@ huggingface-cli download internlm/internlm2_5-7b-chat-gguf internlm2_5-7b-chat-f
|
|
55 |
|
56 |
You can use `llama-cli` for conducting inference. For a detailed explanation of `llama-cli`, please refer to [this guide](https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md)
|
57 |
|
|
|
|
|
58 |
```shell
|
59 |
build/bin/llama-cli \
|
60 |
--model internlm2_5-7b-chat-fp16.gguf \
|
@@ -76,6 +78,49 @@ build/bin/llama-cli \
|
|
76 |
--in-suffix "<|im_end|>\n<|im_start|>assistant\n"
|
77 |
```
|
78 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
79 |
## Serving
|
80 |
|
81 |
`llama.cpp` provides an OpenAI API compatible server - `llama-server`. You can deploy `internlm2_5-7b-chat-fp16.gguf` into a service like this:
|
|
|
55 |
|
56 |
You can use `llama-cli` for conducting inference. For a detailed explanation of `llama-cli`, please refer to [this guide](https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md)
|
57 |
|
58 |
+
### chat example
|
59 |
+
|
60 |
```shell
|
61 |
build/bin/llama-cli \
|
62 |
--model internlm2_5-7b-chat-fp16.gguf \
|
|
|
78 |
--in-suffix "<|im_end|>\n<|im_start|>assistant\n"
|
79 |
```
|
80 |
|
81 |
+
### Function call example
|
82 |
+
|
83 |
+
`llama-cli` example:
|
84 |
+
|
85 |
+
```shell
|
86 |
+
build/bin/llama-cli \
|
87 |
+
--model internlm2_5-7b-chat-fp16.gguf \
|
88 |
+
--predict 512 \
|
89 |
+
--ctx-size 4096 \
|
90 |
+
--gpu-layers 32 \
|
91 |
+
--temp 0.8 \
|
92 |
+
--top-p 0.8 \
|
93 |
+
--top-k 50 \
|
94 |
+
--seed 1024 \
|
95 |
+
--color \
|
96 |
+
--prompt '<|im_start|>system\nYou are InternLM2-Chat, a harmless AI assistant.<|im_end|>\n<|im_start|>system name=<|plugin|>[{"name": "get_current_weather", "parameters": {"required": ["location"], "type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "unit": {"type": "string"}}}, "description": "Get the current weather in a given location"}]<|im_end|>\n<|im_start|>user\n' \
|
97 |
+
--interactive \
|
98 |
+
--multiline-input \
|
99 |
+
--conversation \
|
100 |
+
--verbose \
|
101 |
+
--in-suffix "<|im_end|>\n<|im_start|>assistant\n" \
|
102 |
+
--special
|
103 |
+
```
|
104 |
+
|
105 |
+
Conversation results:
|
106 |
+
|
107 |
+
```text
|
108 |
+
<s><|im_start|>system
|
109 |
+
You are InternLM2-Chat, a harmless AI assistant.<|im_end|>
|
110 |
+
<|im_start|>system name=<|plugin|>[{"name": "get_current_weather", "parameters": {"required": ["location"], "type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "unit": {"type": "string"}}}, "description": "Get the current weather in a given location"}]<|im_end|>
|
111 |
+
<|im_start|>user
|
112 |
+
|
113 |
+
> I want to know today's weather in Shanghai
|
114 |
+
I need to use the get_current_weather function to get the current weather in Shanghai.<|action_start|><|plugin|>
|
115 |
+
{"name": "get_current_weather", "parameters": {"location": "Shanghai"}}<|action_end|>
|
116 |
+
<|im_end|>
|
117 |
+
|
118 |
+
> <|im_start|>environment name=<|plugin|>\n{"temperature": 22}
|
119 |
+
The current temperature in Shanghai is 22 degrees Celsius.<|im_end|>
|
120 |
+
|
121 |
+
>
|
122 |
+
```
|
123 |
+
|
124 |
## Serving
|
125 |
|
126 |
`llama.cpp` provides an OpenAI API compatible server - `llama-server`. You can deploy `internlm2_5-7b-chat-fp16.gguf` into a service like this:
|