Hugging Face's logo

zhuxunyu
/
math-codet5p-770m-py

Text2Text Generation
Transformers
PyTorch
t5
Inference Endpoints
text-generation-inference
Model card Files Community
math-codet5p-770m-py
File size: 4,130 Bytes 
da82b05
 
5d005ec
 
 
 
da82b05
5d005ec
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
 
---
license: apache-2.0
datasets:
- gsm8k
metrics:
- accuracy
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

We distill math reasoning ability from large language model gpt-3.5-turbo to  the open code small language model [Salesforce/codet5p-770m-py](https://huggingface.co/Salesforce/codet5p-770m-py), and math-codet5p-770m-py achieves 44.88% accuracy on GSM8K testing dataset.


### Model Description

<!-- Provide a longer summary of what this model is. -->



- **Developed by:** Xunyu Zhu
- **Model type:** encoder-decoder
- **Language(s) (NLP):** python
- **License:** apache-2.0
- **Finetuned from model:** [Salesforce/codet5p-770m-py](https://huggingface.co/Salesforce/codet5p-770m-py)



## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
This model can be easily loaded using the AutoModelForSeq2SeqLM functionality and employs the same tokenizer as original [Salesforce/codet5p-770m-py](https://huggingface.co/Salesforce/codet5p-770m-py). 
When given a question, the prompt "\nProgram: Let’s design executable python program (return ans) to solve the question." is needed to add as the input to instruct the model to generate reasoning results.

```python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

def safe_execute(code_string: str, keys=None):
    def execute(x):
        try:
            exec(x)
            locals_ = locals()
            if keys is None:
                return locals_.get('ans', None)
            else:
                return [locals_.get(k, None) for k in keys]
        except Exception:
            return None
    try:
        ans = func_timeout.func_timeout(5, execute, args=(code_string,))
    except func_timeout.FunctionTimedOut:
        ans = None
    return ans

checkpoint = "zhuxunyu/math-codet5p-770m-py"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint).to(device)

question = "Question: Janet\u2019s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?\nProgram: Let’s design executable python program (return ans) to solve the question.".
input = tokenizer(question, max_length=256, padding="max_length", truncation=True, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(**input, max_length=256)

generation = tokenizer.decode(output, skip_special_tokens=True)
ans = safe_execute(generation)
print(float(ans))
```




## Training Details

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

We prompt gpt-3.5-turbo to generate reasoning programs to solve questions in GSM8K training dataset, and each question includes 4 reasoning programs. Then, questions in GSM8K training dataset and 
their corresponding reasoning programs are built as a training dataset, and we use the training dataset to fine-tune the LM.

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->



### Testing Data

<!-- This should link to a Dataset Card if possible. -->

The testing data is GSM8K testing dataset.


### Results

math-codet5p-770m-py achieves 44.88% accuracy on GSM8K testing dataset.



## Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

```
@misc{zhu2023mathcodet5plus,
  title={math-codet5p-770m-py},
  author={Xunyu Zhu, Jian Li, Yong Liu, Can Ma, Weiping Wang},
  year={2023}
}
```