shibing624 commited on
Commit
021a4bf
1 Parent(s): e8ab03f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +181 -0
README.md CHANGED
@@ -1,3 +1,184 @@
1
  ---
 
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ pipeline_tag: sentence-similarity
3
  license: apache-2.0
4
+ tags:
5
+ - text2vec
6
+ - feature-extraction
7
+ - sentence-similarity
8
+ - transformers
9
+ datasets:
10
+ - https://huggingface.co/datasets/shibing624/nli-zh-all/tree/main/text2vec-base-multilingual-dataset
11
+ language:
12
+ - zh
13
+ metrics:
14
+ - spearmanr
15
+ library_name: transformers
16
  ---
17
+ # shibing624/text2vec-base-multilingual
18
+ This is a CoSENT(Cosine Sentence) model: shibing624/text2vec-base-multilingual.
19
+
20
+ It maps sentences to a 384 dimensional dense vector space and can be used for tasks
21
+ like sentence embeddings, text matching or semantic search.
22
+
23
+
24
+
25
+ - training dataset: https://huggingface.co/datasets/shibing624/nli-zh-all/tree/main/text2vec-base-multilingual-dataset
26
+ - base model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
27
+ - max_seq_length: 256
28
+ - best epoch: 4
29
+ - sentence embedding dim: 384
30
+
31
+ ## Evaluation
32
+ For an automated evaluation of this model, see the *Evaluation Benchmark*: [text2vec](https://github.com/shibing624/text2vec)
33
+
34
+ ### Release Models
35
+
36
+ - 本项目release模型的中文匹配评测结果:
37
+
38
+ | Arch | BaseModel | Model | ATEC | BQ | LCQMC | PAWSX | STS-B | SOHU-dd | SOHU-dc | Avg | QPS |
39
+ |:-----------|:-------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------|:-----:|:-----:|:-----:|:-----:|:-----:|:-------:|:-------:|:---------:|:-----:|
40
+ | Word2Vec | word2vec | [w2v-light-tencent-chinese](https://ai.tencent.com/ailab/nlp/en/download.html) | 20.00 | 31.49 | 59.46 | 2.57 | 55.78 | 55.04 | 20.70 | 35.03 | 23769 |
41
+ | SBERT | xlm-roberta-base | [sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) | 18.42 | 38.52 | 63.96 | 10.14 | 78.90 | 63.01 | 52.28 | 46.46 | 3138 |
42
+ | Instructor | hfl/chinese-roberta-wwm-ext | [moka-ai/m3e-base](https://huggingface.co/moka-ai/m3e-base) | 41.27 | 63.81 | 74.87 | 12.20 | 76.96 | 75.83 | 60.55 | 57.93 | 2980 |
43
+ | CoSENT | hfl/chinese-macbert-base | [shibing624/text2vec-base-chinese](https://huggingface.co/shibing624/text2vec-base-chinese) | 31.93 | 42.67 | 70.16 | 17.21 | 79.30 | 70.27 | 50.42 | 51.61 | 3008 |
44
+ | CoSENT | hfl/chinese-lert-large | [GanymedeNil/text2vec-large-chinese](https://huggingface.co/GanymedeNil/text2vec-large-chinese) | 32.61 | 44.59 | 69.30 | 14.51 | 79.44 | 73.01 | 59.04 | 53.12 | 2092 |
45
+ | CoSENT | nghuyong/ernie-3.0-base-zh | [shibing624/text2vec-base-chinese-sentence](https://huggingface.co/shibing624/text2vec-base-chinese-sentence) | 43.37 | 61.43 | 73.48 | 38.90 | 78.25 | 70.60 | 53.08 | 59.87 | 3089 |
46
+ | CoSENT | nghuyong/ernie-3.0-base-zh | [shibing624/text2vec-base-chinese-paraphrase](https://huggingface.co/shibing624/text2vec-base-chinese-paraphrase) | 44.89 | 63.58 | 74.24 | 40.90 | 78.93 | 76.70 | 63.30 | **63.08** | 3066 |
47
+ | CoSENT | sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 | [shibing624/text2vec-base-multilingual](https://huggingface.co/shibing624/text2vec-base-multilingual) | 32.39 | 50.33 | 65.64 | 32.56 | 74.45 | 68.88 | 51.17 | 53.67 | 4004 |
48
+
49
+
50
+ 说明:
51
+ - 结果评测指标:spearman系数
52
+ - `shibing624/text2vec-base-chinese`模型,是用CoSENT方法训练,基于`hfl/chinese-macbert-base`在中文STS-B数据训练得到,并在中文STS-B测试集评估达到较好效果,运行[examples/training_sup_text_matching_model.py](https://github.com/shibing624/text2vec/blob/master/examples/training_sup_text_matching_model.py)代码可训练模型,模型文件已经上传HF model hub,中文通用语义匹配任务推荐使用
53
+ - `shibing624/text2vec-base-chinese-sentence`模型,是用CoSENT方法训练,基于`nghuyong/ernie-3.0-base-zh`用人工挑选后的中文STS数据集[shibing624/nli-zh-all/text2vec-base-chinese-sentence-dataset](https://huggingface.co/datasets/shibing624/nli-zh-all/tree/main/text2vec-base-chinese-sentence-dataset)训练得到,并在中文各NLI测试集评估达到较好效果,运行[examples/training_sup_text_matching_model_jsonl_data.py](https://github.com/shibing624/text2vec/blob/master/examples/training_sup_text_matching_model_jsonl_data.py)代码可训练模型,模型文件已经上传HF model hub,中文s2s(句子vs句子)语义匹配任务推荐使用
54
+ - `shibing624/text2vec-base-chinese-paraphrase`模型,是用CoSENT方法训练,基于`nghuyong/ernie-3.0-base-zh`用人工挑选后的中文STS数据集[shibing624/nli-zh-all/text2vec-base-chinese-paraphrase-dataset](https://huggingface.co/datasets/shibing624/nli-zh-all/tree/main/text2vec-base-chinese-paraphrase-dataset),数据集相对于[shibing624/nli-zh-all/text2vec-base-chinese-sentence-dataset](https://huggingface.co/datasets/shibing624/nli-zh-all/tree/main/text2vec-base-chinese-sentence-dataset)加入了s2p(sentence to paraphrase)数据,强化了其长文本的表征能力,并在中文各NLI测试集评估达到SOTA,运行[examples/training_sup_text_matching_model_jsonl_data.py](https://github.com/shibing624/text2vec/blob/master/examples/training_sup_text_matching_model_jsonl_data.py)代码可训练模型,模型文件已经上传HF model hub,中文s2p(句子vs段落)语义匹配任务推荐使用
55
+ - `shibing624/text2vec-base-multilingual`模型,是用CoSENT方法训练,基于`sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`用人工挑选后的多语言STS数据集[shibing624/nli-zh-all/text2vec-base-multilingual-dataset](https://huggingface.co/datasets/shibing624/nli-zh-all/tree/main/text2vec-base-multilingual-dataset)训练得到,并在中英文测试集评估相对于原模型效果有提升,运行[examples/training_sup_text_matching_model_jsonl_data.py](https://github.com/shibing624/text2vec/blob/master/examples/training_sup_text_matching_model_jsonl_data.py)代码可训练模型,模型文件已经上传HF model hub,多语言语义匹配任务推荐使用
56
+ - `w2v-light-tencent-chinese`是腾讯词向量的Word2Vec模型,CPU加载使用,适用于中文字面匹配任务和缺少数据的冷启动情况
57
+ - 各预训练模型均可以通过transformers调用,如MacBERT模型:`--model_name hfl/chinese-macbert-base` 或者roberta模型:`--model_name uer/roberta-medium-wwm-chinese-cluecorpussmall`
58
+ - 为测评模型的鲁棒性,加入了未训练过的SOHU测试集,用于测试模型的泛化能力;为达到开箱即用的实用效果,使用了搜集到的各中文匹配数据集,数据集也上传到HF datasets[链接见下方](#数据集)
59
+ - 中文匹配任务实验表明,pooling最优是`EncoderType.FIRST_LAST_AVG`和`EncoderType.MEAN`,两者预测效果差异很小
60
+ - 中文匹配评测结果复现,可以下载中文匹配数据集到`examples/data`,运行[tests/test_model_spearman.py](https://github.com/shibing624/text2vec/blob/master/tests/test_model_spearman.py)代码复现评测结果
61
+ - QPS的GPU测试环境是Tesla V100,显存32GB
62
+
63
+ 模型训练实验报告:[实验报告](https://github.com/shibing624/text2vec/blob/master/docs/model_report.md)
64
+
65
+ ## Usage (text2vec)
66
+ Using this model becomes easy when you have [text2vec](https://github.com/shibing624/text2vec) installed:
67
+
68
+ ```
69
+ pip install -U text2vec
70
+ ```
71
+
72
+ Then you can use the model like this:
73
+
74
+ ```python
75
+ from text2vec import SentenceModel
76
+ sentences = ['如何更换花呗绑定银行卡', 'How to replace the Huabei bundled bank card']
77
+
78
+ model = SentenceModel('shibing624/text2vec-base-multilingual')
79
+ embeddings = model.encode(sentences)
80
+ print(embeddings)
81
+ ```
82
+
83
+ ## Usage (HuggingFace Transformers)
84
+ Without [text2vec](https://github.com/shibing624/text2vec), you can use the model like this:
85
+
86
+ First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
87
+
88
+ Install transformers:
89
+ ```
90
+ pip install transformers
91
+ ```
92
+
93
+ Then load model and predict:
94
+ ```python
95
+ from transformers import AutoTokenizer, AutoModel
96
+ import torch
97
+
98
+ # Mean Pooling - Take attention mask into account for correct averaging
99
+ def mean_pooling(model_output, attention_mask):
100
+ token_embeddings = model_output[0] # First element of model_output contains all token embeddings
101
+ input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
102
+ return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
103
+
104
+ # Load model from HuggingFace Hub
105
+ tokenizer = AutoTokenizer.from_pretrained('shibing624/text2vec-base-multilingual')
106
+ model = AutoModel.from_pretrained('shibing624/text2vec-base-multilingual')
107
+ sentences = ['如何更换花呗绑定银行卡', 'How to replace the Huabei bundled bank card']
108
+ # Tokenize sentences
109
+ encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
110
+
111
+ # Compute token embeddings
112
+ with torch.no_grad():
113
+ model_output = model(**encoded_input)
114
+ # Perform pooling. In this case, mean pooling.
115
+ sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
116
+ print("Sentence embeddings:")
117
+ print(sentence_embeddings)
118
+ ```
119
+
120
+ ## Usage (sentence-transformers)
121
+ [sentence-transformers](https://github.com/UKPLab/sentence-transformers) is a popular library to compute dense vector representations for sentences.
122
+
123
+ Install sentence-transformers:
124
+ ```
125
+ pip install -U sentence-transformers
126
+ ```
127
+
128
+ Then load model and predict:
129
+
130
+ ```python
131
+ from sentence_transformers import SentenceTransformer
132
+
133
+ m = SentenceTransformer("shibing624/text2vec-base-multilingual")
134
+ sentences = ['如何更换花呗绑定银行卡', 'How to replace the Huabei bundled bank card']
135
+
136
+ sentence_embeddings = m.encode(sentences)
137
+ print("Sentence embeddings:")
138
+ print(sentence_embeddings)
139
+ ```
140
+
141
+
142
+ ## Full Model Architecture
143
+ ```
144
+ CoSENT(
145
+ (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel
146
+ (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_mean_tokens': True})
147
+ )
148
+ ```
149
+
150
+
151
+ ## Intended uses
152
+
153
+ Our model is intented to be used as a sentence and short paragraph encoder. Given an input text, it ouptuts a vector which captures
154
+ the semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks.
155
+
156
+ By default, input text longer than 256 word pieces is truncated.
157
+
158
+
159
+ ## Training procedure
160
+
161
+ ### Pre-training
162
+
163
+ We use the pretrained [`sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) model.
164
+ Please refer to the model card for more detailed information about the pre-training procedure.
165
+
166
+ ### Fine-tuning
167
+
168
+ We fine-tune the model using a contrastive objective. Formally, we compute the cosine similarity from each
169
+ possible sentence pairs from the batch.
170
+ We then apply the rank loss by comparing with true pairs and false pairs.
171
+
172
+
173
+ ## Citing & Authors
174
+ This model was trained by [text2vec](https://github.com/shibing624/text2vec).
175
+
176
+ If you find this model helpful, feel free to cite:
177
+ ```bibtex
178
+ @software{text2vec,
179
+ author = {Ming Xu},
180
+ title = {text2vec: A Tool for Text to Vector},
181
+ year = {2023},
182
+ url = {https://github.com/shibing624/text2vec},
183
+ }
184
+ ```