README.md

---
license: apache-2.0
language: 
- zh
---



# Model Card for Chinese MRC roberta_wwm_ext_large
 
# Model Details
 
## Model Description
 
使用大量中文MRC数据训练的roberta_wwm_ext_large模型，[详情可查看](https://github.com/basketballandlearn/MRC_Competition_Dureader)
 
- **Developed by:** luhua-rain
- **Shared by [Optional]:**  luhua-rain
- **Model type:** Question Answering 
- **Language(s) (NLP):** Chinese
- **License:** Apache 2.0 
- **Parent Model:** BERT
- **Resources for more information:**
  - [GitHub Repo](https://github.com/basketballandlearn/MRC_Competition_Dureader)
 	


# Uses
 

## Direct Use
The model authors also note in the [GitHub Repo](https://github.com/basketballandlearn/MRC_Competition_Dureader)
> 此mrc模型可直接用于open domain，点击体验
 
## Downstream Use [Optional]
 
The model authors also note in the [GitHub Repo](https://github.com/basketballandlearn/MRC_Competition_Dureader)
> 将此模型放到下游 MRC/分类 任务微调可比直接使用预训练语言模型提高2个点/1个点以上
 
## Out-of-Scope Use
 
The model should not be used to intentionally create hostile or alienating environments for people. 
 
# Bias, Risks, and Limitations
 
 
Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.



## Recommendations
 
 
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.


# Training Details
 
## Training Data
 
The model authors also note in the [GitHub Repo](https://github.com/basketballandlearn/MRC_Competition_Dureader)
> 网上收集的大量中文MRC数据 （其中包括公开的MRC数据集以及自己爬取的网页数据等， 囊括了医疗、教育、娱乐、百科、军事、法律、等领域。）
 
## Training Procedure

 
### Preprocessing
The model authors also note in the [GitHub Repo](https://github.com/basketballandlearn/MRC_Competition_Dureader): 
>**清洗**
舍弃：context>1024的舍弃、question>64的舍弃、网页标签占比超过30%的舍弃。
重新标注：若answer>64且不完全出现在文档中，则采用模糊匹配: 计算所有片段与answer的相似度(F1值)，取相似度最高的且高于阈值（0.8）
**数据标注**
收集的数据有一部分是不包含的位置标签的，仅仅是(问题-文章-答案)的三元组形式。 所以，对于只有答案而没有位置标签的数据通过正则匹配进行位置标注：
 若答案片段多次出现在文章中，选择上下文与问题最相似的答案片段作为标准答案（使用F1值计算相似度，答案片段的上文48和下文48个字符作为上下文）；
 若答案片段只出现一次，则默认该答案为标准答案。
采用滑动窗口将长文档切分为多个重叠的子文档，故一个文档可能会生成多个有答案的子文档。
**无答案数据构造**
     在跨领域数据上训练可以增加数据的领域多样性，进而提高模型的泛化能力，而负样本的引入恰好能使得模型编码尽可能多的数据，加强模型对难样本的识别能力：
1.)  对于每一个问题，随机从数据中捞取context，并保留对应的title作为负样本;（50%）
2.)  对于每一个问题，将其正样本中答案出现的句子删除，以此作为负样本；（20%）
3.)  对于每一个问题，使用BM25算法召回得分最高的前十个文档，然后根据得分采样出一个context作为负样本， 对于非实体类答案，剔除得分最高的context（30%）
 
 
 


 
### Speeds, Sizes, Times
More information needed 

 
# Evaluation
 
 
## Testing Data, Factors & Metrics
 
### Testing Data
 
More information needed 
 
 
### Factors
More information needed
 
### Metrics
 
More information needed
 
 
## Results 

* 此库发布的再训练模型，在 阅读理解/分类 等任务上均有大幅提高<br/>
（已有多位小伙伴在Dureader-2021等多个比赛中取得**top5**的成绩😁）

|                模型/数据集                 |  Dureader-2021  |  tencentmedical |
| ------------------------------------------|--------------- | --------------- |
|                                           |    F1-score    |    Accuracy     |
|                                           |  dev / A榜     |     test-1      |
| macbert-large (哈工大预训练语言模型)         | 65.49 / 64.27  |     82.5        |
| roberta-wwm-ext-large (哈工大预训练语言模型) | 65.49 / 64.27  |     82.5        |
| macbert-large (ours)                      | 70.45 / **68.13**|   **83.4**    |
| roberta-wwm-ext-large (ours)              | 68.91 / 66.91   |    83.1        |

       | 68.91 / 66.91   |    83.1        |




 
# Model Examination
 
More information needed
 
# Environmental Impact
 
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
- **Hardware Type:** More information needed
- **Hours used:** More information needed
- **Cloud Provider:** More information needed
- **Compute Region:** More information needed
- **Carbon Emitted:** More information needed
 
# Technical Specifications [optional]
 
## Model Architecture and Objective

More information needed 
 
## Compute Infrastructure
 
More information needed 
 
### Hardware
 
 
More information needed
 
### Software
 
More information needed.
 
# Citation

 
**BibTeX:**
 
 
More information needed 
 
 
 
 
# Glossary [optional]
More information needed 
 
# More Information [optional]
The model authors also note in the [GitHub Repo](https://github.com/basketballandlearn/MRC_Competition_Dureader)
> 代码上传前已经跑通。文件不多，所以如果碰到报错之类的信息，可能是代码路径不对、缺少安装包等问题，一步步解决，可以提issue
环境
 

 
# Model Card Authors [optional]
 
Luhua-rain in collaboration with Ezi Ozoani and the Hugging Face team


# Model Card Contact
 
The model authors also note in the [GitHub Repo](https://github.com/basketballandlearn/MRC_Competition_Dureader)
> 合作
相关训练数据以及使用更多数据训练的模型/一起打比赛 可邮箱联系(luhua98@foxmail.com)~

 
# How to Get Started with the Model
 
Use the code below to get started with the model.
 
<details>
<summary> Click to expand </summary>

```python
 ----- 使用方法 -----
from transformers import AutoTokenizer, AutoModelForQuestionAnswering

model_name = "chinese_pretrain_mrc_roberta_wwm_ext_large" # "chinese_pretrain_mrc_macbert_large"

# Use in Transformers
tokenizer = AutoTokenizer.from_pretrained(f"luhua/{model_name}")
model = AutoModelForQuestionAnswering.from_pretrained(f"luhua/{model_name}")

# Use locally（通过 https://huggingface.co/luhua 下载模型及配置文件）
tokenizer = BertTokenizer.from_pretrained(f'./{model_name}')
model = AutoModelForQuestionAnswering.from_pretrained(f'./{model_name}')
 ```
</details>