Back to all models
Model: ktrapeznikov/biobert_v1.1_pubmed_squad_v2

Monthly model downloads

ktrapeznikov/biobert_v1.1_pubmed_squad_v2 ktrapeznikov/biobert_v1.1_pubmed_squad_v2
- downloads
last 30 days

pytorch

tf

Contributed by

ktrapeznikov Kirill Trapeznikov
3 models

How to use this model directly from the 🤗/transformers library:

			
Copy model
tokenizer = AutoTokenizer.from_pretrained("ktrapeznikov/biobert_v1.1_pubmed_squad_v2") model = AutoModelForQuestionAnswering.from_pretrained("ktrapeznikov/biobert_v1.1_pubmed_squad_v2")

Model

monologg/biobert_v1.1_pubmed fine-tuned on SQuAD V2 using run_squad.py

This model is cased.

Training Parameters

Trained on 4 NVIDIA GeForce RTX 2080 Ti 11Gb

BASE_MODEL=monologg/biobert_v1.1_pubmed
python run_squad.py \
  --version_2_with_negative \
  --model_type albert \
  --model_name_or_path $BASE_MODEL \
  --output_dir $OUTPUT_MODEL \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v2.0.json \
  --predict_file $SQUAD_DIR/dev-v2.0.json \
  --per_gpu_train_batch_size 18 \
  --per_gpu_eval_batch_size 64 \
  --learning_rate 3e-5 \
  --num_train_epochs 3.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --save_steps 2000 \
  --threads 24 \
  --warmup_steps 550 \
  --gradient_accumulation_steps 1 \
  --fp16 \
  --logging_steps 50 \
  --do_train

Evaluation

Evaluation on the dev set. I did not sweep for best threshold.

val
exact 75.97068980038743
f1 79.37043950121722
total 11873.0
HasAns_exact 74.13967611336032
HasAns_f1 80.94892513460755
HasAns_total 5928.0
NoAns_exact 77.79646761984861
NoAns_f1 77.79646761984861
NoAns_total 5945.0
best_exact 75.97068980038743
best_exact_thresh 0.0
best_f1 79.37043950121729
best_f1_thresh 0.0

Usage

See huggingface documentation. Training on SQuAD V2 allows the model to score if a paragraph contains an answer:

start_scores, end_scores = model(input_ids) 
span_scores = start_scores.softmax(dim=1).log()[:,:,None] + end_scores.softmax(dim=1).log()[:,None,:]
ignore_score = span_scores[:,0,0] #no answer scores