ankur310794's picture
updated
c8fc850
metadata
tags:
  - generated_from_keras_callback
  - dpr
license: apache-2.0
model-index:
  - name: dpr-question_encoder_bert_uncased_L-2_H-128_A-2
    results: []

dpr-question_encoder_bert_uncased_L-2_H-128_A-2

This model(google/bert_uncased_L-2_H-128_A-2) was trained from scratch on training data: data.retriever.nq-adv-hn-train(facebookresearch/DPR). It achieves the following results on the evaluation set:

Evaluation data

evaluation dataset: facebook-dpr-dev-dataset from official DPR github

model_name data_name num of queries num of passages R@10 R@20 R@50 R@100 R@100
nlpconnect/dpr-ctx_encoder_bert_uncased_L-2_H-128_A-2(our) nq-dev dataset 6445 199795 60.53% 68.28% 76.07% 80.98% 91.45%
nlpconnect/dpr-ctx_encoder_bert_uncased_L-12_H-128_A-2(our) nq-dev dataset 6445 199795 65.43% 71.99% 79.03% 83.24% 92.11%
*facebook/dpr-ctx_encoder-single-nq-base(hf/fb) nq-dev dataset 6445 199795 40.94% 49.27% 59.05% 66.00% 82.00%

evaluation dataset: UKPLab/beir test data but we have used first 2lac passage only.

model_name data_name num of queries num of passages R@10 R@20 R@50 R@100 R@100
nlpconnect/dpr-ctx_encoder_bert_uncased_L-2_H-128_A-2(our) nq-test dataset 3452 200001 49.68% 59.06% 69.40% 75.75% 89.28%
nlpconnect/dpr-ctx_encoder_bert_uncased_L-12_H-128_A-2(our) nq-test dataset 3452 200001 51.62% 61.09% 70.10% 76.07% 88.70%
*facebook/dpr-ctx_encoder-single-nq-base(hf/fb) nq-test dataset 3452 200001 32.93% 43.74% 56.95% 66.30% 83.92%

Note: * means we have evaluated on same eval dataset.

Usage (HuggingFace Transformers)


passage_encoder = TFAutoModel.from_pretrained("nlpconnect/dpr-ctx_encoder_bert_uncased_L-12_H-128_A-2")
query_encoder = TFAutoModel.from_pretrained("nlpconnect/dpr-question_encoder_bert_uncased_L-12_H-128_A-2")

p_tokenizer = AutoTokenizer.from_pretrained("nlpconnect/dpr-ctx_encoder_bert_uncased_L-12_H-128_A-2")
q_tokenizer = AutoTokenizer.from_pretrained("nlpconnect/dpr-question_encoder_bert_uncased_L-12_H-128_A-2")

def get_title_text_combined(passage_dicts):
    res = []
    for p in passage_dicts:
        res.append(tuple((p['title'], p['text'])))
    return res
    
processed_passages = get_title_text_combined(passage_dicts)

def extracted_passage_embeddings(processed_passages, model_config):
    passage_inputs = tokenizer.batch_encode_plus(
                    processed_passages,
                    add_special_tokens=True,
                    truncation=True,
                    padding="max_length",
                    max_length=model_config.passage_max_seq_len,
                    return_token_type_ids=True
                )
    passage_embeddings = passage_encoder.predict([np.array(passage_inputs['input_ids']), 
                                                np.array(passage_inputs['attention_mask']), 
                                                np.array(passage_inputs['token_type_ids'])], 
                                                batch_size=512, 
                                                verbose=1)
    return passage_embeddings
    
passage_embeddings = extracted_passage_embeddings(processed_passages, model_config)


def extracted_query_embeddings(queries, model_config):
    query_inputs = tokenizer.batch_encode_plus(
                    queries,
                    add_special_tokens=True,
                    truncation=True,
                    padding="max_length",
                    max_length=model_config.query_max_seq_len,
                    return_token_type_ids=True
                )
    query_embeddings = query_encoder.predict([np.array(query_inputs['input_ids']), 
                                                np.array(query_inputs['attention_mask']), 
                                                np.array(query_inputs['token_type_ids'])], 
                                                batch_size=512, 
                                                verbose=1)
    return query_embeddings
    

query_embeddings = extracted_query_embeddings(queries, model_config)

Training hyperparameters

The following hyperparameters were used during training:

  • optimizer: None
  • training_precision: float32

Framework versions

  • Transformers 4.15.0
  • TensorFlow 2.7.0
  • Tokenizers 0.10.3