Repository: beir-cellar/beir
Issue: Elastic Search Connection Error
Root cause: Environment Incompatibility

I tried to replicate BM25 evaluation on trec-covid, but ran into the following problem:
```
2021-04-21 16:16:57 - Downloading trec-covid.zip ...
2021-04-21 16:16:57 - Unzipping trec-covid.zip ...
2021-04-21 16:16:58 - Loaded 171332 TEST Documents.
2021-04-21 16:16:58 - Doc Example: {'text': 'OBJECTIVE: This retrospective chart review describes the epidemiology and clinical features of 40 patients with culture-proven Mycoplasma pneumoniae infections at King Abdulaziz University Hospital, Jeddah, Saudi Arabia. METHODS: Patients with positive M. pneumoniae cultures from respiratory specimens from January 1997 through December 1998 were identified through the Microbiology records. Charts of patients were reviewed. RESULTS: 40 patients were identified, 33 (82.5%) of whom required admission. Most infections (92.5%) were community-acquired. The infection affected all age groups but was most common in infants (32.5%) and pre-school children (22.5%). It occurred year-round but was most common in the fall (35%) and spring (30%). More than three-quarters of patients (77.5%) had comorbidities. Twenty-four isolates (60%) were associated with pneumonia, 14 (35%) with upper respiratory tract infections, and 2 (5%) with bronchiolitis. Cough (82.5%), fever (75%), and malaise (58.8%) were the most common symptoms, and crepitations (60%), and wheezes (40%) were the most common signs. Most patients with pneumonia had crepitations (79.2%) but only 25% had bronchial breathing. Immunocompromised patients were more likely than non-immunocompromised patients to present with pneumonia (8/9 versus 16/31, P = 0.05). Of the 24 patients with pneumonia, 14 (58.3%) had uneventful recovery, 4 (16.7%) recovered following some complications, 3 (12.5%) died because of M pneumoniae infection, and 3 (12.5%) died due to underlying comorbidities. The 3 patients who died of M pneumoniae pneumonia had other comorbidities. CONCLUSION: our results were similar to published data except for the finding that infections were more common in infants and preschool children and that the mortality rate of pneumonia in patients with comorbidities was high.', 'title': 'Clinical features of culture-proven Mycoplasma pneumoniae infections at King Abdulaziz University Hospital, Jeddah, Saudi Arabia'}
2021-04-21 16:16:58 - Loaded 50 TEST Queries.
2021-04-21 16:16:58 - Query Example: what is the origin of COVID-19
2021-04-21 16:16:58 - Activating Elasticsearch....
2021-04-21 16:16:58 - Elastic Search Credentials: {'hostname': 'localhost', 'index_name': 'trec-covid', 'keys': {'title': 'title', 'body': 'txt'}, 'timeout': 100, 'retry_on_timeout': True, 'maxsize': 24}
2021-04-21 16:16:58 - Deleting previous Elasticsearch-Index named - trec-covid
2021-04-21 16:16:58 - Unable to create Index in Elastic Search. Reason: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fa550364110>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fa550364110>: Failed to establish a new connection: [Errno 111] Connection refused)
2021-04-21 16:16:58 - Creating fresh Elasticsearch-Index named - trec-covid
2021-04-21 16:16:58 - Unable to create Index in Elastic Search. Reason: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fa550364c10>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fa550364c10>: Failed to establish a new connection: [Errno 111] Connection refused)
Traceback (most recent call last):
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/urllib3/connection.py", line 170, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 252, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/home/USER/.conda/envs/beir/lib/python3.7/http/client.py", line 1277, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/USER/.conda/envs/beir/lib/python3.7/http/client.py", line 1323, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/USER/.conda/envs/beir/lib/python3.7/http/client.py", line 1272, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/USER/.conda/envs/beir/lib/python3.7/http/client.py", line 1032, in _send_output
    self.send(msg)
  File "/home/USER/.conda/envs/beir/lib/python3.7/http/client.py", line 972, in send
    self.connect()
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fa54912e690>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "evaluate_bm25.py", line 33, in <module>
    results = retriever.retrieve(corpus, queries)
  File "/home/USER/projects/beir-repo/beir/retrieval/evaluation.py", line 22, in retrieve
    return self.retriever.search(corpus, queries, self.top_k, self.score_function)
  File "/home/USER/projects/beir-repo/beir/retrieval/search/lexical/bm25_search.py", line 33, in search
    self.index(corpus)
  File "/home/USER/projects/beir-repo/beir/retrieval/search/lexical/bm25_search.py", line 66, in index
    progress=progress
  File "/home/USER/projects/beir-repo/beir/retrieval/search/lexical/elastic_search.py", line 89, in bulk_add_to_index
    client=self.es, index=self.index_name, actions=generate_actions,
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/elasticsearch/helpers/actions.py", line 326, in streaming_bulk
    **kwargs
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/elasticsearch/helpers/actions.py", line 246, in _process_bulk_chunk
    for item in gen:
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/elasticsearch/helpers/actions.py", line 193, in _process_bulk_chunk_error
    raise error
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/elasticsearch/helpers/actions.py", line 234, in _process_bulk_chunk
    resp = client.bulk("\n".join(bulk_actions) + "\n", *args, **kwargs)
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 153, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/elasticsearch/client/__init__.py", line 460, in bulk
    body=body,
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/elasticsearch/transport.py", line 413, in perform_request
    raise e
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/elasticsearch/transport.py", line 388, in perform_request
    timeout=timeout,
  File "/home/USER/.conda/envs/beir/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 264, in perform_request
    raise ConnectionError("N/A", str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fa54912e690>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fa54912e690>: Failed to establish a new connection: [Errno 111] Connection refused)
```

The only things I've changed in `evaluate_bm25.py` are
1. `dataset = "trec-covid"` (line 17)
2. `hostname = "localhost" #localhost` (line 26)
3. `index_name = dataset # scifact` (line 27)

Elastic Search is new to me so I'm not sure whether I've missed anything. I used `pip install -e .` to install BEIR from source if that's relevant. Thanks!

Hi @Ji-Xin, Seems to me that you might have missed installing Elasticsearch on your desktop (locally), apart from ``pip install beir``. Depending on your OS, you would be able to find how to download Elasticsearch. I like this guide for Ubuntu 18.04 (
[https://linuxize.com/post/how-to-install-elasticsearch-on-ubuntu-18-04/](https://linuxize.com/post/how-to-install-elasticsearch-on-ubuntu-18-04)). For more details, please refer here - [https://www.elastic.co/downloads/elasticsearch](https://www.elastic.co/downloads/elasticsearch).

Meanwhile, you could check out the Google Colab Demo, where we first install elastic search locally and retrieve using BM25 - [https://colab.research.google.com/drive/1HfutiEhHMJLXiWGT8pcipxT5L2TpYEdt?usp=sharing#scrollTo=nqotyXuIBPt6](https://colab.research.google.com/drive/1HfutiEhHMJLXiWGT8pcipxT5L2TpYEdt?usp=sharing#scrollTo=nqotyXuIBPt6)

Kind Regards,
Nandan

@NThakur20 Thanks for the quick and detailed response! Yes it works now after installing Elasticsearch. It would be great for other people if you could mention in README that Elasticsearch needs to be installed before using BM25 search in BEIR.

By the way, Table 8 in the BEIR paper, the row for Trec-covid seems to be P@100 scores instead of Recall@100.

Hi @Ji-Xin, Thanks for the pointer, will include it in the documentation.

Regarding Table 8, often Recall@k scores are unintuitive when the number of relevant documents are high (relevant docs > k). TREC-COVID contains around 500 relevant docs on average. Now, even if you retrieve all relevant documents (100 max, for k=100), pytrec_eval would provide you a score of (100/500) = 0.2 which is quite low. Hence we took an alternate approach, and calculated a capped Recall@k score, where we cap the recall at K, Recall_cap@k would be (100/100) = 1 instead of 0.2. We calculate Recall_cap@100 only for TREC-COVID in table 8 on the paper, hence the Recall@100 results for that row are different. 

Here below is a minimal code for capped Recall@k -
```python
#### Retrieve dense results (format of results is identical to qrels)
results = retriever.retrieve(corpus, queries)

#### Capped Recall Score @ k
recall_cap = 0
k = 100

for qid, scores in results.items():
    results_sorted = sorted(scores.items(), key=lambda item: item[1], reverse=True)[0:k]
    golden_ids = [row[0] for row in results_sorted if row[0] in qrels[qid]]
    max_len = min(len(qrels[qid]), k)
    recall_cap += (len(golden_ids) / max_len)

recall_cap = recall_cap / len(results)
```

Hi @Ji-Xin , the documentation for BM25 has been updated, and now you can also evaluate the capped Recall@K with the beir package.

```python
#### Evaluate your retrieval using NDCG@k, MAP@K ...
ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)
#### Evaluate your retrieval using custom metric such as Capped Recall@K
recall_capped = retriever.evaluate_custom(qrels, results, retriever.k_values, metric="r_cap")
```

Thanks for the update @NThakur20