Repository: yougov/mongo-connector

Issue #279: mongo-connector stuck on fatal error
Created: 2015-07-08T10:34:48Z
I have 2 replicaset in 2 different environnment (dev&prod) with no issue. They run mongo-connector 2.0.3 with pyMongo 2.8. Prod is running with python 2.6, dev with 2.7.
I tried recently to set up a 3thd env with dev data. I ran into fatal issue. This occured with pyMongo 2.0.3, 2.10 and pyMongo 2.8 and 2.8.1.
When starting mongo-connector service, after a constant number of 6000 doc inserted into ElasticSearch. The 6001nth is raising:
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/util.py", line 85, in wrapped
    func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/oplog_manager.py", line 185, in run
    for n, entry in enumerate(cursor):
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1090, in next
    if len(self.__data) or self._refresh():
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1039, in _refresh
    limit, self.__id))
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 958, in __send_message
    self.__compile_re)
  File "/usr/local/lib/python2.7/dist-packages/pymongo/helpers.py", line 121, in _unpack_response
    compile_re)
  File "/usr/local/lib/python2.7/dist-packages/bson/__init__.py", line 537, in decode_all
    tz_aware, uuid_subtype, compile_re))
  File "/usr/local/lib/python2.7/dist-packages/bson/__init__.py", line 331, in _elements_to_dict
    data, position, as_class, tz_aware, uuid_subtype, compile_re)
  File "/usr/local/lib/python2.7/dist-packages/bson/__init__.py", line 320, in _element_to_dict
    data, position, as_class, tz_aware, uuid_subtype, compile_re)
  File "/usr/local/lib/python2.7/dist-packages/bson/__init__.py", line 159, in _get_object
    encoded, as_class, tz_aware, uuid_subtype, compile_re)
  File "/usr/local/lib/python2.7/dist-packages/bson/__init__.py", line 331, in _elements_to_dict
    data, position, as_class, tz_aware, uuid_subtype, compile_re)
  File "/usr/local/lib/python2.7/dist-packages/bson/__init__.py", line 320, in _element_to_dict
    data, position, as_class, tz_aware, uuid_subtype, compile_re)
  File "/usr/local/lib/python2.7/dist-packages/bson/__init__.py", line 232, in _get_date
    dt = EPOCH_AWARE + datetime.timedelta(seconds=seconds)
InvalidBSON: date value out of range

If I set timeZoneAware to false, this is exactly the same, expect EPOCH_AWAREis replaced by EPOCH_NAIVE.
So you think the 6001nth record I imported is bad ? Nope. The first time I launched mongo-connector, I have already imported 180,000+ records, but the second time I juste droped the collection and re-imported all of them so that mongo-connector started to insert doc in ES at the time I start import. Same issue. Setting noDump=true had no effect. The proof the 6001nth doc is not bad ? By spliting import file into 2 parts, (1: 6000 records, 2: remaining), I got 6000 record then 100 imported before first error. 6000 and 100 where constants. By splitingn one more time (6000+6000+remaining), I was hable to import 6000+283 docs. Strange. So pyMongo was not detectin a bad BSON, there is a kind of race condition underneath.
continueOnError and batchSize have no effect, each time I restart mongo-connector without removing oplog-timestamp, even with a dropped collection, it will try to send to ES the same docs again and again. I expected batchSize=1 to force mongo-connector to have an up to date timestamp, but this is not working. the last few records before the fatal error are always "repeated".
The bad part is that I'm not able to launch mongo-connector only to refresh timestamp and exiting, is is crashing very fast and do not update anything. I'm still looking for a workaround : my goal start from an empty collection, start mongo-connector that will remain quiet, then starting to insert doc at a slow rate to see if it matters.
Please upgrade pyMongo dep if it solve this solution (pyMongo 3 is supposed to have a better date support), or forward the issue to pyMongo if the latest has the same issue.
Anyway, you should fix mongo-connector so that oplog-timestamp is updated to avoid doc manager insertion replay.

Duplicate issue #334: mongo-connector stop working if a document contains invalid data  when dumping collections
Created: 2015-10-09T08:41:12Z
While dumping a collection from mongodb to elasticsearch, the mongo-connector exit because one of the document has an invalid date value. The following is the error trace:

2015-10-09 15:46:35,840 [CRITICAL] mongo_connector.oplog_manager:543 - Exception during collection dump
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/mongo_connector/oplog_manager.py", line 495, in do_dump
upsert_all(dm)
File "/usr/local/lib/python2.7/dist-packages/mongo_connector/oplog_manager.py", line 479, in upsert_all
dm.bulk_upsert(docs_to_dump(namespace), mapped_ns, long_ts)
File "/usr/local/lib/python2.7/dist-packages/mongo_connector/util.py", line 32, in wrapped
return f(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/mongo_connector/doc_managers/elastic_doc_manager.py", line 190, in bulk_upsert
for ok, resp in responses:
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/init.py", line 110, in streaming_bulk
for action, data in chunk:
File "/usr/local/lib/python2.7/dist-packages/mongo_connector/doc_managers/elastic_doc_manager.py", line 156, in docs_to_upsert
for doc in docs:
File "/usr/local/lib/python2.7/dist-packages/mongo_connector/oplog_manager.py", line 442, in docs_to_dump
for doc in cursor:
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1090, in next
if len(self.data) or self._refresh():
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1039, in _refresh
limit, self.__id))
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 958, in __send_message
self.__compile_re)
File "/usr/local/lib/python2.7/dist-packages/pymongo/helpers.py", line 121, in _unpack_response
compile_re)
File "/usr/local/lib/python2.7/dist-packages/bson/__init.py", line 537, in decode_all
tz_aware, uuid_subtype, compile_re))
File "/usr/local/lib/python2.7/dist-packages/bson/init.py", line 331, in _elements_to_dict
data, position, as_class, tz_aware, uuid_subtype, compile_re)
File "/usr/local/lib/python2.7/dist-packages/bson/init.py", line 320, in _element_to_dict
data, position, as_class, tz_aware, uuid_subtype, compile_re)
File "/usr/local/lib/python2.7/dist-packages/bson/init.py", line 234, in _get_date
dt = EPOCH_NAIVE + datetime.timedelta(seconds=seconds)
InvalidBSON: date value out of range
2015-10-09 15:46:35,841 [ERROR] mongo_connector.oplog_manager:551 - OplogThread: Failed during dump collection cannot recover! Collection(Database(MongoClient('localhost', 27017), u'local'), u'oplog.rs')
2015-10-09 15:46:35,841 [DEBUG] mongo_connector.oplog_manager:167 - OplogThread: Last entry is the one we already processed.  Up to date.  Sleeping.
2015-10-09 15:46:36,597 [ERROR] mongo_connector.connector:302 - MongoConnector: OplogThread <OplogThread(Thread-2, started 140718959515392)> unexpectedly stopped! Shutting down
2015-10-09 15:46:36,598 [INFO] mongo_connector.connector:359 - MongoConnector: Stopping all OplogThreads
2015-10-09 15:46:36,598 [DEBUG] mongo_connector.oplog_manager:330 - OplogThread: exiting due to join call.

I think the process should never exit if one of the docs fails to be dumped, instead, it should log the error for the doc and then continue to dump next doc in the cursor.

Closing comment (2015-10-13T21:49:25Z): I'm closing this issue as a duplicate of #279. I've added a comment to that ticket explaining what causes this issue and a potential workaround.