Repository: Yelp/elastalert

Issue #996: Another failed to parse date field error
Created: 2017-04-07T10:23:42Z
Hi,
I saw many error like mine but somehow I cannot overcome this:
Config file:
es_host: localhost
es_port: 9200
type: spike
index: metrics
threshold_cur: 1
threshold_ref: 1
spike_type: "both"
filter:
- query:
    query_string:
      query: "M_kernel:>0"
alert:
- "email"
email:
- "my email"

While running elastalert-test-rule i get:
##############################################################3
Successfully loaded Event spike
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost
Got 2 hits from the last 1 day
Available terms in first hit:
M_led_azul_dist
M_upgrades_downloadf_dist
M_led_roxa
@timestamp
M_kernel
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent. To send them, use --verbose.
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost
INFO:elastalert:Queried rule Event spike from 2017-04-06 11:18 WEST to 2017-04-06 11:33 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 11:33 WEST to 2017-04-06 11:48 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 11:48 WEST to 2017-04-06 12:03 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 12:03 WEST to 2017-04-06 12:18 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 12:18 WEST to 2017-04-06 12:33 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 12:33 WEST to 2017-04-06 12:48 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 12:48 WEST to 2017-04-06 13:03 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 13:03 WEST to 2017-04-06 13:18 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 13:18 WEST to 2017-04-06 13:33 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 13:33 WEST to 2017-04-06 13:48 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 13:48 WEST to 2017-04-06 14:03 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 14:03 WEST to 2017-04-06 14:18 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 14:18 WEST to 2017-04-06 14:33 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 14:33 WEST to 2017-04-06 14:48 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 14:48 WEST to 2017-04-06 15:03 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 15:03 WEST to 2017-04-06 15:18 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 15:18 WEST to 2017-04-06 15:33 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 15:33 WEST to 2017-04-06 15:48 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 15:48 WEST to 2017-04-06 16:03 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 16:03 WEST to 2017-04-06 16:18 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 16:18 WEST to 2017-04-06 16:33 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 16:33 WEST to 2017-04-06 16:48 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 16:48 WEST to 2017-04-06 17:03 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 17:03 WEST to 2017-04-06 17:18 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 17:18 WEST to 2017-04-06 17:33 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 17:33 WEST to 2017-04-06 17:48 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 17:48 WEST to 2017-04-06 18:03 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 18:03 WEST to 2017-04-06 18:18 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 18:18 WEST to 2017-04-06 18:33 WEST: 0 / 0 hits
INFO:elastalert:Queried rule Event spike from 2017-04-06 18:33 WEST to 2017-04-06 18:48 WEST: 1 / 1 hits
Traceback (most recent call last):
File "/usr/local/bin/elastalert-test-rule", line 11, in 
load_entry_point('elastalert==0.1.8', 'console_scripts', 'elastalert-test-rule')()
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.8-py2.7.egg/elastalert/test_rule.py", line 333, in main
test_instance.run_rule_test()
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.8-py2.7.egg/elastalert/test_rule.py", line 328, in run_rule_test
self.run_elastalert(rule_yaml, conf, args)
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.8-py2.7.egg/elastalert/test_rule.py", line 248, in run_elastalert
client.run_rule(rule, endtime, starttime)
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.8-py2.7.egg/elastalert/elastalert.py", line 667, in run_rule
if not self.run_query(rule, rule['starttime'], tmp_endtime):
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.8-py2.7.egg/elastalert/elastalert.py", line 476, in run_query
data = self.get_hits(rule, start, end, index, scroll)
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.8-py2.7.egg/elastalert/elastalert.py", line 330, in get_hits
hits = self.process_hits(rule, hits)
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.8-py2.7.egg/elastalert/elastalert.py", line 267, in process_hits
set_es_key(hit['_source'], rule['timestamp_field'], rule'ts_to_dt')
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.8-py2.7.egg/elastalert/util.py", line 109, in ts_to_dt
dt = dateutil.parser.parse(timestamp)
File "/usr/local/lib/python2.7/dist-packages/dateutil/parser.py", line 743, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/dateutil/parser.py", line 300, in parse
res = self._parse(timestr, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/dateutil/parser.py", line 358, in _parse
l = _timelex.split(timestr)
File "/usr/local/lib/python2.7/dist-packages/dateutil/parser.py", line 139, in split
return list(cls(s))
File "/usr/local/lib/python2.7/dist-packages/dateutil/parser.py", line 136, in next
return self.next()  # Python 2.x support
File "/usr/local/lib/python2.7/dist-packages/dateutil/parser.py", line 130, in next
token = self.get_token()
File "/usr/local/lib/python2.7/dist-packages/dateutil/parser.py", line 61, in get_token
nextchar = self.instream.read(1)
AttributeError: 'int' object has no attribute 'read'
################################################################
After putting the  timestamp_field: '@timestamp' and timestamp_type: unix parameters I get:
#################################################################
Successfully loaded Event spike
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost
Got 2 hits from the last 1 day
Available terms in first hit:
M_led_azul_dist
M_upgrades_downloadf_dist
M_upgrades_tentados
M_play_nlinear
M_led_roxa
@timestamp
M_kernel
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent. To send them, use --verbose.
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost
WARNING:elasticsearch:GET http://localhost:9200/metrics/_search?_source_include=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 [status:400 request:0.015s]
ERROR:root:Error running query: TransportError(400, u'search_phase_execution_exception', u'failed to parse date field [1.491474034702163E9] with format [strict_date_optional_time||epoch_millis]')
Would have written the following documents to writeback index (default is elastalert_status):
elastalert_error - {'message': "Error running query: TransportError(400, u'search_phase_execution_exception', u'failed to parse date field [1.491474034702163E9] with format [strict_date_optional_time||epoch_millis]')", 'traceback': ['Traceback (most recent call last):', '  File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.8-py2.7.egg/elastalert/elastalert.py", line 309, in get_hits', "    res = self.current_es.search(scroll=scroll_keepalive, index=index, size=rule['max_query_size'], body=query, ignore_unavailable=True, **extra_args)", '  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 73, in _wrapped', '    return func(*args, params=params, **kwargs)', '  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/init.py", line 625, in search', "    doc_type, '_search'), params=params, body=body)", '  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 318, in perform_request', '    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)', '  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py", line 89, in perform_request', '    self._raise_error(response.status_code, raw_data)', '  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 124, in _raise_error', '    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)', "RequestError: TransportError(400, u'search_phase_execution_exception', u'failed to parse date field [1.491474034702163E9] with format [strict_date_optional_time||epoch_millis]')"], 'data': {'query': {'sort': [{'@timestamp': {'order': 'asc'}}], 'query': {'filtered': {'filter': {'bool': {'must': [{'range': {'@timestamp': {'gt': 1491474034.702163, 'lte': 1491474934.702163}}}, {'query': {'query_string': {'query': 'M_kernel:>0'}}}]}}}}}, 'rule': 'Event spike'}}
##############################################################
What am I doing wrong here??!
Thanks

Duplicate issue #1144: timestamp_field error with unix type.
Created: 2017-06-05T08:40:56Z
ERROR:root:Error running query: TransportError(400, u'search_phase_execution_exception', u'failed to parse date
 field [1.496638310584814E9] with format [epoch_second]')

How can i solve it?

Closing comment (2017-07-27T21:23:28Z): Closing as duplicate of #996