Repository: elastic/logstash-forwarder

Issue #520: send json log file to logstash with logstash forwarder but see wrong log in kibana
Created: 2015-08-31T21:18:16Z
I have an app that writes logs to a json file that named allapp.json, Each log entry is a JSON object. An example of it looks like the following:
"{\"timestamp\":\"2015-08-30 19:42:26.724\",\"MAC_Address\":\"A8:7C:01:CB:2D:09\",\"DeviceID\":\"96f389972de989d1\",\"RunningApp\":\"null{com.tools.app_logs\\/com.tools.app_logs.Main}{com.gtp.nextlauncher\\/com.gtp.nextlauncher.LauncherActivity}{com.android.settings\\/com.android.settings.Settings$WifiSettingsActivity}{com.android.incallui\\/com.android.incallui.InCallActivity}{com.tools.app_logs\\/com.tools.app_logs.Main}{com.gtp.nextlauncher\\/com.gtp.nextlauncher.LauncherActivity}{com.android.settings\\/com.android.settings.Settings$WifiSettingsActivity}{com.android.incallui\\/com.android.incallui.InCallActivity}\",\"PhoneName\":\"rafiee\",\"IP\":\"192.168.1.101\"}"

I want send my log to logstash, so logstash-forwarder.conf configured to:
"network": {  "servers": [ "192.168.1.102:5000" ],  "timeout": 15,  "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"  "files": [  {  "paths":  ["/home/fr/allapp.json"],  "fields": { "type": "json" }  }  ]

and in /etc/logstash/conf.d/ i have two conf file:  _01-lumberjack-input.conf:_
 input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
} }

_30-lumberjack-output.conf_ :
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
} 

when i run     _/opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder.conf_    i see
2015/09/01 01:01:52.194573 Waiting for 1 prospectors to initialise 2015/09/01 01:01:52.194867 Launching harvester on new file: /home/fr/allapp.json 2015/09/01 01:01:52.195326 harvest: "/home/fr/allapp.json" (offset snapshot:0) 2015/09/01 01:01:52.217068 All prospectors initialised with 0 states to persist 2015/09/01 01:01:52.217317 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt 2015/09/01 01:01:52.256222 Connecting to [192.168.1.102]:5000 (192.168.1.102)  2015/09/01 01:01:52.502025 Connected to 192.168.1.102
and at the end in kibana i just see logs like this in far date:
Time    _source     August 30th 2015, 10:21:46.000  message: Aug 29 22:51:46 ubuntu sudo: pam_unix(sudo:session): session closed for user root @version: 1 @timestamp: August 30th 2015, 10:21:46.000 type: syslog file: /var/log/auth.log host: ubuntu offset: 16355 syslog_timestamp: Aug 29 22:51:46 syslog_hostname: ubuntu syslog_program: sudo syslog_message: pam_unix(sudo:session): session closed for user root received_at: August 30th 2015, 10:21:47.899 received_from: ubuntu syslog_severity_code: 5 syslog_facility_code: 1 syslog_facility: user-level syslog_severity: notice _source: {"message":"Aug 29 22:51:46 ubuntu sudo: pam_unix(sudo:session): session closed for user root","@version":"1","@timestamp":"2015-08-30T05:51:46.000Z","type":"syslog","file":"/var/log/auth.log","host":"ubuntu","offset":"16355","syslog_timestamp":"Aug 29 22:51:46","syslog_hostname":"ubuntu","syslog_program":"sudo","syslog_message":"pam_unix(sudo:session): session closed for user root","received_at":"2015-08-30T05:51:47.899Z","received_from":"ubuntu","syslog_severity_code":5,"syslog_facility_code":1,"syslog_facility":"user-level","syslog_severity":"notice"} _id: AU99KWDPEqXW0EMoRjPr _type: syslog _index: logstash-2015.08.30 
as you see at file:
/var/log/auth.log above this is not my logs from allapp.json file, it look like all Elasticsearch, Logstash, ngix and Kibana work without error but the answer i get is wrong and i am not sure where i wrong.  I would sincerely appreciate any help as I'm getting desperate at this point. THANKS

Duplicate issue #521: send json log file to logstash with logstash forwarder but see wrong log in kibana
Created: 2015-08-31T21:34:34Z
I have an app that writes logs to a json file that named allapp.json, Each log entry is a JSON object. An example of it looks like the following:
"{\"timestamp\":\"2015-08-30 19:42:26.724\",\"MAC_Address\":\"A8:7C:01:CB:2D:09\",\"DeviceID\":\"96f389972de989d1\",\"RunningApp\":\"null{com.tools.app_logs\\/com.tools.app_logs.Main}{com.gtp.nextlauncher\\/com.gtp.nextlauncher.LauncherActivity}{com.android.settings\\/com.android.settings.Settings$WifiSettingsActivity}{com.android.incallui\\/com.android.incallui.InCallActivity}{com.tools.app_logs\\/com.tools.app_logs.Main}{com.gtp.nextlauncher\\/com.gtp.nextlauncher.LauncherActivity}{com.android.settings\\/com.android.settings.Settings$WifiSettingsActivity}{com.android.incallui\\/com.android.incallui.InCallActivity}\",\"PhoneName\":\"rafiee\",\"IP\":\"192.168.1.101\"}"

I want send my log to logstash, so logstash-forwarder.conf configured to:
"network": {  "servers": [ "192.168.1.102:5000" ],  "timeout": 15,  "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"  "files": [  {  "paths":  ["/home/fr/allapp.json"],  "fields": { "type": "json" }  }  ]

and in /etc/logstash/conf.d/ i have two conf file:  _01-lumberjack-input.conf:_
 input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
} }

_30-lumberjack-output.conf_ :
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
} 

when i run     _/opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder.conf_    i see
2015/09/01 01:01:52.194573 Waiting for 1 prospectors to initialise 2015/09/01 01:01:52.194867 Launching harvester on new file: /home/fr/allapp.json 2015/09/01 01:01:52.195326 harvest: "/home/fr/allapp.json" (offset snapshot:0) 2015/09/01 01:01:52.217068 All prospectors initialised with 0 states to persist 2015/09/01 01:01:52.217317 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt 2015/09/01 01:01:52.256222 Connecting to [192.168.1.102]:5000 (192.168.1.102)  2015/09/01 01:01:52.502025 Connected to 192.168.1.102
and at the end in kibana i just see logs like this in far date:
Time    _source     August 30th 2015, 10:21:46.000  message: Aug 29 22:51:46 ubuntu sudo: pam_unix(sudo:session): session closed for user root @version: 1 @timestamp: August 30th 2015, 10:21:46.000 type: syslog file: /var/log/auth.log host: ubuntu offset: 16355 syslog_timestamp: Aug 29 22:51:46 syslog_hostname: ubuntu syslog_program: sudo syslog_message: pam_unix(sudo:session): session closed for user root received_at: August 30th 2015, 10:21:47.899 received_from: ubuntu syslog_severity_code: 5 syslog_facility_code: 1 syslog_facility: user-level syslog_severity: notice _source: {"message":"Aug 29 22:51:46 ubuntu sudo: pam_unix(sudo:session): session closed for user root","@version":"1","@timestamp":"2015-08-30T05:51:46.000Z","type":"syslog","file":"/var/log/auth.log","host":"ubuntu","offset":"16355","syslog_timestamp":"Aug 29 22:51:46","syslog_hostname":"ubuntu","syslog_program":"sudo","syslog_message":"pam_unix(sudo:session): session closed for user root","received_at":"2015-08-30T05:51:47.899Z","received_from":"ubuntu","syslog_severity_code":5,"syslog_facility_code":1,"syslog_facility":"user-level","syslog_severity":"notice"} _id: AU99KWDPEqXW0EMoRjPr _type: syslog _index: logstash-2015.08.30 
as you see at file:
/var/log/auth.log above this is not my logs from allapp.json file, it look like all Elasticsearch, Logstash, ngix and Kibana work without error but the answer i get is wrong and i am not sure where i wrong.  I would sincerely appreciate any help as I'm getting desperate at this point. THANKS

Closing comment (2015-08-31T21:46:47Z): Looks like this is a dup of #520