Repository: google/mtail

Issue #165: Huge mtail output logs.
Created: 2018-06-28T10:58:26Z
Hi,
Running the last RC and just processing some not particularly intensive Apache log files. And the INFO output log has gone to 1.8gb in 24 hours.
$ ls -lh mtail*
lrwxrwxrwx 1 root root   55 Jun 28 03:17 mtail.INFO -> mtail..root.log.INFO.20180628-031715.27377
-rw-r--r-- 1 root root 1.8G Jun 28 03:17 mtail..root.log.INFO.20180628-025426.27377
The process is running as
/usr/local/bin/mtail -logs /var/log/httpd/ssl_access_log -progs /usr/local/etc/apache_metrics.mtail -log_dir /var/log
Log file created at: 2018/06/28 03:17:15
Running on machine: lpdosper50136
Binary: Built with gc go1.10.3 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0628 03:17:15.036155   27377 vm.go:86] Prog: apache_metrics.mtail
Metrics
0 Metric: name=apache_http_requests_total program=apache_metrics.mtail kind=Counter type=Int hidden=false keys=[request_method http_version request_status] labelvalues=[LabelValue: [GET HTTP/1.1 200] 9683@1530181005000000000 LabelVa
lue: [GET HTTP/1.1 500] 52@1530176281000000000 LabelValue: [GET HTTP/1.1 400] 56@1530180932000000000 LabelValue: [GET HTTP/1.1 302] 6421@1530180790000000000 LabelValue: [GET HTTP/1.0 500] 74@1530180770000000000 LabelValue: [GET HTTP/1.1 404
] 851@1530181034000000000 LabelValue: [OPTIONS HTTP/1.1 302] 1@1530139497000000000 LabelValue: [POST HTTP/1.1 500] 25@1530168638000000000 LabelValue: [GET HTTP/1.1 503] 403@1530180767000000000 LabelValue: [HEAD HTTP/1.1 302] 9@1530180525000
000000 LabelValue: [POST HTTP/1.0 500] 15@1530180858000000000 LabelValue: [OPTIONS HTTP/1.1 200] 19@1530180878000000000 LabelValue: [POST HTTP/1.1 302] 196@1530180765000000000 LabelValue: [HEAD HTTP/1.0 500] 9@1530180707000000000 LabelValue
: [OPTIONS HTTP/1.1 404] 3@1530180737000000000 LabelValue: [POST HTTP/1.1 404] 31@1530181002000000000 LabelValue: [GET HTTP/1.1 502] 4@1530175752000000000 LabelValue: [PUT HTTP/1.1 404] 1@1530181021000000000] source=apache_metrics.mtail:3:9
-34
1 Metric: name=apache_http_responses_total program=apache_metrics.mtail kind=Counter type=Int hidden=false keys=[request_method http_version request_status] labelvalues=[LabelValue: [GET HTTP/1.1 200] 9683@1530181005000000000 LabelV
alue: [GET HTTP/1.1 500] 52@1530176281000000000 LabelValue: [GET HTTP/1.1 400] 30@1530180778000000000 LabelValue: [GET HTTP/1.1 302] 6086@1530180790000000000 LabelValue: [GET HTTP/1.0 500] 74@1530180770000000000 LabelValue: [POST HTTP/1.1 5
00] 25@1530168638000000000 LabelValue: [GET HTTP/1.1 503] 403@1530180767000000000 LabelValue: [GET HTTP/1.1 404] 173@1530181006000000000 LabelValue: [POST HTTP/1.0 500] 15@1530180858000000000 LabelValue: [POST HTTP/1.1 302] 192@153018076500
0000000 LabelValue: [OPTIONS HTTP/1.1 404] 1@1530175022000000000 LabelValue: [GET HTTP/1.1 502] 4@1530175752000000000 LabelValue: [POST HTTP/1.1 404] 5@1530178243000000000] source=apache_metrics.mtail:4:9-35
...
Etc is there something in this RC with extra debug, or have I mis-configured somewhere ?
Thanks

Duplicate issue #166: INFO, WARNING log file size is getting bigger because of no file handle.
Created: 2018-07-10T08:40:48Z
I ran the mtail with /home/metrics/mtail/mtail --port 9102 --progs /home/metrics/mtail/progs/ --logs /home/www/var/log/logger*.log.
But it looks handle the file that is not matched I wanted.
I wanted to handle only /home/www/var/log/logger*.log files, but mtail tries to handle access*.log files.
I'm doing logrotate and delete old files.
As a result the INFO, WARNING log file size is getting bigger and bigger time goes on.
What should I do? Did I wrong?
Log file created at: 2018/07/09 16:30:50
Running on machine: VLFTCFFR1501
Binary: Built with gc go1.10.3 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0709 16:30:50.675116   16920 main.go:90] mtail version v3.0.0-rc14 git revision bd75e5f3b7e5c77b86ae549364108dca0e4dee3a go version go1.10.3
I0709 16:30:50.675329   16920 main.go:91] Commandline: ["/home/metrics/mtail/mtail" "--port" "9102" "--progs" "/home/metrics/mtail/progs/" "--logs" "/home/www/var/log/logger*.log"]
I0709 16:30:50.675912   16920 types.go:325] Got LUB = "Float"
I0709 16:30:50.675921   16920 types.go:325] Got LUB = "Float"
I0709 16:30:50.675983   16920 checker.go:165] declaration of capture group reference `4' at http_request_duration.mtail:5:1-3 appears to be unused
I0709 16:30:50.676163   16920 codegen.go:442] Conversion: "Int" to "Float"
I0709 16:30:50.676187   16920 store.go:32] Adding a new metric Metric: name=http_request_duration_bucket program=http_request_duration.mtail kind=Counter type=Int hidden=false keys=[le path method status] labelvalues=[] source=http_request_duration.mtail:1:9-48
I0709 16:30:50.676212   16920 store.go:32] Adding a new metric Metric: name=http_request_duration_sum program=http_request_duration.mtail kind=Counter type=Float hidden=false keys=[path method status] labelvalues=[] source=http_request_duration.mtail:2:9-45
I0709 16:30:50.676220   16920 store.go:32] Adding a new metric Metric: name=http_request_duration_count program=http_request_duration.mtail kind=Counter type=Int hidden=false keys=[path method status] labelvalues=[] source=http_request_duration.mtail:3:9-47
I0709 16:30:50.676228   16920 mcp.go:199] Loaded program http_request_duration.mtail
I0709 16:30:50.676234   16920 mcp.go:218] Program http_request_duration.mtail has goroutine marker 0x6674635f
I0709 16:30:50.676258   16920 vm.go:692] Starting program http_request_duration.mtail
I0709 16:30:50.676265   16920 mcp.go:222] Started http_request_duration.mtail
I0709 16:30:50.676449   16920 store.go:32] Adding a new metric Metric: name=http_request_error program=http_request_error.mtail kind=Counter type=Int hidden=false keys=[component function] labelvalues=[] source=http_request_error.mtail:1:9-38
I0709 16:30:50.676464   16920 mcp.go:199] Loaded program http_request_error.mtail
I0709 16:30:50.676467   16920 mcp.go:218] Program http_request_error.mtail has goroutine marker 0x6674635f
I0709 16:30:50.676480   16920 vm.go:692] Starting program http_request_error.mtail
I0709 16:30:50.676486   16920 mcp.go:222] Started http_request_error.mtail
I0709 16:30:50.676644   16920 tail.go:472] Tailing /home/www/var/log/logger.2018-07-09-09.log
I0709 16:30:50.676661   16920 tail.go:472] Tailing /home/www/var/log/logger.2018-07-09-10.log
I0709 16:30:50.676679   16920 tail.go:472] Tailing /home/www/var/log/logger.2018-07-09-11.log
I0709 16:30:50.676691   16920 tail.go:472] Tailing /home/www/var/log/logger.2018-07-09-12.log
I0709 16:30:50.676774   16920 mtail.go:324] Listening on port :9102
W0709 16:30:53.220189   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:30:58.230060   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:31:03.239886   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:31:08.250013   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:31:13.259864   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:31:18.269918   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:31:23.279775   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:31:28.289764   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:31:33.299657   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:31:38.309678   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:31:43.319598   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:31:48.329585   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:31:53.339430   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:31:58.349286   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:32:03.359288   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:32:08.369268   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:32:13.379738   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:32:18.389321   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:32:23.399286   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:32:28.422610   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:32:33.419032   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:32:38.429041   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:32:43.439045   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:32:48.449041   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:32:53.458949   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:32:58.469109   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:33:03.478796   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:33:08.488693   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:33:13.498473   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched
W0709 16:33:18.509068   16920 tail.go:199] No file handle found for "/home/www/var/log/access.2018-07-09-16.log", but is being watched

Closing comment (2018-07-18T07:49:16Z): Ah yep this was also a duplicate of #165

…
On 18 July 2018 at 17:43, Jamie Wilkinson ***@***.***> wrote:
 I removed some logs in commit 2991023 which I think is in the rc15 build.
 Please try that (https://github.com/google/mtail/releases or by checking
 out at HEAD if you prefer).

 On 10 July 2018 at 18:40, gentlejo ***@***.***> wrote:

> I ran the mtail with /home/metrics/mtail/mtail --port 9102 --progs
> /home/metrics/mtail/progs/ --logs /home/www/var/log/logger*.log.
> But it looks handle the file that is not matched I wanted.
> I wanted to handle only /home/www/var/log/logger*.log files, but mtail
> tries to handle access*.log files.
> As a result the INFO, WARNING log file size is getting bigger and bigger
> time goes on.
> What should I do? Did I wrong?
>
> Log file created at: 2018/07/09 16:30:50
> Running on machine: VLFTCFFR1501
> Binary: Built with gc go1.10.3 for linux/amd64
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> I0709 16:30:50.675116 16920 main.go:90] mtail version v3.0.0-rc14 git
> revision bd75e5f
> <bd75e5f>
> go version go1.10.3
> I0709 16:30:50.675329 16920 main.go:91] Commandline:
> ["/home/metrics/mtail/mtail" "--port" "9102" "--progs"
> "/home/metrics/mtail/progs/" "--logs" "/home/www/var/log/logger*.log"]
> I0709 16:30:50.675912 16920 types.go:325] Got LUB = "Float"
> I0709 16:30:50.675921 16920 types.go:325] Got LUB = "Float"
> I0709 16:30:50.675983 16920 checker.go:165] declaration of capture group
> reference `4' at http_request_duration.mtail:5:1-3 appears to be unused
> I0709 16:30:50.676163 16920 codegen.go:442] Conversion: "Int" to "Float"
> I0709 16:30:50.676187 16920 store.go:32] Adding a new metric Metric:
> name=http_request_duration_bucket program=http_request_duration.mtail
> kind=Counter type=Int hidden=false keys=[le path method status]
> labelvalues=[] source=http_request_duration.mtail:1:9-48
> I0709 16:30:50.676212 16920 store.go:32] Adding a new metric Metric:
> name=http_request_duration_sum program=http_request_duration.mtail
> kind=Counter type=Float hidden=false keys=[path method status]
> labelvalues=[] source=http_request_duration.mtail:2:9-45
> I0709 16:30:50.676220 16920 store.go:32] Adding a new metric Metric:
> name=http_request_duration_count program=http_request_duration.mtail
> kind=Counter type=Int hidden=false keys=[path method status] labelvalues=[]
> source=http_request_duration.mtail:3:9-47
> I0709 16:30:50.676228 16920 mcp.go:199] Loaded program
> http_request_duration.mtail
> I0709 16:30:50.676234 16920 mcp.go:218] Program
> http_request_duration.mtail has goroutine marker 0x6674635f
> I0709 16:30:50.676258 16920 vm.go:692] Starting program
> http_request_duration.mtail
> I0709 16:30:50.676265 16920 mcp.go:222] Started
> http_request_duration.mtail
> I0709 16:30:50.676449 16920 store.go:32] Adding a new metric Metric:
> name=http_request_error program=http_request_error.mtail kind=Counter
> type=Int hidden=false keys=[component function] labelvalues=[]
> source=http_request_error.mtail:1:9-38
> I0709 16:30:50.676464 16920 mcp.go:199] Loaded program
> http_request_error.mtail
> I0709 16:30:50.676467 16920 mcp.go:218] Program http_request_error.mtail
> has goroutine marker 0x6674635f
> I0709 16:30:50.676480 16920 vm.go:692] Starting program
> http_request_error.mtail
> I0709 16:30:50.676486 16920 mcp.go:222] Started http_request_error.mtail
> I0709 16:30:50.676644 16920 tail.go:472] Tailing
> /home/www/var/log/logger.2018-07-09-09.log
> I0709 16:30:50.676661 16920 tail.go:472] Tailing
> /home/www/var/log/logger.2018-07-09-10.log
> I0709 16:30:50.676679 16920 tail.go:472] Tailing
> /home/www/var/log/logger.2018-07-09-11.log
> I0709 16:30:50.676691 16920 tail.go:472] Tailing
> /home/www/var/log/logger.2018-07-09-12.log
> I0709 16:30:50.676774 16920 mtail.go:324] Listening on port :9102
> W0709 16:30:53.220189 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:30:58.230060 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:31:03.239886 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:31:08.250013 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:31:13.259864 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:31:18.269918 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:31:23.279775 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:31:28.289764 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:31:33.299657 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:31:38.309678 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:31:43.319598 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:31:48.329585 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:31:53.339430 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:31:58.349286 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:32:03.359288 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:32:08.369268 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:32:13.379738 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:32:18.389321 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:32:23.399286 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:32:28.422610 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:32:33.419032 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:32:38.429041 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:32:43.439045 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:32:48.449041 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:32:53.458949 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:32:58.469109 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:33:03.478796 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:33:08.488693 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:33:13.498473 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
> W0709 16:33:18.509068 16920 tail.go:199] No file handle found for
> "/home/www/var/log/access.2018-07-09-16.log", but is being watched
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <#166>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AC5b-zVSWUMjOECXDKtVn74XOIucEjddks5uFGiYgaJpZM4VI_TS>
> .
>