Repository: lf-edge/ekuiper

Issue #842: Too many open files error when creating more than 1000 rules
Created: 2021-06-18T01:06:02Z
An issue report from EdgeX community.
I'm trying to test the performance of Kuiper, and for that I'm creating a 1000 rules,
While creating rules, I get following error in response to API call,
Create rule error: unable to open database file: too many open files
I'm using Kuiper 1.1.1.

Duplicate issue #971: runtime error when create multiple rules using ./kuiper
Created: 2021-08-24T06:32:43Z
Environment:

eKuiper version (e.g. 1.1.2):
Hardware configuration  RISC-V):
OS (linux):
Others:

What happened and what you expected to happen:
When I  created rules using command "./kuiper create xxxx " almost in the meantime(or with tiny interval),
kuiperd will crash with below message:
_time="2021-08-24 05:31:58" level=info msg="open source node 1 instances" file="sinks/rest_sink.go:140" rule=restRule_devUpdate_res_temphumity_718bc0_AB11007102000001
time="2021-08-24 05:31:58" level=info msg="open rest sink with configuration: {method: PUT, url: http://127.0.0.1:49904/api/v1/device/name/res_temphumity_718bc0_AB11007102000001/updateresponse, bodyType: json, timeout: 5000,header: map[], sendSingle: true, insecureSkipVerify: true" file="sinks/rest_sink.go:140" rule=restRule_devUpdate_res_temphumity_718bc0_AB11007102000001
time="2021-08-24 05:31:58" level=error msg="topo restRule_devUpdate_res_lightstrength_71ab82_AB11007106000001 drain error " file="xstream/streams.go:120" rule=restRule_devUpdate_res_lightstrength_71ab82_AB11007106000001
time="2021-08-24 05:31:58" level=info msg="closing rule restRule_devUpdate_res_lightstrength_71ab82_AB11007106000001" file="server/ruleManager.go:103"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7268ac]
goroutine 29056 [running]:
database/sql.(*Stmt).ExecContext(0x0, 0xd94e40, 0x3f98132058, 0x3f98903ab0, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0)
database/sql/sql.go:2394 +0x4c
database/sql.(*Stmt).Exec(...)
database/sql/sql.go:2423
github.com/emqx/kuiper/common/kv.(*SqliteKVStore).Setnx(0x3f9809a630, 0x3f984a2d40, 0x3c, 0xb5f080, 0x3f989e6ae0, 0x3f98422730, 0x0)
github.com/emqx/kuiper/common/kv/sqliteKV.go:68 +0x1dc
github.com/emqx/kuiper/xsql/processors.(*RuleProcessor).ExecCreate(0x3f98090cc0, 0x3f984a2d40, 0x3c, 0x3f989141e0, 0x1c6, 0x0, 0x0, 0x0)
github.com/emqx/kuiper/xsql/processors/xsql_processor.go:236 +0x140
github.com/emqx/kuiper/xstream/server/server.(*Server).CreateRule(0x3f983c0618, 0x3f98964ba0, 0x3f98b72320, 0x0, 0x0)
github.com/emqx/kuiper/xstream/server/server/rpc.go:79 +0x58
reflect.Value.call(0x3f988d74a0, 0x3f9842a6a0, 0x13, 0xc6e758, 0x4, 0x3f98872f10, 0x3, 0x3, 0x2000000010000, 0x0, ...)
reflect/value.go:460 +0x77c
reflect.Value.Call(0x3f988d74a0, 0x3f9842a6a0, 0x13, 0x3f9870ef10, 0x3, 0x3, 0x3f9853ec40, 0x0, 0x1)
reflect/value.go:321 +0xa8
net/rpc.(*service).call(0x3f988f4640, 0x3f984234a0, 0x3f98d9e580, 0x3f98d9e590, 0x3f987a2c80, 0x3f9877efa0, 0xb4b740, 0x3f98964ba0, 0x16, 0xb3aac0, ...)
net/rpc/server.go:377 +0x12c
created by net/rpc.(*Server).ServeCodec_
Is that because i am using a low version kuiper?

Closing comment (2021-08-24T06:36:07Z): @wallace-hu Thanks for opening the issue. It is like a duplicate of #842
This problem should be fixed since v1.3.0.
If you need to work with previous versions, please try to add more intervals when calling REST API.