Repository: kedacore/keda

Issue #4413: Crash during startup in reconciler
Created: 2023-03-30T12:36:45Z
Report
Let me start with a heads-up: This bug report is best-effort. Keda was in a crash loop, so getting things back up was more important than figuring out what exactly happened. I will try to add all the details I have. I it is not enough, feel free to close.
I cannot tell exactly what triggered it, but after a CI run Keda started to crash loop. Find the relevant part of the log below.
No new scaled objects were introduced. The very same config has been running for quite some time.
Looking at the scaled object, we can see that active and fallback are Unknown instead of True or False.
❯ kubectl get scaledobjects.keda.sh 
NAME                                         SCALETARGETKIND      SCALETARGETNAME                              MIN   MAX   TRIGGERS   AUTHENTICATION   READY   ACTIVE    FALLBACK   AGE
...
master-c96-planningsvc-worker-interactive    apps/v1.Deployment   master-c96-planningsvc-worker-interactive    0     10    cron                        True    Unknown   Unknown    2d2h
...

As obviously something was wrong, I kubectl deleted the scaled object and recreated it. Keda happily started right away.
Expected Behavior
Keda does not crash when "something" (whatever something is) is wrong with a metric.
Actual Behavior
Keda crashes during startup.
Steps to Reproduce the Problem
I have no idea so far. It only happened once.
Logs from KEDA operator
Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"master-c96-planningsvc-worker-interactive","namespace":"mpp-backend-master-c96"}, "namespace": "mpp-backend-master-c96", "name": "master-c96-planningsvc-worker-interactive", "reconcileID": "23c24efb-96df-482e-acc7-ce3b2dd36986"}

panic: runtime error: invalid memory address or nil pointer dereference [recovered]

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2f62194]

goroutine 467 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0x1fa
panic({0x34bf3a0, 0x63ea940})
	/usr/local/go/src/runtime/panic.go:884 +0x212
github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveScaleTargetPodSpec({0x434be30, 0xc000fa4510}, {0x4360950, 0xc0008deea0}, {0x3a94540?, 0xc000ec0a00})
	/workspace/pkg/scaling/resolver/scale_resolvers.go:73 +0xd4
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache(0xc000805110, {0x434be30, 0xc000fa4510}, {0xc0000551d0, 0x4d}, {0x3a94540, 0xc000ec0a00}, 0xc000e3cf00, {0x0, 0x0}, ...)
	/workspace/pkg/scaling/scale_handler.go:347 +0x6e5
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache(0xc000ec0600?, {0x434be30, 0xc000fa4510}, {0x3a94540, 0xc000ec0a00})
	/workspace/pkg/scaling/scale_handler.go:273 +0xf6
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs(0xc000f226c0, {0x434be30, 0xc000fa4510}, {{0x4353fc0?, 0xc000fa4540?}, 0xc001260cc0?}, 0xc000ec0600)
	/workspace/controllers/keda/hpa.go:200 +0xda
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject(0xc000f226c0, {0x434be30?, 0xc000fa4510?}, {{0x4353fc0?, 0xc000fa4540?}, 0x3a221a0?}, 0xc000ec0600, 0xc0012395f0)
	/workspace/controllers/keda/hpa.go:74 +0x66
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded(0xc000f226c0, {0x434be30, 0xc000fa4510}, {{0x4353fc0?, 0xc000fa4540?}, 0xc000fa4510?}, 0xc000ec0600, 0xc00087ba40, 0xc001580580?)
	/workspace/controllers/keda/hpa.go:152 +0x7b
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists(0xc000f226c0, {0x434be30, 0xc000fa4510}, {{0x4353fc0?, 0xc000fa4540?}, 0x4353fc0?}, 0xc000ec0600, 0x0?)
	/workspace/controllers/keda/scaledobject_controller.go:431 +0x238
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject(0xc000f226c0?, {0x434be30, 0xc000fa4510}, {{0x4353fc0?, 0xc000fa4540?}, 0xc0011bae10?}, 0xc000ec0600)
	/workspace/controllers/keda/scaledobject_controller.go:229 +0x1c9
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile(0xc000f226c0, {0x434be30, 0xc000fa4510}, {{{0xc000a7b740?, 0x10?}, {0xc0011bae10?, 0x40da87?}}})
	/workspace/controllers/keda/scaledobject_controller.go:175 +0x526
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x434be30?, {0x434be30?, 0xc000fa4510?}, {{{0xc000a7b740?, 0x32ae080?}, {0xc0011bae10?, 0x0?}}})
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00080b720, {0x434bd88, 0xc00109e180}, {0x3619020?, 0xc000e4a4e0?})
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323 +0x38f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00080b720, {0x434bd88, 0xc00109e180})
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:231 +0x333

KEDA Version
2.10.0
Kubernetes Version
1.25
Platform
None
Scaler Details
cron
Anything else?
No response

Duplicate issue #4557: SIGSEGV crash in RequestScale
Created: 2023-05-21T07:00:08Z
Report
Keda crashes in the middle of the operations
Expected Behavior
Not to crash and continue operations
Actual Behavior
Crash with panic: runtime error: invalid memory address or nil pointer dereference
Steps to Reproduce the Problem
Can't reproduce, it happens in the middle of the regular operations.
Will try to find and correlations.
We have 3 EKS clusters with Keda. And Keda crashes only in the biggest cluster (with 1700 pods).
Logs from KEDA operator
2023-05-21T05:31:44Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"find-reservation-sources","namespace":"default"}, "namespace": "default"
, "name": "find-reservation-sources", "reconcileID": "357b528d-5fd7-488a-809e-d5ff382b3c75"}                                                                                                                                                                
2023-05-21T05:31:44Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"api","namespace":"default"}, "namespace": "default", "name": "api", "rec
oncileID": "0d4960df-8e8f-4233-82ee-b3aa03491a6e"}                                                                                                                                                                                                          
2023-05-21T05:31:44Z    INFO    scaleexecutor    Successfully updated ScaleTarget    {"scaledobject.Name": "communication-attachments-worker", "scaledObject.Namespace": "default", "scaleTarget.Name": "communication-attachments-worker", "Original Replic
as Count": 0, "New Replicas Count": 1}                                                                                                                                                                                                                      
panic: runtime error: invalid memory address or nil pointer dereference                                                                                                                                                                                     
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16fb3c4]                                                                                                                                                                                     
                                                                                                                                                                                                                                                            
goroutine 8740 [running]:                                                                                                                                                                                                                                   
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestScale(0xc000115c70, {0x434bd88, 0xc0062aa5c0}, 0xc006c0da00, 0x0, 0x0)                                                                                                             
    /workspace/pkg/scaling/executor/scale_scaledobjects.go:39 +0x184                                                                                                                                                                                        
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers(0xc00082ee70, {0x434bd88, 0xc0062aa5c0}, {0x3a94540?, 0xc006c0da00?}, {0x4337c18, 0xc007aa0f98})                                                                                       
    /workspace/pkg/scaling/scale_handler.go:234 +0x427                                                                                                                                                                                                      
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop(0x0?, {0x434bd88, 0xc0062aa5c0}, 0xc002845180, {0x3a94540, 0xc006c0da00}, {0x4337c18, 0xc007aa0f98})                                                                                 
    /workspace/pkg/scaling/scale_handler.go:167 +0x351                                                                                                                                                                                                      
created by github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).HandleScalableObject                                                                                                                                                                     
    /workspace/pkg/scaling/scale_handler.go:122 +0x55d

KEDA Version
2.10.0
Kubernetes Version
1.24
Platform
Amazon Web Services
Scaler Details
CPU, Memory, Kafka, RabbitMQ
Anything else?
No response

Closing comment (2023-05-29T14:02:32Z): I guess this is related to #4413 and #4389

I think this was  is a different issue (fixed by 2.10.1 patch)
@alt-dima thanks for letting us know! Please keep us posted.