Self hosted Runners - An error occurred whilst updating runner state to "ONLINE"
Asked by Erick Dzadotz:

Hi,

I have an autoscale runner configured, but when the runner finishes running it does not descale or go ONLINE to run other pipelines or steps.

The Runner executes, it stays ONLINE in the bitbucket workspace, the pods scale to exec the parallel steps and after a while the error occurs and the pods are not descaled.

My configs:

constants:  
default_sleep_time_runner_setup: 5  
default_sleep_time_runner_delete: 5  
runner_api_polling_interval: 60  
runner_cool_down_period: 10  

strategy: "percentageRunnersIdle"  
min: 1   
max: 10   
scale_up_threshold: 0.9   
scale_down_threshold: 0.8   
scale_up_multiplier: 1.5   
scale_down_multiplier: 0.3

resources:   
requests:  
memory: "1Gi"  
cpu: "100m"  
limits:  
memory: "1Gi"  
cpu: "1000m"  

The error of pods that not works:

\[2023-11-09 16:26:16,939\] An error occurred whilst updating runner state to "ONLINE".  
com.atlassian.pipelines.stargate.client.core.exceptions.StargateUnauthorizedException: Response Summary: HttpResponseSummary{httpStatusCode=401, httpStatusMessage=Unauthorized, bodyAsString={"code":401,"message":"Unauthorized"}}  
at jdk.internal.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source)  
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:  
Error has been observed at the following site(s):  
\*__checkpoint ? 401 from PUT [https://api.atlassian.com/ex/bitbucket-pipelines/rest/internal/accounts/123456foo123456/runners/%7B02887258-36ab-5c50-9bea-XPTO/state](https://api.atlassian.com/ex/bitbucket-pipelines/rest/internal/accounts/%7B7b839271-dcd1-415a-b505-497c27e50e11%7D/runners/%7B02887258-36ab-5c50-9bea-b7b4c3cd02eb%7D/state) \[DefaultWebClient\]  
Original Stack Trace:  
at jdk.internal.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source)  
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)  
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)  
at com.atlassian.bitbucketci.client.reactive.ResponseExceptionFactory$ConstructorInvoker.invokeConstructor(ResponseExceptionFactory.java:125)  
at io.vavr.CheckedFunction1.lambda$unchecked$43b513dd$1(CheckedFunction1.java:220)  
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)  
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)  
at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101)  
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)  
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)  
at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)  
at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)  
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)  
at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)  
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)  
at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)  
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)  
at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:413)  
at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:424)  
at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:478)  
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:712)  
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)  
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)  
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)  
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)  
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)  
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)  
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383)  
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)  
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295)  
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)  
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)  
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)  
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)  
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)  
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)  
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)  
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)  
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)  
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)  
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)  
at java.base/java.lang.Thread.run(Unknown Source)


Answers:
author: Theodora Boudale
body: Hi Erick and welcome to the community!

Are you using a Runner Autoscaler for Kubernetes as per this guide?

* <https://community.atlassian.com/t5/Bitbucket-Pipelines-Runner/Bitbucket-Pipelines-Runner-Autoscaler-for-Kubernetes-Getting/ba-p/1989654>

If so, I would suggest joining the following community group and asking your question there, as this group is specifically for Runner Autoscaler questions:

* <https://community.atlassian.com/t5/Bitbucket-Pipelines-Runner/gh-p/bitbucket-runner-autoscaler-4k8s>

Kind regards,  
Theodora

comments: author: Erick Dzadotz
body: Hi Theodora,  
Thank you, I sent a request to join there. I'm using Runner Autoscaler for Kubernetes following this procedure:

* <https://bitbucket.org/bitbucketpipelines/runners-autoscaler/src/master/>

After I receive approval to join the group I can check your first link. For now I don't have sufficient privileges.

Thanks,

Erick


Tags: bitbucket-cloud

bitbucket-pipelines

bitbucket-runners

cloud