Self hosted runners unable to update status to ONLINE
Asked by Bailey Shepherd:

we have a number of self hosted runners on one of our local servers. That server went down recently and now when I try to restart any of the runners i get this:

Updating runner state to "ONLINE".  
\[2022-05-25 19:18:09,670\] An error occurred whilst updating runner state to "ONLINE".  
java.util.concurrent.TimeoutException: null  
at com.netflix.hystrix.AbstractCommand.handleTimeoutViaFallback(AbstractCommand.java:997)  
at com.netflix.hystrix.AbstractCommand.access$500(AbstractCommand.java:60)  
at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:609)  
at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:601)  
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)  
... 14 common frames omitted  
Wrapped by: com.netflix.hystrix.exception.HystrixRuntimeException: STARGATE_BITBUCKETCI_REST_SERVICE_PUT_REPOSITORY_RUNNER_STATE timed-out and no fallback available.  
at com.netflix.hystrix.AbstractCommand$22.call(AbstractCommand.java:822)  
at com.netflix.hystrix.AbstractCommand$22.call(AbstractCommand.java:807)  
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)  
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)  
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)  
at com.netflix.hystrix.AbstractCommand$DeprecatedOnFallbackHookApplication$1.onError(AbstractCommand.java:1472)  
at com.netflix.hystrix.AbstractCommand$FallbackHookApplication$1.onError(AbstractCommand.java:1397)  
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)  
at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:44)  
at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:28)  
at rx.Observable.unsafeSubscribe(Observable.java:10327)  
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)  
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)  
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)  
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)  
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)  
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)  
at rx.Observable.unsafeSubscribe(Observable.java:10327)  
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)  
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)  
at rx.Observable.unsafeSubscribe(Observable.java:10327)  
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)  
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)  
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)  
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)  
at rx.Observable.unsafeSubscribe(Observable.java:10327)  
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)  
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)  
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)  
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)  
at rx.Observable.unsafeSubscribe(Observable.java:10327)  
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:142)  
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)  
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)  
at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$1.run(AbstractCommand.java:1142)  
at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41)  
at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37)  
at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57)  
at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$2.tick(AbstractCommand.java:1159)  
at com.netflix.hystrix.util.HystrixTimer$1.run(HystrixTimer.java:99)  
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)  
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)  
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)  
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)  
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)  
at java.base/java.lang.Thread.run(Unknown Source)


Answers:
author: RyeGordo
body: .


author: Patrik S
body: Hello [@Bailey Shepherd](/t5/user/viewprofilepage/user-id/4945882) ,

Welcome to Atlassian Community!

The error you are facing suggests the runner is timing out while trying to connect to our infrastructure. This is usually caused by a network issue, such as a firewall blocking the connection from the runner to our backend.

In this case, could you please make sure you have whitelisted the atlassian/bitbucket IPs listed below?

* <https://support.atlassian.com/bitbucket-cloud/docs/pipelines-runners-frequently-asked-questions/#What-IP-address-do-I-need-to-whitelist-to-get-Runner-to-work>

Also, in case you are running behind a proxy, you also need to configure the runner to use it by following the steps outlined in the following documentation :

* <https://support.atlassian.com/bitbucket-cloud/docs/configure-a-runner-to-use-a-proxy/>

Hope that helps! Let me know in case you have any questions.

Thank you, [@Bailey Shepherd](/t5/user/viewprofilepage/user-id/4945882)

Kind regards,

Patrik S

comments: author: RyeGordo
body: Good Day:

I came across this issue today.

I have been using self-hosted runners for the greater part of the year and never had this issue in the past.

I recently updated to the latest version of the runners over the weekend and ran them successfully without issue.

Today, after several successful runs, my runners stopped working with the issue explained above.

This is my first time encountering this issue.

My network is not behind a firewall.


author: RyeGordo
body: Hello again:

This is a copy of my error message...it's not entirely the same

```
[2024-06-03 18:58:10,182] Runner version: 2.1.0

[2024-06-03 18:58:10,184] Runner runtime: macos-bash

[2024-06-03 18:58:10,363] Starting websocket listening to RUNNER_UPDATED events.

[2024-06-03 18:58:10,458] Updating runner status to "ONLINE" and checking for new steps assigned to the runner after 0 seconds and then every 30 seconds.

[2024-06-03 18:58:10,604] Updating runner state to "ONLINE".

[2024-06-03 18:58:30,977] An error occurred whilst updating runner state to "ONLINE".

com.atlassian.pipelines.stargate.client.core.exceptions.StargateInternalServerErrorException: Response Summary: HttpResponseSummary{httpStatusCode=500, httpStatusMessage=Internal Server Error, bodyAsString={"code":500,"message":"There was an error processing your request. It has been logged (ID 34515c8fea2fc6eb)."}}

 at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

 Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 

Error has been observed at the following site(s):

 *__checkpoint ? 500 from PUT https://api.atlassian.com/ex/bitbucket-pipelines/rest/internal/accounts/%7Be0a1e971-8422-4bf5-8003-81a5d27b02dd%7D/runners/%7B11f5f14b-d447-52fd-b9e2-b4207de80af1%7D/state [DefaultWebClient]

Original Stack Trace:

  at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

  at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)

  at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)

  at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source)

  at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)

  at com.atlassian.bitbucketci.client.reactive.ResponseExceptionFactory$ConstructorInvoker.invokeConstructor(ResponseExceptionFactory.java:125)

  at io.vavr.CheckedFunction1.lambda$unchecked$43b513dd$1(CheckedFunction1.java:220)

  at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)

  at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)

  at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101)

  at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)

  at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)

  at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)

  at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)

  at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)

  at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)

  at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)

  at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:103)

  at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)

  at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:103)

  at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)

  at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:415)

  at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:439)

  at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:493)

  at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:789)

  at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114)

  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)

  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)

  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)

  at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)

  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)

  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)

  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)

  at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)

  at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)

  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)

  at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)

  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)

  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)

  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)

  at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)

  at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1349)

  at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1389)

  at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)

  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)

  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)

  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)

  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)

  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)

  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)

  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)

  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)

  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)

  at io.netty.channel.kqueue.AbstractKQueueStreamChannel$KQueueStreamUnsafe.readReady(AbstractKQueueStreamChannel.java:544)

  at io.netty.channel.kqueue.AbstractKQueueChannel$AbstractKQueueUnsafe.readReady(AbstractKQueueChannel.java:387)

  at io.netty.channel.kqueue.KQueueEventLoop.processReady(KQueueEventLoop.java:218)

  at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:296)

  at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)

  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

  at java.base/java.lang.Thread.run(Unknown Source)

Wrapped by: reactor.retry.RetryExhaustedException: com.atlassian.pipelines.stargate.client.core.exceptions.StargateInternalServerErrorException: Response Summary: HttpResponseSummary{httpStatusCode=500, httpStatusMessage=Internal Server Error, bodyAsString={"code":500,"message":"There was an error processing your request. It has been logged (ID 34515c8fea2fc6eb)."}}

 at reactor.retry.DefaultRetry.retry(DefaultRetry.java:135)

 at reactor.retry.DefaultRetry.lambda$apply$1(DefaultRetry.java:120)

 at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:376)

 at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251)

 at reactor.core.publisher.FluxIndex$IndexSubscriber.onNext(FluxIndex.java:101)

 at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:89)

 at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)

 at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:537)

 at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:343)

 at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100)

 at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27)

 at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:190)

 at com.atlassian.bitbucketci.client.reactive.trace.TraceContextRestorationSubscriber.lambda$onError$2(TraceContextRestorationSubscriber.java:37)

 at com.atlassian.bitbucketci.client.reactive.trace.TraceVariables.run(TraceVariables.java:66)

 at com.atlassian.bitbucketci.client.reactive.trace.TraceContextRestorationSubscriber.onError(TraceContextRestorationSubscriber.java:37)

 at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192)

 at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259)

 at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onError(Operators.java:2065)

 at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:544)

 at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)

 at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)

 at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:96)

 at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278)

 at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:231)

 at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)

 at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:103)

 at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onComplete(Operators.java:2060)

 at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:89)

 at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:152)

 at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onComplete(FluxOnAssembly.java:549)

 at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)

 at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:103)

 at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)

 at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:103)

 at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)

 at reactor.core.publisher.Operators.complete(Operators.java:137)

 at reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:181)

 at reactor.netty.channel.FluxReceive.subscribe(FluxReceive.java:147)

 at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)

 at reactor.netty.ByteBufFlux.subscribe(ByteBufFlux.java:340)

 at reactor.core.publisher.Mono.subscribe(Mono.java:4491)

 at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)

 at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)

 at reactor.core.publisher.Mono.subscribe(Mono.java:4491)

 at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)

 at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192)

 at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259)

 at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:96)

 at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278)

 at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:231)

 at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)

 at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:103)

 at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onComplete(Operators.java:2060)

 at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:89)

 at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:152)

 at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onComplete(FluxOnAssembly.java:549)

 at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)

 at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:103)

 at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)

 at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:103)

 at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)

 at reactor.core.publisher.Operators.complete(Operators.java:137)

 at reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:181)

 at reactor.netty.channel.FluxReceive.subscribe(FluxReceive.java:147)

 at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)

 at reactor.netty.ByteBufFlux.subscribe(ByteBufFlux.java:340)

 at reactor.core.publisher.Mono.subscribe(Mono.java:4491)

 at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)

 at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)

 at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)

 at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)

 at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)

 at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)

 at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101)

 at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:539)

 at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)

 at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)

 at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)

 at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)

 at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)

 at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)

 at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)

 at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:103)

 at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)

 at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:103)

 at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)

 at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:415)

 at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:439)

 at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:493)

 at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:789)

 at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)

 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)

 at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)

 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)

 at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)

 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)

 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)

 at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)

 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)

 at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)

 at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1349)

 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1389)

 at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)

 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)

 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)

 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)

 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)

 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)

 at io.netty.channel.kqueue.AbstractKQueueStreamChannel$KQueueStreamUnsafe.readReady(AbstractKQueueStreamChannel.java:544)

 at io.netty.channel.kqueue.AbstractKQueueChannel$AbstractKQueueUnsafe.readReady(AbstractKQueueChannel.java:387)

 at io.netty.channel.kqueue.KQueueEventLoop.processReady(KQueueEventLoop.java:218)

 at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:296)

 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)

 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

 at java.base/java.lang.Thread.run(Unknown Source)
```


author: Patrik S
body: Hello [@RyeGordo](/t5/user/viewprofilepage/user-id/4835059) ,

It seems that the time you reported this issue also aligns with an incident that was affecting multiple Atlassian products (including Bitbucket Cloud) that happened yesterday :

* <https://bitbucket.status.atlassian.com/incidents/l1z18sw6ddbh>

I suspect the error you were receiving is related to that incident. Our engineering team already applied a fix for that, so could you confirm if you are still seeing that error in your self-hosted runners?

Thank you, [@RyeGordo](/t5/user/viewprofilepage/user-id/4835059) !

Patrik S


author: RyeGordo
body: Hi, I am no longer seeing the error.

My runners started working again yesterday evening. No problems since.

Thanks,

Ryan


author: Patrik S
body: Thank you for the update, [@RyeGordo](/t5/user/viewprofilepage/user-id/4835059). That confirms the issue was related to yesterday's incident, which has already been fixed.

If you ever need help in the future, feel free to reach out to the community :)


Tags: bitbucket-cloud