Repository: garden-io/garden

Issue #5217: Crash: toKubernetesError encountered an unknown error unexpectedly during Kub...
Created: 2023-10-09T15:40:25Z
Crash report

Error message
Failed processing Build type=container name=asgard (took 431.41 sec). This is what happened:
Error: getaddrinfo ENOTFOUND aks-[hash].[uuid].privatelink.westus2.azmk8s.io
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:84:26)
    at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17)



What did you do?

garden deploy --sync=monolith
I think I had previously run garden cleanup ns, because k9s didn't show anything in my namespace.
Your environment


OS: macOS Sonoma (14.0) 
How I'm running Kubernetes: Azure AKS 
Garden version: 0.13.16 

Frequency

I've only seen this happen once.
Workaround

DNS resolution worked on redeploy.
Additional context

I doubt there's anything you can do about DNS issues, but I think it would be good to have an error handler for this issue.

Duplicate issue #5286: Crash: getaddrinfo ENOTFOUND 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
Created: 2023-10-20T22:36:10Z
Crash report

Error message


Payload:
  {"anonymousId":"d85a6ca6-c782-4f6e-938f-20ffab2101c9","event":"Command Result","properties":{"projectId":"67663e0418ebc48269522093da45c634c181ceff68c51f0362e982d69de2a5d9d41fb8067d4ef8f4707d4dd9168dd394e9b57a3de5adf2b00ebdd117e47db13f","projectIdV2":"unsuitable-belligerent-brother_67663e0418ebc48269522093da45c634","projectName":"998ea67331e2c435208a6fe3b8f73565ab0574993e65961cf391a343c163ecae088a0c746a95207189e4fd434771ab9e91dc9c647f11dce3b25dd18c4d041505","projectNameV2":"ruddy-curved-fortune_998ea67331e2c435208a6fe3b8f73565","enterpriseDomain":"2769c2abae62151b2ebb8658628f7c5f5d0dc0c29fdefdd19a23dec9cb0a7b96d74d82512d1f6906bef65b24a29d84685dfd2fd66964a56fbdaff39fabd69206","enterpriseDomainV2":"ratty-willing-science_2769c2abae62151b2ebb8658628f7c5f","isLoggedIn":false,"ciName":null,"system":{"platform":"darwin","platformVersion":"22.6.0","gardenVersion":"0.13.18"},"isCI":false,"sessionId":"3de417b0-8583-47da-9f7d-21c2aa781dc5","parentSessionId":"3de417b0-8583-47da-9f7d-21c2aa781dc5","projectMetadata":{"modulesCount":42,"moduleTypes":["container"],"tasksCount":46,"servicesCount":49,"testsCount":36,"actionsCount":0,"buildActionCount":0,"testActionCount":0,"deployActionCount":0,"runActionCount":0},"firstRunAt":"2023-09-30T00:10:04.238Z","latestRunAt":"2023-10-20T22:23:00.459Z","isRecurringUser":true,"errors":["runtime"],"lastError":{"error":{"errorType":"runtime","stackTrace":{"functionName":"handleProcessResults","relativeFileName":"commands/base.js","lineNumber":730}},"wrapped":{"errorType":"graph","taskType":"deploy","stackTrace":{"functionName":"GetAddrInfoReqWrap.onlookup [as oncomplete]","lineNumber":84}},"leaf":{"errorType":"crash","stackTrace":{"functionName":"GetAddrInfoReqWrap.onlookup [as oncomplete]","lineNumber":84}}},"result":"failure","name":"deploy","durationMsec":16347}}
1 deploy action(s) failed!
[silly] Error: 1 deploy action(s) failed!
    at handleProcessResults (/snapshot/pkg/core/build/src/commands/base.js:730:23)
    at DeployCommand.action (/snapshot/pkg/core/build/src/commands/deploy.js:291:48)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /snapshot/pkg/core/build/src/commands/base.js:214:30
    at async /snapshot/pkg/core/build/src/util/open-telemetry/spans.js:46:20
    at async /snapshot/pkg/core/build/src/cli/cli.js:258:26
    at async /snapshot/pkg/core/build/src/cli/cli.js:433:33
    at async /snapshot/pkg/core/build/src/util/open-telemetry/spans.js:46:20
    at async GardenCli.run (/snapshot/pkg/core/build/src/cli/cli.js:431:32)
    at async /snapshot/pkg/cli/build/src/cli.js:100:29
    at async /snapshot/pkg/core/build/src/util/open-telemetry/spans.js:46:20
    at async Object.runCli (/snapshot/pkg/cli/build/src/cli.js:95:18)

Error type: runtime

Wrapped errors:
⮑ Error: getaddrinfo ENOTFOUND 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
       at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:84:26)
       at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17)

   Error type: graph

   Wrapped errors:
   ⮑ Error: getaddrinfo ENOTFOUND 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
          at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:84:26)
          at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17)

      Error type: graph

      Wrapped errors:
      ⮑ Error: getaddrinfo ENOTFOUND 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
             at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:84:26)
             at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17)

         Error type: graph

         Wrapped errors:
         ⮑ Error: getaddrinfo ENOTFOUND 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
                at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:84:26)
                at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17)

            Error type: graph

            Wrapped errors:
            ⮑ Error: getaddrinfo ENOTFOUND 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
                   at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:84:26)
                   at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17)

               Error type: crash

What did you do?

I'm getting this consistent error when trying to deploy using the latest versions of garden CLI (0.13.18).  This issue does not appear when I use an older version (0.13.13).
Your environment


OS: mac OS Ventura 13.6 
How I'm running Kubernetes: EKS 
Garden version: 0.13.18 

Frequency

This crash appears to be happening every time I try to deploy.
Workaround

Downrev garden version to 0.13.13.
Additional context

Other developers on my team are also experiencing this issue.  I tried the following command to see if there was some kind of networking issue, the output seems legit.
dig 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com

; <<>> DiG 9.10.6 <<>> 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1743
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com. IN A

;; ANSWER SECTION:
06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com. 31 IN	A 54.241.117.126
06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com. 31 IN	A 54.177.204.177

;; Query time: 23 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Oct 20 15:22:13 PDT 2023
;; MSG SIZE  rcvd: 125

nslookup 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
Name:	06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
Address: 54.241.117.126
Name:	06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
Address: 54.177.204.177

Closing comment (2023-10-31T17:34:36Z): Thank you for investigating this @jghowe 👍
You are right @soren121, this is a duplicate of #5217
We now have a pull request with a possible fix in #5326
I'm closing this in favor of #5217