Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue #816: user pending stop, start server hangs indefinitely
Created: 2018-08-03T00:54:32Z
Hi everyone, I'm having trouble with the Jupyterhub I'm using for an upcoming python class.  It seems to work for a while, then I come back to it 12 hours later and it hangs on the "your server is starting" screen.  I'm not sure what steps I can take to diagnose the problem.  Stopping the hub and letting it restart seems to fix it for a while.
It's running on GKE.  Here are my helm chart, and the recent log from the hub pod.  I'm not sure what else I should look at.  Any tips you have are very much appreciated!
Contents of config.yaml
hub:
  extraConfig: |
    c.JupyterHub.admin_access = True

proxy:
  secretToken: 
singleuser:
  image:
    name: "berkeleydsep/singleuser-data100"
    tag: aab20eb
  memory:
    guarantee: 1G
    limit: 1G
  storage:
    capacity: 10Gi

auth:
  type: github
  github:
    clientId: 
    clientSecret: 
    callbackUrl: "http://35.233.190.214/hub/oauth_callback"
    org_whitelist:
      - "MIDS-INFO-W18"
  whitelist:
    users:
      - 
  admin:
    users:
      - 

recent output of kubectl logs -n jhub hub-c547d8655-r2k9z:

[I 2018-08-02 20:42:28.981 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 5.47ms
[I 180802 20:52:28 cull_idle_servers:68] Culling server for paul-laskowski (inactive since 2018-08-02 19:52:06.310000)
[I 2018-08-02 20:52:28.983 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 7.19ms
[I 2018-08-02 20:52:28.987 JupyterHub proxy:254] Removing user paul-laskowski from proxy (/user/paul-laskowski/)
[W 2018-08-02 20:52:38.989 JupyterHub base:556] User paul-laskowski: server is slow to stop
[I 2018-08-02 20:52:38.992 JupyterHub log:122] 202 DELETE /hub/api/users/paul-laskowski/server (cull-idle@127.0.0.1) 10006.43ms
[I 2018-08-02 21:02:28.982 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 6.34ms
[I 2018-08-02 21:12:28.989 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 5.87ms
[I 2018-08-02 21:22:28.981 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 5.34ms
[I 2018-08-02 21:32:28.981 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 5.84ms
[I 2018-08-02 21:36:35.805 JupyterHub log:122] 302 GET /user/paul-laskowski/api/kernels/bee2f167-5d5b-4925-a4c3-c036514e533a/channels?session_id=d930e59651b740a4b5276d36de32d261 → /hub/user/paul-laskowski/api/kernels/bee2f167-5d5b-4925-a4c3-c036514e533a/channels?session_id=d930e59651b740a4b5276d36de32d261 (@10.24.2.1) 0.64ms
[I 2018-08-02 21:36:38.555 JupyterHub log:122] 302 GET /user/paul-laskowski/api/kernels/bee2f167-5d5b-4925-a4c3-c036514e533a/channels?session_id=d930e59651b740a4b5276d36de32d261 → /hub/user/paul-laskowski/api/kernels/bee2f167-5d5b-4925-a4c3-c036514e533a/channels?session_id=d930e59651b740a4b5276d36de32d261 (@10.24.2.1) 0.61ms
[I 2018-08-02 21:36:43.576 JupyterHub log:122] 302 GET /user/paul-laskowski/api/kernels/bee2f167-5d5b-4925-a4c3-c036514e533a/channels?session_id=d930e59651b740a4b5276d36de32d261 → /hub/user/paul-laskowski/api/kernels/bee2f167-5d5b-4925-a4c3-c036514e533a/channels?session_id=d930e59651b740a4b5276d36de32d261 (@10.24.2.1) 0.68ms
[I 2018-08-02 21:36:54.348 JupyterHub log:122] 302 GET /user/paul-laskowski/api/kernels/bee2f167-5d5b-4925-a4c3-c036514e533a/channels?session_id=d930e59651b740a4b5276d36de32d261 → /hub/user/paul-laskowski/api/kernels/bee2f167-5d5b-4925-a4c3-c036514e533a/channels?session_id=d930e59651b740a4b5276d36de32d261 (@10.24.2.1) 0.62ms
[I 2018-08-02 21:37:12.120 JupyterHub log:122] 302 GET /user/paul-laskowski/api/kernels/bee2f167-5d5b-4925-a4c3-c036514e533a/channels?session_id=d930e59651b740a4b5276d36de32d261 → /hub/user/paul-laskowski/api/kernels/bee2f167-5d5b-4925-a4c3-c036514e533a/channels?session_id=d930e59651b740a4b5276d36de32d261 (@10.24.2.1) 0.94ms
[I 2018-08-02 21:37:44.477 JupyterHub log:122] 302 GET /user/paul-laskowski/api/kernels/bee2f167-5d5b-4925-a4c3-c036514e533a/channels?session_id=d930e59651b740a4b5276d36de32d261 → /hub/user/paul-laskowski/api/kernels/bee2f167-5d5b-4925-a4c3-c036514e533a/channels?session_id=d930e59651b740a4b5276d36de32d261 (@10.24.2.1) 0.68ms
[I 2018-08-02 21:38:49.399 JupyterHub log:122] 302 GET /user/paul-laskowski/api/kernels/bee2f167-5d5b-4925-a4c3-c036514e533a/channels?session_id=d930e59651b740a4b5276d36de32d261 → /hub/user/paul-laskowski/api/kernels/bee2f167-5d5b-4925-a4c3-c036514e533a/channels?session_id=d930e59651b740a4b5276d36de32d261 (@10.24.2.1) 0.80ms
[I 2018-08-02 21:42:28.981 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 5.58ms
[I 2018-08-02 21:52:28.984 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 5.50ms
[I 2018-08-02 21:53:02.023 JupyterHub log:122] 302 GET / → /hub (@10.24.2.1) 0.81ms
[I 2018-08-02 21:53:16.090 JupyterHub log:122] 302 GET / → /hub (@10.24.2.1) 0.73ms
[I 2018-08-02 22:02:28.991 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 6.66ms
[I 2018-08-02 22:12:28.985 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 6.43ms
[I 2018-08-02 22:22:28.990 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 6.00ms
[I 2018-08-02 22:32:28.988 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 5.25ms
[I 2018-08-02 22:42:28.990 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 6.06ms
[I 2018-08-02 22:52:28.967 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 6.63ms
[I 2018-08-02 23:02:28.989 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 5.92ms
[I 2018-08-02 23:12:28.988 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 6.86ms
[I 2018-08-02 23:22:28.982 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 6.01ms
[I 2018-08-02 23:24:57.861 JupyterHub log:122] 302 GET / → /hub (@10.24.2.1) 0.63ms
[I 2018-08-02 23:24:58.389 JupyterHub log:122] 302 GET /hub → /hub/login (@10.24.2.1) 0.62ms
[I 2018-08-02 23:32:28.988 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 6.18ms
[I 2018-08-02 23:42:28.989 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 6.31ms
[I 2018-08-02 23:52:28.982 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 6.41ms
[I 2018-08-03 00:02:28.985 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 5.02ms
[I 2018-08-03 00:12:28.989 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 6.34ms
[I 2018-08-03 00:21:14.069 JupyterHub log:122] 302 GET / → /hub (@10.24.2.1) 0.80ms
[I 2018-08-03 00:22:28.981 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 5.99ms
[I 2018-08-03 00:32:01.715 JupyterHub log:122] 302 GET / → /hub (@10.24.2.1) 0.65ms
[I 2018-08-03 00:32:01.759 JupyterHub log:122] 302 GET /hub → /hub/login (@10.24.2.1) 0.85ms
[I 2018-08-03 00:32:01.814 JupyterHub log:122] 302 GET /hub/login → /hub/ (paul-laskowski@10.24.2.1) 5.80ms
[I 2018-08-03 00:32:01.882 JupyterHub log:122] 302 GET /hub/ → /hub/home (paul-laskowski@10.24.2.1) 2.45ms
[I 2018-08-03 00:32:01.930 JupyterHub log:122] 200 GET /hub/home (paul-laskowski@10.24.2.1) 5.01ms
[I 2018-08-03 00:32:05.950 JupyterHub log:122] 200 GET /hub/admin (paul-laskowski@10.24.2.1) 6.52ms
[I 2018-08-03 00:32:10.507 JupyterHub log:122] 302 GET /hub/spawn → /user/paul-laskowski/ (paul-laskowski@10.24.2.1) 3.58ms
[I 2018-08-03 00:32:10.552 JupyterHub log:122] 302 GET /user/paul-laskowski/ → /hub/user/paul-laskowski/ (@10.24.2.1) 0.70ms
[I 2018-08-03 00:32:10.603 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:32:10.609 JupyterHub log:122] 200 GET /hub/user/paul-laskowski/ (paul-laskowski@10.24.2.1) 6.60ms
[I 2018-08-03 00:32:15.700 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:32:20.797 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:32:25.888 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:32:28.981 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 5.87ms
[I 2018-08-03 00:32:30.978 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:32:36.062 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:32:41.152 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:32:46.244 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:32:51.334 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:32:56.435 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:33:01.522 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:33:06.607 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:33:11.697 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:33:16.785 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:33:21.874 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:33:26.962 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:33:32.052 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:33:37.139 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:33:40.540 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:33:45.727 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:33:50.818 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:33:55.907 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:34:00.997 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:34:06.083 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:34:11.190 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:34:14.852 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:34:19.946 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:34:25.038 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:34:30.147 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:34:35.150 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:34:40.236 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:34:45.324 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:34:50.417 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:34:55.502 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:35:00.588 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:35:05.678 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:35:10.769 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:35:15.855 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:35:20.961 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:35:26.050 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:35:31.312 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:35:36.398 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:35:41.486 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:35:46.758 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:35:51.863 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:35:56.952 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:36:02.048 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:36:07.140 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:36:08.362 JupyterHub base:727] paul-laskowski is pending stop
[W 2018-08-03 00:36:46.426 JupyterHub web:1588] 400 POST /hub/api/users/paul-laskowski/server (10.24.2.1): paul-laskowski is pending stop
[W 2018-08-03 00:36:46.429 JupyterHub log:122] 400 POST /hub/api/users/paul-laskowski/server (paul-laskowski@10.24.2.1) 6.43ms
[I 2018-08-03 00:36:55.762 JupyterHub log:122] 302 GET /hub/spawn → /user/paul-laskowski/ (paul-laskowski@10.24.2.1) 3.29ms
[I 2018-08-03 00:36:55.811 JupyterHub log:122] 302 GET /user/paul-laskowski/ → /hub/user/paul-laskowski/ (@10.24.2.1) 0.72ms
[I 2018-08-03 00:36:55.861 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:37:00.952 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:37:06.056 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:37:11.144 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:37:16.234 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:37:21.321 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:37:26.411 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:37:31.499 JupyterHub base:727] paul-laskowski is pending stop
[W 2018-08-03 00:37:37.457 JupyterHub web:1588] 400 POST /hub/api/users/paul-laskowski/server (10.24.2.1): paul-laskowski is pending stop
[W 2018-08-03 00:37:37.458 JupyterHub log:122] 400 POST /hub/api/users/paul-laskowski/server (paul-laskowski@10.24.2.1) 4.59ms
[I 2018-08-03 00:42:28.981 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 5.37ms
[I 2018-08-03 00:48:41.662 JupyterHub log:122] 302 GET /hub/spawn → /user/paul-laskowski/ (paul-laskowski@10.24.2.1) 3.39ms
[I 2018-08-03 00:48:41.710 JupyterHub log:122] 302 GET /user/paul-laskowski/ → /hub/user/paul-laskowski/ (@10.24.2.1) 0.62ms
[I 2018-08-03 00:48:41.756 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:48:46.984 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:48:54.296 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:49:01.327 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:49:08.216 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:49:15.357 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:49:22.028 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:49:29.171 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:49:36.308 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:49:43.305 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:49:50.404 JupyterHub base:727] paul-laskowski is pending stop
[I 2018-08-03 00:49:56.986 JupyterHub base:727] paul-laskowski is pending stop

Duplicate issue #1022: Sometimes failing to shut down pod
Created: 2018-11-14T14:17:34Z
The hub fails to detect pods that have successfully shut down, sometimes. This happens about 25-33% of the time.
Relevant information
Kubectl does not find the pod any more.
The logs of the proxy sais:
14:02:57.164 - info: [ConfigProxy] Removing route /user/jaime.ferrando_huertas
14:02:57.164 - info: [ConfigProxy] 204 DELETE /api/routes/user/jaime.ferrando_huertas 
14:03:33.473 - info: [ConfigProxy] 200 GET /api/routes

The logs of the hub sais:
# ...
[I 2018-11-14 14:06:15.751 JupyterHub base:1018] jaime is pending stop
[I 2018-11-14 14:06:20.318 JupyterHub base:1018] jaime is pending stop
[I 2018-11-14 14:06:20.833 JupyterHub base:1018] jaime is pending stop
[I 2018-11-14 14:06:25.454 JupyterHub base:1018] jaime is pending stop
[I 2018-11-14 14:06:25.913 JupyterHub base:1018] jaime is pending stop
# ...

Questions raised

What makes the kubespawner realize that the pending stop has succeeded?
What could have caused this event to not take place?

/cc: @jiwidi

Closing comment (2019-05-20T14:54:40Z): @jiwidi
Cool idea, I've tried to click once, but it hasn't help.
Related to: #816