nipunsadvilkar's picture
Saving weights and logs of step 500
74af663
raw history blame
No virus
6.13 kB
2021-07-09 10:34:34,450 INFO MainThread:25100 [internal.py:wandb_internal():88] W&B internal server running at pid: 25100, started at: 2021-07-09 10:34:34.449847
2021-07-09 10:34:34,452 DEBUG HandlerThread:25100 [handler.py:handle_request():124] handle_request: check_version
2021-07-09 10:34:34,452 INFO WriterThread:25100 [datastore.py:open_for_write():80] open: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_103433-d7n9baaw/run-d7n9baaw.wandb
2021-07-09 10:34:34,453 DEBUG SenderThread:25100 [sender.py:send():179] send: header
2021-07-09 10:34:34,453 DEBUG SenderThread:25100 [sender.py:send_request():193] send_request: check_version
2021-07-09 10:34:34,515 DEBUG SenderThread:25100 [sender.py:send():179] send: run
2021-07-09 10:34:34,556 ERROR SenderThread:25100 [internal_api.py:execute():130] 403 response executing GraphQL.
2021-07-09 10:34:34,556 ERROR SenderThread:25100 [internal_api.py:execute():131] {"errors":[{"message":"permission denied","path":["upsertBucket"],"extensions":{"code":"PERMISSION_ERROR"}}],"data":{"upsertBucket":null}}
2021-07-09 10:34:35,516 INFO HandlerThread:25100 [handler.py:finish():638] shutting down handler
2021-07-09 10:34:35,517 INFO WriterThread:25100 [datastore.py:close():288] close: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_103433-d7n9baaw/run-d7n9baaw.wandb
2021-07-09 10:34:35,517 ERROR MainThread:25100 [internal.py:wandb_internal():159] Thread SenderThread:
Traceback (most recent call last):
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/sdk/lib/retry.py", line 102, in __call__
result = self._call_fn(*args, **kwargs)
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/sdk/internal/internal_api.py", line 133, in execute
six.reraise(*sys.exc_info())
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/six.py", line 719, in reraise
raise value
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/sdk/internal/internal_api.py", line 127, in execute
return self.client.execute(*args, **kwargs)
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute
result = self._get_result(document, *args, **kwargs)
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in _get_result
return self.transport.execute(document, *args, **kwargs)
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 39, in execute
request.raise_for_status()
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.wandb.ai/graphql
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper
return func(*args, **kwargs)
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/sdk/internal/internal_api.py", line 922, in upsert_run
response = self.gql(mutation, variable_values=variable_values, **kwargs)
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/sdk/lib/retry.py", line 118, in __call__
if not check_retry_fn(e):
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/util.py", line 727, in no_retry_auth
raise CommError("Permission denied, ask the project owner to grant you access")
wandb.errors.CommError: Permission denied, ask the project owner to grant you access
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/sdk/internal/internal_util.py", line 55, in run
self._run()
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/sdk/internal/internal_util.py", line 105, in _run
self._process(record)
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 292, in _process
self._sm.send(record)
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 181, in send
send_handler(record)
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 604, in send_run
self._init_run(run, config_value_dict)
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 626, in _init_run
server_run, inserted = self._api.upsert_run(
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/apis/normalize.py", line 62, in wrapper
six.reraise(CommError, CommError(message, err), sys.exc_info()[2])
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/six.py", line 718, in reraise
raise value.with_traceback(tb)
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper
return func(*args, **kwargs)
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/sdk/internal/internal_api.py", line 922, in upsert_run
response = self.gql(mutation, variable_values=variable_values, **kwargs)
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/sdk/lib/retry.py", line 118, in __call__
if not check_retry_fn(e):
File "/home/nipunsadvilkar/roberta_mr_env/lib/python3.8/site-packages/wandb/util.py", line 727, in no_retry_auth
raise CommError("Permission denied, ask the project owner to grant you access")
wandb.errors.CommError: Permission denied, ask the project owner to grant you access
2021-07-09 10:34:37,565 INFO MainThread:25100 [internal.py:handle_exit():78] Internal process exited